add: fixup docs

Author: gregfurman

frontend/docs/pages/cookbooks/workflow-aws-s3.mdx

@@ -7,16 +7,29 @@ import FanoutDiagram from "@/components/FanoutDiagram";
 
 # Processing Amazon S3 Objects at Scale
 
-Amazon S3 provides a robust, predictable way to store and retrieve large blobs of data. It has several properties that make it an ideal backend for processing massive datasets at scale. Namely, it provides strong [read-after-write consistency](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#ConsistencyModel) for `PUT` and `DELETE` requests. This guarantees that when we retrieve or modify data within a bucket, we are always working with the most up-to-date state.
+Amazon S3 is a reliable, boring distributed object store with strong [read-after-write consistency](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#ConsistencyModel) guarantees, which makes it an ideal backend for processing massive datasets at scale. Coordinating workloads that actually leverage that scale, while not running up a comically large cloud bill, is non-trivial.
 
-Moreover, S3's data consistency model pairs nicely with Hatchet's [concurrency control](/v1/concurrency) features and dynamic spawning of [child tasks in parallel](/v1/child-spawning#fan-out-spawning-many-children-in-parallel), allowing us to aggressively poll and process S3 objects while sidestepping data races.
+This guide walks through building an _embarrassingly parallel_ Hatchet pipeline for ingesting and processing hundreds of thousands of images stored across Amazon S3. We'll be making extensive use of Hatchet's [concurrency control](/v1/concurrency) features, paired with aggressive fan-outs via [dynamic child spawning](/v1/child-spawning#fan-out-spawning-many-children-in-parallel), to parallelize the heck out of our data processing workflow while ensuring fair allocation of resources and idempotent processing of objects.
 
-This example illustrates how to build an _embarrassingly parallel_ Hatchet pipeline for ingesting and processing hundreds of thousands of images stored across Amazon S3. The constraints for the system are as follows:
+<figure style={{ margin: "2rem auto", maxWidth: "400px", textAlign: "center" }}>
+  <img
+    src="/s3-workflow/spider-man-workers.png"
+    alt="Three workers all pointing at each other, each claiming to have processed the same S3 object"
+    style={{ width: "100%", height: "auto", borderRadius: "8px" }}
+  />
+  <figcaption style={{ marginTop: "0.75rem", fontSize: "0.875rem", fontStyle: "italic", opacity: 0.7 }}>
+    Without concurrency control, nothing stops multiple workers all grabbing, and processing, the same object.
+  </figcaption>
+</figure>
+
+### Scenario
+
+For the purposes of this guide, we'll constrain the system as follows:
 
 - Each bucket corresponds to a separate class of objects.
 - Bucket sizes are unevenly distributed.
-- Objects are ephemeral. Once successfully processed, the object should be deleted.
-- More data can be added or removed dynamically at any point.
+- Objects are ephemeral. Once successfully processed, an object should be deleted.
+- Data can be added or removed at any point, with a sizable chunk being front-loaded at startup.
 
 ## Setup
 
@@ -44,6 +57,10 @@ _Note: `<your-bucket-prefix>` should be scoped at runtime to the specific bucket
   </Tabs.Tab>
 </UniversalTabs>
 
+<Callout type="info">
+  For the purposes of this example, we recommend using `localstack/localstack:4.14`.
+</Callout>
+
 </Steps>
 
 ### Define the Models
@@ -63,16 +80,14 @@ Notice that we only have models defined for our input data, as the task naturall
 
 ## Workflows
 
-To ensure maximum parallelism, we treat the processing of each object within each bucket as an independent unit of work.
-
-The execution sequence is:
+To ensure maximum parallelism, we treat the processing of each object within each bucket as an independent unit of work. Concretely, the execution sequence is:
 
 1. Fetch buckets from S3.
 2. Fetch objects within each bucket.
 3. Process each object.
 4. Delete each object.
 
-We use a double fan-out pattern here to ensure that we can distribute the list operations and the processing operations across a horizontally scaled fleet of workers without hitting bottlenecks.
+We implement this as a _double fan-out_: one fan-out across buckets to spawn per-bucket polling tasks, and a second fan-out within each polling task to spawn one processing task per object. This lets us distribute both the listing and processing work across a horizontally scaled fleet of workers without any single worker becoming a bottleneck.
 
 <FanoutDiagram
   parentLabel="S3::ListBuckets"
@@ -97,9 +112,9 @@ We use a double fan-out pattern here to ensure that we can distribute the list o
 
 We start by fetching all relevant buckets from S3. For simplicity, this pipeline forgoes [S3 Event Notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html) in favor of periodic scanning via a [cron trigger](/v1/cron-runs).
 
-As the root of the pipeline, this task iterates through buckets serially and fans out a child workflow per bucket. Since it produces input for downstream parallelism rather than executing parallel work itself, concurrent execution offers no benefit at this stage.
+As the root of the pipeline, this task paginates across all buckets in our region, fanning out a child workflow per bucket. Since we're just iterating across mini-batches/pages of buckets, there's no meaningful work here to parallelize.
 
-We must guard against overlapping executions. If the task is throttled by the S3 API and the cron fires again before it completes, newer runs should be rejected until the in-flight run finishes.
+However, our strategy should still guard against _overlapping executions_ of the same cron job. If S3 throttles us mid-scan and a run takes longer than the cron interval, the next tick should be rejected until the in-flight run finishes, rather than piling on more concurrent listing calls and making the throttling worse.
 
 <FanoutDiagram
   parentLabel="Fetch Buckets"
@@ -119,9 +134,9 @@ Taking all of this into account, we define our `fetch_s3_buckets` workflow as fo
   </Tabs.Tab>
 </UniversalTabs>
 
-Here, we set the concurrency key to the CEL expression `'singleton'`, a string literal that evaluates to the same value for every run. Combined with `max_runs=1` and `CANCEL_NEWEST`, this rule funnels all runs through **a single slot on a single worker**. While one is in-flight, any new run triggered by the cron is immediately canceled.
+Here, we set the concurrency key to the CEL expression `'singleton'`, a string literal that evaluates to the same value for every run. Combined with `max_runs=1` and `CANCEL_NEWEST`, this rule funnels all runs through **a single slot on a single worker**: while one is in-flight, any new run triggered by the cron is immediately canceled.
 
-Mapping to our implementation, the `fetch_s3_buckets` workflow is responsible for gathering all relevant buckets from our region and spawning a new `fetch_objects` task from the `fetch_s3_objects` workflow for each.
+The task body itself is responsible for gathering all relevant buckets from our region and spawning a new `fetch_objects` task from the `fetch_s3_objects` workflow for each:
 
 <UniversalTabs items={["Typescript", "Python"]}>
   <Tabs.Tab title="Typescript">
@@ -132,7 +147,7 @@ Mapping to our implementation, the `fetch_s3_buckets` workflow is responsible fo
   </Tabs.Tab>
 </UniversalTabs>
 
-For a given trigger, we use the bucket name as a `child_key` to ensure deduplication.
+We use the bucket name as its child key for each spawn to ensure deduplication of downstream `fetch_objects` runs.
 
 ### Fetch Objects
 
@@ -149,7 +164,7 @@ Next, we define a workflow responsible for polling a given S3 bucket for objects
   showCollect={false}
 />
 
-Similarly to [Fetch Buckets](#fetch-buckets-workflow), this workflow needs to ensure each bucket has at most one poller in-flight at a time, preventing overlaps via the `CANCEL_NEWEST` strategy and `max_runs=1`. Here, however, we group by bucket name using the CEL expression `input.bucket` to enforce per-bucket concurrency.
+Similarly to [Fetch Buckets](#fetch-buckets-workflow), this workflow needs to ensure each bucket has at most one poller in-flight at a time, preventing overlaps via the `CANCEL_NEWEST` strategy and constraining to a single run. Here, however, we group by bucket name using the CEL expression `input.bucket` to enforce per-bucket concurrency.
 
 Additionally, we define a global concurrency expression that limits how many bucket pollers can spin up simultaneously. Tasks exceeding this rule are enqueued using the `GROUP_ROUND_ROBIN` strategy and will execute when concurrent slots free up.
 
@@ -175,20 +190,24 @@ The task itself paginates through the bucket's objects and spawns a processing r
   </Tabs.Tab>
 </UniversalTabs>
 
-The `child_key` here combines the bucket name and object key `({bucket}/{key})`, guaranteeing that re-triggering the parent won't spawn duplicate processing runs for the same object. S3 object keys are only unique within a specific bucket, so the bucket prefix is mandatory here.
+The child key here combines the bucket name and object key `({bucket}/{key})`, guaranteeing that re-triggering the parent won't spawn duplicate processing runs for the same object. S3 object keys are only unique within a specific bucket, so the bucket prefix is mandatory here.
 
 ### Process Objects
 
-<AmazonS3ImagePipelineDiagram />
 
-Finally, we define the processing workflow, which represents the bulk of the pipeline's actual computational work. Recall from the previous sections that:
+Lastly, we define the processing workflow, which represents the bulk of the pipeline's actual computational work. Recall from the previous sections that:
 
 1. The distribution of objects per bucket is not uniform.
 2. Processed objects are immediately deleted.
 
+
 Rather than capping concurrency globally, we scope the limit _per bucket_ using `S3_WORKER_MAX_RUNS_PER_BUCKET`. A global cap would allow the largest buckets to monopolize all processing slots. A per-bucket cap keeps throughput fair across the system regardless of individual bucket size. When a bucket hits its limit, additional runs are enqueued and dispatched round-robin via `GROUP_ROUND_ROBIN` as slots open.
 
-There is no child-side deduplication on this workflow (no concurrency expression keyed on `{bucket}/{object-key}`). Because `process-object` runs are only ever spawned by [`fetch_objects`](#fetch-objects-workflow)—which already handles deduplication via `child_key` at spawn time—a secondary concurrency expression would be redundant.
+<AmazonS3ImagePipelineDiagram/>
+
+The diagram above shows this steady state: work interleaves across buckets, the worker pool runs at partial capacity, and a backlog drains in round-robin order. Even when `bucket-1` has the most pending objects, it doesn't monopolize the workers.
+
+There is no child-side deduplication on this workflow (no concurrency expression keyed on `{bucket}/{object-key}`). Because `process-object` runs are only ever spawned by [`fetch_objects`](#fetch-objects-workflow) (which already handles deduplication via child key at spawn time) a secondary concurrency expression would be redundant.
 
 <UniversalTabs items={["Typescript", "Python"]}>
   <Tabs.Tab title="Typescript">
@@ -223,13 +242,13 @@ This architecture provides an idempotent, effectively-exactly-once processing pi
 
 ## Test it
 
-The test suite manages LocalStack via [Docker Compose](https://docs.docker.com/compose/). The start and teardown are fully automatic.
-
-Each test seeds 3 buckets with 3 synthetic PNG objects each, triggers `fetch_s3_buckets` manually, and polls S3 until every object has been deleted by the pipeline, validating the full fan-out chain end-to-end.
-
 <UniversalTabs items={["Python", "Typescript"]}>
   <Tabs.Tab title="Python">
 
+The test suite manages LocalStack via [Docker Compose](https://docs.docker.com/compose/). Startup and teardown are fully automatic.
+
+The suite seeds 3 buckets with 3 synthetic objects each, triggers `fetch_s3_buckets` manually, then polls S3 until every object has been deleted by the pipeline — validating the full fan-out chain end-to-end.
+
 Make sure `boto3` is installed, then run from `sdks/python/`:
 
 ```bash
@@ -239,11 +258,20 @@ pytest examples/aws/s3/test_worker.py
   </Tabs.Tab>
   <Tabs.Tab title="Typescript">
 
-Install the AWS SDK client, then run the e2e suite filtered to the S3 example from `sdks/typescript/`:
+Start LocalStack with Docker Compose from `sdks/python/examples/aws/s3/`:
+
+```bash
+docker compose up -d --wait
+```
+
+Then start the worker from `sdks/typescript/`:
 
 ```bash
-npm install @aws-sdk/client-s3
-pnpm run test:e2e -- --testPathPattern=aws/s3
+S3_WORKER_BUCKET_PREFIX=bucket- \
+S3_WORKER_MAX_CONCURRENT_BUCKET_POLLERS=10 \
+S3_WORKER_MAX_RUNS_PER_BUCKET=20 \
+S3_WORKER_SLOTS=40 \
+npx ts-node src/v1/examples/aws/s3/worker.ts
 ```
 
   </Tabs.Tab>