docs: AWS S3 processing example#3840
Conversation
|
@gregfurman is attempting to deploy a commit to the Hatchet Team on Vercel. A member of the Team first needs to authorize it. |
gregfurman
force-pushed
the
add/cookbook/s3-processing
branch
2 times, most recently
from
1bf2cc9 to
c77f36a
Compare
gregfurman
force-pushed
the
add/cookbook/s3-processing
branch
from
b6de800 to
040061c
Compare
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
gregfurman
force-pushed
the
add/cookbook/s3-processing
branch
from
2e7c78a to
b55804d
Compare
abelanger5
left a comment
abelanger5
left a comment
There was a problem hiding this comment.
looks great!! left some feedback 👍
| import AmazonS3ImagePipelineDiagram from "@/components/AmazonS3ImagePipelineDiagram"; | ||
| import FanoutDiagram from "@/components/FanoutDiagram"; | ||
|
|
||
| # Processing Amazon S3 Objects at Scale with Hatchet |
There was a problem hiding this comment.
A high level comment: I wonder if we want to position this as "image processing at scale" and then mention at the top of this cookbook what this pattern generalizes to: document processing, image processing, video encoding, etc. Would be helpful from both a search perspective, anchoring the content and making people more likely to click on this cookbook. Lmk what you think!
There was a problem hiding this comment.
Fair point. I asssumed since the approach used was so generalisable that I'd keep this as "objects" instead of explicitly mentioning file types -- especially since the approach is agnostic on what classifies "processing" once an s3 object is downloaded.
My thoughts were it's a bit misleading since there really isnt really any image-specifc implementation details here -- hence why I generalised. Maybe we could make mention in the first paragraph how this could be applied to videos, images, etc. to appease the SEO/search gods?
Also, do we want to retain the mention of S3? Or should we try and generalize this to "object storage"?
gregfurman
force-pushed
the
add/cookbook/s3-processing
branch
from
dbc7c3b to
677ae8f
Compare
| import UniversalTabs from "@/components/UniversalTabs"; | ||
| import { snippets } from "@/lib/generated/snippets"; | ||
| import { Snippet } from "@/components/code"; | ||
| import AmazonS3ImagePipelineDiagram from "@/components/AmazonS3ImagePipelineDiagram"; |
There was a problem hiding this comment.
I noticed that we added a wrapper version of this component that avoids SSR. Should we import the wrapper here?
|
|
||
| export default MultiBucketSlotPoolDiagram; | ||
|
|
||
| // import React, { useState, useEffect } from "react"; |
There was a problem hiding this comment.
Can we remove this dead code?
|
Nice read @gregfurman, I love the humor! |
github-actions
Bot
added
documentation
|
|
||
| This guide walks through building an _embarrassingly parallel_ Hatchet pipeline for ingesting and processing hundreds of thousands of images stored across multiple regionally bound S3 buckets. This approach will be making extensive use of Hatchet's [concurrency control](/v1/concurrency) features, paired with aggressive fan-outs via [dynamic child spawning](/v1/child-spawning#fan-out-spawning-many-children-in-parallel), to parallelize the heck out of a data processing workflow while ensuring fair allocation of resources and idempotent processing of objects. | ||
|
|
||
| You can customize your documentation experience by selecting your stack for code examples, including your preferred language (Python, TypeScript, Go, or Ruby) and package manager (`pip`, `poetry`, or `uv`). Click the <svg aria-label="Python" role="img" viewBox="0 0 128 128" style={{ width: "1.1em", height: "1.1em", display: "inline-block", verticalAlign: "-0.2em", color: "currentColor" }}><path fill="currentColor" d="M49.33 62h29.159C86.606 62 93 55.132 93 46.981V19.183c0-7.912-6.632-13.856-14.555-15.176-5.014-.835-10.195-1.215-15.187-1.191-4.99.023-9.612.448-13.805 1.191C37.098 6.188 35 10.758 35 19.183V30h29v4H23.776c-8.484 0-15.914 5.108-18.237 14.811-2.681 11.12-2.8 17.919 0 29.53C7.614 86.983 12.569 93 21.054 93H31V79.952C31 70.315 39.428 62 49.33 62zm-1.838-39.11c-3.026 0-5.478-2.479-5.478-5.545 0-3.079 2.451-5.581 5.478-5.581 3.015 0 5.479 2.502 5.479 5.581-.001 3.066-2.465 5.545-5.479 5.545zm74.789 25.921C120.183 40.363 116.178 34 107.682 34H97v12.981C97 57.031 88.206 65 78.489 65H49.33C41.342 65 35 72.326 35 80.326v27.8c0 7.91 6.745 12.564 14.462 14.834 9.242 2.717 17.994 3.208 29.051 0C85.862 120.831 93 116.549 93 108.126V97H64v-4h43.682c8.484 0 11.647-5.776 14.599-14.66 3.047-9.145 2.916-17.799 0-29.529zm-41.955 55.606c3.027 0 5.479 2.479 5.479 5.547 0 3.076-2.451 5.579-5.479 5.579-3.015 0-5.478-2.502-5.478-5.579 0-3.068 2.463-5.547 5.478-5.547z" /></svg> icon in the upper right-hand corner to get started. |
There was a problem hiding this comment.
@meganeoleary This doc only supports Python + Typescript 😬
github-actions
Bot
added
github_actions
gregfurman
force-pushed
the
add/cookbook/s3-processing
branch
from
56a4fe4 to
bce40c6
Compare
gregfurman
changed the title
gregfurman
added
review: merge when ready
4 participants
Description
Includes a cookbook on running a massively parallel data processing pipeline against Amazon S3.
Type of change
What's Changed
Added a new python and typescript example worker for running S3 workloads.
Included a python test for the worker in
examples/python/aws/s3/test_worker.pyAdded a new dynamic diagram showing objects being polled from each bucket and with process tasks spinning up and executing: