feat(microvm-worker): add HTTP listener and server

[ParidelPooya]

Adds the HTTP listener to aws-durable-execution-sdk-python-microvm-worker. The worker now answers the MicroVM lifecycle hooks and the job routes, and runs each job. It is the Python counterpart of createMicrovmWorkerListener and startMicrovmWorker in @aws/durable-execution-sdk-js-microvm-worker.

This is the second of two PRs for the basic worker. #763 added the job documents, CallbackReporter, and Heartbeats. Later PRs add auto-suspend, and the suspend, terminate, and resume hook actions.

Usage

from aws_durable_execution_sdk_python_microvm_worker import start_microvm_worker

worker = start_microvm_worker(
    handler=run_build,                    # jobs that name no route
    routes={"/clone-build": clone_build},  # jobs sent to a named route
)
worker.wait()

A handler is a plain function (input, context) -> result. Its result must be JSON-serializable.

What it contains

worker.py

• start_microvm_worker(...) binds port 8080 by default, serves in a daemon thread, and returns a MicrovmWorker. MicrovmWorker has port, idle(timeout), wait(timeout), and close().
• MicrovmWorkerListener.create(...) validates the options. Its request_handler_class serves the listener from your own ThreadingHTTPServer.
• MicrovmJobContext is a frozen dataclass with callback_id, microvm_id, region, cancelled, and logger.
• HOOK_PATH_PREFIX and MICROVM_JOB_PATH match the JS constants.

Request handling

1. POST .../run validates the body, starts the job's thread, and answers HTTP 200 at once. A slow or failed run hook can terminate the MicroVM, so the job runs after the answer. Lambda sends the run hook once. So when the job's thread cannot start, the worker fails the job's callback, instead of leaving the job to its heartbeat timeout. Only the first valid run hook is used, because the MicroVM endpoint forwards a request to a hook path from any caller that holds an auth token.
2. POST <route> validates the job request, starts the job's thread, and answers HTTP 202. MICROVM_JOB_PATH runs handler. A thread that cannot start gets HTTP 500, which the durable function retries. After a 202, the durable function does not deliver the job again.
3. A second delivery of a callback ID is answered like the first and ignored. The check and the registration run under one lock, because requests run in parallel threads. The listener remembers the last 1,000 finished callback IDs.
4. An invalid document gets HTTP 400 with the validation message. When it names a callback, the listener fails that callback, so the durable function fails at once instead of at its timeout. Any other read error gets a generic 400 message, so no internal error text reaches the answer.
5. Every other lifecycle hook gets HTTP 200 and no action. Other paths get HTTP 404.
6. The server listens with a backlog of 511, as Node does. With socketserver's default of 5, a burst of 50 parallel job requests got connection resets on Linux.
7. The Server header names the package, not the Python version. A client that resets the connection during the headers prints no traceback.

Each job

1. It runs in its own daemon thread, with its own reporter and client. create_client replaces the default client. The listener never closes a client from create_client.
2. Heartbeats run while the handler runs and while the outcome is reported.
3. A heartbeat that finds the callback gone sets context.cancelled. The outcome is then not reported.
4. The handler's result completes the callback. Its exception fails the callback. A result over 256 KB, or one that is not JSON-serializable, is reported as a failure.

Where Python needs different code from JavaScript

1. Cancellation. Python has no AbortSignal. So context.cancelled is a threading.Event. Python cannot stop a running thread, so the handler must check the event.
2. Async handlers. An async def function, or an object whose __call__ is async def, raises TypeError at create time. A handler that returns an awaitable fails its job with TypeError, and a returned coroutine is closed. Without this, the coroutine would be reported as a result that is not JSON-serializable.
3. Process lifetime. The server runs in a daemon thread. So the image's entry point ends with worker.wait(). Node keeps its process alive while a server listens.
4. Address families. Lambda calls the hooks on localhost. A probe showed curl http://localhost:<port> connect over ::1 on macOS. So the server binds :: with IPV6_V6ONLY off, and falls back to IPv4 when the host has no dual-stack IPv6. Node binds both by default.
5. Request framing. http.server does not decode chunked bodies, and does not answer Expect: 100-continue under HTTP/1.0. Node does both. So the worker decodes chunked bodies, with chunk sizes checked against RFC 9112. The chunk-size lines, their extensions, and the trailers count against a limit of their own, the size of the body limit, as in Node's fix for CVE-2024-22019. It also answers in HTTP/1.1 with Connection: close on every answer. Before this, curl waited 1.04 s before it sent a 2 MB body. After it, curl waited 0.05 s.
6. Oversized bodies. The worker answers a body over the limit (64 KB for the run hook, 6 MiB for a job) without reading it. A connection that closes with unread bytes sends a TCP reset, so the client never read the 400. With Expect: 100-continue, the 400 now replaces 100 Continue. Without it, the worker reads and drops the body after the answer, for at most 10 seconds and 24 MiB. A probe got the 400 in 20 of 20 tries at 6 MiB + 1 byte and at 16 MiB. The JS worker resets at 16 MiB.
7. idle() and failure reports. The failure report for an invalid request starts before the 400 answer. A caller can call idle() from another thread as soon as it has the answer. So the report must already be counted then.
8. SystemExit. A handler that raises SystemExit fails its job. In a thread other than the main one, an uncaught SystemExit ends the thread silently, and the job would never report.
9. Log lines. Listener lines carry callbackId as well as microvmId. An unsendable result logs job failed: its result cannot be sent, where JS logs job succeeded.
10. Factories. MicrovmWorkerListener.create and MicrovmWorker.start hold the logic. The constructors only assign, as the factory rule in CONTRIBUTING.md requires.

Other changes

The closed-callback warning in callback_reporter.py now uses the wording of aws/aws-durable-execution-sdk-js#990, which followed #984. It names both cases that CallbackTimeoutException covers: an earlier attempt delivered the outcome, or the callback timed out first. It ends with "The durable function no longer waits for this callback in either case."

Testing

• hatch run dev-microvm-worker:cov: 259 tests. worker.py has 100% line and branch coverage.
• The tests start a real server on a free port and send real HTTP requests. Only the Lambda client is a fake.
• Each test added for review feedback fails when its fix is reverted. The burst test fails that way on Linux only, because macOS caps the backlog at 128 and passes a burst of 50 either way.
• hatch run test:cov: 4,591 passed, 98% coverage.
• mypy, hatch fmt --check, and lintcommit.py pass.

Known open items

These non-blocking items are still open:

• An answer other than the size rejection still leaves a large body unread, so the client can get a reset. Examples are a 404 on a path without a handler and a 200 on a hook other than run. This matters for a worker with routes only that receives a large job on MICROVM_JOB_PATH: about 1 MiB on macOS, and 4 MiB on Linux.
• HEAD and OPTIONS get 501 from the base class, where JS answers 404.
• The 60-second timeout limits each read, not the whole request.

[yaythomas]

nit, for the squash message: the description and the commit cite aws/aws-durable-execution-sdk-js#984 for the closed-callback wording. #984 ended the warning with "The durable function has the outcome in both cases." The sentence used here, "The durable function no longer waits for this callback in either case.", came from aws/aws-durable-execution-sdk-js#990. The Python text matches #990 exactly.

[ParidelPooya]

Thanks. The PR description now cites aws/aws-durable-execution-sdk-js#990 for the wording, and the squash message will too. c014618 and ce8e394 address the inline comments. The replies are on each thread.