[zhongkechen]

Prepare the coordinated Python SDK 2.1 / OTel 1.1 / testing 1.3 release. This PR changes only the testing-package version from 1.2.1 to 1.3.0 relative to main. Runtime and deployment fixes are incorporated through merged main.

Package	PyPI snapshot verified October 13	Prepared version
SDK	2.0.1	2.1.0, already on main
OTel	1.0.0	1.1.0, already on main
Testing	1.2.1	1.3.0, this PR

Head 5b3a9a3b07324f6fdec3ef49d53d0bfac831ba40 includes main c6383691eacca01ccbf0d76f0959a0de948e2ac3: the JSON logging fix (#792), recoverable testing-image release CI, and earlier MicroVM worker, #755 propagation/model/layer, Telemetry Exporter and revoked-token/local pause-resume changes. The PR diff is now only the testing-package version bump; main already contains the logging fix and its regression test. The SDK layer pin remains 2.1.0, and its Core boto3 minimum comes from #755. The intended combined release remains sdk-v2.1.0,otel-v1.1.0,testing-v1.3.0; the PR author will perform the release merge after the gates below. No package, tag or layer is published here.

Draft release notes

SDK 2.1.0

• Merged: collect optional plugin metadata for new chained-invoke START checkpoints and serialize XAmznTraceId; the Core boto3 minimum is 1.43.114 (feat(plugin): propagate chained invoke trace context in START checkpoints #755).

• Merged: return PENDING after checkpoint-token revocation, including revocation observed during checkpoint cleanup (fix(sdk): answer PENDING on revoked token #757).

• Merged: reject competing OTel views before discovered factories run, with explicit registration metadata and cleanup (fix(otel): reject competing telemetry views #753).

• Merged: run invocation Start, handler/finally, output preparation, resource cleanup and End on the handler worker; isolate registered-plugin branch Contexts (fix: run invocation lifecycle on handler worker #756).

• Merged: pass operation types explicitly rather than inferring them from subtypes (fix(core): require explicit operation types #768).

• Pending prerequisite: callback terminal-notification delivery and replay corrections in fix(otel): isolate invocations and replay telemetry #758.

OTel 1.1.0

• Merged: provide operation Root/Parent/Sampled metadata for new invokes, including unsampled operations; abstain for deferred identities if the installed sampler is replaced (feat(plugin): propagate chained invoke trace context in START checkpoints #755).

• Merged: correct operation/retry status mappings (fix(otel): correct exported status mappings #752), enforce one view and recover log enrichment after rejected-instance reuse (fix(otel): reject competing telemetry views #753), and export sampled fallback roots early with stable anchor metadata (fix(otel): export sampled fallback roots early #754).

• Merged: worker-context propagation and cleanup from fix: run invocation lifecycle on handler worker #756. The optional handler_context API and its marker were removed. OTel 1.1 requires core >=2.1.0, with core released first; no old-core compatibility is claimed for this pair.

• Pending prerequisite: deferred-parent completion handling and expanded 26-case replay/status/callback conformance coverage in fix(otel): isolate invocations and replay telemetry #758 and test: cover OTel replay, status and user callback contexts aws-durable-execution-conformance-tests#131. Current fix(otel): isolate invocations and replay telemetry #758 uses the same core 2.1 floor and removed-handler-API design.

Telemetry completion metadata can be redelivered after an invocation or checkpoint transport failure without a successful response acknowledgment. There is no exporter acknowledgment in durable state, so #758 does not promise exactly-once telemetry across failed/redelivered invocations. This does not mean already-checkpointed step bodies run again; no global exported-operation cache is used to discard a first delivery after a failed attempt.

Testing 1.3.0

• Merged: experimental local pause_execution() / resume_execution() and paused-checkpoint simulation (fix(sdk): answer PENDING on revoked token #757). The cloud runner reports these operations as not implemented; it does not send a pause/resume service request.

• Pending prerequisite fix(otel): isolate invocations and replay telemetry #758 reports callback success/failure/timeout through UpdatedOperationIds, acknowledges invocation-input delivery after a valid response while retaining newer updates and retry delivery, and preserves detailed empty-error history while reconstructing absent-error operation state consistently.

• Main also contains virtual-clock, per-execution worker and local chained-invoke improvements since the published 1.2.1 artifact.

Remaining release gates

Compatibility preparation #774 was intentionally closed unmerged after discussion with the author; the closing decision defers its three metadata-reader fallbacks to the next major release. This Core 2.x minor release retains the existing OperationSubType enum contract, and the testing checkpoint/history parsers still require known enum labels. Those fallbacks add no behavior for the supported Core 2.x subtype contract, so #774 is explicitly no longer a prerequisite for this minor release. This does not provide custom-string subtype support or waive the implementation, conformance or final integrated release gates below.

• fix(otel): correct exported status mappings #752, fix(otel): reject competing telemetry views #753, fix(otel): export sampled fallback roots early #754 and fix: run invocation lifecycle on handler worker #756 are merged; the earlier handler API/floor discrepancy between fix: run invocation lifecycle on handler worker #756 and fix(otel): isolate invocations and replay telemetry #758 is resolved in the current fix(otel): isolate invocations and replay telemetry #758 source.
• Merge fix(otel): isolate invocations and replay telemetry #758 and test: cover OTel replay, status and user callback contexts aws-durable-execution-conformance-tests#131, then refresh this release preparation against that final main commit. feat(plugin): propagate chained invoke trace context in START checkpoints #755 is now merged and included through main; the separate breaking factory migration refactor(plugin): create one plugin instance per invocation #737 remains outside this release.
• Complete the final operation-type/subtype compatibility review using the recorded discussion. fix(core): require explicit operation types #768 is already merged and the reviewer accepted its minor-release classification; fix(sdk): load state with unlisted subtypes #762 was closed without merging. fix: tolerate string subtype metadata #774 is closed unmerged and deferred to the next major release, as recorded above. This is a minor release and preserves the existing public subtype contract. The public enum/type migration and broader custom-configuration, parser and serializer changes proposed in issue [Feature]: Custom subtype for step, callback, and child context operations #749 are reserved for a separately reviewed major release. Coordinated package versions and higher dependency floors do not replace that major-release boundary.
• Resolve the inclusion of aws-durable-execution-sdk-python-telemetry-exporter (renamed from Insight by main refactor(telemetry): rename Insight to Durable Execution Telemetry #786) in the unconditional build/publish matrices in .github/workflows/pypi-publish.yml before the combined SDK/OTel/testing release. The rename does not add this exporter to the coordinated release scope. Any first publication of the new distribution name also requires its own release decision and PyPI trusted-publishing configuration under RELEASING.md. Relative to current main, this version-bump PR changes neither the exporter nor the publisher.
• Obtain green CI on the final release main commit, rebuild final artifacts, verify legal files/dependencies and the combined-release layer pin, and finalize per-package release notes/tag checks under RELEASING.md.
• Satisfy this PR's effective approval and normal repository protections after the prerequisites and release gates are complete. The PR author will perform the release merge after final review.

Validation and boundary

Head 5b3a9a3b07324f6fdec3ef49d53d0bfac831ba40 includes main c6383691eacca01ccbf0d76f0959a0de948e2ac3 (#792), which now contains the identical JSON logging fix and regression test. This merge changes ancestry only: the entire repository tree is byte-identical to 62d49099803200fcbec9d367417a3ca3feca5e19. The existing runtime, script and cloud validation still applies to those unchanged files; no test matrix was repeated. All five workflows for the current version-only diff passed; the release prerequisites and final integrated validation below still apply.

Previous base integration 62d49099803200fcbec9d367417a3ca3feca5e19 included main fdca06707a375e309a0c79af0e9e888c3c340035 (#718). Its nine-file change adds recoverable testing-image release CI, version-parsing helpers, related tests and release documentation. Each composed tree passed 116 scoped script/workflow tests, three helper type checks and formatting. The workflow tests exercise the real shell steps with local AWS/Docker command stubs; they publish nothing. All package sources, the JSON logging repair and qualified OTel references are byte-identical to the preceding head. All six standard SDK workflows passed on 62d49099803200fcbec9d367417a3ca3feca5e19. The remaining implementation and final integrated release gates above still apply.

The JSON logging repair in 9ba654464ab05a4f2044adb5890704c0585f71a7 explicitly sets JSON logging for the 23 durable functions in the plugin conformance template. Lambda rejected the omitted setting during deployment with Only JSON format logs are supported; the other resource failures were cancellations after that rejection. The existing CI role-injection path and SAM Translator 1.115.0 now produce JSON logging for all 23 functions, with all other function properties and test metadata preserved. Both composed branches passed 12 related tests; the exact old template failed the new regression, and the updated test passed type and format checks. Production handlers, SDK/OTel/testing runtime, workflow pins and dependency floors are unchanged. Current-head conformance CI completed successfully with the corrected deployment configuration.

Main 28bf08625d5996b2446f8a6cd2f99865d5c3f604 (Python #782) is integrated. Its changes are confined to the MicroVM worker lifecycle hooks/self-suspend implementation, Extras configuration and composed tests, and the worker Hatch environment. The shared worker tree passed 398 tests, type/format checks and package build; each composed branch passed 259 Extras tests plus type/format checks. Worker wheel sources match the tested checkout. Core, OTel, testing, conformance handlers, workflow/source pins and release gates are unchanged by this integration. Their earlier validation still applies to identical source bytes; new-head CI runs separately.

This head includes merged #755 through main 2b408f444ea791df07c85a4c91c5953a8f500b95: new chained-invoke START checkpoints can carry optional plugin X-Ray metadata using the released Lambda model, and the SDK layer includes its required runtime dependencies. Validation: 4390 Core/OTel/testing tests and five subtests, 701 installed OTel tests, 12 current and 12 released-plugin lifecycle tests, all component type checks, formatting and nine layer/dependency checks passed. Those runtime and installed-plugin checks ran on the preceding base integration; the current template correction has the separate validation above. Wheel Python sources match the validated checkout. New-head GitHub CI runs separately; no tag, package or layer is published.

Main 01c7b4695d605b4773a9110fc8cd190b18985a73 (#790) is integrated. The retry test now uses a real execution and checks retry count, backoff, completion and final result; malformed-output tests retain explicit non-paused state and check that the invoker ran. The three changed tests match upstream, and all other test logic is preserved. Each composed branch passed 135 executor tests, testing type checks and formatting. All runtime sources, qualified conformance references and dependency floors are unchanged; the earlier full runtime and installed-plugin evidence still applies to those identical sources. New-head CI runs separately.

Main 720949e5968eb98098053d8091a658e74efb0838 (#777) is integrated: the testing runner fails malformed or otherwise invalid non-PENDING outputs immediately; PENDING with no pending operations has a three-attempt retry budget. The initial fixture correction set pause/deferred state explicitly; upstream #790 now supplies real execution state for retry coverage and keeps explicit pause guards for malformed-output coverage. This composed branch passed 1702 testing tests, testing type/format checks and exact-source composition checks. Core, OTel, workflow references and package/dependency versions are unchanged. New-head GitHub CI runs separately.

Main 3eee4d7b6529ca9db58c265656f7fba8498b010e workflow updates are integrated. All package/runtime/test sources are byte-for-byte unchanged from the preceding branch head. Ten focused workflow tests, YAML parsing and caller/source checks passed. The OTel workflow uses the upstream 2387bb48 pin; its caller inputs, test ref and resource queues are unchanged. New-head GitHub CI runs separately.

Historical head a7fd58e passed all five workflows: Python package (38486515047), Cloud tests (38486515069), JS Examples (38486515035), CodeQL (38486514000), and AI PR Review (38486515753). The AI review reported no actionable findings; it is not a submitted approval. Those results are historical. Historical head 545db366 passed all five workflows: Python package (38623127133), Cloud tests (38623127148), JS Examples (38623127120), CodeQL (38623119626), and AI PR Review (38623124798, attempt 2 after one targeted activity-recording retry). The base merge passed 98 extras tests and all 87 script tests selected by the workflow. A broader local script run also reproduced one unchanged, pre-existing lint-subject test mismatch. These checks do not validate unmerged prerequisites or replace final release validation.

Historical head 36af7f44 added main #776's Extras HTTP-delivery changes and passed 187 affected Extras tests with 17-file type/format checks. Historical head f819b414 additionally incorporates main #779's MicroVM-session changes: 257 affected Extras tests and 21-file type/format checks passed. Core, OTel, testing, workflow and version-metadata trees are unchanged from 36af7f44. Current-head CI and the final integrated release gates above remain separate validation.

Base integration 9315e8fc20573e11a4dab020e5b970b0ad858090 incorporates main d77d13e6deedf1d57ab3de3047cb597f074da7ef (#757): Core returns PENDING when the checkpoint token is revoked, and the local testing runner supports experimental pause/resume. Paused checkpoints preserve accepted updates while withholding further checkpoint permission; resume continues from durable state. This composed branch passed 4258 Core/testing/OTel tests + 5 subtests; all3component typechecks,Core/testing fmt and exact-source imports passed. The existing PR changes are preserved; OTel, plugin registration/notification code, handlers, workflow/test references and dependency floors are unchanged by this base integration. GitHub CI for the new head runs separately. No tag, package, layer or release is published.

The original release-preparation preflight on ef683592 plus the version bump built wheels/source distributions with Hatch 1.16.5 and Python 3.13.15, verified versions/dependencies/Python floors/LICENSE/NOTICE, checked the OTel wheel contract and layer pin, and passed 18 metadata/layer/wheel tests. That remains historical preparation evidence, not a fresh build of the eventual final release commit.

No tag, GitHub Release, PyPI package or Lambda layer publication is performed by this PR preparation.

[yaythomas]

before next release, please let's double-check how operation type works the type out from the subtype in plugin, since this is a plugin change it could probably do with going out in the same release. still investigating.

[zhongkechen]

@yaythomas I checked this and opened #768. Checkpoint-backed plugin hooks already use the explicit operation type, but local hooks could fall back to inferring it from subtype. #768 removes that fallback and OperationType.from_sub_type: OperationIdentifier now requires operation_type=, and every operation creator supplies it.

The new tests cover built-in operations across suspension/replay and checkpointed/virtual children whose subtype differs from their type. Core (1,821), OTel (668), and Insight (157) tests pass locally, with 98.33% core coverage; type checking and lint also pass. CI is still running.

Requiring operation_type= and removing OperationType.from_sub_type are source-breaking for callers that use those lower-level helpers directly. If these changes are considered non-breaking under the SDK's supported API contract, I recommend including #768 in the minor release; otherwise, it should go in a major release. #762 separately addresses loading unknown subtype values.

[yaythomas]

agree minor release is fine here.

this #749 change to make OperationSubType a str could impact Otel also, so we might want to consider making this 1st to avoid release churn?

Before we release, OTel and testing should accept a subtype that is a plain string.

If we make this change now, #749 won't need another OTel or testing release.

Change:

• OTel, in execution_plugin.py and invocation_plugin.py: replace info.sub_type.value with getattr(info.sub_type, "value", info.sub_type).
• Testing, in model.py: make the same replacement in the sub_type property. In the two places that parse with OperationSubType(...), keep the raw string when the value isn't a member.

Effect today: none. For every subtype the SDK records today, the new code returns the same value.

[zhongkechen]

@yaythomas #774 is now limited to metadata-read preparation in commit 1c2158d: the two OTel attribute builders and testing's history-event projection accept enum values or plain-string labels. Core and testing public subtype declarations remain unchanged.

The earlier claim in this reply that unknown labels are accepted by the checkpoint/typed-history parsers is withdrawn. Real endpoint validation found that passing an unknown string into the current core model can consume an in-memory checkpoint token before serialization fails; returning that string as an enum-typed history result also violates the declared contract. Both parser fallbacks have therefore been removed and are deferred until the core parsing, serialization, replay and public type work in issue #749 can be coordinated. Existing enum parsing and known-label identity are preserved.

The testing suite passes 1,615 tests; the final 210 focused exporter/history/endpoint tests and six-file type check pass, including state/token/retry controls on memory, filesystem and SQLite stores and an existing typed .value caller. The new head's CI is pending. #774 remains a separate preparation dependency for this release; it does not implement the full custom-subtype feature.

[github-actions]

Codex AI review

No actionable findings in the testing-package version bump from 1.2.1 to 1.3.0. Validation was limited to static inspection; tests and package builds were not run under the review constraints.

Reviewed commit 5b3a9a3b07324f6fdec3ef49d53d0bfac831ba40. Workflow run