Skip to content

fix(testing): fail fast on invalid handler output - #777

Open
Thanatos9404 wants to merge 1 commit into
aws:mainfrom
Thanatos9404:fix/670-fail-fast-invalid-output
Open

Thanatos9404 wants to merge 1 commit into
aws:mainfrom
Thanatos9404:fix/670-fail-fast-invalid-output

Conversation

@Thanatos9404

Copy link
Copy Markdown

Issue #, if available: #670

Description of changes:

The local runner sent every invalid invocation output through _retry_invocation: up to five attempts, five seconds apart, then a failure with ErrorObject.from_exception(e). As #670 describes, the service fails the execution on the first invalid output and retries only PENDING with no pending operations, up to three attempts. This implements the fix proposed there.

  • Executor._finish_invocation: a validation failure now fails the execution at once with {ErrorType: "InvalidParameterValueException", ErrorMessage: <validation message>}. That covers FAILED with a Result, SUCCEEDED with an Error, and a status the runner does not accept (such as RETRY). PENDING with no pending operations still goes through _retry_invocation, now capped at MAX_CONSECUTIVE_INVALID_PENDING_ATTEMPTS = 3, and fails with the same error shape.
  • A missing or unknown Status, or an output that is not a JSON object, never reached _finish_invocation: DurableExecutionInvocationOutput.from_dict raised inside the invoker, and the executor retried that as a failed invocation. Both invokers now parse through new InvokeResponse.from_output and InvokeResponse.from_payload factories, which raise a new InvalidInvocationOutputError (a DurableFunctionsTestError). _invoke_handler records the invocation and fails the execution with the same error shape. A stop or timeout that landed during the invocation still stands. LambdaInvoker parses the payload after its try, so a payload that is not JSON is no longer reported as "Unexpected error during Lambda invocation".
  • _retry_invocation takes an optional max_attempts. Handler exceptions, invoke errors and invocation timeouts keep MAX_CONSECUTIVE_FAILED_ATTEMPTS = 5.
  • docs/architecture.md: the retry budget section describes the split.

The PENDING retry shares the existing consecutive_failed_invocation_attempts counter, so earlier consecutive failed attempts count toward its cap of 3 (two handler errors followed by an invalid PENDING fail at once). I can give it its own counter if the service counts them separately.

Testing:

  • New tests/e2e/invalid_invocation_output_test.py: six invalid outputs (FAILED with a Result, SUCCEEDED with an Error, no Status, unknown Status, RETRY, a JSON array) each fail the execution after one handler call with the InvalidParameterValueException error. On main all six time out while the retries run.
  • executor_checkpoint_test.py: PENDING with no pending operations fails on the third attempt. PENDING once, then SUCCEEDED, completes and resets the counter.
  • executor_test.py: the four test_should_retry_when_* tests for outputs that now fail fast are now test_should_fail_when_*. Two new tests cover the unparseable-output path, one with a stop during the invocation. test_should_retry_when_pending_response_has_no_operations now sets seq_counter; without it the test hit a TypeError in max() and retried through the generic error path, not validation.
  • invoker_test.py, exceptions_test.py: both invokers raise InvalidInvocationOutputError for outputs they cannot parse.

Locally (Windows, Python 3.13, Hatch 1.16.5): hatch run test:all has no new failures (only the pre-existing Windows-only ones in the core lambda_service_test.py negative timestamps and the microvm-worker worker_test.py port tests, as on main); hatch run dev-testing:cov passes at 96.08%; hatch run dev-testing:typecheck, hatch run types:check and hatch fmt --check are clean. The asserted error messages are identical on Python 3.11 to 3.14. hatch run dev-testing:js-examples was not run, because the harness needs fcntl; CI runs it. No conformance tests were added or changed.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

The local runner retried every invalid invocation output up to five
times, five seconds apart. The service fails the execution on the first
such output, with an InvalidParameterValueException error, and retries
only PENDING with no pending operations, up to three attempts.

- Fail the execution at once for FAILED with a Result, SUCCEEDED with
  an Error, a missing or unknown Status, and an output that cannot be
  parsed. The error has ErrorType InvalidParameterValueException and
  the validation message as ErrorMessage.
- Keep retrying PENDING with no pending operations, capped at three
  consecutive attempts. Other retries keep the five-attempt budget.
- Parse handler output in InvokeResponse.from_output and from_payload,
  which raise InvalidInvocationOutputError, so the executor can tell an
  output it cannot parse from a failed invocation.
@Thanatos9404
Thanatos9404 deployed to ai-pr-review-runtime October 15, 2026 11:34 — with GitHub Actions Active

This branch was successfully deployed

1 active deployment
ai-pr-review-runtime — d18bba71 Deployed Oct 15, 2026 by Thanatos9404 via notify / Generate notification summary #210
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant