Repository navigation
fix(testing): fail fast on invalid handler output - #777
Open
Thanatos9404 wants to merge 1 commit into
Open
Thanatos9404 wants to merge 1 commit into
Thanatos9404 wants to merge 1 commit into
Conversation
The local runner retried every invalid invocation output up to five times, five seconds apart. The service fails the execution on the first such output, with an InvalidParameterValueException error, and retries only PENDING with no pending operations, up to three attempts. - Fail the execution at once for FAILED with a Result, SUCCEEDED with an Error, a missing or unknown Status, and an output that cannot be parsed. The error has ErrorType InvalidParameterValueException and the validation message as ErrorMessage. - Keep retrying PENDING with no pending operations, capped at three consecutive attempts. Other retries keep the five-attempt budget. - Parse handler output in InvokeResponse.from_output and from_payload, which raise InvalidInvocationOutputError, so the executor can tell an output it cannot parse from a failed invocation.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue #, if available: #670
Description of changes:
The local runner sent every invalid invocation output through
_retry_invocation: up to five attempts, five seconds apart, then a failure withErrorObject.from_exception(e). As #670 describes, the service fails the execution on the first invalid output and retries only PENDING with no pending operations, up to three attempts. This implements the fix proposed there.Executor._finish_invocation: a validation failure now fails the execution at once with{ErrorType: "InvalidParameterValueException", ErrorMessage: <validation message>}. That covers FAILED with a Result, SUCCEEDED with an Error, and a status the runner does not accept (such asRETRY). PENDING with no pending operations still goes through_retry_invocation, now capped atMAX_CONSECUTIVE_INVALID_PENDING_ATTEMPTS = 3, and fails with the same error shape.Status, or an output that is not a JSON object, never reached_finish_invocation:DurableExecutionInvocationOutput.from_dictraised inside the invoker, and the executor retried that as a failed invocation. Both invokers now parse through newInvokeResponse.from_outputandInvokeResponse.from_payloadfactories, which raise a newInvalidInvocationOutputError(aDurableFunctionsTestError)._invoke_handlerrecords the invocation and fails the execution with the same error shape. A stop or timeout that landed during the invocation still stands.LambdaInvokerparses the payload after itstry, so a payload that is not JSON is no longer reported as "Unexpected error during Lambda invocation"._retry_invocationtakes an optionalmax_attempts. Handler exceptions, invoke errors and invocation timeouts keepMAX_CONSECUTIVE_FAILED_ATTEMPTS = 5.docs/architecture.md: the retry budget section describes the split.The PENDING retry shares the existing
consecutive_failed_invocation_attemptscounter, so earlier consecutive failed attempts count toward its cap of 3 (two handler errors followed by an invalid PENDING fail at once). I can give it its own counter if the service counts them separately.Testing:
tests/e2e/invalid_invocation_output_test.py: six invalid outputs (FAILED with a Result, SUCCEEDED with an Error, no Status, unknown Status,RETRY, a JSON array) each fail the execution after one handler call with theInvalidParameterValueExceptionerror. Onmainall six time out while the retries run.executor_checkpoint_test.py: PENDING with no pending operations fails on the third attempt. PENDING once, then SUCCEEDED, completes and resets the counter.executor_test.py: the fourtest_should_retry_when_*tests for outputs that now fail fast are nowtest_should_fail_when_*. Two new tests cover the unparseable-output path, one with a stop during the invocation.test_should_retry_when_pending_response_has_no_operationsnow setsseq_counter; without it the test hit aTypeErrorinmax()and retried through the generic error path, not validation.invoker_test.py,exceptions_test.py: both invokers raiseInvalidInvocationOutputErrorfor outputs they cannot parse.Locally (Windows, Python 3.13, Hatch 1.16.5):
hatch run test:allhas no new failures (only the pre-existing Windows-only ones in the corelambda_service_test.pynegative timestamps and the microvm-workerworker_test.pyport tests, as onmain);hatch run dev-testing:covpasses at 96.08%;hatch run dev-testing:typecheck,hatch run types:checkandhatch fmt --checkare clean. The asserted error messages are identical on Python 3.11 to 3.14.hatch run dev-testing:js-exampleswas not run, because the harness needsfcntl; CI runs it. No conformance tests were added or changed.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.