Skip to content

Treat oversized requests as invalid and stop the backoff loop in flows - #8

Merged
liran merged 5 commits into
mainfrom
fix/oversized-input-and-backoff-loop
Nov 5, 2026
Merged

liran merged 5 commits into
mainfrom
fix/oversized-input-and-backoff-loop

Conversation

@liran

@liran liran commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

A request over gRPC's 4MiB message limit is refused before it reaches Temporal, with ResourceExhausted, the code Temporal also uses for rate limits, so IsInvalidConfigError missed it and a caller that retries everything else, like a queue consumer, retried it forever. It now recognizes the refusal by its gRPC message, as the SDK does, and an integration test sends 3MiB and 5MiB inputs to a real server.

The total backoff behind a step's overall timeout was a loop over every retry inside the flow, so millions of attempts with delays too small to grow tripped the deadlock detector. It is now summed in closed form and rounded up, so it never falls short of the delays Temporal waits, and it only feeds a timeout, which replay does not compare.

@liran liran self-assigned this Nov 5, 2026
@liran
liran force-pushed the fix/oversized-input-and-backoff-loop branch from 87a6e43 to ab363f8 Compare November 5, 2026 20:14
@liran
liran merged commit a8b7b64 into main Nov 5, 2026
4 checks passed
@liran
liran deleted the fix/oversized-input-and-backoff-loop branch November 5, 2026 21:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant