Repository navigation
docs: put the circuit breaker before AsyncTimeout; document rate-mode and event-loop behaviour - #178
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Composition order
The recommended async order changes from
to
With
AsyncTimeoutoutside the breaker, the deadline fires while the breaker is awaiting its inner call. asyncio delivers that as aCancelledErrorinside the breaker, and the breaker classifiesCancelledErroras foreign: it leaves the circuit untouched.AsyncTimeoutonly turns the cancellation into an httpwareTimeoutErrorafter it has passed back through the breaker. The result is that an upstream that hangs never opens the circuit, however many calls time out. Only fast failures (5xx, connection errors) ever count.With the breaker outside the timeout, the httpware
TimeoutErroris what reaches the breaker, and it is a counted failure. Nothing is lost by the move:I checked the old behaviour against
mainbefore changing anything: with[AsyncTimeout(0.01), AsyncCircuitBreaker(failure_threshold=2)]and a hanging transport, the third call is still sent and times out, andstateisCLOSED. With the breaker first, the third call raisesCircuitOpenError.The order stays documented only. httpware still does not enforce it at construction. The sync order (
CircuitBreaker → Bulkhead → Retry) is unchanged, since there is no syncTimeoutand httpx2's per-request timeouts are raised inside the breaker in any order.Updated: the Composition section and its example, the
AsyncTimeoutsection, a note in the breaker's failure classification, and theAsyncTimeoutdocstrings, which said "outermost".Full-stack demo
The demo engine modelled the old order: a timed-out operation was never recorded by the breaker. It now records one as a failure (
brk.res(false, now)), and the deadline still starts only after the breaker admits the request, so an open circuit rejects before any deadline. The order comments inengine.jsmatch the new chain.With that change the scenario told the wrong story. The 4s latency spike in phase 1 opened the breaker, which then stayed open or half-open through the brownout, so the "retry recovers transient errors" beat never happened and the "hard down" stop fired during phase 2. I swapped the first two phases. The incident now runs brownout (2 to 5s), latency spike (5 to 8s), then hard down (8 to 12s), and the walkthrough has a new stop where timeouts open the breaker. Traced in the browser: the breaker stays closed through the brownout, opens at 6.97s from expired deadlines, its probe fails during hard down (reopens at 10.03s), and it closes at 12.24s. The page and chain badge state the new order. No other demo combines a timeout with a breaker.
ADR-0004, updated in place
Its argument against a slow-call rate was that
AsyncTimeoutalready turns a slow call into a countedTimeoutError. Under the old recommended order that was false for the reason above. The ADR now says the argument depends on the breaker sitting outside the deadline, names the order it relies on, and notes that httpware documents but does not enforce that order.Rate mode can open on a success
In rate mode, once the rolling window holds
minimum_callsoutcomes, the failure rate is checked after every recorded outcome, successes included (_record_outcomerecords, then checks, for both kinds). So a success can be the call that opens the circuit: withminimum_calls=4and a threshold of0.5, three failures leave it closed, and a fourth call returning 200 makes the rate 3/4 and opens it. That call still returns its response; the next one getsCircuitOpenError. This matches Resilience4j. Behaviour unchanged, now documented in the rate-mode section. An existing test already pins it (test_circuit_breaker.py, two failures then two successes open the circuit).Clearer cross-loop error
AsyncBulkhead,AsyncCircuitBreakerandAsyncKeyedCircuitBreakerstay strictly bound to the first event loop they run on. Only theRuntimeErrortext changes. It now says the instance is bound to the loop it first ran on (keeping both loop reprs), that async resilience middleware and theAsyncClientholding it must be created once per event loop, for example inside the coroutine passed toasyncio.run(), and that sharing an instance across pytest-asyncio's function-scoped loops is the usual way to hit it. The old hint about the sync primitives for cross-thread use moved to the docs, since a sync breaker can't be used with anAsyncClientanyway.docs/resilience.mdgains a short "One event loop per instance" section covering the same ground, including thathttpx2.AsyncClientalso fails across twoasyncio.run()calls ("Event loop is closed").Release notes (0.22.0)
Not breaking.
AsyncCircuitBreaker → AsyncTimeout → AsyncBulkhead → AsyncRetry, so the breaker counts expired deadlines. Chains that follow the old order keep working but never open the circuit on timeouts; swap the first two entries.RuntimeErrorraised when an async bulkhead or circuit breaker is used from a second event loop has a new message. Code that matched the old "bound to a single event loop" text needs updating.Tests
test_breaker_before_timeout_counts_expired_deadlines: a hanging transport behind[breaker, AsyncTimeout(0.01)]opens the circuit after two timeouts. Swapping the order makes it fail (the third call raisesTimeoutErrorinstead ofCircuitOpenError). Async only, sinceAsyncTimeouthas no sync sibling.just lint,just lint-ci,just test-ci: 1196 passed, 100% coverage.just docs-buildpasses.No effect on the request path: only the error-path message string and docs changed.