Conversation
The local Queues broker registered a timer for every message it received, including messages with no delivery delay. workerd caps a Durable Object at 10000 active timeouts and none of those timers run while the producer is still sending, so enqueuing more than 10000 messages in one burst failed with QuotaExceededError. Messages without a delivery delay are now enqueued directly and only delayed messages use a timer. Miniflare already behaved this way under fake timers, where Timers.setTimeout routes a zero delay through queueMicrotask and runs the closure synchronously - which is why the existing queues tests, all of which enable fake timers, never caught this. Fixes cloudflare#15670
🦋 Changeset detectedLatest commit: 45185f0 The changes in this PR will be included in the next version bump. This PR includes changesets to release 8 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Codeowners approval required for this PR:
Show detailed file reviewers
|
@cloudflare/autoconfig
@cloudflare/build-output-utils
@cloudflare/codemods
@cloudflare/config
@cloudflare/containers-shared
create-cloudflare
@cloudflare/deploy-helpers
@cloudflare/kv-asset-handler
miniflare
@cloudflare/pages-functions
@cloudflare/pages-shared
@cloudflare/runtime-types
@cloudflare/unenv-preset
@cloudflare/vite-plugin
@cloudflare/vitest-plugin
@cloudflare/workers-auth
@cloudflare/workers-editor-shared
@cloudflare/workers-utils
wrangler
commit: |
|
The three red
The same line shows up in the
Could someone with Actions write access re-run the three For reference, locally on this branch: |
|
Codeowners approval required for this PR:
Show detailed file reviewers |
workers-devprod
left a comment
workers-devprod
left a comment
There was a problem hiding this comment.
Codeowners reviews satisfied
Co-authored-by: Christopher Little-Savage <clittle-savage@cloudflare.com>
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Head branch was pushed to by a user without write access
cloudflare#15713 removed `type: "worker"` from the worker config type, so the test added here stopped type-checking once main was merged in, and the miniflare build failed.
3 participants
Fixes #15670.
The local Queues broker hands every incoming message to a timer, even when the
message has no delivery delay:
workerd caps a Durable Object at 10000 active timeouts, and none of those timers
run while the producer is still sending — the reported error says
finished timeouts: 0, so not one of the 10000 had completed. A Worker thatenqueues more messages than that in a single burst therefore fails:
The timer was introduced with delayed delivery in #5570; before that,
#enqueuepushed straight onto
#messages. This change restores that for the undelayedcase and keeps the timer only where a delay was actually requested.
It also makes the real-timer path agree with the fake-timer path. In
Timers.setTimeout, a zero delay under fake timers goes toqueueMicrotask,whose fake implementation invokes the closure synchronously — so under fake
timers the broker has always enqueued undelayed messages immediately. Every
existing test in
test/plugins/queues/exceptcross-process.spec.tsenablesfake timers, which is why the suite never covered this.
The retry path in
#flushalso usesthis.timers.setTimeoutwith apossibly-zero delay, but it is bounded by construction — a retry can only be
scheduled by a flush, and a flush is itself a timer callback — so it is left
alone here.
Measurements
Sending batches of 100 through
sendBatch()sequentially from onefetch()handler, against
packages/miniflarewithdistrebuilt on each side:QuotaExceededErrorQuotaExceededErrorThe boundary sits exactly on workerd's limit: 10000 messages pass, 10001 fail.
That is the arithmetic signature of one live timer per message, and it is what
the new regression test pins — it enqueues 20000 messages, twice the ceiling, so
it can only pass if the broker's live timer count stays bounded well below the
number of messages in flight.
enqueues more undelayed messages than workerd's active timeout limitinpackages/miniflare/test/plugins/queues/index.spec.ts, run in a worktree at themerge base (9515011) with only the test applied:
and on this branch:
File and line references are against
main@9515011dc5ecdc5abf3a0c685d80f78e307fb513.send()/sendBatch()in local dev; no documented behaviour or configuration changes.A picture of a cute animal (not mandatory, but encouraged)