Skip to content

AdmissionControl logic becomes expensive when hit with a thundering herd of shape requests #4291

@alco

Description

@alco

Problem statement

Current admission control logic requires the stack to be up and does a shape handle lookup for each request which can become expensive at a certain number of concurrent requests.

If the stack is not ready, the request registers itself with the StatusMonitor and waits up to 5 seconds to be notified about stack readiness. When many requests arrive at the same time, this momentarily turns the StatusMonitor process into a bottleneck. All requests are let in since admission control step hasn't even been reached yet.

Proposed solution

Add a very cheap gate before expensive shape work.

Properties:

  • ETS counters or another low-overhead primitive.
  • No GenServer call on the reject path.
  • Coarse classification from source, request method, live flag, and offset where possible.
  • Returns 503 with adaptive Retry-After immediately when the instance/source is saturated.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions