Skip to content

feat(cosmos-perf): add open-loop --target-rate mode#4645

Draft
tvaron3 wants to merge 1 commit into
Azure:mainfrom
tvaron3:tvaron3/perf-open-loop-target-rate
Draft

feat(cosmos-perf): add open-loop --target-rate mode#4645
tvaron3 wants to merge 1 commit into
Azure:mainfrom
tvaron3:tvaron3/perf-open-loop-target-rate

Conversation

@tvaron3

@tvaron3 tvaron3 commented Jun 22, 2026

Copy link
Copy Markdown
Member

Summary

Adds an open-loop load-generation mode to the Cosmos perf harness (azure_data_cosmos_perf), driven by a fixed arrival rate instead of fixed concurrency.

Motivation: coordinated omission

The existing harness runs closed-loop: a fixed pool of --concurrency workers each issue a request, wait for it, then issue the next. Under load this hides tail latency — one slow request stalls the next request on that worker, so the slow period is under-sampled (classic coordinated omission). Open-loop mode issues operations on a wall-clock schedule regardless of how long prior operations take, so latency percentiles reflect what a real client population would observe.

What changed

New flags (config.rs):

Flag Default Effect
--target-rate <ops/s> unset Enables open-loop mode. When unset, the harness runs in the existing closed-loop --concurrency mode (fully backward compatible).
--max-in-flight <N> 100000 Caps concurrently outstanding operations in open-loop mode. When saturated, excess scheduled issuances are skipped and counted rather than queued, so arrivals never bunch into a catch-up burst.

Behavior (runner.rs):

  • A 1ms ticker computes the cumulative target issue count from elapsed wall-clock time; the inner loop issues the per-tick delta.
  • Each issuance acquires a semaphore permit (try_acquire_owned) before spawning; on saturation it increments a skip counter and logs a WARN every 10,000 skips.
  • A best-effort 30s drain waits for in-flight operations to complete, then prints a total-skipped summary.
  • The closed-loop path is refactored to share a run_one() helper with the open-loop path; its behavior is unchanged.

Backward compatibility

No behavior change unless --target-rate is passed. Existing closed-loop runs (--concurrency) are byte-for-byte equivalent aside from the extracted run_one() helper.

Validation

  • cargo check, cargo clippy, cargo fmt --check all clean.
  • Exercised end-to-end on an AKS DR-drill cluster: 5 workloads (read/upsert/create/query/feedrange) sustained --target-rate 5000. The feed-range query workload (heavy SELECT VALUE COUNT(1)) was bounded with --max-in-flight 500; the skip-and-count path engaged exactly as designed and the pod stayed stable.

Known follow-ups (not blocking, intentionally out of scope here)

  1. Issue-time latency capture. Latency in open-loop mode is currently measured from operation start inside the spawned task, not from the intended issue time — leaving a small residual omission under executor saturation. A stricter harness captures the scheduled issue instant in the issuing loop and passes it into run_one().
  2. Persist skip count. The total-skipped figure is reported to stdout only, not persisted to the results document.

Draft

Opening as draft pending broader review of the harness changes.

Add an open-loop load-generation mode to the Cosmos perf harness, driven
by a fixed arrival rate rather than fixed concurrency. This eliminates
coordinated-omission bias: in closed-loop mode a single slow request
stalls the next N issuances, hiding tail latency under load. Open-loop
mode issues operations on a wall-clock schedule regardless of how long
prior operations take.

New flags (config.rs):
- --target-rate <ops/s>: enables open-loop mode (no default; when unset
  the harness runs in the existing closed-loop --concurrency mode).
- --max-in-flight <N>: caps concurrently outstanding operations in
  open-loop mode (default 100000). When saturated, excess scheduled
  issuances are skipped and counted rather than queued, so arrivals
  never bunch into a catch-up burst.

Behavior (runner.rs):
- 1ms ticker computes the cumulative target issue count from elapsed
  wall-clock time; the inner loop issues the per-tick delta.
- Each issuance acquires a semaphore permit (try_acquire_owned) before
  spawning; on saturation it increments a skip counter and logs a WARN
  every 10000 skips.
- Best-effort 30s drain waits for in-flight operations to complete, then
  prints a total-skipped summary.
- Closed-loop path is refactored to share a run_one() helper with the
  open-loop path; its behavior is unchanged.

Known follow-ups (not blocking):
- Latency in open-loop mode is measured from operation start inside the
  spawned task, not from the intended issue time, leaving a small
  residual omission under executor saturation.
- Skip count is reported to stdout only, not persisted to the results
  document.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Cosmos The azure_cosmos crate

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

1 participant