feat(cosmos-perf): add open-loop --target-rate mode#4645
Draft
tvaron3 wants to merge 1 commit into
Draft
Conversation
Add an open-loop load-generation mode to the Cosmos perf harness, driven by a fixed arrival rate rather than fixed concurrency. This eliminates coordinated-omission bias: in closed-loop mode a single slow request stalls the next N issuances, hiding tail latency under load. Open-loop mode issues operations on a wall-clock schedule regardless of how long prior operations take. New flags (config.rs): - --target-rate <ops/s>: enables open-loop mode (no default; when unset the harness runs in the existing closed-loop --concurrency mode). - --max-in-flight <N>: caps concurrently outstanding operations in open-loop mode (default 100000). When saturated, excess scheduled issuances are skipped and counted rather than queued, so arrivals never bunch into a catch-up burst. Behavior (runner.rs): - 1ms ticker computes the cumulative target issue count from elapsed wall-clock time; the inner loop issues the per-tick delta. - Each issuance acquires a semaphore permit (try_acquire_owned) before spawning; on saturation it increments a skip counter and logs a WARN every 10000 skips. - Best-effort 30s drain waits for in-flight operations to complete, then prints a total-skipped summary. - Closed-loop path is refactored to share a run_one() helper with the open-loop path; its behavior is unchanged. Known follow-ups (not blocking): - Latency in open-loop mode is measured from operation start inside the spawned task, not from the intended issue time, leaving a small residual omission under executor saturation. - Skip count is reported to stdout only, not persisted to the results document. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an open-loop load-generation mode to the Cosmos perf harness (
azure_data_cosmos_perf), driven by a fixed arrival rate instead of fixed concurrency.Motivation: coordinated omission
The existing harness runs closed-loop: a fixed pool of
--concurrencyworkers each issue a request, wait for it, then issue the next. Under load this hides tail latency — one slow request stalls the next request on that worker, so the slow period is under-sampled (classic coordinated omission). Open-loop mode issues operations on a wall-clock schedule regardless of how long prior operations take, so latency percentiles reflect what a real client population would observe.What changed
New flags (
config.rs):--target-rate <ops/s>--concurrencymode (fully backward compatible).--max-in-flight <N>100000Behavior (
runner.rs):try_acquire_owned) before spawning; on saturation it increments a skip counter and logs aWARNevery 10,000 skips.run_one()helper with the open-loop path; its behavior is unchanged.Backward compatibility
No behavior change unless
--target-rateis passed. Existing closed-loop runs (--concurrency) are byte-for-byte equivalent aside from the extractedrun_one()helper.Validation
cargo check,cargo clippy,cargo fmt --checkall clean.--target-rate 5000. The feed-range query workload (heavySELECT VALUE COUNT(1)) was bounded with--max-in-flight 500; the skip-and-count path engaged exactly as designed and the pod stayed stable.Known follow-ups (not blocking, intentionally out of scope here)
run_one().Draft
Opening as draft pending broader review of the harness changes.