feat: flamegraph CPU-scaling benchmark tooling

[serene-kitfisto-8899]

Summary

This PR adds the profiling infrastructure to generate CPU-scaling flamegraphs for the Cosmian KMS, providing empirical proof that the server scales linearly with CPU count and that the dominant hotspot is cryptographic work (not locking, not I/O).

The --server-workers flag and the http_throughput Criterion bench were landed in previous commits on develop; this PR adds the flamegraph/benchmarking tooling on top.

Changes

New: bench_run_flamegraph.sh

Self-contained script modeled on bench_run_load.sh that:

1. Runs the http_throughput Criterion bench across all worker counts (1/2/4/8 by default)
2. Generates per-worker flamegraph SVGs (ecdsa_sign_w<N>.svg) via cargo-flamegraph / Linux perf
3. Writes an updated cpu_scaling.md report under target/flamegraphs/

Key env controls: WORKER_COUNTS, PROFILE_TIME, PERF_FREQ, SKIP_FLAMEGRAPH, SKIP_THROUGHPUT.

[profile.bench] in Cargo.toml

The release profile uses opt-level = "z" (size) and strip = "symbols", causing a 15-40% throughput penalty in benchmarks and completely unreadable flamegraphs. New [profile.bench] overrides to opt-level = 3, strip = "none", debug = 1, lto = "thin".

.cargo/config.toml - perf flags for x86_64

• force-frame-pointers=yes: forces perf --call-graph fp, reducing perf.data from 4.2 GB to ~20 MB

SQLite pragma tuning

Added cache_size (64 MiB), mmap_size (256 MiB), temp_store=MEMORY to reduce syscall overhead under concurrent read load.

.gitignore

Ignored perf.data, perf.data.old, target/flamegraphs/.

How to run

sudo sysctl kernel.perf_event_paranoid=-1
cargo install flamegraph --locked

# Full run (generates per-worker SVGs)
bash .github/scripts/benchmarks/bench_run_flamegraph.sh --variant non-fips

# Throughput only (no perf required)
SKIP_FLAMEGRAPH=1 bash .github/scripts/benchmarks/bench_run_flamegraph.sh

Outputs are written to target/flamegraphs/:

• ecdsa_sign_w<N>.svg — per-worker interactive flamegraph
• cpu_scaling.md — markdown report linking to all of the above

[Copilot]

Pull request overview

This PR adds CPU-scaling benchmarking + flamegraph tooling around the KMS HTTP server to empirically validate multi-core scaling, and introduces supporting configuration/profile changes (server worker count knob, bench build profile, CI workflow, and benchmark scripts/docs).

Changes:

• Introduces a Criterion benchmark (http_throughput) and a GitHub Actions workflow to generate CPU-scaling throughput reports and perf-based flamegraphs.
• Adds a --server-workers / KMS_SERVER_WORKERS configuration option and wires it into Actix HttpServer::workers.
• Tunes benchmarking build/profile settings (bench profile, cargo config flags) and adjusts SQLite PRAGMAs for concurrent read performance.

Reviewed changes

Copilot reviewed 20 out of 22 changed files in this pull request and generated 14 comments.

Show a summary per file

File	Description
resources/kms.toml	Documents server_workers in sample config.
pkg/kms.toml	Documents server_workers in packaged config.
documentation/mkdocs.yml	Adds navigation entry for CPU scaling/flamegraph page.
documentation/docs/certifications_and_compliance/cryptographic_algorithms/benchmarks/cpu_scaling.md	Adds methodology and reproduction documentation for scaling + flamegraphs.
crate/test_kms_server/Cargo.toml	Registers the new http_throughput bench and adds futures dev-dependency.
crate/test_kms_server/benches/http_throughput.rs	Adds the async HTTP throughput benchmark sweeping worker counts and operations.
crate/server/src/start_kms_server.rs	Applies configurable Actix worker thread count on server startup.
crate/server/src/main.rs	Updates test config struct initialization for new HTTP field.
crate/server/src/config/wizard/tests.rs	Updates wizard round-trip tests for new HTTP field.
crate/server/src/config/wizard/http_wizard.rs	Adds default server_workers: None in wizard HTTP config.
crate/server/src/config/params/server_params.rs	Plumbs server_workers from config into ServerParams.
crate/server/src/config/command_line/http_config.rs	Adds CLI/env support for server_workers.
crate/server/kms_template.toml	Documents server_workers in server template config.
crate/server_database/src/stores/sql/sqlite.rs	Adds SQLite PRAGMA tuning (cache/mmap/temp_store) alongside WAL/synchronous/busy_timeout.
CHANGELOG/develop.md	Adds develop-branch changelog entries describing the new benchmark/tooling.
Cargo.toml	Adds [profile.bench] tuned for flamegraph visibility and throughput realism.
Cargo.lock	Updates lockfile (notably adds futures and bumps some transitive deps).
.gitignore	Ignores perf and flamegraph output artifacts.
.github/workflows/flamegraph.yml	Adds CI workflow to run the benchmark and generate/upload flamegraphs.
.github/scripts/benchmarks/README.md	Documents the new flamegraph benchmark runner script.
.github/scripts/benchmarks/bench_run_flamegraph.sh	Adds a local script to run throughput + flamegraph collection and emit a markdown report.
.cargo/config.toml	Adds a bench alias and x86_64 Linux rustflags intended for perf/flamegraph runs.

[serene-kitfisto-8899]

@copilot disable workflows flamegraph.yml this job will be run manually

[Copilot]

@copilot disable workflows flamegraph.yml this job will be run manually

Done — removed the schedule and pull_request triggers from flamegraph.yml. The workflow now only runs on workflow_dispatch. (commit ef3e27f → new commit on branch)

[serene-kitfisto-8899]

@copilot fix : .github/scripts/benchmarks/bench_run_flamegraph.sh: line 260: unexpected EOF while looking for matching `"'

[Copilot]

@copilot fix : .github/scripts/benchmarks/bench_run_flamegraph.sh: line 260: unexpected EOF while looking for matching `"'

Fixed — there was a stray " on line 212 left over from an earlier edit that opened a string never closed. Removed in the latest commit.

[serene-kitfisto-8899]

@copilot group svg flamefraphs form target/flamegraphs and make one graph in order to show worker scaling

[Copilot]

@copilot group svg flamefraphs form target/flamegraphs and make one graph in order to show worker scaling

Done. Added .github/scripts/benchmarks/combine_flamegraphs.py and wired it into bench_run_flamegraph.sh.

After the per-worker SVGs are generated, the script now calls the combiner which produces target/flamegraphs/cpu_scaling_combined.html — a single self-contained HTML file containing:

• A throughput bar chart (req/s vs worker count) at the top, with pale "ideal linear scaling" reference bars for comparison
• All per-worker interactive flamegraphs stacked vertically below with labelled headers and jump-to navigation

The combined HTML is also linked from the cpu_scaling.md report under a new "Combined scaling overview" section. The individual per-worker SVGs are still kept alongside it.

[serene-kitfisto-8899]

@copilot remove the combine_flamegraphs.py script. this feature is useless.

[Copilot]

@copilot remove the combine_flamegraphs.py script. this feature is useless.

Removed combine_flamegraphs.py and all related wiring from bench_run_flamegraph.sh in the latest commit.

[serene-kitfisto-8899]

@copilot update nix hashes

[Copilot]

@copilot update nix hashes

The Nix hashes are already current. The last Packaging CI run (all 16 jobs passed, commit f58d1a60) confirmed that nix/expected-hashes/ is consistent with the branch's Cargo.lock and ui/pnpm-lock.yaml. The two subsequent commits (ad67b47, 8d811cf) only touched start_kms_server.rs and CHANGELOG/develop.md — no lock file changes — so the hashes remain valid and no update is needed.

If you want to bring the branch up to develop's 5.24.0 package set, that would require rebasing/merging develop first, after which I can pull the new hashes from CI.