Skip to content

fix(metrics): throughput in output tokens/s, measured per program - #116

Merged
YoanSallami merged 2 commits into
mainfrom
fix/per-program-phase-clock
Sep 22, 2026
Merged

YoanSallami merged 2 commits into
mainfrom
fix/per-program-phase-clock

Conversation

@YoanSallami

Copy link
Copy Markdown
Contributor

Summary

  • Throughput now means output tokens per second (reasoning included), instead of LM calls per second. Same for RewardThroughput and OptimizerThroughput. TokensPerSecond still counts input + output tokens. Throughput values recorded before this change are calls/s and are not comparable.
  • The throughput denominator is measured per program. The per-phase wall-clock lived in one thread-local stack, so programs evaluated concurrently on one event loop (e.g. the cells of a tuner sweep) interleaved their op_scope enter/exit on it: time was credited to the wrong phase, and each program's calls were divided by a shared clock.
    • Each trainer now owns a PhaseClock; the trainer's inference / reward / optimizer scopes accrue to it, and the LM/EM operational metrics read the clock of the program they are bound to.
    • A phase accrues while any context has it as its innermost scope on that clock, so concurrent regions of one phase count once, and the optimizer phase still gets its self-time only.
    • op_scope(phase) without a clock keeps the thread-local default, so existing callers are unchanged.
  • Version bump to 0.10.004.

Test plan

  • op_scope_test.py: new tests for self-time nesting, two programs interleaving phases on one loop, concurrent regions of one phase, and a nested program on another clock
  • lm_metrics_test.py updated for output tokens/s
  • uv run pytest synalinks/src/metrics synalinks/src/backend synalinks/src/trainers synalinks/src/modules/language_models synalinks/src/modules/embedding_models: 550 passed
  • ruff check on the changed files

🤖 Generated with Claude Code

Yoan and others added 2 commits September 21, 2026 16:44
`Throughput` (and its reward/optimizer variants) reported LM calls per
second; it now reports output (completion) tokens per second, reasoning
included, over the phase's wall-clock. `TokensPerSecond` keeps input +
output tokens.

The per-phase wall-clock the throughput metrics divide by lived in one
thread-local stack. Programs evaluated concurrently on one event loop (the
cells of a tuner sweep) interleaved their enter/exit on it, crediting time
to the wrong phase and dividing each program's calls by a shared clock.
Each trainer now owns a `PhaseClock`; `op_scope` accrues to it, a phase
counts while any context has it innermost (so concurrent regions of one
phase count once), and the metrics read the clock of the program they are
bound to. `op_scope` without a clock keeps the thread-local default.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@YoanSallami
YoanSallami merged commit a440794 into main Sep 22, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant