fix(metrics): throughput in output tokens/s, measured per program - #116
Merged
Merged
Conversation
`Throughput` (and its reward/optimizer variants) reported LM calls per second; it now reports output (completion) tokens per second, reasoning included, over the phase's wall-clock. `TokensPerSecond` keeps input + output tokens. The per-phase wall-clock the throughput metrics divide by lived in one thread-local stack. Programs evaluated concurrently on one event loop (the cells of a tuner sweep) interleaved their enter/exit on it, crediting time to the wrong phase and dividing each program's calls by a shared clock. Each trainer now owns a `PhaseClock`; `op_scope` accrues to it, a phase counts while any context has it innermost (so concurrent regions of one phase count once), and the metrics read the clock of the program they are bound to. `op_scope` without a clock keeps the thread-local default. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Throughputnow means output tokens per second (reasoning included), instead of LM calls per second. Same forRewardThroughputandOptimizerThroughput.TokensPerSecondstill counts input + output tokens. Throughput values recorded before this change are calls/s and are not comparable.op_scopeenter/exit on it: time was credited to the wrong phase, and each program's calls were divided by a shared clock.PhaseClock; the trainer'sinference/reward/optimizerscopes accrue to it, and the LM/EM operational metrics read the clock of the program they are bound to.op_scope(phase)without a clock keeps the thread-local default, so existing callers are unchanged.0.10.004.Test plan
op_scope_test.py: new tests for self-time nesting, two programs interleaving phases on one loop, concurrent regions of one phase, and a nested program on another clocklm_metrics_test.pyupdated for output tokens/suv run pytest synalinks/src/metrics synalinks/src/backend synalinks/src/trainers synalinks/src/modules/language_models synalinks/src/modules/embedding_models: 550 passed🤖 Generated with Claude Code