Skip to content

Fix #717: run.json totals count subgraph agents' tokens twice - #747

Merged
harperreed merged 1 commit into
mainfrom
jiffy/issue-717-ab466924e62d27a6b3dbaa6c17998e8c
Nov 6, 2026
Merged

harperreed merged 1 commit into
mainfrom
jiffy/issue-717-ab466924e62d27a6b3dbaa6c17998e8c

Conversation

@jiffy-2389

@jiffy-2389 jiffy-2389 Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Issue #717 — run.json counts subgraph agents' tokens twice

Root cause

Two sibling handlers stamp a node id onto events, and they disagreed:

  • pipeline.NodeScopedPipelineHandler (pipeline events) prefixes:
    evt.NodeID = parentNodeID + "/" + evt.NodeID.
  • agent.NodeScopedHandler (agent events) overwrote: evt.NodeID = h.nodeID.

Scopes nest by wrapping. A subgraph's registry factory wraps the parent's agent
handler with a handler scoped to the subgraph node ("Task02"); the child's
codergen handler wraps that again with its own node id ("Implement"). Because
the agent stamp overwrote, the agent's call-level lines (llm_finish,
turn_metrics) landed under "Task02" (or "Implement"), while every pipeline
event for the same node — including the decision_outcome rollup that carries
token_input/token_output — landed under "Task02/Implement".

pipeline/runusage.go keys per-node usage by node id and takes each node's most
direct tier, so it counted the call tier under one id and the rollup tier under
the other. .totals came out at exactly twice combinedUsageForBudget's
snapshot. Reproduced before the fix on a two-agent child subgraph: engine and
the last cost_updated both said 240 tokens; .totals said 480 (ratio exactly
2.00), with the container node Task02 also carrying its children's 200 input
tokens. The same defect showed up for manager_loop children.

Fix

agent/events.go — the agent stamp now composes exactly as the pipeline one
does (new scopeNodeID helper): adopt the scope when the event names no node,
otherwise prefix "<scope>/<existing>". One node now has one id across the agent
and pipeline streams, so runusage.go's per-node tier selection sees a single
node and counts its work once. A top-level agent node is unchanged ("Gen").

Supporting changes: docs/architecture/agent.md (the NodeID invariant now
states the composition rule) and a CHANGELOG ### Fixed entry.

Tests (all written first, all fail before the fix and pass after)

  • agent/events_test.go — TestNodeScopedHandlerComposesWithAnEnclosingScope
    (the registry-factory + codergen nesting), …StampsABareNodeID,
    …MatchesPipelineScoping (table pinning the rule shared with
    NodeScopedPipelineHandler, including the degenerate "Review/Review" case),
    …ComposesANestedSubgraphPath ("L1/L2/Leaf").
  • pipeline/handlers/expand_integration_test.go —
    TestSubgraphAgentEventsShareTheirNodesScopedID: runs a real registry/engine
    with a subgraph and asserts the agent's usage-bearing events and the matching
    decision_outcome carry the same "Sg/Child" id.
  • tracker_capture_totals_test.go —
    TestRunManifestSubgraphTotalsCountEachAgentOnce: end-to-end run with capture,
    comparing run.json against three independent oracles — the engine's own
    accumulated usage, the run's last cost_updated event (the exact discrepancy
    the issue reports), and per-node attribution (agent spend must land on
    Task02/Implement / Task02/Review, not on the container Task02).

Verified by reverting only agent/events.go and re-running: the end-to-end test
then reports run.json totals = 480 tokens, but the last cost_updated says 240.

Verification actually run

  • go build -o /dev/null ./... — pass.
  • go test -vet=off -count=1 ./... — pass, no failures.
  • go test -count=1 ./cmd/tracker-conformance -run TestGoldenTraces — pass (the
    golden traces include node_id and usage; no fixture needed regeneration).
  • go vet ./agent/ ./pipeline/handlers/ . — clean.
  • make complexity — complexity gate OK (135 grandfathered, 0 new).
  • Repo hooks on exactly the changed files (per instruction, not the frozen
    prek run --from-ref FETCH_HEAD --to-ref HEAD command):
    prek run --config .pre-commit-config.yaml --files CHANGELOG.md agent/events.go agent/events_test.go docs/architecture/agent.md pipeline/handlers/expand_integration_test.go tracker_capture_totals_test.go
    — all hooks pass (incl. gofmt, goimports, make test); no files were modified
    by the hooks.

Limitations / notes

  • Archived runs keep their inflated totals. The double count is recorded at
    emission time; a pre-fix log still contains both node ids, so re-assembling it
    with tracker run-json does not recover the correct figure. Only new runs are
    fixed.
  • Out of scope: parallel fan-out shows the same aggregate-vs-children
    shape (the component node's decision_outcome rollup restates its branch
    targets' spend, which is booked under the branches as well). That is a
    separate construct from this issue's subgraph nesting and was left untouched.
  • A bare agent.Completer emits no trace events, so llm_calls is unavailable
    in the end-to-end test (a limit already documented by
    TestRunManifestTotalsMatchEngineUsage); the test therefore asserts token and
    cost totals rather than a call count.

Starting base: c697043bf705b9cfec28326d7b6183cfee2a3832
Verified base: c697043bf705b9cfec28326d7b6183cfee2a3832
Verified commit: 4815e57b29b722f6237e70f773c7d3cf7e5631e8

Verification commands and exit codes:

[{"Argv":["prek","run","--from-ref","FETCH_HEAD","--to-ref","HEAD","--config",".pre-commit-config.yaml"],"ExitCode":0}]

Closes #717

Final review: pass (pi/lunaroute/deepseek-4.1-flash), job 1, round 1.


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

@coderabbitai

coderabbitai Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Important

Review skipped

This PR was authored by a bot without an assigned CodeRabbit review seat. Ask an organization administrator to assign a seat to the bot. See the seat-assignment guide. After the seat is assigned, request a review again.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: d2fc3616-33bc-4198-bddf-f4157f04402c

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@harperreed
harperreed merged commit b4a3b44 into main Nov 6, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

run.json totals count subgraph agents' tokens twice

1 participant