Repository navigation
feat(telemetry): send actual.filter.evaluated from the brief and the gates - #953
Conversation
|
Review rounds on this draft:
The full local suite ran on 687623d, and 4922f9a only moves comments and a table row. It compiles cleanly, and
|
There was a problem hiding this comment.
Review by austinborn's review page (review-austinborn).
Recommendation: Needs additional review · Risk: Unknown · Confidence: Medium · Reviewed head: 4922f9a8668b
One P3 turned up in the brief's counting, and nothing else in the lines the inspector could render. The gate half of the feature runs through src/cli/commands/check_engine.rs, which is past the inspector's 250,000-byte bound, so none of its changed lines were reviewed and the gate-side claims rest on tests I could not run.
Findings: one P3, posted inline on src/cli/commands/rules_brief.rs; nothing else in the lines I could review.
Enumeration report, open questions, skipped checks and coverage ledger
Enumeration report
Each item this change modifies, the other references to it at the reviewed commit, and how many the change updated. Searches ran over the checked-out working tree, which I confirmed matches the reviewed head (the new symbol is present and four files equal the reader's copies in size). A caller is called read only where it went through the bundled reader; the rest were located by search.
- Shared symbol
brief_telemetry::report(newfilterparameter): 1 non-test caller, inrules_brief.rs(read); updated 1, left 0.HookRun(newfilterfield): 6 struct literals, inplan_check.rs,impl_check.rs, threecheck_telemetry.rstests and onecheck_engine.rstest; all 6 set the field, 0 left. The definition and thecheck_engine.rsliteral were located by search only.ScopeIndex::search,search_weighted,search_adrs(bodies now delegate to tallied variants): 3 non-test callers, all inrules_scope.rs(read). Each only consumes the matches or groups, and signatures and results are unchanged, so 0 needed an edit. Test callers were not opened.ScopeIndex::score_document(now returns a three-way enum): 1 caller; updated 1, left 0.retry_by_text(now returns a tuple): 2 callers,governing_talliedand one test; updated 2, left 0.governingstays a wrapper with an unchanged signature, and its 1 outside caller, inplan_check.rs(read), only tests for emptiness, so leaving it alone is correct.prefilter/Prefiltered(new private field, newtallymethod): 2 non-test callers inrules_scope.rs(located by search), which ignore the tally and build no event, and 1 incheck_engine.rsthat uses it (located by search only).AnyValue(newDoubleValuevariant): 1 exhaustive conversion, inotlp.rs, updated; no use outside that file (a control search over the same paths found all 15 hits inside it).actual.event.schema_version(5 to 6): the constant, every test assertion that pinned 5,PRIVACY.mdandCHANGELOG.mdare updated; 0 left. The otherschema_versionhits belong to the ADR content schema and are unrelated.
- Route table: the change modifies none.
- Grant list: the only list of this kind is the re-exports in
scope/mod.rs, which gainFilterTallyandKeptFile; there is no other. I found no name-keyed event registry or allow-list: the previous new event appears in its sender,otlp.rs,PRIVACY.mdandCHANGELOG.md, and this one appears in the matching places plustests/cli_test.rs; 0 missing. - Generated artifact: none found for this change. No event catalogue, schema snapshot or fixture is generated from the event list, and the lockfile is untouched.
- Configuration limit: the change modifies none. The record-count caps (
MAX_RECORDS_PER_REQUESTand the spool's event cap) now see up to one more record per hook round, and my search ofotlp.rsfor size constants found only those record-count caps. - Sibling implementation: three selectors share the new accounting. The brief is updated in
rules_brief.rs. The plan gate and the diff gate are wired the same way inplan_check.rsandimpl_check.rs, but the engine they share,check_engine.rs, was not reviewable. Manualrules selectbuilds no event, which matches the updated privacy text.
Not completed: the check_engine.rs side of items 1 and 6. Corrective hunks: the inspector returns one merged diff and the page forbids reading history, so I could not tell whether any hunk corrects an earlier one in this change.
Open questions and missing evidence
- The changed lines of
src/cli/commands/check_engine.rswere not reviewable: the blob is over the 250,000-byte bound, for the inspector and the bundled reader alike. That file holds the gate's pipeline wrapper, the definition of the newHookRunfield, the per-file split into kept, already settled and outside the judge's window, and the two places the gate sends the event. Unverified: that each hook round sends exactly one event (never a stand-alone one in addition to the one sent with its verdicts) and that the gate's buckets sum to the index size. - Questions for the author: does every exit from a gate round that precedes a verdict send the stand-alone event, and does none send it twice? Which bucket takes a selected gate file that holds no rule for the judge to rule on, and is that the case the inline P3 describes for the brief?
- No check run or status exists at the reviewed head, and the base requires none, so nothing independent confirms the tests, lints or coverage the description cites. I ran none myself.
- A build with the default features off was not exercised, so whether the telemetry-gated wiring compiles without them is unverified.
Skipped or inconclusive checks
- Tests, builds, formatters and linters: skipped. The page forbids running them in the target repository, and no disposable environment was authorized.
- The inspector's one limitation, restated: changed content not reviewable for
src/cli/commands/check_engine.rs(blob exceeds 250000 bytes). Inconclusive for everything that file contributes. - Searches were piped, so their individual exit statuses were not recorded; each conclusion rests on the matches printed. Each absence claim has a positive control that matched: the
AnyValuesearch over the same paths (15 hits, all inotlp.rs) and the sibling event-name search (the four expected places).
Coverage and verification ledger
- Target: pull request 953 on
actual-cli, base branchadr-reported-use. Base and merge base09f78428e9879794e1f6c96594c9d4f6f4164ffc; head4922f9a8668b0049914af6ab0c55e96982a41f58. 15 changed paths:CHANGELOG.md,PRIVACY.md, seven Rust files undersrc/cli/commands/(includingadr/telemetry.rs), four undersrc/rules/scope/(one new),src/telemetry/otlp.rsandtests/cli_test.rs. Intended outcome, taken from the description: one new telemetry event accounting for what the brief and the two gates kept and dropped from the rule index. - Instructions: no
CLAUDE.mdexists at the root or in any ancestor directory of a changed path, nor.claude/CLAUDE.md, so nothing governs. Other tools' instruction files were intentionally not read. - Review lenses: correctness and contracts; security and privacy; data and blast radius; reliability and observability; tests and proof; end-user and operability.
- ADRs:
.actual/rules/does not exist at the head, so the ADR gate does not apply; none selected or omitted, no cap. - Checks: required check names: none configured (rulesets and branch protection both read); check runs and statuses at the head: none. Nothing was executed in the target repository; only bounded read-only searches ran.
- Inspection: the inspector ran twice, both exit 0, and the measured head, base, merge base, changed paths, diff and fingerprint were identical. That proves only that the measured Git objects did not change; the working tree is not fingerprinted in this mode. The pull request's head, read through the review App just before posting, still names the reviewed commit.
- Evidence I checked and found sound:
- The brief's counts conserve: the search, the
--limitand session-memory step, the parse and the render each add exactly the files they remove, so the buckets and kept files sum to the index size, and none of the subtractions can underflow (reachedis de-duplicated by pointer and every entry is in the parsed set). The one problem is a bucket's label, in the P3. Prefiltered::tally's unsigned subtraction is safe: a selection never holds more than the candidates minus those judged unrelated.- The floor comparison was rewritten from "at least" to a negated "less than". The two differ only for a NaN score, and scores are built from guarded, finite coverage values, so none reaches it.
- The brief finds each kept file by pointer identity, and the rendered documents are references into the same vector it searches, so the lookup is sound.
- On the wire, the new double variant is handled by the only exhaustive conversion, non-finite values go as 0, and the kept references and their scores come from one unzip, so their lengths agree.
- The brief's counts conserve: the search, the
This comment review approves nothing and blocks nothing; austinborn still reviews this pull request.
Posted by the operator's software factory.
• City:city· Agent:local-core.external-reviewer
• On behalf of: @austinborn
| Some(KeptFile::of(hit)) | ||
| }) | ||
| .collect(); | ||
| tally.brief_cap += parsed.len() - tally.kept.len(); |
There was a problem hiding this comment.
[P3] Don't count files that contribute no rule line as cut by the brief's limits
Evidence: tally.brief_cap += parsed.len() - tally.kept.len(); puts every parsed rule file that is not in rendered.reached into the bucket for the brief's own limits. reached holds only documents that put at least one line into the brief, and decision_lines in src/rules/brief.rs writes a line only for MUST and MUST NOT rules and skips a statement an earlier file in the same decision already stated. Nothing between the search and the parse drops advice-only files, and the index treats them as ordinary documents, so a file with no obligation, or whose obligations all repeat a sibling's, ends up in actual.filter.dropped.brief_cap.
Impact: That attribute is described in tally.rs, PRIVACY.md and CHANGELOG.md as the files cut by the decision limit, the session memory or the size budget. Files that simply had nothing to show inflate it, so anyone reading the filter-quality signal will overstate what the brief's budget costs. The counts still sum to the index size, so nothing downstream breaks.
Fix: Either widen the three descriptions to say the bucket also holds files with no line of their own, or count those files in a bucket that says so before this line. Add a test with a file that has only a SHOULD rule and one that repeats a sibling's statement, asserting kept_count, brief_cap and that the counts still sum.
Verification: Read render_brief_shown, pack and decision_lines at the reviewed head and checked the three descriptions against them. Nothing was executed, so no test has confirmed the count; the reasoning is from the code alone.
Posted by the operator's software factory.
• City:city· Agent:local-core.external-reviewer
• On behalf of: @austinborn
16447c3 to
eef4054
Compare
|
@austinborn @backlineint, could one of you review this one? It's the bottom of the CLI's ADR telemetry stack, so #958 and #960 can't merge until it does. Main, including #930, is now merged in with an ordinary merge commit.
|
|
Merged The #930 conflicts came from both sides adding to the same places. Every path #930 added sends the event. A Codex shell read or The release filed this PR's CHANGELOG entry under 0.15.0, so the second merge moves it back under Unreleased. Local verificationAt 740c44cc on rustc 1.99.1:
|
…gates Each selection that reaches the agent or the judge now says what it kept and what it dropped at each step, counted in rule files that sum to the index size, with the kept files' ADR references and best scores. The gates send one every hook round; the brief sends one only when it adds an ADR new to the session. actual.event.schema_version is now 6. The Codex hook modes reach the same brief and gate code, so a Codex shell read and a Codex plan gate round send the selection event as Claude Code's do. Both are tested end to end, signed out and signed in, and CHANGELOG and PRIVACY.md say either hook mode sends it. City: factory-main · Agent: local-core.builder-2
330f3a5 to
95397de
Compare
Summary
The rules brief and both gates now send
actual.filter.evaluated, an OpenTelemetry event saying how their selection narrowed the repository's rule files down to what reached the agent or the judge. It records the selection (context,planningorimplementation, the same words asactual.detection.method), the score floor, the index size, the drops at each step by reason, and the kept files' ADR references with their best scores. The counts are in rule files and add up to the index size. Gate events shareactual.action.idwith that round'sactual.adr.detectedverdicts, so kept ADRs the judge callednot_applicablegive a production precision signal for the filter-quality eval (AK-1148, AK-1033).The gates send one every hook round, including rounds that keep nothing. The brief sends one only when it adds an ADR new to the session. Dropped files are never named, and no rule text, plan or diff text, path or glob is sent.
One measured session on a 3,473-file rule set: 2 brief events and 1 gate event, beside 1
actual.action.completed, so the brief ran at 2× the gate count, under the 5× threshold.Where the risk sits. The scope index search gains a per-reason tally, and
gather_rulesnow classifies each selected file.actual.event.schema_versionmoves to6, which assumes #945 lands first with5.Stacked on #945. Its base is
adr-reported-use, so CI's gating workflows don't run here. The checks were run locally (below), most recently after merging #945's review fixes into this branch.Attributes, test plan, the measured session, and what this leaves out
How the event is built
Attributes
actual.filter.selectorcontext(brief),planning(plan gate),implementation(diff gate)actual.filter.min_score0for the gates, which apply none todayactual.filter.index_sizeactual.filter.dropped.inactiveactual.filter.dropped.excluded_pathactual.filter.dropped.no_matchactual.filter.dropped.below_flooractual.filter.dropped.cappedactual.filter.dropped.unrelatedactual.filter.dropped.sessionactual.filter.dropped.windowactual.filter.dropped.brief_capactual.filter.kept_countactual.filter.kept_refsadr_id), best firstactual.filter.kept_scoresThree choices go beyond the approved plan, and each is named here so it can be reversed:
dropped.no_matchis new. Files scoring zero are by far the largest bucket, and without them the counts can't sum to the index size.actual.adr.detectedkeys ADRs by group, sokept_refscan be shorter thankept_count. A kept file without a v2 header has no ref.doubleValuefor the floor and scores. A non-finite value goes as 0.The measured session
Run on 2026-11-18 in a checkout of a large internal TypeScript monorepo that commits 3,474 v2 rule files (3,473 indexed). It was a headless Claude Code 2.1.327 session with the actual-cli plugin 1.8.0 and this branch's debug build.
ACTUAL_TELEMETRY_DEBUG=1was set, OTLP went to a local file andACTUAL_API_URLpointed at a closed local port, so nothing left the machine. Turn one planned a small error-handling change to one server module in plan mode, and turn two implemented it with tests.actual.filter.evaluatedcontextactual.filter.evaluatedimplementationactual.action.completedimpl-checkactual.adr.detectedEvery queued event reached the file sink at the
sentstage. Each event's counts summed to 3,473. The gate's round, for example, had 2,728 excluded by path, 15 no match, 710 capped and 20 kept, naming 7 ADRs. The model wrote its plan as text and never called ExitPlanMode in headless mode, so the plan gate didn't run in this session. Its event is covered byplan_check's hook test, which checks the selector and kept refs. A first attempt, on an older branch of the same repository, sent no brief event, because none of its 425 rule files has a v2 header and the brief only reports ADRs it can identify, the same ruleactual.adr.detectedfollows.Test plan
All on CI's pinned rustc 1.99.1:
cargo fmt --checkandcargo clippy -- -D warnings: clean.cargo test --lib --features integration: 4,979 passed, 1 ignored, run as two shards.cargo test --workspace --features integration --no-fail-fast --test '*' --bins: all 15 binaries pass, includingcli_test's end-to-end brief hook test, which now expects the filter event in the request a mock endpoint receives.cargo llvm-covwith the coverage workflow's exact arguments: every file this PR touches is at 100% line coverage. The only file under 100% isanalysis/cache.rs(817/818), the known detached-HEAD arm that only a named-branch run misses.cargo checkandcargo clippy -- -D warningswith--locked --no-default-features --lib --bins, as the lint workflow now runs them: clean. feat(rules): ask the agent which ADRs it used each turn, and check its answer #945 merged main's opt-out build fix, and this branch's telemetry code compiles out with the feature.After merging #945's review fixes
#945's head moved to take its review fixes and a merge of main, and this branch merges it in. Only
CHANGELOG.mdconflicted, on adjacent entries: it keeps #945's rewritten report-use entry and this branch's filter-event entry after it. The filter event needed no change for #945's telemetry gate. With telemetry off it builds nothing and writes no ledger key, debug mode logs it as not sent, and a build without the feature doesn't compile it, the same three cases #945's question now follows. It keeps no session state, so there's nothing like the plan #945 now drops.Re-run on rustc 1.99.1 after the merge:
cargo fmt --checkand both clippy builds clean;cargo test --lib4,995 passed, and 4,595 with--no-default-features;cargo test --workspace --features integrationpassed in every binary; and the coverage workflow'scargo llvm-covrun passed with every file at 100% except the sameanalysis/cache.rsline. One earlier coverage run failed in #945'sadr::cloudreview-event test, which then poisoned the shared env lock for the tests after it. That test passes alone under coverage, and the full re-run passed.Out of scope
rules select's PostHog event.min_scorereports the floor in force.