Skip to content

feat(telemetry): send actual.filter.evaluated from the brief and the gates - #953

Merged
benw5483 merged 1 commit into
mainfrom
filter-evaluated-event
Nov 20, 2026
Merged

benw5483 merged 1 commit into
mainfrom
filter-evaluated-event

Conversation

@benw5483

@benw5483 benw5483 commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The rules brief and both gates now send actual.filter.evaluated, an OpenTelemetry event saying how their selection narrowed the repository's rule files down to what reached the agent or the judge. It records the selection (context, planning or implementation, the same words as actual.detection.method), the score floor, the index size, the drops at each step by reason, and the kept files' ADR references with their best scores. The counts are in rule files and add up to the index size. Gate events share actual.action.id with that round's actual.adr.detected verdicts, so kept ADRs the judge called not_applicable give a production precision signal for the filter-quality eval (AK-1148, AK-1033).

The gates send one every hook round, including rounds that keep nothing. The brief sends one only when it adds an ADR new to the session. Dropped files are never named, and no rule text, plan or diff text, path or glob is sent.

One measured session on a 3,473-file rule set: 2 brief events and 1 gate event, beside 1 actual.action.completed, so the brief ran at 2× the gate count, under the 5× threshold.

Where the risk sits. The scope index search gains a per-reason tally, and gather_rules now classifies each selected file. actual.event.schema_version moves to 6, which assumes #945 lands first with 5.

Stacked on #945. Its base is adr-reported-use, so CI's gating workflows don't run here. The checks were run locally (below), most recently after merging #945's review fixes into this branch.

Attributes, test plan, the measured session, and what this leaves out

How the event is built

flowchart LR
  I["Scope index search<br/>(tally by reason)"] --> B["Brief: limit, session memory,<br/>size budget"]
  I --> G["Gate: candidate cap and limit,<br/>session exclusion, 40-rule window"]
  B -->|"only when it adds an ADR"| E1["actual.filter.evaluated<br/>selector = context"]
  G -->|"every hook round"| E2["actual.filter.evaluated<br/>selector = planning or implementation"]
  E2 -->|"same actual.action.id"| V["actual.adr.detected<br/>with the judge's verdict"]
Loading

Attributes

Attribute Type Meaning
actual.filter.selector string context (brief), planning (plan gate), implementation (diff gate)
actual.filter.min_score double the floor applied; 0 for the gates, which apply none today
actual.filter.index_size int rule files the index was built from, retired ones included
actual.filter.dropped.inactive int retired status, or no longer readable or active when used
actual.filter.dropped.excluded_path int every named path outside the file's root or excluded by it
actual.filter.dropped.no_match int scored nothing (for the brief's text retry, fewer than two shared words)
actual.filter.dropped.below_floor int scored under the floor
actual.filter.dropped.capped int candidate cap (30) or the caller's limit (gates 20)
actual.filter.dropped.unrelated int stage 2 rank; always 0 in hook mode, which never ranks
actual.filter.dropped.session int gates: every rule already cleared or overridden
actual.filter.dropped.window int gates: no rule inside the 40-rule judge window
actual.filter.dropped.brief_cap int brief: decision limit, once-per-session memory, size budget
actual.filter.kept_count int files that reached the agent or the judge
actual.filter.kept_refs string[] kept files' logical ADRs (group, else adr_id), best first
actual.filter.kept_scores double[] each listed ADR's best score

Three choices go beyond the approved plan, and each is named here so it can be reversed:

  • dropped.no_match is new. Files scoring zero are by far the largest bucket, and without them the counts can't sum to the index size.
  • Counts are in rule files, refs are logical ADRs. The index holds files, and actual.adr.detected keys ADRs by group, so kept_refs can be shorter than kept_count. A kept file without a v2 header has no ref.
  • The OTLP value type gains doubleValue for the floor and scores. A non-finite value goes as 0.

The measured session

Run on 2026-11-18 in a checkout of a large internal TypeScript monorepo that commits 3,474 v2 rule files (3,473 indexed). It was a headless Claude Code 2.1.327 session with the actual-cli plugin 1.8.0 and this branch's debug build. ACTUAL_TELEMETRY_DEBUG=1 was set, OTLP went to a local file and ACTUAL_API_URL pointed at a closed local port, so nothing left the machine. Turn one planned a small error-handling change to one server module in plan mode, and turn two implemented it with tests.

Event Queued Selector
actual.filter.evaluated 2 context
actual.filter.evaluated 1 implementation
actual.action.completed 1 impl-check
actual.adr.detected 11

Every queued event reached the file sink at the sent stage. Each event's counts summed to 3,473. The gate's round, for example, had 2,728 excluded by path, 15 no match, 710 capped and 20 kept, naming 7 ADRs. The model wrote its plan as text and never called ExitPlanMode in headless mode, so the plan gate didn't run in this session. Its event is covered by plan_check's hook test, which checks the selector and kept refs. A first attempt, on an older branch of the same repository, sent no brief event, because none of its 425 rule files has a v2 header and the brief only reports ADRs it can identify, the same rule actual.adr.detected follows.

Test plan

All on CI's pinned rustc 1.99.1:

  • cargo fmt --check and cargo clippy -- -D warnings: clean.
  • cargo test --lib --features integration: 4,979 passed, 1 ignored, run as two shards.
  • cargo test --workspace --features integration --no-fail-fast --test '*' --bins: all 15 binaries pass, including cli_test's end-to-end brief hook test, which now expects the filter event in the request a mock endpoint receives.
  • cargo llvm-cov with the coverage workflow's exact arguments: every file this PR touches is at 100% line coverage. The only file under 100% is analysis/cache.rs (817/818), the known detached-HEAD arm that only a named-branch run misses.
  • New tests cover conservation for each selector, the brief sending nothing when it adds no ADR, the brief's own cuts, the floor reported for the brief and the gates, session and window classification, rank and cap counts, and the record encoding through the official protobuf types.
  • cargo check and cargo clippy -- -D warnings with --locked --no-default-features --lib --bins, as the lint workflow now runs them: clean. feat(rules): ask the agent which ADRs it used each turn, and check its answer #945 merged main's opt-out build fix, and this branch's telemetry code compiles out with the feature.

After merging #945's review fixes

#945's head moved to take its review fixes and a merge of main, and this branch merges it in. Only CHANGELOG.md conflicted, on adjacent entries: it keeps #945's rewritten report-use entry and this branch's filter-event entry after it. The filter event needed no change for #945's telemetry gate. With telemetry off it builds nothing and writes no ledger key, debug mode logs it as not sent, and a build without the feature doesn't compile it, the same three cases #945's question now follows. It keeps no session state, so there's nothing like the plan #945 now drops.

Re-run on rustc 1.99.1 after the merge: cargo fmt --check and both clippy builds clean; cargo test --lib 4,995 passed, and 4,595 with --no-default-features; cargo test --workspace --features integration passed in every binary; and the coverage workflow's cargo llvm-cov run passed with every file at 100% except the same analysis/cache.rs line. One earlier coverage run failed in #945's adr::cloud review-event test, which then poisoned the shared env lock for the tests after it. That test passes alone under coverage, and the full re-run passed.

Out of scope

  • An enforce event (block resolution, plugin delivery and dialog answers, strict mode, Cursor and Codex).
  • Naming dropped ADRs, and rules select's PostHog event.
  • The gates ignoring the score floor: a separate fix, after which min_score reports the floor in force.

Generated by the operator's software factory.
• City: factory-main · Agent: local-core.builder-1
• On behalf of: @benw5483

@benw5483

Copy link
Copy Markdown
Contributor Author

Review rounds on this draft:

  • ADR check, round 1: clean. None of the four ADRs under docs/adr/ is broken. The new event's contract is versioned (schema 6, with the otlp.rs comment, the adr/telemetry.rs pin and the PRIVACY rows), and the new code carries co-located tests.
  • Code review, round 1: two doc-placement fixes, now in 4922f9a. The group_matches doc comment had ended up above the new Scored enum, and the new PRIVACY rows had split the actual.report.* rows. Both are moved back, with no code change. The review also rechecked the bucket conservation on both paths, that Prefiltered::tally can't underflow, and that every verdict outcome carries the filter record.

The full local suite ran on 687623d, and 4922f9a only moves comments and a table row. It compiles cleanly, and cargo fmt --check passes.


Posted by the operator's software factory.
• City: factory-main · Agent: local-core.builder-1
• On behalf of: @benw5483

@benw5483
benw5483 marked this pull request as ready for review November 18, 2026 16:16

@actual-factory actual-factory Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review by austinborn's review page (review-austinborn).

Recommendation: Needs additional review · Risk: Unknown · Confidence: Medium · Reviewed head: 4922f9a8668b

One P3 turned up in the brief's counting, and nothing else in the lines the inspector could render. The gate half of the feature runs through src/cli/commands/check_engine.rs, which is past the inspector's 250,000-byte bound, so none of its changed lines were reviewed and the gate-side claims rest on tests I could not run.

Findings: one P3, posted inline on src/cli/commands/rules_brief.rs; nothing else in the lines I could review.

Enumeration report, open questions, skipped checks and coverage ledger

Enumeration report

Each item this change modifies, the other references to it at the reviewed commit, and how many the change updated. Searches ran over the checked-out working tree, which I confirmed matches the reviewed head (the new symbol is present and four files equal the reader's copies in size). A caller is called read only where it went through the bundled reader; the rest were located by search.

  1. Shared symbol
    • brief_telemetry::report (new filter parameter): 1 non-test caller, in rules_brief.rs (read); updated 1, left 0.
    • HookRun (new filter field): 6 struct literals, in plan_check.rs, impl_check.rs, three check_telemetry.rs tests and one check_engine.rs test; all 6 set the field, 0 left. The definition and the check_engine.rs literal were located by search only.
    • ScopeIndex::search, search_weighted, search_adrs (bodies now delegate to tallied variants): 3 non-test callers, all in rules_scope.rs (read). Each only consumes the matches or groups, and signatures and results are unchanged, so 0 needed an edit. Test callers were not opened.
    • ScopeIndex::score_document (now returns a three-way enum): 1 caller; updated 1, left 0.
    • retry_by_text (now returns a tuple): 2 callers, governing_tallied and one test; updated 2, left 0. governing stays a wrapper with an unchanged signature, and its 1 outside caller, in plan_check.rs (read), only tests for emptiness, so leaving it alone is correct.
    • prefilter / Prefiltered (new private field, new tally method): 2 non-test callers in rules_scope.rs (located by search), which ignore the tally and build no event, and 1 in check_engine.rs that uses it (located by search only).
    • AnyValue (new DoubleValue variant): 1 exhaustive conversion, in otlp.rs, updated; no use outside that file (a control search over the same paths found all 15 hits inside it).
    • actual.event.schema_version (5 to 6): the constant, every test assertion that pinned 5, PRIVACY.md and CHANGELOG.md are updated; 0 left. The other schema_version hits belong to the ADR content schema and are unrelated.
  2. Route table: the change modifies none.
  3. Grant list: the only list of this kind is the re-exports in scope/mod.rs, which gain FilterTally and KeptFile; there is no other. I found no name-keyed event registry or allow-list: the previous new event appears in its sender, otlp.rs, PRIVACY.md and CHANGELOG.md, and this one appears in the matching places plus tests/cli_test.rs; 0 missing.
  4. Generated artifact: none found for this change. No event catalogue, schema snapshot or fixture is generated from the event list, and the lockfile is untouched.
  5. Configuration limit: the change modifies none. The record-count caps (MAX_RECORDS_PER_REQUEST and the spool's event cap) now see up to one more record per hook round, and my search of otlp.rs for size constants found only those record-count caps.
  6. Sibling implementation: three selectors share the new accounting. The brief is updated in rules_brief.rs. The plan gate and the diff gate are wired the same way in plan_check.rs and impl_check.rs, but the engine they share, check_engine.rs, was not reviewable. Manual rules select builds no event, which matches the updated privacy text.

Not completed: the check_engine.rs side of items 1 and 6. Corrective hunks: the inspector returns one merged diff and the page forbids reading history, so I could not tell whether any hunk corrects an earlier one in this change.

Open questions and missing evidence

  • The changed lines of src/cli/commands/check_engine.rs were not reviewable: the blob is over the 250,000-byte bound, for the inspector and the bundled reader alike. That file holds the gate's pipeline wrapper, the definition of the new HookRun field, the per-file split into kept, already settled and outside the judge's window, and the two places the gate sends the event. Unverified: that each hook round sends exactly one event (never a stand-alone one in addition to the one sent with its verdicts) and that the gate's buckets sum to the index size.
  • Questions for the author: does every exit from a gate round that precedes a verdict send the stand-alone event, and does none send it twice? Which bucket takes a selected gate file that holds no rule for the judge to rule on, and is that the case the inline P3 describes for the brief?
  • No check run or status exists at the reviewed head, and the base requires none, so nothing independent confirms the tests, lints or coverage the description cites. I ran none myself.
  • A build with the default features off was not exercised, so whether the telemetry-gated wiring compiles without them is unverified.

Skipped or inconclusive checks

  • Tests, builds, formatters and linters: skipped. The page forbids running them in the target repository, and no disposable environment was authorized.
  • The inspector's one limitation, restated: changed content not reviewable for src/cli/commands/check_engine.rs (blob exceeds 250000 bytes). Inconclusive for everything that file contributes.
  • Searches were piped, so their individual exit statuses were not recorded; each conclusion rests on the matches printed. Each absence claim has a positive control that matched: the AnyValue search over the same paths (15 hits, all in otlp.rs) and the sibling event-name search (the four expected places).

Coverage and verification ledger

  • Target: pull request 953 on actual-cli, base branch adr-reported-use. Base and merge base 09f78428e9879794e1f6c96594c9d4f6f4164ffc; head 4922f9a8668b0049914af6ab0c55e96982a41f58. 15 changed paths: CHANGELOG.md, PRIVACY.md, seven Rust files under src/cli/commands/ (including adr/telemetry.rs), four under src/rules/scope/ (one new), src/telemetry/otlp.rs and tests/cli_test.rs. Intended outcome, taken from the description: one new telemetry event accounting for what the brief and the two gates kept and dropped from the rule index.
  • Instructions: no CLAUDE.md exists at the root or in any ancestor directory of a changed path, nor .claude/CLAUDE.md, so nothing governs. Other tools' instruction files were intentionally not read.
  • Review lenses: correctness and contracts; security and privacy; data and blast radius; reliability and observability; tests and proof; end-user and operability.
  • ADRs: .actual/rules/ does not exist at the head, so the ADR gate does not apply; none selected or omitted, no cap.
  • Checks: required check names: none configured (rulesets and branch protection both read); check runs and statuses at the head: none. Nothing was executed in the target repository; only bounded read-only searches ran.
  • Inspection: the inspector ran twice, both exit 0, and the measured head, base, merge base, changed paths, diff and fingerprint were identical. That proves only that the measured Git objects did not change; the working tree is not fingerprinted in this mode. The pull request's head, read through the review App just before posting, still names the reviewed commit.
  • Evidence I checked and found sound:
    • The brief's counts conserve: the search, the --limit and session-memory step, the parse and the render each add exactly the files they remove, so the buckets and kept files sum to the index size, and none of the subtractions can underflow (reached is de-duplicated by pointer and every entry is in the parsed set). The one problem is a bucket's label, in the P3.
    • Prefiltered::tally's unsigned subtraction is safe: a selection never holds more than the candidates minus those judged unrelated.
    • The floor comparison was rewritten from "at least" to a negated "less than". The two differ only for a NaN score, and scores are built from guarded, finite coverage values, so none reaches it.
    • The brief finds each kept file by pointer identity, and the rendered documents are references into the same vector it searches, so the lookup is sound.
    • On the wire, the new double variant is handled by the only exhaustive conversion, non-finite values go as 0, and the kept references and their scores come from one unzip, so their lengths agree.

This comment review approves nothing and blocks nothing; austinborn still reviews this pull request.


Posted by the operator's software factory.
• City: city · Agent: local-core.external-reviewer
• On behalf of: @austinborn

Some(KeptFile::of(hit))
})
.collect();
tally.brief_cap += parsed.len() - tally.kept.len();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] Don't count files that contribute no rule line as cut by the brief's limits

Evidence: tally.brief_cap += parsed.len() - tally.kept.len(); puts every parsed rule file that is not in rendered.reached into the bucket for the brief's own limits. reached holds only documents that put at least one line into the brief, and decision_lines in src/rules/brief.rs writes a line only for MUST and MUST NOT rules and skips a statement an earlier file in the same decision already stated. Nothing between the search and the parse drops advice-only files, and the index treats them as ordinary documents, so a file with no obligation, or whose obligations all repeat a sibling's, ends up in actual.filter.dropped.brief_cap.
Impact: That attribute is described in tally.rs, PRIVACY.md and CHANGELOG.md as the files cut by the decision limit, the session memory or the size budget. Files that simply had nothing to show inflate it, so anyone reading the filter-quality signal will overstate what the brief's budget costs. The counts still sum to the index size, so nothing downstream breaks.
Fix: Either widen the three descriptions to say the bucket also holds files with no line of their own, or count those files in a bucket that says so before this line. Add a test with a file that has only a SHOULD rule and one that repeats a sibling's statement, asserting kept_count, brief_cap and that the counts still sum.
Verification: Read render_brief_shown, pack and decision_lines at the reviewed head and checked the three descriptions against them. Nothing was executed, so no test has confirmed the count; the reasoning is from the code alone.


Posted by the operator's software factory.
• City: city · Agent: local-core.external-reviewer
• On behalf of: @austinborn

@benw5483

Copy link
Copy Markdown
Contributor Author

@austinborn @backlineint, could one of you review this one? It's the bottom of the CLI's ADR telemetry stack, so #958 and #960 can't merge until it does. Main, including #930, is now merged in with an ordinary merge commit.


Posted by the operator's software factory.
• City: factory-main · Agent: local-core.manager
• On behalf of: @benw5483

@benw5483

Copy link
Copy Markdown
Contributor Author

Merged main twice, both times with an ordinary merge commit: once for #930's Codex hook modes (ef5a7d33) and once for the v0.15.0 release (330f3a55).

The #930 conflicts came from both sides adding to the same places. HookRun now carries #930's host, codex_ask and picker next to this PR's filter. In the brief, the filter account is built before #930's per-session size budget check, and a brief that check turns away is never recorded, so it still sends no actual.filter.evaluated.

Every path #930 added sends the event. A Codex shell read or apply_patch goes through the same brief_for. plan-check --codex-hook and impl-check --codex-hook go through the same run_pipeline_filtered and finish_hook, including the --codex-ask and picker branches, which hand the round's records to the same report. The sign-in split is applied where records are spooled and uploaded, so Codex records follow it as they are. 740c44cc adds three end-to-end tests in tests/cli_test.rs: a Codex shell read signed out (no Authorization header), the same read signed in (the token goes in the header and never reaches the file sink), and a Codex plan gate round that sends the planning selector. It also changes CHANGELOG and PRIVACY.md to say either hook mode sends the event.

The release filed this PR's CHANGELOG entry under 0.15.0, so the second merge moves it back under Unreleased.

Local verification

At 740c44cc on rustc 1.99.1: cargo test --workspace --features integration passed 5,695 tests with 0 failures and 4 ignored. cargo clippy -- -D warnings, cargo fmt --check and cargo check --no-default-features --lib --bins were all clean. The release merge after it changes only CHANGELOG, Cargo.toml and Cargo.lock, and it builds clean. CI runs on the pushed head.


Posted by the operator's software factory.
• City: factory-main · Agent: local-core.builder-3
• On behalf of: @benw5483

@benw5483
benw5483 added this pull request to the merge queue Nov 19, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 19, 2026
@benw5483
benw5483 added this pull request to the merge queue Nov 19, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 19, 2026
…gates

Each selection that reaches the agent or the judge now says what it kept
and what it dropped at each step, counted in rule files that sum to the
index size, with the kept files' ADR references and best scores. The
gates send one every hook round; the brief sends one only when it adds an
ADR new to the session. actual.event.schema_version is now 6.

The Codex hook modes reach the same brief and gate code, so a Codex shell
read and a Codex plan gate round send the selection event as Claude Code's
do. Both are tested end to end, signed out and signed in, and CHANGELOG
and PRIVACY.md say either hook mode sends it.

City: factory-main · Agent: local-core.builder-2
@benw5483
benw5483 force-pushed the filter-evaluated-event branch from 330f3a5 to 95397de Compare November 20, 2026 00:01
@benw5483
benw5483 added this pull request to the merge queue Nov 20, 2026
Merged via the queue into main with commit b8c99c2 Nov 20, 2026
10 checks passed
@benw5483
benw5483 deleted the filter-evaluated-event branch November 20, 2026 00:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants