Skip to content

Commit 6627c9c

Browse files
willwashburnclaude
andauthored
feat(cli): wire burn compare --provider/--workflow/--agent filters (#415)
* feat(cli): wire `burn compare` --provider/--workflow/--agent filters Drops the runtime "not yet wired" branches in `burn compare`: - `--provider` is parsed as a lower-cased CSV allow-list and applied after `query_turns` via `provider_for`, matching `summary --by-provider` classification. - `--workflow` / `--agent` fold through `Query.enrichment` (`workflowId` / `agentId` keys); both required when both flags are passed. - Re-export `filter_turns_by_provider` / `ProviderFilter` from the SDK top-level surface so embedders can reuse the same filter. Also documents the pre-query ingest decision: `burn compare` is a presenter verb and does not run `ingest_all` first (chain `burn ingest && burn compare` for fresh data) — the JSON envelope stays byte-equivalent with the TS golden snapshot. Closes #377. * review: address coderabbit suggestions on #415 - README: `--provider <list>` → `--provider <csv>` to match the hotspots table convention. - CHANGELOG: trim the compare bullet to user-impact-first. - query_verbs: add intersection regression test pinning AND-semantics on combined `--workflow` + `--agent` enrichment filters. * review: re-export ProviderRule + AsTurnLike alongside filter helpers Codex flagged that `filter_turns_by_provider_with_rules` is callable from the lib root but its argument types weren't, leaving the new public surface partially unusable for embedders that want to supply custom rules or implement the trait for their own row type. --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 2f3b7df commit 6627c9c

5 files changed

Lines changed: 246 additions & 30 deletions

File tree

CHANGELOG.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,12 @@ Cross-package release notes for relayburn. Package changelogs contain package-le
44

55
## [Unreleased]
66

7+
### Changed
8+
9+
- `relayburn-cli`: `burn compare` now applies `--provider`, `--workflow`, and
10+
`--agent` filters at runtime. It still reads the current ledger snapshot
11+
(no pre-query ingest); run `burn ingest` first for freshest data.
12+
713
## [2.8.1] - 2026-05-11
814

915
### Changed

README.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,12 +123,19 @@ testing, review, exploration, docs, and refactoring.
123123
| `--since <range>` | Limit to a relative range or ISO timestamp. |
124124
| `--project <path>` | Limit to a project. |
125125
| `--session <id>` | Limit to one session. |
126+
| `--workflow <id>` | Limit to turns folded with stamp `workflowId=<id>`. |
127+
| `--agent <id>` | Limit to turns folded with stamp `agentId=<id>`. |
128+
| `--provider <csv>` | Comma-separated effective providers (case-insensitive). |
126129
| `--min-sample <n>` | Flag cells below the sample threshold. Default: `5`. |
127130
| `--fidelity <class>` | Minimum data quality: `full`, `usage-only`, `aggregate-only`, `cost-only`, or `partial`. |
128131
| `--include-partial` | Include every turn. Shorthand for `--fidelity partial`. |
129132
| `--json` | Emit a stable JSON object. |
130133
| `--csv` | Emit one row per model/activity pair. |
131134

135+
`burn compare` reads the ledger as-is — it does NOT run an ingest sweep
136+
first. Chain `burn ingest && burn compare …` (or run `burn ingest --watch`
137+
in the background) when you need the freshest data.
138+
132139
| Example | Result |
133140
|---|---|
134141
| `burn compare claude-sonnet-4-6,claude-haiku-4-5 --since 30d` | Side-by-side activity table. |

crates/relayburn-cli/src/commands/compare.rs

Lines changed: 96 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -7,15 +7,35 @@
77
//! TS source of truth: `packages/cli/src/commands/compare.ts`. The wire
88
//! shape (cells ordering, rounding rules, fidelity-summary key order)
99
//! mirrors that file byte-for-byte against the cli-golden snapshot.
10+
//!
11+
//! ## Pre-query ingest decision
12+
//!
13+
//! Unlike `burn summary` and `burn hotspots`, this command intentionally does
14+
//! NOT run a pre-query `ingest_all` sweep. Two reasons:
15+
//!
16+
//! 1. The JSON envelope above (`analyzedTurns`, `models`, `categories`,
17+
//! `cells`, `fidelity`) is byte-equivalent with the TS golden snapshot;
18+
//! bolting on an `ingest` block to mirror summary would break that.
19+
//! 2. Compare is a presenter verb — answering "given my ledger, which model
20+
//! won?". Callers who want the freshest answer can chain
21+
//! `burn ingest && burn compare`; the steady-state setup is to run
22+
//! `burn ingest --watch` once per host so every read verb sees current data.
23+
//!
24+
//! Filter wiring: `--project` / `--session` / `--since` lower into the
25+
//! `Query` struct directly; `--workflow` / `--agent` fold through
26+
//! `Query.enrichment` (`workflowId` / `agentId` keys, both required when
27+
//! both flags are passed); `--provider` is a post-query CSV allow-list
28+
//! resolved via `provider_for`, matching `summary --by-provider`'s
29+
//! synthetic-rule-aware classification.
1030
11-
use std::collections::BTreeSet;
31+
use std::collections::{BTreeMap, BTreeSet};
1232

1333
use anyhow::{anyhow, Result};
1434
use relayburn_sdk::{
15-
build_compare_table, has_minimum_fidelity, load_pricing, summarize_fidelity,
35+
build_compare_table, has_minimum_fidelity, load_pricing, provider_for, summarize_fidelity,
1636
AnalyzeCompareOptions as CompareOptions, CompareCell, CompareTable, EnrichedTurn,
17-
FidelityClass, FidelitySummary, Ledger, LedgerOpenOptions, Query, UsageGranularity,
18-
DEFAULT_MIN_SAMPLE,
37+
FidelityClass, FidelitySummary, Ledger, LedgerOpenOptions, ProviderFilter, Query,
38+
UsageGranularity, DEFAULT_MIN_SAMPLE,
1939
};
2040
use serde_json::{json, Value};
2141

@@ -111,17 +131,10 @@ fn run_inner(globals: &GlobalArgs, args: CompareArgs) -> Result<i32> {
111131
return Err(anyhow!("--json and --csv are mutually exclusive; pick one."));
112132
}
113133

114-
// 4. Provider filter. Surfaced as an explicit "not yet wired" error
115-
// rather than a silent no-op — the SDK's provider filter is private
116-
// to the analyze module today, and exposing it through a typed
117-
// top-level surface is part of the broader provider-classifier
118-
// follow-up. The cli-golden corpus exercises compare without a
119-
// provider filter, so this is unblocked for parity.
120-
if args.provider.is_some() {
121-
return Err(anyhow!(
122-
"burn compare: --provider filter is not yet wired through the Rust SDK (#246 follow-up)"
123-
));
124-
}
134+
// 4. Provider filter. Lower-cased CSV; turns whose effective provider
135+
// (per `provider_for`) isn't in the set get dropped after the ledger
136+
// query but before the fidelity gate, matching the TS pipeline order.
137+
let provider_filter = parse_provider_filter(args.provider.as_deref())?;
125138

126139
// 5. min-sample.
127140
let min_sample = args.min_sample.unwrap_or(DEFAULT_MIN_SAMPLE);
@@ -144,14 +157,19 @@ fn run_inner(globals: &GlobalArgs, args: CompareArgs) -> Result<i32> {
144157
if let Some(s) = args.session.as_deref() {
145158
q.session_id = Some(s.to_string());
146159
}
147-
// `workflow` / `agent` flow through the stamp-based enrichment filter
148-
// which the Rust ledger query layer doesn't yet expose. Surface the
149-
// gap explicitly rather than silently dropping the flag — when the
150-
// ledger gains enrichment-filter support, this branch comes out.
151-
if args.workflow.is_some() || args.agent.is_some() {
152-
return Err(anyhow!(
153-
"burn compare: --workflow / --agent filters are not yet wired through the Rust ledger query (#246 follow-up)"
154-
));
160+
// `workflow` / `agent` fold through stamp enrichment. Both keys live in
161+
// the same `Enrichment` map (`workflowId`, `agentId`), and the ledger's
162+
// `Query.enrichment` predicate requires every key/value pair to match —
163+
// so passing both narrows to the intersection.
164+
let mut enrichment = BTreeMap::new();
165+
if let Some(workflow) = args.workflow.as_deref() {
166+
enrichment.insert("workflowId".to_string(), workflow.to_string());
167+
}
168+
if let Some(agent) = args.agent.as_deref() {
169+
enrichment.insert("agentId".to_string(), agent.to_string());
170+
}
171+
if !enrichment.is_empty() {
172+
q.enrichment = Some(enrichment);
155173
}
156174

157175
// 8. Open ledger and walk turns.
@@ -165,9 +183,20 @@ fn run_inner(globals: &GlobalArgs, args: CompareArgs) -> Result<i32> {
165183
progress.set_task("loading turns");
166184
let queried_turns: Vec<EnrichedTurn> = handle.raw().query_turns(&q)?;
167185

168-
// 9. Provider filter is rejected up-front (see step 4). Pipeline
169-
// treats every queried turn as eligible.
170-
let filtered_by_provider: Vec<EnrichedTurn> = queried_turns;
186+
// 9. Drop turns whose effective provider isn't in the allow-list. The
187+
// provider is resolved via `provider_for` (synthetic-rule first,
188+
// then `provider/model` prefix, then collector-implied) so the CLI
189+
// matches `summary --by-provider` semantics 1:1.
190+
let filtered_by_provider: Vec<EnrichedTurn> = match provider_filter.as_ref() {
191+
Some(filter) => queried_turns
192+
.into_iter()
193+
.filter(|et| {
194+
let provider = provider_for(&et.turn).provider;
195+
filter.contains(&provider.to_ascii_lowercase())
196+
})
197+
.collect(),
198+
None => queried_turns,
199+
};
171200

172201
// 10. Fidelity summary is computed BEFORE the fidelity gate so the
173202
// `summary` block in the JSON envelope reflects the queried slice.
@@ -223,6 +252,26 @@ fn run_inner(globals: &GlobalArgs, args: CompareArgs) -> Result<i32> {
223252
// helpers
224253
// ---------------------------------------------------------------------------
225254

255+
/// Parse `--provider` CSV → lower-cased `ProviderFilter`. Mirrors the
256+
/// `summary --provider` parser: trim entries, drop empties, lower-case for
257+
/// case-insensitive matches, and reject an all-empty list with the same
258+
/// error shape (`burn: --provider requires a value`). Returns `Ok(None)`
259+
/// when the flag wasn't passed.
260+
fn parse_provider_filter(raw: Option<&str>) -> Result<Option<ProviderFilter>> {
261+
let Some(raw) = raw else {
262+
return Ok(None);
263+
};
264+
let providers: ProviderFilter = raw
265+
.split(',')
266+
.map(|s| s.trim().to_ascii_lowercase())
267+
.filter(|s| !s.is_empty())
268+
.collect();
269+
if providers.is_empty() {
270+
return Err(anyhow!("--provider requires a value"));
271+
}
272+
Ok(Some(providers))
273+
}
274+
226275
fn parse_fidelity(s: &str) -> Result<FidelityClass> {
227276
match s {
228277
"full" => Ok(FidelityClass::Full),
@@ -1114,6 +1163,27 @@ mod tests {
11141163
assert_eq!(v.to_string(), "0.01125");
11151164
}
11161165

1166+
#[test]
1167+
fn parse_provider_filter_trims_lowercases_and_dedupes() {
1168+
let got = parse_provider_filter(Some(" Anthropic,OPENAI ,, anthropic"))
1169+
.unwrap()
1170+
.unwrap();
1171+
assert!(got.contains("anthropic"));
1172+
assert!(got.contains("openai"));
1173+
assert_eq!(got.len(), 2, "duplicates should collapse: got {got:?}");
1174+
}
1175+
1176+
#[test]
1177+
fn parse_provider_filter_returns_none_when_flag_absent() {
1178+
assert!(parse_provider_filter(None).unwrap().is_none());
1179+
}
1180+
1181+
#[test]
1182+
fn parse_provider_filter_rejects_all_empty_input() {
1183+
let err = parse_provider_filter(Some(" , ,, ")).unwrap_err();
1184+
assert!(format!("{err}").contains("--provider requires a value"));
1185+
}
1186+
11171187
#[test]
11181188
fn parse_fidelity_known_classes() {
11191189
assert!(matches!(parse_fidelity("full").unwrap(), FidelityClass::Full));

crates/relayburn-sdk/src/lib.rs

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -89,10 +89,12 @@ pub use crate::analyze::{
8989
aggregate_by_subagent, aggregate_subagent_type_stats, attribute_hotspots, attribute_overhead,
9090
build_compare_table, build_subagent_tree, build_trim_recommendations, compare_from_archive,
9191
compute_quality, cost_for_turn, cost_for_usage, describe_applies_to, detect_patterns,
92-
detect_tool_call_patterns, detect_tool_output_bloat, find_overhead_files,
93-
findings_from_patterns, has_minimum_fidelity, load_overhead_file, load_pricing, provider_for,
94-
render_unified_diff_for_recommendation, sum_costs, summarize_fidelity,
95-
summarize_replacement_savings, AggregateByProviderOptions, AttributeOverheadInput,
92+
detect_tool_call_patterns, detect_tool_output_bloat, filter_turns_by_provider,
93+
filter_turns_by_provider_with_rules, find_overhead_files, findings_from_patterns,
94+
has_minimum_fidelity, load_overhead_file, load_pricing, provider_for, AsTurnLike,
95+
ProviderFilter, ProviderRule, render_unified_diff_for_recommendation, sum_costs,
96+
summarize_fidelity, summarize_replacement_savings, AggregateByProviderOptions,
97+
AttributeOverheadInput,
9698
AttributionMethod, BashAggregation, BashVerbAggregation, BuildSubagentTreeOptions,
9799
CompareCategory, CompareCell, CompareFromArchiveResult,
98100
CompareOptions as AnalyzeCompareOptions, CompareTable, CompareTotals, ComputeQualityOptions,

crates/relayburn-sdk/src/query_verbs.rs

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4458,6 +4458,137 @@ mod tests {
44584458
assert_eq!(missed.analyzed_turns, 0);
44594459
}
44604460

4461+
#[test]
4462+
fn compare_filters_by_folded_agent_enrichment() {
4463+
let (_dir, mut handle) = fixture_handle();
4464+
let mut enrichment = crate::Enrichment::new();
4465+
enrichment.insert("agentId".into(), "agent-7".into());
4466+
let stamp = crate::Stamp::new(
4467+
"2026-04-22T00:00:00.000Z",
4468+
crate::StampSelector {
4469+
session_id: Some("sess-a".into()),
4470+
..Default::default()
4471+
},
4472+
enrichment,
4473+
)
4474+
.unwrap();
4475+
handle.raw_mut().append_stamp(&stamp).unwrap();
4476+
4477+
let matched = handle
4478+
.compare(CompareOptions {
4479+
models: vec!["claude-sonnet-4-6".into(), "claude-haiku-4-5".into()],
4480+
agent: Some("agent-7".into()),
4481+
min_fidelity: Some(FidelityClass::Partial),
4482+
..CompareOptions::default()
4483+
})
4484+
.unwrap();
4485+
assert_eq!(matched.analyzed_turns, 2);
4486+
4487+
let missed = handle
4488+
.compare(CompareOptions {
4489+
models: vec!["claude-sonnet-4-6".into(), "claude-haiku-4-5".into()],
4490+
agent: Some("agent-missing".into()),
4491+
min_fidelity: Some(FidelityClass::Partial),
4492+
..CompareOptions::default()
4493+
})
4494+
.unwrap();
4495+
assert_eq!(missed.analyzed_turns, 0);
4496+
}
4497+
4498+
#[test]
4499+
fn compare_filters_by_effective_provider() {
4500+
// Fixture has 2 sonnet-4-6 turns (collector-implied `anthropic`).
4501+
// A matching filter keeps them; a non-matching one drops them.
4502+
let (_dir, handle) = fixture_handle();
4503+
4504+
let matched = handle
4505+
.compare(CompareOptions {
4506+
models: vec!["claude-sonnet-4-6".into(), "claude-haiku-4-5".into()],
4507+
provider: Some(vec!["anthropic".into()]),
4508+
min_fidelity: Some(FidelityClass::Partial),
4509+
..CompareOptions::default()
4510+
})
4511+
.unwrap();
4512+
assert_eq!(matched.analyzed_turns, 2);
4513+
4514+
let missed = handle
4515+
.compare(CompareOptions {
4516+
models: vec!["claude-sonnet-4-6".into(), "claude-haiku-4-5".into()],
4517+
provider: Some(vec!["openai".into()]),
4518+
min_fidelity: Some(FidelityClass::Partial),
4519+
..CompareOptions::default()
4520+
})
4521+
.unwrap();
4522+
assert_eq!(missed.analyzed_turns, 0);
4523+
4524+
// Case-insensitive: upper-case input must still match `anthropic`.
4525+
let mixed_case = handle
4526+
.compare(CompareOptions {
4527+
models: vec!["claude-sonnet-4-6".into(), "claude-haiku-4-5".into()],
4528+
provider: Some(vec!["ANTHROPIC".into()]),
4529+
min_fidelity: Some(FidelityClass::Partial),
4530+
..CompareOptions::default()
4531+
})
4532+
.unwrap();
4533+
assert_eq!(mixed_case.analyzed_turns, 2);
4534+
}
4535+
4536+
#[test]
4537+
fn compare_filters_by_workflow_and_agent_intersection() {
4538+
// `Query.enrichment` is AND-semantics: every key/value pair must
4539+
// match. Pin that here so a future drift to OR-semantics regresses
4540+
// visibly. Stamp folds both keys onto sess-a's two turns.
4541+
let (_dir, mut handle) = fixture_handle();
4542+
let mut enrichment = crate::Enrichment::new();
4543+
enrichment.insert("workflowId".into(), "wf-1".into());
4544+
enrichment.insert("agentId".into(), "agent-7".into());
4545+
let stamp = crate::Stamp::new(
4546+
"2026-04-22T00:00:00.000Z",
4547+
crate::StampSelector {
4548+
session_id: Some("sess-a".into()),
4549+
..Default::default()
4550+
},
4551+
enrichment,
4552+
)
4553+
.unwrap();
4554+
handle.raw_mut().append_stamp(&stamp).unwrap();
4555+
4556+
let matched = handle
4557+
.compare(CompareOptions {
4558+
models: vec!["claude-sonnet-4-6".into(), "claude-haiku-4-5".into()],
4559+
workflow: Some("wf-1".into()),
4560+
agent: Some("agent-7".into()),
4561+
min_fidelity: Some(FidelityClass::Partial),
4562+
..CompareOptions::default()
4563+
})
4564+
.unwrap();
4565+
assert_eq!(matched.analyzed_turns, 2);
4566+
4567+
// Workflow matches but agent does not → 0.
4568+
let agent_missing = handle
4569+
.compare(CompareOptions {
4570+
models: vec!["claude-sonnet-4-6".into(), "claude-haiku-4-5".into()],
4571+
workflow: Some("wf-1".into()),
4572+
agent: Some("agent-missing".into()),
4573+
min_fidelity: Some(FidelityClass::Partial),
4574+
..CompareOptions::default()
4575+
})
4576+
.unwrap();
4577+
assert_eq!(agent_missing.analyzed_turns, 0);
4578+
4579+
// Agent matches but workflow does not → 0.
4580+
let workflow_missing = handle
4581+
.compare(CompareOptions {
4582+
models: vec!["claude-sonnet-4-6".into(), "claude-haiku-4-5".into()],
4583+
workflow: Some("wf-missing".into()),
4584+
agent: Some("agent-7".into()),
4585+
min_fidelity: Some(FidelityClass::Partial),
4586+
..CompareOptions::default()
4587+
})
4588+
.unwrap();
4589+
assert_eq!(workflow_missing.analyzed_turns, 0);
4590+
}
4591+
44614592
#[test]
44624593
fn free_function_summary_round_trips_through_ledger_home() {
44634594
let dir = tempfile::tempdir().unwrap();

0 commit comments

Comments
 (0)