Skip to content

Commit 6b43232

Browse files
committed
Agent benchmarks, Python indexer improvements, bump to 4.13.0
- Add agent-level benchmark suite: 25 tasks across 5 repos measuring Claude Code token consumption with vs without wonk (37.4% reduction) - Update README benchmarks section with agent benchmark results - Index Python class-level annotated assignments (e.g. pydantic fields) - Split Python import_from_statement into per-name references - Return Vec<Reference> from match_import_ref for multi-name imports - Add .gitignore entry for __pycache__/
1 parent a7f9f20 commit 6b43232

14 files changed

Lines changed: 2970 additions & 88 deletions

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,3 +2,4 @@
22
.worktrees
33
bench/repos/
44
bench/results/
5+
__pycache__/

Cargo.lock

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Cargo.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[package]
22
name = "wonk"
3-
version = "4.12.1"
3+
version = "4.13.0"
44
edition = "2024"
55

66
[lib]

README.md

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -82,17 +82,19 @@ Wonk pre-indexes your codebase with Tree-sitter so it understands code structure
8282

8383
## Benchmarks
8484

85-
25 queries across 5 repos (ripgrep, tokio, Flask, Django, Express):
85+
25 code-understanding tasks across 5 real-world repos (ripgrep, tokio, httpx, pydantic, fastify), 5 runs each, median reported. Measures Claude Code token consumption with vs without wonk.
8686

87-
| Comparison | Avg token reduction |
88-
|------------|---------------------|
89-
| wonk toon vs rg --json | **60%** |
90-
| wonk json vs rg --json | **57%** |
91-
| wonk grep vs plain rg | **3%** (grep format is already compact) |
87+
| Category | Baseline (avg) | Wonk (avg) | Reduction | Quality (B→W) |
88+
|----------|---------------:|-----------:|----------:|--------------:|
89+
| symbol_location | 100k | 61k | 33% | 0.85→0.85 |
90+
| reference_tracing | 96k | 57k | 28% | 0.92→0.88 |
91+
| architecture | 162k | 101k | 29% | 0.90→0.96 |
92+
| multi_step | 143k | 104k | 23% | 0.93→0.93 |
93+
| structural | 130k | 69k | 46% | 0.95→0.88 |
9294

93-
Definitions found in 92% of queries. Average of 73 re-export dedup annotations per query.
95+
**Overall:** 37.4% total reduction (median per-task 29.7%, best 68.5%). Quality maintained at 0.90 vs 0.91 baseline.
9496

95-
Full results: [`bench/results/report.md`](bench/results/report.md)
97+
Full results: [`bench/results/agent_report.md`](bench/results/agent_report.md)
9698

9799
## Installation
98100

bench/README.md

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
# Token Savings Benchmark
2+
3+
Compares `wonk search` against raw `rg` (ripgrep) across 5 real open-source
4+
codebases. Both tools run from the repo root, producing `file:line:content`
5+
output so byte/token comparisons are apples-to-apples.
6+
7+
## Prerequisites
8+
9+
- `wonk` on PATH (or set `WONK=/path/to/wonk`)
10+
- `rg` (ripgrep) on PATH (or set `RG=/path/to/rg`)
11+
- `git` on PATH
12+
- ~2 GB disk for cloned repos
13+
14+
## Usage
15+
16+
```bash
17+
# From the repo root:
18+
cargo build --release
19+
cd bench
20+
./token_bench.sh
21+
```
22+
23+
The script will:
24+
25+
1. Shallow-clone 5 repos to `bench/repos/` (skips if already present)
26+
2. Run `wonk init` on each repo
27+
3. Run 5 queries per repo through both `rg` and `wonk search`
28+
4. Run a budget sweep (500/1000/2000/4000 tokens) for each query
29+
5. Print markdown tables to stdout and save to `bench/results/report.md`
30+
31+
## What it measures
32+
33+
| Metric | Description |
34+
|--------|-------------|
35+
| Token count | `ceil(bytes / 4)` — same heuristic wonk uses internally |
36+
| Line count | Number of result lines from each tool |
37+
| Reduction % | `(rg_tokens - wonk_tokens) / rg_tokens * 100` |
38+
| Defs found | Whether wonk's `-- definitions --` section appeared |
39+
| Dedup count | Number of `(+N other locations)` annotations |
40+
| Budget fit | What % of raw rg output fits in a given token budget |
41+
42+
## Repos and queries
43+
44+
| Repo | Language | Queries |
45+
|------|----------|---------|
46+
| BurntSushi/ripgrep | Rust | search, match, regex, parse, printer |
47+
| tokio-rs/tokio | Rust | spawn, runtime, task, poll, waker |
48+
| pallets/flask | Python | route, request, response, app, Blueprint |
49+
| django/django | Python | Model, QuerySet, view, middleware, Field |
50+
| expressjs/express | JavaScript | Router, middleware, request, response, next |
51+
52+
## Runtime
53+
54+
Expect ~5 minutes on first run (dominated by cloning django at ~250K LOC).
55+
Subsequent runs skip cloning and take ~2 minutes.
56+
57+
## Output
58+
59+
Results are written to `bench/results/report.md` (gitignored) and also
60+
printed to stdout.

0 commit comments

Comments
 (0)