Skip to content

docs: describe tilth by what it does, retire the cost claim - #231

Merged
jahala merged 2 commits into
mainfrom
docs/retire-cost-claim
Sep 19, 2026
Merged

jahala merged 2 commits into
mainfrom
docs/retire-cost-claim

Conversation

@jahala

@jahala jahala commented Sep 19, 2026

Copy link
Copy Markdown
Owner

Summary

The README, the landing page and the package descriptions led with "−40% cost per correct answer". This PR removes every cost, accuracy and turn figure and describes tilth by what it does.

Why the number goes. It was measured in March 2026 on v0.5.0 with models two generations old, as point estimates with no interval. Regrouping our own 3,791 stored runs on matched tasks flips its sign by period. The grading audit of July 2026 found defects in the harness. No tilth-versus-baseline run exists on a current model. We have nothing measured to put in its place, so the README says that in a short note and makes no cost claim.

What replaces it. tilth parses code and computes answers that text search has to infer: where a symbol is defined, who calls it, what it calls, what depends on a file, and what changed at function level. Anyone can check that by running the tool. The limits are stated too: tilth parses syntax, does not resolve types, and matches callers by name.

Changes

  • README.md: rewritten opening, a "What it doesn't do" section, a note on the retired benchmark. Every example is now real output from the FastAPI and Gin repositories, cut only at whole lines marked .... The old examples were hand-written and no longer matched the tool (faceted search sections, ASCII -> and -- calls --, the deps and grok layouts).
  • index.html: the benchmark table becomes "questions grep can't answer", four rows of explanation plus real output. The hero shows a real callers query. Ligatures are off in code so -> and -- render as the tool prints them. The install grid no longer scrolls sideways on phones. The garden footer gains pollen and weeder, with accents from jahala/plotplot.
  • Facts corrected: about 35,000 lines of Rust (was 20,000), 17 languages with Bash (was 16), and the claim that search time is constant regardless of repository size, which stopped being true for content search in fix(search): disclose the display cap, bound what content collects #220. The speed table is re-measured: median of 15 runs of the release build on an Apple M5 Pro, on Gin (130 files) and FastAPI (2,700 files).
  • Cargo.toml, npm/package.json: descriptions drop "indexed" (tilth has no index) and gain a homepage. npm/README.md is added; package.json already listed it.
  • benchmark/ removed (second commit). The harness is no longer run, and A/B tests moved to copeca. The last commit containing it is tagged benchmark-archive. The four pinned fixture repositories move to scripts/fixtures/setup_repos.py, a standalone script that re-pins with fetch and checkout and never deletes. CLAUDE.md, ARCHITECTURE.md, dependabot.yml, .gitignore and a Cargo profile comment drop their mentions.

Test plan

  • cargo test: 680 passed
  • cargo fmt --check
  • scripts/regen-agents-md.sh is a no-op
  • scripts/fixtures/setup_repos.py: fresh clone, idempotent rerun, unknown name (exit 2), re-pin of a clone on another commit
  • Landing page checked in a browser at 1440 px and 390 px, light and dark: no horizontal overflow, no clipped lines in the output panes

Follow-ups, not in this PR

🤖 Generated with Claude Code

jahala and others added 2 commits September 19, 2026 12:30
The README, the landing page and the package descriptions led with
"-40% cost per correct answer". That number was measured in March 2026
on v0.5.0 against models two generations old, as point estimates with
no interval. Regrouping our own 3,791 stored runs on matched tasks
flips its sign by period, the grading audit found harness defects, and
no tilth-versus-baseline run exists on a current model. We cannot stand
behind it, and we have nothing measured to put in its place, so every
cost, accuracy and turn figure is removed and the README says so in a
short note.

What replaces it is what anyone can check by running the tool: tilth
parses code and computes answers that text search has to infer. The
README examples are now real output from the FastAPI and Gin fixture
repositories. The old ones were hand-written and had drifted from the
current format (faceted search sections, the ASCII `-> ` and
`-- calls --` markers, the deps and grok layouts). The landing page
swaps the benchmark table for four question-and-output rows, and the
hero shows a real callers query.

Facts that had drifted are corrected: about 35,000 lines of Rust (was
20,000), 17 languages with Bash (was 16), and the claim that search
time is constant regardless of repository size, which stopped being
true for content search when #220 made the walk complete. The speed
table is re-measured on the release build: median of 15 runs on an
Apple M5 Pro, on Gin (130 files) and FastAPI (2,700 files).

Also on the page: ligatures are off in code, because the font turned
the tool's plain `->` and `--` into arrows and dashes; the install grid
no longer scrolls sideways on phones; the garden footer gains pollen
and weeder with the accents from the plotplot umbrella repo. npm gets
the README its package.json already listed, and both manifests get a
homepage.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The harness under benchmark/ produced the cost figures that the previous
commit removes, and it is no longer run: A/B tests moved to copeca, and
the grading audit of July 2026 found defects in it (only the last
assistant turn was parsed, a quarter of stored answers were truncated,
three ground-truth strings were wrong). Keeping it on main invites new
numbers from a tool we do not trust. The last commit that contains it
is to be tagged `benchmark-archive`, so the tasks and scripts stay
reachable.

One part is still needed. The four fixture repositories at pinned
commits are what the README examples and the speed table were produced
on, and the output-parity check in the open garden work clones them
too. Their pins move to scripts/fixtures/setup_repos.py, a standalone
script with no imports from the harness. It re-pins an existing clone
with fetch and checkout where the old script deleted and re-cloned, it
defaults to a directory outside the checkout so a fixture never sees
tilth's CLAUDE.md, and it runs on Python 3.9.

CLAUDE.md loses its Benchmarks section and now points instruction
changes at a copeca A/B. ARCHITECTURE.md, dependabot.yml, .gitignore
and a Cargo profile comment drop their mentions. CLAUDE.md also said
install.rs covers 6 hosts; it covers 22.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@jahala
jahala merged commit 237c780 into main Sep 19, 2026
8 checks passed
@jahala
jahala deleted the docs/retire-cost-claim branch September 19, 2026 11:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant