docs: describe tilth by what it does, retire the cost claim - #231
Merged
Merged
Conversation
The README, the landing page and the package descriptions led with "-40% cost per correct answer". That number was measured in March 2026 on v0.5.0 against models two generations old, as point estimates with no interval. Regrouping our own 3,791 stored runs on matched tasks flips its sign by period, the grading audit found harness defects, and no tilth-versus-baseline run exists on a current model. We cannot stand behind it, and we have nothing measured to put in its place, so every cost, accuracy and turn figure is removed and the README says so in a short note. What replaces it is what anyone can check by running the tool: tilth parses code and computes answers that text search has to infer. The README examples are now real output from the FastAPI and Gin fixture repositories. The old ones were hand-written and had drifted from the current format (faceted search sections, the ASCII `-> ` and `-- calls --` markers, the deps and grok layouts). The landing page swaps the benchmark table for four question-and-output rows, and the hero shows a real callers query. Facts that had drifted are corrected: about 35,000 lines of Rust (was 20,000), 17 languages with Bash (was 16), and the claim that search time is constant regardless of repository size, which stopped being true for content search when #220 made the walk complete. The speed table is re-measured on the release build: median of 15 runs on an Apple M5 Pro, on Gin (130 files) and FastAPI (2,700 files). Also on the page: ligatures are off in code, because the font turned the tool's plain `->` and `--` into arrows and dashes; the install grid no longer scrolls sideways on phones; the garden footer gains pollen and weeder with the accents from the plotplot umbrella repo. npm gets the README its package.json already listed, and both manifests get a homepage. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The harness under benchmark/ produced the cost figures that the previous commit removes, and it is no longer run: A/B tests moved to copeca, and the grading audit of July 2026 found defects in it (only the last assistant turn was parsed, a quarter of stored answers were truncated, three ground-truth strings were wrong). Keeping it on main invites new numbers from a tool we do not trust. The last commit that contains it is to be tagged `benchmark-archive`, so the tasks and scripts stay reachable. One part is still needed. The four fixture repositories at pinned commits are what the README examples and the speed table were produced on, and the output-parity check in the open garden work clones them too. Their pins move to scripts/fixtures/setup_repos.py, a standalone script with no imports from the harness. It re-pins an existing clone with fetch and checkout where the old script deleted and re-cloned, it defaults to a directory outside the checkout so a fixture never sees tilth's CLAUDE.md, and it runs on Python 3.9. CLAUDE.md loses its Benchmarks section and now points instruction changes at a copeca A/B. ARCHITECTURE.md, dependabot.yml, .gitignore and a Cargo profile comment drop their mentions. CLAUDE.md also said install.rs covers 6 hosts; it covers 22. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The README, the landing page and the package descriptions led with "−40% cost per correct answer". This PR removes every cost, accuracy and turn figure and describes tilth by what it does.
Why the number goes. It was measured in March 2026 on v0.5.0 with models two generations old, as point estimates with no interval. Regrouping our own 3,791 stored runs on matched tasks flips its sign by period. The grading audit of July 2026 found defects in the harness. No tilth-versus-baseline run exists on a current model. We have nothing measured to put in its place, so the README says that in a short note and makes no cost claim.
What replaces it. tilth parses code and computes answers that text search has to infer: where a symbol is defined, who calls it, what it calls, what depends on a file, and what changed at function level. Anyone can check that by running the tool. The limits are stated too: tilth parses syntax, does not resolve types, and matches callers by name.
Changes
.... The old examples were hand-written and no longer matched the tool (faceted search sections, ASCII->and-- calls --, the deps and grok layouts).->and--render as the tool prints them. The install grid no longer scrolls sideways on phones. The garden footer gains pollen and weeder, with accents fromjahala/plotplot.homepage.npm/README.mdis added;package.jsonalready listed it.benchmark-archive. The four pinned fixture repositories move toscripts/fixtures/setup_repos.py, a standalone script that re-pins with fetch and checkout and never deletes.CLAUDE.md,ARCHITECTURE.md,dependabot.yml,.gitignoreand a Cargo profile comment drop their mentions.Test plan
cargo test: 680 passedcargo fmt --checkscripts/regen-agents-md.shis a no-opscripts/fixtures/setup_repos.py: fresh clone, idempotent rerun, unknown name (exit 2), re-pin of a clone on another commitFollow-ups, not in this PR
scripts/fixtures/setup_repos.py.skills/SKILL.mdstill shows── calls ──; the tool prints-- calls --. feat(mcp): context diet — a pointer at initialize, the guide in skills/SKILL.md #211 rewrites that file.// //), fromsrc/read/outline/code.rs.🤖 Generated with Claude Code