A repair claim needs current evidence.
Software repairs, role workflows, verification receipts and completion claims in one evaluation tool.
Built for the micro1 Eval Hackathon 2026
Software-repair evaluation · By Desan Vasu
Features · Demo · Watch video · Get started · DeepSeek & execution · Documentation
Actual CLI rendering · recorded provider-backed attempts, replayed for inspection.
A coding agent can repair a program and still rely on an old check, skip an independent review, or claim more than its evidence supports. A single pass/fail score hides those differences.
FollowThrough keeps the repaired source, its requirement version and the evidence behind each claim inspectable. It evaluates functional correctness, workflow adherence, verification currency and completion separately, while retaining infrastructure and provider failures as distinct outcomes.
Domain focus: software maintenance, transactional data integrity and cache correctness. The two original task families turn practical repair and verification concerns into executable cases: interrupted retry processing and preview caching after requirements change. Task examples and rationale →
|
Inspect functional, workflow, evidence and completion judgments together. An independent functional pass remains visible even when the overall episode is excluded. |
Candidate SHA-256 and requirement versions identify what a public check, review or protected grade actually covered. Stale evidence cannot silently become current verification. |
|
R1/R2/R3 cover Retry; C1/C2/C3 cover Preview Cache. Ordinary and structured-progress arms share the same task requirements, including changed requirements and authored stale receipts. |
Evaluated roles receive visible workspace files and public checks. Trusted functional grading runs separately against the identified candidate and requirement version. |
|
Navigate the dark/cyan T3 ledger, filter attempts, open saved evidence and inspect resource accounting. Text and JSON modes support non-interactive use. |
Generate JSON, CSV and Markdown reports plus a portable read-only browser view. Estimates, reservation allowances, exclusions and unknowns retain separate labels. |
Watch the narrated 4:45 walkthrough on YouTube → · Submitted public-player mirror · MP4 and captions
The included saved-report input contains three actual DeepSeek-backed attempts. Opening the viewer makes no fresh model request and runs no candidate code.
- Open the actual T3 ledger and select an attempt with
j/kor the arrows. - Inspect all four outcome dimensions and the attempt's resource fields.
- Press
Enterto inspect saved evidence, or/to filter by episode ID, scenario, arm or status. - Compare the same records in the portable browser report.
- Reproduce the authored grader controls and inspect the retained model-output artifacts.
Evidence beside the outcome. Inspect the recorded events supporting a selected attempt. |
A focused terminal view. Filter the retained records without triggering execution. |
The same saved evidence. A portable browser report requires no account or provider access. |
An observed repair trajectory. Two actual saved states were independently assessed posthoc: 6/8 → 8/8. This is not exact onset or a causal workflow comparison. Inspect the source-bound assessment → |
Capture provenance and attribution →
You need Python 3.11+, Git or the source ZIP, and a supported interactive terminal. The saved-results demo uses the Python standard library: no API key, Docker, paid credits or package installation is required.
git clone https://github.com/desanv01/FollowThrough.git
cd FollowThrough
python -B -m reporting.run_ledger --input docs/research/recorded-development-input.json --format tuiUse j/k or arrows to select, Enter for evidence, / to filter, Esc to return and q to close. For logs or an unsupported interactive terminal, use --format text or --format json.
python -B grader/retry_processing.py --self-test
python -B grader/preview_cache.py --self-testThese execute the trusted original fixtures and their intentional defects. Their output distinguishes functional failures, candidate execution errors and infrastructure errors. Evaluated candidate code uses the isolated execution path; the recorded viewer never executes it.
Use a fresh output directory:
python -B -m reporting.report --input docs/research/recorded-development-input.json --output-dir runs/recorded-reportClean setup, browser-viewer command and judging walkthrough →
The adapter calls the official DeepSeek API through separate role histories, exact pre-dispatch reservations and bounded operation limits. The recorded requested and returned alias is deepseek-flash; no immutable provider model snapshot is inferred from that alias.
Credentials remain in an ignored trusted local environment file. A canonical ledger preserves reservation history across operations, and source/run/version bindings keep public checks and protected grades associated with the intended candidate. Docker provides the isolated candidate runtime; the explicit Python public-check backend has native positive and negative control evidence. Native controls →
Fresh provider execution requires separately configured credentials, qualified runtime access, a reviewed operation gate and budget authorization. See the evaluation guide, Python backend and live driver. No key, private ledger or raw private conversation is shipped in the repository.
All three retained attempts received independent functional passes. The original episode was excluded after a public-check infrastructure error; the two later attempts were excluded after invalid provider content. These are development/incomplete-pilot observations, not a frozen formal study or evidence of a coding-capability failure. Results, resources and exact limitations →
The recorded output bundle includes saved parsed model actions, actual candidate source, raw public-check output, model aliases and source hashes. Exact provider wire responses and the rejected final payloads were not retained; the bundle identifies this boundary explicitly.
| Interface | Evaluation & grading | Runtime & accounting | Model access |
|---|---|---|---|
| Python terminal ledger · portable HTML report | Python 3.11 · original Retry and Cache fixtures | Rust controller · Docker · SQLite reservation ledger | Official DeepSeek API · strict JSON-action protocol |
Hosted source CI runs on Ubuntu and Windows. Source packaging records the exact commit, Git tree, ZIP digest and tracked-file inventory. Scoped delivery records →
Original architecture illustration · model actions, trusted checks and outcome judgments remain distinct.
The contribution to micro1 Eval Hackathon 2026 is the original Retry and Preview Cache task/grader design, six scenario packages, bounded role/provider/runtime integration, source-version evidence binding and inspectable terminal/browser reports, developed on 7–9 November 2026. No previous implementation, employer/client data or unrelated project fixture was copied into the tasks. Codex and human-launched DeepSeek harness workers assisted implementation with separate review. Dated contribution ledger →
Desan Vasu · Software-repair evaluation and agent-workflow verification. The video and included records show the actual supported inspection and grading paths; exact observed results and their provenance boundaries remain linked above. No professional tenure or external credential is claimed.
- Setup and testing
- Task examples and scenario packages
- Methods and outcome definitions
- Recorded development evidence
- Terminal ledger and browser viewer
- Contribution and prior-work attribution
Open an issue with the expected behavior and reproducible steps. Keep credentials and private documents out of public reports.
Original FollowThrough code and documentation are MIT licensed. Dependencies and other third-party material retain their own rights and terms; see NOTICE. The owner changed the project licence before submission; the approved video retains the earlier AGPL licence card, which describes its recorded snapshot.



