Add Terminal-Bench adapter (terminal-bench-core==0.1.1) - #3
Merged
Merged
Conversation
Terminal-Bench (laude-institute/terminal-bench) is the docker-per-task resolution benchmark. Adds it as a Benchmark adapter in experiments/benchmark/terminal_bench/, following alfworld's shape. Version lock — terminal-bench-core==0.1.1 (registry.json published entry): commit 91e10457b5410f16c44364da1a34cb6de8c488a5 branch dataset/terminal-bench-core/v0.1.x task_id_subset 80 ids (.base/.easy/.hard variants) -> 70 unique task dirs (repo has no git tags; PyPI terminal-bench-core 404; 0.1.1 is the latest registerable release. main HEAD drifts and is not reproducible, so not used.) Adapter surface: - tasks(): 70 curated dirs; SUBSET_BASES embedded as a module tuple (clone-only source, no dataset-side file). Sampling rewoo_draw(seed). - on_task(): builds + runs the per-task container (base image ghcr.io/laude-institute/t-bench/python-3-13, multi-arch); mounts run-tests.sh + tests/ into /app:ro; the lone `exec` tool docker-exec's in. - score(): runs the official run-tests.sh (which runs `uv run pytest tests/`) in the agent-modified container end-state; correctness = pytest pass (the dataset's own judge, not a reimplementation). - downloads(): pins the codeload tarball @ 91e10457 with the registry note. - Registered in run.py BENCHMARK_FACTORIES; DOWNLOADS.md regenerated to cover it. - .gitignore: ignore .venv/. Gates: - `python -m experiments.scripts.datasets --write` ok - `python -m experiments.scripts.check`: 16 benches · 10 arms · 0 runs, all pass - docker smoke on this arm64 mac: base image builds natively in 27s (no emulation); exec + mounted run-tests path works; end-to-end react × DeepSeek 1 task finishes through official pytest judging. Known limits: - ~10 heavy tasks (qemu kernel / hf-model / pytorch-model-cli ...) may not build on arm64 mac; recorded as env_unavailable. ~60/70 run here; all 70 should run on x86 linux. - per-task ~4-7 min (apt/uv install pytest inside score can't be cached); 70 tasks ≈ 3-6 h. - .easy/.hard variants not split; each unique dir runs once.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Terminal-Bench (laude-institute/terminal-bench) is the docker-per-task resolution benchmark. Adds it as a Benchmark adapter in experiments/benchmark/terminal_bench/, following alfworld's shape.
Version lock — terminal-bench-core==0.1.1 (registry.json published entry):
commit 91e10457b5410f16c44364da1a34cb6de8c488a5
branch dataset/terminal-bench-core/v0.1.x
task_id_subset 80 ids (.base/.easy/.hard variants) -> 70 unique task dirs
(repo has no git tags; PyPI terminal-bench-core 404; 0.1.1 is the latest registerable release. main HEAD drifts and is not reproducible, so not used.)
Adapter surface:
exectool docker-exec's in.uv run pytest tests/) in the agent-modified container end-state; correctness = pytest pass (the dataset's own judge, not a reimplementation).Gates:
python -m experiments.scripts.datasets --writeokpython -m experiments.scripts.check: 16 benches · 10 arms · 0 runs, all passKnown limits:
What this changes
How you know it works
Checklist
npm run checkpassesnpm run typecheckpassesnpm testpassesnpm run demoruns to completionCONTRIBUTING.mddoes not apply here