Skip to content

Add Terminal-Bench adapter (terminal-bench-core==0.1.1) - #3

Merged
Xubqpanda merged 1 commit into
zjunlp:mainfrom
Ylr9933:experiments/add-terminal-bench
Sep 23, 2026
Merged

Xubqpanda merged 1 commit into
zjunlp:mainfrom
Ylr9933:experiments/add-terminal-bench

Conversation

@Ylr9933

@Ylr9933 Ylr9933 commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Terminal-Bench (laude-institute/terminal-bench) is the docker-per-task resolution benchmark. Adds it as a Benchmark adapter in experiments/benchmark/terminal_bench/, following alfworld's shape.

Version lock — terminal-bench-core==0.1.1 (registry.json published entry):
commit 91e10457b5410f16c44364da1a34cb6de8c488a5
branch dataset/terminal-bench-core/v0.1.x
task_id_subset 80 ids (.base/.easy/.hard variants) -> 70 unique task dirs
(repo has no git tags; PyPI terminal-bench-core 404; 0.1.1 is the latest registerable release. main HEAD drifts and is not reproducible, so not used.)

Adapter surface:

  • tasks(): 70 curated dirs; SUBSET_BASES embedded as a module tuple (clone-only source, no dataset-side file). Sampling rewoo_draw(seed).
  • on_task(): builds + runs the per-task container (base image ghcr.io/laude-institute/t-bench/python-3-13, multi-arch); mounts run-tests.sh + tests/ into /app:ro; the lone exec tool docker-exec's in.
  • score(): runs the official run-tests.sh (which runs uv run pytest tests/) in the agent-modified container end-state; correctness = pytest pass (the dataset's own judge, not a reimplementation).
  • downloads(): pins the codeload tarball @ 91e10457 with the registry note.
  • Registered in run.py BENCHMARK_FACTORIES; DOWNLOADS.md regenerated to cover it.
  • .gitignore: ignore .venv/.

Gates:

  • python -m experiments.scripts.datasets --write ok
  • python -m experiments.scripts.check: 16 benches · 10 arms · 0 runs, all pass
  • docker smoke on this arm64 mac: base image builds natively in 27s (no emulation); exec + mounted run-tests path works; end-to-end react × DeepSeek 1 task finishes through official pytest judging.

Known limits:

  • ~10 heavy tasks (qemu kernel / hf-model / pytorch-model-cli ...) may not build on arm64 mac; recorded as env_unavailable. ~60/70 run here; all 70 should run on x86 linux.
  • per-task ~4-7 min (apt/uv install pytest inside score can't be cached); 70 tasks ≈ 3-6 h.
  • .easy/.hard variants not split; each unique dir runs once.

What this changes

How you know it works

Checklist

  • npm run check passes
  • npm run typecheck passes
  • npm test passes
  • npm run demo runs to completion
  • I ran the code in this PR myself
  • Every claim above has a command behind it
  • No new dependencies — or, if there are, the PR says why the rule in CONTRIBUTING.md does not apply here

Terminal-Bench (laude-institute/terminal-bench) is the docker-per-task
resolution benchmark. Adds it as a Benchmark adapter in
experiments/benchmark/terminal_bench/, following alfworld's shape.

Version lock — terminal-bench-core==0.1.1 (registry.json published entry):
  commit 91e10457b5410f16c44364da1a34cb6de8c488a5
  branch dataset/terminal-bench-core/v0.1.x
  task_id_subset 80 ids (.base/.easy/.hard variants) -> 70 unique task dirs
(repo has no git tags; PyPI terminal-bench-core 404; 0.1.1 is the latest
registerable release. main HEAD drifts and is not reproducible, so not used.)

Adapter surface:
- tasks(): 70 curated dirs; SUBSET_BASES embedded as a module tuple
  (clone-only source, no dataset-side file). Sampling rewoo_draw(seed).
- on_task(): builds + runs the per-task container (base image
  ghcr.io/laude-institute/t-bench/python-3-13, multi-arch); mounts
  run-tests.sh + tests/ into /app:ro; the lone `exec` tool docker-exec's in.
- score(): runs the official run-tests.sh (which runs `uv run pytest tests/`)
  in the agent-modified container end-state; correctness = pytest pass
  (the dataset's own judge, not a reimplementation).
- downloads(): pins the codeload tarball @ 91e10457 with the registry note.
- Registered in run.py BENCHMARK_FACTORIES; DOWNLOADS.md regenerated to cover it.
- .gitignore: ignore .venv/.

Gates:
- `python -m experiments.scripts.datasets --write` ok
- `python -m experiments.scripts.check`: 16 benches · 10 arms · 0 runs, all pass
- docker smoke on this arm64 mac: base image builds natively in 27s (no
  emulation); exec + mounted run-tests path works; end-to-end
  react × DeepSeek 1 task finishes through official pytest judging.

Known limits:
- ~10 heavy tasks (qemu kernel / hf-model / pytorch-model-cli ...) may
  not build on arm64 mac; recorded as env_unavailable. ~60/70 run here;
  all 70 should run on x86 linux.
- per-task ~4-7 min (apt/uv install pytest inside score can't be cached);
  70 tasks ≈ 3-6 h.
- .easy/.hard variants not split; each unique dir runs once.
@Xubqpanda
Xubqpanda merged commit 3a57592 into zjunlp:main Sep 23, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants