Repository navigation
Task/environment refactor: task directories, Harbor as environments, every agent Hermit's (#112-#116) - #117
Merged
Merged
Conversation
…tructure only) (#112) * Tasks and environments beside the agents: hermit/task, hermit/environment, hermit/common The first of four stacked changes toward Harbor-format tasks run by Hermit's own agents (design: "Hermit agent 与 benchmark 重构设计", 第二阶段). Structure only: no behavior changes, the requests every agent sends are byte for byte main's. - hermit/environment/: the environments move out of hermit/agent/ to the top level, since a task declares what it needs of one and a judge may run in one; hermit.agent.environment forwards for one release. - hermit/task/: task.py becomes a package. Task holds a public half (what the acting agent is handed, what a run records as its input) and a private half only the judge reads; Benchmark is a task set and its splits; verify.py wraps a judge as a verifier of a Task. Every name importable from hermit.task before still is. - hermit/common.py: RunOutput and load_callable, shared by agents and tasks so that neither imports the other. - The run (hermit/trainer/rollout.py) hands the agent task.given() and the verifier the whole task; the trainer, eval and hermit check read a Benchmark. The validation file the learner gets holds each task's public half. - tests/core/test_layers.py: tasks and agents never import each other; the environments import neither. tests/core/test_task.py covers the new types. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * hermit check: report the probe task by its id (a Task, not a row) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 1: a bare judge reads the whole row again; records keep the row's id - judge_verifier calls the judge with Task.row() (the public half and `_private`), as run_one did before tasks were typed; the agent still gets only the public half. - Records keep the row's own id (7, not "7"); run_tasks keys its results by task. - run_one/run_tasks keep the `judge` parameter name; the Verifier protocol states `takes_task`; agent-layers.md shows hermit/environment/ beside hermit/agent/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rollout: the record's id in one name (line length) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * eval: a record's numeric id finds its task (after the record kept the row's id) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…, the bwrap backend; OfficeQA as task directories (#113) * Tasks and environments beside the agents: hermit/task, hermit/environment, hermit/common The first of four stacked changes toward Harbor-format tasks run by Hermit's own agents (design: "Hermit agent 与 benchmark 重构设计", 第二阶段). Structure only: no behavior changes, the requests every agent sends are byte for byte main's. - hermit/environment/: the environments move out of hermit/agent/ to the top level, since a task declares what it needs of one and a judge may run in one; hermit.agent.environment forwards for one release. - hermit/task/: task.py becomes a package. Task holds a public half (what the acting agent is handed, what a run records as its input) and a private half only the judge reads; Benchmark is a task set and its splits; verify.py wraps a judge as a verifier of a Task. Every name importable from hermit.task before still is. - hermit/common.py: RunOutput and load_callable, shared by agents and tasks so that neither imports the other. - The run (hermit/trainer/rollout.py) hands the agent task.given() and the verifier the whole task; the trainer, eval and hermit check read a Benchmark. The validation file the learner gets holds each task's public half. - tests/core/test_layers.py: tasks and agents never import each other; the environments import neither. tests/core/test_task.py covers the new types. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * hermit check: report the probe task by its id (a Task, not a row) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 1: a bare judge reads the whole row again; records keep the row's id - judge_verifier calls the judge with Task.row() (the public half and `_private`), as run_one did before tasks were typed; the agent still gets only the public half. - Records keep the row's own id (7, not "7"); run_tasks keys its results by task. - run_one/run_tasks keep the `judge` parameter name; the Verifier protocol states `takes_task`; agent-layers.md shows hermit/environment/ beside hermit/agent/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rollout: the record's id in one name (line length) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * eval: a record's numeric id finds its task (after the record kept the row's id) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Task directories in Harbor's format, their own verifier, the bwrap backend; OfficeQA as task directories The second of four stacked changes (design: 第二阶段). A task set may now be a directory of task directories in Harbor's format, run on Hermit's own sandbox (B-class: Harbor's format, not Harbor's runtime). - hermit/task/directory.py: Harbor's task format read without Harbor (moved from hermit/harbor/dataset.py, which imports it): the instruction (line endings kept), settings and metadata public; the task's directory and tests private. Benchmark loads `data.tasks: <directory>` (`data.rollouts: k`). - hermit/task/verify.py: a task's own verifier, `judge: {path: hermit.task.verify, name: script}`, run in the judge's sandbox as Harbor runs it: tests at /tests, a fresh /logs/verifier, reward.txt or reward.json, and feedback.txt for learning methods. A judge is called with Task.row() (the row plus `_private`); an agent still gets only the public half. - hermit/environment/spec.py and bwrap.py: EnvironmentSpec (the run config's `environment` section) and the bubblewrap backend the task sandbox is built by; a task asking for more network or memory than the run gives is refused. - examples/officeqa: the task set moves to utils/benchmark/ (prepare.py writes task directories; judge.py, task-info/, splits, data.json, seed adapter); the agents read each task as the row they always read (officeqa_fields.py), so the prompts are byte for byte the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 1: OfficeQA's methods on the new layout; task directories as Harbor reads them OfficeQA (the methods the move missed): ReHarness's launcher tests the tasks directory; RRSI's rows (byte-identical to before on the real data) and AHE's come from the task directories through officeqa_fields.question_rows; RRSI and AHE score with utils/benchmark/judge.py; Harness²'s source hook reads a directory row; OPRO shows the instruction; utils/.gitignore keeps the old paths ignored; AHE's real-data tests find the prepared data again. Task directories: a step's [verifier] settings come before the task's; [verifier.env] is set for the script (${NAME} resolved as Harbor does); reward.json takes a boolean as Harbor does; a task's [agent] timeout_sec bounds its calls and the judge waits at least the verifier's timeout_sec; an allowlist task's hosts and (with the script verifier) its verifier's network are checked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 2: a view of OfficeQA's rows for agents and learners; Harbor's verifier defaults and allowlists - data.view: <file.py>:<function> turns a task directory's row into the row the task set's agents and learners read. OfficeQA's configs view each task as SkillOpt's row, so records and what a learning method reads (GEPA, ACE, OPRO, SkillOpt) are the same as with its tasks.jsonl (246/246 on the real data); OPRO's question_field is `question` again. - What a task asks of its environment is kept in its private half (`needs`) and checked before each call (hermit.trainer.rollout asks a sandboxed agent's `refuses`), viewed or not. - As Harbor: a verifier's timeout_sec defaults to 600; [verifier.environment] implies a separate verifier; each phase's allowlist (a step's first) is its own; wildcard hosts are refused under an allowlist run (the proxy matches hosts as written). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 3: a step's verifier as Harbor resolves it; a viewed task keeps its timeout; finite rewards - A step's verifier mode and environment are resolved as Harbor's verifier_mode.py does (explicit mode, else an environment implies separate, the step's level before the task's, else shared), and the verifier's network and allowlist come from that effective environment. - A step's tests/ are laid over the task's in the judge's sandbox (its test.sh first), as Harbor uploads them. - A task's agent timeout is kept in its needs, which the run hands the sandboxed agent, so a data.view that leaves it out of the row still bounds the call. - reward.txt must hold a finite number, as in Harbor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 4: multi-step tasks refused on this path; the verifier's workspace and memory - A multi-step task directory is refused here (its steps share one environment in order, which a per-call sandbox does not give); PR 4 brings that. - The script verifier's workspace defaults to the directory the agent worked in, when its output names none. - A separate verifier's own memory_mb is checked against the run's limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ent is Hermit's (#114) * Tasks and environments beside the agents: hermit/task, hermit/environment, hermit/common The first of four stacked changes toward Harbor-format tasks run by Hermit's own agents (design: "Hermit agent 与 benchmark 重构设计", 第二阶段). Structure only: no behavior changes, the requests every agent sends are byte for byte main's. - hermit/environment/: the environments move out of hermit/agent/ to the top level, since a task declares what it needs of one and a judge may run in one; hermit.agent.environment forwards for one release. - hermit/task/: task.py becomes a package. Task holds a public half (what the acting agent is handed, what a run records as its input) and a private half only the judge reads; Benchmark is a task set and its splits; verify.py wraps a judge as a verifier of a Task. Every name importable from hermit.task before still is. - hermit/common.py: RunOutput and load_callable, shared by agents and tasks so that neither imports the other. - The run (hermit/trainer/rollout.py) hands the agent task.given() and the verifier the whole task; the trainer, eval and hermit check read a Benchmark. The validation file the learner gets holds each task's public half. - tests/core/test_layers.py: tasks and agents never import each other; the environments import neither. tests/core/test_task.py covers the new types. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * hermit check: report the probe task by its id (a Task, not a row) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 1: a bare judge reads the whole row again; records keep the row's id - judge_verifier calls the judge with Task.row() (the public half and `_private`), as run_one did before tasks were typed; the agent still gets only the public half. - Records keep the row's own id (7, not "7"); run_tasks keys its results by task. - run_one/run_tasks keep the `judge` parameter name; the Verifier protocol states `takes_task`; agent-layers.md shows hermit/environment/ beside hermit/agent/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rollout: the record's id in one name (line length) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * eval: a record's numeric id finds its task (after the record kept the row's id) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Task directories in Harbor's format, their own verifier, the bwrap backend; OfficeQA as task directories The second of four stacked changes (design: 第二阶段). A task set may now be a directory of task directories in Harbor's format, run on Hermit's own sandbox (B-class: Harbor's format, not Harbor's runtime). - hermit/task/directory.py: Harbor's task format read without Harbor (moved from hermit/harbor/dataset.py, which imports it): the instruction (line endings kept), settings and metadata public; the task's directory and tests private. Benchmark loads `data.tasks: <directory>` (`data.rollouts: k`). - hermit/task/verify.py: a task's own verifier, `judge: {path: hermit.task.verify, name: script}`, run in the judge's sandbox as Harbor runs it: tests at /tests, a fresh /logs/verifier, reward.txt or reward.json, and feedback.txt for learning methods. A judge is called with Task.row() (the row plus `_private`); an agent still gets only the public half. - hermit/environment/spec.py and bwrap.py: EnvironmentSpec (the run config's `environment` section) and the bubblewrap backend the task sandbox is built by; a task asking for more network or memory than the run gives is refused. - examples/officeqa: the task set moves to utils/benchmark/ (prepare.py writes task directories; judge.py, task-info/, splits, data.json, seed adapter); the agents read each task as the row they always read (officeqa_fields.py), so the prompts are byte for byte the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 1: OfficeQA's methods on the new layout; task directories as Harbor reads them OfficeQA (the methods the move missed): ReHarness's launcher tests the tasks directory; RRSI's rows (byte-identical to before on the real data) and AHE's come from the task directories through officeqa_fields.question_rows; RRSI and AHE score with utils/benchmark/judge.py; Harness²'s source hook reads a directory row; OPRO shows the instruction; utils/.gitignore keeps the old paths ignored; AHE's real-data tests find the prepared data again. Task directories: a step's [verifier] settings come before the task's; [verifier.env] is set for the script (${NAME} resolved as Harbor does); reward.json takes a boolean as Harbor does; a task's [agent] timeout_sec bounds its calls and the judge waits at least the verifier's timeout_sec; an allowlist task's hosts and (with the script verifier) its verifier's network are checked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 2: a view of OfficeQA's rows for agents and learners; Harbor's verifier defaults and allowlists - data.view: <file.py>:<function> turns a task directory's row into the row the task set's agents and learners read. OfficeQA's configs view each task as SkillOpt's row, so records and what a learning method reads (GEPA, ACE, OPRO, SkillOpt) are the same as with its tasks.jsonl (246/246 on the real data); OPRO's question_field is `question` again. - What a task asks of its environment is kept in its private half (`needs`) and checked before each call (hermit.trainer.rollout asks a sandboxed agent's `refuses`), viewed or not. - As Harbor: a verifier's timeout_sec defaults to 600; [verifier.environment] implies a separate verifier; each phase's allowlist (a step's first) is its own; wildcard hosts are refused under an allowlist run (the proxy matches hosts as written). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 3: a step's verifier as Harbor resolves it; a viewed task keeps its timeout; finite rewards - A step's verifier mode and environment are resolved as Harbor's verifier_mode.py does (explicit mode, else an environment implies separate, the step's level before the task's, else shared), and the verifier's network and allowlist come from that effective environment. - A step's tests/ are laid over the task's in the judge's sandbox (its test.sh first), as Harbor uploads them. - A task's agent timeout is kept in its needs, which the run hands the sandboxed agent, so a data.view that leaves it out of the row still bounds the call. - reward.txt must hold a finite number, as in Harbor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Harbor environment backend: a task's Harbor environment as a Hermit Environment hermit/environment/harbor.py (HarborEnvironments, HarborEnvironment) and its driver, hermit/environment/harbor_driver.py, run under Harbor's own venv: Harbor's trial of the task with its nop agent, driven a piece at a time (start, exec, upload, download, install, verify, stop) over JSON lines. hermit/harbor/docker.py moves unchanged to hermit/environment/harbor_docker.py (the disk gate, image cleanup, compose_down). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hermit drives each Harbor task: the agent works in the task's Harbor environment, Harbor verifies - Acting agents take the task's environment: run(task, adapter, environment=...). The CLI Base Agents (Codex, Claude Code) run in a remote environment through its exec (the home rendered here and uploaded, the CLI installed by Harbor or mounted); the CLI acting agent deploys the adapter for a session there (hooks rewritten relative to the workspace); the answer-only agent (agent.name: answer) replaces HermitAnswer. - hermit/environment/remote.py: the environment served to the agent's worker in the task sandbox (exec, upload, download, install; never verify). - The run (hermit/trainer/rollout.py) starts each task's environment (environment.harbor), hands it to the agent, verifies with Harbor's verifier and stops it; judge hermit.task.verify:environment reads the verdict. - Deleted: hermit/harbor/runner.py, harborside/ and the runner's parts and tests. - Configs: every agent.name: harbor config moves to agent.base + environment.harbor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Tests for the Harbor environment, the served environment, remote CLI sessions, the answer agent and the run tests/core/test_harbor_environment.py: the driver's protocol with a stand-in driver, the warm start's calls, the verifier's verdicts, Codex and Claude Code in a remote environment, the CLI acting agent's staging, the answer agent, the environment server (no verify), the run's start/verify/stop, the environment judge; and a real Docker + Harbor test (skipped without them). OfficeQA's Harbor config and judge, and the example tests, follow the new path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Docs and changelog: Harbor environments, every agent Hermit's docs/framework/task.md's Benchmarks section describes the new path (environment.harbor, the driver, the warm start, the agents in a remote environment, what is refused); the agent layers, configuration, sandbox design, evaluation, model backends, troubleshooting and compatibility pages, examples/README.md, the benchmark READMEs and the OfficeQA recipe follow it. CHANGELOG: Unreleased, Breaking. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * A ReactAgent in a task's environment: its tools run there, the loop without the environment's budget Codex's install in a remote environment: with a version set, Harbor's installer is asked (it keeps a CLI already at that version). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rollout: wrap a long line (after rebasing onto PR 2) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Harbor environments keep the runner's infra semantics: run again, and never the agent's 0 - hermit/environment/harbor_errors.py: the runner's errors.py ported (Harbor's error policy, INFRA_CAUSES, GRADED, HOST_FAILURE, the verifier that ran no test, classify), with its verifier_timeout_verdict and verifier_verdict, NO_RETRY, the defaults (infra_retries 2, infra_wait_s 300 s), and the causes of an environment that did not start or a session that never did. - HarborEnvironments.verdict reads the verifier's answer as the runner read a trial; EnvironmentUnavailable carries its cause (disk, sandbox_failed, harbor_failed, harbor_timeout). - hermit/trainer/rollout.py: a rollout that ends infra runs again in a fresh environment up to environment.harbor.infra_retries times after infra_wait_s, never for disk or model_rejected; every run's usage is the rollout's (extra.attempts, the runner's _attempts); one that stays infra is marked unscored and the environment judge refuses it (Unscored: a judge_error record, as the runner's judge did). - report.kind counts output.status infra again; tests, task.md, evaluation guide, examples README and CHANGELOG follow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 4: multi-step tasks refused on this path; the verifier's workspace and memory - A multi-step task directory is refused here (its steps share one environment in order, which a per-call sandbox does not give); PR 4 brings that. - The script verifier's workspace defaults to the directory the agent worked in, when its output names none. - A separate verifier's own memory_mb is checked against the run's limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 1 (#114): restore what the Harbor runner did for a session in a task's environment - The CLI on PATH after Harbor's install (nvm's, ~/.local/bin), the prompt on stdin and Harbor's own instruction as the prompt (byte for byte, as the runner gave it); the task's own agent budget, also for the sandboxed worker's deadline; a session is not resumed (refused). - Every sign-in file leaves the environment before the verifier (confirmed, from /); the ChatGPT plan's usage reports come back; web search off on a restricted agent network; the login's hosts allowed; a login's plan waited for before the environment starts; the disk gate and the infra pause stop with the run. - Codex's lost sign-in, provider failure, plan limit and refused configuration read from its notices (codex_stream, ported from signin.py); a refused configuration is model_rejected (never run again); provider and environment failures across the worker boundary are infra; the CLI's provider allowance is the rollout's one. - Usage of every attempt counted; status/failure_cause beside the verdict (AHE); hooks' audit and state collected; python3 ensured for hooks; the image's context file merged within Codex's cap, an unreadable one infra. - Driver: agent commands still running are ended before the verifier (not a server an ended one left); logs copied without following links; installs under the baseline network policy, within Harbor's setup timeout, Debian bullseye sources repaired; a timeout without `timeout` still ends the command; requests written by a writer thread. Harbor task directories are run secrets. Answer agent: Harbor's default /app/answer.txt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * A CLI install that failed spent nothing: its run again's cost is 0, not unknown Seen in the Harbor-install smoke: Harbor's Codex install ran past Harbor's setup timeout on a 1-CPU task (infra, run again, then served); the failed run's cost made the eval's cost unknown, where the runner counted a setup failure as 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 2 (#114): the boundaries the Harbor runner kept, and its reading of a session - Security: no variable an agent's command names touches the driver or the host's compose client (each travels under a HERMIT_ENV_<n> carrier and is exported in the container); a by_fd secret (Claude Code's token) is moved into a pipe by the outermost shell before `timeout` or the CLI start, so no persistent process holds it. - An agent command's process tree is killed (descendants of its marked processes, whatever their environment); installs exclude agent commands while the setup network policy is open; Harbor's scrub of the trial's files runs on verify and stop, and the verifier's exception is scrubbed before Hermit reads it. - The driver's teardown kills a driver whose writer is stuck; the working directory is read from a marked line; environment-operation and host failures are typed infra (host_failed never run again). - The runner's session_verdict ported: a configuration refused before any token is model_rejected, a refused bill or spent quota ends the run (usage_limit), a prompt too long or the model's refusal stays the agent's result; Codex's failure text read. - A failed attempt keeps its usage (the agent returns it as an infra run); the plan's refused sessions' usage counts; a rotated ChatGPT sign-in comes back however the exec ended; the image's context file is read once across provider retries. - On a login the plan's wait and slot are held around the whole environment (the CLI takes none inside; a refused session holds the plan and the task runs again, not counted); the answer and react agents keep the task's budget on a host pipeline. - verifier_reward_raw beside the status (AHE); eval rows keep the rollout's outcome; environment.harbor.* overridable on the command line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: an unused import Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 3 (#114): the runner's results kept on the long paths - A CLI session's provider failure runs the task again in a fresh environment (environment.harbor.busy_retries/busy_wait_s, the runner's allowance over a task's runs, not infra_retries), never again in the environment the session changed. - An adapter that failed before any session (deployment, per-task context, context over Codex's cap) is the agent's result scored 0 (adapter_failed, context_over_limit), not left out of the denominator; a minimal row's context is computed from Harbor's instruction. - The image's linked context file is read through the environment and the link replaced (real Docker test); host failures are host_failed (not run again); a teardown failure after the verdict keeps it; a lost sign-in stops new work before the task's teardown; the verifier watchdog follows Harbor's own verifier timeout. - The install gate admits and counts a command at once and reopens on every path; an admitted session that drew credits holds the plan; a refused session without a plan of the run's ends the run; local sign-in copies go though a take-back fails. - ReAct keeps what its loop spent on a timeout and asks no follow-up after it; the answer agent's whole run is bounded, its log descriptor closed, model.max_retries honoured, attachments sent once, its endpoint's refusals read as the runner read them (bill: usage_limit; configuration: model_rejected; input: its result). - An acting agent's model a CLI reaches is refused for Harbor environments; run_agent and RRSI's smoke see a Harbor rollout's outcome; SWE-bench's token mean filters by outcome.status. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 4 (#114): race-proof log copy, confirmed kills, the runner's Codex cap - The agent's logs reach the trial's folder through O_NOFOLLOW directory descriptors and O_EXCL temp files renamed into place: an entry swapped for a link while the copy runs is replaced, never written through. - A failed restore of the agent phase's network policy breaks the environment: no further agent command, install or ensure; _ensure_python3 re-raises it. - kill_marked confirms its kill (exit 1 when a process is left); _drain raises before the verifier, keeps the mark and tries again; cleanup runs from /. - The sandbox proxy refuses network, hidden-path and readable limits instead of ignoring them. - A hook audit that cannot be read keeps the session's result and usage (hooks_audit_missing). - Codex in a task's environment keeps the runner's project_doc_max_bytes of 262144 unless the config sets one. - Answer: adapter context failures scored 0 (adapter_failed), preparation failures infra, the answer moved into place only within the budget; ReAct's answer-tag follow-up is bounded; a call given up in flight counts as a call with unknown cost. - The disk gate rechecks the stop flag (between gates, after them): nothing starts, the rollout ends as the stopped run's. - logins_of treats an acting agent in environment.harbor as contained; the benchmark-file form of environment.harbor takes nested overrides; config-fields and examples/README updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 5 (#114): the Hermit-side fixes that outlive the bridge redesign - The credentials an agent command is given are scrubbed from the logs handed to the verifier and from the trial's files (the driver records them from each exec's env and by_fd). - A ChatGPT sign-in comes back as the runner took it (signin.take_back, ported into codex_stream): a private download, a regular file of at most 64 KiB with both tokens, stamped at most SKEW/OTHER_CLOCK ahead; a failed local write of the refresh keeps the finished session. - The sandboxed agent provisions and takes back the configured login_root. - A Codex rejection keeps its head (the status) for the verdict, as the runner did. - A remote session that reached its CLI but reported no usage counts as an unknown cost. - A refused session whose plan hold cannot be written pauses HOLD_UNWRITTEN_S before the next environment. - reward_key is authoritative (none there is infra, the runner's classify); verifier_redact restored; an adapter failure has no pass on a benchmark without pass_reward. - The answer agent extracts as the runner did (group 1 only when it took part) and sends max_tokens only when given. - Docs: the agents that work in a Harbor environment; report docstring; the Codex cap comment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Harbor runs the whole trial: one bridge agent relays Hermit's commands (#114) Harbor's own Trial.run() now owns each task's trial: the environment's start, its capability check, the agent's setup (Harbor's setup timeout, the baseline network policy, the agent user), the agent phase (its network policy, user and timeout), the verifier in its mode, the scrub, result.json and the stop. The driver no longer drives the trial a piece at a time. - HermitBridge (a Harbor BaseAgent in the driver, imported by AgentConfig.import_path) has no agent logic: its setup installs what the acting agent declares (Agent.installs: the CLI unless `bin` names one, probed first unless a version is pinned; python3 for an adapter's hooks) with Harbor's own installers; its run starts the phase, relays Hermit's exec/upload/download until `done`, Harbor's agent timeout or a cancellation, and ends every command it started that still runs (their Docker clients too, the kill confirmed) before it returns. A kill not confirmed fails the run (HarborDriverError: no verifier, infra). Hermit's logs reach /logs/agent scrubbed. - Deleted: the Hermit-side network-policy switching and install gate, the install/ensure/verify ops, the separate drain step, the piecewise start/verify/stop. HarborEnvironment.install reports what setup installed; verify sends `done` and reads the trial's verdict. Harbor refusing a task's skills, MCP servers or Windows is unsupported_task (never run again). The agent's own budget ends 60 s before Harbor's agent timeout so its logs arrive first; a phase Harbor ended (AgentPhaseOver) is read as the trial's verdict. - Agents stay Harbor-free: they use Hermit's Environment API; only the rollout of a Harbor task set starts a session. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Bridge: transfers as the agent user, its uid:gid, limits refused, no missed death (#114) - r5 #2: an upload or download the agent's process asks for through the sandbox's proxy (as_agent) runs with the task agent user's own permissions when it is not root (staged as root, copied into place as that user; a tar made as that user, unpacked as data); root only for Hermit's own provisioning. PermissionError, not infra. - r5 #9: the bridge reads the uid:gid the agent runs as (an image's own USER too), and Hermit's provisioning is chowned to it, confirmed (the runner's _owner). - r5 #8: the CLI agent keeps an environment's other limits when it cuts the time, and refuses network, hidden or readable limits in a remote environment; the sandbox's proxy refuses them before it is served. - r5 #13: the driver's reader publishes its death, and replies, under the lock a request registers under. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * One implementation each: usage ledger, credential lifecycle, deadline, retry states (#114) - hermit/agent/model/usage.py: Ledger (an agent run's usage: every attempt, every call given up in flight, every session cut off before its usage; cost unknown when any entry's is; what an empty ledger reports is the caller's word) and Attempts (a rollout's runs and their sum, per-attempt records: the runner's _attempts). The workspace agent, ReAct, the answer agent and base React's total use it; rollout._attempts and the hand sums are gone. - hermit/agent/base/credentials.py: held(provision, take_back, remove), the sign-in's lifecycle with its finally: the CLI's remote session and the sandboxed agent's worker both go through it. - hermit/agent/base/deadline.py: one absolute Deadline (remaining, cut, a chat client bound to it): ReAct's and the answer agent's model calls, the CLI's limits before its session and the workspace's deadline_s. - hermit/trainer/rollout.py: Retries, the rollout's one retry and admission state machine with its states named (DONE, INFRA, PROVIDER_BUSY, PLAN_WAIT, STOP); one run in a fresh environment is _environment_run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 6 (#114): the agent phase's end, outside Harbor's timeout, does what it must Structural fix for the end of the agent phase (findings 1-7, 11): the bridge's end-of-phase work runs as a task of its own that its run awaits through any cancellation (Harbor's agent timeout can no longer cut it short), bounded by END_PHASE_S; anything it cannot confirm raises HarborDriverError, so Harbor runs no verifier. It: lets transfers in flight finish, cancels the commands, ends their clients and kills their processes (running and earlier unconfirmed kills alike), keeps a running command's stream (exec `stream=`, tee'd there as it runs) in the agent logs under its name, removes the credential files the CLI registered before its upload (Environment `secret`, confirmed), and copies Hermit's logs scrubbed. A cancelled wait on any Docker client kills it (compose up after a build timeout too); stop answers outside the lifecycle lock, ending the start's clients before the teardown. - The verdict: a verifier that failed after the agent's timeout is read by its infra policy (the driver keeps every exception Harbor recorded or skipped); a result.json Harbor could not write falls back to the trial's in-memory result, and no verdict at all is the driver's error. - A disabled verifier is refused; Windows refused at the trial's creation is UnsupportedTask. - Agents: install discovery never raises on an adapter (the deployment scores it); a deployment Harbor's phase end cut short is not adapter_failed; python3 for hooks is best effort at setup; the claude-code alias installs Claude Code; the answer agent's log is a folder with the runner's answer.json; eval checks the sign-in before replacing results. Docs: the judge's place with Harbor, installs is built-in only, the driver's intro, AHE's trace note. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Ported from the runner: a mounted CLI's checks, Claude's key, attachments, infra usage (#114) - A mounted binary (`bin`) serves only when it runs there as the pinned version (any, when none is) and the agent's user may execute it (the runner's _use_mounted): linked onto PATH, with the release's rg and this machine's CA bundle for an image that has none (harborside.codex's _after_mount); else the bridge's setup installs the CLI and the session runs the installed program (bin_served, through the proxy too). - Claude Code in a task's environment gets no raw key variable of its binding: it reads its key from its own (session_env), as the runner's launch left it. - The answer agent warns when it leaves out an attachment over 20 MiB. - A run that is not the agent's result after a session ran, with no usage reported, has an unknown cost and token count (the runner's usage None), never 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 7 (#114): no verifier after an unfinished end, the agent's requests as the agent - P1 #1: the phase's end that does not finish (its bound, or any other way) is HarborDriverError, read from the shield's own exception too: Harbor runs no verifier after it (_close_phase, tested past its bound). - P1 #2: nothing the worker asks for runs as root: credential files are removed as the agent user, an upload's missing folders are made by that user (cp -f replaces what it may replace), and the CLI's home there is the bridge's own provisioning, declared by the agent (Agent.installs `home`), made and given to the user at the phase's start. - #3: each command writes its pid at its start, the root of what kill_marked ends where root cannot read another user's environment (no CAP_SYS_PTRACE). #4: the commands are stopped as the phase closes, before a pending transfer is let finish. #5: the driver's environment drops the ambient credentials the runner's harbor_env dropped. #6: a stop Harbor acknowledged with its compose projects still there takes them down by name (a local, deleted Docker environment). #7: cp -f. #8: one absolute deadline from the phase's start (deadline_at, through the proxy): every command's time and ReAct's and the answer agent's model calls come off it. #9: a timed-out rollout's log is the trial's agent logs (the stream the bridge kept, stderr too). #10: the start's and a separate verifier's watchdogs from Harbor's budgets. #11: attachments measured and read in the environment as the agent user (through a link; one over 20 MiB never read). #12: eval checks the acting agent's sign-in only. #13: a pipeline error on a run again keeps what the runs before it spent. #14: the agent logs reach a shared verifier only (documented). - Tests no longer depend on prepared data (the eval test writes its own task set); the affected tests pass in a clean checkout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 8 (#114): the phase's end as confirmed steps, commands by their own word, the runner's checks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Round 8 triage: out-of-scope items (#8, #12, #17, #18) left as the runner had them Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 9 (#114): no worker request as root, archives extracted member by member, the phase's end confirmed - P1 #1: a worker's upload is never promoted to root by its name: the deployment names the agent declares are made the agent user's by the bridge itself as the phase starts (before any agent command runs), and every worker upload stays the agent user's own. - P1 #2: archives another process made (the worker's upload on this machine, the container's tar of a download) are written member by member: regular files and folders only, no absolute or `..` names, files created anew (O_EXCL | O_NOFOLLOW) through folders the extractor made, whatever Python's tarfile filter does. - #3 a transfer still running or failed at the phase's end fails it; #4 a stream that began and cannot be read fails it; #5 the deployment record (.hermit-deploy.json, Claude's .mcp.json) and an image's context link are provisioned; #6 credentials are removed though an earlier step failed; #7 Harbor ending the phase during the answer's delivery keeps its spend; #8 a command whose end is unknown is ended at the phase's end; #9 a zombie is not a live client; #10 the host kills the driver's client groups when it has to kill the driver; #11 keep_containers holds on the fallback too; #12 the loopback check knows `cli` is Codex; #13 an attachment's bytes are read between marks of their own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Harbor tasks back to the simple design: no worker, no proxy, the runner's end of the phase (#114) - A task in Harbor's environment runs its agent in this process (host_pipeline): every command and CLI runs in the task's container, the agent's loop here with the HarborEnvironment. Deleted: hermit/environment/remote.py (EnvironmentServer, ProxyEnvironment, the host-side extraction of a worker's archives), the sandboxed agent's environment server, the worker's environment, the agent-user transfer rules (as_agent) and the deployment and home provisioning they needed; Hermit's uploads are its provisioning again, root's copy given to the agent user. - The end of the agent phase at the runner's level: each command runs under setsid, its group leader's pid in a file (validated by the command's word on its command line); at done or Harbor's timeout the bridge cancels the requests in flight, kills each group still running as root with one check it is gone, removes the registered credential files, copies Hermit's logs, and returns; a kill or a removal that fails is infra (no verifier). Transfers hold one lock with the phase's end. A CLI's stream is written to /logs/agent as it runs (the runner's stream lived there). Deleted: the per-step bounded/confirmed machinery, the stream keeping, the transfer drain, the environment marks. Kept: Harbor's Trial.run, the compose clients' process groups killed on cancellation, keep_containers, the warm start, the disk gate, the cleanup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Harbor: the agent's setup within 20 minutes by default; any failure of the phase's end is infra (#114) - environment.harbor.setup_timeout_sec defaults to 1200 s (Harbor's agent setup, a CLI's install: its own 360 s default ran out on all three attempts of a real run under load); a setting still wins, and the start's watchdog follows it. - A failure of the agent phase's end of any kind is HarborDriverError (no verifier, infra), never the agent's 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Round 10: confine the adapter's code, no implicit sign-in, keep cut-off usage - A Harbor task set's agent call confines the adapter's own code as the worker did: its context tool's Session and its exports' render run in the strict adapter sandbox (hermit.adapter.session.CONFINED, per call). - A CLI in Harbor's environments needs a model binding or a login (the runner's refusal), and a remote session never copies this machine's sign-in. - The usage a CLI reported before Harbor's timeout cut it off is read from the stream it left in the trial's agent logs and counted. - stop() calls end_clients() with its current signature; the host's forced teardown signals only the driver's own group; kill_groups keeps a pid file until its group is confirmed gone; a failed kill exec still removes the credentials; a download of a link is refused; the log a record names is scrubbed of the run's keys. - Docs: no worker or proxy for Harbor; Hermit's judge on the host. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * kill_groups in POSIX sh: no `--` for dash's kill; dead processes gone CI's Ubuntu runner runs `sh` as dash, whose kill reads `--` as a pid ("Illegal number: -"), so the group was never killed and the check failed. The kill is now `kill -KILL -<pgid>` (POSIX: dash, busybox ash, bash), and a process in state X counts as gone with a zombie. Harbor's main container runs the line under bash, so trials were not affected; a sidecar or another shell would be. The tests run kill_groups under sh and bash with a PID 1 that never reaps (Python, as a container's `sleep infinity`), count the group's live processes from /proc (no ps), and once more in Debian's python:3.12-slim (dash). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…read through a view (#115) * Tasks and environments beside the agents: hermit/task, hermit/environment, hermit/common The first of four stacked changes toward Harbor-format tasks run by Hermit's own agents (design: "Hermit agent 与 benchmark 重构设计", 第二阶段). Structure only: no behavior changes, the requests every agent sends are byte for byte main's. - hermit/environment/: the environments move out of hermit/agent/ to the top level, since a task declares what it needs of one and a judge may run in one; hermit.agent.environment forwards for one release. - hermit/task/: task.py becomes a package. Task holds a public half (what the acting agent is handed, what a run records as its input) and a private half only the judge reads; Benchmark is a task set and its splits; verify.py wraps a judge as a verifier of a Task. Every name importable from hermit.task before still is. - hermit/common.py: RunOutput and load_callable, shared by agents and tasks so that neither imports the other. - The run (hermit/trainer/rollout.py) hands the agent task.given() and the verifier the whole task; the trainer, eval and hermit check read a Benchmark. The validation file the learner gets holds each task's public half. - tests/core/test_layers.py: tasks and agents never import each other; the environments import neither. tests/core/test_task.py covers the new types. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * hermit check: report the probe task by its id (a Task, not a row) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 1: a bare judge reads the whole row again; records keep the row's id - judge_verifier calls the judge with Task.row() (the public half and `_private`), as run_one did before tasks were typed; the agent still gets only the public half. - Records keep the row's own id (7, not "7"); run_tasks keys its results by task. - run_one/run_tasks keep the `judge` parameter name; the Verifier protocol states `takes_task`; agent-layers.md shows hermit/environment/ beside hermit/agent/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rollout: the record's id in one name (line length) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * eval: a record's numeric id finds its task (after the record kept the row's id) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Task directories in Harbor's format, their own verifier, the bwrap backend; OfficeQA as task directories The second of four stacked changes (design: 第二阶段). A task set may now be a directory of task directories in Harbor's format, run on Hermit's own sandbox (B-class: Harbor's format, not Harbor's runtime). - hermit/task/directory.py: Harbor's task format read without Harbor (moved from hermit/harbor/dataset.py, which imports it): the instruction (line endings kept), settings and metadata public; the task's directory and tests private. Benchmark loads `data.tasks: <directory>` (`data.rollouts: k`). - hermit/task/verify.py: a task's own verifier, `judge: {path: hermit.task.verify, name: script}`, run in the judge's sandbox as Harbor runs it: tests at /tests, a fresh /logs/verifier, reward.txt or reward.json, and feedback.txt for learning methods. A judge is called with Task.row() (the row plus `_private`); an agent still gets only the public half. - hermit/environment/spec.py and bwrap.py: EnvironmentSpec (the run config's `environment` section) and the bubblewrap backend the task sandbox is built by; a task asking for more network or memory than the run gives is refused. - examples/officeqa: the task set moves to utils/benchmark/ (prepare.py writes task directories; judge.py, task-info/, splits, data.json, seed adapter); the agents read each task as the row they always read (officeqa_fields.py), so the prompts are byte for byte the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 1: OfficeQA's methods on the new layout; task directories as Harbor reads them OfficeQA (the methods the move missed): ReHarness's launcher tests the tasks directory; RRSI's rows (byte-identical to before on the real data) and AHE's come from the task directories through officeqa_fields.question_rows; RRSI and AHE score with utils/benchmark/judge.py; Harness²'s source hook reads a directory row; OPRO shows the instruction; utils/.gitignore keeps the old paths ignored; AHE's real-data tests find the prepared data again. Task directories: a step's [verifier] settings come before the task's; [verifier.env] is set for the script (${NAME} resolved as Harbor does); reward.json takes a boolean as Harbor does; a task's [agent] timeout_sec bounds its calls and the judge waits at least the verifier's timeout_sec; an allowlist task's hosts and (with the script verifier) its verifier's network are checked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 2: a view of OfficeQA's rows for agents and learners; Harbor's verifier defaults and allowlists - data.view: <file.py>:<function> turns a task directory's row into the row the task set's agents and learners read. OfficeQA's configs view each task as SkillOpt's row, so records and what a learning method reads (GEPA, ACE, OPRO, SkillOpt) are the same as with its tasks.jsonl (246/246 on the real data); OPRO's question_field is `question` again. - What a task asks of its environment is kept in its private half (`needs`) and checked before each call (hermit.trainer.rollout asks a sandboxed agent's `refuses`), viewed or not. - As Harbor: a verifier's timeout_sec defaults to 600; [verifier.environment] implies a separate verifier; each phase's allowlist (a step's first) is its own; wildcard hosts are refused under an allowlist run (the proxy matches hosts as written). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 3: a step's verifier as Harbor resolves it; a viewed task keeps its timeout; finite rewards - A step's verifier mode and environment are resolved as Harbor's verifier_mode.py does (explicit mode, else an environment implies separate, the step's level before the task's, else shared), and the verifier's network and allowlist come from that effective environment. - A step's tests/ are laid over the task's in the judge's sandbox (its test.sh first), as Harbor uploads them. - A task's agent timeout is kept in its needs, which the run hands the sandboxed agent, so a data.view that leaves it out of the row still bounds the call. - reward.txt must hold a finite number, as in Harbor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Harbor environment backend: a task's Harbor environment as a Hermit Environment hermit/environment/harbor.py (HarborEnvironments, HarborEnvironment) and its driver, hermit/environment/harbor_driver.py, run under Harbor's own venv: Harbor's trial of the task with its nop agent, driven a piece at a time (start, exec, upload, download, install, verify, stop) over JSON lines. hermit/harbor/docker.py moves unchanged to hermit/environment/harbor_docker.py (the disk gate, image cleanup, compose_down). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hermit drives each Harbor task: the agent works in the task's Harbor environment, Harbor verifies - Acting agents take the task's environment: run(task, adapter, environment=...). The CLI Base Agents (Codex, Claude Code) run in a remote environment through its exec (the home rendered here and uploaded, the CLI installed by Harbor or mounted); the CLI acting agent deploys the adapter for a session there (hooks rewritten relative to the workspace); the answer-only agent (agent.name: answer) replaces HermitAnswer. - hermit/environment/remote.py: the environment served to the agent's worker in the task sandbox (exec, upload, download, install; never verify). - The run (hermit/trainer/rollout.py) starts each task's environment (environment.harbor), hands it to the agent, verifies with Harbor's verifier and stops it; judge hermit.task.verify:environment reads the verdict. - Deleted: hermit/harbor/runner.py, harborside/ and the runner's parts and tests. - Configs: every agent.name: harbor config moves to agent.base + environment.harbor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Tests for the Harbor environment, the served environment, remote CLI sessions, the answer agent and the run tests/core/test_harbor_environment.py: the driver's protocol with a stand-in driver, the warm start's calls, the verifier's verdicts, Codex and Claude Code in a remote environment, the CLI acting agent's staging, the answer agent, the environment server (no verify), the run's start/verify/stop, the environment judge; and a real Docker + Harbor test (skipped without them). OfficeQA's Harbor config and judge, and the example tests, follow the new path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Docs and changelog: Harbor environments, every agent Hermit's docs/framework/task.md's Benchmarks section describes the new path (environment.harbor, the driver, the warm start, the agents in a remote environment, what is refused); the agent layers, configuration, sandbox design, evaluation, model backends, troubleshooting and compatibility pages, examples/README.md, the benchmark READMEs and the OfficeQA recipe follow it. CHANGELOG: Unreleased, Breaking. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * A ReactAgent in a task's environment: its tools run there, the loop without the environment's budget Codex's install in a remote environment: with a version set, Harbor's installer is asked (it keeps a CLI already at that version). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rollout: wrap a long line (after rebasing onto PR 2) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Harbor environments keep the runner's infra semantics: run again, and never the agent's 0 - hermit/environment/harbor_errors.py: the runner's errors.py ported (Harbor's error policy, INFRA_CAUSES, GRADED, HOST_FAILURE, the verifier that ran no test, classify), with its verifier_timeout_verdict and verifier_verdict, NO_RETRY, the defaults (infra_retries 2, infra_wait_s 300 s), and the causes of an environment that did not start or a session that never did. - HarborEnvironments.verdict reads the verifier's answer as the runner read a trial; EnvironmentUnavailable carries its cause (disk, sandbox_failed, harbor_failed, harbor_timeout). - hermit/trainer/rollout.py: a rollout that ends infra runs again in a fresh environment up to environment.harbor.infra_retries times after infra_wait_s, never for disk or model_rejected; every run's usage is the rollout's (extra.attempts, the runner's _attempts); one that stays infra is marked unscored and the environment judge refuses it (Unscored: a judge_error record, as the runner's judge did). - report.kind counts output.status infra again; tests, task.md, evaluation guide, examples README and CHANGELOG follow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 4: multi-step tasks refused on this path; the verifier's workspace and memory - A multi-step task directory is refused here (its steps share one environment in order, which a per-call sandbox does not give); PR 4 brings that. - The script verifier's workspace defaults to the directory the agent worked in, when its output names none. - A separate verifier's own memory_mb is checked against the run's limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 1 (#114): restore what the Harbor runner did for a session in a task's environment - The CLI on PATH after Harbor's install (nvm's, ~/.local/bin), the prompt on stdin and Harbor's own instruction as the prompt (byte for byte, as the runner gave it); the task's own agent budget, also for the sandboxed worker's deadline; a session is not resumed (refused). - Every sign-in file leaves the environment before the verifier (confirmed, from /); the ChatGPT plan's usage reports come back; web search off on a restricted agent network; the login's hosts allowed; a login's plan waited for before the environment starts; the disk gate and the infra pause stop with the run. - Codex's lost sign-in, provider failure, plan limit and refused configuration read from its notices (codex_stream, ported from signin.py); a refused configuration is model_rejected (never run again); provider and environment failures across the worker boundary are infra; the CLI's provider allowance is the rollout's one. - Usage of every attempt counted; status/failure_cause beside the verdict (AHE); hooks' audit and state collected; python3 ensured for hooks; the image's context file merged within Codex's cap, an unreadable one infra. - Driver: agent commands still running are ended before the verifier (not a server an ended one left); logs copied without following links; installs under the baseline network policy, within Harbor's setup timeout, Debian bullseye sources repaired; a timeout without `timeout` still ends the command; requests written by a writer thread. Harbor task directories are run secrets. Answer agent: Harbor's default /app/answer.txt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * A CLI install that failed spent nothing: its run again's cost is 0, not unknown Seen in the Harbor-install smoke: Harbor's Codex install ran past Harbor's setup timeout on a 1-CPU task (infra, run again, then served); the failed run's cost made the eval's cost unknown, where the runner counted a setup failure as 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Task directories for the Claude subscription example; write_task writes one The first benchmark of refactor 4/4: examples/claude_subscription's nine tasks are task directories in Harbor's format under utils/benchmark/ (with the judge, the splits and task-info), each read through data.view as the row {id, prompt} it was in tasks.jsonl, so the prompts and records are byte for byte the same. hermit.task.directory.write_task writes a task directory (instruction.md, task.toml with [metadata] and other tables) whose values tomllib reads back as given. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ASPIRE's engine, Robosuite and LIBERO: tasks as task directories Refactor 4/4: examples/aspire_robosuite's and examples/libero's tasks (each task, its verifier and its rerun) are task directories in Harbor's format under their utils/benchmark/ (prepare.py, judge.py, task-info/, splits), asking for no network and judged on the host, where the simulator is. aspire_fields.py, every config's data.view and the readers on the host (sim_service.py and stage_docs.py --tasks), reads each back as the row it was in tasks.jsonl. Parity: 21/21 Robosuite rows on its four configs and 630/630 LIBERO rows on its fourteen, both presets, every split file byte for byte; the simulator service's rows the same. A short LIBERO run before and after (prompt_only/smoke.yaml, eval of rerun-libero_object-t00, no adapter): the actor's prompt.txt (2471 bytes) and the service's registration identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * RoboDojo-Sim's tasks as task directories Refactor 4/4: examples/robodojo_sim's forty-two tasks are task directories in Harbor's format under utils/benchmark/ (prepare.py, judge.py, task-info/, splits), each asking for no network and judged on the host. robodojo_fields.py, every config's data.view and the simulator service's reader (dojo_service.py --tasks), reads each back as the row it was in tasks.jsonl: 42/42 rows identical on all seven configs, both presets, and the eight split files byte for byte. RoboDojo and Isaac Sim are not on this machine: no smoke. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ServeLearnBench's episodes as task directories Refactor 4/4: examples/servelearnbench's scenario export writes one task directory in Harbor's format per episode under utils/benchmark/ (with prepare.py, judge.py, windows.py, windows.json, task-info/ and the splits), the stamped instruction as instruction.md. slb_fields.py, both configs' data.view and windows.py's reader, reads each back as the row it was: 1161/1161 rows identical (retail_l3, from the old prepare's export) on both configs, the eleven split files byte for byte. Each episode stays one task (the adapter is what carries across the stream), so no multi-step task is needed. The package is not installed here: no smoke. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * CL-Bench's database and poker streams: tasks as task directories Refactor 4/4: each question (hand) of examples/clbench_database's and examples/clbench_poker's ordered streams is a task directory in Harbor's format under utils/benchmark/ (with prepare.py, judge.py, task-info/ and the per-order splits); clbench_fields.py and poker_fields.py, every config's data.view, read each back as the row it was. Each row stays one task: the stream's state is the host's, carried by the agent's own loop in its run directory, so no multi-step task is needed. The runners read the venue texts from utils/benchmark/task-info/mwm. Parity: 200/200 (database) and 520/520 (poker) rows identical on all three configs each, from the old prepare run against ~/code/clbench, splits byte for byte. Short prompt_only runs before and after (order 0's first question, first hand; GLM 5.3 Flash): the recorded input row and the host's first query identical (1184 and 1350 bytes), the evidence README identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 2 (#114): the boundaries the Harbor runner kept, and its reading of a session - Security: no variable an agent's command names touches the driver or the host's compose client (each travels under a HERMIT_ENV_<n> carrier and is exported in the container); a by_fd secret (Claude Code's token) is moved into a pipe by the outermost shell before `timeout` or the CLI start, so no persistent process holds it. - An agent command's process tree is killed (descendants of its marked processes, whatever their environment); installs exclude agent commands while the setup network policy is open; Harbor's scrub of the trial's files runs on verify and stop, and the verifier's exception is scrubbed before Hermit reads it. - The driver's teardown kills a driver whose writer is stuck; the working directory is read from a marked line; environment-operation and host failures are typed infra (host_failed never run again). - The runner's session_verdict ported: a configuration refused before any token is model_rejected, a refused bill or spent quota ends the run (usage_limit), a prompt too long or the model's refusal stays the agent's result; Codex's failure text read. - A failed attempt keeps its usage (the agent returns it as an infra run); the plan's refused sessions' usage counts; a rotated ChatGPT sign-in comes back however the exec ended; the image's context file is read once across provider retries. - On a login the plan's wait and slot are held around the whole environment (the CLI takes none inside; a refused session holds the plan and the task runs again, not counted); the answer and react agents keep the task's budget on a host pipeline. - verifier_reward_raw beside the status (AHE); eval rows keep the rollout's outcome; environment.harbor.* overridable on the command line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: an unused import Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * AgentDyn and AgentDojo: each protocol's frozen cases as task directories Refactor 4/4: AgentDyn's and AgentDojo's prepare scripts, with their judge, task-info/ and DATA.md, move to utils/benchmark/; each protocol directory now holds its cases as task directories in Harbor's format (tasks/<id>/, the case's row as task.toml's [metadata], its id in its place among sorted keys), cases.txt (the rows' order, which a later stage and --full-test read), the splits and the manifest, whose tasks_sha256 is still the SHA-256 of the rows as JSON lines (DATA.md's hashes hold). cases.py's case_row is every config's data.view; the later stages, report.py, compare.py and the experiment runner read a protocol's tasks/ (or a tasks.jsonl prepared before); the experiment snapshot copies a task set's directory and hashes it whole. Each case stays one task (the agent loop is AgentDyn's own). Parity (AgentDyn and recorded collections are not on this machine): the old and the new pipelines run on the same synthetic collections, the native audit set to all eligible: expanded-v2, its full/ inventory, thirty-v1, all-groups-v1, full-test-v1, prepare.py's protocol, AgentDojo's comparison-v1 and prepare.py's protocol give the same rows (54 to 275 each), splits byte for byte and manifests (tasks_sha256 too); all eleven configs read the old rows through data.view. No smoke. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Docs: the examples' task sets in utils/benchmark/; unique test names examples/README.md's layout names utils/benchmark/ (prepare.py writing task directories, judge.py, task-info/), and its rule on the examples whose agent owns its loop says where they now stand: task directories in Harbor's format, each row of a stream one task, on Hermit's own sandbox or the host. docs/framework/task.md names write_task and that route. The examples' new tests get names of their own (pytest refuses two test modules of one basename), and write_task's float case reads more plainly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 1 (#115): an AgentDyn case keeps its nulls; RoboDojo's README on its format - Each case is kept in its task.toml as one JSON line ([metadata] case): a clean case of an old tasks.jsonl (injection_task_id null, injections {}) reads back as it was, key order and hash unchanged; the nullable fixture is back. - RoboDojo's README: its task set is in Harbor's format; its tasks run on the host simulator service, not in Harbor's environments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 3 (#114): the runner's results kept on the long paths - A CLI session's provider failure runs the task again in a fresh environment (environment.harbor.busy_retries/busy_wait_s, the runner's allowance over a task's runs, not infra_retries), never again in the environment the session changed. - An adapter that failed before any session (deployment, per-task context, context over Codex's cap) is the agent's result scored 0 (adapter_failed, context_over_limit), not left out of the denominator; a minimal row's context is computed from Harbor's instruction. - The image's linked context file is read through the environment and the link replaced (real Docker test); host failures are host_failed (not run again); a teardown failure after the verdict keeps it; a lost sign-in stops new work before the task's teardown; the verifier watchdog follows Harbor's own verifier timeout. - The install gate admits and counts a command at once and reopens on every path; an admitted session that drew credits holds the plan; a refused session without a plan of the run's ends the run; local sign-in copies go though a take-back fails. - ReAct keeps what its loop spent on a timeout and asks no follow-up after it; the answer agent's whole run is bounded, its log descriptor closed, model.max_retries honoured, attachments sent once, its endpoint's refusals read as the runner read them (bill: usage_limit; configuration: model_rejected; input: its result). - An acting agent's model a CLI reaches is refused for Harbor environments; run_agent and RRSI's smoke see a Harbor rollout's outcome; SWE-bench's token mean filters by outcome.status. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 4 (#114): race-proof log copy, confirmed kills, the runner's Codex cap - The agent's logs reach the trial's folder through O_NOFOLLOW directory descriptors and O_EXCL temp files renamed into place: an entry swapped for a link while the copy runs is replaced, never written through. - A failed restore of the agent phase's network policy breaks the environment: no further agent command, install or ensure; _ensure_python3 re-raises it. - kill_marked confirms its kill (exit 1 when a process is left); _drain raises before the verifier, keeps the mark and tries again; cleanup runs from /. - The sandbox proxy refuses network, hidden-path and readable limits instead of ignoring them. - A hook audit that cannot be read keeps the session's result and usage (hooks_audit_missing). - Codex in a task's environment keeps the runner's project_doc_max_bytes of 262144 unless the config sets one. - Answer: adapter context failures scored 0 (adapter_failed), preparation failures infra, the answer moved into place only within the budget; ReAct's answer-tag follow-up is bounded; a call given up in flight counts as a call with unknown cost. - The disk gate rechecks the stop flag (between gates, after them): nothing starts, the rollout ends as the stopped run's. - logins_of treats an acting agent in environment.harbor as contained; the benchmark-file form of environment.harbor takes nested overrides; config-fields and examples/README updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 5 (#114): the Hermit-side fixes that outlive the bridge redesign - The credentials an agent command is given are scrubbed from the logs handed to the verifier and from the trial's files (the driver records them from each exec's env and by_fd). - A ChatGPT sign-in comes back as the runner took it (signin.take_back, ported into codex_stream): a private download, a regular file of at most 64 KiB with both tokens, stamped at most SKEW/OTHER_CLOCK ahead; a failed local write of the refresh keeps the finished session. - The sandboxed agent provisions and takes back the configured login_root. - A Codex rejection keeps its head (the status) for the verdict, as the runner did. - A remote session that reached its CLI but reported no usage counts as an unknown cost. - A refused session whose plan hold cannot be written pauses HOLD_UNWRITTEN_S before the next environment. - reward_key is authoritative (none there is infra, the runner's classify); verifier_redact restored; an adapter failure has no pass on a benchmark without pass_reward. - The answer agent extracts as the runner did (group 1 only when it took part) and sends max_tokens only when given. - Docs: the agents that work in a Harbor environment; report docstring; the Codex cap comment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Harbor runs the whole trial: one bridge agent relays Hermit's commands (#114) Harbor's own Trial.run() now owns each task's trial: the environment's start, its capability check, the agent's setup (Harbor's setup timeout, the baseline network policy, the agent user), the agent phase (its network policy, user and timeout), the verifier in its mode, the scrub, result.json and the stop. The driver no longer drives the trial a piece at a time. - HermitBridge (a Harbor BaseAgent in the driver, imported by AgentConfig.import_path) has no agent logic: its setup installs what the acting agent declares (Agent.installs: the CLI unless `bin` names one, probed first unless a version is pinned; python3 for an adapter's hooks) with Harbor's own installers; its run starts the phase, relays Hermit's exec/upload/download until `done`, Harbor's agent timeout or a cancellation, and ends every command it started that still runs (their Docker clients too, the kill confirmed) before it returns. A kill not confirmed fails the run (HarborDriverError: no verifier, infra). Hermit's logs reach /logs/agent scrubbed. - Deleted: the Hermit-side network-policy switching and install gate, the install/ensure/verify ops, the separate drain step, the piecewise start/verify/stop. HarborEnvironment.install reports what setup installed; verify sends `done` and reads the trial's verdict. Harbor refusing a task's skills, MCP servers or Windows is unsupported_task (never run again). The agent's own budget ends 60 s before Harbor's agent timeout so its logs arrive first; a phase Harbor ended (AgentPhaseOver) is read as the trial's verdict. - Agents stay Harbor-free: they use Hermit's Environment API; only the rollout of a Harbor task set starts a session. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Bridge: transfers as the agent user, its uid:gid, limits refused, no missed death (#114) - r5 #2: an upload or download the agent's process asks for through the sandbox's proxy (as_agent) runs with the task agent user's own permissions when it is not root (staged as root, copied into place as that user; a tar made as that user, unpacked as data); root only for Hermit's own provisioning. PermissionError, not infra. - r5 #9: the bridge reads the uid:gid the agent runs as (an image's own USER too), and Hermit's provisioning is chowned to it, confirmed (the runner's _owner). - r5 #8: the CLI agent keeps an environment's other limits when it cuts the time, and refuses network, hidden or readable limits in a remote environment; the sandbox's proxy refuses them before it is served. - r5 #13: the driver's reader publishes its death, and replies, under the lock a request registers under. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * One implementation each: usage ledger, credential lifecycle, deadline, retry states (#114) - hermit/agent/model/usage.py: Ledger (an agent run's usage: every attempt, every call given up in flight, every session cut off before its usage; cost unknown when any entry's is; what an empty ledger reports is the caller's word) and Attempts (a rollout's runs and their sum, per-attempt records: the runner's _attempts). The workspace agent, ReAct, the answer agent and base React's total use it; rollout._attempts and the hand sums are gone. - hermit/agent/base/credentials.py: held(provision, take_back, remove), the sign-in's lifecycle with its finally: the CLI's remote session and the sandboxed agent's worker both go through it. - hermit/agent/base/deadline.py: one absolute Deadline (remaining, cut, a chat client bound to it): ReAct's and the answer agent's model calls, the CLI's limits before its session and the workspace's deadline_s. - hermit/trainer/rollout.py: Retries, the rollout's one retry and admission state machine with its states named (DONE, INFRA, PROVIDER_BUSY, PLAN_WAIT, STOP); one run in a fresh environment is _environment_run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 6 (#114): the agent phase's end, outside Harbor's timeout, does what it must Structural fix for the end of the agent phase (findings 1-7, 11): the bridge's end-of-phase work runs as a task of its own that its run awaits through any cancellation (Harbor's agent timeout can no longer cut it short), bounded by END_PHASE_S; anything it cannot confirm raises HarborDriverError, so Harbor runs no verifier. It: lets transfers in flight finish, cancels the commands, ends their clients and kills their processes (running and earlier unconfirmed kills alike), keeps a running command's stream (exec `stream=`, tee'd there as it runs) in the agent logs under its name, removes the credential files the CLI registered before its upload (Environment `secret`, confirmed), and copies Hermit's logs scrubbed. A cancelled wait on any Docker client kills it (compose up after a build timeout too); stop answers outside the lifecycle lock, ending the start's clients before the teardown. - The verdict: a verifier that failed after the agent's timeout is read by its infra policy (the driver keeps every exception Harbor recorded or skipped); a result.json Harbor could not write falls back to the trial's in-memory result, and no verdict at all is the driver's error. - A disabled verifier is refused; Windows refused at the trial's creation is UnsupportedTask. - Agents: install discovery never raises on an adapter (the deployment scores it); a deployment Harbor's phase end cut short is not adapter_failed; python3 for hooks is best effort at setup; the claude-code alias installs Claude Code; the answer agent's log is a folder with the runner's answer.json; eval checks the sign-in before replacing results. Docs: the judge's place with Harbor, installs is built-in only, the driver's intro, AHE's trace note. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Ported from the runner: a mounted CLI's checks, Claude's key, attachments, infra usage (#114) - A mounted binary (`bin`) serves only when it runs there as the pinned version (any, when none is) and the agent's user may execute it (the runner's _use_mounted): linked onto PATH, with the release's rg and this machine's CA bundle for an image that has none (harborside.codex's _after_mount); else the bridge's setup installs the CLI and the session runs the installed program (bin_served, through the proxy too). - Claude Code in a task's environment gets no raw key variable of its binding: it reads its key from its own (session_env), as the runner's launch left it. - The answer agent warns when it leaves out an attachment over 20 MiB. - A run that is not the agent's result after a session ran, with no usage reported, has an unknown cost and token count (the runner's usage None), never 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 7 (#114): no verifier after an unfinished end, the agent's requests as the agent - P1 #1: the phase's end that does not finish (its bound, or any other way) is HarborDriverError, read from the shield's own exception too: Harbor runs no verifier after it (_close_phase, tested past its bound). - P1 #2: nothing the worker asks for runs as root: credential files are removed as the agent user, an upload's missing folders are made by that user (cp -f replaces what it may replace), and the CLI's home there is the bridge's own provisioning, declared by the agent (Agent.installs `home`), made and given to the user at the phase's start. - #3: each command writes its pid at its start, the root of what kill_marked ends where root cannot read another user's environment (no CAP_SYS_PTRACE). #4: the commands are stopped as the phase closes, before a pending transfer is let finish. #5: the driver's environment drops the ambient credentials the runner's harbor_env dropped. #6: a stop Harbor acknowledged with its compose projects still there takes them down by name (a local, deleted Docker environment). #7: cp -f. #8: one absolute deadline from the phase's start (deadline_at, through the proxy): every command's time and ReAct's and the answer agent's model calls come off it. #9: a timed-out rollout's log is the trial's agent logs (the stream the bridge kept, stderr too). #10: the start's and a separate verifier's watchdogs from Harbor's budgets. #11: attachments measured and read in the environment as the agent user (through a link; one over 20 MiB never read). #12: eval checks the acting agent's sign-in only. #13: a pipeline error on a run again keeps what the runs before it spent. #14: the agent logs reach a shared verifier only (documented). - Tests no longer depend on prepared data (the eval test writes its own task set); the affected tests pass in a clean checkout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 8 (#114): the phase's end as confirmed steps, commands by their own word, the runner's checks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Round 8 triage: out-of-scope items (#8, #12, #17, #18) left as the runner had them Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review round 9 (#114): no worker request as root, archives extracted member by member, the phase's end confirmed - P1 #1: a worker's upload is never promoted to root by its name: the deployment names the agent declares are made the agent user's by the bridge itself as the phase starts (before any agent command runs), and every worker upload stays the agent user's own. - P1 #2: archives another process made (the worker's upload on this machine, the container's tar of a download) are written member by member: regular files and folders only, no absolute or `..` names, files created anew (O_EXCL | O_NOFOLLOW) through folders the extractor made, whatever Python's tarfile filter does. - #3 a transfer still running or failed at the phase's end fails it; #4 a stream that began and cannot be read fails it; #5 the deployment record (.hermit-deploy.json, Claude's .mcp.json) and an image's context link are provisioned; #6 credentials are removed though an earlier step failed; #7 Harbor ending the phase during the answer's delivery keeps its spend; #8 a command whose end is unknown is ended at the phase's end; #9 a zombie is not a live client; #10 the host kills the driver's client groups when it has to kill the driver; #11 keep_containers holds on the fallback too; #12 the loopback check knows `cli` is Codex; #13 an attachment's bytes are read between marks of their own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Harbor tasks back to the simple design: no worker, no proxy, the runner's end of the phase (#114) - A task in Harbor's environment runs its agent in this process (host_pipeline): every command and CLI runs in the task's container, the agent's loop here with the HarborEnvironment. Deleted: hermit/environment/remote.py (EnvironmentServer, ProxyEnvironment, the host-side extraction of a worker's archives), the sandboxed agent's environment server, the worker's environment, the agent-user transfer rules (as_agent) and the deployment and home provisioning they needed; Hermit's uploads are its provisioning again, root's copy given to the agent user. - The end of the agent phase at the runner's level: each command runs under setsid, its group leader's pid in a file (validated by the command's word on its command line); at done or Harbor's timeout the bridge cancels the requests in flight, kills each group still running as root with one check it is gone, removes the registered credential files, copies Hermit's logs, and returns; a kill or a removal that fails is infra (no verifier). Transfers hold one lock with the phase's end. A CLI's stream is written to /logs/agent as it runs (the runner's stream lived there). Deleted: the per-step bounded/confirmed machinery, the stream keeping, the transfer drain, the environment marks. Kept: Harbor's Trial.run, the compose clients' process groups killed on cancellation, keep_containers, the warm start, the disk gate, the cleanup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Harbor: the agent's setup within 20 minutes by default; any failure of the phase's end is infra (#114) - environment.harbor.setup_timeout_sec defaults to 1200 s (Harbor's agent setup, a CLI's install: its own 360 s default ran out on all three attempts of a real run under load); a setting still wins, and the start's watchdog follows it. - A failure of the agent phase's end of any kind is HarborDriverError (no verifier, infra), never the agent's 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Round 10: confine the adapter's code, no implicit sign-in, keep cut-off usage - A Harbor task set's agent call confines the adapter's own code as the worker did: its context tool's Session and its exports' render run in the strict adapter sandbox (hermit.adapter.session.CONFINED, per call). - A CLI in Harbor's environments needs a model binding or a login (the runner's refusal), and a remote session never copies this machine's sign-in. - The usage a CLI reported before Harbor's timeout cut it off is read from the stream it left in the trial's agent logs and counted. - stop() calls end_clients() with its current signature; the host's forced teardown signals only the driver's own group; kill_groups keeps a pid file until its group is confirmed gone; a failed kill exec still removes the credentials; a download of a link is refused; the log a record names is scrubbed of the run's keys. - Docs: no worker or proxy for Harbor; Hermit's judge on the host. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * kill_groups in POSIX sh: no `--` for dash's kill; dead processes gone CI's Ubuntu runner runs `sh` as dash, whose kill reads `--` as a pid ("Illegal number: -"), so the group was never killed and the check failed. The kill is now `kill -KILL -<pgid>` (POSIX: dash, busybox ash, bash), and a process in state X counts as gone with a zombie. Harbor's main container runs the line under bash, so trials were not affected; a sidecar or another shell would be. The tests run kill_groups under sh and bash with a PID 1 that never reaps (Python, as a container's `sleep infinity`), count the group's live processes from /proc (no ps), and once more in Debian's python:3.12-slim (dash). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ent, host faults, usage) (#116) * Final review fixes: verifier script's own environment, host faults, usage - A task's own verifier script (hermit.task.verify:script, shared or separate) runs in a sandbox built from the task's environment only: never the acting model's key variable, provider hosts or forwarded ports. - The strict adapter sandbox is preflighted at startup for a Harbor task set (the runner's sandbox_fault), and a deployment or context that fails because it stopped starting is staging_failed (infra, run again), not a scored 0. - The answer agent counts the usage an HTTP error response reports, on the scored result and on the infra end. - A ledger of only unreported sessions or calls has unknown tokens, not 0. - data.view must keep each task's id. - An unreadable optional feedback.txt leaves the verifier's reward. - AgentDyn/AgentDojo snapshots freeze data.view like the agent and judge. - Docs: Hermit's Harbor judge on the host; the verifier script's sandbox. - Minor: the RRSI trials judge loads its score module once; a dead fixture. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * PR 116 review: edges of the final fixes - sandbox_fault: bubblewrap missing under a required sandbox is a fault (the runner refused it at startup), at startup and at run time. - The answer agent's session construction (its context tool's server) is inside the guard: a sandbox that stopped starting there is infra too. - A staging failure on this machine (the strict sandbox not starting, an OSError such as a full disk) is an infra output, staging_failed: run again, nothing spent, so a later passing run keeps a known cost. - data.view reads a copy of the row: a view that rewrites its input cannot change the task or merge two tasks. - The verifier script's sandbox does not apply sandbox.hide (the agent's), as a trusted judge's never did; the model key, hosts and ports stay out. - What a failed call billed (its error's usage) is counted on the failed record outside Harbor too, across the task sandbox's worker. - Snapshots refuse a module-form data.view (read live where it is installed). - Docs: the verifier script's boundary stated as the grants it lacks. - Minor: the RRSI judge loads under a lock; the 503 test asserts its attempts and their combined cost. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * PR 116 round 2: staging on this machine, a terminal failure's spend - A CLI agent's staging folders that cannot be made (a full disk) in a task's own environment are staging_failed (run again, nothing spent), as the runner guarded its job's folders with the staging; an answer agent whose context server's files cannot be made (OSError) is staging_failed too. - A pipeline's own failure after an infra run again sums every attempt's spend through Attempts, the failed one's included when it said. - An error body's all-zero usage (a session's placeholder) is no report, as hermit.tools.failed_usage reads it. - Docs: a shared judge's verifier script runs in a sandbox of its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
11 tasks done
haizhongzheng
added a commit
that referenced
this pull request
Nov 20, 2026
…eam, refuse multi-step tasks as unsupported (#119) * Docs: Harbor design page, sandbox and benchmark checklists after the refactor After #117 the contributor checklists and the task tutorial still described the framework before it. Add docs/developer/harbor-design.md with three diagrams (the processes, a trial in order, CLI and loop agents) and who does each step, how the agent phase ends and the outcomes; let the sandbox checklist accept hermit.environment and count hermit/environment/ as core; add a benchmark checklist (task directories, public and private halves, what a task asks, parity, a real run) and say when it applies; point the task tutorial and the developer index to task directories; the dual-review reviewer is gpt-6.2-sol. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review fixes: design page and checklists checked against the code; scrub a CLI's live stream; refuse multi-step tasks as unsupported The independent review (/dual-review, 9 rounds) checked every claim of the Harbor design page, its diagrams and the checklists against the code and Harbor 0.25.0, and corrected what was false: Local is not confinement and sandbox evidence must show the isolation probe passed; only the bubblewrap task sandbox refuses a task the run cannot give, and every backend states its effective timeouts; Harbor's trial order, what is infra, which retries apply, what passes through the bridge, CLI install and reuse, the separate verifier. Two driver fixes the maintainer asked to include: - A CLI's output kept in /logs/agent as it ran reached a shared verifier unscrubbed when Harbor's timeout ended the phase before Hermit's logs were copied. The phase's end now scrubs each stream there, best effort, in a local Docker environment (never a failure of the phase's end). - A multi-step task was refused as sandbox_failed and run again; it is now refused as unsupported_task and never run again. Each has a test that fails on main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change
The task/environment refactor (design doc "Hermit agent 与 benchmark 重构设计", 第二阶段), developed on the
integration branch
task-environment-refactoras five reviewed PRs. Merge with a merge commit (not squash), soeach step stays one commit on
main:7e217556hermit/taskandhermit/environmentbeside the agents,hermit/common(structure only)52843e3bdata.view; OfficeQA as task directories125a09f7b56562b122e97d70Design in one paragraph. A task set is a directory of task directories in Harbor's format (or a
tasks.jsonl); atask has a public half (what the agent is handed) and a private half (only the judge reads). Benchmarks of two kinds:
those that really use Harbor (Docker, E2B, Modal; Harbor's verifier) and those that only use Harbor's format and run on
Hermit's own bubblewrap or local environment. For Harbor tasks, Harbor's own
Trial.run()runs each trial and one thinbridge agent relays Hermit's
exec/upload/downloadduring the agent phase, so agents stay independent ofHarbor (they only use Hermit's
EnvironmentAPI); the oldharbor runrunner andharborside(~17.8k lines) are gone.Infrastructure failures run again in a fresh environment and are never scored as the agent's 0. Every benchmark's
prompts, records and learning inputs are byte-identical to before (
data.view).Breaking changes (CHANGELOG, Unreleased): every moved task set needs its
prepare.pyrun again (OfficeQA, LIBERO,Robosuite, RoboDojo-Sim, ServeLearnBench, CL-Bench, AgentDyn/AgentDojo, the claude_subscription toy);
agent.name: harborconfigs moved toagent.base+environment.harbor;hermit.agent.environmentforwards tohermit.environmentfor one release;setup_timeout_secfor Harbor tasks defaults to 1200 s.Not supported yet (refused with a clear error): multi-step Harbor tasks (both paths); adapter tools inside a Harbor
environment; Pi, OpenCode, dsh, mini-swe-agent there; a CLI-backed model behind the answer agent there; session resume
in a remote environment.
Known issues (the old runner behaved the same; listed in #114 and #116): the bridge's pid file in agent-writable
/tmp(checked against the command's token); background processes of a finished command survive; transfer completionnot confirmed; a ChatGPT refresh lost on Harbor's hard timeout; a verifier failing before its tests run is scored 0; the
report and RRSI read
output.status; a billed quota refusal that stops the run is not counted; the SWE-bench README'stoken mean includes retries.
Validation
Linux (RHEL 9, x86_64), bubblewrap, Docker, Harbor 0.25.0 in its own venv, ASPIRE (pinned) with its services on an L40.
pytest --changed --base origin/mainin a clean worktree on this branch's tree (=25daedcb): 1932passed, 118 skipped, 0 failed. CI passed on every PR's final head.
b56562b1, and again on22e97d70where Refactor: fixes from the final review (verifier script's own environment, host faults, usage) #116 touched the path):login: chatgpt, Codex installed by Harbormain)login: chatgpt(on22e97d70)login: chatgpt(on22e97d70)No API key, ChatGPT access/refresh/id token or Fireworks key in any run record, trial directory or log; no leftover
containers, and Harbor removed the task images it built.
246/246, LIBERO 630/630, ServeLearnBench 1161/1161, CL-Bench 200/200 and 520/520, AgentDyn/AgentDojo all 11
configs); OfficeQA's prompts byte-identical; the parity script's model requests byte-identical to
main(Refactor 1/4: hermit/task and hermit/environment beside the agents (structure only) #112).Review
/dual-review: this session and Codex, gpt-6.1-sol then gpt-6.2-sol): Refactor 1/4: hermit/task and hermit/environment beside the agents (structure only) #112 3rounds, Refactor 2/4: task directories in Harbor's format, their own verifier, the bwrap backend; OfficeQA as task directories #113 5, Refactor 3/4: Harbor gives environments, data and verifiers; every agent is Hermit's #114 10 (redesigned around Harbor's
Trial.run()after 5 rounds did not converge, then simplified),Refactor 4/4: every remaining benchmark's tasks as task directories, read through a view #115 2, Refactor: fixes from the final review (verifier script's own environment, host faults, usage) #116 2; then a final whole-branch review (gpt-6.2-sol) whose findings Refactor: fixes from the final review (verifier script's own environment, host faults, usage) #116 fixed. From Refactor 3/4: Harbor gives environments, data and verifiers; every agent is Hermit's #114's round 8 on, the
user's triage standard applied: fix regressions versus main, security problems, and results or cost counted wrong;
the old runner's own issues are the known issues above.
test_layers).General checklist (added after the merge; the items above were the template's only)
hermit/task/(tasks, task sets, task directories, the script verifier),hermit/environment/(the Environment API, the bubblewrap and Docker backends, Harbor's environment, its driver andharbor_errors),hermit/common.py,hermit/trainer/rollout.py(a task's environment started, handed to theagent, verified),
hermit/isolate.py(the strict adapter sandbox for an adapter's own code),hermit/sandbox.py(small additions); the old Harbor runner and
harborsideremoved. Why: the design doc's second phase (Change).and environments import no agent (
test_layers); benchmark agents stay subclasses in their example folders (Refactor 4/4: every remaining benchmark's tasks as task directories, read through a view #115).outcome is read as the old runner read it (
harbor_errors.py, ported); infra runs again in a fresh environment andis never the agent's 0. Timeouts: the agent's budget ends 60 s before Harbor's; a failed end of the agent phase is
infra, with no verifier. Cancellation:
stopcancels the trial through Harbor's teardown; a killed driver'scompose projects are taken down by name. What still behaves as the old runner did is under Known issues.
hermit trainandhermit evalon the real programs in Validation, throughHarbor in Docker (Terminal-Bench, AlgoTune, LAB-Bench), bubblewrap (OfficeQA), ASPIRE on a GPU (LIBERO) and a stream
benchmark (CL-Bench);
tests/core/test_harbor_environment.pyruns a real Harbor trial in Docker where Docker andHarbor's venv are present.
docs/, the README, the CHANGELOG's Unreleased); the contributor checklists and aHarbor design page with diagrams follow in Harbor design page and contributor checklists; scrub a CLI's live stream, refuse multi-step tasks as unsupported #119.
Sandbox checklist
(
hermit/environment/bwrap.py, built onhermit.sandbox; launch recordsruns/<run>/sandbox/*.json) or, for a Harbor task, in the task's container (its trial directory andresult.jsonunderenvironment.harbor.jobs_dir); an adapter's own code (its context tool, its exports' render) runs in the strictadapter sandbox (
hermit/isolate.py). Not confined: Harbor's driver, which runs under Harbor's python on thismachine as
harbor rundid before, since it drives Docker; no agent, adapter or task code runs in it.hermit/environment/,hermit/isolate.py,hermit/sandbox.py).Agreed with the repository's maintainer in the design doc; no separate issue.
Test checklist
tests/core/(test_task.py,test_task_directory.py,test_harbor_environment.py) and the examples' ownutils/tests/;tests/selection.pylistshermit/task/*,hermit/environment/*andhermit/common.pyas shared; no new helper or fixture directory.test_the_environments_moved_to_the_top_level_and_the_old_path_forwards).systematic check that each review fix's regression test fails on the code before the fix.
hermit/trainer/rollout.pyandConfig.load; thedriver is tested against real Harbor in Docker; the model and the CLI are the stand-ins.
tmp_path; the Docker test skips, with its reason, without Docker or Harbor's venv.reharness) predates these PRs, onmain(Refactor: fixes from the final review (verifier script's own environment, host faults, usage) #116).🤖 Generated with Claude Code