diff --git a/AGENTS.md b/AGENTS.md index 08876658..24c77e5b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,7 @@ python -m pytest --changed # the test groups this branch touches (tests/READM - Inspect the relevant implementation, callers and tests to understand current behavior. Agents follow [Model → Base Agent → Acting / Evolver Agent](docs/framework/agent-layers.md): reuse shared model and CLI execution through Base Agents; keep task-specific - logic in `examples/`. Where an agent runs is the environment its call gives (`hermit.agent.environment`), never + logic in `examples/`. Where an agent runs is the environment its call gives (`hermit.environment`), never a setting; a benchmark's own agent subclasses a core agent (a `ReactAgent` with `@tool` methods and hooks) in its example folder instead of copying a loop; a different prompt or model is configuration, not a new agent. - For method reproductions, read the [reproduction guide](docs/developer/reproducing-a-method.md) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2c94bbb1..ca8039e4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,13 +6,81 @@ Changes are grouped by software version. Entries under Unreleased describe work ### Breaking +- Harbor gives a task set its data, its environments and its verifier; every agent is Hermit's + ([Benchmarks](docs/framework/task.md#benchmarks)). `agent.name: harbor` is gone, with `hermit/harbor/runner.py`, + `hermit/harbor/harborside/` (the HermitXXX classes, the scripted agent, the stream driver) and the runner's parts + (`clis`, `launch`, `staging`, `context`, `curate`, `errors`, `judge`, `signin`, `stream`, `tools`). A run config + names the task set's environments as `environment.harbor` (the settings that were `agent.params`: `benchmark`, + `tasks_dir`, `environment`, `verifier`, the multipliers, `agent_timeout_sec`, `docker_cleanup` and the disk gates, + `jobs_dir`, `harbor`, `pass_reward`, `reward_key`, `verifier_timeout`), its acting agent as any other + (`agent.base: codex` or `claude` with `agent.model` and `agent..web_search`; `agent.name: answer` for the + answer-only agent that replaces HermitAnswer), and its judge as `{path: hermit.task.verify, name: environment}`. + Each task's environment is Harbor's own trial of it, run whole by Harbor's `Trial.run()` (`hermit/environment/ + harbor.py` and its driver under Harbor's venv), whose agent is a bridge relaying the Hermit agent's commands + (`HermitBridge`: its setup installs what the agent declares, `Agent.installs`; its run ends its commands still + running before Harbor verifies): the run starts it behind the disk gate, hands it to the agent (`run(task, adapter, + environment=...)`), ends the agent phase and reads the trial's verdict; `hermit/harbor/docker.py` moved unchanged to + `hermit/environment/harbor_docker.py`. Codex and Claude Code run in it through its `exec` (isolation `remote`); the + agent has no worker (its loop runs in the run's process, every command in the task's container; the adapter's own + code, its context tool and its exports' render, in the strict adapter sandbox as before), and a CLI there signs in + by its model binding or a named `login` only (no copy of this machine's own sign-in). A rollout + that is not the agent's result keeps the runner's semantics (`hermit/environment/harbor_errors.py`, ported from + `hermit/harbor/errors.py`): it runs again in a fresh environment (`environment.harbor.infra_retries`, default 2, + after `infra_wait_s`, never for `disk` or `model_rejected`), and one that stays infra is `unscored`, which the judge + refuses (a failed run, never the agent's 0). A session in a task's environment is not resumed (refused: its home + there is made anew for each call). Eval rows of such a task keep its `outcome` (the runner's status); a setting + of `environment.harbor` is overridden on the command line by its path. The CLI agent's usage now counts every + attempt it ran (a provider it ran again on, as the runner counted a trial's runs); a provider that could not serve + runs the task again in a fresh environment (`environment.harbor.busy_retries`, `busy_wait_s`, the runner's). + An acting agent's model a CLI reaches (`agent.params.model: {cli: ...}`) is refused for a task in Harbor's + environments (its plan's wait would fall inside the task's time). What the runner read of a + session (a lost sign-in or a bill refused: the run ends; a configuration refused: `model_rejected`; the plan's + refusal: run again once it allows) is read the same way (`hermit/agent/base/codex_stream.py`, ported). Codex there + keeps the runner's `project_doc_max_bytes` of 262144 unless the config sets one. A hook audit that cannot be read + back leaves the session's result and usage (`hooks_audit_missing`). A task's environment refuses Hermit's network, + hidden-path and mount limits (the task's own say what it reaches). As the runner did: + `verifier_redact`, the named `reward_key` only (none there is infra), the agent's setup (a CLI's install) within + 1200 s unless `environment.harbor.setup_timeout_sec` says (Harbor's own default, 360 s, is too short under load), a ChatGPT sign-in taken back only when valid + (64 KiB, both tokens, its clock), a session's missing usage as an unknown cost. Not carried + over: multi-step tasks (streams), the adapter's tools in a task's environment, the `_bin` mount (a config + mounts a binary through `environment.harbor.environment.mounts` and names it as `agent..bin`), Harbor's own agents + (`acting_agent: {name}`), the Pi, OpenCode, dsh and mini-swe-agent CLIs in a task's environment, and the curated + trial log (the trial's directory under `jobs_dir` is kept as Harbor writes it). Every example config moved. + +- OfficeQA's task set is task directories in Harbor's format under `examples/officeqa/utils/benchmark/` (with its + `prepare.py`, `judge.py`, `task-info/`, splits, `data.json` and seed adapter): run `prepare.py` again. The agents read + each task as the row they always read, so prompts are unchanged. + +- More task sets are task directories in Harbor's format, under each example's `utils/benchmark/` with its + `prepare.py`, `judge.py`, `task-info/` and splits; their configs read each task through `data.view` as the row they + always read, so prompts, records and learning material are unchanged (`hermit.task.directory.write_task` writes + one). `examples/claude_subscription/`: its nine tasks, judge, splits and task-info moved to `utils/benchmark/`. + `examples/aspire_robosuite/` and `examples/libero/`: their launchers run `utils/benchmark/prepare.py`; + `sim_service.py` and `stage_docs.py --tasks` take the task directories. + `examples/robodojo_sim/`: its launcher runs `utils/benchmark/prepare.py`; `dojo_service.py --tasks` takes the task + directories. + `examples/servelearnbench/`: run `utils/benchmark/prepare.py` again (`windows.py` and `windows.json` moved there + too). + `examples/clbench_database/` and `examples/clbench_poker/`: run `utils/benchmark/prepare.py` again (`task-info/` + moved there too). + `examples/agentdyn/` and `examples/agentdojo/`: prepare each protocol again under `utils/benchmark/data/` (each + writes `tasks/`, `cases.txt` with every case in its order, the splits and the manifest, whose `tasks_sha256` is still + that of the rows as JSON lines, as `DATA.md` lists them); `report.py` and `compare.py --tasks` take a protocol's + `tasks/`. + +- Tasks and environments are packages of their own beside the agents: `hermit/environment/` (moved from + `hermit/agent/environment/`, which forwards for one release) and `hermit/task/` (from `hermit/task.py`: `Task` with + a public half the agent is handed and a private half only the judge reads, `Benchmark`, the verifier). `RunOutput` + and `load_callable` live in `hermit/common.py`; every name importable from `hermit.task` before still is. Tasks and + agents never import each other (`tests/core/test_layers.py`). + - One agent abstraction (#111, [agent layers](docs/framework/agent-layers.md#from-the-one-agent-refactor)): where an agent works is an environment given with each call. `BaseAgent.run(prompt, environment=..., tools=, session=, roles=)` replaces `run(prompt, workspace=, limits=)` (a path is a `Local` environment), and a CLI Base Agent refuses `isolation`, `sandbox` or `container` given as settings (use `Sandboxed(path, sandbox=...)` or `Container(path, settings=...)`). A single model call is Layer 1: `Direct(settings, model)` in `hermit.agent.model.direct` returns a `Reply`, and `make("direct")` is refused. `hermit.agent.base.direct` and - `hermit.agent.base.container` (now `hermit.agent.environment.docker`) forward for one release. A run whose acting + `hermit.agent.base.container` (now `hermit.environment.docker`) forward for one release. A run whose acting agent does not take what its method needs of an adapter (`supports` against `evolver.yaml` `needs`) is refused before it starts. @@ -47,6 +115,12 @@ Changes are grouped by software version. Entries under Unreleased describe work ### Added +- Task directories in Harbor's format as a task set (`data.tasks: `, `data.rollouts`), read without Harbor + (`hermit/task/directory.py`), with their own verifier script run in the judge's sandbox (`judge: {path: + hermit.task.verify, name: script}`: `/tests`, `/logs/verifier/reward.txt` or `reward.json`, and `feedback.txt`). + The task sandbox is built from an `EnvironmentSpec` by the bubblewrap backend (`hermit/environment/spec.py`, + `bwrap.py`), which refuses a task that asks for more network or memory than the run's environment gives. + - RoboDojo-Sim (`examples/robodojo_sim/`), a first implementation (its smoke run on the real simulator; no training run yet): its forty-two tasks with a Codex session writing a program that controls the robot, evaluated by RoboDojo as an XPolicyLab policy whose server runs the program in a sandbox with no network and no diff --git a/README.md b/README.md index b66881ea..f4d53ddc 100644 --- a/README.md +++ b/README.md @@ -300,7 +300,7 @@ Prepare the dataset and model endpoint as described in the [OfficeQA recipe](htt ```bash python -m pip install -e . --group skillopt --group gepa -python examples/officeqa/utils/prepare.py --csv /path/to/officeqa_full.csv --docs /path/to/treasury_bulletins_parsed +python examples/officeqa/utils/benchmark/prepare.py --csv /path/to/officeqa_full.csv --docs /path/to/treasury_bulletins_parsed hermit check --config examples/officeqa/gepa/smoke.yaml hermit train --config examples/officeqa/gepa/smoke.yaml hermit eval --config examples/officeqa/gepa/smoke.yaml --adapter final diff --git a/docker/README.md b/docker/README.md index c36546c8..a98deea3 100644 --- a/docker/README.md +++ b/docker/README.md @@ -69,13 +69,13 @@ before the offline runs; running it again re-syncs the clbench venv, so not whil hermit runs every agent, learner and adapter tool in a bubblewrap sandbox of its own (`hermit/sandbox.py`): user, mount, PID and network namespaces, a fresh `/proc` and `/dev`, a positive allowlist of paths. Docker allows an unprivileged process to make namespaces only with `--cap-add SYS_ADMIN`, `--security-opt seccomp=unconfined` and -`--security-opt apparmor=unconfined`, the list of `hermit/agent/environment/docker.py`, which runs Codex's bubblewrap the same way. One +`--security-opt apparmor=unconfined`, the list of `hermit/environment/docker.py`, which runs Codex's bubblewrap the same way. One more is needed for hermit's `--proc /proc`: `--security-opt systempaths=unconfined` lifts the masks Docker lays over `/proc` (`/proc/kcore`, `/proc/keys`, ...), without which a fresh `/proc` cannot be mounted in a namespace (`bwrap: Can't mount proc on /newroot/proc: Operation not permitted`); `docker.py` gives it to Claude Code's containers too, whose own sandbox mounts one. The container's user is unprivileged, so what the masks hid stays root's (`/proc/kcore` is mode 0400, `/proc/sys` is read-only to it). `NET_RAW`, `MKNOD` and -`AUDIT_WRITE` are dropped as `hermit/agent/environment/docker.py` drops them, and `label=disable` keeps an SELinux host from refusing +`AUDIT_WRITE` are dropped as `hermit/environment/docker.py` drops them, and `label=disable` keeps an SELinux host from refusing the bind mounts. The containers are not privileged, have no Docker socket and are not on the host network. `prepare` gets none of the sandbox additions. @@ -85,7 +85,7 @@ the bind mounts. The containers are not privileged, have no Docker socket and ar |---|---|---| | `HERMIT_DIR` | read-write | `PYTHONPATH`; the examples' `utils/` (tasks, splits, `data.json` with absolute paths, the `documents` link) and `runs/`. | | `CLBENCH_DIR` | read-write | `.venv/bin/python` is a link to the interpreter under `.uv-python/`, and `pyvenv.cfg` names it by absolute path; hermit's task sandbox grants `$MWM_CLBENCH` and binds the interpreter the venv names. `prepare` puts the interpreter, the venv and uv's cache under the checkout (`UV_PYTHON_INSTALL_DIR`, `UV_CACHE_DIR`) so they resolve on the host too. A venv made on the host with an interpreter elsewhere does not resolve inside. | -| `OFFICEQA_DIR` | read-only | `examples/officeqa/utils/documents` links to it and `data.json` names it. | +| `OFFICEQA_DIR` | read-only | `examples/officeqa/utils/benchmark/documents` links to it and `data.json` names it. | | `RUN_HOME` | read-write | `HOME` inside: `~/.cache` (uv, Hugging Face, the tests' `hermit-tests`), a CLI agent's home; the launchers' `$HOME/code/...` defaults land here unless overridden. | | `/tmp` | tmpfs | A fresh, memory-backed `/tmp` per container (`exec`, 8 GB); hermit's sandbox bridges live there. | diff --git a/docker/offline.sh b/docker/offline.sh index af7521b0..211bea4d 100755 --- a/docker/offline.sh +++ b/docker/offline.sh @@ -113,11 +113,11 @@ echo "== clbench: the database venue'\''s products.db" "$CLBENCH_DIR/.venv/bin/python" -c "from src.tasks.database_exploration.task import DatabaseExploration as T; T.setup()" cd "$HERMIT_DIR" echo "== examples/clbench_database" -python examples/clbench_database/utils/prepare.py --clbench "$CLBENCH_DIR" +python examples/clbench_database/utils/benchmark/prepare.py --clbench "$CLBENCH_DIR" echo "== examples/clbench_poker" -python examples/clbench_poker/utils/prepare.py --clbench "$CLBENCH_DIR" +python examples/clbench_poker/utils/benchmark/prepare.py --clbench "$CLBENCH_DIR" echo "== examples/officeqa" -python examples/officeqa/utils/prepare.py --csv "$OFFICEQA_DIR/officeqa_full.csv" --docs "$OFFICEQA_DIR/treasury_bulletins_parsed" +python examples/officeqa/utils/benchmark/prepare.py --csv "$OFFICEQA_DIR/officeqa_full.csv" --docs "$OFFICEQA_DIR/treasury_bulletins_parsed" ' ENVFILE= diff --git a/docs/developer/index.md b/docs/developer/index.md index 18952bb4..e050a0de 100644 --- a/docs/developer/index.md +++ b/docs/developer/index.md @@ -18,9 +18,11 @@ Start with the extension boundary you need. A task integration normally requires |---|---| | `hermit/cli.py`, `config.py` | Command parsing and config resolution | | `hermit/agent/` | The [agent layers](../framework/agent-layers.md): `model/` (Layer 1), `base/` (Layer 2, the Base Agents), `acting/` and `evolver/` (Layer 3) | +| `hermit/environment/` | Where an agent or a judge works: the `Environment` interface and its backends (local, bwrap sandbox, Docker) | | `hermit/agent/acting/` | The Acting Agent interface and the built-in acting agents | | `hermit/agent/evolver/` | The Evolver interface and the agent Evolver; `methods/` holds the built-in learning methods | -| `hermit/task.py` | Task records, callable loading, pipeline/judge types | +| `hermit/task/` | Tasks: `Task` (public and private halves), `Benchmark` (a task set and its splits), the judge's verdict, the records a run keeps | +| `hermit/common.py` | What every layer shares: `RunOutput` (an acting agent's result) and callable loading | | `hermit/adapter/` | Manifest loading and acting consumption | | `hermit/adapter/templates/` | Built-in adapter seeds and specifications | | `hermit/trainer/` | Training loop, rollouts, material, budgets, recovery | diff --git a/docs/developer/sandbox-design.md b/docs/developer/sandbox-design.md index 089b541c..738a04c4 100644 --- a/docs/developer/sandbox-design.md +++ b/docs/developer/sandbox-design.md @@ -21,13 +21,13 @@ Three roles run in three kinds of sandbox. | Role | Task sandbox | Judge sandbox | Evolver sandbox | Host | |---|---|---|---|---| | acting (the agent under test) | runs inside | | | only when the pipeline runs on the host (below) | -| judge | `shared`: after the agent, in a new exec | `separate` | | `host`: reads the answer | +| judge | `shared`: after the agent, in a new exec (a task's own verifier script: in a sandbox of its own over the same work directory, below) | `separate` | | `host`: reads the answer | | evolver (the learner) | | | runs inside | | | Sandbox | Contents | Never inside | Backend | |---|---|---|---| | task | the declared task paths, packages and variables, the framework's environment, the acting layer, the frozen adapter, its model's key | the data and split files, the judge and its tests, the runs and trees | bubblewrap; Docker for `environment.image` | -| judge | the task sandbox's settings, the judge's file and `judge.tests`; `separate`: only the collected paths of the agent's work, none of `environment.write` | | bubblewrap: `shared` and `separate` need the bubblewrap task sandbox (with `environment.image`, Harbor or `environment.runner: host` the judge runs on the host) | +| judge | the task sandbox's settings, the judge's file and `judge.tests`; `separate`: only the collected paths of the agent's work, none of `environment.write`; a task's own verifier script (`hermit.task.verify:script`): the task's environment only, with none of the grants the acting model adds (its key variable, its provider's hosts, its forwarded ports); what the task's environment declares still applies (a `public` network is the host's, an allowlisted host is reached), and `sandbox.hide` is not applied (it is the agent's) | | bubblewrap: `shared` and `separate` need the bubblewrap task sandbox (with `environment.image` or `environment.runner: host` the judge runs on the host; with Harbor, Harbor's verifier grades in the task's environment, `[verifier] environment_mode` shared or separate, and Hermit's judge, which only reads its verdict, runs on the host: `judge.mode` stays `host`) | | evolver | the learner CLI, the workspace under fixed names below `/w`, the framework's environment, what the method declares, its models' keys | the framework's code (unless the method needs it), the data, the judge, the test split | bubblewrap | Every sandbox is an allowlist: a path, a variable or a host that is not declared does not exist inside. Python @@ -95,10 +95,9 @@ nested bubblewrap; where it cannot nest, a required sandbox runs no such session With `environment.image` a Codex or Claude Code acting agent runs in a container of its own (the Base Agent's container isolation; `docker/codex/` and `docker/claude/` build the images), started from a worker on the host, one per call; the worker puts the adapter's code the agent renders or serves in the strict sandbox below. The pipeline runs in the host -process instead in three cases, each recorded: `sandbox.enabled: false`; a Harbor task set (`agent.name: harbor`, whose -environment is Harbor's container); and `environment.runner: host`, for a task whose own code isolates the adapter's -code it runs (AgentDyn runs learned hooks in containers of their own); a task's declared tools still run in their -sandbox. +process instead in two cases, each recorded: `sandbox.enabled: false`; and `environment.runner: host`, for a task whose +own code isolates the adapter's code it runs (AgentDyn runs learned hooks in containers of their own); a task's +declared tools still run in their sandbox. Inside the task sandbox the adapter's modules (its tools, its renders) still run in a nested strict sandbox of their own (`HA_ADAPTER_SANDBOX`, set by the worker for the code the agent starts), which only narrows what the task sandbox @@ -114,8 +113,13 @@ The run config's `judge.mode` says where the judge runs. | `shared` | the task sandbox, a new exec over the agent's work directory, with the judge's file and `judge.tests` shown to it alone | the judge reads state in place | | `separate` | a sandbox holding only the paths `judge.collect` names, copied from the agent's work through a sanitized archive and shown at the agent's own path | the result can be extracted and must be graded clean | -With the sandbox off every mode runs the judge on the host. A Harbor task set keeps Harbor's own verifier (shared or -separate as its `task.toml` says), read back on the host. +With the sandbox off every mode runs the judge on the host. A task in its own Harbor environment (`environment.harbor`) +keeps Harbor's own verifier (shared or separate as its `task.toml` says), which Harbor's trial runs in the task's +environment after the agent's phase (`[verifier] environment_mode: separate` gives it an environment of its own); +Hermit's judge (`hermit.task.verify`'s `environment`) only reads that verdict, on the host. Its agent has no worker: every +command and CLI of the agent runs in the task's own container, and the agent's loop runs in the run's process with +that environment (`host_pipeline`), the adapter's own code (its context tool, its exports' render) in the strict +adapter sandbox; the environment's driver, Docker and the task's tests stay outside the container. What a judge's result contains is the task's decision; which splits' results reach the learner is the method's. ## Evolver sandbox diff --git a/docs/framework/adapter-lifecycle.md b/docs/framework/adapter-lifecycle.md index b0226363..6ccdb11d 100644 --- a/docs/framework/adapter-lifecycle.md +++ b/docs/framework/adapter-lifecycle.md @@ -168,7 +168,7 @@ rewrite before returning it. Step 1 and step 2 use different task batches. A higher score on step 2 alone does not show improvement: the questions also changed. Run separate evaluations on the same held-out test split to compare `final`, the adapter the run started -from (`root`, or the `start_from` adapter when the config sets one, such as OfficeQA's `../utils/seed_adapter`), and +from (`root`, or the `start_from` adapter when the config sets one, such as OfficeQA's `../utils/benchmark/seed_adapter`), and `none`; see [OfficeQA evaluation](../recipes/officeqa.md#check-train-evaluate). `final` means the adapter selected by `finalize`, which defaults to the current adapter. It does not mean “the diff --git a/docs/framework/agent-layers.md b/docs/framework/agent-layers.md index c3c8c37e..364713aa 100644 --- a/docs/framework/agent-layers.md +++ b/docs/framework/agent-layers.md @@ -10,9 +10,9 @@ layers. Each layer uses the one below it through a single API and treats it as a The code follows the layers: ```text +hermit/environment/ where an agent works: the Environment API and its implementations (Local, Sandboxed, Container) hermit/agent/ ├── model/ Layer 1: model references, backends, the chat client, a single model call (Direct) -├── environment/ where an agent works: the Environment API and its implementations (Local, Sandboxed, Container) ├── base/ Layer 2: the Base Agent API and its implementations, tools, the tool server, login ├── acting/ Layer 3: the Acting Agent API and the built-in Acting Agents └── evolver/ Layer 3: the Evolver API and the agent Evolver @@ -112,14 +112,16 @@ runs before a call waits). ## Environments -Where an agent works, and what it may reach from there (`hermit/agent/environment/`). Every Base Agent takes its +Where an agent works, and what it may reach from there (`hermit/environment/`, beside the agents: a task declares +what it needs of one and a judge may run in one too, so it belongs to no layer of the agents). Every Base Agent takes its environment from the call; nothing about where it runs is a setting. | Environment | Isolation | What it is | |---|---|---| | `Local(path, limits)` | none | A directory of this machine, as it is: no path can be hidden, the network is not limited | | `Sandboxed(path, limits, sandbox=...)` | sandbox | The directory inside the run's sandbox (`hermit.sandbox.Sandbox`), which decides what the agent reads, writes and reaches | -| `Container(path, limits, settings=...)` | container | A Docker container of the session's own (`hermit/agent/environment/docker.py`: image, network, proxy); a CLI's session runs there | +| `Container(path, limits, settings=...)` | container | A Docker container of the session's own (`hermit/environment/docker.py`: image, network, proxy); a CLI's session runs there | +| `HarborEnvironment` | remote | A task's own Harbor environment (`hermit/environment/harbor.py`): Harbor starts it, runs the agent's commands there and verifies; the run gives it to the agent ([Benchmarks](task.md#benchmarks)) | API: `exec(command, cwd=, env=, timeout=, stdin=) → ExecResult(code, output, timed_out)` runs a command there (a tool of the agent's, a check) within the environment's limits: its readable mounts in place, its `timeout_s` unless @@ -128,15 +130,17 @@ refused (`Unsupported`: a `Local` one hides no path and limits no network). `upl `download(source, target)` move files in and out of the workspace (a link is copied as the link it is), `path(relative)` names a path of the workspace and refuses one outside it, and `with_limits(limits)` is a copy with other limits. `placed(path, sandbox)` is the environment an Evolver's own sessions work in: `Sandboxed` when the run's -sandbox is on, `Local` otherwise. The operations follow Harbor's environments (`exec`, `upload`, `download`), so that -a later adapter can give an agent a Harbor environment the same way; that adapter is not in Hermit yet, and a -`Container` runs a CLI's session but no other command yet (its `exec` is not implemented). +sandbox is on, `Local` otherwise. The operations follow Harbor's environments (`exec`, `upload`, `download`), and a +task's Harbor environment is one (isolation `remote`: a path there is the environment's, not this machine's; Codex and +Claude Code run in it through its `exec`, their home uploaded). A `Container` runs a CLI's session but no other command +yet (its `exec` is not implemented). ## Layer 3: Agent What the Base Agent is used for. Two kinds: -- The **Acting Agent** answers a task. Its API to the Trainer is `run(task, adapter) → output + trajectory`. It +- The **Acting Agent** answers a task. Its API to the Trainer is `run(task, adapter, environment=None) → output + + trajectory` (`environment`: the task's own, when its task set gives one, as a Harbor task set does). It places the task and the adapter in the workspace, calls a Base Agent, and reads the answer from the reply. A task supplies what is its own as **task functions** named in the configuration: `prepare` (the task's files and mounts), `prompt`, `answer` (the output from the last message) and `collect` (the task's artifacts into the @@ -150,7 +154,7 @@ What the Base Agent is used for. Two kinds: CLI has a place and names the rest in the context); `react` takes the context, the context tool and tools. A method says what its adapter needs the acting agent to take (`needs` in its `evolver.yaml`); a run whose agent does not take it is refused before it starts, never run with the export silently dropped. What is checked: the built-in - agents (Harbor's by its acting agent) and a task's own class or function that runs on this machine; a task's class + agents and a task's own class or function that runs on this machine; a task's class that runs only inside its sandbox is not imported to check it, and an agent that declares no `supports` is not checked. - The **Evolver Agent** learns. Its API to the Trainer is `step(ctx)` and `finalize(ctx)`. It reads the acting runs' @@ -328,7 +332,8 @@ Configurations that name an agent, an Evolver or a template (`agent.name: codex` | `agent.run(prompt, workspace=path, limits=Limits(...))` | `agent.run(prompt, environment=Local(path, Limits(...)))` (a path alone is a `Local`) | | `isolation`, `sandbox`, `container` in a Base Agent's settings | the environment: `Sandboxed(path, sandbox=box)`, `Container(path, settings={...})`; given as settings, refused | | `make("direct", settings, model)`, `hermit.agent.base.direct` | `Direct(settings, model)`, `hermit.agent.model.direct` (Layer 1); `make("direct")` is refused; the old module path forwards for one release | -| `hermit.agent.base.container` | `hermit.agent.environment.docker` (the old path forwards for one release) | -| `Mount`, `Limits` from `hermit.agent.base.interface` | `hermit.agent.environment` (still importable from the interface) | +| `hermit.agent.base.container` | `hermit.environment.docker` (the old path forwards for one release) | +| `agent.name: harbor` (Hermit's agents injected into `harbor run`) | Codex, Claude Code, `answer`, `react` or a custom agent that takes the `environment` argument, in the task's Harbor environment (`environment.harbor`; Pi, OpenCode, dsh, mini-swe-agent and Hermes refuse it), verified by Harbor's verifier (`judge: {path: hermit.task.verify, name: environment}`) | +| `Mount`, `Limits` from `hermit.agent.base.interface` | `hermit.environment` (still importable from the interface) | | a benchmark's own copy of the tool loop | a `ReactAgent` subclass with `@tool` methods and hooks | | `@tool(description, parameters)` | also `@tool` alone: the description from the docstring, the schema from the signature | diff --git a/docs/framework/configuration.md b/docs/framework/configuration.md index 1cd5956f..1a65d213 100644 --- a/docs/framework/configuration.md +++ b/docs/framework/configuration.md @@ -50,7 +50,7 @@ Use the same run name for training and evaluation. Unknown configuration keys ar |---|---| | `data` | Task JSONL file, split files, tasks per step, whether to shuffle the train split (`shuffle`, default true), and shuffle seed. | | `task_info` | Directory explaining the task and objective to the learner. | -| `agent.name` | A built-in agent (`react`, `codex`, `claude-code`, `pi`, `mini-swe-agent`, `opencode`, `dsh`, `hermes`, or `harbor`: a benchmark's task set in Harbor's format, [Benchmarks](task.md#benchmarks)), `file.py:Class` for the task's own `Agent` subclass, or `file.py:function` for a plain `run(task, adapter)`. `pipeline.run` is the older spelling. | +| `agent.name` | A built-in agent (`react`, `codex`, `claude-code`, `pi`, `mini-swe-agent`, `opencode`, `dsh`, `hermes`, `answer`), `file.py:Class` for the task's own `Agent` subclass, or `file.py:function` for a plain `run(task, adapter)`. `pipeline.run` is the older spelling. | | `agent.concurrency` | Maximum concurrent task executions during rollout and evaluation. | | `agent.params` | The agent's settings: for a built-in agent its model, prompt and tools; for the task's own agent whatever it reads. | | `judge.path`, `judge.name` | Scoring module/file and function name; the default name is `score`. | @@ -117,6 +117,7 @@ strict preset with the adapter read-only. A task's declared tools run with `envi | `environment.network` | `no-network` (default), `allowlist` (with `environment.allowed_hosts`, through a proxy), or `public`. The agent's model provider (as an allowed host) and the tool endpoint stay reachable. | | `environment.resources` | `memory_mb`, `file_mb`, `cpu_s` per process (positive whole numbers; null: no limit); not with `environment.image`, whose container keeps its own. | | `environment.image` | A Docker image: a Codex or Claude Code acting agent runs in a container of its own; `environment.container` gives its network and egress proxy. | +| `environment.harbor` | A task set in Harbor's environments: each task's own, Harbor's verifier ([Benchmarks](task.md#benchmarks)); its settings, or `benchmark:` the file that holds them | | `environment.runner: host` | The pipeline runs in the Trainer's process, for a task whose own code isolates the adapter's code it runs (AgentDyn's hooks in containers). Warned. `environment.read`, `write` and `packages` then build only the sandbox of the task's declared tools. | Each call gets a work directory of its own (its HOME, logs and workspaces), and a fresh directory in place of the @@ -126,7 +127,8 @@ its hosts are allowed. A Claude subscription (`login: claude`) copies nothing: ` for the call and `api.anthropic.com` is allowed. What the run keeps (the learning material, the adapter) is scrubbed of key values. The adapter's modules, when the agent's code runs them (its tools, a CLI agent's deploy render), run in a nested strict sandbox. The result comes back on a pipe only the worker can write, and its `log` must lie where the call -could write; a link is never followed. A Harbor task set (`agent.name: harbor`) runs in Harbor's container. +could write; a link is never followed. A task in its own Harbor environment (`environment.harbor`) has no worker: its +commands and CLIs run in the task's container, the agent's loop in the run's process. `agent.timeout_min` stops a call that runs longer. ### The judge: `judge.mode` @@ -134,7 +136,7 @@ could write; a link is never followed. A Harbor task set (`agent.name: harbor`) | Mode | Where the judge runs | |---|---| | `host` (default) | the Trainer's process, on the output | -| `shared` | the task sandbox, after the agent, over its work directory, with the judge's file and `judge.tests` shown to it alone | +| `shared` | the task sandbox, after the agent, over its work directory, with the judge's file and `judge.tests` shown to it alone (a task's own verifier script, `hermit.task.verify:script`: a sandbox of its own over that work directory, the task's environment without the acting model's key, hosts or forwarded ports) | | `separate` | a sandbox holding only the paths `judge.collect` names, copied from the agent's work directory through a sanitized archive | With `sandbox.enabled: false` every mode runs the judge in the Trainer's process. @@ -170,7 +172,7 @@ home's `.ssh` and `.bashrc` are absent; the command does not start unless it pas record, `sandbox/-.json` in the run's directory (`eval` writes its own beside its results): what was bound and under which names, which ports were forwarded, which variables passed, and that check. The strict spec and render checks (which show only the adapter and the template), `hermit check`'s probes (which run that check but keep -no run directory), a container's start (`environment.image`, Harbor) and a pipeline in the host process keep no such +no run directory), a container's start (`environment.image`; a Harbor environment, whose trial directory is its record) and a pipeline in the host process keep no such record; the run's card records which ran. The allowlist proxies log each request to `sandbox/network*.jsonl`. diff --git a/docs/framework/task.md b/docs/framework/task.md index cc19359f..e788bf46 100644 --- a/docs/framework/task.md +++ b/docs/framework/task.md @@ -38,7 +38,7 @@ the current `train` and `eval` commands require a configured scoring callable. Only `output` is required when constructing `RunOutput`: -```{literalinclude} ../../hermit/task.py +```{literalinclude} ../../hermit/common.py :language: python :pyobject: RunOutput ``` @@ -57,11 +57,16 @@ log paths available until the Trainer has collected them. ### Feedback result -```{literalinclude} ../../hermit/task.py +```{literalinclude} ../../hermit/task/data.py :language: python :pyobject: JudgeResult ``` +Inside the framework each row becomes a `Task` (`hermit/task/task.py`): its public half is the row, what the acting +agent is handed and what a run records as its input; its private half, a row's `_private` mapping when it has one, +only the judge reads. The run (`hermit/trainer/rollout.py`) gives the agent the public half and the judge the whole +task; the agent and the task set never import each other. + The scoring function decides what feedback to reveal. Reference answers placed in feedback become visible to the learner. The numeric score can be task-specific; the current `eval` report counts a task as solved at least once when any repeated score is at least `1.0`. @@ -90,13 +95,14 @@ agent: | `mini-swe-agent` | mini-swe-agent run in a fresh workspace the adapter is deployed into (no `login`, no roles; [tutorial](../tutorials/mini-swe-agent.md)) | `model`, `model_class`, `config`, `step_limit`, `effort`, `prompt`, `files`, `mounts`, `prepare`, `answer`, `collect`, `context_query`, `env`, `price`, `timeout_s`, `retries`, `retry_wait_s`, `workspaces` | | `opencode` | OpenCode run in a fresh workspace the adapter is deployed into (no `login`, no roles; a model reference only; the deployment's skills only) | `model`, `effort`, `prompt`, `files`, `mounts`, `prepare`, `answer`, `collect`, `context_query`, `args`, `env`, `price`, `timeout_s`, `retries`, `retry_wait_s`, `workspaces` | | `dsh` | DeepSeek Harness run in a fresh workspace the adapter is deployed into (no `login`, no roles; `permission_mode` sets its own sandbox: `read-only`, `workspace-write`, the default, or `danger-full-access`) | `model`, `provider`, `permission_mode`, `image_input`, `effort`, `prompt`, `files`, `mounts`, `prepare`, `answer`, `collect`, `context_query`, `args`, `env`, `price`, `timeout_s`, `retries`, `retry_wait_s`, `workspaces` | +| `answer` | one model call, the answer written where the task's verifier reads it ([Benchmarks](#benchmarks)) | `model`, `instruction_file`, `attachments`, `answer_file`, `answer_pattern`, `answer_format`, `max_tokens`, `wait_s`, `logs` | | `hermes` | Hermes Agent run in a fresh workspace the adapter is deployed into (no `login`, no roles, no web search; a model reference only; `max_turns` caps its model requests a turn, Hermes's 90 by default) | `model`, `provider`, `max_turns`, `effort`, `prompt`, `files`, `mounts`, `prepare`, `answer`, `collect`, `context_query`, `args`, `env`, `price`, `timeout_s`, `retries`, `retry_wait_s`, `workspaces` | ### A coding-agent CLI as the acting agent `codex`, `claude-code`, `pi`, `mini-swe-agent`, `opencode`, `dsh` and `hermes` act with a coding-agent CLI in a -workspace of Hermit's (the other way is `agent.name: harbor` with `acting_agent.cli`: the CLI in a Harbor task's -container, [Benchmarks](#benchmarks)): a task gives what is its own, in configuration and, where its workspace is +workspace of Hermit's (or, for Codex and Claude Code, in a task's own Harbor environment: [Benchmarks](#benchmarks)): +a task gives what is its own, in configuration and, where its workspace is built in code, in functions it names (`file.py:function`); the agent does the rest. OfficeQA's Codex runs are one such configuration (`examples/officeqa/utils/task.py`): @@ -238,412 +244,223 @@ whose manifest names a [`context_tool`](adapter.md#per-task-context), such a loo `session.context(query=, task=)` and records `session.last_context` in `RunOutput.extra["context_tool"]`, as `react` does, so the Evolver knows what the agent read. +## Task directories + +`data.tasks` may name a directory of task directories in [Harbor's format](https://harborframework.com/docs/tasks) +instead of a `tasks.jsonl` (`hermit/task/directory.py`; Harbor is not needed to read them): + +```text +tasks// + instruction.md what the agent is asked + task.toml [metadata]; [agent] timeout_sec; [verifier] timeout_sec, environment_mode; + [environment] network_mode, allowed_hosts, cpus, memory_mb + environment/ what the task needs of its environment (a Dockerfile for a container) + tests/test.sh the verifier; a task without tests/ is judged by the run's judge (judge.mode: host) +``` + +Each task's row (its public half, what the agent is handed) holds the instruction, its settings and its metadata, and +`harbor_task`, the directory's name; where its directory and tests are is its private half, which only the judge +reads. `data.view: :` turns each row into the row the task set's agents and learners read instead +(a benchmark moved from a `tasks.jsonl` to task directories keeps its prompts and its learning material so); what a +task asks of its environment is checked either way. A benchmark's `prepare.py` writes each task with +`hermit.task.directory.write_task(, instruction, metadata={...}, environment={"network_mode": ...})`, whose +`[metadata]` reads back as given, every value in its order (TOML has no null: None is refused). Splits name the directories (``, `:` for a multi-step task's steps, `#` with +`data.rollouts: k`). The run's `environment` must give each task what it asks: a task whose agent needs the public +network (what a task that says nothing asks, Harbor's default) or more memory than `environment.resources.memory_mb` +is refused, never run with less; so is an allowlist task whose `allowed_hosts` the run's allowlist lacks, and, with +the script verifier below, a task whose verifier needs a wider network. A task's `[agent] timeout_sec` bounds each of +its calls (with a minute for the sandbox to start), unless `agent.timeout_min` is stricter. + +A task's own verifier runs as Harbor runs it, in the judge's sandbox: + +```yaml +judge: + path: hermit.task.verify + name: script + mode: shared # over the agent's work; separate: a fresh sandbox with judge.collect's paths +``` + +`tests/` is shown at `/tests` and a fresh log directory at `/logs/verifier`; `test.sh` runs in the agent's workspace +(`HERMIT_WORKSPACE`, from the output's `extra.workspace`) with the agent's output in the file `HERMIT_OUTPUT` names, +and writes `reward.txt` (one number) or `reward.json` (named numbers: `reward` is the score, else their mean). Beyond +Harbor, `feedback.txt` there is the feedback a learning method reads (else the script's output's tail). A task whose +task.toml asks for `environment_mode = "separate"` is refused under `mode: shared`. A step's own `[steps.verifier]` +settings come before the task's, its mode and environment resolved as Harbor resolves them, and its `tests/` are laid +over the task's (its `test.sh` first); a reward that is not a finite number is an error; `[verifier.env]` is set for the script, with `${NAME}` and `${NAME:-default}` taken +from the judge sandbox's environment (`environment.env`), as Harbor resolves them; the judge's deadline is at least the +verifier's `timeout_sec` and a minute. The workspace is the agent's own +(`HERMIT_WORKSPACE`, its output's `extra.workspace`, else the directory it ran in). Not supported here yet: files a task +places in the agent's workspace before it starts, and a multi-step task (its steps share one environment in order), +which is refused. + ## Benchmarks -A benchmark enters Hermit as a task set in [Harbor](https://github.com/laude-institute/harbor)'s format: that is the -one protocol for every task, whether its agent works inside the container (a coding or terminal benchmark, a corpus -searched with shell tools) or only answers (a question stream whose state the environment keeps), instead of Docker -code or a loop of the benchmark's own. The format (`task.toml`, +A benchmark enters Hermit as a task set in [Harbor](https://github.com/laude-institute/harbor)'s format (`task.toml`, `instruction.md`, `environment/` with a Dockerfile or a prebuilt `docker_image`, `tests/test.sh` writing -`/logs/verifier/reward.txt`) is what Terminal-Bench, SWE-bench and the datasets on the Harbor hub share, and the -built-in `harbor` agent makes one rollout one Harbor trial: +`/logs/verifier/reward.txt`): what Terminal-Bench, SWE-bench and the datasets on the Harbor hub share. A task set that +needs the environments its tasks name (an image of its own each) runs in Harbor's environments: Harbor gives the +task's data, its environment and its verifier, and every agent is Hermit's. The run config says so in +`environment.harbor`: ```yaml +data: + tasks: ../utils/tasks.jsonl # the rows `python -m hermit.harbor.dataset prepare` writes (harbor_task) + test: ../utils/rows/test.txt agent: - name: harbor - concurrency: 8 - params: - benchmark: ../utils/benchmark.yaml # the keys every method's config of the example shares (the config overrides) - tasks_dir: tasks # the task directories (python -m hermit.harbor.dataset prepare writes them), - # relative to the benchmark file's folder once `benchmark` is set - model: fireworks/accounts/fireworks/models/glm-5p3 # a / reference: its endpoint and key - acting_agent: {cli: codex} # the CLI in the task's container: codex | claude | pi | opencode | dsh | - # mini-swe-agent; or answer: one model call from Harbor's process - web_search: false # required, either way (but for cli: answer) - agent_timeout_sec: 3600 # a uniform agent budget; null: each task's own + base: codex # any acting agent: Codex or Claude Code in the task's environment, the + concurrency: 6 # answer-only agent, a ReactAgent subclass whose tools use its environment + model: {name: gpt-5.6-luna, login: chatgpt, effort: medium} + codex: {web_search: false} +environment: + harbor: + benchmark: ../utils/benchmark.yaml # the settings every config of the example shares (below) + jobs_dir: ~/.cache/hermit/harbor/jobs/tb20-codex judge: - path: hermit.harbor.judge # the reward the container's verifier wrote + path: hermit.task.verify # the verdict of Harbor's verifier in the task's environment + name: environment ``` -The pipeline of a Harbor task set runs on the host: the task's container is its sandbox, and the verifier runs there -after the agent (Harbor's `[verifier] environment_mode`: shared, or a separate container). - -**A trial.** Harbor starts the task's container and the acting agent runs in it, as one HermitXXX class per CLI -(`hermit.harbor.clis`: `harborside.codex:HermitCodex`, `harborside.claude:HermitClaude`, `harborside.pi:HermitPi`, -`harborside.opencode:HermitOpenCode`, `harborside.dsh:HermitDsh`, `harborside.mini_swe_agent:HermitMiniSweAgent`): -Harbor's install of the CLI (Codex's, on an image with glibc, under the agent user's nvm, which the launch loads as it -does for Pi and dsh), or a binary of this machine mounted read-only (`_bin`, first on PATH, ahead of nvm's, used -only when it answers `--version` (asked as root) as the asked-for version and the task's agent user may execute it, -else the install), and one executor for all of them. What the session is, the CLI's home, its command line and its -environment, Hermit's side renders with the CLI Base Agent its own sessions use (`hermit.harbor.launch`: the trial's -launch plan, `/launch/plan.json`; Pi's and dsh's provider settings take the backend's `developer_role` from the -backends registry, the built-ins and the user's backends file, when it has that backend at the trial's URL, whatever -the run config's `backends:` sets for it, and `keep` otherwise, where Hermit's own sessions take the run config's -`backends:` over the registry: `hermit/harbor/FRAMEWORK_SUPPORT.md`, candidate 14), against the adapter deployed by -the CLI's host exactly as `hermit deploy` lays -it out (`hermit.harbor.staging`: the context file, skills, roles and hooks, the whole adapter in `hermit//` -of the task's workdir; its record, `.hermit-deploy.json`, names the paths of this machine it was staged at, as the -host writes it). harborside uploads the deployment and the home and runs the command as the task's agent user, -the prompt (Harbor's instruction) on its stdin and the session's stream in `/logs/agent/agent-.jsonl`; a -session still running when Harbor's timeout stops the agent is killed with its process group, so that the CLI edits -nothing under the verifier (a command it ran in a session of its own, as Claude Code runs its shell tool's, is outside -that group and goes on unless the CLI ended it within the plan's `kill_grace_s`). That kill and the steps after the -session (the adapter's notes copied out, the collected files, the files returned to this machine) run from `/`, so a -session that removed its workdir is still stopped and what it left still comes back; the notes and the collected files -go each into a new folder of root's in `/logs/agent` (which every user can write in), whatever was there removed first, -so that no link the agent's user put there is followed; a step that fails is recorded in the trial's `result.json` (`after_session_errors` in -the agent's metadata), never over the session's own outcome; a copy of the notes or of a collected file that failed (a -full disk, a read error) is removed, never kept in part, which Hermit's side would take for the whole (without it the -reply is the stream's, and the adapter's notes stay as they were); and a file to return whose check failed is such a -failure, never taken for one the session did not leave. After the trial the runner reads that stream with the same -Base Agent (the reply, whether the session was served, refused by the plan or not served by the provider, the usage at -`agent.params.price` when one is given, else at the backend's price): a trial's session is read as Hermit reads its -own. - -The adapter's tools are served from this machine for the trial (`hermit.harbor.tools`: a ToolServer in the runner's -process, on the Docker bridge's gateway, which a task container reaches, in front of a Session of the adapter in the -strict adapter sandbox, where the adapter's code runs) and named to the CLI as an MCP server, in its home (Claude -Code's on its command line; mini-swe-agent's shell calls them as the `hermit-tool` command), the host's deployment -naming no server of its own (its `prepare(serve_tools=False)`, Pi's and dsh's too); each call is logged -(`tool-calls.jsonl`). An adapter whose context is per task (a `context_tool`, or one only Hermit runs) gets the task's -context, computed before the trial in the adapter sandbox (`hermit.harbor.context`); the notes an adapter keeps -(`hermit//memory/`) come back after the session (`memory/` in the log). A key or a token reaches the container by -its variable's name only (`docker compose exec -e NAME`, Claude Code's subscription token on a descriptor): never in a -command line, a file, the plan or a log. The task's container is the session's sandbox, though: its commands run as -the CLI's user there, and one that read the CLI's memory could still find the key or the token (`harborside.claude`). -`acting_agent.settings.env` passes a variable given whole (`$NAME`) by its name; another `$NAME` inside a value is -this machine's value, written into the plan (`$PATH` is this machine's PATH), and is refused when its name may be a -secret's (holding `KEY`, `TOKEN`, `$MAX_OUTPUT_TOKENS` too, `SECRET`, `PASSWORD`, `CREDENTIAL` or `AUTH`, or with a -word `PASSWD`), as is a value given as it is to a variable named as a secret, a word of its name (between -underscores) being one of them (`DB_PASSWD: hunter2`, `NPM_AUTH: ...`, `HF_TOKEN_READ: ...`; not `MAX_OUTPUT_TOKENS` -or `GIT_AUTHOR_NAME`). The rules go by names: they refuse a setting that only looks like a secret's (`TOKEN_LIMIT: -4096`, `AUTH_MODE`, `SSH_KEY_PATH`: pass it by name, `NAME: $NAME`, which only Harbor's local Docker environment -takes), and let through a secret whose name says none of them (`PGPASSWORD`, `MYSQL_PWD`, `GITHUB_PAT`), whose value -is then written into each trial's plan: keep secrets out of the settings' values. -The container gets no other variable of this machine's: one that main's Base Agents drop from a -session's environment (`dropped_env`, this process's variables) reaches it only when the settings name it, as in -main, and on `login: claude` an `env` that names what would pay instead (`ANTHROPIC_API_KEY`) is refused, as in -Hermit's own sessions. Codex's sign-in on `login: -chatgpt` (`auth.json`) is a file of the CLI's home in the container (mode 600, the agent user's), as in Hermit's own -Codex sessions, so the session's commands can read it; once the session has ended it is copied back and removed from -the container, before a verifier that shares the container (Harbor's shared mode) runs. One whose copy-back failed -(over 64 KiB, an exec that failed) or was cut short (Harbor's second cancellation) stays until the trial's container -is removed, and with `environment.kwargs.keep_containers: true` in the stopped container Harbor keeps. The host's copy -for the trial is removed after the trial. With `login: chatgpt` (Codex) or `login: claude` (Claude Code) a trial -runs on the user's subscription and shares its plan with Hermit's other sessions on it (hermit.agent.base.login): it -waits while the plan holds sessions back, with no bound (main's `plan_max_wait_s` is no setting of the runner's: the -window's reset, a PAUSE file's removal or the run's end ends the wait), its session notes the plan's usage (Codex's as -Claude Code's), so `MAX_USED` -holds trials back as it holds Hermit's own sessions, a session the plan refused runs again once its window allows (on -Codex the refusal is read from Codex's own notices, never from what its log quotes of the session's tool calls), one -the provider could not serve runs again after a pause (`busy_retries` times over all of the trial's runs: -`infra_retries` adds none to it), and none starts or runs again once -another trial ended the run while it waited; the trial's usage is every run's (the tokens they reported, unknown only -when none did, and their model calls; the cost unknown when one run's is; each run's own in `extra.attempts` when the -trial ran more than once, which a training record keeps in its `pipeline_extra` and `hermit eval`'s rows do not; a -trial a lost sign-in ended records nothing, the usage of its earlier runs included: `LoginFailed` ends the run). -Codex's refreshed sign-in comes back from the container (a regular file of at most 64 KiB when checked: a larger one -is not downloaded, and one a process grew after the check is refused when it is taken back), even when Harbor's -timeout lands while it is being copied out (that step, and no other after-step, runs once more), and becomes the -shared copy as a session's refresh does (a later refresh of -the copy's account) when it holds a refresh and an access token, stamped at most a minute ahead of this machine's -clock in Harbor's local Docker environment, whose containers share it, an hour in another (`hermit.harbor.signin`); a -stream's step gets the shared copy at its turn, never when its rollout began. A copy comes back only when its trial -ends: trials that run at once on one copy whose token is due for its refresh race for the copy's one refresh token, -and a loser's refresh refused as already used is a lost sign-in, which ends the run, though the winner's copy is good -(it becomes the shared copy as the winner's trial ends, and `hermit train`'s resume then runs on it; `hermit eval` -stops). The answer-only agent (`cli: answer`, -`harborside.answer:HermitAnswer`) makes one model call from Harbor's process, through an endpoint the runner serves on -this machine (the adapter's context as its system text), and writes the answer into the container, for a task whose -work is a reply (LAB-Bench's questions); a request still in flight when the trial ends (Harbor's timeout stopped the -agent) is given up then, before the trial's usage is read, and leaves the trial's cost unknown. A model a CLI reaches -on a login (`model: {cli, model, login}`, each request one session on the login, in a sandbox of its own: main's -`cli_sandbox`, strict unless `sandbox_enforce: best_effort`, which `HA_ADAPTER_SANDBOX_ENFORCE`, the adapter sandbox's -variable, does not loosen) is given in -the benchmark file, whose model the runner serves itself: the trial waits for that plan before it starts (and does not -start once another trial ended the run meanwhile), its relay waits for it before each request, and a trial whose one -request was still unanswered when the agent's time ran out, held back by that plan at any time (a hold that lifted -before the time ran out too: the time the plan took was not the agent's), is infra (`usage_limit`; the run goes on), -not the agent's timeout; a session the plan refused comes back at once (the relay runs its sessions with `plan_rerun: -false`), its trial infra (`usage_limit`), run again once the plan allows (not counted in `infra_retries`, as often as -it refuses, each time after the plan's hold: an hour when nothing says when its window resets), never -waiting for the window inside the agent's time; a Codex session of the relay whose sign-in was lost ends the run, as a -trial's does. The same model in the run config's own `agent.params.model`, which the pipeline serves, is refused at -startup: its session would wait for the plan inside the request, which the runner cannot tell from the model's own -time. The runner's waits do not cover one for the plan's session slots (`SLOTS`): a request that finds every slot -taken (`SLOTS` below `agent.concurrency`, or the login's other sessions in them) waits for one inside the agent's -time, and a trial whose time ran out there is the agent's timeout, scored 0, so keep `agent.concurrency` at most the -plan's `SLOTS` less the login's other sessions. A task whose agent phase has no public network (task.toml, as Harbor -reads it) gets the model's and the tools' hosts named to Harbor (`extra_allowed_hosts`) and the CLI's web search off. -`hermit check` and `hermit train` warn of an acting agent on `login: claude` whose pipeline runs on the host, which a -Harbor task set's does, when the run config gives the login (one the benchmark file alone gives is not seen); the -unconfined sessions the warning names are a local pipeline's: a Harbor trial's session runs in the task's container, -its token on a descriptor the CLI reads once, not in the CLI's environment (`harborside.claude`). - -A task row (`tasks.jsonl`, written by -`python -m hermit.harbor.dataset index --out tasks.jsonl [--rollouts k]`) names its directory as -`harbor_task` and is the task's name, so a benchmark's existing split files apply with one rollout a task; `k` -rollouts of a task are `k` rows (`#`), which a split file then names one by one. Harbor lives in its own venv -(`~/.cache/hermit/harbor-venv`, `agent.params.harbor` names its command): Hermit's side writes one job config per -rollout and runs `harbor run` as a subprocess, killed with its process group past the trial's wait budget, and the -trial's compose projects then taken down by name (from an empty directory, `/` when none can be made, with none of -this process's `COMPOSE_*` variables: no compose file or `.env` is read, whose volumes would go too), and the half -that runs in Harbor's process, `hermit/harbor/harborside/`, imports only harbor, two of Harbor's own dependencies -(pydantic, and certifi where it is installed: Codex's agent's CA bundle of last resort) and the standard library. -Startup refuses what every trial would refuse: a login for another CLI, a login and a binding at once, no -`web_search`, a model on this machine's loopback, a sign-in or a key that is not there, Harbor's retry on `login: -chatgpt`, a plan the CLI's Base Agent cannot render (mini-swe-agent's: web search, or a model class whose replies -Hermit does not read), mini-swe-agent without its package where the adapter is deployed (the `mini-swe-agent` extra, -in `agent.params.python`, else Hermit's own interpreter) and no bubblewrap, or one that cannot start the strict -adapter sandbox (no user namespaces), while the adapter sandbox is required (`sandbox_enforce`), both for any run, -`hermit eval --adapter none`'s too, which the runner cannot tell from one that deploys adapters; the answer agent's -`attachments` with a model a CLI reaches (a session takes text only), and such a model in the run config (above); a -benchmark file whose `model` is a `/` reference or another mapping (below); `trial_cache` (there is no -trial cache: a resumed run runs its trials again); a `harbor` that is no executable file and, for a task set with a -multi-step task, one with no interpreter beside it (or beside the file a link to it names) for the stream's driver. -`hermit check` builds the runner (`agent.name: harbor` runs on the host), so it reports the first of these refusals -before a run; its probe then runs one real trial (the model's cost, the plan's usage) unless `--no-probe`. An acting -agent of Harbor's own (`acting_agent: {name}`, or a class that is no HermitXXX) takes no adapter: a rollout given one -fails before anything is staged, so `hermit eval --adapter none` is the run it takes. - -| `agent.params` | Meaning | +**A rollout.** The run (`hermit/trainer/rollout.py`) starts the task's own Harbor environment +(`hermit/environment/harbor.py`), hands the acting agent that environment with the task's public half +(`run(task, adapter, environment=env)`), runs Harbor's verifier on what the agent left there, whatever its session's +end (a timeout, a CLI that exited with an error: Harbor verifies all of them), and stops the environment. The +verifier's verdict is the output's `extra["verified"]`, which the judge reads: the reward is the score (the reward +under `reward_key`; none there is infra, as the runner read it), `passed` says whether it reached `pass_reward`, the feedback is the verifier's +`feedback.txt` (Hermit's, beyond Harbor) or its output's tail. A verifier that left no reward, or ran out of its time +on the agent's solution, scores 0, as Harbor grades it. A rollout that is not the agent's result (`infra`, read as +Hermit's Harbor runner read a trial: `hermit/environment/harbor_errors.py`, ported from its `errors.py`) is never the +agent's 0: an environment that did not start (its pull or build, Docker, the healthcheck: `sandbox_failed`; the driver: +`harbor_failed`, `harbor_timeout`), one that failed under the verifier, a verifier that left neither an exception nor a +reward, or ran no test (its tools never arrived, the host's disk full: `verifier_failed`), a session that never +started or that the provider could not serve (`session_failed`, `model_provider_failed`), a provider that refused the run's configuration before serving anything +(`model_rejected`, never run again; a prompt too long or the model's own refusal stays the agent's result), and a +provider that does not bill the account (its quota or credit spent: the run ends, as a lost sign-in does; so does a +session the account refused for its usage limit when the run holds no plan to wait for). An adapter that failed +before any session (its deployment, its per-task context, its context over Codex's cap: `adapter_failed`, +`context_over_limit`) is the agent's result, scored 0 with its message, as the runner graded it. It runs +again, the whole task +in a fresh environment, up to `infra_retries` times (default 2), each after `infra_wait_s` (300 s), unless its cause +runs no better again (a disk gate that waited its time out, `disk`; `model_rejected`); every run's usage is the +rollout's (`extra.attempts`). One that stays infra is marked `unscored` and the judge refuses to score it: the run +records a failed run (status `judge_error`, score 0, its usage and log kept), which a learner sees as one and `hermit +eval` counts among its `errors` and as 0 in its `mean_score`; `python -m hermit.harbor.report ` gives pass@1 +over the runs that are the agent's result ([Evaluation](../guides/evaluation.md)). Each eval row keeps the rollout's +`outcome` (`status`: ok, timeout, agent_error or infra; `failure_cause`, `timed_out`, `exit_code`, `unscored`, +`verifier_reward_raw`), as the runner's output kept it; a setting is overridden on the command line by its path +(`environment.harbor.infra_retries=0`). AlgoTune's judge (`examples/algotune/utils/judge.py`) reads the same verdict +and scores its speedup. + +**The environment.** Harbor runs in its own venv (Python 3.13, `harbor` 0.25.0, `~/.cache/hermit/harbor-venv`), so +each environment is a driver process under Harbor's python (`hermit/environment/harbor_driver.py`) that speaks JSON +lines with Hermit. Harbor runs the whole trial with its own `Trial.run()`: the prebuilt image pulled or the Dockerfile +built (with the build timeout and its multiplier), the healthcheck, Harbor's capability check (a task with MCP servers, +a skills folder or a Windows container is refused: `unsupported_task`, never run again), the agent's setup (within its +setup timeout, under the baseline network policy), the agent phase (its network policy, its user, its timeout), the +task's tests uploaded only for the verifier, the verifier's environment (`[verifier] environment_mode`: shared, or a +separate one), its timeout and multiplier, the reward files, the scrub of credentials, the trial's `result.json` +(under `jobs_dir/__/`) and the environment's stop. The agent in that trial is a bridge with no logic of its +own (`HermitBridge`, a Harbor agent in the driver): its setup installs what the acting agent declares it needs +(`Agent.installs`: the CLI unless `bin` names one, python3 for an adapter's hooks) with Harbor's own installers, +Debian bullseye's archived sources repaired first; its run relays the agent's `exec`, `upload` and `download` to the +environment until Hermit's agent is done, or Harbor's agent timeout ends the phase, then, as the runner's agents did: +kills the process group of each command it started that still runs (each runs under `setsid`; a kill that fails is +infra), removes the credential files the agent registered (a sign-in; one not removed is infra), and copies the +agent's logs (a CLI's stream is also written to `/logs/agent` as it runs); what an ended command left running, a +server the task asked for, stays. Harbor then verifies. The agent's own budget ends 60 s before Harbor's agent timeout (10% of a short one), so that its session's +end and logs reach the trial first. Around it Hermit keeps its warm start +(`hermit/environment/harbor_docker.py`): no environment starts while the host's /tmp or Docker's data root has less +than its gate free (room kept for the task's image when it is not here yet, `image_sizes`, and for the images other +environments of the run are still to pull), the task's prebuilt image is removed after it as `docker_cleanup` says, +and a driver that had to be killed (or the process that ended without stopping it) has its trial's compose projects +taken down by name. Every environment Harbor supports is a `type` (docker by default; e2b, modal, daytona, ... with +Harbor's extra installed in its venv). + +An environment's operations are the Environment API's ([Agent layers](agent-layers.md#environments)): `exec` runs a +command in the task's main container as the task's agent user, from its working directory (the workspace), each +variable passed by name (never on a command line of this machine), under a timeout enforced there (its process group +killed); `upload` and `download` move files (a relative path is in the workspace, an absolute one anywhere there); +`install(cli)` reports the version the setup installed (nothing installs during the phase). The task's directory and tests are the task's +private half: the agent never sees them. The agent has no worker in the task sandbox: every command and CLI of its +runs in the task's container, and its loop runs in the run's process with the environment (the environment, its +driver and Docker there too). + +**The agents.** A CLI Base Agent (Codex, Claude Code) runs where the environment runs its commands (isolation +`remote`, `hermit/agent/base/cli.py`), as the Harbor runner's launch plan ran a session in a task's container: its home +is rendered here and uploaded to `/installed-agent/hermit-home`, the CLI is the environment's (`agent..bin`, a +path there, such as a binary of this machine the environment mounts, `environment.harbor.environment.mounts`, used +when it runs there as `agent..version` (any, when none is pinned) and the agent's user may execute it, linked onto +PATH with, for Codex, the release's `rg` and this machine's CA bundle for an image that has none; else Harbor's setup +installs it, `agent..version`; nvm's and `~/.local/bin` are on PATH), the command runs +from the workspace with the prompt (Harbor's instruction, unless the agent's `prompt` says otherwise) on its stdin, its +output is the stream Hermit reads as it reads its own sessions (Codex's lost sign-in, provider failure, plan limit and +refused configuration read from its own notices, as the runner read them), and the CLI's last message comes back. Its +clock is the task's agent budget unless the agent's `timeout_s` says less. The model's key reaches the exec by name; +Claude Code's subscription token (`login: claude`) is moved into a pipe there before anything that outlives the move +starts (no process there holds it in its environment), and no variable of the agent's reaches this machine's Docker +client; on a login the run takes its admission to the plan (the wait and one of its session slots) around the whole +of the task's environment, never inside the agent's time, and a session the plan refused runs the task again once the +plan allows (not counted). Codex's own sandbox is off there (the environment is the +boundary, as Harbor's own agents run) unless `codex_sandbox` says otherwise, Claude Code runs as the environment's root +may, and the CLI's web search is off where the task's agent phase has no public network. Every sign-in file the home +took is removed from the environment before the verifier runs (a ChatGPT sign-in's refresh, and the plan's usage +reports, come back first); a removal that fails fails the call. A session is not resumed there. The +acting agent deploys the adapter as `hermit deploy` lays it out for a session there (its hooks run from `hermit//` +relative to the workspace with `python3`, installed at the setup when missing; a context file the image already has kept after +Hermit's text, the whole within Codex's cap, 262144 bytes there as the runner's unless `project_doc_max_bytes` says) +and uploads it into the workspace; the adapter's notes and the hooks' audit +and state (`hook-calls.jsonl`, `hook-state/`) come back into the log. The answer-only agent (`agent.name: answer`, +`hermit/agent/acting/answer.py`) makes one model call (the adapter's context its system message, Harbor's instruction +the question) and writes the answer where the verifier reads it (`answer_file`, `/app/answer.txt` unless null; +`answer_pattern`, `answer_format`, `instruction_file`, `attachments`: the environment's images sent with the question). +The agent's logs reach the verifier as `/logs/agent` (the stream, the answer agent's `answer.json`) in Harbor's shared +verifier mode only: a separate verifier gets its own mounts and the task's configured artifacts, not the agent logs. + +Not supported in a Harbor environment yet, and refused rather than dropped: the adapter's tools (the framework's tool +server runs on this machine), the agent's `mounts`, `prepare` and `collect`, readable mounts, hidden paths or a +network limit of Hermit's (the task's task.toml says what its environment reaches), and a multi-step task +(`[[steps]]`, which needs one environment across the Trainer's steps). The credentials an agent command was given are +scrubbed from the logs handed to the verifier and from the trial's files. + +| `environment.harbor` | Meaning | |---|---| -| `tasks_dir`, `jobs_dir`, `tmpdir`, `harbor` | the task directories; one Harbor job per rollout (outside the checkout); TMPDIR for harbor; Harbor's command, a name on PATH or a path (a relative one is the working directory's; default: `~/.cache/hermit/harbor-venv/bin/harbor`), beside which, or beside the file a link to it names, is the interpreter a stream's driver runs under | -| `model`, `login`, `effort`, `web_search`, `price` | the acting model: a `/` reference (its endpoint and key), or with `login` (`chatgpt`, `claude`) the CLI's model name on the user's subscription; the answer agent's may be a model a CLI reaches on its login (`{cli: codex \| claude, model, login}`, in the benchmark file, which the runner serves as written: each request one session of that CLI, which takes text only, each trial first waiting for that login's plan, the mapping's `plan_dir` and `usage_credits`; unlike `llm`'s, no `/` reference is bound there, so one without `login`, Pi's among them, is refused at startup), and its relay takes the client `timeout` and `max_retries` a model mapping gives beside a reference; the reasoning effort (`off`, YAML's false, gives none, as in main's sessions; the answer agent's relay sends it with each request, so an effort its model does not take is each trial's `model_rejected`, not refused at startup); the CLI's web search, required (not for the answer agent); a reference's price (USD per million tokens: `input`, `cached_input`, `output`; the config's, else the benchmark file's; default: the backend's), which prices the HermitXXX trials' usage (on Anthropic's API Claude Code's own cost is kept, as everywhere in Hermit; `hermit train` and `hermit check` check a reference's dialect as a chat client's, main's rule for an agent it does not know, so they refuse Claude Code on a backend that serves only the Messages API, Anthropic's, which `hermit eval` runs, and do not check Codex's need of the Responses API: `hermit/harbor/FRAMEWORK_SUPPORT.md`, candidate 7) and the answer agent's endpoint (one of Harbor's own agents keeps Harbor's cost) | -| `plan_dir`, `usage_credits`, `busy_retries`, `busy_wait_s`, `login_root` | on a login, as a CLI session's: the plan's directory (default: the login's own under `~/.cache/hermit/`); whether a session may draw the account's usage credits (default: no, and one that drew them holds the plan's sessions until its window resets); how many times a session the provider could not serve runs again over all of the trial's runs (3, also when null; `infra_retries` adds none), the n-th after n × `busy_wait_s` (120 s, also when null); the shared copy of the ChatGPT sign-in (`~/.cache/hermit/codex-login/`) | -| `acting_agent` | `{cli: codex}` (the default) or another CLI of the table (not `hermes`: Hermes Agent runs in Hermit's own workspaces only, and the runner refuses it at startup), with `settings` (the CLI Base Agent's own, over the table's; not `login`: a trial's login is `agent.params.login`, and one there is refused at startup), `kwargs` (the HermitXXX class's), `env` (no secrets: its values are written into each trial's job and onto `docker compose exec`'s command line, so a variable Harbor would take for a secret (a name holding `KEY`, `SECRET`, `TOKEN` but not as `TOKENS`, `PASSWORD`, `CREDENTIAL` or `AUTH`: `GIT_AUTHOR_NAME` and `TOKENIZERS_PARALLELISM` too; or with a word `PASSWD`, `DB_PASSWD_FILE` too), or a value naming one, whole as `${NAME}` or `${NAME:-default}`, which Harbor fills in, or otherwise (`$NAME`, or inside a longer value), which it hands on as that text, is refused at startup), `pythonpath`; or `{name}` / `{import_path}`: a Harbor built-in or another class, run as Harbor runs it, which takes no adapter (Harbor's own `claude-code`: its WebSearch and WebFetch off on every task with `web_search: false`, else Harbor's rule, off where the task's agent network is not public; its sign-in the key this process has, never the subscription, so `CLAUDE_FORCE_OAUTH`, or the subscription's token as its only sign-in of a model at Anthropic's API, is refused at startup: the subscription runs with `cli: claude` and `login: claude`) | -| `_bin`, `_version`, `setup_timeout_sec` | a CLI binary of this machine (`codex_bin`, `claude_bin`, `pi_bin`, `opencode_bin`, `dsh_bin`; mini-swe-agent takes none), its folder mounted read-only at `/opt/hermit-`, first on PATH (for Codex's standalone release, a `codex-package.json` in the binary's grandparent, the release's folder, with its `bin/` first on PATH), used only when it answers `--version` (asked as root) as the asked-for version (any, when none is) and the task's agent user may execute it (`test -x`: a binary or folder of mode 700 of another uid is not, unless that user is root), else the container installs the CLI; the version the container installs, the asked-for one (`codex_version`, ..., `mini_swe_agent_version`; default: the `hermit.harbor.clis` table's; its `codex` and `claude` rows give none, so Harbor's own install's); Harbor's agent setup limit | -| `answer` | the answer agent's `instruction_file`, `attachments` (globs of the images sent with the question: one over 20 MiB is left out, as a file that is no image is, and a figure a wildcard would match in a directory the agent user may not list is not found, as by a glob that matches nothing (no error: the model is asked without it; one named outright is found wherever the agent user may enter its directory); attachments whose listing exec failed, and an image that cannot be measured or read, are `launch_failed`: the model is never asked without them), `answer_file`, `answer_pattern`, `answer_format`, `wait_s`, `max_tokens` | -| `pass_reward`, `verifier_redact`, `reward_key` | the reward that is a pass (default 1.0; null: a score scale, where none is); regexes removed from the curated verifier output (a test's expected answer); the verifier's reward key | -| `agent_timeout_sec`, `environment`, `verifier`, `retry`, `verifier_timeout` | the agent's budget; Harbor's environment (`type`, `import_path`, `delete`, `force_build`, `mounts`, `extra_docker_compose`, `kwargs`), verifier and retry configs, passed through (with Harbor's retry on, `retry.max_retries` above its default 0, a failed attempt's usage is not counted: Harbor removes its trial before the next; refused on `login: chatgpt`, whose every attempt would start from the sign-in copied before the first); a verifier that ran out of its time: `agent` (the default, as Harbor grades it: the agent's result, its solution too slow for the tests, as AlgoTune's verifier times it, or a process it left that hung them) or `infra` (not the agent's result, run again, `infra_retries`: a host whose load makes verifiers time out); any other value is refused at startup | -| `verifier_timeout_multiplier`, `environment_build_timeout_multiplier` | Harbor's: each task's verifier limit, and its environment start (pull, build) limit, times this (default 1, never below 1: a slow host); the agent's budget is never scaled | -| `docker_cleanup`, `rmi_threshold_gb`, `disk_gate_gb`, `disk_gate_wait_s`, `tmp_gate_gb`, `image_sizes` | `images_keep`, `rmi_task_image` or `rmi_over_gb` (the task's prebuilt image removed after the trial, only when larger than the threshold); no trial, nor a stream's environment, starts while Docker's data root (or the host's /tmp) has less than the gate free, room kept for the task's image and for the images still to be pulled | -| `infra_retries`, `infra_wait_s` | a trial that is not the agent's result (`infra`, below) runs again this many times at most (default 2), each after this long (default 300 s): never a stream's step, whose environment went on without it, nor a task Harbor runs for none of Hermit's agents (`unsupported_task`), a step whose turn cannot come (`stream_order`), a disk gate that waited its time out, a provider's refusal of the run's configuration (`model_rejected`) or a host failure (`host_failed`: a jobs directory that cannot be made, or past the trial's staging, where its session may have run); the plan of the answer agent's model that refused its request runs the trial again once it allows, not counted; on a login, `busy_retries` counts a provider that could not serve; there is no trial cache (`trial_cache` is refused): a resumed `hermit train` runs its unfinished step's trials again, and an eval run again runs all of its own | -| `tools_host`, `sandbox_enforce`, `context_query`, `python` | where the adapter's tools are served (default: the Docker bridge's gateway); the adapter sandbox's enforce (`required`, the default unless `HA_ADAPTER_SANDBOX_ENFORCE`, the adapter sandbox's own variable, says otherwise, needs bubblewrap: without it every run of Hermit's agents, a table CLI or the answer agent, is refused at startup, `hermit eval --adapter none`'s too, which the runner cannot tell from one that deploys adapters; Harbor's own agents take no adapter and skip the check; `best_effort` renders adapters unconfined, recorded); the row's field a per-task context retrieves for; the interpreter the hosts' deployment and the per-task context run with (default: Hermit's own) | -| `benchmark` | a YAML file holding these same keys, shared by every method's config of the example (the config's params override it, and a relative path that is the config's folder's (below) is then the benchmark file's folder's, the config's own too; the sign-in it gives (`login`, `login_root`, `claude_bin`, each where the config does not give it) is checked at startup as the config's is, its `login_root` and `claude_bin` resolving against the working directory, as the config's do; its `dataset:` block is `dataset prepare`'s); its `model`, which counts only where the config gives none, is a model's name (the CLI's own with a login, or one of Harbor's own agents') or the answer agent's model a CLI reaches (`{cli: ...}`): a `/` reference or another model mapping is refused there, as it belongs in the config's own `agent.params.model`, which the config's loading binds and `hermit check` sees | -| `stream` | a multi-step task's driver: `idle_s` (the environment stops after that long without a step; 4 h, also when null), `start_timeout_s` (1 h, also when null), `turn_wait_s` (how long a step waits for its turn; when null, the budgets of the steps before it, each its own, and `start_timeout_s`) | - -A relative `tasks_dir`, `image_sizes`, `acting_agent.pythonpath`, `acting_agent.settings.config` (mini-swe-agent's) -or `environment.extra_docker_compose` is the config's -folder's, or, with `benchmark` set, the benchmark file's folder's (the config's own values too); `_bin`, -`jobs_dir`, `tmpdir`, `login_root`, `plan_dir` and a `harbor` that is a path resolve against the working directory, -with `~` expanded. The runner follows `tasks_dir`'s links once (the `tasks` link prepare writes), so a prepare that -points it at another set mid-run changes no trial's task set. - -The rollout's output is -`{harbor_task, rollout, step, reward, rewards, status, failure_cause, exception_type, trial, hooks_active}` and the -CLI's last message (`reply`; the answer agent's `answer`), with `status` one of `ok` (a verifier reward), `timeout` -(the agent's time ran out, and the verifier still scored its work: even after a provider notice in its stream that the -session recovered from, and for the answer agent with its model request still in flight, unless that request's session -had itself waited for its login's plan, as the session records its waits, however the plan stands when the trial is -read: `infra`, `usage_limit`; or the verifier's own time ran out on the agent's solution, `solution_timed_out`, unless -`verifier_timeout: infra`), `agent_error` (the CLI exited with an error, its work still scored; the model's safeguards -ended the session; the adapter's context over the CLI's limit beside the image's own; the adapter's own deployment or -per-task context failing in its child process on this machine, `adapter_failed`, its message the feedback, no -container started, unless this process then sees this machine fail any adapter's staging there (a job folder it -cannot write, less than 64 MB free under it, the strict adapter sandbox not starting: `staging_failed`, infra, which -an adapter's own staging that filled the disk reads as too, the host being checked after it; a -child killed or out of its time stays the adapter's), and a trial's launch plan refused for what its adapter is (its -tools beside `text_only`; dsh's MCP server name, made of the adapter's id: 32 characters at most of -`[A-Za-z0-9_-]`), which the plan rendered at startup, with no adapter, does not meet (a manifest this process cannot -load, hermit.adapter.loader's ValueError, is a failed run of the pipeline instead, as on `main`); a verifier that left -no reward Harbor could read, no reward file, an empty one or one that does not parse, `verifier_failed`, scored 0 as -Harbor grades it) or `infra`: not the agent's result (a sandbox, model-provider or Harbor failure, a killed `harbor -run`, the disk gate, a lost stream, an OSError on this machine while it stages the adapter (its deployment, its -per-task context) or writes the trial's job and launch, `staging_failed` (the adapter's own failing in its child -process is `adapter_failed`, above); an OSError on this machine past those steps (a process it could not start, a file -it could not write or read), `host_failed`, its cost unknown (0 for a jobs directory it could not make: nothing ran), -though a lost sign-in the rollout met before it still ends the run; a verifier whose tools never arrived or that the -host's full disk stopped before its tests wrote a report (with one, the words are a test's output), whatever it then -wrote or raised and whether the agent had finished, run out of its time or exited with an error (Harbor verifies all -three), `verifier_failed`; a trial whose environment or agent setup failed before its agent started, which costs 0; a -session that never started or whose tools never came up; for a CLI of the table and the answer agent, a task Harbor -runs for none of them, one that declares MCP servers or a skills folder or targets a Windows container, -`unsupported_task`, never handed to Harbor (Harbor's own agents run it); a Claude Code, Codex, Pi or OpenCode session, -or the answer agent's request, that the provider refused before serving it for the run's configuration, -`model_rejected`, the run going on and the trial never run again (Codex's by the error its turn failed with: a 400, a -402 or a 404 it does not retry, a provider's error body it gives as it is (a 400 it reads as an invalid request: -`{"error": ...}`, `{"detail": ...}`), its words for an API account's quota, "Quota exceeded. Check your plan and -billing details.", or its words for a plan that refuses every session until someone acts: a workspace's credits spent -or its spend cap reached, a member's limit its owner set, a ChatGPT plan without Codex; a 401 or a 403 there is a -lost sign-in): its key, its permission, its model (not found, not existing, unknown, invalid or not supported), its -account's bill (a 401, a 402, a 403 or a 404 where the failure writes it as a status, never a -number among its words, or the CLI's words for one) or a request setting it does not take (an unsupported parameter or -value, an unrecognized request argument, an unsupported operation: `max_tokens` where the model takes -`max_completion_tokens`, an effort it has not); the answer agent's by its endpoint's status, or by a 400 whose message -says one of these (the last 1,000 characters of the error's `message`, all that Hermit's endpoint says of a failed CLI -session, of an error that is itself a string, else of the body's own `message` or `detail`) or whose code says that -the runner's relay refused what a CLI's session cannot take or start with; of these, a refusal of the account's bill -(a 402 written as a status, payment required, a credit balance too low, an insufficient quota, balance, credits or -funds, Codex's quota and plan words above, billing) is `usage_limit` instead and ends the run, as an API account's -usage limit does: nothing later can be served, and so does a 429 for an API account's spent quota (OpenAI's -`insufficient_quota`, the error's code or type, or a CLI's words for it), which no pause ends, though a 429 is -otherwise the provider's (`model_provider_failed`); harborside's own setup failing: no launch plan (or a stream -step's plan the agent cannot take), the CLI's install (a package, an npm or GitHub download, the mounted binary's -probe), the workdir or its owner, the run directory, the upload of the deployment, the home, the bundle or the launch -script, the `hermit-tool` script's install, Hermit's mini-swe-agent runner that cannot run in the container, a -workdir it cannot enter, the image's own context file that cannot be read or captured whole (its copy failed -part-way), a secret the plan passes by name that is not in Harbor's environment (or an environment other than -Harbor's local `docker` one, which cannot pass it by name), a launch that exited non-zero and left no stream (the CLI -never started, as when Docker could not run its exec; the stream an earlier step of a stream left is removed before -the launch), a launch whose exec this machine could not run (an OSError: the compose client not spawned, EMFILE or -EAGAIN), or, after its non-zero exit, whose stream it could not look for (whether the CLI started cannot be told), -the answer agent's question (its instruction file not there in time) or figure that cannot be read in the container, -its attachments whose listing exec failed, or its answer that could not be kept in `answer.json` (an OSError on this -machine) or written to `answer_file` (its exec failed, or this machine could not run it: an OSError), -`launch_failed`; a plan's usage limit, a lost sign-in). On Codex a refused sign-in or key (a 401, an invalid key, not -logged in, no Codex credentials found, a refused proxy token, the ChatGPT workspace routing's 401, a refresh of the -ChatGPT sign-in the provider refused, a 403 its turn failed with; only since the session last worked, never a refusal -of a side request it worked after; never one in a notice of an error Codex recovers from and goes on: that it is -reconnecting, or falling back from WebSockets to HTTPS; nor one in what Codex's log quotes of the session's work, the -model's own text: its records of the session's tool calls, as of a rejected command or a patch whose expected lines -it did not find, and every record's further lines, a line of a JSON object among them, unless it is in Codex's own -form, an event of one of Codex's types or a log record of one of Codex's targets (`codex_core::...`), which is read -as Codex's: a session's work that prints one, rarely, can read as a lost sign-in or a refusal) is a lost sign-in, as -a refused login is on Claude Code; a refusal of the run's configuration or bill in its last failure is -`model_rejected` (above), the plan's usage limit is read from Codex's own notices alone (never what its log quotes), -and any other refusal of Codex's requests by the provider is the agent's result, as dsh's and mini-swe-agent's are; -whether the provider could not serve a Codex session is read from its last failure (the error its turn failed with, -else its last error or stderr line since the session last worked, never a warning or what its log quotes; and a turn -that failed with the error Codex was retrying, its retries run out, or a stream that ended while it retried, unless -the sign-in was lost, as on a ChatGPT login that cannot reach chatgpt.com, whose turn fails with `workspace routing -discovery failed`; never a session that failed on its own after a retry it went on from, nor a response the -provider's content filter ended, which Codex writes as a disconnected stream: the agent's result; any line when no -session began, as for a container docker could not start, whose error ends with docker's `--help` hint), as Claude -Code's, Pi's, OpenCode's and dsh's is: a notice it recovered from leaves a later failure of its own the agent's -result. A session the provider refused for what its input made the request (a prompt too long, an invalid request, a -content filter) keeps the trial's own status, `agent_error` for a CLI that exited with the error: the agent's result, -as Harbor scores it, and so is such a request of one of Harbor's own agents (litellm's `BadRequestError` or -`UnprocessableEntityError`). Harbor grades a model provider's other failures as the agent's too, a refused sign-in or -model aside (a rate limit, a lost connection or a server error, after its retries; a parameter, a response or a -budget the provider refused), where Hermit keeps them `infra` and runs them again: a provider that could not serve, -or refused the configuration, is not the agent's result. An infra rollout is `unscored` (its cause in `extra`): the -runner runs it again (never a stream's step, nor `unsupported_task`, `stream_order`, a disk gate that waited its time -out, `model_rejected` or `host_failed`), up to `infra_retries` times, each after `infra_wait_s`, every run's usage -the rollout's (`extra.attempts`), and one that ends `infra` all the same is no verdict: the judge refuses it, so the -run records a failed run (status `judge_error`, score 0, its usage and log kept), which a learner sees as one and -`hermit eval` counts among its `errors` and as 0 in its `mean_score`; `python -m hermit.harbor.report ` -gives the eval's pass@1 over the trials that are the agent's result ([Evaluation](../guides/evaluation.md)). The -judge scores the reward whenever the verifier gave one, a `timeout` or an `agent_error` included (Harbor's own job -mean does), and says the trial passed at `pass_reward`; a `timeout` or an `agent_error` whose verifier then left no -reward scores 0, Harbor keeping the agent's exception as the trial's. The log is a curated copy of the trial -(`hermit.harbor.curate`): at its top, where the methods' readers look in the log of a session of Hermit's own, the -CLI's stream under the name such a session keeps it (`agent-.jsonl`, Harbor's own Codex's and Claude Code's -too), `task.md`, `prompt.txt`, `last_message.txt` and the hook audit (`hook-calls.jsonl`, `hook-state/`); beside them -`result.json`, the agent's other outputs (`agent/`, with the ATIF `trajectory.json` of Harbor's own agents), the -tools' calls, the adapter's notes (`memory/`) and the verifier's output and reward; never the CLI's home, a sign-in -or the agent's session transcripts. A lost sign-in, or an API account's usage limit, ends the run: the rollout raises -`LoginFailed`, as a lost sign-in of a session of Hermit's own does (its message names the trial's job and the cause), -and so does every rollout of the runner that starts, or waits for the plan, a slot, the disk or a run again, after -it; nothing is written that a run of another process would meet. A lost sign-in is never run again, on a login -neither as a session the plan refused nor as one the provider could not serve. `hermit train` stops, unfinished, -before the learner is called on the step's rollouts (after the learner's call, for one in `run_agent`'s trials), and -resumes after its last finished step once the sign-in or the account is restored, that step's trials run again; -`hermit eval` stops. - -**Streams.** A task with `[[steps]]` in its task.toml (`steps//instruction.md`, `steps//tests/`) is one -environment that lives through every step: a benchmark whose next question depends on what the earlier ones left in -the environment. `dataset index` writes one row per step (`:`, with `step`, `step_index` and `steps`), so -the Trainer sees the stream as consecutive tasks and the learner acts between them; the `harbor` agent starts a driver -with the first step (`harborside/driver.py`, Harbor's `MultiStepTrial` fed one step at a time over a socket named by -a digest of the stream's directory, under `$XDG_RUNTIME_DIR` or `~/.cache/hermit/harbor/`; the stream's directory is -`jobs_dir/streams/__s__`, the id the runner's own, so that two runs on one jobs -directory, at once or one after the other, a resumed run among them, never share a stream), asks it for each step in -order with the step's own launch plan (its adapter deployed over the last step's; the adapter's tools one server for -the stream's life) and the hosts its agent phase may reach, a step whose turn has not come waiting for the earlier -ones, running in other rollouts of the batch (one whose turn cannot come ends at once: the limits below), and stops it -after the last step, an aborted step, a step whose turn could not come, or `stream.idle_s` without a request. At the -runner's exit, or once the runner is freed, the drivers of the streams that did not finish are stopped, one after -another (no thread may start at exit, so N hung drivers hold it up N × 2 minutes, and up to 40 s more for each kill, -SIGTERM then SIGKILL, and its environment's `docker compose down`), each killed if it has not stopped within 2 -minutes (a process killed by a signal leaves them up for `stream.idle_s`): no stream is resumed. A resumed `hermit -train` (a new runner) has no driver for a stream under way, so that stream's remaining steps in the epoch end `infra` -(`stream_order`), counted 0 in the step's mean and given to the learner as such, and the stream starts over at its -first step's next rollout (the next epoch). A stream that finished runs again from its first step in a new -environment in the same directory (an eval's next repeat, a later epoch), the earlier run's trial set aside -(`earlier//`), once the earlier run's driver has exited (its record says it is ending before its last answers; the -new one waits for it up to 30 s, then kills it). A step's -verifier is read as a trial's (`verifier_timeout`, the check that its tools downloaded) and takes the timeout -multipliers. With a frozen adapter, `harbor run` on the same task evaluates the whole stream in one trial. A stream's -steps run in their order, each once while the stream runs: keep its rows in file order with `data.shuffle: false` (the -Trainer shuffles a split's rows one by one), and list every step of a stream in a split that lists any; a step out of -its turn ends `infra` (`stream_order`). A stream's driver outlives the rollout that started it; it runs on the host -with the pipeline. +| `benchmark` | a YAML file holding these same keys, shared by every config of the example (the config's override it; its relative paths are its folder's; its `dataset:` block is `dataset prepare`'s) | +| `tasks_dir` | the task directories (a row's `harbor_task` names one; a directory task's private half names its own) | +| `environment` | Harbor's environment config: `type`, `import_path`, `kwargs`, `delete`, `force_build`, `mounts`, `extra_docker_compose`, `env`, `override_cpus`, `override_memory_mb`, ... | +| `verifier`, `verifier_timeout_multiplier`, `environment_build_timeout_multiplier` | Harbor's verifier config, and its multipliers (at least 1) | +| `agent_timeout_sec`, `setup_timeout_sec` | the agent's budget (Harbor's `override_timeout_sec`; null: each task's own), the agent's setup's, its CLI's install (Harbor's `override_setup_timeout_sec`; default 1200 s, not Harbor's 360 s) | +| `allowed_hosts` | hosts a task whose agent phase has no public network lets the agent reach (Harbor's `extra_allowed_hosts`); the run adds its agent's model hosts | +| `docker_cleanup`, `rmi_threshold_gb`, `disk_gate_gb`, `disk_gate_wait_s`, `tmp_gate_gb`, `image_sizes` | the warm start: `images_keep`, `rmi_task_image` or `rmi_over_gb`; the gates; the images' compressed sizes | +| `jobs_dir`, `harbor` | where each environment's trial goes; Harbor's command, beside which (or beside the file a link to it names) is the python the driver runs under | +| `start_timeout_s`, `verify_timeout_s` | how long Hermit waits for the driver's start and its verifier | +| `busy_retries`, `busy_wait_s` | how many times a session the provider could not serve runs the task again in a fresh environment, over all of its runs (default 3, not counted in `infra_retries`), the n-th after n x `busy_wait_s` (120 s) | +| `infra_retries`, `infra_wait_s` | how many times a rollout that is not the agent's result runs again in a fresh environment (default 2), each after how long (300 s); never one whose cause runs no better again (`disk`, `model_rejected`) | +| `verifier_timeout`, `pass_reward`, `reward_key` | a verifier that ran out of its time: `agent` (scored 0) or `infra`; the reward that passes (1.0; null: a scale); the reward's key (a verifier that left no reward under it is infra) | +| `verifier_redact` | regexes removed from the verifier's output the feedback and an infra verdict quote (a test's expected answer the learner must not read) | + +A task row (`tasks.jsonl`, written by `python -m hermit.harbor.dataset index --out tasks.jsonl +[--rollouts k]`) names its directory as `harbor_task`, so a benchmark's split files apply with one rollout a task; +`k` rollouts of a task are `k` rows (`#`). A directory of task directories as `data.tasks` works too. **Packaging a benchmark.** `python -m hermit.harbor.dataset prepare utils/benchmark.yaml` makes an example's data from -the `dataset:` block of its benchmark file: the registry's tasks (`ref: @`: those already in the data -directory are reused, unless the `.hermit-source.json` a download leaves there names another source, the ref with -`repo`, `registry_url` and `registry_path`, or another patch (or none where the block has one, or one where it has -none), refused before the patch runs on them, else they are downloaded into a folder of their own, `..partial` -in the data directory when it exists (else `.partial` beside it), patched there, and moved into place when -complete (a second prepare downloading there meanwhile is refused, on a filesystem that keeps file locks), and a file -where the data directory goes, or a `` already there that holds none of the set's tasks (a file, a link, a -directory not empty), is in the download's way: refused before anything is downloaded, never replaced; the data -directory is the ref, every character but a letter, a digit, `_`, `.`, `@` and `-` made `_` (`org/name@1.0` is -`org_name@1.0`; `+` of the registry options and the patch, its hook and its file's digest, after it when the -block sets any, so another registry's set of the ref, or one patched otherwise, is never taken for it, and a changed -patch's set is downloaded anew), under `~/.cache/hermit/harbor-tasks` or `$HERMIT_HARBOR_TASKS` (a relative one in the -working directory), outside the checkout, unless `data:` names another, a relative one in the benchmark file's folder, -but never the `tasks` link below nor a path in it, refused before anything is downloaded or reused) and, beside the -file, `tasks`, a link to them (it names a path of this machine: keep it out of git; anything but a link there, as a -directory of its own, is refused before anything is downloaded or reused), `tasks.jsonl`, the splits (task counts, or -`in_sample`), smoke rows, a patch of the download and, with `image_sizes: true`, the images' sizes (an image whose -size cannot be read anew keeps the one written before), all made before any is written (a split asking for more tasks -than there are, or of a fraction, a smoke task named twice, leave every file as it was), checked against the lock -beside the file (which holds no absolute path, a `registry_path` of this machine named by its file name, and is -rewritten only when its tasks or their source change, the source, its ref and registry options, another one's a -mismatch; one that cannot be read, JSON that is no lock included, is a mismatch, refused unless `--force`; -hermit/harbor/dataset.py has the block's keys). A benchmark of your own is written as task directories by your -`utils/prepare.py` (the judge goes into `tests/`, uploaded after the agent ran, so the answer is never where the agent -can read it; see `examples/officeqa/utils/harbor/build_tasks.py`). `dataset lock` records the sha256 of every task -directory (its files' paths, bytes and executable bits, not their modes, and a link by its own text, never by what it -leads to) and `dataset check` compares; `dataset splits` writes deterministic train/val/test lists by task. The task -set is checked without a model by running it with the scripted agent -(`acting_agent: {import_path: harborside.scripted:ScriptedAgent, kwargs: {script: "echo 42 > /app/answer.txt"}}`): a -script that produces the right answer must score 1 and a wrong one 0 (`tests/core/test_harbor_smoke.py` does this -against Docker). - -Limits: a stream's steps take no login's holds (they run as their turn comes) and never run again when `infra`, and a -step on a login notes nothing of the plan's usage, so what the steps used holds back none of the plan's other -sessions; a step a driver already ran is refused while its stream goes on (`infra`, `stream_order`), and its first -step run again after the stream's last step starts the stream over in a new environment (its trial directory and -replies set aside in `earlier//`: a step that leaves no output of its own reads none of the first run's, nor the -tool calls it left, its own read only once its turn came), where a later step run again ends `stream_order` (no -driver, and no rollout of its first step running), so `run_agent` runs a training step in the Trainer's own stream of -its task, out of its turn unless it is the step the stream waits for ([Known -limitations](../reference/compatibility.md#known-limitations)), a first step it runs once the Trainer's pass of the -stream finished starts a new pass that the next epoch inherits (that epoch's first step `stream_order`, its later -steps in the environment the candidate's pass left), and a validation step it runs a second time ends `stream_order` -unless it is the first step of a stream that had finished; a step whose turn cannot come ends `infra` (`stream_order`) -at once, not after `stream.turn_wait_s`, and starts no driver: its driver waits for a step no rollout is running (one -that ended before it reached the driver, as its staging, its tools or its launch plan failed on this machine or the -disk gate held it; or one that ran in an earlier training step), or it has no driver (stopped, or idled out) and no -rollout of its first step is running to start one, and then it ends before anything is staged for it; a step that -ended in any way before its driver was asked for it (those, its own folder not made on this machine, `host_failed`, -its turn's wait running out, its turn's sign-in or tools -not readied, an exception of any of them) gives its stream up, so that the driver is stopped once the stream's -rollouts have ended (that step's own, when it was the last) and every later step of the pass ends so, and the stream's -next pass starts from its first step in a new environment; a lost sign-in in a step ends the run, whose resume loses -the stream's remaining steps in that epoch (above); a driver that answers no stop within 15 minutes (hung in a step) -is killed with its process group and its trial's compose projects are taken down by name, as an expired trial's are -(best effort: `docker compose ls` lists what is left); whether the adapter's tools came up is read from Claude Code's -and dsh's streams (Codex, Pi and OpenCode run on without them, unsaid, as in Hermit's own sessions; mini-swe-agent's -runner lists them before the model's first call, so a tool server it cannot reach then ends the session before it -started, `session_failed`, and one lost later makes each `hermit-tool` call an error the model reads); a process the -agent started in a session of its own outlives the kill until the container stops; Harbor's own built-in agents take -no adapter. A config for `hermit eval` alone, such as a baseline's, still gives the keys every config needs: -`data.train`, which the eval reads and checks, `template`, whose manifest it reads, and `evolver.name`, which with the -template names the run's folder unless `trainer.run_name` does (`//eval//`: two such configs -of the same evolver and template share it, the second eval's results replacing the first's). - -How the Harbor runner fits the framework's interfaces without changing them, what that costs, and the candidates for -framework support are in `hermit/harbor/FRAMEWORK_SUPPORT.md`. +the `dataset:` block of its benchmark file: the registry's tasks (`ref: @`, downloaded under +`~/.cache/hermit/harbor-tasks` or `$HERMIT_HARBOR_TASKS`, patched when the block names a patch, reused when already +there from the same source), and beside the file `tasks` (a link to them: keep it out of git), `tasks.jsonl`, the +splits (task counts, or `in_sample`), smoke rows and, with `image_sizes: true`, the images' sizes, checked against the +lock beside the file (`hermit/harbor/dataset.py` has the block's keys). `dataset lock` records the sha256 of every task +directory and `dataset check` compares; `dataset splits` writes deterministic train/val/test lists by task. A benchmark +of your own is written as task directories by your `utils/benchmark/prepare.py` (`write_task`; a verifier of its own +goes into `tests/`, uploaded after the agent ran, so the answer is never where the agent can read it; see +`examples/officeqa/utils/harbor/build_tasks.py`). A task set of task directories that Hermit runs in its own sandbox or +on the host, with a judge on the host (`judge.mode: host`, no `tests/`), is what the examples whose agent owns its loop +use (`examples/aspire_robosuite/`, `clbench_*`, `servelearnbench/`, `agentdyn/`, ...; [Task directories](#task-directories)). +A task set is checked without a model by running a command of a known answer in each task's environment and verifying +it (`tests/core/test_harbor_environment.py` does this against Docker). ## What the task owner provides | Input | Purpose | Used by | |---|---|---| -| `tasks.jsonl` | Task dictionaries with unique IDs. | Pipeline and scoring function. | +| `tasks.jsonl`, or task directories | Task dictionaries with unique IDs ([Task directories](#task-directories): one each, read through `data.view`). | Pipeline and scoring function. | | Split files | One task ID per line for train, optional validation, and test. | Trainer and evaluation. | | `task-info/` | Description of the task, objective, feedback, and execution. | Evolver. | | Pipeline callable | Adapter integration and acting execution. | Training, evaluation, and candidate experiments. | diff --git a/docs/get-started/installation.md b/docs/get-started/installation.md index 82cafad1..54e95c4e 100644 --- a/docs/get-started/installation.md +++ b/docs/get-started/installation.md @@ -90,10 +90,7 @@ never shown). ## Install mini-swe-agent The `mini-swe-agent` acting agent and `hermit deploy --host mini-swe-agent` need mini-swe-agent (2.4.6, which brings -litellm), published as an extra, and so does a Harbor task set acting with it (`acting_agent: {cli: mini-swe-agent}`): -the task's container installs its own, but this machine deploys each trial's adapter from the package's config, so -startup refuses the run (`hermit eval --adapter none` too, which the runner cannot tell from a run that deploys -adapters) when the interpreter the deployment runs with (`agent.params.python`, else Hermit's own) does not find it: +litellm), published as an extra: ```bash python -m pip install -e ".[mini-swe-agent]" # a checkout diff --git a/docs/guides/claude-subscription.md b/docs/guides/claude-subscription.md index e81b7f79..0f242cc8 100644 --- a/docs/guides/claude-subscription.md +++ b/docs/guides/claude-subscription.md @@ -101,7 +101,7 @@ unconfined as everything else does, and `sandbox.enforce: best_effort` runs it u A Codex text session can still run commands inside its sandbox; a Claude Code one has no tools at all. `cli_problems` in `hermit/agent/base/endpoint.py` has the rule for each. -A container for Claude Code is Codex's (`hermit/agent/environment/docker.py`, +A container for Claude Code is Codex's (`hermit/environment/docker.py`, [the limits](../framework/agent-layers.md#limits)). `docker/claude/` builds the image from the machine's own Claude Code, one file of the version the image is named for (it does not update itself there), root's and execute-only for the session's user: the kernel then keeps the running CLI's memory and descriptors from every other process of that user. diff --git a/docs/guides/evaluation.md b/docs/guides/evaluation.md index 1eeafcf0..c2e23b3f 100644 --- a/docs/guides/evaluation.md +++ b/docs/guides/evaluation.md @@ -14,9 +14,7 @@ file-check success, and a learner's own claims are different evidence. | Adapter directory | That directory, resolved relative to the config | How does an included or external adapter perform? | `final` is the Evolver's final selection, defaulting to the current adapter. It is not a universal best-score -checkpoint. GEPA and SkillOpt use their retained validation candidates for final selection. A Harbor task set whose -acting agent is one of Harbor's own takes no adapter: only `--adapter none` evaluates it (given any other, its -rollouts fail). +checkpoint. GEPA and SkillOpt use their retained validation candidates for final selection. ## Keep the protocol fixed @@ -39,8 +37,6 @@ hermit eval --config run.yaml --adapter final eval.repeats=3 Results are written under `//eval//`. Re-evaluating the same selector replaces its `results.jsonl` and `summary.json`; archive records or use separate output directories when preserving comparisons. -On a Harbor task set each repeat runs a multi-step task's stream again from its first step, in a new environment, -once the repeat before it has finished the stream ([Benchmarks](../framework/task.md#benchmarks)). | Summary field | Meaning | |---|---| @@ -55,19 +51,16 @@ once the repeat before it has finished the stream ([Benchmarks](../framework/tas The `solved_once` threshold is fixed at 1.0. For a 0–100 score or a non-success metric, it is not a meaningful solved rate; use the benchmark's own definition and raw scores. It is also distinct from mean per-attempt accuracy. -On a Harbor task set, a trial that is not the agent's result (`infra`: the host, the model provider, a session that -never started) runs again in the runner itself (`agent.params.infra_retries`, default 2, each after `infra_wait_s`, -default 300 s), unless its cause runs no better again: a disk gate that waited its time out, a provider's refusal of -the run's configuration (`model_rejected`), a host failure (`host_failed`: a jobs directory that cannot be made, or -past the trial's staging, where its session may have run), a task Harbor runs for none of Hermit's agents, or a -multi-step task's step. One that ends `infra` all the -same gets no score: the judge refuses it, so its record has status `judge_error`, its cost and log kept, and it counts -in `errors` and as 0 in `mean_score`. A benchmark's own figure leaves such trials out: `python -m hermit.harbor.report +On a task set in Harbor's environments, a run that is not the agent's result (`infra`: the task's environment did not +start, failed under the verifier or its verifier ran no test, the session never started) runs again in a fresh +environment (`environment.harbor.infra_retries`, default 2, each after `infra_wait_s`, default 300 s), unless its cause +runs no better again: a disk gate that waited its time out, a provider's refusal of the run's configuration +(`model_rejected`). One that ends `infra` all the same gets no score: the judge refuses it, so its record has status +`judge_error`, its cost and log kept, and it counts in `errors` and as 0 in `mean_score`. A benchmark's own figure leaves such runs out: `python -m hermit.harbor.report //eval/` reads the eval's `results.jsonl` and gives, over all repeats and for each, the trials scored, infra and otherwise failed, pass@1 over the scored ones (the judge's `passed`), their mean score -and `mean_score`. A lost sign-in, an API account's usage limit, or a provider that does not bill the account (its -quota or credit spent), stops the eval -([Benchmarks](../framework/task.md#benchmarks)): it writes no row of the repeat it stopped in (a repeat's rows are +and `mean_score` ([Benchmarks](../framework/task.md#benchmarks)). A lost sign-in, or an API account's usage limit, +stops the eval: it writes no row of the repeat it stopped in (a repeat's rows are written once all its trials are done), so its `results.jsonl` holds the repeats it finished, none with one repeat, beside the `summary.json` of the eval before, if any (the eval writes its summary when it ends). The report says such an eval did not finish (`unfinished`, and `finished: false` with `--json`). Run it again before reading either. Its diff --git a/docs/guides/model-backends.md b/docs/guides/model-backends.md index 2ae043cb..f636ef6e 100644 --- a/docs/guides/model-backends.md +++ b/docs/guides/model-backends.md @@ -31,7 +31,6 @@ hermit train --config run.yaml model=vllm/Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 | `agent.params.model` with `agent.name: opencode`, `evolver.params.model` of the `opencode` Evolver | OpenCode | chat completions | | `agent.params.model` with `agent.name: dsh`, `evolver.params.model` of the `dsh` Evolver | DeepSeek Harness | chat completions, else the Messages API, else the Responses API: the first the backend serves | | `agent.params.model` with `agent.name: hermes`, `evolver.params.model` of the `hermes` Evolver | Hermes Agent | chat completions, else the Messages API, else the Responses API: the first the backend serves | -| `agent.params.model` with `agent.name: harbor` (else the top-level `model`, else the benchmark file's `model`, a model's name or, for the answer agent, one a CLI reaches: a `/` reference goes in the config) | a Harbor task set's acting agent: the CLI of `acting_agent.cli` in the task's container, or the answer agent (`cli: answer`) from Harbor's process ([Benchmarks](../framework/task.md#benchmarks)) | that CLI's, as in the rows above; chat completions for `cli: answer` (`hermit check` and `train` check it as chat completions whatever the CLI: below) | | `model` (top level) | the default for all of the above when they name none | | | the examples' task agents (`agent.params.model`, or the launchers' `MW_MODEL`) | CL-Bench's litellm host, ServeLearnBench's loop, the SkillOpt rollout | chat completions | @@ -134,11 +133,8 @@ speaks the OpenAI Responses API only** (`wire_api = "chat"` is rejected since Co Anthropic Messages API only**. A backend's `apis` says what it serves, and the framework checks a model's dialect against its backend before a run starts: `hermit check` reports it, `train` refuses to start on a mismatch, and `hermit backends check ` sends -one short request in each dialect (and lists `GET /models`) so a wrong claim shows up before anything is spent. A -Harbor task set's acting model (`agent.name: harbor`) is checked as chat completions whatever its CLI, as for an agent -the check does not know: Claude Code on a backend that serves only the Messages API (Anthropic's) is refused by -`check` and `train`, though `hermit eval` runs it, and a Codex model on a backend without the Responses API passes -them, every trial then failing at the provider (`hermit/harbor/FRAMEWORK_SUPPORT.md`, candidate 7). +one short request in each dialect (and lists `GET /models`) so a wrong claim shows up before anything is spent. An +acting agent in a task's Harbor environment is checked as the same agent anywhere (its CLI's dialect). | Backend | chat | responses (Codex) | messages (Claude Code) | |---|---|---|---| diff --git a/docs/guides/sandbox.md b/docs/guides/sandbox.md index e12f6ddf..f61b6c6a 100644 --- a/docs/guides/sandbox.md +++ b/docs/guides/sandbox.md @@ -22,7 +22,7 @@ them. | Sandbox | Who runs inside | What is inside | Never inside | |---|---|---|---| | task | the acting agent, one worker per call; a task's declared tools; a `shared` judge after the agent | the declared task paths, packages and variables; the framework's Python environment; the acting layer; the frozen adapter; its model's key | the data and split files, the judge and its tests, the runs and trees | -| judge | the judge in `shared` mode (a new exec in the task sandbox) or `separate` mode (a fresh sandbox) | the judge's file and `judge.tests`; `separate`: only the paths `judge.collect` names, copied through a sanitized archive | in `separate` mode, everything else the agent left | +| judge | the judge in `shared` mode (a new exec in the task sandbox) or `separate` mode (a fresh sandbox); a task's own verifier script (`hermit.task.verify:script`) in a sandbox of its own in either mode, over the agent's work directory when `shared` | the judge's file and `judge.tests`; `separate`: only the paths `judge.collect` names, copied through a sanitized archive; a verifier script: the task's environment, none of the acting model's grants (its key, its provider's hosts, its forwarded ports) | in `separate` mode, everything else the agent left | | evolver | the learner's sessions, its roles, the programs it runs | the workspace under `/w`, the learner CLI, what the method declares, its models' keys | the framework's code (unless the method runs it), the data, the judge, the test split | With `judge.mode: host` (the default) the judge runs in the host process on the downloaded output. Three cases run the diff --git a/docs/guides/troubleshooting.md b/docs/guides/troubleshooting.md index 8f2eb68d..6a8c1767 100644 --- a/docs/guides/troubleshooting.md +++ b/docs/guides/troubleshooting.md @@ -13,7 +13,7 @@ can make paid calls. | `no such option: --group` | pip version | Run `python -m pip install --upgrade "pip>=25.1"`, then retry the group install | | Learner authentication fails | Learner endpoint, key variable, backend CLI | Verify the configured variable is set without printing it; authenticate the selected CLI | | `login: claude signs in with the token ...`, `Not logged in`, or a run that stopped with `hermit_login_failed` | `CLAUDE_CODE_OAUTH_TOKEN` | Run `claude setup-token` and export the token, then rerun `hermit train`: it resumes after the last finished step. See [Claude subscription](claude-subscription.md) | -| On a Harbor task set, `hermit train` or `hermit eval` stopped with `a Harbor trial (): login_failed ...` (Harbor's own agents: `model_provider_failed (AgentAuthenticationError)`, or `(ApiUsageLimitError)` for an API account's usage limit; a table CLI's: `usage_limit` on an API account, or on one the provider does not bill, its quota or credit spent) | The trial's job and curated log under `agent.params.jobs_dir`; the model reference's key variable, or the login's sign-in | Export a valid key, or sign in again (`codex login` on `login: chatgpt`: [Run with your coding plan](../tutorials/codex-plan.md); `claude setup-token` on `login: claude`), then rerun: `hermit train` resumes after its last finished step, running that step's trials again (a multi-step task's stream under way loses its remaining steps in that epoch, `stream_order`), and `hermit eval` runs again; a stopped eval wrote no row of the repeat it stopped in (its `results.jsonl` holds the repeats it finished, beside the summary of the eval before), which `python -m hermit.harbor.report` marks unfinished. See [Benchmarks](../framework/task.md#benchmarks) | +| A task's Harbor environment did not start (`the task's environment could not be given`) | `driver.log` and the trial's `trial.log` under `environment.harbor.jobs_dir/__/` | The image's pull or build, the disk gate (`disk_gate_gb`, `tmp_gate_gb`), Docker's daemon; Harbor's venv (`~/.cache/hermit/harbor-venv`). See [Benchmarks](../framework/task.md#benchmarks) | | Sessions on a login wait and nothing runs | The plan's files and report under `~/.cache/hermit/claude-login/plan/` (Codex: `codex-login/plan/`) | `plan.json` says which window is over and until when; remove `PAUSE`, raise `MAX_USED`, or wait for the reset. See [The plan](claude-subscription.md#the-plan) | | `train refuses to start: ... serves [...], not 'responses'` (or `'messages'`) | The model's backend and the caller's dialect | Codex speaks the Responses API only, Claude Code the Messages API only: bind a backend that serves it (vLLM, SGLang, Fireworks, DeepSeek) or a translating proxy; `hermit backends check ` tries each dialect. See [Model backends](model-backends.md) | | Actor works but learner fails, or conversely | Separate actor/learner configurations | The pipeline owns acting-model settings; `llm` and backend parameters own learner settings | diff --git a/docs/methods/ahe_adapter.md b/docs/methods/ahe_adapter.md index d27bbb97..6015799f 100644 --- a/docs/methods/ahe_adapter.md +++ b/docs/methods/ahe_adapter.md @@ -155,7 +155,7 @@ this framework's own code. | Dimension | AHE (8b2a55d) | This Evolver | Why | |---|---|---|---| | When learning happens | Once per iteration, after k = 2 rollouts of every task; 10 iterations | The same: a step is every trial row, ten steps | Same | -| What the learner sees | The Debugger: each task's NexAU traces (with a per-turn timing breakdown for a timed-out rollout) and verifier output; the Evolve Agent: the overview, per-task reports, verdicts, history | The same, with the acting agent's event stream (the one its CLI wrote; whole, and one file per standard step) and, on the hosts that run hooks (Codex, Claude Code), the hook audit as the traces, and the judge's feedback as the verifier output; no timing breakdown, which comes only from an ATIF trajectory (a Harbor trial's `agent/trajectory.json`, which only Harbor's own agents write: a HermitXXX trial's log holds its CLI's stream, `agent-.jsonl`, as Hermit's own sessions' logs do) | Framework: the host's trace format | +| What the learner sees | The Debugger: each task's NexAU traces (with a per-turn timing breakdown for a timed-out rollout) and verifier output; the Evolve Agent: the overview, per-task reports, verdicts, history | The same, with the acting agent's event stream (the one its CLI wrote; whole, and one file per standard step) and, on the hosts that run hooks (Codex, Claude Code), the hook audit as the traces, and the judge's feedback as the verifier output; no timing breakdown, which comes only from an ATIF trajectory (a Harbor trial's `agent/trajectory.json`, which only Harbor's own agents write: a trial of Hermit's agent holds its CLI's stream, `agent-.jsonl`, as Hermit's own sessions' logs do) | Framework: the host's trace format | | What is learned | NexAU's prompt, tool descriptions and implementations, middleware classes, skills, sub-agents, memory | The adapter's files (the table above), the same on either host; the host's loop and tools are fixed | Framework: the acting agent is fixed; the reason for the name | | The update | Edits plus a manifest of predictions, attributed at the next iteration; Algorithm 1 reverts HARMFUL changes, the released code leaves it to the agent | The same manifest and rules; `rollback: auto` (the paper) or `agent` (the code) | Decision: the paper by default | | Final harness | The best iteration by pass@1 | The best-ever commit, the harness its rollouts ran (Algorithm 1 line 15 names the harness after the next edit, which no rollout measured) | Decision | diff --git a/docs/methods/aspire.md b/docs/methods/aspire.md index 5079c42a..191ad502 100644 --- a/docs/methods/aspire.md +++ b/docs/methods/aspire.md @@ -23,7 +23,7 @@ agent: environment: # the task sandbox: its code, ASPIRE's prompts and docs, the service read: [../utils/agent, $ASPIRE_DOCS, ...] env: [ASPIRE_SIM_URL] -judge: {path: ../utils/judge.py} # on the host: Stage 2 +judge: {path: ../utils/benchmark/judge.py} # on the host: Stage 2 sandbox: {enabled: true, enforce: required} evolver: name: aspire # the coordinator: a Codex session with ASPIRE's guide @@ -46,7 +46,7 @@ the template's paths and commands onto the workspace. The sandbox shows nothing shipped library, worked solutions or examples. It has no GPU, so the simulator runs on the host as a service (`utils/agent/sim_service.py`), which `bin/sim` reaches on a forwarded loopback port; the session registers there before the agent starts (its workspace and task id: one session per rollout), and the service runs only the seeds of -the task row it reads on the host, the unseen check once and a verifier's two programs. The judge (`utils/judge.py`, +the task row it reads on the host, the unseen check once and a verifier's two programs. The judge (`utils/benchmark/judge.py`, on the host) runs the selected program once on the held-out seeds: ASPIRE's Stage 2, the coordinator's in ASPIRE. The verifier is the same actor on the task's verifier row (`verify-`, `data.val`): ASPIRE's verifier prompt, five seeds, two attempts, scored on its own seeds with its own verdict. @@ -99,7 +99,7 @@ is the evaluation on the test split (`rerun-` rows, the same seeds), compa The Codex actor's settings (`agent.params`) are `aspire_root`, `aspire_docs` (the docs as the launcher stages them), `max_steps` and `evosearch` (default true; false leaves out the evolutionary search: its part of the prompt, its page in the workspace, and its program as the scored one); its seeds are the task rows' -(`utils/prepare.py`, which the launcher runs: `ASPIRE_EVAL_SEEDS` for the held-out ones), and the simulator's +(`utils/benchmark/prepare.py`, which the launcher runs: `ASPIRE_EVAL_SEEDS` for the held-out ones), and the simulator's environment and workers are the simulator service's (`ASPIRE_SIM_ENV`). The direct comparison actor (`run_direct.yaml`) reads `debug_seeds`, `workers`, `aspire_root` and `sim_env` from `agent.params`, a task row's own seeds winning; the held-out seeds are the task rows' for both actors (the judge reads them there). The verifier's rows diff --git a/docs/methods/evosafe.md b/docs/methods/evosafe.md index 4703df9b..9f9505d2 100644 --- a/docs/methods/evosafe.md +++ b/docs/methods/evosafe.md @@ -29,7 +29,7 @@ a domain plugs in through the [task contract](#the-task-contract). The first dom ```yaml template: evosafe@1.0 agent: {name: ../utils/bridge.py:run, base: direct, model: vllm/nvidia/nvidia/nemotron-3.5-lightning} # the domain's hook bridge -judge: {path: ../utils/judge.py} # native verdicts as metrics in JudgeResult.extra +judge: {path: ../utils/benchmark/judge.py} # native verdicts as metrics in JudgeResult.extra evolver: name: evosafe base: codex diff --git a/docs/recipes/agentdojo.md b/docs/recipes/agentdojo.md index d61e5894..b92a79b5 100644 --- a/docs/recipes/agentdojo.md +++ b/docs/recipes/agentdojo.md @@ -11,7 +11,7 @@ upstream logs, removes prior trajectories/scores, audits native initialization, and splits by user task. It has 57 TRAIN, 54 reserved validation and 406 TEST cases. Both Codex recipes select on TRAIN only and reserve TEST for frozen no-defense/initial/final evaluation. See the -[data protocol](https://github.com/Infini-AI-Lab/hermit/blob/main/examples/agentdojo/utils/DATA.md) +[data protocol](https://github.com/Infini-AI-Lab/hermit/blob/main/examples/agentdojo/utils/benchmark/DATA.md) for source revision, exclusions, ownership and hashes. Both methods use the same fixed Nemotron 3.5 Lightning victim, 25 tool iterations, diff --git a/docs/recipes/clbench.md b/docs/recipes/clbench.md index 1751d70e..7e53d9ec 100644 --- a/docs/recipes/clbench.md +++ b/docs/recipes/clbench.md @@ -25,7 +25,7 @@ export MW_RUNS="$PWD/.demo/clbench-runs" The CL-Bench host environment also uses the benchmark checkout's `.venv/bin/python`. For database, the launcher checks that the benchmark prompt contains the accuracy-scoring contract (`SCORING_UNLIMITED`). Inspect -`examples/clbench_database/utils/task-info/` and the task runners before starting; a vanilla checkout missing that +`examples/clbench_database/utils/benchmark/task-info/` and the task runners before starting; a vanilla checkout missing that contract is insufficient. ## Prepare and launch @@ -33,18 +33,20 @@ contract is insufficient. From the framework repository root: ```bash -python examples/clbench_database/utils/prepare.py --clbench "$MWM_CLBENCH" +python examples/clbench_database/utils/benchmark/prepare.py --clbench "$MWM_CLBENCH" MW_TAG=mwm-db WM_PORT=3401 MW_ORDERS="0" examples/clbench_database/mwm/launch.sh ``` For poker, use its own run tag and free port pair: ```bash -python examples/clbench_poker/utils/prepare.py --clbench "$MWM_CLBENCH" +python examples/clbench_poker/utils/benchmark/prepare.py --clbench "$MWM_CLBENCH" MW_TAG=mwm-poker WM_PORT=3411 MW_ORDERS="0" examples/clbench_poker/mwm/launch.sh ``` -These commands make real model calls. Begin with one order. `MW_ORDERS="0 1 2 3 4"` selects all five. +`prepare.py` writes each question (hand) as a task directory in Harbor's format under `utils/benchmark/tasks/`, with +the splits beside it; each config reads a task back as the row it always was (`data.view`), so the host and the +learning material are unchanged. These commands make real model calls. Begin with one order. `MW_ORDERS="0 1 2 3 4"` selects all five. `MW_TRAIN_SPLIT` can select a prepared prefix split for a smoke run; a prefix is not the full experiment. The launcher uses `WM_PORT` and `WM_PORT+1` for host and learner wire meters. diff --git a/docs/recipes/index.md b/docs/recipes/index.md index ced251b7..1e7022ac 100644 --- a/docs/recipes/index.md +++ b/docs/recipes/index.md @@ -21,7 +21,7 @@ Every recipe's code, configs and trained adapters are under `examples//`; to run one are in [`examples/README.md`](https://github.com/Infini-AI-Lab/hermit/blob/main/examples/README.md). Configuration paths and `eval --adapter` directories resolve against the method config's directory. Stream launchers can override output directories; use their recipe's explicit output settings. MWM venue text lives -under `utils/task-info/mwm/`, with a configurable deployment path. +under `utils/benchmark/task-info/mwm/`, with a configurable deployment path. ## What counts as a result diff --git a/docs/recipes/officeqa.md b/docs/recipes/officeqa.md index e2e4c21b..bba19136 100644 --- a/docs/recipes/officeqa.md +++ b/docs/recipes/officeqa.md @@ -46,14 +46,16 @@ pages in either: ## Prepare the data ```bash -python examples/officeqa/utils/prepare.py --csv /absolute/path/to/officeqa_full.csv \ +python examples/officeqa/utils/benchmark/prepare.py --csv /absolute/path/to/officeqa_full.csv \ --docs /absolute/path/to/treasury_bulletins_parsed ``` -It writes into `examples/officeqa/utils/`: +It writes into `examples/officeqa/utils/benchmark/`: ```text -tasks.jsonl the 246 questions, without answers +tasks// the 246 questions as task directories in Harbor's format: instruction.md (the question), + task.toml (its category and sources under [metadata]); no answers and no tests/ (the + judge runs on the host) splits/ train.txt, val.txt and test.txt (50 / 24 / 172 ids), and smoke_train.txt, smoke_val.txt and smoke_test.txt: the first four training, two validation and four test questions data.json absolute paths of the corpus, the answers and the CSV, and drifted_from_split @@ -62,6 +64,11 @@ seed_adapter/ SkillOpt's initial OfficeQA skill as a prompt-only@1.0 documents a link to the corpus: what the agent's sandbox shows (`environment.read`); `.private/` is not ``` +The configs read `data.tasks: ../utils/benchmark/tasks` with `data.view: ../utils/agent/officeqa_fields.py:officeqa_task`: +every task is the row SkillOpt's loader gives (the instruction as `question`, the metadata beside it), so the agents' +prompts, the run records and what a learning method reads are the same as when the task set was one `tasks.jsonl`. Prepared before this layout? Run `prepare.py` again: the files moved from +`utils/` to `utils/benchmark/`. + The split comes from SkillOpt's repository at a pinned commit unless `--split-dir` names a local copy of its `data/officeqa_id_split`. Questions, answers and source fields come from the CSV, so they follow the dataset revision you downloaded; `drifted_from_split` lists the items whose sources differ from the split's, which is the @@ -70,7 +77,7 @@ paper's revision. `prepare.py` prints the split sizes and those ids. `data.json` ## Check, train, evaluate -Inspect `examples/officeqa/gepa/smoke.yaml`. It selects `prompt-only@1.0`, starts from `../utils/seed_adapter`, +Inspect `examples/officeqa/gepa/smoke.yaml`. It selects `prompt-only@1.0`, starts from `../utils/benchmark/seed_adapter`, uses the `gepa` Evolver with GLM 5.3 as its model, batches of two tasks, two validation tasks, one epoch, and a training budget of USD 5 / three hours. Usage becomes visible at call boundaries, so the budget is not a provider-side spending cap. The separate `check` probe and `eval` calls are outside the training budget. @@ -79,7 +86,7 @@ provider-side spending cap. The separate `check` probe and `eval` calls are outs hermit check --config examples/officeqa/gepa/smoke.yaml hermit train --config examples/officeqa/gepa/smoke.yaml hermit eval --config examples/officeqa/gepa/smoke.yaml --adapter final -hermit eval --config examples/officeqa/gepa/smoke.yaml --adapter ../utils/seed_adapter +hermit eval --config examples/officeqa/gepa/smoke.yaml --adapter ../utils/benchmark/seed_adapter hermit eval --config examples/officeqa/gepa/smoke.yaml --adapter none ``` @@ -120,7 +127,7 @@ SkillOpt its best skill, handed back as is (`promote_final: false`). Each method folder has six configs, each with its own run name and adapter tree, so runs can train side by side; do not mix their run-name or tree settings. All use the `prompt-only@1.0` template and start from -`../utils/seed_adapter`; all but the Flash-row and GPT-5.6-Luna configs (`run_codex_flash.yaml`, +`../utils/benchmark/seed_adapter`; all but the Flash-row and GPT-5.6-Luna configs (`run_codex_flash.yaml`, `run_codex_luna.yaml`: one pass) evaluate three passes per test question. The full runs run eight rollouts at a time, the smoke runs four. @@ -163,7 +170,7 @@ hermit train --config examples/officeqa/skillopt/run.yaml hermit eval --config examples/officeqa/gepa/run.yaml --adapter final hermit eval --config examples/officeqa/skillopt/run.yaml --adapter final hermit eval --config examples/officeqa/skillopt/run.yaml --adapter none -hermit eval --config examples/officeqa/skillopt/run.yaml --adapter ../utils/seed_adapter +hermit eval --config examples/officeqa/skillopt/run.yaml --adapter ../utils/benchmark/seed_adapter ``` Each row of [Results](#results) is the `mean_score` of one evaluation, in the folder named after its selector: @@ -171,7 +178,7 @@ Each row of [Results](#results) is the `mean_score` of one evaluation, in the fo | Row | Config | `--adapter` | `summary.json` under `examples/officeqa/` | |---|---|---|---| | no skill | `skillopt/run.yaml` | `none` | `skillopt/runs/paper-skillopt/eval/none/` | -| initial skill | `skillopt/run.yaml` | `../utils/seed_adapter` | `skillopt/runs/paper-skillopt/eval/seed_adapter/` | +| initial skill | `skillopt/run.yaml` | `../utils/benchmark/seed_adapter` | `skillopt/runs/paper-skillopt/eval/seed_adapter/` | | GEPA | `gepa/run.yaml` | `final` | `gepa/runs/paper-gepa/eval/final/` | | SkillOpt | `skillopt/run.yaml` | `final` | `skillopt/runs/paper-skillopt/eval/final/` | | ReHarness, direct chat | `reharness/run.yaml`, through `reharness/launch.sh run eval` ([ReHarness](#reharness)) | `final` | `reharness/runs/paper-reharness/eval/final/` | @@ -201,7 +208,7 @@ trace summary for its analyst. Harness², Continual Harness, ReHarness's Codex s made with this agent's deployment and session (formerly `utils/agent/codex_agent.py`, now removed). Compare rows made with one harness; SkillOpt's Evolver now builds its Codex trace summary from the session's event stream. -Each Codex session runs in a Docker container of its own (`environment.image`, `hermit/agent/environment/docker.py`): +Each Codex session runs in a Docker container of its own (`environment.image`, `hermit/environment/docker.py`): - Files: only its workspace, a fresh Codex home and the corpus (read-only), with a private `/tmp`. - Network: the containers sit on an internal Docker network with no route out; their only way out is an HTTPS proxy @@ -254,7 +261,7 @@ hermit train --config examples/officeqa/skillopt/run_codex.yaml hermit eval --config examples/officeqa/gepa/run_codex.yaml --adapter final hermit eval --config examples/officeqa/skillopt/run_codex.yaml --adapter final hermit eval --config examples/officeqa/skillopt/run_codex.yaml --adapter none -hermit eval --config examples/officeqa/skillopt/run_codex.yaml --adapter ../utils/seed_adapter +hermit eval --config examples/officeqa/skillopt/run_codex.yaml --adapter ../utils/benchmark/seed_adapter ``` Codex's workspaces are under `~/.cache/hermit/officeqa-codex-actor/` (`agent.params.workspaces`), outside the @@ -264,13 +271,14 @@ checkout, so no repository instructions reach the session; each rollout's events ### Through Harbor The same Codex harness runs through [Harbor](../framework/task.md#benchmarks) instead of a container of the -acting agent's own (`environment.image`): each question is a Harbor task, and Harbor starts the container, installs Codex, deploys the -skill and runs the judge inside, after the session. Build the task set once (private: it holds the answers), then +acting agent's own (`environment.image`): each question is a Harbor task in its own Harbor environment, where Harbor +starts the container and installs Codex, Hermit's Codex agent deploys the skill and runs the session, and Harbor runs +the judge inside, after the session. Build the task set once (private: it holds the answers), then run a method's `run_codex_harbor.yaml` with the corpus mounted where the tasks expect it: ```bash python -m pip install --group skillopt # the task text is rendered by SkillOpt's own code -python examples/officeqa/utils/harbor/build_tasks.py # utils/.private/harbor_tasks//, tasks.jsonl, MOUNTS.json +python examples/officeqa/utils/harbor/build_tasks.py # utils/benchmark/.private/harbor_tasks//, tasks.jsonl, MOUNTS.json export OFFICEQA_DOCS_ROOT=... # the mount the build prints: the corpus at /app/docs/root_1 hermit check --config examples/officeqa/gepa/run_codex_harbor.yaml hermit train --config examples/officeqa/gepa/run_codex_harbor.yaml @@ -279,11 +287,11 @@ hermit eval --config examples/officeqa/gepa/run_codex_harbor.yaml --adapter fin What differs from the Codex route above: the container is Harbor's (`ubuntu:24.04` with python3 and ripgrep; `build_tasks.py --base-image` takes one with Codex installed, which saves Harbor's install per trial only when its -`codex --version` is exactly the config's `codex_version`, `0.134.0`: with another, Harbor still installs that one in -each trial), its network is the task's (no egress proxy: the model's endpoint and key reach Codex through Harbor), the +`codex --version` is exactly the config's `agent.codex.version`, `0.134.0`: with another, Harbor still installs that +one for each task), its network is the task's (no egress proxy: the model's endpoint and key reach Codex by name), the question and its parsed pages are the instruction rather than a `task.md`, and the judge (`utils/harbor/judge.py`, SkillOpt's exact match) reads the `` from the session's final message, in the -Codex stream the trial keeps (`/logs/agent/agent-codex.jsonl`; `codex.txt` for Harbor's own Codex agent), or +Codex stream Hermit hands the verifier (`/logs/agent/agent-codex.jsonl`; `codex.txt` for Harbor's own Codex agent), or `/app/answer.txt`, inside the container. The build copies the judge into each task's `tests/`: a task set built before a change to it needs building again. The scores of this route are not in the results table: a run through it is a different setting and is reported as one. @@ -520,7 +528,7 @@ Paired per question, 95% bootstrap intervals; bold where the interval lies above ## OPRO [OPRO](https://arxiv.org/abs/2309.03409) (Yang et al., ICLR 2024) searches one instruction document, `ADAPTER.md` on -the `prompt-only` template, from SkillOpt's initial skill (`utils/seed_adapter`). Its optimizer reads the best +the `prompt-only` template, from SkillOpt's initial skill (`utils/benchmark/seed_adapter`). Its optimizer reads the best instructions scored so far with their training scores and three training problems, and writes four new ones per step, each scored on the 50 training questions; `final` is the best validation mean of the top 3 on training ([OPRO](../methods/opro.md)). @@ -542,7 +550,7 @@ evaluation pass, and one evaluation of the initial skill in the same setting. | Adapter | accuracy | − the initial skill, points | |---|---|---| -| the initial skill (`utils/seed_adapter`), this setting | 0.355 [0.285, 0.424] | | +| the initial skill (`utils/benchmark/seed_adapter`), this setting | 0.355 [0.285, 0.424] | | | OPRO `final` (candidate 2 of 13, validation 0.625) | 0.436 [0.366, 0.512] | **+8.1** [+1.7, +14.5] | Paired per question, 95% bootstrap intervals; bold where the interval excludes zero. 24 questions better and 10 @@ -580,7 +588,7 @@ A run's records are under `examples/officeqa//runs//` (`` `` is `final`, `none`, `adapter`, or a directory's name such as `seed_adapter`; re-evaluating a selector replaces its records. -- Each answer is scored by SkillOpt's evaluator (`utils/judge.py`): exact match after its normalisation (lower case; +- Each answer is scored by SkillOpt's evaluator (`utils/benchmark/judge.py`): exact match after its normalisation (lower case; commas, punctuation and unit words removed), with no numeric tolerance. Token F1 is recorded as `f1` and is not the score. An attempt without an answer scores 0. - Accuracy is `mean_score` over 172 test questions × 3 passes = 516 attempts per adapter. The framework computes no @@ -604,7 +612,7 @@ agents are the acting model, GLM 5.3 Flash (medium) ([ReHarness](#reharness)). T | Skill | Direct chat | Codex harness | |---|---|---| | none: the agent without an adapter | 0.415 | 0.543 | -| SkillOpt's initial skill (`utils/seed_adapter/`) | 0.465 | 0.564 | +| SkillOpt's initial skill (`utils/benchmark/seed_adapter/`) | 0.465 | 0.564 | | GEPA's trained skill | 0.564 ([card](https://github.com/Infini-AI-Lab/hermit/blob/main/examples/officeqa/gepa/adapter/.hermit/CARD.md)) | 0.595 | | SkillOpt's trained skill | 0.576 ([card](https://github.com/Infini-AI-Lab/hermit/blob/main/examples/officeqa/skillopt/adapter/.hermit/CARD.md)) | 0.628 | | ReHarness's bank (`reharness/run.yaml`): the initial skill plus the five reflections retrieved per question, written from the 50 training questions | 0.579 | not provided | @@ -890,7 +898,7 @@ gains over the reference execution overlap GLM's intervals: they cannot be calle editor sessions. All runs together took the plan's weekly window from 29% to 38% (whole percents, account-wide: an upper bound on their share); no session was refused for the plan's limit and the credits balance did not change. - **Two defects of the ports**, found on this row and not fixed here, which touch the GLM row too: - `utils/task-info/README.md` tells the learner the judge removes "unit words", where `normalize_answer` removes + `utils/benchmark/task-info/README.md` tells the learner the judge removes "unit words", where `normalize_answer` removes million, billion, dollar and nominal and keeps `%`, thousand and trillion; only Continual Harness's Refiner reads it, and it wrote "required unit words" into its policy (steps 2–172; "any required unit words" from step 5). The Continual Harness Evolver counts Codex's own notices (the hook-trust warning `codex_agent.py` always causes, diff --git a/docs/recipes/servelearnbench.md b/docs/recipes/servelearnbench.md index 66d6a2be..df38fb6f 100644 --- a/docs/recipes/servelearnbench.md +++ b/docs/recipes/servelearnbench.md @@ -83,5 +83,6 @@ test episodes do not update the learning method. Since window serving counts do 11 cases of a window can remain unlearned at that window's test. Preserve this timing when comparing runs. `MW_TRAIN_SPLIT` can select a prepared prefix for smoke testing; prefix results do not reproduce the full stream. -The task contract is implemented under `examples/servelearnbench/utils/`. For the world-model requirements see +The task contract is implemented under `examples/servelearnbench/utils/` (the task set, as task directories in +Harbor's format, and its judge under `utils/benchmark/`). For the world-model requirements see [MWM](../methods/mwm.md), and for experiment reporting see [Evaluation](../guides/evaluation.md). diff --git a/docs/reference/compatibility.md b/docs/reference/compatibility.md index c07c5852..af84b722 100644 --- a/docs/reference/compatibility.md +++ b/docs/reference/compatibility.md @@ -34,33 +34,22 @@ framework's stdio server. Neither command guarantees that arbitrary tool depende - `agent.name` is the current acting-agent selector. Legacy `pipeline.run` settings remain supported when no agent is named; function-based agents receive `params` and `base` when their signatures declare those keywords. - Trainer acceptance checks structure; performance selection belongs to the Evolver or your experiment protocol. -- The Trainer's own process is not sandboxed; the acting agent, the judge (when sandboxed), the declared tools and the learner run in sandboxes of their own ([Sandbox design](../developer/sandbox-design.md)), except where the pipeline runs on the host by design (`sandbox.enabled: false`, Harbor, `environment.runner: host`). +- The Trainer's own process is not sandboxed; the acting agent, the judge (when sandboxed), the declared tools and the learner run in sandboxes of their own ([Sandbox design](../developer/sandbox-design.md)), except where the pipeline runs on the host by design (`sandbox.enabled: false`, `environment.runner: host`). A task in its own Harbor environment has no worker: every command and CLI of the agent runs in the task's container, the agent's loop and Hermit's judge in the Trainer's process (the adapter's own code, its context tool and its exports' render, in the strict adapter sandbox), Harbor's verifier in the task's environment (`[verifier] environment_mode` gives it a separate one). - An isolated network is a network namespace: the only ways out are the forwarded ports and the allowlist proxy, so a program that ignores proxy settings has no route out. - Budget observation can overshoot with in-flight work; missing usage remains unknown. - DeepSeek Harness reports no cost: the cost recorded follows the price (the agent's own `price`, else the backend's). Web searches and the requests that summarize a long session's history are not counted, though the session's cost still reads as known, so `budget.usd` can undercount it. - Hermes Agent's own cost figure is ignored: the cost recorded follows the price. Its auxiliary calls (a compression's summary, ...) are counted from its state database at the turn's end, a resumed session counting only this call's share. The summary Hermes asks for at `max_turns` is reported as a call without tokens, so that session's cost is unknown. - Crash recovery restores the last completed framework step, not arbitrary external service state. - The evaluator's `solved_once` threshold is fixed at score 1.0. -- Per-task adapter context comes only through `exports.yaml`'s `context_tool`, consumed by `Session`, the `react` agent, the `codex`, `claude-code`, `pi`, `mini-swe-agent`, `opencode`, `dsh` and `hermes` agents, and the `harbor` agent (computed on this machine before each trial, `hermit.harbor.context`); its template must declare `plugin-portable: false`, and `deploy` and `convert` refuse such adapters. +- Per-task adapter context comes only through `exports.yaml`'s `context_tool`, consumed by `Session`, the `react` agent, the `codex`, `claude-code`, `pi`, `mini-swe-agent`, `opencode`, `dsh` and `hermes` agents (in a task's Harbor environment too, computed before the session); its template must declare `plugin-portable: false`, and `deploy` and `convert` refuse such adapters. - The proposed manifest `serve` section, health checks, standalone startup, and relay lifecycle are not supported. - A model reached through a CLI (`llm: {cli: ...}`, `agent.model: {cli: ...}`) takes text and gives text: tool calls, sampling and structured-output requests are refused ([the model contract](../guides/claude-subscription.md#the-model-contract)). A subscription has no per-token cost, so `budget.usd` bounds nothing on it. - Harness²'s editors are told of upstream's Codex surface (the `harness2-codex` template); a Claude Code acting agent runs the same tree in its own places, and its execution's stream reaches them rendered in upstream's grammar by `harness2_upstream/render.py`. -- On a Harbor task set, a multi-step task's training steps run in the Trainer's own stream of that task, in order, - each once ([Benchmarks](../framework/task.md#benchmarks)), and - [`run_agent`](../framework/evolver.md#interaction-with-the-trainer) runs such a step in that same stream: a step - that already ran ends `stream_order` while the stream goes on, a judge failure scored 0, and so does a later one - while no rollout runs the step the stream waits for, which also stops the stream, so the Trainer's later steps of it - end the same way; the step the stream waits for runs there, on the candidate, in the Trainer's turn, which the - Trainer's own rollout of it then finds taken; and the stream's first step, run once the Trainer's pass of the stream - has finished, starts a new pass there on the candidate, in a new environment that stays up, which the Trainer's next - epoch inherits: its own first step ends `stream_order`, and its later steps run in the environment the candidate's - step left, scored as the agent's (the runner cannot tell `run_agent`'s rollout from the Trainer's: candidate 3 of - `hermit/harbor/FRAMEWORK_SUPPORT.md`). No check refuses a method that tests its candidates on training rows through - `run_agent` on such a set: GEPA's children, ACE's retry rounds, OPRO's new instructions, SkillOpt's slow update and - meta skill (two skills compared on training tasks), Harness²'s probes and compositions, and RRSI Adapter's smoke - when its domain names such a row get these failures in place of runs. Such a set trains with a method that learns - from the Trainer's own rollouts: a CLI Evolver whose `evolver.params.tools` leaves `run_agent` out (its default, - null, offers every Trainer tool, `run_agent` among them), or Meta-Harness Adapter. +- A task set in Harbor's environments (`environment.harbor`) runs single-step tasks only: a multi-step task + (`[[steps]]`) needs one environment across the Trainer's steps, which the run does not keep yet. In a task's Harbor + environment Codex and Claude Code run (the other CLIs are refused there), the adapter's tools are refused (the + framework's tool server runs on this machine), and so are an agent's `mounts`, `prepare` and `collect` + ([Benchmarks](../framework/task.md#benchmarks)). ## Earlier command interfaces diff --git a/docs/reference/config-fields.md b/docs/reference/config-fields.md index 3e2138ad..019cfba9 100644 --- a/docs/reference/config-fields.md +++ b/docs/reference/config-fields.md @@ -32,7 +32,7 @@ guarantee that their consumer supports arbitrary fields. | `data.shuffle` | boolean / true | Shuffle training order; set false for ordered streams | | `data.shuffle_seed` | integer / 0 | Training shuffle seed; does not seed every model or method | | `task_info` | directory / null | Learner-facing task description; provide `README.md` | -| `agent.name` | string / null; required | Built-in `react`, `codex`, `claude-code`, `pi`, `mini-swe-agent`, `opencode`, `dsh`, `hermes`, `harbor` (a benchmark's task set in Harbor's format, see [Benchmarks](../framework/task.md#benchmarks)), an Agent class, or a `file.py:function` / `module:function` | +| `agent.name` | string / null; required | Built-in `react`, `codex`, `claude-code`, `pi`, `mini-swe-agent`, `opencode`, `dsh`, `hermes`, `answer` (one model call, its answer written where the task's verifier reads it), an Agent class, or a `file.py:function` / `module:function`; a benchmark's task set in Harbor's environments is `environment.harbor` ([Benchmarks](../framework/task.md#benchmarks)), where `codex`, `claude-code`, `answer`, `react` and a custom agent that takes the `environment` argument work (the other CLIs refuse it) | | `agent.concurrency` | integer / 4 | Maximum concurrent rollout/evaluation task executions; use a positive value | | `agent.timeout_min` | number / null | A sandboxed task call (its worker) running longer is stopped and recorded as an error; null: no limit | | `agent.params` | mapping / {} | Passed to Agent constructors; plain functions receive it when they declare a `params` argument | @@ -145,6 +145,7 @@ Checkpoint frequency controls tags; completed steps still have records and accep | Field | Type / default | Meaning | |---|---|---| +| `environment.harbor` | mapping / path / null | A task set in Harbor's environments: each task's own, verified by Harbor's verifier; its settings, or the benchmark file that holds them (`benchmark:`); each setting overridable by its path ([Benchmarks](../framework/task.md#benchmarks)) | | `environment.runner` | string / `sandbox` | `sandbox`: the acting agent runs in the task sandbox; `host`: in the Trainer's process (a task whose code isolates what it runs) | | `environment.read` | list / [] | Task paths shown read-only to the acting agent; never the data, the judge, its tests or the runs | | `environment.write` | list / [] | Paths it may write besides its own work directory | diff --git a/examples/README.md b/examples/README.md index 9d87cf58..8021d561 100644 --- a/examples/README.md +++ b/examples/README.md @@ -9,13 +9,16 @@ examples// utils/ everything the task itself needs: the agent, the data preparation, the judge, the texts agent/ the acting agent: a prompt for a built-in agent, or agent.py with the task's own Agent class when a benchmark owns the loop; the config names it in agent.name - prepare.py writes tasks.jsonl and splits/ beside itself, from the task's data (which stays where it is) - benchmark.yaml a benchmark of Harbor's registry, in place of prepare.py: the params its configs share and a + benchmark/ the task set, in Harbor's task format, run on Hermit's own sandbox or on the host: + prepare.py writes tasks/ (one task directory a task: instruction.md, task.toml, tests/ when the task + has a verifier of its own) and splits/ beside itself, from the task's data (which stays where + it is); the configs' data.view reads each task as the row the agents and learners read + judge.py scores one output and writes the feedback (a task without tests/ is judged so) + task-info/ what the learner is told: README.md, and mwm/ with what MWM needs to know about the task + benchmark.yaml a benchmark of Harbor's registry, in place of benchmark/: the params its configs share and a dataset: block, from which python -m hermit.harbor.dataset prepare writes tasks.jsonl, splits/ and the rest beside it (Add an example, below) - judge.py scores one output and writes the feedback tools.py the task's tools for a built-in agent, where it has any - task-info/ what the learner is told: README.md, and mwm/ with what MWM needs to know about the task tests/ the task's own tests, where it has any / one folder per method run on the task: smoke.yaml, run.yaml, a launcher where a run needs one, a trained adapter/ where included @@ -42,10 +45,12 @@ a `.hermit/CARD.md`. ## Harbor benchmarks Five benchmarks of Harbor's registry, each with a baseline and the method whose paper's protocol fits it, packaged so -that any of their runs can be handed to someone with a Docker host. Each folder's README gives the protocol, the -commands and the results table, empty until the runs are made: the configs were checked only on each benchmark's two -smoke rows, with an earlier version of Hermit's Harbor runner (LAB-Bench FigQA's with the reworked one too, before -its last review fixes). Every config comes +that any of their runs can be handed to someone with a Docker host. Each task runs in its own Harbor environment +(`environment.harbor`): Harbor starts the task's container and verifies the agent's work with the task's tests; the +agent is Hermit's ([Benchmarks](../docs/framework/task.md#benchmarks)). Each folder's README gives the protocol, the +commands and the results table, empty until the runs are made: Terminal-Bench 2.0's two smoke tasks were run on this +path with Codex on GLM 5.3 Flash (both passed); the configs as written (their logins) were checked on the smoke rows +with Hermit's earlier Harbor runner only. Every config comes twice, Codex with GPT-5.6-Luna on a ChatGPT login (`*codex*.yaml`) and Claude Code with Claude Haiku 5.5 on a Claude subscription (`*claude*.yaml`), both at medium effort, acting and learning. @@ -59,13 +64,8 @@ Claude Haiku 5.5 on a Claude subscription (`*claude*.yaml`), both at medium effo **What a machine needs.** -- Linux with Docker, which runs every trial in the task's container, and bubblewrap, for the learners' sandbox and for - every trial that deploys an adapter (its exports are rendered in the strict adapter sandbox: each `--adapter final` - or `root`, a transfer, AHE's seed as a directory). `hermit eval` and `hermit train` refuse to start without it, or - with one that cannot start the strict sandbox (as on Ubuntu 24.04 when its AppArmor policy keeps unprivileged user - namespaces off), a baseline's `--adapter none` too, as the runner cannot tell a run that deploys none; on such a - machine a baseline runs with the override `agent.params.sandbox_enforce=best_effort`, which would render an adapter - unconfined (recorded) ([Installation](../docs/get-started/installation.md)). +- Linux with Docker, which runs every task's environment, and bubblewrap, for the acting agent's and the learners' + sandboxes ([Installation](../docs/get-started/installation.md)). - Harbor 0.25.0 in a venv of its own, where the configs look for it: ```bash @@ -76,23 +76,20 @@ Claude Haiku 5.5 on a Claude subscription (`*claude*.yaml`), both at medium effo - Hermit from this checkout, with SkillOpt's group for its configs: `python -m pip install -e . --group skillopt`. - The sign-in of the profile's CLI: `codex login` on a ChatGPT plan for the Codex configs; `claude setup-token`, its token exported as `CLAUDE_CODE_OAUTH_TOKEN`, for the Claude Code configs. Harbor installs the CLI in each task's - container; a method run also needs it on this machine, where its learners run, and so does `hermit check` of a - Claude Code config, a baseline's too: `main`'s check of a `login: claude` run wants `claude` on PATH, at Hermit's - minimum version, though `hermit eval` makes no such check (`hermit/harbor/FRAMEWORK_SUPPORT.md`, candidate 7). The - sign-in reaches the CLI in each task's container (Codex's sign-in file; Claude Code's token by name, in no process's - environment), where the session's commands run as the CLI's user and, in these task sets, reach the public network: - a command that read it could send it out. `hermit train` and `hermit check` warn of the Claude Code token, in - `main`'s words for a session on this machine (unconfined, the token in its CLI's environment), which a trial's is - not. Run them on a sign-in you accept that risk for. -- Room. No trial starts while Docker's disk has less than 25 GB free (the benchmark files' `disk_gate_gb`); a trial's - image is built or pulled for it: Terminal-Bench 2.0's prebuilt images from Docker Hub, 120-150 GB in all, each removed - after its trials and so pulled again at every AHE step (sign in with `docker login` for Docker Hub's pull limits, or + environment (or a config mounts one of this machine's and names it as `agent..bin`); a method run also needs it + on this machine, where its learners run. The sign-in reaches the CLI in each task's environment (Codex's sign-in + file, taken back and removed before the verifier; Claude Code's token by name, then on a descriptor), where the + session's commands run and, in these task sets, reach the public network: a command that read it could send it out. + Run them on a sign-in you accept that risk for. +- Room. No task's environment starts while Docker's disk has less than 25 GB free (the benchmark files' + `disk_gate_gb`); a task's image is built or pulled for it: Terminal-Bench 2.0's prebuilt images from Docker Hub, 120-150 GB in all, each removed + after its rollouts and so pulled again at every AHE step (sign in with `docker login` for Docker Hub's pull limits, or keep them where the disk holds them: [Terminal-Bench 2.0](terminal_bench_2_0/README.md#prepare)); SWE-bench's image of the issue, one for each of the 500, which stays in Docker's build cache (their size on disk is not measured here: this machine pulled three of them); AlgoTune's 8 GB, the same layers for every task. An AlgoTune container takes 8 CPUs and - up to 64 GB, six at a time. No trial starts either while the host's `/tmp` has less than 1 GB free - (`agent.params.tmp_gate_gb`). Each gate holds a trial up to six hours (`disk_gate_wait_s`), one after the other (so up - to twelve), then the trial ends infra (`disk`, which the runner does not run again). + up to 64 GB, six at a time. No environment starts either while the host's `/tmp` has less than 1 GB free + (`tmp_gate_gb`). Each gate holds a task up to six hours (`disk_gate_wait_s`), one after the other (so up to twelve), + then the task's run fails as not the agent's result. **Run a benchmark** from the repository root: @@ -112,28 +109,15 @@ and that no hard budget is set, as it does for every config here (their budgets also reports that its splits share their two rows, which is what a smoke runs on; of a SkillOpt config it warns that the validation data is given to the Evolver, whose gate runs it (a row holds no answer: its task's tests keep that). -A trial that is not the agent's result (infra: the host, the model provider, a session that never started) runs again in -the runner, twice at most and five minutes apart (`agent.params.infra_retries`, `infra_wait_s`), unless it is one that -does not run again (a host failure but its staging's, a disk gate that waited its time out, a provider's refusal of -the run's configuration: [Benchmarks](../docs/framework/task.md#benchmarks) lists them) or a session the provider -could not serve, which on these configs' logins runs again under `busy_retries` instead (below); one still infra is a -failed run, counted among `hermit eval`'s `errors` and as 0 in its `mean_score`, and -`python -m hermit.harbor.report ` gives the eval's pass@1 over the trials that are the agent's result, with -the infra ones counted apart. Run again, `hermit train` resumes after its last finished step (the step it stopped in -runs again, all of it; a run stopped in its first step starts over in its folder, which `main`'s Trainer leaves as it -was, so AHE's history can then hold its first iteration twice, when the run stopped after that step's Evolver call -began) and `hermit eval` runs all of its trials again: no trial is taken from an earlier run. A lost sign-in ends the -run at the trial that meets it (`LoginFailed`), or at a Claude Code learner's session that meets it first, which -`main` reads; a Codex learner that meets it first fails its step if it is AHE's Evolve session, as on `main`, while -SkillOpt's optimizer skips a call that failed, as it skips any, and its step goes on without that call's edit (after -the run's last trials, its last slow update and meta skill then come out empty, with no error); the next trial ends -the run (`hermit/harbor/FRAMEWORK_SUPPORT.md`, candidate 10): sign in again and run the command again. An eval it -stopped writes no row of the repeat it stopped in (these configs run one: its -`results.jsonl` is left empty), beside the `summary.json` of the eval before, if any, which the report and the two -scripts below say (`unfinished`); the trials it finished are only in their job folders under the profile's `jobs_dir`, -which no run reads back. A plan's usage limit ends nothing on these configs' logins: the trials wait for the plan's -window, then run. A provider that does not bill the account (a 402, its credit spent) would end the run, as an API -account's usage limit does; a subscription should not meet one. +A run that is not the agent's result (infra: its environment did not start or failed under the verifier, its verifier +ran no test, its session never started) runs again in a fresh environment, twice at most and five minutes apart +(`environment.harbor.infra_retries`, `infra_wait_s`), unless it is one that does not run again (a disk gate that waited +its time out, a provider's refusal of the run's configuration); one still infra is a failed run, counted among `hermit +eval`'s `errors` and as 0 in its `mean_score`, and `python -m hermit.harbor.report ` gives the eval's pass@1 over the runs that are the agent's result, with the infra ones counted apart. Run +again, `hermit train` resumes after its last finished step and `hermit eval` runs all of its tasks again. A lost +sign-in ends the run (`LoginFailed`): sign in again and run the command again. A plan's usage limit ends nothing on +these configs' logins: the sessions wait for the plan's window, then run. **Hand a run to someone.** Each run below is one machine's and shares nothing with the others, so each can go to a different person (a method's seed or empty skill, `--adapter root`, goes with the method's training: like `final`, it is @@ -165,17 +149,17 @@ table. An AHE run also sends back its best iteration's in-sample pass@1, numbers | | the empty skill | `hermit eval --config skillopt/run_

.yaml --adapter root`, after SkillOpt's training | 127 | | | | SkillOpt | `hermit train --config skillopt/run_

.yaml`, then `hermit eval ... --adapter final` | at most 408 (more if resumed mid-step), then 127 | | -`

` is `codex` or `claude`, and a config is `examples//...`. The trials are counted once each, though one -can run more than once: the runner runs an infra trial up to twice more (above), and a trial on a login again when -the plan refused its session (once the plan's window allows, as often as it refuses) or the provider could not serve -it (`busy_retries`, three times by default over all of its runs); a row's tokens are all its runs'. Before a long +`

` is `codex` or `claude`, and a config is `examples//...`. The trials are counted once each; a task runs +again in a fresh environment when the provider could not serve its session (`environment.harbor.busy_retries`, 3, the +n-th after n x `busy_wait_s`, 120 s, not counted in `infra_retries`) or its login's plan refused it (once the plan +allows, not counted), and a row's tokens are all its runs'. Before a long training, its method's `smoke_

.yaml` checks the task set, the CLI in the containers and the method's learners together. SWE-bench Verified trains nothing and has no smoke config: its baseline config runs its 2 smoke tasks apart from the baseline's run and trials, as the methods' smokes run apart from theirs, with ```bash hermit eval --config examples/swebench_verified/baseline/

.yaml --adapter none data.test=../utils/splits/smoke.txt \ - trainer.run_name=

-smoke agent.params.jobs_dir=~/.cache/hermit/harbor/jobs/swebv-

-smoke + trainer.run_name=

-smoke environment.harbor.jobs_dir=~/.cache/hermit/harbor/jobs/swebv-

-smoke ``` The transfer runs take Terminal-Bench 2.0's final adapter of the same profile as a directory, ``, which whoever @@ -183,12 +167,12 @@ ran that training writes out (`examples/terminal_bench_2_0/ahe_adapter/runs/ahe- [Terminal-Bench 2.0](terminal_bench_2_0/README.md#ahe-adapter) says how) and hands on. LAB-Bench's runs are also reported on its 102 questions with a new figure, from the same trials ([LAB-Bench FigQA](labbench/README.md#prepare)). -**Another model or CLI.** A config's models are its `agent.params.model` (the acting agent), its learners' -(`evolver.model` for AHE, `llm.model` for SkillOpt) and their `login`s: copy the config, change them, and give the -copy its own `trainer.run_name` and `tree_dir`. `acting_agent: {cli: ...}` takes another CLI (`codex`, `claude`, -`pi`, `opencode`, `dsh`, `mini-swe-agent`: [Benchmarks](../docs/framework/task.md#benchmarks)), and a +**Another model or CLI.** A config's models are its `agent.model` (the acting agent), its learners' (`evolver.model` +for AHE, `llm.model` for SkillOpt) and their `login`s: copy the config, change them, and give the copy its own +`trainer.run_name` and `tree_dir`. `agent.base` takes `codex` or `claude` in a task's Harbor environment (or +`agent.name: answer`, the answer-only agent: [Benchmarks](../docs/framework/task.md#benchmarks)), and a `/` reference with no `login` runs on that backend's key -([Model backends](../docs/guides/model-backends.md)); these configs were checked on the two logins only. +([Model backends](../docs/guides/model-backends.md)). ## Demos @@ -210,7 +194,7 @@ evaluate with a method's config: ```bash python -m pip install -e . --group skillopt --group gepa -python examples/officeqa/utils/prepare.py --csv /path/to/officeqa_full.csv --docs /path/to/treasury_bulletins_parsed +python examples/officeqa/utils/benchmark/prepare.py --csv /path/to/officeqa_full.csv --docs /path/to/treasury_bulletins_parsed export FIREWORKS_API_KEY=... # the configs name GLM models on Fireworks hermit check --config examples/officeqa/gepa/smoke.yaml hermit train --config examples/officeqa/gepa/smoke.yaml @@ -227,23 +211,26 @@ SGLang server with one override, `model=vllm/Qwen/Qwen3-30B-A3B-Instruct-2507-FP ## Add an example -- A task: `utils/prepare.py`, the task's tools or an `utils/agent/agent.py` with an `Agent` subclass, `utils/judge.py`, - `utils/task-info/README.md` (see [Task integration](../docs/framework/task.md) and the +- A task: `utils/benchmark/prepare.py` writing task directories (`hermit.task.directory.write_task`), the task's + tools or an `utils/agent/agent.py` with an `Agent` subclass, `utils/benchmark/judge.py`, + `utils/benchmark/task-info/README.md` (see [Task integration](../docs/framework/task.md) and the [built-in agents](../docs/framework/task.md#built-in-agents)), and a row in the table above. - A benchmark comes as a task set in Harbor's format, the one protocol every task enters by, not as Docker code or a - loop of its own. A benchmark of Harbor's registry is packaged as `utils/benchmark.yaml`: the params every method's - config shares (`agent.params.benchmark` names the file) and a `dataset:` block naming the set + loop of its own. A benchmark of Harbor's registry is packaged as `utils/benchmark.yaml`: the Harbor settings every + method's config shares (`environment.harbor.benchmark` names the file) and a `dataset:` block naming the set (`ref: @`), from which `python -m hermit.harbor.dataset prepare utils/benchmark.yaml` downloads the tasks outside the checkout (by default under `~/.cache/hermit/harbor-tasks`, or `$HERMIT_HARBOR_TASKS`, a relative one the working directory's) and writes beside the file the `tasks` link to them (git-ignored: it names a path of this machine), `tasks.jsonl`, `dataset.lock` (the tasks' contents, which every later prepare checks), `splits/` and, with `image_sizes: true`, `image_sizes.json`. A benchmark of your own has a `utils/prepare.py` that builds the task directories (`task.toml`, `environment/`, `tests/test.sh` writing the reward) and indexes them into `tasks.jsonl`. The - configs name `agent.name: harbor` and `judge.path: hermit.harbor.judge`, and a benchmark whose tasks carry state from - one to the next (CL-Bench, ServeLearnBench) is one multi-step task per stream. See + configs name `environment.harbor` and `judge: {path: hermit.task.verify, name: environment}`. See [Benchmarks](../docs/framework/task.md#benchmarks). - The examples that still own their loop (CL-Bench, ServeLearnBench, OfficeQA's direct harness) predate this rule - and are to be moved onto it; no new example drives Docker or a model loop itself. + The examples whose agent owns its loop (CL-Bench, ServeLearnBench, AgentDyn, AgentDojo, OfficeQA's direct + harness) predate this rule: their task sets are task directories in Harbor's format (`utils/benchmark/`), each + row of a stream one task, its state carried by the agent's own loop, and they run on Hermit's own sandbox or on + the host; their loops are still to be moved onto Hermit's agents. No new example drives Docker or a model loop + itself. - A method on an existing task: `/smoke.yaml` and `run.yaml` (a stream's launcher may take a prefix split in place of a smoke config, `MW_TRAIN_SPLIT=...`, as the CL-Bench folders do), and a row in `docs/methods/.md`. - A method: `docs/methods/.md` in the shape of the others: what it does, one step, knobs, configs, commands. diff --git a/examples/agentdojo/README.md b/examples/agentdojo/README.md index 0fc85903..a9ff0c4e 100644 --- a/examples/agentdojo/README.md +++ b/examples/agentdojo/README.md @@ -50,18 +50,20 @@ scores and other fields are discarded. For example, the row shape is: Replace the placeholder with an existing recorded payload. Prepare your collection: ```bash -python examples/agentdojo/utils/prepare.py /path/to/frozen-cases.jsonl examples/agentdojo/utils/data +python examples/agentdojo/utils/benchmark/prepare.py /path/to/frozen-cases.jsonl examples/agentdojo/utils/benchmark/data # Or import existing native benchmark logs for one suite: -python examples/agentdojo/utils/prepare.py /path/to/native-logs examples/agentdojo/utils/data --from-logs --suite workspace +python examples/agentdojo/utils/benchmark/prepare.py /path/to/native-logs examples/agentdojo/utils/benchmark/data --from-logs --suite workspace ``` Preparation deduplicates rows, adds one clean case per user task, and writes -`tasks.jsonl`, `train.txt`, `val.txt`, `test.txt` and `manifest.json`. All variants +the cases as task directories in Harbor's format (`tasks/`, each read back as its row through AgentDyn's +`utils/benchmark/cases.py`, the configs' `data.view`), `cases.txt`, `train.txt`, `val.txt`, `test.txt` and +`manifest.json`. All variants of `(suite, user_task_id)` stay together. With seed 0 by default, the shuffled groups assign `max(1, groups // 5)` groups each to validation and test, and the rest to training. At least five groups and injected coverage in every split are required. The manifest records group ownership, counts, reference revision and -task-file SHA-256. This is a local split protocol, not an official AgentDojo split. +the SHA-256 of the rows as JSON lines. This is a local split protocol, not an official AgentDojo split. Native task and injection-vector IDs are checked by the suite at execution time. Prepared cases, logs, training artifacts and model credentials stay local. Do not @@ -74,7 +76,7 @@ not case labels, judge predicates or hidden environment state. proposer in `llm` for training; the supplied config leaves `llm: null` deliberately. It uses the existing `evosafe@1.0` template and EvoSafe Evolver without algorithm changes. See [the example](evosafe/README.md) and -[the domain specification](utils/task-info/README.md). +[the domain specification](utils/benchmark/task-info/README.md). ```bash hermit check --config examples/agentdojo/evosafe/run.yaml @@ -118,15 +120,15 @@ from these measured configurations. ## Shared SHE and EvoSafe comparison -The [frozen comparison protocol](utils/DATA.md) matches AgentDyn's 57 TRAIN / 54 +The [frozen comparison protocol](utils/benchmark/DATA.md) matches AgentDyn's 57 TRAIN / 54 reserved-validation budgets, uses all four official suites, and evaluates 406 TEST cases (55 clean, 351 injected). Install the SHE dependency alongside the official benchmark. Use the existing upstream logs, without generating attacks: ```bash uv pip install --python .venv-agentdojo/bin/python -e '.[test]' --group she -e ../agentdojo -python examples/agentdojo/utils/prepare_benchmark.py \ - ../agentdojo/runs/gpt-4o-2024-05-13 examples/agentdojo/utils/data/comparison-v1 +python examples/agentdojo/utils/benchmark/prepare_benchmark.py \ + ../agentdojo/runs/gpt-4o-2024-05-13 examples/agentdojo/utils/benchmark/data/comparison-v1 ``` Both `evosafe/run_nemotron_codex.yaml` and `she/run_nemotron_codex.yaml` freeze @@ -190,6 +192,6 @@ unconditional or statistical-significance claim. SHE rounds 6 and 9 measured the same artifact, so their best-record promotions include repeat variability. Training used only the 57 TRAIN cases; TEST never feeds the search. -The public tool schemas/descriptions in `utils/task-info/tools.json` are generated +The public tool schemas/descriptions in `utils/benchmark/task-info/tools.json` are generated from the pinned official AgentDojo interfaces and retain the upstream [MIT notice](AGENTDOJO_LICENSE.txt). diff --git a/examples/agentdojo/evosafe/README.md b/examples/agentdojo/evosafe/README.md index bc60a4bf..2277bbfe 100644 --- a/examples/agentdojo/evosafe/README.md +++ b/examples/agentdojo/evosafe/README.md @@ -3,7 +3,7 @@ [`run.yaml`](run.yaml) connects the existing EvoSafeHarness method and `evosafe@1.0` adapter to official AgentDojo's frozen replay task. Follow [installation and data preparation](../README.md) first. The config's -paths are relative to this folder; preparation writes `../utils/data/`. +paths are relative to this folder; preparation writes `../utils/benchmark/data/`. The victim defaults to `openai/gpt-4o-mini-2024-07-18`. Configure its backend and credentials, and replace `llm: null` with your proposer before training. The @@ -31,7 +31,7 @@ are errors, not successful defenses. The Lightning/Codex comparison below record ## Matching Lightning/Codex experiment [`run_nemotron_codex.yaml`](run_nemotron_codex.yaml) uses the shared -[57/54/406 protocol](../utils/DATA.md) and the same fixed Lightning actor settings +[57/54/406 protocol](../utils/benchmark/DATA.md) and the same fixed Lightning actor settings as SHE. TRAIN has 57 cases; reserved validation is unused. The Designer, Criticizer, Cascade and Analyzer use the existing `evosafe` implementation, with three proposals, cascade sizes 12/57, source gates, 0.07 improvement/tolerance diff --git a/examples/agentdojo/evosafe/run.yaml b/examples/agentdojo/evosafe/run.yaml index 037057a0..93e4e363 100644 --- a/examples/agentdojo/evosafe/run.yaml +++ b/examples/agentdojo/evosafe/run.yaml @@ -1,11 +1,12 @@ data: - tasks: ../utils/data/tasks.jsonl - train: ../utils/data/train.txt - val: ../utils/data/val.txt - test: ../utils/data/test.txt + tasks: ../utils/benchmark/data/tasks + view: ../../agentdyn/utils/benchmark/cases.py:case_row # each case as the row it always was + train: ../utils/benchmark/data/train.txt + val: ../utils/benchmark/data/val.txt + test: ../utils/benchmark/data/test.txt step_size: 4 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info template: evosafe@1.0 agent: name: ../utils/task.py:run @@ -20,7 +21,7 @@ agent: logs: runs/episodes benchmark: agentdojo # the experiment supervisor's provenance reads it judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py evolver: name: evosafe params: diff --git a/examples/agentdojo/evosafe/run_nemotron_codex.yaml b/examples/agentdojo/evosafe/run_nemotron_codex.yaml index df4b0b3c..1292eb04 100644 --- a/examples/agentdojo/evosafe/run_nemotron_codex.yaml +++ b/examples/agentdojo/evosafe/run_nemotron_codex.yaml @@ -1,11 +1,12 @@ # Shared Lightning victim and the AgentDyn Codex search budget; frozen AgentDojo comparison-v1. data: - tasks: ../utils/data/comparison-v1/tasks.jsonl - train: ../utils/data/comparison-v1/train.txt - test: ../utils/data/comparison-v1/test.txt + tasks: ../utils/benchmark/data/comparison-v1/tasks + view: ../../agentdyn/utils/benchmark/cases.py:case_row # each case as the row it always was + train: ../utils/benchmark/data/comparison-v1/train.txt + test: ../utils/benchmark/data/comparison-v1/test.txt step_size: 57 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info template: evosafe@1.0 agent: name: ../utils/task.py:run @@ -32,7 +33,7 @@ agent: timeout_s: 45 benchmark: agentdojo judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py evolver: name: evosafe base: codex diff --git a/examples/agentdojo/she/README.md b/examples/agentdojo/she/README.md index 8f5ee4bb..4b54cafa 100644 --- a/examples/agentdojo/she/README.md +++ b/examples/agentdojo/she/README.md @@ -10,7 +10,7 @@ No new method or replacement selector is introduced for AgentDojo. ## Setup and protocol Follow the [AgentDojo comparison guide](../README.md#shared-she-and-evosafe-comparison) -and [data protocol](../utils/DATA.md). Install the additional dependency in the +and [data protocol](../utils/benchmark/DATA.md). Install the additional dependency in the same resolver invocation as the benchmark and Hermit: ```bash diff --git a/examples/agentdojo/she/run.yaml b/examples/agentdojo/she/run.yaml index 4f358f97..1e782bed 100644 --- a/examples/agentdojo/she/run.yaml +++ b/examples/agentdojo/she/run.yaml @@ -1,10 +1,11 @@ data: - tasks: ../utils/data/comparison-v1/tasks.jsonl - train: ../utils/data/comparison-v1/train.txt - test: ../utils/data/comparison-v1/test.txt + tasks: ../utils/benchmark/data/comparison-v1/tasks + view: ../../agentdyn/utils/benchmark/cases.py:case_row # each case as the row it always was + train: ../utils/benchmark/data/comparison-v1/train.txt + test: ../utils/benchmark/data/comparison-v1/test.txt step_size: 57 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info template: she@1.0 agent: name: ../utils/task.py:run @@ -26,7 +27,7 @@ agent: max_tokens: 1024 timeout_s: 45 judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py evolver: name: she params: diff --git a/examples/agentdojo/she/run_nemotron_codex.yaml b/examples/agentdojo/she/run_nemotron_codex.yaml index 61f76e6e..c4b47653 100644 --- a/examples/agentdojo/she/run_nemotron_codex.yaml +++ b/examples/agentdojo/she/run_nemotron_codex.yaml @@ -1,11 +1,12 @@ # Shared Lightning victim and the AgentDyn Codex search budget; frozen AgentDojo comparison-v1. data: - tasks: ../utils/data/comparison-v1/tasks.jsonl - train: ../utils/data/comparison-v1/train.txt - test: ../utils/data/comparison-v1/test.txt + tasks: ../utils/benchmark/data/comparison-v1/tasks + view: ../../agentdyn/utils/benchmark/cases.py:case_row # each case as the row it always was + train: ../utils/benchmark/data/comparison-v1/train.txt + test: ../utils/benchmark/data/comparison-v1/test.txt step_size: 57 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info template: she@1.0 agent: name: ../utils/task.py:run @@ -32,7 +33,7 @@ agent: base: direct model: vllm/nvidia/nvidia/nemotron-3.5-lightning judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py evolver: name: she params: diff --git a/examples/agentdojo/utils/DATA.md b/examples/agentdojo/utils/benchmark/DATA.md similarity index 91% rename from examples/agentdojo/utils/DATA.md rename to examples/agentdojo/utils/benchmark/DATA.md index 4b661e79..61c356f3 100644 --- a/examples/agentdojo/utils/DATA.md +++ b/examples/agentdojo/utils/benchmark/DATA.md @@ -28,11 +28,12 @@ unrecorded goals are not synthesized. “All goals” below means all eligible From the repository root: ```bash -python examples/agentdojo/utils/prepare_benchmark.py \ - ../agentdojo/runs/gpt-4o-2024-05-13 examples/agentdojo/utils/data/comparison-v1 +python examples/agentdojo/utils/benchmark/prepare_benchmark.py \ + ../agentdojo/runs/gpt-4o-2024-05-13 examples/agentdojo/utils/benchmark/data/comparison-v1 ``` -The command refuses to overwrite an output directory. It writes task rows and +The command refuses to overwrite an output directory. It writes the cases as task +directories (`tasks/`, the row as `task.toml`'s `[metadata]`; `cases.txt` keeps their order) and split ID files, a source audit, and a manifest with hashes, native inventory, source file paths/hashes and group ownership. Generated cases and audit logs stay ignored; public tool schemas in `task-info/tools.json` are reusable interfaces. @@ -61,7 +62,7 @@ variant independently of outcomes. | Workspace | 8 / 16 | 8 / 16 | 24 / 144 | | Total | **19 / 38** | **18 / 36** | **55 / 351** | -The task JSONL SHA-256 is +The SHA-256 of the rows as JSON lines (the manifest's `tasks_sha256`; what `tasks.jsonl` held) is `d50945a77e81afb21a7556a10b29d1989a99c4822eabd45f81c4d67013f0d65e`. Both Codex recipes select on TRAIN only and leave validation unused, following the corresponding AgentDyn Codex protocols. TEST is evaluated only with frozen diff --git a/examples/agentdojo/utils/judge.py b/examples/agentdojo/utils/benchmark/judge.py similarity index 73% rename from examples/agentdojo/utils/judge.py rename to examples/agentdojo/utils/benchmark/judge.py index 3f1d5408..16cde2c5 100644 --- a/examples/agentdojo/utils/judge.py +++ b/examples/agentdojo/utils/benchmark/judge.py @@ -2,7 +2,7 @@ from pathlib import Path import sys -sys.path.insert(0, str(Path(__file__).resolve().parents[2] / "agentdyn/utils")) +sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "agentdyn/utils")) from bridge import score __all__ = ["score"] diff --git a/examples/agentdojo/utils/prepare.py b/examples/agentdojo/utils/benchmark/prepare.py similarity index 93% rename from examples/agentdojo/utils/prepare.py rename to examples/agentdojo/utils/benchmark/prepare.py index 79f26fb3..cb7463d2 100644 --- a/examples/agentdojo/utils/prepare.py +++ b/examples/agentdojo/utils/benchmark/prepare.py @@ -5,12 +5,12 @@ from pathlib import Path import sys -REPLAY = Path(__file__).resolve().parents[2] / "agentdyn/utils" +REPLAY = Path(__file__).resolve().parents[3] / "agentdyn/utils" sys.path[:0] = [str(REPLAY), str(REPLAY.parents[2])] # AgentDyn's runtime; hermit from this checkout from bridge import BENCHMARKS # AgentDyn's prepare.py, by its path: this file has the same name. -_spec = importlib.util.spec_from_file_location("agentdyn_prepare", REPLAY / "prepare.py") +_spec = importlib.util.spec_from_file_location("agentdyn_prepare", REPLAY / "benchmark" / "prepare.py") _replay = importlib.util.module_from_spec(_spec) _spec.loader.exec_module(_replay) prepare_replay, read_logs = _replay.prepare, _replay.read_logs diff --git a/examples/agentdojo/utils/prepare_benchmark.py b/examples/agentdojo/utils/benchmark/prepare_benchmark.py similarity index 93% rename from examples/agentdojo/utils/prepare_benchmark.py rename to examples/agentdojo/utils/benchmark/prepare_benchmark.py index acedf5d7..30af6e96 100644 --- a/examples/agentdojo/utils/prepare_benchmark.py +++ b/examples/agentdojo/utils/benchmark/prepare_benchmark.py @@ -12,10 +12,12 @@ import random import sys -sys.path[:0] = [str(Path(__file__).resolve().parents[2] / 'agentdyn/utils'), # AgentDyn's runtime - str(Path(__file__).resolve().parents[3])] # hermit from this checkout -from bridge import AGENTDOJO_REVISION, case_fields, validate_case -from prepare_expanded import audit_native +sys.path[:0] = [str(Path(__file__).resolve().parents[3] / 'agentdyn/utils'), # AgentDyn's runtime + str(Path(__file__).resolve().parents[3] / 'agentdyn/utils/benchmark'), # its prepare scripts + str(Path(__file__).resolve().parents[4])] # hermit from this checkout +from bridge import AGENTDOJO_REVISION, case_fields, validate_case # noqa: E402 +from cases import write_cases # noqa: E402 +from prepare_expanded import audit_native # noqa: E402 SUITES = ('banking', 'slack', 'travel', 'workspace') VERSION = 'v1.2.2' @@ -110,8 +112,7 @@ def prepare(source, output, *, attack='tool_knowledge', seed=20261117, train_gro inventory, audit = audit_native(cases, suite_names=SUITES, benchmark_version=VERSION) rows, splits, assignments, selected = select(cases, audit, seed, train_groups, val_groups) output.mkdir(parents=True) - tasks = output / 'tasks.jsonl' - tasks.write_text(''.join(json.dumps(row, sort_keys=True) + '\n' for row in rows)) + tasks_sha256 = write_cases(output, rows, benchmark='agentdojo', sort_keys=True) for split, ids in splits.items(): (output / f'{split}.txt').write_text('\n'.join(ids) + '\n') manifest = {'protocol': 'AgentDojo frozen replay comparison v1; local split, not official benchmark score', @@ -123,7 +124,7 @@ def prepare(source, output, *, attack='tool_knowledge', seed=20261117, train_gro 'source_cases': len(cases), 'eligible_cases': sum(row['eligible'] for row in audit.values()), 'selected_sources': {key: sources[key] for key in selected}, 'audit_scope': 'native IDs, vectors, environment initialization and false no-op verdicts; no model outcomes', - 'tasks_sha256': hashlib.sha256(tasks.read_bytes()).hexdigest(), + 'tasks_sha256': tasks_sha256, 'split_sha256': {s: hashlib.sha256((output / f'{s}.txt').read_bytes()).hexdigest() for s in splits}} (output / 'manifest.json').write_text(json.dumps(manifest, indent=2)) (output / 'source-audit.json').write_text(json.dumps(audit, indent=2)) diff --git a/examples/agentdojo/utils/task-info/README.md b/examples/agentdojo/utils/benchmark/task-info/README.md similarity index 100% rename from examples/agentdojo/utils/task-info/README.md rename to examples/agentdojo/utils/benchmark/task-info/README.md diff --git a/examples/agentdojo/utils/task-info/tools.json b/examples/agentdojo/utils/benchmark/task-info/tools.json similarity index 100% rename from examples/agentdojo/utils/task-info/tools.json rename to examples/agentdojo/utils/benchmark/task-info/tools.json diff --git a/examples/agentdojo/utils/tests/test_protocol.py b/examples/agentdojo/utils/tests/test_protocol.py index d4d51b01..a7f76f30 100644 --- a/examples/agentdojo/utils/tests/test_protocol.py +++ b/examples/agentdojo/utils/tests/test_protocol.py @@ -15,7 +15,7 @@ from hermit.config import Config ROOT = Path(__file__).resolve().parents[4] -sys.path.insert(0, str(ROOT / 'examples/agentdojo/utils')) +sys.path.insert(0, str(ROOT / 'examples/agentdojo/utils/benchmark')) from prepare_benchmark import SUITES, digest, read_collection, select @@ -107,7 +107,8 @@ def test_comparison_recipes_share_victim_and_test_data_while_retaining_method_se assert config['agent']['params']['benchmark'] == 'agentdojo' assert config['agent']['params']['max_iters'] == 25 assert config['data']['step_size'] == 57 - assert config.path(config['data']['test']) == ROOT / 'examples/agentdojo/utils/data/comparison-v1/test.txt' + assert config.path(config['data']['test']) == \ + ROOT / 'examples/agentdojo/utils/benchmark/data/comparison-v1/test.txt' assert not config['data'].get('val') assert config['evolver']['params']['backend'] == 'codex' if method == 'she': @@ -143,8 +144,9 @@ def test_the_entry_points_load_agentdyns_runtime_beside_an_installed_examples_pa (tmp_path / 'examples').mkdir() (tmp_path / 'examples/__init__.py').write_text('') code = ("import runpy\n" - "for name in ('agentdojo/utils/task.py', 'agentdojo/utils/judge.py', 'agentdojo/utils/prepare.py',\n" - " 'agentdojo/utils/prepare_benchmark.py', 'agentdyn/utils/prepare.py'):\n" + "for name in ('agentdojo/utils/task.py', 'agentdojo/utils/benchmark/judge.py',\n" + " 'agentdojo/utils/benchmark/prepare.py', 'agentdojo/utils/benchmark/prepare_benchmark.py',\n" + " 'agentdyn/utils/benchmark/prepare.py'):\n" " runpy.run_path('examples/' + name)\n") env = {**os.environ, 'PYTHONPATH': os.pathsep.join([str(tmp_path), str(ROOT)])} result = subprocess.run([sys.executable, '-c', code], cwd=ROOT, env=env, capture_output=True, text=True) diff --git a/examples/agentdojo/utils/tests/test_replay.py b/examples/agentdojo/utils/tests/test_replay.py index ceba01fa..c349b0e8 100644 --- a/examples/agentdojo/utils/tests/test_replay.py +++ b/examples/agentdojo/utils/tests/test_replay.py @@ -23,9 +23,11 @@ UTILS = ROOT / 'examples/agentdojo/utils' sys.path.insert(0, str(ROOT / 'examples/agentdyn/utils')) from bridge import AGENTDOJO_REVISION, validate_case +sys.path.insert(0, str(ROOT / 'examples/agentdyn/utils/benchmark')) +from cases import read_cases # AgentDojo's prepare.py by its path: AgentDyn's tests import AgentDyn's own as `prepare`. -_spec = importlib.util.spec_from_file_location('agentdojo_prepare', UTILS / 'prepare.py') +_spec = importlib.util.spec_from_file_location('agentdojo_prepare', UTILS / 'benchmark' / 'prepare.py') _prepare = importlib.util.module_from_spec(_spec) _spec.loader.exec_module(_prepare) prepare = _prepare.prepare @@ -37,8 +39,8 @@ def test_preparation_keeps_variants_together_and_discards_old_answers(tmp_path): messages=['OLD ANSWER'], utility=True) for i in range(5) for variant in range(2)] manifest = prepare(cases, tmp_path / 'data') - rows = [json.loads(line) for line in (tmp_path / 'data/tasks.jsonl').read_text().splitlines()] - assert len(rows) == 15 + rows = read_cases(tmp_path / 'data') # each case's task directory, read back as its row + assert len(rows) == 15 and (tmp_path / 'data/tasks' / rows[0]['id'] / 'task.toml').is_file() assert all('messages' not in row and 'utility' not in row for row in rows) groups = {} for split in ('train', 'val', 'test'): @@ -61,11 +63,11 @@ def test_prepare_cli_imports_native_logs_from_outside_the_checkout(tmp_path): 'suite_name': 'workspace', 'user_task_id': f'user_task_{i}', 'injection_task_id': 'injection_task_0', 'injections': {'vector': 'frozen'}, 'messages': ['old private answer']})) - result = subprocess.run([sys.executable, str(UTILS / 'prepare.py'), str(logs), str(tmp_path / 'data'), + result = subprocess.run([sys.executable, str(UTILS / 'benchmark' / 'prepare.py'), str(logs), str(tmp_path / 'data'), '--from-logs', '--suite', 'workspace'], cwd=tmp_path, capture_output=True, text=True, check=True) assert json.loads(result.stdout)['counts'] == {'train': 6, 'val': 2, 'test': 2} - assert 'old private answer' not in (tmp_path / 'data/tasks.jsonl').read_text() + assert not any('old private answer' in p.read_text() for p in (tmp_path / 'data/tasks').rglob('*') if p.is_file()) def test_recipe_loads_the_task_entry_point_and_shared_template(): @@ -188,7 +190,7 @@ def test_function_agent_replays_native_tools_and_scores_attack_direction( case, output = replay(tmp_path, model_server, injected=injected) assert output.output == {'utility': True, 'attack_success': attack if injected else None, 'kind': 'injected' if injected else 'clean'} - score = load_callable(str(UTILS / 'judge.py') + ':score', tmp_path)(case, output) + score = load_callable(str(UTILS / 'benchmark' / 'judge.py') + ':score', tmp_path)(case, output) assert score.score == float(not attack) assert score.extra['metrics'] == output.output assert output.extra['benchmark'] == 'agentdojo' @@ -221,7 +223,7 @@ def test_incomplete_native_episodes_are_errors_not_secure_verdicts(tmp_path, nat model_server[1]['value'] = mode case, output = replay(tmp_path, model_server, injected=True, max_iters=1) assert 'episode_error' in output.output - score = load_callable(str(UTILS / 'judge.py') + ':score', tmp_path) + score = load_callable(str(UTILS / 'benchmark' / 'judge.py') + ':score', tmp_path) with pytest.raises(RuntimeError): score(case, output) assert output.tokens > 0 diff --git a/examples/agentdyn/README.md b/examples/agentdyn/README.md index 089ef205..5027f30f 100644 --- a/examples/agentdyn/README.md +++ b/examples/agentdyn/README.md @@ -6,7 +6,7 @@ judges. Nothing here generates or optimizes attacks. | Folder | Contents | |---|---| -| [`utils/`](utils/) | The benchmark integration: the hook bridge to AgentDyn's agent loop and environments (`bridge.py`), the OpenAI-compatible victim (`victim.py`), the optional same-victim classifier (`auditor.py`), the native judge (`judge.py`), case preparation (`prepare*.py`), reports (`report.py`, `compare.py`), the domain specification (`task-info/`) and the data notes ([`DATA.md`](utils/DATA.md)) | +| [`utils/`](utils/) | The benchmark integration: the hook bridge to AgentDyn's agent loop and environments (`bridge.py`), the OpenAI-compatible victim (`victim.py`), the optional same-victim classifier (`auditor.py`), reports (`report.py`, `compare.py`); under `utils/benchmark/`, the task set: case preparation (`prepare*.py`, each protocol's cases as task directories in Harbor's format, `cases.py`), the native judge (`judge.py`), the domain specification (`task-info/`) and the data notes ([`DATA.md`](utils/benchmark/DATA.md)) | | [`evosafe/`](evosafe/) | Experiments with the [EvoSafeHarness](../../docs/methods/evosafe.md) method: run configs and how to run and read them | The replay runtime also supports [official AgentDojo](../agentdojo/) through its @@ -38,7 +38,7 @@ Install both projects in one resolver invocation, so AgentDyn's dependencies can ## Prepare the cases ```bash -cd examples/agentdyn/utils +cd examples/agentdyn/utils/benchmark python prepare_expanded.py /path/to/AgentDyn/runs/google_gemini-2.5-flash data/expanded-v2 python prepare_thirty.py data/expanded-v2 data/thirty-v1 python prepare_all_groups.py data/expanded-v2 data/thirty-v1 data/all-groups-v1 @@ -48,7 +48,11 @@ python prepare_all_groups.py data/expanded-v2 data/thirty-v1 data/full-test-v1 - The importer keeps only suite, user task, injection task and frozen payloads; old traces and scores are discarded. `full-test-v1` has 57 train, 54 validation and 186 test cases. `--full-test` includes every frozen variant in the 18 TEST-owned groups; TRAIN and validation retain their clean case and two injection goals per group. -Both EvoSafe recipes use `full-test-v1`. See [DATA.md](utils/DATA.md) for split ownership and preparation details. +Both EvoSafe recipes use `full-test-v1`. Each protocol directory holds its cases as task directories in Harbor's +format (`tasks//`: the case's row as `task.toml`'s `[metadata]`), `cases.txt` (every case, in the rows' order), the +splits and `manifest.json`; the configs read each case back as its row (`data.view: ../utils/benchmark/cases.py:case_row`), +so the agent loop, the judge and the records see what they saw in `tasks.jsonl`. See [DATA.md](utils/benchmark/DATA.md) +for split ownership and preparation details. ## The bridge and the hooks @@ -70,7 +74,7 @@ hooks around every tool call. | `on_post_tool_call(ctx, result)` | Rewrite or refuse the output. An effect that already happened is not undone. | The hooks get the trusted user instruction and public tool schemas, never task IDs, injection labels, environment -state or judge predicates. The template's `.hermit/EVOLVER.md` and `utils/task-info/README.md` state the contract. +state or judge predicates. The template's `.hermit/EVOLVER.md` and `utils/benchmark/task-info/README.md` state the contract. When `agent.params.auditor.enabled` is set, structured hooks also get `ctx.classify_output(text)`, a bounded, side-effect-free classifier that uses the same frozen victim model. diff --git a/examples/agentdyn/evosafe/README.md b/examples/agentdyn/evosafe/README.md index db3db8d1..6de5714f 100644 --- a/examples/agentdyn/evosafe/README.md +++ b/examples/agentdyn/evosafe/README.md @@ -60,7 +60,7 @@ cases or a complete-run defense objective. These are descriptive fixed-case meas results have not been measured. Recorded on 2026-11-11 in run `agentdyn-evosafe-codex` at framework revision `9c45a65`, using the -[full TEST protocol](../utils/DATA.md#full-test-variants) and the documented AgentDyn webpage patch. +[full TEST protocol](../utils/benchmark/DATA.md#full-test-variants) and the documented AgentDyn webpage patch. The run's backend mapping resolved to `nvidia/nvidia/nemotron-3.5-lightning` (also recorded as `victim_model_id` in its manifest). The learner used local `sandbox.enabled=false` because bubblewrap was unavailable; learned hooks still ran in networkless Docker containers. Committed sandbox defaults @@ -71,7 +71,7 @@ records locally; generated artifacts are not committed. ```bash python ../utils/report.py runs/agentdyn-evosafe-codex/eval/final/results.jsonl \ - --tasks ../utils/data/full-test-v1/tasks.jsonl --split ../utils/data/full-test-v1/test.txt + --tasks ../utils/benchmark/data/full-test-v1/tasks --split ../utils/benchmark/data/full-test-v1/test.txt ``` The experiment runner exports the three-way comparison automatically. For method differences, diff --git a/examples/agentdyn/evosafe/REPRODUCTION.md b/examples/agentdyn/evosafe/REPRODUCTION.md index 2bcb14fa..21c76399 100644 --- a/examples/agentdyn/evosafe/REPRODUCTION.md +++ b/examples/agentdyn/evosafe/REPRODUCTION.md @@ -48,7 +48,7 @@ and GPT-6 Sol with high effort for the learner roles. It uses 57 training cases held-out cases from `full-test-v1`, grouped by user task. Candidate selection uses the training cascade. The [Claude recipe](run_nemotron_claude.yaml) uses the same Lightning victim and TEST set, with a disjoint validation split for candidate confirmation. Both recipes keep `max_iters: 25`. -The dataset revision, construction and split hashes are in [DATA.md](../utils/DATA.md). +The dataset revision, construction and split hashes are in [DATA.md](../utils/benchmark/DATA.md). The experiment runner evaluates `none`, `root`, and `final` on the same test split with the same victim settings. Clean utility is measured on completed clean cases. diff --git a/examples/agentdyn/evosafe/run.yaml b/examples/agentdyn/evosafe/run.yaml index 02bcb914..a83ecfc2 100644 --- a/examples/agentdyn/evosafe/run.yaml +++ b/examples/agentdyn/evosafe/run.yaml @@ -1,11 +1,12 @@ data: - tasks: ../utils/data/tasks.jsonl - train: ../utils/data/train.txt - val: ../utils/data/val.txt - test: ../utils/data/test.txt + tasks: ../utils/benchmark/data/tasks + view: ../utils/benchmark/cases.py:case_row # each case as the row it always was + train: ../utils/benchmark/data/train.txt + val: ../utils/benchmark/data/val.txt + test: ../utils/benchmark/data/test.txt step_size: 4 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info template: evosafe@1.0 agent: name: ../utils/bridge.py:run @@ -18,7 +19,7 @@ agent: max_iters: 15 logs: runs/episodes judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py evolver: name: evosafe params: diff --git a/examples/agentdyn/evosafe/run_nemotron_claude.yaml b/examples/agentdyn/evosafe/run_nemotron_claude.yaml index 2fca0b43..a7382c13 100644 --- a/examples/agentdyn/evosafe/run_nemotron_claude.yaml +++ b/examples/agentdyn/evosafe/run_nemotron_claude.yaml @@ -1,13 +1,14 @@ # Nemotron 3.5 Lightning victim: 57 train / 54 validation / 186 test cases. # Claude Code roles, full TRAIN scoring, bounded error handling, 12 concurrent victim episodes. data: - tasks: ../utils/data/full-test-v1/tasks.jsonl - train: ../utils/data/full-test-v1/train.txt - val: ../utils/data/full-test-v1/val.txt - test: ../utils/data/full-test-v1/test.txt + tasks: ../utils/benchmark/data/full-test-v1/tasks + view: ../utils/benchmark/cases.py:case_row # each case as the row it always was + train: ../utils/benchmark/data/full-test-v1/train.txt + val: ../utils/benchmark/data/full-test-v1/val.txt + test: ../utils/benchmark/data/full-test-v1/test.txt step_size: 57 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info template: evosafe@1.0 agent: name: ../utils/bridge.py:run @@ -34,7 +35,7 @@ agent: max_tokens: 2048 timeout_s: 45 judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py evolver: name: evosafe base: claude diff --git a/examples/agentdyn/evosafe/run_nemotron_codex.yaml b/examples/agentdyn/evosafe/run_nemotron_codex.yaml index 8af09b0f..32f86515 100644 --- a/examples/agentdyn/evosafe/run_nemotron_codex.yaml +++ b/examples/agentdyn/evosafe/run_nemotron_codex.yaml @@ -1,11 +1,12 @@ # Nemotron 3.5 Lightning victim with Codex defense-search roles. data: - tasks: ../utils/data/full-test-v1/tasks.jsonl - train: ../utils/data/full-test-v1/train.txt - test: ../utils/data/full-test-v1/test.txt + tasks: ../utils/benchmark/data/full-test-v1/tasks + view: ../utils/benchmark/cases.py:case_row # each case as the row it always was + train: ../utils/benchmark/data/full-test-v1/train.txt + test: ../utils/benchmark/data/full-test-v1/test.txt step_size: 57 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info template: evosafe@1.0 agent: name: ../utils/bridge.py:run @@ -31,7 +32,7 @@ agent: max_tokens: 2048 timeout_s: 45 judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py evolver: name: evosafe base: codex diff --git a/examples/agentdyn/she/README.md b/examples/agentdyn/she/README.md index 14742dca..d847e3bd 100644 --- a/examples/agentdyn/she/README.md +++ b/examples/agentdyn/she/README.md @@ -17,7 +17,7 @@ source .venv-agentdyn/bin/activate ``` Follow the [shared preparation commands](../README.md#prepare-the-cases), including `--full-test`, to create -`../utils/data/full-test-v1`. The prepared data contains 57 TRAIN, 54 reserved validation and 186 TEST cases, from disjoint +`../utils/benchmark/data/full-test-v1`. The prepared data contains 57 TRAIN, 54 reserved validation and 186 TEST cases, from disjoint user-task groups. SHE uses TRAIN for official best-harness selection and leaves validation unused. TRAIN is unchanged; TEST has 18 clean and 168 injected cases. Preparation never uses model outcomes, and injection payloads and native judges remain unchanged. diff --git a/examples/agentdyn/she/run.yaml b/examples/agentdyn/she/run.yaml index 45f9c8a8..919f32ed 100644 --- a/examples/agentdyn/she/run.yaml +++ b/examples/agentdyn/she/run.yaml @@ -1,10 +1,11 @@ data: - tasks: ../utils/data/full-test-v1/tasks.jsonl - train: ../utils/data/full-test-v1/train.txt - test: ../utils/data/full-test-v1/test.txt + tasks: ../utils/benchmark/data/full-test-v1/tasks + view: ../utils/benchmark/cases.py:case_row # each case as the row it always was + train: ../utils/benchmark/data/full-test-v1/train.txt + test: ../utils/benchmark/data/full-test-v1/test.txt step_size: 57 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info template: she@1.0 agent: name: ../utils/bridge.py:run @@ -25,7 +26,7 @@ agent: base: direct model: vllm/nvidia/nvidia/nemotron-3.5-lightning judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py evolver: name: she params: diff --git a/examples/agentdyn/she/run_nemotron_claude.yaml b/examples/agentdyn/she/run_nemotron_claude.yaml index ef90a61d..62bd49d6 100644 --- a/examples/agentdyn/she/run_nemotron_claude.yaml +++ b/examples/agentdyn/she/run_nemotron_claude.yaml @@ -1,10 +1,11 @@ data: - tasks: ../utils/data/full-test-v1/tasks.jsonl - train: ../utils/data/full-test-v1/train.txt - test: ../utils/data/full-test-v1/test.txt + tasks: ../utils/benchmark/data/full-test-v1/tasks + view: ../utils/benchmark/cases.py:case_row # each case as the row it always was + train: ../utils/benchmark/data/full-test-v1/train.txt + test: ../utils/benchmark/data/full-test-v1/test.txt step_size: 57 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info template: she@1.0 agent: name: ../utils/bridge.py:run @@ -30,7 +31,7 @@ agent: base: direct model: vllm/nvidia/nvidia/nemotron-3.5-lightning judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py evolver: name: she params: diff --git a/examples/agentdyn/she/run_nemotron_codex.yaml b/examples/agentdyn/she/run_nemotron_codex.yaml index 29a7796e..48bec53c 100644 --- a/examples/agentdyn/she/run_nemotron_codex.yaml +++ b/examples/agentdyn/she/run_nemotron_codex.yaml @@ -1,10 +1,11 @@ data: - tasks: ../utils/data/full-test-v1/tasks.jsonl - train: ../utils/data/full-test-v1/train.txt - test: ../utils/data/full-test-v1/test.txt + tasks: ../utils/benchmark/data/full-test-v1/tasks + view: ../utils/benchmark/cases.py:case_row # each case as the row it always was + train: ../utils/benchmark/data/full-test-v1/train.txt + test: ../utils/benchmark/data/full-test-v1/test.txt step_size: 57 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info template: she@1.0 agent: name: ../utils/bridge.py:run @@ -30,7 +31,7 @@ agent: base: direct model: vllm/nvidia/nvidia/nemotron-3.5-lightning judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py evolver: name: she params: diff --git a/examples/agentdyn/she/run_nemotron_codex_daybreak.yaml b/examples/agentdyn/she/run_nemotron_codex_daybreak.yaml index 779a29e6..fccaf557 100644 --- a/examples/agentdyn/she/run_nemotron_codex_daybreak.yaml +++ b/examples/agentdyn/she/run_nemotron_codex_daybreak.yaml @@ -1,10 +1,11 @@ data: - tasks: ../utils/data/full-test-v1/tasks.jsonl - train: ../utils/data/full-test-v1/train.txt - test: ../utils/data/full-test-v1/test.txt + tasks: ../utils/benchmark/data/full-test-v1/tasks + view: ../utils/benchmark/cases.py:case_row # each case as the row it always was + train: ../utils/benchmark/data/full-test-v1/train.txt + test: ../utils/benchmark/data/full-test-v1/test.txt step_size: 57 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info template: she@1.0 agent: name: ../utils/bridge.py:run @@ -30,7 +31,7 @@ agent: base: direct model: vllm/nvidia/nvidia/nemotron-3.5-lightning judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py evolver: name: she params: diff --git a/examples/agentdyn/utils/DATA.md b/examples/agentdyn/utils/benchmark/DATA.md similarity index 89% rename from examples/agentdyn/utils/DATA.md rename to examples/agentdyn/utils/benchmark/DATA.md index 5a97378a..1bc72dd0 100644 --- a/examples/agentdyn/utils/DATA.md +++ b/examples/agentdyn/utils/benchmark/DATA.md @@ -46,8 +46,10 @@ Other tasks were inspected at the source level but are not certified free of def ## Inventory and split ownership `prepare_expanded.py` (seed `20261101`) writes `full/`: 569 cases from 55 eligible user groups, 55 clean plus 514 -injected. Each group is owned by exactly one split: 19 train, 18 validation, 18 test. The SHA-256 of -`full/tasks.jsonl` is `4cc68ea19e1adc518b0a0965641b75c174853396231b55770006b5ac8fac791e`. +injected. Each group is owned by exactly one split: 19 train, 18 validation, 18 test. Every protocol directory holds +its cases as task directories (`tasks/`, the row as `task.toml`'s `[metadata]`) and `cases.txt`, their order; a +SHA-256 below, the manifest's `tasks_sha256`, is that of the rows as JSON lines in that order (what `tasks.jsonl` +held before the task directories). The SHA-256 of `full/`'s rows is `4cc68ea19e1adc518b0a0965641b75c174853396231b55770006b5ac8fac791e`. All variants of a user task stay in their owner's split, and a larger protocol never moves a group to another split. @@ -67,7 +69,7 @@ Three steps build it, each keeping the cases of the previous one: | Test | 18 | 18 | 36 | 54 | The two injection goals per group are chosen by a fixed seed from the goals that group has. The SHA-256 of -`all-groups-v1/tasks.jsonl` is `2f5d3943caf7d56d0d50c1998c6a2154f524fb2178ca4119fe89d7b2846d69f4`. +`all-groups-v1`'s rows is `2f5d3943caf7d56d0d50c1998c6a2154f524fb2178ca4119fe89d7b2846d69f4`. Validation cases serve only to confirm a candidate. Test cases are used only by `hermit eval` and are never shown to the roles. @@ -78,7 +80,7 @@ owned by the existing TEST groups. This produces 186 TEST cases: 18 clean and 168 injected across the same 18 groups, including the original 54 cases plus 132 additional injected variants. TRAIN (57 cases), validation (54 cases), split ownership and injection payloads are unchanged. Unselected variants in TRAIN and validation groups remain excluded. -Selection does not use model outcomes. The SHA-256 of `full-test-v1/tasks.jsonl` is +Selection does not use model outcomes. The SHA-256 of `full-test-v1`'s rows is `b529dfa72103fe8c55972e1960dcf492adda2be7dfdd6b1c0a87433007e043fb`. The Codex EvoSafe recipe uses TRAIN for search and TEST for frozen evaluation, with no validation gate. diff --git a/examples/agentdyn/utils/benchmark/cases.py b/examples/agentdyn/utils/benchmark/cases.py new file mode 100644 index 00000000..d139ecff --- /dev/null +++ b/examples/agentdyn/utils/benchmark/cases.py @@ -0,0 +1,74 @@ +"""A protocol's frozen cases as task directories (Harbor's format), shared by AgentDyn's and AgentDojo's prepare +scripts: + + /tasks// instruction.md (the case in a line) and task.toml, whose [metadata] `case` is the case's + row as one JSON line (TOML holds no null, which a clean case's injection_task_id is) + /cases.txt every case's id, in the order the rows were written (tasks.jsonl's lines before) + /train.txt ... the splits, and manifest.json, written by the prepare script + +`case_row` is the configs' data.view: a case's task directory read back as exactly the row tasks.jsonl held, so the +agent loop, the judge, the records and the learning material are what they were. `write_cases` returns the SHA-256 +of the rows as JSON lines (the bytes tasks.jsonl held), which the manifests keep as `tasks_sha256` and DATA.md lists. +A case asks for no network (its environment is AgentDyn's or AgentDojo's own simulation; the victim's model is the +run's) and has no tests/: the judge (judge.py) reads the native verdicts from the agent's output. +""" +from __future__ import annotations + +import hashlib +import json +from pathlib import Path +import shutil + +from hermit.task.directory import write_task + +NAMES = {"agentdyn": "AgentDyn", "agentdojo": "AgentDojo"} + + +def case_row(task: dict) -> dict: + """data.view: a case's task directory as its row (the row as it is when it is one already).""" + if "metadata" not in task: + return dict(task) + return json.loads(task["metadata"]["case"]) + + +def instruction(row: dict, benchmark: str) -> str: + text = f"{NAMES.get(benchmark, benchmark)} {row['suite']} user task `{row['user_task_id']}`" + if row.get("injection_task_id"): + text += f", with the frozen injection of `{row['injection_task_id']}`" + return text + ": complete the user's task in the suite's native environment." + + +def write_cases(output: Path, rows: list[dict], *, benchmark: str, sort_keys: bool = False) -> str: + """Every row as a task directory under output/tasks (made anew) and their order in output/cases.txt; the SHA-256 + of the rows as JSON lines (`sort_keys` as the JSON lines were written).""" + output = Path(output) + # As the JSON lines read back: sort_keys sorts every mapping's keys, the injections' too + rows = [json.loads(json.dumps(r, sort_keys=sort_keys)) for r in rows] + if (output / "tasks").exists(): + shutil.rmtree(output / "tasks") + for row in rows: + write_task(output / "tasks" / row["id"], instruction(row, benchmark), metadata={"case": json.dumps(row)}, + environment={"network_mode": "no-network"}) + (output / "cases.txt").write_text("".join(r["id"] + "\n" for r in rows)) + return hashlib.sha256("".join(json.dumps(r) + "\n" for r in rows).encode()).hexdigest() + + +def read_cases(protocol: Path) -> list[dict]: + """A protocol's rows in their order: its task directories through case_row (cases.txt's order), or the + tasks.jsonl of a protocol prepared before the task directories.""" + protocol = Path(protocol) + if (protocol / "tasks").is_dir(): + from hermit.task.directory import tasks_of + found = {tid: t.given() for tid, t in tasks_of(protocol / "tasks", view=case_row).items()} + order = [i for i in (protocol / "cases.txt").read_text().splitlines() if i.strip()] \ + if (protocol / "cases.txt").is_file() else sorted(found) + return [found[i] for i in order] + return [json.loads(s) for s in (protocol / "tasks.jsonl").read_text().splitlines() if s.strip()] + + +def rows(path: Path) -> list[dict]: + """The rows a report reads: of a protocol directory, of its tasks/ directory, or of a tasks.jsonl.""" + path = Path(path) + if path.is_file(): + return [json.loads(s) for s in path.read_text().splitlines() if s.strip()] + return read_cases(path.parent if path.name == "tasks" and not (path / "tasks").is_dir() else path) diff --git a/examples/agentdyn/utils/judge.py b/examples/agentdyn/utils/benchmark/judge.py similarity index 63% rename from examples/agentdyn/utils/judge.py rename to examples/agentdyn/utils/benchmark/judge.py index 9bd87433..83688b0e 100644 --- a/examples/agentdyn/utils/judge.py +++ b/examples/agentdyn/utils/benchmark/judge.py @@ -1,6 +1,10 @@ """AgentDyn's judge for EvoSafe: native verdicts as per-episode metrics in the judge's `extra` (bridge.score). Task-level defense metrics are computed from the run records by the method's contract (`contract.summary`), in the Evolver and in `report.py`, not by the framework.""" -from bridge import score +from pathlib import Path +import sys + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) # AgentDyn's runtime (bridge.py) +from bridge import score # noqa: E402 __all__ = ["score"] diff --git a/examples/agentdyn/utils/prepare.py b/examples/agentdyn/utils/benchmark/prepare.py similarity index 89% rename from examples/agentdyn/utils/prepare.py rename to examples/agentdyn/utils/benchmark/prepare.py index a8d95c28..e53ebc82 100644 --- a/examples/agentdyn/utils/prepare.py +++ b/examples/agentdyn/utils/benchmark/prepare.py @@ -1,11 +1,16 @@ -"""Normalize fixed replay cases; split by user task, never by injection variant.""" +"""Normalize fixed replay cases; split by user task, never by injection variant. Writes the protocol's cases as task +directories (cases.py), its splits and its manifest.""" import argparse import hashlib import json from pathlib import Path import random +import sys -from bridge import benchmark_settings, case_fields, validate_case +sys.path[:0] = [str(Path(__file__).resolve().parents[1]), # AgentDyn's runtime (bridge.py) + str(Path(__file__).resolve().parent)] # cases.py +from bridge import benchmark_settings, case_fields, validate_case # noqa: E402 +from cases import write_cases # noqa: E402 def read_logs(directory, suite): @@ -59,13 +64,13 @@ def prepare(cases, output, seed=0, *, benchmark="agentdyn"): raise ValueError(f"{name} has no injected cases; provide coverage for more user tasks") output = Path(output) output.mkdir(parents=True, exist_ok=False) - (output / "tasks.jsonl").write_text("".join(json.dumps(case_fields(r)) + "\n" for r in rows)) + tasks_sha256 = write_cases(output, [case_fields(r) for r in rows], benchmark=benchmark) for name, members in splits.items(): (output / f"{name}.txt").write_text("".join(r["id"] + "\n" for r in members)) manifest = {"seed": seed, "benchmark": benchmark, "reference_revision": revision, "groups": [{"suite": s, "user_task_id": u, "split": assignment[(s,u)]} for s,u in ordered], "counts": {k: len(v) for k,v in splits.items()}, - "tasks_sha256": hashlib.sha256((output / "tasks.jsonl").read_bytes()).hexdigest()} + "tasks_sha256": tasks_sha256} (output / "manifest.json").write_text(json.dumps(manifest, indent=2)) return manifest diff --git a/examples/agentdyn/utils/prepare_all_groups.py b/examples/agentdyn/utils/benchmark/prepare_all_groups.py similarity index 89% rename from examples/agentdyn/utils/prepare_all_groups.py rename to examples/agentdyn/utils/benchmark/prepare_all_groups.py index 812c8f37..2a596f14 100644 --- a/examples/agentdyn/utils/prepare_all_groups.py +++ b/examples/agentdyn/utils/benchmark/prepare_all_groups.py @@ -7,12 +7,15 @@ Reads only the inventory and split files, never model outcomes. """ import argparse -import hashlib import json from pathlib import Path import random +import sys -from bridge import case_fields +sys.path[:0] = [str(Path(__file__).resolve().parents[1]), # AgentDyn's runtime (bridge.py) + str(Path(__file__).resolve().parent)] # cases.py +from bridge import case_fields # noqa: E402 +from cases import read_cases, write_cases # noqa: E402 SPLITS = ('train', 'val', 'test') @@ -23,7 +26,7 @@ def prepare(inventory, base, output, goals=2, seed=20261102, *, full_test=False) raise ValueError('goals must be a positive integer') if output.exists(): raise FileExistsError(output) - rows = [json.loads(s) for s in (inventory / 'full/tasks.jsonl').read_text().splitlines() if s.strip()] + rows = read_cases(inventory / 'full') by_id = {r['id']: r for r in rows} owners = {(g['suite'], g['user_task_id']): g['split'] for g in json.loads((inventory / 'full/manifest.json').read_text())['groups']} @@ -54,7 +57,8 @@ def prepare(inventory, base, output, goals=2, seed=20261102, *, full_test=False) splits[split] = selected output.mkdir(parents=True) used = {i for ids in splits.values() for i in ids} - (output / 'tasks.jsonl').write_text(''.join(json.dumps(case_fields(by_id[i]), sort_keys=True) + '\n' for i in sorted(used))) + tasks_sha256 = write_cases(output, [case_fields(by_id[i]) for i in sorted(used)], benchmark='agentdyn', + sort_keys=True) for split, ids in splits.items(): (output / f'{split}.txt').write_text('\n'.join(ids) + '\n') clean = {s: sum(not by_id[i].get('injection_task_id') for i in ids) for s, ids in splits.items()} @@ -67,7 +71,7 @@ def prepare(inventory, base, output, goals=2, seed=20261102, *, full_test=False) 'counts': {s: len(ids) for s, ids in splits.items()}, 'clean': clean, 'groups': [{'suite': s, 'user_task_id': t, 'split': owners[(s, t)]} for s, t in sorted({group(by_id[i]) for i in used})], - 'tasks_sha256': hashlib.sha256((output / 'tasks.jsonl').read_bytes()).hexdigest()} + 'tasks_sha256': tasks_sha256} (output / 'manifest.json').write_text(json.dumps(report, indent=2) + '\n') return report diff --git a/examples/agentdyn/utils/prepare_expanded.py b/examples/agentdyn/utils/benchmark/prepare_expanded.py similarity index 95% rename from examples/agentdyn/utils/prepare_expanded.py rename to examples/agentdyn/utils/benchmark/prepare_expanded.py index 766ef4d1..9ee00a69 100644 --- a/examples/agentdyn/utils/prepare_expanded.py +++ b/examples/agentdyn/utils/benchmark/prepare_expanded.py @@ -11,8 +11,12 @@ import json from pathlib import Path import random +import sys -from bridge import AGENTDYN_REVISION, case_fields, validate_case +sys.path[:0] = [str(Path(__file__).resolve().parents[1]), # AgentDyn's runtime (bridge.py) + str(Path(__file__).resolve().parent)] # cases.py +from bridge import AGENTDYN_REVISION, case_fields, validate_case # noqa: E402 +from cases import write_cases # noqa: E402 SUITES = ("shopping", "github", "dailylife") # Source-level judge discrepancies, fixed before any model-backed expanded run. @@ -139,7 +143,7 @@ def export_full(output, cases, audit, assignments, seed): rows.extend({"id": f"{s}-{t}-{c['injection_task_id']}-{k[:8]}", **c} for k, c in sorted(eligible.items()) if (c["suite"], c["user_task_id"]) == (s, t)) output.mkdir(parents=True, exist_ok=False) - (output / "tasks.jsonl").write_text("".join(json.dumps(case_fields(r), sort_keys=True) + "\n" for r in rows)) + tasks_sha256 = write_cases(output, [case_fields(r) for r in rows], benchmark="agentdyn", sort_keys=True) counts = {} for split in ("train", "val", "test"): buckets = {s: [r["id"] for r in rows if r["suite"] == s @@ -151,7 +155,7 @@ def export_full(output, cases, audit, assignments, seed): manifest = {"protocol": "full eligible frozen inventory; not a completed live evaluation", "reference_revision": AGENTDYN_REVISION, "seed": seed, "counts": counts, "groups": [{"suite": s, "user_task_id": t, "split": owners[(s, t)]} for s, t in groups], - "tasks_sha256": hashlib.sha256((output / "tasks.jsonl").read_bytes()).hexdigest()} + "tasks_sha256": tasks_sha256} (output / "manifest.json").write_text(json.dumps(manifest, indent=2)) return manifest @@ -175,11 +179,11 @@ def prepare(source, output, seed=20261101, groups_per_split=2, goals_per_group=2 "audit_scope": "native ID/vector/environment initialization and no-op verdicts; not semantic pairing certification", } output.mkdir(parents=True) - (output / "tasks.jsonl").write_text("".join(json.dumps(case_fields(r), sort_keys=True) + "\n" for r in rows)) + tasks_sha256 = write_cases(output, [case_fields(r) for r in rows], benchmark="agentdyn", sort_keys=True) for split, ids in splits.items(): (output / f"{split}.txt").write_text("\n".join(ids) + "\n") (output / "source-audit.json").write_text(json.dumps({"inventory": inventory, "cases": audit}, indent=2)) - manifest["tasks_sha256"] = hashlib.sha256((output / "tasks.jsonl").read_bytes()).hexdigest() + manifest["tasks_sha256"] = tasks_sha256 manifest["split_sha256"] = {s: hashlib.sha256((output / f"{s}.txt").read_bytes()).hexdigest() for s in splits} (output / "manifest.json").write_text(json.dumps(manifest, indent=2)) export_full(output / "full", cases, audit, assignments, seed) diff --git a/examples/agentdyn/utils/prepare_thirty.py b/examples/agentdyn/utils/benchmark/prepare_thirty.py similarity index 87% rename from examples/agentdyn/utils/prepare_thirty.py rename to examples/agentdyn/utils/benchmark/prepare_thirty.py index 4ff9bf8e..6a289479 100644 --- a/examples/agentdyn/utils/prepare_thirty.py +++ b/examples/agentdyn/utils/benchmark/prepare_thirty.py @@ -4,12 +4,15 @@ Retains old cases verbatim; adds frozen payloads from already-owned groups. """ import argparse -import hashlib import json from pathlib import Path import random +import sys -from bridge import case_fields +sys.path[:0] = [str(Path(__file__).resolve().parents[1]), # AgentDyn's runtime (bridge.py) + str(Path(__file__).resolve().parent)] # cases.py +from bridge import case_fields # noqa: E402 +from cases import read_cases, write_cases # noqa: E402 SUITES = ('shopping', 'github', 'dailylife') SPLITS = ('train', 'val', 'test') @@ -19,7 +22,7 @@ def prepare(source, output, seed=20261102): source, output = Path(source), Path(output) if output.exists(): raise FileExistsError(output) - rows = [json.loads(s) for s in (source / 'full/tasks.jsonl').read_text().splitlines()] + rows = read_cases(source / 'full') by_id = {r['id']: r for r in rows} manifest = json.loads((source / 'full/manifest.json').read_text()) owners = {(g['suite'], g['user_task_id']): g['split'] for g in manifest['groups']} @@ -55,7 +58,8 @@ def prepare(source, output, seed=20261102): splits[split] = selected output.mkdir(parents=True) used = {i for ids in splits.values() for i in ids} - (output / 'tasks.jsonl').write_text(''.join(json.dumps(case_fields(by_id[i]), sort_keys=True) + '\n' for i in sorted(used))) + tasks_sha256 = write_cases(output, [case_fields(by_id[i]) for i in sorted(used)], benchmark='agentdyn', + sort_keys=True) for split, ids in splits.items(): (output / f'{split}.txt').write_text('\n'.join(ids) + '\n') report = {'protocol': '30-case grouped frozen replay: 10 clean + 20 injected per split', @@ -64,7 +68,7 @@ def prepare(source, output, seed=20261102): 'groups': [{'suite': s, 'user_task_id': t, 'split': owners[(s, t)]} for s, t in sorted({group(by_id[i]) for i in used})], 'preserved_prior_split_ownership': True, - 'tasks_sha256': hashlib.sha256((output / 'tasks.jsonl').read_bytes()).hexdigest()} + 'tasks_sha256': tasks_sha256} (output / 'manifest.json').write_text(json.dumps(report, indent=2) + '\n') return report diff --git a/examples/agentdyn/utils/task-info/README.md b/examples/agentdyn/utils/benchmark/task-info/README.md similarity index 100% rename from examples/agentdyn/utils/task-info/README.md rename to examples/agentdyn/utils/benchmark/task-info/README.md diff --git a/examples/agentdyn/utils/task-info/tools.json b/examples/agentdyn/utils/benchmark/task-info/tools.json similarity index 100% rename from examples/agentdyn/utils/task-info/tools.json rename to examples/agentdyn/utils/benchmark/task-info/tools.json diff --git a/examples/agentdyn/utils/compare.py b/examples/agentdyn/utils/compare.py index c62e6a4f..f0f22ad7 100644 --- a/examples/agentdyn/utils/compare.py +++ b/examples/agentdyn/utils/compare.py @@ -88,6 +88,9 @@ def compare(tasks, ids, runs, repeats=1): if __name__ == '__main__': + import sys + sys.path.insert(0, str(Path(__file__).resolve().parent / 'benchmark')) + from cases import rows as case_rows # a protocol's tasks/, or a tasks.jsonl parser = argparse.ArgumentParser(description=__doc__) parser.add_argument('--tasks',type=Path,required=True) parser.add_argument('--split',type=Path,required=True) @@ -100,7 +103,8 @@ def compare(tasks, ids, runs, repeats=1): name,sep,path=entry.partition('=') if not sep or name in runs: raise ValueError('provide unique LABEL=path') runs[name]=load(path) - result=compare(load(args.tasks),[s for s in args.split.read_text().splitlines() if s and not s.startswith('#')],runs,args.repeats) + result=compare(case_rows(args.tasks), + [s for s in args.split.read_text().splitlines() if s and not s.startswith('#')],runs,args.repeats) text=json.dumps(result,indent=2) if args.output: args.output.write_text(text+'\n') print(text) diff --git a/examples/agentdyn/utils/experiment_runtime.py b/examples/agentdyn/utils/experiment_runtime.py index 4b91b6e0..fa84d621 100644 --- a/examples/agentdyn/utils/experiment_runtime.py +++ b/examples/agentdyn/utils/experiment_runtime.py @@ -6,6 +6,7 @@ import os from pathlib import Path import re +import shutil import signal import subprocess import tarfile @@ -110,6 +111,18 @@ def run_owned(command, *, cwd, stream, registry, heartbeat=lambda: None, started cleanup_containers(registry) +def digest(path): + """A data input's SHA-256: a file's bytes; a directory's (a task set's task directories), every file's path and + bytes, in order.""" + path = Path(path) + if not path.is_dir(): # a file (one that is missing raises) + return hashlib.sha256(path.read_bytes()).hexdigest() + h = hashlib.sha256() + for p in sorted(q for q in path.rglob('*') if q.is_file()): + h.update(str(p.relative_to(path)).encode() + b'\0' + hashlib.sha256(p.read_bytes()).digest()) + return h.hexdigest() + + def snapshot(root, config_path, config, *, sandbox_enabled=None): """Archive the committed tree; never bind live repository files into a measured run.""" root = Path(root).resolve() @@ -144,7 +157,10 @@ def snapshot(root, config_path, config, *, sandbox_enabled=None): continue source = config.path(config.data['data'][key]) target = inputs / (key + source.suffix) - target.write_bytes(source.read_bytes()) + if source.is_dir(): # a task set of task directories + shutil.copytree(source, target, symlinks=False) + else: + target.write_bytes(source.read_bytes()) effective['data'][key] = str(target) # Public instructions and the template must come from the committed tree. for key, source in [('task_info', config.path(config.data['task_info'])), @@ -164,13 +180,20 @@ def snapshot(root, config_path, config, *, sandbox_enabled=None): if target is None or not target.exists(): raise ValueError(f'{name} must be committed inside the repository') holder[key] = str(target) - # File-backed application entry points must resolve inside the archived source. - for section, key in [('agent', 'name'), ('judge', 'path')]: - value = effective[section][key] + # File-backed application entry points must resolve inside the archived source: the agent, the judge, and the + # view the task set's rows are read through (data.view). + for section, key in [('agent', 'name'), ('judge', 'path'), ('data', 'view')]: + value = (effective.get(section) or {}).get(key) + if not value: + continue filename, separator, function = value.partition(':') + if section == 'data' and not filename.endswith('.py'): + # a module's view would be read from wherever it is installed, live and unhashed + raise ValueError('data.view must be a committed file entry point (.py:)') if filename.endswith('.py'): - target = frozen / config.path(filename).relative_to(root) - if not target.is_file(): + source = config.path(filename).resolve() + target = frozen / source.relative_to(root) if source.is_relative_to(root) else None + if target is None or not target.is_file(): raise ValueError(f'{section}.{key} must be a committed entry point') effective[section][key] = str(target) + (separator + function if separator else '') effective['trainer']['output_dir'] = str(config.run_dir().parent) diff --git a/examples/agentdyn/utils/report.py b/examples/agentdyn/utils/report.py index 469eaef2..3fa61fed 100644 --- a/examples/agentdyn/utils/report.py +++ b/examples/agentdyn/utils/report.py @@ -55,11 +55,16 @@ def summarize(rows, expected_ids, repeats=1, *, tasks=None): if __name__ == "__main__": + import sys + sys.path.insert(0, str(Path(__file__).resolve().parent / "benchmark")) + from cases import rows as case_rows # a protocol's tasks/, or a tasks.jsonl parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("results", type=Path) - parser.add_argument("--tasks", type=Path, required=True, help="task JSONL for ID-to-kind mapping") + parser.add_argument("--tasks", type=Path, required=True, + help="the protocol's tasks/ (or a tasks.jsonl) for ID-to-kind mapping") parser.add_argument("--split", type=Path, required=True, help="expected test IDs, one per line") parser.add_argument("--repeats", type=int, default=1) args = parser.parse_args() ids = [s.strip() for s in args.split.read_text().splitlines() if s.strip() and not s.startswith("#")] - print(json.dumps(summarize([json.loads(s) for s in args.results.read_text().splitlines() if s.strip()], ids, args.repeats, tasks=[json.loads(s) for s in args.tasks.read_text().splitlines() if s.strip()]), indent=2)) + print(json.dumps(summarize([json.loads(s) for s in args.results.read_text().splitlines() if s.strip()], ids, + args.repeats, tasks=case_rows(args.tasks)), indent=2)) diff --git a/examples/agentdyn/utils/run_experiment.py b/examples/agentdyn/utils/run_experiment.py index c5b5d5fa..da11b1f0 100644 --- a/examples/agentdyn/utils/run_experiment.py +++ b/examples/agentdyn/utils/run_experiment.py @@ -15,7 +15,10 @@ from hermit.tree import Tree from report import summarize from compare import compare -from experiment_runtime import (ExperimentInterrupted, interruption_handlers, run_owned, snapshot, verify_snapshot) +from experiment_runtime import (ExperimentInterrupted, digest, interruption_handlers, run_owned, snapshot, + verify_snapshot) +sys.path.insert(0, str(Path(__file__).resolve().parent / 'benchmark')) +from cases import rows as case_rows # noqa: E402 def read_rows(path): @@ -301,7 +304,7 @@ def started(pid): for key in ('tasks', 'train', 'val', 'test'): if key == 'val' and not config.data['data'].get(key): continue - manifest['data_sha256'][key] = hashlib.sha256(config.path(config.data['data'][key]).read_bytes()).hexdigest() + manifest['data_sha256'][key] = digest(config.path(config.data['data'][key])) for source in config.path(config.data['task_info']).iterdir(): if source.is_file(): manifest['task_info_sha256'][source.name] = hashlib.sha256(source.read_bytes()).hexdigest() @@ -346,7 +349,7 @@ def started(pid): method=config.data['evolver']['name']) save() runs = {} - tasks = read_rows(config.path(config.data['data']['tasks'])) + tasks = case_rows(config.path(config.data['data']['tasks'])) ids = config.path(config.data['data']['test']).read_text().splitlines() ids = [i.strip() for i in ids if i.strip() and not i.startswith('#')] for adapter in ('none', 'root', 'final'): diff --git a/examples/algotune/baseline/claude.yaml b/examples/algotune/baseline/claude.yaml index 7b484505..190ea40b 100644 --- a/examples/algotune/baseline/claude.yaml +++ b/examples/algotune/baseline/claude.yaml @@ -16,15 +16,14 @@ data: train: ../utils/splits/smoke.txt # placeholder, as template and evolver are: hermit eval reads it, trains nothing test: ../utils/splits/test.txt # 108 tasks (SkillOpt's split: 2:1:7, seed 42) agent: - name: harbor + base: claude # Claude Code in each task's own Harbor environment, installed there by Harbor concurrency: 6 - params: - benchmark: ../utils/benchmark.yaml - acting_agent: {cli: claude} # Claude Code in the task's container, installed there by Harbor - model: claude-haiku-5-5 # a Claude Code model name, on the subscription - login: claude # the subscription's plan, shared with Hermit's other sessions on it - effort: medium - jobs_dir: ~/.cache/hermit/harbor/jobs/algotune-claude # the Claude Code configs' trials + model: {name: claude-haiku-5-5, login: claude, effort: medium} # the subscription's plan, shared + claude: {web_search: false} # the CLI's web search off +environment: + harbor: # each task in its own Harbor environment, verified by Harbor's verifier + benchmark: ../utils/benchmark.yaml # the settings every config of this example shares + jobs_dir: ~/.cache/hermit/harbor/jobs/algotune-claude # the Claude Code configs' environments (each task's trial) judge: path: ../utils/judge.py # 1 - 1/speedup (utils/judge.py) template: prompt-only@1.0 # placeholder (see data.train) diff --git a/examples/algotune/baseline/codex.yaml b/examples/algotune/baseline/codex.yaml index cbf27329..2dcabcf0 100644 --- a/examples/algotune/baseline/codex.yaml +++ b/examples/algotune/baseline/codex.yaml @@ -15,15 +15,14 @@ data: train: ../utils/splits/smoke.txt # placeholder, as template and evolver are: hermit eval reads it, trains nothing test: ../utils/splits/test.txt # 108 tasks (SkillOpt's split: 2:1:7, seed 42) agent: - name: harbor + base: codex # Codex in each task's own Harbor environment, installed there by Harbor concurrency: 6 - params: - benchmark: ../utils/benchmark.yaml - acting_agent: {cli: codex} # Codex in the task's container, installed there by Harbor - model: gpt-5.6-luna # a Codex model name, on the ChatGPT login - login: chatgpt # the ChatGPT plan, shared with Hermit's other sessions on it - effort: medium - jobs_dir: ~/.cache/hermit/harbor/jobs/algotune-codex # the Codex configs' trials + model: {name: gpt-5.6-luna, login: chatgpt, effort: medium} # the ChatGPT plan, shared + codex: {web_search: false} # the CLI's web search off +environment: + harbor: # each task in its own Harbor environment, verified by Harbor's verifier + benchmark: ../utils/benchmark.yaml # the settings every config of this example shares + jobs_dir: ~/.cache/hermit/harbor/jobs/algotune-codex # the Codex configs' environments (each task's trial) judge: path: ../utils/judge.py # 1 - 1/speedup (utils/judge.py) template: prompt-only@1.0 # placeholder (see data.train) diff --git a/examples/algotune/skillopt/run_claude.yaml b/examples/algotune/skillopt/run_claude.yaml index 7f6705aa..6c2b909c 100644 --- a/examples/algotune/skillopt/run_claude.yaml +++ b/examples/algotune/skillopt/run_claude.yaml @@ -27,15 +27,14 @@ data: shuffle_seed: 42 task_info: ../utils/task-info agent: - name: harbor + base: claude # Claude Code in each task's own Harbor environment, installed there by Harbor concurrency: 6 - params: - benchmark: ../utils/benchmark.yaml - acting_agent: {cli: claude} # Claude Code in the task's container, installed there by Harbor - model: claude-haiku-5-5 # a Claude Code model name, on the subscription - login: claude # the subscription's plan, shared with Hermit's other sessions on it - effort: medium - jobs_dir: ~/.cache/hermit/harbor/jobs/algotune-claude # the Claude Code configs' trials + model: {name: claude-haiku-5-5, login: claude, effort: medium} # the subscription's plan, shared + claude: {web_search: false} # the CLI's web search off +environment: + harbor: # each task in its own Harbor environment, verified by Harbor's verifier + benchmark: ../utils/benchmark.yaml # the settings every config of this example shares + jobs_dir: ~/.cache/hermit/harbor/jobs/algotune-claude # the Claude Code configs' environments (each task's trial) judge: path: ../utils/judge.py # 1 - 1/speedup (utils/judge.py) template: prompt-only@1.0 # the skill is ADAPTER.md, with no instructions at first (`--adapter root`) diff --git a/examples/algotune/skillopt/run_codex.yaml b/examples/algotune/skillopt/run_codex.yaml index 35e9767e..957863b3 100644 --- a/examples/algotune/skillopt/run_codex.yaml +++ b/examples/algotune/skillopt/run_codex.yaml @@ -26,15 +26,14 @@ data: shuffle_seed: 42 task_info: ../utils/task-info agent: - name: harbor + base: codex # Codex in each task's own Harbor environment, installed there by Harbor concurrency: 6 - params: - benchmark: ../utils/benchmark.yaml - acting_agent: {cli: codex} # Codex in the task's container, installed there by Harbor - model: gpt-5.6-luna # a Codex model name, on the ChatGPT login - login: chatgpt # the ChatGPT plan, shared with Hermit's other sessions on it - effort: medium - jobs_dir: ~/.cache/hermit/harbor/jobs/algotune-codex # the Codex configs' trials + model: {name: gpt-5.6-luna, login: chatgpt, effort: medium} # the ChatGPT plan, shared + codex: {web_search: false} # the CLI's web search off +environment: + harbor: # each task in its own Harbor environment, verified by Harbor's verifier + benchmark: ../utils/benchmark.yaml # the settings every config of this example shares + jobs_dir: ~/.cache/hermit/harbor/jobs/algotune-codex # the Codex configs' environments (each task's trial) judge: path: ../utils/judge.py # 1 - 1/speedup (utils/judge.py) template: prompt-only@1.0 # the skill is ADAPTER.md, with no instructions at first (`--adapter root`) diff --git a/examples/algotune/skillopt/smoke_claude.yaml b/examples/algotune/skillopt/smoke_claude.yaml index be65918e..a5255ec8 100644 --- a/examples/algotune/skillopt/smoke_claude.yaml +++ b/examples/algotune/skillopt/smoke_claude.yaml @@ -13,16 +13,15 @@ data: shuffle_seed: 42 task_info: ../utils/task-info agent: - name: harbor + base: claude # Claude Code in each task's own Harbor environment, installed there by Harbor concurrency: 2 - params: - benchmark: ../utils/benchmark.yaml - acting_agent: {cli: claude} # Claude Code in the task's container, installed there by Harbor - model: claude-haiku-5-5 # a Claude Code model name, on the subscription - login: claude # the subscription's plan, shared with Hermit's other sessions on it - effort: medium - jobs_dir: ~/.cache/hermit/harbor/jobs/algotune-claude-smoke # the smoke's trials, apart from the run's - agent_timeout_sec: 1800 # the smoke's cap on a session; the run keeps each task's own + model: {name: claude-haiku-5-5, login: claude, effort: medium} # the subscription's plan, shared + claude: {web_search: false} # the CLI's web search off +environment: + harbor: # each task in its own Harbor environment, verified by Harbor's verifier + benchmark: ../utils/benchmark.yaml # the settings every config of this example shares + jobs_dir: ~/.cache/hermit/harbor/jobs/algotune-claude-smoke # the smoke's environments, apart from the run's + agent_timeout_sec: 1800 # the smoke's cap on a session; the run keeps each task's own judge: path: ../utils/judge.py # 1 - 1/speedup (utils/judge.py) template: prompt-only@1.0 # the skill is ADAPTER.md, with no instructions at first (`--adapter root`) diff --git a/examples/algotune/skillopt/smoke_codex.yaml b/examples/algotune/skillopt/smoke_codex.yaml index b43c7038..ec3ba299 100644 --- a/examples/algotune/skillopt/smoke_codex.yaml +++ b/examples/algotune/skillopt/smoke_codex.yaml @@ -13,16 +13,15 @@ data: shuffle_seed: 42 task_info: ../utils/task-info agent: - name: harbor + base: codex # Codex in each task's own Harbor environment, installed there by Harbor concurrency: 2 - params: - benchmark: ../utils/benchmark.yaml - acting_agent: {cli: codex} # Codex in the task's container, installed there by Harbor - model: gpt-5.6-luna # a Codex model name, on the ChatGPT login - login: chatgpt # the ChatGPT plan, shared with Hermit's other sessions on it - effort: medium - jobs_dir: ~/.cache/hermit/harbor/jobs/algotune-codex-smoke # the smoke's trials, apart from the run's - agent_timeout_sec: 1800 # the smoke's cap on a session; the run keeps each task's own + model: {name: gpt-5.6-luna, login: chatgpt, effort: medium} # the ChatGPT plan, shared + codex: {web_search: false} # the CLI's web search off +environment: + harbor: # each task in its own Harbor environment, verified by Harbor's verifier + benchmark: ../utils/benchmark.yaml # the settings every config of this example shares + jobs_dir: ~/.cache/hermit/harbor/jobs/algotune-codex-smoke # the smoke's environments, apart from the run's + agent_timeout_sec: 1800 # the smoke's cap on a session; the run keeps each task's own judge: path: ../utils/judge.py # 1 - 1/speedup (utils/judge.py) template: prompt-only@1.0 # the skill is ADAPTER.md, with no instructions at first (`--adapter root`) diff --git a/examples/algotune/utils/benchmark.yaml b/examples/algotune/utils/benchmark.yaml index 7f830c66..d07f65bd 100644 --- a/examples/algotune/utils/benchmark.yaml +++ b/examples/algotune/utils/benchmark.yaml @@ -1,17 +1,17 @@ -# AlgoTune through Harbor: the agent.params every config of this example shares (agent.params.benchmark; a config's -# own params override these), and the task set (the dataset: block, which `python -m hermit.harbor.dataset prepare` -# reads). One rollout is one Harbor trial of a task: Harbor builds the task's image (Python with the solver packages -# the instruction lists), the acting agent writes /app/solver.py, and the task's verifier (uploaded only after the -# agent stopped) times it against the reference solver on 100 instances, interleaved. Its reward is the speedup when -# every output is valid and the solver is faster, else 1.0 (AlgoTune's floor). AlgoTune's score of a set of tasks is -# the harmonic mean of the speedups; judge.py scores a trial 1 - 1/speedup, whose mean m gives that harmonic mean as -# 1 / (1 - m). The acting agent, its model and its sign-in are each config's own. +# AlgoTune in Harbor's environments: the environment.harbor settings every config of this example shares +# (environment.harbor.benchmark; a config's own settings override these), and the task set (the dataset: block, which +# `python -m hermit.harbor.dataset prepare` reads). One rollout is one task in its own Harbor environment: Harbor +# builds the task's image (Python with the solver packages the instruction lists), the acting agent writes +# /app/solver.py, and the task's verifier (uploaded only after the agent stopped) times it against the reference +# solver on 100 instances, interleaved. Its reward is the speedup when every output is valid and the solver is faster, +# else 1.0 (AlgoTune's floor). AlgoTune's score of a set of tasks is the harmonic mean of the speedups; judge.py +# scores a trial 1 - 1/speedup, whose mean m gives that harmonic mean as 1 / (1 - m). The acting agent, its model and +# its sign-in are each config's own. dataset: ref: algotune@1.0 # 154 tasks split: {train: 31, val: 15, test: rest, seed: 42} # SkillOpt's split: 2:1:7 with its seed 42 (108 test tasks) smoke: [algotune-aes-gcm-encryption, algotune-count-connected-components] # rows #0 (splits/smoke.txt) tasks_dir: tasks # the 154 task directories (prepare's link, outside the checkout) -web_search: false # the CLI's web search off pass_reward: null # a speedup is a scale: no trial "passes" agent_timeout_sec: null # each task's own limit (3,600 s for every task) environment: @@ -30,5 +30,5 @@ verifier: verifier_timeout: agent # a verifier that ran out of its time ran out on the agent's solver (too slow # to time): the agent's result, not run again docker_cleanup: images_keep # Harbor removes the image it built -disk_gate_gb: 25 # no trial starts with less than this free on Docker's data root +disk_gate_gb: 25 # no environment starts with less than this free on Docker's data root disk_gate_wait_s: 21600 diff --git a/examples/algotune/utils/judge.py b/examples/algotune/utils/judge.py index 7cebd070..d30eeafd 100644 --- a/examples/algotune/utils/judge.py +++ b/examples/algotune/utils/judge.py @@ -1,4 +1,5 @@ -"""AlgoTune's judge: the Harbor judge's verdict (hermit.harbor.judge), with a trial scored 1 - 1/speedup. +"""AlgoTune's judge: the verdict of Harbor's verifier in the task's environment (hermit.task.verify.environment), with +a trial scored 1 - 1/speedup. The verifier's reward is the trial's speedup: the reference solver's time over the agent's when every output is valid and the solver is faster, else 1.0 (AlgoTune's floor); 0 when the verifier wrote none (it crashed before the timing @@ -6,8 +7,8 @@ speedups. The score of a trial here is q = 1 - 1/speedup (0 for no speedup), so that the mean m of the scores, the figure every mean of Hermit's takes (the Trainer's, the Evolvers', `hermit eval`'s), is 1 - 1/H: H = 1 / (1 - m), and a method that raises the mean raises AlgoTune's own score. The speedup is kept beside it (`speedup`). A trial that is -not the agent's result (infra) gets no score: the Harbor judge refuses it (`Unscored`), so the run records a failed -run, scored 0 in `hermit eval`'s mean. +not the agent's result (its environment failed) gets no score: the environment judge refuses it (`Unscored`), so the +run records a failed run, scored 0 in `hermit eval`'s mean. judge: path: ../utils/judge.py @@ -20,7 +21,8 @@ import sys from pathlib import Path -from hermit.harbor import judge as harbor, report +from hermit.harbor import report +from hermit.task.verify import environment def speedup_score(speedup: float | None) -> float: @@ -29,10 +31,10 @@ def speedup_score(speedup: float | None) -> float: def score(task: dict, output) -> dict: - record = harbor.score(task, output) # an infra trial: Unscored, raised before any reward is read - out = output.output if isinstance(output.output, dict) else {} - raw = out.get("reward") + verdict = environment(task, output) # an infra trial: Unscored, raised before any reward is read + raw = (output.extra or {})["verified"].get("reward") speedup = float(raw) if raw is not None else None + record = {"score": verdict.score, "feedback": verdict.feedback, **verdict.extra} record["speedup"] = speedup record["score"] = speedup_score(speedup) if speedup is not None: diff --git a/examples/algotune/utils/tests/test_judge.py b/examples/algotune/utils/tests/test_judge.py index 0462fb98..b9fbcefc 100644 --- a/examples/algotune/utils/tests/test_judge.py +++ b/examples/algotune/utils/tests/test_judge.py @@ -1,6 +1,7 @@ """AlgoTune's judge (utils/judge.py): a trial's score is 1 - 1/speedup, so the mean m of the scores is 1 - 1/H for the -harmonic mean H of the speedups, AlgoTune's own score of a set of tasks. The trials are the runner's outputs as the -Harbor judge reads them (hermit.harbor.judge): the verifier's reward is the speedup, with no pass reward.""" +harmonic mean H of the speedups, AlgoTune's own score of a set of tasks. The trials are outputs with the verdict of +Harbor's verifier in the task's environment (hermit.task.verify.environment): the verifier's reward is the speedup, +with no pass reward.""" from __future__ import annotations import importlib.util @@ -21,10 +22,11 @@ def judge(): return module -def trial(status="ok", reward=None, log=None): - """A Harbor trial's output as the runner hands it to the judge, on a benchmark with no pass reward.""" - out = {"status": status, "reward": reward, "rewards": None if reward is None else {"reward": reward}} - return SimpleNamespace(output=out, log=log, extra={"judge": {"pass_reward": None}}) +def trial(status="ok", reward=None, feedback="", timed_out=False): + """An output as the run hands it to the judge, the environment's verdict in it, with no pass reward.""" + verified = {"status": status, "reward": reward, "rewards": None if reward is None else {"reward": reward}, + "passed": None, "feedback": feedback, "failure_cause": None if status == "ok" else "verifier_failed"} + return SimpleNamespace(output=None, log=None, extra={"verified": verified, "timed_out": timed_out}) @pytest.mark.parametrize("speedup, score", [(None, 0.0), (0.0, 0.0), (0.5, 0.0), (1.0, 0.0), (2.0, 0.5), @@ -33,10 +35,8 @@ def test_a_speedup_scores_one_less_its_inverse(judge, speedup, score): assert judge.speedup_score(speedup) == pytest.approx(score) -def test_a_trial_keeps_its_speedup_beside_the_score(judge, tmp_path): - (tmp_path / "verifier").mkdir() - (tmp_path / "verifier" / "test-stdout.txt").write_text("100 instances valid; speedup 2.5\n") - record = judge.score({}, trial(reward=2.5, log=tmp_path)) +def test_a_trial_keeps_its_speedup_beside_the_score(judge): + record = judge.score({}, trial(reward=2.5, feedback="100 instances valid; speedup 2.5\n")) assert record["speedup"] == 2.5 and record["score"] == pytest.approx(0.6) assert record["passed"] is None # a speedup has no pass assert record["feedback"].startswith("speedup 2.5x (score 0.600): reward 2.5") @@ -46,19 +46,19 @@ def test_a_trial_keeps_its_speedup_beside_the_score(judge, tmp_path): def test_no_speedup_and_a_timed_out_agent_score_their_reward(judge): floor = judge.score({}, trial(reward=1.0)) # invalid outputs, or not faster: AlgoTune's 1x assert floor["speedup"] == 1.0 and floor["score"] == 0.0 - late = judge.score({}, trial(status="timeout", reward=3.0)) # the verifier still timed the agent's solver + late = judge.score({}, trial(reward=3.0, timed_out=True)) # the verifier still timed the agent's solver assert late["speedup"] == 3.0 and late["score"] == pytest.approx(2 / 3) def test_an_infra_trial_gets_no_score(judge): - from hermit.harbor.judge import Unscored + from hermit.task.verify import Unscored with pytest.raises(Unscored): # the run records a failed run, never a speedup's 0 judge.score({}, trial(status="infra", reward=5.0)) def infra_row(tid: str) -> dict: """A trial that is not the agent's result as `hermit eval` writes its row: the judge refused it.""" - return {"id": tid, "status": "judge_error", "output": {"status": "infra", "failure_cause": "host_failed"}, + return {"id": tid, "status": "judge_error", "error": "Unscored: the task's environment failed", "judge": {"score": 0.0, "feedback": "the judge failed: Unscored: ..."}} diff --git a/examples/aspire_robosuite/README.md b/examples/aspire_robosuite/README.md index a5b96101..b52dfe34 100644 --- a/examples/aspire_robosuite/README.md +++ b/examples/aspire_robosuite/README.md @@ -19,9 +19,10 @@ only if a fresh verifier re-solves the task from the library and the proposals a | `utils/agent/sim.py` | The robot execution engine from the harness's side (the held-out run) and the trace digest | | `utils/agent/prompts/` | The pages that map ASPIRE's fix-loop, verifier and evolutionary-search templates onto the workspace (and the direct actor's prompts) | | `utils/agent/agent.py` | The structured direct-chat actor of `aspire/run_direct.yaml` (a comparison, not the method) | -| `utils/prepare.py` | The seven Robosuite tasks with their seeds; the verifier's rows; the reruns with the final library | +| `utils/benchmark/prepare.py` | The seven Robosuite tasks with their seeds, as task directories in Harbor's format (`tasks/`, each read as its row through `utils/agent/aspire_fields.py`, the configs' `data.view`); the verifier's tasks; the reruns with the final library; `splits/` | | `utils/seg_service/` | A SAM3-wire-compatible segmentation service from ungated models (optional) | -| `utils/judge.py` | Stage 2 on the host: the selected program on the held-out seeds (a verifier's on its own), its success rate | +| `utils/benchmark/judge.py` | Stage 2 on the host: the selected program on the held-out seeds (a verifier's on its own), its success rate | +| `utils/benchmark/task-info/` | What a task and its record are, for the learner | | `aspire/` | `run.yaml` (ASPIRE's seeds), `smoke.yaml`, the `*_direct.yaml` comparison, `launch.sh` | ## Setup diff --git a/examples/aspire_robosuite/aspire/launch.sh b/examples/aspire_robosuite/aspire/launch.sh index fdd9f08d..15a70d2c 100755 --- a/examples/aspire_robosuite/aspire/launch.sh +++ b/examples/aspire_robosuite/aspire/launch.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash # Train, or evaluate, the ASPIRE method on Robosuite. Run it on the host that serves ASPIRE's providers. # (Another benchmark of ASPIRE's harness starts it from its own folder, as ASPIRE_EXAMPLE: its configs, in the -# method folder ASPIRE_METHOD (default aspire), its utils/prepare.py and task rows, its runs; ../../libero/utils/launch.sh.) +# method folder ASPIRE_METHOD (default aspire), its utils/benchmark/prepare.py and tasks, its runs; ../../libero/utils/launch.sh.) # # ASPIRE_ROOT=/path/to/ASPIRE ./launch.sh # train run.yaml # ASPIRE_CONFIG=smoke.yaml ./launch.sh # train the smoke (its seeds: ASPIRE_SEEDS=smoke) @@ -12,7 +12,7 @@ # ASPIRE_OUT=/big/disk/aspire ./launch.sh # the runs, the sessions, Stage 2 (default ../runs) # ASPIRE_EVAL_SEEDS=1-50 ./launch.sh # held-out seeds other than the preset's # -# The launcher writes the seeds into the task rows (prepare.py; ASPIRE_SEEDS=smoke for the smoke's), starts the +# The launcher writes the seeds into the tasks (prepare.py; ASPIRE_SEEDS=smoke for the smoke's), starts the # simulator service the actors reach through bin/sim (utils/agent/sim_service.py, on ASPIRE_SIM_PORT, default 8120; # stopped when the run ends), and runs hermit. ASPIRE_SIM_ENV names a YAML mapping of environment variables for the # simulator, e.g. {MUJOCO_GL: egl, PYOPENGL_PLATFORM: egl, EGL_PLATFORM: device, MUJOCO_EGL_DEVICE_ID: "0"} @@ -22,7 +22,7 @@ set -euo pipefail HERE="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" : "${ASPIRE_ROOT:?set ASPIRE_ROOT to an ASPIRE checkout with .venv-robosuite}" -EXAMPLE="$(cd "${ASPIRE_EXAMPLE:-$HERE/..}" && pwd)" # the folder with / and utils/prepare.py +EXAMPLE="$(cd "${ASPIRE_EXAMPLE:-$HERE/..}" && pwd)" # the folder with / and utils/benchmark/ METHOD="${ASPIRE_METHOD:-aspire}" # the folder of the configs # Absolute, from where the launcher was called: the service, the judge and the sandboxed workers each start elsewhere ASPIRE_ROOT="$(cd "$ASPIRE_ROOT" && pwd)" @@ -51,17 +51,17 @@ cd "$EXAMPLE/$METHOD" # The model key, for a config whose models are Fireworks' (one on the machine's ChatGPT login needs none) ! grep -q "ref: fireworks/" "$CONFIG" || : "${FIREWORKS_API_KEY:?set FIREWORKS_API_KEY}" -# The seeds, in the task rows: the simulator service and the judge read them there. The example's prepare.py holds -# its benchmark's (ASPIRE's defaults and the smoke's). +# The seeds, in the tasks (task directories, each read as its row): the simulator service and the judge read them +# there. The example's utils/benchmark/prepare.py holds its benchmark's (ASPIRE's defaults and the smoke's). PRESET="${ASPIRE_SEEDS:-$([[ "${CONFIG##*/}" == smoke* ]] && echo smoke || echo aspire)}" [[ "$PRESET" == smoke || "$PRESET" == aspire ]] || { echo "ASPIRE_SEEDS is smoke or aspire" >&2; exit 2; } SEEDS=(--preset "$PRESET") [[ -z "${ASPIRE_EVAL_SEEDS:-}" ]] || SEEDS+=(--eval "$ASPIRE_EVAL_SEEDS") -(cd ../utils && "$HERMIT" prepare.py "${SEEDS[@]}") +(cd ../utils/benchmark && "$HERMIT" prepare.py "${SEEDS[@]}") # ASPIRE's docs as the actors may read them, outside its checkout: the task sandbox is shown these, not the # checkout's pages, which name ASPIRE's worked programs per task. export ASPIRE_DOCS="$EXAMPLE/utils/aspire-docs" -"$HERMIT" "$HERE/../utils/agent/stage_docs.py" --aspire-root "$ASPIRE_ROOT" --tasks ../utils/tasks.jsonl \ +"$HERMIT" "$HERE/../utils/agent/stage_docs.py" --aspire-root "$ASPIRE_ROOT" --tasks ../utils/benchmark/tasks \ --out "$ASPIRE_DOCS" # Where the runs go (ASPIRE_OUT): the trainer's records, the actors' sessions (their simulator runs' records and @@ -79,7 +79,7 @@ export ASPIRE_ROOT ASPIRE_OUT="$OUT" ASPIRE_SIM_ENV="$SIM_ENV_JSON" ASPIRE_SIM_G [[ -n "${ASPIRE_SIM_GPU:-}" ]] || echo "note: ASPIRE_SIM_GPU names no GPU: the simulator renders in software" >&2 PORT="${ASPIRE_SIM_PORT:-8120}" NONCE="$$-$RANDOM-$(date +%s%N)" # this launch's service answers /health with it, another run's not -"$HERMIT" "$HERE/../utils/agent/sim_service.py" --port "$PORT" --aspire-root "$ASPIRE_ROOT" --tasks ../utils/tasks.jsonl \ +"$HERMIT" "$HERE/../utils/agent/sim_service.py" --port "$PORT" --aspire-root "$ASPIRE_ROOT" --tasks ../utils/benchmark/tasks \ --workspaces "$OUT/actor-logs" --state "$OUT/sim-service" --sim-env "$SIM_ENV_JSON" --nonce "$NONCE" \ --gpu "${ASPIRE_SIM_GPU:-}" > "$OUT/sim-service.log" 2>&1 & SERVICE=$! diff --git a/examples/aspire_robosuite/aspire/run.yaml b/examples/aspire_robosuite/aspire/run.yaml index 0cfba931..037fee61 100644 --- a/examples/aspire_robosuite/aspire/run.yaml +++ b/examples/aspire_robosuite/aspire/run.yaml @@ -2,13 +2,14 @@ # coordinator chooses (its curriculum); ASPIRE's seeds (development 101-125, unseen-check 126-130, held-out 1-100, # written into the task rows by launch.sh); Codex acting and coordinating, GLM 5.3 Flash. Started by launch.sh. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/all.txt - val: ../utils/splits/verify.txt # the verifier's rows: run only by the coordinator, through run_agent - test: ../utils/splits/rerun.txt # every task again with the final library: ASPIRE's backward transfer + tasks: ../utils/benchmark/tasks + view: ../utils/agent/aspire_fields.py:aspire_task # each task as the row it always was + train: ../utils/benchmark/splits/all.txt + val: ../utils/benchmark/splits/verify.txt # the verifier's rows: run only by the coordinator, through run_agent + test: ../utils/benchmark/splits/rerun.txt # every task again with the final library: ASPIRE's backward transfer step_size: 1 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: base: codex concurrency: 1 @@ -38,7 +39,7 @@ environment: # the task sandbox: the actor's code, ASPIRE - $ASPIRE_ROOT/.claude/skills/evosearch/skills/evosearch-iteration.md env: [ASPIRE_SIM_URL] # the host's simulator service (launch.sh): its 127.0.0.1 port is forwarded judge: - path: ../utils/judge.py # on the host: Stage 2 on the held-out seeds + path: ../utils/benchmark/judge.py # on the host: Stage 2 on the held-out seeds template: aspire evolver: name: aspire diff --git a/examples/aspire_robosuite/aspire/run_direct.yaml b/examples/aspire_robosuite/aspire/run_direct.yaml index 4bbb2e2b..140c3df7 100644 --- a/examples/aspire_robosuite/aspire/run_direct.yaml +++ b/examples/aspire_robosuite/aspire/run_direct.yaml @@ -4,13 +4,14 @@ # every task the coordinator repairs and grows the skill library through the verifier gate. Started by launch.sh, # inside the host that runs ASPIRE's perception, IK and planning services (see ../README.md). data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/all.txt - val: ../utils/splits/verify.txt # the verifier's rows: run only by the coordinator, through run_agent - test: ../utils/splits/rerun.txt # every task again with the final library: ASPIRE's backward transfer + tasks: ../utils/benchmark/tasks + view: ../utils/agent/aspire_fields.py:aspire_task # each task as the row it always was + train: ../utils/benchmark/splits/all.txt + val: ../utils/benchmark/splits/verify.txt # the verifier's rows: run only by the coordinator, through run_agent + test: ../utils/benchmark/splits/rerun.txt # every task again with the final library: ASPIRE's backward transfer step_size: 1 # one task per step: the coordinator acts on each task as it finishes shuffle: false # the curriculum in the split's order -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: name: ../utils/agent/agent.py:AspireActor concurrency: 1 # one simulator sweep at a time on the engine's GPU @@ -24,7 +25,7 @@ agent: evo: {k: 8, rounds: 5, threshold: 0.8, qualify_below: 0.9, parents: 3} workers: 4 judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: aspire evolver: # the coordinator, as in run.yaml name: aspire diff --git a/examples/aspire_robosuite/aspire/smoke.yaml b/examples/aspire_robosuite/aspire/smoke.yaml index ead40237..35cff3a4 100644 --- a/examples/aspire_robosuite/aspire/smoke.yaml +++ b/examples/aspire_robosuite/aspire/smoke.yaml @@ -2,13 +2,14 @@ # the task rows: ASPIRE_SEEDS=smoke); Codex acting (ASPIRE's fix-loop subagent and verifier prompts) and coordinating, # GLM 5.3 Flash. Started by launch.sh: ASPIRE_CONFIG=smoke.yaml ./launch.sh data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/smoke.txt - val: ../utils/splits/verify.txt # the verifier's rows: run only by the coordinator, through run_agent - test: ../utils/splits/smoke_rerun.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/aspire_fields.py:aspire_task # each task as the row it always was + train: ../utils/benchmark/splits/smoke.txt + val: ../utils/benchmark/splits/verify.txt # the verifier's rows: run only by the coordinator, through run_agent + test: ../utils/benchmark/splits/smoke_rerun.txt step_size: 1 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: base: codex concurrency: 1 @@ -38,7 +39,7 @@ environment: # the task sandbox: the actor's code, ASPIRE - $ASPIRE_ROOT/.claude/skills/evosearch/skills/evosearch-iteration.md env: [ASPIRE_SIM_URL] # the host's simulator service (launch.sh): its 127.0.0.1 port is forwarded judge: - path: ../utils/judge.py # on the host: Stage 2 on the held-out seeds + path: ../utils/benchmark/judge.py # on the host: Stage 2 on the held-out seeds template: aspire evolver: name: aspire diff --git a/examples/aspire_robosuite/aspire/smoke_direct.yaml b/examples/aspire_robosuite/aspire/smoke_direct.yaml index ed5a8bfb..8d6b7b61 100644 --- a/examples/aspire_robosuite/aspire/smoke_direct.yaml +++ b/examples/aspire_robosuite/aspire/smoke_direct.yaml @@ -2,13 +2,14 @@ # Smoke: two tasks, three development seeds, three held-out seeds, two versions and no search; the verifier's own # rows (5 seeds, 2 attempts). Started by launch.sh: ASPIRE_CONFIG=smoke_direct.yaml ./launch.sh data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/smoke.txt - val: ../utils/splits/verify.txt - test: ../utils/splits/smoke_rerun.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/aspire_fields.py:aspire_task # each task as the row it always was + train: ../utils/benchmark/splits/smoke.txt + val: ../utils/benchmark/splits/verify.txt + test: ../utils/benchmark/splits/smoke_rerun.txt step_size: 1 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: name: ../utils/agent/agent.py:AspireActor concurrency: 1 @@ -21,7 +22,7 @@ agent: search: debug workers: 3 judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: aspire evolver: # the coordinator, as in run.yaml name: aspire diff --git a/examples/aspire_robosuite/utils/.gitignore b/examples/aspire_robosuite/utils/.gitignore index bc46a5ce..d006f309 100644 --- a/examples/aspire_robosuite/utils/.gitignore +++ b/examples/aspire_robosuite/utils/.gitignore @@ -1,3 +1,5 @@ +aspire-docs/ +runs/ +# Where utils/prepare.py wrote before utils/benchmark/: kept out of git in a checkout prepared then tasks.jsonl splits/ -aspire-docs/ diff --git a/examples/aspire_robosuite/utils/agent/agent.py b/examples/aspire_robosuite/utils/agent/agent.py index cdb3117f..b942cd73 100644 --- a/examples/aspire_robosuite/utils/agent/agent.py +++ b/examples/aspire_robosuite/utils/agent/agent.py @@ -3,7 +3,7 @@ (ASPIRE §2.1, the iterative-debugging loop of NVlabs/ASPIRE .claude/skills/iterative-debugging/subagent-prompt.md); optionally continue with the evolutionary search over the program (ASPIRE §2.3, Algorithm 1; .claude/skills/evosearch/); then write the findings the coordinator distills skills from. The judge runs the -selected program on the held-out seeds (utils/judge.py). +selected program on the held-out seeds (utils/benchmark/judge.py). The adapter is the shared skill library: its context (ADAPTER.md, and the skills it exports) is the "Shared skill library" section of the actor's instructions, the only thing that carries from one task to the next. adapter=None diff --git a/examples/aspire_robosuite/utils/agent/aspire_fields.py b/examples/aspire_robosuite/utils/agent/aspire_fields.py new file mode 100644 index 00000000..09ac8e63 --- /dev/null +++ b/examples/aspire_robosuite/utils/agent/aspire_fields.py @@ -0,0 +1,20 @@ +"""A task of ASPIRE's engine as the actor, the simulator service and the judge read it: {id, (benchmark), suite, +task_id, the seeds, (role, max_versions, search, findings)}, the row each task was in tasks.jsonl. A task directory +(utils/benchmark/prepare.py, and LIBERO's) has every field but its id under task.toml's [metadata], in the row's +order; this puts them back, so the prompts, the service's rules and the records are what they were. LIBERO's tasks +are read so too (their configs' data.view).""" +from __future__ import annotations + +from pathlib import Path + + +def aspire_task(task: dict) -> dict: + if "instruction" not in task: # already a row of that shape + return dict(task) + return {"id": task["id"], **(task.get("metadata") or {})} + + +def task_rows(tasks: Path) -> dict[str, dict]: + """Every task's row by id: from the task directories prepare.py writes, or from a tasks.jsonl.""" + from hermit.task import Benchmark + return {tid: t.given() for tid, t in Benchmark.load(Path(tasks), view=aspire_task).tasks.items()} diff --git a/examples/aspire_robosuite/utils/agent/sim_service.py b/examples/aspire_robosuite/utils/agent/sim_service.py index 732599ef..6b933475 100644 --- a/examples/aspire_robosuite/utils/agent/sim_service.py +++ b/examples/aspire_robosuite/utils/agent/sim_service.py @@ -4,8 +4,9 @@ services; a session reaches it through `bin/sim` (sim_cli.py) on a forwarded 127.0.0.1 port (ASPIRE_SIM_URL in environment.env). The service is the trusted side of the session's rules, and it trusts nothing the session sends but the workspace and its task's id: the seeds, the role and the budget are the task row's (--tasks, read here on -the host), and a rollout's directory (//-, made by the framework outside the -sandbox, which can write only inside it) registers one session, of the task its name carries: +the host, each task as its row: aspire_fields.py), and a rollout's directory (//-, +made by the framework outside the sandbox, which can write only inside it) registers one session, of the task its +name carries: POST /register {workspace, task, max_steps} once per rollout, by the task's `prepare`, before the agent starts (a retry's `prepare`, in the @@ -25,7 +26,7 @@ Every refusal is a 200 with {ok: false, error}. The ledger (registrations, unseen checks, a verifier's programs) is kept in --state, outside every sandbox. Started by launch.sh: - python sim_service.py --port 8120 --aspire-root $ASPIRE_ROOT --tasks utils/tasks.jsonl --workspaces + python sim_service.py --port 8120 --aspire-root $ASPIRE_ROOT --tasks utils/benchmark/tasks --workspaces --state

[--sim-env '{"MUJOCO_GL": "egl", ...}'] [--workers 4] """ from __future__ import annotations @@ -49,6 +50,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parent)) import sim # noqa: E402 from sim import seeds_of # noqa: E402 +from aspire_fields import task_rows # noqa: E402 class Ledger: @@ -361,7 +363,8 @@ def main() -> None: ap.add_argument("--host", default="127.0.0.1") ap.add_argument("--port", type=int, default=8120) ap.add_argument("--aspire-root", type=Path, required=True) - ap.add_argument("--tasks", type=Path, required=True, help="the task rows (utils/tasks.jsonl): their seeds") + ap.add_argument("--tasks", type=Path, required=True, + help="the tasks (utils/benchmark/tasks, or a tasks.jsonl): their seeds") ap.add_argument("--workspaces", type=Path, action="append", required=True, help="a directory whose sessions this service serves (repeatable)") ap.add_argument("--state", type=Path, required=True) @@ -378,7 +381,7 @@ def main() -> None: sys.exit("sim_service: bubblewrap (bwrap) is not on PATH: every simulator run is confined (sim.confine)") gpu_args(sim.gpus_of(a.gpu)) # a GPU the machine lacks raises here a.state.mkdir(parents=True, exist_ok=True) - tasks = {r["id"]: r for r in (json.loads(line) for line in a.tasks.read_text().splitlines() if line.strip())} + tasks = task_rows(a.tasks) service = Service(a.aspire_root.resolve(), tasks, a.workspaces, Ledger(a.state / "ledger.json"), json.loads(a.sim_env), a.workers, run_timeout=a.run_timeout, gpu=sim.gpus_of(a.gpu)) service.nonce = a.nonce diff --git a/examples/aspire_robosuite/utils/agent/stage_docs.py b/examples/aspire_robosuite/utils/agent/stage_docs.py index c6f033d3..4c1ed997 100644 --- a/examples/aspire_robosuite/utils/agent/stage_docs.py +++ b/examples/aspire_robosuite/utils/agent/stage_docs.py @@ -2,27 +2,28 @@ (environment.read, agent.params.aspire_docs), never the checkout's own pages, which name ASPIRE's worked programs per task. launch.sh runs it on the host before a run: - python stage_docs.py --aspire-root $ASPIRE_ROOT --tasks utils/tasks.jsonl --out utils/aspire-docs + python stage_docs.py --aspire-root $ASPIRE_ROOT --tasks utils/benchmark/tasks --out utils/aspire-docs """ from __future__ import annotations import argparse -import json from pathlib import Path import shutil import sys sys.path.insert(0, str(Path(__file__).resolve().parent)) import sim # noqa: E402 +from aspire_fields import task_rows # noqa: E402 def main() -> None: ap = argparse.ArgumentParser() ap.add_argument("--aspire-root", type=Path, required=True) - ap.add_argument("--tasks", type=Path, required=True, help="the task rows: their benchmarks' pages are staged") + ap.add_argument("--tasks", type=Path, required=True, + help="the tasks (utils/benchmark/tasks, or a tasks.jsonl): their benchmarks' pages are staged") ap.add_argument("--out", type=Path, required=True) a = ap.parse_args() - rows = [json.loads(line) for line in a.tasks.read_text().splitlines() if line.strip()] + rows = list(task_rows(a.tasks).values()) pages = {} for benchmark in sorted({str(r.get("benchmark") or "robosuite") for r in rows}): pages.update(sim.workspace_docs(a.aspire_root, benchmark)) diff --git a/examples/aspire_robosuite/utils/agent/task.py b/examples/aspire_robosuite/utils/agent/task.py index 7d7eb1d8..d6c64610 100644 --- a/examples/aspire_robosuite/utils/agent/task.py +++ b/examples/aspire_robosuite/utils/agent/task.py @@ -12,7 +12,7 @@ ASPIRE's checkout comes from agent.params.aspire_root (its prompt templates and snapshot program must be declared readable: environment.read; its docs are read from agent.params.aspire_docs, the staged copies); the service's URL from ASPIRE_SIM_URL (environment.env). A task row carries the -seeds: development, unseen-check and held-out (utils/prepare.py). +seeds: development, unseen-check and held-out (utils/benchmark/prepare.py). """ from __future__ import annotations @@ -44,7 +44,8 @@ def settings(task: dict, params: dict) -> dict: - """The session's assignment, from its task row (the seeds; utils/prepare.py) and agent.params (the checkout).""" + """The session's assignment, from its task row (the seeds; utils/benchmark/prepare.py) and agent.params (the + checkout).""" role = "verifier" if task.get("role") == "verifier" else "actor" # a rerun is an actor with the final library root = params.get("aspire_root") or os.environ.get("ASPIRE_ROOT") if not root: diff --git a/examples/aspire_robosuite/utils/benchmark/.gitignore b/examples/aspire_robosuite/utils/benchmark/.gitignore new file mode 100644 index 00000000..d296b16f --- /dev/null +++ b/examples/aspire_robosuite/utils/benchmark/.gitignore @@ -0,0 +1,2 @@ +tasks/ +splits/ diff --git a/examples/aspire_robosuite/utils/judge.py b/examples/aspire_robosuite/utils/benchmark/judge.py similarity index 98% rename from examples/aspire_robosuite/utils/judge.py rename to examples/aspire_robosuite/utils/benchmark/judge.py index 1429ff0c..5b3c2f41 100644 --- a/examples/aspire_robosuite/utils/judge.py +++ b/examples/aspire_robosuite/utils/benchmark/judge.py @@ -15,7 +15,7 @@ from pathlib import Path import sys -sys.path.insert(0, str(Path(__file__).resolve().parent / "agent")) +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "agent")) import sim # noqa: E402 diff --git a/examples/aspire_robosuite/utils/prepare.py b/examples/aspire_robosuite/utils/benchmark/prepare.py similarity index 67% rename from examples/aspire_robosuite/utils/prepare.py rename to examples/aspire_robosuite/utils/benchmark/prepare.py index d9a5abdf..ae04112a 100644 --- a/examples/aspire_robosuite/utils/prepare.py +++ b/examples/aspire_robosuite/utils/benchmark/prepare.py @@ -1,11 +1,15 @@ -"""Write the Robosuite tasks of ASPIRE's harness (cap_harness/robosuite/registry.py) as hermit tasks: one row per -task with its seeds, which the simulator service and the judge take from the row (ASPIRE's Robosuite defaults: -development 101-125, unseen-check 126-130, held-out 1-100). +"""Write the Robosuite tasks of ASPIRE's harness (cap_harness/robosuite/registry.py) as hermit tasks: one task +directory (Harbor's format) per task with its seeds, which the simulator service and the judge take from the task's +row (ASPIRE's Robosuite defaults: development 101-125, unseen-check 126-130, held-out 1-100). python prepare.py # ASPIRE's seeds python prepare.py --preset smoke # the smoke's: three development seeds, one unseen, three held-out python prepare.py --dev 101-103 --unseen 126 --eval 1-3 # any others - # tasks.jsonl, splits/{all,smoke,single_arm,verify,rerun,smoke_rerun,single_arm_rerun}.txt + # tasks//, splits/{all,smoke,single_arm,verify,rerun,smoke_rerun,single_arm_rerun}.txt + +A task directory's task.toml holds the row's fields but its id under [metadata], in the row's order, and asks for no +network (the actor reaches only its model and the simulator service's forwarded port); it has no tests/: the judge +(judge.py) runs on the host, where the simulator is. utils/agent/aspire_fields.py reads each back as its row. The verifier rows (splits/verify.txt) are the Evolver's: a run gives them as `data.val`, so only `run_agent` runs them. @@ -16,12 +20,14 @@ from __future__ import annotations import argparse -import json from pathlib import Path +import shutil import sys +from hermit.task.directory import write_task as write_task_dir + HERE = Path(__file__).resolve().parent -sys.path.insert(0, str(HERE / "agent")) +sys.path.insert(0, str(HERE.parent / "agent")) from sim import seeds_of # noqa: E402 TASKS = [ # suite, what it is (task-info only; the actor reads the language from get_task_context()) ("cube_lifting", "Lift"), @@ -61,9 +67,25 @@ def seeds_from(presets: dict, argv=None) -> tuple[dict, str]: chosen["verify"] +def instruction(row: dict) -> str: + """What a task asks, in a line (the actor's prompt is utils/agent/task.py's, built from the row).""" + what = f"{row.get('benchmark') or 'robosuite'} suite `{row['suite']}`, task id {row['task_id']}" + if row.get("role") == "verifier": + return f"Verify the proposals for {what}: solve it again from the library and the proposals alone." + if row.get("role") == "rerun": + return f"Solve {what} again with the final library." + return f"Write the program that solves {what} in ASPIRE's Code-as-Policy harness." + + +def write_task(tasks: Path, row: dict) -> Path: + """One row as a task directory: its fields but the id under [metadata], in the row's order; no network.""" + return write_task_dir(tasks / row["id"], instruction(row), metadata={k: v for k, v in row.items() if k != "id"}, + environment={"network_mode": "no-network"}) + + def write(here: Path, tasks: list[dict], seeds: dict, verify: str, splits: dict[str, list[str]]) -> None: - """tasks.jsonl and splits/ under `here`: each task (`id`, `suite`, `task_id`, and `benchmark` when it is not - Robosuite) with its seeds, its verifier row and its rerun row; splits/.txt for each of `splits`, with + """tasks/ and splits/ under `here`: each task (`id`, `suite`, `task_id`, and `benchmark` when it is not + Robosuite) with its seeds, its verifier task and its rerun task; splits/.txt for each of `splits`, with _rerun.txt beside it (rerun.txt for `all`), and verify.txt.""" rows = [{**t, **seeds} for t in tasks] # The verifier of each task (ASPIRE verifier-prompt.md): a fresh actor that re-solves the task from the library @@ -73,14 +95,17 @@ def write(here: Path, tasks: list[dict], seeds: dict, verify: str, splits: dict[ # The rerun of each task with the final library (ASPIRE §2.4, backward transfer): the same task, the same seeds # and a fresh actor; its own id so that the test split does not share ids with the training split. reruns = [{**t, "id": f"rerun-{t['id']}", "role": "rerun", "findings": False, **seeds} for t in tasks] - (here / "tasks.jsonl").write_text("".join(json.dumps(r) + "\n" for r in rows + verifiers + reruns)) + if (here / "tasks").exists(): # this script's own output: rewritten whole + shutil.rmtree(here / "tasks") + for row in rows + verifiers + reruns: + write_task(here / "tasks", row) out = here / "splits" out.mkdir(exist_ok=True) for name, ids in splits.items(): (out / f"{name}.txt").write_text("".join(f"{i}\n" for i in ids)) (out / ("rerun.txt" if name == "all" else f"{name}_rerun.txt")).write_text("".join(f"rerun-{i}\n" for i in ids)) (out / "verify.txt").write_text("".join(f"verify-{t['id']}\n" for t in tasks)) - print(f"{len(rows)} tasks -> {here / 'tasks.jsonl'}") + print(f"{len(rows)} tasks -> {here / 'tasks'}") def main() -> None: diff --git a/examples/aspire_robosuite/utils/task-info/README.md b/examples/aspire_robosuite/utils/benchmark/task-info/README.md similarity index 100% rename from examples/aspire_robosuite/utils/task-info/README.md rename to examples/aspire_robosuite/utils/benchmark/task-info/README.md diff --git a/examples/claude_subscription/README.md b/examples/claude_subscription/README.md index 52f7b6c9..0bdff66f 100644 --- a/examples/claude_subscription/README.md +++ b/examples/claude_subscription/README.md @@ -11,6 +11,11 @@ integration smoke test, and it promises no gain. | `run_react.yaml` | `react`, whose model is the endpoint (`agent.model: {cli: claude}`) | ACE, the same | A pipeline that takes a model instead of a workspace | | `run_workspace_learner.yaml` | Claude Code in a workspace | The `claude` Evolver edits the adapter | The learner's Base Agent path: tools, files, sessions | +The nine tasks are task directories in [Harbor's format](../../docs/framework/task.md#task-directories) +(`utils/benchmark/tasks//`: the review to classify as `instruction.md`), with their splits, the judge (`judge.py`, +which keeps the labels) and the learner's `task-info/` beside them in `utils/benchmark/`. Each config reads a task as +the row `{id, prompt}` it always was (`data.view: utils/benchmark/sentiment_fields.py:sentiment_task`). + The [guide](../../docs/guides/claude-subscription.md) has the sign-in, the boundaries, the plan, what a failed session becomes, the model contract and which methods go which way. diff --git a/examples/claude_subscription/run.yaml b/examples/claude_subscription/run.yaml index e6772669..73a5d771 100644 --- a/examples/claude_subscription/run.yaml +++ b/examples/claude_subscription/run.yaml @@ -13,15 +13,16 @@ # the task sandbox, each chat call's in one of its own. Sessions on the login share its plan: the files under # ~/.cache/hermit/claude-login/plan/ pause them, cap the plan's usage and the sessions at once (hermit.agent.base.login). data: - tasks: tasks.jsonl - train: train.txt - val: val.txt - test: test.txt + tasks: utils/benchmark/tasks # one task directory a review (Harbor's format) + view: utils/benchmark/sentiment_fields.py:sentiment_task # each read as the row {id, prompt} it always was + train: utils/benchmark/splits/train.txt + val: utils/benchmark/splits/val.txt + test: utils/benchmark/splits/test.txt step_size: 3 shuffle: false -task_info: task-info +task_info: utils/benchmark/task-info judge: - path: judge.py + path: utils/benchmark/judge.py template: ace@1.0 agent: base: claude # Layer 2: the Claude Code CLI, in a workspace diff --git a/examples/claude_subscription/run_react.yaml b/examples/claude_subscription/run_react.yaml index 2bb99646..5ab2bb15 100644 --- a/examples/claude_subscription/run_react.yaml +++ b/examples/claude_subscription/run_react.yaml @@ -4,15 +4,16 @@ # endpoint's local URL. A pipeline fits this route when it needs text back: `react` here has no tools to offer; a # request with tools is refused (docs/guides/claude-subscription.md). data: - tasks: tasks.jsonl - train: train.txt - val: val.txt - test: test.txt + tasks: utils/benchmark/tasks # one task directory a review (Harbor's format) + view: utils/benchmark/sentiment_fields.py:sentiment_task # each read as the row {id, prompt} it always was + train: utils/benchmark/splits/train.txt + val: utils/benchmark/splits/val.txt + test: utils/benchmark/splits/test.txt step_size: 3 shuffle: false -task_info: task-info +task_info: utils/benchmark/task-info judge: - path: judge.py + path: utils/benchmark/judge.py template: ace@1.0 agent: base: react diff --git a/examples/claude_subscription/run_workspace_learner.yaml b/examples/claude_subscription/run_workspace_learner.yaml index 3b4cb837..737610e4 100644 --- a/examples/claude_subscription/run_workspace_learner.yaml +++ b/examples/claude_subscription/run_workspace_learner.yaml @@ -2,15 +2,16 @@ # the learning material and edits the adapter. It is the Base Agent's own route (tools, files, session continuation), in # the evolver sandbox (/w), which lets the session reach its model and nothing else. data: - tasks: tasks.jsonl - train: train.txt - val: val.txt - test: test.txt + tasks: utils/benchmark/tasks # one task directory a review (Harbor's format) + view: utils/benchmark/sentiment_fields.py:sentiment_task # each read as the row {id, prompt} it always was + train: utils/benchmark/splits/train.txt + val: utils/benchmark/splits/val.txt + test: utils/benchmark/splits/test.txt step_size: 3 shuffle: false -task_info: task-info +task_info: utils/benchmark/task-info judge: - path: judge.py + path: utils/benchmark/judge.py template: prompt-only@1.0 agent: base: claude diff --git a/examples/claude_subscription/tasks.jsonl b/examples/claude_subscription/tasks.jsonl deleted file mode 100644 index 5fcda6ba..00000000 --- a/examples/claude_subscription/tasks.jsonl +++ /dev/null @@ -1,9 +0,0 @@ -{"id":"train-001","prompt":"Classify the sentiment of this review: The service was wonderful and I loved the meal."} -{"id":"train-002","prompt":"Classify the sentiment of this review: The product broke immediately and support was awful."} -{"id":"train-003","prompt":"Classify the sentiment of this review: The package contains three cables."} -{"id":"val-001","prompt":"Classify the sentiment of this review: The staff were friendly and the room was spotless."} -{"id":"val-002","prompt":"Classify the sentiment of this review: The delivery was late and the box arrived crushed."} -{"id":"val-003","prompt":"Classify the sentiment of this review: The manual is printed in English and French."} -{"id":"test-001","prompt":"Classify the sentiment of this review: This was a delightful experience."} -{"id":"test-002","prompt":"Classify the sentiment of this review: I regret buying this disappointing device."} -{"id":"test-003","prompt":"Classify the sentiment of this review: The shop opens at nine."} diff --git a/examples/claude_subscription/judge.py b/examples/claude_subscription/utils/benchmark/judge.py similarity index 100% rename from examples/claude_subscription/judge.py rename to examples/claude_subscription/utils/benchmark/judge.py diff --git a/examples/claude_subscription/utils/benchmark/sentiment_fields.py b/examples/claude_subscription/utils/benchmark/sentiment_fields.py new file mode 100644 index 00000000..9df73e0c --- /dev/null +++ b/examples/claude_subscription/utils/benchmark/sentiment_fields.py @@ -0,0 +1,8 @@ +"""A sentiment task as the acting agent reads it: {id, prompt}, the row each task was before it became a task directory +(its instruction is the prompt), so the prompt and the records stay what they were.""" + + +def sentiment_task(task: dict) -> dict: + if "instruction" not in task: # already a row of that shape + return dict(task) + return {"id": task["id"], "prompt": task["instruction"]} diff --git a/examples/claude_subscription/test.txt b/examples/claude_subscription/utils/benchmark/splits/test.txt similarity index 100% rename from examples/claude_subscription/test.txt rename to examples/claude_subscription/utils/benchmark/splits/test.txt diff --git a/examples/claude_subscription/train.txt b/examples/claude_subscription/utils/benchmark/splits/train.txt similarity index 100% rename from examples/claude_subscription/train.txt rename to examples/claude_subscription/utils/benchmark/splits/train.txt diff --git a/examples/claude_subscription/val.txt b/examples/claude_subscription/utils/benchmark/splits/val.txt similarity index 100% rename from examples/claude_subscription/val.txt rename to examples/claude_subscription/utils/benchmark/splits/val.txt diff --git a/examples/claude_subscription/task-info/README.md b/examples/claude_subscription/utils/benchmark/task-info/README.md similarity index 100% rename from examples/claude_subscription/task-info/README.md rename to examples/claude_subscription/utils/benchmark/task-info/README.md diff --git a/examples/claude_subscription/utils/benchmark/tasks/test-001/instruction.md b/examples/claude_subscription/utils/benchmark/tasks/test-001/instruction.md new file mode 100644 index 00000000..6d192c70 --- /dev/null +++ b/examples/claude_subscription/utils/benchmark/tasks/test-001/instruction.md @@ -0,0 +1 @@ +Classify the sentiment of this review: This was a delightful experience. diff --git a/examples/claude_subscription/utils/benchmark/tasks/test-001/task.toml b/examples/claude_subscription/utils/benchmark/tasks/test-001/task.toml new file mode 100644 index 00000000..9f983731 --- /dev/null +++ b/examples/claude_subscription/utils/benchmark/tasks/test-001/task.toml @@ -0,0 +1,4 @@ +version = "1.0" + +[environment] +network_mode = "no-network" diff --git a/examples/claude_subscription/utils/benchmark/tasks/test-002/instruction.md b/examples/claude_subscription/utils/benchmark/tasks/test-002/instruction.md new file mode 100644 index 00000000..3c8cd365 --- /dev/null +++ b/examples/claude_subscription/utils/benchmark/tasks/test-002/instruction.md @@ -0,0 +1 @@ +Classify the sentiment of this review: I regret buying this disappointing device. diff --git a/examples/claude_subscription/utils/benchmark/tasks/test-002/task.toml b/examples/claude_subscription/utils/benchmark/tasks/test-002/task.toml new file mode 100644 index 00000000..9f983731 --- /dev/null +++ b/examples/claude_subscription/utils/benchmark/tasks/test-002/task.toml @@ -0,0 +1,4 @@ +version = "1.0" + +[environment] +network_mode = "no-network" diff --git a/examples/claude_subscription/utils/benchmark/tasks/test-003/instruction.md b/examples/claude_subscription/utils/benchmark/tasks/test-003/instruction.md new file mode 100644 index 00000000..4e5b003d --- /dev/null +++ b/examples/claude_subscription/utils/benchmark/tasks/test-003/instruction.md @@ -0,0 +1 @@ +Classify the sentiment of this review: The shop opens at nine. diff --git a/examples/claude_subscription/utils/benchmark/tasks/test-003/task.toml b/examples/claude_subscription/utils/benchmark/tasks/test-003/task.toml new file mode 100644 index 00000000..9f983731 --- /dev/null +++ b/examples/claude_subscription/utils/benchmark/tasks/test-003/task.toml @@ -0,0 +1,4 @@ +version = "1.0" + +[environment] +network_mode = "no-network" diff --git a/examples/claude_subscription/utils/benchmark/tasks/train-001/instruction.md b/examples/claude_subscription/utils/benchmark/tasks/train-001/instruction.md new file mode 100644 index 00000000..ca450c59 --- /dev/null +++ b/examples/claude_subscription/utils/benchmark/tasks/train-001/instruction.md @@ -0,0 +1 @@ +Classify the sentiment of this review: The service was wonderful and I loved the meal. diff --git a/examples/claude_subscription/utils/benchmark/tasks/train-001/task.toml b/examples/claude_subscription/utils/benchmark/tasks/train-001/task.toml new file mode 100644 index 00000000..9f983731 --- /dev/null +++ b/examples/claude_subscription/utils/benchmark/tasks/train-001/task.toml @@ -0,0 +1,4 @@ +version = "1.0" + +[environment] +network_mode = "no-network" diff --git a/examples/claude_subscription/utils/benchmark/tasks/train-002/instruction.md b/examples/claude_subscription/utils/benchmark/tasks/train-002/instruction.md new file mode 100644 index 00000000..3e15c1e0 --- /dev/null +++ b/examples/claude_subscription/utils/benchmark/tasks/train-002/instruction.md @@ -0,0 +1 @@ +Classify the sentiment of this review: The product broke immediately and support was awful. diff --git a/examples/claude_subscription/utils/benchmark/tasks/train-002/task.toml b/examples/claude_subscription/utils/benchmark/tasks/train-002/task.toml new file mode 100644 index 00000000..9f983731 --- /dev/null +++ b/examples/claude_subscription/utils/benchmark/tasks/train-002/task.toml @@ -0,0 +1,4 @@ +version = "1.0" + +[environment] +network_mode = "no-network" diff --git a/examples/claude_subscription/utils/benchmark/tasks/train-003/instruction.md b/examples/claude_subscription/utils/benchmark/tasks/train-003/instruction.md new file mode 100644 index 00000000..fdf12f9c --- /dev/null +++ b/examples/claude_subscription/utils/benchmark/tasks/train-003/instruction.md @@ -0,0 +1 @@ +Classify the sentiment of this review: The package contains three cables. diff --git a/examples/claude_subscription/utils/benchmark/tasks/train-003/task.toml b/examples/claude_subscription/utils/benchmark/tasks/train-003/task.toml new file mode 100644 index 00000000..9f983731 --- /dev/null +++ b/examples/claude_subscription/utils/benchmark/tasks/train-003/task.toml @@ -0,0 +1,4 @@ +version = "1.0" + +[environment] +network_mode = "no-network" diff --git a/examples/claude_subscription/utils/benchmark/tasks/val-001/instruction.md b/examples/claude_subscription/utils/benchmark/tasks/val-001/instruction.md new file mode 100644 index 00000000..d13b8ebe --- /dev/null +++ b/examples/claude_subscription/utils/benchmark/tasks/val-001/instruction.md @@ -0,0 +1 @@ +Classify the sentiment of this review: The staff were friendly and the room was spotless. diff --git a/examples/claude_subscription/utils/benchmark/tasks/val-001/task.toml b/examples/claude_subscription/utils/benchmark/tasks/val-001/task.toml new file mode 100644 index 00000000..9f983731 --- /dev/null +++ b/examples/claude_subscription/utils/benchmark/tasks/val-001/task.toml @@ -0,0 +1,4 @@ +version = "1.0" + +[environment] +network_mode = "no-network" diff --git a/examples/claude_subscription/utils/benchmark/tasks/val-002/instruction.md b/examples/claude_subscription/utils/benchmark/tasks/val-002/instruction.md new file mode 100644 index 00000000..7a30af50 --- /dev/null +++ b/examples/claude_subscription/utils/benchmark/tasks/val-002/instruction.md @@ -0,0 +1 @@ +Classify the sentiment of this review: The delivery was late and the box arrived crushed. diff --git a/examples/claude_subscription/utils/benchmark/tasks/val-002/task.toml b/examples/claude_subscription/utils/benchmark/tasks/val-002/task.toml new file mode 100644 index 00000000..9f983731 --- /dev/null +++ b/examples/claude_subscription/utils/benchmark/tasks/val-002/task.toml @@ -0,0 +1,4 @@ +version = "1.0" + +[environment] +network_mode = "no-network" diff --git a/examples/claude_subscription/utils/benchmark/tasks/val-003/instruction.md b/examples/claude_subscription/utils/benchmark/tasks/val-003/instruction.md new file mode 100644 index 00000000..27d9fedd --- /dev/null +++ b/examples/claude_subscription/utils/benchmark/tasks/val-003/instruction.md @@ -0,0 +1 @@ +Classify the sentiment of this review: The manual is printed in English and French. diff --git a/examples/claude_subscription/utils/benchmark/tasks/val-003/task.toml b/examples/claude_subscription/utils/benchmark/tasks/val-003/task.toml new file mode 100644 index 00000000..9f983731 --- /dev/null +++ b/examples/claude_subscription/utils/benchmark/tasks/val-003/task.toml @@ -0,0 +1,4 @@ +version = "1.0" + +[environment] +network_mode = "no-network" diff --git a/examples/claude_subscription/utils/tests/test_sentiment_task_directories.py b/examples/claude_subscription/utils/tests/test_sentiment_task_directories.py new file mode 100644 index 00000000..3feeb737 --- /dev/null +++ b/examples/claude_subscription/utils/tests/test_sentiment_task_directories.py @@ -0,0 +1,37 @@ +"""The sentiment tasks as task directories (utils/benchmark/tasks/) read back, through each config's data.view, as +exactly the rows {id, prompt} they were in tasks.jsonl, in the same splits: the prompts and the records stay +byte for byte what they were.""" +import json +from pathlib import Path + +import pytest + +from hermit.config import Config +from hermit.task import Benchmark + +HERE = Path(__file__).resolve().parents[2] +PREFIX = "Classify the sentiment of this review: " +REVIEWS = { + "train-001": "The service was wonderful and I loved the meal.", + "train-002": "The product broke immediately and support was awful.", + "train-003": "The package contains three cables.", + "val-001": "The staff were friendly and the room was spotless.", + "val-002": "The delivery was late and the box arrived crushed.", + "val-003": "The manual is printed in English and French.", + "test-001": "This was a delightful experience.", + "test-002": "I regret buying this disappointing device.", + "test-003": "The shop opens at nine.", +} +ROWS = [{"id": tid, "prompt": PREFIX + review} for tid, review in REVIEWS.items()] # tasks.jsonl, line by line + + +@pytest.mark.parametrize("config", ["run.yaml", "run_react.yaml", "run_workspace_learner.yaml"]) +def test_each_config_reads_the_rows_tasks_jsonl_held(config): + bench = Benchmark.from_config(Config.load(HERE / config)) + by_id = sorted(ROWS, key=lambda r: r["id"]) # the same keys, in order, and types + assert [json.dumps(t.given()) for t in bench.tasks.values()] == [json.dumps(r) for r in by_id] + for name in ("train", "val", "test"): + assert [t.given() for t in bench.split(name)] == [r for r in ROWS if r["id"].startswith(name)] + task = bench.tasks["test-003"] + assert task.private["tests"] is None # judged on the host (judge.py) + assert task.private["needs"]["network"]["agent"] == "no-network" # what every config gives diff --git a/examples/clbench_database/README.md b/examples/clbench_database/README.md index 93b42236..a54b18ac 100644 --- a/examples/clbench_database/README.md +++ b/examples/clbench_database/README.md @@ -21,7 +21,7 @@ Fireworks). A local server: ```bash git clone git@github.com:haizhongzheng/clbench-mwm.git ~/code/clbench && (cd ~/code/clbench && uv sync) -python examples/clbench_database/utils/prepare.py --clbench ~/code/clbench # the five ordered streams +python examples/clbench_database/utils/benchmark/prepare.py --clbench ~/code/clbench # the five ordered streams ``` ## Run @@ -81,8 +81,10 @@ venue's observation names after each answer. ```text utils/agent/ CLBenchDatabaseAgent (agent.py) starts database_runner.py in clbench's virtualenv through clbench_host.py; host.py is the acting host, built over hermit.Session -utils/prepare.py tasks.jsonl and one ordered split per order utils/judge.py the venue's score and feedback -utils/task-info/ the task for the learner; task-info/mwm/ is what MWM needs to know about the venue +utils/benchmark/ prepare.py: the tasks (task directories in Harbor's format, read back as their rows by + utils/agent/*_fields.py, the configs' data.view) and one ordered split per order; judge.py: the + venue's score and feedback; task-info/: the task for the learner (task-info/mwm/ is what MWM + needs to know about the venue) utils/tests/ golden tests of the MWM texts (both venues) and the host test, which runs under clbench's own interpreter: PYTHONPATH= MWM_CLBENCH= /.venv/bin/python -m pytest utils/tests/test_host.py mwm/ prompt_only/ run.yaml and launch.sh per method; mwm/sql_tool.py is the learner's SQL channel diff --git a/examples/clbench_database/mwm/launch.sh b/examples/clbench_database/mwm/launch.sh index 067133f3..be55ed78 100755 --- a/examples/clbench_database/mwm/launch.sh +++ b/examples/clbench_database/mwm/launch.sh @@ -37,7 +37,7 @@ HOST_MODEL=${MW_HOST_MODEL:-$MODEL} EVOLVER_MODEL=${MW_EVOLVER_MODEL:-$MODEL} EFFORT=${MW_EFFORT:-high} META_PORT=$((WM_PORT+1)) -TASKDIR=$FRAMEWORK/examples/clbench_database/utils +TASKDIR=$FRAMEWORK/examples/clbench_database/utils/benchmark export PYTHONPATH=$FRAMEWORK${PYTHONPATH:+:$PYTHONPATH} # The two models resolved on their backends (the shell form of hermit.agent.model.backends: HOST_* and EVOLVER_*). @@ -76,7 +76,7 @@ if [ "$VENUE" = database ]; then fi command -v codex >/dev/null || { echo "REFUSING: codex CLI not on PATH"; exit 1; } -[ -f "$TASKDIR/tasks.jsonl" ] || { echo "REFUSING: run $TASKDIR/prepare.py first"; exit 1; } +[ -d "$TASKDIR/tasks" ] || { echo "REFUSING: run $TASKDIR/prepare.py first"; exit 1; } # The meters this launch starts stop within a second or two of the processes that use them ending (#82): its own and # those of any launch that reuses one. Each launch on a port lists its processes in the port's users file, emptied @@ -137,7 +137,7 @@ for i in ${MW_ORDERS:-0}; do mkdir -p "$WD/evidence" : > "$WD/current_db" # The store's README exists before the bootstrap session, as natively (the wrapper wrote it before block 0). - "$PY" -m hermit.agent.evolver.methods.mwm.evidence.readme --evidence "$WD/evidence" --from "$HERE/../utils/task-info/mwm/contract/env_evidence.md" + "$PY" -m hermit.agent.evolver.methods.mwm.evidence.readme --evidence "$WD/evidence" --from "$HERE/../utils/benchmark/task-info/mwm/contract/env_evidence.md" if [ "$VENUE" = database ]; then # the first question's database, registered before the bootstrap, as natively "$CLBENCH/.venv/bin/python" "$FRAMEWORK/examples/clbench_database/utils/agent/database_runner.py" --register-first $i \ --workdir "$WD" --clbench "$CLBENCH" --framework "$FRAMEWORK" diff --git a/examples/clbench_database/mwm/run.yaml b/examples/clbench_database/mwm/run.yaml index 1456077d..c71cb521 100644 --- a/examples/clbench_database/mwm/run.yaml +++ b/examples/clbench_database/mwm/run.yaml @@ -1,16 +1,17 @@ # MWM on the CL-Bench database venue: one order's 40-question stream, one MWM block per question (learn_every 1), # accuracy scoring, unlimited host queries. Started per order by `launch.sh`, which sets the per-order paths. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/order0.txt # launch.sh: order.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/clbench_fields.py:clbench_task # each task as the row it always was + train: ../utils/benchmark/splits/order0.txt # launch.sh: order.txt step_size: 1 # one block per question (prepare.py --step-size 1) shuffle: false # the stream's own order -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: name: ../utils/agent/agent.py:CLBenchDatabaseAgent concurrency: 1 judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: mwm evolver: name: mwm @@ -31,7 +32,7 @@ evolver_access: input_schema: {type: object, properties: {query: {type: string}}, required: [query], additionalProperties: false} output_schema: {type: object} environment: # the task sandbox: the host runner, clbench's checkout and venv - read: [../utils/agent, ../utils/task-info/mwm, $MWM_CLBENCH] + read: [../utils/agent, ../utils/benchmark/task-info/mwm, $MWM_CLBENCH] write: [$MWM_TASK_WORKDIR] # the run's task directory: evidence, host records env: ["MWM_*", "CLB_*", "MW_*", "LITELLM_*"] # the launcher's settings (launch.sh) # + the host's litellm key variable (launch.sh): the benchmark's own loop calls diff --git a/examples/clbench_database/prompt_only/launch.sh b/examples/clbench_database/prompt_only/launch.sh index 99cc5904..a24bf1e1 100755 --- a/examples/clbench_database/prompt_only/launch.sh +++ b/examples/clbench_database/prompt_only/launch.sh @@ -39,7 +39,7 @@ EFFORT=${MW_EFFORT:-high} # The learner model's context window, told to codex: MW_CONTEXT_WINDOW, else what the backend declares, else 1M # for Fireworks (the runs in the README used it); empty leaves codex's own default. META_PORT=$((WM_PORT+1)) -TASKDIR=$FRAMEWORK/examples/clbench_database/utils +TASKDIR=$FRAMEWORK/examples/clbench_database/utils/benchmark export PYTHONPATH=$FRAMEWORK${PYTHONPATH:+:$PYTHONPATH} # The two models resolved on their backends (the shell form of hermit.agent.model.backends: HOST_* and EVOLVER_*). @@ -80,7 +80,7 @@ if [ "$VENUE" = database ]; then fi command -v codex >/dev/null || { echo "REFUSING: codex CLI not on PATH"; exit 1; } -[ -f "$TASKDIR/tasks.jsonl" ] || { echo "REFUSING: run $TASKDIR/prepare.py first"; exit 1; } +[ -d "$TASKDIR/tasks" ] || { echo "REFUSING: run $TASKDIR/prepare.py first"; exit 1; } # The meters this launch starts stop within a second or two of the processes that use them ending (#82): its own and # those of any launch that reuses one. Each launch on a port lists its processes in the port's users file, emptied @@ -142,7 +142,7 @@ for i in ${MW_ORDERS:-0}; do : > "$WD/current_db" # The host runners keep the venue's record store under the task directory (the learner here reads the # framework's learning material, not this store); the database runner needs the first question's database. - "$PY" -m hermit.agent.evolver.methods.mwm.evidence.readme --evidence "$WD/evidence" --from "$HERE/../utils/task-info/mwm/contract/env_evidence.md" + "$PY" -m hermit.agent.evolver.methods.mwm.evidence.readme --evidence "$WD/evidence" --from "$HERE/../utils/benchmark/task-info/mwm/contract/env_evidence.md" if [ "$VENUE" = database ]; then "$CLBENCH/.venv/bin/python" "$FRAMEWORK/examples/clbench_database/utils/agent/database_runner.py" --register-first $i \ --workdir "$WD" --clbench "$CLBENCH" --framework "$FRAMEWORK" diff --git a/examples/clbench_database/prompt_only/run.yaml b/examples/clbench_database/prompt_only/run.yaml index 5a00b982..b9137858 100644 --- a/examples/clbench_database/prompt_only/run.yaml +++ b/examples/clbench_database/prompt_only/run.yaml @@ -4,16 +4,17 @@ # (one fresh Codex session per step, GLM 5.3 Flash through the learner's meter, reasoning effort high). # Started per order by `launch.sh`, which sets the per-order paths and the learner's provider arguments. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/order0.txt # launch.sh: order.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/clbench_fields.py:clbench_task # each task as the row it always was + train: ../utils/benchmark/splits/order0.txt # launch.sh: order.txt step_size: 1 # one step per question (prepare.py --step-size 1) shuffle: false # the stream's own order -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: name: ../utils/agent/agent.py:CLBenchDatabaseAgent concurrency: 1 judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: prompt-only@1.0 evolver: name: codex # the built-in Codex AgentEvolver, with GLM 5.3 Flash through the learner's meter @@ -24,7 +25,7 @@ evolver: home: [] # no login files: the provider key is the backend's variable tools: [check_spec] # no run_agent, llm or budget: the MWM learner of this venue has none either environment: # the task sandbox: the host runner, clbench's checkout and venv - read: [../utils/agent, ../utils/task-info/mwm, $MWM_CLBENCH] + read: [../utils/agent, ../utils/benchmark/task-info/mwm, $MWM_CLBENCH] write: [$MWM_TASK_WORKDIR] # the run's task directory: evidence, host records env: ["MWM_*", "CLB_*", "MW_*", "LITELLM_*"] # the launcher's settings (launch.sh) # + the host's litellm key variable (launch.sh): the benchmark's own loop calls diff --git a/examples/clbench_database/reharness/launch.sh b/examples/clbench_database/reharness/launch.sh index 2b9bee18..720535a9 100755 --- a/examples/clbench_database/reharness/launch.sh +++ b/examples/clbench_database/reharness/launch.sh @@ -48,7 +48,7 @@ EMBED_MODEL=${MW_EMBED_MODEL:-fireworks/accounts/fireworks/models/qwen3-embeddin EFFORT=${MW_EFFORT:-high} META_PORT=$((WM_PORT+1)) EMBED_PORT=$((WM_PORT+2)) -TASKDIR=$FRAMEWORK/examples/clbench_database/utils +TASKDIR=$FRAMEWORK/examples/clbench_database/utils/benchmark export PYTHONPATH=$FRAMEWORK${PYTHONPATH:+:$PYTHONPATH} # The three models resolved on their backends (the shell form of hermit.agent.model.backends: HOST_*, EVOLVER_* and EMBED_*). @@ -88,7 +88,7 @@ if [ "$VENUE" = database ]; then || { echo "REFUSING: task prompt lacks the accuracy scoring sentence"; exit 1; } fi -[ -f "$TASKDIR/tasks.jsonl" ] || { echo "REFUSING: run $TASKDIR/prepare.py first"; exit 1; } +[ -d "$TASKDIR/tasks" ] || { echo "REFUSING: run $TASKDIR/prepare.py first"; exit 1; } # The meters this launch starts stop within a second or two of the processes that use them ending (#82): its own and # those of any launch that reuses one. Each launch on a port lists its processes in the port's users file, emptied @@ -161,7 +161,7 @@ for i in ${MW_ORDERS:-0}; do # The task directory keeps MWM's layout (current_db, the evidence README; the database's first question # registered), as ../mwm/launch.sh writes it, because the host runners are MWM's; ReHarness's learner reads the # framework's learning material, none of this. - "$PY" -m hermit.agent.evolver.methods.mwm.evidence.readme --evidence "$WD/evidence" --from "$HERE/../utils/task-info/mwm/contract/env_evidence.md" + "$PY" -m hermit.agent.evolver.methods.mwm.evidence.readme --evidence "$WD/evidence" --from "$HERE/../utils/benchmark/task-info/mwm/contract/env_evidence.md" if [ "$VENUE" = database ]; then "$CLBENCH/.venv/bin/python" "$FRAMEWORK/examples/clbench_database/utils/agent/database_runner.py" --register-first $i \ --workdir "$WD" --clbench "$CLBENCH" --framework "$FRAMEWORK" diff --git a/examples/clbench_database/reharness/run.yaml b/examples/clbench_database/reharness/run.yaml index b9ad0da1..d7cc32ef 100644 --- a/examples/clbench_database/reharness/run.yaml +++ b/examples/clbench_database/reharness/run.yaml @@ -7,17 +7,18 @@ # Flash through the learner's meter at reasoning effort high. Started per order by `launch.sh`, which sets the # per-order paths, the meters and the embedding relay. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/order0.txt # launch.sh: order.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/clbench_fields.py:clbench_task # each task as the row it always was + train: ../utils/benchmark/splits/order0.txt # launch.sh: order.txt step_size: 1 # one step per question: ReHarness learns per instance (the `block` field of - # tasks.jsonl groups questions for MWM only; it names their evidence directory) + # a task's row groups questions for MWM only; it names their evidence directory) shuffle: false # the stream's own order -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: name: ../utils/agent/agent.py:CLBenchDatabaseAgent concurrency: 1 judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: reharness evolver: name: reharness @@ -29,7 +30,7 @@ llm: # the revision and reflection agents; l model: fireworks/accounts/fireworks/models/glm-5p3-flash # launch.sh: learner/ extra_body: {reasoning_effort: high} # launch.sh: MW_EFFORT, the host's effort environment: # the task sandbox: the host runner, clbench's checkout and venv - read: [../utils/agent, ../utils/task-info/mwm, $MWM_CLBENCH] + read: [../utils/agent, ../utils/benchmark/task-info/mwm, $MWM_CLBENCH] write: [$MWM_TASK_WORKDIR] # the run's task directory: evidence, host records env: ["MWM_*", "CLB_*", "MW_*", "LITELLM_*", "REHARNESS_*"] # the launcher's settings, the reader's relay # + the host's litellm key variable (launch.sh): the benchmark's own loop calls diff --git a/examples/clbench_database/utils/.gitignore b/examples/clbench_database/utils/.gitignore index 87a26c7f..a05ea4bc 100644 --- a/examples/clbench_database/utils/.gitignore +++ b/examples/clbench_database/utils/.gitignore @@ -1,2 +1,3 @@ +# Where utils/prepare.py wrote before utils/benchmark/: kept out of git in a checkout prepared then tasks.jsonl splits/ diff --git a/examples/clbench_database/utils/agent/clbench_fields.py b/examples/clbench_database/utils/agent/clbench_fields.py new file mode 100644 index 00000000..8af2bd59 --- /dev/null +++ b/examples/clbench_database/utils/agent/clbench_fields.py @@ -0,0 +1,10 @@ +"""A CL-Bench question as the agent and its learners read it: {id, order, position, question_id, block}, the row each +task was in tasks.jsonl. A task directory (utils/benchmark/prepare.py) has every field but its id under task.toml's +[metadata], in the row's order; this puts them back, so the host, its records and the learning material are what they +were.""" + + +def clbench_task(task: dict) -> dict: + if "instruction" not in task: # already a row of that shape + return dict(task) + return {"id": task["id"], **(task.get("metadata") or {})} diff --git a/examples/clbench_database/utils/agent/database_runner.py b/examples/clbench_database/utils/agent/database_runner.py index b390364a..f5911c7a 100644 --- a/examples/clbench_database/utils/agent/database_runner.py +++ b/examples/clbench_database/utils/agent/database_runner.py @@ -89,7 +89,7 @@ def main() -> None: host_ws = workdir / "host" host_ws.mkdir(parents=True, exist_ok=True) from hermit.agent.evolver.methods.mwm import deployment as _dep - venue_texts = TASKS / "task-info" / "mwm" + venue_texts = TASKS / "benchmark" / "task-info" / "mwm" deployment = _dep.host_settings(venue_texts) task = dbtask.DatabaseExploration(schedule="default", run_index=order) diff --git a/examples/clbench_database/utils/benchmark/.gitignore b/examples/clbench_database/utils/benchmark/.gitignore new file mode 100644 index 00000000..d296b16f --- /dev/null +++ b/examples/clbench_database/utils/benchmark/.gitignore @@ -0,0 +1,2 @@ +tasks/ +splits/ diff --git a/examples/clbench_database/utils/judge.py b/examples/clbench_database/utils/benchmark/judge.py similarity index 100% rename from examples/clbench_database/utils/judge.py rename to examples/clbench_database/utils/benchmark/judge.py diff --git a/examples/clbench_database/utils/prepare.py b/examples/clbench_database/utils/benchmark/prepare.py similarity index 60% rename from examples/clbench_database/utils/prepare.py rename to examples/clbench_database/utils/benchmark/prepare.py index 1365a1ca..10b0f23c 100644 --- a/examples/clbench_database/utils/prepare.py +++ b/examples/clbench_database/utils/benchmark/prepare.py @@ -1,9 +1,13 @@ #!/usr/bin/env python3 -"""Write tasks.jsonl and one ordered split per order for the CL-Bench database venue. +"""Write the tasks (task directories in Harbor's format, tasks//) and one ordered split per order for the CL-Bench +database venue. python prepare.py [--orders 0 1 2 3 4] [--step-size 1] [--clbench ~/code/clbench] -A task is one question of one order's stream: {id, order, position, question_id, block}. `block` is the MWM block +A task is one question of one order's stream: {id, order, position, question_id, block}, its fields but the id under +task.toml's [metadata] (utils/agent/clbench_fields.py reads it back as that row); the stream's state is the host's +(the agent's own loop, one question a call, in its run's directory), and the judge (judge.py) reads the official +task's verdict from the agent's output, so a task has no tests/. `block` is the MWM block the question belongs to (position // step_size + 1), which names its evidence directory; the run must use the same `data.step_size` and `data.shuffle: false`, so a Trainer step is exactly one block. The question ids come from the official task itself (clbench's environment), so a stream mismatch is caught again when a question is played. @@ -11,8 +15,11 @@ import argparse import json from pathlib import Path +import shutil import subprocess +from hermit.task.directory import write_task as write_task_dir + HERE = Path(__file__).resolve().parent LIST = """ @@ -30,6 +37,15 @@ """ +def write_task(tasks: Path, row: dict) -> Path: + """One question as a task directory: its fields but the id under [metadata], in the row's order. The host needs + no network but its model's (the run's sandbox reaches that, and the launcher's meter).""" + instruction = (f"Answer question {row['question_id']} of CL-Bench's database exploration stream (order " + f"{row['order']}, position {row['position']}) as the host of its loop.") + return write_task_dir(tasks / row["id"], instruction, metadata={k: v for k, v in row.items() if k != "id"}, + environment={"network_mode": "no-network"}) + + def main(): ap = argparse.ArgumentParser() ap.add_argument("--orders", type=int, nargs="+", default=[0, 1, 2, 3, 4]) @@ -49,7 +65,10 @@ def main(): "block": position // a.step_size + 1}) ids.append(tid) (HERE / "splits" / f"order{order}.txt").write_text("\n".join(ids) + "\n") - (HERE / "tasks.jsonl").write_text("".join(json.dumps(r) + "\n" for r in rows)) + if (HERE / "tasks").exists(): # this script's own output: rewritten whole + shutil.rmtree(HERE / "tasks") + for row in rows: + write_task(HERE / "tasks", row) print(f"{len(rows)} tasks, {len(a.orders)} orders, step size {a.step_size}") diff --git a/examples/clbench_database/utils/task-info/README.md b/examples/clbench_database/utils/benchmark/task-info/README.md similarity index 100% rename from examples/clbench_database/utils/task-info/README.md rename to examples/clbench_database/utils/benchmark/task-info/README.md diff --git a/examples/clbench_database/utils/task-info/mwm/contract/domain_what.md b/examples/clbench_database/utils/benchmark/task-info/mwm/contract/domain_what.md similarity index 100% rename from examples/clbench_database/utils/task-info/mwm/contract/domain_what.md rename to examples/clbench_database/utils/benchmark/task-info/mwm/contract/domain_what.md diff --git a/examples/clbench_database/utils/task-info/mwm/contract/env_consumption.md b/examples/clbench_database/utils/benchmark/task-info/mwm/contract/env_consumption.md similarity index 100% rename from examples/clbench_database/utils/task-info/mwm/contract/env_consumption.md rename to examples/clbench_database/utils/benchmark/task-info/mwm/contract/env_consumption.md diff --git a/examples/clbench_database/utils/task-info/mwm/contract/env_evidence.md b/examples/clbench_database/utils/benchmark/task-info/mwm/contract/env_evidence.md similarity index 100% rename from examples/clbench_database/utils/task-info/mwm/contract/env_evidence.md rename to examples/clbench_database/utils/benchmark/task-info/mwm/contract/env_evidence.md diff --git a/examples/clbench_database/utils/task-info/mwm/contract/env_executable.md b/examples/clbench_database/utils/benchmark/task-info/mwm/contract/env_executable.md similarity index 100% rename from examples/clbench_database/utils/task-info/mwm/contract/env_executable.md rename to examples/clbench_database/utils/benchmark/task-info/mwm/contract/env_executable.md diff --git a/examples/clbench_database/utils/task-info/mwm/contract/env_verification.md b/examples/clbench_database/utils/benchmark/task-info/mwm/contract/env_verification.md similarity index 100% rename from examples/clbench_database/utils/task-info/mwm/contract/env_verification.md rename to examples/clbench_database/utils/benchmark/task-info/mwm/contract/env_verification.md diff --git a/examples/clbench_database/utils/task-info/mwm/delegation b/examples/clbench_database/utils/benchmark/task-info/mwm/delegation similarity index 100% rename from examples/clbench_database/utils/task-info/mwm/delegation rename to examples/clbench_database/utils/benchmark/task-info/mwm/delegation diff --git a/examples/clbench_database/utils/task-info/mwm/environment/README.md b/examples/clbench_database/utils/benchmark/task-info/mwm/environment/README.md similarity index 100% rename from examples/clbench_database/utils/task-info/mwm/environment/README.md rename to examples/clbench_database/utils/benchmark/task-info/mwm/environment/README.md diff --git a/examples/clbench_database/utils/task-info/mwm/environment/bin/task-env b/examples/clbench_database/utils/benchmark/task-info/mwm/environment/bin/task-env similarity index 100% rename from examples/clbench_database/utils/task-info/mwm/environment/bin/task-env rename to examples/clbench_database/utils/benchmark/task-info/mwm/environment/bin/task-env diff --git a/examples/clbench_database/utils/task-info/mwm/host.json b/examples/clbench_database/utils/benchmark/task-info/mwm/host.json similarity index 100% rename from examples/clbench_database/utils/task-info/mwm/host.json rename to examples/clbench_database/utils/benchmark/task-info/mwm/host.json diff --git a/examples/clbench_database/utils/tests/test_clbench_task_directories.py b/examples/clbench_database/utils/tests/test_clbench_task_directories.py new file mode 100644 index 00000000..6cbff0a8 --- /dev/null +++ b/examples/clbench_database/utils/tests/test_clbench_task_directories.py @@ -0,0 +1,44 @@ +"""Both CL-Bench venues' task directories (utils/benchmark/prepare.py's write_task) read back, through every config's +data.view, as exactly the rows tasks.jsonl held: the host, its records and the learning material are unchanged. Each +row stays one task; the stream's state is the host's, in the agent's own loop.""" +import importlib.util +import json +from pathlib import Path + +import pytest + +from hermit.config import Config +from hermit.task import Benchmark + +EXAMPLES = Path(__file__).resolve().parents[3] +ROWS = {"clbench_database": [{"id": "o0_q001", "order": 0, "position": 0, "question_id": "1", "block": 1}, + {"id": "o0_q002", "order": 0, "position": 1, "question_id": "5", "block": 2}], + "clbench_poker": [{"id": "o0_h001", "order": 0, "hand_num": 1, "block": 1}, + {"id": "o0_h003", "order": 0, "hand_num": 3, "block": 1}]} + + +def _prepare(venue): + spec = importlib.util.spec_from_file_location(f"{venue}_prepare", EXAMPLES / venue / "utils/benchmark/prepare.py") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) # its work is in main(): importing it writes nothing + return module + + +@pytest.mark.parametrize("venue", sorted(ROWS)) +def test_a_venues_tasks_read_back_as_their_rows_in_every_config(venue, tmp_path, monkeypatch): + prepare = _prepare(venue) + for row in ROWS[venue]: + prepare.write_task(tmp_path / "tasks", row) + (tmp_path / "order0.txt").write_text("".join(r["id"] + "\n" for r in ROWS[venue])) + configs = sorted((EXAMPLES / venue).glob("*/run.yaml")) + assert len(configs) == 3 + for path in configs: + config = Config.load(path) + data = config.data["data"] + assert config.path(data["tasks"]) == (EXAMPLES / venue / "utils/benchmark/tasks").resolve(), path + monkeypatch.setitem(data, "tasks", str(tmp_path / "tasks")) + monkeypatch.setitem(data, "train", str(tmp_path / "order0.txt")) + bench = Benchmark.from_config(config) + assert [json.dumps(t.given()) for t in bench.split("train")] == [json.dumps(r) for r in ROWS[venue]], path + task = bench.split("train")[0] + assert task.private["tests"] is None and task.private["needs"]["network"]["agent"] == "no-network" diff --git a/examples/clbench_database/utils/tests/test_host.py b/examples/clbench_database/utils/tests/test_host.py index af8b1342..78612444 100644 --- a/examples/clbench_database/utils/tests/test_host.py +++ b/examples/clbench_database/utils/tests/test_host.py @@ -38,7 +38,7 @@ def setUp(self): shutil.copytree(TEMPLATE, self.adapter, ignore=shutil.ignore_patterns("__pycache__")) (self.adapter / "ADAPTER.md").write_text("CONTEXT FOR THE HOST\n") from hermit.agent.evolver.methods.mwm import deployment as _dep - deployment = _dep.host_settings(TASK / "utils/task-info/mwm") + deployment = _dep.host_settings(TASK / "utils/benchmark/task-info/mwm") from hermit import Session session = Session(self.adapter, evidence=self.tmp / "evidence", deployment=deployment) self.system = host.AdapterHost(session, model="m", workspace=self.tmp / "ws", venue="database") diff --git a/examples/clbench_database/utils/tests/test_parity_golden.py b/examples/clbench_database/utils/tests/test_parity_golden.py index 11533d0a..44ec5af2 100644 --- a/examples/clbench_database/utils/tests/test_parity_golden.py +++ b/examples/clbench_database/utils/tests/test_parity_golden.py @@ -59,7 +59,7 @@ def _session(tmp: Path, domain: str, block: int, tick: int) -> tuple[Path, Path] shutil.copytree(TEMPLATE, adapter, ignore=shutil.ignore_patterns("__pycache__")) session = tmp / "session" (session / "control").mkdir(parents=True) - shutil.copytree(VENUES[domain] / "utils/task-info/mwm", session / "deployment") + shutil.copytree(VENUES[domain] / "utils/benchmark/task-info/mwm", session / "deployment") PROMPT.prepare_session(adapter, session, block=block, tick=tick) return adapter, session @@ -112,7 +112,7 @@ def test_push_package_matches_native(self): f"i=wm_iface.WorldModelInterface({str(wrepo)!r}, {str(evidence)!r}, delegation='none', " "code_channel=True); print(i.package(wm_iface.transport_note()), end='')") from hermit.agent.evolver.methods.mwm import deployment as _dep - host = _dep.host_settings(TASKS / "utils/task-info/mwm") + host = _dep.host_settings(TASKS / "utils/benchmark/task-info/mwm") # The pipeline reads ADAPTER.md as the Evolver wrote it after an adopted step; write it the same way # first (interface.context with the venue's host.json), since no step has run on this copy. from hermit.adapter.loader import Adapter @@ -169,7 +169,7 @@ def test_readmes_match_native(self): for domain in ("clbench", "poker"): native = _native(f"from mwm import env_surface; print(env_surface.readme({domain!r}), end='')", MW_META_SQL="1") - ours = (VENUES[domain] / "utils/task-info/mwm/environment/README.md").read_text() + ours = (VENUES[domain] / "utils/benchmark/task-info/mwm/environment/README.md").read_text() self.assertEqual(native.strip(), ours.strip(), domain) diff --git a/examples/clbench_poker/README.md b/examples/clbench_poker/README.md index 70185b26..931bc0a0 100644 --- a/examples/clbench_poker/README.md +++ b/examples/clbench_poker/README.md @@ -21,7 +21,7 @@ Fireworks). A local server: ```bash git clone git@github.com:haizhongzheng/clbench-mwm.git ~/code/clbench && (cd ~/code/clbench && uv sync) -python examples/clbench_poker/utils/prepare.py --clbench ~/code/clbench # the five ordered streams +python examples/clbench_poker/utils/benchmark/prepare.py --clbench ~/code/clbench # the five ordered streams ``` ## Run @@ -73,8 +73,10 @@ ReHarness has no reported number on this venue. ```text utils/agent/ CLBenchPokerAgent (agent.py) starts poker_runner.py in clbench's virtualenv through clbench_host.py; host.py is the acting host, built over hermit.Session -utils/prepare.py tasks.jsonl and one ordered split per order utils/judge.py the venue's score and feedback -utils/task-info/ the task for the learner; task-info/mwm/ is what MWM needs to know about the venue +utils/benchmark/ prepare.py: the tasks (task directories in Harbor's format, read back as their rows by + utils/agent/*_fields.py, the configs' data.view) and one ordered split per order; judge.py: the + venue's score and feedback; task-info/: the task for the learner (task-info/mwm/ is what MWM + needs to know about the venue) mwm/ prompt_only/ run.yaml and launch.sh per method reharness/ run.yaml and launch.sh for ReHarness; its template and Evolver are the framework's built-ins ``` diff --git a/examples/clbench_poker/mwm/launch.sh b/examples/clbench_poker/mwm/launch.sh index c083e428..0f35ffd0 100755 --- a/examples/clbench_poker/mwm/launch.sh +++ b/examples/clbench_poker/mwm/launch.sh @@ -37,7 +37,7 @@ HOST_MODEL=${MW_HOST_MODEL:-$MODEL} EVOLVER_MODEL=${MW_EVOLVER_MODEL:-$MODEL} EFFORT=${MW_EFFORT:-high} META_PORT=$((WM_PORT+1)) -TASKDIR=$FRAMEWORK/examples/clbench_poker/utils +TASKDIR=$FRAMEWORK/examples/clbench_poker/utils/benchmark export PYTHONPATH=$FRAMEWORK${PYTHONPATH:+:$PYTHONPATH} # The two models resolved on their backends (the shell form of hermit.agent.model.backends: HOST_* and EVOLVER_*). @@ -71,7 +71,7 @@ export LITELLM_DROP_PARAMS=${LITELLM_DROP_PARAMS:-True} LITELLM_LOCAL_MODEL_COST export MWM_CLBENCH=$CLBENCH MWM_METAWORLD=$METAWORLD command -v codex >/dev/null || { echo "REFUSING: codex CLI not on PATH"; exit 1; } -[ -f "$TASKDIR/tasks.jsonl" ] || { echo "REFUSING: run $TASKDIR/prepare.py first"; exit 1; } +[ -d "$TASKDIR/tasks" ] || { echo "REFUSING: run $TASKDIR/prepare.py first"; exit 1; } # The meters this launch starts stop within a second or two of the processes that use them ending (#82): its own and # those of any launch that reuses one. Each launch on a port lists its processes in the port's users file, emptied @@ -132,7 +132,7 @@ for i in ${MW_ORDERS:-0}; do mkdir -p "$WD/evidence" : > "$WD/current_db" # The store's README exists before the bootstrap session, as natively (the wrapper wrote it before block 0). - "$PY" -m hermit.agent.evolver.methods.mwm.evidence.readme --evidence "$WD/evidence" --from "$HERE/../utils/task-info/mwm/contract/env_evidence.md" + "$PY" -m hermit.agent.evolver.methods.mwm.evidence.readme --evidence "$WD/evidence" --from "$HERE/../utils/benchmark/task-info/mwm/contract/env_evidence.md" # The learner's model through its meter: the backend copied to the metered URL, the Evolver bound to it (the # Trainer derives codex's provider settings from the binding and passes its key variable into the learner's # sandbox, the meter's port forwarded). diff --git a/examples/clbench_poker/mwm/run.yaml b/examples/clbench_poker/mwm/run.yaml index 29761019..8a14338e 100644 --- a/examples/clbench_poker/mwm/run.yaml +++ b/examples/clbench_poker/mwm/run.yaml @@ -3,16 +3,17 @@ # `launch.sh`, which sets the per-order paths. The venue metric (bb/hand over all 120 hands) is in # /outcomes.jsonl. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/order0.txt # launch.sh: order.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/poker_fields.py:poker_task # each task as the row it always was + train: ../utils/benchmark/splits/order0.txt # launch.sh: order.txt step_size: 5 # = learn_every (prepare.py --step-size 5) shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: name: ../utils/agent/agent.py:CLBenchPokerAgent concurrency: 1 judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: mwm evolver: name: mwm @@ -25,7 +26,7 @@ evolver_access: environment: [] # launch.sh: [/evidence] tools: [] # the poker venue declares no channel environment: # the task sandbox: the host runner, clbench's checkout and venv - read: [../utils/agent, ../utils/task-info/mwm, $MWM_CLBENCH] + read: [../utils/agent, ../utils/benchmark/task-info/mwm, $MWM_CLBENCH] write: [$MWM_TASK_WORKDIR] # the run's task directory: evidence, host records env: ["MWM_*", "CLB_*", "MW_*", "LITELLM_*"] # the launcher's settings (launch.sh) # + the host's litellm key variable (launch.sh): the benchmark's own loop calls diff --git a/examples/clbench_poker/prompt_only/launch.sh b/examples/clbench_poker/prompt_only/launch.sh index f6485ad0..78f19e26 100755 --- a/examples/clbench_poker/prompt_only/launch.sh +++ b/examples/clbench_poker/prompt_only/launch.sh @@ -39,7 +39,7 @@ EFFORT=${MW_EFFORT:-high} # The learner model's context window, told to codex: MW_CONTEXT_WINDOW, else what the backend declares, else 1M # for Fireworks (the runs in the README used it); empty leaves codex's own default. META_PORT=$((WM_PORT+1)) -TASKDIR=$FRAMEWORK/examples/clbench_poker/utils +TASKDIR=$FRAMEWORK/examples/clbench_poker/utils/benchmark export PYTHONPATH=$FRAMEWORK${PYTHONPATH:+:$PYTHONPATH} # The two models resolved on their backends (the shell form of hermit.agent.model.backends: HOST_* and EVOLVER_*). @@ -75,7 +75,7 @@ export LITELLM_DROP_PARAMS=${LITELLM_DROP_PARAMS:-True} LITELLM_LOCAL_MODEL_COST export MWM_CLBENCH=$CLBENCH MWM_METAWORLD=$METAWORLD command -v codex >/dev/null || { echo "REFUSING: codex CLI not on PATH"; exit 1; } -[ -f "$TASKDIR/tasks.jsonl" ] || { echo "REFUSING: run $TASKDIR/prepare.py first"; exit 1; } +[ -d "$TASKDIR/tasks" ] || { echo "REFUSING: run $TASKDIR/prepare.py first"; exit 1; } # The meters this launch starts stop within a second or two of the processes that use them ending (#82): its own and # those of any launch that reuses one. Each launch on a port lists its processes in the port's users file, emptied @@ -137,7 +137,7 @@ for i in ${MW_ORDERS:-0}; do : > "$WD/current_db" # The host runners keep the venue's record store under the task directory (the learner here reads the # framework's learning material, not this store); the database runner needs the first question's database. - "$PY" -m hermit.agent.evolver.methods.mwm.evidence.readme --evidence "$WD/evidence" --from "$HERE/../utils/task-info/mwm/contract/env_evidence.md" + "$PY" -m hermit.agent.evolver.methods.mwm.evidence.readme --evidence "$WD/evidence" --from "$HERE/../utils/benchmark/task-info/mwm/contract/env_evidence.md" # The learner's model through its meter: the backend copied to the metered URL, the Evolver bound to it (the # Trainer derives codex's provider settings from the binding and passes its key variable into the learner's # sandbox, the meter's port forwarded). diff --git a/examples/clbench_poker/prompt_only/run.yaml b/examples/clbench_poker/prompt_only/run.yaml index f2257fc8..fbf0c6a0 100644 --- a/examples/clbench_poker/prompt_only/run.yaml +++ b/examples/clbench_poker/prompt_only/run.yaml @@ -4,16 +4,17 @@ # session per step, the learner model through its meter, reasoning effort high). # Started per order by `launch.sh`, which sets the per-order paths and the learner's provider arguments. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/order0.txt # launch.sh: order.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/poker_fields.py:poker_task # each task as the row it always was + train: ../utils/benchmark/splits/order0.txt # launch.sh: order.txt step_size: 5 # one step per block of 5 host hands shuffle: false # the stream's own order -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: name: ../utils/agent/agent.py:CLBenchPokerAgent concurrency: 1 judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: prompt-only@1.0 evolver: name: codex # the built-in Codex AgentEvolver, with GLM 5.3 Flash through the learner's meter @@ -24,7 +25,7 @@ evolver: home: [] # no login files: the provider key is the backend's variable tools: [check_spec] # no run_agent, llm or budget: the MWM learner of this venue has none either environment: # the task sandbox: the host runner, clbench's checkout and venv - read: [../utils/agent, ../utils/task-info/mwm, $MWM_CLBENCH] + read: [../utils/agent, ../utils/benchmark/task-info/mwm, $MWM_CLBENCH] write: [$MWM_TASK_WORKDIR] # the run's task directory: evidence, host records env: ["MWM_*", "CLB_*", "MW_*", "LITELLM_*"] # the launcher's settings (launch.sh) # + the host's litellm key variable (launch.sh): the benchmark's own loop calls diff --git a/examples/clbench_poker/reharness/launch.sh b/examples/clbench_poker/reharness/launch.sh index d2e26eec..96899e8f 100755 --- a/examples/clbench_poker/reharness/launch.sh +++ b/examples/clbench_poker/reharness/launch.sh @@ -48,7 +48,7 @@ EMBED_MODEL=${MW_EMBED_MODEL:-fireworks/accounts/fireworks/models/qwen3-embeddin EFFORT=${MW_EFFORT:-high} META_PORT=$((WM_PORT+1)) EMBED_PORT=$((WM_PORT+2)) -TASKDIR=$FRAMEWORK/examples/clbench_poker/utils +TASKDIR=$FRAMEWORK/examples/clbench_poker/utils/benchmark export PYTHONPATH=$FRAMEWORK${PYTHONPATH:+:$PYTHONPATH} # The three models resolved on their backends (the shell form of hermit.agent.model.backends: HOST_*, EVOLVER_* and EMBED_*). @@ -83,7 +83,7 @@ export CLB_SKIP_BASELINE=1 CLB_REQUEST_TIMEOUT=600 CLB_REASONING_EFFORT=$EFFORT export LITELLM_DROP_PARAMS=${LITELLM_DROP_PARAMS:-True} LITELLM_LOCAL_MODEL_COST_MAP=${LITELLM_LOCAL_MODEL_COST_MAP:-True} export MWM_CLBENCH=$CLBENCH MWM_METAWORLD=$METAWORLD -[ -f "$TASKDIR/tasks.jsonl" ] || { echo "REFUSING: run $TASKDIR/prepare.py first"; exit 1; } +[ -d "$TASKDIR/tasks" ] || { echo "REFUSING: run $TASKDIR/prepare.py first"; exit 1; } # The meters this launch starts stop within a second or two of the processes that use them ending (#82): its own and # those of any launch that reuses one. Each launch on a port lists its processes in the port's users file, emptied @@ -155,7 +155,7 @@ for i in ${MW_ORDERS:-0}; do : > "$WD/current_db" # The task directory keeps MWM's layout (current_db, the evidence README), as ../mwm/launch.sh writes it, # because the host runner is MWM's; ReHarness's learner reads the framework's learning material, none of this. - "$PY" -m hermit.agent.evolver.methods.mwm.evidence.readme --evidence "$WD/evidence" --from "$HERE/../utils/task-info/mwm/contract/env_evidence.md" + "$PY" -m hermit.agent.evolver.methods.mwm.evidence.readme --evidence "$WD/evidence" --from "$HERE/../utils/benchmark/task-info/mwm/contract/env_evidence.md" # The learner model through its meter: the backend copied to the metered URL, the run's `llm` tool bound to it at # the host's reasoning effort (`llm` is a free-form section: a whole mapping on the command line). The embedding # model through its relay, once for the Evolver (evolver.params.embedding, with the key variable) and once for the diff --git a/examples/clbench_poker/reharness/run.yaml b/examples/clbench_poker/reharness/run.yaml index acbf636b..87806ddb 100644 --- a/examples/clbench_poker/reharness/run.yaml +++ b/examples/clbench_poker/reharness/run.yaml @@ -7,18 +7,19 @@ # in the paper), GLM 5.3 Flash through the learner's meter at reasoning effort high. Started per order by # `launch.sh`, which sets the per-order paths, the meters and the embedding relay. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/order0.txt # launch.sh: order.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/poker_fields.py:poker_task # each task as the row it always was + train: ../utils/benchmark/splits/order0.txt # launch.sh: order.txt step_size: 1 # one step per hand: ReHarness learns per instance, not per block of 5 (the - # `block` field of tasks.jsonl groups hands for MWM only, naming their evidence + # `block` field of a task's row groups hands for MWM only, naming their evidence # directory) shuffle: false # the stream's own order -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: name: ../utils/agent/agent.py:CLBenchPokerAgent concurrency: 1 judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: reharness evolver: name: reharness @@ -30,7 +31,7 @@ llm: # the revision and reflection agents; l model: fireworks/accounts/fireworks/models/glm-5p3-flash # launch.sh: learner/ extra_body: {reasoning_effort: high} # launch.sh: MW_EFFORT, the host's effort environment: # the task sandbox: the host runner, clbench's checkout and venv - read: [../utils/agent, ../utils/task-info/mwm, $MWM_CLBENCH] + read: [../utils/agent, ../utils/benchmark/task-info/mwm, $MWM_CLBENCH] write: [$MWM_TASK_WORKDIR] # the run's task directory: evidence, host records env: ["MWM_*", "CLB_*", "MW_*", "LITELLM_*", "REHARNESS_*"] # the launcher's settings, the reader's relay # + the host's litellm key variable (launch.sh): the benchmark's own loop calls diff --git a/examples/clbench_poker/utils/.gitignore b/examples/clbench_poker/utils/.gitignore index 87a26c7f..a05ea4bc 100644 --- a/examples/clbench_poker/utils/.gitignore +++ b/examples/clbench_poker/utils/.gitignore @@ -1,2 +1,3 @@ +# Where utils/prepare.py wrote before utils/benchmark/: kept out of git in a checkout prepared then tasks.jsonl splits/ diff --git a/examples/clbench_poker/utils/agent/poker_fields.py b/examples/clbench_poker/utils/agent/poker_fields.py new file mode 100644 index 00000000..df68e608 --- /dev/null +++ b/examples/clbench_poker/utils/agent/poker_fields.py @@ -0,0 +1,9 @@ +"""A CL-Bench hand as the agent and its learners read it: {id, order, hand_num, block}, the row each task was in +tasks.jsonl. A task directory (utils/benchmark/prepare.py) has every field but its id under task.toml's [metadata], in +the row's order; this puts them back, so the host, its records and the learning material are what they were.""" + + +def poker_task(task: dict) -> dict: + if "instruction" not in task: # already a row of that shape + return dict(task) + return {"id": task["id"], **(task.get("metadata") or {})} diff --git a/examples/clbench_poker/utils/agent/poker_runner.py b/examples/clbench_poker/utils/agent/poker_runner.py index 360d79d5..8caf02ec 100644 --- a/examples/clbench_poker/utils/agent/poker_runner.py +++ b/examples/clbench_poker/utils/agent/poker_runner.py @@ -67,7 +67,7 @@ def main() -> None: graded = graded[:hand_num - 1] ledger.write_text("".join(json.dumps(o) + "\n" for o in graded)) from hermit.agent.evolver.methods.mwm import deployment as _dep - venue_texts = TASKS / "task-info" / "mwm" + venue_texts = TASKS / "benchmark" / "task-info" / "mwm" deployment = _dep.host_settings(venue_texts) task = Poker(schedule="default", run_index=order) diff --git a/examples/clbench_poker/utils/benchmark/.gitignore b/examples/clbench_poker/utils/benchmark/.gitignore new file mode 100644 index 00000000..d296b16f --- /dev/null +++ b/examples/clbench_poker/utils/benchmark/.gitignore @@ -0,0 +1,2 @@ +tasks/ +splits/ diff --git a/examples/clbench_poker/utils/judge.py b/examples/clbench_poker/utils/benchmark/judge.py similarity index 100% rename from examples/clbench_poker/utils/judge.py rename to examples/clbench_poker/utils/benchmark/judge.py diff --git a/examples/clbench_poker/utils/prepare.py b/examples/clbench_poker/utils/benchmark/prepare.py similarity index 56% rename from examples/clbench_poker/utils/prepare.py rename to examples/clbench_poker/utils/benchmark/prepare.py index 5c23ec28..36f45805 100644 --- a/examples/clbench_poker/utils/prepare.py +++ b/examples/clbench_poker/utils/benchmark/prepare.py @@ -1,20 +1,28 @@ #!/usr/bin/env python3 -"""Write tasks.jsonl and one ordered split per order for the CL-Bench poker venue. +"""Write the tasks (task directories in Harbor's format, tasks//) and one ordered split per order for the CL-Bench +poker venue. python prepare.py [--orders 0 1 2 3 4] [--step-size 5] [--clbench ~/code/clbench] -A task is one hand the HOST plays: {id, order, hand_num, block}. Natively a hand the opponent ends before the host -acts is settled by the task inside the previous hand's last step; it never reaches the host and does not count -toward a learning block (the poker system learns every `learn_every` hands it observed). Which hands those are is -fixed by the deal (each hand restores its pre-generated RNG state before the opponent's first action), so they are -found here by replaying the stream through the task's own resume path, without a host. `block` is the MWM block -(index among the host's hands // step_size + 1); the run must use the same data.step_size and data.shuffle: false. +A task is one hand the HOST plays: {id, order, hand_num, block}, its fields but the id under task.toml's +[metadata] (utils/agent/poker_fields.py reads it back as that row; the stream's state is the host's, in the agent's +own loop, and the judge reads the hand's result from the agent's output, so a task has no tests/). + +Natively a hand the opponent ends before the host acts is settled by the task inside the previous hand's last step; +it never reaches the host and does not count toward a learning block (the poker system learns every `learn_every` +hands it observed). Which hands those are is fixed by the deal (each hand restores its pre-generated RNG state +before the opponent's first action), so they are found here by replaying the stream through the task's own resume +path, without a host. `block` is the MWM block (index among the host's hands // step_size + 1); the run must use the +same data.step_size and data.shuffle: false. """ import argparse import json from pathlib import Path +import shutil import subprocess +from hermit.task.directory import write_task as write_task_dir + HERE = Path(__file__).resolve().parent LIST = """ @@ -40,6 +48,14 @@ """ +def write_task(tasks: Path, row: dict) -> Path: + """One hand as a task directory: its fields but the id under [metadata], in the row's order; no network.""" + instruction = (f"Play hand {row['hand_num']} of CL-Bench's exploitable-poker stream (order {row['order']}) as " + "the host.") + return write_task_dir(tasks / row["id"], instruction, metadata={k: v for k, v in row.items() if k != "id"}, + environment={"network_mode": "no-network"}) + + def main(): ap = argparse.ArgumentParser() ap.add_argument("--orders", type=int, nargs="+", default=[0, 1, 2, 3, 4]) @@ -62,7 +78,10 @@ def main(): ids.append(tid) (HERE / "splits" / f"order{order}.txt").write_text("\n".join(ids) + "\n") print(f"order {order}: the host plays {len(hands)} of {streams[str(order)]['total']} hands") - (HERE / "tasks.jsonl").write_text("".join(json.dumps(r) + "\n" for r in rows)) + if (HERE / "tasks").exists(): # this script's own output: rewritten whole + shutil.rmtree(HERE / "tasks") + for row in rows: + write_task(HERE / "tasks", row) if __name__ == "__main__": diff --git a/examples/clbench_poker/utils/task-info/README.md b/examples/clbench_poker/utils/benchmark/task-info/README.md similarity index 100% rename from examples/clbench_poker/utils/task-info/README.md rename to examples/clbench_poker/utils/benchmark/task-info/README.md diff --git a/examples/clbench_poker/utils/task-info/mwm/contract/domain_what.md b/examples/clbench_poker/utils/benchmark/task-info/mwm/contract/domain_what.md similarity index 100% rename from examples/clbench_poker/utils/task-info/mwm/contract/domain_what.md rename to examples/clbench_poker/utils/benchmark/task-info/mwm/contract/domain_what.md diff --git a/examples/clbench_poker/utils/task-info/mwm/contract/env_consumption.md b/examples/clbench_poker/utils/benchmark/task-info/mwm/contract/env_consumption.md similarity index 100% rename from examples/clbench_poker/utils/task-info/mwm/contract/env_consumption.md rename to examples/clbench_poker/utils/benchmark/task-info/mwm/contract/env_consumption.md diff --git a/examples/clbench_poker/utils/task-info/mwm/contract/env_evidence.md b/examples/clbench_poker/utils/benchmark/task-info/mwm/contract/env_evidence.md similarity index 100% rename from examples/clbench_poker/utils/task-info/mwm/contract/env_evidence.md rename to examples/clbench_poker/utils/benchmark/task-info/mwm/contract/env_evidence.md diff --git a/examples/clbench_poker/utils/task-info/mwm/contract/env_executable.md b/examples/clbench_poker/utils/benchmark/task-info/mwm/contract/env_executable.md similarity index 100% rename from examples/clbench_poker/utils/task-info/mwm/contract/env_executable.md rename to examples/clbench_poker/utils/benchmark/task-info/mwm/contract/env_executable.md diff --git a/examples/clbench_poker/utils/task-info/mwm/contract/env_verification.md b/examples/clbench_poker/utils/benchmark/task-info/mwm/contract/env_verification.md similarity index 100% rename from examples/clbench_poker/utils/task-info/mwm/contract/env_verification.md rename to examples/clbench_poker/utils/benchmark/task-info/mwm/contract/env_verification.md diff --git a/examples/clbench_poker/utils/task-info/mwm/delegation b/examples/clbench_poker/utils/benchmark/task-info/mwm/delegation similarity index 100% rename from examples/clbench_poker/utils/task-info/mwm/delegation rename to examples/clbench_poker/utils/benchmark/task-info/mwm/delegation diff --git a/examples/clbench_poker/utils/task-info/mwm/environment/README.md b/examples/clbench_poker/utils/benchmark/task-info/mwm/environment/README.md similarity index 100% rename from examples/clbench_poker/utils/task-info/mwm/environment/README.md rename to examples/clbench_poker/utils/benchmark/task-info/mwm/environment/README.md diff --git a/examples/clbench_poker/utils/task-info/mwm/host.json b/examples/clbench_poker/utils/benchmark/task-info/mwm/host.json similarity index 100% rename from examples/clbench_poker/utils/task-info/mwm/host.json rename to examples/clbench_poker/utils/benchmark/task-info/mwm/host.json diff --git a/examples/labbench/baseline/claude.yaml b/examples/labbench/baseline/claude.yaml index 3e5c1867..fd4cc49e 100644 --- a/examples/labbench/baseline/claude.yaml +++ b/examples/labbench/baseline/claude.yaml @@ -16,17 +16,17 @@ data: train: ../utils/splits/smoke.txt # placeholder, as template and evolver are: hermit eval reads it, trains nothing test: ../utils/splits/test.txt # 127 questions (SkillOpt's split: 2:1:7, seed 42) agent: - name: harbor + base: claude # Claude Code in each task's own Harbor environment, installed there by Harbor concurrency: 8 - params: - benchmark: ../utils/benchmark.yaml - acting_agent: {cli: claude} # Claude Code in the task's container, installed there by Harbor - model: claude-haiku-5-5 # a Claude Code model name, on the subscription - login: claude # the subscription's plan, shared with Hermit's other sessions on it - effort: medium - jobs_dir: ~/.cache/hermit/harbor/jobs/labbench-claude # the Claude Code configs' trials + model: {name: claude-haiku-5-5, login: claude, effort: medium} # the subscription's plan, shared + claude: {web_search: false} # the CLI's web search off +environment: + harbor: # each task in its own Harbor environment, verified by Harbor's verifier + benchmark: ../utils/benchmark.yaml # the settings every config of this example shares + jobs_dir: ~/.cache/hermit/harbor/jobs/labbench-claude # the Claude Code configs' environments (each task's trial) judge: - path: hermit.harbor.judge # the verifier's reward + path: hermit.task.verify # Harbor's verifier, in the task's environment: the verifier's reward + name: environment template: prompt-only@1.0 # placeholder (see data.train) task_info: ../utils/task-info # what hermit check asks of a training config evolver: {name: eval-only} # placeholder: no Evolver of this name, so hermit train refuses this config diff --git a/examples/labbench/baseline/codex.yaml b/examples/labbench/baseline/codex.yaml index 2c52120b..cedc1bcc 100644 --- a/examples/labbench/baseline/codex.yaml +++ b/examples/labbench/baseline/codex.yaml @@ -14,17 +14,17 @@ data: train: ../utils/splits/smoke.txt # placeholder, as template and evolver are: hermit eval reads it, trains nothing test: ../utils/splits/test.txt # 127 questions (SkillOpt's split: 2:1:7, seed 42) agent: - name: harbor + base: codex # Codex in each task's own Harbor environment, installed there by Harbor concurrency: 8 - params: - benchmark: ../utils/benchmark.yaml - acting_agent: {cli: codex} # Codex in the task's container, installed there by Harbor - model: gpt-5.6-luna # a Codex model name, on the ChatGPT login - login: chatgpt # the ChatGPT plan, shared with Hermit's other sessions on it - effort: medium - jobs_dir: ~/.cache/hermit/harbor/jobs/labbench-codex # the Codex configs' trials + model: {name: gpt-5.6-luna, login: chatgpt, effort: medium} # the ChatGPT plan, shared + codex: {web_search: false} # the CLI's web search off +environment: + harbor: # each task in its own Harbor environment, verified by Harbor's verifier + benchmark: ../utils/benchmark.yaml # the settings every config of this example shares + jobs_dir: ~/.cache/hermit/harbor/jobs/labbench-codex # the Codex configs' environments (each task's trial) judge: - path: hermit.harbor.judge # the verifier's reward + path: hermit.task.verify # Harbor's verifier, in the task's environment: the verifier's reward + name: environment template: prompt-only@1.0 # placeholder (see data.train) task_info: ../utils/task-info # what hermit check asks of a training config evolver: {name: eval-only} # placeholder: no Evolver of this name, so hermit train refuses this config diff --git a/examples/labbench/skillopt/run_claude.yaml b/examples/labbench/skillopt/run_claude.yaml index 3309cf8f..37d99bd9 100644 --- a/examples/labbench/skillopt/run_claude.yaml +++ b/examples/labbench/skillopt/run_claude.yaml @@ -25,17 +25,17 @@ data: shuffle_seed: 42 task_info: ../utils/task-info agent: - name: harbor + base: claude # Claude Code in each task's own Harbor environment, installed there by Harbor concurrency: 8 - params: - benchmark: ../utils/benchmark.yaml - acting_agent: {cli: claude} # Claude Code in the task's container, installed there by Harbor - model: claude-haiku-5-5 # a Claude Code model name, on the subscription - login: claude # the subscription's plan, shared with Hermit's other sessions on it - effort: medium - jobs_dir: ~/.cache/hermit/harbor/jobs/labbench-claude # the Claude Code configs' trials + model: {name: claude-haiku-5-5, login: claude, effort: medium} # the subscription's plan, shared + claude: {web_search: false} # the CLI's web search off +environment: + harbor: # each task in its own Harbor environment, verified by Harbor's verifier + benchmark: ../utils/benchmark.yaml # the settings every config of this example shares + jobs_dir: ~/.cache/hermit/harbor/jobs/labbench-claude # the Claude Code configs' environments (each task's trial) judge: - path: hermit.harbor.judge # the verifier's reward + path: hermit.task.verify # Harbor's verifier, in the task's environment: the verifier's reward + name: environment template: prompt-only@1.0 # the skill is ADAPTER.md, with no instructions at first (`--adapter root`) evolver: name: skillopt diff --git a/examples/labbench/skillopt/run_codex.yaml b/examples/labbench/skillopt/run_codex.yaml index 427e55cf..95a6e3f5 100644 --- a/examples/labbench/skillopt/run_codex.yaml +++ b/examples/labbench/skillopt/run_codex.yaml @@ -24,17 +24,17 @@ data: shuffle_seed: 42 task_info: ../utils/task-info agent: - name: harbor + base: codex # Codex in each task's own Harbor environment, installed there by Harbor concurrency: 8 - params: - benchmark: ../utils/benchmark.yaml - acting_agent: {cli: codex} # Codex in the task's container, installed there by Harbor - model: gpt-5.6-luna # a Codex model name, on the ChatGPT login - login: chatgpt # the ChatGPT plan, shared with Hermit's other sessions on it - effort: medium - jobs_dir: ~/.cache/hermit/harbor/jobs/labbench-codex # the Codex configs' trials + model: {name: gpt-5.6-luna, login: chatgpt, effort: medium} # the ChatGPT plan, shared + codex: {web_search: false} # the CLI's web search off +environment: + harbor: # each task in its own Harbor environment, verified by Harbor's verifier + benchmark: ../utils/benchmark.yaml # the settings every config of this example shares + jobs_dir: ~/.cache/hermit/harbor/jobs/labbench-codex # the Codex configs' environments (each task's trial) judge: - path: hermit.harbor.judge # the verifier's reward + path: hermit.task.verify # Harbor's verifier, in the task's environment: the verifier's reward + name: environment template: prompt-only@1.0 # the skill is ADAPTER.md, with no instructions at first (`--adapter root`) evolver: name: skillopt diff --git a/examples/labbench/skillopt/smoke_claude.yaml b/examples/labbench/skillopt/smoke_claude.yaml index a490beaa..8a778f14 100644 --- a/examples/labbench/skillopt/smoke_claude.yaml +++ b/examples/labbench/skillopt/smoke_claude.yaml @@ -13,17 +13,17 @@ data: shuffle_seed: 42 task_info: ../utils/task-info agent: - name: harbor + base: claude # Claude Code in each task's own Harbor environment, installed there by Harbor concurrency: 2 - params: - benchmark: ../utils/benchmark.yaml - acting_agent: {cli: claude} # Claude Code in the task's container, installed there by Harbor - model: claude-haiku-5-5 # a Claude Code model name, on the subscription - login: claude # the subscription's plan, shared with Hermit's other sessions on it - effort: medium - jobs_dir: ~/.cache/hermit/harbor/jobs/labbench-claude-smoke # the smoke's trials, apart from the run's + model: {name: claude-haiku-5-5, login: claude, effort: medium} # the subscription's plan, shared + claude: {web_search: false} # the CLI's web search off +environment: + harbor: # each task in its own Harbor environment, verified by Harbor's verifier + benchmark: ../utils/benchmark.yaml # the settings every config of this example shares + jobs_dir: ~/.cache/hermit/harbor/jobs/labbench-claude-smoke # the smoke's environments, apart from the run's judge: - path: hermit.harbor.judge # the verifier's reward + path: hermit.task.verify # Harbor's verifier, in the task's environment: the verifier's reward + name: environment template: prompt-only@1.0 # the skill is ADAPTER.md, with no instructions at first (`--adapter root`) evolver: name: skillopt diff --git a/examples/labbench/skillopt/smoke_codex.yaml b/examples/labbench/skillopt/smoke_codex.yaml index 921f38f2..8c0c4075 100644 --- a/examples/labbench/skillopt/smoke_codex.yaml +++ b/examples/labbench/skillopt/smoke_codex.yaml @@ -13,17 +13,17 @@ data: shuffle_seed: 42 task_info: ../utils/task-info agent: - name: harbor + base: codex # Codex in each task's own Harbor environment, installed there by Harbor concurrency: 2 - params: - benchmark: ../utils/benchmark.yaml - acting_agent: {cli: codex} # Codex in the task's container, installed there by Harbor - model: gpt-5.6-luna # a Codex model name, on the ChatGPT login - login: chatgpt # the ChatGPT plan, shared with Hermit's other sessions on it - effort: medium - jobs_dir: ~/.cache/hermit/harbor/jobs/labbench-codex-smoke # the smoke's trials, apart from the run's + model: {name: gpt-5.6-luna, login: chatgpt, effort: medium} # the ChatGPT plan, shared + codex: {web_search: false} # the CLI's web search off +environment: + harbor: # each task in its own Harbor environment, verified by Harbor's verifier + benchmark: ../utils/benchmark.yaml # the settings every config of this example shares + jobs_dir: ~/.cache/hermit/harbor/jobs/labbench-codex-smoke # the smoke's environments, apart from the run's judge: - path: hermit.harbor.judge # the verifier's reward + path: hermit.task.verify # Harbor's verifier, in the task's environment: the verifier's reward + name: environment template: prompt-only@1.0 # the skill is ADAPTER.md, with no instructions at first (`--adapter root`) evolver: name: skillopt diff --git a/examples/labbench/utils/benchmark.yaml b/examples/labbench/utils/benchmark.yaml index 52b125a8..2ff9c2cf 100644 --- a/examples/labbench/utils/benchmark.yaml +++ b/examples/labbench/utils/benchmark.yaml @@ -1,10 +1,11 @@ -# LAB-Bench FigQA through Harbor: the agent.params every config of this example shares (agent.params.benchmark; a -# config's own params override these), and the task set (the dataset: block, which `python -m hermit.harbor.dataset -# prepare` reads). One rollout is one Harbor trial of a task: Harbor builds the task's image (Python with the -# question's figure), the container's entrypoint shuffles the answer choices into /app/instruction.md and deletes the -# file that names the right one before the agent starts, the acting agent looks at the figure and writes its letter -# to /app/answer.txt, and the verifier (uploaded only after the agent stopped) scores it: 1 for the right letter, else -# 0. The acting agent, its model and its sign-in are each config's own. +# LAB-Bench FigQA in Harbor's environments: the environment.harbor settings every config of this example shares +# (environment.harbor.benchmark; a config's own settings override these), and the task set (the dataset: block, which +# `python -m hermit.harbor.dataset prepare` reads). One rollout is one task in its own Harbor environment: Harbor +# builds the task's image (Python with the question's figure), the container's entrypoint shuffles the answer choices +# into /app/instruction.md and deletes the file that names the right one before the agent starts, the acting agent +# looks at the figure and writes its letter to /app/answer.txt, and the verifier (uploaded only after the agent +# stopped) scores it: 1 for the right letter, else 0. The acting agent, its model and its sign-in are each config's +# own. dataset: ref: labbench@1.0 # 181 multiple-choice questions about figures of biology papers split: {train: 36, val: 18, test: rest, seed: 42} # SkillOpt's split: 2:1:7 with its seed 42 (127 test tasks) @@ -12,8 +13,7 @@ dataset: patch: patches.py:patch_tasks # the shuffle's seed not saved where the agent reads it, and the verifier # not running /app/setup.py, the agent's to rewrite (patches.py) tasks_dir: tasks # the 181 task directories (prepare's link, outside the checkout) -web_search: false # the CLI's web search off: the papers and their figures are public agent_timeout_sec: null # each task's own limit (300 s for every task) docker_cleanup: images_keep # Harbor removes the image it built -disk_gate_gb: 25 # no trial starts with less than this free on Docker's data root +disk_gate_gb: 25 # no environment starts with less than this free on Docker's data root disk_gate_wait_s: 21600 diff --git a/examples/libero/README.md b/examples/libero/README.md index 617021c4..6b64c097 100644 --- a/examples/libero/README.md +++ b/examples/libero/README.md @@ -11,9 +11,9 @@ on held-out seeds by the task's own success check. | Folder | What it is | |---|---| -| `utils/prepare.py` | The 210 tasks (a suite and a task id each) with their seeds and splits; the verifier's rows and the reruns | +| `utils/benchmark/prepare.py` | The 210 tasks (a suite and a task id each) with their seeds, as task directories in Harbor's format (`tasks/`, read as their rows by the engine's `aspire_fields.py`), and splits; the verifier's tasks and the reruns | | `utils/agent/` | A method-neutral acting prompt (`prompt.md`, written here: LIBERO has none) and the workspace without ASPIRE's method pages, for a method that brings no prompt of its own | -| `utils/task-info/` | What a task and its record are, for the learner | +| `utils/benchmark/task-info/` | What a task and its record are, for the learner | | `utils/launch.sh` | Starts a method folder's run: the simulator service and hermit | | [`aspire/`](aspire/) | The ASPIRE method: its fix-loop prompts, verifier and coordinator. `run.yaml` (Object, Spatial, Goal, Long: 40 tasks), `run_90.yaml` (LIBERO-90), `run_pro.yaml` (LIBERO-Pro: 80), `run_pro_long.yaml` (LIBERO-Long-Pro), `run_luna.yaml` (`run.yaml` on Luna), `run_sample_luna.yaml` (six of the forty on Luna, for a run of a day or two), `smoke*.yaml`, `launch.sh` | | [`prompt_only/`](prompt_only/) | The prompt-only template: one `ADAPTER.md` the built-in Codex Evolver revises after each task, with the method-neutral prompt. `run.yaml`, `run_pro.yaml`, `smoke.yaml`, `smoke_luna.yaml`, `launch.sh` | diff --git a/examples/libero/aspire/run.yaml b/examples/libero/aspire/run.yaml index e2a4b310..0f8450db 100644 --- a/examples/libero/aspire/run.yaml +++ b/examples/libero/aspire/run.yaml @@ -3,13 +3,14 @@ # (development 51-65, unseen-check 66-80, held-out 1-50, written into the task rows by launch.sh; every run # --init-mode seeded); Codex acting and coordinating, GLM 5.3 Flash. Started by launch.sh. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/libero.txt - val: ../utils/splits/verify.txt # the verifier's rows: run only by the coordinator, through run_agent - test: ../utils/splits/libero_rerun.txt # every task again with the final library: ASPIRE's backward transfer + tasks: ../utils/benchmark/tasks + view: ../../aspire_robosuite/utils/agent/aspire_fields.py:aspire_task # each task as the row it always was + train: ../utils/benchmark/splits/libero.txt + val: ../utils/benchmark/splits/verify.txt # the verifier's rows: run only by the coordinator, through run_agent + test: ../utils/benchmark/splits/libero_rerun.txt # every task again with the final library: ASPIRE's backward transfer step_size: 1 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: base: codex concurrency: 1 @@ -39,7 +40,7 @@ environment: # the task sandbox: the actor's code, ASPIRE - $ASPIRE_ROOT/.claude/skills/evosearch/skills/evosearch-iteration.md env: [ASPIRE_SIM_URL] # the host's simulator service (launch.sh): its 127.0.0.1 port is forwarded judge: - path: ../../aspire_robosuite/utils/judge.py # on the host: Stage 2 on the held-out seeds + path: ../../aspire_robosuite/utils/benchmark/judge.py # on the host: Stage 2 on the held-out seeds template: aspire evolver: name: aspire diff --git a/examples/libero/aspire/run_90.yaml b/examples/libero/aspire/run_90.yaml index d9e471c3..dacaf106 100644 --- a/examples/libero/aspire/run_90.yaml +++ b/examples/libero/aspire/run_90.yaml @@ -1,13 +1,14 @@ # ASPIRE on LIBERO-90: the ninety tasks of LIBERO's libero_90 suite, as run.yaml runs the other four (the same # seeds, roles and model). Started by launch.sh: ASPIRE_CONFIG=run_90.yaml ./launch.sh data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/libero_90.txt - val: ../utils/splits/verify.txt # the verifier's rows: run only by the coordinator, through run_agent - test: ../utils/splits/libero_90_rerun.txt # every task again with the final library: ASPIRE's backward transfer + tasks: ../utils/benchmark/tasks + view: ../../aspire_robosuite/utils/agent/aspire_fields.py:aspire_task # each task as the row it always was + train: ../utils/benchmark/splits/libero_90.txt + val: ../utils/benchmark/splits/verify.txt # the verifier's rows: run only by the coordinator, through run_agent + test: ../utils/benchmark/splits/libero_90_rerun.txt # every task again with the final library: ASPIRE's backward transfer step_size: 1 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: base: codex concurrency: 1 @@ -37,7 +38,7 @@ environment: # the task sandbox: the actor's code, ASPIRE - $ASPIRE_ROOT/.claude/skills/evosearch/skills/evosearch-iteration.md env: [ASPIRE_SIM_URL] # the host's simulator service (launch.sh): its 127.0.0.1 port is forwarded judge: - path: ../../aspire_robosuite/utils/judge.py # on the host: Stage 2 on the held-out seeds + path: ../../aspire_robosuite/utils/benchmark/judge.py # on the host: Stage 2 on the held-out seeds template: aspire evolver: name: aspire diff --git a/examples/libero/aspire/run_luna.yaml b/examples/libero/aspire/run_luna.yaml index ca0516a1..e297052a 100644 --- a/examples/libero/aspire/run_luna.yaml +++ b/examples/libero/aspire/run_luna.yaml @@ -1,13 +1,14 @@ # run.yaml with the models changed and nothing else: GPT-5.6-Luna at medium effort acts and coordinates on the # machine's own ChatGPT login (`codex login`; no API key). Started by launch.sh: ASPIRE_CONFIG=run_luna.yaml ./launch.sh data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/libero.txt - val: ../utils/splits/verify.txt # the verifier's rows: run only by the coordinator, through run_agent - test: ../utils/splits/libero_rerun.txt # every task again with the final library: ASPIRE's backward transfer + tasks: ../utils/benchmark/tasks + view: ../../aspire_robosuite/utils/agent/aspire_fields.py:aspire_task # each task as the row it always was + train: ../utils/benchmark/splits/libero.txt + val: ../utils/benchmark/splits/verify.txt # the verifier's rows: run only by the coordinator, through run_agent + test: ../utils/benchmark/splits/libero_rerun.txt # every task again with the final library: ASPIRE's backward transfer step_size: 1 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: base: codex concurrency: 1 @@ -37,7 +38,7 @@ environment: # the task sandbox: the actor's code, ASPIRE - $ASPIRE_ROOT/.claude/skills/evosearch/skills/evosearch-iteration.md env: [ASPIRE_SIM_URL] # the host's simulator service (launch.sh): its 127.0.0.1 port is forwarded judge: - path: ../../aspire_robosuite/utils/judge.py # on the host: Stage 2 on the held-out seeds + path: ../../aspire_robosuite/utils/benchmark/judge.py # on the host: Stage 2 on the held-out seeds template: aspire evolver: name: aspire diff --git a/examples/libero/aspire/run_pro.yaml b/examples/libero/aspire/run_pro.yaml index f5d1b8b3..62537701 100644 --- a/examples/libero/aspire/run_pro.yaml +++ b/examples/libero/aspire/run_pro.yaml @@ -4,13 +4,14 @@ # rows by launch.sh; every run --init-mode seeded); Codex acting and coordinating, GLM 5.3 Flash. Started by # launch.sh. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/libero_pro.txt - val: ../utils/splits/verify.txt # the verifier's rows: run only by the coordinator, through run_agent - test: ../utils/splits/libero_pro_rerun.txt # every task again with the final library: ASPIRE's backward transfer + tasks: ../utils/benchmark/tasks + view: ../../aspire_robosuite/utils/agent/aspire_fields.py:aspire_task # each task as the row it always was + train: ../utils/benchmark/splits/libero_pro.txt + val: ../utils/benchmark/splits/verify.txt # the verifier's rows: run only by the coordinator, through run_agent + test: ../utils/benchmark/splits/libero_pro_rerun.txt # every task again with the final library: ASPIRE's backward transfer step_size: 1 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: base: codex concurrency: 1 @@ -40,7 +41,7 @@ environment: # the task sandbox: the actor's code, ASPIRE - $ASPIRE_ROOT/.claude/skills/evosearch/skills/evosearch-iteration.md env: [ASPIRE_SIM_URL] # the host's simulator service (launch.sh): its 127.0.0.1 port is forwarded judge: - path: ../../aspire_robosuite/utils/judge.py # on the host: Stage 2 on the held-out seeds + path: ../../aspire_robosuite/utils/benchmark/judge.py # on the host: Stage 2 on the held-out seeds template: aspire evolver: name: aspire diff --git a/examples/libero/aspire/run_pro_long.yaml b/examples/libero/aspire/run_pro_long.yaml index d440f064..8964ba6b 100644 --- a/examples/libero/aspire/run_pro_long.yaml +++ b/examples/libero/aspire/run_pro_long.yaml @@ -1,13 +1,14 @@ # ASPIRE on LIBERO-Long-Pro: the twenty tasks of libero_10_swap and libero_10_task, as run_pro.yaml runs all eighty (the # same seeds, roles and model). Started by launch.sh: ASPIRE_CONFIG=run_pro_long.yaml ./launch.sh data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/libero_pro_long.txt - val: ../utils/splits/verify.txt # the verifier's rows: run only by the coordinator, through run_agent - test: ../utils/splits/libero_pro_long_rerun.txt # every task again with the final library: ASPIRE's backward transfer + tasks: ../utils/benchmark/tasks + view: ../../aspire_robosuite/utils/agent/aspire_fields.py:aspire_task # each task as the row it always was + train: ../utils/benchmark/splits/libero_pro_long.txt + val: ../utils/benchmark/splits/verify.txt # the verifier's rows: run only by the coordinator, through run_agent + test: ../utils/benchmark/splits/libero_pro_long_rerun.txt # every task again with the final library: ASPIRE's backward transfer step_size: 1 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: base: codex concurrency: 1 @@ -37,7 +38,7 @@ environment: # the task sandbox: the actor's code, ASPIRE - $ASPIRE_ROOT/.claude/skills/evosearch/skills/evosearch-iteration.md env: [ASPIRE_SIM_URL] # the host's simulator service (launch.sh): its 127.0.0.1 port is forwarded judge: - path: ../../aspire_robosuite/utils/judge.py # on the host: Stage 2 on the held-out seeds + path: ../../aspire_robosuite/utils/benchmark/judge.py # on the host: Stage 2 on the held-out seeds template: aspire evolver: name: aspire diff --git a/examples/libero/aspire/run_sample_luna.yaml b/examples/libero/aspire/run_sample_luna.yaml index dceec134..27164d2b 100644 --- a/examples/libero/aspire/run_sample_luna.yaml +++ b/examples/libero/aspire/run_sample_luna.yaml @@ -3,13 +3,14 @@ # ./launch.sh. Its baseline, the same tasks with ASPIRE's empty library: # ASPIRE_CONFIG=run_sample_luna.yaml ./launch.sh eval --adapter root data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/libero_sample.txt - val: ../utils/splits/verify.txt # the verifier's rows: run only by the coordinator, through run_agent - test: ../utils/splits/libero_sample_rerun.txt # every task again with an adapter given + tasks: ../utils/benchmark/tasks + view: ../../aspire_robosuite/utils/agent/aspire_fields.py:aspire_task # each task as the row it always was + train: ../utils/benchmark/splits/libero_sample.txt + val: ../utils/benchmark/splits/verify.txt # the verifier's rows: run only by the coordinator, through run_agent + test: ../utils/benchmark/splits/libero_sample_rerun.txt # every task again with an adapter given step_size: 1 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: base: codex concurrency: 1 @@ -39,7 +40,7 @@ environment: # the task sandbox: the actor's code, ASPIRE - $ASPIRE_ROOT/.claude/skills/evosearch/skills/evosearch-iteration.md env: [ASPIRE_SIM_URL] # the host's simulator service (launch.sh): its 127.0.0.1 port is forwarded judge: - path: ../../aspire_robosuite/utils/judge.py # on the host: Stage 2 on the held-out seeds + path: ../../aspire_robosuite/utils/benchmark/judge.py # on the host: Stage 2 on the held-out seeds template: aspire evolver: name: aspire diff --git a/examples/libero/aspire/smoke.yaml b/examples/libero/aspire/smoke.yaml index a23f383d..5648f30e 100644 --- a/examples/libero/aspire/smoke.yaml +++ b/examples/libero/aspire/smoke.yaml @@ -2,13 +2,14 @@ # (launch.sh writes them into the task rows: ASPIRE_SEEDS=smoke); Codex acting (ASPIRE's fix-loop subagent and # verifier prompts) and coordinating, GLM 5.3 Flash. Started by launch.sh: ASPIRE_CONFIG=smoke.yaml ./launch.sh data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/smoke.txt - val: ../utils/splits/verify.txt # the verifier's rows: run only by the coordinator, through run_agent - test: ../utils/splits/smoke_rerun.txt + tasks: ../utils/benchmark/tasks + view: ../../aspire_robosuite/utils/agent/aspire_fields.py:aspire_task # each task as the row it always was + train: ../utils/benchmark/splits/smoke.txt + val: ../utils/benchmark/splits/verify.txt # the verifier's rows: run only by the coordinator, through run_agent + test: ../utils/benchmark/splits/smoke_rerun.txt step_size: 1 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: base: codex concurrency: 1 @@ -38,7 +39,7 @@ environment: # the task sandbox: the actor's code, ASPIRE - $ASPIRE_ROOT/.claude/skills/evosearch/skills/evosearch-iteration.md env: [ASPIRE_SIM_URL] # the host's simulator service (launch.sh): its 127.0.0.1 port is forwarded judge: - path: ../../aspire_robosuite/utils/judge.py # on the host: Stage 2 on the held-out seeds + path: ../../aspire_robosuite/utils/benchmark/judge.py # on the host: Stage 2 on the held-out seeds template: aspire evolver: name: aspire diff --git a/examples/libero/aspire/smoke_luna.yaml b/examples/libero/aspire/smoke_luna.yaml index 305a86da..2f746c60 100644 --- a/examples/libero/aspire/smoke_luna.yaml +++ b/examples/libero/aspire/smoke_luna.yaml @@ -2,13 +2,14 @@ # machine's own ChatGPT login (`codex login`; no API key). Started by launch.sh: ASPIRE_CONFIG=smoke_luna.yaml # ./launch.sh (its seeds are the smoke's). data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/smoke.txt - val: ../utils/splits/verify.txt # the verifier's rows: run only by the coordinator, through run_agent - test: ../utils/splits/smoke_rerun.txt + tasks: ../utils/benchmark/tasks + view: ../../aspire_robosuite/utils/agent/aspire_fields.py:aspire_task # each task as the row it always was + train: ../utils/benchmark/splits/smoke.txt + val: ../utils/benchmark/splits/verify.txt # the verifier's rows: run only by the coordinator, through run_agent + test: ../utils/benchmark/splits/smoke_rerun.txt step_size: 1 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: base: codex concurrency: 1 @@ -38,7 +39,7 @@ environment: # the task sandbox: the actor's code, ASPIRE - $ASPIRE_ROOT/.claude/skills/evosearch/skills/evosearch-iteration.md env: [ASPIRE_SIM_URL] # the host's simulator service (launch.sh): its 127.0.0.1 port is forwarded judge: - path: ../../aspire_robosuite/utils/judge.py # on the host: Stage 2 on the held-out seeds + path: ../../aspire_robosuite/utils/benchmark/judge.py # on the host: Stage 2 on the held-out seeds template: aspire evolver: name: aspire diff --git a/examples/libero/aspire/smoke_pro.yaml b/examples/libero/aspire/smoke_pro.yaml index 80c4aef7..638e5f0b 100644 --- a/examples/libero/aspire/smoke_pro.yaml +++ b/examples/libero/aspire/smoke_pro.yaml @@ -2,13 +2,14 @@ # (launch.sh writes them into the task rows: ASPIRE_SEEDS=smoke); Codex acting (ASPIRE's fix-loop subagent and # verifier prompts) and coordinating, GLM 5.3 Flash. Started by launch.sh: ASPIRE_CONFIG=smoke_pro.yaml ./launch.sh data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/smoke_pro.txt - val: ../utils/splits/verify.txt # the verifier's rows: run only by the coordinator, through run_agent - test: ../utils/splits/smoke_pro_rerun.txt + tasks: ../utils/benchmark/tasks + view: ../../aspire_robosuite/utils/agent/aspire_fields.py:aspire_task # each task as the row it always was + train: ../utils/benchmark/splits/smoke_pro.txt + val: ../utils/benchmark/splits/verify.txt # the verifier's rows: run only by the coordinator, through run_agent + test: ../utils/benchmark/splits/smoke_pro_rerun.txt step_size: 1 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: base: codex concurrency: 1 @@ -38,7 +39,7 @@ environment: # the task sandbox: the actor's code, ASPIRE - $ASPIRE_ROOT/.claude/skills/evosearch/skills/evosearch-iteration.md env: [ASPIRE_SIM_URL] # the host's simulator service (launch.sh): its 127.0.0.1 port is forwarded judge: - path: ../../aspire_robosuite/utils/judge.py # on the host: Stage 2 on the held-out seeds + path: ../../aspire_robosuite/utils/benchmark/judge.py # on the host: Stage 2 on the held-out seeds template: aspire evolver: name: aspire diff --git a/examples/libero/aspire/smoke_pro_luna.yaml b/examples/libero/aspire/smoke_pro_luna.yaml index 6fc584c3..6e1e381f 100644 --- a/examples/libero/aspire/smoke_pro_luna.yaml +++ b/examples/libero/aspire/smoke_pro_luna.yaml @@ -2,13 +2,14 @@ # machine's own ChatGPT login (`codex login`; no API key). Started by launch.sh: ASPIRE_CONFIG=smoke_pro_luna.yaml # ./launch.sh (its seeds are the smoke's). data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/smoke_pro.txt - val: ../utils/splits/verify.txt # the verifier's rows: run only by the coordinator, through run_agent - test: ../utils/splits/smoke_pro_rerun.txt + tasks: ../utils/benchmark/tasks + view: ../../aspire_robosuite/utils/agent/aspire_fields.py:aspire_task # each task as the row it always was + train: ../utils/benchmark/splits/smoke_pro.txt + val: ../utils/benchmark/splits/verify.txt # the verifier's rows: run only by the coordinator, through run_agent + test: ../utils/benchmark/splits/smoke_pro_rerun.txt step_size: 1 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: base: codex concurrency: 1 @@ -38,7 +39,7 @@ environment: # the task sandbox: the actor's code, ASPIRE - $ASPIRE_ROOT/.claude/skills/evosearch/skills/evosearch-iteration.md env: [ASPIRE_SIM_URL] # the host's simulator service (launch.sh): its 127.0.0.1 port is forwarded judge: - path: ../../aspire_robosuite/utils/judge.py # on the host: Stage 2 on the held-out seeds + path: ../../aspire_robosuite/utils/benchmark/judge.py # on the host: Stage 2 on the held-out seeds template: aspire evolver: name: aspire diff --git a/examples/libero/prompt_only/run.yaml b/examples/libero/prompt_only/run.yaml index 6b726af9..f8668473 100644 --- a/examples/libero/prompt_only/run.yaml +++ b/examples/libero/prompt_only/run.yaml @@ -3,12 +3,13 @@ # seeds (development 51-65, unseen-check 66-80, held-out 1-50); Codex acting on the method-neutral prompt, GLM 5.3 # Flash. Started by launch.sh. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/libero.txt - test: ../utils/splits/libero_rerun.txt # `launch.sh eval`: the tasks again, with an adapter given + tasks: ../utils/benchmark/tasks + view: ../../aspire_robosuite/utils/agent/aspire_fields.py:aspire_task # each task as the row it always was + train: ../utils/benchmark/splits/libero.txt + test: ../utils/benchmark/splits/libero_rerun.txt # `launch.sh eval`: the tasks again, with an adapter given step_size: 1 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: base: codex concurrency: 1 @@ -36,7 +37,7 @@ environment: # the task sandbox: the engine's code, the s - $ASPIRE_ROOT/.claude/skills/iterative-debugging/skills/task-exploration.md env: [ASPIRE_SIM_URL] # the host's simulator service (launch.sh): its 127.0.0.1 port is forwarded judge: - path: ../../aspire_robosuite/utils/judge.py # on the host: the program once on the held-out seeds + path: ../../aspire_robosuite/utils/benchmark/judge.py # on the host: the program once on the held-out seeds template: prompt-only@1.0 evolver: name: codex # the built-in Codex AgentEvolver: it revises ADAPTER.md after each task diff --git a/examples/libero/prompt_only/run_pro.yaml b/examples/libero/prompt_only/run_pro.yaml index 77e76157..f62382b6 100644 --- a/examples/libero/prompt_only/run_pro.yaml +++ b/examples/libero/prompt_only/run_pro.yaml @@ -2,12 +2,13 @@ # runs LIBERO's forty (the same seeds, template and model). Started by launch.sh: ASPIRE_CONFIG=run_pro.yaml # ./launch.sh data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/libero_pro.txt - test: ../utils/splits/libero_pro_rerun.txt # `launch.sh eval`: the tasks again, with an adapter given + tasks: ../utils/benchmark/tasks + view: ../../aspire_robosuite/utils/agent/aspire_fields.py:aspire_task # each task as the row it always was + train: ../utils/benchmark/splits/libero_pro.txt + test: ../utils/benchmark/splits/libero_pro_rerun.txt # `launch.sh eval`: the tasks again, with an adapter given step_size: 1 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: base: codex concurrency: 1 @@ -35,7 +36,7 @@ environment: # the task sandbox: the engine's code, the s - $ASPIRE_ROOT/.claude/skills/iterative-debugging/skills/task-exploration.md env: [ASPIRE_SIM_URL] # the host's simulator service (launch.sh): its 127.0.0.1 port is forwarded judge: - path: ../../aspire_robosuite/utils/judge.py # on the host: the program once on the held-out seeds + path: ../../aspire_robosuite/utils/benchmark/judge.py # on the host: the program once on the held-out seeds template: prompt-only@1.0 evolver: name: codex # the built-in Codex AgentEvolver: it revises ADAPTER.md after each task diff --git a/examples/libero/prompt_only/smoke.yaml b/examples/libero/prompt_only/smoke.yaml index 293db9dd..83c28c19 100644 --- a/examples/libero/prompt_only/smoke.yaml +++ b/examples/libero/prompt_only/smoke.yaml @@ -3,12 +3,13 @@ # Evolver revises after each task; Codex acting on the method-neutral prompt, GLM 5.3 Flash. Started by launch.sh: # ASPIRE_CONFIG=smoke.yaml ./launch.sh data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/smoke.txt - test: ../utils/splits/smoke_rerun.txt # `launch.sh eval`: the tasks again, with an adapter given + tasks: ../utils/benchmark/tasks + view: ../../aspire_robosuite/utils/agent/aspire_fields.py:aspire_task # each task as the row it always was + train: ../utils/benchmark/splits/smoke.txt + test: ../utils/benchmark/splits/smoke_rerun.txt # `launch.sh eval`: the tasks again, with an adapter given step_size: 1 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: base: codex concurrency: 1 @@ -36,7 +37,7 @@ environment: # the task sandbox: the engine's code, the s - $ASPIRE_ROOT/.claude/skills/iterative-debugging/skills/task-exploration.md env: [ASPIRE_SIM_URL] # the host's simulator service (launch.sh): its 127.0.0.1 port is forwarded judge: - path: ../../aspire_robosuite/utils/judge.py # on the host: the program once on the held-out seeds + path: ../../aspire_robosuite/utils/benchmark/judge.py # on the host: the program once on the held-out seeds template: prompt-only@1.0 evolver: name: codex # the built-in Codex AgentEvolver: it revises ADAPTER.md after each task diff --git a/examples/libero/prompt_only/smoke_luna.yaml b/examples/libero/prompt_only/smoke_luna.yaml index ff5ccafb..527e6d2c 100644 --- a/examples/libero/prompt_only/smoke_luna.yaml +++ b/examples/libero/prompt_only/smoke_luna.yaml @@ -2,12 +2,13 @@ # machine's own ChatGPT login (`codex login`; no API key). Started by launch.sh: ASPIRE_CONFIG=smoke_luna.yaml # ./launch.sh (its seeds are the smoke's). data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/smoke.txt - test: ../utils/splits/smoke_rerun.txt # `launch.sh eval`: the tasks again, with an adapter given + tasks: ../utils/benchmark/tasks + view: ../../aspire_robosuite/utils/agent/aspire_fields.py:aspire_task # each task as the row it always was + train: ../utils/benchmark/splits/smoke.txt + test: ../utils/benchmark/splits/smoke_rerun.txt # `launch.sh eval`: the tasks again, with an adapter given step_size: 1 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: base: codex concurrency: 1 @@ -35,7 +36,7 @@ environment: # the task sandbox: the engine's code, the s - $ASPIRE_ROOT/.claude/skills/iterative-debugging/skills/task-exploration.md env: [ASPIRE_SIM_URL] # the host's simulator service (launch.sh): its 127.0.0.1 port is forwarded judge: - path: ../../aspire_robosuite/utils/judge.py # on the host: the program once on the held-out seeds + path: ../../aspire_robosuite/utils/benchmark/judge.py # on the host: the program once on the held-out seeds template: prompt-only@1.0 evolver: name: codex # the built-in Codex AgentEvolver: it revises ADAPTER.md after each task diff --git a/examples/libero/utils/.gitignore b/examples/libero/utils/.gitignore index bc46a5ce..d006f309 100644 --- a/examples/libero/utils/.gitignore +++ b/examples/libero/utils/.gitignore @@ -1,3 +1,5 @@ +aspire-docs/ +runs/ +# Where utils/prepare.py wrote before utils/benchmark/: kept out of git in a checkout prepared then tasks.jsonl splits/ -aspire-docs/ diff --git a/examples/libero/utils/benchmark/.gitignore b/examples/libero/utils/benchmark/.gitignore new file mode 100644 index 00000000..d296b16f --- /dev/null +++ b/examples/libero/utils/benchmark/.gitignore @@ -0,0 +1,2 @@ +tasks/ +splits/ diff --git a/examples/libero/utils/prepare.py b/examples/libero/utils/benchmark/prepare.py similarity index 85% rename from examples/libero/utils/prepare.py rename to examples/libero/utils/benchmark/prepare.py index 250f0b43..35136984 100644 --- a/examples/libero/utils/prepare.py +++ b/examples/libero/utils/benchmark/prepare.py @@ -1,11 +1,12 @@ """Write LIBERO's tasks (`cap-harness --benchmark libero-pro`, the suites of the LIBERO it pins) as hermit tasks: one -row per (suite, task id) with its seeds, which the simulator service and the judge take from the row (ASPIRE's +task per (suite, task id) with its seeds, which the simulator service and the judge take from its row (ASPIRE's LIBERO defaults: development 51-65, unseen-check 66-80, held-out 1-50; every run with `--init-mode seeded`, which the engine sets, so that a seed is a scene of its own). python prepare.py # ASPIRE's seeds python prepare.py --preset smoke # the smoke's: three development seeds, one unseen, three held-out - # tasks.jsonl, splits/{all,libero,libero_90,libero_pro,libero_pro_long,,smoke,smoke_pro,libero_sample}.txt, + # tasks// (task directories), + # splits/{all,libero,libero_90,libero_pro,libero_pro_long,,smoke,smoke_pro,libero_sample}.txt, # their *_rerun.txt (rerun.txt for all), verify.txt LIBERO's own five suites (Liu et al., 2023): `libero_object`, `libero_spatial`, `libero_goal` and `libero_10` @@ -17,8 +18,8 @@ splits/libero_pro_long.txt for LIBERO-Long-Pro (`libero_10_swap`, `libero_10_task`). A suite and its perturbations share task ids, so a row's id carries the suite. -The rows, the verifier's and the reruns are written by the Robosuite example's prepare.py (`write`), whose engine -this benchmark runs on. +The tasks, the verifier's and the reruns are written by the Robosuite example's prepare.py (`write`), whose engine +this benchmark runs on: one task directory each, read back as its row by its utils/agent/aspire_fields.py. """ from __future__ import annotations @@ -26,7 +27,7 @@ import sys HERE = Path(__file__).resolve().parent -sys.path.insert(0, str(HERE.parents[1] / "aspire_robosuite" / "utils")) +sys.path.insert(0, str(HERE.parents[2] / "aspire_robosuite" / "utils" / "benchmark")) from prepare import seeds_from, write # noqa: E402 BENCHMARK = "libero-pro" # cap-harness's name for its LIBERO adapter diff --git a/examples/libero/utils/task-info/README.md b/examples/libero/utils/benchmark/task-info/README.md similarity index 100% rename from examples/libero/utils/task-info/README.md rename to examples/libero/utils/benchmark/task-info/README.md diff --git a/examples/libero/utils/launch.sh b/examples/libero/utils/launch.sh index 141362b0..1bcd66fd 100755 --- a/examples/libero/utils/launch.sh +++ b/examples/libero/utils/launch.sh @@ -10,7 +10,7 @@ # ./launch.sh eval --adapter root # the same with the empty one # # The engine is the Robosuite example's (../../aspire_robosuite): its launcher runs the method folder's configs, with -# the seeds of utils/prepare.py, and takes the same variables (ASPIRE_SIM_ENV, ASPIRE_SIM_GPU, ASPIRE_OUT, +# the seeds of utils/benchmark/prepare.py, and takes the same variables (ASPIRE_SIM_ENV, ASPIRE_SIM_GPU, ASPIRE_OUT, # ASPIRE_EVAL_SEEDS, ASPIRE_SIM_PORT, HERMIT_PYTHON; see its header). ASPIRE_ROOT needs .venv-libero. set -euo pipefail HERE="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" diff --git a/examples/officeqa/README.md b/examples/officeqa/README.md index 33d75ffc..ffe42019 100644 --- a/examples/officeqa/README.md +++ b/examples/officeqa/README.md @@ -27,11 +27,11 @@ harness also needs Docker, set up once as in the [recipe](../../docs/recipes/off ```bash python -m pip install -e . --group skillopt --group gepa -python examples/officeqa/utils/prepare.py --csv /path/to/officeqa_full.csv --docs /path/to/treasury_bulletins_parsed +python examples/officeqa/utils/benchmark/prepare.py --csv /path/to/officeqa_full.csv --docs /path/to/treasury_bulletins_parsed ``` -`prepare.py` writes the questions, the paper's split, SkillOpt's initial skill (`utils/seed_adapter/`) and a link to the -corpus (`utils/documents`, what the agent's sandbox is granted) to `utils/`. +`prepare.py` writes the questions, the paper's split, SkillOpt's initial skill (`utils/benchmark/seed_adapter/`) and a link to the +corpus (`utils/benchmark/documents`, what the agent's sandbox is granted) to `utils/`. ## Run @@ -41,7 +41,7 @@ hermit train --config examples/officeqa/skillopt/run.yaml # SkillOpt hermit eval --config examples/officeqa/gepa/run.yaml --adapter final # GEPA's skill hermit eval --config examples/officeqa/skillopt/run.yaml --adapter final # SkillOpt's skill hermit eval --config examples/officeqa/skillopt/run.yaml --adapter none # no skill -hermit eval --config examples/officeqa/skillopt/run.yaml --adapter ../utils/seed_adapter # initial skill +hermit eval --config examples/officeqa/skillopt/run.yaml --adapter ../utils/benchmark/seed_adapter # initial skill FIREWORKS_API_KEY=... WM_PORT=3601 examples/officeqa/reharness/launch.sh run # ReHarness: the 50 as one stream FIREWORKS_API_KEY=... WM_PORT=3601 examples/officeqa/reharness/launch.sh run eval # its bank on the 172, three passes FIREWORKS_API_KEY=... WM_PORT=3601 examples/officeqa/reharness/launch.sh stream # the paper's protocol: the 172 as one stream @@ -83,7 +83,7 @@ hermit train --config examples/officeqa/skillopt/run_codex_luna.yaml hermit eval --config examples/officeqa/gepa/run_codex_luna.yaml --adapter final # GEPA's skill hermit eval --config examples/officeqa/skillopt/run_codex_luna.yaml --adapter final # SkillOpt's skill hermit eval --config examples/officeqa/skillopt/run_codex_luna.yaml --adapter none # no skill -hermit eval --config examples/officeqa/skillopt/run_codex_luna.yaml --adapter ../utils/seed_adapter # initial skill +hermit eval --config examples/officeqa/skillopt/run_codex_luna.yaml --adapter ../utils/benchmark/seed_adapter # initial skill python examples/officeqa/continual_harness/make_seed.py # Continual Harness hermit train --config examples/officeqa/continual_harness/stream_codex_luna.yaml # the 172 as one stream FIREWORKS_API_KEY=... WM_PORT=3601 examples/officeqa/reharness/launch.sh stream_codex_luna # ReHarness (embeddings on Fireworks) @@ -244,11 +244,14 @@ utils/task.py OfficeQA's task functions for the framework's Codex agent (t Code's image utils/harbor/ the Codex harness through Harbor: build_tasks.py writes one Harbor task per question (private, with the answers in tests/), judge.py is the exact-match judge run inside the container; - gepa/run_codex_harbor.yaml runs it (agent.name: harbor; recipe: "Through Harbor") + gepa/run_codex_harbor.yaml runs it (environment.harbor; recipe: "Through Harbor") utils/tests/ test_harbor_judge.py: utils/harbor/judge.py finds the of the final message in either - Codex stream a trial leaves in /logs/agent (HermitCodex's, Harbor's own Codex agent's) -utils/prepare.py tasks.jsonl, the splits, data.json and seed_adapter/ utils/judge.py SkillOpt's exact match -utils/task-info/ the task for the learner + Codex stream a task leaves in /logs/agent (Hermit's Codex agent's, Harbor's own Codex agent's) +utils/benchmark/ the task set: prepare.py writes tasks/ (one task directory per question, Harbor's format), the + splits, data.json and seed_adapter/; judge.py is SkillOpt's exact match (on the host); + task-info/ is the task for the learner +utils/agent/ OfficeQAAgent (SkillOpt's direct-chat rollout) and officeqa_fields.py (a task directory's row + as the row the agents read) gepa/ skillopt/ smoke.yaml and run.yaml per method, smoke_codex.yaml and run_codex.yaml for the Codex harness, run_codex_flash.yaml and run_codex_luna.yaml for the first two columns of Results, and skillopt/run_claude_haiku.yaml for the third (its no-skill evaluation); @@ -273,6 +276,6 @@ rrsi_adapter/ RRSI Adapter on the GPT-5.6-Luna column: run_luna.yaml and s upstream's), trials.py and make_rows.py (two trials per training question as rows #k), make_seed.py opro/ OPRO on the prompt-only template: run_codex_luna.yaml and smoke_codex_luna.yaml in the - GPT-5.6-Luna setting, starting from utils/seed_adapter/; adapter/ is the measured run's final + GPT-5.6-Luna setting, starting from utils/benchmark/seed_adapter/; adapter/ is the measured run's final instruction, with its .hermit/CARD.md ``` diff --git a/examples/officeqa/ace/adapter/.hermit/CARD.md b/examples/officeqa/ace/adapter/.hermit/CARD.md index f6ed2cf6..b844b3ab 100644 --- a/examples/officeqa/ace/adapter/.hermit/CARD.md +++ b/examples/officeqa/ace/adapter/.hermit/CARD.md @@ -3,7 +3,7 @@ - Template: `ace@1.0` - Started from: `0cf6d3ec6911d9682df1309ca98c1bccd9234b73` - Evolver: `ace` -- Task: ../utils/task-info/ +- Task: ../utils/benchmark/task-info/ - Training: 2 epoch(s), 100 step(s) - Spending: rollouts $0.0000, Evolver $0.0000 (+386 calls of unknown cost); 1.45 h diff --git a/examples/officeqa/ace/make_seed.py b/examples/officeqa/ace/make_seed.py index 4552639e..fab49a48 100644 --- a/examples/officeqa/ace/make_seed.py +++ b/examples/officeqa/ace/make_seed.py @@ -1,8 +1,8 @@ #!/usr/bin/env python3 -"""Write seed_adapter/ beside this file: SkillOpt's initial OfficeQA skill (../utils/seed_adapter/ADAPTER.md, written -by ../utils/prepare.py) as an `ace` adapter: the skill text first, unchanged, then ACE's generator instructions and -its empty playbook. ACE then starts from exactly the text the GEPA and SkillOpt runs start from, plus an empty -playbook. +"""Write seed_adapter/ beside this file: SkillOpt's initial OfficeQA skill (../utils/benchmark/seed_adapter/ADAPTER.md, +written by ../utils/benchmark/prepare.py) as an `ace` adapter: the skill text first, unchanged, then ACE's generator +instructions and its empty playbook. ACE then starts from exactly the text the GEPA and SkillOpt runs start from, plus +an empty playbook. python examples/officeqa/ace/make_seed.py """ @@ -15,7 +15,7 @@ HERE = Path(__file__).resolve().parent TEMPLATE = Path(hermit.__file__).resolve().parent / "adapter" / "templates" / "ace" -skill = (HERE.parent / "utils" / "seed_adapter" / "ADAPTER.md").read_text() +skill = (HERE.parent / "utils" / "benchmark" / "seed_adapter" / "ADAPTER.md").read_text() out = HERE / "seed_adapter" if out.exists(): shutil.rmtree(out) diff --git a/examples/officeqa/ace/run_codex_luna.yaml b/examples/officeqa/ace/run_codex_luna.yaml index 76156879..34298bbb 100644 --- a/examples/officeqa/ace/run_codex_luna.yaml +++ b/examples/officeqa/ace/run_codex_luna.yaml @@ -3,13 +3,14 @@ # login (`login: chatgpt`), and the Reflector and the Curator are the same model, one Codex session per chat call # (`llm: {cli: codex}`). The image and the egress proxy: the recipe's "The Flash row with GPT-5.6-Luna". data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/train.txt - val: ../utils/splits/val.txt - test: ../utils/splits/test.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: ../utils/benchmark/splits/train.txt + val: ../utils/benchmark/splits/val.txt + test: ../utils/benchmark/splits/test.txt step_size: 1 # upstream's per-sample loop shuffle_seed: 42 -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: base: codex # Layer 2: Codex as a Base Agent (hermit/agent/base/cli.py) concurrency: 4 @@ -29,9 +30,9 @@ agent: project_doc_max_bytes: 409600 # 400 KiB, the ace template's limit: the skill with a growing playbook; past # Codex's default 32 KiB it would cut AGENTS.md (deploy refuses that) judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: ace@1.0 -start_from: seed_adapter # make_seed.py writes it from ../utils/seed_adapter +start_from: seed_adapter # make_seed.py writes it from ../utils/benchmark/seed_adapter evolver: name: ace params: diff --git a/examples/officeqa/ace/smoke_codex_luna.yaml b/examples/officeqa/ace/smoke_codex_luna.yaml index 9192e0d5..9e996617 100644 --- a/examples/officeqa/ace/smoke_codex_luna.yaml +++ b/examples/officeqa/ace/smoke_codex_luna.yaml @@ -1,13 +1,14 @@ # Smoke of run_codex_luna.yaml: 4 training questions one at a time, 2 validation questions, 4 test questions, one # epoch, one retry round; shows that ACE's levers work (bullets named, counters moving, bullets added). data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/smoke_train.txt - val: ../utils/splits/smoke_val.txt - test: ../utils/splits/smoke_test.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: ../utils/benchmark/splits/smoke_train.txt + val: ../utils/benchmark/splits/smoke_val.txt + test: ../utils/benchmark/splits/smoke_test.txt step_size: 1 # upstream's per-sample loop shuffle_seed: 42 -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: base: codex # Layer 2: Codex as a Base Agent (hermit/agent/base/cli.py) concurrency: 2 @@ -27,9 +28,9 @@ agent: project_doc_max_bytes: 409600 # 400 KiB, the ace template's limit: the skill with a growing playbook; past # Codex's default 32 KiB it would cut AGENTS.md (deploy refuses that) judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: ace@1.0 -start_from: seed_adapter # make_seed.py writes it from ../utils/seed_adapter +start_from: seed_adapter # make_seed.py writes it from ../utils/benchmark/seed_adapter evolver: name: ace params: diff --git a/examples/officeqa/ahe_adapter/README.md b/examples/officeqa/ahe_adapter/README.md index 2858e258..fd7eec2e 100644 --- a/examples/officeqa/ahe_adapter/README.md +++ b/examples/officeqa/ahe_adapter/README.md @@ -21,7 +21,7 @@ with the machine's standalone Codex, and `oq-egress-login` on the `oq-sandbox` n and auth.openai.com). Then, from the repository root: ```bash -python examples/officeqa/utils/prepare.py --csv ... --docs ... # once (../README.md) +python examples/officeqa/utils/benchmark/prepare.py --csv ... --docs ... # once (../README.md) python examples/officeqa/ahe_adapter/make_rows.py # rows/: k = 2 trial rows python examples/officeqa/ahe_adapter/make_seed.py # seed_adapter/: H_0 hermit check --config examples/officeqa/ahe_adapter/smoke_codex_luna.yaml diff --git a/examples/officeqa/ahe_adapter/make_rows.py b/examples/officeqa/ahe_adapter/make_rows.py index baa9f5be..31869261 100644 --- a/examples/officeqa/ahe_adapter/make_rows.py +++ b/examples/officeqa/ahe_adapter/make_rows.py @@ -1,23 +1,27 @@ #!/usr/bin/env python3 -"""Write rows/ beside this file: tasks.jsonl (every question's own row of ../utils/tasks.jsonl, then each training +"""Write rows/ beside this file: tasks.jsonl (every question's own row of ../utils/benchmark/tasks, then each training question k times as `#`, interleaved by trial: every question's first rollout, then every question's second) -and train_k.txt / smoke_k.txt (those trial rows for ../utils/splits/train.txt and, when it is there, +and train_k.txt / smoke_k.txt (those trial rows for ../utils/benchmark/splits/train.txt and, when it is there, smoke_train.txt). AHE evolves on k rollouts of every question per iteration (the paper's k = 2): a trial row is the question's own row under the id of its j-th rollout, with `uid` (the question's id, by which the Evolver groups the k trials: evolver.params.task_field) and `rollout` (j). The validation and test rows are the questions' own, unchanged. - python examples/officeqa/ahe_adapter/make_rows.py [--k 2] # after ../utils/prepare.py + python examples/officeqa/ahe_adapter/make_rows.py [--k 2] # after ../utils/benchmark/prepare.py """ import argparse import json from pathlib import Path +from hermit.common import load_callable from hermit.task import read_ids HERE = Path(__file__).resolve().parent UTILS = HERE.parent / "utils" +question_rows = load_callable(str(UTILS / "agent" / "officeqa_fields.py") + ":question_rows", UTILS) + + def trials(ids: list[str], k: int) -> list[str]: """`#` for j = 1..k, by trial.""" return [f"{uid}#{j}" for j in range(1, k + 1) for uid in ids] @@ -26,12 +30,13 @@ def trials(ids: list[str], k: int) -> list[str]: def main(argv=None) -> None: parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) parser.add_argument("--k", type=int, default=2, help="rollouts of each training question per iteration") - parser.add_argument("--tasks", type=Path, default=UTILS / "tasks.jsonl") - parser.add_argument("--splits", type=Path, default=UTILS / "splits") + parser.add_argument("--tasks", type=Path, default=UTILS / "benchmark" / "tasks") + parser.add_argument("--splits", type=Path, default=UTILS / "benchmark" / "splits") parser.add_argument("--out", type=Path, default=HERE / "rows") args = parser.parse_args(argv) - lines = [line for line in args.tasks.read_text(encoding="utf-8").splitlines() if line.strip()] - by_id = {row["id"]: row for row in map(json.loads, lines)} + questions = question_rows(args.tasks, args.splits) + lines = [json.dumps(row, ensure_ascii=False) for row in questions] + by_id = {row["id"]: row for row in questions} splits = {"train": args.splits / "train.txt", "smoke": args.splits / "smoke_train.txt"} names = {} for name, path in splits.items(): diff --git a/examples/officeqa/ahe_adapter/make_seed.py b/examples/officeqa/ahe_adapter/make_seed.py index 86046faa..149315ef 100644 --- a/examples/officeqa/ahe_adapter/make_seed.py +++ b/examples/officeqa/ahe_adapter/make_seed.py @@ -1,8 +1,9 @@ #!/usr/bin/env python3 """Write seed_adapter/ beside this file: the `ahe` seed harness (the paper's seed system prompt, an empty -long-term memory, the seed middleware) with SkillOpt's initial OfficeQA skill (../utils/seed_adapter/ADAPTER.md, -written by ../utils/prepare.py) appended to its system prompt, so H_0 holds the same text the GEPA, SkillOpt, -Harness², Continual Harness and Meta-Harness Adapter runs start from. AGENTS.md and its copy ADAPTER.md carry it. +long-term memory, the seed middleware) with SkillOpt's initial OfficeQA skill +(../utils/benchmark/seed_adapter/ADAPTER.md, written by ../utils/benchmark/prepare.py) appended to its system prompt, so +H_0 holds the same text the GEPA, SkillOpt, Harness², Continual Harness and Meta-Harness Adapter runs start from. +AGENTS.md and its copy ADAPTER.md carry it. python examples/officeqa/ahe_adapter/make_seed.py """ @@ -14,7 +15,7 @@ HERE = Path(__file__).resolve().parent TEMPLATE = Path(hermit.__file__).resolve().parent / "adapter" / "templates" / "ahe" -skill = (HERE.parent / "utils" / "seed_adapter" / "ADAPTER.md").read_text(encoding="utf-8").strip() +skill = (HERE.parent / "utils" / "benchmark" / "seed_adapter" / "ADAPTER.md").read_text(encoding="utf-8").strip() out = HERE / "seed_adapter" if out.exists(): shutil.rmtree(out) diff --git a/examples/officeqa/ahe_adapter/run_codex_luna.yaml b/examples/officeqa/ahe_adapter/run_codex_luna.yaml index d9675c89..35ac783a 100644 --- a/examples/officeqa/ahe_adapter/run_codex_luna.yaml +++ b/examples/officeqa/ahe_adapter/run_codex_luna.yaml @@ -16,10 +16,10 @@ data: tasks: rows/tasks.jsonl # make_rows.py: own rows, then k = 2 trial rows per training question train: rows/train_k2.txt # 50 questions x k = 2, by trial - test: ../utils/splits/test.txt + test: ../utils/benchmark/splits/test.txt step_size: 100 # one step = one AHE iteration over every trial shuffle: false -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: base: codex # Layer 2: Codex as a Base Agent (hermit/agent/base/cli.py) concurrency: 8 @@ -39,7 +39,7 @@ agent: project_doc_max_bytes: 65536 # 64 KiB: the skill (up to 40 KB) with the deployment's notes; past # Codex's default 32 KiB it would cut AGENTS.md (deploy refuses that) judge: - path: trials.py # the question's score (../utils/judge.py) + path: trials.py # the question's score (../utils/benchmark/judge.py) template: ahe@1.0 start_from: seed_adapter # make_seed.py: H_0 evolver: diff --git a/examples/officeqa/ahe_adapter/smoke_codex_luna.yaml b/examples/officeqa/ahe_adapter/smoke_codex_luna.yaml index 0e239bf8..f5db6c6d 100644 --- a/examples/officeqa/ahe_adapter/smoke_codex_luna.yaml +++ b/examples/officeqa/ahe_adapter/smoke_codex_luna.yaml @@ -7,10 +7,10 @@ data: tasks: rows/tasks.jsonl # make_rows.py: own rows, then k = 2 trial rows per training question train: rows/smoke_k2.txt # 4 questions x k = 2, by trial - test: ../utils/splits/smoke_test.txt + test: ../utils/benchmark/splits/smoke_test.txt step_size: 8 # one step = one AHE iteration over every trial shuffle: false -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: base: codex # Layer 2: Codex as a Base Agent (hermit/agent/base/cli.py) concurrency: 4 @@ -30,7 +30,7 @@ agent: project_doc_max_bytes: 65536 # 64 KiB: the skill (up to 40 KB) with the deployment's notes; past # Codex's default 32 KiB it would cut AGENTS.md (deploy refuses that) judge: - path: trials.py # the question's score (../utils/judge.py) + path: trials.py # the question's score (../utils/benchmark/judge.py) template: ahe@1.0 start_from: seed_adapter # make_seed.py: H_0 evolver: diff --git a/examples/officeqa/ahe_adapter/trials.py b/examples/officeqa/ahe_adapter/trials.py index 9d21ce25..af5f6ca2 100644 --- a/examples/officeqa/ahe_adapter/trials.py +++ b/examples/officeqa/ahe_adapter/trials.py @@ -8,7 +8,7 @@ answer: ../utils/task.py:answer # unchanged: the last ... collect: trials.py:collect # ../utils/task.py:collect, then the adapter's hook logs judge: - path: trials.py # score: ../utils/judge.py + path: trials.py # score: ../utils/benchmark/judge.py The adapter's hook runner (hermit/adapter/templates/ahe/hook_runner.py) logs every hook call to `.hermit-hooks/hook-calls.jsonl` under the session's workspace and each session's state to `.hermit-hooks/hook-state/`; @@ -68,5 +68,5 @@ def collect(task: dict, workspace: Path, log: Path) -> dict: def score(task: dict, output) -> dict: - """The question's score against its reference answer (../utils/judge.py:score).""" - return utils("judge.py:score")(question(task), output) + """The question's score against its reference answer (../utils/benchmark/judge.py:score).""" + return utils("benchmark/judge.py:score")(question(task), output) diff --git a/examples/officeqa/continual_harness/make_seed.py b/examples/officeqa/continual_harness/make_seed.py index e09b4ff1..88e3ff12 100644 --- a/examples/officeqa/continual_harness/make_seed.py +++ b/examples/officeqa/continual_harness/make_seed.py @@ -1,8 +1,8 @@ #!/usr/bin/env python3 -"""Write seed_adapter/ beside this file: SkillOpt's initial OfficeQA skill (../utils/seed_adapter/ADAPTER.md, written -by ../utils/prepare.py) as a `continual-harness` adapter: the skill text is AGENTS.md (the base policy p; ADAPTER.md -is its copy), and the sub-agents, skills and memory start empty, as the template leaves them. Continual Harness then -starts from exactly the text the GEPA, SkillOpt and Harness² runs start from. +"""Write seed_adapter/ beside this file: SkillOpt's initial OfficeQA skill (../utils/benchmark/seed_adapter/ADAPTER.md, +written by ../utils/benchmark/prepare.py) as a `continual-harness` adapter: the skill text is AGENTS.md (the base policy +p; ADAPTER.md is its copy), and the sub-agents, skills and memory start empty, as the template leaves them. Continual +Harness then starts from exactly the text the GEPA, SkillOpt and Harness² runs start from. python examples/officeqa/continual_harness/make_seed.py """ @@ -14,7 +14,7 @@ HERE = Path(__file__).resolve().parent TEMPLATE = Path(hermit.__file__).resolve().parent / "adapter" / "templates" / "continual_harness" -seed = (HERE.parent / "utils" / "seed_adapter" / "ADAPTER.md").read_text() +seed = (HERE.parent / "utils" / "benchmark" / "seed_adapter" / "ADAPTER.md").read_text() out = HERE / "seed_adapter" if out.exists(): shutil.rmtree(out) diff --git a/examples/officeqa/continual_harness/run_codex.yaml b/examples/officeqa/continual_harness/run_codex.yaml index 72077a17..7637475c 100644 --- a/examples/officeqa/continual_harness/run_codex.yaml +++ b/examples/officeqa/continual_harness/run_codex.yaml @@ -7,12 +7,13 @@ # place; no validation gate. GLM 5.3 Flash acts and refines, at medium effort; one evaluation pass, as the all-Flash # GEPA and SkillOpt runs (gepa/run_codex_flash.yaml, skillopt/run_codex_flash.yaml). data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/train.txt - test: ../utils/splits/test.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: ../utils/benchmark/splits/train.txt + test: ../utils/benchmark/splits/test.txt step_size: 3 # the Refiner's window: three questions' trajectories per step shuffle_seed: 42 -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: base: codex # Layer 2: Codex as a Base Agent (hermit/agent/base/cli.py) concurrency: 3 @@ -31,9 +32,9 @@ agent: project_doc_max_bytes: 65536 # 64 KiB: the skill (up to 40 KB) with the deployment's notes; past # Codex's default 32 KiB it would cut AGENTS.md (deploy refuses that) judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: continual-harness@1.1 -start_from: seed_adapter # make_seed.py writes it from ../utils/seed_adapter +start_from: seed_adapter # make_seed.py writes it from ../utils/benchmark/seed_adapter evolver: name: continual_harness params: diff --git a/examples/officeqa/continual_harness/smoke_codex.yaml b/examples/officeqa/continual_harness/smoke_codex.yaml index 5a7b9bf5..2580b0ff 100644 --- a/examples/officeqa/continual_harness/smoke_codex.yaml +++ b/examples/officeqa/continual_harness/smoke_codex.yaml @@ -1,12 +1,13 @@ # Smoke: 4 training questions in 2 steps of 2, 4 test questions, one epoch. Confirms the Continual Harness Evolver # runs end to end on the Codex harness (check -> train -> eval), not a measurement. The full run is run_codex.yaml. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/smoke_train.txt - test: ../utils/splits/smoke_test.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: ../utils/benchmark/splits/smoke_train.txt + test: ../utils/benchmark/splits/smoke_test.txt step_size: 2 shuffle_seed: 42 -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: base: codex # Layer 2: Codex as a Base Agent (hermit/agent/base/cli.py) concurrency: 2 @@ -25,9 +26,9 @@ agent: project_doc_max_bytes: 65536 # 64 KiB: the skill (up to 40 KB) with the deployment's notes; past # Codex's default 32 KiB it would cut AGENTS.md (deploy refuses that) judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: continual-harness@1.1 -start_from: seed_adapter # make_seed.py writes it from ../utils/seed_adapter +start_from: seed_adapter # make_seed.py writes it from ../utils/benchmark/seed_adapter evolver: name: continual_harness params: diff --git a/examples/officeqa/continual_harness/stream_codex.yaml b/examples/officeqa/continual_harness/stream_codex.yaml index 865ed9bb..82c5a1ea 100644 --- a/examples/officeqa/continual_harness/stream_codex.yaml +++ b/examples/officeqa/continual_harness/stream_codex.yaml @@ -8,11 +8,12 @@ # clock and starting text (SkillOpt's initial skill as AGENTS.md) are those of ../harness2_stream/run_codex.yaml. # The stream's accuracy is the mean of the steps' rollout scores (steps.jsonl); no `hermit eval` follows. data: - tasks: ../utils/tasks.jsonl - train: stream_s0.txt # upstream Harness²'s seed-0 order of ../utils/splits/test.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: stream_s0.txt # upstream Harness²'s seed-0 order of ../utils/benchmark/splits/test.txt step_size: 1 # one question per step shuffle: false # the stream's own order -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: base: codex # Layer 2: Codex as a Base Agent (hermit/agent/base/cli.py) concurrency: 1 @@ -31,7 +32,7 @@ agent: project_doc_max_bytes: 65536 # 64 KiB: the skill (up to 40 KB) with the deployment's notes; past # Codex's default 32 KiB it would cut AGENTS.md (deploy refuses that) judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: continual-harness@1.1 start_from: seed_adapter # make_seed.py: SkillOpt's initial skill as AGENTS.md, G / K / M empty evolver: diff --git a/examples/officeqa/continual_harness/stream_codex_luna.yaml b/examples/officeqa/continual_harness/stream_codex_luna.yaml index 1367f165..4e150cf3 100644 --- a/examples/officeqa/continual_harness/stream_codex_luna.yaml +++ b/examples/officeqa/continual_harness/stream_codex_luna.yaml @@ -7,11 +7,12 @@ # (docker/codex/Dockerfile.standalone) and the egress proxy lets through chatgpt.com and auth.openai.com (the login's # refresh): the recipe's "The Flash row with GPT-5.6-Luna". (Ported from PR #54.) data: - tasks: ../utils/tasks.jsonl - train: stream_s0.txt # upstream Harness²'s seed-0 order of ../utils/splits/test.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: stream_s0.txt # upstream Harness²'s seed-0 order of ../utils/benchmark/splits/test.txt step_size: 1 # one question per step shuffle: false # the stream's own order -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: base: codex # Layer 2: Codex as a Base Agent (hermit/agent/base/cli.py) concurrency: 1 @@ -31,7 +32,7 @@ agent: project_doc_max_bytes: 65536 # 64 KiB: the skill (up to 40 KB) with the deployment's notes; past # Codex's default 32 KiB it would cut AGENTS.md (deploy refuses that) judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: continual-harness@1.1 start_from: seed_adapter # make_seed.py: SkillOpt's initial skill as AGENTS.md, G / K / M empty evolver: diff --git a/examples/officeqa/gepa/run.yaml b/examples/officeqa/gepa/run.yaml index fde64c65..d7f779b2 100644 --- a/examples/officeqa/gepa/run.yaml +++ b/examples/officeqa/gepa/run.yaml @@ -2,20 +2,21 @@ # default reflection minibatch of 3, Pareto selection over the 24 validation questions, 4 epochs over the 50 training # questions. GLM 5.3 Flash acts. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/train.txt - val: ../utils/splits/val.txt - test: ../utils/splits/test.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: ../utils/benchmark/splits/train.txt + val: ../utils/benchmark/splits/val.txt + test: ../utils/benchmark/splits/test.txt step_size: 3 shuffle_seed: 42 -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: name: ../utils/agent/agent.py:OfficeQAAgent concurrency: 8 judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: prompt-only@1.0 -start_from: ../utils/seed_adapter # SkillOpt's initial OfficeQA skill +start_from: ../utils/benchmark/seed_adapter # SkillOpt's initial OfficeQA skill evolver: name: gepa model: fireworks/accounts/fireworks/models/glm-5p3-flash # the agent's (agent.params.model), named so its key and @@ -24,7 +25,7 @@ llm: # the optimizer: GLM 5.3 at medium effort (the pa model: fireworks/accounts/fireworks/models/glm-5p3 # /; the price is the backend's extra_body: {reasoning_effort: medium} environment: # the task sandbox: the agent's code, its data and the corpus - read: [../utils/agent, ../utils/data.json, ../utils/documents] + read: [../utils/agent, ../utils/benchmark/data.json, ../utils/benchmark/documents] packages: [skillopt] sandbox: enabled: true diff --git a/examples/officeqa/gepa/run_codex.yaml b/examples/officeqa/gepa/run_codex.yaml index db6fdd67..f1c5ae83 100644 --- a/examples/officeqa/gepa/run_codex.yaml +++ b/examples/officeqa/gepa/run_codex.yaml @@ -6,13 +6,14 @@ # per session). The published Codex-harness numbers were made with SkillOpt's own Codex harness (the skill as # SKILL.md, 180 s sessions), which this replaces. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/train.txt - val: ../utils/splits/val.txt - test: ../utils/splits/test.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: ../utils/benchmark/splits/train.txt + val: ../utils/benchmark/splits/val.txt + test: ../utils/benchmark/splits/test.txt step_size: 3 shuffle_seed: 42 -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: base: codex # Layer 2: Codex as a Base Agent (hermit/agent/base/cli.py) concurrency: 8 @@ -31,9 +32,9 @@ agent: project_doc_max_bytes: 65536 # 64 KiB: the skill (up to 40 KB) with the deployment's notes; past # Codex's default 32 KiB it would cut AGENTS.md (deploy refuses that) judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: prompt-only@1.0 -start_from: ../utils/seed_adapter # SkillOpt's initial OfficeQA skill +start_from: ../utils/benchmark/seed_adapter # SkillOpt's initial OfficeQA skill evolver: name: gepa llm: # the optimizer: GLM 5.3 at medium effort (the paper: GPT-5.5, medium) diff --git a/examples/officeqa/gepa/run_codex_flash.yaml b/examples/officeqa/gepa/run_codex_flash.yaml index 2fb5340d..fc946924 100644 --- a/examples/officeqa/gepa/run_codex_flash.yaml +++ b/examples/officeqa/gepa/run_codex_flash.yaml @@ -9,13 +9,14 @@ # are evaluated once, and the Codex image is officeqa-codex:0.162.0 (the # README table: Codex 0.134.0, GLM 5.3 learns, three passes). The recipe's Flash row. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/train.txt - val: ../utils/splits/val.txt - test: ../utils/splits/test.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: ../utils/benchmark/splits/train.txt + val: ../utils/benchmark/splits/val.txt + test: ../utils/benchmark/splits/test.txt step_size: 3 shuffle_seed: 42 -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: base: codex # Layer 2: Codex as a Base Agent (hermit/agent/base/cli.py) concurrency: 8 @@ -34,9 +35,9 @@ agent: project_doc_max_bytes: 65536 # 64 KiB: the skill (up to 40 KB) with the deployment's notes; past # Codex's default 32 KiB it would cut AGENTS.md (deploy refuses that) judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: prompt-only@1.0 -start_from: ../utils/seed_adapter # SkillOpt's initial OfficeQA skill +start_from: ../utils/benchmark/seed_adapter # SkillOpt's initial OfficeQA skill evolver: name: gepa llm: # the optimizer: GLM 5.3 Flash at medium effort, the acting model (the README table: GLM 5.3) diff --git a/examples/officeqa/gepa/run_codex_harbor.yaml b/examples/officeqa/gepa/run_codex_harbor.yaml index 0babab41..935727b0 100644 --- a/examples/officeqa/gepa/run_codex_harbor.yaml +++ b/examples/officeqa/gepa/run_codex_harbor.yaml @@ -1,38 +1,42 @@ # OfficeQA in the SkillOpt paper's setting with the GEPA Evolver, the Codex harness through Harbor: as # gepa/run_codex.yaml (GLM 5.3 Flash acts in Codex, the skill as AGENTS.md, the whole corpus under docs/), but each -# question is a Harbor task (utils/harbor/build_tasks.py) and the session runs in a container Harbor starts, installs -# Codex into and verifies (the judge inside the container, utils/harbor/judge.py), instead of environment.image; -# HermitCodex deploys the adapter there as hermit deploy --host codex does, its ADAPTER.md (the skill) as AGENTS.md. +# question is a Harbor task (utils/harbor/build_tasks.py) and the session runs in the task's own Harbor environment, a +# container Harbor starts, installs Codex into and verifies (the judge inside the container, utils/harbor/judge.py, +# which reads the session's stream Hermit hands it in /logs/agent), instead of environment.image; the Codex agent +# deploys the adapter there as hermit deploy --host codex does, its ADAPTER.md (the skill) as AGENTS.md. # Before: python examples/officeqa/utils/harbor/build_tasks.py, and export OFFICEQA_DOCS_ROOT as it prints (the corpus, # mounted read-only at /app/docs/root_1). The recipe's "Through Harbor" section has the rest. data: - tasks: ../utils/.private/harbor_tasks/tasks.jsonl # the same UIDs as ../utils/tasks.jsonl: the splits apply - train: ../utils/splits/train.txt - val: ../utils/splits/val.txt - test: ../utils/splits/test.txt + tasks: ../utils/benchmark/.private/harbor_tasks/tasks.jsonl # the same UIDs as ../utils/benchmark/tasks: the splits apply + train: ../utils/benchmark/splits/train.txt + val: ../utils/benchmark/splits/val.txt + test: ../utils/benchmark/splits/test.txt step_size: 3 shuffle_seed: 42 -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: - name: harbor + base: codex concurrency: 8 - params: - tasks_dir: ../utils/.private/harbor_tasks - model: fireworks/accounts/fireworks/models/glm-5p3-flash # the backend's endpoint and key go to Codex in the container - effort: medium + model: {ref: fireworks/accounts/fireworks/models/glm-5p3-flash, effort: medium} # the key goes to Codex by name + codex: web_search: false # as the Codex route: the answer is in the corpus, not on the web - codex_version: "0.134.0" # the Codex in run_codex.yaml's image (docker/codex/Dockerfile) + version: "0.134.0" # the Codex in run_codex.yaml's image (docker/codex/Dockerfile), installed by + # Harbor in each task's environment +environment: + harbor: + tasks_dir: ../utils/benchmark/.private/harbor_tasks agent_timeout_sec: 3600 # one session's wall clock, as the Codex route's timeout_s environment: delete: true mounts: # the corpus, where the instruction's candidate paths point - {type: bind, source: $OFFICEQA_DOCS_ROOT, target: /app/docs/root_1, read_only: true} jobs_dir: ~/.cache/hermit/officeqa-harbor/jobs - docker_cleanup: images_keep # one image for every task: nothing to reclaim between trials + docker_cleanup: images_keep # one image for every task: nothing to reclaim between tasks judge: - path: hermit.harbor.judge # the reward the container's judge wrote (exact match), its feedback line + path: hermit.task.verify # the reward the container's judge wrote (exact match), its feedback line + name: environment template: prompt-only@1.0 -start_from: ../utils/seed_adapter # SkillOpt's initial OfficeQA skill +start_from: ../utils/benchmark/seed_adapter # SkillOpt's initial OfficeQA skill evolver: name: gepa llm: # the optimizer: GLM 5.3 at medium effort (the paper: GPT-5.5, medium) diff --git a/examples/officeqa/gepa/run_codex_luna.yaml b/examples/officeqa/gepa/run_codex_luna.yaml index df2d2a9e..16b6ef6c 100644 --- a/examples/officeqa/gepa/run_codex_luna.yaml +++ b/examples/officeqa/gepa/run_codex_luna.yaml @@ -7,13 +7,14 @@ # (docker/codex/Dockerfile.standalone) and the egress proxy lets through chatgpt.com and auth.openai.com (the login's # refresh): the recipe's "The Flash row with GPT-5.6-Luna". (Ported from PR #54.) data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/train.txt - val: ../utils/splits/val.txt - test: ../utils/splits/test.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: ../utils/benchmark/splits/train.txt + val: ../utils/benchmark/splits/val.txt + test: ../utils/benchmark/splits/test.txt step_size: 3 shuffle_seed: 42 -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: base: codex # Layer 2: Codex as a Base Agent (hermit/agent/base/cli.py) concurrency: 8 @@ -33,9 +34,9 @@ agent: project_doc_max_bytes: 65536 # 64 KiB: the skill (up to 40 KB) with the deployment's notes; past # Codex's default 32 KiB it would cut AGENTS.md (deploy refuses that) judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: prompt-only@1.0 -start_from: ../utils/seed_adapter # SkillOpt's initial OfficeQA skill +start_from: ../utils/benchmark/seed_adapter # SkillOpt's initial OfficeQA skill evolver: name: gepa llm: # the learner: GPT-5.6-Luna at medium effort, the acting model diff --git a/examples/officeqa/gepa/smoke.yaml b/examples/officeqa/gepa/smoke.yaml index fc6b5b5e..d738f8fd 100644 --- a/examples/officeqa/gepa/smoke.yaml +++ b/examples/officeqa/gepa/smoke.yaml @@ -1,19 +1,20 @@ # Smoke: 4 training questions in 2 steps, 2 validation questions, 4 test questions. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/smoke_train.txt - val: ../utils/splits/smoke_val.txt - test: ../utils/splits/smoke_test.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: ../utils/benchmark/splits/smoke_train.txt + val: ../utils/benchmark/splits/smoke_val.txt + test: ../utils/benchmark/splits/smoke_test.txt step_size: 2 shuffle_seed: 42 -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: name: ../utils/agent/agent.py:OfficeQAAgent concurrency: 4 judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: prompt-only@1.0 -start_from: ../utils/seed_adapter # SkillOpt's initial OfficeQA skill +start_from: ../utils/benchmark/seed_adapter # SkillOpt's initial OfficeQA skill evolver: name: gepa model: fireworks/accounts/fireworks/models/glm-5p3-flash # the agent's (agent.params.model), named so its key and @@ -22,7 +23,7 @@ llm: # the optimizer: GLM 5.3 at medium effort (the pa model: fireworks/accounts/fireworks/models/glm-5p3 # /; the price is the backend's extra_body: {reasoning_effort: medium} environment: # the task sandbox: the agent's code, its data and the corpus - read: [../utils/agent, ../utils/data.json, ../utils/documents] + read: [../utils/agent, ../utils/benchmark/data.json, ../utils/benchmark/documents] packages: [skillopt] sandbox: enabled: true diff --git a/examples/officeqa/gepa/smoke_codex.yaml b/examples/officeqa/gepa/smoke_codex.yaml index 84ccfaf3..704a0ad6 100644 --- a/examples/officeqa/gepa/smoke_codex.yaml +++ b/examples/officeqa/gepa/smoke_codex.yaml @@ -4,13 +4,14 @@ # per session). The published Codex-harness numbers were made with SkillOpt's own Codex harness (the skill as # SKILL.md, 180 s sessions), which this replaces. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/smoke_train.txt - val: ../utils/splits/smoke_val.txt - test: ../utils/splits/smoke_test.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: ../utils/benchmark/splits/smoke_train.txt + val: ../utils/benchmark/splits/smoke_val.txt + test: ../utils/benchmark/splits/smoke_test.txt step_size: 2 shuffle_seed: 42 -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: base: codex # Layer 2: Codex as a Base Agent (hermit/agent/base/cli.py) concurrency: 4 @@ -29,9 +30,9 @@ agent: project_doc_max_bytes: 65536 # 64 KiB: the skill (up to 40 KB) with the deployment's notes; past # Codex's default 32 KiB it would cut AGENTS.md (deploy refuses that) judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: prompt-only@1.0 -start_from: ../utils/seed_adapter # SkillOpt's initial OfficeQA skill +start_from: ../utils/benchmark/seed_adapter # SkillOpt's initial OfficeQA skill evolver: name: gepa llm: # the optimizer: GLM 5.3 at medium effort (the paper: GPT-5.5, medium) diff --git a/examples/officeqa/harness2_stream/README.md b/examples/officeqa/harness2_stream/README.md index 6308d08b..5755abf3 100644 --- a/examples/officeqa/harness2_stream/README.md +++ b/examples/officeqa/harness2_stream/README.md @@ -84,7 +84,7 @@ Paths are relative to this directory: ```bash LIBRARY_FROM=runs/h2-codex-s0-first10 TAG=h2-codex-s0-rest STREAM=stream_rest_s0.txt SKIP_TASKS=10 \ - SPLITS=../utils/splits/test.txt CLAUDE_BIN=runs/.claude-pin//claude \ + SPLITS=../utils/benchmark/splits/test.txt CLAUDE_BIN=runs/.claude-pin//claude \ FIREWORKS_API_KEY=... SEED=0 WM_PORT=3591 examples/officeqa/harness2_stream/launch.sh ``` diff --git a/examples/officeqa/harness2_stream/launch.sh b/examples/officeqa/harness2_stream/launch.sh index 51c9ecfe..cc92c6c0 100755 --- a/examples/officeqa/harness2_stream/launch.sh +++ b/examples/officeqa/harness2_stream/launch.sh @@ -17,7 +17,7 @@ # MAX_TASKS=2 TAG=h2-smoke ... # a smoke run on the first questions of the stream # SKIP_TASKS=10 ... # drop the stream's first questions (continuing a run that took them) # LIBRARY_FROM=runs/h2-codex-s0 TAG=h2-codex-s0-rest STREAM=stream_rest_s0.txt \ -# SPLITS="../utils/splits/train.txt ../utils/splits/val.txt" CLAUDE_BIN=runs/.claude-pin//claude ... +# SPLITS="../utils/benchmark/splits/train.txt ../utils/benchmark/splits/val.txt" CLAUDE_BIN=runs/.claude-pin//claude ... # # that stream continued on other questions (paths: from here) # # Writes upstream's stream order for SEED, regenerates H0 (seed_adapter_codex) and trains. With Claude Code editors @@ -61,7 +61,7 @@ if [ "$ON_CLAUDE_LOGIN" = yes ] && [ -z "${CLAUDE_CODE_OAUTH_TOKEN:-}" ]; then fi RUN_NAME=$("$PY" -c "import yaml; print(yaml.safe_load(open('$HERE/$CONFIG'))['trainer']['run_name'])") case "$RUN_NAME" in *s0) TAG=${TAG:-${RUN_NAME%0}$SEED} ;; *) TAG=${TAG:-$RUN_NAME} ;; esac -SPLITS=${SPLITS:-../utils/splits/test.txt} # the stream's questions (several files: their union) +SPLITS=${SPLITS:-../utils/benchmark/splits/test.txt} # the stream's questions (several files: their union) STREAM=${STREAM:-stream_s$SEED.txt} LIBRARY_FROM=${LIBRARY_FROM:-} # a finished run whose library this run starts from: a continuation case "$EDITORS/$LOGIN" in @@ -169,7 +169,7 @@ else fi # The answer files and the run continued (its records carry the judge's feedback): the sessions' sandbox is checked to # show none of them. -ABSENT=$("$PY" -c "import json, sys; d = json.load(open('../utils/data.json')); +ABSENT=$("$PY" -c "import json, sys; d = json.load(open('../utils/benchmark/data.json')); from pathlib import Path; print(json.dumps([str(Path(d['answers']).parent), d['csv'], *sys.argv[1:]]))" \ ${LIBRARY_FROM:+"$LIBRARY_FROM"}) if [ "$METER" = yes ]; then diff --git a/examples/officeqa/harness2_stream/make_seed_codex.py b/examples/officeqa/harness2_stream/make_seed_codex.py index 37b97c5f..6cbafbb5 100644 --- a/examples/officeqa/harness2_stream/make_seed_codex.py +++ b/examples/officeqa/harness2_stream/make_seed_codex.py @@ -1,8 +1,9 @@ #!/usr/bin/env python3 -"""Write seed_adapter_codex/ beside this file: SkillOpt's initial OfficeQA skill (../utils/seed_adapter/ADAPTER.md, -written by ../utils/prepare.py) as a `harness2-codex` adapter: the skill text is AGENTS.md (and ADAPTER.md, its -copy), and the on-demand directories hold only the template's placeholders. Harness² under Codex then starts from -exactly the text the GEPA and SkillOpt runs start from, as Codex reads it. +"""Write seed_adapter_codex/ beside this file: SkillOpt's initial OfficeQA skill +(../utils/benchmark/seed_adapter/ADAPTER.md, written by ../utils/benchmark/prepare.py) as a `harness2-codex` adapter: +the skill text is AGENTS.md (and ADAPTER.md, its copy), and the on-demand directories hold only the template's +placeholders. Harness² under Codex then starts from exactly the text the GEPA and SkillOpt runs start from, as Codex +reads it. python examples/officeqa/harness2_stream/make_seed_codex.py """ @@ -14,7 +15,7 @@ HERE = Path(__file__).resolve().parent TEMPLATE = Path(hermit.__file__).resolve().parent / "adapter" / "templates" / "harness2_codex" -seed = (HERE.parent / "utils" / "seed_adapter" / "ADAPTER.md").read_text() +seed = (HERE.parent / "utils" / "benchmark" / "seed_adapter" / "ADAPTER.md").read_text() out = HERE / "seed_adapter_codex" if out.exists(): shutil.rmtree(out) diff --git a/examples/officeqa/harness2_stream/run_claude_haiku.yaml b/examples/officeqa/harness2_stream/run_claude_haiku.yaml index fa33d6b0..4dc87ec1 100644 --- a/examples/officeqa/harness2_stream/run_claude_haiku.yaml +++ b/examples/officeqa/harness2_stream/run_claude_haiku.yaml @@ -13,11 +13,12 @@ # them, cap the plan's usage and the sessions at once (hermit.agent.base.login). Started by launch.sh: # CONFIG=run_claude_haiku.yaml. data: - tasks: ../utils/tasks.jsonl - train: stream_s0.txt # launch.sh: upstream's order of ../utils/splits/test.txt for SEED + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: stream_s0.txt # launch.sh: upstream's order of ../utils/benchmark/splits/test.txt for SEED step_size: 1 # one question per step shuffle: false # the stream's own order -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: base: claude # Layer 2: Claude Code as a Base Agent (hermit/agent/base/cli.py) concurrency: 2 # the INV and DOM probes of a question run side by side @@ -35,7 +36,7 @@ agent: web_search: false # set explicitly: WebSearch and WebFetch denied (they run at Anthropic) timeout_s: 3600 # the paper's wall clock per execution judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: harness2-codex@1.1 # the same tree as the Codex columns: Claude Code takes each part in its place start_from: seed_adapter_codex # H0 (launch.sh runs make_seed_codex.py) evolver: diff --git a/examples/officeqa/harness2_stream/run_codex.yaml b/examples/officeqa/harness2_stream/run_codex.yaml index d44e21ef..a67beb09 100644 --- a/examples/officeqa/harness2_stream/run_codex.yaml +++ b/examples/officeqa/harness2_stream/run_codex.yaml @@ -7,11 +7,12 @@ # at medium effort: the executor as GEPA's and SkillOpt's, the editors as Claude Code sessions (`claude -p`). # Started by launch.sh. data: - tasks: ../utils/tasks.jsonl - train: stream_s0.txt # launch.sh: upstream's order of ../utils/splits/test.txt for SEED + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: stream_s0.txt # launch.sh: upstream's order of ../utils/benchmark/splits/test.txt for SEED step_size: 1 # one question per step shuffle: false # the stream's own order -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: base: codex # Layer 2: Codex as a Base Agent (hermit/agent/base/cli.py) concurrency: 2 # the INV and DOM probes of a question run side by side @@ -30,7 +31,7 @@ agent: project_doc_max_bytes: 65536 # 64 KiB: the skill (up to 40 KB) with the deployment's notes; past # Codex's default 32 KiB it would cut AGENTS.md (deploy refuses that) judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: harness2-codex@1.1 start_from: seed_adapter_codex # H0 (launch.sh runs make_seed_codex.py) evolver: diff --git a/examples/officeqa/harness2_stream/run_codex_luna.yaml b/examples/officeqa/harness2_stream/run_codex_luna.yaml index fac255f8..3ebcceeb 100644 --- a/examples/officeqa/harness2_stream/run_codex_luna.yaml +++ b/examples/officeqa/harness2_stream/run_codex_luna.yaml @@ -8,11 +8,12 @@ # (docker/codex/Dockerfile.standalone) and the egress proxy lets through chatgpt.com and auth.openai.com (the login's # refresh): the recipe's "The Flash row with GPT-5.6-Luna". (Ported from PR #54.) data: - tasks: ../utils/tasks.jsonl - train: stream_s0.txt # launch.sh: upstream's order of ../utils/splits/test.txt for SEED + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: stream_s0.txt # launch.sh: upstream's order of ../utils/benchmark/splits/test.txt for SEED step_size: 1 # one question per step shuffle: false # the stream's own order -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: base: codex # Layer 2: Codex as a Base Agent (hermit/agent/base/cli.py) concurrency: 2 # the INV and DOM probes of a question run side by side @@ -32,7 +33,7 @@ agent: project_doc_max_bytes: 65536 # 64 KiB: the skill (up to 40 KB) with the deployment's notes; past # Codex's default 32 KiB it would cut AGENTS.md (deploy refuses that) judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: harness2-codex@1.1 start_from: seed_adapter_codex # H0 (launch.sh runs make_seed_codex.py) evolver: diff --git a/examples/officeqa/harness2_stream/slots_claude.yaml b/examples/officeqa/harness2_stream/slots_claude.yaml index 36aad34b..4a8a4427 100644 --- a/examples/officeqa/harness2_stream/slots_claude.yaml +++ b/examples/officeqa/harness2_stream/slots_claude.yaml @@ -3,7 +3,7 @@ # Glob or Grep, so it searches in Bash; slots_codex.yaml names rg, this file does not). The per-task text its stage # prompts interpolate. They describe the work, never its measurement: upstream refuses the words benchmark, evaluation, # grader, rubric and score here. `work` names the deliverable's form, as upstream's own benchmarks do: the final answer -# is the value alone, which is how the task's answers are taken (../utils/task-info/README.md). No session sees an +# is the value alone, which is how the task's answers are taken (../utils/benchmark/task-info/README.md). No session sees an # answer. work: >- Answering a question about U.S. Treasury Bulletins (1939-2025). The agent is a Claude Code session in a workspace: diff --git a/examples/officeqa/harness2_stream/slots_codex.yaml b/examples/officeqa/harness2_stream/slots_codex.yaml index 728fc737..1d1fa9c6 100644 --- a/examples/officeqa/harness2_stream/slots_codex.yaml +++ b/examples/officeqa/harness2_stream/slots_codex.yaml @@ -1,7 +1,7 @@ # Upstream Harness²'s prompt slots for OfficeQA answered by Codex (run_codex.yaml): the per-task text its stage prompts # interpolate. They describe the work, never its measurement: upstream refuses the words benchmark, evaluation, grader, # rubric and score here. `work` names the deliverable's form, as upstream's own benchmarks do: the final answer is the -# value alone, which is how the task's answers are taken (../utils/task-info/README.md). No session sees an answer. +# value alone, which is how the task's answers are taken (../utils/benchmark/task-info/README.md). No session sees an answer. work: >- Answering a question about U.S. Treasury Bulletins (1939-2025). The agent is a Codex session in a workspace: `task.md` holds the question, the parsed pages its answer comes from and the bulletin files those pages are in; the diff --git a/examples/officeqa/harness2_stream/sources.py b/examples/officeqa/harness2_stream/sources.py index a0405c11..72a94d51 100644 --- a/examples/officeqa/harness2_stream/sources.py +++ b/examples/officeqa/harness2_stream/sources.py @@ -8,8 +8,9 @@ def task_files(task: dict, dest: Path) -> Path: from skillopt.envs.officeqa.tool_runtime import resolve_candidate_files, resolve_docs_roots - docs = json.loads((HERE.parent / "utils" / "data.json").read_text())["docs"] - files = resolve_candidate_files(task["source_files"], resolve_docs_roots([docs])) + docs = json.loads((HERE.parent / "utils" / "benchmark" / "data.json").read_text())["docs"] + meta = task.get("metadata") or {} # a task directory's row keeps them under its metadata + files = resolve_candidate_files(task.get("source_files", meta.get("source_files")), resolve_docs_roots([docs])) if not files: raise FileNotFoundError(f"no source files for {task['id']} under {docs}") dest.mkdir(parents=True, exist_ok=True) diff --git a/examples/officeqa/harness2_stream/stream.py b/examples/officeqa/harness2_stream/stream.py index d6274413..5a28fdd7 100755 --- a/examples/officeqa/harness2_stream/stream.py +++ b/examples/officeqa/harness2_stream/stream.py @@ -2,7 +2,7 @@ """Write the task stream for data.train: a split in upstream Harness²'s order for a seed (harness2.core.schedule .stream_order: the sorted ids, shuffled by Random(f"{seed}:{domain}")), one id per line. Several splits: their union. - python stream.py --seed 0 --split ../utils/splits/test.txt --out stream_s0.txt [--max 2] [--skip 10] + python stream.py --seed 0 --split ../utils/benchmark/splits/test.txt --out stream_s0.txt [--max 2] [--skip 10] """ import argparse from pathlib import Path diff --git a/examples/officeqa/meta_harness_adapter/demo.yaml b/examples/officeqa/meta_harness_adapter/demo.yaml index f5cde8a7..b8f37546 100644 --- a/examples/officeqa/meta_harness_adapter/demo.yaml +++ b/examples/officeqa/meta_harness_adapter/demo.yaml @@ -5,9 +5,10 @@ # officeqa-codex:0.162.0) with the whole adapter deployed in its workspace; GLM 5.3 Flash acts and proposes, at # medium effort. No test evaluation in the demo. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/train.txt # the search set - test: ../utils/splits/test.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: ../utils/benchmark/splits/train.txt # the search set + test: ../utils/benchmark/splits/test.txt step_size: 50 # the whole search set per step: each step evaluates one candidate shuffle: false task_info: task-info/ @@ -29,9 +30,9 @@ agent: project_doc_max_bytes: 65536 # 64 KiB: the skill (up to 40 KB) with the deployment's notes; past # Codex's default 32 KiB it would cut AGENTS.md (deploy refuses that) judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: meta-harness-adapter@1.0 -start_from: seed_adapter # make_seed.py writes it from ../utils/seed_adapter +start_from: seed_adapter # make_seed.py writes it from ../utils/benchmark/seed_adapter evolver: name: meta_harness_adapter params: # Layer 3: the method @@ -43,7 +44,7 @@ evolver: ref: fireworks/accounts/fireworks/models/glm-5p3-flash effort: medium evolver_access: - environment: [../utils/documents] # what the learner reads beside its workspace + environment: [../utils/benchmark/documents] # what the learner reads beside its workspace environment: # the acting agent: Codex in a container of its own image: "officeqa-codex:0.162.0" container: {"network": "oq-sandbox", "proxy": "http://oq-egress:3128"} diff --git a/examples/officeqa/meta_harness_adapter/make_seed.py b/examples/officeqa/meta_harness_adapter/make_seed.py index 5655866b..68edd3a8 100644 --- a/examples/officeqa/meta_harness_adapter/make_seed.py +++ b/examples/officeqa/meta_harness_adapter/make_seed.py @@ -1,7 +1,8 @@ #!/usr/bin/env python3 -"""Write seed_adapter/ beside this file: SkillOpt's initial OfficeQA skill (../utils/seed_adapter/ADAPTER.md, written -by ../utils/prepare.py) as a `meta-harness-adapter` adapter, its ADAPTER.md and nothing else. Meta-Harness Adapter -then starts from exactly the text the GEPA, SkillOpt, Harness² and Continual Harness runs start from. +"""Write seed_adapter/ beside this file: SkillOpt's initial OfficeQA skill (../utils/benchmark/seed_adapter/ADAPTER.md, +written by ../utils/benchmark/prepare.py) as a `meta-harness-adapter` adapter, its ADAPTER.md and nothing else. +Meta-Harness Adapter then starts from exactly the text the GEPA, SkillOpt, Harness² and Continual Harness runs start +from. python examples/officeqa/meta_harness_adapter/make_seed.py """ @@ -13,7 +14,7 @@ HERE = Path(__file__).resolve().parent TEMPLATE = Path(hermit.__file__).resolve().parent / "adapter" / "templates" / "meta_harness_adapter" -seed = (HERE.parent / "utils" / "seed_adapter" / "ADAPTER.md").read_text() +seed = (HERE.parent / "utils" / "benchmark" / "seed_adapter" / "ADAPTER.md").read_text() out = HERE / "seed_adapter" if out.exists(): shutil.rmtree(out) diff --git a/examples/officeqa/meta_harness_adapter/run.yaml b/examples/officeqa/meta_harness_adapter/run.yaml index d65d5047..78de6205 100644 --- a/examples/officeqa/meta_harness_adapter/run.yaml +++ b/examples/officeqa/meta_harness_adapter/run.yaml @@ -6,9 +6,10 @@ # (the framework's Codex agent, image officeqa-codex:0.162.0) with the whole adapter deployed in its workspace; # GLM 5.3 Flash acts and proposes, at medium effort. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/train.txt # the search set - test: ../utils/splits/test.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: ../utils/benchmark/splits/train.txt # the search set + test: ../utils/benchmark/splits/test.txt step_size: 50 # the whole search set per step: each step evaluates one candidate shuffle: false task_info: task-info/ @@ -30,9 +31,9 @@ agent: project_doc_max_bytes: 65536 # 64 KiB: the skill (up to 40 KB) with the deployment's notes; past # Codex's default 32 KiB it would cut AGENTS.md (deploy refuses that) judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: meta-harness-adapter@1.0 -start_from: seed_adapter # make_seed.py writes it from ../utils/seed_adapter +start_from: seed_adapter # make_seed.py writes it from ../utils/benchmark/seed_adapter evolver: name: meta_harness_adapter params: @@ -42,7 +43,7 @@ evolver: k: 2 max_rounds: 6 # six rounds of two: twelve candidates, each evaluated once evolver_access: - environment: [../utils/documents] # what the learner reads beside its workspace + environment: [../utils/benchmark/documents] # what the learner reads beside its workspace environment: # the acting agent: Codex in a container of its own image: "officeqa-codex:0.162.0" container: {"network": "oq-sandbox", "proxy": "http://oq-egress:3128"} diff --git a/examples/officeqa/opro/README.md b/examples/officeqa/opro/README.md index 6df80af8..803ee827 100644 --- a/examples/officeqa/opro/README.md +++ b/examples/officeqa/opro/README.md @@ -36,7 +36,7 @@ learning (the [recipe](../../../docs/recipes/officeqa.md#opro) has the run and i | Adapter | accuracy | − the initial skill, points | |---|---|---| -| the initial skill (`../utils/seed_adapter`) | 0.355 | | +| the initial skill (`../utils/benchmark/seed_adapter`) | 0.355 | | | **`final`, candidate 2 of 13 (validation 0.625)** | **0.436** | **+8.1** [+1.7, +14.5] | Paired per question, 95% bootstrap interval. `final` is one 398-character instruction in place of the 886-character diff --git a/examples/officeqa/opro/adapter/.hermit/CARD.md b/examples/officeqa/opro/adapter/.hermit/CARD.md index ad19e3bd..c65d2b22 100644 --- a/examples/officeqa/opro/adapter/.hermit/CARD.md +++ b/examples/officeqa/opro/adapter/.hermit/CARD.md @@ -3,7 +3,7 @@ - Template: `prompt-only@1.0` - Started from: `3dc113889b760d0d9c0bc21b963fe8a9fd543920` - Evolver: `opro` -- Task: ../utils/task-info/ +- Task: ../utils/benchmark/task-info/ - Training: 3 epoch(s), 3 step(s) - Spending: rollouts $0.0000, Evolver $0.0000 (+835 calls of unknown cost); 2.41 h diff --git a/examples/officeqa/opro/run_codex_luna.yaml b/examples/officeqa/opro/run_codex_luna.yaml index 998f1264..2c5cd232 100644 --- a/examples/officeqa/opro/run_codex_luna.yaml +++ b/examples/officeqa/opro/run_codex_luna.yaml @@ -3,13 +3,14 @@ # login (`login: chatgpt`), and the optimizer is the same model, one Codex session per chat call # (`llm: {cli: codex}`). The image and the egress proxy: the recipe's "The Flash row with GPT-5.6-Luna". data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/train.txt - val: ../utils/splits/val.txt - test: ../utils/splits/test.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: ../utils/benchmark/splits/train.txt + val: ../utils/benchmark/splits/val.txt + test: ../utils/benchmark/splits/test.txt step_size: 50 # every training question: one OPRO step per training step shuffle_seed: 42 -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: base: codex # Layer 2: Codex as a Base Agent (hermit/agent/base/cli.py) concurrency: 4 @@ -29,9 +30,9 @@ agent: project_doc_max_bytes: 65536 # 64 KiB: the skill (up to 40 KB) with the deployment's notes; past # Codex's default 32 KiB it would cut AGENTS.md (deploy refuses that) judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: prompt-only@1.0 -start_from: ../utils/seed_adapter # SkillOpt's initial OfficeQA skill +start_from: ../utils/benchmark/seed_adapter # SkillOpt's initial OfficeQA skill evolver: name: opro params: diff --git a/examples/officeqa/opro/smoke_codex_luna.yaml b/examples/officeqa/opro/smoke_codex_luna.yaml index 171da320..6916a64e 100644 --- a/examples/officeqa/opro/smoke_codex_luna.yaml +++ b/examples/officeqa/opro/smoke_codex_luna.yaml @@ -1,13 +1,14 @@ # Smoke of run_codex_luna.yaml: 4 training questions, two OPRO steps of 2 new instructions, 2 validation questions, # 4 test questions. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/smoke_train.txt - val: ../utils/splits/smoke_val.txt - test: ../utils/splits/smoke_test.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: ../utils/benchmark/splits/smoke_train.txt + val: ../utils/benchmark/splits/smoke_val.txt + test: ../utils/benchmark/splits/smoke_test.txt step_size: 4 # every training question: one OPRO step per training step shuffle_seed: 42 -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: base: codex # Layer 2: Codex as a Base Agent (hermit/agent/base/cli.py) concurrency: 2 @@ -27,9 +28,9 @@ agent: project_doc_max_bytes: 65536 # 64 KiB: the skill (up to 40 KB) with the deployment's notes; past # Codex's default 32 KiB it would cut AGENTS.md (deploy refuses that) judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: prompt-only@1.0 -start_from: ../utils/seed_adapter # SkillOpt's initial OfficeQA skill +start_from: ../utils/benchmark/seed_adapter # SkillOpt's initial OfficeQA skill evolver: name: opro params: diff --git a/examples/officeqa/reharness/launch.sh b/examples/officeqa/reharness/launch.sh index f762c58b..44e73f68 100755 --- a/examples/officeqa/reharness/launch.sh +++ b/examples/officeqa/reharness/launch.sh @@ -44,7 +44,7 @@ case "${1:-}" in train|eval|check) COMMAND=$1; shift;; esac WM_PORT=${WM_PORT:?set WM_PORT, the port of the wire meter (3601, say)} export PYTHONPATH=$FRAMEWORK${PYTHONPATH:+:$PYTHONPATH} cd "$HERE" -[ -f ../utils/tasks.jsonl ] || { echo "REFUSING: run ../utils/prepare.py first"; exit 1; } +[ -d ../utils/benchmark/tasks ] || { echo "REFUSING: run ../utils/benchmark/prepare.py first"; exit 1; } case "$CONFIG" in stream|stream_codex|stream_codex_luna) [ "$COMMAND" != eval ] \ || { echo "REFUSING: $CONFIG.yaml has no test split; its number is the mean of rollout.mean_score over the run's steps.jsonl"; exit 1; };; esac # hermit's options (--adapter, --no-probe) go before its overrides (key=value), whatever order they were given in. diff --git a/examples/officeqa/reharness/make_seed.py b/examples/officeqa/reharness/make_seed.py index 447f7967..268f5b91 100755 --- a/examples/officeqa/reharness/make_seed.py +++ b/examples/officeqa/reharness/make_seed.py @@ -1,10 +1,11 @@ #!/usr/bin/env python3 """Write seed_adapter/ beside this file: the reharness template with SkillOpt's initial OfficeQA skill as ADAPTER.md. -The skill is skillopt/envs/officeqa/skills/initial.md of the installed skillopt package, the text ../utils/prepare.py -gives the prompt-only seed the GEPA and SkillOpt runs start from; below it, after a blank line, the template's own -ADAPTER.md (the framing of the retrieved reflections). Everything else is the template's: index.json, the reader, -reharness_core/, an empty bank. Rewrites the directory each time (launch.sh runs it before every command). +The skill is skillopt/envs/officeqa/skills/initial.md of the installed skillopt package, the text +../utils/benchmark/prepare.py gives the prompt-only seed the GEPA and SkillOpt runs start from; below it, after a blank +line, the template's own ADAPTER.md (the framing of the retrieved reflections). Everything else is the template's: +index.json, the reader, reharness_core/, an empty bank. Rewrites the directory each time (launch.sh runs it before every +command). python examples/officeqa/reharness/make_seed.py """ diff --git a/examples/officeqa/reharness/run.yaml b/examples/officeqa/reharness/run.yaml index f31ab880..a48eb40c 100644 --- a/examples/officeqa/reharness/run.yaml +++ b/examples/officeqa/reharness/run.yaml @@ -4,17 +4,18 @@ # on the 172 test questions, three passes each. GLM 5.3 Flash acts and writes the reflections. Started by launch.sh, # which puts the embedding model behind a loopback relay for the adapter's reader. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/train.txt - test: ../utils/splits/test.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: ../utils/benchmark/splits/train.txt + test: ../utils/benchmark/splits/test.txt step_size: 1 # one question per step: one ReHarness instance (read, revise, write) shuffle: false # the stream in the split's order -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: name: ../utils/agent/agent.py:OfficeQAAgent concurrency: 8 # the eval; the stream itself is one question at a time judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: reharness start_from: seed_adapter # SkillOpt's initial OfficeQA skill over the reharness template (make_seed.py) evolver: @@ -28,7 +29,7 @@ llm: # the reflection and revision agents: the acting model: fireworks/accounts/fireworks/models/glm-5p3-flash extra_body: {reasoning_effort: medium} environment: # the task sandbox: the agent's code, its data and the corpus - read: [../utils/agent, ../utils/data.json, ../utils/documents] + read: [../utils/agent, ../utils/benchmark/data.json, ../utils/benchmark/documents] packages: [skillopt] env: ["REHARNESS_*"] # the reader's embedding relay (launch.sh) sandbox: diff --git a/examples/officeqa/reharness/run_codex_flash.yaml b/examples/officeqa/reharness/run_codex_flash.yaml index 8600f137..2251fccf 100644 --- a/examples/officeqa/reharness/run_codex_flash.yaml +++ b/examples/officeqa/reharness/run_codex_flash.yaml @@ -9,12 +9,13 @@ # effort. Run four times (trainer.run_name=flash-codex-reharness, -r2, -r3, -r4, each with its own tree_dir) and # report the mean, as the row's GEPA and SkillOpt numbers are. Started by launch.sh. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/train.txt # the 50 training questions, the stream - test: ../utils/splits/test.txt # the 172 test questions, evaluated once with the frozen bank + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: ../utils/benchmark/splits/train.txt # the 50 training questions, the stream + test: ../utils/benchmark/splits/test.txt # the 172 test questions, evaluated once with the frozen bank step_size: 1 # one question per step: one ReHarness instance (read, revise, write) shuffle: false # the split's own order -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: base: codex # Layer 2: Codex as a Base Agent (hermit/agent/base/cli.py) concurrency: 4 # the eval; the stream itself is one question at a time @@ -33,7 +34,7 @@ agent: project_doc_max_bytes: 204800 # 200 KiB: five retrieved reflections exceed Codex's default 32 KiB, # past which it silently drops the rest of AGENTS.md judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: reharness start_from: seed_adapter # make_seed.py: SkillOpt's initial skill over the reharness template, an empty bank evolver: diff --git a/examples/officeqa/reharness/smoke.yaml b/examples/officeqa/reharness/smoke.yaml index e7293809..03965822 100644 --- a/examples/officeqa/reharness/smoke.yaml +++ b/examples/officeqa/reharness/smoke.yaml @@ -1,17 +1,18 @@ # Smoke: the 4 smoke training questions as a stream of 4 steps (one question each), then the 4 smoke test questions, # one pass. Started by launch.sh, which puts the embedding model behind a loopback relay for the adapter's reader. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/smoke_train.txt - test: ../utils/splits/smoke_test.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: ../utils/benchmark/splits/smoke_train.txt + test: ../utils/benchmark/splits/smoke_test.txt step_size: 1 # one question per step: one ReHarness instance (read, revise, write) shuffle: false # the stream in the split's order -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: name: ../utils/agent/agent.py:OfficeQAAgent concurrency: 4 # the eval; the stream itself is one question at a time judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: reharness start_from: seed_adapter # SkillOpt's initial OfficeQA skill over the reharness template (make_seed.py) evolver: @@ -25,7 +26,7 @@ llm: # the reflection and revision agents: the acting model: fireworks/accounts/fireworks/models/glm-5p3-flash extra_body: {reasoning_effort: medium} environment: # the task sandbox: the agent's code, its data and the corpus - read: [../utils/agent, ../utils/data.json, ../utils/documents] + read: [../utils/agent, ../utils/benchmark/data.json, ../utils/benchmark/documents] packages: [skillopt] env: ["REHARNESS_*"] # the reader's embedding relay (launch.sh) sandbox: diff --git a/examples/officeqa/reharness/stream.yaml b/examples/officeqa/reharness/stream.yaml index e7167e22..725a2e01 100644 --- a/examples/officeqa/reharness/stream.yaml +++ b/examples/officeqa/reharness/stream.yaml @@ -7,17 +7,18 @@ # question set is learned from, and each question is answered once. Started by launch.sh, which puts the embedding # model behind a loopback relay for the adapter's reader. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/test.txt # the stream: the test questions, in the split's order + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: ../utils/benchmark/splits/test.txt # the stream: the test questions, in the split's order step_size: 1 # one question per step: one ReHarness instance (read, revise, write) shuffle: false # the stream in the split's order # no data.test: nothing is held out; `hermit eval` has nothing to run here -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: name: ../utils/agent/agent.py:OfficeQAAgent concurrency: 1 # the stream is one question at a time, and there is no eval judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: reharness start_from: seed_adapter # SkillOpt's initial OfficeQA skill over the reharness template (make_seed.py) evolver: @@ -31,7 +32,7 @@ llm: # the reflection and revision agents: the acting model: fireworks/accounts/fireworks/models/glm-5p3-flash extra_body: {reasoning_effort: medium} environment: # the task sandbox: the agent's code, its data and the corpus - read: [../utils/agent, ../utils/data.json, ../utils/documents] + read: [../utils/agent, ../utils/benchmark/data.json, ../utils/benchmark/documents] packages: [skillopt] env: ["REHARNESS_*"] # the reader's embedding relay (launch.sh) sandbox: diff --git a/examples/officeqa/reharness/stream_codex.yaml b/examples/officeqa/reharness/stream_codex.yaml index d49a42fb..6bdd1132 100644 --- a/examples/officeqa/reharness/stream_codex.yaml +++ b/examples/officeqa/reharness/stream_codex.yaml @@ -11,11 +11,12 @@ # scores (steps.jsonl); no `hermit eval` follows. Started by launch.sh, which puts the embedding model behind a # loopback relay for the adapter's reader. data: - tasks: ../utils/tasks.jsonl - train: stream_s0.txt # upstream Harness²'s seed-0 order of ../utils/splits/test.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: stream_s0.txt # upstream Harness²'s seed-0 order of ../utils/benchmark/splits/test.txt step_size: 1 # one question per step: one ReHarness instance (read, revise, write) shuffle: false # the stream's own order -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: base: codex # Layer 2: Codex as a Base Agent (hermit/agent/base/cli.py) concurrency: 1 # one question at a time, and there is no eval @@ -34,7 +35,7 @@ agent: project_doc_max_bytes: 204800 # 200 KiB: five retrieved reflections exceed Codex's default 32 KiB, # past which it silently drops the rest of AGENTS.md judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: reharness start_from: seed_adapter # make_seed.py: SkillOpt's initial skill over the reharness template, an empty bank evolver: diff --git a/examples/officeqa/reharness/stream_codex_luna.yaml b/examples/officeqa/reharness/stream_codex_luna.yaml index 451c80f9..70537dea 100644 --- a/examples/officeqa/reharness/stream_codex_luna.yaml +++ b/examples/officeqa/reharness/stream_codex_luna.yaml @@ -7,11 +7,12 @@ # (docker/codex/Dockerfile.standalone) and the egress proxy lets through chatgpt.com and auth.openai.com (the login's # refresh): the recipe's "The Flash row with GPT-5.6-Luna". (Ported from PR #54.) data: - tasks: ../utils/tasks.jsonl - train: stream_s0.txt # upstream Harness²'s seed-0 order of ../utils/splits/test.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: stream_s0.txt # upstream Harness²'s seed-0 order of ../utils/benchmark/splits/test.txt step_size: 1 # one question per step: one ReHarness instance (read, revise, write) shuffle: false # the stream's own order -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: base: codex # Layer 2: Codex as a Base Agent (hermit/agent/base/cli.py) concurrency: 1 # one question at a time, and there is no eval @@ -31,7 +32,7 @@ agent: project_doc_max_bytes: 204800 # 200 KiB: five retrieved reflections exceed Codex's default 32 KiB, # past which it silently drops the rest of AGENTS.md judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: reharness start_from: seed_adapter # make_seed.py: SkillOpt's initial skill over the reharness template, an empty bank evolver: diff --git a/examples/officeqa/rrsi_adapter/domain.py b/examples/officeqa/rrsi_adapter/domain.py index 6aca15f4..a9679df2 100644 --- a/examples/officeqa/rrsi_adapter/domain.py +++ b/examples/officeqa/rrsi_adapter/domain.py @@ -35,7 +35,7 @@ class OfficeQADomain(AdapterDomain): critic_patterns = [ (r"\bUID\d{4}\b", "question id in diff"), (r"treasury_bulletin_\d{4}_\d{2}", "a specific bulletin file name in diff"), - (r"answers\.jsonl|/\.private/|utils/judge\.py|skillopt\.envs\.officeqa|normalize_answer|exact_match", + (r"answers\.jsonl|/\.private/|utils/(benchmark/)?judge\.py|skillopt\.envs\.officeqa|normalize_answer|exact_match", "answers, judge or scoring code referenced in diff"), ] diff --git a/examples/officeqa/rrsi_adapter/make_rows.py b/examples/officeqa/rrsi_adapter/make_rows.py index fa84eaa0..30f205e9 100644 --- a/examples/officeqa/rrsi_adapter/make_rows.py +++ b/examples/officeqa/rrsi_adapter/make_rows.py @@ -1,8 +1,8 @@ #!/usr/bin/env python3 -"""Write rows/: tasks.jsonl (every row of ../utils/tasks.jsonl, then each training question k times as `#`, -interleaved by trial so that every question's first trial comes before any second one) and train_k.txt (those -trial rows; smoke_k.txt, the same for ../utils/splits/smoke_train.txt). RRSI evaluates each candidate on the whole evolve set with k trials per task; the validation and test -rows are the questions' own, unchanged. +"""Write rows/: tasks.jsonl (every row of ../utils/benchmark/tasks, then each training question k times as `#`, +interleaved by trial so that every question's first trial comes before any second one) and train_k.txt (those trial +rows; smoke_k.txt, the same for ../utils/benchmark/splits/smoke_train.txt). RRSI evaluates each candidate on the +whole evolve set with k trials per task; the validation and test rows are the questions' own, unchanged. python examples/officeqa/rrsi_adapter/make_rows.py [--k 2] """ @@ -10,14 +10,18 @@ import json from pathlib import Path +from hermit.common import load_callable + HERE = Path(__file__).resolve().parent UTILS = HERE.parent / "utils" parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) parser.add_argument("--k", type=int, default=2) args = parser.parse_args() -rows = [json.loads(l) for l in (UTILS / "tasks.jsonl").read_text().splitlines() if l.strip()] +BENCH = UTILS / "benchmark" +rows = load_callable(str(UTILS / "agent" / "officeqa_fields.py") + ":question_rows", UTILS)(BENCH / "tasks", + BENCH / "splits") by_id = {r["id"]: r for r in rows} -train = [l.strip() for l in (UTILS / "splits" / "train.txt").read_text().splitlines() if l.strip()] +train = [l.strip() for l in (BENCH / "splits" / "train.txt").read_text().splitlines() if l.strip()] out = HERE / "rows" out.mkdir(exist_ok=True) trials = [f"{q}#{j}" for j in range(1, args.k + 1) for q in train] @@ -27,7 +31,7 @@ for t in trials: f.write(json.dumps({**by_id[t.rsplit("#", 1)[0]], "id": t}, ensure_ascii=False) + "\n") (out / f"train_k{args.k}.txt").write_text("".join(t + "\n" for t in trials)) -smoke = [l.strip() for l in (UTILS / "splits" / "smoke_train.txt").read_text().splitlines() if l.strip()] +smoke = [l.strip() for l in (BENCH / "splits" / "smoke_train.txt").read_text().splitlines() if l.strip()] (out / f"smoke_k{args.k}.txt").write_text("".join(f"{q}#{j}\n" for j in range(1, args.k + 1) for q in smoke if q in train)) print(f"wrote {out}: {len(rows)} rows + {len(trials)} trial rows ({len(train)} questions x k={args.k})") diff --git a/examples/officeqa/rrsi_adapter/make_seed.py b/examples/officeqa/rrsi_adapter/make_seed.py index 1d749d2b..c045e19b 100644 --- a/examples/officeqa/rrsi_adapter/make_seed.py +++ b/examples/officeqa/rrsi_adapter/make_seed.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 -"""Write seed_adapter/ beside this file: SkillOpt's initial OfficeQA skill (../utils/seed_adapter/ADAPTER.md, written -by ../utils/prepare.py) as an `rrsi-adapter` adapter, its ADAPTER.md and nothing else: H_0 is exactly the text the -GEPA, SkillOpt, Harness², Continual Harness and Meta-Harness Adapter runs start from. +"""Write seed_adapter/ beside this file: SkillOpt's initial OfficeQA skill (../utils/benchmark/seed_adapter/ADAPTER.md, +written by ../utils/benchmark/prepare.py) as an `rrsi-adapter` adapter, its ADAPTER.md and nothing else: H_0 is exactly +the text the GEPA, SkillOpt, Harness², Continual Harness and Meta-Harness Adapter runs start from. python examples/officeqa/rrsi_adapter/make_seed.py """ @@ -13,7 +13,7 @@ HERE = Path(__file__).resolve().parent TEMPLATE = Path(hermit.__file__).resolve().parent / "adapter" / "templates" / "rrsi_adapter" -seed = (HERE.parent / "utils" / "seed_adapter" / "ADAPTER.md").read_text() +seed = (HERE.parent / "utils" / "benchmark" / "seed_adapter" / "ADAPTER.md").read_text() out = HERE / "seed_adapter" if out.exists(): shutil.rmtree(out) diff --git a/examples/officeqa/rrsi_adapter/run_luna.yaml b/examples/officeqa/rrsi_adapter/run_luna.yaml index 19f62714..439b5f63 100644 --- a/examples/officeqa/rrsi_adapter/run_luna.yaml +++ b/examples/officeqa/rrsi_adapter/run_luna.yaml @@ -19,9 +19,9 @@ # sandbox.evolver.allowed_hosts=[chatgpt.com], except that the learner on a ChatGPT login is now also allowed # auth.openai.com (README.md). data: - tasks: rows/tasks.jsonl # make_rows.py: ../utils/tasks.jsonl plus k trial rows per training question + tasks: rows/tasks.jsonl # make_rows.py: ../utils/benchmark/tasks plus k trial rows per training question train: rows/train_k2.txt # the evolve set: 50 questions x k = 2 - test: ../utils/splits/test.txt + test: ../utils/benchmark/splits/test.txt step_size: 100 # the whole evolve set per step: each step evaluates one candidate shuffle: false task_info: task-info/ @@ -45,9 +45,9 @@ agent: project_doc_max_bytes: 65536 # 64 KiB, as every OfficeQA Codex config: past Codex's default 32 KiB it would # cut AGENTS.md (deploy refuses that) judge: - path: trials.py # ../utils/judge.py on the row's question + path: trials.py # ../utils/benchmark/judge.py on the row's question template: rrsi-adapter@1.0 -start_from: seed_adapter # make_seed.py writes it from ../utils/seed_adapter +start_from: seed_adapter # make_seed.py writes it from ../utils/benchmark/seed_adapter evolver: name: rrsi_adapter params: # Layer 3: the method diff --git a/examples/officeqa/rrsi_adapter/run_luna_eng.yaml b/examples/officeqa/rrsi_adapter/run_luna_eng.yaml index 54f62ac0..5c4c7681 100644 --- a/examples/officeqa/rrsi_adapter/run_luna_eng.yaml +++ b/examples/officeqa/rrsi_adapter/run_luna_eng.yaml @@ -22,9 +22,9 @@ # sandbox.evolver.allowed_hosts=[chatgpt.com], except that the learner on a ChatGPT login is now also allowed # auth.openai.com (README.md). data: - tasks: rows/tasks.jsonl # make_rows.py: ../utils/tasks.jsonl plus k trial rows per training question + tasks: rows/tasks.jsonl # make_rows.py: ../utils/benchmark/tasks plus k trial rows per training question train: rows/train_k2.txt # the evolve set: 50 questions x k = 2 - test: ../utils/splits/test.txt + test: ../utils/benchmark/splits/test.txt step_size: 100 # the whole evolve set per step: each step evaluates one candidate shuffle: false task_info: task-info/ @@ -48,9 +48,9 @@ agent: project_doc_max_bytes: 65536 # 64 KiB, as every OfficeQA Codex config: past Codex's default 32 KiB it would # cut AGENTS.md (deploy refuses that) judge: - path: trials.py # ../utils/judge.py on the row's question + path: trials.py # ../utils/benchmark/judge.py on the row's question template: rrsi-adapter@1.0 -start_from: seed_adapter # make_seed.py writes it from ../utils/seed_adapter +start_from: seed_adapter # make_seed.py writes it from ../utils/benchmark/seed_adapter evolver: name: rrsi_adapter params: # Layer 3: the method diff --git a/examples/officeqa/rrsi_adapter/smoke_luna.yaml b/examples/officeqa/rrsi_adapter/smoke_luna.yaml index f87c0558..dabfc00f 100644 --- a/examples/officeqa/rrsi_adapter/smoke_luna.yaml +++ b/examples/officeqa/rrsi_adapter/smoke_luna.yaml @@ -1,9 +1,9 @@ # A smoke of run_luna.yaml: the 4 smoke questions x k = 2, one round of two candidates (three steps), real Codex # sessions on the ChatGPT login for the agent, the proposer and the roles; the 4 smoke test questions for `hermit eval`. data: - tasks: rows/tasks.jsonl # make_rows.py: ../utils/tasks.jsonl plus k trial rows per training question + tasks: rows/tasks.jsonl # make_rows.py: ../utils/benchmark/tasks plus k trial rows per training question train: rows/smoke_k2.txt # the 4 smoke questions x k = 2 - test: ../utils/splits/smoke_test.txt + test: ../utils/benchmark/splits/smoke_test.txt step_size: 8 shuffle: false task_info: task-info/ @@ -27,9 +27,9 @@ agent: project_doc_max_bytes: 65536 # 64 KiB, as every OfficeQA Codex config: past Codex's default 32 KiB it would # cut AGENTS.md (deploy refuses that) judge: - path: trials.py # ../utils/judge.py on the row's question + path: trials.py # ../utils/benchmark/judge.py on the row's question template: rrsi-adapter@1.0 -start_from: seed_adapter # make_seed.py writes it from ../utils/seed_adapter +start_from: seed_adapter # make_seed.py writes it from ../utils/benchmark/seed_adapter evolver: name: rrsi_adapter params: # Layer 3: the method diff --git a/examples/officeqa/rrsi_adapter/trials.py b/examples/officeqa/rrsi_adapter/trials.py index 5c1d9db7..3c33cb3d 100644 --- a/examples/officeqa/rrsi_adapter/trials.py +++ b/examples/officeqa/rrsi_adapter/trials.py @@ -5,14 +5,13 @@ from pathlib import Path import re -import sys +import threading from hermit.agent.acting.workspace import CliAgent +from hermit.common import load_callable HERE = Path(__file__).resolve().parent -sys.path.insert(0, str(HERE.parent / "utils")) - -import judge as _judge # noqa: E402 +UTILS = HERE.parent / "utils" def question(row_id: str) -> str: @@ -27,6 +26,13 @@ def run(self, task: dict, adapter): return super().run({**task, "id": question(task["id"])}, adapter) +_JUDGE: list = [] # the score module, loaded once (its answers with it) +_LOADING = threading.Lock() + + def score(task: dict, output) -> dict: - """`judge.path: trials.py`: SkillOpt's exact match (../utils/judge.py) on the row's question.""" - return _judge.score({**task, "id": question(task["id"])}, output) + """`judge.path: trials.py`: SkillOpt's exact match (../utils/benchmark/judge.py) on the row's question.""" + with _LOADING: + if not _JUDGE: + _JUDGE.append(load_callable(str(UTILS / "benchmark" / "judge.py") + ":score", UTILS)) + return _JUDGE[0]({**task, "id": question(task["id"])}, output) diff --git a/examples/officeqa/skillopt/run.yaml b/examples/officeqa/skillopt/run.yaml index 52145853..cd3e8c37 100644 --- a/examples/officeqa/skillopt/run.yaml +++ b/examples/officeqa/skillopt/run.yaml @@ -2,20 +2,21 @@ # match, and its SkillOpt hyperparameters: 4 epochs of batches of 40 (2 steps each), minibatch 8, merge 8, learning # rate 4 decaying to 2, validation gate, gated slow update, meta skill. GLM 5.3 Flash acts. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/train.txt - val: ../utils/splits/val.txt - test: ../utils/splits/test.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: ../utils/benchmark/splits/train.txt + val: ../utils/benchmark/splits/val.txt + test: ../utils/benchmark/splits/test.txt step_size: 40 shuffle_seed: 42 -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: name: ../utils/agent/agent.py:OfficeQAAgent concurrency: 8 judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: prompt-only@1.0 -start_from: ../utils/seed_adapter # SkillOpt's initial OfficeQA skill +start_from: ../utils/benchmark/seed_adapter # SkillOpt's initial OfficeQA skill evolver: name: skillopt params: @@ -39,7 +40,7 @@ llm: # the optimizer: GLM 5.3 at medium effort (the pa model: fireworks/accounts/fireworks/models/glm-5p3 # /; the price is the backend's extra_body: {reasoning_effort: medium} environment: # the task sandbox: the agent's code, its data and the corpus - read: [../utils/agent, ../utils/data.json, ../utils/documents] + read: [../utils/agent, ../utils/benchmark/data.json, ../utils/benchmark/documents] packages: [skillopt] sandbox: enabled: true diff --git a/examples/officeqa/skillopt/run_claude_haiku.yaml b/examples/officeqa/skillopt/run_claude_haiku.yaml index ecc420e4..260159c6 100644 --- a/examples/officeqa/skillopt/run_claude_haiku.yaml +++ b/examples/officeqa/skillopt/run_claude_haiku.yaml @@ -9,13 +9,14 @@ # the egress proxy lets through api.anthropic.com; each learner call is one session in a bubblewrap sandbox of its own, # as the luna learner's is (docs/guides/claude-subscription.md; the recipe's "The Flash row with Claude Haiku 5.5"). data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/train.txt - val: ../utils/splits/val.txt - test: ../utils/splits/test.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: ../utils/benchmark/splits/train.txt + val: ../utils/benchmark/splits/val.txt + test: ../utils/benchmark/splits/test.txt step_size: 40 shuffle_seed: 42 -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: base: claude # Layer 2: Claude Code as a Base Agent (hermit/agent/base/cli.py) concurrency: 8 @@ -33,9 +34,9 @@ agent: web_search: false # set explicitly: WebSearch and WebFetch denied (they run at Anthropic) timeout_s: 3600 judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: prompt-only@1.0 -start_from: ../utils/seed_adapter # SkillOpt's initial OfficeQA skill +start_from: ../utils/benchmark/seed_adapter # SkillOpt's initial OfficeQA skill evolver: name: skillopt params: diff --git a/examples/officeqa/skillopt/run_codex.yaml b/examples/officeqa/skillopt/run_codex.yaml index 1fb734ba..62c83f1e 100644 --- a/examples/officeqa/skillopt/run_codex.yaml +++ b/examples/officeqa/skillopt/run_codex.yaml @@ -6,13 +6,14 @@ # per session). The published Codex-harness numbers were made with SkillOpt's own Codex harness (the skill as # SKILL.md, 180 s sessions), which this replaces. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/train.txt - val: ../utils/splits/val.txt - test: ../utils/splits/test.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: ../utils/benchmark/splits/train.txt + val: ../utils/benchmark/splits/val.txt + test: ../utils/benchmark/splits/test.txt step_size: 40 shuffle_seed: 42 -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: base: codex # Layer 2: Codex as a Base Agent (hermit/agent/base/cli.py) concurrency: 8 @@ -31,9 +32,9 @@ agent: project_doc_max_bytes: 65536 # 64 KiB: the skill (up to 40 KB) with the deployment's notes; past # Codex's default 32 KiB it would cut AGENTS.md (deploy refuses that) judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: prompt-only@1.0 -start_from: ../utils/seed_adapter # SkillOpt's initial OfficeQA skill +start_from: ../utils/benchmark/seed_adapter # SkillOpt's initial OfficeQA skill evolver: name: skillopt params: diff --git a/examples/officeqa/skillopt/run_codex_flash.yaml b/examples/officeqa/skillopt/run_codex_flash.yaml index ada568e0..d34e9279 100644 --- a/examples/officeqa/skillopt/run_codex_flash.yaml +++ b/examples/officeqa/skillopt/run_codex_flash.yaml @@ -9,13 +9,14 @@ # are evaluated once, and the Codex image is officeqa-codex:0.162.0 (the # README table: Codex 0.134.0, GLM 5.3 learns, three passes). The recipe's Flash row. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/train.txt - val: ../utils/splits/val.txt - test: ../utils/splits/test.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: ../utils/benchmark/splits/train.txt + val: ../utils/benchmark/splits/val.txt + test: ../utils/benchmark/splits/test.txt step_size: 40 shuffle_seed: 42 -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: base: codex # Layer 2: Codex as a Base Agent (hermit/agent/base/cli.py) concurrency: 8 @@ -34,9 +35,9 @@ agent: project_doc_max_bytes: 65536 # 64 KiB: the skill (up to 40 KB) with the deployment's notes; past # Codex's default 32 KiB it would cut AGENTS.md (deploy refuses that) judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: prompt-only@1.0 -start_from: ../utils/seed_adapter # SkillOpt's initial OfficeQA skill +start_from: ../utils/benchmark/seed_adapter # SkillOpt's initial OfficeQA skill evolver: name: skillopt params: diff --git a/examples/officeqa/skillopt/run_codex_luna.yaml b/examples/officeqa/skillopt/run_codex_luna.yaml index 9a7e1594..f5a32354 100644 --- a/examples/officeqa/skillopt/run_codex_luna.yaml +++ b/examples/officeqa/skillopt/run_codex_luna.yaml @@ -7,13 +7,14 @@ # (docker/codex/Dockerfile.standalone) and the egress proxy lets through chatgpt.com and auth.openai.com (the login's # refresh): the recipe's "The Flash row with GPT-5.6-Luna". (Ported from PR #54.) data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/train.txt - val: ../utils/splits/val.txt - test: ../utils/splits/test.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: ../utils/benchmark/splits/train.txt + val: ../utils/benchmark/splits/val.txt + test: ../utils/benchmark/splits/test.txt step_size: 40 shuffle_seed: 42 -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: base: codex # Layer 2: Codex as a Base Agent (hermit/agent/base/cli.py) concurrency: 8 @@ -33,9 +34,9 @@ agent: project_doc_max_bytes: 65536 # 64 KiB: the skill (up to 40 KB) with the deployment's notes; past # Codex's default 32 KiB it would cut AGENTS.md (deploy refuses that) judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: prompt-only@1.0 -start_from: ../utils/seed_adapter # SkillOpt's initial OfficeQA skill +start_from: ../utils/benchmark/seed_adapter # SkillOpt's initial OfficeQA skill evolver: name: skillopt params: diff --git a/examples/officeqa/skillopt/smoke.yaml b/examples/officeqa/skillopt/smoke.yaml index 142d4ea3..11a436bc 100644 --- a/examples/officeqa/skillopt/smoke.yaml +++ b/examples/officeqa/skillopt/smoke.yaml @@ -1,19 +1,20 @@ # Smoke: 4 training questions in 2 epochs of 2 steps, 2 validation questions, 4 test questions. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/smoke_train.txt - val: ../utils/splits/smoke_val.txt - test: ../utils/splits/smoke_test.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: ../utils/benchmark/splits/smoke_train.txt + val: ../utils/benchmark/splits/smoke_val.txt + test: ../utils/benchmark/splits/smoke_test.txt step_size: 2 shuffle_seed: 42 -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: name: ../utils/agent/agent.py:OfficeQAAgent concurrency: 4 judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: prompt-only@1.0 -start_from: ../utils/seed_adapter # SkillOpt's initial OfficeQA skill +start_from: ../utils/benchmark/seed_adapter # SkillOpt's initial OfficeQA skill evolver: name: skillopt params: @@ -37,7 +38,7 @@ llm: # the optimizer: GLM 5.3 at medium effort (the pa model: fireworks/accounts/fireworks/models/glm-5p3 # /; the price is the backend's extra_body: {reasoning_effort: medium} environment: # the task sandbox: the agent's code, its data and the corpus - read: [../utils/agent, ../utils/data.json, ../utils/documents] + read: [../utils/agent, ../utils/benchmark/data.json, ../utils/benchmark/documents] packages: [skillopt] sandbox: enabled: true diff --git a/examples/officeqa/skillopt/smoke_codex.yaml b/examples/officeqa/skillopt/smoke_codex.yaml index a4832a35..06edb005 100644 --- a/examples/officeqa/skillopt/smoke_codex.yaml +++ b/examples/officeqa/skillopt/smoke_codex.yaml @@ -4,13 +4,14 @@ # per session). The published Codex-harness numbers were made with SkillOpt's own Codex harness (the skill as # SKILL.md, 180 s sessions), which this replaces. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/smoke_train.txt - val: ../utils/splits/smoke_val.txt - test: ../utils/splits/smoke_test.txt + tasks: ../utils/benchmark/tasks + view: ../utils/agent/officeqa_fields.py:officeqa_task # each task as the row SkillOpt's loader gives + train: ../utils/benchmark/splits/smoke_train.txt + val: ../utils/benchmark/splits/smoke_val.txt + test: ../utils/benchmark/splits/smoke_test.txt step_size: 2 shuffle_seed: 42 -task_info: ../utils/task-info/ +task_info: ../utils/benchmark/task-info/ agent: base: codex # Layer 2: Codex as a Base Agent (hermit/agent/base/cli.py) concurrency: 4 @@ -29,9 +30,9 @@ agent: project_doc_max_bytes: 65536 # 64 KiB: the skill (up to 40 KB) with the deployment's notes; past # Codex's default 32 KiB it would cut AGENTS.md (deploy refuses that) judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: prompt-only@1.0 -start_from: ../utils/seed_adapter # SkillOpt's initial OfficeQA skill +start_from: ../utils/benchmark/seed_adapter # SkillOpt's initial OfficeQA skill evolver: name: skillopt params: diff --git a/examples/officeqa/utils/.gitignore b/examples/officeqa/utils/.gitignore index 7967661f..f431ecee 100644 --- a/examples/officeqa/utils/.gitignore +++ b/examples/officeqa/utils/.gitignore @@ -1,3 +1,5 @@ +# Where utils/prepare.py wrote before utils/benchmark/ (answers among them): kept out of git in a checkout prepared +# then .private/ tasks.jsonl splits/ diff --git a/examples/officeqa/utils/agent/agent.py b/examples/officeqa/utils/agent/agent.py index f090140e..bc5d0df5 100644 --- a/examples/officeqa/utils/agent/agent.py +++ b/examples/officeqa/utils/agent/agent.py @@ -26,9 +26,10 @@ from hermit.agent.acting import Agent from hermit.task import RunOutput +from officeqa_fields import officeqa_task # beside this file (load_callable puts its folder on the path) HERE = Path(__file__).resolve().parent -DATA = json.loads((HERE.parent / "data.json").read_text()) +DATA = json.loads((HERE.parent / "benchmark" / "data.json").read_text()) DEFAULT_MODEL = "fireworks/accounts/fireworks/models/glm-5p3-flash" EFFORT = "medium" # paper p.8: teacher and student at medium MAX_TURNS, MAX_TOKENS = 24, 16384 # configs/officeqa/default.yaml; issue #13 @@ -151,6 +152,7 @@ class OfficeQAAgent(Agent): def run(self, task: dict, adapter: Path | None) -> RunOutput: _setup(self.params) from skillopt.envs.officeqa.rollout import process_one + task = officeqa_task(task) # a task directory's row as the row the rollout always read session = self.session(adapter) # The question is the instance's input: what the adapter's context tool, if it declares one, retrieves for. @@ -162,7 +164,7 @@ def run(self, task: dict, adapter: Path | None) -> RunOutput: from skillopt.envs.officeqa.tool_runtime import build_oracle_parsed_pages_context, resolve_docs_roots if not build_oracle_parsed_pages_context(item["source_files"], item["source_docs"], resolve_docs_roots([DATA["docs"]])): # cached: the rollout rereads them - raise RuntimeError(f"no parsed pages for {task['id']} under {DATA['docs']}: see utils/prepare.py") + raise RuntimeError(f"no parsed pages for {task['id']} under {DATA['docs']}: see utils/benchmark/prepare.py") started = time.time() for attempt in range(3): # the API was unavailable: run the whole rollout again later out_root = Path(tempfile.mkdtemp(prefix=f"{task['id']}-", dir=logs)) diff --git a/examples/officeqa/utils/agent/officeqa_fields.py b/examples/officeqa/utils/agent/officeqa_fields.py new file mode 100644 index 00000000..ec133018 --- /dev/null +++ b/examples/officeqa/utils/agent/officeqa_fields.py @@ -0,0 +1,29 @@ +"""An OfficeQA task as the agents read it: {id, question, category, source_files, source_docs}, the row SkillOpt's +OfficeQA loader gives. A task directory (utils/benchmark/prepare.py) has the question as its instruction and the rest +under task.toml's [metadata]; this puts them back, so the agent, its prompt and an adapter's context tool see exactly +the row they saw when the task set was a tasks.jsonl.""" +from __future__ import annotations + +from pathlib import Path + + +def officeqa_task(task: dict) -> dict: + if "instruction" not in task: # already a row of that shape + return dict(task) + meta = task.get("metadata") or {} + return {"id": task["id"], "question": task["instruction"], "category": meta["category"], + "source_files": list(meta["source_files"]), "source_docs": list(meta["source_docs"])} + + +def question_rows(tasks: Path, splits: Path) -> list[dict]: + """Every question's row as the agents read it, in the splits' order (train, validation, test; a split file that + is not there is skipped): from the task directories of utils/benchmark/prepare.py, or from a tasks.jsonl.""" + import json + from hermit.task import read_ids + from hermit.task.directory import tasks_of + if not Path(tasks).is_dir(): + return [json.loads(line) for line in Path(tasks).read_text(encoding="utf-8").splitlines() if line.strip()] + found = {tid: officeqa_task(task.given()) for tid, task in tasks_of(Path(tasks)).items()} + order = [i for name in ("train", "val", "test") if (Path(splits) / f"{name}.txt").is_file() + for i in read_ids(Path(splits) / f"{name}.txt")] + return [found[i] for i in dict.fromkeys([*order, *sorted(found)]) if i in found] diff --git a/examples/officeqa/utils/benchmark/.gitignore b/examples/officeqa/utils/benchmark/.gitignore new file mode 100644 index 00000000..f9b74285 --- /dev/null +++ b/examples/officeqa/utils/benchmark/.gitignore @@ -0,0 +1,6 @@ +.private/ +tasks/ +splits/ +data.json +seed_adapter/ +documents diff --git a/examples/officeqa/utils/judge.py b/examples/officeqa/utils/benchmark/judge.py similarity index 100% rename from examples/officeqa/utils/judge.py rename to examples/officeqa/utils/benchmark/judge.py diff --git a/examples/officeqa/utils/benchmark/prepare.py b/examples/officeqa/utils/benchmark/prepare.py new file mode 100644 index 00000000..51c620b1 --- /dev/null +++ b/examples/officeqa/utils/benchmark/prepare.py @@ -0,0 +1,139 @@ +#!/usr/bin/env python3 +"""Write the SkillOpt paper's OfficeQA split (arXiv 2605.23904: 50 train, 24 validation, 172 test) for this example. + + python prepare.py --csv officeqa_full.csv --docs treasury_bulletins_parsed --split-dir + SkillOpt/data/officeqa_id_split + +--csv databricks/officeqa officeqa_full.csv (questions and answers) +--docs databricks/officeqa treasury_bulletins_parsed/ with transformed/ (text) and jsons/ (page-level parse) +--split-dir SkillOpt's data/officeqa_id_split (ids, source files and pages, at the paper's dataset revision); + without it the files are fetched from github.com/microsoft/SkillOpt at the pinned commit. + +Writes here (utils/benchmark/): tasks/ (one task directory per question in Harbor's format: instruction.md, the +question; task.toml, its category and sources under [metadata]; no tests/: the judge, judge.py, runs on the host on +the answers only it reads), splits/, data.json, seed_adapter/ (SkillOpt's initial OfficeQA skill as a prompt-only +adapter) and documents (a link to --docs, what the agent's sandbox is granted), and the answers to +.private/answers.jsonl. +""" +import argparse +import csv +import json +from pathlib import Path +import shutil +import urllib.request + +HERE = Path(__file__).resolve().parent +SKILLOPT_COMMIT = "79124b37e9a6371e13b753f8bcd7adb1e493ade1" + + +def split_items(split_dir: str | None, name: str) -> list[dict]: + if split_dir: + return json.loads((Path(split_dir) / name / "items.json").read_text()) + url = (f"https://raw.githubusercontent.com/microsoft/SkillOpt/{SKILLOPT_COMMIT}/data/officeqa_id_split/" + f"{name}/items.json") + with urllib.request.urlopen(url, timeout=60) as response: + return json.loads(response.read()) + + +def as_list(value) -> list[str]: + from skillopt.envs.officeqa.dataloader import _parse_list_field # SkillOpt's own parsing of these fields + return _parse_list_field(value) + + +def _toml_string(value: str) -> str: + return json.dumps(value, ensure_ascii=False) # a JSON string is a TOML basic string + + +def write_task(tasks: Path, row: dict) -> Path: + """One question as a task directory: instruction.md (the question), task.toml ([metadata]: category, + source_files, source_docs; [environment]: no network). Read back by hermit.task.directory; + utils/agent/officeqa_fields.py turns it into the row the agent reads.""" + d = tasks / row["id"] + d.mkdir(parents=True, exist_ok=True) + (d / "instruction.md").write_text(row["question"] + "\n", encoding="utf-8") + lines = ['version = "1.0"', "", "[metadata]", f"category = {_toml_string(row['category'])}", + f"source_files = [{', '.join(_toml_string(s) for s in row['source_files'])}]", + f"source_docs = [{', '.join(_toml_string(s) for s in row['source_docs'])}]", "", + # The agent needs no network but its model (the run's sandbox reaches that): said, since a task that says + # nothing asks for the public network (Harbor's default) + "[environment]", 'network_mode = "no-network"'] + (d / "task.toml").write_text("\n".join(lines) + "\n", encoding="utf-8") + return d + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--csv", required=True, type=Path) + parser.add_argument("--docs", required=True, type=Path) + parser.add_argument("--split-dir") + args = parser.parse_args() + + docs = args.docs.resolve() + for sub in ("transformed", "jsons"): + if not (docs / sub).is_dir(): + raise SystemExit(f"{docs / sub} is missing: --docs is treasury_bulletins_parsed/ with transformed/ and " + "jsons/") + # The documents, linked beside the task: the configs grant the agent's sandbox this link (.private/ is never + # granted). + link = HERE / "documents" + if link.is_symlink() or link.exists(): + link.unlink() + link.symlink_to(docs, target_is_directory=True) + rows = {r["uid"]: r for r in csv.DictReader(args.csv.open(encoding="utf-8", newline=""))} + private = HERE / ".private" + private.mkdir(mode=0o700, exist_ok=True) + (HERE / "splits").mkdir(exist_ok=True) + tasks, answers, splits, drifted = [], [], {}, [] + for name in ("train", "val", "test"): + items = split_items(args.split_dir, name) + splits[name] = [item["uid"] for item in items] + for item in items: + row = rows[item["uid"]] + # Every field comes from the CSV, so question, answer and pages agree. Items whose sources differ from the + # split's (the paper's dataset revision) are listed in data.json as drifted. + source_files, source_docs = as_list(row["source_files"]), as_list(row["source_docs"]) + if (source_files, source_docs) != (as_list(item["source_files"]), as_list(item["source_docs"])): + drifted.append(item["uid"]) + tasks.append({"id": item["uid"], "question": row["question"].strip(), "category": item["category"], + "source_files": source_files, "source_docs": source_docs}) + answers.append({"id": item["uid"], "answer": row["answer"].strip()}) + if (HERE / "tasks").exists(): # this script's own output: rewritten whole + shutil.rmtree(HERE / "tasks") + for row in tasks: + write_task(HERE / "tasks", row) + with open(private / "answers.jsonl", "w") as f: + f.writelines(json.dumps(a, ensure_ascii=False) + "\n" for a in answers) + (private / "answers.jsonl").chmod(0o600) + for name, ids in splits.items(): + (HERE / "splits" / f"{name}.txt").write_text("\n".join(ids) + "\n") + (HERE / "splits" / "smoke_train.txt").write_text("\n".join(splits["train"][:4]) + "\n") + (HERE / "splits" / "smoke_val.txt").write_text("\n".join(splits["val"][:2]) + "\n") + (HERE / "splits" / "smoke_test.txt").write_text("\n".join(splits["test"][:4]) + "\n") + # SkillOpt reads a question's parsed pages from jsons/ beside the directory transformed/ resolves to, symlinks + # followed: a transformed/ linked in from a folder without jsons/ leaves every prompt without its pages. + from skillopt.envs.officeqa.tool_runtime import build_oracle_parsed_pages_context, resolve_docs_roots + roots = resolve_docs_roots([str(docs)]) + without = [t["id"] for t in tasks + if not build_oracle_parsed_pages_context(t["source_files"], t["source_docs"], roots)] + if without: + raise SystemExit(f"{len(without)} of {len(tasks)} questions get no parsed pages: SkillOpt looks for jsons/ " + "beside " + f"{roots[0]}. Put transformed/ and jsons/ side by side as real directories.") + (HERE / "data.json").write_text(json.dumps({"docs": str(docs), "answers": str(private / "answers.jsonl"), + "csv": str(args.csv.resolve()), "drifted_from_split": drifted}, + indent=1)) + + # SkillOpt's starting skill for OfficeQA, as a prompt-only adapter (start_from in the configs). + import hermit + import skillopt + seed = HERE / "seed_adapter" + if seed.exists(): + shutil.rmtree(seed) + shutil.copytree(Path(hermit.__file__).parent / "adapter" / "templates" / "prompt_only", seed) + (seed / "ADAPTER.md").write_text((Path(skillopt.__file__).parent / "envs" / "officeqa" / "skills" / "initial.md") + .read_text()) + print({name: len(ids) for name, ids in splits.items()}, "sources differing from the split:", drifted) + + +if __name__ == "__main__": + main() diff --git a/examples/officeqa/utils/task-info/README.md b/examples/officeqa/utils/benchmark/task-info/README.md similarity index 100% rename from examples/officeqa/utils/task-info/README.md rename to examples/officeqa/utils/benchmark/task-info/README.md diff --git a/examples/officeqa/utils/harbor/build_tasks.py b/examples/officeqa/utils/harbor/build_tasks.py index 51cb252a..ed2b0619 100644 --- a/examples/officeqa/utils/harbor/build_tasks.py +++ b/examples/officeqa/utils/harbor/build_tasks.py @@ -1,22 +1,25 @@ #!/usr/bin/env python3 -"""OfficeQA in the SkillOpt paper's setting as a task set in Harbor's format: one task directory per question, run by -`agent.name: harbor` (the Codex harness in a container Harbor starts, instead of one the acting agent's environment.image starts). +"""OfficeQA in the SkillOpt paper's setting as a task set in Harbor's format: one task directory per question, each run +in its own Harbor environment (`environment.harbor`: the Codex harness in a container Harbor starts, instead of one the +acting agent's environment.image starts). - python examples/officeqa/utils/harbor/build_tasks.py # after utils/prepare.py - [--out utils/.private/harbor_tasks] [--ids utils/splits/test.txt] [--limit N] [--base-image ubuntu:24.04] + python examples/officeqa/utils/harbor/build_tasks.py # after utils/benchmark/prepare.py + [--out utils/benchmark/.private/harbor_tasks] [--ids utils/benchmark/splits/test.txt] [--limit N] [--base-image + ubuntu:24.04] Each task is `//`: `task.toml` (the question's metadata, a 3600 s agent budget as the Codex route's), -`instruction.md` (the task text SkillOpt's Codex harness writes to task.md: the question, its oracle parsed pages, -the corpus note, the candidate files, the source hints, with the answer instruction), `environment/Dockerfile` +`instruction.md` (the task text SkillOpt's Codex harness writes to task.md: the question, its oracle parsed pages, the +corpus note, the candidate files, the source hints, with the answer instruction), `environment/Dockerfile` (`--base-image` plus python3, ripgrep and CA certificates; the same for every task, so Harbor builds one image), and -`tests/` (`test.sh`, `judge.py`, `gold.json`: the exact-match judge of utils/judge.py, run inside the container after -the agent, so the answer is never where the agent can read it). The corpus is not in the image: the config mounts -it read-only at `/app/docs/root_1`, where the instruction's candidate paths point (`MOUNTS.json` beside the tasks -says which host directory). `/tasks.jsonl` holds the rows for `data.tasks` (the same UIDs as utils/tasks.jsonl, -so utils/splits/ apply unchanged). +`tests/` (`test.sh`, `judge.py`, `gold.json`: the exact-match judge of utils/benchmark/judge.py, run inside the +container after the agent, so the answer is never where the agent can read it). The corpus is not in the image: the +config mounts it read-only at `/app/docs/root_1`, where the instruction's candidate paths point (`MOUNTS.json` beside +the tasks says which host directory). `/tasks.jsonl` holds the rows for `data.tasks` (the same UIDs as +utils/benchmark/tasks, so utils/benchmark/splits/ apply unchanged). Needs the skillopt extra (the instruction is rendered with SkillOpt's own `_build_user`) and the answers -utils/prepare.py wrote to utils/.private/answers.jsonl. The output is private: it holds the gold answers. +utils/benchmark/prepare.py wrote to utils/benchmark/.private/answers.jsonl. The output is private: it holds the gold +answers. """ from __future__ import annotations @@ -67,6 +70,14 @@ def task_toml(row: dict) -> str: "[environment]", "build_timeout_sec = 900.0", "cpus = 2", "memory_mb = 4096", 'workdir = "/app"', ""]) +def question_rows() -> list[dict]: + """Every question's row as the agents read it (utils/agent/officeqa_fields.py), from the task directories of + utils/benchmark/prepare.py, in the splits' order (train, validation, test).""" + from hermit.common import load_callable + rows = load_callable(str(UTILS / "agent" / "officeqa_fields.py") + ":question_rows", UTILS) + return rows(UTILS / "benchmark" / "tasks", UTILS / "benchmark" / "splits") + + def build(out: Path, rows: list[dict], answers: dict[str, str], docs: str, base_image: str) -> list[str]: from task import task_text # SkillOpt's task.md, and the corpus roots it names out.mkdir(parents=True, exist_ok=True) @@ -97,15 +108,15 @@ def build(out: Path, rows: list[dict], answers: dict[str, str], docs: str, base_ def main(argv=None) -> int: parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - parser.add_argument("--out", type=Path, default=UTILS / ".private" / "harbor_tasks") + parser.add_argument("--out", type=Path, default=UTILS / "benchmark" / ".private" / "harbor_tasks") parser.add_argument("--ids", type=Path, help="only these task ids (one per line; default: every task)") parser.add_argument("--limit", type=int) parser.add_argument("--base-image", default="ubuntu:24.04", help="the image the tasks build on; one with Codex installed saves Harbor's install per trial") - parser.add_argument("--docs", help="treasury_bulletins_parsed/ (default: utils/data.json's)") + parser.add_argument("--docs", help="treasury_bulletins_parsed/ (default: utils/benchmark/data.json's)") args = parser.parse_args(argv) - data = json.loads((UTILS / "data.json").read_text()) - rows = [json.loads(l) for l in (UTILS / "tasks.jsonl").read_text().splitlines() if l.strip()] + data = json.loads((UTILS / "benchmark" / "data.json").read_text()) + rows = question_rows() if args.ids: wanted = {l.strip() for l in args.ids.read_text().splitlines() if l.strip() and not l.startswith("#")} rows = [r for r in rows if r["id"] in wanted] @@ -114,7 +125,7 @@ def main(argv=None) -> int: answers = {r["id"]: r["answer"] for r in map(json.loads, Path(data["answers"]).read_text().splitlines()) if r} missing = [r["id"] for r in rows if r["id"] not in answers] if missing: - raise SystemExit(f"no answer for {missing[:5]}: run utils/prepare.py first") + raise SystemExit(f"no answer for {missing[:5]}: run utils/benchmark/prepare.py first") out = args.out.resolve() written = build(out, rows, answers, args.docs or data["docs"], args.base_image) from hermit.harbor import dataset @@ -125,7 +136,7 @@ def main(argv=None) -> int: dataset.write_jsonl(index, out / "tasks.jsonl") mounts = json.loads((out / "MOUNTS.json").read_text()) print(f"{len(written)} tasks -> {out} (tasks.jsonl beside them)") - print("the corpus mount for the config's agent.params.environment.mounts:") + print("the corpus mount for the config's environment.harbor.environment.mounts:") print(" export OFFICEQA_DOCS_ROOT=" + mounts[0]["source"]) return 0 diff --git a/examples/officeqa/utils/harbor/judge.py b/examples/officeqa/utils/harbor/judge.py index 512c67f3..3bbdd9d5 100644 --- a/examples/officeqa/utils/harbor/judge.py +++ b/examples/officeqa/utils/harbor/judge.py @@ -1,13 +1,13 @@ #!/usr/bin/env python3 """The OfficeQA judge inside the task container (copied to every task's tests/judge.py by build_tasks.py). -The answer is the text inside the last `...` of the agent's final message, read from Codex's -`--json` event stream in /logs/agent: HermitCodex's agent-codex.jsonl (Hermit's own launch of the session), else the -codex.txt Harbor's own Codex agent writes; or the content of /app/answer.txt when the agent wrote one (that file -wins). Scoring is SkillOpt's: exact match after its normalisation, no tolerance; token F1 is recorded beside -it. The normalisation is SkillOpt's envs/officeqa/evaluator.py (microsoft/SkillOpt, MIT), copied so that the -container needs nothing but python3. Writes /logs/verifier/reward.txt (the exact match) and reward.json -({reward, f1, answered}); prints the feedback utils/judge.py gives on the host.""" +The answer is the text inside the last `...` of the agent's final message, read from Codex's `--json` +event stream in /logs/agent: agent-codex.jsonl (the Codex agent's session, whose log Hermit hands the verifier there), +else the codex.txt Harbor's own Codex agent writes; or the content of /app/answer.txt when the agent wrote one (that +file wins). Scoring is SkillOpt's: exact match after its normalisation, no tolerance; token F1 is recorded beside it. +The normalisation is SkillOpt's envs/officeqa/evaluator.py (microsoft/SkillOpt, MIT), copied so that the container needs +nothing but python3. Writes /logs/verifier/reward.txt (the exact match) and reward.json ({reward, f1, answered}); prints +the feedback utils/benchmark/judge.py gives on the host.""" from __future__ import annotations from collections import Counter @@ -20,7 +20,7 @@ _NUMERIC_CHARS = set("0123456789.-") VERIFIER = Path("/logs/verifier") ANSWER_FILE = Path("/app/answer.txt") -STREAMS = (Path("/logs/agent/agent-codex.jsonl"), # HermitCodex's session (hermit/harbor/launch.py's stream) +STREAMS = (Path("/logs/agent/agent-codex.jsonl"), # the Codex agent's session (its log, Hermit's) Path("/logs/agent/codex.txt")) # Harbor's own Codex agent's (its `codex exec --json | tee`) diff --git a/examples/officeqa/utils/prepare.py b/examples/officeqa/utils/prepare.py deleted file mode 100644 index 19d3aeac..00000000 --- a/examples/officeqa/utils/prepare.py +++ /dev/null @@ -1,103 +0,0 @@ -#!/usr/bin/env python3 -"""Write the SkillOpt paper's OfficeQA split (arXiv 2605.23904: 50 train, 24 validation, 172 test) for this example. - - python prepare.py --csv officeqa_full.csv --docs treasury_bulletins_parsed --split-dir SkillOpt/data/officeqa_id_split - ---csv databricks/officeqa officeqa_full.csv (questions and answers) ---docs databricks/officeqa treasury_bulletins_parsed/ with transformed/ (text) and jsons/ (page-level parse) ---split-dir SkillOpt's data/officeqa_id_split (ids, source files and pages, at the paper's dataset revision); - without it the files are fetched from github.com/microsoft/SkillOpt at the pinned commit. - -Writes tasks.jsonl (questions only), splits/, data.json, seed_adapter/ (SkillOpt's initial OfficeQA skill as a -prompt-only adapter) and documents (a link to --docs, what the agent's sandbox is granted) here, and the answers to -.private/answers.jsonl. -""" -import argparse -import csv -import json -from pathlib import Path -import shutil -import urllib.request - -HERE = Path(__file__).resolve().parent -SKILLOPT_COMMIT = "79124b37e9a6371e13b753f8bcd7adb1e493ade1" - - -def split_items(split_dir: str | None, name: str) -> list[dict]: - if split_dir: - return json.loads((Path(split_dir) / name / "items.json").read_text()) - url = (f"https://raw.githubusercontent.com/microsoft/SkillOpt/{SKILLOPT_COMMIT}/data/officeqa_id_split/" - f"{name}/items.json") - with urllib.request.urlopen(url, timeout=60) as response: - return json.loads(response.read()) - - -def as_list(value) -> list[str]: - from skillopt.envs.officeqa.dataloader import _parse_list_field # SkillOpt's own parsing of these fields - return _parse_list_field(value) - - -parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) -parser.add_argument("--csv", required=True, type=Path) -parser.add_argument("--docs", required=True, type=Path) -parser.add_argument("--split-dir") -args = parser.parse_args() - -docs = args.docs.resolve() -for sub in ("transformed", "jsons"): - if not (docs / sub).is_dir(): - raise SystemExit(f"{docs / sub} is missing: --docs is treasury_bulletins_parsed/ with transformed/ and jsons/") -# The documents, linked beside the task: the configs grant the agent's sandbox this link (.private/ is never granted). -link = HERE / "documents" -if link.is_symlink() or link.exists(): - link.unlink() -link.symlink_to(docs, target_is_directory=True) -rows = {r["uid"]: r for r in csv.DictReader(args.csv.open(encoding="utf-8", newline=""))} -private = HERE / ".private" -private.mkdir(mode=0o700, exist_ok=True) -(HERE / "splits").mkdir(exist_ok=True) -tasks, answers, splits, drifted = [], [], {}, [] -for name in ("train", "val", "test"): - items = split_items(args.split_dir, name) - splits[name] = [item["uid"] for item in items] - for item in items: - row = rows[item["uid"]] - # Every field comes from the CSV, so question, answer and pages agree. Items whose sources differ from the - # split's (the paper's dataset revision) are listed in data.json as drifted. - source_files, source_docs = as_list(row["source_files"]), as_list(row["source_docs"]) - if (source_files, source_docs) != (as_list(item["source_files"]), as_list(item["source_docs"])): - drifted.append(item["uid"]) - tasks.append({"id": item["uid"], "question": row["question"].strip(), "category": item["category"], - "source_files": source_files, "source_docs": source_docs}) - answers.append({"id": item["uid"], "answer": row["answer"].strip()}) -with open(HERE / "tasks.jsonl", "w") as f: - f.writelines(json.dumps(t, ensure_ascii=False) + "\n" for t in tasks) -with open(private / "answers.jsonl", "w") as f: - f.writelines(json.dumps(a, ensure_ascii=False) + "\n" for a in answers) -(private / "answers.jsonl").chmod(0o600) -for name, ids in splits.items(): - (HERE / "splits" / f"{name}.txt").write_text("\n".join(ids) + "\n") -(HERE / "splits" / "smoke_train.txt").write_text("\n".join(splits["train"][:4]) + "\n") -(HERE / "splits" / "smoke_val.txt").write_text("\n".join(splits["val"][:2]) + "\n") -(HERE / "splits" / "smoke_test.txt").write_text("\n".join(splits["test"][:4]) + "\n") -# SkillOpt reads a question's parsed pages from jsons/ beside the directory transformed/ resolves to, symlinks -# followed: a transformed/ linked in from a folder without jsons/ leaves every prompt without its pages. -from skillopt.envs.officeqa.tool_runtime import build_oracle_parsed_pages_context, resolve_docs_roots -roots = resolve_docs_roots([str(docs)]) -without = [t["id"] for t in tasks if not build_oracle_parsed_pages_context(t["source_files"], t["source_docs"], roots)] -if without: - raise SystemExit(f"{len(without)} of {len(tasks)} questions get no parsed pages: SkillOpt looks for jsons/ beside " - f"{roots[0]}. Put transformed/ and jsons/ side by side as real directories.") -(HERE / "data.json").write_text(json.dumps({"docs": str(docs), "answers": str(private / "answers.jsonl"), - "csv": str(args.csv.resolve()), "drifted_from_split": drifted}, indent=1)) - -# SkillOpt's starting skill for OfficeQA, as a prompt-only adapter (start_from in the configs). -import hermit -import skillopt -seed = HERE / "seed_adapter" -if seed.exists(): - shutil.rmtree(seed) -shutil.copytree(Path(hermit.__file__).parent / "adapter" / "templates" / "prompt_only", seed) -(seed / "ADAPTER.md").write_text((Path(skillopt.__file__).parent / "envs" / "officeqa" / "skills" / "initial.md") - .read_text()) -print({name: len(ids) for name, ids in splits.items()}, "sources differing from the split:", drifted) diff --git a/examples/officeqa/utils/task.py b/examples/officeqa/utils/task.py index c4402d14..cc2a7e0a 100644 --- a/examples/officeqa/utils/task.py +++ b/examples/officeqa/utils/task.py @@ -11,7 +11,8 @@ The workspace holds the question as `task.md` (the question with its oracle parsed pages and its source files as hints: the text SkillOpt's own Codex harness writes) and the whole corpus under `docs/` (read-only). The corpus is bound at its own path too, so that its links (prepare.py may link `transformed/` and `jsons/`) resolve. The corpus -is `agent.params.docs`, else ../data.json's (utils/prepare.py writes it). +is `agent.params.docs`, else benchmark/data.json's (utils/benchmark/prepare.py writes it). A task directory's row is +read as the row SkillOpt's loader gives (agent/officeqa_fields.py). """ from __future__ import annotations @@ -20,14 +21,17 @@ import shutil from pathlib import Path +from hermit.common import load_callable + HERE = Path(__file__).resolve().parent +officeqa_task = load_callable(str(HERE / "agent" / "officeqa_fields.py") + ":officeqa_task", HERE) ANSWER = re.compile(r"(.*?)", re.IGNORECASE | re.DOTALL) CORPUS_NOTE = ("The full OfficeQA document corpus is available under `docs/`. The candidate files below are source " "hints or likely starting points; search the full corpus if needed.") def corpus(params: dict, base: Path) -> Path: - docs = params.get("docs") or json.loads((HERE / "data.json").read_text())["docs"] + docs = params.get("docs") or json.loads((HERE / "benchmark" / "data.json").read_text())["docs"] path = Path(docs).expanduser() return (path if path.is_absolute() else Path(base) / path).resolve() @@ -35,13 +39,14 @@ def corpus(params: dict, base: Path) -> Path: def task_text(task: dict, docs: Path) -> tuple[str, list[str]]: """`task.md` as SkillOpt's Codex harness writes it (its `_build_user`: the question, its oracle parsed pages, the corpus note, the candidate files as workspace paths, the source hints), and the corpus roots.""" + task = officeqa_task(task) from skillopt.envs.officeqa.rollout import _build_user, _workspace_doc_path from skillopt.envs.officeqa.tool_runtime import (build_oracle_parsed_pages_context, resolve_candidate_files, resolve_docs_roots) roots = resolve_docs_roots([str(docs)]) oracle = build_oracle_parsed_pages_context(task["source_files"], task["source_docs"], roots) if not oracle: - raise RuntimeError(f"no parsed pages for {task['id']} under {docs}: see utils/prepare.py") + raise RuntimeError(f"no parsed pages for {task['id']} under {docs}: see utils/benchmark/prepare.py") files = resolve_candidate_files(task["source_files"], roots) item = {"id": task["id"], "question": task["question"], "task_type": task.get("category", "officeqa"), "source_files": task["source_files"], "source_docs": task["source_docs"]} @@ -64,7 +69,7 @@ def prepare(task: dict, workspace: Path, params: dict, base: Path) -> list[dict] docs = corpus(params, base) for sub in ("transformed", "jsons"): if not (docs / sub).is_dir(): - raise RuntimeError(f"the corpus {docs} lacks {sub}/: see utils/prepare.py") + raise RuntimeError(f"the corpus {docs} lacks {sub}/: see utils/benchmark/prepare.py") text, roots = task_text(task, docs) (workspace / "task.md").write_text(text, encoding="utf-8") for i, root in enumerate(roots[1:], start=2): diff --git a/examples/officeqa/utils/tests/test_harbor_judge.py b/examples/officeqa/utils/tests/test_harbor_judge.py index 6931727e..4359bf5d 100644 --- a/examples/officeqa/utils/tests/test_harbor_judge.py +++ b/examples/officeqa/utils/tests/test_harbor_judge.py @@ -1,9 +1,10 @@ """OfficeQA's judge in a Harbor task's container (utils/harbor/judge.py, which build_tasks.py copies into every task's tests/): `find_answer` takes the `` of the final message from the Codex event stream a trial leaves in -/logs/agent, HermitCodex's agent-codex.jsonl or the codex.txt Harbor's own Codex agent writes. The container's paths +/logs/agent, the Codex agent's agent-codex.jsonl or the codex.txt Harbor's own Codex agent writes. The container's paths (the module's own ANSWER_FILE and STREAMS) are moved under tmp_path; the stream has the shape of those the OfficeQA GPT-5.6-Luna runs kept (Codex 0.164.0, `codex exec --json`): Codex's stderr line first, then its events. The name of -HermitCodex's stream is checked against the launch plan Hermit renders for a Codex trial (hermit.harbor.launch).""" +Codex agent's stream is checked against the name its log keeps it under (hermit.agent.acting.workspace), which the +run hands the verifier in /logs/agent.""" from __future__ import annotations import importlib.util @@ -55,32 +56,27 @@ def write(path: Path, stream: list) -> None: @pytest.mark.parametrize("stream", ["agent-codex.jsonl", "codex.txt"]) def test_an_answer_in_the_final_message_is_found_in_either_codex_stream(judge, stream): - """HermitCodex keeps the session's stream in agent-codex.jsonl; the judge read only codex.txt, which Harbor's own - Codex agent writes, so every answer given in a HermitCodex trial's final message scored 0.""" + """The Codex agent keeps the session's stream in agent-codex.jsonl; the judge read only codex.txt, which Harbor's + own Codex agent writes, so every answer given in a Hermit session's final message scored 0.""" found = [path for path in judge.STREAMS if path.name == stream] assert len(found) == 1 write(found[0], STREAM) assert judge.find_answer() == ("44,463", "final message") -def test_the_judge_reads_the_stream_the_launch_plan_names_for_a_codex_trial(tmp_path, monkeypatch): - """harborside writes a session's stream to /logs/agent/, a name Hermit's launch plan gives - (agent-.jsonl): a judge reading another name finds no final message, and every answer scores 0.""" - from hermit.agent.model.backends import Binding - from hermit.harbor import launch - monkeypatch.setenv("FAKE_MODEL_KEY", "fake-model-key") - binding = Binding.from_spec({"ref": "fake/glm-5", "backend": "fake", "model": "glm-5", - "base_url": "https://models.example/v1", "api_key_env": "FAKE_MODEL_KEY", - "key_required": True, "apis": ["chat", "responses", "messages"]}) - plan = json.loads(launch.render("codex", launch.settings_for("codex", binding=binding, web_search=False), - deploy=None, tools_url=None, job=tmp_path / "job").read_text()) - assert load().STREAMS[0] == Path("/logs/agent") / plan["stream"] +def test_the_judge_reads_the_stream_the_codex_agent_keeps(tmp_path): + """The Codex agent keeps a session's stream in its log as agent-.jsonl, the log the run hands Harbor's + verifier in /logs/agent: a judge reading another name finds no final message, and every answer scores 0.""" + from hermit.agent.base.cli import Codex + source = (Path(__file__).resolve().parents[4] / "hermit" / "agent" / "acting" / "workspace.py").read_text() + assert 'log=log / f"agent-{backend}.jsonl"' in source + assert load().STREAMS[0] == Path("/logs/agent") / f"agent-{Codex.backend}.jsonl" def test_a_line_of_codexs_stderr_that_is_json_but_no_object_is_skipped(judge): - """Codex's stderr shares the stream's file (harborside: `> stream 2>&1`): a line of it that parses as JSON but is - no object, a bare number, raised AttributeError in the judge before it wrote its reward, and test.sh's fallback - scored the trial 0: a correct answer lost.""" + """Codex's stderr shares the stream's file (the session's output, stdout and stderr together): a line of it that + parses as JSON but is no object, a bare number, raised AttributeError in the judge before it wrote its reward, and + test.sh's fallback scored the trial 0: a correct answer lost.""" write(judge.STREAMS[0], ["429", "null", *STREAM]) assert judge.find_answer() == ("44,463", "final message") diff --git a/examples/officeqa/utils/tests/test_task_directories.py b/examples/officeqa/utils/tests/test_task_directories.py new file mode 100644 index 00000000..618a2828 --- /dev/null +++ b/examples/officeqa/utils/tests/test_task_directories.py @@ -0,0 +1,34 @@ +"""OfficeQA's task directories (utils/benchmark/prepare.py) read back as the rows its agents always read: a question +written as a task directory and loaded by hermit.task gives exactly its row again (agent/officeqa_fields.py).""" +import importlib.util +from pathlib import Path + +from hermit.common import load_callable +from hermit.task import Benchmark + +UTILS = Path(__file__).resolve().parents[1] + + +def _prepare(): + spec = importlib.util.spec_from_file_location("officeqa_prepare", UTILS / "benchmark" / "prepare.py") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) # its work is in main(): importing it writes nothing + return module + + +def test_a_question_round_trips_through_its_task_directory(tmp_path): + officeqa_task = load_callable(str(UTILS / "agent" / "officeqa_fields.py") + ":officeqa_task", UTILS) + rows = [{"id": "UID0001", "question": 'What was the "total"\r\nin 1953? Use C:\\path and \u2019quotes\u2019.', + "category": "Hard", "source_files": ["treasury_bulletin_1953_01.txt"], + "source_docs": ["https://example.org/a?b=1&c=\"d\""]}, + {"id": "UID0002", "question": "Plain?", "category": "Easy", "source_files": [], "source_docs": []}] + prepare = _prepare() + for row in rows: + prepare.write_task(tmp_path / "tasks", row) + (tmp_path / "train.txt").write_text("UID0001\nUID0002\n") + bench = Benchmark.load(tmp_path / "tasks", tmp_path / "train.txt") + assert [officeqa_task(t.given()) for t in bench.split("train")] == rows + assert list(officeqa_task(bench.tasks["UID0001"].given())) == list(rows[0]) # the same keys, in order + assert officeqa_task(rows[1]) == rows[1] # a row as it is + assert bench.tasks["UID0001"].private["tests"] is None # judged on the host + assert bench.tasks["UID0001"].given()["network"]["agent"] == "no-network" # no network but its model diff --git a/examples/robodojo_sim/README.md b/examples/robodojo_sim/README.md index 2dbd4c22..22a571bc 100644 --- a/examples/robodojo_sim/README.md +++ b/examples/robodojo_sim/README.md @@ -27,8 +27,8 @@ what is left is listed under [Not verified](#not-verified). | `utils/agent/split_eval.sh` | One evaluation with the simulator in RoboDojo's container (`ROBODOJO_IMAGE`) | | `utils/agent/dojo_service.py`, `sim_cli.py` | The simulator service on the host and `bin/sim`, the actor's client of it (the ASPIRE example's service with this engine) | | `utils/agent/task.py`, `prompts/`, `docs/interface.md` | The Codex actor's task functions, its prompt and the interface page it is given | -| `utils/prepare.py` | The forty-two tasks with their development and held-out layouts | -| `utils/judge.py` | The program on the held-out layout sets, on the host | +| `utils/benchmark/prepare.py` | The forty-two tasks with their development and held-out layouts, as task directories in Harbor's format (`tasks/`, each read as its row through `utils/agent/robodojo_fields.py`, the configs' `data.view`), and `splits/` | +| `utils/benchmark/judge.py` | The program on the held-out layout sets, on the host | | `utils/control/press_by_number.py` | The positive control: press_by_number with one layout's digits written in (not shown to a session) | | `prompt_only/` | `run.yaml`, `smoke.yaml`, `run_luna.yaml` and `smoke_luna.yaml` (the same with GPT-5.6-Luna on the machine's ChatGPT login), `run_sample_luna.yaml` (three of the tasks on Luna, for a run of a day or two), `run_sample_sol.yaml` (the same on GPT-6.1-Sol), `run_sample_opus.yaml` (the same with Claude Code on Claude Opus 5.5, on a Claude subscription: `CLAUDE_CODE_OAUTH_TOKEN`), `launch.sh`: the prompt-only template with the built-in Codex Evolver | @@ -37,7 +37,7 @@ what is left is listed under [Not verified](#not-verified). RoboDojo's saved layouts come in sets (`--seed` 0, 1, 2) and an evaluation runs a set's first N. A task's row keeps development and scoring apart by set: a session develops on set 2 (its first 10 layouts), and the judge scores on sets 0 and 1, each as RoboDojo evaluates a set: its first 50 layouts, or 25 for the twelve tasks whose -`eval_nums` RoboDojo sets to 25 (`task/RoboDojo/config/_task.yml`; `utils/prepare.py`: `EVAL_25`). The score is the +`eval_nums` RoboDojo sets to 25 (`task/RoboDojo/config/_task.yml`; `utils/benchmark/prepare.py`: `EVAL_25`). The score is the fraction of those layouts that succeeded; a layout with no result counts as failed (RoboDojo drops a layout whose episode raised or whose scene went unstable and scores a later one instead, which a program must not gain by, so the later one is not counted). An evaluation that leaves no result at all (a simulator that crashed, an environment @@ -94,9 +94,9 @@ score is not a held-out score. ## Setup -RoboDojo is not in Harbor's task format, as `examples/README.md` asks of a benchmark: a task is developed and scored -on one Isaac Sim simulator per GPU, shared through a host service that keeps the development and held-out layout -sets apart, which Harbor's one container per trial does not express. Where Isaac Sim does not install natively, the +RoboDojo's task set is in Harbor's task format (`utils/benchmark/`), but its tasks do not run in Harbor's +environments: a task is developed and scored on one Isaac Sim simulator per GPU, shared through a host service that +keeps the development and held-out layout sets apart, which Harbor's one container per trial does not express. Where Isaac Sim does not install natively, the simulator runs in RoboDojo's own container image (below), the only Docker this example drives. 1. RoboDojo, as its documentation installs it (Python 3.11, Isaac Sim 5.1, Isaac Lab 2.3; an NVIDIA GPU with 16 GB @@ -132,7 +132,7 @@ ROBODOJO_GPU=0 ./launch.sh # the forty-two tasks ``` Run one launch at a time on a checkout of this repository and a RoboDojo checkout: each launch rewrites the task -rows and splits (`utils/prepare.py`) and reinstalls the policy into RoboDojo's checkout, which a running one reads. +rows and splits (`utils/benchmark/prepare.py`) and reinstalls the policy into RoboDojo's checkout, which a running one reads. ## Not verified diff --git a/examples/robodojo_sim/prompt_only/launch.sh b/examples/robodojo_sim/prompt_only/launch.sh index 215dd8c6..ca75e443 100755 --- a/examples/robodojo_sim/prompt_only/launch.sh +++ b/examples/robodojo_sim/prompt_only/launch.sh @@ -16,7 +16,7 @@ # # and the judge read both from the environment # ROBODOJO_OUT=/big/disk/robodojo ./launch.sh # the runs, the sessions, the held-out evaluations # -# The launcher writes the layouts into the task rows (utils/prepare.py), installs the harness's XPolicyLab policy +# The launcher writes the layouts into the tasks (utils/benchmark/prepare.py), installs the harness's XPolicyLab policy # into the checkout (utils/policy/install.py), starts the simulator service the actors reach through bin/sim # (utils/agent/dojo_service.py, on ROBODOJO_SIM_PORT, default 8130; stopped when the run ends) and runs hermit. The # model key (FIREWORKS_API_KEY), or the Claude subscription's token (CLAUDE_CODE_OAUTH_TOKEN), comes from the @@ -37,7 +37,7 @@ ACTUAL="$(git -C "$ROBODOJO_ROOT" rev-parse HEAD 2>/dev/null || echo unknown)" [[ "$ACTUAL" == "$PINNED" ]] || echo "note: ROBODOJO_ROOT is at $ACTUAL, not the pinned $PINNED" >&2 PRESET="${ROBODOJO_PRESET:-$([[ "${CONFIG##*/}" == smoke* ]] && echo smoke || echo full)}" -(cd ../utils && "$HERMIT" prepare.py --preset "$PRESET") +(cd ../utils/benchmark && "$HERMIT" prepare.py --preset "$PRESET") "$HERMIT" ../utils/policy/install.py --root "$ROBODOJO_ROOT" OUT="${ROBODOJO_OUT:-$HERE/../runs}" @@ -51,7 +51,7 @@ export ROBODOJO_ROOT ROBODOJO_OUT="$OUT" ROBODOJO_GPU="${ROBODOJO_GPU:-0}" \ PORT="${ROBODOJO_SIM_PORT:-8130}" NONCE="$$-$RANDOM-$(date +%s%N)" -"$HERMIT" ../utils/agent/dojo_service.py --port "$PORT" --robodojo-root "$ROBODOJO_ROOT" --tasks ../utils/tasks.jsonl \ +"$HERMIT" ../utils/agent/dojo_service.py --port "$PORT" --robodojo-root "$ROBODOJO_ROOT" --tasks ../utils/benchmark/tasks \ --workspaces "$OUT/actor-logs" --state "$OUT/sim-service" --gpu "$ROBODOJO_GPU" --nonce "$NONCE" \ --policy-env "$ROBODOJO_POLICY_ENV" --eval-env "$ROBODOJO_EVAL_ENV" --env-cfg "$ROBODOJO_ENV_CFG" \ --action-type "$ROBODOJO_ACTION_TYPE" > "$OUT/sim-service.log" 2>&1 & diff --git a/examples/robodojo_sim/prompt_only/run.yaml b/examples/robodojo_sim/prompt_only/run.yaml index e836ec1d..057dcdec 100644 --- a/examples/robodojo_sim/prompt_only/run.yaml +++ b/examples/robodojo_sim/prompt_only/run.yaml @@ -4,12 +4,13 @@ # Evolver revises the one learned ADAPTER.md, the next session's context, from the sessions' notes and scores. # Codex acting and learning, GLM 5.3 Flash. Started by launch.sh. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/all.txt - test: ../utils/splits/all.txt # `launch.sh eval`: the tasks again, with an adapter given + tasks: ../utils/benchmark/tasks + view: ../utils/agent/robodojo_fields.py:robodojo_task # each task as the row it always was + train: ../utils/benchmark/splits/all.txt + test: ../utils/benchmark/splits/all.txt # `launch.sh eval`: the tasks again, with an adapter given step_size: 1 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: base: codex concurrency: 1 # one simulator: an evaluation takes a GPU @@ -30,7 +31,7 @@ environment: # the task sandbox: the actor's code and the - ../utils/agent env: [ROBODOJO_SIM_URL] # the host's simulator service (launch.sh): its 127.0.0.1 port is forwarded judge: - path: ../utils/judge.py # on the host: the program on the held-out layout sets + path: ../utils/benchmark/judge.py # on the host: the program on the held-out layout sets template: prompt-only@1.0 evolver: name: codex # the built-in Codex AgentEvolver diff --git a/examples/robodojo_sim/prompt_only/run_luna.yaml b/examples/robodojo_sim/prompt_only/run_luna.yaml index e66971a1..dc05e2d6 100644 --- a/examples/robodojo_sim/prompt_only/run_luna.yaml +++ b/examples/robodojo_sim/prompt_only/run_luna.yaml @@ -1,12 +1,13 @@ # run.yaml with the models changed and nothing else: GPT-5.6-Luna at medium effort acts and learns on the machine's # own ChatGPT login (`codex login`; no API key). Started by launch.sh: ROBODOJO_CONFIG=run_luna.yaml ./launch.sh data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/all.txt - test: ../utils/splits/all.txt # `launch.sh eval`: the tasks again, with an adapter given + tasks: ../utils/benchmark/tasks + view: ../utils/agent/robodojo_fields.py:robodojo_task # each task as the row it always was + train: ../utils/benchmark/splits/all.txt + test: ../utils/benchmark/splits/all.txt # `launch.sh eval`: the tasks again, with an adapter given step_size: 1 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: base: codex concurrency: 1 # one simulator: an evaluation takes a GPU @@ -27,7 +28,7 @@ environment: # the task sandbox: the actor's code and the - ../utils/agent env: [ROBODOJO_SIM_URL] # the host's simulator service (launch.sh): its 127.0.0.1 port is forwarded judge: - path: ../utils/judge.py # on the host: the program on the held-out layout sets + path: ../utils/benchmark/judge.py # on the host: the program on the held-out layout sets template: prompt-only@1.0 evolver: name: codex # the built-in Codex AgentEvolver diff --git a/examples/robodojo_sim/prompt_only/run_sample_luna.yaml b/examples/robodojo_sim/prompt_only/run_sample_luna.yaml index 76b6bb9e..d418e022 100644 --- a/examples/robodojo_sim/prompt_only/run_sample_luna.yaml +++ b/examples/robodojo_sim/prompt_only/run_sample_luna.yaml @@ -2,12 +2,13 @@ # task of three dimensions), for a run of a day or two. Started by launch.sh: ROBODOJO_CONFIG=run_sample_luna.yaml # ./launch.sh data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/sample.txt - test: ../utils/splits/sample.txt # `launch.sh eval`: the tasks again, with an adapter given + tasks: ../utils/benchmark/tasks + view: ../utils/agent/robodojo_fields.py:robodojo_task # each task as the row it always was + train: ../utils/benchmark/splits/sample.txt + test: ../utils/benchmark/splits/sample.txt # `launch.sh eval`: the tasks again, with an adapter given step_size: 1 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: base: codex concurrency: 1 # one simulator: an evaluation takes a GPU @@ -28,7 +29,7 @@ environment: # the task sandbox: the actor's code and the - ../utils/agent env: [ROBODOJO_SIM_URL] # the host's simulator service (launch.sh): its 127.0.0.1 port is forwarded judge: - path: ../utils/judge.py # on the host: the program on the held-out layout sets + path: ../utils/benchmark/judge.py # on the host: the program on the held-out layout sets template: prompt-only@1.0 evolver: name: codex # the built-in Codex AgentEvolver diff --git a/examples/robodojo_sim/prompt_only/run_sample_opus.yaml b/examples/robodojo_sim/prompt_only/run_sample_opus.yaml index 7337d7c6..eea8fa1e 100644 --- a/examples/robodojo_sim/prompt_only/run_sample_opus.yaml +++ b/examples/robodojo_sim/prompt_only/run_sample_opus.yaml @@ -5,12 +5,13 @@ # Claude Code session per step that edits ADAPTER.md. Started by launch.sh: ROBODOJO_CONFIG=run_sample_opus.yaml # ./launch.sh data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/sample.txt - test: ../utils/splits/sample.txt # `launch.sh eval`: the tasks again, with an adapter given + tasks: ../utils/benchmark/tasks + view: ../utils/agent/robodojo_fields.py:robodojo_task # each task as the row it always was + train: ../utils/benchmark/splits/sample.txt + test: ../utils/benchmark/splits/sample.txt # `launch.sh eval`: the tasks again, with an adapter given step_size: 1 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: base: claude concurrency: 1 # one simulator: an evaluation takes a GPU @@ -29,7 +30,7 @@ environment: # the task sandbox: the actor's code and the - ../utils/agent env: [ROBODOJO_SIM_URL] # the host's simulator service (launch.sh): its 127.0.0.1 port is forwarded judge: - path: ../utils/judge.py # on the host: the program on the held-out layout sets + path: ../utils/benchmark/judge.py # on the host: the program on the held-out layout sets template: prompt-only@1.0 evolver: name: claude # the built-in Claude Code AgentEvolver: it revises ADAPTER.md after each task diff --git a/examples/robodojo_sim/prompt_only/run_sample_sol.yaml b/examples/robodojo_sim/prompt_only/run_sample_sol.yaml index a0c7d2d6..4269a9d1 100644 --- a/examples/robodojo_sim/prompt_only/run_sample_sol.yaml +++ b/examples/robodojo_sim/prompt_only/run_sample_sol.yaml @@ -1,12 +1,13 @@ # run_sample_luna.yaml with the models changed and nothing else: GPT-6.1-Sol at medium effort acts and learns. # Started by launch.sh: ROBODOJO_CONFIG=run_sample_sol.yaml ./launch.sh data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/sample.txt - test: ../utils/splits/sample.txt # `launch.sh eval`: the tasks again, with an adapter given + tasks: ../utils/benchmark/tasks + view: ../utils/agent/robodojo_fields.py:robodojo_task # each task as the row it always was + train: ../utils/benchmark/splits/sample.txt + test: ../utils/benchmark/splits/sample.txt # `launch.sh eval`: the tasks again, with an adapter given step_size: 1 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: base: codex concurrency: 1 # one simulator: an evaluation takes a GPU @@ -27,7 +28,7 @@ environment: # the task sandbox: the actor's code and the - ../utils/agent env: [ROBODOJO_SIM_URL] # the host's simulator service (launch.sh): its 127.0.0.1 port is forwarded judge: - path: ../utils/judge.py # on the host: the program on the held-out layout sets + path: ../utils/benchmark/judge.py # on the host: the program on the held-out layout sets template: prompt-only@1.0 evolver: name: codex # the built-in Codex AgentEvolver diff --git a/examples/robodojo_sim/prompt_only/smoke.yaml b/examples/robodojo_sim/prompt_only/smoke.yaml index f796456b..f28cf351 100644 --- a/examples/robodojo_sim/prompt_only/smoke.yaml +++ b/examples/robodojo_sim/prompt_only/smoke.yaml @@ -5,12 +5,13 @@ # Evolver revises the one learned ADAPTER.md, the next session's context, from the sessions' notes and scores. # Codex acting and learning, GLM 5.3 Flash. Started by launch.sh. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/smoke.txt - test: ../utils/splits/smoke.txt # `launch.sh eval`: the tasks again, with an adapter given + tasks: ../utils/benchmark/tasks + view: ../utils/agent/robodojo_fields.py:robodojo_task # each task as the row it always was + train: ../utils/benchmark/splits/smoke.txt + test: ../utils/benchmark/splits/smoke.txt # `launch.sh eval`: the tasks again, with an adapter given step_size: 1 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: base: codex concurrency: 1 # one simulator: an evaluation takes a GPU @@ -31,7 +32,7 @@ environment: # the task sandbox: the actor's code and the - ../utils/agent env: [ROBODOJO_SIM_URL] # the host's simulator service (launch.sh): its 127.0.0.1 port is forwarded judge: - path: ../utils/judge.py # on the host: the program on the held-out layout sets + path: ../utils/benchmark/judge.py # on the host: the program on the held-out layout sets template: prompt-only@1.0 evolver: name: codex # the built-in Codex AgentEvolver diff --git a/examples/robodojo_sim/prompt_only/smoke_luna.yaml b/examples/robodojo_sim/prompt_only/smoke_luna.yaml index 32904b73..12c2e145 100644 --- a/examples/robodojo_sim/prompt_only/smoke_luna.yaml +++ b/examples/robodojo_sim/prompt_only/smoke_luna.yaml @@ -2,12 +2,13 @@ # own ChatGPT login (`codex login`; no API key). Started by launch.sh: ROBODOJO_CONFIG=smoke_luna.yaml ./launch.sh # (its layouts are the smoke's). data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/smoke.txt - test: ../utils/splits/smoke.txt # `launch.sh eval`: the tasks again, with an adapter given + tasks: ../utils/benchmark/tasks + view: ../utils/agent/robodojo_fields.py:robodojo_task # each task as the row it always was + train: ../utils/benchmark/splits/smoke.txt + test: ../utils/benchmark/splits/smoke.txt # `launch.sh eval`: the tasks again, with an adapter given step_size: 1 shuffle: false -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: base: codex concurrency: 1 # one simulator: an evaluation takes a GPU @@ -28,7 +29,7 @@ environment: # the task sandbox: the actor's code and the - ../utils/agent env: [ROBODOJO_SIM_URL] # the host's simulator service (launch.sh): its 127.0.0.1 port is forwarded judge: - path: ../utils/judge.py # on the host: the program on the held-out layout sets + path: ../utils/benchmark/judge.py # on the host: the program on the held-out layout sets template: prompt-only@1.0 evolver: name: codex # the built-in Codex AgentEvolver diff --git a/examples/robodojo_sim/utils/.gitignore b/examples/robodojo_sim/utils/.gitignore index 87a26c7f..b111fc05 100644 --- a/examples/robodojo_sim/utils/.gitignore +++ b/examples/robodojo_sim/utils/.gitignore @@ -1,2 +1,4 @@ +runs/ +# Where utils/prepare.py wrote before utils/benchmark/: kept out of git in a checkout prepared then tasks.jsonl splits/ diff --git a/examples/robodojo_sim/utils/agent/dojo.py b/examples/robodojo_sim/utils/agent/dojo.py index b0f3d2b6..40e63074 100644 --- a/examples/robodojo_sim/utils/agent/dojo.py +++ b/examples/robodojo_sim/utils/agent/dojo.py @@ -12,7 +12,7 @@ copies must be byte for byte the ones here, or the evaluation is refused. An evaluation is a layout set (RoboDojo's `--seed`: a directory of saved layouts) and the first N of its layouts -(`--eval-num`); development and held-out evaluations use different layout sets (utils/prepare.py). +(`--eval-num`); development and held-out evaluations use different layout sets (utils/benchmark/prepare.py). Where Isaac Sim does not install on the host, the simulator runs in RoboDojo's container (`image`, ROBODOJO_IMAGE) and the policy server, with the program's worker, on the host (split_eval.sh): the same policy, the same result. @@ -182,7 +182,7 @@ def run(self, program: Path, *, task: str, layout_seed: int, layouts: int, out: """Run `program` on layout set `layout_seed` until RoboDojo has scored `layouts` layouts; one record per scored layout: layout_id, task_success, score. RoboDojo takes the set's layouts from the first and replaces one it abandons (an unstable scene) by the next, so a record's layout_id can be `layouts` or more: a caller - that scores counts the ids below `layouts` and treats a missing one as failed (utils/judge.py). + that scores counts the ids below `layouts` and treats a missing one as failed (utils/benchmark/judge.py). `out` receives what the program's author may see (the result, the videos, the program's own log and stderr); `private` (default: a directory beside `out`) the rest: RoboDojo's log and the sandbox's record, diff --git a/examples/robodojo_sim/utils/agent/dojo_service.py b/examples/robodojo_sim/utils/agent/dojo_service.py index 2fb4094c..70313889 100644 --- a/examples/robodojo_sim/utils/agent/dojo_service.py +++ b/examples/robodojo_sim/utils/agent/dojo_service.py @@ -10,7 +10,7 @@ evaluates a set's first N); a held-out set is never run here GET /health - python dojo_service.py --port 8130 --robodojo-root $ROBODOJO_ROOT --tasks utils/tasks.jsonl --workspaces + python dojo_service.py --port 8130 --robodojo-root $ROBODOJO_ROOT --tasks utils/benchmark/tasks --workspaces --state [--gpu 0] [--policy-env RoboDojo] [--eval-env RoboDojo] [--env-cfg arx_x5] [--action-type joint] [--run-timeout 21600] [--nonce ] """ @@ -27,6 +27,7 @@ HERE = Path(__file__).resolve().parent sys.path.insert(0, str(HERE)) import dojo # noqa: E402 +from robodojo_fields import task_rows # noqa: E402 sys.path.append(str(HERE.parents[2] / "aspire_robosuite" / "utils" / "agent")) import sim_service as base # noqa: E402 (the ASPIRE example's) @@ -93,7 +94,8 @@ def main() -> None: ap.add_argument("--host", default="127.0.0.1") ap.add_argument("--port", type=int, default=8130) ap.add_argument("--robodojo-root", type=Path, required=True) - ap.add_argument("--tasks", type=Path, required=True, help="the task rows (utils/tasks.jsonl): their layouts") + ap.add_argument("--tasks", type=Path, required=True, + help="the tasks (utils/benchmark/tasks, or a tasks.jsonl), each read as its row: their layouts") ap.add_argument("--workspaces", type=Path, action="append", required=True) ap.add_argument("--state", type=Path, required=True) ap.add_argument("--gpu", default="0") @@ -110,7 +112,7 @@ def main() -> None: action_type=a.action_type, gpu=a.gpu, timeout_s=a.run_timeout) engine._installed() # refused now rather than on a session's first run a.state.mkdir(parents=True, exist_ok=True) - tasks = {r["id"]: r for r in (json.loads(line) for line in a.tasks.read_text().splitlines() if line.strip())} + tasks = task_rows(a.tasks) service = Service(engine, tasks, a.workspaces, base.Ledger(a.state / "ledger.json"), run_timeout=a.run_timeout) service.nonce = a.nonce diff --git a/examples/robodojo_sim/utils/agent/robodojo_fields.py b/examples/robodojo_sim/utils/agent/robodojo_fields.py new file mode 100644 index 00000000..f20157ea --- /dev/null +++ b/examples/robodojo_sim/utils/agent/robodojo_fields.py @@ -0,0 +1,19 @@ +"""A RoboDojo-Sim task as the actor, the simulator service and the judge read it: {id, benchmark, suite, task_id, +dimension, layout_set, debug_seeds, eval_layout_sets, eval_layouts}, the row each task was in tasks.jsonl. A task +directory (utils/benchmark/prepare.py) has every field but its id under task.toml's [metadata], in the row's order; +this puts them back, so the prompts, the service's rules and the records are what they were.""" +from __future__ import annotations + +from pathlib import Path + + +def robodojo_task(task: dict) -> dict: + if "instruction" not in task: # already a row of that shape + return dict(task) + return {"id": task["id"], **(task.get("metadata") or {})} + + +def task_rows(tasks: Path) -> dict[str, dict]: + """Every task's row by id: from the task directories prepare.py writes, or from a tasks.jsonl.""" + from hermit.task import Benchmark + return {tid: t.given() for tid, t in Benchmark.load(Path(tasks), view=robodojo_task).tasks.items()} diff --git a/examples/robodojo_sim/utils/benchmark/.gitignore b/examples/robodojo_sim/utils/benchmark/.gitignore new file mode 100644 index 00000000..d296b16f --- /dev/null +++ b/examples/robodojo_sim/utils/benchmark/.gitignore @@ -0,0 +1,2 @@ +tasks/ +splits/ diff --git a/examples/robodojo_sim/utils/judge.py b/examples/robodojo_sim/utils/benchmark/judge.py similarity index 98% rename from examples/robodojo_sim/utils/judge.py rename to examples/robodojo_sim/utils/benchmark/judge.py index 5411bfe4..df718110 100644 --- a/examples/robodojo_sim/utils/judge.py +++ b/examples/robodojo_sim/utils/benchmark/judge.py @@ -23,7 +23,7 @@ from pathlib import Path import sys -sys.path.insert(0, str(Path(__file__).resolve().parent / "agent")) +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "agent")) import dojo # noqa: E402 JUDGE_TIMEOUT_S = 48 * 3600 diff --git a/examples/robodojo_sim/utils/prepare.py b/examples/robodojo_sim/utils/benchmark/prepare.py similarity index 77% rename from examples/robodojo_sim/utils/prepare.py rename to examples/robodojo_sim/utils/benchmark/prepare.py index 639079c8..34d06ddb 100644 --- a/examples/robodojo_sim/utils/prepare.py +++ b/examples/robodojo_sim/utils/benchmark/prepare.py @@ -1,10 +1,15 @@ """Write RoboDojo-Sim's forty-two tasks (RoboDojo `scripts/internal/task_inventory.py`, its five capability -dimensions) as hermit tasks: one row per task with the layouts it is developed and scored on. +dimensions) as hermit tasks: one task directory (Harbor's format) per task with the layouts it is developed and scored +on. python prepare.py # development: layout set 2, its first 10 layouts; held-out: sets 0 and 1, # as many of each as RoboDojo evaluates (50, or the task's own 25) python prepare.py --preset smoke # development: 2 layouts; held-out: set 0, 2 layouts - # tasks.jsonl, splits/{all,,smoke,sample}.txt + # tasks//, splits/{all,,smoke,sample}.txt + +A task directory's task.toml holds the row's fields but its id under [metadata], in the row's order, and asks for no +network (the actor reaches only its model and the simulator service's forwarded port); it has no tests/: the judge +(judge.py) runs on the host, where the simulator is. utils/agent/robodojo_fields.py reads each back as its row. RoboDojo's saved layouts come in sets (`--seed`: `Assets/Eval_Layout/RoboDojo///_.json`; its protocol evaluates sets 0, 1 and 2) and an evaluation runs a set's first N. A row keeps the two apart by set: @@ -16,8 +21,10 @@ from __future__ import annotations import argparse -import json from pathlib import Path +import shutil + +from hermit.task.directory import write_task as write_task_dir HERE = Path(__file__).resolve().parent BENCHMARK = "robodojo" @@ -47,6 +54,15 @@ "smoke": {"dev_set": 2, "dev": 2, "eval_sets": "0", "eval": 2}} +def write_task(tasks: Path, row: dict) -> Path: + """One row as a task directory: its fields but the id under [metadata], in the row's order; no network.""" + instruction = (f"Write the program that controls the bimanual robot in RoboDojo-Sim's `{row['suite']}` " + f"({row['dimension']}): developed on layout set {row['layout_set']}, scored on the held-out sets " + f"{', '.join(map(str, row['eval_layout_sets']))}.") + return write_task_dir(tasks / row["id"], instruction, metadata={k: v for k, v in row.items() if k != "id"}, + environment={"network_mode": "no-network"}) + + def main(argv=None) -> None: ap = argparse.ArgumentParser() ap.add_argument("--preset", choices=sorted(PRESETS), default="full") @@ -67,7 +83,10 @@ def main(argv=None) -> None: "eval_layout_sets": eval_sets, "eval_layouts": min(chosen["eval"] or EVAL_NUMS, 25 if task in EVAL_25 else EVAL_NUMS)} for dimension, tasks in DIMENSIONS.items() for task in tasks] - (HERE / "tasks.jsonl").write_text("".join(json.dumps(r) + "\n" for r in rows)) + if (HERE / "tasks").exists(): # this script's own output: rewritten whole + shutil.rmtree(HERE / "tasks") + for row in rows: + write_task(HERE / "tasks", row) splits = HERE / "splits" splits.mkdir(exist_ok=True) of = lambda dimension: [r["id"] for r in rows if r["dimension"] == dimension] @@ -75,7 +94,7 @@ def main(argv=None) -> None: for name, members in {"all": ids, **{d: of(d) for d in DIMENSIONS}, "smoke": ids[:2], "sample": [f"robodojo-{t}" for t in SAMPLE]}.items(): (splits / f"{name}.txt").write_text("".join(f"{i}\n" for i in members)) - print(f"{len(rows)} tasks -> {HERE / 'tasks.jsonl'}") + print(f"{len(rows)} tasks -> {HERE / 'tasks'}") if __name__ == "__main__": diff --git a/examples/robodojo_sim/utils/task-info/README.md b/examples/robodojo_sim/utils/benchmark/task-info/README.md similarity index 100% rename from examples/robodojo_sim/utils/task-info/README.md rename to examples/robodojo_sim/utils/benchmark/task-info/README.md diff --git a/examples/robodojo_sim/utils/tests/test_robodojo.py b/examples/robodojo_sim/utils/tests/test_robodojo.py index e853ce6e..93f45260 100644 --- a/examples/robodojo_sim/utils/tests/test_robodojo.py +++ b/examples/robodojo_sim/utils/tests/test_robodojo.py @@ -449,16 +449,33 @@ def test_a_program_is_scored_by_what_it_does_and_reads_nothing_of_robodojos_chec def test_the_rows_keep_development_and_held_out_layouts_in_different_sets(tmp_path, monkeypatch): - P = _load("robodojo_prepare", UTILS / "prepare.py") + """prepare.py's task directories, each read as the row it was in tasks.jsonl (robodojo_fields.py: its keys in + order, its types), by every config through its data.view.""" + from hermit.config import Config + from hermit.task import Benchmark + P = _load("robodojo_prepare", UTILS / "benchmark" / "prepare.py") + F = _load("robodojo_fields", UTILS / "agent" / "robodojo_fields.py") assert sum(len(tasks) for tasks in P.DIMENSIONS.values()) == 42 assert len({t for tasks in P.DIMENSIONS.values() for t in tasks}) == 42 monkeypatch.setattr(P, "HERE", tmp_path) P.main([]) - rows = [json.loads(line) for line in (tmp_path / "tasks.jsonl").read_text().splitlines()] + found = F.task_rows(tmp_path / "tasks") + rows = [found[i] for i in (tmp_path / "splits" / "all.txt").read_text().split()] assert len(rows) == 42 and all(r["layout_set"] not in r["eval_layout_sets"] for r in rows) - assert rows[0] == {"id": "robodojo-stack_bowls", "benchmark": "robodojo", "suite": "stack_bowls", "task_id": 0, - "dimension": "generalization", "layout_set": 2, "debug_seeds": "0-9", - "eval_layout_sets": [0, 1], "eval_layouts": 25} + assert json.dumps(rows[0]) == json.dumps( + {"id": "robodojo-stack_bowls", "benchmark": "robodojo", "suite": "stack_bowls", "task_id": 0, + "dimension": "generalization", "layout_set": 2, "debug_seeds": "0-9", "eval_layout_sets": [0, 1], + "eval_layouts": 25}) + task = Benchmark.load(tmp_path / "tasks", view=F.robodojo_task).tasks["robodojo-stack_bowls"] + assert task.private["tests"] is None and task.private["needs"]["network"]["agent"] == "no-network" + for path in sorted((EXAMPLE / "prompt_only").glob("*.yaml")): + config = Config.load(path) + data = config.data["data"] + assert config.path(data["tasks"]) == (UTILS / "benchmark" / "tasks").resolve(), path + monkeypatch.setitem(data, "tasks", str(tmp_path / "tasks")) + for key in ("train", "test"): + monkeypatch.setitem(data, key, str(tmp_path / "splits" / Path(data[key]).name)) + assert {t.id: t.given() for t in Benchmark.from_config(config).tasks.values()} == found, path assert len((tmp_path / "splits" / "all.txt").read_text().split()) == 42 assert (tmp_path / "splits" / "sample.txt").read_text().split() == [ "robodojo-stack_bowls", "robodojo-press_by_number", "robodojo-general_pickup"] @@ -468,8 +485,7 @@ def test_the_rows_keep_development_and_held_out_layouts_in_different_sets(tmp_pa assert P.EVAL_25 <= set(held) and held["push_T"] == 25 and held["press_by_number"] == 50 assert sorted(held.values()).count(25) == 12 and set(held.values()) == {25, 50} P.main(["--eval", "40"]) # at most RoboDojo's own, as its main.py - capped = {json.loads(line)["suite"]: json.loads(line)["eval_layouts"] - for line in (tmp_path / "tasks.jsonl").read_text().splitlines()} + capped = {r["suite"]: r["eval_layouts"] for r in F.task_rows(tmp_path / "tasks").values()} assert capped["press_by_number"] == 40 and capped["push_T"] == 25 with pytest.raises(SystemExit): P.main(["--dev-set", "0"]) # the development set is a held-out set @@ -552,7 +568,7 @@ def run(self, program, **options): def test_the_judge_scores_the_program_on_the_held_out_sets_only(tmp_path, monkeypatch): - J = _load("robodojo_judge", UTILS / "judge.py") + J = _load("robodojo_judge", UTILS / "benchmark" / "judge.py") ran = [] made = [] diff --git a/examples/servelearnbench/README.md b/examples/servelearnbench/README.md index d2aa98ff..dfa3426d 100644 --- a/examples/servelearnbench/README.md +++ b/examples/servelearnbench/README.md @@ -13,7 +13,7 @@ Install the framework first ([Installation](../../docs/get-started/installation. ```bash pip install "git+https://github.com/Infini-AI-Lab/ServeLearnBench" # the benchmark -python examples/servelearnbench/utils/prepare.py --scenario retail_l3 --block 12 # tasks.jsonl, splits/, windows.json, the documents +python examples/servelearnbench/utils/benchmark/prepare.py --scenario retail_l3 --block 12 # tasks/, splits/, windows.json, the documents ``` Also needed for training: the Codex CLI on PATH, bubblewrap, and the key of the models' backend (`FIREWORKS_API_KEY` @@ -31,7 +31,7 @@ examples/servelearnbench/prompt_only/launch.sh retail_l3 # prompt-only, same Each launcher starts the wire meters and one `hermit train` in the background (`$MW_RUNS//run.log`), then replays every window's held-out tests on the adapter the run had after that window and scores the result file with `slb score` (`test.log` and `task/results/retail_l3_.jsonl` in the run directory). The tests can be rerun with -`launch.sh retail_l3 test`. A smoke on the first 12 serving tasks: `MW_TRAIN_SPLIT=examples/servelearnbench/utils/splits/smoke_serving.txt`. +`launch.sh retail_l3 test`. A smoke on the first 12 serving tasks: `MW_TRAIN_SPLIT=examples/servelearnbench/utils/benchmark/splits/smoke_serving.txt`. ## Results @@ -66,10 +66,12 @@ Leaderboard rows are the benchmark's published numbers for GLM 5.3 Flash on reta ```text utils/agent/agent.py ServeLearnBenchAgent: one episode through the benchmark's loop, the adapter plugged in through Session -utils/prepare.py exports one scenario: tasks.jsonl, splits/ (serving, test_w, smoke), windows.json, the documents -utils/judge.py the benchmark's reward; the feedback is the one line its methods see ("Score: n/100.") -utils/windows.py the per-window test replay and the `slb score` result file -utils/task-info/ the task for the learner; task-info/mwm/ is what MWM needs to know about the venue (see docs/methods/mwm.md) +utils/agent/slb_fields.py a task directory read back as the row the agent always read (the configs' data.view) +utils/benchmark/ prepare.py exports one scenario: tasks/ (one task directory in Harbor's format per episode), + splits/ (serving, test_w, smoke), windows.json, the documents; judge.py, the benchmark's + reward (the feedback is the one line its methods see, "Score: n/100."); windows.py, the + per-window test replay and the `slb score` result file; task-info/, the task for the learner + (task-info/mwm/ is what MWM needs to know about the venue, see docs/methods/mwm.md) mwm/ prompt_only/ retail_l3.yaml and launch.sh per method ``` diff --git a/examples/servelearnbench/mwm/launch.sh b/examples/servelearnbench/mwm/launch.sh index 5d48f497..c8b0d843 100755 --- a/examples/servelearnbench/mwm/launch.sh +++ b/examples/servelearnbench/mwm/launch.sh @@ -7,7 +7,7 @@ # # `train` starts two wire meters (the host's on WM_PORT, the learner's on WM_PORT+1), each in front of its model's # backend, and one `hermit train` in the background, logging to //run.log. `test` replays every -# window's held-out tests on the adapter the run had at that window (examples/servelearnbench/utils/windows.py) and +# window's held-out tests on the adapter the run had at that window (examples/servelearnbench/utils/benchmark/windows.py) and # scores the result file; it can run while training is still going (windows not trained yet are skipped) and again # later. # @@ -43,7 +43,7 @@ EVOLVER_MODEL=${MW_EVOLVER_MODEL:-$MODEL} EFFORT=${MW_EFFORT:-high} METER=${MW_WIRE_METER:-$FRAMEWORK/hermit/runtime/wire_meter.py} META_PORT=$((WM_PORT+1)) -TASKDIR=$FRAMEWORK/examples/servelearnbench/utils +TASKDIR=$FRAMEWORK/examples/servelearnbench/utils/benchmark CONFIG=$FRAMEWORK/examples/servelearnbench/$METHOD/$SCENARIO.yaml WD="$RUNS/$TAG/task" export PYTHONPATH=$FRAMEWORK${PYTHONPATH:+:$PYTHONPATH} @@ -52,8 +52,8 @@ export PYTHONPATH=$FRAMEWORK${PYTHONPATH:+:$PYTHONPATH} [ -f "$METER" ] || { echo "REFUSING: no wire meter at $METER (MW_WIRE_METER)"; exit 1; } "$PY" -c "import servelearnbench" 2>/dev/null \ || { echo "REFUSING: servelearnbench is not installed: $PY -m pip install git+https://github.com/Infini-AI-Lab/ServeLearnBench"; exit 1; } -# The scenario as a task (tasks.jsonl, splits, windows.json, the reference documents), exported once. -if ! grep -q "\"scenario\": \"$SCENARIO\"" "$TASKDIR/tasks.jsonl" 2>/dev/null; then +# The scenario as a task (its task directories, splits, windows.json, the reference documents), exported once. +if ! { [ -d "$TASKDIR/tasks" ] && grep -q "\"scenario\": \"$SCENARIO\"" "$TASKDIR/windows.json" 2>/dev/null; }; then BLOCK=$(grep -E '^\s*step_size:' "$CONFIG" | head -1 | sed -E 's/.*step_size:\s*([0-9]+).*/\1/') "$PY" "$TASKDIR/prepare.py" --scenario "$SCENARIO" --block "${BLOCK:-12}" fi diff --git a/examples/servelearnbench/mwm/retail_l3.yaml b/examples/servelearnbench/mwm/retail_l3.yaml index 24b6bd21..a8b3e318 100644 --- a/examples/servelearnbench/mwm/retail_l3.yaml +++ b/examples/servelearnbench/mwm/retail_l3.yaml @@ -1,18 +1,19 @@ # MWM on ServeLearnBench retail_l3: the scenario's serving stream (624 cases in 8 windows), one MWM block per 12 # cases, the benchmark's own agent loop and verifier. Started by `launch.sh retail_l3`, which sets the run's paths. -# The per-window test sets are played afterwards by examples/servelearnbench/windows.py. +# The per-window test sets are played afterwards by examples/servelearnbench/utils/benchmark/windows.py. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/serving.txt # launch.sh: splits/smoke_serving.txt for a smoke - test: ../utils/splits/test_w0.txt # windows.py: test_w.txt per window + tasks: ../utils/benchmark/tasks + view: ../utils/agent/slb_fields.py:slb_task # each task as the row it always was + train: ../utils/benchmark/splits/serving.txt # launch.sh: splits/smoke_serving.txt for a smoke + test: ../utils/benchmark/splits/test_w0.txt # windows.py: test_w.txt per window step_size: 12 # = prepare.py --block (the benchmark's update cadence) shuffle: false # the stream's own order -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: name: ../utils/agent/agent.py:ServeLearnBenchAgent concurrency: 12 # the cases of one block, in parallel judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: mwm evolver: name: mwm @@ -26,7 +27,7 @@ evolver_access: environment: [] # launch.sh: [/evidence] tools: [] # the venue has no channel: the learner reads /evidence and /environment only environment: # the task sandbox: the agent, MWM's venue texts, the benchmark - read: [../utils/agent, ../utils/task-info/mwm, ../utils/windows.json] # windows.json: the task -> window map + read: [../utils/agent, ../utils/benchmark/task-info/mwm, ../utils/benchmark/windows.json] # windows.json: the task -> window map # (prepare.py; its ids are the package's own test sets, shown already) packages: [servelearnbench] # its environment and its scoring live in one package write: [$SLB_WORKDIR] # the run's task directory: rows, episodes, evidence diff --git a/examples/servelearnbench/prompt_only/retail_l3.yaml b/examples/servelearnbench/prompt_only/retail_l3.yaml index 0f455bdc..35c27942 100644 --- a/examples/servelearnbench/prompt_only/retail_l3.yaml +++ b/examples/servelearnbench/prompt_only/retail_l3.yaml @@ -3,17 +3,18 @@ # as its context, no tools) and the Codex Evolver of the CL-Bench baseline (one fresh session per step, GLM 5.3 Flash # through the learner's meter, reasoning effort high, tools gated to check_spec). Started by `launch.sh retail_l3`. data: - tasks: ../utils/tasks.jsonl - train: ../utils/splits/serving.txt # launch.sh: splits/smoke_serving.txt for a smoke - test: ../utils/splits/test_w0.txt # windows.py: test_w.txt per window + tasks: ../utils/benchmark/tasks + view: ../utils/agent/slb_fields.py:slb_task # each task as the row it always was + train: ../utils/benchmark/splits/serving.txt # launch.sh: splits/smoke_serving.txt for a smoke + test: ../utils/benchmark/splits/test_w0.txt # windows.py: test_w.txt per window step_size: 12 # = prepare.py --block (the benchmark's update cadence) shuffle: false # the stream's own order -task_info: ../utils/task-info +task_info: ../utils/benchmark/task-info agent: name: ../utils/agent/agent.py:ServeLearnBenchAgent concurrency: 12 # the cases of one block, in parallel judge: - path: ../utils/judge.py + path: ../utils/benchmark/judge.py template: prompt-only@1.0 evolver: name: codex # the built-in Codex AgentEvolver, with GLM 5.3 Flash through the learner's meter @@ -24,7 +25,7 @@ evolver: home: [] # no login files: the provider key is the backend's variable tools: [check_spec] # no run_agent, llm or budget: the MWM learner of this venue has none either environment: # the task sandbox: the agent, MWM's venue texts, the benchmark - read: [../utils/agent, ../utils/task-info/mwm, ../utils/windows.json] # windows.json: the task -> window map + read: [../utils/agent, ../utils/benchmark/task-info/mwm, ../utils/benchmark/windows.json] # windows.json: the task -> window map # (prepare.py; its ids are the package's own test sets, shown already) packages: [servelearnbench] # its environment and its scoring live in one package write: [$SLB_WORKDIR] # the run's task directory: rows, episodes, evidence diff --git a/examples/servelearnbench/utils/.gitignore b/examples/servelearnbench/utils/.gitignore index 14370bf0..f77919ba 100644 --- a/examples/servelearnbench/utils/.gitignore +++ b/examples/servelearnbench/utils/.gitignore @@ -1,3 +1,5 @@ +# Where utils/prepare.py wrote before utils/benchmark/: kept out of git in a checkout prepared then tasks.jsonl splits/ windows.json +task-info/mwm/environment/docs/ diff --git a/examples/servelearnbench/utils/agent/agent.py b/examples/servelearnbench/utils/agent/agent.py index a52f99bb..af496e1e 100644 --- a/examples/servelearnbench/utils/agent/agent.py +++ b/examples/servelearnbench/utils/agent/agent.py @@ -34,8 +34,8 @@ HERE = Path(__file__).resolve().parent -UTILS = HERE.parent -MWM = HERE.parent / "task-info" / "mwm" # the venue's texts for MWM +BENCHMARK = HERE.parent / "benchmark" # prepare.py's: the tasks, windows.json, task-info/ +MWM = BENCHMARK / "task-info" / "mwm" # the venue's texts for MWM from hermit.agent.evolver.methods.mwm.evidence import servelearnbench as evidence # the MWM learner's records @@ -74,7 +74,7 @@ def _setup(scenario: str, params: dict | None = None) -> dict: for w in range(n_windows(bundle)): for t in bundle["test_set"](w): tasks[t.task_id] = t - windows = json.loads((UTILS / "windows.json").read_text()) + windows = json.loads((BENCHMARK / "windows.json").read_text()) if windows["scenario"] != scenario: raise RuntimeError(f"windows.json is for {windows['scenario']}, the task row says {scenario}") window_of = {} diff --git a/examples/servelearnbench/utils/agent/slb_fields.py b/examples/servelearnbench/utils/agent/slb_fields.py new file mode 100644 index 00000000..6967c03c --- /dev/null +++ b/examples/servelearnbench/utils/agent/slb_fields.py @@ -0,0 +1,10 @@ +"""A ServeLearnBench episode as the agent and its learners read it: {id, scenario, phase, timestamp, (block), +instruction}, the row each task was in tasks.jsonl. A task directory (utils/benchmark/prepare.py) has the stamped +instruction as instruction.md and the other fields under task.toml's [metadata], in the row's order; this puts them +back, so the instruction the benchmark's loop is given and the records are what they were.""" + + +def slb_task(task: dict) -> dict: + if "metadata" not in task: # already a row of that shape + return dict(task) + return {"id": task["id"], **(task.get("metadata") or {}), "instruction": task["instruction"]} diff --git a/examples/servelearnbench/utils/benchmark/.gitignore b/examples/servelearnbench/utils/benchmark/.gitignore new file mode 100644 index 00000000..eebcbbe7 --- /dev/null +++ b/examples/servelearnbench/utils/benchmark/.gitignore @@ -0,0 +1,3 @@ +tasks/ +splits/ +windows.json diff --git a/examples/servelearnbench/utils/judge.py b/examples/servelearnbench/utils/benchmark/judge.py similarity index 100% rename from examples/servelearnbench/utils/judge.py rename to examples/servelearnbench/utils/benchmark/judge.py diff --git a/examples/servelearnbench/utils/prepare.py b/examples/servelearnbench/utils/benchmark/prepare.py similarity index 78% rename from examples/servelearnbench/utils/prepare.py rename to examples/servelearnbench/utils/benchmark/prepare.py index 60bcc1e0..5d11706e 100755 --- a/examples/servelearnbench/utils/prepare.py +++ b/examples/servelearnbench/utils/benchmark/prepare.py @@ -1,6 +1,6 @@ #!/usr/bin/env python3 -"""Write one ServeLearnBench scenario as a framework task: tasks.jsonl, the splits, windows.json, and the -learner's read-only copy of the reference documents. +"""Write one ServeLearnBench scenario as a framework task: its tasks (task directories in Harbor's format, tasks//), +the splits, windows.json, and the learner's read-only copy of the reference documents. python prepare.py --scenario retail_l3 [--block 12] [--smoke 12] @@ -13,6 +13,11 @@ category, family) are not in the rows: windows.json maps them on the host side, and nothing a learner reads says where a window begins. +A task directory has the stamped instruction as instruction.md and the other fields under task.toml's [metadata], in +the row's order (utils/agent/slb_fields.py reads it back as that row); it asks for no network (the benchmark's client +reaches its model through the launcher's meter) and has no tests/: the judge (judge.py) reads the benchmark's own +reward from the agent's output. Each task is one episode; what carries across the stream is the method's adapter. + `--smoke N` also writes `splits/smoke_serving.txt` (the first N serving tasks) and `splits/smoke_test.txt` (the first 4 test tasks of window 0), for a short end-to-end check. @@ -24,6 +29,9 @@ import argparse import json from pathlib import Path +import shutil + +from hermit.task.directory import write_task as write_task_dir HERE = Path(__file__).resolve().parent @@ -61,6 +69,14 @@ def export_docs(bundle, dest: Path) -> int: return n +def write_task(tasks: Path, row: dict) -> Path: + """One episode as a task directory: the instruction as instruction.md, the other fields (scenario, phase, + timestamp, block) under [metadata] in the row's order; no network.""" + meta = {k: v for k, v in row.items() if k not in ("id", "instruction")} + return write_task_dir(tasks / row["id"], row["instruction"], metadata=meta, + environment={"network_mode": "no-network"}) + + def main() -> None: ap = argparse.ArgumentParser() ap.add_argument("--scenario", default="retail_l3") @@ -96,7 +112,10 @@ def main() -> None: ids = [r["id"] for r in rows] if len(set(ids)) != len(ids): raise SystemExit("task ids are not unique across serving and test") - (HERE / "tasks.jsonl").write_text("".join(json.dumps(r, ensure_ascii=False) + "\n" for r in rows)) + if (HERE / "tasks").exists(): # this script's own output: rewritten whole + shutil.rmtree(HERE / "tasks") + for row in rows: + write_task(HERE / "tasks", row) (HERE / "windows.json").write_text(json.dumps({"scenario": a.scenario, "block": a.block, "windows": windows}, indent=1)) if a.smoke: diff --git a/examples/servelearnbench/utils/task-info/.gitignore b/examples/servelearnbench/utils/benchmark/task-info/.gitignore similarity index 100% rename from examples/servelearnbench/utils/task-info/.gitignore rename to examples/servelearnbench/utils/benchmark/task-info/.gitignore diff --git a/examples/servelearnbench/utils/task-info/README.md b/examples/servelearnbench/utils/benchmark/task-info/README.md similarity index 100% rename from examples/servelearnbench/utils/task-info/README.md rename to examples/servelearnbench/utils/benchmark/task-info/README.md diff --git a/examples/servelearnbench/utils/task-info/mwm/contract/domain_what.md b/examples/servelearnbench/utils/benchmark/task-info/mwm/contract/domain_what.md similarity index 100% rename from examples/servelearnbench/utils/task-info/mwm/contract/domain_what.md rename to examples/servelearnbench/utils/benchmark/task-info/mwm/contract/domain_what.md diff --git a/examples/servelearnbench/utils/task-info/mwm/contract/env_consumption.md b/examples/servelearnbench/utils/benchmark/task-info/mwm/contract/env_consumption.md similarity index 100% rename from examples/servelearnbench/utils/task-info/mwm/contract/env_consumption.md rename to examples/servelearnbench/utils/benchmark/task-info/mwm/contract/env_consumption.md diff --git a/examples/servelearnbench/utils/task-info/mwm/contract/env_evidence.md b/examples/servelearnbench/utils/benchmark/task-info/mwm/contract/env_evidence.md similarity index 100% rename from examples/servelearnbench/utils/task-info/mwm/contract/env_evidence.md rename to examples/servelearnbench/utils/benchmark/task-info/mwm/contract/env_evidence.md diff --git a/examples/servelearnbench/utils/task-info/mwm/contract/env_executable.md b/examples/servelearnbench/utils/benchmark/task-info/mwm/contract/env_executable.md similarity index 100% rename from examples/servelearnbench/utils/task-info/mwm/contract/env_executable.md rename to examples/servelearnbench/utils/benchmark/task-info/mwm/contract/env_executable.md diff --git a/examples/servelearnbench/utils/task-info/mwm/contract/env_verification.md b/examples/servelearnbench/utils/benchmark/task-info/mwm/contract/env_verification.md similarity index 100% rename from examples/servelearnbench/utils/task-info/mwm/contract/env_verification.md rename to examples/servelearnbench/utils/benchmark/task-info/mwm/contract/env_verification.md diff --git a/examples/servelearnbench/utils/task-info/mwm/delegation b/examples/servelearnbench/utils/benchmark/task-info/mwm/delegation similarity index 100% rename from examples/servelearnbench/utils/task-info/mwm/delegation rename to examples/servelearnbench/utils/benchmark/task-info/mwm/delegation diff --git a/examples/servelearnbench/utils/task-info/mwm/environment/README.md b/examples/servelearnbench/utils/benchmark/task-info/mwm/environment/README.md similarity index 100% rename from examples/servelearnbench/utils/task-info/mwm/environment/README.md rename to examples/servelearnbench/utils/benchmark/task-info/mwm/environment/README.md diff --git a/examples/servelearnbench/utils/task-info/mwm/host.json b/examples/servelearnbench/utils/benchmark/task-info/mwm/host.json similarity index 100% rename from examples/servelearnbench/utils/task-info/mwm/host.json rename to examples/servelearnbench/utils/benchmark/task-info/mwm/host.json diff --git a/examples/servelearnbench/utils/windows.py b/examples/servelearnbench/utils/benchmark/windows.py similarity index 95% rename from examples/servelearnbench/utils/windows.py rename to examples/servelearnbench/utils/benchmark/windows.py index e32ee77c..5e6fa50f 100755 --- a/examples/servelearnbench/utils/windows.py +++ b/examples/servelearnbench/utils/benchmark/windows.py @@ -34,6 +34,14 @@ HERE = Path(__file__).resolve().parent +def task_rows(tasks: Path) -> dict[str, dict]: + """Every task's row by id (the task directories prepare.py writes, as utils/agent/slb_fields.py reads them).""" + from hermit.task import Benchmark + sys.path.insert(0, str(HERE.parent / "agent")) + from slb_fields import slb_task + return {tid: t.given() for tid, t in Benchmark.load(tasks, view=slb_task).tasks.items()} + + def _steps(run_dir: Path) -> list[dict]: p = run_dir / "steps.jsonl" if not p.exists(): @@ -98,7 +106,7 @@ def main() -> None: from hermit.tree import ROOT_TAG windows = json.loads((HERE / "windows.json").read_text()) - tasks = {json.loads(l)["id"]: json.loads(l) for l in (HERE / "tasks.jsonl").read_text().splitlines() if l.strip()} + tasks = task_rows(HERE / "tasks") stream_pos = {tid: int(r["timestamp"]) for tid, r in tasks.items() if r["phase"] == "serving"} base = Config.load(a.config, [f"trainer.run_name={a.run}", *a.overrides]) run_dir = base.run_dir() diff --git a/examples/servelearnbench/utils/tests/test_slb_task_directories.py b/examples/servelearnbench/utils/tests/test_slb_task_directories.py new file mode 100644 index 00000000..6294d3a4 --- /dev/null +++ b/examples/servelearnbench/utils/tests/test_slb_task_directories.py @@ -0,0 +1,44 @@ +"""ServeLearnBench's task directories (utils/benchmark/prepare.py's write_task) read back, through both configs' +data.view, as exactly the rows tasks.jsonl held: the stamped instruction the benchmark's loop is given, the records and +the learning material are unchanged. Each episode stays one task; what carries across the stream is the adapter.""" +import importlib.util +import json +from pathlib import Path + +from hermit.config import Config +from hermit.task import Benchmark + +EXAMPLE = Path(__file__).resolve().parents[2] +ROWS = [{"id": "l3a_t0001_p_amex_ret", "scenario": "retail_l3", "phase": "serving", "timestamp": 1, "block": 1, + "instruction": 'Current timestamp: 1\n\nCustomer: Aisha (zip 37759) writes:\n"I want to send back the ' + 'Cookware Set — no longer needed. \\ \\"quoted\\""'}, + {"id": "l3a_test_w0_007", "scenario": "retail_l3", "phase": "test", "timestamp": 77, + "instruction": "Current timestamp: 77\n\nCustomer writes:\n\tTab and\r\nCRLF kept."}] + + +def _module(name, path): + spec = importlib.util.spec_from_file_location(name, path) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) # its work is in main(): importing it writes nothing + return module + + +def test_an_episode_reads_back_as_its_row_in_every_config(tmp_path, monkeypatch): + prepare = _module("slb_prepare", EXAMPLE / "utils/benchmark/prepare.py") + for row in ROWS: + prepare.write_task(tmp_path / "tasks", row) + (tmp_path / "serving.txt").write_text(ROWS[0]["id"] + "\n") + (tmp_path / "test_w0.txt").write_text(ROWS[1]["id"] + "\n") + for path in sorted(EXAMPLE.glob("*/retail_l3.yaml")): + config = Config.load(path) + data = config.data["data"] + assert config.path(data["tasks"]) == (EXAMPLE / "utils/benchmark/tasks").resolve(), path + monkeypatch.setitem(data, "tasks", str(tmp_path / "tasks")) + monkeypatch.setitem(data, "train", str(tmp_path / "serving.txt")) + monkeypatch.setitem(data, "test", str(tmp_path / "test_w0.txt")) + bench = Benchmark.from_config(config) + given = [t.given() for t in bench.split("train") + bench.split("test")] + assert [json.dumps(g) for g in given] == [json.dumps(r) for r in ROWS], path + assert bench.split("train")[0].private["needs"]["network"]["agent"] == "no-network" + windows = _module("slb_windows", EXAMPLE / "utils/benchmark/windows.py") + assert windows.task_rows(tmp_path / "tasks") == {r["id"]: r for r in ROWS} # the replay's stream positions diff --git a/examples/swebench_verified/README.md b/examples/swebench_verified/README.md index 10149edd..43acae18 100644 --- a/examples/swebench_verified/README.md +++ b/examples/swebench_verified/README.md @@ -49,9 +49,8 @@ of them trains on it (DGM, SICA and HGM do, and are not ported). The gains are small at this level, and AHE gives the tokens a trial beside them, a mean over its completed trials (an infra or timed-out one left out, §4.1). Here a trial's tokens are `tokens` in the eval's `results.jsonl`; leave out the -same trials (an `output.status` of `infra` or `timeout`) for a mean beside AHE's. A trial the runner ran again after -an infra run holds the tokens of all its runs, and its row does not say so. Neither paper says how many runs a task -had; here each runs once. With the final adapter of +same trials (a row whose `outcome.status` is not `ok`: `infra`, `timeout` or `agent_error`) for a mean beside +AHE's. Neither paper says how many runs a task had; here each runs once. With the final adapter of [Terminal-Bench 2.0's AHE run](../terminal_bench_2_0/README.md#ahe-adapter) written out as a directory (that README says how), the same acting agent tests it: @@ -74,10 +73,9 @@ config's folder's. The Claude Code configs take `ahe-final-claude`. adapter is what such a statement alone adds. - Some tasks cannot pass on a given host whatever the agent does. `psf__requests-1724`'s tests call `httpbin.org` over HTTPS with the 2014 `requests` its repository pins, whose certificates no longer verify: its reference fix scores 0 - here (`SSLError: CERTIFICATE_VERIFY_FAILED`). Harbor's oracle agent finds such tasks before a run spends model calls - on them: `hermit eval ... --adapter none "agent.params.acting_agent={name: oracle}" agent.params.login=null - agent.params.model=null agent.params.effort=null trainer.run_name=oracle` runs each task's reference fix, in - `runs/oracle/` (without the run name it would write over the baseline's `runs/codex-baseline/eval/none/`). + here (`SSLError: CERTIFICATE_VERIFY_FAILED`). Harbor's oracle agent (`harbor run -a oracle` on the task set, in + Harbor's own venv) finds such tasks before a run spends model calls on them; every agent of a Hermit run is + Hermit's. - Both arms of a comparison meet the same broken tasks, so differences stay comparable; absolute rates are lower than the leaderboard's by those tasks. diff --git a/examples/swebench_verified/baseline/claude.yaml b/examples/swebench_verified/baseline/claude.yaml index 86adfbe2..8076b4ac 100644 --- a/examples/swebench_verified/baseline/claude.yaml +++ b/examples/swebench_verified/baseline/claude.yaml @@ -14,17 +14,17 @@ data: train: ../utils/splits/smoke.txt # placeholder, as template and evolver are: hermit eval reads it, trains nothing test: ../utils/splits/test.txt # the 500 tasks agent: - name: harbor + base: claude # Claude Code in each task's own Harbor environment, installed there by Harbor concurrency: 8 - params: - benchmark: ../utils/benchmark.yaml - acting_agent: {cli: claude} # Claude Code in the task's container, installed there by Harbor - model: claude-haiku-5-5 # a Claude Code model name, on the subscription - login: claude # the subscription's plan, shared with Hermit's other sessions on it - effort: medium - jobs_dir: ~/.cache/hermit/harbor/jobs/swebv-claude # the Claude Code configs' trials + model: {name: claude-haiku-5-5, login: claude, effort: medium} # the subscription's plan, shared + claude: {web_search: false} # the CLI's web search off +environment: + harbor: # each task in its own Harbor environment, verified by Harbor's verifier + benchmark: ../utils/benchmark.yaml # the settings every config of this example shares + jobs_dir: ~/.cache/hermit/harbor/jobs/swebv-claude # the Claude Code configs' environments (each task's trial) judge: - path: hermit.harbor.judge # the verifier's reward + path: hermit.task.verify # Harbor's verifier, in the task's environment: the verifier's reward + name: environment template: prompt-only@1.0 # placeholder (see data.train) task_info: ../utils/task-info # what hermit check asks of a training config evolver: {name: eval-only} # placeholder: no Evolver of this name, so hermit train refuses this config diff --git a/examples/swebench_verified/baseline/codex.yaml b/examples/swebench_verified/baseline/codex.yaml index 446a5151..d1c759d1 100644 --- a/examples/swebench_verified/baseline/codex.yaml +++ b/examples/swebench_verified/baseline/codex.yaml @@ -13,17 +13,17 @@ data: train: ../utils/splits/smoke.txt # placeholder, as template and evolver are: hermit eval reads it, trains nothing test: ../utils/splits/test.txt # the 500 tasks agent: - name: harbor + base: codex # Codex in each task's own Harbor environment, installed there by Harbor concurrency: 8 - params: - benchmark: ../utils/benchmark.yaml - acting_agent: {cli: codex} # Codex in the task's container, installed there by Harbor - model: gpt-5.6-luna # a Codex model name, on the ChatGPT login - login: chatgpt # the ChatGPT plan, shared with Hermit's other sessions on it - effort: medium - jobs_dir: ~/.cache/hermit/harbor/jobs/swebv-codex # the Codex configs' trials + model: {name: gpt-5.6-luna, login: chatgpt, effort: medium} # the ChatGPT plan, shared + codex: {web_search: false} # the CLI's web search off +environment: + harbor: # each task in its own Harbor environment, verified by Harbor's verifier + benchmark: ../utils/benchmark.yaml # the settings every config of this example shares + jobs_dir: ~/.cache/hermit/harbor/jobs/swebv-codex # the Codex configs' environments (each task's trial) judge: - path: hermit.harbor.judge # the verifier's reward + path: hermit.task.verify # Harbor's verifier, in the task's environment: the verifier's reward + name: environment template: prompt-only@1.0 # placeholder (see data.train) task_info: ../utils/task-info # what hermit check asks of a training config evolver: {name: eval-only} # placeholder: no Evolver of this name, so hermit train refuses this config diff --git a/examples/swebench_verified/utils/benchmark.yaml b/examples/swebench_verified/utils/benchmark.yaml index 2f477f88..340c0e61 100644 --- a/examples/swebench_verified/utils/benchmark.yaml +++ b/examples/swebench_verified/utils/benchmark.yaml @@ -1,17 +1,16 @@ -# SWE-bench Verified through Harbor: the agent.params every config of this example shares (agent.params.benchmark; a -# config's own params override these), and the task set (the dataset: block, which `python -m hermit.harbor.dataset -# prepare` reads). One rollout is one Harbor trial of a task: Harbor builds the task's image (the SWE-bench image of the -# repository at the issue's commit, swebench/sweb.eval.x86_64. with the id's __ written _1776_), the acting agent -# edits the repository in /testbed, and the task's tests (the harness's FAIL_TO_PASS and PASS_TO_PASS lists, uploaded -# only after the agent stopped) score it: 1 if the issue is resolved, else 0. The acting agent, its model and its -# sign-in are each config's. +# SWE-bench Verified in Harbor's environments: the environment.harbor settings every config of this example shares +# (environment.harbor.benchmark; a config's own settings override these), and the task set (the dataset: block, which +# `python -m hermit.harbor.dataset prepare` reads). One rollout is one task in its own Harbor environment: Harbor +# builds the task's image (the SWE-bench image of the repository at the issue's commit, swebench/sweb.eval.x86_64. +# with the id's __ written _1776_), the acting agent edits the repository in /testbed, and the task's tests (the +# harness's FAIL_TO_PASS and PASS_TO_PASS lists, uploaded only after the agent stopped) score it: 1 if the issue is +# resolved, else 0. The acting agent, its model and its sign-in are each config's. dataset: ref: swebench-verified@1.0 # 500 tasks of 12 repositories split: in_sample # every task is a test task: the transfer protocol (AHE, RRSI) evaluates all # 500, and no method trains here smoke: [psf__requests-1142, pallets__flask-5014] # two small issues, rows #0 (splits/smoke.txt) tasks_dir: tasks # the 500 task directories (prepare's link, outside the checkout) -web_search: false # the CLI's web search off: the issues and their fixes are public agent_timeout_sec: null # each task's own limit (3,000 s for every task) environment: env: # thread pools sized to the task's CPU (every task has cpus = 1): Docker @@ -21,5 +20,5 @@ environment: NUMEXPR_NUM_THREADS: "1" docker_cleanup: images_keep # Harbor removes the image it built; the SWE-bench base layers stay in # Docker's build cache -disk_gate_gb: 25 # no trial starts with less than this free on Docker's data root +disk_gate_gb: 25 # no environment starts with less than this free on Docker's data root disk_gate_wait_s: 21600 diff --git a/examples/terminal_bench_2_0/README.md b/examples/terminal_bench_2_0/README.md index 7272641d..1e2927f4 100644 --- a/examples/terminal_bench_2_0/README.md +++ b/examples/terminal_bench_2_0/README.md @@ -96,12 +96,12 @@ hermit eval --config examples/terminal_bench_2_0/ahe_adapter/run_codex.yaml --a adapter (the baseline), the seed (`root`), the final adapter, each on rollouts #3 and #4, and the paper's own measure, the best iteration's pass@1 on its training rollouts (`runs/codex-ahe/workspace/state/best_ever.json`); the last is selected on the rollouts it reports, so it overstates what the adapter does on fresh ones. The two kinds count two -trials differently. One that is not the agent's result (infra, still after the runner ran it again): the iteration's +trials differently. One that is not the agent's result (infra: its environment could not be given or failed): the iteration's pass@1 counts it as a failure, as the paper does, and so does `hermit eval`'s `mean_score`, while `python -m hermit.harbor.report` leaves it out of its pass@1 and counts it (`infra`): report that pass@1 for each eval, and give that count beside each number. And one whose agent ran out of its hour or ended in an error while the tests passed: the iteration's pass@1 counts it as a failure, as the paper's counts a timed-out trial (§4.1), and `hermit eval` -scores it by the tests' reward, as Harbor does; count those in the eval's `results.jsonl` (an `output.status` of +scores it by the tests' reward, as Harbor does; count those in the eval's `results.jsonl` (an `outcome.status` of `timeout` or `agent_error` with a `judge.score` of 1) and give that count too. The learners run on the same plan as the acting agent: on the smoke's two tasks, one step's Debugger and Evolve diff --git a/examples/terminal_bench_2_0/ahe_adapter/run_claude.yaml b/examples/terminal_bench_2_0/ahe_adapter/run_claude.yaml index 6a7dcde6..a99d9bee 100644 --- a/examples/terminal_bench_2_0/ahe_adapter/run_claude.yaml +++ b/examples/terminal_bench_2_0/ahe_adapter/run_claude.yaml @@ -27,17 +27,17 @@ data: shuffle: false task_info: ../utils/task-info agent: - name: harbor + base: claude # Claude Code in each task's own Harbor environment, installed there by Harbor concurrency: 6 - params: - benchmark: ../utils/benchmark.yaml - acting_agent: {cli: claude} # Claude Code in the task's container, the adapter deployed as for `hermit deploy` - model: claude-haiku-5-5 # a Claude Code model name, on the subscription - login: claude # the subscription's plan, shared with Hermit's other sessions on it - effort: medium - jobs_dir: ~/.cache/hermit/harbor/jobs/tb20-claude # the Claude Code configs' trials + model: {name: claude-haiku-5-5, login: claude, effort: medium} # the subscription's plan, shared + claude: {web_search: false} # the CLI's web search off +environment: + harbor: # each task in its own Harbor environment, verified by Harbor's verifier + benchmark: ../utils/benchmark.yaml # the settings every config of this example shares + jobs_dir: ~/.cache/hermit/harbor/jobs/tb20-claude # the Claude Code configs' environments (each task's trial) judge: - path: hermit.harbor.judge # the verifier's reward; its output is what the Debugger reads + path: hermit.task.verify # Harbor's verifier's reward; its output is what the Debugger reads + name: environment template: ahe@1.0 # H_0 is the template's own adapter (`--adapter root`) evolver: name: ahe_adapter # the Agent Debugger and the Evolve Agent diff --git a/examples/terminal_bench_2_0/ahe_adapter/run_codex.yaml b/examples/terminal_bench_2_0/ahe_adapter/run_codex.yaml index 9b502209..f7e79a9e 100644 --- a/examples/terminal_bench_2_0/ahe_adapter/run_codex.yaml +++ b/examples/terminal_bench_2_0/ahe_adapter/run_codex.yaml @@ -27,17 +27,17 @@ data: shuffle: false task_info: ../utils/task-info agent: - name: harbor + base: codex # Codex in each task's own Harbor environment, installed there by Harbor concurrency: 6 - params: - benchmark: ../utils/benchmark.yaml - acting_agent: {cli: codex} # Codex in the task's container, the adapter deployed as for `hermit deploy` - model: gpt-5.6-luna # a Codex model name, on the ChatGPT login - login: chatgpt # the ChatGPT plan, shared with Hermit's other sessions on it - effort: medium - jobs_dir: ~/.cache/hermit/harbor/jobs/tb20-codex # the Codex configs' trials + model: {name: gpt-5.6-luna, login: chatgpt, effort: medium} # the ChatGPT plan, shared + codex: {web_search: false} # the CLI's web search off +environment: + harbor: # each task in its own Harbor environment, verified by Harbor's verifier + benchmark: ../utils/benchmark.yaml # the settings every config of this example shares + jobs_dir: ~/.cache/hermit/harbor/jobs/tb20-codex # the Codex configs' environments (each task's trial) judge: - path: hermit.harbor.judge # the verifier's reward; its output is what the Debugger reads + path: hermit.task.verify # Harbor's verifier's reward; its output is what the Debugger reads + name: environment template: ahe@1.0 # H_0 is the template's own adapter (`--adapter root`) evolver: name: ahe_adapter # the Agent Debugger and the Evolve Agent diff --git a/examples/terminal_bench_2_0/ahe_adapter/smoke_claude.yaml b/examples/terminal_bench_2_0/ahe_adapter/smoke_claude.yaml index 89555c1d..d922e567 100644 --- a/examples/terminal_bench_2_0/ahe_adapter/smoke_claude.yaml +++ b/examples/terminal_bench_2_0/ahe_adapter/smoke_claude.yaml @@ -13,17 +13,17 @@ data: shuffle: false task_info: ../utils/task-info agent: - name: harbor + base: claude # Claude Code in each task's own Harbor environment, installed there by Harbor concurrency: 2 - params: - benchmark: ../utils/benchmark.yaml - acting_agent: {cli: claude} - model: claude-haiku-5-5 - login: claude - effort: medium - jobs_dir: ~/.cache/hermit/harbor/jobs/tb20-claude-smoke # the smoke's trials, apart from the run's + model: {name: claude-haiku-5-5, login: claude, effort: medium} # the subscription's plan, shared + claude: {web_search: false} # the CLI's web search off +environment: + harbor: # each task in its own Harbor environment, verified by Harbor's verifier + benchmark: ../utils/benchmark.yaml # the settings every config of this example shares + jobs_dir: ~/.cache/hermit/harbor/jobs/tb20-claude-smoke # the smoke's environments, apart from the run's judge: - path: hermit.harbor.judge + path: hermit.task.verify # Harbor's verifier, in the task's environment: the verifier's reward + name: environment template: ahe@1.0 evolver: name: ahe_adapter diff --git a/examples/terminal_bench_2_0/ahe_adapter/smoke_codex.yaml b/examples/terminal_bench_2_0/ahe_adapter/smoke_codex.yaml index cb4ab8ba..11ad477a 100644 --- a/examples/terminal_bench_2_0/ahe_adapter/smoke_codex.yaml +++ b/examples/terminal_bench_2_0/ahe_adapter/smoke_codex.yaml @@ -13,17 +13,17 @@ data: shuffle: false task_info: ../utils/task-info agent: - name: harbor + base: codex # Codex in each task's own Harbor environment, installed there by Harbor concurrency: 2 - params: - benchmark: ../utils/benchmark.yaml - acting_agent: {cli: codex} - model: gpt-5.6-luna - login: chatgpt - effort: medium - jobs_dir: ~/.cache/hermit/harbor/jobs/tb20-codex-smoke # the smoke's trials, apart from the run's + model: {name: gpt-5.6-luna, login: chatgpt, effort: medium} # the ChatGPT plan, shared + codex: {web_search: false} # the CLI's web search off +environment: + harbor: # each task in its own Harbor environment, verified by Harbor's verifier + benchmark: ../utils/benchmark.yaml # the settings every config of this example shares + jobs_dir: ~/.cache/hermit/harbor/jobs/tb20-codex-smoke # the smoke's environments, apart from the run's judge: - path: hermit.harbor.judge + path: hermit.task.verify # Harbor's verifier, in the task's environment: the verifier's reward + name: environment template: ahe@1.0 evolver: name: ahe_adapter diff --git a/examples/terminal_bench_2_0/baseline/claude.yaml b/examples/terminal_bench_2_0/baseline/claude.yaml index 6b5e8c66..8b08a255 100644 --- a/examples/terminal_bench_2_0/baseline/claude.yaml +++ b/examples/terminal_bench_2_0/baseline/claude.yaml @@ -14,17 +14,17 @@ data: train: ../utils/splits/smoke.txt # placeholder, as template and evolver are: hermit eval reads it, trains nothing test: ../utils/rows/test.txt # 89 tasks x rollouts #3 and #4 agent: - name: harbor + base: claude # Claude Code in each task's own Harbor environment, installed there by Harbor concurrency: 6 - params: - benchmark: ../utils/benchmark.yaml - acting_agent: {cli: claude} # Claude Code in the task's container, installed there by Harbor - model: claude-haiku-5-5 # a Claude Code model name, on the subscription - login: claude # the subscription's plan, shared with Hermit's other sessions on it - effort: medium - jobs_dir: ~/.cache/hermit/harbor/jobs/tb20-claude # the Claude Code configs' trials + model: {name: claude-haiku-5-5, login: claude, effort: medium} # the subscription's plan, shared + claude: {web_search: false} # the CLI's web search off +environment: + harbor: # each task in its own Harbor environment, verified by Harbor's verifier + benchmark: ../utils/benchmark.yaml # the settings every config of this example shares + jobs_dir: ~/.cache/hermit/harbor/jobs/tb20-claude # the Claude Code configs' environments (each task's trial) judge: - path: hermit.harbor.judge # the verifier's reward + path: hermit.task.verify # Harbor's verifier, in the task's environment: the verifier's reward + name: environment template: prompt-only@1.0 # placeholder (see data.train) task_info: ../utils/task-info # what hermit check asks of a training config evolver: {name: eval-only} # placeholder: no Evolver of this name, so hermit train refuses this config diff --git a/examples/terminal_bench_2_0/baseline/codex.yaml b/examples/terminal_bench_2_0/baseline/codex.yaml index 22e79219..e93dc72c 100644 --- a/examples/terminal_bench_2_0/baseline/codex.yaml +++ b/examples/terminal_bench_2_0/baseline/codex.yaml @@ -13,17 +13,17 @@ data: train: ../utils/splits/smoke.txt # placeholder, as template and evolver are: hermit eval reads it, trains nothing test: ../utils/rows/test.txt # 89 tasks x rollouts #3 and #4 agent: - name: harbor + base: codex # Codex in each task's own Harbor environment, installed there by Harbor concurrency: 6 - params: - benchmark: ../utils/benchmark.yaml - acting_agent: {cli: codex} # Codex in the task's container, installed there by Harbor - model: gpt-5.6-luna # a Codex model name, on the ChatGPT login - login: chatgpt # the ChatGPT plan, shared with Hermit's other sessions on it - effort: medium - jobs_dir: ~/.cache/hermit/harbor/jobs/tb20-codex # the Codex configs' trials + model: {name: gpt-5.6-luna, login: chatgpt, effort: medium} # the ChatGPT plan, shared + codex: {web_search: false} # the CLI's web search off +environment: + harbor: # each task in its own Harbor environment, verified by Harbor's verifier + benchmark: ../utils/benchmark.yaml # the settings every config of this example shares + jobs_dir: ~/.cache/hermit/harbor/jobs/tb20-codex # the Codex configs' environments (each task's trial) judge: - path: hermit.harbor.judge # the verifier's reward + path: hermit.task.verify # Harbor's verifier, in the task's environment: the verifier's reward + name: environment template: prompt-only@1.0 # placeholder (see data.train) task_info: ../utils/task-info # what hermit check asks of a training config evolver: {name: eval-only} # placeholder: no Evolver of this name, so hermit train refuses this config diff --git a/examples/terminal_bench_2_0/utils/benchmark.yaml b/examples/terminal_bench_2_0/utils/benchmark.yaml index 5fadceb2..6cf21597 100644 --- a/examples/terminal_bench_2_0/utils/benchmark.yaml +++ b/examples/terminal_bench_2_0/utils/benchmark.yaml @@ -1,8 +1,9 @@ -# Terminal-Bench 2.0 through Harbor: the agent.params every config of this example shares (agent.params.benchmark; a -# config's own params override these), and the task set (the dataset: block, which `python -m hermit.harbor.dataset -# prepare` reads). One rollout is one Harbor trial of a task: Harbor pulls the task's prebuilt image, the acting agent -# works in the container until it stops or its time runs out, and the task's tests, uploaded only after the agent -# stopped, score it: 1 if every test passes, else 0. The acting agent, its model and its sign-in are each config's own. +# Terminal-Bench 2.0 in Harbor's environments: the environment.harbor settings every config of this example shares +# (environment.harbor.benchmark; a config's own settings override these), and the task set (the dataset: block, which +# `python -m hermit.harbor.dataset prepare` reads). One rollout is one task in its own Harbor environment: Harbor pulls +# the task's prebuilt image and starts its container, the acting agent (Hermit's) works in it until it stops or its +# time runs out, and the task's tests, uploaded only after the agent stopped, score it: 1 if every test passes, else 0. +# The acting agent, its model and its sign-in are each config's own. dataset: ref: terminal-bench@2.0 # 89 tasks: 4 easy, 55 medium, 30 hard rollouts: 4 # rows #1 ... #4: #1 and #2 are the training rollouts (k = 2), #3 and #4 @@ -11,15 +12,14 @@ dataset: smoke: [fix-git, log-summary-date-ranges] # two short tasks, rows #0 (splits/smoke.txt) image_sizes: true # image_sizes.json: each task image's size, for the disk gate tasks_dir: tasks # the 89 task directories (prepare's link, outside the checkout) -web_search: false # the CLI's web search off: the tasks and their solutions are public agent_timeout_sec: 3600 # the AHE paper's hour for every task (the tasks' own limits: 900 s for 49 of # them, up to 12,000 s); null runs each task's own limit, as the leaderboard verifier_timeout_multiplier: 5 # the verifier's limit only: a container gets its task's CPU quota (1 or 2 for # most tasks) but sees all of the host's cores, and a thread pool sized to them # runs slowly in that quota (torch-pipeline-parallelism's tests) environment_build_timeout_multiplier: 3 # the start includes the image's pull: 8.8 GB for the two mteb tasks -docker_cleanup: rmi_task_image # a task's image is removed after its trials: the 89 images take about +docker_cleanup: rmi_task_image # a task's image is removed after its rollouts: the 89 images take about # 120-150 GB on disk image_sizes: image_sizes.json # the disk gate keeps room for the images still to be pulled -disk_gate_gb: 25 # no trial starts with less than this free on Docker's data root +disk_gate_gb: 25 # no environment starts with less than this free on Docker's data root disk_gate_wait_s: 21600 diff --git a/examples/terminal_bench_science/ahe_adapter/run_claude.yaml b/examples/terminal_bench_science/ahe_adapter/run_claude.yaml index a6886c66..16ab6e2c 100644 --- a/examples/terminal_bench_science/ahe_adapter/run_claude.yaml +++ b/examples/terminal_bench_science/ahe_adapter/run_claude.yaml @@ -26,17 +26,17 @@ data: shuffle: false task_info: ../utils/task-info agent: - name: harbor + base: claude # Claude Code in each task's own Harbor environment, installed there by Harbor concurrency: 6 - params: - benchmark: ../utils/benchmark.yaml - acting_agent: {cli: claude} # Claude Code in the task's container, installed there by Harbor - model: claude-haiku-5-5 # a Claude Code model name, on the subscription - login: claude # the subscription's plan, shared with Hermit's other sessions on it - effort: medium - jobs_dir: ~/.cache/hermit/harbor/jobs/tbsci-claude # the Claude Code configs' trials + model: {name: claude-haiku-5-5, login: claude, effort: medium} # the subscription's plan, shared + claude: {web_search: false} # the CLI's web search off +environment: + harbor: # each task in its own Harbor environment, verified by Harbor's verifier + benchmark: ../utils/benchmark.yaml # the settings every config of this example shares + jobs_dir: ~/.cache/hermit/harbor/jobs/tbsci-claude # the Claude Code configs' environments (each task's trial) judge: - path: hermit.harbor.judge # the verifier's reward; its output is what the Debugger reads + path: hermit.task.verify # Harbor's verifier's reward; its output is what the Debugger reads + name: environment template: ahe@1.0 # H_0 is the template's own adapter (`--adapter root`) evolver: name: ahe_adapter # the Agent Debugger and the Evolve Agent diff --git a/examples/terminal_bench_science/ahe_adapter/run_codex.yaml b/examples/terminal_bench_science/ahe_adapter/run_codex.yaml index a6693460..3d580d86 100644 --- a/examples/terminal_bench_science/ahe_adapter/run_codex.yaml +++ b/examples/terminal_bench_science/ahe_adapter/run_codex.yaml @@ -26,17 +26,17 @@ data: shuffle: false task_info: ../utils/task-info agent: - name: harbor + base: codex # Codex in each task's own Harbor environment, installed there by Harbor concurrency: 6 - params: - benchmark: ../utils/benchmark.yaml - acting_agent: {cli: codex} # Codex in the task's container, installed there by Harbor - model: gpt-5.6-luna # a Codex model name, on the ChatGPT login - login: chatgpt # the ChatGPT plan, shared with Hermit's other sessions on it - effort: medium - jobs_dir: ~/.cache/hermit/harbor/jobs/tbsci-codex # the Codex configs' trials + model: {name: gpt-5.6-luna, login: chatgpt, effort: medium} # the ChatGPT plan, shared + codex: {web_search: false} # the CLI's web search off +environment: + harbor: # each task in its own Harbor environment, verified by Harbor's verifier + benchmark: ../utils/benchmark.yaml # the settings every config of this example shares + jobs_dir: ~/.cache/hermit/harbor/jobs/tbsci-codex # the Codex configs' environments (each task's trial) judge: - path: hermit.harbor.judge # the verifier's reward; its output is what the Debugger reads + path: hermit.task.verify # Harbor's verifier's reward; its output is what the Debugger reads + name: environment template: ahe@1.0 # H_0 is the template's own adapter (`--adapter root`) evolver: name: ahe_adapter # the Agent Debugger and the Evolve Agent diff --git a/examples/terminal_bench_science/ahe_adapter/smoke_claude.yaml b/examples/terminal_bench_science/ahe_adapter/smoke_claude.yaml index 24a3f0c7..aad0a003 100644 --- a/examples/terminal_bench_science/ahe_adapter/smoke_claude.yaml +++ b/examples/terminal_bench_science/ahe_adapter/smoke_claude.yaml @@ -13,18 +13,18 @@ data: shuffle: false task_info: ../utils/task-info agent: - name: harbor + base: claude # Claude Code in each task's own Harbor environment, installed there by Harbor concurrency: 2 - params: - benchmark: ../utils/benchmark.yaml - acting_agent: {cli: claude} # Claude Code in the task's container, installed there by Harbor - model: claude-haiku-5-5 # a Claude Code model name, on the subscription - login: claude # the subscription's plan, shared with Hermit's other sessions on it - effort: medium - jobs_dir: ~/.cache/hermit/harbor/jobs/tbsci-claude-smoke # the smoke's trials, apart from the run's - agent_timeout_sec: 1800 # the smoke's cap on a session; the run keeps each task's own + model: {name: claude-haiku-5-5, login: claude, effort: medium} # the subscription's plan, shared + claude: {web_search: false} # the CLI's web search off +environment: + harbor: # each task in its own Harbor environment, verified by Harbor's verifier + benchmark: ../utils/benchmark.yaml # the settings every config of this example shares + jobs_dir: ~/.cache/hermit/harbor/jobs/tbsci-claude-smoke # the smoke's environments, apart from the run's + agent_timeout_sec: 1800 # the smoke's cap on a session; the run keeps each task's own judge: - path: hermit.harbor.judge # the verifier's reward; its output is what the Debugger reads + path: hermit.task.verify # Harbor's verifier's reward; its output is what the Debugger reads + name: environment template: ahe@1.0 # H_0 is the template's own adapter (`--adapter root`) evolver: name: ahe_adapter # the Agent Debugger and the Evolve Agent diff --git a/examples/terminal_bench_science/ahe_adapter/smoke_codex.yaml b/examples/terminal_bench_science/ahe_adapter/smoke_codex.yaml index 1e667c7c..9c52643e 100644 --- a/examples/terminal_bench_science/ahe_adapter/smoke_codex.yaml +++ b/examples/terminal_bench_science/ahe_adapter/smoke_codex.yaml @@ -13,18 +13,18 @@ data: shuffle: false task_info: ../utils/task-info agent: - name: harbor + base: codex # Codex in each task's own Harbor environment, installed there by Harbor concurrency: 2 - params: - benchmark: ../utils/benchmark.yaml - acting_agent: {cli: codex} # Codex in the task's container, installed there by Harbor - model: gpt-5.6-luna # a Codex model name, on the ChatGPT login - login: chatgpt # the ChatGPT plan, shared with Hermit's other sessions on it - effort: medium - jobs_dir: ~/.cache/hermit/harbor/jobs/tbsci-codex-smoke # the smoke's trials, apart from the run's - agent_timeout_sec: 1800 # the smoke's cap on a session; the run keeps each task's own + model: {name: gpt-5.6-luna, login: chatgpt, effort: medium} # the ChatGPT plan, shared + codex: {web_search: false} # the CLI's web search off +environment: + harbor: # each task in its own Harbor environment, verified by Harbor's verifier + benchmark: ../utils/benchmark.yaml # the settings every config of this example shares + jobs_dir: ~/.cache/hermit/harbor/jobs/tbsci-codex-smoke # the smoke's environments, apart from the run's + agent_timeout_sec: 1800 # the smoke's cap on a session; the run keeps each task's own judge: - path: hermit.harbor.judge # the verifier's reward; its output is what the Debugger reads + path: hermit.task.verify # Harbor's verifier's reward; its output is what the Debugger reads + name: environment template: ahe@1.0 # H_0 is the template's own adapter (`--adapter root`) evolver: name: ahe_adapter # the Agent Debugger and the Evolve Agent diff --git a/examples/terminal_bench_science/baseline/claude.yaml b/examples/terminal_bench_science/baseline/claude.yaml index c2db04ca..01a0245f 100644 --- a/examples/terminal_bench_science/baseline/claude.yaml +++ b/examples/terminal_bench_science/baseline/claude.yaml @@ -15,17 +15,17 @@ data: train: ../utils/splits/smoke.txt # placeholder, as template and evolver are: hermit eval reads it, trains nothing test: ../utils/rows/test.txt # 70 tasks x rollouts #3 and #4 agent: - name: harbor + base: claude # Claude Code in each task's own Harbor environment, installed there by Harbor concurrency: 6 - params: - benchmark: ../utils/benchmark.yaml - acting_agent: {cli: claude} # Claude Code in the task's container, installed there by Harbor - model: claude-haiku-5-5 # a Claude Code model name, on the subscription - login: claude # the subscription's plan, shared with Hermit's other sessions on it - effort: medium - jobs_dir: ~/.cache/hermit/harbor/jobs/tbsci-claude # the Claude Code configs' trials + model: {name: claude-haiku-5-5, login: claude, effort: medium} # the subscription's plan, shared + claude: {web_search: false} # the CLI's web search off +environment: + harbor: # each task in its own Harbor environment, verified by Harbor's verifier + benchmark: ../utils/benchmark.yaml # the settings every config of this example shares + jobs_dir: ~/.cache/hermit/harbor/jobs/tbsci-claude # the Claude Code configs' environments (each task's trial) judge: - path: hermit.harbor.judge # the verifier's reward + path: hermit.task.verify # Harbor's verifier, in the task's environment: the verifier's reward + name: environment template: prompt-only@1.0 # placeholder (see data.train) task_info: ../utils/task-info # what hermit check asks of a training config evolver: {name: eval-only} # placeholder: no Evolver of this name, so hermit train refuses this config diff --git a/examples/terminal_bench_science/baseline/codex.yaml b/examples/terminal_bench_science/baseline/codex.yaml index 333a1c66..7b1a8465 100644 --- a/examples/terminal_bench_science/baseline/codex.yaml +++ b/examples/terminal_bench_science/baseline/codex.yaml @@ -14,17 +14,17 @@ data: train: ../utils/splits/smoke.txt # placeholder, as template and evolver are: hermit eval reads it, trains nothing test: ../utils/rows/test.txt # 70 tasks x rollouts #3 and #4 agent: - name: harbor + base: codex # Codex in each task's own Harbor environment, installed there by Harbor concurrency: 6 - params: - benchmark: ../utils/benchmark.yaml - acting_agent: {cli: codex} # Codex in the task's container, installed there by Harbor - model: gpt-5.6-luna # a Codex model name, on the ChatGPT login - login: chatgpt # the ChatGPT plan, shared with Hermit's other sessions on it - effort: medium - jobs_dir: ~/.cache/hermit/harbor/jobs/tbsci-codex # the Codex configs' trials + model: {name: gpt-5.6-luna, login: chatgpt, effort: medium} # the ChatGPT plan, shared + codex: {web_search: false} # the CLI's web search off +environment: + harbor: # each task in its own Harbor environment, verified by Harbor's verifier + benchmark: ../utils/benchmark.yaml # the settings every config of this example shares + jobs_dir: ~/.cache/hermit/harbor/jobs/tbsci-codex # the Codex configs' environments (each task's trial) judge: - path: hermit.harbor.judge # the verifier's reward + path: hermit.task.verify # Harbor's verifier, in the task's environment: the verifier's reward + name: environment template: prompt-only@1.0 # placeholder (see data.train) task_info: ../utils/task-info # what hermit check asks of a training config evolver: {name: eval-only} # placeholder: no Evolver of this name, so hermit train refuses this config diff --git a/examples/terminal_bench_science/utils/benchmark.yaml b/examples/terminal_bench_science/utils/benchmark.yaml index 39ca1b43..17ff5846 100644 --- a/examples/terminal_bench_science/utils/benchmark.yaml +++ b/examples/terminal_bench_science/utils/benchmark.yaml @@ -1,9 +1,9 @@ -# Terminal-Bench-Science through Harbor: the agent.params every config of this example shares -# (agent.params.benchmark; a config's own params override these), and the task set (the dataset: block, which -# `python -m hermit.harbor.dataset prepare` reads). One rollout is one Harbor trial of a task: Harbor builds the -# task's image (its Dockerfile, data and pinned packages), the acting agent works in the container until it stops or -# its time runs out, and the task's tests, uploaded only after the agent stopped and run with no network, score it: 1 -# if every test passes, else 0. The acting agent, its model and its sign-in are each config's own. +# Terminal-Bench-Science in Harbor's environments: the environment.harbor settings every config of this example shares +# (environment.harbor.benchmark; a config's own settings override these), and the task set (the dataset: block, which +# `python -m hermit.harbor.dataset prepare` reads). One rollout is one task in its own Harbor environment: Harbor +# builds the task's image (its Dockerfile, data and pinned packages), the acting agent works in the container until it +# stops or its time runs out, and the task's tests, uploaded only after the agent stopped and run with no network, +# score it: 1 if every test passes, else 0. The acting agent, its model and its sign-in are each config's own. dataset: ref: terminal-bench-science/terminal-bench-science@0.1.0 # 70 research workflows of the Harbor hub rollouts: 4 # rows #1 ... #4: #1 and #2 are the training rollouts (k = 2), #3 and #4 @@ -11,8 +11,7 @@ dataset: split: in_sample # every task is trained on and tested on, as the AHE paper does smoke: [clinical-metadata-recovery, variable-star-vetting] # rows #0 (splits/smoke.txt) tasks_dir: tasks # the 70 task directories (prepare's link, outside the checkout) -web_search: false # the CLI's web search off (the agent's network is public) agent_timeout_sec: null # each task's own limit: 8 hours (28,800 s) for every task docker_cleanup: images_keep # Harbor removes the image it built -disk_gate_gb: 25 # no trial starts with less than this free on Docker's data root +disk_gate_gb: 25 # no environment starts with less than this free on Docker's data root disk_gate_wait_s: 21600 diff --git a/hermit/adapter/session.py b/hermit/adapter/session.py index 62af4872..84884273 100644 --- a/hermit/adapter/session.py +++ b/hermit/adapter/session.py @@ -27,6 +27,7 @@ import asyncio import concurrent.futures from datetime import datetime, timezone +import contextvars import json import os from pathlib import Path @@ -43,6 +44,39 @@ ENV_SANDBOX = "HA_ADAPTER_SANDBOX" # "1": sandbox by default; "0": this is that sandbox's server ENV_ENFORCE = "HA_ADAPTER_SANDBOX_ENFORCE" # the run's sandbox.enforce ENV_PYTHON = "HA_FRAMEWORK_PYTHON" # the framework's interpreter, for a server started elsewhere +# The same, for an acting agent that runs in the run's own process (a Harbor task set's: its commands run in the +# task's container, its loop here): the enforce mode while the call runs, None outside one. Per call (a context +# variable), never this process's environment, which the learner's sessions share. +CONFINED: contextvars.ContextVar = contextvars.ContextVar("hermit_adapter_confined", default=None) + + +def confined() -> str | None: + """The enforce mode the adapter's code is confined with here ("required", "best_effort"), or None.""" + if os.environ.get(ENV_SANDBOX) == "1": + return os.environ.get(ENV_ENFORCE, "required") + return CONFINED.get() + + +def host_fault(exc: BaseException) -> str | None: + """This machine's failure behind `exc` (an adapter's deployment or context that failed), when the strict adapter + sandbox it ran in does not start here: the runner's staging_failed (infra, run again; never the adapter's scored + zero), as its message; None when the sandbox starts, or when nothing here is confined.""" + from hermit.sandbox import sandbox_fault + enforce = confined() + fault = sandbox_fault(enforce) if enforce else None + if fault is None: + return None + return f"staging_failed: {fault} (the adapter's: {type(exc).__name__}: {str(exc)[-1500:]})" + + +def staging_failed(message: str, started: float, log=None): + """A rollout whose staging failed on this machine before any session (the runner's staging_failed): infra, run + again, nothing spent.""" + import time + from hermit.common import RunOutput + return RunOutput(output=None, log=log, cost_usd=0.0, tokens=0, wall_s=round(time.time() - started, 1), + api_speed={"model_calls": 0}, + extra={"infra": "staging_failed", "infra_final": False, "exception_message": message[-2000:]}) def _adapter_sandbox(): @@ -50,8 +84,7 @@ def _adapter_sandbox(): the environment's keys; the framework's environment (sandbox.hide covered, carried in HA_SANDBOX_HIDE).""" from hermit.sandbox import LOCAL_ANY, Sandbox loopback = [k for k, v in os.environ.items() if LOCAL_ANY.search(str(v))] - return Sandbox(role="adapter", network="isolated", env_allow=loopback, - enforce=os.environ.get(ENV_ENFORCE, "required")) + return Sandbox(role="adapter", network="isolated", env_allow=loopback, enforce=confined() or "required") class Session: @@ -61,7 +94,7 @@ def __init__(self, adapter: Path | str, *, resources: dict | None = None, deploy self._remote = None self._sandboxed = False self.last_context: dict | None = None # the context tool's whole result from the last context(query=...) - if sandbox is None and os.environ.get(ENV_SANDBOX) == "1": + if sandbox is None and confined(): sandbox = _adapter_sandbox() owned = True else: diff --git a/hermit/agent/acting/agent.py b/hermit/agent/acting/agent.py index 391425a3..8e430758 100644 --- a/hermit/agent/acting/agent.py +++ b/hermit/agent/acting/agent.py @@ -2,13 +2,13 @@ class Agent: def __init__(self, params: dict, base: Path) # agent.params from the config, and its directory - def run(self, task: dict, adapter: Path | None) -> RunOutput + def run(self, task: dict, adapter: Path | None, environment=None) -> RunOutput A config names the agent: agent: name: react # a built-in: react | codex | claude-code | pi | mini-swe-agent | opencode | dsh - # | hermes + # | hermes | answer name: agent/agent.py:CLBenchAgent # or the task's own class (file.py:Class, module:Class) name: agent/pipeline.py:run # or a plain function run(task, adapter), wrapped as a FunctionAgent concurrency: 4 @@ -16,7 +16,10 @@ def run(self, task: dict, adapter: Path | None) -> RunOutput The Trainer, `eval` and `check` call nothing but `run`; what happens inside (a tool loop, a CLI process, a benchmark's own loop with the adapter plugged in) is the agent's business. `Session` gives any agent the -adapter's exports; `self.session(adapter)` builds one. +adapter's exports; `self.session(adapter)` builds one. `environment` is the task's own environment when the task set +gives one (a task's Harbor environment: hermit.environment.harbor), where the agent works through its `exec`, +`upload` and `download`; None for a task the agent works on in a workspace of its own. A run hands it only to an +agent whose `run` takes it. What an agent can take of an adapter is its `supports`: the exports (exports.yaml: tools, context_tool, skills, agents, hooks; every agent takes the context) it gives the session. A method says what of these it needs (its @@ -30,7 +33,7 @@ def run(self, task: dict, adapter: Path | None) -> RunOutput from pathlib import Path from typing import Callable -from hermit.task import RunOutput, load_callable +from hermit.common import RunOutput, load_callable EXPORTS = ("context", "context_tool", "tools", "skills", "agents", "hooks") # an adapter's, as exports.yaml names them @@ -44,9 +47,17 @@ def __init__(self, params: dict | None = None, base: Path | str | None = None): self.params = dict(params or {}) self.base = Path(base).resolve() if base is not None else Path.cwd() - def run(self, task: dict, adapter: Path | None) -> RunOutput: + def run(self, task: dict, adapter: Path | None, environment=None) -> RunOutput: raise NotImplementedError(f"{type(self).__name__}.run") + def installs(self, adapter: Path | None) -> list[dict]: + """What this agent needs installed in a task's own environment before its run (an environment that installs, + a task's Harbor one, does it at its setup; nothing installs while the agent runs): items of {agent, version, + probe} (a CLI by the environment's installer of that name, unless `probe`, a shell line, finds it and no + version is pinned) or {packages} (best effort). Nothing by default. Read on this machine for a built-in agent + only (a task's own class is not loaded here): a custom agent brings what it needs in the task's image.""" + return [] + # --- helpers for subclasses --------------------------------------------------------------------------------- def session(self, adapter: Path | None, **kwargs): """The adapter's exports as a Session, or None without an adapter (a baseline run).""" @@ -79,8 +90,14 @@ def __init__(self, fn: Callable, params: dict | None = None, base: Path | str | self.extra["params"] = self.params if "base" in wanted: self.extra["base"] = self.base - - def run(self, task: dict, adapter: Path | None) -> RunOutput: + self.takes_environment = "environment" in wanted + + def run(self, task: dict, adapter: Path | None, environment=None) -> RunOutput: + if environment is not None: + if not self.takes_environment: + name = getattr(self.fn, "__name__", self.fn) + raise TypeError(f"{name} takes no environment, and the task gives one: add an `environment` parameter") + return self.fn(task, adapter, environment=environment, **self.extra) return self.fn(task, adapter, **self.extra) @@ -97,14 +114,17 @@ def builtin(name: str): if backend is None: return CliAgent return lambda params, base: CliAgent({"backend": backend, **(params or {})}, base) + if name == "answer": + from hermit.agent.acting.answer import AnswerAgent + return AnswerAgent if name == "harbor": - from hermit.harbor.runner import HarborAgent - return HarborAgent + raise ValueError("agent.name: harbor is gone: a Harbor task set gives each task its environment " + "(environment.harbor), and any acting agent works in it; see docs/framework/task.md") return None BUILTIN_NAMES = ("react", "codex", "claude-code", "pi", "mini-swe-agent", "opencode", "dsh", "hermes", "cli", - "harbor") + "answer") def load_agent(name: str, params: dict | None, base: Path) -> Agent: @@ -126,19 +146,15 @@ def load_agent(name: str, params: dict | None, base: Path) -> Agent: def supports_of(name: str, base: Path, params: dict | None = None, *, may_import: bool) -> tuple[str, ...] | None: """What the agent `agent.name` names takes of an adapter (its `supports`), without running it: a built-in's from - its class (Harbor's from its acting agent, read from `params` and its benchmark file); a task's own class or - function only when `may_import` (it runs on this machine anyway), else None.""" + its class; a task's own class or function only when `may_import` (it runs on this machine anyway), else None.""" factory = builtin(str(name)) if factory is not None: if str(name) == "react": from hermit.agent.acting.react import ReactAgent return ReactAgent.supports - if str(name) == "harbor": - from hermit.harbor.runner import supports_of_params - try: - return supports_of_params(params, base) - except (OSError, ValueError): # an unreadable benchmark file: its own settings check reports it - return None + if str(name) == "answer": + from hermit.agent.acting.answer import AnswerAgent + return AnswerAgent.supports from hermit.agent.acting.workspace import CliAgent return CliAgent.supports if not may_import: diff --git a/hermit/agent/acting/answer.py b/hermit/agent/acting/answer.py new file mode 100644 index 00000000..fa279a23 --- /dev/null +++ b/hermit/agent/acting/answer.py @@ -0,0 +1,300 @@ +"""The `answer` agent: an acting agent that only answers, one model call, its answer written where the task's verifier +reads it. For a task whose work is a reply (a question about a figure, a value from a document) rather than a +session of work in the task's environment. + + agent: + name: answer + params: + model: fireworks/accounts/fireworks/models/glm-5p3 # a / reference, as react's + instruction_file: /app/instruction.md # optional: the question is this file of the task's environment (a + # task whose entrypoint writes it), waited for up to wait_s; else the + # environment's instruction (Harbor's), else the task's prompt-like + # field + attachments: ["/app/*.png"] # optional: globs of images in the environment, sent with the question + answer_file: /app/answer.txt # where the answer is written in the environment (the default, as the + # Harbor runner's answer agent wrote it; null: nowhere) + answer_pattern: '(.*?)' # optional: the answer is the reply's last match (group 1) + answer_format: "{answer}" # optional: the text written, with {answer} in it + max_tokens: 16384 + wait_s: 30 + +The adapter's context (ADAPTER.md, and what its context tool renders for the question) is the call's system message; +without an adapter there is none. The output is {answer, reply}; the log keeps the request's messages (an image as +its path, never its bytes) and the reply. Without the task's environment (a task set that gives none) the answer is +only returned. A figure over MAX_ATTACHMENT_BYTES is left out, as one that is no image. +""" +from __future__ import annotations + +import base64 +import json +import mimetypes +from pathlib import Path +import re +import shlex +import tempfile +import sys +import time + +from hermit.agent.acting.agent import Agent +from hermit.agent.base.deadline import Deadline +from hermit.agent.model.usage import Ledger +from hermit.agent.model.direct import Direct +from hermit.agent.model.llm import ChatAborted, ChatClient, ChatHTTPError +from hermit.common import RunOutput + +INPUT_FIELDS = ("prompt", "question", "instruction", "input") +ANSWER_FILE = "/app/answer.txt" # where the answer goes in the environment unless answer_file says +MAX_ATTACHMENT_BYTES = 20 * 2**20 +B64_BEGIN, B64_END = "hermit-attachment-begin", "hermit-attachment-end" # around an attachment's bytes there +DEFAULT_WAIT_S = 30.0 +POLL_S = 1.0 + + + +def _error_usage(client, exc) -> list: + """The usage an endpoint's error response reports (a relay's or provider's final body), priced as a reply's + would be (the runner's relay priced it too); none when the body holds none.""" + from hermit.agent.model.usage import Usage + try: + payload = json.loads(exc.body or b"null") + except ValueError: + return [] + u = payload.get("usage") if isinstance(payload, dict) else None + if not isinstance(u, dict): + return [] + try: + prompt, completion = int(u.get("prompt_tokens") or 0), int(u.get("completion_tokens") or 0) + cached = int((u.get("prompt_tokens_details") or {}).get("cached_tokens") or 0) + except (TypeError, ValueError, AttributeError): + return [] + if not (prompt or cached or completion): # a session's placeholder (hermit.tools.failed_usage's rule) + return [] + cost = client.cost(prompt, cached, completion) if callable(getattr(client, "cost", None)) else None + return [Usage(getattr(client, "model", ""), prompt, cached, completion, 1, cost)] + +class AnswerAgent(Agent): + supports = ("context", "context_tool") # the adapter's context is the call's system message + + def run(self, task: dict, adapter: Path | None, environment=None) -> RunOutput: + p = self.params + model = p.get("model") + if not isinstance(model, dict) or not model.get("base_url") or not model.get("model"): + raise ValueError("agent.params.model is required: a `/` reference") + started = time.time() + client = ChatClient.from_spec(model) + if model.get("max_retries") is not None: # the relay forwarded it; the chat client takes it here + client.max_retries = int(model["max_retries"]) + budget = getattr(getattr(environment, "limits", None), "timeout_s", None) if environment is not None else None + # the task's agent budget bounds the whole call (the run's one Deadline), as Harbor's agent timeout cancelled + # the runner's + at = getattr(environment, "deadline_at", None) # the phase's own absolute deadline, when it has one + caller = (Deadline(at) if at else Deadline.within(budget)).bound(client) if budget else client + try: + return self._answer(task, adapter, environment, model, client, caller, budget, started) + finally: + if budget: + caller.close() + + def _answer(self, task, adapter, environment, model, client, caller, budget, started) -> RunOutput: + p = self.params + question = self.question(task, environment) + session = None + try: + session = self.session(adapter) # a context tool's server starts here, in its sandbox + system = session.context(query=question, task=task) if session is not None else "" + record = session.last_context if session is not None else None + except Exception as exc: # the adapter's own (its context tool): the agent's result, scored 0 + from hermit.adapter.session import host_fault, staging_failed + fault = host_fault(exc) # unless the sandbox it runs in does not start here: infra + if fault is None and session is None and isinstance(exc, OSError): # its server not made: this machine's + fault = f"staging_failed: {type(exc).__name__}: {exc}" + if fault is not None: + if getattr(environment, "isolation", None) == "remote": # a task's own environment: run again + return staging_failed(fault, started) + raise RuntimeError(fault) from exc + message = f"{type(exc).__name__}: {exc}"[-2000:] + return RunOutput(output=None, cost_usd=0.0, tokens=0, wall_s=round(time.time() - started, 1), + api_speed={"model_calls": 0}, + extra={"adapter_failed": "adapter_failed", "exception_message": message, + "session_failed": message[-600:]}) + finally: + if session is not None: + session.close() + images, named = self.attachments(environment) + content = [{"type": "text", "text": question}, *({"type": "image_url", "image_url": {"url": url}} + for url in images)] if images else question + messages = ([{"role": "system", "content": system}] if system.strip() else []) + \ + [{"role": "user", "content": content}] + timed_out = False + try: + # max_tokens only when given (the runner's answer agent sent none otherwise) + reply = Direct({"max_tokens": int(p["max_tokens"]) if p.get("max_tokens") else None}, caller).run(messages) + except ChatAborted: + if not (budget and caller.expired): + raise + from hermit.agent.model.direct import Reply + reply, timed_out = Reply(reply=""), True + except ChatHTTPError as exc: # its body's usage counted, whatever the refusal means + spent = _error_usage(client, exc) + try: + reply, timed_out = self._refused(exc, spent), False + except (ChatHTTPError, RuntimeError) as refused: + if spent: # an infra end: the rollout keeps what the call spent + total = Ledger(getattr(client, "model", "")).add(spent).total() + refused.spent = RunOutput(output=None, cost_usd=total.cost_usd, + tokens=total.input_tokens + total.output_tokens, + api_speed={"model_calls": total.calls}) + raise + answer = self.answer(reply.reply) + # what the call spent, as soon as it answered: kept through a delivery that fails + usage = Ledger(getattr(client, "model", "")).add(reply.usage).unreported( + caller.abandoned if budget else 0).total() + try: + timed_out = self._deliver(environment, answer, caller, budget, timed_out) + except Exception as exc: # noqa: BLE001 - the environment's: infra, its spend kept + message = f"{type(exc).__name__}: {exc}"[-2000:] + if "AgentPhaseOver" in message: # Harbor ended the phase: the trial's verdict, its spend kept + return RunOutput(output=None, cost_usd=usage.cost_usd, wall_s=round(time.time() - started, 1), + tokens=usage.input_tokens + usage.output_tokens, + api_speed={"model_calls": usage.calls}, + extra={"timed_out": True, "session_failed": message[-600:]}) + return RunOutput(output=None, cost_usd=usage.cost_usd, wall_s=round(time.time() - started, 1), + tokens=usage.input_tokens + usage.output_tokens, api_speed={"model_calls": usage.calls}, + extra={"infra": "harbor_failed", "infra_final": False, "exception_message": message}) + # the run's log, a folder: the session's own record, and answer.json as the runner's answer agent published it + # in the agent logs (a verifier may read it, answer_file null or not), handed to the trial with the verdict + log = Path(tempfile.mkdtemp(prefix=f"{task.get('id', 'task')}-", dir=self.log_dir())) + shown = [{**m, "content": question if isinstance(m["content"], list) else m["content"]} for m in messages] + (log / "session.json").write_text(json.dumps({"task": task.get("id"), "model": model["model"], + "messages": shown, "attachments": named, "reply": reply.reply, + "answer": answer}, ensure_ascii=False, indent=1), + encoding="utf-8") + (log / "answer.json").write_text(json.dumps( + {"reply": reply.reply, "answer": answer, "usage": {"prompt_tokens": usage.input_tokens, + "completion_tokens": usage.output_tokens}, + "model": model["model"], "attachments": named}, indent=1, ensure_ascii=False) + "\n", encoding="utf-8") + extra = {"attachments": named, **({"timed_out": True} if timed_out else {})} + if record is not None: + extra["context_tool"] = record + return RunOutput(output={"answer": answer, "reply": reply.reply}, log=log, cost_usd=usage.cost_usd, + wall_s=round(time.time() - started, 1), tokens=usage.input_tokens + usage.output_tokens, + api_speed={"model_calls": usage.calls}, + messages=[*shown, {"role": "assistant", "content": reply.reply}], extra=extra) + + def _deliver(self, environment, answer: str, caller, budget, timed_out: bool) -> bool: + """The answer written where the verifier reads it (answer_file), whole within the budget or not at all; + whether the budget ran out.""" + p = self.params + answer_file = None if "answer_file" in p and p["answer_file"] is None else p.get("answer_file") or ANSWER_FILE + timed_out = timed_out or bool(budget and caller.expired) + if environment is not None and answer_file and not timed_out: + text = str(p.get("answer_format") or "{answer}").replace("{answer}", answer) + with tempfile.TemporaryDirectory(prefix="hermit-answer-") as tmp: + local = Path(tmp) / "answer.txt" + local.write_text(text, encoding="utf-8") + if not budget: + environment.upload(local, str(answer_file)) + else: # delivered whole within the budget, or not at all: put beside, then moved + part = f"{answer_file}.hermit-part" + environment.upload(local, part) + left = caller.remaining() + done = environment.exec(["mv", "-f", part, str(answer_file)], timeout=left) if left > 0 else None + if done is None or done.timed_out: # the budget spent first: no answer delivered + environment.exec(["rm", "-f", part, *([] if done is None else [str(answer_file)])], timeout=30) + timed_out = True + elif done.code != 0: + raise RuntimeError(f"the task's environment did not provide {answer_file} for the answer: " + f"{done.output[-300:]}") + return timed_out + + def question(self, task: dict, environment) -> str: + """The instruction file of the environment (waited for), else the task's prompt-like field.""" + path = self.params.get("instruction_file") + if path and environment is not None: + deadline = time.time() + float(self.params.get("wait_s") or DEFAULT_WAIT_S) + while True: + found = environment.exec(["cat", str(path)], timeout=60) + if found.code == 0: + return found.output.strip() + if time.time() >= deadline: # the environment's: infra, run again (the runner's launch_failed) + raise RuntimeError(f"the task's environment did not provide {path} (waited " + f"{self.params.get('wait_s') or DEFAULT_WAIT_S} s)") + time.sleep(POLL_S) + if getattr(environment, "instruction", None) is not None: + return environment.instruction # Harbor's instruction, as the runner's answer agent was given it + for field in INPUT_FIELDS: + if isinstance(task.get(field), str): + return task[field] + raise ValueError(f"task {task.get('id')!r} has none of {INPUT_FIELDS}") + + def attachments(self, environment) -> tuple[list[str], list[str]]: + """(the images as data URLs, their paths) the attachment globs match in the environment.""" + globs = list(self.params.get("attachments") or []) + if not globs or environment is None: + return [], [] + listing = environment.exec(["sh", "-c", "for f in " + " ".join(globs) + "; do [ -f \"$f\" ] && echo \"$f\"; " + "done; true"], timeout=60) + if listing.code != 0: + raise RuntimeError(f"the task's environment did not provide its attachments' listing: " + f"{listing.output[-500:]}") + urls, named = [], [] + found = list(dict.fromkeys(line for line in listing.output.splitlines() if line.strip())) # each image once + for path in found: + kind = mimetypes.guess_type(path)[0] or "" + if not kind.startswith("image/"): + continue + # measured and read there, as the task's agent user, through a link to its target (the runner's): one over + # MAX_ATTACHMENT_BYTES is left out with a warning, never brought here whole + quoted = shlex.quote(path) + size = environment.exec(["sh", "-c", f"wc -c < {quoted}"], timeout=60) + if size.code != 0 or not size.output.strip().split()[-1:] or not size.output.split()[-1].isdigit(): + raise RuntimeError(f"the task's environment did not provide the attachment {path}: " + f"{size.output[-300:]}") + if int(size.output.split()[-1]) > MAX_ATTACHMENT_BYTES: + print(f"WARNING: {path} is over {MAX_ATTACHMENT_BYTES} bytes: it is not sent", file=sys.stderr, + flush=True) + continue + # its own exit decides; the payload between marks of its own (the runner's), never a diagnostic line the + # environment's client printed beside it + data = environment.exec(["sh", "-c", f"echo {B64_BEGIN}; base64 < {quoted} || exit 1; echo {B64_END}"], + timeout=120) + framed = data.output.split(B64_BEGIN, 1)[-1].split(B64_END, 1)[0] if B64_BEGIN in data.output else "" + encoded = "".join(framed.split()) + try: + image = base64.b64decode(encoded, validate=True) + except ValueError: + image = b"" + if data.code != 0 or len(image) != int(size.output.split()[-1]): # the bytes measured, or a failure + raise RuntimeError(f"the task's environment did not provide the attachment {path}: " + f"{data.output[-300:]}") + urls.append(f"data:{kind};base64,{encoded}") + named.append(path) + return urls, named + + @staticmethod + def _refused(exc, spent=()): + """The model endpoint's refusal, read as the Harbor runner read the answer agent's: a bill or a spent quota ends + the run (usage_limit), the run's configuration refused (401-404, a request setting it does not take) is + model_rejected (never run again), a provider that could not serve is infra; a request refused for what its + input made it (a prompt too long, an invalid request) is the agent's result: no answer.""" + from hermit.agent.base.codex_stream import BILLING_REFUSED, CONFIG_REFUSED, QUOTA_SAID + from hermit.agent.base.interface import LoginFailed + from hermit.agent.model.direct import Reply + said = f"{exc.code} {exc.body.decode(errors='replace')[-1000:]}" + if exc.code == 402 or BILLING_REFUSED.search(said) or QUOTA_SAID.search(said): + raise LoginFailed(f"agent answer: usage_limit: the provider does not bill the account: {said}") from None + if exc.code in (401, 403, 404) or (exc.code == 400 and CONFIG_REFUSED.search(said)): + raise RuntimeError(f"agent answer: model_rejected: {said}") from None + if exc.code in (408, 409, 429) or exc.code >= 500: + raise exc + return Reply(reply="", usage=list(spent)) + + def answer(self, reply: str) -> str: + pattern = self.params.get("answer_pattern") + if not pattern: + return reply.strip() + found = list(re.finditer(str(pattern), reply)) # the runner's extract: its last match, group 1 when it has + if not found: # one and it took part, else the whole match + return reply.strip() + last = found[-1] + return (last.group(1) if last.re.groups and last.group(1) is not None else last.group(0)).strip() diff --git a/hermit/agent/acting/react.py b/hermit/agent/acting/react.py index e36c72d5..513903e3 100644 --- a/hermit/agent/acting/react.py +++ b/hermit/agent/acting/react.py @@ -40,13 +40,24 @@ from hermit.adapter.session import Session from hermit.agent.acting.agent import Agent from hermit.agent.acting.tools import MARK, as_tools, load_tools, tool # noqa: F401 (tool: for subclasses) -from hermit.agent.base.react import React, total as usage_total +from hermit.agent.base.deadline import Deadline +from hermit.agent.base.react import React +from hermit.agent.model.usage import Ledger from hermit.agent.base.tool import Tool from hermit.agent.model.direct import Direct -from hermit.agent.model.llm import ChatClient -from hermit.task import RunOutput +from hermit.agent.model.llm import ChatAborted, ChatClient +from hermit.common import RunOutput CONTEXT_HEADING = "# Instructions from the harness adapter" + + +def Bounded(client, budget_s: float): + """A chat client whose every call is given up once the task's whole agent budget is spent (a task's own + environment's `timeout_s`): the run's one Deadline, bound to the client.""" + from hermit.agent.base.deadline import Deadline + return Deadline.within(budget_s).bound(client) + + USER_FIELDS = ("prompt", "question", "instruction", "input") _TOOLS: dict[tuple, list] = {} @@ -108,8 +119,8 @@ def before_tool(self, loop, call): ... # and the loop's (hermit.agent.ba supports = ("context", "context_tool", "tools") # the adapter exports it takes (hermit.agent.acting.agent) environment = None # the current task's environment (environment_for), if any - def run(self, task: dict, adapter: Path | None) -> RunOutput: - return _run(task, adapter, self.params, self.base, agent=self) + def run(self, task: dict, adapter: Path | None, environment=None) -> RunOutput: + return _run(task, adapter, self.params, self.base, agent=self, environment=environment) # ---------- task-level hooks ---------- def system_prompt(self, task: dict, context: str) -> str: @@ -129,8 +140,11 @@ def answer(self, result, task: dict) -> str: return result.reply def environment_for(self, task: dict): - """The environment the task's tools work in (hermit.agent.environment), or None: the loop needs none.""" - return None + """The environment the task's tools work in (hermit.environment), or None: the loop needs none. By default the + call's own (the task's environment, when its task set gives one).""" + return self.given_environment + + given_environment = None # the environment the call was given (run's `environment`) # ---------- the loop's hooks (React's), forwarded ---------- def call_model(self, loop, messages, specs): @@ -182,13 +196,15 @@ def run(task: dict, adapter: Path | None, *, params: dict, base: Path) -> RunOut return ReactAgent(params, base).run(task, adapter) -def _run(task: dict, adapter: Path | None, params: dict, base: Path, agent: ReactAgent) -> RunOutput: +def _run(task: dict, adapter: Path | None, params: dict, base: Path, agent: ReactAgent, + environment=None) -> RunOutput: model = params.get("model") if not isinstance(model, dict) or not model.get("base_url") or not model.get("model"): raise ValueError("agent.params.model is required: a `/` reference, or {base_url, model, " "api_key_env, price} (the run's `model` is the default)") client = ChatClient.from_spec(model) # one client for the loop and the answer-tag call: its total is the run's agent = copy.copy(agent) # this task's agent: its environment, its tools' self, its loop + agent.given_environment = environment agent.environment = agent.environment_for(task) session = Session(adapter) if adapter is not None else None tools = agent.tools(task) + (session.as_tools() if session else []) @@ -204,16 +220,53 @@ def _run(task: dict, adapter: Path | None, params: dict, base: Path, agent: Reac {"role": "user", "content": user}] started = time.time() loop_agent = agent._loop_agent = _Loop(settings, client, agent) - result = loop_agent.run(messages, environment=agent.environment, tools=tools) + # The loop cannot stop a model call at a time limit (react refuses one): a task's environment that has its own + # budget (a Harbor task's) bounds each of its commands, and the whole of the loop's model calls (Bounded: the call + # in flight given up once it is spent, as Harbor's agent timeout cancels its agents), whatever runs the pipeline. + place = agent.environment + bounded = None + if place is not None and getattr(place, "limits", None) is not None and place.limits.timeout_s: + at = getattr(place, "deadline_at", None) # the phase's own absolute deadline, when it has one + bounded = Deadline(at).bound(client) if at else Bounded(client, place.limits.timeout_s) + loop_agent.model = bounded + place = place.with_limits(dataclasses.replace(place.limits, timeout_s=None)) + loop = loop_agent.start(messages, environment=place, tools=tools) + timed_out = False + try: + while not loop.done: + if bounded is not None and bounded.expired: # spent between steps (a tool ran it out) + timed_out = True + break + loop_agent.step(loop) + except ChatAborted: + if bounded is None or not bounded.expired: + raise + timed_out = True # the call in flight given up: what ran before is kept + result = loop.result() + if timed_out: + result = dataclasses.replace(result, timed_out=True) messages, turns, usages = result.session, result.extra["turns"], list(result.usage) - if tag and f"<{tag}>".upper() not in (result.reply or "").upper(): # the tag in the model's reply, not answer()'s - follow = Direct({"max_tokens": int(params.get("max_tokens", 16384)), - "temperature": params.get("temperature")}, client) - reply = follow.run(f"Stop using tools. Give your final answer now inside <{tag}>....", session=messages) - messages = reply.session[:-1] + [{"role": "assistant", "content": reply.reply}] - result = dataclasses.replace(result, reply=reply.reply, session=messages) - usages += reply.usage - turns += 1 + try: + if tag and not timed_out and f"<{tag}>".upper() not in (result.reply or "").upper(): # the reply's tag + follow = Direct({"max_tokens": int(params.get("max_tokens", 16384)), + "temperature": params.get("temperature")}, bounded or client) + try: + reply = follow.run(f"Stop using tools. Give your final answer now inside <{tag}>....", + session=messages) + messages = reply.session[:-1] + [{"role": "assistant", "content": reply.reply}] + result = dataclasses.replace(result, reply=reply.reply, session=messages) + usages += reply.usage + turns += 1 + except ChatAborted: + if bounded is None or not bounded.expired: + raise + result = dataclasses.replace(result, timed_out=True) + finally: + if bounded is not None: + bounded.close() + ledger = Ledger(getattr(client, "model", "")).add(usages) + if bounded is not None: # a request given up in flight may have been billed: its cost is not known + ledger.unreported(bounded.abandoned) answer = agent.answer(result, task) logs = Path(base) / params.get("logs", ".logs") @@ -222,8 +275,9 @@ def _run(task: dict, adapter: Path | None, params: dict, base: Path, agent: Reac log.write_text(json.dumps({"task": task.get("id"), "model": model["model"], "adapter": str(adapter) if adapter else None, "messages": messages}, ensure_ascii=False, indent=1)) # What this run's calls used, as call_model reported it (a subclass may call another client) and the follow-up's - total, model_s = usage_total(client, usages), result.extra["model_s"] - extra = {"turns": turns, "tool_calls": result.extra["tool_calls"]} + total, model_s = ledger.total(), result.extra["model_s"] + extra = {"turns": turns, "tool_calls": result.extra["tool_calls"], + **({"timed_out": True} if result.timed_out else {})} # the task's agent budget ran out if session is not None and session.last_context is not None: extra["context_tool"] = session.last_context # what the adapter's context tool rendered for this task return RunOutput(output=answer, log=log, cost_usd=total.cost_usd, wall_s=round(time.time() - started, 1), diff --git a/hermit/agent/acting/workspace.py b/hermit/agent/acting/workspace.py index 35db287a..106558d7 100644 --- a/hermit/agent/acting/workspace.py +++ b/hermit/agent/acting/workspace.py @@ -28,7 +28,7 @@ collect: utils/task.py:collect # optional: (task, workspace, log) -> facts, after the run context_query: question # the task field an adapter's context tool retrieves for # (the CLI, Codex or Claude Code, in a Docker container of its own: the run config's environment.image and - # environment.container, not these params; hermit.agent.environment.docker) + # environment.container, not these params; hermit.environment.docker) project_doc_max_bytes: null # Codex's cap on AGENTS.md; null: Codex's default shell_env: core # the variables Codex's commands get: core (no keys, no PYTHONPATH) or all web_search: false # web search, set explicitly either way: Codex's hosted search (it runs @@ -54,29 +54,43 @@ backend, model, effort, web_search, deploy, isolation, deployed, exit_code, timed_out, attempts, workspace, mounts (where each task mount inside the workspace is: bound or linked), and context_tool, task and session_failed (a session that gave no answer) when there are any. Usage is read from the event stream, as the AgentEvolver reads it. + +In a task's own environment (`run(task, adapter, environment=...)`, a remote one: a task's Harbor environment, +hermit.environment.harbor) the workspace is the environment's: the adapter is deployed here into a staging folder as +`hermit deploy` lays it out for a session there (the hosts' deployment with the environment's `python3`, its tools +not served, its hooks run from `hermit//` relative to the session's directory: `remote_hooks`), the staging's +files uploaded into the environment's workspace (Hermit's context first in a context file the image already has, +then the image's own text), the `files` uploaded beside them, and the CLI runs there (its Base Agent's remote +isolation). The adapter's notes come back from there. The adapter's tools, `mounts`, `prepare` and `collect` are not +supported there (refused). """ from __future__ import annotations +import dataclasses import inspect import json import os from pathlib import Path import re +import shlex import shutil import sys import time import uuid from hermit.agent.acting.agent import ALL, Agent -from hermit.agent.base.interface import BaseAgent, LoginFailed, make -from hermit.agent.environment import Container, Environment, Limits, Mount, Local, docker as container +from hermit.agent.base.interface import BaseAgent, LoginFailed, Unsupported, make +from hermit.environment import Container, Environment, Limits, Mount, Local, docker as container from hermit.agent.base.login import LOGINS, NO_LOGIN -from hermit.task import RunOutput, load_callable +from hermit.agent.model.usage import Ledger +from hermit.common import RunOutput, load_callable BACKENDS = ("codex", "claude", "pi", "mini-swe-agent", "opencode", "dsh", "hermes") DEPLOYS = ("host",) ISOLATIONS = ("none", "container") INPUT_FIELDS = ("question", "prompt", "instruction", "input", "query") +REMOTE_CODEX_DOC_BYTES = 262144 # Codex's project_doc_max_bytes in a task's own environment (the runner's) +MODEL_REJECTED = "model_rejected" # the provider refused the configuration (hermit.environment.harbor_errors) MEMORY = Path("memory") / "NOTES.md" # an adapter's notes the agent keeps across tasks (continual-harness) MEMORY_NOTE = ("`{path}` is your memory across questions: read it first, and record under a `## ` " "heading anything worth keeping for later questions (where figures live, how a source is laid out, " @@ -101,8 +115,10 @@ def _call(fn, *args, params: dict, base: Path): return fn(*args, **{k: v for k, v in (("params", params), ("base", Path(base))) if k in wanted}) -def _prompt(params: dict, base: Path, task: dict) -> str: +def _prompt(params: dict, base: Path, task: dict, environment=None) -> str: value = params.get("prompt") + if not value and getattr(environment, "instruction", None) is not None: + return environment.instruction # a task's own environment: Harbor's instruction, as the runner gave it if not value: for field in ("prompt", "question", "instruction", "input"): if isinstance(task.get(field), str): @@ -137,41 +153,102 @@ def _context(task: dict, adapter: Path, params: dict) -> tuple[str, dict | None] def deploy(adapter: Path, workspace: Path, backend: str, *, context: str | None = None, - context_limit: int | None = None, approve_tools: bool = False, serve_tools: bool = True) -> dict: + context_limit: int | None = None, approve_tools: bool = False, serve_tools: bool = True, + python: str | None = None) -> dict: """`hermit deploy`'s deployment into a workspace; `context` is this task's, for an adapter whose context is per task, `context_limit` Codex's project_doc_max_bytes for the session (None: Codex's default), `approve_tools` the - adapter's tools called without asking.""" + adapter's tools called without asking, `python` the interpreter the deployment's commands run with where the + session runs (default: this one).""" + python = python or sys.executable if backend == "codex": from hermit.hosts.codex import DOC_BYTES, prepare - return prepare(adapter, workspace, sys.executable, context=context, context_limit=context_limit or DOC_BYTES, + return prepare(adapter, workspace, python, context=context, context_limit=context_limit or DOC_BYTES, approve_tools=approve_tools, serve_tools=serve_tools) if backend == "pi": from hermit.hosts.pi import prepare as prepare_pi - return prepare_pi(adapter, workspace, sys.executable, context=context) + return prepare_pi(adapter, workspace, python, context=context) if backend == "mini-swe-agent": from hermit.hosts.mini_swe_agent import prepare as prepare_mini - return prepare_mini(adapter, workspace, sys.executable, context=context, serve_tools=serve_tools) + return prepare_mini(adapter, workspace, python, context=context, serve_tools=serve_tools) if backend == "opencode": from hermit.hosts.opencode import prepare as prepare_opencode - return prepare_opencode(adapter, workspace, sys.executable, context=context, serve_tools=serve_tools) + return prepare_opencode(adapter, workspace, python, context=context, serve_tools=serve_tools) if backend == "dsh": from hermit.hosts.dsh import prepare as prepare_dsh - return prepare_dsh(adapter, workspace, sys.executable, context=context) + return prepare_dsh(adapter, workspace, python, context=context) if backend == "hermes": from hermit.hosts.hermes import prepare as prepare_hermes - return prepare_hermes(adapter, workspace, sys.executable, context=context, serve_tools=serve_tools) + return prepare_hermes(adapter, workspace, python, context=context, serve_tools=serve_tools) from hermit.hosts.claude_code import prepare - return prepare(adapter, workspace, sys.executable, context=context, serve_tools=serve_tools) - - -def _deploy(task: dict, adapter: Path | None, work: Path, backend: str, params: dict) -> tuple[dict, dict | None]: - """The adapter in the workspace; (what was deployed, the context tool's record or None).""" + return prepare(adapter, workspace, python, context=context, serve_tools=serve_tools) + + +REMOTE_HOOK_HOSTS = {"codex": "hermit.hosts.codex", "claude": "hermit.hosts.claude_code"} +CONTEXT_FILES = {"codex": "AGENTS.md", "claude": "CLAUDE.md"} + + +def remote_hooks(adapter: Path, manifest: dict, deploy_dir: Path, backend: str) -> bool: + """The deployed hooks rewritten for a session in a remote environment: the ones the host's deployment connects (its + HOOK_TYPES), every command run from the adapter's copy by its workspace-relative path (Codex's + `.codex/hooks.json`; the `hooks` of Claude Code's `.claude/settings.json`, from `$CLAUDE_PROJECT_DIR`, since Claude + Code runs a hook in the shell's current directory), not from this machine's folder. True when the adapter has + hooks the CLI runs.""" + from importlib import import_module + import shlex + from hermit.adapter.loader import hook_definitions + from hermit.hosts.inputs import ADAPTER_DIR, hook_commands, native_hooks + if backend not in REMOTE_HOOK_HOSTS: + return False + hooks = native_hooks(hook_definitions(adapter, manifest), import_module(REMOTE_HOOK_HOSTS[backend]).HOOK_TYPES) + relative = hook_commands(hooks, Path(ADAPTER_DIR) / manifest["id"]) if hooks else None + if backend == "claude": + if relative is not None: + folder = f"{ADAPTER_DIR}/{manifest['id']}" + plain = f"cd {shlex.quote(folder)} || exit 1\n" + for groups in relative["hooks"].values(): + for hook in (h for g in groups for h in g["hooks"]): + if hook.get("type") == "command" and hook["command"].startswith(plain): + hook["command"] = f'cd "${{CLAUDE_PROJECT_DIR:-.}}"/{shlex.quote(folder)} || exit 1\n' + \ + hook["command"][len(plain):] + path = deploy_dir / ".claude" / "settings.json" + settings = json.loads(path.read_text()) if path.is_file() else {} + if relative is None: + if settings.pop("hooks", None) is not None: + path.write_text(json.dumps(settings, indent=2) + "\n") + return False + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps({**settings, "hooks": relative["hooks"]}, indent=2) + "\n") + return True + path = deploy_dir / ".codex" / "hooks.json" + if relative is None: + path.unlink(missing_ok=True) + return False + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(relative, indent=2) + "\n") + return True + + +def _deploy(task: dict, adapter: Path | None, work: Path, backend: str, params: dict, *, + remote: bool = False) -> tuple[dict, dict | None]: + """The adapter in the workspace (for a remote environment: in the staging folder `work`, laid out for a session + there); (what was deployed, the context tool's record or None).""" if adapter is None: return {"mode": "host"}, None adapter = Path(adapter) from hermit.adapter.loader import agent_tools, load_manifest from hermit.adapter.manifest import PORTABLE manifest = load_manifest(adapter) + if remote: + if backend not in REMOTE_HOOK_HOSTS: + raise Unsupported(f"agent {backend} does not run in a remote environment (codex and claude do)") + if agent_tools(manifest): + raise Unsupported(f"the adapter {manifest['id']} has tools, and a session in a remote environment (a " + "task's Harbor environment) cannot reach the framework's tool server yet") + context, record = _context(task, adapter, params) if not manifest.get(PORTABLE, True) else (None, None) + deployed = deploy(adapter, work, backend, context=context, context_limit=params.get("project_doc_max_bytes"), + approve_tools=True, serve_tools=False, python="python3") + hooks = remote_hooks(adapter, manifest, work, backend) + return {"mode": "host", **deployed, "hooks": hooks}, record # A container session cannot start the tools' server on this machine, mini-swe-agent reaches tools only # through its Base Agent, OpenCode waits 2 s at most for a starting server (STARTUP_GRACE_MS in 2.2; the # adapter's takes longer, so its first request would go without them), and Hermes reads MCP servers from its home @@ -280,13 +357,157 @@ def _output(last: str, task: dict, params: dict, base: Path, code: int, timed_ou class CliAgent(Agent): supports = ALL # a deployment connects each export where the CLI has a place, and names the rest in the context - def run(self, task: dict, adapter: Path | None) -> RunOutput: - return _run(task, adapter, self.params, self.base) + def run(self, task: dict, adapter: Path | None, environment=None) -> RunOutput: + return _run(task, adapter, self.params, self.base, environment) + + def installs(self, adapter: Path | None) -> list[dict]: + return installs(self.params, adapter) + + def usage_of(self, logs: Path) -> list: + """The usage a session's stream left in `logs` (a trial's agent logs: the stream the CLI wrote there as it ran), + read and priced as the session's own would have been: what a session Harbor's timeout cut off had already + reported (the runner parsed the retained stream after its trial too).""" + backend = _check(self.params.get("backend", "codex"), self.params) + agent = base_agent(backend, self.params) + stream = Path(logs) / f"agent-{agent.backend}.jsonl" + if not stream.is_file() or stream.is_symlink(): + return [] + from hermit.agent.base.records import price_usage + s = agent.settings + _, usage = agent.parse(stream, (s.get("binding") or {}).get("ref") or s.get("model") or agent.backend) + price_usage(usage, s) + return list(usage) + + +def installs(params: dict, adapter: Path | None) -> list[dict]: + """What a CLI agent needs in a task's own environment before its session (Agent.installs): its CLI, unless `bin` + names one there (by the environment's installer of the CLI's name, unless it is on the session's PATH there and no + `version` is pinned, as the Harbor runner installed it), and python3 when the adapter has hooks the CLI runs.""" + from hermit.agent.base.cli import REMOTE_INSTALL, REMOTE_PATH + backend = str(params.get("backend") or "codex") + backend = "claude" if backend == "claude-code" else backend # as _check names it + out = [] + if params.get("bin") and backend in REMOTE_INSTALL: # a mounted binary: used when it serves, else installed + out.append({"agent": REMOTE_INSTALL[backend], "version": params.get("version"), "bin": str(params["bin"]), + "program": backend}) + elif backend in REMOTE_INSTALL: + out.append({"agent": REMOTE_INSTALL[backend], "version": params.get("version"), + "probe": "bash -c " + shlex.quote(REMOTE_PATH + f"command -v {backend}")}) + if adapter is not None and backend in REMOTE_HOOK_HOSTS: + from importlib import import_module + from hermit.adapter.loader import hook_definitions, load_manifest + from hermit.hosts.inputs import native_hooks + try: + manifest = load_manifest(Path(adapter)) + hooks = native_hooks(hook_definitions(Path(adapter), manifest), + import_module(REMOTE_HOOK_HOSTS[backend]).HOOK_TYPES) + except Exception: # noqa: BLE001 - the deployment says why (adapter_failed) + hooks = None + if hooks: + out.append({"packages": ["python3"]}) + return out -def run(task: dict, adapter: Path | None, *, params: dict, base: Path) -> RunOutput: +def run(task: dict, adapter: Path | None, *, params: dict, base: Path, environment=None) -> RunOutput: """The function form; CliAgent is the class (`agent.name: codex` or `claude-code`).""" - return _run(task, adapter, params, base) + return _run(task, adapter, params, base, environment) + + +def _infra_run(cause: str, message: str, usages: Ledger, log: Path, started: float, *, final: bool) -> RunOutput: + """A run in a task's own environment that is not the agent's result (`extra.infra`, which the run reads: + hermit.trainer.rollout), with what its sessions spent: `final`, never run again (the provider allowance spent, the + configuration refused).""" + ledger = usages # a session ran: none reported is not known (the runner's usage None), never 0 + return RunOutput(output=None, log=log, cost_usd=ledger.cost(None), wall_s=round(time.time() - started, 1), + tokens=ledger.tokens(None), api_speed={"model_calls": ledger.calls()}, + extra={"infra": cause, "infra_final": final, "exception_message": message[-2000:]}) + + +def _copy_files(params: dict, base: Path, work: Path) -> None: + """The config's `files`, copied into the workspace before the run.""" + for item in params.get("files") or []: + source = Path(base) / item + target = work / source.name + shutil.copytree(source, target) if source.is_dir() else shutil.copy2(source, target) + + +class AdapterContextTooLong(ValueError): + """The adapter's context with the image's own over the CLI's cap: the adapter's result (context_over_limit).""" + + +def _adapter_failed(cause: str, message: str, usages: list, log: Path, started: float) -> RunOutput: + """The adapter failed before any session (its deployment, its per-task context, its context over the CLI's cap): + the agent's result, scored 0 with its message (`extra.adapter_failed`, which the run reads).""" + (log / "adapter_failed.txt").write_text(message, encoding="utf-8") + return RunOutput(output=None, log=log, cost_usd=0.0, wall_s=round(time.time() - started, 1), tokens=0, + api_speed={"model_calls": 0}, extra={"adapter_failed": cause, "exception_message": message[-2000:], + "session_failed": message[-600:]}) + + +def _upload_staging(environment: Environment, work: Path, backend: str, deployed: dict, params: dict, + image: dict | None = None) -> None: + """The staging folder's files into a remote environment's workspace. A context file the image already has there + is kept after Hermit's text (Hermit's first), as the Harbor runner merged them; a file that is there but cannot be + read fails the call (the environment's fault: never the adapter's text alone in its place), and for Codex the + merged text must fit its cap on AGENTS.md (project_doc_max_bytes), as the runner checked it.""" + name = CONTEXT_FILES.get(backend) + mine = work / name if name else None + if mine is not None and mine.is_file() and deployed.get("adapter_id"): + image = {} if image is None else image # the image's own file, read once: a run again finds the merge + if "text" not in image: + image["text"] = None + if environment.exec(["test", "-e", name], timeout=60).code == 0: + # read there, a link followed there (AGENTS.md -> CLAUDE.md), as the runner read it in the container + got = environment.exec(["sh", "-c", f'printf "{IMAGE_MARK}"; cat -- {name}'], timeout=120) + if got.code != 0 or IMAGE_MARK not in got.output: + del image["text"] + raise RuntimeError(f"{IMAGE_CONTEXT_UNREAD} ({name}): {got.output[-300:]}") + image["text"] = got.output.split(IMAGE_MARK, 1)[1] + if image["text"] is not None: + mine.write_text(mine.read_text(encoding="utf-8") + "\n\n" + image["text"], encoding="utf-8") + # the merge replaces a link there, never writes through it to the file it names + environment.exec(["sh", "-c", f"if [ -L {name} ]; then rm -f -- {name}; fi"], timeout=60) + if backend == "codex": + from hermit.hosts.codex import DOC_BYTES + limit = int(params.get("project_doc_max_bytes") or DOC_BYTES) + size = len(mine.read_bytes()) + if size > limit: + raise AdapterContextTooLong(f"the adapter's context with the image's own {name} is {size} bytes, over " + f"Codex's limit of {limit} (project_doc_max_bytes): the session would not " + "read all of it") + for entry in sorted(work.iterdir()): + environment.upload(entry, entry.name) + + +IMAGE_CONTEXT_UNREAD = "the image's own context file could not be read" +IMAGE_MARK = "hermit-image-context:" # what precedes the file's text in the reading command's output + + +def _ensure_python3(environment: Environment) -> bool: + """Whether python3 is where the session runs, for the adapter's hooks (rendered for `python3`): installed at the + environment's setup when the agent asked for it (installs: an adapter with hooks), as the Harbor runner ensured it + before a session with hooks. Missing: a warning, and the session runs with hooks that may fail open.""" + if environment.exec(["sh", "-c", "command -v python3"], timeout=60).code == 0: + return True + print("WARNING: python3 for the adapter's hooks is not in the environment", file=sys.stderr, flush=True) + return False + + +def _collect_hooks(environment: Environment, log: Path) -> None: + """The adapter's hook audit and state (`.hermit-hooks/` in the workspace: hook-calls.jsonl, hook-state/) into the + log, where the Harbor runner's curated log kept them and the AHE Debugger and learner read them.""" + if environment.exec(["test", "-d", ".hermit-hooks"], timeout=60).code != 0: + return + got = log.parent / "hook-logs" + environment.download(".hermit-hooks", got) + for entry in sorted(got.iterdir()): + target = log / entry.name + if target.is_dir() and not target.is_symlink(): + shutil.rmtree(target) + elif os.path.lexists(target): + target.unlink() + shutil.move(str(entry), str(target)) + shutil.rmtree(got, ignore_errors=True) def _check(backend: str, params: dict) -> str: @@ -352,7 +573,7 @@ def base_agent(backend: str, params: dict) -> BaseAgent: """The Base Agent (Layer 2) these params describe: the CLI and how it is run, nothing of the task.""" from hermit.agent.base.cli import HERMES_PARAMS, PI_PARAMS settings = {k: params[k] for k in ("model", "binding", "login", "plan_dir", "plan_max_wait_s", "usage_credits", - "effort", "codex_sandbox", "shell_env", + "effort", "codex_sandbox", "shell_env", "bin", "version", "remote_home", "project_doc_max_bytes", "args", "env", "price", *PI_PARAMS, *(p for p in HERMES_PARAMS if p != "skills_dir"), "model_class", "config", "step_limit") @@ -364,8 +585,25 @@ def base_agent(backend: str, params: dict) -> BaseAgent: return make(backend, {**settings, "web_search": params["web_search"]}) -def _run(task: dict, adapter: Path | None, params: dict, base: Path) -> RunOutput: +def _run(task: dict, adapter: Path | None, params: dict, base: Path, given: Environment | None = None) -> RunOutput: backend = _check(params.get("backend", "codex"), params) + remote = given is not None and given.isolation == "remote" + if given is not None and not remote: + raise Unsupported(f"agent {backend} makes its own workspace; the environment a task gives it is a remote one " + f"(a task's Harbor environment), not {given!r}") + if remote: + unsupported = [k for k in ("mounts", "prepare", "collect") if params.get(k)] + lim = given.limits + if lim.readable or lim.hidden or lim.network is not None: + raise Unsupported(f"agent {backend} in a remote environment takes no readable mounts, hidden paths or " + "network limit of Hermit's (the task's own say what its environment reaches)") + if unsupported or params.get("isolation") == "container": + raise Unsupported(f"agent {backend} in a remote environment (a task's Harbor environment) takes none of " + f"{', '.join(unsupported) or 'isolation: container'}: the environment is the task's own") + if remote and backend == "codex" and not params.get("project_doc_max_bytes"): + # the Harbor runner's cap on AGENTS.md for Codex in a task's container (its CLI table's: 256 KiB), for the + # deployment's check and the session's own setting alike + params = {**params, "project_doc_max_bytes": REMOTE_CODEX_DOC_BYTES} if params.get("config"): # mini-swe-agent's config file, beside the run config params = {**params, "config": str(Path(base) / os.path.expanduser(str(params["config"])))} agent = base_agent(backend, params) @@ -379,20 +617,65 @@ def _run(task: dict, adapter: Path | None, params: dict, base: Path) -> RunOutpu work, home, log = run_dir / "work", run_dir / "home", run_dir / "log" agent = agent.configured(home=home, log=log / f"agent-{backend}.jsonl", **({"login_root": params["login_root"]} if params.get("login_root") else {})) - timeout_s = float(params.get("timeout_s", 3600)) - retries = max(1, int(params.get("retries", 3))) + # In a task's own environment the session's clock is the task's agent budget unless the agent's own `timeout_s` + # says less (the Harbor runner gave each task its own: Terminal-Bench-Science's 8 hours) + own_budget = given.limits.timeout_s if remote else None + default_s = float(own_budget) if own_budget else 3600.0 + timeout_s = float(params.get("timeout_s") or default_s) + # In a task's own environment a provider that could not serve is run again by the run, in a fresh environment + # (the Harbor runner's busy_retries, one allowance over the task's runs: environment.harbor.busy_retries), never + # here in the environment the session already changed + retries = 1 if remote else max(1, int(params.get("retries", 3))) + usages = Ledger() # every attempt's usage: a run again spends too + image_context: dict = {} # a task environment's own context file, read before the first deployment started = time.time() # `deadline_s`: the whole call's time (attempts and waits), set by a caller that is stopped at it (a container # session must end, and remove its container, before its worker is killed) + from hermit.agent.base.deadline import Deadline budget = float(params["deadline_s"]) if params.get("deadline_s") else None - if budget is not None: # and the session's own waits (a login's plan) come off it too - agent = agent.configured(deadline=started + budget) + deadline = Deadline(started + budget if budget is not None else None) # the run's one deadline + if deadline.at is not None: # and the session's own waits (a login's plan) come off it too + agent = agent.configured(deadline=deadline.at) for attempt in range(1, retries + 1): - for d in (work, home, log): - shutil.rmtree(d, ignore_errors=True) - d.mkdir(parents=True) - deployed, context_tool = _deploy(task, adapter, work, backend, params) - session = agent.configured(args=[*(agent.settings.get("args") or []), *_session_args(backend, deployed, work)], + try: + for d in (work, home, log): + shutil.rmtree(d, ignore_errors=True) + d.mkdir(parents=True) + except OSError as exc: + if not remote: + raise + from hermit.adapter.session import staging_failed # this machine's (a full disk): the runner's staging + return staging_failed(f"staging_failed: {type(exc).__name__}: {exc}", started) + if remote: + # the adapter's own failure (its deployment, its per-task context, a context over the CLI's cap) is the + # agent's result, scored 0 with its message, as the runner graded adapter_failed and context_over_limit; + # the environment's failing under it stays infra + row = task if any(isinstance(task.get(f), str) for f in INPUT_FIELDS) or params.get("context_query") \ + else {**task, "instruction": given.instruction} if given.instruction is not None else task + try: + deployed, context_tool = _deploy(row, adapter, work, backend, params, remote=True) + _copy_files(params, base, work) + _upload_staging(given, work, backend, deployed, params, image_context) + except AdapterContextTooLong as exc: + return _adapter_failed("context_over_limit", str(exc), usages, log, started) + except Unsupported: + raise + except OSError as exc: # this machine's (a full disk): the runner's staging_failed + from hermit.adapter.session import staging_failed + return staging_failed(f"staging_failed: {type(exc).__name__}: {exc}", started, log) + except Exception as exc: + said = type(exc).__name__ + str(exc) + if IMAGE_CONTEXT_UNREAD in said or "EnvironmentUnavailable" in said or "AgentPhaseOver" in said: + raise # the environment's, or Harbor ended the phase: not the adapter + from hermit.adapter.session import host_fault, staging_failed + fault = host_fault(exc) # this machine's, whatever the adapter (the runner's staging) + if fault is not None: + return staging_failed(fault, started, log) + return _adapter_failed("adapter_failed", f"{type(exc).__name__}: {exc}", usages, log, started) + else: + deployed, context_tool = _deploy(task, adapter, work, backend, params, remote=False) + where = Path(str(given.workspace)) if remote else work # the workspace as the session sees it + session = agent.configured(args=[*(agent.settings.get("args") or []), *_session_args(backend, deployed, where)], **({"tools_name": deployed["harness_tools"].replace("-", "_")} if deployed.get("harness_tools") else {}), # OpenCode loads only the skills it is allowed: the deployment's @@ -401,72 +684,123 @@ def _run(task: dict, adapter: Path | None, params: dict, base: Path) -> RunOutpu **({"skills_dir": deployed["skills_dir"]} if backend == "hermes" and deployed.get("skills_dir") else {})) memory = _memory(work, deployed) - for item in params.get("files") or []: - source = Path(base) / item - target = work / source.name - shutil.copytree(source, target) if source.is_dir() else shutil.copy2(source, target) - mounts = _mounts(task, work, params, base) - prompt = _prompt(params, base, task) + ( + if not remote: + _copy_files(params, base, work) + if remote and deployed.get("hooks"): + _ensure_python3(given) + mounts = [] if remote else _mounts(task, work, params, base) + prompt = _prompt(params, base, task, given if remote else None) + ( "\n" + MEMORY_NOTE.format(path=memory.relative_to(work).as_posix()) if memory else "") (log / "prompt.txt").write_text(prompt, encoding="utf-8") (log / "deployment.json").write_text(json.dumps(deployed, indent=1, default=str)) from hermit.adapter.session import Session served = Session(adapter) if deployed.get("harness_tools") else None if budget is not None: # what is left once the deployment, the files and the server are ready - timeout_s = min(float(params.get("timeout_s", 3600)), budget - (time.time() - started)) + timeout_s = min(float(params.get("timeout_s") or default_s), deadline.remaining()) if timeout_s <= 0: if served is not None: served.close() raise RuntimeError(f"agent {backend} could not be served within deadline_s={budget:.0f}") + if remote: + own = given.limits.timeout_s + place = given.with_limits(dataclasses.replace(given.limits, timeout_s=min(timeout_s, float(own)) + if own else timeout_s)) # its other limits kept + else: + place = environment(work, params, Limits(readable=mounts, timeout_s=timeout_s)) try: - result = session.run(prompt, environment=environment(work, params, Limits(readable=mounts, - timeout_s=timeout_s)), - tools=served.as_tools() if served else ()) + result = session.run(prompt, environment=place, tools=served.as_tools() if served else ()) finally: if served is not None: served.close() + usages.add(result.usage) + if remote and not result.usage and result.extra.get("started"): + # a session that reached its CLI's events but reported no usage (cut off before turn.completed) may have + # been billed: its cost is not known (the runner's usage None), never 0 + usages.unreported(0, str(params.get("model") or backend), sessions=1) if result.extra.get("login_failed"): # before a timeout: a session that then hung is no attempt raise LoginFailed(f"agent {backend}: {result.extra.get('failure')}") + if remote: + # what the Harbor runner read of a session beyond its own result (codex_stream.session_verdict): the + # account's bill refused (nothing later can be served: the run ends), the configuration refused (never + # run again); a refusal of what the input made the request, and the model's own, are the agent's result + from hermit.agent.base.codex_stream import session_verdict + spent = sum(u.input_tokens + u.output_tokens for u in (result.usage or [])) + said = session_verdict(result.extra, spent) + if said == "usage_limit": + raise LoginFailed(f"agent {backend}: usage_limit: the provider does not bill the account (its quota " + f"or credit spent, or a payment it needs): {result.extra.get('failure')}") + if said == MODEL_REJECTED: + return _infra_run(MODEL_REJECTED, f"agent {backend}: the provider refused the run's configuration: " + f"{result.extra.get('failure')}", usages, log, started, final=True) + if result.extra.get("refused") and not result.extra["served"] and not result.timed_out: + if not LOGINS.get(params.get("login")): + # no plan of the run's to wait for (the user's own sign-in): nothing later can be served + raise LoginFailed(f"agent {backend}: usage_limit: the account refused the session for its usage " + f"limit and the run holds no plan to wait for: {result.extra.get('failure')}") + # the login's plan refused it (its window): run again once the plan allows, never counted + out = _infra_run("usage_limit", f"agent {backend}: the plan refused the session: " + f"{result.extra.get('failure')}", usages, log, started, final=False) + if result.extra.get("hold_unwritten"): # no hold to wait for: the run pauses (the runner's 5 min) + out.extra["hold_unwritten"] = True + return out if result.extra["served"] or result.timed_out: break if not result.extra["unavailable"]: if result.exit_code and not result.extra["started"]: # The CLI exited before it wrote one event: the run never started, which is no wrong answer. - raise RuntimeError(f"agent {backend} exited {result.exit_code} before starting a session: " - f"{result.extra['output_tail']}") + said = f"agent {backend} exited {result.exit_code} before starting a session: " \ + f"{result.extra['output_tail']}" + if remote: + return _infra_run("session_failed", said, usages, log, started, final=False) + raise RuntimeError(said) break + if remote: # the run runs it again in a fresh environment (busy_retries) + return _infra_run("model_provider_failed", f"agent {backend}: the provider could not serve the session: " + f"{result.extra['output_tail']}", usages, log, started, final=False) if attempt == retries: raise RuntimeError(f"agent {backend} could not be served after {retries} attempts: " f"{result.extra['output_tail']}") wait = float(params.get("retry_wait_s", 120)) - time.sleep(wait if budget is None else max(0.0, min(wait, budget - (time.time() - started) - 1))) - code, timed_out, last, usage = result.exit_code, result.timed_out, result.reply, result.usage + time.sleep(wait if deadline.at is None else max(0.0, min(wait, deadline.remaining() - 1))) + code, timed_out, last, usage = result.exit_code, result.timed_out, result.reply, usages (log / "last_message.txt").write_text(last, encoding="utf-8") - if memory is not None and memory.parent.is_dir(): # the notes the session left + if memory is not None and remote: # the notes the session left, from there + try: + given.download(memory.parent.relative_to(work).as_posix(), log / "memory") + except Exception: # noqa: BLE001 - none left + pass + elif memory is not None and memory.parent.is_dir(): # the notes the session left shutil.copytree(memory.parent, log / "memory", dirs_exist_ok=True) + hooks_missing = None + if remote and deployed.get("hooks"): + try: # an audit that cannot be read leaves the session as it ended + _collect_hooks(given, log) + except Exception as exc: # noqa: BLE001 - recorded, the verifier decides the rest + hooks_missing = f"{type(exc).__name__}: {exc}"[-500:] # A session the CLI itself reports as failed has no answer to read or to judge (trainer.rollout: an error record). failed = result.extra.get("failed") output = None if failed else _output(last, task, params, base, code, timed_out, timeout_s) facts = None - fn = _task_function(params.get("collect"), base) + fn = None if remote else _task_function(params.get("collect"), base) if fn is not None: facts = _call(fn, task, work, log, params=params, base=base) _clean(work, [m.target for m in mounts]) - cost = tokens = None - if usage: # every model of the session (Hermes's auxiliary calls are a usage of their own) - cost = None if any(u.cost_usd is None for u in usage) else sum(u.cost_usd for u in usage) - tokens = sum(u.input_tokens + u.output_tokens for u in usage) + # every model of the session (Hermes's auxiliary calls are a usage of their own); none reported: not known + cost, tokens = usage.cost(None), usage.tokens(None) extra = {"backend": backend, "model": params.get("model"), "effort": params.get("effort"), "web_search": params["web_search"], - "deploy": deployed.get("mode"), "isolation": params.get("isolation") or "none", + "deploy": deployed.get("mode"), "isolation": "remote" if remote else params.get("isolation") or "none", "deployed": {k: v for k, v in deployed.items() if k not in ("mode", "files")}, - "exit_code": code, "timed_out": timed_out, "attempts": attempt, "workspace": str(work), + "exit_code": code, "timed_out": timed_out, "attempts": attempt, + "workspace": str(given.workspace) if remote else str(work), "mounts": result.extra.get("mounts") or {}} # a task's data in the workspace: bound or linked if failed: extra["session_failed"] = str(result.extra.get("failure") or f"the CLI exited {code} without a result") + if hooks_missing: + extra["hooks_audit_missing"] = hooks_missing if context_tool is not None: extra["context_tool"] = context_tool if facts: extra["task"] = facts return RunOutput(output=output, log=log, cost_usd=cost, wall_s=round(time.time() - started, 1), tokens=tokens, - api_speed={"model_calls": sum(u.calls for u in usage)} if usage else None, extra=extra) + api_speed={"model_calls": usage.calls()} if not usage.empty else None, extra=extra) diff --git a/hermit/agent/base/cli.py b/hermit/agent/base/cli.py index 011e83d8..f0432011 100644 --- a/hermit/agent/base/cli.py +++ b/hermit/agent/base/cli.py @@ -42,7 +42,10 @@ timeout_s the session's wall clock (default 3600); Limits.timeout_s overrides it cancel a threading.Event: the session is stopped once it is set tools_name the MCP server's name for the call's tools (default hermit_tools) - bin the CLI's program (default codex / claude) + bin the CLI's program (default codex / claude); in a remote environment, its path there + version in a remote environment without `bin`: the version the environment installs (default: its + installer's) + remote_home in a remote environment: the session's home there (default /installed-agent/hermit-home) cwd Codex: the working directory it is told, when the isolation shows the workspace under another path (MWM's /w) tools_env a variable that gets the tools' JSON entrance (GET /tools, POST /call/) @@ -96,26 +99,39 @@ hermit's plugin turns off; default true). Hermes takes no roles, has no web search, takes a prompt of one argument (ARG_MAX bytes) at most and runs under isolation none or sandbox. -The isolation is the call's environment's (hermit.agent.environment), never a setting: none (Local: the CLI on -this machine), sandbox (Sandboxed: inside the run's hermit.sandbox.Sandbox) or container (Container: a Docker container -of its own, its settings image, network, proxy: hermit.agent.environment.docker, where the CLI's own sandbox holds the -session's commands: Codex's `codex_sandbox`, Claude Code's CLAUDE_SANDBOX). Limits are the environment's, enforced by -its isolation, or the call is refused (Unsupported): +The isolation is the call's environment's (hermit.environment), never a setting: none (Local: the CLI on +this machine), sandbox (Sandboxed: inside the run's hermit.sandbox.Sandbox), container (Container: a Docker container +of its own, its settings image, network, proxy: hermit.environment.docker, where the CLI's own sandbox holds the +session's commands: Codex's `codex_sandbox`, Claude Code's CLAUDE_SANDBOX) or remote (an environment that runs the +CLI where it runs its commands, through its `exec`: a task's Harbor environment, hermit.environment.harbor). In a +remote environment (Codex and Claude Code) the +session's home is rendered here and uploaded to `remote_home` there, the CLI is the environment's (`bin`, else +installed by the environment's own installer when it has none), the command runs from the workspace with the prompt +on its stdin where the CLI reads it there, and the stream is its output. A secret the CLI reads from a descriptor +(Claude Code's token on `login: claude`) reaches the environment's exec by name and is moved there into a pipe before +the CLI starts, as in a container of its own. Codex's own sandbox is off there unless `codex_sandbox` says otherwise +(the environment is the boundary, as Harbor's own agents run), and Claude Code runs as the environment's root may +(IS_SANDBOX), with no update mid-session. The call's tools, readable mounts, hidden paths and a network limit are +refused (the environment is the task's own: it says what it reaches). +Limits are the environment's, enforced by its isolation, or the call is refused (Unsupported): isolation readable hidden network none linked into the workspace refused refused (any list) sandbox shown at its target (a bind) refused when the sandbox shows it the sandbox's own: [] only container bound read-only at their target all but workspace, home, data the container's proxy: [] only + remote refused refused refused (the task's own) """ from __future__ import annotations import contextlib import copy import dataclasses +import inspect import json import os from pathlib import Path import re +import shlex import shutil import signal import stat @@ -128,8 +144,9 @@ import urllib.parse from hermit.agent.base import login as codex_login -from hermit.agent.environment import docker as container -from hermit.agent.environment.base import Environment, _descendants, as_environment +from hermit.agent.base.credentials import held +from hermit.environment import docker as container +from hermit.environment.base import Environment, ExecResult, _descendants, as_environment from hermit.agent.base.interface import BaseAgent, Limits, Mount, Result, Unsupported from hermit.agent.base.records import (parse_claude, parse_codex, parse_dsh, parse_hermes, parse_mini_swe_agent, parse_opencode, parse_pi, price_usage) @@ -138,8 +155,15 @@ mini_swe_agent_steps, opencode_steps, pi_steps) from hermit.agent.model.backends import CLAUDE_ENV -ISOLATIONS = ("none", "sandbox", "container") -ENVIRONMENT_KEYS = ("isolation", "sandbox", "container") # the environment's (hermit.agent.environment), not settings +ISOLATIONS = ("none", "sandbox", "container", "remote") +REMOTE_HOME = "/installed-agent/hermit-home" # a session's home in a remote environment (a task's Harbor one) +# The installed agent of Harbor's that installs each CLI in a remote environment that installs CLIs (its `install`) +REMOTE_INSTALL = {"codex": "codex", "claude": "claude-code"} +# Where those installs put the CLI, put on PATH first in a remote session as the Harbor runner's launch did: nvm's +# (Harbor's Codex on a glibc image, loaded with the user's own HOME) and ~/.local/bin (Harbor's Claude Code) +REMOTE_PATH = ('export NVM_DIR="$HOME/.nvm"; if [ -s "$NVM_DIR/nvm.sh" ]; then . "$NVM_DIR/nvm.sh" >/dev/null 2>&1; ' + 'fi; export PATH="$HOME/.local/bin:$PATH"\n') +ENVIRONMENT_KEYS = ("isolation", "sandbox", "container") # the environment's (hermit.environment), not settings # The CLI's own reports of a provider it could not reach (it retries those itself first), and of a container that # could not start, not the session's text UNAVAILABLE = re.compile(r"unexpected status (?:429|5\d\d)|exceeded retry limit|stream disconnected|" @@ -196,6 +220,9 @@ CLAUDE_DOWN = re.compile(r"not your usage limit|api error: (?:429|5\d\d)|request timed out|connection (?:error|refused|" r"to the api was lost)|no response from api|stream idle timeout|high load|overloaded", re.I) FINISH = {"max_tokens": "length", "pause_turn": "length", "refusal": "content_filter"} +# In a remote environment (a task's own, whose container is the boundary): bypassPermissions as its root, and no update +# of the CLI mid-session +CLAUDE_REMOTE_ENV = {"IS_SANDBOX": "1", "DISABLE_AUTOUPDATER": "1"} _ready: set = set() @@ -520,7 +547,14 @@ def _prompt_file(self, flag: str, text: str, home: Path) -> list[str]: one place every isolation shows it.""" if len(text.encode()) <= ARG_MAX: return [flag, text] - path = home / f"{flag.lstrip('-')}.md" # system-prompt.md, append-system-prompt.md: one file each + name = f"{flag.lstrip('-')}.md" # system-prompt.md, append-system-prompt.md: one file each + remote = getattr(self, "_remote_home", None) # (this machine's home, its path in a remote environment) + if remote is not None and str(home) == str(remote[1]): + path = Path(remote[0]) / name # written here, uploaded with the home + _clear(path) + path.write_text(text, encoding="utf-8") + return [f"{flag}-file", str(remote[1] / name)] + path = home / name _clear(path) # a link the last session left would take the text elsewhere path.write_text(text, encoding="utf-8") return [f"{flag}-file", str(path)] @@ -603,6 +637,11 @@ def _isolation(self, workspace: Path, limits: Limits): why = "container runs Codex" if isolation == "container" else f"{isolation} is not for {self.label}" raise Unsupported(f"isolation: {why}; {self.label} takes isolation {' or '.join(self.isolations)}") network = None if limits.network is None else list(limits.network) + if isolation == "remote": + if limits.readable or limits.hidden or network is not None: + raise Unsupported(f"{self.name} in a remote environment takes no readable mounts, hidden paths or " + "network limits: the environment is the task's own") + return isolation, None, None if isolation == "none": if limits.hidden: raise Unsupported(f"{self.name} without isolation cannot hide paths: use isolation sandbox or container") @@ -711,6 +750,8 @@ def _sign_in(self, home: Path) -> list[Path]: return [home / codex_login.FILE] if s.get("login") or s.get("binding"): # login: claude signs in with the token in the environment return [] + if self.environment is not None and self.environment.isolation == "remote": + return [] # never this machine's own sign-in carried to a task's container copied = [] for name in self.home_files if s.get("home_files") is None else s["home_files"]: source = self.user_home() / name @@ -740,7 +781,7 @@ def run(self, prompt, *, environment: Environment | Path | None = None, tools: S given = sorted(k for k in ENVIRONMENT_KEYS if k in self.settings) if given: # never ignored: a session that took them for settings would run without its isolation raise ValueError(f"{self.name}: {', '.join(given)} belong to the call's environment " - "(hermit.agent.environment: Local, Sandboxed, Container), not to the agent's settings") + "(hermit.environment: Local, Sandboxed, Container), not to the agent's settings") self = copy.copy(self) # this call's agent: it holds the call's environment self.environment = env s = self.settings @@ -756,11 +797,14 @@ def run(self, prompt, *, environment: Environment | Path | None = None, tools: S if login == "claude" and not os.environ.get(codex_login.TOKEN_ENV): return self._not_run("login", None) plan, cancel = codex_login.plan_dir(s.get("plan_dir"), login), s.get("cancel") + if getattr(env, "plan_admitted", False): + return self._admitted_session(prompt, env, plan, tools=tools, session=session, roles=roles) max_s = s.get("plan_max_wait_s", codex_login.MAX_WAIT_S) deadline = None if max_s is None else time.time() + float(max_s) # over every wait of this session if s.get("deadline") is not None: deadline = float(s["deadline"]) if deadline is None else min(deadline, float(s["deadline"])) say = lambda m: print(f"hermit {self.name}: {m}", file=sys.stderr, flush=True) + spent: list = [] # the usage of sessions the plan refused, which a run again adds to while True: credits = codex_login.opted_in(s.get("usage_credits")) waited = codex_login.wait(plan, cancel, say, None if deadline is None else max(0.0, deadline - time.time()), @@ -794,34 +838,50 @@ def run(self, prompt, *, environment: Environment | Path | None = None, tools: S if not done: # never ran: no answer; every session waits for the window (the refusal's reset) codex_login.hold(plan, until=result.extra.get("held_until") or None, cancel=cancel, max_s=None if deadline is None else max(0.0, deadline - time.time())) - if done: - return result - if str(s.get("plan_rerun", True)).strip().lower() in ("false", "0", "no", "off"): - say("a session was refused for the plan's usage limit; it is returned (plan_rerun: false)") - return result + if done or str(s.get("plan_rerun", True)).strip().lower() in ("false", "0", "no", "off"): + if not done: + say("a session was refused for the plan's usage limit; it is returned (plan_rerun: false)") + return dataclasses.replace(result, usage=[*spent, *(result.usage or [])]) if spent else result + spent += list(result.usage or []) say("a session was refused for the plan's usage limit; it runs again once the plan allows") + def _admitted_session(self, prompt, env: Environment, plan, *, tools, session, roles) -> Result: + """A session on a login whose plan the run already admitted (it waited and took a slot around the task's + environment, as the Harbor runner held a trial's: `plan_admitted`): no wait or slot of its own. What it says of + the plan is noted; one the plan refused holds the plan for the window and is returned (the run runs the task + again once the plan allows), never waited for inside the environment.""" + result = self._session(prompt, environment=env, tools=tools, session=session, roles=roles) + report = result.extra.pop("plan_report", None) + if report is not None: + codex_login.note_usage(plan, None, report) + if report["using_credits"] and not codex_login.opted_in(self.settings.get("usage_credits")): + # as run()'s: a session that drew the account's usage credits holds the sessions to come + codex_login.hold(plan, report.get("credits_resets_at") or report["resets_at"], "usage_credits", + cancel=self.settings.get("cancel"), max_s=0.0) + refused = not result.extra.get("served") and result.extra.get("refused") and \ + not result.extra.get("login_failed") + if refused and codex_login.hold(plan, until=result.extra.get("held_until") or None, + cancel=self.settings.get("cancel"), max_s=0.0) is False: + result.extra["hold_unwritten"] = True # the run waits HOLD_UNWRITTEN_S itself, outside the env + return result + def _within(self, limits: Limits | None): """`limits` with its timeout cut to what is left before the `deadline` setting (as given without one); False when nothing is left.""" - if self.settings.get("deadline") is None: + from hermit.agent.base.deadline import Deadline + deadline = Deadline(self.settings.get("deadline")) + if deadline.at is None: return limits - left = float(self.settings["deadline"]) - time.time() - if left <= 0: - return False limits = limits or Limits() own = limits.timeout_s or self.settings.get("timeout_s", 3600) # as the session's own default - return dataclasses.replace(limits, timeout_s=min(left, float(own)) if own else left) + left = deadline.cut(own) + return False if left is False else dataclasses.replace(limits, timeout_s=left) def _left(self, timeout: float | None): """`timeout` cut to what the preparation left before the `deadline` setting, just before the CLI starts; False when nothing is.""" - if self.settings.get("deadline") is None: - return timeout - left = float(self.settings["deadline"]) - time.time() - if left <= 0: - return False - return min(timeout, left) if timeout else left + from hermit.agent.base.deadline import Deadline + return Deadline(self.settings.get("deadline")).cut(timeout) def _not_run(self, why: str, max_s) -> Result: """A session that never started: its caller stopped it, the caller's deadline passed, the plan held it back @@ -839,6 +899,8 @@ def _not_run(self, why: str, max_s) -> Result: def _session(self, prompt, *, environment: Environment, tools: Sequence[Tool] = (), session: Any = None, roles: Sequence[dict] = ()) -> Result: + if environment.isolation == "remote": + return self._remote_session(prompt, environment=environment, tools=tools, session=session, roles=roles) workspace, limits = environment.workspace, environment.limits if not isinstance(prompt, str): raise Unsupported(f"{self.name} takes a prompt, not chat messages") @@ -1021,6 +1083,205 @@ def _session(self, prompt, *, environment: Environment, tools: Sequence[Tool] = "cancelled": cancelled, "isolation": isolation, "wall_s": round(wall_s, 1), "mounts": placed, "output_tail": raw[-2000:]}) + def _is_file(self, path) -> bool: + """Whether `path` is a file where the session runs: in a remote environment, there.""" + env = self.environment + if env is not None and env.isolation == "remote": + return env.exec(["test", "-f", str(path)], timeout=60).code == 0 + return Path(path).is_file() + + def _remote_program(self, environment: Environment) -> None: + """The CLI in a remote environment: `bin` as it is, else the program found there (on REMOTE_PATH's PATH; with + `version` set, the installer is asked anyway: Harbor's keeps a CLI already at that version), else installed by + the environment's own installer (Harbor's installed agent of the CLI).""" + s = self.settings + if s.get("bin"): + served = getattr(environment, "bin_served", None) or {} + if served.get(REMOTE_INSTALL.get(self.backend)) is not False: + return + # the environment's setup found the mounted binary unusable there and installed the CLI instead + s = self.settings = {**s, "bin": None} + if not s.get("version"): + found = environment.exec(["bash", "-c", REMOTE_PATH + f"command -v {self.program}"], timeout=60) + if found.code == 0 and found.output.strip(): + return + installer = REMOTE_INSTALL.get(self.backend) + if installer is None or not hasattr(environment, "install"): + raise Unsupported(f"{self.label} is not in the environment and the environment installs none: give its " + "path there as `bin`") + environment.install(installer, s.get("version")) + + def _remote_session(self, prompt, *, environment: Environment, tools: Sequence[Tool] = (), session: Any = None, + roles: Sequence[dict] = ()) -> Result: + """One session where a remote environment runs its commands (the module docstring): the home rendered here + and uploaded, the command run through the environment's exec, its output the stream.""" + from pathlib import PurePosixPath + s, limits = self.settings, environment.limits or Limits() + if not isinstance(prompt, str): + raise Unsupported(f"{self.name} takes a prompt, not chat messages") + if not isinstance(s.get("web_search"), bool): + raise ValueError(f"{self.name}: set web_search, true or false") + if s.get("text_only") and (tools or roles): + raise Unsupported(f"{self.name}: a session with text_only has no tools or roles") + if tools: + raise Unsupported(f"{self.name} in a remote environment cannot reach the call's tools (the framework's " + "tool server runs on this machine)") + if session: + raise Unsupported(f"{self.name} in a remote environment does not resume a session: its home there is " + "made anew for each call") + self._isolation(PurePosixPath(str(environment.workspace)), limits) + # As the Harbor runner's launch plan ran a session in a task's container: the prompt on stdin, whatever its + # length (Codex reads it as `-`); the CLI's web search off where the task's agent phase has no public network + overrides = {"stdin": True} + if getattr(environment, "network_public", True) is False: + overrides["web_search"] = False + self.settings = s = {**s, **overrides} + prompt = self.prompt_text(prompt) + workspace = PurePosixPath(str(environment.workspace)) + remote_home = PurePosixPath(str(s.get("remote_home") or REMOTE_HOME)) + home, fresh = self._home() + if s.get("log"): + stream = Path(s["log"]) + elif fresh: + fd, name = tempfile.mkstemp(prefix=f"agent-{self.backend}-", suffix=".jsonl") + os.close(fd) # only its name: the stream is written below + stream = Path(name) + else: + stream = home / f"agent-{self.backend}.jsonl" + _session_output(stream, (home,)) + stream.parent.mkdir(parents=True, exist_ok=True) + last_file = home / "last_message.txt" + _clear(last_file) + last_file.unlink(missing_ok=True) + timeout = limits.timeout_s or s.get("timeout_s", 3600) + timeout = float(timeout) if timeout else None + self._remote_home = (home, remote_home) + code, timed_out, raw, wall_s = -1, False, "", 0.0 + chatgpt = s.get("login") == "chatgpt" + + def take_back(copied): + # the refreshed sign-in and the plan's usage reports back here, however the session's exec ended (a + # session that rotated its token before its exec failed), before the sign-in leaves the environment + if chatgpt and copied: + with contextlib.suppress(Exception): + self._signin_back(environment, remote_home, home) + with contextlib.suppress(Exception): + environment.download(str(remote_home / "sessions"), home / "sessions") + try: + if chatgpt: + codex_login.keep_refreshed(Path(s.get("login_root") or codex_login.ROOT), home) + codex_login.note_usage(codex_login.plan_dir(s.get("plan_dir")), home) + except OSError as exc: # this machine's (the shared copy's folder full): the session stands, its + print(f"WARNING: the sign-in or plan report the session refreshed could not be kept ({exc}); a " + "later session refreshes again", file=sys.stderr, flush=True) # refresh lost (the runner's) + + def remove(copied): + # every sign-in file the home took (the login's, or the user's own CLI home's) out of the environment, + # from a directory the session cannot have removed, before its verifier runs, and the local copies; a + # removal there not confirmed fails the call (no verifier then reads them) + left = [str(remote_home / path.name) for path in copied] + gone = None + try: + if left: + try: + gone = environment.exec(["rm", "-f", *left], cwd="/", timeout=120) + except Exception as exc: # noqa: BLE001 - reported below + gone = ExecResult(-1, f"{type(exc).__name__}: {exc}") + finally: # the local copies go however the take-back went + for path in copied: + path.unlink(missing_ok=True) + if gone is not None and gone.code != 0: + raise RuntimeError(f"{self.name}: the sign-in could not be removed from the environment: " + f"{gone.output[-300:]}") + + try: + with held(lambda: self._sign_in(home), take_back, remove) as copied: + env = {k: os.path.expandvars(str(v)) for k, v in (s.get("env") or {}).items()} # $KEY_VAR -> the key + env.update(self.session_env()) + self.prepare_home(home, roles, None, s.get("tools_name") or "hermit_tools") + env.update(self.home_env(remote_home)) + key_env = self._key_env() + # by name to the environment's exec, never in a file; Claude Code reads its key from its own variable + # (session_env), as the runner's launch left the binding's out for it + if key_env and self.backend != "claude": + env[key_env] = os.environ.get(key_env, "") + self._remote_program(environment) # the CLI there (the mounted one, or the installed one) + command = self.command(prompt, workspace, remote_home / "last_message.txt", session=session, + roles=roles, tools_url=None) + by_fd = self.by_fd() + missing = set(by_fd) - set(env) + if missing: + raise ValueError(f"by_fd names variables the session does not pass: {sorted(missing)}") + # the installs' places on PATH (nvm's, ~/.local/bin); each by_fd secret the environment moves into a + # pipe before anything that outlives the move starts (its exec's by_fd) + command = ["bash", "-c", REMOTE_PATH + 'exec "$@"', "bash", *command] + environment.exec(["rm", "-rf", str(remote_home)], cwd="/", timeout=120) + register = getattr(environment, "secret", None) # removed at the phase's end, whatever ends it + if register is not None and copied: + register([str(remote_home / path.name) for path in copied]) + environment.upload(home, str(remote_home)) + timeout = self._left(timeout) + if timeout is False: + return self._not_run("deadline", None) + started = time.time() + accepts = inspect.signature(environment.exec).parameters + kept = {"stream": stream.name} if "stream" in accepts else {} + if "secret_env" in accepts: # the credentials the session is given: scrubbed by name + kept["secret_env"] = [n for n in (key_env, *by_fd) if n] + done = environment.exec(command, cwd=str(workspace), env=env, timeout=timeout, + stdin=prompt if self.on_stdin() else None, + **({"by_fd": by_fd} if by_fd else {}), **kept) + wall_s = time.time() - started + code, timed_out, raw = done.code, done.timed_out, done.output + stream.write_text(raw, encoding="utf-8") + with contextlib.suppress(Exception): # a session that wrote none: the stream's last message + environment.download(str(remote_home / "last_message.txt"), last_file) + finally: + self._remote_home = None + said = self.outcome(raw, stream, last_file) + said["failed"] = said.get("failed", False) and not timed_out + began = said.pop("started", None) + handle, usage = self.parse(stream, (s.get("binding") or {}).get("ref") or s.get("model") or self.backend) + usage = self.adjust_usage(home, handle, usage, resumed=bool(session)) + price_usage(usage, s) + steps = self.steps(events(stream)) + if fresh: + shutil.rmtree(home, ignore_errors=True) + return Result(reply=said.pop("reply"), finish_reason=said.pop("finish_reason", None), trajectory=steps, + usage=usage, session=handle, exit_code=code, timed_out=timed_out, log=stream, + extra={"failed": False, "login_failed": False, "rejected": False, **said, + "started": any(line.lstrip().startswith("{") for line in raw.splitlines()) + if began is None else bool(began), + "cancelled": False, "isolation": "remote", "wall_s": round(wall_s, 1), + "mounts": {}, "output_tail": raw[-2000:]}) + + @staticmethod + def _signin_back(environment, remote_home, home: Path) -> None: + """The ChatGPT sign-in the session left in the environment, as the Harbor runner took it back (signin. + take_back): downloaded into a private folder, then the local copy only when it is a regular file of at most + SIGNIN_MAX_BYTES holding a ChatGPT sign-in stamped at most SKEW (a local Docker environment, sharing this + machine's clock) or OTHER_CLOCK past this machine's clock.""" + from hermit.agent.base import codex_stream + skew = codex_stream.SKEW if getattr(environment, "same_clock", True) else codex_stream.OTHER_CLOCK + there = shlex.quote(str(remote_home / codex_login.FILE)) + # a regular file of at most SIGNIN_MAX_BYTES there, measured there (the runner's _return_files): nothing larger + # or of another kind is brought here + size = environment.exec(["sh", "-c", f"[ -f {there} ] && [ ! -L {there} ] && wc -c < {there}"], timeout=60) + words = size.output.split() + if size.code != 0 or not words or not words[-1].isdigit() or int(words[-1]) > codex_stream.SIGNIN_MAX_BYTES: + return + with tempfile.TemporaryDirectory(prefix="hermit-signin-") as private: + got = Path(private) / codex_login.FILE + environment.download(str(remote_home / codex_login.FILE), got) + data = codex_stream.regular_bytes(got, codex_stream.SIGNIN_MAX_BYTES) + if data is None or not codex_stream.signed_in(data, skew): + return + local = home / codex_login.FILE + fd = os.open(f"{local}.taken", os.O_WRONLY | os.O_CREAT | os.O_TRUNC | os.O_NOFOLLOW, 0o600) + with os.fdopen(fd, "wb") as f: + f.write(data) + os.replace(f"{local}.taken", local) + def on_stdin(self) -> bool: """Whether the prompt is given on stdin, not on the command line.""" return bool(self.settings.get("stdin")) @@ -1092,10 +1353,12 @@ def adjust_usage(self, home, handle, usage, resumed): def command(self, prompt, workspace, last_file, *, session=None, roles=(), tools_url=None): s = self.settings effort = s.get("effort") - hooks = (workspace / ".codex" / "hooks.json").is_file() # the deployment's hooks, trusted as deployed + hooks = self._is_file(workspace / ".codex" / "hooks.json") # the deployment's hooks, trusted as deployed + remote = self.environment is not None and self.environment.isolation == "remote" return [str(s.get("bin") or self.program), "exec", "--json", "--skip-git-repo-check", "-C", str(s.get("cwd") or workspace), - "-s", s.get("codex_sandbox") or "workspace-write", "-o", str(last_file), + "-s", s.get("codex_sandbox") or ("danger-full-access" if remote else "workspace-write"), + "-o", str(last_file), *(["-m", s["model"]] if s.get("model") else []), *(["-c", f'model_reasoning_effort="{effort}"'] if effort else []), *(["--dangerously-bypass-hook-trust"] if hooks else []), @@ -1104,9 +1367,20 @@ def command(self, prompt, workspace, last_file, *, session=None, roles=(), tools def outcome(self, raw, stream, last_file): notes, done = self.notices(raw), self.served(raw) - return {"reply": self.last_message(stream, last_file), "served": done, + said = {"reply": self.last_message(stream, last_file), "served": done, "unavailable": bool(UNAVAILABLE.search(notes)), "refused": codex_login.refused(notes) and not done, "failure": notes[-600:]} + if not done and self.environment is not None and self.environment.isolation == "remote": + # a session in a task's container, read as the Harbor runner read one (hermit.agent.base.codex_stream): + # from Codex's own notices only, a lost sign-in or key, the provider, the plan, a refused configuration + from hermit.agent.base import codex_stream + rejected = codex_stream.codex_rejected(raw) + said.update(login_failed=codex_stream.codex_signin_lost(raw), + unavailable=codex_stream.codex_unavailable(raw), refused=codex_stream.codex_refused(raw), + rejected=bool(rejected)) + if rejected: # what session_verdict reads: Codex's own failure + said["failure"] = rejected[:600] # what refused it, its status first (the runner's head) + return said class Claude(CliAgent): @@ -1181,6 +1455,8 @@ def prompt_text(self, prompt): def session_env(self): s, env = self.settings, {} + if self.environment is not None and self.environment.isolation == "remote": + env.update(CLAUDE_REMOTE_ENV) if s.get("text_only"): env.update(CLAUDE_TEXT_ENV) if s.get("login") == "claude": diff --git a/hermit/harbor/signin.py b/hermit/agent/base/codex_stream.py similarity index 69% rename from hermit/harbor/signin.py rename to hermit/agent/base/codex_stream.py index 9a9e43bb..ea2c7748 100644 --- a/hermit/harbor/signin.py +++ b/hermit/agent/base/codex_stream.py @@ -1,49 +1,15 @@ -"""What Hermit's side reads of a sign-in in a Harbor trial, beside what the CLI's Base Agent reads of the session. - -The Base Agents (hermit.agent.base.cli, .login) are used as they are: a trial's stream is read with the CLI's own -`outcome`, and a refreshed ChatGPT sign-in goes back to the shared copy through `login.take_back`. Three things a -trial in a task's container needs beyond them are here: - -- `codex_signin_lost(raw)`: whether a trial's Codex stream says that its sign-in or its key was refused, which no wait - restores. Codex's `outcome` reports no lost sign-in (a session on this machine that met one fails on its own); a - Harbor trial that met one would leave every later trial on the same sign-in failing the same way, so the runner ends - the run (hermit.agent.base.interface.LoginFailed, which hermit.trainer.rollout passes up: `hermit train` resumes - after the last finished step, `hermit eval` stops). Codex's own lines are told from what its log quotes of the - session's work (`lines`), and only the notices since the session last worked count (a side request's 401 that the - session worked after is no lost sign-in). -- `codex_unavailable(raw)`: whether a Codex session that served no turn failed on its provider (an outage, a rate - limit, the network). Codex's `outcome` takes any notice of its stream for one, a reconnect Codex recovered from too: - a trial that worked on after one and then failed on its own (its context full, say) would be run again and left - out of the scored trials, though it is the agent's result. Here only the session's last failure counts, or a retry - Codex was still making when the turn failed with the error it retried, or when the stream ended; a response the - provider's content filter ended is the agent's result, never an outage. -- `codex_refused(raw)` and `codex_rejected(raw)`: the plan's usage limit (the Base Agent's words, login.LIMIT) and a - refusal of the run's configuration or bill (CODEX_REJECTED: a 400, 402 or 404 Codex does not retry, an API - account's quota), each read from Codex's own notices only (`lines`), never from what its log quotes of the session's - work, which the Base Agent's reading takes for its notices too. -- `take_back(root, returned, local=...)`: the sign-in a trial's container handed back becomes the shared copy only - when it is a regular file of at most MAX_BYTES, a ChatGPT sign-in (its refresh and access tokens), stamped at most - SKEW past this machine's clock in Harbor's local Docker environment (whose containers share it; OTHER_CLOCK in any - other): read once, never through a link, into a private copy that `login.take_back` then takes (which keeps the - copy's account and takes only a later refresh). A session that can read its sign-in can also write a well-formed - one, which no check can tell from a refresh. +"""What Hermit reads of a CLI session's stream beyond its Base Agent's own `outcome`, for a session in a remote +environment (a task's Harbor environment): for Codex, a lost sign-in or key, a provider that could not serve, the +plan's usage limit and a refusal of the run's configuration, each read from Codex's own notices only, never from what +its log quotes of the session's work; for every CLI, whether a refusal was of the run's configuration or of the +account's bill (`session_verdict`). Ported unchanged from Hermit's Harbor runner (hermit/harbor/signin.py's readers +and the runner's CONFIG_REFUSED, BILLING_REFUSED, QUOTA_SAID and session_verdict), which applied them to every session +in a task's container. """ from __future__ import annotations -from datetime import datetime, timedelta, timezone import json -import os -from pathlib import Path import re -import shutil -import stat - -FILE = "auth.json" # hermit.agent.base.login.FILE: Codex's sign-in file -MAX_BYTES = 64 << 10 # a sign-in larger than this is none, never read whole (Codex's auth.json is a few KB: two - # tokens of about 2 KB each, a refresh token, the account and a time) -SKEW = timedelta(minutes=1) # a refresh stamped on this machine's clock (a local Docker container shares it) -OTHER_CLOCK = timedelta(hours=1) # ... on another machine's (a cloud sandbox): the hour login.take_back allows -TAKEN_DIR = "taken" # under a job's launch/: the private copy login.take_back reads # Codex's report of a sign-in or a key the provider refused, which no wait restores (the wording of Harbor's # AgentAuthenticationError, harbor/agents/installed/base.py ERROR_PATTERNS, and Codex 0.164's own for no sign-in at all, @@ -243,9 +209,76 @@ def codex_rejected(raw: str) -> str | None: return last if CODEX_REJECTED.search(last) else None -def _regular_bytes(path: Path, limit: int) -> bytes | None: +# A rejection that refused the run's configuration, which no input to the session changes, as the CLI's own notices +# (the session's `failure`) name it: its key or token (401, unauthorized, a failed authentication, an invalid key or +# token, a key not valid, not logged in, OpenCode's provider.auth), its permission (403, forbidden, no access), its +# model or endpoint (404, not found, a model that does not or may not exist, an invalid or unknown model, one that is +# not a valid model), its account's bill (402, a credit balance too low, an insufficient quota, Codex's "Quota exceeded. +# Check your plan and billing details.", billing; not a rate limit's advice, "billing details" or a link to the billing +# page). A status is read only where one is written: leading +# the text or a line of it (Pi's "401 {...}"), after the prefix Hermit's endpoint gives a CLI's failure in its reply +# ("...(exit 1): 401 {...}", "...(timed out): "; endpoint._no_reply), after "API Error: " (Claude Code's) or +# "Error code: " (OpenAI's SDK), in "(HTTP 401)" (OpenCode's) or in "API error (401)" (Pi's), after "status" ("status +# code", "status_code") or a JSON "code"; never another number of the text (a token count: "you requested 4402 tokens +# (402 in the messages, ...)", "input tokens (402): 403 over"). So is a request setting the model does not take (an +# unsupported parameter or value: max_tokens where it takes max_completion_tokens, an effort it has not; Azure's +# unrecognized request argument; OpenCode's provider.unsupported-operation), and what Hermit's own endpoint refuses a +# CLI session cannot take (its code, tools.UNSUPPORTED: config_refused). Any other rejection (a +# prompt too long, an invalid request, a content filter) is what the adapter's or the task's input made the request: +# the agent's result. (A rate limit or an overload is `unavailable`, read before any rejection.) +CONFIG_REFUSED = re.compile(r"(?:^|\((?:exit -?\d+|timed out)\): |\bAPI error(?:: | \()|\bError code: |" + r"(?:\bstatus(?:[ _]code)?|\"code\")\"?:? ?\"?)40[1-4]\b|\(HTTP 40[1-4]\)|" + r"unauthori[sz]ed|forbidden|authenticat|\bauth\b|not logged in|" + r"invalid[\w -]{0,20}?(?:key|token)\b|\b(?:key|token) not valid|permission|access denied|" + r"not have access|not[ _]found|(?:may|does|model) not exist|(?:invalid|unknown)[ _]model|" + r"not a valid model|credit balance is too low|insufficient[_ ]?quota|" + r"^\s*Quota exceeded\. Check your plan and billing details\b|" + r"(? str | None: + """What a session that served nothing says beyond its own result, as the Harbor runner's `session_verdict` read + it from any CLI's facts (its Base Agent's outcome): "usage_limit" when the provider does not bill the account (a + refusal of the configuration that is billing's, BILLING_REFUSED, or a provider that could not serve for a quota + spent, QUOTA_SAID: nothing later can be served, the run ends), "model_rejected" when it refused the run's + configuration before serving anything (CONFIG_REFUSED, no model refusal, no tokens: never run again), else None: + a request refused for what the adapter's or the task's input made it (a prompt too long, an invalid request, a + content filter) and the model's own refusal are the agent's result.""" + if facts.get("served"): + return None + failure = str(facts.get("failure") or "") + if facts.get("unavailable") and not facts.get("refused") and QUOTA_SAID.search(failure): + return "usage_limit" + if facts.get("rejected") and not facts.get("model_refusal") and not tokens and CONFIG_REFUSED.search(failure): + return "usage_limit" if BILLING_REFUSED.search(failure) else "model_rejected" + return None + + +# ---- the sign-in a session hands back (ported from hermit/harbor/signin.py) ---------------------------------------- +SIGNIN_MAX_BYTES = 64 << 10 # a sign-in larger than this is none, never read whole (Codex's auth.json is a few KB) +SKEW = 60.0 # seconds a refresh may lie past this machine's clock: a local Docker container's +OTHER_CLOCK = 3600.0 # ... another machine's (a cloud sandbox): the hour login.take_back allows + + +def regular_bytes(path, limit: int): """At most `limit` bytes of `path` opened as a regular file at its own name, never through a link there (O_NOFOLLOW), read once; None for none there, a link, no regular file, or one larger than `limit`.""" + import os + import stat try: fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK | os.O_CLOEXEC) except OSError: @@ -264,10 +297,11 @@ def _regular_bytes(path: Path, limit: int) -> bytes | None: return data if len(data) <= limit else None -def signed_in(data: bytes, skew: timedelta) -> bool: - """Whether a sign-in file's bytes are a ChatGPT sign-in Codex refreshed at most `skew` past this machine's clock: a - JSON object whose tokens hold a refresh and an access token (each a string, not empty), and a last_refresh that +def signed_in(data: bytes, skew_s: float) -> bool: + """Whether a sign-in file's bytes are a ChatGPT sign-in Codex refreshed at most `skew_s` past this machine's clock: + a JSON object whose tokens hold a refresh and an access token (each a string, not empty), and a last_refresh that parses and lies no further ahead (a file the session could write must not pin the shared copy).""" + from datetime import datetime, timedelta, timezone try: parsed = json.loads(data.decode("utf-8")) except (ValueError, UnicodeDecodeError, RecursionError): @@ -282,32 +316,4 @@ def signed_in(data: bytes, skew: timedelta) -> bool: return False if when.tzinfo is None: when = when.replace(tzinfo=timezone.utc) - return when <= datetime.now(timezone.utc) + skew - - -def take_back(root: Path, returned: Path, *, local: bool) -> bool: - """The sign-in a trial's container handed back (`returned`/auth.json) taken for the shared copy under `root`, when - it passes `signed_in` with SKEW (`local`: Harbor's local Docker environment) or OTHER_CLOCK and is at most - MAX_BYTES: its bytes, read once, go to a private copy beside `returned` (launch/taken/), which - hermit.agent.base.login.take_back takes when it is a later refresh of the copy, for the copy's account; the - private copy is removed after. Whether it was offered to the copy.""" - from hermit.agent.base import login - data = _regular_bytes(Path(returned) / FILE, MAX_BYTES) - if data is None or not signed_in(data, SKEW if local else OTHER_CLOCK): - return False - taken = Path(returned).parent / TAKEN_DIR - if os.path.lexists(taken): - if taken.is_dir() and not taken.is_symlink(): - shutil.rmtree(taken) - else: - taken.unlink() - taken.mkdir(mode=0o700) - copy = taken / FILE - try: - fd = os.open(copy, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW | os.O_CLOEXEC, 0o600) - with os.fdopen(fd, "wb") as f: - f.write(data) - login.take_back(Path(root), copy, within=taken) - finally: - shutil.rmtree(taken, ignore_errors=True) - return True + return when <= datetime.now(timezone.utc) + timedelta(seconds=skew_s) diff --git a/hermit/agent/base/container.py b/hermit/agent/base/container.py index 43cb3272..3acda95a 100644 --- a/hermit/agent/base/container.py +++ b/hermit/agent/base/container.py @@ -1,4 +1,4 @@ -"""Moved to hermit.agent.environment.docker.""" +"""Moved to hermit.environment.docker.""" from hermit._moved import forward -forward(__name__, "hermit.agent.environment.docker") +forward(__name__, "hermit.environment.docker") diff --git a/hermit/agent/base/credentials.py b/hermit/agent/base/credentials.py new file mode 100644 index 00000000..a448b5c6 --- /dev/null +++ b/hermit/agent/base/credentials.py @@ -0,0 +1,32 @@ +"""A credential's lifecycle, one implementation for every place a sign-in is handed to a session: provisioned, used +by the session (the block), taken back (a refresh the session made becomes the shared copy) and removed, the last two +however the block ends. A take-back that fails never keeps the copies (the removal runs regardless); a removal that +fails is raised, unless the block itself failed, whose error then stands (the removal's is printed).""" +from __future__ import annotations + +import contextlib +import sys + + +@contextlib.contextmanager +def held(provision, take_back=None, remove=None): + """`provision()` -> the copies, handed to the block; then `take_back(copies)` and `remove(copies)`.""" + copies = provision() + failed = False + try: + yield copies + except BaseException: + failed = True + raise + finally: + try: + if take_back is not None: + take_back(copies) + finally: + if remove is not None: + try: + remove(copies) + except Exception as exc: # noqa: BLE001 - the block's own error stands + if not failed: + raise + print(f"WARNING: a credential copy could not be removed: {exc}", file=sys.stderr, flush=True) diff --git a/hermit/agent/base/deadline.py b/hermit/agent/base/deadline.py new file mode 100644 index 00000000..53914c55 --- /dev/null +++ b/hermit/agent/base/deadline.py @@ -0,0 +1,73 @@ +"""One absolute deadline for a whole agent run, the one implementation every agent uses (ReAct's loop, the answer +agent, a CLI's session and its preparation): a point in time (or none), what is left of it, a time limit cut to it, +and a chat client bound to it (`bound`: every call given up once it passes, as Harbor's agent timeout cancelled its +agents; the call in flight closed, ChatAborted, and counted as abandoned: maybe billed, its usage never reported).""" +from __future__ import annotations + +import threading +import time + + +class Deadline: + def __init__(self, at: float | None = None): + self.at = None if at is None else float(at) # time.time() seconds; None: no deadline + + @classmethod + def within(cls, seconds: float | None) -> "Deadline": + return cls(time.time() + float(seconds) if seconds else None) + + def remaining(self) -> float | None: + return None if self.at is None else max(0.0, self.at - time.time()) + + @property + def expired(self) -> bool: + return self.at is not None and time.time() >= self.at + + def cut(self, seconds: float | None): + """`seconds` cut to what is left (what is left when none is given); False when nothing is left.""" + left = self.remaining() + if left is None: + return seconds + if left <= 0: + return False + return min(float(seconds), left) if seconds else left + + def bound(self, client) -> "BoundClient": + return BoundClient(client, self) + + +class BoundClient: + """A chat client whose calls end at the deadline (Deadline.bound). `expired` says whether it passed.""" + + def __init__(self, client, deadline: Deadline): + self.client, self.deadline, self.abort = client, deadline, threading.Event() + self.abandoned = 0 # calls given up in flight: sent, maybe billed, their usage never reported + left = deadline.remaining() + self.timer = threading.Timer(left if left is not None else 1e9, self.abort.set) + self.timer.daemon = True + if left is not None: + self.timer.start() + + @property + def expired(self) -> bool: + return self.abort.is_set() + + def remaining(self) -> float: + left = self.deadline.remaining() + return float("inf") if left is None else left + + def chat(self, messages, **kwargs): + from hermit.agent.model.llm import ChatAborted + if self.abort.is_set(): + raise ChatAborted("the task's agent budget is spent") + try: + return self.client.chat(messages, abort=self.abort, **kwargs) + except ChatAborted: + self.abandoned += 1 + raise + + def close(self) -> None: + self.timer.cancel() + + def __getattr__(self, name): + return getattr(self.client, name) diff --git a/hermit/agent/base/endpoint.py b/hermit/agent/base/endpoint.py index ef84514d..e89fa390 100644 --- a/hermit/agent/base/endpoint.py +++ b/hermit/agent/base/endpoint.py @@ -377,7 +377,7 @@ def cli_answer(agent: BaseAgent, request: dict, abort: threading.Event | None = own["writable"] = [*(agent.settings.get("writable") or ()), str(work)] # a sandbox shows it no other way try: session, prompt = agent.configured(**own).instructed(system, prompt) - from hermit.agent.environment import Local + from hermit.environment import Local result = session.run(prompt, environment=(place or Local)(work)) finally: shutil.rmtree(work, ignore_errors=True) @@ -468,7 +468,7 @@ def cli_problems(spec: dict) -> list[str]: "(a Docker container of its own, with container: {image, network, proxy}); without bubblewrap, " "sandbox.enforce: best_effort runs it unconfined and records that") elif isolation == "container": - from hermit.agent.environment import docker as container + from hermit.environment import docker as container try: container.settings(spec, "claude") except ValueError as e: @@ -492,7 +492,7 @@ def cli_endpoint(spec: dict, log_dir=None, *, sandbox=None) -> ChatEndpoint: problems = cli_problems(spec) if problems: raise ValueError("; ".join(problems)) - from hermit.agent.environment import Container, Local, Sandboxed + from hermit.environment import Container, Local, Sandboxed settings = {k: v for k, v in spec.items() if k not in ("cli", "timeout", "isolation", "container")} isolation = spec.get("isolation") or "sandbox" # where each session runs: its environment, not a setting if isolation == "sandbox": diff --git a/hermit/agent/base/interface.py b/hermit/agent/base/interface.py index de7b1986..46f81cfe 100644 --- a/hermit/agent/base/interface.py +++ b/hermit/agent/base/interface.py @@ -5,7 +5,7 @@ result.reply, result.trajectory, result.usage, result.session Layer 3 hands everything for one call through `run`: what to do (`prompt`), where and within what (`environment`: -hermit.agent.environment, its workspace, limits and isolation; a path is a Local environment there), with what +hermit.environment, its workspace, limits and isolation; a path is a Local environment there), with what (`tools`), whether to continue an earlier call (`session`), and named sub-agents (`roles`). A Base Agent enforces every limit with the environment's isolation, or refuses the call (`Unsupported`); it never ignores one. What an agent can be given of an adapter is its `exports` (`supports`). @@ -20,7 +20,7 @@ from hermit.agent.base.tool import Tool from hermit.agent.base.trajectory import Step -from hermit.agent.environment.base import (Environment, Limits, Mount, Unsupported, # noqa: F401 (re-exported) +from hermit.environment.base import (Environment, Limits, Mount, Unsupported, # noqa: F401 (re-exported) as_environment) from hermit.agent.model.usage import Usage diff --git a/hermit/agent/base/login.py b/hermit/agent/base/login.py index 6af2c482..19642179 100644 --- a/hermit/agent/base/login.py +++ b/hermit/agent/base/login.py @@ -533,20 +533,25 @@ def refused(output: str) -> bool: def hold(plan: Path | str | None = None, until: float | None = None, why: str = "session_refused", *, cancel=None, - max_s: float | None = None) -> None: + max_s: float | None = None) -> bool: """A session was refused for the plan's limit: hold every session back until `until` (a reset time the refusal named), else until a report says otherwise, or for an hour when no report says when the window resets. `why` is the hold's reason when the plan names none yet: usage_credits (a session drew the account's usage credits) holds only the sessions that may not draw them. If the hold cannot be written, this session waits five minutes itself rather - than ask again at once (at most `max_s`, until `cancel` is set).""" + than ask again at once (at most `max_s`, until `cancel` is set). Whether the hold was written.""" try: _hold(plan, until, why) + return True except OSError: - pause = 300.0 if max_s is None else min(300.0, max(0.0, max_s)) + pause = HOLD_UNWRITTEN_S if max_s is None else min(HOLD_UNWRITTEN_S, max(0.0, max_s)) if cancel is not None: cancel.wait(pause) else: time.sleep(pause) + return False + + +HOLD_UNWRITTEN_S = 300.0 # a refused session's own wait when its plan's hold cannot be written def _hold(plan: Path | str | None = None, until: float | None = None, why: str = "session_refused") -> None: @@ -584,12 +589,14 @@ def _hold(plan: Path | str | None = None, until: float | None = None, why: str = def logins_of(data: dict, evolver_params: dict | None = None) -> list[tuple[str, str, str | None, bool, str | None]]: """Every subscription a run signs in to, from the normalized config: (who, login, the CLI's program if named, whether its sessions run in a container, with the image's CLI: the acting agent's with environment.image and the - sandbox on, a model's through a CLI with isolation: container; never the learner's or its roles' (RRSI's, on the - learner's CLI), which run in the evolver sandbox on this machine's CLI; the Codex login's shared copy if named, - `login_root`). The program and the copy are the ones the sessions use: `bin`, or Harness²'s `claude_bin`.""" + sandbox on or in a task's own environment (environment.harbor: its `bin` is a path there), a model's through a CLI + with isolation: container; never the learner's or its roles' (RRSI's, on the learner's CLI), which run in the + evolver sandbox on this machine's CLI; the Codex login's shared copy if named, `login_root`). The program and the + copy are the ones the sessions use: `bin`, or Harness²'s `claude_bin`.""" params = (data.get("agent") or {}).get("params") or {} - image = bool((data.get("environment") or {}).get("image")) and \ - bool((data.get("sandbox") or {}).get("enabled", True)) # agent_pipeline's container (ContainerAgent) + image = (bool((data.get("environment") or {}).get("image")) and + bool((data.get("sandbox") or {}).get("enabled", True))) or \ + bool((data.get("environment") or {}).get("harbor")) # agent_pipeline's container, or a task's own environment learner = evolver_params or {} roles = learner.get("roles") if isinstance(learner.get("roles"), dict) else {} out = [] diff --git a/hermit/agent/base/react.py b/hermit/agent/base/react.py index d3326497..1c93ce00 100644 --- a/hermit/agent/base/react.py +++ b/hermit/agent/base/react.py @@ -208,10 +208,6 @@ def stopped(self, loop: Loop, message: dict) -> bool: def total(client, usages: list): - """The usage of this run: the sum of its calls' usage.""" - from hermit.agent.model.usage import Usage - costs = [u.cost_usd for u in usages] - return Usage(model=getattr(client, "model", ""), input_tokens=sum(u.input_tokens for u in usages), - cached_input_tokens=sum(u.cached_input_tokens for u in usages), - output_tokens=sum(u.output_tokens for u in usages), calls=sum(u.calls for u in usages), - cost_usd=None if not costs or any(c is None for c in costs) else sum(costs)) + """The usage of this run: the sum of its calls' usage (the Ledger's).""" + from hermit.agent.model.usage import Ledger + return Ledger(getattr(client, "model", "")).add(usages).total() diff --git a/hermit/agent/environment/__init__.py b/hermit/agent/environment/__init__.py index 6b09bed4..ab2a3278 100644 --- a/hermit/agent/environment/__init__.py +++ b/hermit/agent/environment/__init__.py @@ -1,14 +1,5 @@ -"""Environments: where an agent works and what it may reach (docs/framework/agent-layers.md#environments). +"""Moved to hermit.environment (the environments every layer shares: the task's, the agent's, the judge's).""" +from hermit._moved import forward - base.py the interface (Environment, Limits, Mount, ExecResult) and its implementations on this machine: - Local (isolation none), Sandboxed (the run's sandbox), Container (a Docker container) - docker.py a CLI session in a Docker container of its own (the Container environment's runner) - -A Base Agent (hermit.agent.base) works in the environment its call gives; the environment never imports a layer -above it. -""" -from hermit.agent.environment.base import (Container, Environment, ExecResult, Limits, Local, Mount, Sandboxed, - Unsupported, as_environment, placed) - -__all__ = ["Container", "Environment", "ExecResult", "Limits", "Local", "Mount", "Sandboxed", "Unsupported", - "as_environment", "placed"] +__getattr__ = forward(__name__, "hermit.environment", {"base": "hermit.environment.base", + "docker": "hermit.environment.docker"}) diff --git a/hermit/agent/evolver/agent.py b/hermit/agent/evolver/agent.py index 4e7366cd..9aac971d 100644 --- a/hermit/agent/evolver/agent.py +++ b/hermit/agent/evolver/agent.py @@ -41,7 +41,7 @@ import time from hermit.agent.base.interface import LoginFailed, make -from hermit.agent.environment import placed +from hermit.environment import placed from hermit.agent.base.records import parse_log, price_usage # noqa: F401 (re-exported) from hermit.agent.evolver.base import Context, Evolver, StepContext, StepResult, Usage diff --git a/hermit/agent/evolver/methods/ahe_adapter/debugger.py b/hermit/agent/evolver/methods/ahe_adapter/debugger.py index baaa1b8f..e526139e 100644 --- a/hermit/agent/evolver/methods/ahe_adapter/debugger.py +++ b/hermit/agent/evolver/methods/ahe_adapter/debugger.py @@ -51,7 +51,7 @@ import time from hermit.agent.base.interface import LoginFailed, Unsupported, make -from hermit.agent.environment import placed +from hermit.environment import placed from hermit.agent.base.trajectory import CODEX_STREAM, STREAMS, log_steps from hermit.agent.evolver.base import Usage from hermit.agent.evolver.methods.ahe_adapter import trajectory as _trajectory diff --git a/hermit/agent/evolver/methods/evosafe/cli_roles.py b/hermit/agent/evolver/methods/evosafe/cli_roles.py index f6b89214..a3fbd336 100644 --- a/hermit/agent/evolver/methods/evosafe/cli_roles.py +++ b/hermit/agent/evolver/methods/evosafe/cli_roles.py @@ -10,7 +10,7 @@ from hermit.agent.base.cli import PI_PARAMS from hermit.agent.base.interface import LoginFailed, make -from hermit.agent.environment import Limits, placed +from hermit.environment import Limits, placed from .experience import role_tools # The Layer 1/2 settings a configuration may give the role CLI; the rest are set per call below. diff --git a/hermit/agent/evolver/methods/harness2_upstream/channel.py b/hermit/agent/evolver/methods/harness2_upstream/channel.py index 23342551..bfc5974e 100644 --- a/hermit/agent/evolver/methods/harness2_upstream/channel.py +++ b/hermit/agent/evolver/methods/harness2_upstream/channel.py @@ -24,7 +24,7 @@ from hermit.agent.base.cli import PI_PARAMS from hermit.agent.base.interface import LoginFailed, make -from hermit.agent.environment import placed +from hermit.environment import placed from hermit.agent.base.records import price_usage from hermit.agent.evolver.base import Usage diff --git a/hermit/agent/evolver/methods/mwm/evolver.py b/hermit/agent/evolver/methods/mwm/evolver.py index 4de2a726..d3158a88 100644 --- a/hermit/agent/evolver/methods/mwm/evolver.py +++ b/hermit/agent/evolver/methods/mwm/evolver.py @@ -56,7 +56,7 @@ import uuid from hermit.agent.base.interface import LoginFailed, make -from hermit.agent.environment import Sandboxed +from hermit.environment import Sandboxed from hermit.agent.evolver.base import Context, Evolver, FinalResult, StepContext, StepResult, Usage HERE = Path(__file__).resolve().parent diff --git a/hermit/agent/evolver/methods/rrsi_adapter/domain.py b/hermit/agent/evolver/methods/rrsi_adapter/domain.py index c68474a0..dd8a9859 100644 --- a/hermit/agent/evolver/methods/rrsi_adapter/domain.py +++ b/hermit/agent/evolver/methods/rrsi_adapter/domain.py @@ -108,7 +108,9 @@ def smoke_failure(self, result: dict) -> bool: (timeouts included). Default: an output that carries a status (Harbor's) other than ok, or a rollout `run_agent` reports as timed out.""" out = result.get("output") if isinstance(result.get("output"), dict) else {} - return ("status" in out and out.get("status") != "ok") or bool(result.get("timed_out")) + outcome = result.get("outcome") if isinstance(result.get("outcome"), dict) else {} # a Harbor environment's + return ("status" in out and out.get("status") != "ok") or bool(result.get("timed_out")) or \ + outcome.get("status") not in (None, "ok") def halted_result(self, result: dict) -> bool: """Whether a smoke rollout was refused for a reason outside the candidate (a plan's usage limit): the diff --git a/hermit/agent/evolver/methods/rrsi_adapter/evolver.py b/hermit/agent/evolver/methods/rrsi_adapter/evolver.py index 372b9b49..f3ccb69e 100644 --- a/hermit/agent/evolver/methods/rrsi_adapter/evolver.py +++ b/hermit/agent/evolver/methods/rrsi_adapter/evolver.py @@ -55,7 +55,7 @@ from hermit.agent.base import login as codex_login from hermit.agent.base.interface import LoginFailed, make -from hermit.agent.environment import placed +from hermit.environment import placed from hermit.agent.model.backends import CLAUDE_ENV from hermit.agent.evolver.agent import AgentEvolver from hermit.agent.evolver.base import FinalResult, StepContext, StepResult, Usage diff --git a/hermit/agent/evolver/methods/rrsi_adapter/roles.py b/hermit/agent/evolver/methods/rrsi_adapter/roles.py index fc029071..beb92a91 100644 --- a/hermit/agent/evolver/methods/rrsi_adapter/roles.py +++ b/hermit/agent/evolver/methods/rrsi_adapter/roles.py @@ -39,7 +39,7 @@ from hermit.agent.base import login as codex_login from hermit.agent.base.interface import LoginFailed, Result, make -from hermit.agent.environment import placed +from hermit.environment import placed from hermit.agent.evolver.base import Usage ROLE_MODULES = ("rrsi.analyst", "rrsi.digester", "rrsi.critic") diff --git a/hermit/agent/model/usage.py b/hermit/agent/model/usage.py index 7845f44c..a6b349f0 100644 --- a/hermit/agent/model/usage.py +++ b/hermit/agent/model/usage.py @@ -13,3 +13,83 @@ class Usage: calls: int = 0 cost_usd: float | None = None # None when no price applies (for example a subscription) unreported_calls: int = 0 # calls whose tokens the CLI did not report: no price makes a cost of them + + +class Ledger: + """One run's usage, written by every attempt, every call given up in flight and every run again, and read once: + the one place tokens, calls and cost are summed for an agent's run and for a rollout's (hermit.trainer.rollout). + The cost is unknown (None) when any entry's is; an entry that may have been billed but reported nothing + (`unreported`) makes it unknown. What an empty ledger reports is the caller's word: None (nothing known: a session + ran and said nothing) or 0 (nothing ran).""" + + def __init__(self, model: str = ""): + self.model = model + self.entries: list[Usage] = [] + + def add(self, usages) -> "Ledger": + self.entries += list(usages or []) + return self + + def unreported(self, calls: int = 1, model: str | None = None, *, sessions: int = 0) -> "Ledger": + """`calls` requests given up in flight, or `sessions` sessions cut off before their usage: maybe billed, their + usage never reported (the cost is then unknown).""" + if calls or sessions: + self.entries.append(Usage(model=str(model or self.model or ""), calls=calls, + unreported_calls=calls + sessions, cost_usd=None)) + return self + + @property + def empty(self) -> bool: + return not self.entries + + def cost(self, empty: float | None = None) -> float | None: + if not self.entries: + return empty + return None if any(u.cost_usd is None for u in self.entries) else sum(u.cost_usd for u in self.entries) + + def tokens(self, empty: int | None = None) -> int | None: + """The tokens reported; unknown (None) when no entry reported any: only sessions or calls cut off before + their usage (`unreported`), never a measured 0.""" + if not self.entries: + return empty + if all(u.unreported_calls and not (u.input_tokens or u.output_tokens) for u in self.entries): + return None + return sum(u.input_tokens + u.output_tokens for u in self.entries) + + def calls(self) -> int: + return sum(u.calls for u in self.entries) + + def total(self) -> Usage: + """The sum as one Usage (its cost None when the ledger is empty or any entry's is unknown).""" + return Usage(model=self.model, input_tokens=sum(u.input_tokens for u in self.entries), + cached_input_tokens=sum(u.cached_input_tokens for u in self.entries), + output_tokens=sum(u.output_tokens for u in self.entries), calls=self.calls(), + cost_usd=self.cost(None), unreported_calls=sum(u.unreported_calls for u in self.entries)) + + +class Attempts: + """A rollout's runs, each one's own record and their sum (Hermit's Harbor runner's `_attempts`): the tokens the + runs reported (unknown only when none did), their model calls (a run that reported none adds 0) and their cost + (unknown when any run's is); each run's own in `extra.attempts` (its usage alone and its infra cause, `unscored`, + None for a run that is the agent's result). One run alone is returned as it is.""" + + def __init__(self): + self.runs: list = [] + + def add(self, out) -> None: + self.runs.append(out) + + def output(self): + import dataclasses + last = self.runs[-1] + if len(self.runs) == 1: + return last + tokens = [r.tokens for r in self.runs if r.tokens is not None] + costs = [r.cost_usd for r in self.runs] + calls = sum(int((r.api_speed or {}).get("model_calls") or 0) for r in self.runs) + each = [{"tokens": r.tokens, "cost_usd": r.cost_usd, "model_calls": (r.api_speed or {}).get("model_calls"), + "unscored": (r.extra or {}).get("unscored")} for r in self.runs] + return dataclasses.replace(last, tokens=sum(tokens) if tokens else None, + cost_usd=None if None in costs else sum(costs), + api_speed={**(last.api_speed or {}), "model_calls": calls}, + extra={**(last.extra or {}), "attempts": each}) diff --git a/hermit/agents/__init__.py b/hermit/agents/__init__.py index 2ee54f7f..24003c93 100644 --- a/hermit/agents/__init__.py +++ b/hermit/agents/__init__.py @@ -4,4 +4,4 @@ __getattr__ = forward(__name__, "hermit.agent.acting", { "base": "hermit.agent.acting.agent", "cli": "hermit.agent.acting.workspace", "react": "hermit.agent.acting.react", "tools": "hermit.agent.acting.tools", - "container": "hermit.agent.environment.docker", "login": "hermit.agent.base.login"}) + "container": "hermit.environment.docker", "login": "hermit.agent.base.login"}) diff --git a/hermit/common.py b/hermit/common.py new file mode 100644 index 00000000..09ea0f0e --- /dev/null +++ b/hermit/common.py @@ -0,0 +1,46 @@ +"""What every layer shares and none owns: what an acting agent returns (RunOutput), read by the run and the judge, and +how a `file.py:function` or `module:function` spec is loaded. It imports nothing of Hermit, so the agents +(hermit.agent), the tasks (hermit.task) and the environments (hermit.environment) can all use it without depending +on each other.""" +from __future__ import annotations + +from dataclasses import dataclass, field +import importlib +import importlib.util +from pathlib import Path +import sys +from typing import Any, Callable + + +@dataclass +class RunOutput: + output: Any # what the judge scores + log: Path | None = None # the host's original log (the trajectory), a file or a directory + cost_usd: float | None = None # None = unknown + wall_s: float = 0.0 + api_speed: dict | None = None # for example output tokens per second + messages: list | None = None # optional normalised message list of the trajectory + tokens: int | None = None # input + output tokens, if known + extra: dict = field(default_factory=dict) + + +def load_callable(spec: str, base: Path) -> Callable: + """'file.py:function' relative to `base`, or 'package.module:function'.""" + module_part, _, name = spec.partition(":") + if not name: + raise ValueError(f"Expected 'file.py:function' or 'module:function', got {spec!r}") + if module_part.endswith(".py"): + path = (Path(base) / module_part).resolve() + mod_name = f"_task_{path.stem}_{abs(hash(path))}" + mod_spec = importlib.util.spec_from_file_location(mod_name, path) + module = importlib.util.module_from_spec(mod_spec) + sys.modules[mod_name] = module + if str(path.parent) not in sys.path: + sys.path.insert(0, str(path.parent)) + mod_spec.loader.exec_module(module) + else: + module = importlib.import_module(module_part) + fn = getattr(module, name, None) + if not callable(fn): + raise ValueError(f"{spec} is not callable") + return fn diff --git a/hermit/config.py b/hermit/config.py index 92a0b240..2910faaa 100644 --- a/hermit/config.py +++ b/hermit/config.py @@ -44,6 +44,8 @@ def _merge(base: dict, extra: dict, prefix: str = "") -> dict: FREE = ("params", "llm", "backends", "container", "resources", # sections whose keys the framework does not know + "harbor", # (environment.harbor: Harbor's, checked by + # hermit.environment.harbor, overridable one by one) "model", *("codex", "claude", "pi", "mini-swe-agent", "opencode", "dsh", "hermes", "react", "direct")) # a layer's (agent-layers.md) BASES = {"codex": "codex", "claude": "claude-code", "claude-code": "claude-code", "pi": "pi", @@ -237,6 +239,9 @@ def load(cls, path: Path, overrides: list[str] = ()) -> "Config": defaults = yaml.safe_load(DEFAULTS.read_text()) user = yaml.safe_load(Path(path).read_text()) or {} data = _merge(defaults, user) + harbor = (data.get("environment") or {}).get("harbor") + if isinstance(harbor, str): # the benchmark file alone: a mapping, so a setting can be + data["environment"]["harbor"] = {"benchmark": harbor} # overridden by its path for item in overrides: _override(data, item) if data["pipeline"].get("run") and not data["agent"].get("name"): # the older spelling diff --git a/hermit/defaults.yaml b/hermit/defaults.yaml index cd9410dc..6a9a5b63 100644 --- a/hermit/defaults.yaml +++ b/hermit/defaults.yaml @@ -4,6 +4,8 @@ data: train: null val: null test: null + rollouts: 1 # data.tasks a directory of tasks (Harbor's format): k rows of each, #1..k + view: null # a directory's rows as its agents and learners read them: : step_size: 8 shuffle: true # false: every epoch takes the train split in file order (an ordered stream) shuffle_seed: 0 @@ -36,7 +38,7 @@ environment: # the task sandbox, where the acting agent runs ( # code isolates what it runs (AgentDyn's hooks in containers); warned image: null # a Docker image: a CLI acting agent runs in a container of its own (Codex or # Claude Code) - container: {} # with image: the container's network and proxy (hermit/agent/environment/docker.py) + container: {} # with image: the container's network and proxy (hermit/environment/docker.py) packages: [] # no image: Python distributions shown besides the framework's environment (an editable # install's package, one on PYTHONPATH), with their dependencies read: [] # no image: task paths shown read-only (relative to this file); never the data, the @@ -49,6 +51,8 @@ environment: # the task sandbox, where the acting agent runs ( resources: {} # memory_mb, file_mb, cpu_s: per process gpu: [] # NVIDIA GPUs shown, by the N of /dev/nvidia (CUDA and EGL; inside they count from 0); # none: no GPU (EGL renders in software where Mesa is installed) + harbor: null # a task set in Harbor's environments: each task's own (hermit/environment/harbor.py), its + # settings or the benchmark file that holds them; the agent works there, Harbor verifies judge: path: null name: score diff --git a/hermit/environment/__init__.py b/hermit/environment/__init__.py new file mode 100644 index 00000000..fead86c3 --- /dev/null +++ b/hermit/environment/__init__.py @@ -0,0 +1,14 @@ +"""Environments: where an agent works and what it may reach (docs/framework/agent-layers.md#environments). + + base.py the interface (Environment, Limits, Mount, ExecResult) and its implementations on this machine: + Local (isolation none), Sandboxed (the run's sandbox), Container (a Docker container) + docker.py a CLI session in a Docker container of its own (the Container environment's runner) + +A Base Agent (hermit.agent.base) works in the environment its call gives; the environment never imports a layer +above it. +""" +from hermit.environment.base import (Container, Environment, ExecResult, Limits, Local, Mount, Sandboxed, + Unsupported, as_environment, placed) + +__all__ = ["Container", "Environment", "ExecResult", "Limits", "Local", "Mount", "Sandboxed", "Unsupported", + "as_environment", "placed"] diff --git a/hermit/agent/environment/base.py b/hermit/environment/base.py similarity index 99% rename from hermit/agent/environment/base.py rename to hermit/environment/base.py index 1621e7e2..7407f418 100644 --- a/hermit/agent/environment/base.py +++ b/hermit/environment/base.py @@ -68,7 +68,7 @@ class Environment: call gives one, and a limit it cannot keep refused (Unsupported).""" isolation = "none" sandbox = None # the run's hermit.sandbox.Sandbox, under isolation sandbox - container: dict = {} # the container's settings (hermit.agent.environment.docker), under container + container: dict = {} # the container's settings (hermit.environment.docker), under container def __init__(self, workspace: Path | str, limits: Limits | None = None): self.workspace = Path(workspace) @@ -296,7 +296,7 @@ def exec(self, command, *, cwd=None, env=None, timeout=None, stdin=None): class Container(Environment): - """A Docker container of the session's own (isolation container, hermit.agent.environment.docker): the workspace + """A Docker container of the session's own (isolation container, hermit.environment.docker): the workspace and the session's home bound into it, its network the egress proxy's. `settings` are the container settings (image, network, proxy, ...: docker.settings). A CLI's session runs there; other commands are not run there yet.""" isolation = "container" diff --git a/hermit/environment/bwrap.py b/hermit/environment/bwrap.py new file mode 100644 index 00000000..633f41ea --- /dev/null +++ b/hermit/environment/bwrap.py @@ -0,0 +1,42 @@ +"""The bubblewrap backend: the sandbox a task's agent (and its judge, and a task's declared tools) runs in, built from +an EnvironmentSpec (docs/developer/sandbox-design.md). An allowlist: it shows the spec's read paths read-only, its +write paths read-write, passes its variables, and reaches the network its mode allows, through an allowlist proxy +where hosts are allowed. A caller adds what is its own (the agent's model hosts and key variable, a login's plan, a +CLI's installation) through the `extra_*` arguments; nothing of an agent or a task is known here. + + box, proxy = task_sandbox(spec, enforce="required", records=..., role="rollout", probe_absent=[...]) +""" +from __future__ import annotations + +from pathlib import Path +from typing import Iterable + +from hermit.environment.spec import EnvironmentSpec +from hermit.sandbox import AllowlistProxy, Sandbox, gpu_list, network_mode + + +def task_sandbox(spec: EnvironmentSpec, *, enforce: str, records: Path | None, role: str, + probe_absent: Iterable[Path] = (), extra_readable: Iterable[Path] = (), + extra_writable: Iterable[Path] = (), extra_env: Iterable[str] = (), extra_hosts: Iterable[str] = (), + framework: bool = True, proxy_log: str | None = None) -> tuple[Sandbox, AllowlistProxy | None]: + """(the sandbox, its allowlist proxy or None). Hosts in `extra_hosts` are reached under any network mode short of + public (the mode becomes an allowlist that holds them); the proxy logs to records/network-.jsonl. + The caller closes both.""" + network, allow = spec.network or "no-network", list(spec.allowed_hosts) + wanted = list(extra_hosts) + if network != "public" and wanted: + network, allow = "allowlist", [*allow, *[h for h in wanted if h not in allow]] + isolated, wants_proxy = network_mode(network) + proxy = AllowlistProxy(allow, log=records / f"network-{proxy_log or role}.jsonl" if records else None) \ + if (isolated == "isolated" and wants_proxy and allow) else None + try: + box = Sandbox(enabled=True, network=network, allow=allow, readable=[*spec.read, *extra_readable], + writable=[*spec.write, *extra_writable], proxy_url=proxy.url if proxy else None, + env_allow=[*spec.env, *extra_env], enforce=enforce, packages=list(spec.packages), + framework=framework, record_dir=records, probe_absent=list(probe_absent), + limits=dict(spec.resources), gpu=gpu_list(spec.gpu), role=role) + except BaseException: + if proxy is not None: + proxy.close() + raise + return box, proxy diff --git a/hermit/agent/environment/docker.py b/hermit/environment/docker.py similarity index 99% rename from hermit/agent/environment/docker.py rename to hermit/environment/docker.py index 130d8f2c..9896915e 100644 --- a/hermit/agent/environment/docker.py +++ b/hermit/environment/docker.py @@ -36,7 +36,7 @@ import threading import time -from hermit.agent.environment.base import Mount # noqa: F401 (re-exported) +from hermit.environment.base import Mount # noqa: F401 (re-exported) DEFAULTS = {"image": ("HERMIT_CODEX_IMAGE", "hermit-codex:0.162.0"), "network": ("HERMIT_CODEX_NETWORK", "hermit-sandbox"), diff --git a/hermit/environment/harbor.py b/hermit/environment/harbor.py new file mode 100644 index 00000000..88104f89 --- /dev/null +++ b/hermit/environment/harbor.py @@ -0,0 +1,769 @@ +"""A task's own Harbor environment, as a Hermit environment: Harbor gives the task's data, its environment and its +verifier; the agent working in it is Hermit's (docs/framework/task.md#benchmarks). + + environments = HarborEnvironments(settings, base=folder) # the run's: its settings, disk gate and cleanup + with environments.session(task_dir, name) as env: # started (the disk gate first), stopped after + agent.run(task, adapter, environment=env) # exec / upload / download, as any environment + verdict = env.verify(agent_logs=log_dir) # Harbor's verifier: {rewards, reward, feedback, ...} + +Harbor runs in its own venv (Python 3.13, `harbor` 0.25.0), not in Hermit's interpreter, so each environment is a +driver process of its own under Harbor's python (hermit/environment/harbor_driver.py: Harbor's own `Trial.run()` of +the task, its agent a bridge that relays this side's commands) and this side speaks JSON lines with it. The whole trial +is Harbor's code: the start of a prebuilt image (`docker_image`, pulled) or a Dockerfile's build, the agent's setup +(what the agent declares it needs installed: Agent.installs), the network policy and user of the agent phase, its +timeout, the task's tests uploaded only for the verifier, the verifier's environment (`[verifier] environment_mode`: +shared, or a separate one), the reward files, the trial's result.json. +What Hermit adds around it is the warm start Hermit's runner had (hermit/environment/harbor_docker.py, unchanged): +no environment starts while the host's /tmp or Docker's data root has less than its gate free (room kept for the +task's image when it is not here yet, and for the images other environments of the run are still to pull), the +task's prebuilt image removed after it as `docker_cleanup` says, and a driver that had to be killed has its trial's +compose projects taken down by name. + +Settings (a run config's `environment.harbor`, merged over the benchmark file it names as `benchmark`): + + tasks_dir the task directories (relative to the benchmark file's folder; its links followed once) + environment Harbor's environment config: type (docker by default; e2b, modal, daytona, ...: whatever Harbor + supports, its own extra installed in Harbor's venv), import_path, kwargs, delete, force_build, + mounts, extra_docker_compose (relative to the benchmark file's folder), env, override_cpus, + override_memory_mb, ... + verifier Harbor's verifier config (override_timeout_sec, max_timeout_sec, env, disable, ...) + verifier_timeout_multiplier, environment_build_timeout_multiplier Harbor's, at least 1 + agent_timeout_sec the agent's budget (Harbor's override_timeout_sec); null: each task's own + allowed_hosts hosts a task whose agent phase has no public network lets the agent reach (Harbor's + extra_allowed_hosts); the run adds its agent's model hosts + docker_cleanup, rmi_threshold_gb, disk_gate_gb, disk_gate_wait_s, tmp_gate_gb, image_sizes + the warm start (above): images_keep | rmi_task_image | rmi_over_gb; GB; seconds; the images' + compressed sizes (a JSON file), for the room the gate keeps + jobs_dir where each environment's trial directory goes (default ~/.cache/hermit/harbor/jobs) + harbor Harbor's command (default ~/.cache/hermit/harbor-venv/bin/harbor): its venv's python, beside it or + beside the file a link to it names, runs the driver + start_timeout_s, verify_timeout_s how long this side waits for the trial's agent phase (pull, build, setup) and + for the trial's end after it (the verifier) + verifier_timeout agent (a verifier that ran out of its time scores the agent's solution: Harbor's grading) or + infra (not the agent's result) + pass_reward, reward_key the reward that is a pass (1.0; null: a scale), the verifier's reward key (reward) + infra_retries, infra_wait_s a rollout that is not the agent's result (infra: hermit.environment.harbor_errors) runs + again in a fresh environment this many times at most (default 2), each after this long (default + 300 s), unless its cause runs no better again (NO_RETRY: the disk gate, model_rejected) + busy_retries, busy_wait_s a session the provider could not serve runs again in a fresh environment this many times + over the task's runs (default 3, not counted in infra_retries), the n-th after n x this (120 s), as + the runner's busy_retries + +An environment runs the task's commands where Harbor runs them (isolation `remote`): `exec` runs a command in the +task's main container as the task's agent user, from its working directory (the workspace), a variable passed by name +(never on a command line of this machine), a timeout enforced there (`timeout`, its process group killed); `upload` and +`download` move files between this machine and it (a relative path is in the workspace, an absolute one anywhere in +the environment); `install(cli)` reports the version Harbor's setup installed there (nothing installs during the +phase). `verify` ends the agent's phase (the bridge ends its commands still running) and reads the trial's verdict. +""" +from __future__ import annotations + +import atexit +import contextlib +import json +import os +from pathlib import Path, PurePosixPath +import queue +import re +import shlex +import shutil +import subprocess +import sys +import threading +import time +import uuid + +import yaml + +from hermit.environment import harbor_docker as docker +from hermit.environment import harbor_errors as errors +from hermit.environment.base import Environment, ExecResult, Limits, Unsupported + +HARBOR = "~/.cache/hermit/harbor-venv/bin/harbor" +JOBS_DIR = "~/.cache/hermit/harbor/jobs" +DRIVER = Path(__file__).resolve().with_name("harbor_driver.py") +DISK_GATE_GB = 20.0 +DISK_GATE_WAIT_S = 7200.0 +TMP_GATE_GB = 1.0 +HOST_TMP = Path("/tmp") +START_TIMEOUT_S = 3600.0 # the driver's start up to the agent phase, at least (start_timeout) +START_ALLOWANCE_S = 1800.0 # beside Harbor's build and setup timeouts: the healthcheck, the logs +BUILD_TIMEOUT_S = 600.0 # Harbor's default build_timeout_sec +SETUP_TIMEOUT_S = 1200.0 # the agent's setup (a CLI's install) when setup_timeout_sec is not set: Harbor's + # own 360 s default is too short for an apt and npm install in a 1-CPU container + # under load +VERIFY_TIMEOUT_S = 7200.0 # ... and its verifier when Harbor names no timeout (its own ends it first) +VERIFY_ALLOWANCE_S = 1800.0 # beside Harbor's verifier timeout: the tests' upload, a separate environment's + # start, the logs +STOP_TIMEOUT_S = 600.0 +SETUP_ALLOWANCE_S = 360.0 # beside a task's agent budget: the home's upload, the logs (the runner's + # SETUP_BUDGET_S) +DELIVERY_MARGIN_S = 60.0 # the agent's own budget ends this much before Harbor's agent timeout, so that its + # session's end and logs reach the trial before Harbor cancels the phase +PASS_REWARD = 1.0 +MULTIPLIERS = ("verifier_timeout_multiplier", "environment_build_timeout_multiplier") +SETTINGS = (*MULTIPLIERS, "benchmark", "tasks_dir", "environment", "verifier", "agent_timeout_sec", + "setup_timeout_sec", "allowed_hosts", "docker_cleanup", "rmi_threshold_gb", "disk_gate_gb", + "disk_gate_wait_s", "tmp_gate_gb", "image_sizes", "jobs_dir", "harbor", "start_timeout_s", + "verify_timeout_s", "verifier_timeout", "pass_reward", "reward_key", "infra_retries", "infra_wait_s", + "busy_retries", "busy_wait_s", "verifier_redact", + "dataset") +VERIFIER_TIMEOUTS = ("agent", "infra") +FEEDBACK_CHARS = 4000 + + +class EnvironmentUnavailable(RuntimeError): + """The task's environment could not be given (the disk gate waited its time out, the image did not pull or build, + the driver died): not the agent's result. `cause` is its infra cause (hermit.environment.harbor_errors: disk, + sandbox_failed, harbor_failed, harbor_timeout).""" + + def __init__(self, message: str, cause: str = "harbor_failed"): + super().__init__(message) + self.cause = cause + + +def _merge(base: dict, over: dict) -> dict: + out = dict(base) + for key, value in over.items(): + out[key] = {**out[key], **value} if isinstance(value, dict) and isinstance(out.get(key), dict) else value + return out + + +def settings_of(section, base: Path) -> tuple[dict, Path]: + """(the settings, the folder relative paths resolve against) of a run config's `environment.harbor`: a mapping, + merged over the YAML file its `benchmark` names (whose `dataset:` block is `dataset prepare`'s, not a setting), + or the path of such a file alone.""" + if isinstance(section, (str, os.PathLike)): + section = {"benchmark": str(section)} + if not isinstance(section, dict): + raise ValueError("environment.harbor is a mapping of Harbor's settings, or the path of a benchmark file") + given = dict(section) + folder = Path(base) + if given.get("benchmark"): + path = Path(os.path.expanduser(str(given["benchmark"]))) + path = path if path.is_absolute() else (Path(base) / path) + shared = yaml.safe_load(path.read_text()) or {} + if not isinstance(shared, dict): + raise ValueError(f"environment.harbor.benchmark {path} is not a mapping") + shared.pop("dataset", None) + given = _merge(shared, {k: v for k, v in given.items() if k != "benchmark"}) + folder = path.resolve().parent + given.pop("dataset", None) + unknown = sorted(set(given) - set(SETTINGS)) + if unknown: + raise ValueError(f"environment.harbor: unknown keys {unknown}; known: {sorted(SETTINGS)}") + return given, folder + + +class HarborEnvironments: + """The run's Harbor environments: one for each task, sharing the settings, the disk gate's room for pulls and the + image cleanup's lock.""" + + def __init__(self, settings: dict, *, base: Path, allowed_hosts=(), log=None, drop_env=()): + self.settings, self.base = settings_of(settings, base) + s = self.settings + if not s.get("tasks_dir"): + raise ValueError("environment.harbor.tasks_dir is required (the task directories)") + self.tasks_dir = Path(os.path.realpath(self.path(s["tasks_dir"]))) + self.jobs_dir = Path(os.path.expanduser(str(s.get("jobs_dir") or JOBS_DIR))).resolve() + self.harbor = self._harbor(s.get("harbor")) + self.python = str(Path(os.path.realpath(self.harbor)).parent / "python") + if not (os.path.isfile(self.python) and os.access(self.python, os.X_OK)): + raise ValueError(f"{self.python} is not an executable file: a Harbor environment's driver runs under the " + f"python of Harbor's venv, beside environment.harbor.harbor ({self.harbor}) or beside " + f"the file a link of that name points to (default {HARBOR})") + self.cleanup = str(s.get("docker_cleanup") or "images_keep") + if self.cleanup not in docker.CLEANUP_MODES: + raise ValueError(f"environment.harbor.docker_cleanup must be one of {', '.join(docker.CLEANUP_MODES)}, " + f"not {self.cleanup!r}") + env_config = s.get("environment") if isinstance(s.get("environment"), dict) else {} + # a local Docker environment Harbor deletes after the trial (a kept one, delete false, is left as it is) + self.keep_containers = bool((env_config.get("kwargs") or {}).get("keep_containers")) + self.local_docker = env_config.get("type") in (None, "docker") and env_config.get("delete", True) is not False \ + and not self.keep_containers # Harbor told to keep them: kept + self.verifier_timeout = str(s.get("verifier_timeout") or "agent") + if self.verifier_timeout not in VERIFIER_TIMEOUTS: + raise ValueError(f"environment.harbor.verifier_timeout is agent or infra, not {self.verifier_timeout!r}") + for key in MULTIPLIERS: + if s.get(key) is not None and float(s[key]) < 1.0: + raise ValueError(f"environment.harbor.{key} must be at least 1, not {s[key]}") + self.pass_reward = PASS_REWARD if "pass_reward" not in s else \ + (None if s["pass_reward"] is None else float(s["pass_reward"])) + self.reward_key = str(s.get("reward_key") or "reward") + self.redactions = self._redactions(s.get("verifier_redact")) + if (s.get("verifier") or {}).get("disable"): + raise ValueError("environment.harbor.verifier.disable: a disabled verifier gives no reward, so every " + "rollout would be infra and run again; a Harbor task set is scored by its verifier") + self.infra_retries = self._count("infra_retries", errors.INFRA_RETRIES, int) + self.infra_wait_s = self._count("infra_wait_s", errors.INFRA_WAIT_S, float) + self.busy_retries = self._count("busy_retries", errors.BUSY_RETRIES, int) + self.busy_wait_s = self._count("busy_wait_s", errors.BUSY_WAIT_S, float) + sizes = json.loads(self.path(s["image_sizes"]).read_text()) if s.get("image_sizes") else {} + self.image_sizes = {k: v for k, v in sizes.items() if not str(k).startswith("_")} + self.pulls = docker.Pulls() + self.allowed_hosts = [*(s.get("allowed_hosts") or []), *(h for h in allowed_hosts + if h not in (s.get("allowed_hosts") or []))] + self.log = log or (lambda m: print(m, file=sys.stderr, flush=True)) + self.drop_env = set(drop_env) # this process's variables the driver (and Harbor's compose) never sees + self._root: Path | None = None + self.live: set = set() # the environments started and not stopped: stopped at exit + self.stopping = threading.Event() # set once the run ends (a lost sign-in): no gate or pause waits on + self.admission = None # the agent's login's plan admission around each environment (a + # context manager factory: hermit.isolate._plan_admission) + self._lock = threading.Lock() + atexit.register(self.stop_all) + + def stop_all(self) -> None: + """Every environment still up (this process is ending): stopped, its compose projects taken down.""" + with self._lock: + left = list(self.live) + for env in left: + with contextlib.suppress(Exception): + env.stop() + + def _count(self, key: str, default, kind): + value = self.settings.get(key) + try: + value = default if value is None else kind(value) + except (TypeError, ValueError): + value = -1 + if value < 0: + raise ValueError(f"environment.harbor.{key} is a number, at least 0, not {self.settings.get(key)!r}") + return value + + def path(self, value) -> Path: + p = Path(os.path.expandvars(str(value))).expanduser() + return p if p.is_absolute() else (self.base / p).resolve() + + @staticmethod + def _harbor(value) -> str: + value = os.path.expanduser(str(value or HARBOR)) + if "/" not in value: + value = shutil.which(value) or value + return os.path.abspath(value) if "/" in value else value + + def task_dir(self, task) -> Path: + """A task's directory: its private half's (a directory task), else its row's `harbor_task` under + tasks_dir.""" + private = getattr(task, "private", None) or {} + public = getattr(task, "public", task) + if private.get("task_dir"): + return Path(private["task_dir"]) + name = public.get("harbor_task") + if not name: + raise ValueError(f"task {public.get('id')!r} names no Harbor task directory (harbor_task)") + return self.tasks_dir / str(name) + + # --- the warm start: the disk gate and the image cleanup (hermit.environment.harbor_docker, unchanged) ------- + def _gate(self, key: str, default: float) -> float: + value = self.settings.get(key) + return float(default if value is None else value) + + def docker_root(self) -> Path: + if self._root is None: + self._root = docker.data_root() + return self._root + + def image_gb(self, image: str) -> float: + compressed = self.image_sizes.get(image) + return float(compressed) * docker.ON_DISK / docker.GB if compressed else 0.0 + + def wait_for_disk(self, task_dir: Path, admitted: list, cancel=None) -> bool: + """The disk gate: the host's /tmp first, then Docker's data root with room for the task's prebuilt image when + it is not here yet (`image_sizes`), beside the room other environments still to pull hold (`admitted` gets + the key of the room this one holds, given back by `done`).""" + wait_s = float(self.settings.get("disk_gate_wait_s") or DISK_GATE_WAIT_S) + cancel = cancel or self.stopping + if not docker.wait_for_disk(HOST_TMP, self._gate("tmp_gate_gb", TMP_GATE_GB), wait_s, cancel=cancel, + log=self.log): + return False + if cancel.is_set(): # the run ended between the gates + return False + image = docker.task_image(task_dir) + return docker.wait_for_disk(self.docker_root(), self._gate("disk_gate_gb", DISK_GATE_GB), wait_s, + image=image, image_gb=self.image_gb(image) if image else 0.0, pulls=self.pulls, + admitted=admitted, cancel=cancel, log=self.log) + + def disk_held(self, task_dir: Path) -> str: + """Which gate held an environment back once wait_for_disk gave up.""" + tmp_gate = self._gate("tmp_gate_gb", TMP_GATE_GB) + if tmp_gate > 0 and docker.free_gb(HOST_TMP) < tmp_gate: + return f"the host's {HOST_TMP} stayed under {tmp_gate:g} GB free" + gate = self._gate("disk_gate_gb", DISK_GATE_GB) + held = f"the Docker data root {self.docker_root()} stayed under {gate:g} GB free" + image = docker.task_image(task_dir) + image_gb = self.image_gb(image) if image else 0.0 + if image_gb and docker.image_size(image) is None: + held += f" with {image_gb:.1f} GB more for the task's image {image}, not here yet" + with self.pulls.lock: + taken = self.pulls.taken_gb() + if taken: + held += f", beside {taken:.1f} GB held for the images other environments are still to pull" + return held + + def cleanup_images(self, task_dir: Path) -> None: + docker.cleanup_images(self.cleanup, task_dir, self.jobs_dir / ".locks", + threshold_gb=float(self.settings.get("rmi_threshold_gb") or 1.0)) + + # --- what each environment is given -------------------------------------------------------------------------- + def environment_config(self) -> dict: + given = self.settings.get("environment") or {} + if not isinstance(given, dict): + raise ValueError("environment.harbor.environment is a mapping (Harbor's environment config)") + env = {"delete": True, **{k: v for k, v in given.items() if v is not None}} + if env.get("extra_docker_compose"): + env["extra_docker_compose"] = [str(self.path(p)) for p in env["extra_docker_compose"]] + if env.get("mounts"): + env["mounts"] = [{**m, "source": str(self.path(m["source"]))} if m.get("type", "bind") == "bind" and + m.get("source") else m for m in env["mounts"]] + return env + + def start_request(self, task_dir: Path, trials_dir: Path, name: str, installs=None) -> dict: + s = self.settings + agent = {"override_timeout_sec": s.get("agent_timeout_sec"), + "override_setup_timeout_sec": float(s.get("setup_timeout_sec") or SETUP_TIMEOUT_S), + "extra_allowed_hosts": list(self.allowed_hosts) or None} + return {"op": "start", "task_dir": str(task_dir), "trials_dir": str(trials_dir), "trial_name": name, + "agent": {k: v for k, v in agent.items() if v is not None}, "environment": self.environment_config(), + "verifier": {k: v for k, v in (s.get("verifier") or {}).items() if v is not None}, + "trial": {k: float(s[k]) for k in MULTIPLIERS if s.get(k) is not None}, + "installs": list(installs or [])} + + def open(self, task_dir: Path, name: str | None = None, installs=None) -> "HarborEnvironment": + """The environment of one task, not started yet; `installs`, what the agent needs installed in it before its + phase (Agent.installs: Harbor's setup runs them).""" + return HarborEnvironment(task_dir, owner=self, name=name, installs=installs) + + @contextlib.contextmanager + def session(self, task_dir: Path, name: str | None = None, installs=None): + """One task's environment, started (the disk gate first, then Harbor's trial up to its agent phase) and + stopped after, however the block ends.""" + env = self.open(task_dir, name, installs) + try: + env.start() + yield env + finally: + env.stop() + + @staticmethod + def _redactions(patterns) -> list[str]: + """verifier_redact: regexes removed from the verifier's output the feedback and an infra verdict quote (what + the learner must not read, a test's expected answer), the Harbor runner's setting.""" + patterns = [patterns] if isinstance(patterns, str) else list(patterns or []) + for pattern in patterns: + try: + re.compile(str(pattern)) + except re.error as exc: + raise ValueError(f"environment.harbor.verifier_redact: {pattern!r} is no regular expression " + f"({exc})") from None + return [str(p) for p in patterns] + + def start_timeout(self, task_dir: Path) -> float: + """How long this side waits for the trial's agent phase: start_timeout_s, else Harbor's own budgets for what + comes before it (the task's build timeout times its multiplier, the setup timeout) and START_ALLOWANCE_S, at + least START_TIMEOUT_S.""" + import tomllib + if self.settings.get("start_timeout_s"): + return float(self.settings["start_timeout_s"]) + try: + config = tomllib.loads((Path(task_dir) / "task.toml").read_text()) + except (OSError, ValueError): + config = {} + env = config.get("environment") if isinstance(config.get("environment"), dict) else {} + build = float(env.get("build_timeout_sec") or BUILD_TIMEOUT_S) * \ + float(self.settings.get("environment_build_timeout_multiplier") or 1.0) + setup = float(self.settings.get("setup_timeout_sec") or SETUP_TIMEOUT_S) + return max(START_TIMEOUT_S, build + setup + START_ALLOWANCE_S) + + def redact(self, text: str) -> str: + for pattern in self.redactions: + text = re.sub(pattern, "[redacted]", text) + return text + + def verdict(self, verified: dict) -> dict: + """What a verifier's answer is for a judge, as Hermit's Harbor runner read a trial (hermit.environment. + harbor_errors): the reward (`reward_key`'s only), the rewards, the feedback + (feedback.txt, else the verifier's output's tail), a status and its cause: ok; `timeout` for a verifier that + ran out of its time on the agent's solution (solution_timed_out, scored 0; infra with verifier_timeout: infra); + `agent_error` for one that left no reward Harbor could read (verifier_failed, scored 0, as Harbor grades it); + `infra` for an environment that failed under the verifier, a verifier that ran no test (its tools never arrived, + the host's disk full) or no exception and no reward: not the agent's result.""" + rewards = verified.get("rewards") or None + exception = verified.get("exception") or None + reward = (rewards or {}).get(self.reward_key) # the named reward only: none there is the runner's infra + reward = float(reward) if reward is not None else None + # Harbor's trial as the runner read its result.json: the reward under its key, the exception Harbor recorded + result = {"verifier_result": {"rewards": dict(rewards or {})}, + "exception_info": {"exception_type": exception.get("type"), "exception_message": + exception.get("message") or ""} if exception else None} + later = [e for e in verified.get("later") or [] if e.get("type") not in (None, "AgentTimeoutError", + "CancelledError")] + if exception and exception.get("type") == "AgentTimeoutError" and later: + # Harbor kept the agent's timeout; a verifier that then failed is read by its own policy when it is infra + # (its timeout with verifier_timeout infra, a separate verifier environment that did not start) + after = {**result, "exception_info": {"exception_type": later[0]["type"], + "exception_message": later[0].get("message") or ""}} + other = errors.verifier_timeout_verdict(errors.classify(after, reward_key=self.reward_key), + self.verifier_timeout) + if other["status"] == "infra": + result = after + read = errors.classify(result, reward_key=self.reward_key) + read = errors.verifier_timeout_verdict(read, self.verifier_timeout) + logs = Path(verified.get("verifier_dir") or "") if verified.get("verifier_dir") else None + read = errors.verifier_verdict(read, logs, self.redactions) + feedback = "" + for name in ("feedback.txt", "test-stdout.txt"): # optional: one this process cannot read leaves the reward + try: + if logs is not None and (logs / name).is_file() and not (logs / name).is_symlink(): + text = (logs / name).read_text(encoding="utf-8", errors="replace") + else: + continue + except OSError: + continue + feedback = self.redact(text)[-FEEDBACK_CHARS:] + break + passed = None if self.pass_reward is None else (reward is not None and reward >= self.pass_reward) + return {"reward": reward, "rewards": rewards, "status": read["status"], "failure_cause": read["failure_cause"], + "passed": passed, + "exception": {**exception, "message": self.redact(str(exception.get("message") or ""))} + if exception else None, + "exception_message": self.redact(str(read["exception_message"] or "")), + "feedback": feedback, "verifier_dir": verified.get("verifier_dir"), + "agent_dir": verified.get("agent_dir"), + "trial_dir": verified.get("trial_dir")} + + +class _Driver: + """The driver process and its JSON lines: requests by id, answered in any order by a reader thread.""" + + def __init__(self, python: str, log_path: Path, env: dict): + log_path.parent.mkdir(parents=True, exist_ok=True) + self.log = open(log_path, "ab") + self.proc = subprocess.Popen([python, "-I", str(DRIVER)], stdin=subprocess.PIPE, stdout=subprocess.PIPE, + stderr=self.log, env=env, start_new_session=True, cwd=str(log_path.parent)) + self.lock = threading.Lock() + self.waiting: dict[int, queue.Queue] = {} + self.next_id = 0 + self.dead: str | None = None + self.outbox: queue.Queue = queue.Queue() # the requests to write: a writer thread of its own, so that + self.reader = threading.Thread(target=self._read, daemon=True) # a hung driver's full pipe holds no caller + self.reader.start() # past its request's timeout + self.writer = threading.Thread(target=self._write, daemon=True) + self.writer.start() + + def _write(self) -> None: + while True: + item = self.outbox.get() + if item is None: + return + rid, data = item + try: + self.proc.stdin.write(data) + self.proc.stdin.flush() + except (BrokenPipeError, OSError, ValueError) as exc: + with self.lock: + box = self.waiting.pop(rid, None) + if box is not None: + box.put({"ok": False, "error": f"the Harbor driver is gone: {exc}", "type": "DriverGone"}) + + def _read(self) -> None: + for line in self.proc.stdout: + try: + reply = json.loads(line) + except ValueError: + continue + with self.lock: + box = self.waiting.pop(reply.get("id"), None) + if box is not None: + box.put(reply) + dead = f"the Harbor driver exited ({self.proc.wait()})" + with self.lock: # a request registers under it: none misses the death + self.dead = dead + boxes, self.waiting = list(self.waiting.values()), {} + for box in boxes: + box.put({"ok": False, "error": dead, "type": "DriverGone"}) + + def request(self, op: str, timeout: float | None, **fields) -> dict: + box: queue.Queue = queue.Queue() + with self.lock: + if self.dead: + raise EnvironmentUnavailable(self.dead, "harbor_failed") + self.next_id += 1 + rid = self.next_id + self.waiting[rid] = box + self.outbox.put((rid, (json.dumps({"id": rid, "op": op, **fields}, default=str) + "\n").encode())) + try: + reply = box.get(timeout=timeout) + except queue.Empty: + with self.lock: + self.waiting.pop(rid, None) + raise TimeoutError(f"the Harbor driver did not answer {op} in {timeout:.0f} s") from None + if not reply.get("ok"): + if reply.get("type") == "DriverGone": + raise EnvironmentUnavailable(str(reply.get("error")), "harbor_failed") + error = RuntimeError(str(reply.get("error") or f"{op} failed")) + error.kind = reply.get("type") + raise error + return reply + + def close(self, timeout: float) -> bool: + """Close its stdin (it stops its trial and exits); True when it exited by itself, else it is killed. A writer + still blocked on a driver that stopped reading holds the stream: the driver is killed first, never waited on + through a close that would block on the writer's lock.""" + self.outbox.put(None) + self.writer.join(timeout=5) + if self.writer.is_alive(): + self._kill() + self.writer.join(timeout=10) + self.reader.join(timeout=5) + self.log.close() + return False + with contextlib.suppress(OSError): + self.proc.stdin.close() + try: + self.proc.wait(timeout=timeout) + done = True + except subprocess.TimeoutExpired: + done = False + self._kill() + self.reader.join(timeout=5) + self.log.close() + return done + + def _kill(self) -> None: + for sig in (15, 9): + with contextlib.suppress(ProcessLookupError, PermissionError): + os.killpg(self.proc.pid, sig) + try: + self.proc.wait(timeout=20) + return + except subprocess.TimeoutExpired: + continue + + +class HarborEnvironment(Environment): + """One task's Harbor environment (module docstring). Its workspace is the task's working directory in the + environment, a path there, not on this machine.""" + isolation = "remote" + + def __init__(self, task_dir: Path, *, owner: HarborEnvironments, name: str | None = None, + limits: Limits | None = None, installs=None): + super().__init__(PurePosixPath("/"), limits) + self.task_dir = Path(task_dir) + self.installs = list(installs or []) + self.owner = owner + safe = "".join(c if c.isalnum() or c in "-_." else "_" for c in (name or self.task_dir.name)) + self.name = f"{safe[:48]}__{uuid.uuid4().hex[:8]}" + self.trials_dir = owner.jobs_dir / self.name + self.driver: _Driver | None = None + self.started: dict = {} + self.instruction: str | None = None + self.network_public = True + self.plan_admitted = False # the run holds the agent's login's plan admission around this environment + self.setup_budget_s = SETUP_ALLOWANCE_S + self.admitted: list = [] + self.verified: dict | None = None + self.deadline_at: float | None = None # the agent phase's absolute deadline (less the delivery margin) + + def __repr__(self) -> str: + return f"HarborEnvironment({self.task_dir.name!r}, workspace={str(self.workspace)!r})" + + # --- the lifecycle ------------------------------------------------------------------------------------------- + def start(self) -> "HarborEnvironment": + owner = self.owner + if not self.task_dir.is_dir(): + raise ValueError(f"no Harbor task directory {self.task_dir}") + if owner.stopping.is_set(): + raise EnvironmentUnavailable("the run ended: no environment started", "stopped") + held = not owner.wait_for_disk(self.task_dir, self.admitted) + if owner.stopping.is_set(): # the run ended while it waited: nothing started, room back + for key in self.admitted: + owner.pulls.done(key) + self.admitted = [] + raise EnvironmentUnavailable("the run ended while the disk gate held the task: no environment started", + "stopped") + if held: + raise EnvironmentUnavailable(f"{owner.disk_held(self.task_dir)}: no environment started", "disk") + with owner._lock: + owner.live.add(self) + try: # this machine's failing: infra, never run again (host_failed) + self.trials_dir.mkdir(parents=True, exist_ok=True) + tmp = owner.jobs_dir / "tmp" + tmp.mkdir(parents=True, exist_ok=True) + env = {**{k: v for k, v in os.environ.items() if k not in owner.drop_env}, "TMPDIR": str(tmp)} + self.driver = _Driver(owner.python, self.trials_dir / "driver.log", env) + except OSError as exc: + raise EnvironmentUnavailable(f"this machine could not start the environment's driver: " + f"{type(exc).__name__}: {exc}", "host_failed") from None + try: + started = self.driver.request(**owner.start_request(self.task_dir, self.trials_dir, self.name, + self.installs), + timeout=owner.start_timeout(self.task_dir)) + except (RuntimeError, TimeoutError) as exc: + kind = "TimeoutError" if isinstance(exc, TimeoutError) else getattr(exc, "kind", None) + if kind == "UnsupportedTask": # Harbor runs it for none of Hermit's agents: never again + raise EnvironmentUnavailable(f"Harbor refused the task for Hermit's agent: {exc}"[-3000:], + "unsupported_task") from None + raise EnvironmentUnavailable(f"the task's environment did not start: {exc}"[-3000:], + errors.start_cause(kind, str(exc))) from None + self.started = started + self.workspace = PurePosixPath(started.get("workdir") or "/") + self.instruction = started.get("instruction") # Harbor's own, as its agents are given it + self.network_public = (started.get("agent_network") or "public") == "public" + self.bin_served = dict(started.get("bins") or {}) # cli -> whether its mounted binary serves (setup's word) + kind = (owner.settings.get("environment") or {}).get("type") if isinstance( + owner.settings.get("environment"), dict) else None + self.same_clock = kind in (None, "docker") # a local Docker container shares this machine's clock + self.setup_budget_s = float(started.get("setup_timeout_sec") or 0) + SETUP_ALLOWANCE_S + budget = started.get("agent_timeout_sec") + if budget: # ends before Harbor's timeout, its logs delivered first + margin = min(DELIVERY_MARGIN_S, float(budget) * 0.1) + budget = float(budget) - margin + # one absolute deadline from the phase's start as the bridge saw it begin (Harbor's timer): every command's + # time and an agent's whole run come off it + epoch = started.get("deadline_epoch") + self.deadline_at = (float(epoch) - margin) if epoch else time.time() + budget + self.limits = Limits(readable=self.limits.readable, hidden=self.limits.hidden, network=self.limits.network, + timeout_s=float(budget) if budget else self.limits.timeout_s) + return self + + def stop(self) -> None: + """The trial's result written and the environment down; the room its pull held given back, its image + removed as docker_cleanup says. A driver that does not stop within STOP_TIMEOUT_S is killed and its compose + projects taken down by name.""" + driver, self.driver = self.driver, None + try: + if driver is not None: + stopped = False + with contextlib.suppress(RuntimeError, TimeoutError, EnvironmentUnavailable): + driver.request("stop", timeout=STOP_TIMEOUT_S) + stopped = True + if not driver.close(30 if stopped else 60) or not stopped: + if not self.owner.keep_containers: # Harbor told to keep them: never taken down + docker.compose_down(self.name) + elif self.owner.local_docker and docker.compose_projects(docker.compose_project(self.name)): + docker.compose_down(self.name) # Harbor logged a down that failed: its projects, by name + self.owner.cleanup_images(self.task_dir) + finally: + for key in self.admitted: + self.owner.pulls.done(key) + self.admitted = [] + with self.owner._lock: + self.owner.live.discard(self) + + def _live(self) -> _Driver: + if self.driver is None: + raise EnvironmentUnavailable("the environment is not started (or was stopped)") + if self.verified is not None: + raise Unsupported("the task was verified: its environment takes no more commands") + return self.driver + + # --- the agent's operations ---------------------------------------------------------------------------------- + def path(self, relative: str | Path = "", *, follow: bool = True) -> PurePosixPath: + """A path of the environment: a relative one in the workspace (one that leaves it is refused), an absolute + one as it is.""" + rel = PurePosixPath(str(relative)) + if rel.is_absolute(): + return rel + parts: list[str] = [] + for part in rel.parts: + if part == "..": + if not parts: + raise ValueError(f"{relative} lies outside the workspace") + parts.pop() + elif part not in (".", ""): + parts.append(part) + return PurePosixPath(self.workspace, *parts) + + def exec(self, command, *, cwd=None, env=None, timeout=None, stdin=None, user=None, by_fd=None, + stream: str | None = None, secret_env=()) -> ExecResult: + """A command in the task's main container (Environment.exec). `by_fd` ({a variable of `env`: the variable that + names its descriptor}): each such secret reaches the command on a descriptor instead, out of every process's + environment there (the driver's container_prologue). `stream`, a log name: the command's output is kept as it + runs, and is the agent log of that name should the phase end while it runs (a CLI's stream).""" + if self.limits.readable or self.limits.hidden or self.limits.network is not None: + raise Unsupported("a Harbor environment is the task's own: it takes no readable mounts, hidden paths or " + "network limits of Hermit's (the task's task.toml says what it reaches)") + line = command if isinstance(command, str) else shlex.join([str(c) for c in command]) + timeout = timeout if timeout is not None else self.limits.timeout_s + if self.deadline_at is not None: # what is left of the phase's deadline, never more + left = max(1.0, self.deadline_at - time.time()) + timeout = min(float(timeout), left) if timeout else left + reply = self._operate("exec", None if timeout is None else float(timeout) + 300, command=line, + cwd=str(self.path(cwd or "")), env={str(k): str(v) for k, v in (env or {}).items()}, + timeout_sec=timeout, stdin=stdin, user=user, by_fd=dict(by_fd or {}), + **({"stream": str(stream)} if stream else {}), + **({"secret_env": [str(n) for n in secret_env]} if secret_env else {})) + return ExecResult(code=int(reply.get("code", -1)), output=str(reply.get("output") or ""), + timed_out=bool(reply.get("timed_out"))) + + def _operate(self, op: str, timeout, **fields) -> dict: + """One of the agent's operations: a failure the environment's (its driver's, Docker's) is EnvironmentUnavailable + (infra), as the Harbor runner took a trial's environment failing under its agent.""" + try: + return self._live().request(op, timeout, **fields) + except EnvironmentUnavailable: + raise + except (RuntimeError, TimeoutError) as exc: + if getattr(exc, "kind", None) == "AgentPhaseOver": + raise + raise EnvironmentUnavailable(f"{op} failed in the task's environment: {exc}"[-3000:], + errors.start_cause(getattr(exc, "kind", None), str(exc))) from None + + def upload(self, source, target) -> PurePosixPath: + """`source` to `target` (root's copy, given to the agent user: the runner's provisioning).""" + dest = self.path(target) + self._operate("upload", 1800, source=str(Path(source).absolute()), target=str(dest)) + return dest + + def download(self, source, target) -> Path: + target = Path(target) + self._operate("download", 1800, source=str(self.path(source)), target=str(target.absolute())) + return target + + def secret(self, paths) -> None: + """Files there that hold a credential (a CLI home's sign-in): removed from the environment when the agent's + phase ends, however it ends, before any verifier; a removal not confirmed fails the trial.""" + self._operate("secret", 60, paths=[str(self.path(p)) for p in paths]) + + def install(self, cli: str, version: str | None = None) -> str | None: + """The CLI Harbor's setup installed before the agent phase (Agent.installs asked for it): its version. Nothing + installs during the phase: one not asked for is refused.""" + installed = self.started.get("installed") or {} + if cli not in installed: + raise Unsupported(f"{cli} was not installed at the environment's setup (the agent's installs name what " + "Harbor's setup installs before its phase): give its path there as `bin`") + return installed[cli] + + def ensure(self, packages) -> None: + """System packages: installed at the environment's setup when the agent asked for them; none installs during + the phase (the caller checks what is there).""" + return None + + def verify(self, agent_logs: Path | str | None = None) -> dict: + """The agent's phase ended (the bridge ends its commands still running), Harbor's trial run on to its end (its + verifier on the agent's work); the verdict (HarborEnvironments.verdict). `agent_logs`, a folder of this + machine, becomes the trial's agent logs (/logs/agent), which a verifier may read.""" + if self.verified is not None: + return self.verified + driver = self._live() + # Harbor's own verifier timeout (its overrides and multiplier) ends a verifier; this side waits for that and the + # verifier's preparation and the logs beside it (verify_timeout_s overrides) + own = self.started.get("verifier_timeout_sec") + # a separate verifier's environment starts first, within Harbor's build timeout + build = float(self.started.get("build_timeout_sec") or 0) if self.started.get("verifier_mode") == "separate" \ + else 0.0 + watchdog = float(self.owner.settings.get("verify_timeout_s") or + (float(own) + build + VERIFY_ALLOWANCE_S if own else VERIFY_TIMEOUT_S + build)) + try: # the agent phase ends; Harbor's trial runs on to its end + verified = driver.request("done", watchdog, + agent_logs=str(Path(agent_logs).absolute()) if agent_logs else None) + except (RuntimeError, TimeoutError) as exc: # the driver's or this side's, not Harbor's verifier's + said = f"{getattr(exc, 'kind', None) or type(exc).__name__}: {exc}"[-3000:] + verified = {"rewards": None, "exception": {"type": errors.DRIVER_ERROR, "message": said}} + verified["trial_dir"] = self.started.get("trial_dir") + self.verified = self.owner.verdict(verified) + return self.verified + + +__all__ = ["EnvironmentUnavailable", "HarborEnvironment", "HarborEnvironments", "settings_of"] diff --git a/hermit/harbor/docker.py b/hermit/environment/harbor_docker.py similarity index 97% rename from hermit/harbor/docker.py rename to hermit/environment/harbor_docker.py index c9a7028f..12a78094 100644 --- a/hermit/harbor/docker.py +++ b/hermit/environment/harbor_docker.py @@ -1,6 +1,6 @@ -"""The Docker daemon as the Harbor runner sees it: the data root's free space (the disk gate), a task's prebuilt -image and its removal, a trial's compose projects (its environment's, its separate verifiers') taken down after a -kill. +"""The Docker daemon as a Harbor environment sees it (hermit.environment.harbor): the data root's free space (the disk +gate), a task's prebuilt image and its removal, a trial's compose projects (its environment's, its separate +verifiers') taken down after a kill. Nothing here touches an image or a container that is not the task's own: every removal names the exact image reference a task.toml declares, runs under a per-image file lock, and is a quiet no-op when Docker refuses it diff --git a/hermit/environment/harbor_driver.py b/hermit/environment/harbor_driver.py new file mode 100644 index 00000000..a12b4874 --- /dev/null +++ b/hermit/environment/harbor_driver.py @@ -0,0 +1,905 @@ +"""One task's Harbor trial, served to Hermit: run under Harbor's own interpreter, it imports harbor and the standard +library only (never hermit), and speaks JSON lines on its stdin and stdout. + + /bin/python -I harbor_driver.py # hermit.environment.harbor starts it, one per task + +Harbor runs the whole trial with its own `Trial.run()`: the environment's start (the prebuilt image pulled or the +Dockerfile built, its build timeout and multiplier, the healthcheck), Harbor's capability check (a task's skills or +MCP servers, a Windows container: refused for this bridge), +the agent's setup (under Harbor's setup timeout, the baseline network policy, the agent user), the agent phase (the +task's agent network policy, its user, its timeout), the verifier (`[verifier] environment_mode` shared or separate, +the tests uploaded only now, its timeout), the scrub of the trial's files, result.json and the environment's stop. +The agent in it is HermitBridge, a Harbor agent with no logic of its own (an external agent, as Harbor calls one): +its setup installs what Hermit asked for with Harbor's own installers, and its run relays Hermit's agent commands to +the environment until Hermit says it is done, or Harbor's agent timeout or a cancellation ends it; either way it ends +every command it started that still runs before it returns (what a command that ended left running, a server the +task asked for, stays, as the runner's agents left it). Harbor then verifies. + +Requests, one JSON object a line: {"id": n, "op": , ...}; each is answered, in any order, by one line +{"id": n, "ok": true, ...} or {"id": n, "ok": false, "error": ": ", "type": ""}. + + start {task_dir, trials_dir, trial_name, agent, environment, verifier, trial, installs} + -> {workdir, user, agent_timeout_sec, setup_timeout_sec, verifier_timeout_sec, trial_dir, instruction, + agent_network, installed, verifier_mode}: answered once the agent phase began (the environment up, the + installs done); a trial that ended before it is answered with its exception (its type: Harbor's, or + UnsupportedTask: a task Harbor runs for none of Hermit's agents) + exec {command, cwd, env, timeout_sec, stdin, user, by_fd} -> {code, output, timed_out}: `command` (a shell + line) in the main container, as the task's agent user unless `user` names another; with a timeout, under + `timeout` there, and `stdin` given through a file; only during the agent phase + upload {source, target}: a file or a folder of this machine to `target`, given to the agent user when it is not + root (Hermit's provisioning, as the runner's); download {source, target}: the other way + done {agent_logs} -> {rewards, exception, verifier_dir, verifier_mode, trial_dir}: the agent phase ends (the + process group of each command still running killed, the registered credential files removed, + `agent_logs`, a folder of this machine, copied scrubbed into the trial's agent logs, /logs/agent, which a + verifier may read), and the trial runs on to its end + secret {paths}: credential files there (a CLI home's sign-in), removed at the agent phase's end + stop -> {result}: a trial still running is cancelled (Harbor's own teardown); the driver exits + +A variable `exec` passes reaches a local Docker environment's `docker compose exec` by name (`-e NAME`, its value in +the compose client's own environment): never on a command line of this machine. The driver ends with the trial +stopped when its stdin closes (Hermit's process ended); a driver killed by a signal leaves the containers up, which +Hermit takes down by the trial's compose project names (hermit.environment.harbor_docker.compose_down). +""" +from __future__ import annotations + +import asyncio +import contextlib +import json +import os +from pathlib import Path +import shlex +import shutil +import stat +import sys +import tempfile +import time +import traceback +import uuid + +from harbor.agents.base import BaseAgent +from harbor.agents.factory import AgentFactory +from harbor.models.agent.name import AgentName +from harbor.models.task.verifier_mode import resolve_task_verifier_mode +from harbor.models.trial.config import AgentConfig, EnvironmentConfig, TaskConfig, TrialConfig, VerifierConfig +from harbor.models.trial.paths import EnvironmentPaths +from harbor.trial.trial import Trial + +MODULE = "hermit_harbor_driver" # this module's name for Harbor's agent import (AgentConfig.import_path) +BRIDGE = f"{MODULE}:HermitBridge" +UNSUPPORTED = ("does not support", "cannot forward HTTP MCP") # Harbor's capability refusals of an agent +VERSION_BEGIN, VERSION_END = "hermit-version-begin", "hermit-version-end" # around a mounted binary's --version +# Where TLS clients find the system's CA bundle (the runner's harborside.codex) +CA_BUNDLES = ("/etc/ssl/certs/ca-certificates.crt", "/etc/pki/tls/certs/ca-bundle.crt", "/etc/ssl/ca-bundle.pem", + "/etc/pki/tls/cacert.pem", "/etc/ssl/cert.pem") + + +def host_ca_bundle() -> Path | None: + """The CA bundle OpenSSL trusts on this machine, else certifi's; None when there is neither.""" + import ssl + paths = ssl.get_default_verify_paths() + for name in (os.environ.get(paths.openssl_cafile_env or ""), paths.cafile, paths.openssl_cafile): + if name and Path(name).is_file(): + return Path(name).resolve() + try: + import certifi + except ImportError: + return None + return Path(certifi.where()) + +TRIAL_FIELDS = ("verifier_timeout_multiplier", "environment_build_timeout_multiplier", "agent_timeout_multiplier", + "agent_setup_timeout_multiplier", "timeout_multiplier") +AGENT_FIELDS = ("override_timeout_sec", "override_setup_timeout_sec", "max_timeout_sec", "extra_allowed_hosts") +OWNER_MARK = "hermit-owner:" # the agent user's uid:gid line (the runner's OWNER_LINE) +WORKDIR_MARK = "hermit-workdir:" # the line the working directory's probe prints it on (shell noise aside) +KILL_GRACE_S = 10 # `timeout -k`: after the TERM, the KILL +BACKSTOP_S = 60 # Harbor's own exec timeout past the command's, should `timeout` not end it +# Debian 11 (bullseye) images: the bullseye pools on deb.debian.org no longer serve the release's packages, so an +# install fails before the CLI starts. Point main and security at archive.debian.org, accept its expired Release files, +# refresh the lists, once per environment; other images are untouched (the Harbor runner's _repair_apt_sources). +APT_REPAIR = ("if grep -qs 'VERSION_CODENAME=bullseye' /etc/os-release && command -v apt-get >/dev/null 2>&1; " + "then sed -i " + "-e 's|http://deb.debian.org/debian-security|http://archive.debian.org/debian-security|g' " + "-e 's|http://security.debian.org/debian-security|http://archive.debian.org/debian-security|g' " + "-e 's|http://deb.debian.org/debian |http://archive.debian.org/debian |g' /etc/apt/sources.list " + "/etc/apt/sources.list.d/*.list 2>/dev/null; " + "echo 'Acquire::Check-Valid-Until \"false\";' > /etc/apt/apt.conf.d/99hermit-archive; " + "apt-get update; fi") + + +TOKEN = "hermit-mark-" # each agent command's own word on its command line (bash's $0) +PIDS = "/tmp/hermit-pid-" # where each running command's process group leader is written (the runner's) + + +def kill_groups(pattern: str) -> str: + """A shell line (run as root) that kills the process group of each agent command whose pid file `pattern` + matches, as the runner's _stop_session killed a session's (each command runs under setsid: its group holds what + it started), the pid taken only when that process's command line holds the command's own word (never another + process a file names); then one check that no process of those groups is left but as a zombie or dead (exit 1 if + one is, within 5 s): a container's PID 1 need not reap them. POSIX sh throughout (`kill -KILL -`, no `--`, + which dash's kill reads as a pid), whatever shell runs it.""" + return (f"t='{TOKEN[:-1]}'; groups=''; files=''; for f in {pattern}; do [ -f \"$f\" ] || continue; " + f"m=${{f#{PIDS}}}; p=$(tr -cd '0-9' < \"$f\"); [ -n \"$p\" ] || {{ rm -f -- \"$f\"; continue; }}; " + "c=$(tr '\\0' ' ' < /proc/$p/cmdline 2>/dev/null); " + "case \"$c\" in *\"$t-$m\"*) kill -KILL -$p 2>/dev/null; groups=\"$groups $p\"; " + "files=\"$files $f\";; *) rm -f -- \"$f\";; esac; done; " + "n=0; while [ $n -lt 25 ]; do left=; for s in /proc/[0-9]*/stat; do l=$(cat $s 2>/dev/null) || continue; " + "l=${l##*) }; set -- $l; for g in $groups; do [ \"$3\" = \"$g\" ] && [ \"$1\" != Z ] && [ \"$1\" != X ] " + "&& left=1; done; done; if [ -z \"$left\" ]; then for f in $files; do rm -f -- \"$f\"; done; exit 0; fi; " + "sleep 0.2; n=$((n+1)); done; exit 1") # a group not confirmed gone keeps its file: tried again + + +ENV_PREFIX = "HERMIT_ENV_" +NAME = __import__("re").compile(r"[A-Za-z_][A-Za-z0-9_]*\Z") + + +def container_prologue(values: dict, by_fd: dict) -> tuple[str, dict]: + """(the shell lines that set a command's variables in the container, the variables that carry their values + there): each value travels under a name of this driver's (HERMIT_ENV_), the lines export it under its own name + (or, for a secret `by_fd` names, move it into a pipe on a descriptor of its own, its variable naming the + descriptor, as hermit.environment.docker's by_fd) and unset the carrier, before the command is exec'd: no process + that outlives the lines (`timeout`, the CLI) holds a by_fd secret in its environment.""" + lines, carried, fd = [], {}, 3 + for i, (name, value) in enumerate(values.items()): + if not NAME.match(name): + raise ValueError(f"not a variable name: {name!r}") + carrier = f"{ENV_PREFIX}{i}" + carried[carrier] = value + if name in by_fd: + if not NAME.match(str(by_fd[name])): + raise ValueError(f"not a variable name: {by_fd[name]!r}") + lines.append(f"exec {fd}< bytes: + """`data` with each of `secrets` replaced, as Harbor's trial scrubs its files (text only: a binary file is left).""" + if not secrets or b"\0" in data[:8192]: + return data + for secret in sorted(secrets, key=len, reverse=True): + data = data.replace(secret.encode(), b"[REDACTED]") + return data + + +def scrub_tree(root: Path, secrets) -> None: + """Every regular file under `root` scrubbed of `secrets` (links never followed).""" + if not secrets: + return + for path in root.rglob("*"): + try: + if path.is_symlink() or not path.is_file(): + continue + data = path.read_bytes() + clean = scrubbed(data, secrets) + if clean != data: + path.write_bytes(clean) + except OSError as exc: + log(f"scrub: {path}: {exc}") + + +def safe_copy(source: Path, target: Path, secrets=()) -> None: + """`source`'s files under `target`, never through a link there, however the agent's environment changes the folder + it shares with this machine (/logs/agent) meanwhile: every step goes through a directory descriptor opened without + following a link (O_NOFOLLOW), an entry where a folder goes is moved aside for one of this process's own, and a + file is written to a fresh name in its folder (O_EXCL) and renamed over the entry there, which replaces a link + instead of writing through it. `target` itself is this machine's (the trial's folder, the mount's root).""" + flags = os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW + + def aside(parent: int, name: str) -> None: # an entry in the way, never followed: renamed away + with contextlib.suppress(FileNotFoundError): + os.rename(name, f".hermit-gone-{uuid.uuid4().hex}", src_dir_fd=parent, dst_dir_fd=parent) + + def folder_in(parent: int, name: str) -> int: + try: + return os.open(name, flags, dir_fd=parent) + except OSError: # a link, a file, nothing: a folder of this process's own + aside(parent, name) + os.mkdir(name, 0o755, dir_fd=parent) + return os.open(name, flags, dir_fd=parent) # replaced meanwhile: raised, nothing followed + + def put(parent: int, name: str, path: Path) -> None: + temp = f".hermit-{uuid.uuid4().hex}" + if path.is_symlink(): # a link is copied as the link it is + os.symlink(os.readlink(path), temp, dir_fd=parent) + else: + # a regular file only, opened without blocking (a named pipe, a device: left out, as the runner's curator + # left them), never through a link + try: + src = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK | os.O_CLOEXEC) + except OSError: + return + if not stat.S_ISREG(os.fstat(src).st_mode): + os.close(src) + return + fd = os.open(temp, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o644, dir_fd=parent) + with os.fdopen(fd, "wb") as out, os.fdopen(src, "rb") as given: + if secrets: # the agent's credentials never reach the verifier's folder + out.write(scrubbed(given.read(), secrets)) + else: + shutil.copyfileobj(given, out) + try: + os.rename(temp, name, src_dir_fd=parent, dst_dir_fd=parent) + except OSError: # a folder in the way + aside(parent, name) + os.rename(temp, name, src_dir_fd=parent, dst_dir_fd=parent) + + target.mkdir(parents=True, exist_ok=True) + root = os.open(str(target), flags) + try: + for folder, dirs, files in os.walk(source): + rel = Path(folder).relative_to(source) + fds = [root] + try: + for part in rel.parts: + fds.append(folder_in(fds[-1], part)) + for name in [*files, *(d for d in dirs if (Path(folder) / d).is_symlink())]: + put(fds[-1], name, Path(folder) / name) + finally: + for fd in fds[1:]: + os.close(fd) + finally: + os.close(root) + + +def log(message: str) -> None: + try: + print(f"[harbor-driver {time.strftime('%H:%M:%S')}] {message}", file=sys.stderr, flush=True) + except OSError: + pass + + + +# Every Docker client this process starts (Harbor's and its own: asyncio.create_subprocess_exec), with the operation it +# serves: Harbor's collector leaves a client running when its awaiting task is cancelled, so the bridge ends the +# clients of the agent's commands before it returns, and the driver every client before a teardown on Hermit's exit. +CLIENTS: set = set() # the Docker clients this process started, each its own process group +_spawn = asyncio.create_subprocess_exec + + +async def _tracked_spawn(*args, **kwargs): + kwargs.setdefault("start_new_session", True) # its own process group: docker and its compose plugin, ended + process = await _spawn(*args, **kwargs) # together + for known in [p for p in CLIENTS if p.returncode is not None and not group_alive(p)]: + CLIENTS.discard(known) + CLIENTS.add(process) + return process + +asyncio.create_subprocess_exec = _tracked_spawn +_communicate = asyncio.subprocess.Process.communicate + + +async def _communicate_or_kill(self, input=None): + """Process.communicate, the client killed when the wait for it is cancelled (Harbor's collector leaves it + running): a compose up its build timeout gave up on, a transfer or an exec the phase's end cancelled, never left + to finish after the teardown or the verifier began.""" + try: + return await _communicate(self, input) + except asyncio.CancelledError: + kill_group(self) + raise + + +def kill_group(process) -> None: + """The client's whole process group killed (docker runs its compose plugin as a child, which would go on with an + up or a cp after its parent alone was killed).""" + with contextlib.suppress(ProcessLookupError, PermissionError): + os.killpg(process.pid, 9) + with contextlib.suppress(ProcessLookupError): + process.kill() + + +def group_alive(process) -> bool: + """Whether a process of the client's group still runs (a zombie, not reaped yet, does not).""" + try: + os.killpg(process.pid, 0) + except (ProcessLookupError, PermissionError): + return False + for stat_file in Path("/proc").glob("[0-9]*/stat"): + try: + fields = stat_file.read_text().rsplit(")", 1)[1].split() + except (OSError, IndexError): + continue + if len(fields) > 2 and fields[2] == str(process.pid) and fields[0] != "Z": # its state, its group + return True + return False + +asyncio.subprocess.Process.communicate = _communicate_or_kill +END_PHASE_S = 300 # the end of the agent phase's own bound + + +async def end_clients() -> None: + """Every client still running killed with its process group, and waited for (before a teardown on Hermit's + exit).""" + for process in list(CLIENTS): + if process.returncode is None or group_alive(process): + kill_group(process) + with contextlib.suppress(Exception): + await asyncio.wait_for(process.wait(), 30) + CLIENTS.discard(process) + + +class HarborDriverError(RuntimeError): + """The bridge's own failure in the trial (a command it could not confirm ended, the working directory unread): a + trial Harbor then records with this exception and does not verify, which Hermit reads as infra (harbor_failed).""" + + +class Failed(RuntimeError): + """A request's failure with the type its reply names (`kind`).""" + + def __init__(self, kind: str, message: str): + super().__init__(message) + self.kind = kind + + +DRIVER: "Driver | None" = None # the process's one driver, which the bridge reaches + + +class HermitBridge(BaseAgent): + """Harbor's agent for a Hermit agent: no logic of its own (module docstring). Its capabilities are Harbor's + defaults, so Harbor itself refuses a Windows task, a task's skills or MCP servers for it.""" + + @staticmethod + def name() -> str: + return "hermit" + + def version(self) -> str | None: + return None + + async def setup(self, environment) -> None: + await DRIVER.setup(environment) + + async def run(self, instruction: str, environment, context) -> None: + await DRIVER.phase(instruction, environment) + + +class Driver: + def __init__(self, out): + self.out = out + self.trial: Trial | None = None + self.run_task: asyncio.Task | None = None + self.installs: list = [] + self.installed: dict = {} + self.bins: dict = {} # cli -> whether its mounted binary serves (setup's word) + self.phase_started: asyncio.Future = asyncio.get_running_loop().create_future() + self.done = asyncio.Event() + self.agent_logs: str | None = None + self.env = None # the environment, while the agent phase is on + self.in_phase = False + self.user = None + self.owner: str | None = None # the agent user's uid:gid when not root (the runner's _owner) + self.workdir: str | None = None # the workspace there (the agent phase's working directory) + self.has_timeout: bool | None = None + self.mark = uuid.uuid4().hex # this environment's: each command's pid file starts with it + self.agent_ops: set = set() # the agent's operations in flight + self.transfers = asyncio.Lock() # a transfer at a time, never beside the phase's end + self.secrets: set = set() # the credentials the agent's commands were given (scrubbed) + self.secret_files: set = set() # files there holding a credential: removed at the phase's end + self.exceptions: list = [] # every exception the trial recorded or skipped, in order + self.repaired = False + self.lock = asyncio.Lock() # one request that changes the trial's phase at a time + + def reply(self, rid, payload: dict) -> None: + self.out.write(json.dumps({"id": rid, **payload}, default=str) + "\n") + self.out.flush() + + # --- the trial ------------------------------------------------------------------------------------------------ + async def start(self, req: dict) -> dict: + if self.trial is not None: + raise RuntimeError("the trial is already started") + agent = {k: v for k, v in (req.get("agent") or {}).items() if k in AGENT_FIELDS and v is not None} + fields = {k: v for k, v in (req.get("trial") or {}).items() if k in TRIAL_FIELDS and v is not None} + config = TrialConfig(task=TaskConfig(path=Path(req["task_dir"])), trial_name=req["trial_name"], + trials_dir=Path(req["trials_dir"]), agent=AgentConfig(import_path=BRIDGE, **agent), + environment=EnvironmentConfig(**(req.get("environment") or {})), + verifier=VerifierConfig(**(req.get("verifier") or {})), job_id=uuid.uuid4(), **fields) + try: + trial = await Trial.create(config) + except Exception as exc: # Harbor's refusal of the task before any trial + if "windows" in str(exc).lower() and "support" in str(exc).lower(): + raise Failed("UnsupportedTask", str(exc)) from None + raise + recorded = trial._record_exception + + def record(exc, _recorded=recorded): # Harbor keeps the first; the driver keeps them all + self.exceptions.append(exc) + _recorded(exc) + trial._record_exception = record + if trial.task.has_steps: + raise ValueError(f"{trial.task.name} has [[steps]]: a multi-step task is not served here") + self.trial, self.installs = trial, list(req.get("installs") or []) + self.run_task = asyncio.create_task(trial.run()) + await asyncio.wait({self.run_task, self.phase_started}, return_when=asyncio.FIRST_COMPLETED) + if self.phase_started.done() and not self.phase_started.cancelled(): + return self.phase_started.result() + with contextlib.suppress(BaseException): # the trial ended before its agent phase: its exception + self.run_task.result() + info = trial.result.exception_info + kind = getattr(info, "exception_type", None) or "TrialEnded" + message = getattr(info, "exception_message", None) or "the trial ended before its agent phase" + if any(word in message for word in UNSUPPORTED): + kind = "UnsupportedTask" + raise Failed(kind, message) + + async def setup(self, environment) -> None: + """Harbor's agent setup (its timeout, the baseline network policy, the agent user): each CLI Hermit asked + for, by Harbor's own installed agent of that name (unless its probe finds it there and no version is + pinned), and the system packages asked for, by Harbor's own installer.""" + for item in self.installs: + if item.get("home") or item.get("deploy"): # the bridge's own provisioning, at the phase's start and + continue # on a deployment's upload + if item.get("bin"): # a binary of this machine the environment mounts + served = await self._use_mounted(environment, item) + self.bins[str(item["agent"])] = served + if served: + continue + elif item.get("probe") and not item.get("version"): + found = await environment.exec(str(item["probe"])) + if found.return_code == 0 and (found.stdout or "").strip(): + continue + if not self.repaired: + self.repaired = True + with contextlib.suppress(Exception): # a failure here is left to the install to report + await environment.exec(APT_REPAIR, user="root") + if item.get("agent"): + agent = self._installer(str(item["agent"]), item.get("version")) + await agent.setup(environment) + self.installed[str(item["agent"])] = agent.version() + if item.get("packages"): # best effort, as the runner's: the hooks may fail open + try: + await self._installer("codex").ensure_system_dependencies(environment, tuple(item["packages"])) + for package in item["packages"]: + self.installed[str(package)] = True + except Exception as exc: # noqa: BLE001 - the session runs; its hooks are warned + log(f"setup: {', '.join(item['packages'])} could not be installed: {type(exc).__name__}: {exc}") + + async def _use_mounted(self, environment, item: dict) -> bool: + """The mounted binary, when it runs there as the asked-for version (any, when none is asked for) and the user + the session runs as may execute it (the runner's _use_mounted): linked onto PATH, and for Codex the release's + rg when the image has none and a CA bundle when the image has none (its _after_mount); else False, and the + CLI is installed instead (the session then runs the installed one).""" + binary, version = str(item["bin"]), item.get("version") + q = shlex.quote + got = await environment.exec(f"echo {VERSION_BEGIN}; {q(binary)} --version 2>&1; c=$?; " + f"printf '\\n{VERSION_END}\\n'; exit \"$c\"", user="root") + text = got.stdout or "" + said = text.split(VERSION_BEGIN, 1)[-1].split(VERSION_END, 1)[0].strip() if VERSION_BEGIN in text else "" + found = self._installer(str(item["agent"])).parse_version(said) if got.return_code == 0 and said else None + if not found or (version and found != version): + log(f"setup: the mounted {binary} does not run here as {version or 'any version'} ({said[:200]!r}): " + "installing it") + return False + if (await environment.exec(f"test -x {q(binary)}")).return_code != 0: + log(f"setup: the mounted {binary} may not be executed here by the session's user: installing it") + return False + program = str(item.get("program") or Path(binary).name) + await environment.exec(f"mkdir -p /usr/local/bin && ln -sf {q(binary)} /usr/local/bin/{q(program)}", + user="root") + if item["agent"] == "codex": + rg = q(str(Path(binary).parent.parent / "codex-path" / "rg")) + await environment.exec(f"if ! command -v rg >/dev/null 2>&1 && [ -x {rg} ]; then mkdir -p /usr/local/bin " + f"&& ln -sf {rg} /usr/local/bin/rg; fi", user="root") + await self._ensure_ca_bundle(environment) + self.installed[str(item["agent"])] = found + return True + + async def _ensure_ca_bundle(self, environment) -> None: + """TLS roots for a mounted Codex: the image's own bundle, else this machine's copied to the Debian path (the + runner's: without them every request fails and the session serves nothing).""" + check = " || ".join(f"[ -s {path} ]" for path in CA_BUNDLES) + if (await environment.exec(check, user="root")).return_code == 0: + return + source = host_ca_bundle() + if source is None: + log("setup: the image has no CA bundle and neither has this machine: Codex may not reach its API") + return + target = CA_BUNDLES[0] + await environment.exec(f"mkdir -p {shlex.quote(str(Path(target).parent))}", user="root") + await environment.upload_file(source, target) + await environment.exec(f"chmod 644 {shlex.quote(target)}", user="root") + + def _installer(self, name: str, version=None): + logs = self.trial.paths.trial_dir / "install" / name + logs.mkdir(parents=True, exist_ok=True) + return AgentFactory.create_agent_from_name(AgentName(name), logs_dir=logs, version=version) + + async def phase(self, instruction: str, environment) -> None: + """The bridge's run: the agent phase, Hermit's commands relayed until `done`, Harbor's timeout or a + cancellation; the commands still running ended (confirmed: one that is not fails the trial before any + verifier) and Hermit's logs copied, scrubbed, into the trial's agent logs.""" + self.phase_began = time.time() # as Harbor's agent timeout began (it awaits this run) + trial = self.trial + self.env, self.user = environment, trial.task.config.agent.user + found = await environment.exec(f"printf '{WORKDIR_MARK}%s\\n' \"$PWD\"") + marked = [line[len(WORKDIR_MARK):] for line in ((found.stdout or "") + (found.stderr or "")).splitlines() + if line.startswith(WORKDIR_MARK)] + if found.return_code != 0 or not marked or not marked[-1].startswith("/"): + raise HarborDriverError(f"the environment's working directory could not be read: {found.stdout!r}"[:500]) + self.workdir = marked[-1] + said = await environment.exec(f'echo "{OWNER_MARK}$(id -u):$(id -g)"') # the user the agent runs as + owner = [line[len(OWNER_MARK):].strip() for line in ((said.stdout or "") + (said.stderr or "")).splitlines() + if line.startswith(OWNER_MARK)] + if said.return_code != 0 or not owner or not __import__("re").fullmatch(r"\d+:\d+", owner[-1]): + raise HarborDriverError(f"the task's agent user has no uid:gid here: {said.stdout!r}"[:500]) + self.owner = None if owner[-1].split(":")[0] == "0" else owner[-1] + plan = trial._network_plan(None) + self.in_phase = True + self.phase_started.set_result({ + "workdir": self.workdir, "user": self.user, "agent_timeout_sec": trial.agent_timeout_sec, + "setup_timeout_sec": trial.agent_setup_timeout_sec, "verifier_timeout_sec": trial.verifier_timeout_sec, + "build_timeout_sec": trial.environment_build_timeout_sec, + # Harbor's timer started as it called this run: the phase's absolute end, on this machine's clock + "deadline_epoch": self.phase_began + float(trial.agent_timeout_sec) if trial.agent_timeout_sec else None, + "trial_dir": str(trial.paths.trial_dir), "os": str(trial.task.config.environment.os.value), + "verifier_mode": resolve_task_verifier_mode(trial.task.config).value, "instruction": instruction, + "agent_network": str(getattr(plan.agent_phase.network_mode, "value", plan.agent_phase.network_mode)), + "installed": self.installed, "bins": self.bins}) + cancelled = False + try: + await self.done.wait() + except asyncio.CancelledError: # Harbor's agent timeout, or the trial cancelled + cancelled = True + await self._close_phase(cancelled) + + async def _close_phase(self, cancelled: bool) -> None: + """The phase's end, a task of its own awaited through any cancellation (Harbor's timeout cancels this run, and + waits for it): never cut short by the deadline it follows, bounded by its own END_PHASE_S; one that did not + finish, past its bound or for any other reason, is HarborDriverError (Harbor runs no verifier after it).""" + self.in_phase = False + ending = asyncio.ensure_future(asyncio.wait_for(self._end_phase(), END_PHASE_S)) + while not ending.done(): + try: + await asyncio.shield(ending) + except asyncio.CancelledError: + if ending.cancelled(): + break + cancelled = True + except BaseException: # noqa: BLE001 - the end's own outcome, read below + break + try: + ending.result() + except (asyncio.TimeoutError, asyncio.CancelledError): # not finished: never a verifier after it + raise HarborDriverError(f"the agent phase's end did not finish in {END_PHASE_S} s") from None + except HarborDriverError: + raise + except Exception as exc: # any other failure of it: infra too, never the agent's 0 + raise HarborDriverError(f"the agent phase's end failed: {type(exc).__name__}: {exc}") from exc + if cancelled: + raise asyncio.CancelledError() + + async def _end_phase(self) -> None: + """The end of the agent phase, at the runner's: the agent's requests in flight cancelled (their Docker + clients killed with them), no transfer running (they hold the same lock), the process group of each command + still running killed as root and checked gone (kill_groups), the credential files the agent registered removed + (each confirmed gone), and Hermit's logs copied, scrubbed, into the trial's agent logs. A command that ended + left what it started (a server the task asked for), as the runner's sessions did. A kill or a removal that + fails is HarborDriverError: Harbor records it and runs no verifier (infra).""" + for task in list(self.agent_ops): + task.cancel() + if self.agent_ops: + await asyncio.gather(*self.agent_ops, return_exceptions=True) + async with self.transfers: + try: + killed = await self.env.exec(kill_groups(f"{PIDS}{self.mark}-*"), cwd="/", user="root") + failed = None if killed.return_code == 0 else "an agent command's process group could not be ended" + except Exception as exc: # noqa: BLE001 - the credentials removed all the same + failed = f"an agent command's process group could not be ended: {type(exc).__name__}: {exc}" + try: + await self._remove_credentials() # however the kill went + except HarborDriverError as exc: + failed = failed or str(exc) + if failed: + raise HarborDriverError(failed) + await self._publish_logs() + + async def _remove_credentials(self) -> None: + if not self.secret_files: + return + listed = " ".join(shlex.quote(f) for f in sorted(self.secret_files)) + # as the agent user (the files are its own, given to it at their upload): a path the agent's process named is + # never removed past that user's permissions; rm's own failure fails, and each must be gone, a link included + gone = await self.env.exec(f"rm -f -- {listed} || exit 1; for f in {listed}; do " + "if [ -e \"$f\" ] || [ -L \"$f\" ]; then exit 1; fi; done; exit 0", cwd="/") + if gone.return_code != 0: + raise HarborDriverError("a credential file could not be removed from the environment: " + f"{((gone.stdout or '') + (gone.stderr or ''))[-300:]}") + + async def _publish_logs(self) -> None: + if self.agent_logs and Path(self.agent_logs).is_dir(): + await asyncio.to_thread(safe_copy, Path(self.agent_logs), self.trial.paths.agent_dir, self.secrets) + + async def done_op(self, req: dict) -> dict: + if self.run_task is None: + raise RuntimeError("the trial is not started") + if not self.done.is_set(): + self.agent_logs = req.get("agent_logs") + self.done.set() + failed = None + try: + await asyncio.shield(self.run_task) + except BaseException as exc: # noqa: BLE001 - the trial itself raised (its finalize) + failed = exc + trial = self.trial + scrub_tree(trial.paths.trial_dir, self.secrets) # Harbor's scrub ran in its trial; the agent's own too + try: + result = json.loads(trial.paths.result_path.read_text()) + except (OSError, ValueError): # result.json not written (a full volume): the trial's own + result = json.loads(trial.result.model_dump_json()) if trial.result is not None else {} + rewards = (result.get("verifier_result") or {}).get("rewards") + info = result.get("exception_info") or None + if failed is not None and not rewards and not info: + raise HarborDriverError(f"the trial ended with no verdict: {type(failed).__name__}: {failed}") + + def said(kind, message): + return {"type": kind, "message": scrubbed(str(message or "").encode(), self.secrets).decode()} + exception = said(info.get("exception_type"), info.get("exception_message")) if info else None + # every exception after the one Harbor kept (it records the first): a verifier that failed after the agent's + # timeout is read by its own policy + later = [said(type(e).__name__, str(e)) for e in self.exceptions[1:]] + return {"rewards": rewards or None, "exception": exception, "later": later, + "agent_dir": str(trial.paths.agent_dir), + "verifier_dir": str(trial.paths.verifier_dir), + "verifier_mode": resolve_task_verifier_mode(trial.task.config).value, + "trial_dir": str(trial.paths.trial_dir)} + + async def stop(self, req: dict | None = None) -> dict: + """A trial still running cancelled (Harbor's own teardown: its environment down, result.json written), the + clients of its start ended first (a compose up never outlives the down); never behind another request.""" + task = self.run_task + if task is not None and not task.done(): + await end_clients() + task.cancel() + with contextlib.suppress(BaseException): + await task + if not self.phase_started.done(): # a start waiting on it: answered with the stop + self.phase_started.cancel() + await end_clients() + if self.trial is not None: + scrub_tree(self.trial.paths.trial_dir, self.secrets) + return {"result": str(self.trial.paths.result_path)} + return {"result": None} + + # --- the agent's operations --------------------------------------------------------------------------------- + async def _timeout_ok(self) -> bool: + if self.has_timeout is None: + found = await self.env.exec("command -v timeout >/dev/null 2>&1 && echo yes", user="root") + self.has_timeout = "yes" in (found.stdout or "") + return self.has_timeout + + def _note_secrets(self, req: dict) -> None: + """The credentials an agent command is given, kept to scrub: a value of a variable Harbor names as a secret, + or one passed on a descriptor, as Harbor's own scrub reads its agents' env.""" + from harbor.utils.env import is_bool_env_value, is_sensitive_env_key + by_fd = req.get("by_fd") or {} + named = set(req.get("secret_env") or []) # the binding's own key variable, whatever its name + for key, value in (req.get("env") or {}).items(): + value = str(value or "") + if value and (key in by_fd or key in named or is_sensitive_env_key(str(key))) and \ + not is_bool_env_value(value): + self.secrets.add(value) + + async def exec(self, req: dict) -> dict: + self._note_secrets(req) + return await self._run_command(req, str(req["command"]), req.get("timeout_sec"), req.get("user", None), + f"{self.mark}-{uuid.uuid4().hex[:8]}") + + async def _run_command(self, req: dict, command: str, timeout, user, mark: str) -> dict: + """The command under setsid in a process group of its own (the runner's launch), its leader's pid in a file + until it ends (what the phase's end, or its own timeout, kills the group from), its word on its command + line; a stream it writes kept as it runs in the agent logs under its name (/logs/agent: the runner's stream + lived there too).""" + env = self.env + staged = None + if req.get("stdin") is not None: + staged = f"/tmp/hermit-stdin-{uuid.uuid4().hex}" + with tempfile.NamedTemporaryFile("w", delete=False, encoding="utf-8") as f: + f.write(req["stdin"]) + local = f.name + try: + await env.upload_file(local, staged) + finally: + os.unlink(local) + await env.exec(f"chmod 644 {shlex.quote(staged)}", user="root") + command = f"( {command} ) < {shlex.quote(staged)}" + if req.get("stream"): + logged = shlex.quote(f"{EnvironmentPaths.agent_dir}/{Path(str(req['stream'])).name}") + command = f"set -o pipefail; {{ {command} ; }} | {{ tee {logged} 2>/dev/null; true; }}" + q, pids = shlex.quote, shlex.quote(f"{PIDS}{mark}") + # its leader stays this bash (with the word on its command line), never exec'd into the command's last one + inner = f"bash -c {q(command + chr(10) + 'exit $?')} {TOKEN}{mark}" + group = (f"if command -v setsid >/dev/null 2>&1; then setsid {inner} & else set -m; {inner} & fi; " + f"pid=$!; echo $pid > {pids}; wait $pid; code=$?; rm -f {pids}; exit $code") + outer = None + if timeout: + timeout = max(1, int(float(timeout) + 0.999)) + if await self._timeout_ok(): + group = f"timeout -k {KILL_GRACE_S} {timeout} bash -c {q(group)}" + outer = timeout + KILL_GRACE_S + BACKSTOP_S + else: + outer = timeout + group = f"bash -c {q(group)}" + else: + group = f"bash -c {q(group)}" + started = time.time() + try: + result = await self._exec(group, cwd=req.get("cwd"), env=dict(req.get("env") or {}), + by_fd=req.get("by_fd") or {}, timeout=outer, user=user) + code = result.return_code + output = (result.stdout or "") + (result.stderr or "") + except Exception as exc: # Harbor's own client timeout (no `timeout` in the image) + if not (timeout and time.time() - started >= timeout - 1): + raise + code, output = -1, f"{type(exc).__name__}: {exc}" + finally: + if staged is not None: + with contextlib.suppress(Exception): + await env.exec(f"rm -f {shlex.quote(staged)}", user="root") + timed_out = bool(timeout) and (code in (124, 137, -1) and time.time() - started >= timeout - 1) + if timed_out: # its group ended now (else at the phase's end, from its file) + with contextlib.suppress(Exception): + await env.exec(kill_groups(f"{PIDS}{mark}"), cwd="/", user="root") + return {"code": -1 if timed_out else code, "output": output, "timed_out": timed_out} + + async def _exec(self, command: str, *, cwd, env: dict, by_fd: dict, timeout, user): + """`command` (a shell line that execs the command: `timeout ...` or `bash -c ...`) in the main container, + its variables delivered there and nowhere on this machine (container_prologue): no variable the request names + touches this process's environment or the compose client's own (PATH, DOCKER_HOST stay this machine's), and + none is on a command line.""" + environment = self.env + try: + from harbor.environments.docker.docker import (DockerEnvironment, + _sanitize_docker_compose_project_name) + except ImportError: # noqa: BLE001 + DockerEnvironment, _sanitize_docker_compose_project_name = (), None + values = {str(k): str(v) for k, v in ((environment._merge_env(env) or {}) if env else {}).items()} + prologue, carried = container_prologue(values, by_fd) + line = prologue + "exec " + command + if isinstance(environment, DockerEnvironment) and not environment._is_windows_container: + # DockerEnvironment._compose_exec and _run_docker_compose_command, with the request's variables under + # names of this driver's own (HERMIT_ENV_), passed by name to `docker compose exec` from a private + # copy of the client's environment + workdir = cwd or environment.task_env_config.workdir + argv = ["exec", *(["-w", str(workdir)] if workdir else [])] + for name in carried: + argv += ["-e", name] + resolved = environment._resolve_user(user) + if resolved is not None: + argv += ["-u", str(resolved)] + argv += ["main", *environment._platform.exec_shell_args(line)] + runtime = type(environment).runtime() + full = [*runtime.compose, "--project-name", _sanitize_docker_compose_project_name(environment.session_id)] + if runtime.supports_compose_project_directory: + full += ["--project-directory", str(environment.environment_dir.resolve().absolute())] + for path in environment._docker_compose_paths: + full += ["-f", str(path.resolve().absolute())] + client = {**environment._compose_env_vars(include_os_env=True), **carried} + process = await asyncio.create_subprocess_exec( + *full, *argv, env=client, stdin=asyncio.subprocess.DEVNULL, stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.STDOUT, + cwd=str(environment.environment_dir.resolve().absolute()) if environment.environment_dir.is_dir() + else None) + return await DockerEnvironment._collect_buffered_output(process, timeout_sec=timeout) + return await environment.exec(line, cwd=cwd, env=carried or None, timeout_sec=timeout, user=user) + + async def _own(self, target: str) -> None: + """What Hermit provisioned (an upload lands as root's), the agent user's: its uid:gid as the environment says it + (an image's own USER too), the chown confirmed (the runner's _owner and its chown).""" + if self.owner is not None: + done = await self.env.exec(f"chown -R {shlex.quote(self.owner)} {shlex.quote(target)}", user="root") + if done.return_code != 0: + raise RuntimeError(f"{target} could not be given to the agent user {self.owner}: " + f"{(done.stdout or '')[-300:]}") + + async def upload(self, req: dict) -> dict: + """Hermit's provisioning (the runner's): root's copy, given to the agent user.""" + source, target = Path(req["source"]), str(req["target"]) + async with self.transfers: + await self.env.exec(f"mkdir -p {shlex.quote(str(Path(target).parent))}", user="root") + if source.is_dir() and not source.is_symlink(): + await self.env.upload_dir(source, target) + else: + await self.env.upload_file(source, target) + await self._own(target) + return {} + + async def download(self, req: dict) -> dict: + source, target = str(req["source"]), Path(req["target"]) + async with self.transfers: + # never through a link the agent left (a hook audit, its notes): the runner refused one at the notes' folder + link = await self.env.exec(f"[ -L {shlex.quote(source)} ]", user="root") + if link.return_code == 0: + raise Failed("PermissionError", f"{source} is a link: never followed out of the environment") + target.parent.mkdir(parents=True, exist_ok=True) + if await self.env.is_dir(source, user="root"): + await self.env.download_dir(source, target) + else: + await self.env.download_file(source, target) + return {} + + async def handle(self, req: dict) -> None: + rid, op = req.get("id"), req.get("op") + try: + if op == "stop": # never behind a start or a done that waits on the trial + payload = await self.stop(req) + elif op in ("start", "done"): + async with self.lock: + payload = await getattr(self, "done_op" if op == "done" else op)(req) + elif op == "secret": # a credential file there: removed at the phase's end + self.secret_files.update(str(p) for p in req.get("paths") or []) + payload = {} + elif op in ("exec", "upload", "download"): + if not self.in_phase: + raise Failed("AgentPhaseOver", "the agent's phase is not on (not begun, or ended)") + task = asyncio.current_task() + self.agent_ops.add(task) + try: + payload = await getattr(self, op)(req) + except asyncio.CancelledError: # the phase ended: done, Harbor's timeout, a cancellation + self.reply(rid, {"ok": False, "error": f"AgentPhaseOver: {op}: the agent's phase ended", + "type": "AgentPhaseOver"}) + return + finally: + self.agent_ops.discard(task) + else: + raise ValueError(f"unknown op {op!r}") + self.reply(rid, {"ok": True, **payload}) + except Exception as exc: # noqa: BLE001 - the caller gets the error + log(f"{op}: {type(exc).__name__}: {exc}\n{traceback.format_exc()[-2000:]}") + kind = getattr(exc, "kind", None) or type(exc).__name__ + self.reply(rid, {"ok": False, "error": f"{kind}: {exc}"[-4000:], "type": kind}) + + +async def serve(out) -> int: + global DRIVER + sys.modules.setdefault(MODULE, sys.modules[__name__]) # Harbor imports the bridge by this name + driver = DRIVER = Driver(out) + loop = asyncio.get_running_loop() + reader = asyncio.StreamReader(limit=1 << 30) + await loop.connect_read_pipe(lambda: asyncio.StreamReaderProtocol(reader), sys.stdin) + tasks: set[asyncio.Task] = set() + stopping = None + while True: + line = await reader.readline() + if not line: + break + try: + req = json.loads(line) + except ValueError: + continue + task = asyncio.create_task(driver.handle(req)) + tasks.add(task) + task.add_done_callback(tasks.discard) + if req.get("op") == "stop": + stopping = task + break + if stopping is not None: + await stopping + else: # Hermit went away: the trial is cancelled all the same + with contextlib.suppress(Exception): + await driver.stop() + for task in list(tasks): + task.cancel() + if tasks: + await asyncio.gather(*tasks, return_exceptions=True) + return 0 + + +def main() -> int: + # The protocol's own stream: a copy of the stdout this process was given. Whatever Harbor or a library prints + # goes to stderr, never into the replies. + out = os.fdopen(os.dup(1), "w", encoding="utf-8") + os.dup2(2, 1) + try: + return asyncio.run(serve(out)) + except Exception as exc: # noqa: BLE001 - the log says why the driver died + log(f"fatal: {type(exc).__name__}: {exc}\n{traceback.format_exc()}") + return 2 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/hermit/harbor/errors.py b/hermit/environment/harbor_errors.py similarity index 54% rename from hermit/harbor/errors.py rename to hermit/environment/harbor_errors.py index d5cdd267..2adef084 100644 --- a/hermit/harbor/errors.py +++ b/hermit/environment/harbor_errors.py @@ -1,47 +1,34 @@ -"""What a Harbor trial's `result.json` (or a step's result) says about the rollout: status, failure cause, exception, -reward, and whether the run must halt. +"""What a task's Harbor environment says about a rollout: status, failure cause, exception, reward. Ported from Hermit's +Harbor runner (hermit/harbor/errors.py, before the runner was retired) with its policy unchanged; the run's verdict of +an environment (hermit.environment.harbor.HarborEnvironments.verdict) is `classify` of what Harbor's trial recorded, +then `verifier_timeout_verdict` and `verifier_verdict`, as the runner read a trial. Harbor 0.25.0's error policy (harbor/models/trial/error_policy.py) maps exception types to failure causes. It is copied here so that Hermit's side never imports Harbor: Harbor lives in its own venv (Python 3.13). An unlisted exception -type is `unclassified`. What the policy grades as the agent's result although its cause is the verifier's or the -provider's (GRADED) is the agent's here too. Harbor grades a model provider's other failures as the agent's too, a -refused sign-in or model aside (a rate limit, a lost connection or a server error, after its retries; a parameter, a -response or a budget the provider refused); Hermit keeps them infra and runs them again, since a provider that could -not serve the run, or refused its configuration, is not the agent's result. +type is `unclassified`. What the policy grades as the agent's result although its cause is the verifier's (GRADED) is +the agent's here too. status condition ok no exception, a verifier reward - timeout AgentTimeoutError (the verifier may still have run); a verifier that ran out of its time on the - agent's solution (VerifierTimeoutError: solution_timed_out, runner.verifier_timeout_verdict) - agent_error unclassified (an agent exit with an unmatched message: a CLI that exited non-zero, as one does whose - first request the provider refused for what the input made it, a prompt too long), a request of one - of Harbor's own agents that the provider refused for what it held (BadRequestError, - UnprocessableEntityError: model_provider_failed), a verifier that left no reward Harbor could read - (no reward file, an empty one, one that does not parse: verifier_failed), a session the model's - safeguards ended (agent_refused_task: runner.session_verdict), the adapter's context over the CLI's - limit beside the image's own (context_over_limit), the adapter's deployment or per-task context - failing in its child process on this machine, before any container (adapter_failed) - infra a sandbox, model provider or Harbor failure; harbor killed by the wait budget; no trial directory; - no exception and no reward; an unlisted exception whose message is the host's (HOST_FAILURE); - harborside's launch of the session failing for a reason not the agent's (launch_failed); a verifier + timeout a verifier that ran out of its time on the agent's solution (VerifierTimeoutError: + solution_timed_out; infra with verifier_timeout: infra) + agent_error a verifier that left no reward Harbor could read (no reward file, an empty one, one that does not + parse: verifier_failed); an unclassified exception whose message is not the host's + infra the environment did not come up (sandbox_failed: its build or pull, Docker, the healthcheck; the disk + gate, `disk`; the driver's own failure, harbor_failed; a wait past its budget, harbor_timeout); no + exception and no reward; an unlisted exception whose message is the host's (HOST_FAILURE); a verifier the host kept from its tests (verifier_failed: its tools never arrived, or the host's disk was full, - runner.verifier_verdict; one that ran out of its time with verifier_timeout: infra); and what the - runner reads from the session or meets before the trial: the adapter's staging failing on this - machine (staging_failed), the plan's usage limit, or an account the provider does not bill (its quota - or credit spent, a 402: usage_limit, a halt, as an API account's usage limit), a lost sign-in - (login_failed), a provider that refused the run's configuration before it served any request - (model_rejected: a key it does not take, a permission it does not give, a model it does not have; no - halt, and never run again), a session that never started or whose tools never came up - (session_failed), the adapter's tools not served (tools_failed) - -The judge (judge.py) scores the verifier's reward whenever there is one (ok, timeout, agent_error), else 0. An infra -rollout is not the agent's result: the runner marks it `unscored` and runs it again (agent.params.infra_retries; not a -cause that runs no better again, runner.NO_RETRY); the judge refuses to score one that stays so (judge.Unscored), and -the run records a failed run (judge_error). Tokens and cost are Hermit's reading of the CLI's stream (runner.session); -Harbor's own for its built-in agents. + `verifier_verdict`); a session that never started, or that the provider could not serve + (session_failed, model_provider_failed: what the acting agent raised) + +An infra rollout is not the agent's result: the run (hermit.trainer.rollout) runs the task again in a fresh +environment (environment.harbor.infra_retries; not a cause that runs no better again, NO_RETRY), marks one that stays +so `unscored`, and the environment judge refuses to score it (hermit.task.verify.Unscored): the run records a failed +run (judge_error), which a learner sees as one and `hermit eval` counts among its errors. """ from __future__ import annotations +from pathlib import Path import re FAILURE_CAUSE: dict[str, str] = {name: cause for cause, names in { @@ -58,8 +45,9 @@ "BudgetExceededError", "AgentAuthenticationError", "ModelNotFoundError", "ApiProviderResourceNotFoundError", "AuthenticationError", "PermissionDeniedError", "NotFoundError"), + # (HarborDriverError is Hermit's own: the environment's driver failed, or did not answer, outside Harbor's verifier) "harbor_failed": ("ServiceOperationsUnsupportedError", "VercelScopeError", "DspyImportError", - "MissingExtraError", "RegradeError"), + "MissingExtraError", "RegradeError", "HarborDriverError"), "unclassified": ("NonZeroAgentExitCodeError",), "agent_exhausted_budget": ("AgentTimeoutError", "ContextWindowExceededError", "OutputTokenExceededError", "ContextLengthExceededError", "OutputLengthExceededError"), @@ -72,6 +60,7 @@ "launch_failed": ("HermitLaunchError",), "context_over_limit": ("HermitContextLimitError",), }.items() for name in names} +DRIVER_ERROR = "HarborDriverError" INFRA_CAUSES = {"sandbox_failed", "model_provider_failed", "verifier_failed", "harbor_failed", "cancelled", "harbor_timeout", "disk", "stream_lost", "stream_order", "launch_failed", # read from the session's stream by the runner (runner.session_verdict), or met before the trial @@ -90,6 +79,16 @@ GRADED = {"VerifierTimeoutError", "RewardFileNotFoundError", "RewardFileEmptyError", "VerifierOutputParseError", "BadRequestError", "UnprocessableEntityError"} HALTING = {"ApiUsageLimitError", "AgentAuthenticationError"} # nothing after this trial can succeed +# the causes of an infra rollout that run no better again (the runner's NO_RETRY): a disk gate that waited its time out, +# a request the provider refused for the run's configuration (its key, permission, model or bill: no input to the +# session changes it) +NO_RETRY = ("disk", "model_rejected", "host_failed", # and this machine's own failure, which the runner never ran + "unsupported_task") # again, nor a task Harbor runs for none of Hermit's agents + # again (an OSError starting the environment's driver) +INFRA_RETRIES = 2 # environment.harbor.infra_retries' default +INFRA_WAIT_S = 300.0 # infra_wait_s' +BUSY_RETRIES = 3 # busy_retries' default: the provider's allowance +BUSY_WAIT_S = 120.0 # busy_wait_s' # An exception Harbor does not list whose message shows the host failing, not the agent: the agent's setup could not # exec in the container (the host's /tmp full, where runc writes each exec's process file; the daemon gone), or the # task's environment did not come up: Harbor's Docker environment raises a plain RuntimeError "Docker compose command @@ -172,16 +171,95 @@ def classify(result: dict | None, *, expired: bool = False, harbor_exit: int | N float(reward) if reward is not None else None, halt=exc_type in HALTING) -def adapter_failed(message: str) -> dict: - """The verdict of a rollout whose adapter failed on this machine before any container: its deployment, or its - per-task context, raised in their child process (hermit.harbor.staging, hermit.harbor.context). That is the - agent's result, scored 0 (agent_error), as a session's failure on its adapter is, its message the feedback; no - session ran, so it spent nothing (`ran` False).""" - return {**_verdict("agent_error", "adapter_failed", None, message, None), "ran": False} - - def no_trial(cause: str, message: str) -> dict: - """An infra verdict for a rollout that never reached a container (the disk gate, the adapter's staging, a stream - step its driver was never asked for): no session ran, so it spent nothing (`ran` False: its cost is 0, never - unknown). A stream step lost after its request was sent may have run one: its caller sets `ran` None.""" + """An infra verdict for a rollout that never reached its environment (the disk gate, an environment that did not + start): no session ran, so it spent nothing (`ran` False: its cost is 0, never unknown).""" return {**_verdict("infra", cause, None, message, None), "ran": False} + + +def verifier_timeout_verdict(verdict: dict, verifier_timeout: str = "agent") -> dict: + """A verifier that ran out of its time (VerifierTimeoutError) is the agent's result with `verifier_timeout: + agent`, the default, as Harbor grades it: a timeout on the agent's solution (solution_timed_out), scored its + reward or 0 and not run again (a solution too slow for its tests, as AlgoTune's verifier times it, or a process + it left that hung them). With `verifier_timeout: infra`, for a host whose load makes verifiers time out, it is + infra: run again. Either way the session did not run out of its time: `timed_out` stays false.""" + if verdict.get("exception_type") != "VerifierTimeoutError": + return verdict + if verifier_timeout == "infra": + return {**verdict, "status": "infra", "failure_cause": "verifier_failed", "timed_out": False} + return {**verdict, "status": "timeout", "failure_cause": "solution_timed_out", "timed_out": False} + + +def _text(path: Path) -> str: + """A regular file's text, or "" when it is missing, unreadable or no regular file (never read through a link).""" + try: + return path.read_text(encoding="utf-8", errors="replace") if path.is_file() and not path.is_symlink() else "" + except OSError: + return "" + + +def verifier_verdict(verdict: dict, verifier_dir: Path | None, redact=()) -> dict: + """A rollout whose verifier ran no test is infra: run again, not kept as the agent's result, whether the verifier + wrote 0, left no reward or ran out of its time. Either the verifier's tools never arrived, their download failed by + a server (a 5xx, GitHub's release download) so its test runner is missing and it wrote no report + (verifier_unreached), or the host's disk was full while it ran, before its tests wrote a report + (verifier_disk_full: with a report, the words are a test's output). A rollout the verifier passed ran its tests; + one whose verifier never ran (no output of it) stays as it was. The output quoted has `redact`'s patterns removed + (verifier_redact).""" + if verifier_dir is None or verdict["status"] == "infra" or float(verdict.get("reward") or 0) >= 1: + return verdict + out = Path(verifier_dir) + stdout = _text(out / "test-stdout.txt") + if verifier_unreached(stdout, (out / "ctrf.json").exists()): + why = "the verifier's tools never arrived (their download failed), so its tests never ran" + elif verifier_disk_full(stdout, (out / "ctrf.json").exists()): + why = "the host's disk was full while the verifier ran (ENOSPC)" + else: + return verdict + return {**verdict, "status": "infra", "failure_cause": "verifier_failed", + "exception_message": f"{why}: {_redacted(stdout, redact).strip()[-300:]}"} + + +def _redacted(text: str, patterns) -> str: + """The runner's curate.redact_verifier: each match of `patterns` replaced.""" + for pattern in patterns or (): + text = re.sub(str(pattern), "[redacted]", text) + return text + + +def start_cause(kind: str | None, message: str) -> str: + """The infra cause of an environment that did not start (Harbor's start, its healthcheck, the driver): Harbor's + policy's cause when it lists one among INFRA_CAUSES, else sandbox_failed (a trial whose environment failed before + its agent started is infra, whatever raised).""" + cause = FAILURE_CAUSE.get(kind or "", "unclassified") + if kind == "TimeoutError": + return "harbor_timeout" + if kind == "DriverGone": + return "harbor_failed" + return cause if cause in INFRA_CAUSES else "sandbox_failed" + + +# What an acting agent raises that is not its result (the runner read the same of a trial's session): an environment +# that failed under it (its driver gone, the image's context file unreadable), a session that never started, a provider +# that could not serve it after the agent's own runs again (the Codex agent's `retries`: the runner's busy_retries, +# one allowance over all of a rollout's runs, so it is not run again here), a model endpoint that failed (the answer +# agent's: a 408, 409, 429 or 5xx, or no answer, after the chat client's own retries), and a refusal of the run's +# configuration (a CLI's, read by the Codex agent's stream reader; the answer agent's endpoint's 401-404), which no +# run again changes. (cause, whether the rollout runs again). +PIPELINE_INFRA = ((re.compile(r"\bmodel_rejected\b|chat/completions HTTP 40[1-4]\b"), "model_rejected", False), + (re.compile(r"could not be served after"), "model_provider_failed", False), + (re.compile(r"before starting a session"), "session_failed", True), + (re.compile(r"chat/completions HTTP (?:408|409|429|5\d\d)\b|chat/completions failed: "), + "model_provider_failed", True), + (re.compile(r"\bEnvironmentUnavailable\b|the Harbor driver|the environment is not started|" + r"the image's own context file could not be read|" + r"the task's environment did not provide"), "harbor_failed", True)) + + +AGENT_PHASE_OVER = "AgentPhaseOver" # an agent command refused or ended because Harbor ended the agent phase + + +def pipeline_cause(message: str) -> tuple[str, bool] | None: + """(the infra cause of an acting agent's failure, whether it runs again) by PIPELINE_INFRA, or None: the + pipeline's own failure, a failed run as any.""" + return next(((cause, again) for pattern, cause, again in PIPELINE_INFRA if pattern.search(message or "")), None) diff --git a/hermit/environment/spec.py b/hermit/environment/spec.py new file mode 100644 index 00000000..68edeb2d --- /dev/null +++ b/hermit/environment/spec.py @@ -0,0 +1,73 @@ +"""What an environment must give: the paths it shows read-only and read-write, the variables that pass, the network, +the Python distributions it shows, its resources and GPUs, or a container image. A run config's `environment` +section is one; a task directory's task.toml asks for some of it ([environment] network_mode, memory_mb), which the +run's must cover: a task is never run with less than it asks, nor given more than the run declares. + + spec = EnvironmentSpec.from_section(config.data["environment"], resolve) # resolve(where, values) -> [Path] + spec.problems_for(task_row) # [] when the run's covers the task +""" +from __future__ import annotations + +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Callable, Mapping + +NETWORK_ORDER = {"no-network": 0, "allowlist": 1, "public": 2} # how much each network mode lets through + + +@dataclass(frozen=True) +class EnvironmentSpec: + read: tuple[Path, ...] = () + write: tuple[Path, ...] = () + env: tuple[str, ...] = () + network: str = "no-network" + allowed_hosts: tuple[str, ...] = () + packages: tuple[str, ...] = () + resources: Mapping[str, Any] = field(default_factory=dict) + gpu: Any = None + image: str | None = None + container: Mapping[str, Any] = field(default_factory=dict) + + @classmethod + def from_section(cls, section: Mapping[str, Any], resolve: Callable[[str, Any], list[Path]]) -> "EnvironmentSpec": + """A run config's `environment` section; `resolve(where, values)` turns its paths into paths (the config's + own resolution, which refuses an unset variable).""" + s = dict(section or {}) + return cls(read=tuple(resolve("environment.read", s.get("read"))), + write=tuple(resolve("environment.write", s.get("write"))), + env=tuple(s.get("env") or ()), network=s.get("network") or "no-network", + allowed_hosts=tuple(s.get("allowed_hosts") or ()), packages=tuple(s.get("packages") or ()), + resources=dict(s.get("resources") or {}), gpu=s.get("gpu"), image=s.get("image"), + container=dict(s.get("container") or {})) + + def problems_for(self, task: Mapping[str, Any], *, verifier: bool = False) -> list[str]: + """What a task asks of its environment (a directory task's row: `network`, `allowed_hosts`, `memory_mb`) that + this one does not give: a wider network than the run declares (for its verifier too when `verifier`: a task's + own verifier runs in a sandbox like the agent's), hosts its allowlist lacks, more memory than its limit. A row + that asks nothing (a row of tasks.jsonl) has none.""" + out = [] + modes = task.get("network") if isinstance(task.get("network"), Mapping) else {} + for phase in ("agent", "verifier") if verifier else ("agent",): + asked = modes.get(phase) + if asked in NETWORK_ORDER and NETWORK_ORDER[asked] > NETWORK_ORDER.get(self.network, 0): + out.append(f"the task's {phase} needs network {asked}; the run's environment.network is " + f"{self.network}") + elif asked == "allowlist": + hosts = task.get("allowed_hosts") or {} + hosts = list(hosts.get(phase) or []) if isinstance(hosts, Mapping) else list(hosts) + wild = [h for h in hosts if "*" in h] + missing = [h for h in hosts if h not in self.allowed_hosts and h not in wild] + if wild and self.network == "allowlist": # its proxy matches a host as written + out.append(f"the task's {phase} allows {wild}: wildcard hosts are not supported; name the hosts") + if missing and self.network == "allowlist": + out.append(f"the task's {phase} needs to reach {missing}, which environment.allowed_hosts lacks") + if task.get("steps"): # each step would start where the last one left nothing + out.append(f"the task has {task['steps']} steps, which run in one environment one after another: a " + "multi-step task is not supported here yet") + limit = self.resources.get("memory_mb") + if verifier and limit and task.get("verifier_memory_mb") and int(task["verifier_memory_mb"]) > int(limit): + out.append(f"the task's verifier needs {task['verifier_memory_mb']} MB of memory; " + f"environment.resources.memory_mb is {limit}") + if limit and task.get("memory_mb") and int(task["memory_mb"]) > int(limit): + out.append(f"the task needs {task['memory_mb']} MB of memory; environment.resources.memory_mb is {limit}") + return out diff --git a/hermit/evaluate.py b/hermit/evaluate.py index 05582e3f..8aaaec74 100644 --- a/hermit/evaluate.py +++ b/hermit/evaluate.py @@ -8,19 +8,25 @@ from hermit.adapter.manifest import CARD, MANIFEST from hermit.config import Config -from hermit.isolate import agent_pipeline, judge_for +from hermit.isolate import agent_pipeline, judge_for, task_environments from hermit.redact import redact_path, redact_value -from hermit.task import TaskData +from hermit.task import Benchmark from hermit.trainer.budget import Budget from hermit.trainer.rollout import run_tasks from hermit.tree import ROOT_TAG, Tree +def sign_in_problems(data: dict) -> list[str]: + """What stops the acting agent's sessions before any request (its login and its model's: eval never runs the + learner, whose login is not checked here).""" + from hermit.agent.base import login as subscription + return [f"{who}: {p}" for who, login, program, contained, root in subscription.logins_of(data) + if who in ("agent", "agent's model") for p in subscription.problems_of(login, program, contained, root)] + + def evaluate(config: Config, adapter: str, *, log=print) -> dict: d = config.data - data = TaskData(config.path(d["data"]["tasks"]), config.path(d["data"]["train"]), - config.path(d["data"]["val"]), config.path(d["data"]["test"])) - tests = data.rows("test") + tests = Benchmark.from_config(config).split("test") if not tests: raise SystemExit("eval needs data.test") run_name = config.run_name() @@ -48,23 +54,32 @@ def evaluate(config: Config, adapter: str, *, log=print) -> dict: pipeline, close = agent_pipeline(config, "eval", records=out / "sandbox") try: judge = judge_for(config, pipeline) + environments = task_environments(config, log=log) + if environments is not None: # the sign-in, before any result is replaced or a trial starts (the runner's) + problems = sign_in_problems(d) + if problems: + raise SystemExit("eval refuses to start: " + "; ".join(problems)) except BaseException: close() # the pipeline's sandbox and proxies, before the error leaves raise budget = Budget() t0 = time.time() - per_task: dict[str, list[float]] = {t["id"]: [] for t in tests} + per_task: dict[str, list[float]] = {t.id: [] for t in tests} errors = 0 # runs the pipeline or judge failed on (scored 0) try: with open(out / "results.jsonl", "w") as f: for r in range(1, repeats + 1): log(f"eval {label}: repeat {r}/{repeats} on {len(tests)} test tasks") for rec in run_tasks(tests, target, pipeline, judge, concurrency=int(d["agent"]["concurrency"]), - budget=budget, part="rollout"): - per_task[rec["id"]].append(rec["judge"]["score"]) + budget=budget, part="rollout", environments=environments): + per_task[str(rec["id"])].append(rec["judge"]["score"]) # Task.id: the row's id as text errors += rec["status"] != "ok" row = {"repeat": r, **{k: rec[k] for k in ("id", "status", "error", "output", "judge", "cost_usd", "wall_s", "api_speed", "tokens", "log")}} + extra = rec.get("pipeline_extra") or {} + if "verified" in extra: # a task in its own environment: the rollout's outcome, as the runner's was + row["outcome"] = {k: extra.get(k) for k in ("status", "failure_cause", "timed_out", + "exit_code", "unscored", "verifier_reward_raw")} f.write(json.dumps(redact_value(row), default=str) + "\n") # no model key is kept finally: close() diff --git a/hermit/harbor/FRAMEWORK_SUPPORT.md b/hermit/harbor/FRAMEWORK_SUPPORT.md deleted file mode 100644 index 6c83211d..00000000 --- a/hermit/harbor/FRAMEWORK_SUPPORT.md +++ /dev/null @@ -1,96 +0,0 @@ -# Harbor and the Hermit core - -The Harbor integration changes nothing in `hermit/` outside this package but one thing: Pi's and dsh's host -deployments (`hermit/hosts/pi.py`, `hermit/hosts/dsh.py`) take `serve_tools`, as the other five hosts already do, so -that a trial in a task's container gets the adapter's tools from the runner, not from a server its workspace would -start on this machine. The three agent layers (`hermit/agent/`), the Trainer, `hermit eval`, `hermit check`, the -configuration and the sandbox are unchanged. This note records how Harbor's needs fit the existing interfaces, and -what could later become framework support. Nothing here is required to run a Harbor task set. - -## How Harbor fits the existing interfaces - -| Need | Where it lives | -|---|---| -| A trial's verdict, usage and final message, from the CLI's stream in the task's container | `HarborAgent.session` (`runner.py`) copies the stream and the CLI's last message out of the trial's outputs, each read once as a regular file and never through a link, then reads the copies with the CLI Base Agent's `outcome` and `parse`, as a local session is read. | -| Codex in a trial: a lost sign-in, and a provider that could not serve | `signin.codex_signin_lost` and `signin.codex_unavailable` read a Codex trial's stream. The Base Agent's Codex reading reports no lost sign-in, since a local session that meets one fails on its own. It also takes a reconnect Codex recovered from for an outage. | -| A lost sign-in, or an API account's usage limit, ends the run | The runner raises `LoginFailed` (`hermit.agent.base.interface`), which the rollout passes up: `hermit train` stops and resumes after its last finished step, and `hermit eval` stops. The answer agent's relay turns a Codex session's lost sign-in into the endpoint's 401 with `LOGIN_FAILED`, as the endpoint already does for Claude Code. | -| Subscription plans shared by trials (Codex's ChatGPT login, Claude's token) | A trial waits for the plan, takes one of its slots and records what the session says of it with the Base Agent's login functions (`held`, `wait`, `slot`, `hold`, `note_usage`), as a local session does. | -| A sign-in a trial refreshed, copied back | `signin.take_back` checks the file first: a regular file of at most 64 KiB holding a ChatGPT sign-in's tokens, refreshed at most a minute ahead of this machine's clock (an hour for a container on another machine). Then `login.take_back` takes it. | -| The answer agent's model when only a CLI reaches it | `tools.ModelEndpoint` serves the benchmark file's `model: {cli: ...}` itself. Before each request it waits for the login's plan (`login.held`, `login.wait`), then answers through `endpoint.cli_answer`, and it counts what failed calls used from their error replies. A run config's own `{cli}` model is refused at startup: the pipeline serves it and waits for the plan inside the request, where the runner cannot tell the wait from the model's own time. | -| Checks before the first trial | `HarborAgent.check_agent`, run as the runner is built, checks the sign-in a trial is handed, web search, a model the containers can reach, bubblewrap, mini-swe-agent's package, and no secret in `acting_agent.env` (Harbor's own rule for secret names). `hermit check` builds the runner too (`agent.name: harbor` runs on the host) and reports the first of these refusals. A run config's own `{cli}` model for the answer agent is refused both as `hermit check` hands it (as written) and as `hermit eval` and `hermit train` do (the endpoint the pipeline serves it from, `PIPELINE_CLI_SPEC`, which must stay the keys of `main`'s `endpoint_spec`). | -| The acting model in a trial | `launch.binding_of` and the model layer's `bind_agent_params` bind it for the CLI. A benchmark file names a model by name, or `{cli: ...}` on the CLI's own login (one without `login` is refused: no reference is bound there); a `/` reference belongs in the run config (`check_shared_model`). | -| The adapter in the task's container | `staging.deploy_here` calls the host's own `prepare(..., context=..., serve_tools=False)` in a child process under the strict adapter sandbox. The runner serves the adapter's tools over HTTP (`tools.AdapterTools`), and the Base Agent's `prepare_home` names them for the session. A deployment or per-task context that fails in that child is the agent's result (`adapter_failed`: `agent_error`, scored 0, its message the feedback); an OSError on this machine is infra (`staging_failed`). | -| Evolver methods reading Harbor trials | `curate.py` lays out each trial's log as a local session's: the CLI's stream (`agent-.jsonl`), the hooks' `hook-calls.jsonl` and `hook-state/`, `task.md`, `prompt.txt` and `last_message.txt` at its top. The trajectory readers (`hermit.agent.base.trajectory`'s `log_steps`), the AHE Debugger, RRSI and SkillOpt read it unchanged. | -| Continual Harness telling the agent's timeout from the verifier's | `HarborAgent.verifier_timeout_verdict` never marks a verifier that ran out of time as the agent's `timed_out`. | -| Infrastructure failures (Docker, the network, the provider, Harbor itself) | `HarborAgent._run_trial` runs such a trial again, `agent.params.infra_retries` times (2) after `infra_wait_s` (300 s), unless its cause runs no better again (`NO_RETRY`: a disk gate that waited its time out, a provider's refusal of the run's configuration, `model_rejected`). One that still fails is refused by the judge (`judge.Unscored`), so the rollout records a failed run (`status: judge_error`), metered and with its log, as EvoSafe's judge does. `hermit eval` counts it as 0 in `mean_score` and among its `errors`. `python -m hermit.harbor.report ` gives pass@1 over the trials that are the agent's result, with the infrastructure failures apart. | -| Multi-step tasks (streams) | Each runner process names its own streams (`/streams/__s__`) and stops the unfinished ones when the runner is freed or at exit. A stream's steps run in the split's order (`data.shuffle: false`); a step whose turn cannot come is `stream_order`, a failed run. A step that ends in any way before its driver is asked for it (its own folder not made among them) gives its stream up, so that the driver is stopped and the stream's next pass starts anew, its driver once the earlier one has exited (`StreamClient.running` takes no driver that is ending). A resumed `hermit train` (a new runner) loses a stream's remaining steps in that epoch (`stream_order`). | -| Evaluation-only configs | `Config.validate` asks every config for `data.train`, `template` and `evolver.name`; a config that only evaluates gives placeholders. | -| Harbor's own agents (`acting_agent: {name: ...}` or `{import_path: ...}`) take no adapter | Each rollout's `run()` refuses an adapter for one (a ValueError), before anything is staged. | -| A model a CLI reaches, for the answer agent | Its sessions run on this machine in the sandbox main gives such sessions (`hermit.isolate.cli_sandbox`, role `agent-model`), the user's own secrets proven absent (`HOME_SECRETS`). | - -`hermit/harbor/` uses four private helpers of main: `hermit.adapter.session._adapter_sandbox` and -`hermit.agent.base.endpoint._DROP` (`tools.py`), `hermit.agent.acting.workspace._query` (`context.py`) and -`._session_args` (`launch.py`). `launch.check_settings` repeats `CliAgent.run`'s and `MiniSweAgent.run`'s checks of -their settings, so that a trial is refused before its container. A change of main to any of them breaks or silently changes the Harbor runner. - -There is no trial cache: a resumed eval or training step runs its trials again. - -## Candidates for framework support (open for discussion) - -The Trainer and `hermit eval`: - -1. **A run that is not a verdict.** A `RunOutput` could say it is not the agent's result (an infrastructure - failure), so that the Trainer and `hermit eval` leave it out of the mean and can run it again, instead of a task - routing it through its judge and a separate report. -2. **Evaluation-only configs**, which need no `data.train`, `template` or `evolver.name`, and an eval folder of their - own: two such configs of one evolver and template share `/eval//`, the second's results replacing - the first's. -3. **Rows that name their epoch, their repeat and who runs them.** If the Trainer handed the pipeline the epoch and - repeat of each row, a pipeline could keep its trials, and a resumed eval or training step could reuse the finished - ones; if a row said whether the Trainer's rollout or an Evolver's `run_agent` runs it, a candidate's run of a - multi-step task's step could run in a stream of its own, never in the Trainer's (out of its turn there, or a pass - of its first step that the Trainer's next epoch inherits). -4. **A run identity for the pipeline**, so that a resumed run continues its multi-step streams, instead of losing a - stream's remaining steps in the epoch it resumes. -5. **A multi-step task's step order**, checked by the Trainer rather than kept by `data.shuffle: false`. -6. **Paths a config declares**, kept off Harbor's task and job directories (the overlap check in `hermit/isolate.py`). -7. **`hermit check` and `hermit train` for Harbor task sets**: a trial's sign-in, the acting model's binding for its - CLI (`agent.name: harbor` names no CLI to `_cli_kind`, so `agent.params.model` is checked as a chat client's: - Claude Code on a backend that serves only the Messages API, Anthropic's, is refused, which `hermit eval` runs, and - Codex's need of the Responses API is not checked), and an adapter refused for Harbor's own agents; and their - refusal of a `login: claude` run with no `claude` on PATH and no `claude_bin`, though a trial runs the container's - CLI. -8. A refused `hermit eval` that keeps the earlier eval's files; `hermit check`'s check of a test row's own `repeat`; - one-line config errors. - -The agent layers: - -9. **Reading CLI streams**: lines split on `\n` alone (a message's U+2028), very deep JSON, Codex's quotes of tool - calls in its log, OpenCode's HTTP status. -10. **Codex's lost sign-in and outage**, read in the Base Agent as `signin.py` reads a trial's, so that Hermit's own - Codex sessions end the run on a lost sign-in and take a recovered reconnect for no outage. -11. **Stricter checks on a sign-in copied back** from Hermit's own sandboxed sessions. -12. **The pipeline's CLI endpoint saying that a request waited for its login's plan**, so that a run config's own - `{cli}` model could serve the answer agent; and its failed reply naming the plan's refusal before an outage - (`_no_reply` gives 503 first, so a relayed Codex session the plan refused after a reconnect notice reads - `model_provider_failed`, run again after `infra_wait_s`, not `usage_limit`, run again once the plan allows). -13. **Hosts**: Pi's tool server in the strict adapter sandbox, as the others' are; `hermit/agent/acting/workspace.py` - passing `serve_tools` to Pi and dsh for its container sessions. -14. **`Binding.spec` keeping `developer_role`**, which a spec the config's loading kept drops: the runner takes the - backends registry's (the built-ins and the user's backends file) for a backend the registry has at the trial's - URL (`launch._with_role`), so that Pi's and dsh's trials on it get the provider settings Hermit's own sessions - do. The run config's `backends:` is not the runner's to read: a backend it declares (`from:` another included) - or moves to another URL keeps the spec's `keep`, and a role it sets for a registry backend at the registry's URL - gives way to the registry's, where Hermit's own sessions take the run config's. -15. **Clocks of AHE and its Debugger** that leave out a wait for the login's plan. -16. **Harness²** for more CLIs; MWM's file names and its round count; RRSI Adapter's `halt_file`, which still names - Harbor's `/HALT`, a file the runner no longer writes. -17. **Usage of a failed session** counted for learners' `llm: {cli: ...}` calls. - -The rollout, the sandbox and `main`'s private helpers: - -18. **A pipeline's usage when `LoginFailed` ends the run**: a trial a lost sign-in ended records nothing, the usage of - its earlier runs included. -19. **The run's own secret paths** (`run_secrets`) for the sandbox of the answer agent's CLI sessions, which needs the - config the runner never gets. -20. **Public names** for the four private helpers above. diff --git a/hermit/harbor/__init__.py b/hermit/harbor/__init__.py index 1fdf497c..a5e8f6c5 100644 --- a/hermit/harbor/__init__.py +++ b/hermit/harbor/__init__.py @@ -1,33 +1,9 @@ -"""Benchmarks in Hermit: one acting agent for every task set in Harbor's format, the protocol a benchmark enters by. +"""Benchmarks in Harbor's format: the task sets Harbor's registry holds (Terminal-Bench, SWE-bench, ...), made into an +example's data, and an eval's figures read from its results. - agent: - name: harbor - params: - tasks_dir: ../utils/tasks # the Harbor task directories (task.toml, environment/, tests/) - acting_agent: {cli: codex} # the CLI in the container: codex, claude, pi, opencode, dsh, - # mini-swe-agent, or answer (one model call); or a Harbor built-in by name - model: fireworks/glm-5 # a `/` reference, or with `login` the CLI's model name - web_search: false # the CLI's web search, always set - agent_timeout_sec: 3600 # uniform agent budget; null: each task's own - jobs_dir: ~/.cache/hermit/harbor/jobs + python -m hermit.harbor.dataset prepare utils/benchmark.yaml # the download, tasks.jsonl, splits, a lock + python -m hermit.harbor.report runs//eval/ # pass@1 over the runs that are the agent's result -The runner (`runner.HarborAgent`) turns one Hermit rollout into one Harbor trial. Hermit renders the trial and -harborside carries it out: the adapter is deployed for the acting CLI (`staging`, by the CLI's own host, with the -task's context when the adapter's is per task: `context`), its tools are served from this machine (`tools`), and the -CLI's launch plan is rendered by Hermit's own CLI Base Agent (`launch`: the CLI's home, command and environment, with -`clis`, the table of what differs per CLI). Harbor then starts the task's container and the HermitXXX class runs the -plan in it (harborside/hermit.py), then the task's own verifier; the trial's reward and the CLI's own stream come back, -read as Hermit reads its own sessions (`errors` for the trial's verdict, the Base Agent's `outcome` and `parse` for -the session's, `signin` for what a trial in a container adds to them: a Codex sign-in or key refused, a refreshed -ChatGPT sign-in handed back), and `judge.score` scores it. A trial that is not the agent's result (infra) runs again -in the runner, and one that stays so gets no score (the run records a failed run, which `hermit eval` counts as 0); -`report` gives an eval's pass@1 over the trials that are the agent's result. A lost sign-in or an API account's usage -limit ends the run (hermit.agent.base.interface.LoginFailed). A task with `[[steps]]` (a stream: one environment that lives through -many steps, each with its instruction and verifier) is run one step per rollout by a driver (`stream`, -`harborside/driver.py`) that keeps the environment between the Trainer's steps, so a learner acts between them. - -`dataset` makes a benchmark's data from its benchmark file (`python -m hermit.harbor.dataset prepare`: the download, -`tasks.jsonl` rows, splits, a content lock); `curate` and `docker` are the runner's parts. `harborside/` is the half -that runs under Harbor's own interpreter (harbor, two of Harbor's own dependencies, pydantic and certifi where it is -installed, and the standard library only): the container-side agents and the stream driver. +Harbor itself gives each task its environment and its verifier (hermit.environment.harbor, `environment.harbor` in a +run config); the agent that works in it is Hermit's, any acting agent (docs/framework/task.md#benchmarks). """ diff --git a/hermit/harbor/clis.py b/hermit/harbor/clis.py deleted file mode 100644 index 6c1cc3a5..00000000 --- a/hermit/harbor/clis.py +++ /dev/null @@ -1,89 +0,0 @@ -"""The CLIs a Harbor trial runs Hermit's way, one row each: the one place the runner learns what differs per CLI. - - acting_agent: {cli: codex} # or claude, pi, opencode, dsh, mini-swe-agent, answer - -A row names the HermitXXX class Harbor imports (`harborside.:`, under Harbor's interpreter), the CLI -Base Agent (`hermit.agent.base.cli.CLIS`) whose home, command, environment and stream reader the trial uses, the -deployment's context file and the limit on it once merged with the image's own (less a byte for each occurrence in -Hermit's text of `context_escaped`, which the CLI escapes by one more), and the settings a container session needs -besides the run's (Codex's own sandbox off: the task's container is the boundary, as Harbor's own -`--dangerously-bypass-approvals-and-sandbox`; dsh's likewise). `version` is what an in-container install installs -when the config names none (the builds Hermit's tests were recorded on, docs/get-started/installation.md), not a -pin: a mounted binary (`_bin`) runs whatever it is. `nvm`: the CLI is an npm program the install put under the -agent user's nvm, sourced before the session's own HOME is set. Every CLI's home in the container is CONTAINER_HOME. - -harborside never imports this module (it runs under Harbor's interpreter: harbor, two of Harbor's own dependencies -and the standard library only): what a trial needs of a row reaches it in the trial's launch plan -(hermit.harbor.launch). -""" -from __future__ import annotations - -from hermit.agent.base.cli import DSH_INSTRUCTION_BYTES, DSH_INSTRUCTION_FRAME - -CONTAINER_HOME = "/installed-agent/hermit-home" # the CLI's home in the container, for every CLI -BIN_MOUNT = "/opt/hermit-{cli}" # where `_bin`'s folder is mounted, read-only -DEFAULT_CLI = "codex" - -HARBOR_CLIS: dict[str, dict] = { - # Harbor's install puts codex under the agent user's nvm on a glibc image (it links none into /usr/local/bin); a - # mounted codex_bin's folder still comes first on PATH, put there after nvm is loaded - "codex": {"import_path": "harborside.codex:HermitCodex", "backend": "codex", "context_file": "AGENTS.md", - "context_limit": None, # Codex's own cap is project_doc_max_bytes (launch: the limit after the merge) - "nvm": True, "settings": {"codex_sandbox": "danger-full-access", "project_doc_max_bytes": 262144}}, - "claude": {"import_path": "harborside.claude:HermitClaude", "backend": "claude", "context_file": "CLAUDE.md", - "context_limit": None, - # bypassPermissions as the container's root, and no update of the CLI mid-trial - "env": {"IS_SANDBOX": "1", "DISABLE_AUTOUPDATER": "1"}}, - "pi": {"import_path": "harborside.pi:HermitPi", "backend": "pi", "context_file": "AGENTS.md", "context_limit": None, - "nvm": True, "version": "1.5.0"}, - "opencode": {"import_path": "harborside.opencode:HermitOpenCode", "backend": "opencode", - "context_file": "AGENTS.md", "context_limit": None, "version": "2.2.6"}, - # dsh cuts the project's instructions at DSH_INSTRUCTION_BYTES, frame included, and tells only the model; in its - # frame each `` of the text is escaped by one more byte (hermit.hosts.dsh counts them too) - "dsh": {"import_path": "harborside.dsh:HermitDsh", "backend": "dsh", "context_file": "AGENTS.md", - "context_limit": DSH_INSTRUCTION_BYTES - DSH_INSTRUCTION_FRAME, "context_escaped": "", - "nvm": True, "version": "0.2.2-rc.1", "settings": {"permission_mode": "danger-full-access"}}, - "mini-swe-agent": {"import_path": "harborside.mini_swe_agent:HermitMiniSweAgent", "backend": "mini-swe-agent", - "context_file": None, "context_limit": None, "version": "2.4.6"}, - # The answer-only agent: one model call from Harbor's own process, its answer written into the container - "answer": {"import_path": "harborside.answer:HermitAnswer", "backend": None, "context_file": None, - "context_limit": None}, -} - - -def entry(cli: str) -> dict: - """The row of `cli`, every key present (absent: none, false, empty).""" - row = HARBOR_CLIS[cli] - return {"settings": {}, "env": {}, "nvm": False, "version": None, "context_escaped": None, **row} - - -def cli_of(acting_agent: dict | None) -> str | None: - """The table's CLI an `acting_agent` spec names: `{cli: }`, `{import_path: }` (a harborside - class by its name), the default `codex` for a spec that names no agent; None for a Harbor built-in by `name` or - another class, which runs as Harbor installs it (no adapter, no launch plan).""" - spec = dict(acting_agent or {}) - cli, path, name = spec.get("cli"), spec.get("import_path"), spec.get("name") - if name and path: - raise ValueError("agent.params.acting_agent names both a name and an import_path") - if cli and (name or path): - raise ValueError(f"agent.params.acting_agent names cli {cli!r} and {'name' if name else 'import_path'} " - f"{name or path!r}: give one (cli names the class itself)") - if cli: - if cli not in HARBOR_CLIS: - raise ValueError(f"agent.params.acting_agent.cli is one of {', '.join(HARBOR_CLIS)}, not {cli!r}") - return str(cli) - if path: - module, _, cls = str(path).partition(":") - for key, row in HARBOR_CLIS.items(): - own_class = row["import_path"].partition(":")[2] - if path == row["import_path"] or (module.startswith("harborside.") and cls == own_class): - return key - return None - if name: - return None - return DEFAULT_CLI - - -def bin_mount(cli: str) -> str: - """Where a trial's container sees the folder of `_bin`.""" - return BIN_MOUNT.format(cli=cli) diff --git a/hermit/harbor/context.py b/hermit/harbor/context.py deleted file mode 100644 index f9f0af23..00000000 --- a/hermit/harbor/context.py +++ /dev/null @@ -1,71 +0,0 @@ -"""An adapter's context for one Harbor task, computed on this machine before the trial: what a host deploys in place -of the context it would render itself, for an adapter whose context is per task (a `context_tool`, or any adapter -that is not plugin-portable), and the whole system text of the answer-only agent. - - python -m hermit.harbor.context --adapter --task --out [--query-field ] - -It runs in a child process with HA_ADAPTER_SANDBOX=1, so its Session runs the adapter's tools in the strict adapter -sandbox (no network but the loopback endpoints the environment names, none of this process's keys) and the adapter's -code is never imported into the runner's process: `Session(adapter).context(query=..., task=row)`, as the acting -agents' workspace computes it (hermit.agent.acting.workspace._context; the query is the row's `context_query` field, -else its instruction). The result file holds the text and the context tool's whole result (`last_context`, which the -rollout records as `extra["context_tool"]`: ReHarness revises what it retrieved). -""" -from __future__ import annotations - -import argparse -import json -import os -from pathlib import Path -import subprocess -import sys - -TIMEOUT_S = 900.0 # a context tool that renders a library, or retrieves - - -def compute(adapter: Path, task: dict, out_dir: Path, *, python: str | None = None, enforce: str | None = None, - query_field: str | None = None, timeout_s: float = TIMEOUT_S) -> tuple[str, dict | None]: - """(the adapter's context for `task`, the context tool's record or None), computed in a child process under the - strict adapter sandbox; the row and the result are kept in `out_dir` (context-task.json, context.json).""" - from hermit.adapter.session import ENV_ENFORCE, ENV_SANDBOX - from hermit.sandbox import framework_env - out_dir.mkdir(parents=True, exist_ok=True) - row, result = out_dir / "context-task.json", out_dir / "context.json" - row.write_text(json.dumps(task, ensure_ascii=False, default=str), encoding="utf-8") - result.unlink(missing_ok=True) - env = framework_env({ENV_SANDBOX: "1", ENV_ENFORCE: enforce or os.environ.get(ENV_ENFORCE, "required")}) - command = [python or sys.executable, "-m", "hermit.harbor.context", "--adapter", str(Path(adapter).resolve()), - "--task", str(row), "--out", str(result), *(["--query-field", query_field] if query_field else [])] - try: - run = subprocess.run(command, capture_output=True, text=True, env=env, timeout=timeout_s) - except subprocess.TimeoutExpired: - raise RuntimeError(f"the adapter's per-task context took longer than {timeout_s:.0f} s") from None - if run.returncode != 0 or not result.is_file(): - raise RuntimeError(f"the adapter's per-task context failed: {(run.stderr or run.stdout).strip()[-1500:]}") - data = json.loads(result.read_text(encoding="utf-8")) - return str(data.get("text") or ""), data.get("last_context") - - -def main(argv=None) -> int: - from hermit.adapter.session import Session - from hermit.agent.acting.workspace import _query - parser = argparse.ArgumentParser(description="An adapter's context for one task, in the adapter sandbox.") - parser.add_argument("--adapter", type=Path, required=True) - parser.add_argument("--task", type=Path, required=True, help="the task's row (JSON)") - parser.add_argument("--out", type=Path, required=True) - parser.add_argument("--query-field", help="the row's field the context tool retrieves for (context_query)") - args = parser.parse_args(argv) - task = json.loads(args.task.read_text(encoding="utf-8")) - session = Session(args.adapter) - try: - text = session.context(query=_query(task, {"context_query": args.query_field}), task=task) - record = session.last_context - finally: - session.close() - args.out.write_text(json.dumps({"text": text, "last_context": record}, ensure_ascii=False, default=str), - encoding="utf-8") - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/hermit/harbor/curate.py b/hermit/harbor/curate.py deleted file mode 100644 index f2794a67..00000000 --- a/hermit/harbor/curate.py +++ /dev/null @@ -1,312 +0,0 @@ -"""The rollout's log: a curated copy of what the trial (or the step) left, without the agent's raw session -transcripts, the CLI's home or a sign-in. - - hermit-log/ - agent-.jsonl the CLI's stream, under the name Hermit's own sessions keep it (hermit.agent.base. - trajectory.STREAMS, which the methods' readers look for first at a log's top): a table - CLI's (agent/agent-.jsonl), Harbor's own codex's (agent/codex.txt, as - agent-codex.jsonl) or claude-code's (agent/claude-code.txt, as agent-claude.jsonl) - task.md the task's (or the step's) instruction, as Harbor gives it to the agent - prompt.txt the prompt the CLI was given: the instruction, then the launch plan's prompt_suffix (the - memory note), with its prompt_slash_prefix before one that starts with `/` - last_message.txt the CLI's last message, as its Base Agent read it (the session's reply) - hook-calls.jsonl, hook-state/ what the adapter's hooks wrote (their audit, each session's state) - result.json the trial's or step's result as Harbor wrote it - exception.txt the exception type, message and traceback, when there was one - harbor.log, job.yaml the job's own log and config (a trial; a stream's step has the driver's log) - agent/ the agent's other outputs: the answer agent's answer.json, Harbor's own agents' text - outputs and trajectory.json (ATIF), hermit-collect/ (the CLI's files brought out: its - last message); never sessions/ (a CLI's session logs, collected ones included), nor a - binary file of the CLI's home (OpenCode's database) - memory/ the notes the deployed adapter kept (hermit//memory, copied back after the session: - Continual Harness reads memory/NOTES.md) - tool-calls.jsonl each call of the adapter's tools served for the trial (hermit.harbor.tools) - verifier/ test-stdout.txt (its last 20,000 characters, with `verifier_redact`'s patterns - removed), reward.* - -Every file kept is scrubbed of this process's key values (hermit.redact). The job's `launch/` (the CLI's home, the -sign-in that came back from the container) is never copied, and nothing is read through a link the container left. -""" -from __future__ import annotations - -from datetime import datetime -import json -import os -from pathlib import Path -import re -import shutil -import stat - -STDOUT_TAIL = 20_000 -AGENT_SKIP = {"sessions"} # directories of the agent's logs never copied -AGENT_FILE_LIMIT = 50 * 1024 * 1024 # a file larger than this is not copied -AGENT_SUFFIXES = (".txt", ".json", ".jsonl", ".md", ".log") -COPY_DIR = "hermit-copy" # the workspace's folders harborside copied back (copy_back) -COLLECT_DIR = "hermit-collect" # the CLI's home files it brought out (collect) -MEMORY = "memory" -TOOL_AUDIT = "tool-calls.jsonl" -REDACTED = "[redacted]" -HOOK_AUDIT, HOOK_STATE = "hook-calls.jsonl", "hook-state" # an adapter's hooks' audit and state (HERMIT_HOOK_LOGS) -# the CLI's stream in a trial's agent/ and its name at the log's top: a table CLI's under its own name, Harbor's own -# agents' under the name Hermit's sessions of that CLI keep (hermit.agent.base.trajectory.CODEX_STREAM, CLAUDE_STREAM) -TOP_STREAMS = {**{f"agent-{b}.jsonl": f"agent-{b}.jsonl" - for b in ("codex", "claude", "pi", "mini-swe-agent", "opencode", "dsh")}, - "codex.txt": "agent-codex.jsonl", "claude-code.txt": "agent-claude.jsonl"} - - -def seconds(timing: dict | None) -> float | None: - """A Harbor TimingInfo's duration in seconds.""" - if not timing or not timing.get("started_at") or not timing.get("finished_at"): - return None - stamps = [datetime.fromisoformat(str(timing[k]).replace("Z", "+00:00")) for k in ("started_at", "finished_at")] - return round((stamps[1] - stamps[0]).total_seconds(), 1) - - -def model_calls(trajectory: Path) -> int: - """The ATIF steps carrying token metrics: one per model call (none of a file that is missing, no JSON object, or - nested past the JSON parser's depth).""" - try: - steps = json.loads(trajectory.read_text(encoding="utf-8")).get("steps") or [] - except (OSError, ValueError, AttributeError, RecursionError): - return 0 - return sum(1 for s in steps if isinstance(s, dict) and s.get("metrics")) - - -def open_regular(path: Path) -> int | None: - """`path` opened for reading as a regular file, never through a link at its name (O_NOFOLLOW), else None: what the - task's container could write is read only so, since a link it left there could name a file of this machine.""" - try: - fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK | os.O_CLOEXEC) - except OSError: - return None - try: - if stat.S_ISREG(os.fstat(fd).st_mode): - return fd - except OSError: - pass - os.close(fd) - return None - - -def regular_text(path: Path) -> str | None: - """A regular file's text (open_regular), or None: missing, unreadable, a link, a pipe or a directory.""" - fd = open_regular(path) - if fd is None: - return None - with os.fdopen(fd, "rb") as f: - try: - return f.read().decode("utf-8", errors="replace") - except OSError: - return None - - -def redact_verifier(text: str, patterns) -> str: - """The verifier's output with each match of `patterns` (verifier_redact: what the learner must not read, a - test's expected answer) replaced by REDACTED.""" - for pattern in patterns or (): - text = re.sub(str(pattern), REDACTED, text) - return text - - -DIR_FLAGS = os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW | os.O_CLOEXEC -FILE_FLAGS = os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK | os.O_CLOEXEC - - -def _files(src: Path, dst: Path, *, text_only: bool = False, skip=()) -> None: - """The regular files below `src` (`text_only`: AGENT_SUFFIXES' only), each at most AGENT_FILE_LIMIT, copied under - `dst`, none in a directory named in `skip`. A link is never followed nor copied: the container wrote `src`, and a - link there could name a file of this machine. The walk holds each directory open and opens what is in it through - it, never following a link at any name (O_NOFOLLOW), so a link that a process the agent left running (a stream's - environment stays up between steps) swaps in for a file or a folder while the walk runs is met as a link, never - followed; an entry that vanishes or cannot be read meanwhile is left out. A failure to write under `dst` (this - machine's disk) raises.""" - try: - top = os.open(src, DIR_FLAGS) - except OSError: - return - _walk(top, dst, text_only, frozenset(skip)) - - -def _walk(dir_fd: int, dst: Path, text_only: bool, skip: frozenset) -> None: - """_files' walk of the directory open at `dir_fd` (closed after it) into `dst`.""" - try: - try: - names = sorted(os.listdir(dir_fd)) - except OSError: - return - for name in names: - try: - st = os.stat(name, dir_fd=dir_fd, follow_symlinks=False) - except OSError: - continue - if stat.S_ISDIR(st.st_mode): - if name in skip: - continue - try: - child = os.open(name, DIR_FLAGS, dir_fd=dir_fd) - except OSError: - continue - _walk(child, dst / name, text_only, skip) - elif stat.S_ISREG(st.st_mode) and not (text_only and Path(name).suffix not in AGENT_SUFFIXES): - try: - fd = os.open(name, FILE_FLAGS, dir_fd=dir_fd) - except OSError: - continue - _copy_open(fd, dst / name) - finally: - os.close(dir_fd) - - -def _subdir(parent: Path, name: str) -> int | None: - """`parent/name` open as a directory, each of the two opened without following a link (DIR_FLAGS), the second - through the first, so that neither a link at `name` nor one swapped in for `parent` is followed; None otherwise.""" - try: - top = os.open(parent, DIR_FLAGS) - except OSError: - return None - try: - return os.open(name, DIR_FLAGS, dir_fd=top) - except OSError: - return None - finally: - os.close(top) - - -def _copy_open(fd: int, target: Path) -> None: - """The file open at `fd` (closed after it) copied to `target` with its times, when it is a regular file of at most - AGENT_FILE_LIMIT; at most that much of it all the same when it grows meanwhile. A read that fails (the - container's file) ends the copy with what was read; a write that fails (this machine's disk) raises.""" - with os.fdopen(fd, "rb") as f: - try: - st = os.fstat(f.fileno()) - except OSError: - return - if not stat.S_ISREG(st.st_mode) or st.st_size > AGENT_FILE_LIMIT: - return - target.parent.mkdir(parents=True, exist_ok=True) - with open(target, "wb") as out: - left = AGENT_FILE_LIMIT - while left > 0: - try: - chunk = f.read(min(1 << 20, left)) - except OSError: - break - if not chunk: - break - out.write(chunk) - left -= len(chunk) - try: - os.utime(target, ns=(st.st_atime_ns, st.st_mtime_ns)) - except OSError: - pass - - -def curate(outputs: Path | None, job: Path, verdict: dict, *, result: dict | None = None, - extra_files: dict[str, Path] | None = None, redact=(), audit: Path | None = None, - instruction: str | None = None, plan: Path | None = None, reply: str | None = None) -> Path: - """The curated log directory under `job`: the result and the agent's and verifier's outputs from `outputs` - (a trial directory, or a step's directory holding agent/ and verifier/), the verdict's exception, - `extra_files` ({name: path}: the job's harbor.log, the stream's driver.log), the tools' `audit` log; `redact` - the verifier_redact patterns. At its top, where Hermit's own logs keep them (the module's docstring), the CLI's - stream, `instruction` (task.md) and the prompt the launch `plan` made of it (prompt.txt), the session's `reply` - (last_message.txt) and the hooks' files.""" - from hermit.redact import redact_tree - log = job / "hermit-log" - shutil.rmtree(log, ignore_errors=True) - (log / "agent").mkdir(parents=True) - (log / "verifier").mkdir() - for name, path in {"job.yaml": job / "job.yaml", "harbor.log": job / "harbor.log", **(extra_files or {})}.items(): - if path.is_file(): - shutil.copy2(path, log / name) - if audit is not None and Path(audit).is_file(): - shutil.copy2(audit, log / TOOL_AUDIT) - if verdict.get("exception_type") or verdict.get("exception_message"): - (log / "exception.txt").write_text(f"{verdict.get('exception_type') or verdict['failure_cause']}: " - f"{verdict.get('exception_message', '')}\n", encoding="utf-8") - if result is not None: - (log / "result.json").write_text(json.dumps(result, indent=1, default=str) + "\n", encoding="utf-8") - trace = ((result.get("exception_info") or {}).get("exception_traceback")) - if trace: - with open(log / "exception.txt", "a", encoding="utf-8") as f: - f.write("\n" + str(trace)) - if outputs is not None: - _outputs(Path(outputs), log, result, redact, _plan_stream(plan)) - if instruction: - (log / "task.md").write_text(instruction if instruction.endswith("\n") else instruction + "\n", - encoding="utf-8") - (log / "prompt.txt").write_text(_prompt(instruction, plan), encoding="utf-8") - if reply is not None: - (log / "last_message.txt").write_text(str(reply), encoding="utf-8") - redact_tree(log) - return log - - -def _prompt(instruction: str, plan: Path | None) -> str: - """The prompt harborside gives the CLI (harborside/hermit.py): the instruction and the plan's prompt_suffix, the - plan's prompt_slash_prefix before one that starts with `/`; the instruction alone without a plan.""" - try: - data = json.loads(Path(plan).read_text(encoding="utf-8")) if plan is not None else {} - except (OSError, ValueError): - data = {} - prompt = instruction + str(data.get("prompt_suffix") or "") - if prompt.startswith("/") and data.get("prompt_slash_prefix"): - prompt = str(data["prompt_slash_prefix"]) + prompt - return prompt - - -def _plan_stream(plan: Path | None) -> str | None: - """The stream the launch `plan` names, "" for one that is no CLI's stream (the answer agent's answer.json) or a - plan that cannot be read; None without a plan, or one that names none (Harbor's own agents: their stream found by - TOP_STREAMS).""" - if plan is None: - return None - try: - data = json.loads(Path(plan).read_text(encoding="utf-8")) - except (OSError, ValueError): - return "" - name = data.get("stream") if isinstance(data, dict) else None - if name is None: - return None - return name if name in TOP_STREAMS else "" - - -def _outputs(outputs: Path, log: Path, result: dict | None, redact, named: str | None = None) -> None: - """The trial's outputs into the log; at its top, the CLI's stream: the one its plan `named` (a trial's agent may - write another CLI's), else the first TOP_STREAMS finds.""" - if result is None and (outputs / "result.json").is_file(): - shutil.copy2(outputs / "result.json", log / "result.json") - agent = outputs / "agent" - if agent.is_dir(): - names = TOP_STREAMS if named is None else [named] if named else [] - stream = next((n for n in names if (agent / n).is_file() and not (agent / n).is_symlink()), None) - for child in sorted(agent.iterdir()): - if child.is_symlink(): - continue - if child.is_dir(): - if child.name in AGENT_SKIP: - continue - if child.name == HOOK_STATE: # at the top, beside the stream - _files(child, log / HOOK_STATE) - elif child.name == ".hermit": - _files(child, log / "agent" / child.name) - elif child.name == COLLECT_DIR: # Codex's collected session logs are its transcripts - _files(child, log / "agent" / COLLECT_DIR, text_only=True, skip=AGENT_SKIP) - elif child.name == COPY_DIR and (memory := _subdir(child, MEMORY)) is not None: - _walk(memory, log / MEMORY, False, frozenset()) # through the copy's folder, never a link - continue - # a regular file only, opened at its own name (open_regular): a FIFO or a device the session left under a - # text name is never read, nor a link swapped in for the file since the checks above - if child.suffix in AGENT_SUFFIXES and (fd := open_regular(child)) is not None: - top = TOP_STREAMS[child.name] if child.name == stream else HOOK_AUDIT if child.name == HOOK_AUDIT \ - else None - _copy_open(fd, log / top if top else log / "agent" / child.name) - verifier = outputs / "verifier" - text = regular_text(verifier / "test-stdout.txt") - if text is not None: - text = redact_verifier(text, redact) - if len(text) > STDOUT_TAIL: - text = f"[... {len(text) - STDOUT_TAIL} characters dropped ...]\n" + text[-STDOUT_TAIL:] - (log / "verifier" / "test-stdout.txt").write_text(text, encoding="utf-8") - for reward in verifier.glob("reward.*") if verifier.is_dir() else []: - if (fd := open_regular(reward)) is not None: # never a link swapped in since the glob - _copy_open(fd, log / "verifier" / reward.name) diff --git a/hermit/harbor/dataset.py b/hermit/harbor/dataset.py index 26a8cdf3..bdf898bc 100644 --- a/hermit/harbor/dataset.py +++ b/hermit/harbor/dataset.py @@ -76,9 +76,10 @@ import stat import subprocess import sys -import tomllib -TASK_FILE = "task.toml" +from hermit.task.directory import (TASK_FILE, is_task, network_modes, resources, # noqa: F401 + rows_of, step_names, task_config, task_dirs) + SKIP_DIRS = {".git", "__pycache__"} LOCK_SCHEMA = 1 TASKS_ROOT_ENV = "HERMIT_HARBOR_TASKS" # the root of every prepared task set (prepare's data default) @@ -91,113 +92,6 @@ SOURCE_FILE = ".hermit-source.json" # in a data directory: its download's source and patch -def is_task(path: Path) -> bool: - return path.is_dir() and (path / TASK_FILE).is_file() - - -def task_dirs(tasks_dir: Path) -> list[Path]: - """The task directories under `tasks_dir`, by name.""" - tasks_dir = Path(tasks_dir) - if is_task(tasks_dir): - return [tasks_dir] - return sorted(p for p in tasks_dir.iterdir() if is_task(p)) if tasks_dir.is_dir() else [] - - -def task_config(task_dir: Path) -> dict: - return tomllib.loads((Path(task_dir) / TASK_FILE).read_text()) - - -def step_names(config: dict) -> list[str]: - return [str(s["name"]) for s in (config.get("steps") or [])] - - -def _baseline(env) -> str: - """An environment's network mode (Harbor's baseline): its network_mode, else the deprecated allow_internet (true: - public, false: no-network), else public.""" - env = env if isinstance(env, dict) else {} - if env.get("network_mode"): - return str(env["network_mode"]) - if env.get("allow_internet") is not None and env.get("allowed_hosts") is None: - return "public" if env["allow_internet"] else "no-network" - return "public" - - -def network_modes(config: dict, step: str | None = None) -> dict[str, str]: - """{"agent": mode, "verifier": mode} of a task (or of its step `step`), as Harbor resolves them - (harbor/trial/network_policy.py): a phase's own `network_mode` ([agent], [verifier], a step's first), else the - environment's baseline (the verifier's own environment's when it has one); modes public, no-network, allowlist.""" - step_cfg = next((s for s in config.get("steps") or [] if str(s.get("name")) == str(step)), None) if step else None - step_cfg = step_cfg or {} - - def explicit(role: str): - return ((step_cfg.get(role) or {}).get("network_mode")) or ((config.get(role) or {}).get("network_mode")) - base = _baseline(config.get("environment")) - verifier_env = ((step_cfg.get("verifier") or {}).get("environment") - or (config.get("verifier") or {}).get("environment")) - return {"agent": str(explicit("agent") or base), - "verifier": str(explicit("verifier") or (_baseline(verifier_env) if verifier_env else base))} - - -def _size_mb(value) -> int | None: - """A size as Harbor reads the deprecated `memory`/`storage` fields ("2G", "512M"), in MB; None when unreadable.""" - text = str(value).strip().upper() - try: - if text.endswith("G"): - return int(float(text[:-1]) * 1024) - if text.endswith("M"): - return int(float(text[:-1])) - if text.endswith("K"): - return int(float(text[:-1]) / 1024) - except ValueError: - return None - return None - - -def resources(config: dict) -> dict: - """The task's resources ([environment]): cpus, memory_mb, storage_mb, gpus (None where it sets none).""" - env = config.get("environment") or {} - memory = env.get("memory_mb") if env.get("memory_mb") is not None else \ - (_size_mb(env["memory"]) if env.get("memory") is not None else None) - storage = env.get("storage_mb") if env.get("storage_mb") is not None else \ - (_size_mb(env["storage"]) if env.get("storage") is not None else None) - return {"cpus": env.get("cpus"), "memory_mb": memory, "storage_mb": storage, "gpus": env.get("gpus")} - - -def _instruction(path: Path) -> str: - try: - return path.read_text(encoding="utf-8").strip() - except OSError: - return "" - - -def rows_of(task_dir: Path, *, rollouts: int = 1) -> list[dict]: - """The Hermit rows of one task: by rollout, and by step for a multi-step task.""" - task_dir = Path(task_dir) - config = task_config(task_dir) - env, agent, verifier = (config.get("environment") or {}), (config.get("agent") or {}), (config.get("verifier") or {}) - base = {"harbor_task": task_dir.name, "agent_timeout_sec": agent.get("timeout_sec"), - "verifier_timeout_sec": verifier.get("timeout_sec"), "build_timeout_sec": env.get("build_timeout_sec"), - "docker_image": env.get("docker_image"), "metadata": config.get("metadata") or {}, **resources(config), - "network": network_modes(config)} - steps = step_names(config) - out = [] - for k in range(1, rollouts + 1): - suffix = f"#{k}" if rollouts > 1 else "" - if not steps: - out.append({"id": f"{task_dir.name}{suffix}", **base, "rollout": k, - "instruction": _instruction(task_dir / "instruction.md")}) - continue - for i, step in enumerate(steps, start=1): - step_cfg = next(s for s in config["steps"] if str(s["name"]) == step) - out.append({"id": f"{task_dir.name}:{step}{suffix}", **base, "rollout": k, "step": step, "step_index": i, - "steps": len(steps), "instruction": _instruction(task_dir / "steps" / step / "instruction.md"), - "network": network_modes(config, step), - "agent_timeout_sec": (step_cfg.get("agent") or {}).get("timeout_sec") or base["agent_timeout_sec"], - "verifier_timeout_sec": ((step_cfg.get("verifier") or {}).get("timeout_sec") - or base["verifier_timeout_sec"])}) - return out - - def index(tasks_dir: Path, *, rollouts: int = 1, include: list[str] | None = None) -> list[dict]: """The rows of every task under `tasks_dir` (or of those named in `include`).""" wanted = set(include) if include else None diff --git a/hermit/harbor/harborside/__init__.py b/hermit/harbor/harborside/__init__.py deleted file mode 100644 index e2801ff5..00000000 --- a/hermit/harbor/harborside/__init__.py +++ /dev/null @@ -1,25 +0,0 @@ -"""The half of Hermit's Harbor support that runs under Harbor's own interpreter: imported by `harbor run` (the agents) -or started by the stream client (the driver). Harbor lives in its own venv, so nothing here imports `hermit`: harbor, -two of Harbor's own dependencies (pydantic, which its agents are built on; certifi, where it is installed, for -Codex's CA bundle of last resort) and the standard library only. Hermit puts this directory's parent on Harbor's -PYTHONPATH. - -Every acting CLI runs inside the task container from the launch plan Hermit renders for its trial -(hermit.harbor.launch writes it; harborside.hermit carries it out the same way for each CLI): - - harborside.hermit the launch plan's executor: HermitInstalled, mixed in over Harbor's agent - harborside.codex:HermitCodex Codex (Harbor's install, or a mounted codex_bin with its rg and CA bundle) - harborside.claude:HermitClaude Claude Code (Harbor's install, or a mounted claude_bin; the subscription's - token by descriptor) - harborside.pi:HermitPi Pi (Harbor's npm install, or a mounted pi_bin) - harborside.opencode:HermitOpenCode OpenCode 2.2 (its GitHub release, musl or glibc, or a mounted opencode_bin) - harborside.dsh:HermitDsh DeepSeek Harness (Node 24 by nvm and npm, or a mounted dsh_bin) - harborside.mini_swe_agent:HermitMiniSweAgent mini-swe-agent (Harbor's uv install), running Hermit's runner from - the plan's bundle (HermitLaunchError when that cannot run there: no other - agent in its place) - harborside.answer:HermitAnswer an agent that only answers: one request to Hermit's model from Harbor's - process, its answer written into the container - harborside.scripted:ScriptedAgent a shell script as the agent (smoke tests of task sets, no model) - harborside.driver one multi-step task, run a step at a time on request (a stream), each step - with its own launch plan -""" diff --git a/hermit/harbor/harborside/answer.py b/hermit/harbor/harborside/answer.py deleted file mode 100644 index 8abbeca2..00000000 --- a/hermit/harbor/harborside/answer.py +++ /dev/null @@ -1,377 +0,0 @@ -"""HermitAnswer: an acting agent that only answers, one request to Hermit's model, its answer written into the task. - -Imported by Harbor as `harborside.answer:HermitAnswer`, for a task whose work is a reply (a question with a figure, -a value from a document) rather than a session in the container. It is an external agent: it runs in Harbor's own -process on the host, outside the task container's network policy, and installs nothing there. Its launch plan -(`cli: answer`, written by hermit.harbor.launch) has no command but an `answer` block: - - endpoint Hermit's chat endpoint for the trial (http://127.0.0.1:/), which serves the run's - model with Hermit's binding and prices (a direct model, or a CLI-reached one) - model, system the request's model, and its system message (the adapter's per-task context; may be empty) - instruction_file null: Harbor's instruction is the question; else a file in the container that holds it (a task - whose entrypoint writes it, such as LAB-Bench's /app/instruction.md), waited for up to wait_s - attachments globs of files in the container sent with the question as image data URLs (base64 over exec), - each image they match but one over MAX_ATTACHMENT_BYTES, left out with a warning - answer_file where the answer is written in the container, as the task's agent user (base64 through exec, - so no quoting of the text is involved); null: written nowhere - answer_pattern a regex: the answer is its last match in the reply (group 1 when it has one), else the reply - answer_format the text written, with `{answer}` in it (the tagged form a verifier reads) - max_tokens, effort the request's max_tokens and reasoning_effort, when given - -One POST /chat/completions (the standard library's urllib, in a thread of its own, so that Harbor's timeout -can stop the trial; an endpoint on this machine is reached directly, never through the http(s)_proxy of Harbor's -environment). `/logs/agent/answer.json` keeps {reply, answer, usage, model, attachments} (the attachments' paths, -never their bytes, and never the endpoint, whose path is its token). A model that failed (an HTTP error, no reply, no -connection) raises NonZeroAgentExitCodeError, so the verifier still runs; what the task's container did not provide -(no instruction file in time, attachments whose listing exec failed, a question or a figure that cannot be read there: -no base64 or tr, a file the agent user may not open, no wc for a figure's size; no answer file written) and what this -machine could not do with the model's answer (keep it in answer.json; run the exec that writes it: an OSError, the -compose client not spawned) raise HermitLaunchError, which Hermit's side counts as infra (harborside.hermit). A figure -a wildcard would match in a directory the agent user may not list is not found, as by a glob that matches nothing (no -error: the model is asked without it); one named outright is found wherever the agent user may enter its directory. -The agent declares no skills or MCP servers, so a task that names some is refused by Harbor, saying so, rather than -answered without them. -""" -from __future__ import annotations - -import asyncio -import base64 -import json -import mimetypes -import os -from pathlib import Path, PurePosixPath -import re -import shlex -import tempfile -import threading -import time -from typing import Any -import urllib.error -import urllib.parse -import urllib.request - -from harbor.agents.base import BaseAgent -from harbor.agents.capabilities import AgentCapabilities -from harbor.agents.installed.base import NonZeroAgentExitCodeError -from harbor.agents.options import AgentOptions -from harbor.environments.base import BaseEnvironment -from harbor.models.agent.context import AgentContext -from pydantic import Field, field_validator - -from harborside.hermit import HermitLaunchError, load_plan, marked - -# What is read from the container is printed after a mark on a line of its own (harborside.hermit.marked): Harbor's -# docker environment merges an exec's stderr into its stdout -INSTRUCTION_LINE = "hermit-instruction:" # the instruction file, in base64 -FILE_LINE = "hermit-file:" # an attachment's path, one line each -SIZE_LINE = "hermit-size:" # its size in bytes -DATA_LINE = "hermit-data:" # its bytes, in base64 -UNREADABLE = 3 # a read's exit code: the file cannot be read in base64 here -POLL_S = 1.0 -DEFAULT_WAIT_S = 30.0 -REQUEST_TIMEOUT_S = 3600 # one request at most; Harbor's agent timeout stops the trial before -MAX_ATTACHMENT_BYTES = 20 * 2**20 # one attachment's bytes at most (it is sent inline) -CHUNK = 60_000 # base64 characters per exec when the answer is written -GLOB = re.compile(r"/[A-Za-z0-9_./*?\[\]+@%:,=-]*") # a container glob that needs no quoting to expand - - -def answer_problems(spec: Any) -> list[str]: - """What is wrong with a plan's `answer` block, as messages naming the field; [] when none.""" - if not isinstance(spec, dict): - return ["answer must be a mapping (the answer agent's plan)"] - problems = [] - if not str(spec.get("endpoint") or "").startswith(("http://", "https://")): - problems.append("answer.endpoint must be an http(s) URL") - if not isinstance(spec.get("model"), str) or not spec["model"]: - problems.append("answer.model must name the model") - if not isinstance(spec.get("system") or "", str): - problems.append("answer.system must be text") - for key in ("instruction_file", "answer_file"): - if spec.get(key) is not None and not str(spec[key]).startswith("/"): - problems.append(f"answer.{key} must be an absolute container path or null") - wait = spec.get("wait_s", DEFAULT_WAIT_S) - if isinstance(wait, bool) or not isinstance(wait, (int, float)) or wait < 0: - problems.append("answer.wait_s must be a number of seconds") - for pattern in spec.get("attachments") or []: - if not isinstance(pattern, str) or not GLOB.fullmatch(pattern): - problems.append(f"answer.attachments must be absolute globs of plain characters, not {pattern!r}") - if spec.get("answer_pattern") is not None: - try: - re.compile(str(spec["answer_pattern"])) - except re.error as exc: - problems.append(f"answer.answer_pattern is not a regex: {exc}") - if "{answer}" not in str(spec.get("answer_format") or "{answer}"): - problems.append("answer.answer_format must hold {answer}") - tokens = spec.get("max_tokens") - if tokens is not None and (isinstance(tokens, bool) or not isinstance(tokens, int) or tokens <= 0): - problems.append("answer.max_tokens must be a positive integer or null") - return problems - - -def extract(reply: str, pattern: str | None) -> str: - """The answer in a reply: the last match of `pattern` (its group 1 when it has one), else the whole reply.""" - if pattern: - matches = list(re.finditer(pattern, reply)) - if matches: - last = matches[-1] - found = last.group(1) if last.re.groups and last.group(1) is not None else last.group(0) - return found.strip() - return reply.strip() - - -def _read_base64(path: str, mark: str) -> str: - """The command that prints the file at `path` in base64 on a line after `mark`, as the context capture of - harborside.hermit reads the image's own file: base64 and tr checked first, the read under pipefail, so that a - file it cannot read exits UNREADABLE saying why (without them, `base64 < f | tr -d '\\n'` succeeds with nothing) - and is never an empty value.""" - q = shlex.quote(path) - return ("command -v base64 >/dev/null 2>&1 && command -v tr >/dev/null 2>&1 || " - f"{{ echo 'hermit: no base64 or tr in the image' >&2; exit {UNREADABLE}; }}; " - f"v=$(set -o pipefail; base64 < {q} | tr -d '\\n') || exit {UNREADABLE}; printf '{mark}%s\\n' \"$v\"") - - -def _text(content: Any) -> str: - """A chat message's content as text (a string, or the text of its parts).""" - if isinstance(content, str): - return content - if isinstance(content, list): - return "".join(str(p.get("text") or "") for p in content if isinstance(p, dict) and p.get("type") in - (None, "text", "output_text")) - return "" - - -def _opener(url: str) -> urllib.request.OpenerDirector: - """urllib's opener for the endpoint: without proxies for one on this machine (Hermit's endpoint is; a proxy from - the environment would not reach it), the environment's otherwise.""" - host = (urllib.parse.urlsplit(url).hostname or "").lower() - if host in ("localhost", "::1") or host.startswith("127."): - return urllib.request.build_opener(urllib.request.ProxyHandler({})) - return urllib.request.build_opener() - - -def _request(url: str, body: dict) -> dict: - """One chat completion; the parsed response, or NonZeroAgentExitCodeError (never naming the URL: its path is the - endpoint's token).""" - data = json.dumps(body).encode() - request = urllib.request.Request(url, data=data, method="POST", headers={"Content-Type": "application/json"}) - try: - with _opener(url).open(request, timeout=REQUEST_TIMEOUT_S) as response: - raw = response.read() - except urllib.error.HTTPError as error: - said = (error.read() or b"").decode("utf-8", errors="replace")[:600] - raise NonZeroAgentExitCodeError(f"hermit: the answer's model failed: HTTP {error.code}: {said}") from None - except (urllib.error.URLError, OSError) as error: - reason = getattr(error, "reason", error) - raise NonZeroAgentExitCodeError(f"hermit: the answer's model was not reached: {reason}") from None - try: - answer = json.loads(raw or b"null") - except ValueError: - raise NonZeroAgentExitCodeError("hermit: the answer's model sent no JSON response") from None - if not isinstance(answer, dict): - raise NonZeroAgentExitCodeError("hermit: the answer's model sent no JSON object") - return answer - - -class HermitAnswerOptions(AgentOptions): - """The answer agent's kwargs: its plan. `extra="forbid"`.""" - - plan: str | None = Field(default=None, description="The launch plan Hermit rendered for the trial (cli: answer).") - - @field_validator("plan") - @classmethod - def _plan_is_a_file(cls, value: str | None) -> str | None: - if value is not None and not Path(value).expanduser().is_file(): - raise ValueError(f"the launch plan {value} is not a file") - return value - - -class HermitAnswer(BaseAgent): - capabilities = AgentCapabilities() # no skills, no MCP servers: Harbor refuses such a task, saying so - options_model = HermitAnswerOptions - options: HermitAnswerOptions - - def __init__(self, *args: Any, **kwargs: Any) -> None: - super().__init__(*args, **kwargs) - self._plan: dict | None = None - if self.options.plan: - self.set_plan(self.options.plan) - - @staticmethod - def name() -> str: - return "hermit-answer" - - def version(self) -> str: - return "1" - - def set_plan(self, path: str | None) -> None: - """The plan the next `run` follows (a stream's driver gives each step its own); None: no plan.""" - if not path: - self._plan = None - return - plan = load_plan(path, "answer") - problems = answer_problems(plan.get("answer")) - if problems: - raise ValueError(f"hermit: the launch plan {path} is not usable: " + "; ".join(problems)) - self._plan = plan - - async def setup(self, environment: BaseEnvironment) -> None: - """Nothing in the container: the agent runs in Harbor's process.""" - - async def run(self, instruction: str, environment: BaseEnvironment, context: AgentContext) -> None: - if self._plan is None: - raise HermitLaunchError("hermit: HermitAnswer answers from the launch plan Hermit renders for each trial " - "(kwargs.plan), and it was given none") - spec = self._plan["answer"] - question = await self._question(environment, spec, instruction) + str(self._plan.get("prompt_suffix") or "") - images = await self._attachments(environment, spec) - content: Any = question if not images else [{"type": "text", "text": question}, *( - {"type": "image_url", "image_url": {"url": url}} for _, url in images)] - messages = ([{"role": "system", "content": spec["system"]}] if spec.get("system") else []) + \ - [{"role": "user", "content": content}] - body: dict = {"model": spec["model"], "messages": messages} - if spec.get("max_tokens"): - body["max_tokens"] = int(spec["max_tokens"]) - if spec.get("effort"): - body["reasoning_effort"] = str(spec["effort"]) - response = await self._post(str(spec["endpoint"]).rstrip("/") + "/chat/completions", body) - choices = response.get("choices") or [] - message = choices[0].get("message") if choices and isinstance(choices[0], dict) else None - if not isinstance(message, dict): - raise NonZeroAgentExitCodeError("hermit: the answer's model sent no message") - reply = _text(message.get("content")) - answer = extract(reply, spec.get("answer_pattern")) - written = str(spec.get("answer_format") or "{answer}").replace("{answer}", answer) - usage = response.get("usage") if isinstance(response.get("usage"), dict) else {} - answer_json = self.logs_dir / "answer.json" - part = None - try: - self.logs_dir.mkdir(parents=True, exist_ok=True) - # a file of its own, then renamed: the task's container writes this folder too, and a link it left at - # answer.json is replaced, never followed onto this machine - fd, part = tempfile.mkstemp(dir=self.logs_dir, prefix=".answer-", suffix=".json") - with os.fdopen(fd, "w", encoding="utf-8") as f: - f.write(json.dumps( - {"reply": reply, "answer": answer, "usage": usage, "model": response.get("model") or spec["model"], - "attachments": [path for path, _ in images]}, indent=1, ensure_ascii=False) + "\n") - os.replace(part, answer_json) - part = None - except OSError as exc: # this machine's (a full disk, its quota): not the agent's 0 - if part is not None: - Path(part).unlink(missing_ok=True) - raise HermitLaunchError(f"hermit: the answer could not be kept in {answer_json}: {type(exc).__name__}: " - f"{exc}") from exc - if spec.get("answer_file"): - await self._write(environment, str(spec["answer_file"]), written) - details = usage.get("prompt_tokens_details") if isinstance(usage.get("prompt_tokens_details"), dict) else {} - context.n_input_tokens = int(usage.get("prompt_tokens") or 0) - context.n_output_tokens = int(usage.get("completion_tokens") or 0) - context.n_cache_tokens = int(details.get("cached_tokens") or 0) - context.metadata = {**(context.metadata or {}), **(self._plan.get("metadata") or {}), "hermit_cli": "answer", - "answer_file": spec.get("answer_file"), "attachments": len(images), - "instruction_from": "file" if spec.get("instruction_file") else "harbor"} - - async def _question(self, environment: BaseEnvironment, spec: dict, instruction: str) -> str: - """Harbor's instruction, or the task's instruction file once it is there (polled up to wait_s); a file there - that cannot be read (no base64 or tr in the image, a read that failed) is HermitLaunchError at once, never an - empty question.""" - path = spec.get("instruction_file") - if not path: - return instruction - deadline = time.monotonic() + float(spec.get("wait_s", DEFAULT_WAIT_S)) - while True: - got = await environment.exec(command=f"test -f {shlex.quote(path)} || exit 1; " - + _read_base64(path, INSTRUCTION_LINE)) - value = marked(got.stdout, INSTRUCTION_LINE) if got.return_code == 0 else None - if value is not None: - return base64.b64decode(value).decode("utf-8", errors="replace") - if got.return_code in (0, UNREADABLE): - raise HermitLaunchError(f"hermit: the task's instruction file {path} could not be read: " - f"{(got.stdout or got.stderr or '').strip()[-300:]}") - if time.monotonic() >= deadline: - raise HermitLaunchError(f"hermit: the task's instruction file {path} was not there after " - f"{spec.get('wait_s', DEFAULT_WAIT_S)} s (its environment writes it): no " - "question to answer") - await asyncio.sleep(POLL_S) - - async def _attachments(self, environment: BaseEnvironment, spec: dict) -> list[tuple[str, str]]: - """The attachments' files as (path, image data URL), in the order of their globs, each once: the images among - them, but one over MAX_ATTACHMENT_BYTES, which is left out with a warning (as a file that is no image is). - A listing whose exec failed (which files the globs match is not known) and an image that cannot be measured or - read (no wc, base64 or tr in the image, a file the agent user may not open, an exec that failed) are - HermitLaunchError: the model is never asked without them. A figure a wildcard would match in a directory the - agent user may not list is not found, as by a glob that matches nothing (no error: the model is asked without - it); one named outright is found wherever the agent user may enter its directory.""" - patterns = spec.get("attachments") or [] - if not patterns: - return [] - listed = await environment.exec(command="shopt -s nullglob; for f in " + " ".join(patterns) + - f"; do [ -f \"$f\" ] && printf '{FILE_LINE}%s\\n' \"$f\"; done; true") - if listed.return_code != 0: # the command ends in `true`: the exec itself failed - raise HermitLaunchError(f"hermit: the attachments {' '.join(patterns)} could not be listed: " - f"{(listed.stdout or listed.stderr or '').strip()[-300:]}") - paths = list(dict.fromkeys(line.split(FILE_LINE, 1)[1] for line in (listed.stdout or "").splitlines() - if FILE_LINE in line)) - out = [] - for path in paths: - kind = mimetypes.guess_type(path)[0] or "" - if not kind.startswith("image/"): - self.logger.warning("hermit: %s is no image: it is not sent", path) - continue - size = await environment.exec(command=f'n=$(wc -c < {shlex.quote(path)}) && echo "{SIZE_LINE}$n"') - n = marked(size.stdout, SIZE_LINE) if size.return_code == 0 else None - if n is None or not n.isdecimal(): - raise HermitLaunchError(f"hermit: the attachment {path} could not be measured: " - f"{(size.stdout or size.stderr or '').strip()[-300:]}") - if int(n) > MAX_ATTACHMENT_BYTES: - self.logger.warning("hermit: %s is over %d bytes: it is not sent", path, MAX_ATTACHMENT_BYTES) - continue - data = await environment.exec(command=_read_base64(path, DATA_LINE)) - value = marked(data.stdout, DATA_LINE) if data.return_code == 0 else None - if value is None: - raise HermitLaunchError(f"hermit: the attachment {path} could not be read: " - f"{(data.stdout or data.stderr or '').strip()[-300:]}") - out.append((path, f"data:{kind};base64,{value}")) - return out - - async def _post(self, url: str, body: dict) -> dict: - """_request in a daemon thread: Harbor's timeout cancels the wait, and the thread holds no exit.""" - loop = asyncio.get_running_loop() - future: asyncio.Future = loop.create_future() - - def settle(result, error) -> None: - if not future.done(): - future.set_exception(error) if error is not None else future.set_result(result) - - def work() -> None: - try: - result, error = _request(url, body), None - except BaseException as exc: # noqa: BLE001 - handed to the waiting coroutine - result, error = None, exc - try: - loop.call_soon_threadsafe(settle, result, error) - except RuntimeError: # the loop is gone: the trial no longer waits - pass - - threading.Thread(target=work, daemon=True, name="hermit-answer").start() - return await future - - async def _write(self, environment: BaseEnvironment, path: str, text: str) -> None: - """The answer into the container as the task's agent user: base64 through exec, a chunk at a time. An exec - that exited non-zero, or that this machine could not run (an OSError: the compose client not spawned, EMFILE, - EAGAIN), is HermitLaunchError.""" - encoded = base64.b64encode(text.encode("utf-8")).decode() - chunks = [encoded[i:i + CHUNK] for i in range(0, len(encoded), CHUNK)] or [""] - target = shlex.quote(path) - parent = shlex.quote(str(PurePosixPath(path).parent)) - for i, chunk in enumerate(chunks): - into = ">" if i == 0 else ">>" - command = (f"mkdir -p {parent} && printf '%s' {shlex.quote(chunk)} | base64 -d {into} {target}" - if chunk else f"mkdir -p {parent} && : > {target}") - try: - result = await environment.exec(command=command) - except OSError as exc: # this machine's: the compose client not spawned (EMFILE) - raise HermitLaunchError(f"hermit: the answer could not be written to {path}: its exec could not be " - f"run here: {type(exc).__name__}: {exc}") from exc - if result.return_code != 0: - raise HermitLaunchError(f"hermit: the answer could not be written to {path} as the task's agent " - f"user: {(result.stdout or result.stderr or '').strip()[-300:]}") diff --git a/hermit/harbor/harborside/claude.py b/hermit/harbor/harborside/claude.py deleted file mode 100644 index 281dac94..00000000 --- a/hermit/harbor/harborside/claude.py +++ /dev/null @@ -1,46 +0,0 @@ -"""HermitClaude: Claude Code in a Harbor trial, its session launched from Hermit's plan (harborside.hermit). - -Imported by Harbor as `harborside.claude:HermitClaude`. Harbor's Claude Code brings the install (its bootstrap script -into ~/.local/bin, npm on Alpine) at `version`, after the apt repair of an image whose release left the mirrors, -unless the binary Hermit's runner mounts (`claude_bin`: the machine's Claude Code, its folder mounted read-only at -/opt/hermit-claude) answers `--version` as `version` and the task's agent user may execute it; then it is linked onto -PATH and nothing is installed. - -The session is the plan's, as harborside.hermit runs every CLI's: Hermit's own `claude -p --output-format stream-json -...` (its denied tools, web search either way, on a login the last `--settings` that keeps it on the subscription) -with its own CLAUDE_CONFIG_DIR (/installed-agent/hermit-home), the prompt on stdin, the deployment's CLAUDE.md and -.claude/ in the workdir (hooks run from the adapter's copy under $CLAUDE_PROJECT_DIR), and the stream -agent-claude.jsonl. On the subscription (`login: claude`) the token CLAUDE_CODE_OAUTH_TOKEN is the plan's env_by_fd -secret: passed to `docker compose exec` by name, never on a command line or in a file, then into a pipe on descriptor -3, which the CLI reads once (CLAUDE_CODE_OAUTH_TOKEN_FILE_DESCRIPTOR). In the container only the exec's bash, until it -execs the launch script (the token unset, and `env -u`), and the subshell that writes the token into the pipe (a fork -of that bash), until its printf ends, have it in their environment: the launch script, the CLI and what they start do -not. The task's container is the session's sandbox: its commands run as the CLI's user there, and a command that read -the CLI's memory could still find the token. - -Harbor's own run (its command, its claude-code.txt and CLAUDE_CONFIG_DIR under /logs/agent/sessions, its auth -environment) is not used. -""" -from __future__ import annotations - -from harbor.agents.installed.claude_code import ClaudeCode, ClaudeCodeOptions -from pydantic import Field - -from harborside.hermit import HermitInstalled, HermitOptions - - -class HermitClaudeOptions(HermitOptions, ClaudeCodeOptions): - """Harbor's Claude Code kwargs (accepted, unused: the plan holds the session) and Hermit's. `extra="forbid"`.""" - - claude_bin: str | None = Field(default=None, description="A Claude Code binary mounted in the container, used when " - "its --version is `version` and the task's agent user " - "may execute it, instead of Harbor's per-trial install.") - - -class HermitClaude(HermitInstalled, ClaudeCode): - CLI = "claude" - PROGRAM = "claude" - BIN_OPTION = "claude_bin" - RUN_PATH = ("$HOME/.local/bin",) # where Harbor's bootstrap install puts `claude` - options_model = HermitClaudeOptions - options: HermitClaudeOptions diff --git a/hermit/harbor/harborside/codex.py b/hermit/harbor/harborside/codex.py deleted file mode 100644 index d7a9a703..00000000 --- a/hermit/harbor/harborside/codex.py +++ /dev/null @@ -1,95 +0,0 @@ -"""HermitCodex: Codex in a Harbor trial, its session launched from Hermit's plan (harborside.hermit). - -Imported by Harbor as `harborside.codex:HermitCodex`. Harbor's Codex brings the install (apt curl nodejs npm ripgrep, -then `npm install -g @openai/codex` at `version`: on a glibc image under the agent user's nvm with Node 22, on musl -into npm's global bin), after the apt repair of an image whose release left the mirrors, unless a Codex binary is -mounted (`codex_bin`: the standalone release's bin/codex, its release folder mounted read-only at /opt/hermit-codex by -Hermit's runner), answers `--version` as `version` and may be executed by the task's agent user. Harbor's link of codex -into /usr/local/bin looks for it without nvm, so on glibc it finds none: the plan loads nvm (`nvm: true`) with the agent -user's own HOME, and a mount's bin/ goes on PATH after that, ahead of nvm's. A mount that serves is linked onto PATH -with the release's rg (when the image has none) and, for an image with no CA bundle at all (TB2.0's bare ubuntu:24.04), -the host's bundle where Debian keeps one: Codex trusts the system's roots, and without them every request fails ("error -sending request", then "workspace routing discovery failed") and the trial ends in seconds with no request served (a -provider that could not serve: infra, run again to the same end). Harbor's install would have brought ca-certificates -with curl. - -The session is the plan's, as harborside.hermit runs every CLI's: Hermit's own `codex exec --json ...` with its home -(CODEX_HOME=/installed-agent/hermit-home: config.toml, roles, a sign-in), the deployment's .codex/ in the workdir -(trusted as a project there, its hooks run from hermit//), and the stream agent-codex.jsonl. Harbor's own -run (its config.toml, its codex.txt, its deletion of CODEX_HOME after the session) is not used. -""" -from __future__ import annotations - -import os -from pathlib import Path, PurePosixPath -import shlex -import ssl - -from harbor.agents.installed.codex import Codex, CodexOptions -from harbor.environments.base import BaseEnvironment -from pydantic import Field - -from harborside.hermit import HermitInstalled, HermitOptions - -# Where TLS clients find the system's CA bundle. Codex trusts the system's roots (unless SSL_CERT_FILE or -# CODEX_CA_CERTIFICATE names a bundle); Harbor's install brings them (ca-certificates, with curl), a bare image has -# none. -CA_BUNDLES = ("/etc/ssl/certs/ca-certificates.crt", "/etc/pki/tls/certs/ca-bundle.crt", "/etc/ssl/ca-bundle.pem", - "/etc/pki/tls/cacert.pem", "/etc/ssl/cert.pem") - - -def host_ca_bundle() -> Path | None: - """The CA bundle OpenSSL trusts on this host, else certifi's; None when there is neither.""" - paths = ssl.get_default_verify_paths() - for name in (os.environ.get(paths.openssl_cafile_env or ""), paths.cafile, paths.openssl_cafile): - if name and Path(name).is_file(): - return Path(name).resolve() - try: - import certifi # Harbor's own dependency; optional here - except ImportError: - return None - return Path(certifi.where()) - - -class HermitCodexOptions(HermitOptions, CodexOptions): - """Harbor's Codex kwargs (accepted, unused: the plan holds the session) and Hermit's. `extra="forbid"`.""" - - codex_bin: str | None = Field(default=None, description="A Codex binary mounted in the container (the standalone " - "release's bin/codex, its release folder mounted " - "read-only), used when it runs there as `version`, instead " - "of Harbor's per-trial install (apt curl nodejs npm " - "ripgrep, nvm, npm).") - - -class HermitCodex(HermitInstalled, Codex): - CLI = "codex" - PROGRAM = "codex" - BIN_OPTION = "codex_bin" - options_model = HermitCodexOptions - options: HermitCodexOptions - - @staticmethod - def name() -> str: - return "codex" # Harbor keeps treating the trial as a Codex one - - async def _after_mount(self, environment: BaseEnvironment, binary: PurePosixPath) -> None: - """The release's rg when the image has none (Codex's file search runs it), and a CA bundle.""" - rg = shlex.quote(str(binary.parent.parent / "codex-path" / "rg")) - await self.exec_as_root(environment, command=f"if ! command -v rg >/dev/null 2>&1 && [ -x {rg} ]; then " - f"mkdir -p /usr/local/bin && ln -sf {rg} /usr/local/bin/rg; fi") - await self._ensure_ca_bundle(environment) - - async def _ensure_ca_bundle(self, environment: BaseEnvironment) -> None: - """TLS roots for the mounted binary: the image's own bundle, else the host's copied to the Debian path.""" - check = " || ".join(f"[ -s {path} ]" for path in CA_BUNDLES) - if (await environment.exec(command=check, user="root")).return_code == 0: - return - source = host_ca_bundle() - if source is None: - self.logger.warning("The image has no CA bundle and neither has the host: Codex may not reach its API") - return - target = PurePosixPath(CA_BUNDLES[0]) - await self.exec_as_root(environment, command=f"mkdir -p {shlex.quote(str(target.parent))}") - await environment.upload_file(source, str(target)) - await self.exec_as_root(environment, command=f"chmod 644 {shlex.quote(str(target))}") - self.logger.info("The image has no CA bundle: copied the host's %s to %s", source, target) diff --git a/hermit/harbor/harborside/driver.py b/hermit/harbor/harborside/driver.py deleted file mode 100644 index 8b851b64..00000000 --- a/hermit/harbor/harborside/driver.py +++ /dev/null @@ -1,346 +0,0 @@ -"""One multi-step Harbor task, run a step at a time on request: a stream's environment that lives between the -Trainer's steps. - - python -m harborside.driver --task-dir --stream-dir --name --socket - --agent-json '{"import_path": ..., "model_name": ..., "kwargs": {...}}' --environment-json '{...}' - [--verifier-json '{...}'] [--trial-json '{"verifier_timeout_multiplier": 2.0, ...}'] [--idle-s 14400] - -`--trial-json` holds the run's fields of Harbor's TrialConfig (the verifier's and the environment build's timeout -multipliers: TRIAL_FIELDS), which the stream's trial takes as a job's trials do; any other key is refused. - -Harbor's own `MultiStepTrial` runs every step of a task in one go. This subclass keeps its lifecycle (the -environment, the agent's install, each step's setup, agent phase, verifier and archive, the trial's result at the -end) and hands the step loop to whoever connects to the socket: a request names the next step (and the launch plan -Hermit rendered for it, which the agent takes before the step: `set_plan`, and the hosts the step's agent phase lets -the agent reach: the trial's agent config's extra_allowed_hosts, which Harbor reads for each step's network plan), the -driver runs it and answers with the step's result; `stop` ends the trial (the aggregate reward, the environment down). Between -requests the environment waits; after `--idle-s` without one it stops by itself. Steps run in order; a request for -another step is refused with the index the driver is at, so concurrent rollouts of one stream queue up outside, and -one for a step the task has not gets an error reply. A step whose launch plan the agent cannot take fails as -HermitLaunchError (infra on Hermit's side). - -The driver writes `driver.json` (pid, socket, trial name and dir, the task's step names) in the stream directory -once it listens and logs to its -stderr (the client sends that to driver.log); Harbor's trial log, each step's agent/ and verifier/ and the final -result.json are under the trial directory. Each step's result is also kept as `replies/.json`, for a client -that asked, lost the connection and asks again (`result`). A step whose result cannot be kept (a full disk: its reply's -file, an aborted step's agent/ moved under the step) is answered with an error, and a log line that cannot be written is -lost: neither leaves a request unanswered. - -Protocol: one JSON object per line, one request per connection (see hermit/harbor/stream.py). -""" -from __future__ import annotations - -import argparse -import asyncio -import json -import os -from pathlib import Path -import signal -import sys -import time -import traceback -from uuid import uuid4 - -from harbor.models.trial.config import AgentConfig, EnvironmentConfig, TaskConfig, TrialConfig, VerifierConfig -from harbor.models.trial.result import ExceptionInfo, StepResult -from harbor.trial.multi_step import MultiStepTrial - -from harborside.hermit import HermitLaunchError - -# --trial-json's keys: the run's fields of Harbor's TrialConfig, which a stream's trial takes as a job's trials do -TRIAL_FIELDS = ("verifier_timeout_multiplier", "environment_build_timeout_multiplier") - - -def log(message: str) -> None: - try: - print(f"[driver {time.strftime('%H:%M:%S')}] {message}", file=sys.stderr, flush=True) - except OSError: # a full disk: the line is lost, never a request's answer - pass - - -class StepDriverTrial(MultiStepTrial): - """MultiStepTrial whose step loop is fed by requests.""" - - @classmethod - async def create(cls, config: TrialConfig) -> "StepDriverTrial": - cls._resolve_agent_skills(config) - task, download = await cls._load_task(config) - if not task.has_steps: - raise ValueError(f"{task.name} has no [[steps]]: a single-step task is a trial, not a stream") - return cls(config, _task=task, _task_download_result=download) - - def __init__(self, *args, **kwargs): - super().__init__(*args, **kwargs) - self.commands: asyncio.Queue = asyncio.Queue() - self.prepared = asyncio.Event() - self.done = asyncio.Event() # trial.run() returned: result.json written, environment down - self.error: str | None = None - self.running: str | None = None - self.next_index = 1 - self.aborted: str | None = None - self.loop_done = False # the step loop ended; the environment may still be stopping - self.finished = False # the environment is down - self.replies_dir = Path(self.config.trials_dir).parent / "replies" - self.replies_dir.mkdir(parents=True, exist_ok=True) - - # --- lifecycle --------------------------------------------------------------------------------------------- - async def _prepare(self) -> None: - try: - await super()._prepare() - except BaseException as exc: - self.error = f"{type(exc).__name__}: {exc}" - raise - finally: - self.prepared.set() - - async def _run(self) -> None: - self.result.step_results = [] - steps = {s.name: s for s in self.task.config.steps or []} - total = len(steps) - while True: - command, future = await self.commands.get() - if command.get("op") == "stop": - self._resolve(future, {"ok": True, "stopped": True}) - break - try: - step = steps[command["name"]] - except KeyError: # answered; the loop waits for the next request - self._resolve(future, {"ok": False, "error": f"{self.task.name} has no step {command.get('name')!r} " - f"(its steps: {', '.join(steps)})"}) - continue - index = int(command["index"]) - self.running = step.name - step_result = StepResult(step_name=step.name) - self.result.step_results.append(step_result) - try: - self._apply(command) - await self._run_step(step, step_result, index=index, total=total) - except Exception as exc: # the step's own errors are recorded inside _run_step - if step_result.exception_info is None: - step_result.exception_info = ExceptionInfo.from_exception(exc) - log(f"step {step.name}: {type(exc).__name__}: {exc}") - stop = self._should_stop_after_step(step, step_result) - if stop: - self.aborted = (f"step {step.name} failed" if step_result.exception_info and not step_result.verifier_result - else f"step {step.name} under min_reward") - self.running = None - self.next_index = index + 1 - reply = {"ok": True, "index": index, "step": step_result.model_dump(mode="json"), - "paths": {"trial_dir": str(self.paths.trial_dir), "step_dir": str(self.paths.step_dir(step.name))}, - "aborted": self.aborted, "last": index >= total} - try: - if stop: - self._move_agent_dir_to_step(step) - (self.replies_dir / f"{index}.json").write_text(json.dumps(reply, default=str)) - except OSError as exc: # a full disk: answered all the same, as an error - reply = {"ok": False, "error": f"step {step.name}'s result could not be kept: {type(exc).__name__}: " - f"{exc}"} - self._resolve(future, reply) - if stop or index >= total: - break - self.loop_done = True - self._drain({"ok": True, "stopped": True}) - self.result.verifier_result = self._select_multi_step_reward() - try: - await self._stop_agent_environment() - self.paths.cleanup_empty_mount_dirs() - finally: - self.finished = True - - def _drain(self, reply: dict) -> None: - """Answer whatever was queued after the step loop ended.""" - while not self.commands.empty(): - _, future = self.commands.get_nowait() - self._resolve(future, reply) - - def _apply(self, command: dict) -> None: - """What a step changes on the agent: the hosts its agent phase lets the agent reach (the request's - `allowed_hosts`, the step's own: Harbor's extra_allowed_hosts, read for each step's network plan, none for a - step whose agent phase is public; a request that names none leaves them as they are), and the launch plan - Hermit rendered for it (its deployment, the CLI's home, the command), given as the request's `plan` or in its - agent kwargs (the stream client sends the step's). A step that gives no plan runs with the one the agent - already has; hosts or a plan the agent cannot use fail the step as HermitLaunchError (infra on Hermit's side, - not the agent's 0).""" - if "allowed_hosts" in command: - from harbor.models.task.config import normalize_allowed_hosts - try: - self.config.agent.extra_allowed_hosts = normalize_allowed_hosts(list(command["allowed_hosts"] or [])) - except (ValueError, TypeError) as exc: - raise HermitLaunchError(f"the step's allowed hosts: {exc}") from exc - plan = command.get("plan") or (command.get("agent_kwargs") or {}).get("plan") - if not plan: - log(f"step {command['name']} gives no launch plan: it runs with the agent's current one") - elif hasattr(self.agent, "set_plan"): - try: - self.agent.set_plan(plan) - except ValueError as exc: - raise HermitLaunchError(str(exc)) from exc - else: - log(f"the agent {type(self.agent).__name__} takes no launch plan: step {command['name']} runs as " - "configured") - - @staticmethod - def _resolve(future: asyncio.Future, value: dict) -> None: - if not future.done(): - future.set_result(value) - - # --- requests ---------------------------------------------------------------------------------------------- - def status(self) -> dict: - return {"ok": True, "ready": self.prepared.is_set() and self.error is None, "error": self.error, - "next_index": self.next_index, "running": self.running, "finished": self.finished, - "aborted": self.aborted, "trial_dir": str(self.paths.trial_dir), "pid": os.getpid()} - - async def handle(self, request: dict) -> dict: - op = request.get("op") - if op == "status": - return self.status() - if op == "result": - path = self.replies_dir / f"{int(request.get('index', 0))}.json" - if path.is_file(): - return json.loads(path.read_text()) - return {"ok": False, "error": f"no result for step {request.get('index')}"} - if op == "stop": - if not (self.loop_done or self.error): - future: asyncio.Future = asyncio.get_running_loop().create_future() - await self.commands.put(({"op": "stop"}, future)) - await future - # Answer once the environment is down and the trial's result.json is written (serve() sets `done`). - try: - await asyncio.wait_for(self.done.wait(), timeout=900) - except asyncio.TimeoutError: - pass - return {"ok": True, "stopped": True, "finished": self.finished, "done": self.done.is_set()} - if op == "step": - if self.error: - return {"ok": False, "error": self.error} - if self.loop_done or self.aborted: - return {"ok": False, "error": f"the stream {'aborted: ' + self.aborted if self.aborted else 'finished'}", - "aborted": self.aborted, "finished": self.finished} - if self.running is not None: - return {"ok": False, "busy": self.running, "next_index": self.next_index} - if int(request.get("index", 0)) != self.next_index: - return {"ok": False, "out_of_order": True, "next_index": self.next_index} - future = asyncio.get_running_loop().create_future() - await self.commands.put((request, future)) - return await future - return {"ok": False, "error": f"unknown op {op!r}"} - - -async def serve(args) -> int: - agent = json.loads(args.agent_json) - environment = json.loads(args.environment_json) - verifier = json.loads(args.verifier_json) if args.verifier_json else {} - fields = json.loads(args.trial_json) if args.trial_json else {} - if not isinstance(fields, dict) or set(fields) - set(TRIAL_FIELDS): - raise ValueError(f"--trial-json takes a JSON object of {', '.join(TRIAL_FIELDS)} (Harbor's TrialConfig " - f"fields), not {args.trial_json}") - stream_dir = Path(args.stream_dir) - config = TrialConfig(task=TaskConfig(path=Path(args.task_dir)), trial_name=args.name, - trials_dir=stream_dir / "trials", agent=AgentConfig(**agent), - environment=EnvironmentConfig(**environment), verifier=VerifierConfig(**verifier), - job_id=uuid4(), **fields) - trial = await StepDriverTrial.create(config) - last_request = time.time() - pending: list[asyncio.Future] = [] - - async def on_connect(reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None: - nonlocal last_request - last_request = time.time() - try: - line = await reader.readline() - request = json.loads(line.decode() or "{}") - reply = await trial.handle(request) - except Exception as exc: # noqa: BLE001 - the client gets the error - reply = {"ok": False, "error": f"{type(exc).__name__}: {exc}", "traceback": traceback.format_exc()[-2000:]} - try: - writer.write((json.dumps(reply, default=str) + "\n").encode()) - await writer.drain() - except (ConnectionError, OSError): - pass - finally: - writer.close() - last_request = time.time() - - sock = Path(args.socket) - sock.unlink(missing_ok=True) - server = await asyncio.start_unix_server(on_connect, path=str(sock)) - (stream_dir / "driver.json").write_text(json.dumps({"pid": os.getpid(), "socket": str(sock), "name": args.name, - "steps": [s.name for s in trial.task.config.steps or []], - "trial_dir": str(trial.paths.trial_dir), - "started": time.strftime("%Y-%m-%dT%H:%M:%S")}) + "\n") - log(f"listening on {sock}; trial {trial.paths.trial_dir}") - - async def request_stop(reason: str) -> None: - if trial.loop_done: - return - log(f"stopping: {reason}") - future: asyncio.Future = asyncio.get_running_loop().create_future() - pending.append(future) - await trial.commands.put(({"op": "stop"}, future)) - - async def idle_watch() -> None: - while not trial.loop_done: - await asyncio.sleep(30) - if trial.running is None and time.time() - last_request > args.idle_s: - await request_stop(f"idle for {args.idle_s:.0f} s") - return - - loop = asyncio.get_running_loop() - for sig in (signal.SIGTERM, signal.SIGINT): - loop.add_signal_handler(sig, lambda s=sig: asyncio.ensure_future(request_stop(f"signal {s}"))) - watcher = asyncio.create_task(idle_watch()) - result = await trial.run() # _prepare, then _run (the request loop), then _finalize - watcher.cancel() - _mark_ending(stream_dir) - # A request queued after the loop ended (a step after an environment failure) is answered now. - while not trial.commands.empty(): - _, future = trial.commands.get_nowait() - if not future.done(): - future.set_result({"ok": False, "error": trial.error or "the stream finished", "finished": True}) - trial.done.set() - await asyncio.sleep(0.5) # a `stop` waiting on `done` writes its answer - server.close() - await server.wait_closed() - sock.unlink(missing_ok=True) - log(f"done: {'error ' + trial.error if trial.error else 'reward ' + str((result.verifier_result or {}) and result.verifier_result.rewards)}") - return 0 if trial.error is None else 1 - - -def _mark_ending(stream_dir: Path) -> None: - """The driver's record says it is ending (`ending`), before any request is answered from here on (a `stop` waits - on `done`): a client never takes a driver that is closing for a live one (StreamClient.running), and starts the - stream's next driver only once this process has ended (StreamClient.ensure), whose exit removes the socket's path - that the next driver binds. Written whole (os.replace); a disk that refuses it leaves the record as it was.""" - path = stream_dir / "driver.json" - try: - record = json.loads(path.read_text()) - record["ending"] = True - part = path.with_name(path.name + ".part") - part.write_text(json.dumps(record) + "\n") - os.replace(part, path) - except (OSError, ValueError, AttributeError, TypeError): - pass - - -def main(argv=None) -> int: - parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - parser.add_argument("--task-dir", required=True) - parser.add_argument("--stream-dir", required=True) - parser.add_argument("--name", required=True) - parser.add_argument("--socket", required=True) - parser.add_argument("--agent-json", required=True) - parser.add_argument("--environment-json", default="{}") - parser.add_argument("--verifier-json", default="{}") - parser.add_argument("--trial-json", default="{}") - parser.add_argument("--idle-s", type=float, default=4 * 3600) - args = parser.parse_args(argv) - try: - return asyncio.run(serve(args)) - except Exception as exc: # noqa: BLE001 - the log says why the driver died - log(f"fatal: {type(exc).__name__}: {exc}\n{traceback.format_exc()}") - return 2 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/hermit/harbor/harborside/dsh.py b/hermit/harbor/harborside/dsh.py deleted file mode 100644 index 3ab58fea..00000000 --- a/hermit/harbor/harborside/dsh.py +++ /dev/null @@ -1,73 +0,0 @@ -"""HermitDsh: DeepSeek Harness (dsh) in a Harbor trial, its session launched from Hermit's plan (harborside.hermit). - -Imported by Harbor as `harborside.dsh:HermitDsh` (Harbor ships no dsh agent). The install is the one Hermit documents -(docs/get-started/installation.md): Node.js 24.9 or later, here from Harbor's nvm snippet with Node 24 (checked to be -at least 24.9; nvm's builds need glibc, so a musl image is refused with that reason), then `npm install -g -@deepseek-ai/dsh` at `version` (Hermit's runner gives its tested build, a prerelease that npm's `latest` tag may not -name, when the config names none), as the agent user; or a dsh mounted at /opt/hermit-dsh (`dsh_bin`) that answers -`--version` as `version` and that the task's agent user may execute. Setup may take longer than Harbor's 360 s default -under load (600 s here; the runner's setup_timeout_sec overrides it). - -The session is the plan's, as harborside.hermit runs every CLI's: Hermit's own `dsh --profile headless ... --patch -/hermit.patch.yml --json --permission-mode danger-full-access --approval-policy never` (the task's container is -the session's sandbox: dsh's own needs privileges a Harbor container lacks), its home's DSH_HOME and HOME under -/installed-agent/hermit-home, the deployment's AGENTS.md (within dsh's 65,536-byte limit once the image's own follows -it, or the trial fails saying so) and .dsh/ in the workdir, stderr in the stream agent-dsh.jsonl (where dsh says a -tool server did not start). The plan loads nvm with the agent user's own HOME before dsh's HOME replaces it. dsh exits -0 when SIGTERM stops it: Hermit's side reads its verdict from the stream, not its exit code. -""" -from __future__ import annotations - -from harbor.agents.installed.node_install import nvm_node_install_snippet -from harbor.environments.base import BaseEnvironment -from pydantic import Field - -from harborside.hermit import MUSL, VERSION, HermitInstalled, HermitOptions - -PACKAGE = "@deepseek-ai/dsh" -NODE_MAJOR = 24 -NODE_MIN = (24, 9) -NODE_CHECK = ("node -e 'const [a, b] = process.versions.node.split(\".\").map(Number); " - f"if (a < {NODE_MIN[0]} || (a === {NODE_MIN[0]} && b < {NODE_MIN[1]})) {{ console.error(\"hermit: dsh " - f"needs Node {NODE_MIN[0]}.{NODE_MIN[1]} or later, not \" + process.versions.node); process.exit(1); }}'") - - -class HermitDshOptions(HermitOptions): - """Hermit's kwargs for dsh (Harbor has no options of its own for it). `extra="forbid"`.""" - - dsh_bin: str | None = Field(default=None, description="A dsh mounted in the container (at /opt/hermit-dsh), used " - "when it runs there as `version`, instead of the per-trial " - "install (nvm, Node 24, npm).") - - -class HermitDsh(HermitInstalled): - CLI = "dsh" - PROGRAM = "dsh" - BIN_OPTION = "dsh_bin" - DEFAULT_SETUP_TIMEOUT_SEC = 600.0 - options_model = HermitDshOptions - options: HermitDshOptions - - @staticmethod - def name() -> str: - return "dsh" - - def get_version_command(self) -> str | None: - return "if [ -s ~/.nvm/nvm.sh ]; then . ~/.nvm/nvm.sh; fi; dsh --version" - - def parse_version(self, stdout: str) -> str: - found = VERSION.search(stdout or "") - return found.group(1) if found else (stdout or "").strip() - - def install_command(self) -> str: - spec = f"@{self._version}" if self._version else "@latest" - return ("set -euo pipefail; " - f"if {MUSL}; then echo 'hermit: dsh needs Node {NODE_MIN[0]}.{NODE_MIN[1]} or later from nvm, whose " - "builds need glibc, and this image is musl' >&2; exit 1; fi; " - f"{nvm_node_install_snippet(node_major=NODE_MAJOR)} && {NODE_CHECK} && " - f"npm install -g {PACKAGE}{spec} && dsh --version") - - async def install_cli(self, environment: BaseEnvironment) -> None: - """curl (nvm's installer), then Node 24 and dsh as the agent user (the module's docstring).""" - await self.ensure_system_dependencies(environment, ("curl",)) - await self.exec_as_agent(environment, command=self.install_command()) diff --git a/hermit/harbor/harborside/hermit.py b/hermit/harbor/harborside/hermit.py deleted file mode 100644 index d70215c4..00000000 --- a/hermit/harbor/harborside/hermit.py +++ /dev/null @@ -1,1166 +0,0 @@ -"""What every HermitXXX does in a Harbor trial: carry out the launch plan Hermit rendered for its session. - -Hermit's side (`hermit.harbor.launch`, in Hermit's venv) renders each trial's session of the acting CLI with the -CLI's own Base Agent (`hermit.agent.base.cli`: prepare_home, home_env, session_env, command), against host staging -paths translated to the container's, and writes it as `/launch/plan.json` (schema version 1). All per-CLI -knowledge stays there; the class Harbor imports (harborside.codex:HermitCodex, ...) is this mixin over Harbor's agent -of that CLI, which brings Harbor's install of the CLI (or uses a mounted binary) and then, in `run`, carries the plan -out the same way for every CLI: - - /hermit// the adapter's copy (removed, then uploaded) - / what the host wrote beside it (.codex/, .claude/, .pi/, .dsh/, opencode.json, - mini-swe-agent.yaml, ...); a stream's earlier step's files that this - deployment does not replace are removed - / Hermit's context first, then the image's own file (captured once per container - in /installed-agent/hermit/original-), within the plan's context_limit - (the image's text counted as the CLI escapes it: context_escaped) - /.hermit-deploy.json the deployment's record, always (an adapter's hook runner reads its host there) - /installed-agent/hermit-home/ the CLI's home as Hermit rendered it: @@HERMIT_WORKDIR@@ filled in in its - files, its sign-in files mode 600, all of it the task's agent user's - /installed-agent/hermit/run/ the launch script, the prompt and the session's pid - /logs/agent/ the session's stdout and stderr (agent-.jsonl): Hermit reads it with - the CLI's Base Agent after the trial (outcome, parse, steps) - /logs/agent/hermit-copy// the plan's copy_back directories (an adapter's memory), after the session - /logs/agent/hermit-collect/ the plan's collect files (last_message.txt, a session's database) - -A link in the deployment or the home is uploaded as the link it is, as `docker cp` keeps the adapter's own copy: it is -never followed to a file of this host (an evolved adapter's link would otherwise carry that file into the container). - -The command runs as the task's agent user, in a process group of its own (setsid; its pid in a file), from , -with nvm loaded when the plan says so (with the user's own HOME, before a variable of the plan replaces it), the -class's install directories and the plan's own first on PATH, the plan's variables, HERMIT_HOOK_LOGS=/logs/agent (where -an adapter's hooks write their audit and state, which come back with the trial), and the prompt (Harbor's instruction -and the plan's prompt_suffix, with its prompt_slash_prefix before one that starts with `/`) in its arguments -(@@HERMIT_PROMPT@@) or on its stdin, read from a file in the container. - -A key or a token never leaves Harbor's environment by value (Codex's ChatGPT sign-in is a file of the plan's home, mode -600): an `env_from_host` variable reaches `docker compose exec` by name -(`-e NAME`: the compose client reads the value from its own environment, Harbor's); one the plan gives another name is -set under it first thing in the launch script, every source read as Harbor passed it before nvm, a variable of the -plan or another entry could replace one, and a source no entry sets is unset there, so the plan's own variables apply -over them (Claude Code on Anthropic's own API: ANTHROPIC_AUTH_TOKEN takes the key from ANTHROPIC_API_KEY, which the -plan then empties). An `env_by_fd` one (Claude Code's subscription token) goes on into a pipe on descriptor 3 (the CLI -reads it from there once), never in the environment of the launch script or of anything it starts. Neither is in a -command line, a file or a log; only a local Docker environment can pass a variable by name, so a plan with secrets is -refused elsewhere. - -However the session ends (its exit, a non-zero exit, Harbor's timeout cancelling `run`), a session still running is -killed with its group (TERM, the plan's grace, KILL): Harbor's timeout cancels only Harbor's wait on the -`docker compose exec` client, and a CLI left running would go on editing under the verifier. On Harbor's default path -(an exec with no log callback streaming the trial's output, as under `harbor run`, and every `docker compose cp`) a -client whose wait was cancelled goes on running on this host, as Harbor ends one only when the call's own timeout_sec -runs out (the session's exec has none), and a client Harbor ends (an exec's with a log callback, on the cancellation as -well) leaves the CLI running in the container all the same. A command the CLI ran in a session of its own (Claude Code -runs its shell tool's commands so) is not in that group: unless the CLI ends it within the grace, it goes on. A session -that ended by itself leaves what it started in the background running (a server the task asked for), as Harbor's own -agents and Hermit's own sessions do. Then the copy-back, the collected files and the files returned to the host (a -refreshed ChatGPT sign-in) are taken, each on its own and from / (Docker refuses an exec in a workdir the session -removed), so that none of them hides the session's own error, and Harbor's timeout landing in one of them does not keep -the others from running (the one it cut short is not run again, but for the return of the files, which runs once more: -its copy may overlap the cancelled one's, a `docker compose cp` left running on this host that writes the same file), -nor does it hide that error: the cancellation goes on after the last, unless the session raised an error of its own, -which then goes on as it is (Harbor's wait_for turns only a cancellation into the agent's timeout, so a launch failure -or a non-zero exit is not taken for one). Once the run is cancelled, each of them has AFTER_CANCEL_S at most (a hung -container would hold them), and a second cancellation (the job's, after Harbor's timeout) goes on at once. A non-zero -exit raises Harbor's NonZeroAgentExitCodeError (the verifier still runs; Hermit's side reads the session's verdict from -its stream). What keeps the session from starting that is not the agent's doing (no plan, secrets by name outside a -local Docker environment or missing from Harbor's environment, no agent user, a CLI that cannot be installed or run -there, one of harborside's own steps before the session's exec failing: an upload, a chown, a probe of the container; a -launch script that cannot enter the workdir; a non-zero exit that left no stream, which the launch script makes before -the CLI starts (a stream's earlier step's is removed before the exec), as when the exec failed in Docker; the session's -exec this machine could not run (an OSError: the compose client not spawned, EMFILE, EAGAIN), and, after a non-zero -exit, a look for its stream it could not run: whether the CLI started cannot be told) raises HermitLaunchError, which -Hermit's side counts as infra; a context file over the CLI's limit once merged with the image's own raises -HermitContextLimitError, the adapter's result for the task (hermit.harbor.errors). - -Harbor's model connection is not used: the model, its endpoint and its key are the plan's. MODEL_CONNECTION is None, -so Harbor resolves no key, infers no endpoint and allowlists no host from one (Hermit's runner names the hosts), and a -base_url, an API key name or an API format given to the agent is refused. Nor are Harbor's readers of the CLI's own -logs: `populate_context_post_run` records metadata only (the CLI, whether an adapter was deployed and its id, whether -hooks and python3 were there, a mounted binary, the exit code, whether the session had to be killed, the session's -stream, the after-steps that failed or were cancelled (after_session_errors), and the plan's own metadata). -""" -from __future__ import annotations - -import asyncio -import base64 -import contextlib -import json -import os -from pathlib import Path, PurePosixPath -import re -import shlex -import shutil -import tempfile -from typing import Any, ClassVar - -from harbor.agents.agent_auth_resolution import ResolvedModelConnection -from harbor.agents.capabilities import AgentCapabilities -from harbor.agents.installed.base import BaseInstalledAgent, NonZeroAgentExitCodeError, with_prompt_template -from harbor.agents.options import InstalledAgentOptions -from harbor.environments.base import BaseEnvironment -from harbor.models.agent.context import AgentContext -from pydantic import Field, field_validator - -PLAN_VERSION = 1 -CLIS = ("codex", "claude", "pi", "opencode", "dsh", "mini-swe-agent", "answer") -HOME = "/installed-agent/hermit-home" # every CLI's home in the container -STATE = "/installed-agent/hermit" # harborside's own: the image's original context files, run/, bin/ -RUN = f"{STATE}/run" # the launch script, the prompt, the pid (the agent user's) -PIDFILE = f"{RUN}/session.pid" -TOOLS_BIN = f"{STATE}/bin" # the hermit-tool script (tools_script) -LOGS = "/logs/agent" # Harbor's agent log directory: it comes back with the trial -COPY_DIR = f"{LOGS}/hermit-copy" -COLLECT_DIR = f"{LOGS}/hermit-collect" -WORKDIR_MARK = "@@HERMIT_WORKDIR@@" -PROMPT_MARK = "@@HERMIT_PROMPT@@" -PROMPT_VAR = "HERMIT_PROMPT_TEXT" # the launch script's variable that holds the prompt (not exported) -ADAPTER_DIR = "hermit" -DEPLOY_RECORD = ".hermit-deploy.json" -HOOK_LOGS = "HERMIT_HOOK_LOGS" # where an adapter's hook runner writes its audit and state -LAUNCH_FAILED = 97 # the launcher's exit code when it cannot enter the workdir -CANNOT_ENTER = "hermit: cannot enter the workdir" # ... and what it says then: both, not a CLI's own exit 97 -FIRST_FD = 3 # env_by_fd's descriptors: 3, then 4, ... -DEFAULT_KILL_GRACE_S = 10 -AFTER_CANCEL_S = 120 # an after-step's time at most once the run was cancelled -COMPOSE_EXEC = ("_run_docker_compose_command", "_merge_env", "_resolve_user", "_output_callback") -NAME = re.compile(r"[A-Za-z_][A-Za-z0-9_]*") -VERSION = re.compile(r"(\d+\.\d+\.\d+(?:-[0-9A-Za-z.]+)?)") -# Harbor's own test for a musl image, made to hold under pipefail (the prefix of Harbor's exec_as_root and -# exec_as_agent; dsh's install sets its own): musl's ldd says what it is and exits 1 -MUSL = "{ ldd --version 2>&1 || true; } | grep -qi musl || [ -f /etc/alpine-release ]" -# A value read from the container is printed on a line of its own after its mark, and `marked` takes it from there: -# Harbor's docker environment merges an exec's stderr into its stdout, so other lines (a compose warning, an echo of -# the image's BASH_ENV) may come before and after it. A command's whole output (what a mounted binary says to -# --version) is printed between a begin and an end line of its own, and `between` takes it from there. -OWNER_LINE = "hermit-owner:" -PWD_LINE = "hermit-pwd:" -CONTEXT_LINE = "hermit-context:" # the image's own context file, in base64 -RETURN_LINE = "hermit-return:" # a file to return: its bytes (a file, no link there), or no -VERSION_BEGIN = "hermit-version-begin" # what a mounted binary says to --version, between these lines -VERSION_END = "hermit-version-end" - - -class HermitLaunchError(RuntimeError): - """What keeps the launch plan's session from starting that is not the agent's doing: no plan, secrets by name - outside a local Docker environment or missing from Harbor's, a container without the task's agent user, a CLI - that cannot be installed (its install failed: a download, a package, an image it refuses) or run there, a step of - harborside's own before the session's exec that failed (an upload, a chown, a probe of the container), a launch - script that could not enter the workdir, a non-zero exit that left no stream (the CLI never started, as when the - exec failed in Docker), the session's exec this machine could not run (an OSError: the compose client not - spawned, EMFILE, EAGAIN), and, after a non-zero exit, a look for its stream it could not run (whether the CLI - started cannot be told). Hermit's side counts the trial as infra (run again), not as the agent's 0.""" - - -class HermitContextLimitError(RuntimeError): - """The deployment's context file, merged with the image's own, is over the CLI's limit: the adapter's result for - the task (its context too long beside the task's own), scored as the agent's.""" - - -@contextlib.contextmanager -def _setup_step(what: str): - """One of harborside's own steps before the session's exec starts: an exception there (an exec of Harbor's that - exited non-zero, a `docker compose cp` that failed) is not the agent's doing, so it goes on as HermitLaunchError - saying what failed; HermitLaunchError, HermitContextLimitError and a cancellation go on as they are.""" - try: - yield - except (HermitLaunchError, HermitContextLimitError): - raise - except Exception as exc: # noqa: BLE001 - harborside's own, whatever it was - raise HermitLaunchError(f"hermit: {what} failed: {type(exc).__name__}: {exc}") from exc - - -async def _bounded(work) -> None: - """An after-step once the run was cancelled: AFTER_CANCEL_S at most (a hung container would hold it for good), - then a RuntimeError saying so; a cancellation goes on as it is.""" - try: - await asyncio.wait_for(work, AFTER_CANCEL_S) - except asyncio.TimeoutError: - raise RuntimeError(f"not done within {AFTER_CANCEL_S} s of the run's cancellation") from None - - -# The hermit-tool command for a CLI whose model runs bash commands only (mini-swe-agent): an adapter's tools, served -# by Hermit's tool server on the host, as one command. It answers as hermit.agent.base.mini_swe_agent_env answers the -# same command inside Hermit's environment class (which intercepts it before bash when that class runs), except when -# GET /tools answers with no list of tools (an HTTP error's JSON object, a list of anything but objects with a -# name and a description): here that is an `Error: RuntimeError: ...` line. -TOOL_SCRIPT = '''#!__PYTHON__ -"""hermit-tool: the adapter's tools as one command (written by harborside for the launch plan's tools_script). - - hermit-tool the tools, one per line - hermit-tool '' the tool's result; an error exits 1 - -The tools are the trial's tool server on the host, at the URL in $__URL_ENV__ (GET /tools, POST /call/). -""" -import json -import os -import sys -import urllib.error -import urllib.request - -URL = os.environ.get("__URL_ENV__", "").rstrip("/") - - -def request(url, body=None): - data = None if body is None else json.dumps(body).encode() - req = urllib.request.Request(url, data=data, method="GET" if body is None else "POST", - headers={"Content-Type": "application/json"}) - try: - with urllib.request.urlopen(req, timeout=600) as response: - return json.loads(response.read() or b"null") - except urllib.error.HTTPError as error: - return json.loads(error.read() or b"{}") - - -def text(result): - if isinstance(result, dict) and isinstance(result.get("text"), str): - return result["text"] - return result if isinstance(result, str) else json.dumps(result, ensure_ascii=False, default=str) - - -def main(words): - try: - tools = request(URL + "/tools") if URL else [] - # Anything but a list of tools (objects with a name and a description) is not the tools (an HTTP error's JSON - # object): the error said is the object's "error" (or FastAPI's "detail"), else the answer itself - if not isinstance(tools, list) or not all(isinstance(t, dict) and "name" in t and "description" in t - for t in tools): - raise RuntimeError(tools.get("error", tools.get("detail", tools)) if isinstance(tools, dict) else tools) - except Exception as exc: - return "Error: %s: %s" % (type(exc).__name__, exc), 1 - if not tools: - return "Error: no tools are available here", 1 - if not words: - return "\\n".join("%s: %s" % (t["name"], t["description"]) for t in tools), 0 - if len(words) > 2: - return ("Error: run hermit-tool as a command of its own: hermit-tool '', with the " - "arguments as one JSON object in single quotes"), 1 - try: - if words[0] not in {t["name"] for t in tools}: - raise ValueError("Unknown tool: " + words[0]) - arguments = json.loads(words[1]) if len(words) > 1 else {} - if not isinstance(arguments, dict): - raise ValueError("tool arguments must be a JSON object") - answer = request(URL + "/call/" + words[0], arguments) - if "error" in answer: - raise RuntimeError(answer["error"]) - return text(answer.get("result")), 0 - except Exception as exc: - return "Error: %s: %s" % (type(exc).__name__, exc), 1 - - -if __name__ == "__main__": - out, code = main(sys.argv[1:]) - print(out) - sys.exit(code) -''' - - -def _q(value: str) -> str: - return shlex.quote(str(value)) - - -def _fresh_dir(path: str) -> str: - """A root's shell step that makes `path` (under /logs/agent, which Harbor leaves writable by every user) a new - directory no other user can write in, and enters it: whatever was there removed first (a link the agent's user - planted to a directory of root's, a folder of its own holding links), and the directory entered checked to be - `path` itself, never a link put there meanwhile; else the step exits 1. What follows works in it by relative - names, so that nothing the agent's user can rename or replace is followed.""" - where = PurePosixPath(path) - return (f"rm -rf {_q(path)} && mkdir -m 755 {_q(path)} && cd -P {_q(path)} && " - f'[ "$(pwd -P)" = "$(cd -P {_q(str(where.parent))} && pwd -P)/{where.name}" ] || exit 1') - - -def marked(output: str | None, mark: str) -> str | None: - """The value a command printed after `mark` (`echo ""`), from the last line that carries the mark, - stripped; None when no line does. Other lines of the exec's output (its stderr merged in) are not the value.""" - lines = [line for line in (output or "").splitlines() if mark in line] - return lines[-1].split(mark, 1)[1].strip() if lines else None - - -def between(output: str | None, begin: str, end: str) -> str | None: - """What a command printed between a line `begin` and a line `end` it printed around it (`echo ; - ; printf '\\n\\n'`), after the last line `begin`; None when no such lines are there. Other lines of - the exec's output (its stderr merged in) are not the command's.""" - lines = (output or "").splitlines() - begins = [i for i, line in enumerate(lines) if line.strip() == begin] - rest = lines[begins[-1] + 1:] if begins else [] - ends = [i for i, line in enumerate(rest) if line.strip() == end] - return "\n".join(rest[:ends[0]]) if ends else None - - -def _relative(path: str) -> bool: - """A relative POSIX path that stays below where it is put (no `..`, not absolute, not empty).""" - parts = PurePosixPath(str(path)).parts - return bool(parts) and not PurePosixPath(str(path)).is_absolute() and ".." not in parts - - -def _absolute(path: str) -> bool: - """An absolute container path, or one under the workdir's placeholder.""" - text = str(path) - return text.startswith("/") or text.startswith(WORKDIR_MARK) - - -def _regular(path: Path) -> bool: - """A regular file reached through no link: what harborside reads on the host. A link in a deployment or a home - is uploaded as the link it is (as `docker cp` keeps the adapter's own), never followed to a file of this host.""" - return path.is_file() and not path.is_symlink() - - -def _copy_tree(source: Path, target: Path, workdir: str, *, left_out: set = frozenset(), - unfilled: set = frozenset()) -> set: - """`source`'s entries but `left_out` (names at its top) copied to `target`, links as links, @@HERMIT_WORKDIR@@ - filled in in the regular files but `unfilled` (paths relative to `source`); the copied files' and links' paths - relative to `target`.""" - target.mkdir(parents=True, exist_ok=True) - for entry in sorted(source.iterdir()): - if entry.name in left_out: - continue - if entry.is_symlink(): - (target / entry.name).symlink_to(os.readlink(entry)) - elif entry.is_dir(): - shutil.copytree(entry, target / entry.name, symlinks=True) - elif entry.is_file(): - shutil.copy2(entry, target / entry.name) - entries = set() - for path in sorted(target.rglob("*")): # a link to a folder is an entry, not descended into - rel = path.relative_to(target).as_posix() - if _regular(path) and rel not in unfilled: - data = path.read_bytes() - if WORKDIR_MARK.encode() in data: - path.write_bytes(data.replace(WORKDIR_MARK.encode(), workdir.encode())) - if path.is_symlink() or path.is_file(): - entries.add(rel) - return entries - - -def plan_problems(plan: dict) -> list[str]: - """What is wrong with a launch plan's fields (schema version 1), as messages naming the field; [] when none. - The answer agent's own block is HermitAnswer's to check.""" - problems: list[str] = [] - - def names(field: str, value: Any, *, values_are_names: bool) -> None: - if value is None: - return - if not isinstance(value, dict): - problems.append(f"{field} must be a mapping") - return - for key, item in value.items(): - if not isinstance(key, str) or not NAME.fullmatch(key): - problems.append(f"{field} has a variable name that is not one: {key!r}") - if not isinstance(item, str) or (values_are_names and not NAME.fullmatch(item)): - problems.append(f"{field}[{key}] must be {'a variable name' if values_are_names else 'a string'}") - - cli = plan.get("cli") - if cli != "answer": - command = plan.get("command") - if not isinstance(command, list) or not command or not all(isinstance(a, str) for a in command): - problems.append("command must be a non-empty list of strings") - stream = plan.get("stream") - if not isinstance(stream, str) or not stream or "/" in stream or stream in (".", ".."): - problems.append("stream must be a file name in /logs/agent") - home = plan.get("home") - if not isinstance(home, dict) or not home.get("host_dir"): - problems.append("home must name its host_dir") - else: - if not Path(str(home["host_dir"])).is_dir(): - problems.append(f"home.host_dir is not a directory: {home['host_dir']}") - if not str(home.get("container_dir") or HOME).startswith("/"): - problems.append("home.container_dir must be an absolute container path") - for name in home.get("secret_files") or []: - if not _relative(name): - problems.append(f"home.secret_files must be paths relative to the home, not {name!r}") - names("env", plan.get("env"), values_are_names=False) - names("env_from_host", plan.get("env_from_host"), values_are_names=True) - names("env_by_fd", plan.get("env_by_fd"), values_are_names=True) - deploy = plan.get("deploy") - if deploy is not None: - if not isinstance(deploy, dict) or not deploy.get("work_dir"): - problems.append("deploy must name its work_dir") - else: - if not Path(str(deploy["work_dir"])).is_dir(): - problems.append(f"deploy.work_dir is not a directory: {deploy['work_dir']}") - name = deploy.get("context_file") - if name is not None and (not isinstance(name, str) or not _relative(name) or "/" in name): - problems.append(f"deploy.context_file must be a file name, not {name!r}") - limit = deploy.get("context_limit") - if limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0): - problems.append("deploy.context_limit must be a positive number of bytes or null") - escaped = deploy.get("context_escaped") - if escaped is not None and (not isinstance(escaped, str) or not escaped): - problems.append("deploy.context_escaped must be the text the CLI escapes by a byte, or null") - for rel in deploy.get("copy_back") or []: - if not isinstance(rel, str) or not _relative(rel): - problems.append(f"deploy.copy_back must hold paths relative to the workdir, not {rel!r}") - bundle = plan.get("bundle") - if bundle is not None and (not isinstance(bundle, dict) or not Path(str(bundle.get("host_dir") or "")).is_dir() - or not str(bundle.get("container_dir") or "").startswith("/")): - problems.append("bundle must name an existing host_dir and an absolute container_dir") - for entry in plan.get("collect") or []: - if not isinstance(entry, dict) or not _absolute(entry.get("from", "")) or not _relative(entry.get("to", "")): - problems.append(f"collect entries are {{from: a container path, to: a path relative to " - f"{COLLECT_DIR}}}, not {entry!r}") - for entry in plan.get("return_files") or []: - limit = entry.get("max_bytes") if isinstance(entry, dict) else None - if not isinstance(entry, dict) or not _absolute(entry.get("from", "")) or \ - not Path(str(entry.get("to", ""))).is_absolute() or \ - (limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit < 0)): - problems.append(f"return_files entries are {{from: a container path, to: an absolute host path, " - f"max_bytes: a number of bytes or null}}, not {entry!r}") - if not isinstance(plan.get("path_prepend") or [], list): - problems.append("path_prepend must be a list of directories") - if not isinstance(plan.get("prompt_slash_prefix") or "", str): - problems.append("prompt_slash_prefix must be text or null") - tools = plan.get("tools_script") - if tools is not None and (not isinstance(tools, dict) or not NAME.fullmatch(str(tools.get("url_env") or ""))): - problems.append("tools_script must name its url_env variable") - grace = plan.get("kill_grace_s", DEFAULT_KILL_GRACE_S) - if isinstance(grace, bool) or not isinstance(grace, (int, float)) or grace < 0: - problems.append("kill_grace_s must be a number of seconds") - if not isinstance(plan.get("metadata") or {}, dict): - problems.append("metadata must be a mapping") - return problems - - -def load_plan(path: str | Path, cli: str | None = None) -> dict: - """The launch plan at `path` (schema version 1), checked; with `cli`, it must be that CLI's.""" - source = Path(str(path)).expanduser() - try: - plan = json.loads(source.read_text(encoding="utf-8")) - except (OSError, ValueError) as exc: - raise ValueError(f"hermit: the launch plan {source} cannot be read: {exc}") from None - if not isinstance(plan, dict): - raise ValueError(f"hermit: the launch plan {source} is not a JSON object") - if plan.get("version") != PLAN_VERSION: - raise ValueError(f"hermit: the launch plan {source} is version {plan.get('version')!r} and this harborside " - f"reads version {PLAN_VERSION}: render it with the matching Hermit") - if plan.get("cli") not in CLIS: - raise ValueError(f"hermit: the launch plan {source} names cli {plan.get('cli')!r}, not one of " - f"{', '.join(CLIS)}") - if cli is not None and plan["cli"] != cli: - raise ValueError(f"hermit: the launch plan {source} is for {plan['cli']}, and this agent runs {cli}: give the " - "agent its own CLI's plan (hermit.harbor.clis names each CLI's class)") - problems = plan_problems(plan) - if problems: - raise ValueError(f"hermit: the launch plan {source} is not usable: " + "; ".join(problems)) - return plan - - -class HermitOptions(InstalledAgentOptions): - """The kwargs every HermitXXX takes besides its Harbor class's. `extra="forbid"` is inherited: anything else is - refused at Harbor's preflight.""" - - plan: str | None = Field(default=None, description="The launch plan Hermit rendered for the trial (plan.json on " - "this machine): the deployment, the CLI's home, its command.") - image_repairs: bool = Field(default=True, description="Repair the apt sources of images whose release left the " - "mirrors (Debian bullseye) before a package is installed.") - ensure_python3: bool = Field(default=True, description="Install python3 when the deployment has hooks and the " - "image lacks it (best effort).") - - @field_validator("plan") - @classmethod - def _plan_is_a_file(cls, value: str | None) -> str | None: - if value is not None and not Path(value).expanduser().is_file(): - raise ValueError(f"the launch plan {value} is not a file") - return value - - -class HermitInstalled(BaseInstalledAgent): - """The generic executor of a launch plan (the module's docstring), mixed in first over Harbor's agent of the CLI: - `class HermitCodex(HermitInstalled, Codex)`. A subclass names its CLI and what differs in its install.""" - - CLI: ClassVar[str] = "" # the plan's `cli` - PROGRAM: ClassVar[str] = "" # the CLI's program: a mounted binary is linked as /usr/local/bin/ - BIN_OPTION: ClassVar[str | None] = None # the option naming a mounted binary (codex_bin, ...) - RUN_PATH: ClassVar[tuple[str, ...]] = () # where the class's install puts the CLI: first on PATH, before the - # plan's variables ($HOME is the agent user's own there) - MODEL_CONNECTION = None # the model is the plan's (the module's docstring) - # Hermit's plan sets web search either way, off for a task whose agent network is not public (the runner's rule, - # Harbor's own), so Harbor's switch has nothing left to do; no skills or MCP servers of Harbor's are wired (the - # adapter's are the plan's), so a task that names some is refused by Harbor, not run without them. - capabilities = AgentCapabilities(native_web_tools=True) - options_model = HermitOptions - options: HermitOptions - - def __init__(self, *args: Any, **kwargs: Any) -> None: - super().__init__(*args, **kwargs) - self._plan: dict | None = None - self._mounted = False - self._python3_ok: bool | None = None - self._apt_repaired = False - self._hooks_active = False - self._placed: dict = {"files": set(), "copies": set(), "context": None, "record": False} - self._facts: dict = {} - if self.options.plan: - self.set_plan(self.options.plan) - - # ----- the plan --------------------------------------------------------------------------------------------------- - - def set_plan(self, path: str | None) -> None: - """The plan the next `run` carries out (a stream's driver gives each step its own); None: no plan.""" - self._plan = load_plan(path, self.CLI) if path else None - - @classmethod - def resolve_model_connection_config(cls, model_name, resolve_env, **options): - """No connection of Harbor's: the plan holds the model, its endpoint and its key.""" - given = [k for k in ("base_url", "model_api_key_name", "model_api_format") if options.get(k) is not None] - if given: - raise ValueError(f"Agent '{cls.name()}' takes its model from Hermit's launch plan: give it no " - f"{', '.join(given)}") - return ResolvedModelConnection() - - @classmethod - def check_model_credentials(cls, connection, **_: Any) -> None: - """Nothing to check before the trial: the plan's secrets are checked by name when the session starts.""" - - async def _refresh_web_tool_config(self, environment: BaseEnvironment) -> None: - """Nothing written to refresh: web search is the plan's.""" - - @property - def _bin(self) -> str | None: - return getattr(self.options, self.BIN_OPTION, None) if self.BIN_OPTION else None - - # ----- install ---------------------------------------------------------------------------------------------------- - - async def install(self, environment: BaseEnvironment) -> None: - """The mounted binary when it runs here as the asked-for version and the task's agent user may execute it, - else the class's install (after the apt repair of an image whose release left the mirrors). A failure in any - of it (the mount's probe, a package, an npm or GitHub download, an image the CLI refuses) is not the agent's - doing: HermitLaunchError, which Harbor's setup passes on as it is.""" - with _setup_step(f"installing {self.CLI}"): - if self._bin and await self._use_mounted(environment): - return - if self.options.image_repairs: - await self._repair_apt_sources(environment) - await self.install_cli(environment) - - async def install_cli(self, environment: BaseEnvironment) -> None: - """The CLI's own install: Harbor's agent's, unless the class has its own.""" - await super().install(environment) - - async def _use_mounted(self, environment: BaseEnvironment) -> bool: - """The mounted binary, when it runs here and is the asked-for version (any, when none was asked for), and the - user the session runs as may execute it: linked onto PATH (as root), no install. Its version is asked as root - (Codex under a HOME it cannot write, as an agent user's may be, warns first, a line Harbor's parse_version - would take for the version); whether it may be executed is asked as the task's agent user (Harbor's default - user for the install as for the session, unless a step of a multi-step task names its own) with `test -x`, - which needs search permission on every folder on the way: a mount that user may not execute (its mode and - owner are this machine's) is not used. Otherwise the class's install, and the mount is left off PATH. What it - says to --version is read between lines of its own (`between`), not among the exec's other lines.""" - binary = PurePosixPath(str(self._bin)) - got = await environment.exec(command=f"echo {VERSION_BEGIN}; {_q(binary)} --version 2>&1; c=$?; " - f"printf '\\n{VERSION_END}\\n'; exit \"$c\"", user="root") - said = between(got.stdout, VERSION_BEGIN, VERSION_END) - version = self.parse_version(said) if got.return_code == 0 and (said or "").strip() else None - if not version or (self._version and version != self._version): - self.logger.warning("The mounted %s %s does not run here as %s (%s): installing it", self.CLI, binary, - self._version or "any version", (said or got.stdout or got.stderr or "").strip()[:200]) - return False - if (await environment.exec(command=f"test -x {_q(binary)}")).return_code != 0: - self.logger.warning("The mounted %s %s may not be executed here by the session's user: installing it", - self.CLI, binary) - return False - await self.exec_as_root(environment, command=f"mkdir -p /usr/local/bin && ln -sf {_q(binary)} " - f"/usr/local/bin/{_q(self.PROGRAM)}") - await self._after_mount(environment, binary) - self._mounted = True - return True - - async def _after_mount(self, environment: BaseEnvironment, binary: PurePosixPath) -> None: - """What a mounted binary needs besides its link (nothing, unless the class says so).""" - - async def _repair_apt_sources(self, environment: BaseEnvironment) -> None: - """Debian 11 (bullseye) images: the bullseye pools on deb.debian.org no longer serve the release's packages, - so an `apt-get install` (curl for an install, python3 for hooks) fails and the trial dies before the CLI - starts. Point main and security at archive.debian.org, accept its expired Release files, refresh the lists, - once per agent. Other images are untouched; a failure here is left to the install to report.""" - if self._apt_repaired: - return - self._apt_repaired = True - command = ("if grep -qs 'VERSION_CODENAME=bullseye' /etc/os-release && command -v apt-get >/dev/null 2>&1; " - "then sed -i " - "-e 's|http://deb.debian.org/debian-security|http://archive.debian.org/debian-security|g' " - "-e 's|http://security.debian.org/debian-security|http://archive.debian.org/debian-security|g' " - "-e 's|http://deb.debian.org/debian |http://archive.debian.org/debian |g' /etc/apt/sources.list " - "/etc/apt/sources.list.d/*.list 2>/dev/null; " - "echo 'Acquire::Check-Valid-Until \"false\";' > /etc/apt/apt.conf.d/99hermit-archive; " - "apt-get update; fi") - try: - await self.exec_as_root(environment, command=command) - except Exception as exc: # noqa: BLE001 - reported by the install if it matters - self.logger.warning("apt source repair skipped: %s", exc) - - async def _ensure_python3(self, environment: BaseEnvironment) -> bool: - """python3 on PATH, installed with the image's package manager when there is one. False, never an - exception: a container without Python runs the trial with hooks that may fail open.""" - check = "command -v python3 >/dev/null 2>&1" - if (await environment.exec(command=check)).return_code == 0: - return True - if self.options.image_repairs: - await self._repair_apt_sources(environment) - try: - await self.ensure_system_dependencies(environment, ("python3",)) - except Exception as exc: # apt failure, no network: fall back, do not fail the trial - self.logger.warning("Installing python3 failed: %s", exc) - return False - return (await environment.exec(command=check)).return_code == 0 - - # ----- the session ------------------------------------------------------------------------------------------------ - - @with_prompt_template - async def run(self, instruction: str, environment: BaseEnvironment, context: AgentContext) -> None: - plan = self._plan - if plan is None: - raise HermitLaunchError(f"hermit: {type(self).__name__} runs the launch plan Hermit renders for each trial " - "(kwargs.plan), and it was given none") - self._facts = {"exit_code": None, "killed": False, "after_session_errors": []} - self._hooks_active = False # this step's deployment's (_deploy), never the step before's - with _setup_step("finding the workdir"): - workdir = await self._workdir(environment, plan) - self._facts["workdir"] = workdir - cancelled = failed = False - try: - with _setup_step("reading the task's agent user"): - owner = await self._owner(environment) - with _setup_step("preparing the run directory"): - await self.exec_as_root(environment, command=f"mkdir -p {_q(RUN)} {_q(TOOLS_BIN)} && " - f"chown {_q(owner)} {_q(RUN)} && chmod 700 {_q(RUN)} && " - f"rm -f {_q(PIDFILE)}") - with _setup_step("deploying the adapter"): - await self._deploy(environment, plan, workdir, owner) - with _setup_step("uploading the CLI's home"): - await self._upload_home(environment, plan, workdir, owner) - with _setup_step("uploading the bundle"): - await self._upload_bundle(environment, plan) - if plan.get("tools_script"): - with _setup_step("installing the hermit-tool script"): - await self._install_tool_script(environment, plan) - await self._launch(environment, plan, workdir, prompt_of(plan, instruction), owner) - except asyncio.CancelledError: - cancelled = True # Harbor's timeout: the after-steps have a bounded time - raise - except Exception: - failed = True # the session's own error: it stands after the after-steps - raise - finally: - await self._after_session(environment, plan, workdir, cancelled=cancelled, failed=failed) - - async def _workdir(self, environment: BaseEnvironment, plan: dict) -> str: - """Where the session runs: the plan's workdir (the task's [environment].workdir), else where Harbor's execs - start (`pwd` as the agent user).""" - workdir = plan.get("workdir") - if not workdir: - said = (await self.exec_as_agent(environment, command=f'echo "{PWD_LINE}$(pwd)"')).stdout - workdir = marked(said, PWD_LINE) or "/" - return str(workdir).rstrip("/") or "/" - - async def _owner(self, environment: BaseEnvironment) -> str: - """The task's agent user as uid:gid, for what is uploaded as root and must be that user's.""" - said = (await self.exec_as_agent(environment, command=f'echo "{OWNER_LINE}$(id -u):$(id -g)"')).stdout - owner = marked(said, OWNER_LINE) or "" - if not re.fullmatch(r"\d+:\d+", owner): - raise HermitLaunchError(f"hermit: the task's agent user has no uid:gid here ({(said or '').strip()!r})") - return owner - - @staticmethod - def _fill(text: str, workdir: str) -> str: - return str(text).replace(WORKDIR_MARK, workdir) - - async def _upload_text(self, environment: BaseEnvironment, text: str, remote: str) -> None: - with tempfile.TemporaryDirectory(prefix="hermit-upload-") as tmp: - local = Path(tmp) / (PurePosixPath(remote).name or "file") - local.write_text(text, encoding="utf-8") - await environment.upload_file(local, remote) - - # ----- the deployment --------------------------------------------------------------------------------------------- - - async def _deploy(self, environment: BaseEnvironment, plan: dict, workdir: str, owner: str) -> None: - """The deployment into the workdir (the module's docstring), replacing a stream's earlier step's.""" - deploy = plan.get("deploy") or None - root = workdir.rstrip("/") - new: dict = {"files": set(), "copies": set(), "context": None, "record": False} - with tempfile.TemporaryDirectory(prefix="hermit-deploy-") as tmp: - rest = Path(tmp) / "rest" - rest.mkdir() - work = Path(deploy["work_dir"]) if deploy else None - context = None - if work is not None: - adapters = work / ADAPTER_DIR - new["copies"] = {p.name for p in adapters.iterdir() if p.is_dir() and not p.is_symlink()} \ - if adapters.is_dir() and not adapters.is_symlink() else set() - context = deploy.get("context_file") - if context and _regular(work / context): - new["context"] = context - new["record"] = _regular(work / DEPLOY_RECORD) - # The rest of the deployment (the host's files beside the copy), with the workdir filled in - new["files"] = _copy_tree(work, rest, workdir, left_out={ADAPTER_DIR, DEPLOY_RECORD, - *([context] if context else [])}) - await self._remove_stale(environment, root, new) - self._hooks_active = bool(deploy and deploy.get("hooks")) - if work is None: - self._placed = new - return - if self._hooks_active and self.options.ensure_python3 and self._python3_ok is None: - self._python3_ok = await self._ensure_python3(environment) - owned: list[str] = [] - if new["copies"]: - await self.exec_as_root(environment, command="rm -rf " + " ".join( - _q(f"{root}/{ADAPTER_DIR}/{name}") for name in sorted(new["copies"])) + - f" && mkdir -p {_q(f'{root}/{ADAPTER_DIR}')}") - await environment.upload_dir(work / ADAPTER_DIR, f"{root}/{ADAPTER_DIR}") - owned += [f"{root}/{ADAPTER_DIR}/{name}" for name in sorted(new["copies"])] - if new["files"]: - await environment.upload_dir(rest, root or "/") # a workdir of / is root "" - tops = {PurePosixPath(f).parts[0] for f in new["files"]} - owned += sorted(f"{root}/{top}" for top in tops if not (rest / top).is_symlink()) - if new["context"]: - await self._place_context(environment, deploy, work, root) - owned.append(f"{root}/{new['context']}") - if new["record"]: # always: an adapter's hook runner finds its host there - await environment.upload_file(work / DEPLOY_RECORD, f"{root}/{DEPLOY_RECORD}") - owned.append(f"{root}/{DEPLOY_RECORD}") - else: - self.logger.warning("hermit: the deployment in %s has no %s: an adapter's hooks cannot find their host", - work, DEPLOY_RECORD) - if owned: - await self.exec_as_root(environment, command=f"chown -R {_q(owner)} {shlex.join(owned)} && " - f"chmod -R u+rwX {shlex.join(owned)}") - self._placed = new - - async def _remove_stale(self, environment: BaseEnvironment, root: str, new: dict) -> None: - """What the last deployment in this container placed and this one does not: removed (a stream's step that - deploys less than the one before), and the image's own context file put back when this one has none.""" - old = self._placed - gone = [f"{root}/{f}" for f in sorted(old["files"] - new["files"])] - gone += [f"{root}/{ADAPTER_DIR}/{c}" for c in sorted(old["copies"] - new["copies"])] - if old["record"] and not new["record"]: - gone.append(f"{root}/{DEPLOY_RECORD}") - commands = [f"rm -rf {shlex.join(gone)}"] if gone else [] - if old["context"] and old["context"] != new["context"]: - original = f"{STATE}/original-{old['context']}" - target = _q(f"{root}/{old['context']}") - commands.append(f"if [ -f {_q(original)} ]; then cat {_q(original)} > {target}; else rm -f {target}; fi") - if commands: - await self.exec_as_root(environment, command=" && ".join(commands)) - - async def _place_context(self, environment: BaseEnvironment, deploy: dict, work: Path, root: str) -> None: - """The context file: Hermit's text first, then the image's own file as the container had it before the first - deployment (captured then, once per container), within the plan's context_limit (the image's text a byte - longer for each `context_escaped` in it, as the CLI escapes it). An image's own file that cannot be read (no - base64 or tr there, a read that failed) or captured whole (its copy failed part-way: a full disk) is a launch - failure (HermitLaunchError), never taken for none (it would be overwritten) and never kept in part (it would be - merged, and put back as the image's own file by a later step without a context file).""" - name = deploy["context_file"] - ours = (work / name).read_text(encoding="utf-8") - original = f"{STATE}/original-{name}" - target = f"{root}/{name}" - # Captured whole or not at all: copied beside it, then renamed into its place; a copy that failed is removed - part = _q(original + ".part") - failed = _q(f"hermit: the image's own {name} could not be captured in {original}") - capture = (f"mkdir -p {_q(STATE)} || exit 3; if [ ! -e {_q(original)} ] && [ ! -e {_q(original + '.none')} ]; " - f"then if [ -f {_q(target)} ]; then cat {_q(target)} > {part} && mv -f {part} {_q(original)}; " - f"else : > {_q(original + '.none')}; fi || {{ rm -f {part}; echo {failed} >&2; exit 3; }}; fi; " - f"v=; if [ -f {_q(original)} ]; then " - "command -v base64 >/dev/null 2>&1 && command -v tr >/dev/null 2>&1 || " - "{ echo 'hermit: no base64 or tr in the image' >&2; exit 3; }; " - f"v=$(set -o pipefail; base64 < {_q(original)} | tr -d '\\n') || exit 3; fi; " - f"printf '{CONTEXT_LINE}%s\\n' \"$v\"") - said = (await self.exec_as_root(environment, command=capture)).stdout - value = marked(said, CONTEXT_LINE) - if value is None: # never the image's own file taken for none (it would be overwritten) - raise HermitLaunchError(f"hermit: the image's own {name} could not be read: {(said or '').strip()[-300:]}") - image = base64.b64decode(value).decode("utf-8", errors="replace") - merged = ours.rstrip("\n") + "\n\n" + image if image.strip() else ours - limit = deploy.get("context_limit") - # the image's own text as the CLI counts it, a byte more for each text it escapes (Hermit's own are taken off - # the plan's limit: hermit.harbor.launch.context_limit) - escaped = deploy.get("context_escaped") - extra = image.count(escaped) if escaped and image.strip() else 0 - size = len(merged.encode("utf-8")) + extra - if limit is not None and size > int(limit): - raise HermitContextLimitError( - f"hermit: {name} for {plan_label(self._plan)} would be {size} bytes (Hermit's context " - f"{len(ours.encode('utf-8'))}, then the image's own {len(image.encode('utf-8')) + extra}), over the " - f"CLI's limit of {limit} bytes: shorten the adapter's context") - # A link there (an image whose AGENTS.md links its CLAUDE.md) is replaced, not written through - await self.exec_as_root(environment, command=f"rm -f {_q(target)}") - await self._upload_text(environment, merged, target) - - # ----- the home, the bundle, the tool script ---------------------------------------------------------------------- - - async def _upload_home(self, environment: BaseEnvironment, plan: dict, workdir: str, owner: str) -> None: - """The CLI's home as Hermit rendered it, replacing the last one: @@HERMIT_WORKDIR@@ filled in in its files - (not in its sign-in files), all of it the agent user's and private, the sign-in files mode 600.""" - home = plan.get("home") or {} - if not home: - return - target = str(home.get("container_dir") or HOME).rstrip("/") - secrets = sorted({PurePosixPath(str(s)).as_posix() for s in home.get("secret_files") or []}) - with tempfile.TemporaryDirectory(prefix="hermit-home-") as tmp: - staged = Path(tmp) / "home" - _copy_tree(Path(home["host_dir"]), staged, workdir, unfilled=set(secrets)) - present = [s for s in secrets if _regular(staged / s)] - await self.exec_as_root(environment, command=f"rm -rf {_q(target)} && " - f"mkdir -p {_q(str(PurePosixPath(target).parent))}") - await environment.upload_dir(staged, target) - command = f"chown -R {_q(owner)} {_q(target)} && chmod -R u+rwX,go-rwx {_q(target)}" - if present: - command += " && chmod 600 " + " ".join(_q(f"{target}/{s}") for s in present) - await self.exec_as_root(environment, command=command) - - async def _upload_bundle(self, environment: BaseEnvironment, plan: dict) -> None: - """Extra files the command needs (mini-swe-agent's: Hermit's modules), readable by everyone.""" - bundle = plan.get("bundle") - if not bundle: - return - target = str(bundle["container_dir"]).rstrip("/") - await self.exec_as_root(environment, command=f"rm -rf {_q(target)} && " - f"mkdir -p {_q(str(PurePosixPath(target).parent))}") - await environment.upload_dir(Path(bundle["host_dir"]), target) - await self.exec_as_root(environment, command=f"chmod -R a+rX {_q(target)}") - - async def tool_python(self, environment: BaseEnvironment) -> str: - """The interpreter of the hermit-tool script: python3 from PATH, unless the class has its own.""" - return "/usr/bin/env python3" - - async def _install_tool_script(self, environment: BaseEnvironment, plan: dict) -> None: - """The stdlib hermit-tool command (TOOL_SCRIPT), on PATH for the session.""" - script = TOOL_SCRIPT.replace("__PYTHON__", await self.tool_python(environment)) \ - .replace("__URL_ENV__", str(plan["tools_script"]["url_env"])) - target = f"{TOOLS_BIN}/hermit-tool" - await self._upload_text(environment, script, target) - await self.exec_as_root(environment, command=f"chmod 755 {_q(target)}") - - # ----- the command ------------------------------------------------------------------------------------------------ - - async def session(self, environment: BaseEnvironment, plan: dict, workdir: str) -> dict: - """What the launch script runs: {argv, env (variables over the plan's), env_refs (a variable set from another - one in the container), stream}. The plan's own, unless the class adjusts it (mini-swe-agent's Python); a - mounted binary that did not serve (the class's install ran instead) is not the program: the class's program - is, the installed CLI found on PATH.""" - argv = [self._fill(a, workdir) for a in plan["command"]] - if self._bin and not self._mounted and PurePosixPath(argv[0]) == PurePosixPath(str(self._bin)): - argv[0] = self.PROGRAM - return {"argv": argv, "env": {}, "env_refs": {}, "stream": plan["stream"]} - - def run_path(self, plan: dict) -> list[str]: - """The class's install directories, put on PATH before the plan's variables (double-quoted: $HOME is the - agent user's own there).""" - return list(self.RUN_PATH) - - def _path_prepend(self, plan: dict, workdir: str) -> list[str]: - """The plan's directories for PATH; a mount whose binary did not serve (another version: the class's - install runs instead) is left off, so that the installed CLI is the one found.""" - dirs = [self._fill(d, workdir) for d in plan.get("path_prepend") or []] - if self._bin and not self._mounted: - mount = str(PurePosixPath(*PurePosixPath(str(self._bin)).parts[:3])) # /opt/hermit- - dirs = [d for d in dirs if not (d == mount or d.startswith(mount + "/"))] - return dirs - - @staticmethod - def _word(arg: str) -> str: - """One argument as a word of the launch script: quoted, the prompt's placeholder as its variable.""" - if PROMPT_MARK not in arg: - return _q(arg) - return f'"${PROMPT_VAR}"'.join(_q(part) if part else "" for part in arg.split(PROMPT_MARK)) - - def launch_script(self, plan: dict, session: dict, workdir: str) -> str: - """The bash script that runs the session (the module's docstring).""" - env = {k: self._fill(v, workdir) for k, v in (plan.get("env") or {}).items()} - env.update(session.get("env") or {}) - from_host = plan.get("env_from_host") or {} - by_fd = plan.get("env_by_fd") or {} - argv = session["argv"] - lines = ["# written by harborside (hermit/harbor/harborside/hermit.py): one session of the launch plan", - f"cd -- {_q(workdir)} || {{ echo {_q(f'{CANNOT_ENTER} {workdir}')} >&2; exit {LAUNCH_FAILED}; }}"] - # A secret under another name first, as Harbor passed it: one export, whose words are all expanded before any - # is set, so that neither nvm nor a variable of the plan, nor another entry, replaces a source before it is - # read (Claude Code's key on Anthropic's own API is ANTHROPIC_API_KEY, which the plan then empties); a source - # no entry sets is then unset, and the plan's variables come after - renamed = {var: name for var, name in from_host.items() if var != name} - if renamed: - lines.append("export " + " ".join(f'{var}="${{{name}-}}"' for var, name in renamed.items())) - gone = sorted(set(renamed.values()) - set(from_host)) - if gone: - lines.append(f"unset {' '.join(gone)}") - if plan.get("nvm"): # with the user's own HOME: before the plan's variables - lines.append('export NVM_DIR="$HOME/.nvm"; if [ -s "$NVM_DIR/nvm.sh" ]; then . "$NVM_DIR/nvm.sh" ' - '>/dev/null 2>&1; fi') - for directory in reversed(self.run_path(plan)): - lines.append(f'export PATH="{directory}:$PATH"') - for key, value in env.items(): - lines.append(f"export {key}={_q(value)}") - for key, ref in (session.get("env_refs") or {}).items(): - lines.append(f'export {key}="${{{ref}-}}"') - prepend = self._path_prepend(plan, workdir) + ([TOOLS_BIN] if plan.get("tools_script") else []) - for directory in reversed(prepend): # after the variables: first on PATH whatever they set - lines.append(f'export PATH={_q(directory)}:"$PATH"') - lines.append(f"export {HOOK_LOGS}={_q(LOGS)}") - for i, fd_var in enumerate(by_fd.values()): - lines.append(f"export {fd_var}={FIRST_FD + i}") - prompt_file = f"{RUN}/prompt.txt" - if any(PROMPT_MARK in a for a in argv): - lines.append(f"IFS= read -r -d '' {PROMPT_VAR} < {_q(prompt_file)} || true") - stdin = _q(prompt_file) if plan.get("stdin") else "/dev/null" - words = " ".join(self._word(a) for a in argv) - stream = _q(f"{LOGS}/{session['stream']}") - lines += ["# a process group of its own: the kill after the session reaches whatever it started", - "if command -v setsid >/dev/null 2>&1; then", - f" setsid {words} > {stream} 2>&1 < {stdin} &", - "else", - " set -m", - f" {words} > {stream} 2>&1 < {stdin} &", - "fi", - "pid=$!", - f'echo "$pid" > {_q(PIDFILE)}', - 'wait "$pid"', - "code=$?", - f"rm -f {_q(PIDFILE)}", - 'exit "$code"'] - return "\n".join(lines) + "\n" - - def _outer_command(self, plan: dict) -> str: - """The exec's command: the launch script, after the env_by_fd secrets went into pipes on descriptors 3, ... - and out of the environment (`exec env -u`: the shell that had them is replaced, so neither the launch script - nor anything it starts has them in its environment; the subshell that writes one into its pipe, a fork of that - shell, has it there until its printf ends).""" - script = f"{RUN}/launch.sh" - by_fd = list((plan.get("env_by_fd") or {}).keys()) - if not by_fd: - return f"exec bash {_q(script)}" - pipes = " ".join(f'{FIRST_FD + i}< <(printf "%s\\n" "${name}")' for i, name in enumerate(by_fd)) - return (f"exec {pipes}; unset {' '.join(by_fd)}; exec env {' '.join(f'-u {n}' for n in by_fd)} " - f"bash {_q(script)}") - - async def _launch(self, environment: BaseEnvironment, plan: dict, workdir: str, prompt: str, owner: str) -> None: - with _setup_step("preparing the launch"): - session = await self.session(environment, plan, workdir) - self._facts["stream"] = session["stream"] - stream = f"{LOGS}/{session['stream']}" - await self._upload_text(environment, prompt, f"{RUN}/prompt.txt") - await self._upload_text(environment, self.launch_script(plan, session, workdir), f"{RUN}/launch.sh") - # The prompt and the script the agent user's, and the stream an earlier step of a stream left (collected - # with that step) removed: the one there after the exec is this session's, made by its launch script - await self.exec_as_root(environment, command=f"chown {_q(owner)} {_q(RUN + '/prompt.txt')} " - f"{_q(RUN + '/launch.sh')} && chmod 600 " - f"{_q(RUN + '/prompt.txt')} && " - f"chmod 700 {_q(RUN + '/launch.sh')} && rm -f {_q(stream)}") - names = sorted({*(plan.get("env_from_host") or {}).values(), *(plan.get("env_by_fd") or {}).keys()}) - command = self._outer_command(plan) - try: - if names: - result = await self._exec_by_name(environment, command, names) - else: - result = await environment.exec(command=command) - except OSError as exc: # this machine's: the compose client not spawned (EMFILE) - raise HermitLaunchError(f"hermit: {plan_label(plan)} did not start: its exec could not be run here: " - f"{type(exc).__name__}: {exc}") from exc - self._facts["exit_code"] = result.return_code - if result.return_code != 0: - said = (result.stdout or result.stderr or "").strip()[-1000:] - if result.return_code == LAUNCH_FAILED and CANNOT_ENTER in (result.stdout or "") + (result.stderr or ""): - raise HermitLaunchError(f"hermit: {plan_label(plan)} did not start: {said}") - # The launch script's redirect makes the stream before the CLI starts (an earlier step's was removed): with - # none, the CLI never started (the exec failed in Docker: runc could not start it, compose's service was not - # running), not the agent's doing. Asked from /: Docker refuses an exec in a workdir the agent removed. - try: - probe = await environment.exec(command=f"test -e {_q(stream)}", cwd="/", user="root") - except OSError as exc: # whether the CLI started cannot be told: not the agent's 0 - raise HermitLaunchError(f"hermit: {plan_label(plan)} exited with {result.return_code}, and its stream " - f"({stream}) could not be looked for here: {type(exc).__name__}: " - f"{exc}") from exc - if probe.return_code != 0: - raise HermitLaunchError(f"hermit: {plan_label(plan)} did not start: its exec exited with " - f"{result.return_code} and left no stream ({stream})" + - (f"\n{said}" if said else "")) - raise NonZeroAgentExitCodeError(f"hermit: {plan_label(plan)} exited with {result.return_code} (its stream: " - f"{stream})" + (f"\n{said}" if said else "")) - - async def _exec_by_name(self, environment: BaseEnvironment, command: str, names: list[str]): - """The command as Harbor's exec runs it (the task's workdir, its user, the merged variables), with the - variables `names` passed to `docker compose exec` by name: their values come from this process's environment - (the compose client reads them) and never reach a command line.""" - if not all(callable(getattr(environment, n, None)) for n in COMPOSE_EXEC) or \ - getattr(environment, "_is_windows_container", False): - raise HermitLaunchError(f"hermit: the session's secrets ({', '.join(names)}) go to `docker compose exec` " - "by name, which only a local Docker environment does: run the trial in Harbor's " - "docker environment") - missing = [n for n in names if not os.environ.get(n)] - if missing: - raise HermitLaunchError(f"hermit: {', '.join(missing)} {'is' if len(missing) == 1 else 'are'} not in " - "Harbor's environment, and the launch plan passes it to the session by name: set " - "it where Hermit's runner starts harbor") - argv = ["exec"] - workdir = getattr(getattr(environment, "task_env_config", None), "workdir", None) - if workdir: - argv += ["-w", str(workdir)] - for key, value in (environment._merge_env(None) or {}).items(): - if key not in names: - argv += ["-e", f"{key}={value}"] - for name in names: - argv += ["-e", name] - user = environment._resolve_user(None) - if user is not None: - argv += ["-u", str(user)] - argv += ["main", "bash", "-c", command] - return await environment._run_docker_compose_command(argv, check=False, - on_output=environment._output_callback()) - - # ----- after the session ------------------------------------------------------------------------------------------ - - async def _after_session(self, environment: BaseEnvironment, plan: dict, workdir: str, *, - cancelled: bool = False, failed: bool = False) -> None: - """The kill, the copy-back, the collected files and the returned ones, each on its own and with its execs - from / (Docker refuses an exec in a workdir the session removed): a failure is logged and recorded - (after_session_errors), never raised over the session's own outcome. Harbor's timeout landing in one of them - is recorded as its cancellation, the steps after it still run, and the cancellation goes on after the last - one, unless the session raised an error of its own (`failed`: a launch failure, a non-zero exit), which then - goes on as it is (Harbor's wait_for turns only a cancellation into the agent's timeout, so that error is not - taken for one). The return of the files is the one step run once more when the timeout landed in it (a refreshed - sign-in still comes back): its copy of a file can overlap the cancelled attempt's (a `docker compose cp` whose - wait was cancelled goes on running on this host), the same file written twice, while the copy-back's and the - collect's `rm -rf` then `cp -a` could nest a copy inside the other. Once the run is cancelled (there, or earlier - in `run`: `cancelled`), each step left (the return run once more included) has AFTER_CANCEL_S at most (a hung - container would hold every one), and a second cancellation (the job's, after Harbor's timeout) goes on at once, - the steps after it not run.""" - caught: asyncio.CancelledError | None = None - steps = [self._stop_session, self._copy_back, self._collect, self._return_files] - while steps: - step = steps.pop(0) - name = step.__name__.strip("_") - work = step(environment, plan, workdir) - try: - await (_bounded(work) if cancelled else work) - except asyncio.CancelledError as exc: - if cancelled: - raise # the second: no step waits any longer - self.logger.warning("hermit: %s after the session was cancelled", name) - self._facts.setdefault("after_session_errors", []).append(f"{name}: cancelled") - caught, cancelled = exc, True - if step == self._return_files: # the sign-in's return never finished: once more, bounded - steps.insert(0, step) - except Exception as exc: # noqa: BLE001 - the session's own error, if any, goes on - self.logger.warning("hermit: %s after the session failed: %s", name, exc) - self._facts.setdefault("after_session_errors", []).append(f"{name}: {exc}") - if caught is not None and not failed: # a session that failed: its own error goes on instead - raise caught - - async def _stop_session(self, environment: BaseEnvironment, plan: dict, workdir: str) -> None: - """A session still running (Harbor's timeout cancelled `run`, or the launch failed around it) is killed with - its process group: TERM, up to the plan's grace for it to exit, then KILL. One that ended by itself has no - pid file left, and what it started in the background goes on.""" - grace = int(plan.get("kill_grace_s", DEFAULT_KILL_GRACE_S)) - alive = '{ kill -0 "$p" 2>/dev/null && ! grep -qs "^State:[[:space:]]*Z" /proc/"$p"/status; }' - command = (f'p=$(cat {_q(PIDFILE)} 2>/dev/null); if [ -n "$p" ] && {alive}; then ' - f'kill -TERM -- -"$p" 2>/dev/null; i=0; while [ "$i" -lt {grace} ] && {alive}; do sleep 1; ' - f'i=$((i+1)); done; kill -KILL -- -"$p" 2>/dev/null; echo hermit-killed; fi; ' - f'rm -f {_q(PIDFILE)}; true') - result = await environment.exec(command=command, cwd="/", user="root", timeout_sec=grace + 30) - if "hermit-killed" in (result.stdout or ""): - self._facts["killed"] = True - - async def _copy_back(self, environment: BaseEnvironment, plan: dict, workdir: str) -> None: - """The plan's copy_back directories (relative to the workdir) to /logs/agent/hermit-copy/: what the - agent wrote that Hermit keeps (an adapter's memory), into a new directory of root's (_fresh_dir). A link there - is not followed. A copy that failed (a full disk, a read error) is removed, never kept in part (Hermit's side - would take it for the whole: a cut memory merged into the adapter's), and fails the step (its exec exits 1), - after the other copies and the chmod that keeps what was copied readable.""" - dirs = (plan.get("deploy") or {}).get("copy_back") or [] - if not dirs: - return - root = workdir.rstrip("/") - parts = [_fresh_dir(COPY_DIR), "f=0"] # f: 1 once a part failed; every part still runs - for rel in dirs: - source, target = _q(f"{root}/{rel}"), _q(f"./{PurePosixPath(rel).name}") - parts.append(f"rm -rf {target} || f=1; if [ -d {source} ] && [ ! -L {source} ]; then " - f"cp -a {source} {target} || {{ f=1; rm -rf {target}; }}; fi") - parts += ["chmod -R a+rX . || f=1", 'exit "$f"'] - await self.exec_as_root(environment, command="; ".join(parts), cwd="/") - - async def _collect(self, environment: BaseEnvironment, plan: dict, workdir: str) -> None: - """The plan's collect files (a file or a directory) to /logs/agent/hermit-collect/, a new directory of - root's (_fresh_dir); never the home's sign-in files, and never through a link. A copy that failed (a full disk, - a read error) is removed, never kept in part (Hermit's side would take it for the whole: a cut last message for - the reply, where without the file it reads the stream's), and fails the step (its exec exits 1), after the - other copies and the chmod that keeps what was copied readable.""" - entries = plan.get("collect") or [] - if not entries: - return - home = plan.get("home") or {} - target_home = str(home.get("container_dir") or HOME).rstrip("/") - secret = {f"{target_home}/{PurePosixPath(str(s)).as_posix()}" for s in home.get("secret_files") or []} - parts = [_fresh_dir(COLLECT_DIR), "f=0"] # f: 1 once a part failed; every part still runs - for entry in entries: - source = self._fill(entry["from"], workdir) - if str(PurePosixPath(source)) in secret: - self.logger.warning("hermit: %s is a sign-in file of the home: it is not collected into %s", source, - LOGS) - continue - target = f"./{PurePosixPath(entry['to']).as_posix()}" - s, t = _q(source), _q(target) - failed = f"{{ f=1; rm -rf {t}; }}" # a copy that failed is removed, never kept in part - parts.append(f"mkdir -p {_q(str(PurePosixPath(target).parent))} || f=1; rm -rf {t} || f=1; " - f"if [ -L {s} ]; then :; elif [ -d {s} ]; then cp -a {s} {t} || {failed}; " - f"elif [ -f {s} ]; then cp {s} {t} || {failed}; fi") - parts += ["chmod -R a+rX . || f=1", 'exit "$f"'] - await self.exec_as_root(environment, command="; ".join(parts), cwd="/") - - async def _return_files(self, environment: BaseEnvironment, plan: dict, workdir: str) -> None: - """The plan's return_files from the container to their host paths (Hermit's side takes them from there: a - refreshed ChatGPT sign-in), never into the logs. One the session did not leave (or left as a link) is skipped; - one larger than the entry's max_bytes when checked is not downloaded and raises (the session wrote something - else there: a sign-in is a few KB), though a process still writing it can grow it between the check and the - download (the take-back then refuses it by its size); one whose check could not tell (its exec failed: - compose's service stopped; no wc in the image) raises, never taken for none. Each one returned is then removed - from the container (a sign-in: the verifier's commands, or a process the session left, would read it), as far - as the container lets it: its removal failing changes nothing else.""" - for entry in plan.get("return_files") or []: - source = self._fill(entry["from"], workdir) - s = _q(source) - check = await environment.exec(command=f'test -f {s} && test ! -L {s} && {{ n=$(wc -c < {s}) && ' - f'echo "{RETURN_LINE}$n" || exit 3; }} || echo "{RETURN_LINE}no"', - cwd="/", user="root") - there = marked(check.stdout, RETURN_LINE) if check.return_code == 0 else None - if there == "no": - continue - if there is None or not there.isdecimal(): - raise RuntimeError(f"whether the session left {source} could not be told (exit {check.return_code}): " - f"{(check.stdout or check.stderr or '').strip()[-300:]}") - if entry.get("max_bytes") is not None and int(there) > entry["max_bytes"]: - raise RuntimeError(f"{source} is {there} bytes, over the {entry['max_bytes']} it may hold: not " - "returned") - target = Path(entry["to"]) - target.parent.mkdir(parents=True, exist_ok=True) - await environment.download_file(source, target) - try: - await environment.exec(command=f"rm -f {s}", cwd="/", user="root") - except Exception: # noqa: BLE001 - its copy is home already - pass - - # ----- what Harbor records ---------------------------------------------------------------------------------------- - - def populate_context_post_run(self, context: AgentContext) -> None: - """Metadata only: Hermit reads the session from its stream, not Harbor.""" - plan = self._plan or {} - deploy = plan.get("deploy") or {} - facts = {"hermit_cli": self.CLI, "adapter_deployed": bool(deploy), "adapter_id": deploy.get("adapter_id"), - "hooks_active": self._hooks_active, "python3": self._python3_ok, "mounted": self._mounted, - "exit_code": self._facts.get("exit_code"), "killed": bool(self._facts.get("killed")), - "stream": self._facts.get("stream") or plan.get("stream")} - if self._facts.get("after_session_errors"): - facts["after_session_errors"] = list(self._facts["after_session_errors"]) - context.metadata = {**(context.metadata or {}), **(plan.get("metadata") or {}), **facts} - - -def prompt_of(plan: dict, instruction: str) -> str: - """The session's prompt: Harbor's instruction, then the plan's prompt_suffix; the plan's prompt_slash_prefix before - one that starts with `/` (which Claude Code would otherwise run as one of its commands).""" - prompt = instruction + str(plan.get("prompt_suffix") or "") - prefix = plan.get("prompt_slash_prefix") - return prefix + prompt if prefix and prompt.startswith("/") else prompt - - -def plan_label(plan: dict | None) -> str: - """The CLI's name in messages.""" - plan = plan or {} - return str(plan.get("label") or plan.get("cli") or "the CLI") diff --git a/hermit/harbor/harborside/mini_swe_agent.py b/hermit/harbor/harborside/mini_swe_agent.py deleted file mode 100644 index 1c4a263a..00000000 --- a/hermit/harbor/harborside/mini_swe_agent.py +++ /dev/null @@ -1,105 +0,0 @@ -"""HermitMiniSweAgent: mini-swe-agent in a Harbor trial, its session launched from Hermit's plan (harborside.hermit). - -Imported by Harbor as `harborside.mini_swe_agent:HermitMiniSweAgent`. Harbor's mini-swe-agent brings the install: uv, -Python 3.12 and `uv tool install mini-swe-agent== --with litellm ...` as the agent user (Hermit's runner gives -its tested build when the config names none), after the apt repair of an image whose release left the mirrors. - -Hermit's mini-swe-agent session is no command of mini-swe-agent's own but Hermit's runner of one task -(hermit.agent.base.mini_swe_agent_run), whose stdout is Hermit's event stream. The plan therefore brings a bundle of -Hermit's modules (`bundle`, uploaded under its container_dir) and a command whose interpreter is @@HERMIT_PYTHON@@ -(Hermit's `bin` setting; a host interpreter's path there is taken for it too): this class finds mini-swe-agent's own -Python, the uv tool's (`uv tool dir`, else the shebang of the tool's `mini-swe-agent`), puts the bundle first on -PYTHONPATH and checks once that the runner imports there. Then the session runs Hermit's runner as every other CLI's -plan runs (the prompt on stdin, the stream agent-mini-swe-agent.jsonl, the tools through HERMIT_TOOLS_URL). - -If the runner cannot run there (the plan has no bundle, mini-swe-agent's Python is not found, or the runner does not -import in it), the trial raises HermitLaunchError saying why: another agent (mini-swe-agent's own command) in its -place would be a result of something else. -""" -from __future__ import annotations - -from pathlib import PurePosixPath -import re -import shlex - -from harbor.agents.installed.mini_swe_agent import MiniSweAgent, MiniSweAgentOptions -from harbor.environments.base import BaseEnvironment - -from harborside.hermit import HermitInstalled, HermitLaunchError, HermitOptions, marked - -PYTHON_MARK = "@@HERMIT_PYTHON@@" -RUNNER = "hermit.agent.base.mini_swe_agent_run" -INTERPRETER = re.compile(r"python[0-9.]*") -PYTHON_LINE = "hermit-python:" # FIND_PYTHON's answer, on a marked line (harborside.hermit.marked) -UV_ENV = ('if [ -f "$HOME/.local/bin/env" ]; then . "$HOME/.local/bin/env"; fi; ' - 'export PATH="$HOME/.local/bin:$PATH"; ') -FIND_PYTHON = (UV_ENV + 'd=$(uv tool dir 2>/dev/null || true); ' - 'if [ -n "$d" ] && [ -x "$d/mini-swe-agent/bin/python" ]; then ' - f'echo "{PYTHON_LINE}$d/mini-swe-agent/bin/python"; ' - 'exit 0; fi; b=$(command -v mini-swe-agent 2>/dev/null || true); ' - 'if [ -n "$b" ]; then l=$(head -n 1 "$b"); case "$l" in "#!/"*) set -- ${l#??}; ' - f'if [ -x "$1" ]; then echo "{PYTHON_LINE}$1"; exit 0; fi;; esac; fi; exit 1') - - -class HermitMiniSweAgentOptions(HermitOptions, MiniSweAgentOptions): - """Harbor's mini-swe-agent kwargs (accepted, unused: the plan holds the session) and Hermit's. `extra="forbid"`.""" - - -class HermitMiniSweAgent(HermitInstalled, MiniSweAgent): - CLI = "mini-swe-agent" - PROGRAM = "mini-swe-agent" - options_model = HermitMiniSweAgentOptions - options: HermitMiniSweAgentOptions - - def __init__(self, *args, **kwargs) -> None: - super().__init__(*args, **kwargs) - self._python: str | None = None - - async def _find_python(self, environment: BaseEnvironment) -> str | None: - """mini-swe-agent's own Python (the uv tool's), found once per container.""" - if self._python is None: - found = await environment.exec(command=FIND_PYTHON) - path = marked(found.stdout, PYTHON_LINE) if found.return_code == 0 else None - self._python = path if path and path.startswith("/") else None - return self._python - - async def tool_python(self, environment: BaseEnvironment) -> str: - return await self._find_python(environment) or "/usr/bin/env python3" - - def _pythonpath(self, plan: dict, workdir: str) -> str: - """The bundle first on PYTHONPATH, then the rest of the plan's own PYTHONPATH (which may name it too).""" - bundle = str(plan["bundle"]["container_dir"]).rstrip("/") - own = self._fill((plan.get("env") or {}).get("PYTHONPATH") or "", workdir).split(":") - return ":".join([bundle, *(p for p in own if p and p.rstrip("/") != bundle)]) - - async def _bundle_problem(self, environment: BaseEnvironment, plan: dict, workdir: str) -> str | None: - """Why Hermit's runner cannot run here, or None: no bundle, no mini-swe-agent Python, or no import.""" - if not plan.get("bundle"): - return "the plan has no bundle of Hermit's modules" - python = await self._find_python(environment) - if python is None: - return "mini-swe-agent's Python was not found (uv tool dir, the tool's shebang)" - probe = await environment.exec(command=f"cd {_q(workdir)} && {_q(python)} -P -c {_q('import ' + RUNNER)} 2>&1", - env={"PYTHONPATH": self._pythonpath(plan, workdir)}) - if probe.return_code != 0: - return f"{RUNNER} does not import in {python}: {(probe.stdout or probe.stderr or '').strip()[-400:]}" - return None - - async def session(self, environment: BaseEnvironment, plan: dict, workdir: str) -> dict: - own = await super().session(environment, plan, workdir) - problem = await self._bundle_problem(environment, plan, workdir) - if problem is not None: - raise HermitLaunchError(f"hermit: Hermit's mini-swe-agent runner cannot run in this container: {problem}") - python = self._python - argv = [python if a == PYTHON_MARK or (i == 0 and _interpreter(a)) else a.replace(PYTHON_MARK, python) - for i, a in enumerate(own["argv"])] - return {**own, "argv": argv, "env": {"PYTHONPATH": self._pythonpath(plan, workdir)}} - - -def _q(value: str) -> str: - return shlex.quote(str(value)) - - -def _interpreter(arg: str) -> bool: - """A Python interpreter's path or name (a host's sys.executable a plan left in the command).""" - return bool(INTERPRETER.fullmatch(PurePosixPath(arg).name)) diff --git a/hermit/harbor/harborside/opencode.py b/hermit/harbor/harborside/opencode.py deleted file mode 100644 index 3791623e..00000000 --- a/hermit/harbor/harborside/opencode.py +++ /dev/null @@ -1,84 +0,0 @@ -"""HermitOpenCode: OpenCode 2.2 in a Harbor trial, its session launched from Hermit's plan (harborside.hermit). - -Imported by Harbor as `harborside.opencode:HermitOpenCode`. Harbor's OpenCode installs the npm package `opencode-ai`, -which is OpenCode's 1.x line; Hermit's configs (its home's opencode.json, the inline config, `run --standalone`) are -OpenCode 2.2's. So the install here is the GitHub release Hermit documents (docs/get-started/installation.md): the -`anomalyco/opencode` release v's `opencode-linux-x64.tar.gz`, or `-musl.tar.gz` on a musl image (the test -Harbor's own installs use), downloaded with curl, else wget, else python3's urllib (curl installed first when the -image has none of them), unpacked into /installed-agent/opencode/bin and linked as /usr/local/bin/opencode; or an -OpenCode binary mounted at /opt/hermit-opencode (`opencode_bin`) that answers `--version` as `version` and that the -task's agent user may execute. The version is the runner's (Hermit's tested build when the config names none); without -one, the latest release. The release is not checked against a digest here: pinning a CLI's build is not this class's -business. - -The session is the plan's, as harborside.hermit runs every CLI's: Hermit's own `opencode run --standalone --format -json -m hermit/ ...` with the prompt on stdin, its home's HOME and XDG folders under /installed-agent/hermit-home -(its session database is collected for Hermit's usage reader), the deployment's opencode.json and .opencode/skills in -the workdir, and the stream agent-opencode.jsonl. Harbor's own run (its opencode.txt, its config) is not used. -""" -from __future__ import annotations - -from harbor.agents.installed.opencode import OpenCode, OpenCodeOptions -from harbor.environments.base import BaseEnvironment -from pydantic import Field - -from harborside.hermit import MUSL, VERSION, HermitInstalled, HermitOptions - -REPOSITORY = "anomalyco/opencode" -INSTALL_DIR = "/installed-agent/opencode" -DOWNLOADERS = "command -v curl >/dev/null 2>&1 || command -v wget >/dev/null 2>&1 || command -v python3 >/dev/null 2>&1" - - -class HermitOpenCodeOptions(HermitOptions, OpenCodeOptions): - """Harbor's OpenCode kwargs (accepted, unused: the plan holds the session) and Hermit's. `extra="forbid"`.""" - - opencode_bin: str | None = Field(default=None, description="An OpenCode binary mounted in the container (at " - "/opt/hermit-opencode), used when it runs there as " - "`version`, instead of the per-trial download of the " - "GitHub release.") - - -class HermitOpenCode(HermitInstalled, OpenCode): - CLI = "opencode" - PROGRAM = "opencode" - BIN_OPTION = "opencode_bin" - RUN_PATH = (f"{INSTALL_DIR}/bin",) - options_model = HermitOpenCodeOptions - options: HermitOpenCodeOptions - - def parse_version(self, stdout: str) -> str: - found = VERSION.search(stdout or "") - return found.group(1) if found else (stdout or "").strip() - - def release_url(self) -> str: - """The release's download folder: v's, or the latest's when no version is given.""" - tail = f"download/v{self._version}" if self._version else "latest/download" - return f"https://github.com/{REPOSITORY}/releases/{tail}" - - def install_command(self) -> str: - """Download, unpack and link the release's binary (as root: /installed-agent and /usr/local/bin are root's).""" - d = INSTALL_DIR - return "\n".join([ - "set -eu", - f"d={d}", - 'arch=x64; case "$(uname -m)" in aarch64|arm64) arch=arm64;; esac', - f'if {MUSL}; then asset="opencode-linux-$arch-musl.tar.gz"; else asset="opencode-linux-$arch.tar.gz"; fi', - f'url="{self.release_url()}/$asset"', - 'rm -rf "$d" && mkdir -p "$d/bin" "$d/unpack"', - 'if command -v curl >/dev/null 2>&1; then curl -fsSL --retry 3 -o "$d/$asset" "$url"', - 'elif command -v wget >/dev/null 2>&1; then wget -q -O "$d/$asset" "$url"', - "elif command -v python3 >/dev/null 2>&1; then python3 -c 'import sys, urllib.request; " - "urllib.request.urlretrieve(sys.argv[1], sys.argv[2])' \"$url\" \"$d/$asset\"", - "else echo 'hermit: no curl, wget or python3 here to download OpenCode' >&2; exit 1; fi", - 'tar -xzf "$d/$asset" -C "$d/unpack"', - 'b=$(find "$d/unpack" -type f -name opencode | head -n 1)', - '[ -n "$b" ] || { echo "hermit: $asset holds no opencode binary" >&2; exit 1; }', - 'mv "$b" "$d/bin/opencode" && chmod 755 "$d/bin/opencode" && rm -rf "$d/unpack" "$d/$asset"', - 'mkdir -p /usr/local/bin && ln -sf "$d/bin/opencode" /usr/local/bin/opencode', - '"$d/bin/opencode" --version']) - - async def install_cli(self, environment: BaseEnvironment) -> None: - """The GitHub release (the module's docstring); curl and its CA roots first on an image with no downloader.""" - if (await environment.exec(command=DOWNLOADERS, user="root")).return_code != 0: - await self.ensure_system_dependencies(environment, ("curl", "ca_certificates")) - await self.exec_as_root(environment, command=self.install_command()) diff --git a/hermit/harbor/harborside/pi.py b/hermit/harbor/harborside/pi.py deleted file mode 100644 index 95e5cf3f..00000000 --- a/hermit/harbor/harborside/pi.py +++ /dev/null @@ -1,37 +0,0 @@ -"""HermitPi: Pi in a Harbor trial, its session launched from Hermit's plan (harborside.hermit). - -Imported by Harbor as `harborside.pi:HermitPi`. Harbor's Pi brings the install: nvm and Node 22, then `npm install -g ---ignore-scripts @earendil-works/pi-coding-agent` at `version` (Hermit's runner gives its tested build when the -config names none; Harbor's own default would be the latest), after the apt repair of an image whose release left the -mirrors; or a Pi binary mounted at /opt/hermit-pi (`pi_bin`) that answers `--version` as `version` and that the task's -agent user may execute. - -The session is the plan's, as harborside.hermit runs every CLI's: Hermit's own `pi --mode json --provider hermit ...` -(models.json, mcp.json, settings.json and the command-limit extension in its home, PI_CODING_AGENT_DIR and HOME both -/installed-agent/hermit-home, offline), the deployment's .pi/ in the workdir trusted for the session (--approve), and -the stream agent-pi.jsonl. The plan loads nvm (`nvm: true`) with the agent user's own HOME, before Pi's HOME replaces -it, so the install's Node and `pi` are found. Pi exits 0 however a session ended: Hermit's side reads its verdict from -the stream. Harbor's own run (its --session-dir under /logs/agent, its pi.txt, its models.json) is not used. -""" -from __future__ import annotations - -from harbor.agents.installed.pi import Pi, PiOptions -from pydantic import Field - -from harborside.hermit import HermitInstalled, HermitOptions - - -class HermitPiOptions(HermitOptions, PiOptions): - """Harbor's Pi kwargs (accepted, unused: the plan holds the session) and Hermit's. `extra="forbid"`.""" - - pi_bin: str | None = Field(default=None, description="A Pi binary mounted in the container (at /opt/hermit-pi), " - "used when it runs there as `version`, instead of Harbor's " - "per-trial install (nvm, Node 22, npm).") - - -class HermitPi(HermitInstalled, Pi): - CLI = "pi" - PROGRAM = "pi" - BIN_OPTION = "pi_bin" - options_model = HermitPiOptions - options: HermitPiOptions diff --git a/hermit/harbor/harborside/scripted.py b/hermit/harbor/harborside/scripted.py deleted file mode 100644 index 0b76af36..00000000 --- a/hermit/harbor/harborside/scripted.py +++ /dev/null @@ -1,76 +0,0 @@ -"""A shell script as Harbor's agent: what a task set's packaging is checked with, without a model. - - agents: - - import_path: harborside.scripted:ScriptedAgent - kwargs: {script: "echo 42 > /app/answer.txt"} - -The script runs in the task's environment as its default user, with the instruction in the file -`$HERMIT_INSTRUCTION_FILE` (`/logs/agent/instruction.md`); its output is kept in the agent's logs as `script.txt`. -A non-zero exit is the agent's failure (NonZeroAgentExitCodeError is not raised: the verifier still runs, as it -would after a real agent gave up). Tokens are zero. A script still running after `timeout_sec` is not waited for: -Harbor's docker environment ends its `docker compose exec` client and raises a RuntimeError ("Command timed out -after seconds") out of `run`, before `script.txt` is written: an agent error. A single-step trial ends -there, without its verifier (Hermit's side scores it 0); in a step of a multi-step task the error is the step's, and -its verifier still runs (Hermit's side scores the step's reward). (The task's own agent timeout, when it comes first, -is Harbor's AgentTimeoutError instead, after which the verifier runs.) -""" -from __future__ import annotations - -from pathlib import Path -import shlex -import tempfile -from typing import Any - -from harbor.agents.base import BaseAgent -from harbor.agents.capabilities import AgentCapabilities -from harbor.agents.options import AgentOptions -from harbor.environments.base import BaseEnvironment -from harbor.models.agent.context import AgentContext -from pydantic import Field - - -class ScriptedOptions(AgentOptions): - script: str = Field(default="true", description="The shell script to run in the environment.") - shell: str = Field(default="bash", description="The shell that runs it.") - timeout_sec: int | None = Field(default=600, description="The script's own time limit.") - - -class ScriptedAgent(BaseAgent): - capabilities = AgentCapabilities() - options_model = ScriptedOptions - options: ScriptedOptions - - def __init__(self, *args: Any, **kwargs: Any) -> None: - super().__init__(*args, **kwargs) - self.exit_code: int | None = None - - @staticmethod - def name() -> str: - return "scripted" - - def version(self) -> str: - return "1" - - async def setup(self, environment: BaseEnvironment) -> None: - pass - - async def run(self, instruction: str, environment: BaseEnvironment, context: AgentContext) -> None: - logs = self.environment_logs_dir - remote = (logs / "instruction.md").as_posix() - await environment.exec(command=f"mkdir -p {shlex.quote(logs.as_posix())}", user="root") - with tempfile.TemporaryDirectory(prefix="hermit-scripted-") as tmp: - local = Path(tmp) / "instruction.md" - local.write_text(instruction) - await environment.upload_file(local, remote) - await environment.exec(command=f"chmod -R a+rwX {shlex.quote(logs.as_posix())}", user="root") - result = await environment.exec(command=f"{self.options.shell} -c {shlex.quote(self.options.script)}", - env={"HERMIT_INSTRUCTION_FILE": remote}, - timeout_sec=self.options.timeout_sec) - self.exit_code = result.return_code - self.logs_dir.mkdir(parents=True, exist_ok=True) - (self.logs_dir / "script.txt").write_text(f"$ {self.options.script}\n[exit {result.return_code}]\n" - f"--- stdout ---\n{result.stdout or ''}\n--- stderr ---\n" - f"{result.stderr or ''}\n") - context.n_input_tokens = 0 - context.n_output_tokens = 0 - context.metadata = {**(context.metadata or {}), "exit_code": result.return_code} diff --git a/hermit/harbor/judge.py b/hermit/harbor/judge.py deleted file mode 100644 index 33eaf58a..00000000 --- a/hermit/harbor/judge.py +++ /dev/null @@ -1,100 +0,0 @@ -"""The judge of a benchmark run through Harbor: the verifier's reward, as the container wrote it. - - judge: - path: hermit.harbor.judge # name: score (the default) - -The score is the trial's reward (`/logs/verifier/reward.txt` or `reward.json`'s `reward_key`, carried in the runner's -output) whenever the verifier gave one, whatever the trial's status: a trial that ran out of the agent's time or -whose agent exited with an error still has its work scored, as Harbor's own job mean scores it. A trial without a -reward scores 0. A trial that is not the agent's result (infra: the host, the provider, a session that never started; -the runner marks it `unscored` after its own runs again, agent.params.infra_retries) gets no score: `score` raises -Unscored, so that the run records a failed run (hermit.trainer.rollout: status judge_error, its usage and log kept), -which `hermit eval` counts among its errors and as 0 in its mean, as EvoSafe's judge does with a run it cannot score; -`python -m hermit.harbor.report ` gives the pass@1 over the trials that are the agent's result beside it. -A trial "passed" when its reward reaches the benchmark's pass reward (`agent.params.pass_reward`, -1.0 unless the benchmark sets another; null for a benchmark whose reward is a scale with no pass, where none is said -to pass); `passed` says so (None without a pass reward). The feedback names the -status (and, for a context over the CLI's limit, by how much: the session never started) and, for a trial that did -not pass, the tail of the verifier's output (as curated: `verifier_redact` applied), which is what a learner can act -on. The verifier's other reward keys are kept in `rewards`, for every status. -""" -from __future__ import annotations - -from pathlib import Path - -TAIL = 1500 # characters of the verifier's stdout in the feedback -PASS_REWARD = 1.0 - - -class Unscored(RuntimeError): - """A trial that is not the agent's result (infra): no score is the agent's.""" - - -def pass_reward(output) -> float | None: - """The run's pass reward, as the runner gave it (`extra["judge"]["pass_reward"]`), else PASS_REWARD.""" - extra = getattr(output, "extra", None) - judge = extra.get("judge") if isinstance(extra, dict) else None - if isinstance(judge, dict) and "pass_reward" in judge: - return None if judge["pass_reward"] is None else float(judge["pass_reward"]) - return PASS_REWARD - - -def unscored(output) -> str | None: - """Why a rollout is not the agent's result (the runner's `unscored` cause, or an infra status), or None.""" - out = output.output if isinstance(output.output, dict) else {} - extra = getattr(output, "extra", None) - why = extra.get("unscored") if isinstance(extra, dict) else None - if why or (out.get("status") or "infra") == "infra": - return str(why or out.get("failure_cause") or "infra") - return None - - -def score(task: dict, output) -> dict: - why = unscored(output) - if why is not None: - extra = getattr(output, "extra", None) - said = str((extra.get("exception_message") if isinstance(extra, dict) else None) or "")[:600] - raise Unscored(f"the trial is not the agent's result ({why}{': ' + said if said else ''}): it has no score") - out = output.output if isinstance(output.output, dict) else {} - status = out.get("status") or "infra" - raw = out.get("reward") - reward = float(raw) if raw is not None else None - bar = pass_reward(output) - passed = reward is not None and bar is not None and reward >= bar - cause = out.get("failure_cause") or status - if status == "ok": - feedback = "passed" if passed else f"reward {reward:g}" if reward is not None else "no reward" - else: - detail = out.get("exception_type") or "" - feedback = f"{status} ({cause}{': ' + detail if detail else ''})" - if cause == "solution_timed_out": - feedback += ("; the verifier ran out of its time on the agent's solution (too slow to finish its tests, " - "or a process it left that hung them)") - elif cause in ("context_over_limit", "adapter_failed"): # no session ran: the message says why - extra = getattr(output, "extra", None) or {} - feedback += "; " + str(extra.get("exception_message") or "").removeprefix("hermit: ")[:600] - elif status == "timeout": - feedback += "; the agent ran out of its time budget" - if reward is not None: - feedback += "; its work passed the verifier" if passed else f"; the verifier scored its work {reward:g}" - if not passed: - feedback += _verifier_tail(output.log) - record = {"score": reward if reward is not None else 0.0, "feedback": feedback, "status": status, - "passed": passed if bar is not None else None, "rewards": out.get("rewards")} - if status != "ok": - record.update(failure_cause=cause, verifier_reward_raw=raw) - return record - - -def _verifier_tail(log) -> str: - """The end of the verifier's stdout from the curated log directory, when there is one.""" - if not log: - return "" - path = Path(log) / "verifier" / "test-stdout.txt" - try: - text = path.read_text(encoding="utf-8", errors="replace").strip() - except OSError: - return "" - if not text: - return "" - return "\nverifier output (tail):\n" + (text[-TAIL:] if len(text) > TAIL else text) diff --git a/hermit/harbor/launch.py b/hermit/harbor/launch.py deleted file mode 100644 index 53be4124..00000000 --- a/hermit/harbor/launch.py +++ /dev/null @@ -1,478 +0,0 @@ -"""A Harbor trial's launch plan (`/launch/plan.json`, schema 1): the acting CLI's home, command line, -environment and stream, rendered on Hermit's side by the CLI Base Agent Hermit's own sessions use -(hermit.agent.base.cli: `prepare_home`, `home_env`, `session_env`, `command`), and carried out in the task's -container by harborside/hermit.py, which imports nothing of Hermit. - - /launch/ - plan.json what harborside uploads and runs; every host path in it absolute - home/ the CLI's home as the Base Agent renders it, uploaded to /installed-agent/hermit-home - work/ the empty workspace a trial without an adapter is rendered against - bundle/ mini-swe-agent: the Hermit modules its runner imports (hermit.agent.base.mini_swe_agent_run) - return/ what the container hands back to this machine (a ChatGPT sign-in Codex refreshed, at most - the entry's max_bytes: hermit.harbor.signin.take_back reads it) - -The Base Agent renders against host paths (the home above, the staged deployment or work/ as the workspace); the plan -names container paths: the home is CONTAINER_HOME, the workspace `@@HERMIT_WORKDIR@@` (the task's working directory, -which harborside finds), in the command, the environment and the home's text files. A path of the job's directory -(where the home and the workspace are rendered) that would remain in them is refused (`check`), and so is a secret's -value anywhere but a secret file: a key reaches the container by its variable's name (`env_from_host`: the value is in -Harbor's process environment, never in an argument, a file or the plan), Claude Code's subscription token on a -descriptor (`env_by_fd`), the ChatGPT sign-in as a secret file of the home that comes back after the trial -(`return_files`; a stream's step copies it in at its turn: `sign_in`). The prompt goes on stdin for every CLI: Harbor's instruction, then `prompt_suffix`, the memory note -(hermit.agent.acting.workspace's MEMORY_NOTE) when the deployed adapter keeps notes, which are copied back after the -session (`copy_back`); `prompt_slash_prefix` goes before a prompt that starts with `/`, as the Base Agent's -prompt_text puts it (Claude Code's line break: it would run the prompt as a command). After the trial the runner reads -`agent/` with the same Base Agent (`outcome`, `parse`, `adjust_usage`), from the files `collect` brings out -(on a ChatGPT login, Codex's session logs too: their reports of the plan's usage). What the Base Agent's `run` would -refuse before a session (mini-swe-agent's settings, a login of another CLI) is refused when the plan is rendered -(`check_settings`), as the plan is carried out without `run`. - -The answer-only agent (`cli: answer`) has no home and no command: its plan's `answer` block names the model endpoint -the runner serves on this machine, the adapter's context as the system text, and where the answer goes. -""" -from __future__ import annotations - -import json -import os -from pathlib import Path -import re -import shutil - -from hermit.harbor.clis import CONTAINER_HOME, entry - -PLAN_VERSION = 1 -PLAN_FILE = "plan.json" -LAUNCH_DIR = "launch" -HOME_DIR = "home" # under LAUNCH_DIR: the CLI's home as rendered -RETURN_DIR = "return" # ... and what the container hands back (return_files) -WORKDIR = "@@HERMIT_WORKDIR@@" -PROMPT = "@@HERMIT_PROMPT@@" -BUNDLE_DIR = "/installed-agent/hermit-bundle" # mini-swe-agent's Hermit modules in the container -KILL_GRACE_S = 10 -LAST_MESSAGE = "last_message.txt" -# What mini-swe-agent's runner imports of Hermit (its own modules, mini-swe-agent's and the standard library besides); -# tests/core/test_harbor_launch.py checks that nothing else is imported -MINI_BUNDLE = ("hermit/__init__.py", "hermit/agent/__init__.py", "hermit/agent/base/__init__.py", - "hermit/agent/base/mini_swe_agent.py", "hermit/agent/base/mini_swe_agent.yaml", - "hermit/agent/base/mini_swe_agent_run.py", "hermit/agent/base/mini_swe_agent_env.py", - "hermit/agent/model/__init__.py", "hermit/agent/model/backends.py", "hermit/agent/model/backends.yaml") -# Files of the home outcome() and adjust_usage() read after the session, brought out of the container: the last -# message the CLI writes itself (Codex's -o, mini-swe-agent's runner's --last), OpenCode's session database with its -# write-ahead log (OpenCode keeps it in WAL mode: a session cut off leaves its last rows in -wal; a file the session did -# not leave is not collected) -COLLECT = {"codex": [(LAST_MESSAGE, LAST_MESSAGE)], "mini-swe-agent": [(LAST_MESSAGE, LAST_MESSAGE)], - "opencode": [("data/opencode/opencode.db", "opencode.db"), - ("data/opencode/opencode.db-wal", "opencode.db-wal"), - ("data/opencode/opencode.db-shm", "opencode.db-shm")]} -# ... and on a login, what says how much of the plan the session used: Codex's session logs (their rate-limit reports, -# which hermit.agent.base.login.usage_of reads as it does in a Hermit session's own home) -COLLECT_LOGIN = {"chatgpt": [("sessions", "sessions")]} -SECRET_NAME = re.compile(r"(KEY|TOKEN|SECRET|PASSWORD|CREDENTIAL)", re.I) -# a variable named as a secret: one of the words of its name (between underscores) is a secret's (OPENAI_API_KEY, -# GH_TOKEN, HF_TOKEN_READ, SECRET_ID, DB_PASSWD), never a count (MAX_OUTPUT_TOKENS) or a word that holds one -# (GIT_AUTHOR_NAME, TOKENIZERS_PARALLELISM) -SECRET_VAR = re.compile(r"(?:^|_)(?:API_?KEY|KEY|TOKEN|SECRET|PASSWORD|PASSWD|CREDENTIALS?|AUTH)(?:_|$)", re.I) -# a name Harbor treats as a secret's (harbor.utils.env's rule: OPENAI_API_KEY, GH_TOKEN, NPM_AUTH, GIT_AUTHOR_NAME too) -HARBOR_SECRET = re.compile(r"KEY|SECRET|TOKEN(?!S)|PASSWORD|CREDENTIAL|AUTH", re.I) -VAR_REF = re.compile(r"\$(?:\{(\w+)\}|(\w+))") -TEXT_LIMIT = 4 << 20 # a home file larger than this is left as it is (no path in it) - - -# ---- the model and the Base Agent's settings ---------------------------------------------------------------------- -def binding_of(params: dict): - """The model binding of the runner's params, or None for a plain model name: the Binding spec the config's - loading kept (`binding`, or `model` as a loaded spec), else a `/` reference resolved here against - the backends registry (a runner built without the config's loading).""" - from hermit.agent.model.backends import Binding, as_binding, registry - spec = params.get("binding") - if isinstance(spec, dict) and spec.get("base_url") and spec.get("model"): - return _with_role(Binding.from_spec(spec)) - model = params.get("model") - if isinstance(model, dict) and model.get("base_url") and model.get("model"): - return _with_role(Binding.from_spec(model)) - if isinstance(model, (str, dict)) and model: - return as_binding(model, registry()) - return None - - -def _with_role(binding): - """`binding` with its backend's developer_role, which a spec the config's loading kept drops (Binding.spec: - FRAMEWORK_SUPPORT.md, candidate 14) and Pi's and dsh's provider settings read (`compat`): the backends registry's - (the built-ins and the user's backends file), for a backend it has at the spec's URL; any other keeps the spec's - (keep). The run config's `backends:` is not read here: a role it sets gives way to the registry's, and a backend - it declares, or moves to another URL, keeps keep.""" - import dataclasses - from hermit.agent.model.backends import registry - try: - known = registry().get(binding.backend.name) - except (OSError, ValueError): # a registry that cannot be read: the spec as it is - return binding - if known is None or known.developer_role == binding.backend.developer_role or \ - known.base_url.rstrip("/") != binding.backend.base_url.rstrip("/"): - return binding - return dataclasses.replace(binding, backend=dataclasses.replace(binding.backend, - developer_role=known.developer_role)) - - -def bind(settings: dict, kind: str, binding) -> dict: - """`bind_agent_params` for `kind` over one resolved binding: the model id, the provider settings where the CLI - reads them (Codex's `args`, Claude Code's `env`, Pi's, OpenCode's and dsh's `provider`), the price, the spec.""" - from hermit.agent.model.backends import bind_agent_params - ref = {"backend": binding.backend.name, "model": binding.model, "price": binding.price, - "extra_body": dict(binding.extra_body)} - return bind_agent_params({**settings, "model": ref}, kind, {binding.backend.name: binding.backend}) - - -def settings_for(cli: str, *, model=None, binding=None, effort=None, web_search: bool = False, - login: str | None = None, extra: dict | None = None) -> dict: - """The Base Agent's settings for a trial of `cli`: the table's, then the run's own (`acting_agent.settings`), the - model bound for the CLI's dialect (or its plain name, on a login), the effort, web search set either way, the - login, and the prompt on stdin.""" - row = entry(cli) - settings = {**json.loads(json.dumps(row["settings"])), **dict(extra or {})} - settings.update(web_search=bool(web_search), stdin=True) - if binding is not None and row["backend"] is None: # the answer agent: the model's id, priced; its spec served - settings.update(model=binding.model, price=binding.price, binding=binding.spec()) - elif binding is not None: - settings = bind(settings, row["backend"], binding) - elif isinstance(model, dict): # a model a CLI reaches (the answer agent's): the session's - settings["model"] = str(model.get("model") or "") - elif model: - settings["model"] = str(model) - if effort is not None and effort != "": # YAML's bare off (false) as the Base Agents take it: no flag - settings["effort"] = effort if isinstance(effort, bool) else str(effort) - if login: - settings["login"] = str(login) - return settings - - -# ---- rendering ---------------------------------------------------------------------------------------------------- -def check_settings(agent, tools: bool = False) -> None: - """What the CLI Base Agent's `run` refuses before any session (hermit.agent.base.cli), refused for a plan, which - harborside carries out without `run`: a login that is not the CLI's own (CliAgent.run), `text_only` with the - adapter's tools (`tools`: CliAgent._session), and mini-swe-agent's settings (MiniSweAgent.run: no login, no web - search, a model, a model class whose messages Hermit reads), the same checks with the same messages.""" - from hermit.agent.base.interface import Unsupported - from hermit.agent.base.login import LOGINS - s = agent.settings - if s.get("login") and LOGINS.get(s["login"]) != agent.backend: - raise Unsupported(f"{agent.name} does not sign in with login: {s['login']}") - if s.get("text_only") and tools: - raise Unsupported(f"{agent.name}: a session with text_only has no tools or roles") - if agent.backend != "mini-swe-agent": - return - if s.get("login"): - raise Unsupported("mini-swe-agent has no login: give a / reference or a litellm model") - if s.get("web_search"): - raise Unsupported("mini-swe-agent has no web search: set web_search false") - if not (s.get("binding") or s.get("model")): - raise ValueError("mini-swe-agent needs a model: a / reference or a litellm model id") - model_class = s.get("model_class") or "litellm" - if s.get("binding") and model_class not in agent.binding_classes: - raise Unsupported(f"mini-swe-agent's model_class {model_class!r} calls its own service; a " - f"/ reference takes {' or '.join(agent.binding_classes)}") - if model_class not in agent.binding_classes: - raise Unsupported(f"mini-swe-agent's model_class {model_class!r}: Hermit reads " - f"{' and '.join(agent.binding_classes)} only; another provider's model goes through " - "litellm (openrouter/, ...)") - - -def render(cli: str, settings: dict, *, deploy: dict | None, tools_url: str | None, job: Path, - login: str | None = None, tools_name: str | None = None, workdir: str | None = None, - bin_path: str | None = None, path_prepend: list[str] | None = None, metadata: dict | None = None, - login_root: Path | None = None, sign_in: bool | str = True) -> Path: - """The launch plan of a trial of `cli` (a table key other than answer) under `/launch/`; its path. `settings` - are the Base Agent's (settings_for), `deploy` the staging record (hermit.harbor.staging.stage) or None, - `tools_url` the MCP URL of the adapter's tools served for the trial (hermit.harbor.tools) or None, `tools_name` - their server's name (the adapter id with `-` as `_`, as the context names them). `bin_path` and `path_prepend`: - a mounted CLI's program and folder in the container. On login: chatgpt the home gets the shared copy of the - sign-in (hermit.agent.base.login.prepare) as a secret file that comes back after the trial; `sign_in` "later" lists - it in the plan and leaves the copy to `sign_in()` (a stream's step, at its turn), false renders without it (a - check).""" - from hermit.agent.base.interface import make - from hermit.agent.base.login import NO_TOKEN, TOKEN_ENV - row = entry(cli) - backend = row["backend"] - if backend is None: - raise ValueError(f"{cli} has no CLI to launch: its plan is render_answer's") - login = login or settings.get("login") - launch = Path(job) / LAUNCH_DIR - shutil.rmtree(launch, ignore_errors=True) - launch.mkdir(parents=True, mode=0o700) # its plan and home name the trial's tool URL and token: this user's only - home = launch / HOME_DIR - home.mkdir() - work = Path(deploy["deploy"]) if deploy else launch / "work" - work.mkdir(parents=True, exist_ok=True) - work = work.resolve() - home = home.resolve() - settings = dict(settings) - settings["stdin"] = True - if bin_path: - settings["bin"] = bin_path - elif backend == "mini-swe-agent": - settings["bin"] = "python3" # Hermit's runner on mini-swe-agent's interpreter (HermitMiniSweAgent) - if tools_url: - settings["tools_name"] = tools_name or "hermit_tools" - if deploy: - from hermit.agent.acting.workspace import _session_args - settings["args"] = [*(settings.get("args") or []), *_session_args(backend, deploy, work)] - if backend == "opencode": # the only skills OpenCode may load: the deployment's - settings["skills"] = list(deploy.get("skills") or []) - agent = make(backend, settings) - check_settings(agent, tools=bool(tools_url)) # what its run() refuses: a trial's plan is carried out without it - if login == "claude" and not os.environ.get(TOKEN_ENV): - raise ValueError(NO_TOKEN) - agent.prepare_home(home, [], tools_url, settings.get("tools_name") or "hermit_tools") - command = [str(a) for a in agent.command(PROMPT, work, home / LAST_MESSAGE, session=None, roles=(), - tools_url=tools_url)] - by_fd = dict(agent.by_fd()) - env, from_host = {}, {} - for name, value in (settings.get("env") or {}).items(): - _env_entry(str(name), str(value), env, from_host) - env.update({k: v for k, v in agent.session_env().items() if k not in by_fd}) - tools_env = agent.settings.get("tools_env") # the Base Agent's own default too (mini-swe-agent's) - if tools_url and tools_env: - env[tools_env] = tools_url.removesuffix("/mcp") # the tools' JSON entrance, for a CLI that calls them by shell - env.update(agent.home_env(home)) - env.update(row["env"]) - binding = settings.get("binding") or {} - key_env = _key_env(binding) - if key_env and backend != "claude": # the CLI reads the key from its own variable (Claude Code: env above) - from_host.setdefault(key_env, key_env) - secret_files, return_files = [], [] - if login == "chatgpt" and sign_in: - from hermit.agent.base import login as codex_login - from hermit.harbor.signin import MAX_BYTES - if sign_in != "later": - codex_login.prepare(Path(login_root or codex_login.ROOT), home) - secret_files.append(codex_login.FILE) - (launch / RETURN_DIR).mkdir() - return_files.append({"from": f"{CONTAINER_HOME}/{codex_login.FILE}", - "to": str((launch / RETURN_DIR / codex_login.FILE).resolve()), "max_bytes": MAX_BYTES}) - try: - return _finish(cli, row, agent, settings, command, env, from_host, by_fd, home=home, work=work, launch=launch, - job=Path(job), deploy=deploy, tools_url=tools_url, workdir=workdir, path_prepend=path_prepend, - metadata=metadata, secret_files=secret_files, return_files=return_files, login=login) - except BaseException: - for name in secret_files: # a plan refused: no copy of the sign-in is left behind - (home / name).unlink(missing_ok=True) - raise - - -def sign_in(plan_path: Path, login_root: Path | None = None) -> None: - """The ChatGPT sign-in copied into the home of a plan rendered with `sign_in="later"` (a stream's step, at its turn): - the shared copy as it is then (hermit.agent.base.login.prepare), never as it was when the step's rollout began, - which an earlier step's session may since have refreshed, and so spent. A plan with no sign-in file gets none.""" - from hermit.agent.base import login as codex_login - plan = json.loads(Path(plan_path).read_text(encoding="utf-8")) - home = plan.get("home") or {} - if codex_login.FILE in (home.get("secret_files") or []): - codex_login.prepare(Path(login_root or codex_login.ROOT), Path(home["host_dir"])) - - -def _finish(cli, row, agent, settings, command, env, from_host, by_fd, *, home: Path, work: Path, launch: Path, - job: Path, deploy, tools_url, workdir, path_prepend, metadata, secret_files, return_files, - login: str | None = None) -> Path: - """The rest of `render`, after the sign-in: the paths translated, the plan assembled, checked and written.""" - backend = row["backend"] - places = {str(home): CONTAINER_HOME, str(work): WORKDIR} - command = [_translate(a, places) for a in command] - env = {k: _translate(str(v), places) for k, v in env.items()} - _translate_home(home, places, skip=set(secret_files)) - bundle = None - if backend == "mini-swe-agent": - bundle = {"host_dir": str(_mini_bundle(launch / "bundle")), "container_dir": BUNDLE_DIR} - env["PYTHONPATH"] = BUNDLE_DIR - prompt_suffix, copy_back = "", [] - deploy_block = None - if deploy: - from hermit.agent.acting.workspace import MEMORY, MEMORY_NOTE - folder = f"hermit/{deploy['adapter_id']}" - if (work / folder / MEMORY).is_file(): - note = (Path(folder) / MEMORY).as_posix() - prompt_suffix = "\n" + MEMORY_NOTE.format(path=note) - copy_back.append((Path(folder) / MEMORY.parent).as_posix()) - ours = work / row["context_file"] if row["context_file"] else None - text = ours.read_text(encoding="utf-8", errors="replace") if ours is not None and ours.is_file() else "" - deploy_block = {"work_dir": str(work), "adapter_id": deploy["adapter_id"], "context_file": row["context_file"], - "context_limit": context_limit(cli, settings, text), - "context_escaped": row["context_escaped"] if row["context_limit"] else None, - "hooks": bool(deploy.get("hooks")), "copy_back": copy_back} - for name in [k for k in from_host if k in env]: # a variable the session's own settings set later wins - del from_host[name] - collect = [{"from": f"{CONTAINER_HOME}/{src}", "to": dst} - for src, dst in [*COLLECT.get(backend, []), *COLLECT_LOGIN.get(login or "", [])]] - # what the Base Agent puts before a prompt that starts with `/` (Claude Code would run it as one of its commands) - slash = agent.prompt_text("/") - plan = {"version": PLAN_VERSION, "cli": cli, "label": agent.label, "workdir": workdir, "deploy": deploy_block, - "home": {"host_dir": str(home), "container_dir": CONTAINER_HOME, "secret_files": secret_files}, - "bundle": bundle, "command": command, "stdin": bool(agent.on_stdin()), "prompt_suffix": prompt_suffix, - "prompt_slash_prefix": slash[:-1] if slash != "/" and slash.endswith("/") else None, - "env": env, "env_from_host": from_host, "env_by_fd": by_fd, "nvm": bool(row["nvm"]), - "path_prepend": list(path_prepend or []), "stream": f"agent-{backend}.jsonl", "collect": collect, - "return_files": return_files, - "tools_script": {"url_env": agent.settings["tools_env"]} if tools_url and agent.settings.get("tools_env") - else None, - "kill_grace_s": KILL_GRACE_S, - "metadata": {"hermit_cli": cli, "backend": backend, **(metadata or {})}} - check(plan, job=Path(job), secret_files=secret_files) - path = launch / PLAN_FILE - path.write_text(json.dumps(plan, indent=1, ensure_ascii=False) + "\n", encoding="utf-8") - return path - - -def render_answer(settings: dict, *, endpoint: str, job: Path, system: str = "", options: dict | None = None, - metadata: dict | None = None) -> Path: - """The answer-only agent's launch plan: no home and no command, the `answer` block (the endpoint on this - machine that serves the model, the model's id, the adapter's context as the system text, the answer's options: - instruction_file, wait_s, attachments, answer_file, answer_pattern, answer_format, max_tokens); its path. An - answer_file of null is written nowhere (/app/answer.txt when none is given); a wait_s of null is the default - 30 s.""" - options = dict(options or {}) - unknown = set(options) - {"instruction_file", "wait_s", "attachments", "answer_file", "answer_pattern", - "answer_format", "max_tokens"} - if unknown: - raise ValueError(f"agent.params.answer takes instruction_file, wait_s, attachments, answer_file, " - f"answer_pattern, answer_format and max_tokens, not {sorted(unknown)}") - if options.get("answer_pattern"): - re.compile(str(options["answer_pattern"])) - answer_format = str(options.get("answer_format") or "{answer}") - if "{answer}" not in answer_format: - raise ValueError("agent.params.answer.answer_format must hold {answer}") - launch = Path(job) / LAUNCH_DIR - shutil.rmtree(launch, ignore_errors=True) - launch.mkdir(parents=True, mode=0o700) # its plan names the relay's URL and token: this user's only - effort = settings.get("effort") - plan = {"version": PLAN_VERSION, "cli": "answer", "label": "answer", "workdir": None, "deploy": None, - "home": None, "bundle": None, "command": None, "stdin": False, "prompt_suffix": "", - "prompt_slash_prefix": None, "env": {}, - "env_from_host": {}, "env_by_fd": {}, "nvm": False, "path_prepend": [], "stream": "answer.json", - "collect": [], "return_files": [], "tools_script": None, "kill_grace_s": KILL_GRACE_S, - "metadata": {"hermit_cli": "answer", **(metadata or {})}, - "answer": {"endpoint": endpoint, "model": str(settings.get("model") or ""), "system": system or "", - "instruction_file": options.get("instruction_file"), - "wait_s": float(30 if options.get("wait_s") is None else options["wait_s"]), - "attachments": [str(a) for a in options.get("attachments") or []], - "answer_file": None if options.get("answer_file", "") is None else - options.get("answer_file") or "/app/answer.txt", - "answer_pattern": options.get("answer_pattern"), "answer_format": answer_format, - "max_tokens": int(options["max_tokens"]) if options.get("max_tokens") else None, - "effort": None if effort in (None, "", False) else str(effort)}} - check(plan, job=Path(job), secret_files=[]) - path = launch / PLAN_FILE - path.write_text(json.dumps(plan, indent=1, ensure_ascii=False) + "\n", encoding="utf-8") - return path - - -def context_limit(cli: str, settings: dict, text: str = "") -> int | None: - """The bytes the deployment's context file may hold once merged with the image's own: the table's limit, less a - byte for each `context_escaped` in `text` (Hermit's own context: dsh escapes each `` by one more - byte, as hermit.hosts.dsh counts it; the image's own file's are counted where it is merged, the plan's - `deploy.context_escaped`), else Codex's own cap (project_doc_max_bytes, past which Codex would cut the file without - a word).""" - row = entry(cli) - if row["context_limit"]: - return int(row["context_limit"]) - (text.count(row["context_escaped"]) if row["context_escaped"] else 0) - if row["backend"] == "codex" and settings.get("project_doc_max_bytes"): - return int(settings["project_doc_max_bytes"]) - return None - - -def _key_env(binding: dict) -> str | None: - if not binding: - return None - from hermit.agent.model.backends import Binding - return Binding.from_spec(binding).key_env_if_set() - - -def _env_entry(name: str, value: str, env: dict, from_host: dict) -> None: - """A setting's variable into the plan: `$NAME` alone passes that variable by name (its value never written); - other `$NAME`s are expanded with this machine's values, as a session's are (`$PATH` is this machine's PATH), - unless one names a secret (SECRET_NAME's, SECRET_VAR's or Harbor's rule), which is refused, as is a value given - as it is to a variable named as a secret (SECRET_VAR's: a word of its name is a secret's): either would be written - into the plan.""" - whole = re.fullmatch(r"\$(?:\{(\w+)\}|(\w+))", value) - if whole: - from_host[name] = whole.group(1) or whole.group(2) - return - named = [a or b for a, b in VAR_REF.findall(value)] - secret = [n for n in named if SECRET_NAME.search(n) or SECRET_VAR.search(n) or HARBOR_SECRET.search(n)] - if secret: - raise ValueError(f"the acting agent's env {name} holds ${secret[0]} inside other text: a secret reaches the " - f"container only whole, by its variable's name ({name}: ${secret[0]})") - from hermit.agent.model.backends import PLACEHOLDER_KEY - if value and value != PLACEHOLDER_KEY and SECRET_VAR.search(name): # a keyless backend's placeholder: no secret - raise ValueError(f"the acting agent's env {name} is named as a secret and given a value: it would be written " - f"into each trial's plan; export it here and pass it by name ({name}: ${name})") - env[name] = os.path.expandvars(value) if named else value - - -def _translate(text: str, places: dict[str, str]) -> str: - for host, container in sorted(places.items(), key=lambda kv: -len(kv[0])): # the longest first - text = text.replace(host, container) - return text - - -def _translate_home(home: Path, places: dict[str, str], skip: set[str]) -> None: - """The home's text files with host paths made container paths (a binary file holds none of them).""" - for path in sorted(p for p in home.rglob("*") if p.is_file() and not p.is_symlink()): - rel = path.relative_to(home).as_posix() - if rel in skip or path.stat().st_size > TEXT_LIMIT: - continue - try: - text = path.read_text(encoding="utf-8") - except (UnicodeDecodeError, OSError): - continue - new = _translate(text, places) - if new != text: - path.write_text(new, encoding="utf-8") - - -def _mini_bundle(target: Path) -> Path: - """The Hermit modules mini-swe-agent's runner imports, copied under `target`.""" - from hermit.sandbox import FRAMEWORK - for rel in MINI_BUNDLE: - (target / rel).parent.mkdir(parents=True, exist_ok=True) - shutil.copy2(Path(FRAMEWORK) / rel, target / rel) - return target.resolve() - - -# ---- what a plan must never hold ---------------------------------------------------------------------------------- -def secret_values() -> dict[str, str]: - """{variable: value} of this process's secrets a plan must not hold: every key-like variable's value - (hermit.redact's rule) and Claude Code's subscription token.""" - from hermit.agent.base.login import TOKEN_ENV - from hermit.redact import key_values - out = {name: value.decode() for name, value in key_values().items()} - if len(os.environ.get(TOKEN_ENV) or "") >= 8: - out[TOKEN_ENV] = os.environ[TOKEN_ENV] - return out - - -def check(plan: dict, *, job: Path, secret_files) -> None: - """Refuse a plan that names a path of this machine's job directory where the container reads it (its command, - its environment, its home's text files) or that holds a secret's value anywhere but a secret file.""" - secrets = secret_values() - home = Path((plan.get("home") or {}).get("host_dir") or "") - texts = [("the command", " ".join(plan.get("command") or [])), - ("the environment", json.dumps(plan.get("env") or {})), - ("the answer block", json.dumps(plan.get("answer") or {})), - ("the prompt suffix", plan.get("prompt_suffix") or "")] - if plan.get("home"): - for path in sorted(p for p in home.rglob("*") if p.is_file() and not p.is_symlink()): - rel = path.relative_to(home).as_posix() - if rel in set(secret_files) or path.stat().st_size > TEXT_LIMIT: - continue - try: - texts.append((f"the home's {rel}", path.read_text(encoding="utf-8"))) - except (UnicodeDecodeError, OSError): - continue - job_text = str(Path(job).resolve()) - for where, text in texts: - for name, value in secrets.items(): - if value and value in text: - raise RuntimeError(f"hermit: the launch plan would hold the value of {name} in {where}: a secret " - "reaches the container by its variable's name only") - if job_text in text and where != "the answer block": - raise RuntimeError(f"hermit: the launch plan names a path of this machine in {where} ({job_text}): the " - "container would not find it") diff --git a/hermit/harbor/report.py b/hermit/harbor/report.py index ca911cce..dfb10de2 100644 --- a/hermit/harbor/report.py +++ b/hermit/harbor/report.py @@ -2,15 +2,15 @@ python -m hermit.harbor.report ... [--json] -`hermit eval` scores a trial that is not the agent's result (infra: the host, the provider, a session that never -started, once the runner's own runs again are spent: agent.params.infra_retries) as a failed run: its judge -(hermit.harbor.judge) gives it no score, so it counts among the summary's `errors` and as 0 in its `mean_score`. A -benchmark's own figure leaves such trials out. This reads the eval's results.jsonl (`/eval//`) and -gives, over all its repeats and for each repeat: +`hermit eval` scores a trial that is not the agent's result (infra, once the run's runs again are spent: +environment.harbor.infra_retries) as a failed run: its judge (hermit.task.verify.environment) gives it no score, so it +counts among the summary's `errors` and as 0 in its `mean_score`. A benchmark's own figure leaves such trials out. This +reads the eval's results.jsonl (`/eval//`) and gives, over all its repeats and for each repeat: trials the rows of the eval scored the trials the judge scored (status ok): the agent's results - infra the trials that are not the agent's result (the runner's verdict: output.status infra) + infra the trials that are not the agent's result after the run's runs again (the row's output.status or + its outcome.status infra, or the judge's Unscored) failed the other failed runs (the pipeline or the judge failed for another reason) pass_at_1 of the scored trials, the share that passed (the judge's `passed`, which hermit eval keeps in the row's judge.extra: the reward reached the run's pass_reward); null when the judge says no trial @@ -113,12 +113,20 @@ def later(path: Path) -> str | None: "not finish, beside the summary of the eval before") +INFRA_SAID = ("Unscored", "the task's environment could not be given") + + def kind(row: dict) -> str: """scored, infra or failed (the module's docstring).""" output = row.get("output") if isinstance(row.get("output"), dict) else {} - if output.get("status") == "infra": + outcome = row.get("outcome") if isinstance(row.get("outcome"), dict) else {} + if output.get("status") == "infra" or outcome.get("status") == "infra": # a rollout that stayed infra return "infra" - return "scored" if row.get("status") == "ok" else "failed" + if row.get("status") == "ok": + return "scored" + judged = row.get("judge") if isinstance(row.get("judge"), dict) else {} + said = f"{row.get('error') or ''} {judged.get('feedback') or ''}" + return "infra" if any(word in said for word in INFRA_SAID) else "failed" def passed(row: dict): diff --git a/hermit/harbor/runner.py b/hermit/harbor/runner.py deleted file mode 100644 index dce13e23..00000000 --- a/hermit/harbor/runner.py +++ /dev/null @@ -1,2371 +0,0 @@ -"""`agent.name: harbor`: one Hermit rollout as one Harbor trial, or one step of a Harbor multi-step task. - - agent: - name: harbor - concurrency: 8 - params: - benchmark: ../utils/benchmark.yaml # optional: these same keys, shared by every method's config (its - # model a name, or a model a CLI reaches: a `/` - # reference goes in the run config, where hermit check sees it) - tasks_dir: ../utils/tasks # the Harbor task directories (dataset.py exports or indexes them) - model: openai/gpt-5.6-luna # a `/` reference (the backend's endpoint and key), or - # with `login` the CLI's model name; the answer agent's model a CLI - # reaches ({cli: codex | claude, model, login}, on the CLI's login) goes - # in the benchmark file: here it is refused (CONFIG_CLI_MODEL) - login: chatgpt # Codex on the user's ChatGPT sign-in; claude: Claude Code on the user's - # subscription (CLAUDE_CODE_OAUTH_TOKEN; plan_dir, usage_credits, - # busy_retries, busy_wait_s, login_root as a CLI session's, - # hermit.agent.base.login) - acting_agent: # the agent in the container - cli: codex # codex (the default) | claude | pi | opencode | dsh | mini-swe-agent | - # answer (one model call, no tools: its answer written in the container); - # or name: / import_path: (no adapter) - settings: {} # the CLI Base Agent's own settings over the table's (codex `args`, ...) - kwargs: {} # options the HermitXXX class declares - env: {} # variables of every agent exec in the container (no secrets) - pythonpath: [] # directories Harbor must import an agent class from (an example's own) - effort: medium # the CLI's reasoning effort - web_search: false # the CLI's web search, set either way (required; off for a task whose - # agent phase has no public network) - codex_bin: ~/.codex/packages/standalone/releases/0.164.0-x86_64-unknown-linux-musl/bin/codex - # a CLI's binary on this machine (codex_bin, claude_bin, pi_bin, - # opencode_bin, dsh_bin), its folder mounted read-only at - # /opt/hermit- and first on PATH, used when it answers --version - # (asked as root) as _version and the task's agent user may - # execute it; else the container installs - codex_version: "0.164.0" # what that install installs (_version; default: clis.HARBOR_CLIS's) - setup_timeout_sec: null # Harbor's agent setup limit (an npm install under load needs more - # than Harbor's 360 s) - answer: {} # the answer agent's: instruction_file, attachments, answer_file, - # answer_pattern, answer_format, wait_s, max_tokens - pass_reward: 1.0 # the reward that is a pass ("passed"); null: none is (a score scale) - verifier_redact: [] # regexes removed from the curated verifier output (the feedback's tail) - infra_retries: 2 # a trial that ended infra (not the agent's result) runs again this many - # times at most (a stream's step never) - infra_wait_s: 300 # ... each after this long - agent_timeout_sec: 3600 # uniform agent budget; null: each task's own - environment: {delete: true} # Harbor's environment config (type, import_path, force_build, kwargs, - # env, extra_docker_compose: overlays, relative to the benchmark file) - verifier: {} # Harbor's verifier config (override_timeout_sec, ...) - verifier_timeout: agent # a verifier that ran out of its time: the agent's result, kept, as - # Harbor grades it (its solution too slow for the tests, or a process it - # left that hung them); infra: not the agent's result (a host whose load - # makes verifiers time out), run again (infra_retries) - verifier_timeout_multiplier: 1.0 # Harbor's: each task's verifier limit times this (a slow host) - environment_build_timeout_multiplier: 1.0 # Harbor's: each task's environment start (pull, build) limit - # times this; the agent's limit is never scaled - retry: {max_retries: 0} # Harbor's retry of a failed trial (not of a stream's step; none on - # login: chatgpt, whose every attempt would start from the sign-in copied - # before the first); a failed attempt's usage is not counted (Harbor - # removes its trial): infra_retries runs each attempt as a trial of its own - reward_key: reward # the verifier's reward the judge reads - jobs_dir: ~/.cache/hermit/harbor/jobs # one Harbor job per rollout, outside the checkout - tmpdir: /tmp # TMPDIR for harbor; never /tmp - image_sizes: ../utils/image_sizes.json # compressed bytes per task image: the disk gate keeps room for an - # image not pulled yet (2.5x on disk), and for the pulls of other trials - harbor: ~/.cache/hermit/harbor-venv/bin/harbor # Harbor's command: a name is found on PATH, a relative - # path in the working directory - docker_cleanup: images_keep # images_keep | rmi_task_image | rmi_over_gb (docker.py) - rmi_threshold_gb: 1.0 - disk_gate_gb: 20 # no trial starts while Docker's data root has less than this free - disk_gate_wait_s: 7200 # ... each gate at most this long, in turn; then infra (disk) - tmp_gate_gb: 1.0 # ... nor while the host's /tmp has less (runc writes each exec's - # process file there: full, every agent setup fails) - tools_host: null # the address the adapter's tools are served on (default: the Docker - # bridge's gateway, which a task container reaches) - sandbox_enforce: required # the adapter sandbox's enforce (its tools, its per-task context): - # required | best_effort - context_query: null # the row's field a per-task context retrieves for (default: the - # instruction) - python: # the one the hosts' deployment and the per-task context run with - stream: {idle_s: 14400, start_timeout_s: 3600, turn_wait_s: null} # a multi-step task's driver: idle stop, - # environment start, how long a step waits for its turn (null: a trial's - # budget for each step before it, at least one, and the start's; a turn - # that cannot come ends at once) - -A task row (`dataset.index`) names its directory as `harbor_task`; a step row adds `step`, `step_index`, `steps`. - -A trial: the adapter is staged for the acting CLI (`staging.stage`: its host's deployment, the task's context when the -adapter's context is per task), its tools served from this machine (`tools.AdapterTools`), the CLI's launch plan -rendered by Hermit's own CLI Base Agent (`launch.render`: the CLI's home, command and environment for the container), -the job config written, `harbor run -c job.yaml` run in Harbor's own venv as a subprocess, and the trial read back: the -verdict of its `result.json` (`errors.classify`), then the CLI's own stream (`agent/agent-.jsonl`) read as -Hermit reads a session's, by the CLI's Base Agent (`outcome`, `parse`: served, refused for a plan's limit, a provider -that could not serve or that refused its requests, a lost sign-in, a session that never started or whose tools never -came up; the usage, priced at the run's price: agent.params.price, else the backend's), and for Codex whether its -sign-in or key was refused (`signin.codex_signin_lost`). What the container wrote is read only from regular files, -never through a link it left there (which could name a file of this machine): the stream and the CLI's last message -are copied into the job's own `launch/read/` first, and the Base Agent reads those copies. -A step: the stream's driver (`stream.StreamClient`, `harborside/driver.py`) is started with the first step and keeps -the task's environment; each rollout stages its adapter, renders its plan, waits for its turn, asks for its step (with -its own allowed hosts) and reads the step's result; the last step (or an abort) stops the driver. A step whose turn -cannot come (its driver waits for a step no rollout of this runner is running: one that ended before it reached the -driver, or ran in an earlier training step; or it has no driver, and no rollout of its first step is running to start -one) ends at once, infra (stream_order), and so does every later step of its stream, whose driver the last of them -stops; a driver that answers no stop (hung in a step) is killed, and its trial's compose projects are taken down. A new -driver in a stream's directory finds the earlier one's trial set aside (`earlier//`). A stream is the row's rollout -of the task in this runner: `/streams/__s__` (each runner its own id, so two -runs on one jobs directory, at once or one after the other, a resumed run among them, never share a driver, its -environment or its step logs). Its steps run in their order: a training split that runs one keeps `data.shuffle: -false`, so that the Trainer hands its rows in the split's order. When the runner is freed, or at exit for one still -held then (`_close_runner_streams`), the drivers of the streams that did not finish are stopped, one after another, -each killed if it has not stopped within CLOSE_STOP_S, which would otherwise hold their tasks' environments for -`stream.idle_s`, and every stream's tool server is closed: no stream is resumed. - -What the rollout returns: `output` {harbor_task, rollout, step, reward, rewards, status, failure_cause, -exception_type, trial, hooks_active, reply (the CLI's last message), answer (the answer agent's)}, the usage, the -curated log (`curate.py`), and `extra` with the verdict, the session's facts, the job, what was deployed and its -per-task context (`context_tool`), `judge` (pass_reward) and, for a trial that is not the agent's result (infra: -the host, the provider, a disk gate, a session that never started), `unscored` (its cause). Such a trial runs again, -up to `infra_retries` times, each after `infra_wait_s` (not a stream's step, whose environment went on without it, -nor a task Harbor runs for none of Hermit's agents, a step whose turn cannot come, a disk gate that already waited -its time, a provider's refusal of the run's configuration, model_rejected, or a host failure, host_failed: the jobs -directory that cannot be made, or a failure past the trial's staging, whose session may have run); what every run -used is the rollout's (`extra.attempts`). One that ends -infra all the same is no verdict: -the judge (hermit.harbor.judge) raises on it, so the run records a failed run (status judge_error), metered and with -its log, which `hermit eval` scores 0 in its mean and counts among its errors; `python -m hermit.harbor.report` gives -an eval's pass@1 over the trials that are the agent's result beside it. A lost sign-in, or the usage limit of an API -account (no login waits it out), or a provider that does not bill the account (its quota or credit spent, a 402), -ends the run: the rollout raises hermit.agent.base.interface.LoginFailed, which -hermit.trainer.rollout passes up (no further rollout starts; `hermit train` resumes after its last finished step, -`hermit eval` stops), and so does every rollout of this runner that starts, or was waiting for the plan or a slot, -after it. - -On a login (`login: chatgpt` or `claude`) a trial shares the plan with every Hermit session on the login: it waits -while the plan holds sessions back (its PAUSE, MAX_USED, a refusal's hold) and runs in one of its slots; what the -session says of the plan (Claude Code's stream; Codex's session logs, which the launch plan collects) is kept before -the slot is given back, and so is whether the plan refused the session: then every session is held until the window -resets and the trial runs again, so that a usage limit costs time, never a trial. A session the provider could not -serve runs again `busy_retries` times, a count over all of the trial's runs that infra_retries adds none to, unless it -ran out of the agent's time (its result, as Hermit's own sessions keep it). The answer agent's model reached -through a CLI on a login that the runner serves itself (`model: {cli, login}` in the benchmark file: each request one session on the login, in a sandbox of its own) waits for that plan before each -trial, and its relay waits for it before each request (`tools.ModelEndpoint`): a trial whose one request was still -unanswered when the agent's time ran out, held back by that plan, is infra (usage_limit), not the agent's timeout; -its sessions run with `plan_rerun: false`, so that one the plan refused comes back at once (the relay's 429: infra, -usage_limit, run again once the plan allows, not counted in infra_retries), never waiting for the window inside the -request. That wait does not -cover a slot, which the request's session takes itself (CliAgent.run): a request waiting for one -(the plan's SLOTS all taken) waits inside the agent's time, and its trial is the agent's timeout (its result) if that -time runs out; so keep agent.concurrency at most SLOTS less the login's other sessions. Such a model given in the run -config's own agent.params.model would be served by the pipeline (hermit.isolate.agent_pipeline, as for any acting -agent), whose session waits for the plan inside the request, which the runner could not tell from the model's own -time: the runner refuses it at startup (PIPELINE_CLI_SPEC; as written, as hermit check hands it the run config's -params), and the benchmark file is where it goes. Codex's -refreshed sign-in comes back from the container and becomes the shared copy (`signin.take_back`); a stream's step is -given the shared copy at its turn, never when its rollout began. -""" -from __future__ import annotations - -import dataclasses -import glob -import json -import os -from pathlib import Path -import re -import shutil -import signal -import stat -import subprocess -import sys -import threading -import time -import uuid -import weakref - -import yaml - -from hermit.agent.acting.agent import ALL, Agent -from hermit.agent.base import login as subscription -from hermit.agent.base.interface import LoginFailed -from hermit.harbor import clis -from hermit.harbor import curate as curate_mod -from hermit.harbor import dataset -from hermit.harbor import docker -from hermit.harbor import launch -from hermit.harbor import signin -from hermit.harbor import stream as stream_mod -from hermit.harbor.errors import adapter_failed, classify, no_trial, verifier_disk_full, verifier_unreached -from hermit.harbor.staging import stage -from hermit.harbor.stream import (DRIVER_LOG, StepAlreadyRunError, StreamClient, StreamError, StreamOrderError, - TurnLostError) -from hermit.harbor.tools import UNSUPPORTED -from hermit.task import RunOutput - -HARBOR = "~/.cache/hermit/harbor-venv/bin/harbor" -JOBS_DIR = "~/.cache/hermit/harbor/jobs" -HARBORSIDE_ROOT = Path(__file__).resolve().parent # the directory holding the harborside package -SETUP_BUDGET_S = 360 # Harbor's agent setup: apt + nvm + npm + python3 -TEARDOWN_BUDGET_S = 600 # image pull, container start, compose down -DEFAULT_AGENT_TIMEOUT_S = 3600 -TIMEOUT_MULTIPLIERS = ("verifier_timeout_multiplier", "environment_build_timeout_multiplier") # into Harbor's job -DISK_GATE_GB = 20.0 -DISK_GATE_WAIT_S = 7200.0 -TMP_GATE_GB = 1.0 -HOST_TMP = Path("/tmp") -INFRA_RETRIES = 2 # infra_retries' default -INFRA_WAIT_S = 300.0 # infra_wait_s' -# the causes of an infra trial that run no better again: a disk gate that waited its time out, a request the provider -# refused for the run's configuration (its key, permission, model or bill: no input to the session changes it). A task -# Harbor runs for none of Hermit's agents (unsupported_task) and a stream's step never run again: neither reaches -# _run_trial -NO_RETRY = ("disk", "model_rejected") -# a variable an acting_agent.env value names: a whole value ${NAME} or ${NAME:-default}, which Harbor fills in from -# this process's environment (harbor.utils.env.resolve_env_vars), and any other reference, $NAME or one inside a longer -# value, which it hands on as that text -ENV_REF = re.compile(r"\$(?:\{(\w+)(?::-[^}]*)?\}|(\w+))") -# a variable named as a secret, as Harbor names one (harbor.utils.env._SENSITIVE_KEY_RE): OPENAI_API_KEY, GH_TOKEN, -# HF_TOKEN_READ, AWS_SECRET_ACCESS_KEY, ...; never a count (MAX_OUTPUT_TOKENS) -HARBOR_SECRET = launch.HARBOR_SECRET -NESTED = ("acting_agent", "environment", "verifier", "retry", "stream", "answer") # params merged one level deep -BINS = ("codex", "claude", "pi", "opencode", "dsh") # the CLIs a binary of this machine may be mounted for -# What has Harbor's own codex agent sign in with the user's ChatGPT file (~/.codex/auth.json, or the file named) -CODEX_SIGN_IN = ("CODEX_FORCE_AUTH_JSON", "CODEX_AUTH_JSON_PATH") -TOOLS_PLACEHOLDER = "http://0.0.0.0:0/" + "0" * 32 + "/mcp" # a dry render's tools URL (the startup check's) -ANSWER_PLACEHOLDER = "http://127.0.0.1:0/" + "0" * 32 # ... and the answer agent's endpoint -# The keys of the spec hermit.agent.base.endpoint.endpoint_spec gives a pipeline for a model only a CLI reaches, which an -# endpoint the pipeline started serves (hermit.isolate.agent_pipeline: a run config's own `agent.params.model: {cli: -# ...}`); a run config's model, once the config is loaded, is a Binding's spec, which has more (its ref, backend, apis) -PIPELINE_CLI_SPEC = frozenset({"base_url", "model", "api_key_env", "key_required", "price", "timeout"}) -CONFIG_CLI_MODEL = ("the answer agent's model is one a CLI reaches that the run config gives (agent.params.model: " - "{cli: ...}), which the pipeline serves: its session waits for its login's plan inside the " - "request, which the runner cannot tell from the model's own time, so a trial whose time ran out " - "as it waited would be scored as the agent's. Give it in the benchmark file (agent.params." - "benchmark, its model: {cli: ...}), whose model the runner serves itself, waiting for the plan " - "apart") -SESSION_FACTS = ("served", "refused", "unavailable", "login_failed", "rejected", "model_refusal", "held_until", - "served_model", "plan_report", "started", "tools_down", "finish_reason", "calls", "statuses") -# A rejection that refused the run's configuration, which no input to the session changes, as the CLI's own notices -# (the session's `failure`) name it: its key or token (401, unauthorized, a failed authentication, an invalid key or -# token, a key not valid, not logged in, OpenCode's provider.auth), its permission (403, forbidden, no access), its -# model or endpoint (404, not found, a model that does not or may not exist, an invalid or unknown model, one that is -# not a valid model), its account's bill (402, a credit balance too low, an insufficient quota, Codex's "Quota exceeded. -# Check your plan and billing details.", billing; not a rate limit's advice, "billing details" or a link to the billing -# page). A status is read only where one is written: leading -# the text or a line of it (Pi's "401 {...}"), after the prefix Hermit's endpoint gives a CLI's failure in its reply -# ("...(exit 1): 401 {...}", "...(timed out): "; endpoint._no_reply), after "API Error: " (Claude Code's) or -# "Error code: " (OpenAI's SDK), in "(HTTP 401)" (OpenCode's) or in "API error (401)" (Pi's), after "status" ("status -# code", "status_code") or a JSON "code"; never another number of the text (a token count: "you requested 4402 tokens -# (402 in the messages, ...)", "input tokens (402): 403 over"). So is a request setting the model does not take (an -# unsupported parameter or value: max_tokens where it takes max_completion_tokens, an effort it has not; Azure's -# unrecognized request argument; OpenCode's provider.unsupported-operation), and what Hermit's own endpoint refuses a -# CLI session cannot take (its code, tools.UNSUPPORTED: config_refused). Any other rejection (a -# prompt too long, an invalid request, a content filter) is what the adapter's or the task's input made the request: -# the agent's result. (A rate limit or an overload is `unavailable`, read before any rejection.) -CONFIG_REFUSED = re.compile(r"(?:^|\((?:exit -?\d+|timed out)\): |\bAPI error(?:: | \()|\bError code: |" - r"(?:\bstatus(?:[ _]code)?|\"code\")\"?:? ?\"?)40[1-4]\b|\(HTTP 40[1-4]\)|" - r"unauthori[sz]ed|forbidden|authenticat|\bauth\b|not logged in|" - r"invalid[\w -]{0,20}?(?:key|token)\b|\b(?:key|token) not valid|permission|access denied|" - r"not have access|not[ _]found|(?:may|does|model) not exist|(?:invalid|unknown)[ _]model|" - r"not a valid model|credit balance is too low|insufficient[_ ]?quota|" - r"^\s*Quota exceeded\. Check your plan and billing details\b|" - r"(? bool: - """A regular file at `src` (never through a link: curate.open_regular) copied to `dst`; whether it was.""" - fd = curate_mod.open_regular(src) - if fd is None: - return False - with os.fdopen(fd, "rb") as f: - try: - dst.parent.mkdir(parents=True, exist_ok=True) - with open(dst, "wb") as out: - shutil.copyfileobj(f, out) - except OSError: - return False - return True - - -def _folder(path: Path) -> bool: - """Whether `path` is a directory itself, not a link to one (lstat).""" - try: - return stat.S_ISDIR(os.lstat(path).st_mode) - except OSError: - return False - - -def _text(path: Path) -> str: - """A regular file's text, or "" when it is missing, unreadable or no regular file (never read through a link).""" - return _read(path) or "" - - -def _close_runner_streams(drivers: dict, streams: dict, lock) -> None: - """What a runner's streams still hold, closed (HarborAgent.close): the driver still running of each stream that did - not finish is stopped (no later runner meets it again: each runner names its own streams, and each would hold its - task's environment for stream.idle_s; a stream that finished was stopped by its last step), and every stream's tool - server is closed. It is the runner's finalizer too (weakref.finalize): it runs when the runner is freed, and at exit - for a runner still held then, since main's hermit.isolate.agent_pipeline hands the runner over without a close of - its own (`hermit check`'s probe, `hermit eval`, `hermit train`, a run a lost sign-in stopped). It holds the runner's - state, never the runner.""" - while True: - with lock: - if not drivers: - break - _, client = drivers.popitem() - try: - if client.running() is not None: - client.stop(timeout_s=CLOSE_STOP_S) # one after another, at exit too: a hung one is killed sooner - except Exception: # noqa: BLE001 - best effort, as at exit - pass - while True: - with lock: - if not streams: - break - _, served = streams.popitem() - try: - served.close() - except Exception: # noqa: BLE001 - best effort, as at exit - pass - - -def _events(raw: str): - """The JSON objects of a CLI's stream, one per line (a line that is none, or one nested past the JSON parser's - depth, is skipped; U+0085, U+2028 and U+2029 end no line, though main's reader, hermit.agent.base.cli._objects, - ends one there: `hermit/harbor/FRAMEWORK_SUPPORT.md`, candidate 9).""" - for line in raw.split("\n"): - if not line.lstrip().startswith("{"): - continue - try: - event = json.loads(line) - except (ValueError, RecursionError): # no JSON, or nested past the parser's depth - continue - if isinstance(event, dict): - yield event - - -def _safe(text: str) -> str: - return re.sub(r"[^A-Za-z0-9_.-]", "_", str(text)) - - -def billing_refused(facts: dict) -> bool: - """Whether a refusal of the run's configuration (config_refused) is the provider's refusal to bill the account - (BILLING_REFUSED): the answer agent's model endpoint's last status 402, or the message.""" - statuses = facts.get("statuses") or [] - return (bool(statuses) and statuses[-1] == 402) or facts.get("code") == QUOTA_SPENT or \ - bool(BILLING_REFUSED.search(str(facts.get("failure") or ""))) - - -def quota_spent(facts: dict) -> bool: - """Whether a session the provider could not serve (`unavailable`: a 429 among them) was refused because the API - account's quota or credit is spent (QUOTA_SPENT: the relay's code, a CLI's words), which no pause ends.""" - return facts.get("code") == QUOTA_SPENT or bool(QUOTA_SAID.search(str(facts.get("failure") or ""))) - - -_SANDBOX_STARTED: list[bool] = [] # a strict adapter sandbox started in this process (a runner's startup) - - -def sandbox_fault(enforce: str | None, *, fresh: bool = True) -> str | None: - """Why the strict adapter sandbox that renders each adapter does not start on this machine (bubblewrap that cannot - make its namespaces, say), seen from this process: a child of it running this interpreter (the deployment's own - starts it as its child does, for its own interpreter); None when it starts, or when it is not required (enforce - best_effort, or HA_ADAPTER_SANDBOX_ENFORCE). `fresh` false (a runner's startup): once it started in this process, - it is not tried again.""" - from hermit.adapter.session import ENV_ENFORCE - from hermit.sandbox import run_in_strict_sandbox - if (enforce or os.environ.get(ENV_ENFORCE, "required")) != "required" or not shutil.which("bwrap"): - return None - if not fresh and _SANDBOX_STARTED: - return None - try: - run = run_in_strict_sandbox([sys.executable, "-c", "pass"], role="render", readable=[], enforce="required", - timeout=120) - except (OSError, subprocess.SubprocessError) as exc: - return f"the strict adapter sandbox did not start: {type(exc).__name__}: {exc}" - if run.returncode != 0: - return (f"the strict adapter sandbox did not start (exit {run.returncode}): " - f"{(run.stderr or run.stdout or '').strip()[-300:]}") - _SANDBOX_STARTED.append(True) - return None - - -def config_refused(facts: dict) -> bool: - """Whether a session's rejection refused the run's configuration (CONFIG_REFUSED): the answer agent's by its model - endpoint's last status, or by its message beside a 400 (the one status Hermit's endpoint answers a CLI session the - provider rejected with, whatever the reason: endpoint._no_reply), or by that 400's code when Hermit's endpoint - refused what a CLI session cannot take (UNSUPPORTED: a figure, tool calls, another model; settings the session - cannot start with); a CLI's by its own notices.""" - statuses = facts.get("statuses") or [] - said = bool(CONFIG_REFUSED.search(str(facts.get("failure") or ""))) - if statuses: - return statuses[-1] in CONFIG_STATUSES or (statuses[-1] == 400 and (said or facts.get("code") == UNSUPPORTED)) \ - or facts.get("code") == QUOTA_SPENT - return said - - -def _merge(base: dict, over: dict) -> dict: - out = dict(base) - for key, value in over.items(): - if key in NESTED and isinstance(value, dict) and isinstance(out.get(key), dict): - out[key] = {**out[key], **value} - else: - out[key] = value - return out - - -class _TurnUnready(Exception): - """A stream step's turn came and this machine could not ready it (the sign-in's copy, the adapter's tools): `cause` - is the step's verdict's.""" - - def __init__(self, cause: str, message: str): - self.cause = cause - super().__init__(message) - - -def supports_of_cli(cli: str | None) -> tuple[str, ...]: - """What a trial's agent takes of an adapter: a CLI all of it (staged as `hermit deploy` deploys it), the answer - agent its context, a context tool's computed for the task beforehand (staging.stage; it has no tools, skills, - roles or hooks), Harbor's own agents nothing (they take no adapter).""" - if cli is None: - return () - return ("context", "context_tool") if cli == "answer" else ALL - - -def supports_of_params(params: dict | None, base: Path) -> tuple[str, ...]: - """HarborAgent.supports from its params alone (its benchmark file's acting_agent under the params' own), without - the Harbor installation building the agent needs.""" - given = dict(params or {}) - if given.get("benchmark"): - shared = yaml.safe_load((Path(base) / Path(given["benchmark"]).expanduser()).read_text()) or {} - given = _merge(shared if isinstance(shared, dict) else {}, {k: v for k, v in given.items() if k != "benchmark"}) - spec = dict(given.get("acting_agent") or {}) # as the runner merges it (a null resets it) - if not spec.get("name") and not spec.get("import_path") and not spec.get("cli"): - spec["cli"] = clis.DEFAULT_CLI - return supports_of_cli(clis.cli_of(spec)) - - -class HarborAgent(Agent): - """`agent.name: harbor`.""" - - def __init__(self, params: dict | None = None, base: Path | str | None = None): - super().__init__(params, base) - given = dict(self.params) - if given.get("benchmark"): - shared = yaml.safe_load(self.path(given["benchmark"]).read_text()) or {} - if not isinstance(shared, dict): - raise ValueError(f"agent.params.benchmark {given['benchmark']} is not a mapping") - shared.pop("dataset", None) # the task set's preparation (dataset.prepare), not a param - if "model" not in given: - self.check_shared_model(shared.get("model"), given["benchmark"]) - self.params = _merge(shared, {k: v for k, v in given.items() if k != "benchmark"}) - self.benchmark_base = self.path(given["benchmark"]).parent - else: - self.benchmark_base = self.base - if not self.params.get("tasks_dir"): - raise ValueError("agent.params.tasks_dir is required (the Harbor task directories)") - if "trial_cache" in self.params: - raise ValueError("agent.params.trial_cache: there is no trial cache any more (no trial is taken from an " - "earlier run's); remove the key, from the benchmark file too: a resumed `hermit train` " - "runs its unfinished step's trials again, and an eval run again runs all of its own") - self.infra_retries # and an infra retry count or wait that is no number - self.infra_wait_s - self.docker_cleanup # an unknown mode fails here, before any container - self.verifier_timeout # and an unknown verifier_timeout - self.acting_agent # and an agent spec without a name - if self.cli == "answer" and isinstance(given.get("model"), dict) and given["model"].get("cli"): - # the run config's own model a CLI reaches, as hermit check hands it the config's params (hermit eval and - # train hand it the endpoint the pipeline serves that model from: check_agent, PIPELINE_CLI_SPEC) - raise ValueError(CONFIG_CLI_MODEL) - for key in TIMEOUT_MULTIPLIERS: # and a multiplier below 1 - self.timeout_multiplier(key) - self.pass_reward # and a pass reward or a redaction that cannot be read - self.redactions - harbor = self.harbor # and a harbor that is not there: no rollout could start it - if not (os.path.isabs(harbor) and os.path.isfile(harbor) and os.access(harbor, os.X_OK)): - raise ValueError(f"agent.params.harbor {harbor} is not an executable file" - f"{'' if os.path.isabs(harbor) else ', nor a command on PATH'}: the `harbor` command of " - f"Harbor's own venv runs every trial (default {HARBOR})") - python = self.harbor_python # and a python no stream's driver could start under - if self.multi_step() and not (os.path.isfile(python) and os.access(python, os.X_OK)): - raise ValueError(f"{python} is not an executable file: a multi-step task's driver runs under the python of " - f"Harbor's venv, beside agent.params.harbor ({harbor}) or beside the file a link of that " - "name points to; give the venv's own `harbor`, or a link to it, not a wrapper elsewhere") - self.pulls = docker.Pulls() # the room trials hold for images still to be pulled - sizes = json.loads(self._path(self.params["image_sizes"]).read_text()) if self.params.get("image_sizes") else {} - self.image_sizes = {k: v for k, v in sizes.items() if not str(k).startswith("_")} - self._streams: dict[str, object] = {} # a stream's tool server, for the stream's life - self._drivers: dict[str, StreamClient] = {} # the drivers of the streams it stepped, until they finish - self._in_flight: dict[str, list[int]] = {} # a stream's steps whose rollouts are in _run_step now - self._given_up: dict[str, str] = {} # a stream whose next turn cannot come: why - self._streams_lock = threading.Lock() - self._stop = threading.Event() # set once a rollout met a lost sign-in (end_run) - self._stop_why = "" # ... and why - self.run_id = uuid.uuid4().hex[:8] # this runner's own: its streams' names (_run_step) - # its streams closed when it is freed, or at exit (_close_runner_streams) - self._finalizer = weakref.finalize(self, _close_runner_streams, self._drivers, self._streams, - self._streams_lock) - self.check_agent() - - # --- params --------------------------------------------------------------------------------------------- - @staticmethod - def check_shared_model(model, where) -> None: - """A model the benchmark file gives (the run config giving none) is a name (the CLI's own, with a login; one of - Harbor's own agents') or a model a CLI reaches on its login (`{cli: ..., login: ...}`, which the runner serves - itself, as written). A `/` reference or another model mapping is refused: it belongs in the run - config's own agent.params.model, which the config's loading resolves and hermit check and hermit train check - (its backend, its key), and which they never read in the benchmark file; and so is a `{cli: ...}` model - without a login, which only a config's loading would bind, as it binds `llm`'s (its reference's key, endpoint - and hosts).""" - if isinstance(model, dict) and model.get("cli"): - if not model.get("login"): - # served as it is written (tools.ModelEndpoint): no config's loading binds a reference here, as it - # binds `llm`'s, so a session would run with no key, no endpoint and no host it may reach - raise ValueError(f"agent.params.benchmark {where} gives model {{cli: {model['cli']}}} with no login: " - "the runner serves the benchmark file's model as written, on the CLI's own sign-in " - "(login: chatgpt for codex, claude for claude), and binds no / " - "reference there (Pi, which has no login, cannot be such a model); give its login, " - "or a / reference in the run config's own agent.params.model") - return - if model is None: - return - if isinstance(model, str) and launch.binding_of({"model": model}) is None: - return - raise ValueError(f"agent.params.benchmark {where} gives model {model!r}, a / reference or a " - "model mapping: give it in the run config's own agent.params.model (where the config's " - "loading binds it and hermit check sees its backend and key); the benchmark file names a " - "model by name, or one a CLI reaches (model: {cli: ...})") - - @property - def infra_retries(self) -> int: - """How many times a trial that ended infra runs again (`infra_retries`, INFRA_RETRIES).""" - value = self.params.get("infra_retries") - try: - value = INFRA_RETRIES if value is None else int(value) - except (TypeError, ValueError): - value = -1 - if value < 0: - raise ValueError(f"agent.params.infra_retries is a count, at least 0, not {self.params.get('infra_retries')!r}") - return value - - @property - def infra_wait_s(self) -> float: - """How long before each run again (`infra_wait_s`, INFRA_WAIT_S).""" - value = self.params.get("infra_wait_s") - try: - value = INFRA_WAIT_S if value is None else float(value) - except (TypeError, ValueError): - value = -1.0 - if not value >= 0: - raise ValueError(f"agent.params.infra_wait_s is a number of seconds, at least 0, not " - f"{self.params.get('infra_wait_s')!r}") - return value - - def _path(self, value) -> Path: - p = Path(os.path.expandvars(str(value))).expanduser() - return p if p.is_absolute() else (self.benchmark_base / p).resolve() - - @property - def harbor(self) -> str: - """The `harbor` command, resolved once, so that the startup check and every trial's Popen (in the job's - directory) start the same one: a name without a slash found on PATH, a path made absolute (`~` expanded, - relative to the working directory); a name not on PATH stays as given (the startup check refuses it).""" - if getattr(self, "_harbor", None) is None: - value = os.path.expanduser(str(self.params.get("harbor") or HARBOR)) - if "/" not in value: - value = shutil.which(value) or value - self._harbor = os.path.abspath(value) if "/" in value else value - return self._harbor - - @property - def harbor_python(self) -> str: - """The interpreter of Harbor's venv (a stream's driver runs under it): the one beside `harbor`, or beside the - file a link to it names (uv tool's and pipx's ~/.local/bin/harbor are links into the tool's venv).""" - return str(Path(os.path.realpath(self.harbor)).parent / "python") - - @property - def tasks_dir(self) -> Path: - """The task set's directory, its links followed once (`utils/tasks`, the link `python -m - hermit.harbor.dataset prepare` writes): a prepare that points the link elsewhere mid-run changes no trial's - task set, nor the key its trial is kept under.""" - if getattr(self, "_tasks_dir", None) is None: - self._tasks_dir = Path(os.path.realpath(self._path(self.params["tasks_dir"]))) - return self._tasks_dir - - def multi_step(self) -> bool: - """Whether the task set holds a multi-step task (its steps run as a stream, whose driver runs under - harbor_python); a task whose task.toml cannot be read counts as none here (its own rollout says why).""" - for task_dir in dataset.task_dirs(self.tasks_dir): - try: - if dataset.step_names(dataset.task_config(task_dir)): - return True - except (OSError, ValueError, KeyError, TypeError): - continue - return False - - def unsupported(self, task: dict) -> list[str]: - """Why Harbor runs the task for none of Hermit's own agents (a table CLI, the answer agent), or nothing: Harbor - refuses a task that declares MCP servers (`[[environment.mcp_servers]]`) or a skills folder - (`environment.skills_dir`), or that targets a Windows container (`environment.os`), for an agent whose - capabilities lack them (harbor/trial/trial.py, _validate_agent_capabilities), as harborside's agents' do, and - its refusal leaves no stream: such a rollout is infra (`unsupported_task`), never started (run). A task.toml - that cannot be read says nothing here (its trial says why).""" - env = self.task_config(task).get("environment") or {} - if not isinstance(env, dict): - return [] - return [w for w, on in (("MCP servers", env.get("mcp_servers")), ("a skills folder", env.get("skills_dir")), - ("a Windows container", str(env.get("os") or "").lower() == "windows")) if on] - - @property - def jobs_dir(self) -> Path: - return Path(os.path.expanduser(str(self.params.get("jobs_dir") or JOBS_DIR))).resolve() - - @property - def tmpdir(self) -> Path: - value = self.params.get("tmpdir") - return Path(os.path.expanduser(str(value))).resolve() if value else self.jobs_dir / "tmp" - - @property - def python(self) -> str: - return str(self.params.get("python") or sys.executable) - - @property - def acting_agent(self) -> dict: - spec = dict(self.params.get("acting_agent") or {}) - clis.cli_of(spec) # names one agent, and a known CLI - if not spec.get("name") and not spec.get("import_path") and not spec.get("cli"): - spec["cli"] = clis.DEFAULT_CLI - return spec - - @property - def cli(self) -> str | None: - """The table's CLI the acting agent is (hermit.harbor.clis), or None for a Harbor built-in or another class.""" - return clis.cli_of(self.acting_agent) - - @property - def supports(self) -> tuple[str, ...]: - return supports_of_cli(self.cli) - - @property - def deploys_adapter(self) -> bool: - return self.cli is not None - - @property - def host(self) -> str | None: - """The host the adapter is deployed for: the acting agent's CLI.""" - return self.cli - - @property - def login(self) -> str | None: - return str(self.params["login"]) if self.params.get("login") else None - - @property - def login_root(self) -> Path: - value = self.params.get("login_root") - return Path(os.path.expanduser(str(value))).resolve() if value else subscription.ROOT - - @property - def binding(self): - """The model's binding (hermit.harbor.launch.binding_of), or None for a plain model name or a model a CLI - reaches; priced at `price` when the params give one (an explicit price, which bind_agent_params keeps beside - the backend's in the binding's spec), so that every HermitXXX trial's usage (one of Harbor's own agents keeps - Harbor's cost) and the answer agent's endpoint follow it.""" - if not hasattr(self, "_binding"): - self._binding = None if self.cli_model is not None else launch.binding_of(self.params) - if self._binding is not None and isinstance(self.params.get("price"), dict): - self._binding.price = dict(self.params["price"]) - return self._binding - - @property - def cli_model(self) -> dict | None: - """`model: {cli: codex | claude, model, login, ...}`: a model only a CLI reaches (the answer agent's, each - request one session of that CLI on this machine: hermit.agent.base.endpoint.cli_endpoint), or None.""" - model = self.params.get("model") - return dict(model) if isinstance(model, dict) and model.get("cli") else None - - def answer_model(self) -> dict: - """The answer agent's model as its endpoint serves it: a model a CLI reaches, or the binding's spec with the - client settings the model's spec gives beside it (`timeout`, `max_retries`, which the relay's client would - otherwise take as its own 600 s and retries).""" - if self.cli_model is not None: - return self.cli_model - if self.binding is None: - raise ValueError("the answer agent needs agent.params.model: a / reference, or a model a " - "CLI reaches on its login, in the benchmark file (model: {cli: codex | claude, model: " - "..., login: ...})") - spec, bound, model = self.binding.spec(), self.params.get("binding"), self.params.get("model") - # the spec the binding was made from (launch.binding_of): the bound one, else the model's own mapping - given = bound if isinstance(bound, dict) and bound.get("base_url") and bound.get("model") else model - if isinstance(given, dict): - spec.update({k: given[k] for k in ("timeout", "max_retries") if given.get(k) is not None}) - return spec - - def model_plan(self) -> tuple[Path, bool] | None: - """(the plan, whether its sessions may draw usage credits) of the login the answer agent's model runs on when - the runner serves that model itself (`model: {cli, login, plan_dir, usage_credits}`, the benchmark file's): - each request one session on the login, which the relay holds back while the plan holds sessions back - (tools.ModelEndpoint); None for any other agent or model, and for such a model the pipeline serves (the run - config's own: hermit.isolate.agent_pipeline), whose CLI and login the runner never learns.""" - model = self.cli_model if self.cli == "answer" else None - if not model or not model.get("login"): - return None - return (subscription.plan_dir(model.get("plan_dir"), str(model["login"])), - subscription.opted_in(model.get("usage_credits"))) - - def plan_login(self) -> str | None: - """The login whose plan the trial's sessions draw on: the acting CLI's (`login`), or the answer agent's model's - when the runner serves it (model_plan); None for none, or for a login the runner does not know (that of a - model the pipeline serves).""" - if self.cli == "answer": - return str(self.cli_model["login"]) if self.model_plan() is not None else None - return self.login - - def bin(self, cli: str | None = None) -> Path | None: - """`_bin`: a binary of this machine mounted for the acting CLI, or None.""" - cli = cli or self.cli - value = self.params.get(f"{cli}_bin") if cli in BINS else None - return Path(os.path.expanduser(str(value))).resolve() if value else None - - def bin_mount(self, cli: str | None = None) -> tuple[Path, str]: - """(the folder mounted for `_bin`, the binary's path in the container): Codex's standalone release's - folder (codex-package.json; its rg and code-mode host beside the binary), else the binary's own folder.""" - cli = cli or self.cli - binary = self.bin(cli) - root = binary.parent - if cli == "codex" and (binary.parent.parent / "codex-package.json").is_file(): - root = binary.parent.parent - return root, f"{clis.bin_mount(cli)}/{binary.relative_to(root).as_posix()}" - - def version(self, cli: str | None = None) -> str | None: - """`_version`, else the table's default: what an install in the container installs.""" - cli = cli or self.cli - value = self.params.get(f"{cli.replace('-', '_')}_version") if cli else None - return str(value) if value else (clis.entry(cli)["version"] if cli else None) - - @property - def agent_timeout_sec(self) -> int | None: - value = self.params.get("agent_timeout_sec") - return int(value) if value else None - - @property - def setup_timeout_sec(self) -> float | None: - value = self.params.get("setup_timeout_sec") - return float(value) if value else None - - @property - def docker_cleanup(self) -> str: - mode = str(self.params.get("docker_cleanup") or "images_keep") - if mode not in docker.CLEANUP_MODES: - raise ValueError(f"agent.params.docker_cleanup must be one of {', '.join(docker.CLEANUP_MODES)}, not " - f"{mode!r}") - return mode - - @property - def verifier_timeout(self) -> str: - """What a verifier that ran out of its time is: `agent` (the default) or `infra` (verifier_timeout_verdict).""" - mode = str(self.params.get("verifier_timeout") or "agent") - if mode not in ("agent", "infra"): - raise ValueError(f"agent.params.verifier_timeout is agent or infra (a verifier that ran out of its time: " - f"the agent's result, or run again), not {mode!r}") - return mode - - @property - def reward_key(self) -> str: - return str(self.params.get("reward_key") or "reward") - - @property - def pass_reward(self) -> float | None: - value = self.params.get("pass_reward", 1.0) - if value is None: - return None - try: - return float(value) - except (TypeError, ValueError): - raise ValueError(f"agent.params.pass_reward is a number or null (no reward is a pass), not {value!r}") \ - from None - - @property - def redactions(self) -> list[str]: - patterns = self.params.get("verifier_redact") or [] - if isinstance(patterns, str): - patterns = [patterns] - for pattern in patterns: - try: - re.compile(str(pattern)) - except re.error as exc: - raise ValueError(f"agent.params.verifier_redact: {pattern!r} is no regular expression " - f"({exc})") from None - return [str(p) for p in patterns] - - def model_fields(self) -> dict: - """A Harbor built-in's model fields from `model`: a name, or a bound backend's {model, base_url, - api_key_env}. Harbor's claude-code is given the backend's Messages API root (Claude Code appends /v1/messages - to it, as Hermit's own sessions are given it: Binding.claude_env), and a model at Anthropic's own API by name - alone (`anthropic/`), as Harbor signs in to it for a model given by name: its key, ANTHROPIC_API_KEY.""" - model = self.params.get("model") - if not isinstance(model, dict): - return {"model_name": str(model)} if model else {} - base = model.get("base_url") - if base and self.acting_agent.get("name") == "claude-code": - base = str(model.get("messages_url") or re.sub(r"/v1/?$", "", str(base))).rstrip("/") - from urllib.parse import urlsplit - if urlsplit(base).hostname == "api.anthropic.com": # with its key's variable, which Harbor reads - named = {"model_name": f"anthropic/{model.get('model')}", - "model_api_key_name": model.get("api_key_env")} - return {k: v for k, v in named.items() if v} - out = {"model_name": model.get("model"), "base_url": base, "model_api_key_name": model.get("api_key_env")} - return {k: v for k, v in out.items() if v} - - def task_dir(self, task: dict) -> Path: - return self.tasks_dir / str(task.get("harbor_task") or task.get("task") or task["id"]) - - def instruction(self, task: dict) -> str: - """The task's (or the step's) instruction, as Harbor gives it to the agent: the row's (dataset.index), else its - instruction.md (a step's under steps//).""" - if task.get("instruction"): - return str(task["instruction"]) - rel = Path("steps") / str(task["step"]) / "instruction.md" if task.get("step") else Path("instruction.md") - return _text(self.task_dir(task) / rel).strip() - - def task_config(self, task: dict) -> dict: - try: - return dataset.task_config(self.task_dir(task)) - except (OSError, ValueError): - return {} - - def restricted(self, task: dict) -> bool: - """Whether the task's (or the step's) agent phase has no public network (task.toml): its model's and tools' - hosts are named to Harbor, and the CLI's web search is off (Harbor's own rule).""" - return dataset.network_modes(self.task_config(task), task.get("step"))["agent"] != "public" - - def wait_budget(self, task: dict, *, build: bool = True) -> float: - """How long a trial may take: setup (Harbor's limit, `setup_timeout_sec` when longer), the agent's budget, the - verifier's (verifier.override_timeout_sec, else the task's, times verifier_timeout_multiplier), the image - build (a Dockerfile task; with environment_build_timeout_multiplier, its scaled limit, and for a pulled image - the start's added room), teardown; or the explicit `wait_budget_s`.""" - if self.params.get("wait_budget_s"): - return float(self.params["wait_budget_s"]) - agent = self.agent_timeout_sec or int(task.get("agent_timeout_sec") or DEFAULT_AGENT_TIMEOUT_S) - verifier = int(self.verifier_config().get("override_timeout_sec") or task.get("verifier_timeout_sec") or agent) - verifier = int(verifier * self.timeout_multiplier("verifier_timeout_multiplier")) - start = self.timeout_multiplier("environment_build_timeout_multiplier") - build_s = int(task.get("build_timeout_sec") or 0) - built = int(build_s * (start if build and not task.get("docker_image") else start - 1)) - setup = max(int(self.params.get("setup_budget_s") or SETUP_BUDGET_S), int(self.setup_timeout_sec or 0)) - return agent + setup + verifier + built + int(self.params.get("teardown_budget_s") or TEARDOWN_BUDGET_S) - - def _turn_wait(self, task: dict, index: int, settings: dict) -> float: - """How long a stream's step waits for its turn: `stream.turn_wait_s`, else the budgets of the steps before it, - each its own (a step may set its own agent and verifier timeouts: dataset.rows_of), or the first step's own, - and the driver's start (`stream.start_timeout_s`).""" - if settings.get("turn_wait_s"): - return float(settings["turn_wait_s"]) - start = float(settings["start_timeout_s"]) - try: - earlier = [r for r in dataset.rows_of(self.task_dir(task)) if 0 < int(r.get("step_index") or 0) < index] - except (OSError, ValueError, KeyError, TypeError, StopIteration): # read at startup: the step's own then - earlier = [] - if len(earlier) != index - 1: # its steps not as at startup: this step's budget for each - return max(1, index - 1) * self.wait_budget(task, build=False) + start - own = sum(self.wait_budget(r, build=False) for r in earlier) if earlier else self.wait_budget(task, build=False) - return own + start - - def timeout_multiplier(self, key: str) -> float: - """Harbor's job-level multiplier `key` (TIMEOUT_MULTIPLIERS), 1 when unset; never below 1.""" - value = float(self.params.get(key) or 1.0) - if value < 1.0: - raise ValueError(f"{key} must be at least 1, not {value}") - return value - - # --- the acting CLI: its settings, checked at startup --------------------------------------------------------- - def settings(self, task: dict | None = None) -> dict: - """The CLI Base Agent's settings for a trial (hermit.harbor.launch.settings_for); web search off for a task - whose agent phase has no public network.""" - web = bool(self.params.get("web_search")) - if task is not None and self.restricted(task): - web = False - extra = dict(self.acting_agent.get("settings") or {}) - if extra.get("config"): # mini-swe-agent's config file: a relative one as the params' paths - extra["config"] = str(self._path(extra["config"])) - return launch.settings_for(self.cli, model=self.params.get("model"), binding=self.binding, - effort=self.params.get("effort"), web_search=web, login=self.login, extra=extra) - - def check_agent(self) -> None: - """What every trial of the acting agent would refuse, refused before the first one: for a Harbor built-in its - kwargs from the run's settings (no login, an effort it takes: builtin_kwargs) and, for its claude-code, a - sign-in it is handed (check_builtin_sign_in); for a table CLI its sign-in (a binding with its key set, or its - CLI's own login: the Codex sign-in, Claude's token, a mounted claude_bin's version), web search set, a model - this machine's containers can reach, no Harbor retry on login: chatgpt, the settings its Base Agent renders (a - dry render of the launch plan), and for mini-swe-agent the package its adapter's deployment reads on this - machine; bubblewrap when the adapter sandbox that renders each adapter (`sandbox_enforce`) is required; for the - answer agent its model's key, no attachments on a model a CLI reaches that the runner serves (`model: - {cli: ...}` in the benchmark file), whose sessions take text only, and no model a CLI reaches that the pipeline - serves (the run config's own `model: {cli: ...}`: as the pipeline's endpoint spec, PIPELINE_CLI_SPEC, and as - written, which __init__ refuses, as hermit check hands it the config's params). The runner never learns - whether the run deploys an adapter, so a run that deploys none (hermit eval --adapter none) is checked for - bubblewrap and - mini-swe-agent's package too. A task Harbor runs for none of Hermit's agents is not refused here, as the run - may never take it: its rollout is infra (unsupported). For every agent, no secret in acting_agent.env - (check_env).""" - if self.cli is None: - self.builtin_kwargs() - self.check_builtin_sign_in() # before check_env: its CLAUDE_FORCE_OAUTH says what to do - self.check_env() - return - self.check_env() - cli, login, binding = self.cli, self.login, self.binding - model = self.params.get("model") - if not isinstance(self.acting_agent.get("settings") or {}, dict): - raise ValueError("agent.params.acting_agent.settings is a mapping of the CLI Base Agent's settings") - if "login" in (self.acting_agent.get("settings") or {}): - # the session would sign in with it (launch.render_plan) where the runner, which reads agent.params.login - # alone, neither checks it nor hands the trial its sign-in, its plan and its refusals - raise ValueError("agent.params.acting_agent.settings.login: a trial's login is agent.params.login (its " - "sign-in, its plan, the refusals of a login its CLI cannot take); give it there") - if not shutil.which("bwrap"): - from hermit.adapter.session import ENV_ENFORCE - if (self.params.get("sandbox_enforce") or os.environ.get(ENV_ENFORCE, "required")) == "required": - # staging renders every adapter (its exports, its per-task context) in it: each trial would fail - raise ValueError("a trial deploys its adapter by rendering it in the strict adapter sandbox, whose " - "enforce is required (agent.params.sandbox_enforce), and bubblewrap (bwrap) is not " - "installed: install it, or set agent.params.sandbox_enforce: best_effort (unconfined, " - "recorded), which a run that deploys no adapter (hermit eval --adapter none) needs " - "too: the runner cannot tell it from one that does") - elif (fault := sandbox_fault(self.params.get("sandbox_enforce"), fresh=False)) is not None: - raise ValueError(f"a trial deploys its adapter by rendering it in the strict adapter sandbox, and {fault}: " - "every trial's staging would fail; let bubblewrap make its namespaces (user namespaces " - "on), or set agent.params.sandbox_enforce: best_effort (unconfined, recorded)") - if cli == "answer": - if login: - raise ValueError("the answer agent takes no login: its model is a / reference, or " - "one a CLI reaches (model: {cli: ..., login: ...})") - if isinstance(model, dict) and set(model) == PIPELINE_CLI_SPEC and model.get("api_key_env") is None \ - and model.get("key_required") is False: - # its session waits for its login's plan inside the request (CliAgent.run), where the runner cannot - # tell that wait from the model's own time: a trial whose time ran out as it waited would be the - # agent's timeout, scored 0, though it is the plan's - raise ValueError(CONFIG_CLI_MODEL) - spec = self.answer_model() - reached = (self.cli_model or {}).get("cli") - if reached and (self.params.get("answer") or {}).get("attachments"): - raise ValueError(f"agent.params.answer.attachments sends the task's images with the question, and the " - f"answer agent's model is one a CLI reaches (model: {{cli: {reached}}}), whose " - "session takes text only: give a / reference whose model takes " - "images, or no attachments") - if self.cli_model is not None: - from hermit.agent.base.endpoint import cli_problems - problems = cli_problems(spec) - enforce = str(self.params.get("sandbox_enforce") or "required") - if (spec.get("isolation") or "sandbox") == "sandbox" and enforce == "required" and \ - not shutil.which("bwrap"): - problems.append(f"model: cli: {spec['cli']} runs each session in a bubblewrap sandbox of its own " - "(isolation: sandbox), and bubblewrap (bwrap) is not installed") - if spec.get("login") and not problems: - # its sign-in, as the acting CLI's is checked (a session on this machine: its CLI too, a session - # in a container of its own the image's), so that a run with none ends at startup, not with - # every trial unserved, run again and then a failed run - problems = subscription.problems_of(str(spec["login"]), program=spec.get("bin"), - contained=spec.get("isolation") == "container", - root=spec.get("login_root")) - if problems: - raise ValueError("; ".join(problems)) - elif binding.key_env_if_set() and not os.environ.get(binding.key_env_if_set()): - raise ValueError(f"{binding.key_env_if_set()} is not set: the answer agent's model endpoint on this " - "machine sends each request on with the key it reads from this process's environment") - self._dry_render(None, tools=False) - return - if self.cli_model is not None: - raise ValueError(f"model: {{cli: ...}} is a model the answer agent reaches through a CLI; a {cli} trial's " - "model is its own: a / reference, or the CLI's model name with its login") - if login is not None and login not in subscription.LOGINS: - raise ValueError(f"agent.params.login is {' or '.join(sorted(subscription.LOGINS))} or absent, not " - f"{login!r}") - own = clis.entry(cli)["backend"] - if login and subscription.LOGINS[login] != own: - raise ValueError(f"login: {login} is {'Codex' if login == 'chatgpt' else 'Claude Code'}'s sign-in; the " - f"acting CLI {cli} " + ("takes no login: its model is a / reference" - if own in subscription.NO_LOGIN else - f"takes login: {'chatgpt' if own == 'codex' else 'claude'}")) - if login and binding is not None: - raise ValueError(f"login: {login} and a / reference are two bills: give one") - if not login and binding is None: - raise ValueError(f"agent.params.model {model!r} is no / reference: a {cli} trial needs one" - + (f", or login: {'chatgpt' if own == 'codex' else 'claude'} with the CLI's model name" - if own in ("codex", "claude") else " (its only sign-in)")) - if login and (not isinstance(model, str) or not model): - raise ValueError(f"with login: {login}, agent.params.model is the CLI's model name") - if not isinstance(self.params.get("web_search"), bool): - raise ValueError("agent.params.web_search must be set, true or false: the CLI's web search (Codex's runs " - "at the provider, past any sandbox or proxy) is never left to a default") - if binding is not None: - from urllib.parse import urlsplit - url = binding.base_url if own != "claude" else binding.backend.messages_root - if (urlsplit(url).hostname or "") in ("127.0.0.1", "localhost", "::1", "0.0.0.0"): - raise ValueError(f"the model's endpoint {url} is on this machine's loopback, which a task container " - f"cannot reach: serve it on an address the container reaches (_BASE_URL); a " - f"model a CLI reaches (model: {{cli: ...}}, which a run config's loading serves on " - f"the loopback) is the answer agent's alone") - if login: - # a mounted claude_bin is this machine's Claude Code, checked here as that CLI (problems_of runs it for its - # version); without one a Claude session runs the container's install (contained: only the token is checked) - mounted = self.bin("claude") if login == "claude" else None - problems = subscription.problems_of(login, program=str(mounted) if mounted else None, - contained=mounted is None, root=self.login_root) - if problems: - raise ValueError("; ".join(problems)) - retry = self.params.get("retry") or {} - if login == "chatgpt" and isinstance(retry, dict) and int(retry.get("max_retries") or 0) > 0: - raise ValueError("agent.params.retry.max_retries > 0 with login: chatgpt: each of Harbor's attempts " - "of a trial would start from the sign-in copied before the first, which Codex may " - "have refreshed (and so spent) in an earlier attempt; set retry.max_retries: 0 and " - "let infra_retries run a trial that is not the agent's result again, each run with " - "the shared copy as it is then") - if own == "dsh": - from hermit.agent.base.cli import dsh_problems - problems = dsh_problems(self.settings(), "agent") - if problems: - raise ValueError("; ".join(problems)) - binary = self.bin(cli) - if binary is not None and not (binary.is_file() and os.access(binary, os.X_OK)): - raise ValueError(f"agent.params.{cli}_bin {binary} is not an executable file") - if cli == "mini-swe-agent" and not self._finds("minisweagent"): - from hermit.agent.base.mini_swe_agent import INSTALL - raise ValueError(f"{INSTALL}, for {self.python} (agent.params.python, else this interpreter): it deploys " - "the adapter of each mini-swe-agent trial (hermit.hosts.mini_swe_agent), which reads the " - "package's own config") - self._dry_render(None, tools=False, check=True) - - def _finds(self, package: str) -> bool: - """Whether the interpreter the adapter's deployment runs with (`python`, staging.stage) finds `package` as a - package, as hermit.agent.base.mini_swe_agent.render_config looks for it: in this process when that interpreter - is this one, else in a short child of it.""" - if self.python == sys.executable: - from importlib.util import find_spec - spec = find_spec(package) - return spec is not None and bool(spec.submodule_search_locations) - from hermit.sandbox import framework_env - probe = (f"import importlib.util, sys; s = importlib.util.find_spec({package!r}); " - "sys.exit(0 if s is not None and s.submodule_search_locations else 1)") - try: - return subprocess.run([self.python, "-c", probe], env=framework_env(), capture_output=True, - timeout=60).returncode == 0 - except (OSError, subprocess.SubprocessError): - return False - - def _dry_render(self, task: dict | None, *, tools: bool, check: bool = False) -> None: - """The launch plan a trial of `task` renders, without its deployment, sign-in or tools' server, rendered in a - scratch job that is removed after: the settings the CLI's Base Agent refuses are refused here, at startup. - With `check`, what harborside would refuse in every trial is refused here too (check_secrets).""" - scratch = self.jobs_dir / ".renders" / uuid.uuid4().hex[:12] - scratch.mkdir(parents=True) - try: - if self.cli == "answer": - plan = launch.render_answer(self.settings(task), endpoint=ANSWER_PLACEHOLDER, job=scratch, - options=self.params.get("answer") or {}) - else: - plan = launch.render(self.cli, self.settings(task), deploy=None, sign_in=False, job=scratch, - tools_url=TOOLS_PLACEHOLDER if tools else None, login=self.login, - tools_name="adapter" if tools else None, **self._mounted()) - if check: - self.check_secrets(json.loads(plan.read_text(encoding="utf-8"))) - finally: - shutil.rmtree(scratch, ignore_errors=True) - - def check_secrets(self, plan: dict) -> None: - """A plan's secrets reach the session by name (harborside: `docker compose exec -e NAME`, or a descriptor): - each must be set here, where harbor starts, and the task's environment must be Harbor's local Docker one.""" - names = sorted({*(plan.get("env_from_host") or {}).values(), *(plan.get("env_by_fd") or {}).keys()}) - if not names: - return - missing = [n for n in names if not os.environ.get(n)] - if missing: - raise ValueError(f"{', '.join(missing)} {'is' if len(missing) == 1 else 'are'} not set: the acting " - f"agent's session reads {'it' if len(missing) == 1 else 'them'} from this process's " - "environment, by name") - kind = (self.params.get("environment") or {}).get("type") - if kind not in (None, "docker"): - raise ValueError(f"agent.params.environment.type {kind!r}: the acting agent's secrets ({', '.join(names)}) " - "reach its session by name, which only Harbor's local docker environment does") - - def _mounted(self) -> dict: - """The launch's program and PATH for a mounted `_bin`.""" - if self.bin() is None: - return {} - program = self.bin_mount()[1] - return {"bin_path": program, "path_prepend": [str(Path(program).parent)]} - - # --- the job -------------------------------------------------------------------------------------------- - def agent_config(self, task: dict, plan: Path | None = None, deployed: dict | None = None, *, - tools_host: str | None = None) -> dict: - """Harbor's agent config for this rollout: a table CLI's class with its launch plan (and its own options: the - mounted binary, the install's version), or a Harbor built-in with its model fields, as Harbor runs it.""" - spec = self.acting_agent - cli = self.cli - kwargs = dict(spec.get("kwargs") or {}) - env = {str(k): str(v) for k, v in (spec.get("env") or {}).items()} - if cli is not None: - agent: dict = {"import_path": clis.entry(cli)["import_path"]} - if plan is not None: - kwargs["plan"] = str(plan) - if cli != "answer": - if self.version(cli): - kwargs.setdefault("version", self.version(cli)) - if self.bin(cli) is not None: - kwargs.setdefault(f"{cli}_bin", self.bin_mount(cli)[1]) - hosts = self.allowed_hosts(task, tools_host) - if hosts: - agent["extra_allowed_hosts"] = hosts - else: - if deployed is not None: - raise ValueError(f"the acting agent {spec.get('import_path') or spec.get('name')} takes no adapter; " - f"use acting_agent.cli ({', '.join(k for k in clis.HARBOR_CLIS)}), or run with " - "--adapter none") - agent = {("import_path" if spec.get("import_path") else "name"): spec.get("import_path") or spec["name"]} - agent.update(self.model_fields()) - if agent.get("base_url"): - agent["skip_model_parsing"] = True # a bound endpoint serves the id as given (accounts/...) - kwargs = {**self.builtin_kwargs(), **kwargs} - if self.agent_timeout_sec: - agent["override_timeout_sec"] = self.agent_timeout_sec - if self.setup_timeout_sec: - agent["override_setup_timeout_sec"] = self.setup_timeout_sec - agent["kwargs"] = kwargs - if env: - agent["env"] = env - return agent - - def check_env(self) -> None: - """`acting_agent.env` holds no secret: its values are written into the trial's job (job.yaml, agent.json) and - handed to each `docker compose exec` on its command line, which every user of this machine can read. A - variable named as a secret, or a value that names one, is refused, a secret's name being one Harbor names so - (HARBOR_SECRET, harbor.utils.env's: OPENAI_API_KEY, GH_TOKEN, HF_TOKEN_READ, GIT_AUTHOR_NAME too) or one of - launch.SECRET_VAR's (DB_PASSWD): a value that is `${NAME}` or `${NAME:-default}` alone, which Harbor fills in - from this process's environment, and any other reference (`$NAME`, or one inside a longer value), which it - hands on as that text (never the secret the config meant). A table CLI's secret is passed by name (the CLI's - settings.env, NAME: $NAME), and Harbor's own agents read theirs from this process's environment.""" - def secret(n: str) -> bool: - return bool(HARBOR_SECRET.search(n) or launch.SECRET_VAR.search(n)) - for name, value in (self.acting_agent.get("env") or {}).items(): - named = [a or b for a, b in ENV_REF.findall(str(value))] - found = str(name) if secret(str(name)) else next((n for n in named if secret(n)), None) - if found: - raise ValueError(f"agent.params.acting_agent.env {name} holds a secret ({found}, a name Harbor treats " - "as one): its values are written into each trial's job and onto `docker compose " - "exec`'s command line (Harbor fills a whole ${NAME} or ${NAME:-default} in, and hands " - "any other reference on as that text); pass a table CLI's by name in its " - "settings.env (NAME: $NAME), and export one of Harbor's own agents in this process's " - "environment") - - def builtin_kwargs(self) -> dict: - """A Harbor built-in's kwargs from the run's settings: Codex's and Claude Code's effort and version, Claude - Code's web search off on every task with `web_search: false` (true, or unset: Harbor's own rule, off where the - task's agent network is not public, which any `disable_web_search` Harbor is given would override) and the - tools that outlive the session denied. Its sign-in is Harbor's own, never a login: Harbor's Claude Code would - put the token on a command line.""" - name = self.acting_agent.get("name") - out: dict = {} - if name not in ("codex", "claude-code"): - if self.login: - raise ValueError(f"login: {self.login} runs with acting_agent.cli, whose launch passes the sign-in " - f"safely; {name or self.acting_agent.get('import_path')} would put it on a command " - "line") - return out - if self.login: - raise ValueError(f"login: {self.login} runs with acting_agent.cli: " - f"{'codex' if name == 'codex' else 'claude'}, which passes the sign-in safely; Harbor's " - f"own {name} would put it on a command line") - if self.params.get("effort"): - if name == "claude-code": - from hermit.agent.base import cli as cli_mod - if str(self.params["effort"]) not in cli_mod.Claude.efforts: - raise ValueError(f"Claude Code takes effort {', '.join(cli_mod.Claude.efforts)}, not " - f"{self.params['effort']!r}") - out["reasoning_effort"] = str(self.params["effort"]) - version = self.params.get("codex_version" if name == "codex" else "claude_version") - if version: - out["version"] = str(version) - if name == "claude-code": - from hermit.agent.base import cli as cli_mod - web = self.params.get("web_search") - if web is not None and not isinstance(web, bool): - raise ValueError(f"agent.params.web_search is true or false, not {web!r}") - if web is False: - out["disable_web_search"] = True - out["disallowed_tools"] = ",".join(cli_mod.CLAUDE_BEYOND_SESSION) - return out - - def check_builtin_sign_in(self) -> None: - """Harbor's own claude-code bills the key this process gives it (harbor_env), never the subscription, whose - token it would put on a command line: refused at startup, CLAUDE_FORCE_OAUTH (Harbor's switch to the - subscription: each trial would bill the key instead, unsaid), and the subscription's token as the only - sign-in of a model at Anthropic's API (each trial would run signed out and halt the run). CLAUDE_FORCE_OAUTH, - and a bill beside the token (Claude Code's billing variables, the model's own key variable), count in this - process's environment and in `acting_agent.env`, which Harbor's agent reads too.""" - if self.acting_agent.get("name") != "claude-code": - return - from hermit.agent.base import cli as cli_mod - route = ("run acting_agent.cli: claude with login: claude, which hands Claude Code the subscription's token " - "safely") - given = {**os.environ, **{str(k): str(v) for k, v in (self.acting_agent.get("env") or {}).items()}} - if (given.get("CLAUDE_FORCE_OAUTH") or "").strip().lower() not in ("", "false", "0", "no"): - raise ValueError("CLAUDE_FORCE_OAUTH asks Harbor's own claude-code for the subscription's sign-in, whose " - "token Hermit never hands it (it would put it on a command line), so each trial would " - f"bill ANTHROPIC_API_KEY instead: {route}, or unset CLAUDE_FORCE_OAUTH to bill the key") - model = self.model_fields() - at_anthropic = not model.get("base_url") and "/" not in str(model.get("model_name") or "").removeprefix( - "anthropic/") - bills = (*cli_mod.CLAUDE_BILLING, *([model["model_api_key_name"]] if model.get("model_api_key_name") else [])) - if (os.environ.get(subscription.TOKEN_ENV) or "").strip() and at_anthropic and \ - not any(given.get(n) for n in bills): - raise ValueError(f"{subscription.TOKEN_ENV} is the only sign-in here, and Harbor's own claude-code is " - "never handed it (it would put it on a command line), so each trial would run signed " - f"out: {route}, or export the key it is to bill (ANTHROPIC_API_KEY)") - - def allowed_hosts(self, task: dict, tools_host: str | None = None) -> list[str]: - """The hosts a task whose agent phase has no public network must let the CLI reach (Harbor's - extra_allowed_hosts, which then replaces Harbor's own pick of the model's host): the model's (the binding's - endpoint, a login's hosts) and the address of the adapter's tools. None for a public one.""" - if not self.restricted(task) or self.cli in (None, "answer"): - return [] - from urllib.parse import urlsplit - hosts = [] - if self.binding is not None: - for url in (self.binding.base_url, self.binding.backend.messages_root): - host = urlsplit(url).hostname - if host and host not in hosts: - hosts.append(host) - hosts += [h for h in subscription.hosts(self.login) if h not in hosts] - if tools_host and tools_host not in hosts: - hosts.append(tools_host) - return hosts - - def environment_config(self) -> dict: - env = {"delete": True, **(self.params.get("environment") or {})} - if env.get("extra_docker_compose"): # Harbor's compose overlays, found where the benchmark - env["extra_docker_compose"] = [str(self._path(p)) for p in env["extra_docker_compose"]] # file is - cli = self.cli if self.cli in BINS else None - if cli is not None and self.bin(cli) is not None: - if not os.access(self.bin(cli), os.X_OK): - raise ValueError(f"agent.params.{cli}_bin {self.bin(cli)} is not an executable file") - mounts = list(env.get("mounts") or []) - mounts.append({"type": "bind", "source": str(self.bin_mount(cli)[0]), "target": clis.bin_mount(cli), - "read_only": True}) - env["mounts"] = mounts - return {k: v for k, v in env.items() if v is not None} - - def verifier_config(self) -> dict: - return {k: v for k, v in (self.params.get("verifier") or {}).items() if v is not None} - - def job_config(self, task: dict, job_name: str, agent: dict) -> dict: - """Harbor's job YAML for this one trial (a config file: `--agent` on the CLI discards the agents block).""" - config = {"job_name": job_name, "jobs_dir": str(self.jobs_dir), "n_attempts": 1, "n_concurrent_trials": 1, - "tasks": [{"path": str(self.task_dir(task))}], "agents": [agent], - "environment": self.environment_config()} - if self.verifier_config(): - config["verifier"] = self.verifier_config() - for key in TIMEOUT_MULTIPLIERS: - if self.params.get(key) is not None: - config[key] = self.timeout_multiplier(key) - if self.params.get("retry"): - config["retry"] = dict(self.params["retry"]) - return config - - def harbor_env(self, keep=()) -> dict: - """The environment of Harbor's processes: the harborside package (and the example's agents) importable, - a TMPDIR outside /tmp, the variables a plan passes by name (`keep`) among this process's. For HermitClaude - nothing of another bill or of a parent Claude Code session; for Harbor's own Claude Code, which signs in with - this process's key (ANTHROPIC_API_KEY, ...), nothing of a parent Claude Code session or another model's - settings; the subscription's token only on its login; and never the switches that have Harbor's own codex sign - in with the user's ChatGPT file (CODEX_SIGN_IN), so that a ChatGPT plan is used only on `login: chatgpt`, in - the plan's slots and under its MAX_USED.""" - self.tmpdir.mkdir(parents=True, exist_ok=True) - paths = [str(HARBORSIDE_ROOT), *(str(self._path(p)) for p in (self.acting_agent.get("pythonpath") or []))] - if os.environ.get("PYTHONPATH"): - paths.append(os.environ["PYTHONPATH"]) - env = {**os.environ, "PYTHONPATH": os.pathsep.join(paths), "TMPDIR": str(self.tmpdir)} - for name in CODEX_SIGN_IN: - if name not in keep: - env.pop(name, None) - if self.cli == "claude" or self.acting_agent.get("name") == "claude-code": - from hermit.agent.base import cli - drop = (*cli.CLAUDE_BILLING, *cli.CLAUDE_INHERITED, "CLAUDE_FORCE_OAUTH") - if self.cli != "claude": # Harbor's own: its bill is the one it is given here - drop = tuple(n for n in drop if n not in cli.CLAUDE_BILLING) - for name in drop: - if name not in keep: - env.pop(name, None) - if self.login != "claude": # the subscription's token: only a Claude login's trial - env.pop(subscription.TOKEN_ENV, None) - elif not env.get(subscription.TOKEN_ENV): - raise ValueError(subscription.NO_TOKEN) - return env - - def harbor_run(self, job: Path, config: Path, task: dict, keep=()) -> tuple[int | None, bool]: - """`harbor run` on the job config in its own process group; (exit code, expired). On expiry the group is - killed and the trial's compose projects taken down, best effort: Harbor would otherwise leave the containers - running after its client died.""" - with open(job / "harbor.log", "w") as log: - proc = subprocess.Popen([self.harbor, "run", "-c", str(config), "-y", "-q"], env=self.harbor_env(keep), - stdout=log, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL, - start_new_session=True, cwd=str(job)) - try: - return proc.wait(timeout=self.wait_budget(task)), False - except subprocess.TimeoutExpired: - pass - for sig, grace in ((signal.SIGTERM, 30), (signal.SIGKILL, 10)): - try: - os.killpg(os.getpgid(proc.pid), sig) - except ProcessLookupError: - break - try: - proc.wait(timeout=grace) - break - except subprocess.TimeoutExpired: - continue - for trial in self.trial_dirs(job, task): - docker.compose_down(trial.name) - return proc.returncode, True - - @staticmethod - def trial_dirs(job: Path, task: dict) -> list[Path]: - """Harbor's trial directories of the job: `__<7-char id>` (one, for one trial), with the task's name - cut as Harbor cuts it (harbor/models/trial/config.py generate_trial_name: the last path part, its first 32 - characters, trailing '_' and '-' removed): a longer name is never found under its full spelling.""" - name = str(task.get("harbor_task") or task.get("task") or task["id"]).split("/")[-1][:32].rstrip("_-") - return sorted(p for p in job.glob(f"{glob.escape(name)}__*") if p.is_dir()) - - # --- the run's end and the disk gate --------------------------------------------------------------------- - def end_run(self, verdict: dict, job: Path): - """A halting verdict (a lost sign-in, an API account's usage limit: nothing later in the run can succeed) ends - the run: every rollout of this runner waiting for the plan, a slot, the disk or a run again stops, and none - starts (stopped), and this one raises LoginFailed (hermit.agent.base.interface), which hermit.trainer.rollout - passes up, as a lost sign-in of a Hermit session ends the run: `hermit train` resumes after its last finished - step (the trials of the step it ended in run again), `hermit eval` stops. The trial's job and curated log stay - under the jobs directory.""" - cause = verdict["failure_cause"] or verdict["exception_type"] - named = f"{cause} ({verdict['exception_type']})" if verdict["exception_type"] not in (None, cause) else cause - why = (f"a Harbor trial ({Path(job).name}): {named}: {str(verdict['exception_message'] or '')[:1500]}; nothing " - "later in the run can succeed: restore the sign-in or the account's limit, then resume") - with self._streams_lock: - if not self._stop.is_set(): - self._stop_why = why - self._stop.set() - print(f"WARNING: {why} (the trial's job: {job})", file=sys.stderr, flush=True) - raise LoginFailed(why) - - def stopped(self) -> None: - """Raise LoginFailed when another rollout of this runner ended the run (end_run).""" - if self._stop.is_set(): - raise LoginFailed(self._stop_why) - - def wait_for_disk(self, task: dict | None = None, admitted: list | None = None) -> bool: - """The disk gate; for a task whose prebuilt image is not here yet and whose size `image_sizes` knows (a JSON - file: compressed bytes per image reference), room for the image too, with the room other trials of this - process are still to pull counted as taken (`admitted` gets the key of the room this trial holds).""" - gate = self.params.get("disk_gate_gb") - gate = float(DISK_GATE_GB if gate is None else gate) - tmp_gate = self.params.get("tmp_gate_gb") - tmp_gate = float(TMP_GATE_GB if tmp_gate is None else tmp_gate) - wait_s = float(self.params.get("disk_gate_wait_s") or DISK_GATE_WAIT_S) - self.stopped() - if not docker.wait_for_disk(HOST_TMP, tmp_gate, wait_s, cancel=self._stop): # the host's /tmp, before any room - self.stopped() # is held - return False - image = docker.task_image(self.task_dir(task)) if task is not None else None - free = docker.wait_for_disk(self._docker_root(), gate, wait_s, image=image, - image_gb=self.image_gb(image) if image else 0.0, pulls=self.pulls, - admitted=admitted, cancel=self._stop) - self.stopped() # another rollout ended the run while this one waited - return free - - def _disk_held(self, task: dict | None = None) -> str: - """Which disk gate held a trial back once wait_for_disk gave up, for its verdict: the host's /tmp when it is - short of its own gate now (that gate is waited for first), else Docker's data root, with the room the task's - image takes when it is not here yet (`image_sizes`) and the room other trials hold for their pulls, which - wait_for_disk counts as taken.""" - tmp_gate = self.params.get("tmp_gate_gb") - tmp_gate = float(TMP_GATE_GB if tmp_gate is None else tmp_gate) - if tmp_gate > 0 and docker.free_gb(HOST_TMP) < tmp_gate: - return f"the host's {HOST_TMP} stayed under {tmp_gate:g} GB free" - gate = self.params.get("disk_gate_gb") - gate = float(DISK_GATE_GB if gate is None else gate) - held = f"the Docker data root {self._docker_root()} stayed under {gate:g} GB free" - image = docker.task_image(self.task_dir(task)) if task is not None else None - image_gb = self.image_gb(image) if image else 0.0 - if image_gb and docker.image_size(image) is None: - held += f" with {image_gb:.1f} GB more for the task's image {image}, not here yet" - with self.pulls.lock: - taken = self.pulls.taken_gb() - if taken: - held += f", beside {taken:.1f} GB held for the images other trials are still to pull" - return held - - def image_gb(self, image: str) -> float: - """An image's estimated size on disk (GB), from `image_sizes`; 0 when unknown.""" - compressed = self.image_sizes.get(image) - return float(compressed) * docker.ON_DISK / docker.GB if compressed else 0.0 - - def _docker_root(self) -> Path: - if getattr(self, "_root", None) is None: - self._root = docker.data_root() - return self._root - - def cleanup_images(self, task: dict) -> None: - docker.cleanup_images(self.docker_cleanup, self.task_dir(task), self.jobs_dir / ".locks", - threshold_gb=float(self.params.get("rmi_threshold_gb") or 1.0)) - - # --- the rollout ------------------------------------------------------------------------------------------ - def run(self, task: dict, adapter: Path | None) -> RunOutput: - started = time.time() - self.stopped() # another rollout lost the sign-in: no container - try: - self.jobs_dir.mkdir(parents=True, exist_ok=True) - except OSError as exc: # this machine's, before anything: no session ran - return self._output(task, no_trial("host_failed", f"{type(exc).__name__}: {exc}"[-2000:]), None, None, - started) - if not self.task_dir(task).is_dir(): - raise ValueError(f"no Harbor task directory {self.task_dir(task)} for {task['id']}") - why = self.unsupported(task) if self.cli is not None else [] - if why: # Harbor would refuse it for this agent: no container - return self._output(task, no_trial( - "unsupported_task", f"Harbor runs {self.task_dir(task).name} for none of Hermit's agents " - f"(acting_agent.cli: {self.cli}): it declares {' and '.join(why)}; leave it out of the split, or run " - "it with one of Harbor's own agents (acting_agent.name)"), None, None, started) - if adapter is not None and self.cli is None: - self.agent_config(task, deployed={}) # refuses, before anything is staged - try: - if task.get("step") is not None: - return self._run_step(task, adapter, started) - return self._run_trial(task, adapter, started) - except OSError as exc: - # this machine's, past the guarded steps (no process left to start Harbor, a file it could not write or - # read): the host's failure, not the agent's result; a session may have run, so its cost is unknown. A run - # another rollout ended meanwhile ends here too - self.stopped() - failed = f"{type(exc).__name__}: {exc}" - return self._output(task, {**no_trial("host_failed", failed[-2000:]), "ran": None}, None, None, started) - - def _run_trial(self, task: dict, adapter: Path | None, started: float) -> RunOutput: - """A trial, run again while it ends infra (not the agent's result: `unscored`), up to infra_retries times, - each after infra_wait_s, unless its cause runs no better again (NO_RETRY) or has a rule of its own: on a login, - a provider that could not serve the session runs again within busy_retries, one count over all of the trial's - runs (_login_trial), and not again here once that is spent; the plan of the answer agent's model that refused - its request (usage_limit) has the trial run again once that plan allows, as a login's own refusal does, not - counted. What every run used is the trial's (_attempts). One that ends infra all the same is returned as such: - the judge then refuses to score it (hermit.harbor.judge), so that the run records it as a failed run - (judge_error), never as the agent's 0.""" - say = lambda m: print(f"hermit harbor {task['id']}: {m}", file=sys.stderr, flush=True) - retries, pause, runs, again = self.infra_retries, self.infra_wait_s, [], 0 - busy: dict = {"n": 0} # the login's busy_retries, spent over all of the runs - on_login = bool(self.login and self.cli is not None) - while True: - if on_login: - out = self._login_trial(task, adapter, started, busy) - else: - self._model_plan_wait(task) - out = self._trial_once(task, adapter, started) - runs.append(out) - cause = out.extra.get("unscored") - if cause is None or cause in NO_RETRY or busy.pop("spent", False): - return self._attempts(runs) - plan = None if on_login else self.model_plan() - if cause == "usage_limit" and plan is not None: - # the session's refusal held that plan (CliAgent.run, or a wait of its own when the hold cannot be - # written), which _model_plan_wait waits for - say("the plan of the answer agent's model refused its request (its usage limit): the trial runs again " - "once that plan allows") - continue - again += 1 - if again > retries: - return self._attempts(runs) - say(f"not the agent's result ({cause}: {str(out.extra.get('exception_message') or '')[:300]}); again in " - f"{pause:.0f} s ({again}/{retries})") - self._pause(pause) - - def _pause(self, seconds: float) -> None: - """The pause before a trial runs again; LoginFailed as soon as another rollout ends the run meanwhile.""" - if self._stop.wait(seconds): - self.stopped() - - def _model_plan_wait(self, task: dict) -> None: - """Before a trial of the answer agent whose model the runner serves on a login (model_plan): the wait while that - plan holds sessions back, which the trial's one request would otherwise wait out inside the agent's time (its - relay waits again before each request: tools.ModelEndpoint); LoginFailed when another rollout ended the run - meanwhile. A slot is not waited for here: the request's session takes one itself (CliAgent.run), so a request - waiting for one (the plan's SLOTS all taken) waits inside the agent's time, and its trial is the agent's timeout - (its result) if that time runs out; keep agent.concurrency at most SLOTS less the login's other sessions.""" - plan = self.model_plan() - if plan is None: - return - say = lambda m: print(f"hermit harbor {task['id']}: {m}", file=sys.stderr, flush=True) - subscription.wait(plan[0], self._stop, say, None, plan[1]) - self.stopped() - - @staticmethod - def _attempts(runs: list[RunOutput]) -> RunOutput: - """A trial's output when it ran more than once (_run_trial, _login_trial): its last run's, carrying what every - run used (one the plan refused, the provider could not serve or that ended infra may have spent tokens before it - ended): the tokens the runs reported (unknown only when none did), their model calls (a run that reported none - adds 0) and their cost (unknown when any run's is), and each run's own in `extra.attempts` (its usage alone, and - its infra cause, `unscored`, None for a run that is the agent's result; a run that itself ran more than once - gives each of its own: a run's verdict and job are the trial's own, shown to the learner as the trial is). One - run alone is returned as it is, with no `extra.attempts`.""" - last = runs[-1] - if len(runs) == 1: - return last - tokens = [r.tokens for r in runs if r.tokens is not None] - costs = [r.cost_usd for r in runs] - calls = sum(int((r.api_speed or {}).get("model_calls") or 0) for r in runs) - each = [a for r in runs for a in (r.extra.get("attempts") or - [{"tokens": r.tokens, "cost_usd": r.cost_usd, - "model_calls": (r.api_speed or {}).get("model_calls"), - "unscored": r.extra.get("unscored")}])] - return dataclasses.replace(last, tokens=sum(tokens) if tokens else None, - cost_usd=None if None in costs else sum(costs), - api_speed={**(last.api_speed or {}), "model_calls": calls}, - extra={**last.extra, "attempts": each}) - - def _login_trial(self, task: dict, adapter: Path | None, started: float, busy: dict | None = None) -> RunOutput: - """A trial on a login (ChatGPT's, Claude's): after the plan's holds, in one of its slots; again after the window - when the plan refused the session, again after a pause when the provider could not serve it, busy_retries - times at most over the trial (`busy`: the count, shared by _run_trial's runs; "spent" set when the last run is - one that count left as it was), not when it ran out of the agent's time (that is its result), never a lost - sign-in, which a provider's notice beside it does not make busy; never once another rollout ended the run while - it waited (LoginFailed). What every run used is the trial's (_attempts).""" - plan = subscription.plan_dir(self.params.get("plan_dir"), self.login) - credits = subscription.opted_in(self.params.get("usage_credits")) - retries, pause = self.params.get("busy_retries"), self.params.get("busy_wait_s") - retries, pause = int(3 if retries is None else retries), float(120 if pause is None else pause) - say = lambda m: print(f"hermit harbor {task['id']}: {m}", file=sys.stderr, flush=True) - busy = {"n": 0} if busy is None else busy - runs = [] - while True: - subscription.wait(plan, self._stop, say, None, credits) # as long as the plan holds sessions back - self.stopped() # another rollout ended the run while this one waited - with subscription.slot(plan, self._stop, say) as got: - self.stopped() - if got is not True or subscription.held(plan, credits): - continue - out = self._trial_once(task, adapter, started) - runs.append(out) - # What the session says of the plan is written before its slot is given back, as CliAgent.run writes - # it: a trial waiting for the slot reads the plan the moment it has it, so with SLOTS: 1 the one after - # a session that drew credits (or was refused) is held, never started in between. - facts = out.extra.get("session") or {} - report = facts.get("plan_report") - if report is not None: - subscription.note_usage(plan, None, report) - if report.get("using_credits") and not credits: - say("a session drew the account's usage credits, billed apart from the plan: sessions wait " - "for the plan's window (usage_credits: true allows them)") - subscription.hold(plan, report.get("credits_resets_at") or report.get("resets_at"), - "usage_credits", max_s=0.0) - # a lost sign-in is returned (it halts), never held or run again, as CliAgent.run returns it - refused = bool(facts.get("refused")) and not facts.get("served") and not facts.get("login_failed") - if refused: - # a hold that cannot be written makes this trial wait five minutes itself (in its slot, so that - # no session asks again at once), which the run's end cuts short - subscription.hold(plan, until=facts.get("held_until") or None, cancel=self._stop) - if refused: - say("the plan refused the session (its usage limit): the trial runs again once the plan allows") - continue - ran_out = out.extra.get("exception_type") == "AgentTimeoutError" # the agent's own time: its result - if facts.get("unavailable") and not facts.get("served") and not ran_out and not facts.get("login_failed"): - if busy["n"] < retries: - busy["n"] += 1 - say(f"the provider could not serve the session; again in {pause * busy['n']:.0f} s " - f"({busy['n']}/{retries})") - self._pause(pause * busy["n"]) - continue - busy["spent"] = True # busy_retries is its budget: infra_retries adds none - return self._attempts(runs) - - def _trial_once(self, task: dict, adapter: Path | None, started: float) -> RunOutput: - admitted: list = [] - try: - return self._trial_admitted(task, adapter, started, admitted) - finally: - for key in admitted: # the room it held for its image's pull - self.pulls.done(key) - - def _stage(self, task: dict, adapter: Path | None, where: Path) -> dict | None: - """The adapter staged for the acting CLI under `where`, with the task's context when it is per task (computed - from the task's row). It runs - on this machine before any container: an OSError there (a full disk, a folder it cannot write) is the host's, - and the trial or the step that met it is infra (staging_failed). A RuntimeError is the adapter's: its - deployment or its per-task context failed in their child process, the agent's result (errors.adapter_failed: - agent_error, scored 0, its message the feedback), which its caller returns; what such a child prints cannot - tell the host's failure from the adapter's, which could print one.""" - if self.cli is None: - return None - from hermit.harbor.launch import context_limit - return stage(adapter, where, self.python, self.params.get("sandbox_enforce"), self.cli, - task={**task, "instruction": self.instruction(task)}, # a row without one: Harbor's file's - context_query=self.params.get("context_query"), - context_limit=context_limit(self.cli, self.settings(task)) if self.cli == "codex" else None) - - def _staging_verdict(self, exc: Exception, where: Path) -> dict: - """The verdict of a staging whose child failed (_stage's RuntimeError) under `where`: the adapter's - (adapter_failed: scored 0, its message the feedback), unless this process sees this machine fail any adapter's - staging there (_host_fault): then the host's (staging_failed, infra). What the child printed is not taken for - either: the adapter could print one.""" - fault = self._host_fault(where) - if fault is not None: - return no_trial("staging_failed", f"{fault} (the staging's child: {str(exc)[-1500:]})"[-2000:]) - return adapter_failed(str(exc)[-2000:]) - - def _host_fault(self, where: Path) -> str | None: - """What this process sees on this machine that fails a staging child whatever the adapter: a folder it cannot - write, less than STAGING_FREE_MB free under it, the strict adapter sandbox not starting (sandbox_fault); None - when none of them is so. A child killed or run out of its time is the adapter's all the same.""" - try: - if not os.access(where, os.W_OK | os.X_OK): - return f"{where} cannot be written by this process" - free = shutil.disk_usage(where).free - except OSError as exc: - return f"{where}: {type(exc).__name__}: {exc}" - if free < STAGING_FREE_MB << 20: - return f"{where} has {free >> 20} MB free" - return sandbox_fault(self.params.get("sandbox_enforce")) - - @staticmethod - def _plan_refused(exc: Exception) -> dict: - """A trial's launch plan refused for what this trial's adapter is (dsh's MCP server name from its id, ...): the - adapter's result, as a deployment of it that failed (adapter_failed), since the plan rendered at startup for the - same config with a placeholder adapter (check_agent).""" - return adapter_failed(f"the trial's launch plan refused the adapter: {exc}"[-2000:]) - - def _tools_host(self) -> str: - from hermit.harbor.tools import docker_gateway - return str(self.params.get("tools_host") or docker_gateway()) - - def _launch(self, task: dict, job: Path, deployed: dict | None, served, sign_in=True) -> Path | None: - """The trial's launch plan under `job` (none for an agent outside the table); `sign_in` as launch.render's (a - stream's step: "later", copied at its turn).""" - if self.cli is None: - return None - settings = self.settings(task) - meta = {"adapter_id": (deployed or {}).get("adapter_id"), - "tools": served.names if served is not None and hasattr(served, "names") else []} - if self.cli == "answer": - system = "" - if deployed and deployed.get("context_file"): - system = Path(deployed["context_file"]).read_text(encoding="utf-8") - return launch.render_answer(settings, endpoint=served.url, job=job, system=system, - options=self.params.get("answer") or {}, metadata=meta) - config = self.task_config(task) - tools_url = served.mcp_url if served is not None else None - return launch.render(self.cli, settings, deploy=deployed, tools_url=tools_url, job=job, login=self.login, - tools_name=str(deployed["adapter_id"]).replace("-", "_") if tools_url else None, - workdir=(config.get("environment") or {}).get("workdir"), metadata=meta, - login_root=self.login_root, sign_in=sign_in, **self._mounted()) - - def _serve(self, deployed: dict | None, job: Path, adapter: Path | None): - """What the trial's agent reaches on this machine: the answer agent's model endpoint, or the adapter's tools - (none without agent tools); started.""" - if self.cli == "answer": - from hermit.harbor.tools import ModelEndpoint - return ModelEndpoint(self.answer_model(), records=job, enforce=self.params.get("sandbox_enforce"), - plan=self.model_plan()).start() - if self.cli is None or not deployed or not deployed.get("tools"): - return None - from hermit.harbor.tools import AdapterTools - return AdapterTools(adapter, audit_log=job / curate_mod.TOOL_AUDIT, host=self._tools_host(), - enforce=self.params.get("sandbox_enforce")).start() - - def _trial_admitted(self, task: dict, adapter: Path | None, started: float, admitted: list) -> RunOutput: - if not self.wait_for_disk(task, admitted): - return self._output(task, no_trial("disk", f"{self._disk_held(task)}: no container started"), None, None, - started) - job_name = f"{_safe(task['id'])}-{uuid.uuid4().hex[:8]}" - job = self.jobs_dir / job_name - try: - job.mkdir() - deployed = self._stage(task, adapter, job) - except OSError as exc: # this machine's, before any container: no trial - return self._output(task, no_trial("staging_failed", str(exc)[-2000:]), None, None, started, - extra={"harbor_job": str(job)}) - except RuntimeError as exc: # the adapter's (_stage), unless the host's - return self._output(task, self._staging_verdict(exc, job), None, None, started, - extra={"harbor_job": str(job)}) - served = plan = None - try: - try: - served = self._serve(deployed, job, adapter) - except (RuntimeError, OSError) as exc: # no trial without its tools (or its model) - return self._output(task, no_trial("tools_failed", str(exc)[-2000:]), None, None, started, - extra={"harbor_job": str(job), "deployed": deployed}) - try: - try: - plan = self._launch(task, job, deployed, served) - except ValueError as exc: - if adapter is None: # the config's: rendered at startup (check_agent) - raise - return self._output(task, self._plan_refused(exc), None, None, started, - extra={"harbor_job": str(job), "deployed": deployed}) - agent = self.agent_config(task, plan, deployed, tools_host=getattr(served, "host", None)) - (job / "job.yaml").write_text(yaml.safe_dump(self.job_config(task, job_name, agent), - sort_keys=False)) - keep = set(json.loads(plan.read_text()).get("env_from_host", {}).values()) if plan else set() - except OSError as exc: # this machine's, writing the trial's launch and job: no container yet - return self._output(task, no_trial("staging_failed", str(exc)[-2000:]), None, None, started, - extra={"harbor_job": str(job), "deployed": deployed}) - code, expired = self.harbor_run(job, job / "job.yaml", task, keep) - finally: - if served is not None: - served.close() - self.sign_in_back(job) - trials = [t for t in self.trial_dirs(job, task) if (t / "result.json").is_file()] - trial = trials[-1] if trials else None - result = None - if trial is not None: - try: - result = json.loads((trial / "result.json").read_text(encoding="utf-8")) - except ValueError: - result = None - verdict = classify(result, expired=expired, harbor_exit=code, reward_key=self.reward_key) - verdict = self.verifier_timeout_verdict(verdict) - facts, usage, said = self.session(trial, plan, served, job, task) - if facts is not None: - verdict = self.session_verdict(verdict, facts, self.plan_login()) - verdict = self.verifier_verdict(verdict, trial, self.redactions) - try: - log = curate_mod.curate(trial, job, verdict, redact=self.redactions, audit=job / curate_mod.TOOL_AUDIT, - instruction=self.instruction(task), plan=plan, reply=said.get("reply")) - self.cleanup_images(task) - finally: - if verdict["halt"]: # the trial's log kept first, however that went - self.end_run(verdict, job) - extra = {"harbor_job": str(job), "harbor_exit": code, "deployed": deployed} - return self._output(task, verdict, result, log, started, trial=trial.name if trial else None, extra=extra, - trajectory=trial / "agent" / "trajectory.json" if trial else None, facts=facts, - usage=usage, said=said, deployed=deployed) - - def sign_in_back(self, job: Path) -> None: - """After the trial (or the step) under `job`, however it ended: the ChatGPT sign-in Codex refreshed in the - container becomes the shared copy when it passes signin.take_back's checks (a regular file of at most - signin.MAX_BYTES, a ChatGPT sign-in, stamped at most SKEW ahead of this machine's clock in Harbor's local Docker - environment, whose containers share that clock, and OTHER_CLOCK in another, whose clock may differ) and is a - later refresh of the copy, for its account (hermit.agent.base.login.take_back); and no copy of a sign-in is - left in the job (the home's, the one that came back).""" - where = Path(job) / launch.LAUNCH_DIR - returned = where / launch.RETURN_DIR - try: - if self.login == "chatgpt" and _folder(returned): - local = (self.params.get("environment") or {}).get("type") in (None, "docker") - signin.take_back(self.login_root, returned, local=local) - except OSError as exc: # this machine's (the shared copy's folder full): the trial stands, its refresh lost - print(f"WARNING: the sign-in Codex refreshed in {Path(job).name} could not be taken back ({exc}); a " - "later session refreshes again", file=sys.stderr, flush=True) - for folder in (returned, where / launch.HOME_DIR): - if folder.is_dir() and not folder.is_symlink(): - subscription.remove(folder / subscription.FILE, folder) - - def verifier_timeout_verdict(self, verdict: dict) -> dict: - """A verifier that ran out of its time (VerifierTimeoutError) is the agent's result with `verifier_timeout: - agent`, the default, as Harbor grades it: a timeout on the agent's solution (solution_timed_out), scored its - reward or 0 and not run again (a solution too slow for its tests, as AlgoTune's verifier times it, or a process - it left that hung them). With `verifier_timeout: infra`, for a host whose load makes verifiers time out, it is - infra: run again. Either way the session did not run out of its time: `timed_out` stays false (a method that - reads it, Continual Harness's, tells the agent its session hit its wall clock).""" - if verdict.get("exception_type") != "VerifierTimeoutError": - return verdict - if self.verifier_timeout == "infra": - return {**verdict, "status": "infra", "failure_cause": "verifier_failed", "timed_out": False} - return {**verdict, "status": "timeout", "failure_cause": "solution_timed_out", "timed_out": False} - - # --- the acting CLI's session, from its stream ------------------------------------------------------------------ - def session(self, outputs: Path | None, plan: Path | None, served, job: Path, task: dict | None = None): - """(the session's facts, its usage, what its stream said) of a trial or a step whose outputs are under - `outputs` (agent/), read as Hermit reads a session of the CLI (its Base Agent's `outcome`, `parse`, - `adjust_usage`, priced at the binding's price): served, refused for the plan's limit (and until when), the - provider unavailable, the sign-in lost, rejected, a model's refusal, the plan's report (Claude Code's stream's; - on a ChatGPT login Codex's session logs', collected), started, the tools down, the tokens used; for Codex, a - session that served nothing and whose stream says its sign-in or key was refused (signin.codex_signin_lost) has - lost its sign-in, as Claude Code's Base Agent says of its own, and its provider was unavailable only by its last - failure (signin.codex_unavailable); (None, None, {}) for an agent outside the table - or a session that never ran (no stream). The container could write all of it: each file is read only as a - regular file, never through a link (a link in the stream's place is no stream, a link among the collected files - is not read), and the stream and the CLI's last message are read once, into this process's own copies under - `launch/read/`, which the Base Agent then reads.""" - if plan is None or self.cli is None: - return None, None, {} - data = json.loads(plan.read_text(encoding="utf-8")) - agent_dir = outputs / "agent" if outputs is not None and _folder(outputs / "agent") else None - if self.cli == "answer": - said = {} - if agent_dir is not None: - try: - said = json.loads(_read(agent_dir / "answer.json") or "{}") - except (ValueError, RecursionError): # no JSON, or nested past the parser's depth - said = {} - said = said if isinstance(said, dict) else {} # JSON, but no object: no answer either - facts = served.facts() if served is not None else {"started": False} - if facts.get("plan_waited") and facts.get("calls") == 1 and \ - not any(s is not None for s in facts.get("statuses") or []): - # its one request never answered, and the relay held it for its login's plan (tools.ModelEndpoint: the - # plan's hold it met, before the session): why the plan held it back - facts["plan_held"] = facts["plan_waited"] - usage = served.usage if served is not None and served.usage.calls else None - if served is not None and usage is None and not facts.get("calls"): # no request reached the relay: - usage = dataclasses.replace(served.usage, cost_usd=0.0) # nothing was spent - return facts, usage, {"reply": said.get("reply"), "answer": said.get("answer")} - raw = _read(agent_dir / data["stream"]) if agent_dir is not None else None - if raw is None: - return None, None, {} - # its lines end as in Hermit's own run, which reads a stream with Path.read_text: "\r\n" and "\r" are "\n" (an - # event's strings hold a "\r" escaped, so no event is split by it) - raw = raw.replace("\r\n", "\n").replace("\r", "\n") - from hermit.agent.base.cli import DSH_TOOLS_DOWN - from hermit.agent.base.interface import make - from hermit.agent.base.records import price_usage - settings = self.settings_of(data, task) - agent = make(clis.entry(self.cli)["backend"], settings) - read = job / launch.LAUNCH_DIR / "read" # what is read of the session, as this process took it - shutil.rmtree(read, ignore_errors=True) - read.mkdir(parents=True) - stream = read / data["stream"] - stream.write_text(raw, encoding="utf-8") - collected = agent_dir / curate_mod.COLLECT_DIR - collected = collected if _folder(collected) else None - last = read / launch.LAST_MESSAGE # the CLI's last message, as this process copied it - if collected is None or not _copy(collected / launch.LAST_MESSAGE, last): - last = None - said = agent.outcome(raw, stream, last) - if agent.backend == "codex": - # Codex's own notices only (signin.lines: never what its log quotes of the session's work, which the Base - # Agent's reading takes for notices too): a sign-in or a key refused, which no wait restores; the plan's - # usage limit; the provider unavailable only by the session's last failure (the Base Agent's reading takes - # a reconnect it recovered from for one too); a refusal of the run's configuration or bill (a 400, 402 or - # 404, an API account's quota), which no retry changes, as the other CLIs' Base Agents read theirs - # (`rejected`: model_rejected when it names the configuration, CONFIG_REFUSED; a halt when it is the bill, - # BILLING_REFUSED) - served = bool(said.get("served")) - said["login_failed"] = not served and signin.codex_signin_lost(raw) - said["unavailable"] = not served and signin.codex_unavailable(raw) - said["refused"] = not served and signin.codex_refused(raw) - refusal = None if served or said["login_failed"] else signin.codex_rejected(raw) - said["rejected"] = refusal is not None - if refusal is not None: - said["failure"] = refusal[:600] # what refused it, as Codex wrote it (its status first) - started = said.pop("started", None) - if started is None: - started = any(line.lstrip().startswith("{") for line in raw.split("\n")) - model = (settings.get("binding") or {}).get("ref") or settings.get("model") or agent.backend - handle, usage = agent.parse(stream, model) - stream.unlink() # read: the trial's own copy stays in its outputs - if last is not None: - last.unlink() # read: the trial's own copy stays in its outputs - for src, dst in launch.COLLECT.get(agent.backend, []) if collected is not None else []: - if dst != launch.LAST_MESSAGE: # what adjust_usage reads of the home, back where the home had it - _copy(collected / dst, read / src) - db = read / "data" / "opencode" / "opencode.db" - if db.is_file() and db.with_name(db.name + "-wal").is_file(): - # OpenCode keeps its database in WAL mode: a session cut off leaves its last rows in the -wal file, which - # the reader (opened immutable) never opens; this copy is the runner's own, so they are merged into it - import contextlib - import sqlite3 - try: - with contextlib.closing(sqlite3.connect(db)) as c: - c.execute("PRAGMA wal_checkpoint(TRUNCATE)") - except sqlite3.Error: - pass - usage = agent.adjust_usage(read, handle, usage, resumed=False) - price_usage(usage, settings) - down = DSH_TOOLS_DOWN.findall(agent.notices(raw)) if agent.backend == "dsh" else [] - if agent.backend == "claude" and settings.get("tools_name"): - name = settings["tools_name"] - for event in _events(raw): # Claude Code's init event names each MCP server's status - if event.get("type") == "system" and event.get("subtype") == "init": - status = {s.get("name"): s.get("status") for s in event.get("mcp_servers") or [] - if isinstance(s, dict)} - if status.get(name) not in (None, "connected"): - down.append((name, str(status.get(name)))) - break - facts = {k: said.get(k) for k in SESSION_FACTS if k in said} - facts.update(started=bool(started), tools_down=bool(down), session=handle, - failure=str(said.get("failure") or "")[-600:], - tokens=usage[0].input_tokens + usage[0].output_tokens if usage else 0) - if down: - facts["failure"] = f"the adapter's tools did not come up ({down[0][0]}: {down[0][1]})"[-600:] - if self.login == "chatgpt" and collected is not None: - try: # Codex's session logs, collected: the plan's rate limits - report = subscription.usage_of(collected) - except Exception: # noqa: BLE001 - the container wrote them: one that - report = None # cannot be read is no report (kept as defence: usage_of - # skips a line that is no object itself) - if report is not None: - facts["plan_report"] = report - return facts, (usage[0] if usage else None), {"reply": said.get("reply")} - - def settings_of(self, plan: dict, task: dict | None = None) -> dict: - """The Base Agent's settings a plan was rendered with, for reading its stream (its tools' name, the model).""" - settings = self.settings(task) - if plan.get("metadata", {}).get("tools"): - settings["tools_name"] = str(plan["metadata"].get("adapter_id") or "hermit_tools").replace("-", "_") - return settings - - @staticmethod - def session_verdict(verdict: dict, facts: dict, login: str | None = None) -> dict: - """The trial's verdict with what the session's stream adds. A lost sign-in halts (the run ends: end_run). A - plan's refusal or a provider - that could not serve is not the agent's doing (infra: the trial is run again), unless the session ran out of - the agent's time (its result, kept: a timed-out session has no completed turn, and an earlier notice it - recovered from is no failure, as Hermit's own sessions keep it); nor is a provider that refused the run's - configuration before it served any request (a key it does not take, a permission it does not give, a model it - does not have, a request setting it does not take: model_rejected, config_refused; no halt), or the account's - bill (its quota or credit spent, a payment it needs: billing_refused; usage_limit, a halt, as nothing later can - be served), nor a session that never started or whose tools never came up (session_failed). A first request - refused for what the adapter's or the task's input made it (a prompt too long, an invalid request, a content - filter) is the agent's result, as Harbor scores it, and so is the model's own refusal (agent_refused_task, - kept). A usage limit on a `login` (the acting CLI's, or that of the answer agent's model the runner serves: - HarborAgent.plan_login) is the login's to wait for (the plan's window resets): no halt; without one (an API - account's), nothing later can be served: a halt. The plan of the login the answer agent's model runs on is that - login's to wait for too (`plan_held`: its one request unanswered, the relay having held it for that plan; a hold - that came while the session ran is no wait of its own): a trial that ran out of the agent's time with it is - infra (usage_limit, no halt), not the agent's timeout.""" - def message(): - return verdict.get("exception_message") or facts.get("failure") or "" - if facts.get("login_failed"): - return {**verdict, "status": "infra", "failure_cause": "login_failed", "halt": True, - "exception_type": verdict.get("exception_type") or "AgentAuthenticationError", - "exception_message": message()} - ran_out = verdict.get("exception_type") == "AgentTimeoutError" - if ran_out and facts.get("plan_held") and not facts.get("served"): - return {**verdict, "status": "infra", "failure_cause": "usage_limit", "halt": False, - "exception_message": f"the model's one request waited for its login's plan, which held its " - f"session back ({facts['plan_held']}), and had no answer when the agent's " - "time ran out"} - if facts.get("unavailable") and not facts.get("refused") and not facts.get("served") and \ - quota_spent(facts): # a 429 for a quota spent: nothing later can be served, as after a 402, - # however long the CLI went on retrying it (a session that ran out of the agent's time all the same) - return {**verdict, "status": "infra", "failure_cause": "usage_limit", "halt": True, - "exception_message": f"the provider does not bill the account (its quota spent): {message()}"} - if (facts.get("refused") or facts.get("unavailable")) and not facts.get("served") and not ran_out: - cause = "usage_limit" if facts.get("refused") else "model_provider_failed" - return {**verdict, "status": "infra", "failure_cause": cause, - "halt": bool(facts.get("refused")) and not login, "exception_message": message()} - if facts.get("rejected") and not facts.get("served") and not facts.get("model_refusal") and \ - not facts.get("tokens") and config_refused(facts): - if billing_refused(facts): # nothing later can be served, as after an API account's usage limit - return {**verdict, "status": "infra", "failure_cause": "usage_limit", "halt": True, - "exception_message": f"the provider does not bill the account (its quota or credit spent, " - f"or a payment it needs): {message()}"} - return {**verdict, "status": "infra", "failure_cause": "model_rejected", "halt": False, - "exception_message": message()} - if verdict["status"] != "infra" and (not facts.get("started") or facts.get("tools_down")): - why = facts.get("failure") if facts.get("tools_down") else \ - f"the session never started (no event in its stream): {message()}" - return {**verdict, "status": "infra", "failure_cause": "session_failed", "halt": False, - "exception_message": str(why)[-2000:]} - if facts.get("model_refusal") and verdict["status"] in ("infra", "agent_error"): - # The model's safeguards stopped the session (a refusal stop reason): the agent's own end, as Harbor's - # AgentSafetyRefusalError is (agent_refused_task), kept as its result, not run again - return {**verdict, "status": "agent_error", "failure_cause": "agent_refused_task", "halt": False, - "exception_message": message()} - waits = bool(login) and verdict.get("exception_type") == "ApiUsageLimitError" # the login loop's to hold - return {**verdict, "halt": verdict["halt"] and not waits} - - @staticmethod - def verifier_verdict(verdict: dict, trial: Path | None, redact=()) -> dict: - """A trial whose verifier ran no test is infra: run again, not kept as the agent's result, whether the agent - finished, ran out of its time or exited with an error (Harbor verifies all three and grades the reward), and - whether the verifier wrote 0, left no reward or ran out of its time (the verifier's own failure, which is - otherwise the agent's result, as Harbor grades it). Either the verifier's tools never arrived, their download - failed by a server (a 5xx, GitHub's release download) so its test runner is missing and it wrote no report - (hermit.harbor.errors.verifier_unreached), or the host's disk was full while it ran, before its tests wrote a - report (verifier_disk_full: with a report, the words are a test's output). A trial the verifier passed ran its - tests; a trial whose verifier never ran (no output of it) stays as it was. - The verifier's output the verdict quotes has `redact`'s patterns removed (verifier_redact), as the curated - output has.""" - if trial is None or verdict["status"] == "infra" or float(verdict.get("reward") or 0) >= 1: - return verdict - out = trial / "verifier" - stdout = _text(out / "test-stdout.txt") - if verifier_unreached(stdout, (out / "ctrf.json").exists()): - why = "the verifier's tools never arrived (their download failed), so its tests never ran" - elif verifier_disk_full(stdout, (out / "ctrf.json").exists()): - why = "the host's disk was full while the verifier ran (ENOSPC)" - else: - return verdict - return {**verdict, "status": "infra", "failure_cause": "verifier_failed", - "exception_message": f"{why}: {curate_mod.redact_verifier(stdout, redact).strip()[-300:]}"} - - # --- streams -------------------------------------------------------------------------------------------------- - def _stream_tools(self, stream: str, deployed: dict | None, step_dir: Path, adapter: Path | None): - """The stream's tool server: one for the stream's life, started by the first rollout of the stream whose - adapter has tools (with that step's adapter and log: whichever step serves next swaps its own in); None for a - step whose adapter has none.""" - if self.cli in (None, "answer") or not deployed or not deployed.get("tools"): - return None - from hermit.harbor.tools import AdapterTools - with self._streams_lock: - served = self._streams.get(stream) - if served is not None: - return served - served = AdapterTools(adapter, audit_log=step_dir / curate_mod.TOOL_AUDIT, host=self._tools_host(), - enforce=self.params.get("sandbox_enforce")).start() - self._streams[stream] = served - return served - - def _stream_done(self, stream: str) -> None: - """A stream this runner is done with (its driver stopped, or gone): its tool server closed, its driver no longer - this runner's to stop.""" - with self._streams_lock: - served = self._streams.pop(stream, None) - self._drivers.pop(stream, None) - if served is not None: - served.close() - - def close(self) -> None: - """The streams this runner stepped that are still open, closed (_close_runner_streams, which also runs when the - runner is freed, and at exit).""" - _close_runner_streams(self._drivers, self._streams, self._streams_lock) - - def _run_step(self, task: dict, adapter: Path | None, started: float) -> RunOutput: - """One step of a multi-step task, through the stream's driver. While it runs, its index is one of its stream's - steps in flight (`_in_flight`): a step whose turn cannot come ends at once (infra, stream_order: its driver - waits for a step no rollout of this runner is running, or it has none and no rollout of its first step is - running to start one), never waiting for its turn until the deadline, and the last of its stream's rollouts to - end then stops the driver (_leave).""" - index = int(task["step_index"]) - # the row's task and rollout, in this runner (its own id): two runs on one jobs directory, at once or one after - # the other (a resumed train among them: its new runner has no driver for a stream under way, so that stream's - # remaining steps in the epoch end stream_order, and its first step's next rollout starts it over), never share - # a driver, its environment or its steps' directories. One runner's rollouts of one stream after it finished - # (hermit eval's repeats, a training epoch after another) start a new driver there, which sets the earlier - # trial aside (earlier//); - # each rollout of a step works in a directory of its own (_step) - stream = _safe(f"{task.get('harbor_task') or task['id']}__s{task.get('rollout', 1)}__{self.run_id}") - with self._streams_lock: - self._in_flight.setdefault(stream, []).append(index) - try: - return self._step(task, adapter, started, stream) - finally: - self._leave(stream, index) - - def _give_up(self, stream: str, index: int, cause: str) -> None: - """A step that ends before it reaches its stream's driver (its staging, its adapter, its tools, its launch plan - or the disk gate failed, its turn cannot come or did not come within its wait, its turn's sign-in or tools - could not be readied, or any of them raised): the driver waits for it, so no later step's turn can come. The - stream is given up, and the last of its rollouts to end stops its driver (_leave), whose environment no step - can use any more: the stream's next pass (an eval's next repeat, a later epoch) starts from its first step in a - new environment. The first reason given is kept.""" - with self._streams_lock: - self._given_up.setdefault(stream, f"its step {index} ended before it reached the stream's driver ({cause})") - - def _leave(self, stream: str, index: int) -> None: - """A step's rollout ending: its index out of its stream's steps in flight. The last of a stream whose next turn - cannot come (`_given_up`) stops its driver, whose environment none of the stream's steps can use any more, and - is done with the stream, which its first step may then start again.""" - with self._streams_lock: - steps = self._in_flight[stream] - steps.remove(index) - if steps: - return - del self._in_flight[stream] - if stream not in self._given_up: - return - client = StreamClient(self.jobs_dir / "streams" / stream, python=self.harbor_python) - if client.running() is not None: - client.stop() - self._stream_done(stream) - with self._streams_lock: - if stream not in self._in_flight: # no step of it came in the meantime - self._given_up.pop(stream, None) - - def _first_step_missing(self, stream: str, client: StreamClient) -> str | None: - """Why a stream that has no driver gets none for a later step: no rollout of its first step is running in this - runner (none either after TURN_GRACE_POLLS polls, in which one submitted just before this one has begun), or its - next turn already cannot come; None once a driver or such a rollout is there.""" - for poll in range(stream_mod.TURN_GRACE_POLLS + 1): - if poll: - time.sleep(stream_mod.POLL_S) - with self._streams_lock: - if stream in self._given_up: - return self._given_up[stream] - if 1 in self._in_flight.get(stream, ()): - return None - if client.running() is not None: - return None - return ("the stream has no driver (none started in this run, or the one its earlier steps ran in was stopped " - "or idled out) and no rollout of its first step is running to start one") - - def _step_missing(self, stream: str, nxt: int) -> str | None: - """Why no rollout will ask the stream's driver, idle, for the step `nxt` it waits for (stream.wait_turn's - `missing`): none of this runner's is running it (it ended before it reached the driver, or ran in an earlier - training step), or the stream's next turn already cannot come; None while one of its rollouts runs.""" - if self._stop.is_set(): - return f"the run ended ({self._stop_why})" - with self._streams_lock: - if stream in self._given_up: - return self._given_up[stream] - if nxt in self._in_flight.get(stream, ()): - return None - return (f"the driver waits for step {nxt}, which no rollout is running (it ended before it reached the driver, " - "or ran in an earlier training step)") - - def _step(self, task: dict, adapter: Path | None, started: float, stream: str) -> RunOutput: - """A step's rollout in its stream (_run_step).""" - index, name = int(task["step_index"]), str(task["step"]) - total = int(task.get("steps") or 0) - stream_dir = self.jobs_dir / "streams" / stream - settings = {"idle_s": 4 * 3600, "start_timeout_s": 3600, # one left empty (null) is its default - **{k: v for k, v in (self.params.get("stream") or {}).items() if v is not None}} - client = StreamClient(stream_dir, python=self.harbor_python) - # this rollout's own: a rollout of the step after another (an eval's repeat, a later epoch) keeps its records - step_dir = stream_dir / "steps" / f"{index:03d}-{_safe(name)}-{uuid.uuid4().hex[:8]}" - try: - step_dir.mkdir(parents=True) - except OSError: # this machine's (host_failed): the step never reaches the driver either - self._give_up(stream, index, "host_failed") - raise - if index > 1 and client.running() is None and (why := self._first_step_missing(stream, client)): - # its turn cannot come (no driver, nor a rollout of the first step to start one): it ends before anything - # is staged or served for it, as the check before the driver's start below would end it - with self._streams_lock: - self._given_up.setdefault(stream, why) - verdict = no_trial("stream_order", str(TurnLostError(index, why))) - log = curate_mod.curate(None, step_dir, verdict, result=None, - extra_files={DRIVER_LOG: stream_dir / DRIVER_LOG}, redact=self.redactions, - audit=None, instruction=self.instruction(task), plan=None, reply=None) - return self._output(task, verdict, None, log, started, trial=stream, step=name, - extra={"harbor_job": str(step_dir), "stream_dir": str(stream_dir)}) - # every way the step can end before its driver is asked for it (its staging, its tools, its launch plan, the - # disk gate, a turn that cannot come or did not come in its wait, its turn's own setup; an exception of any of - # them) gives the stream up: the driver waits for this step, so no later step's turn can come, and the last of - # the stream's rollouts to end stops it (_leave); the stream's next pass then starts anew. A step the stream - # already ran (StepAlreadyRunError) gives nothing up: the stream went on without this rollout. - asked = passed = False - reply, result, error, cause = None, None, None, "stream_lost" - try: - try: - deployed = self._stage(task, adapter, step_dir) - except OSError as exc: # this machine's, before the step: not run - self._give_up(stream, index, "staging_failed") - return self._output(task, no_trial("staging_failed", str(exc)[-2000:]), None, None, started, - extra={"harbor_job": str(step_dir), "stream_dir": str(stream_dir)}) - except RuntimeError as exc: # the adapter's (_stage), unless the host's - verdict = self._staging_verdict(exc, step_dir) - self._give_up(stream, index, verdict["failure_cause"]) - return self._output(task, verdict, None, None, started, - extra={"harbor_job": str(step_dir), "stream_dir": str(stream_dir)}) - served = endpoint = plan = None - lost = turn = False - try: - try: - if self.cli == "answer": - from hermit.harbor.tools import ModelEndpoint - endpoint = ModelEndpoint(self.answer_model(), records=step_dir, - enforce=self.params.get("sandbox_enforce"), - plan=self.model_plan()).start() - else: - served = self._stream_tools(stream, deployed, step_dir, adapter) - except (RuntimeError, OSError) as exc: - self._give_up(stream, index, "tools_failed") - return self._output(task, no_trial("tools_failed", str(exc)[-2000:]), None, None, started, - extra={"harbor_job": str(step_dir), "deployed": deployed}) - try: - # the sign-in is copied at the step's turn (below), never now: an earlier step of the stream may - # still be running, and refresh the shared copy before this one starts - try: - plan = self._launch(task, step_dir, deployed, endpoint or served, sign_in="later") - except ValueError as exc: - if adapter is None: # the config's: rendered at startup (check_agent) - raise - self._give_up(stream, index, "adapter_failed") - return self._output(task, self._plan_refused(exc), None, None, started, - extra={"harbor_job": str(step_dir), "stream_dir": str(stream_dir), - "deployed": deployed}) - agent = self.agent_config(task, plan, deployed, tools_host=getattr(served, "host", None)) - (step_dir / "agent.json").write_text(json.dumps(agent, indent=1) + "\n") - keep = set(json.loads(plan.read_text()).get("env_from_host", {}).values()) if plan else set() - except OSError as exc: # this machine's, as the staging's: the step not run - self._give_up(stream, index, "staging_failed") - return self._output(task, no_trial("staging_failed", str(exc)[-2000:]), None, None, started, - extra={"harbor_job": str(step_dir), "stream_dir": str(stream_dir), - "deployed": deployed}) - try: - if client.running() is None and index > 1 and (why := self._first_step_missing(stream, client)): - raise TurnLostError(index, why) # no driver (an environment) started for it - # the environment's start: the disk gate, with room for the task's image as a single trial's - # (held until the driver is up, its image then pulled) - room: list = [] - try: - if client.running() is None and not self.wait_for_disk(task, room): - self._give_up(stream, index, "disk") - return self._output(task, no_trial("disk", f"{self._disk_held(task)}: the stream's " - "environment was not started"), None, None, - started) - with self._streams_lock: # this runner's to stop until it finishes (close) - self._drivers.setdefault(stream, client) - client.ensure(task_dir=self.task_dir(task), name=stream, agent=agent, - environment=self.environment_config(), verifier=self.verifier_config(), - idle_s=float(settings["idle_s"]), env=self.harbor_env(keep), - start_timeout_s=float(settings["start_timeout_s"]), - trial={k: self.timeout_multiplier(k) for k in TIMEOUT_MULTIPLIERS}) - finally: - for key in room: - self.pulls.done(key) - # Earlier steps of the stream may be running in other rollouts of the same batch: wait for them, - # each for its own budget at most (or `stream.turn_wait_s`); not for one no rollout is running (rows - # are submitted in order: an earlier step of the batch has begun by then), whose turn never comes. - budget = self.wait_budget(task, build=False) - turn_wait = self._turn_wait(task, index, settings) - client.wait_turn(index, timeout_s=turn_wait, missing=lambda nxt: self._step_missing(stream, nxt)) - self.stopped() # another rollout ended the run while this one waited - turn = True - try: - if plan is not None and self.login == "chatgpt": - try: - launch.sign_in(plan, self.login_root) # the shared copy as it is at this step's turn - except (OSError, RuntimeError) as exc: # this machine's (no shared copy): no trial - raise _TurnUnready("staging_failed", f"the sign-in's copy for the step's turn: {exc}") - if served is not None and (served.adapter, served.audit_log) != \ - (Path(adapter), step_dir / curate_mod.TOOL_AUDIT): - # this step's turn: its adapter behind the same URL, its calls in its own log (the server - # may have been started by a later step's rollout, or served the step before) - try: - served.swap(Path(adapter), step_dir / curate_mod.TOOL_AUDIT) - except (RuntimeError, OSError) as exc: # its tools: none for the step - raise _TurnUnready("tools_failed", f"the adapter's tools for the step's turn: {exc}") - except _TurnUnready as exc: - # the driver waits for this step, which is not asked for: the stream is given up (below), never - # this machine's failure taken for the driver's (its stream lost, its driver left running) - error, cause = str(exc)[-2000:], exc.cause - else: - asked = True # from here the driver may run the step's session - reply = client.step(name=name, index=index, plan=str(plan) if plan else None, - agent_kwargs=agent["kwargs"], timeout_s=budget, - allowed_hosts=agent.get("extra_allowed_hosts") or []) - except StreamOrderError as exc: - error, cause = str(exc), "stream_order" - passed = isinstance(exc, StepAlreadyRunError) # the stream went on past it: not this rollout's - if isinstance(exc, TurnLostError): # nor can a later step's: the stream is given up - with self._streams_lock: - self._given_up.setdefault(stream, exc.why) - except (StreamError, RuntimeError) as exc: # the driver's, or a tool server another step closed - # a driver that is gone (it closed the connection without an answer: killed while it answered) is - # lost as one whose socket refuses is - error, lost = str(exc), isinstance(exc, StreamError) and client.gone() - except OSError as exc: # the driver gone (killed), or silent past a request's time (busy) - error, lost = f"the stream's driver could not be reached: {exc}", not isinstance(exc, TimeoutError) - finally: - if endpoint is not None: - endpoint.close() - self.sign_in_back(step_dir) - finally: - if not asked and not passed: - self._give_up(stream, index, cause if error else "it ended before its driver was asked for it") - if reply and reply.get("ok"): - result = reply.get("step") or {} - verdict = self.verifier_timeout_verdict(classify(result, reward_key=self.reward_key)) - elif reply and reply.get("expired"): - verdict = classify(None, expired=True) - else: - if (reply or {}).get("out_of_order"): - cause = "stream_order" - verdict = no_trial(cause, error or (reply or {}).get("error") or json.dumps(reply)[:500]) - if asked and cause != "stream_order": # its session may have run, its usage unread: unknown - verdict["ran"] = None - outputs = Path(reply["paths"]["step_dir"]) if reply and reply.get("ok") and reply.get("paths") else None - facts = usage = None - said: dict = {} - if outputs is not None: - facts, usage, said = self.session(outputs, plan, endpoint or served, step_dir, task) - if facts is not None: - verdict = self.session_verdict(verdict, facts, self.plan_login()) - verdict = self.verifier_verdict(verdict, outputs, self.redactions) - ran = bool(reply and reply.get("ok")) - aborted = ran and bool(reply.get("aborted")) - finished = ran and ((total and index >= total) or bool(reply.get("last")) or aborted) - if finished or (reply or {}).get("expired") or verdict["halt"]: - client.stop() # the environment has no further use - self._stream_done(stream) - if finished: - self.cleanup_images(task) - elif lost: - # no step after it reaches this driver: its tools closed (not for one only slow to answer, which may still - # be running the step before, on these tools) - self._stream_done(stream) - # the step's tool calls once its turn came; a step whose turn never came made none (the log there is an - # earlier run's of the step) - try: - log = curate_mod.curate(outputs, step_dir, verdict, result=result, - extra_files={DRIVER_LOG: stream_dir / DRIVER_LOG}, redact=self.redactions, - audit=step_dir / curate_mod.TOOL_AUDIT if turn else None, - instruction=self.instruction(task), plan=plan, reply=said.get("reply")) - finally: - if verdict["halt"]: # the step's log kept first, however that went - self.end_run(verdict, step_dir) - return self._output(task, verdict, result, log, started, trial=stream, step=name, - extra={"harbor_job": str(step_dir), "stream_dir": str(stream_dir), "deployed": deployed, - "aborted": aborted}, - trajectory=outputs / "agent" / "trajectory.json" if outputs else None, facts=facts, - usage=usage, said=said, deployed=deployed) - - # --- the rollout's output ------------------------------------------------------------------------------------- - def _output(self, task: dict, verdict: dict, result: dict | None, log: Path | None, started: float, *, - trial: str | None = None, step: str | None = None, extra: dict | None = None, - trajectory: Path | None = None, facts: dict | None = None, usage=None, said: dict | None = None, - deployed: dict | None = None) -> RunOutput: - agent_result = (result or {}).get("agent_result") or {} - metadata = agent_result.get("metadata") or {} - hooks_active = metadata.get("hooks_active") # None: nothing was deployed - status = verdict["status"] - ok = status == "ok" - if self.cli is not None: - # Hermit's reading of the session: its tokens and calls, priced at the backend's price (Harbor counts - # nothing for an agent whose model it does not run) - tokens = usage.input_tokens + usage.output_tokens if usage is not None else None - cost = usage.cost_usd if usage is not None else None - calls = usage.calls if usage is not None else 0 - cache_tokens = usage.cached_input_tokens if usage is not None else None - else: - n_in, n_out = agent_result.get("n_input_tokens"), agent_result.get("n_output_tokens") - tokens = int(n_in) + int(n_out) if ok and n_in is not None and n_out is not None else None - cost = agent_result.get("cost_usd") - calls = curate_mod.model_calls(trajectory) if trajectory else 0 - cache_tokens = agent_result.get("n_cache_tokens") - wall = (curate_mod.seconds((result or {}).get("agent_execution")) - or curate_mod.seconds({"started_at": (result or {}).get("started_at"), - "finished_at": (result or {}).get("finished_at")}) - or round(time.time() - started, 1)) - never = result is not None and not ((result.get("agent_execution") or {}).get("started_at")) - if (verdict.get("ran") is False or never) and usage is None: - # no container, no session (no_trial), or no agent started in it (Harbor records none: its environment - # or the agent's setup failed first): nothing spent - tokens, cost, calls = 0, 0.0, 0 - rewards = ((result or {}).get("verifier_result") or {}).get("rewards") if result else None - output = {"harbor_task": task.get("harbor_task") or task.get("task") or task["id"], - "rollout": task.get("rollout"), "step": step, "reward": verdict["reward"], "rewards": rewards, - "status": status, "failure_cause": verdict["failure_cause"], - "exception_type": verdict["exception_type"], "trial": trial, "hooks_active": hooks_active} - said = said or {} - if said.get("reply") is not None: - output["reply"] = said["reply"] - if self.cli == "answer": - output["answer"] = said.get("answer") - more = {"status": status, "timed_out": verdict["timed_out"], "exception_type": verdict["exception_type"], - "exception_message": verdict["exception_message"][:2000], "failure_cause": verdict["failure_cause"], - "halt": verdict["halt"], "n_cache_tokens": cache_tokens, "verifier_reward_raw": verdict["reward"], - "hooks_active": hooks_active, "agent_metadata": metadata, "judge": {"pass_reward": self.pass_reward}} - if self.cli is not None: - more["cli"] = self.cli - if facts is not None: - more["session"] = facts - if deployed and deployed.get("context_tool") is not None: - more["context_tool"] = deployed["context_tool"] - if status == "infra": - more["unscored"] = verdict["failure_cause"] or "infra" - return RunOutput(output=output, log=log, cost_usd=cost, wall_s=wall, tokens=tokens, - api_speed={"model_calls": calls, "status": status}, extra={**more, **(extra or {})}) diff --git a/hermit/harbor/staging.py b/hermit/harbor/staging.py deleted file mode 100644 index 4f3df66a..00000000 --- a/hermit/harbor/staging.py +++ /dev/null @@ -1,198 +0,0 @@ -"""The adapter, deployed for the task container. - -`stage` prepares the acting CLI's workspace from the adapter exactly as `hermit deploy` would, by the host of that -CLI (`HOSTS`: hermit.hosts.codex, claude_code, pi, opencode, dsh, mini_swe_agent): the context block in the CLI's -context file, the native roles, skills and hooks where the CLI looks for them, the whole adapter copied to -`hermit//`: the host's own `prepare`, called as Hermit's acting agents call it -(hermit.agent.acting.workspace.deploy), in a child process of this module (`python -m hermit.harbor.staging`) that -renders the adapter's exports in the strict sandbox (HA_ADAPTER_SANDBOX=1: the adapter read-only, no network, none of -this process's keys). What differs from a host deployment, because the workspace goes into a container whose paths -are not this machine's: - -- the hooks (Codex's `.codex/hooks.json`, Claude Code's in `.claude/settings.json`) are rewritten with every command - run from `hermit//` relative to the session's directory (the task's workdir, where the CLI runs; for - Claude Code, which runs a hook in the shell's current directory, under `$CLAUDE_PROJECT_DIR`), not from the absolute - host folder; -- no host starts the adapter's tools (`prepare(serve_tools=False)`): a trial's tools are served from this machine - (hermit.harbor.tools) and named to the CLI by the launch plan (hermit.harbor.launch), in its home (Claude Code's on - its command line, `--mcp-config`; mini-swe-agent's shell calls them as the `hermit-tool` command); -- an adapter whose context is per task (a `context_tool`, or one that is not plugin-portable) gets this task's - context, computed beforehand in the adapter sandbox (hermit.harbor.context) and handed to the host - (`prepare(context=...)`), as the acting agents' workspace hands it; the context tool's result is kept in the record - (`context_tool`). Codex's deployment checks the context against the CLI's limit on its AGENTS.md - (`context_limit`: the session's project_doc_max_bytes). - -The answer-only agent (`host` answer) has no workspace: its whole deployment is the adapter's context for the task, the -system text of its one model call (`context.md`). - -The staging directory holds `deploy/` (the workspace), `staging.json` (what was deployed) and the context's files. -Without an adapter nothing is staged and the CLI runs without one. -""" -from __future__ import annotations - -import argparse -import json -import os -from pathlib import Path -import subprocess -import sys - -STAGING_INFO = "staging.json" -CONTEXT_FILE = "context.md" -HOSTS = {"codex": "hermit.hosts.codex", "claude": "hermit.hosts.claude_code", "pi": "hermit.hosts.pi", - "opencode": "hermit.hosts.opencode", "dsh": "hermit.hosts.dsh", - "mini-swe-agent": "hermit.hosts.mini_swe_agent"} -HOOK_HOSTS = ("codex", "claude") # the CLIs whose hooks a deployment connects natively - - -def prepare_workspace(adapter: Path, deploy: Path, python: str, enforce: str | None = None, host: str = "codex", *, - context_file: Path | None = None, context_limit: int | None = None) -> dict: - """The host's `prepare` (`HOSTS`) in a child process under the strict adapter sandbox (`main`, run by `python`), - its tools not served; its JSON report. `context_limit` is Codex's alone.""" - from hermit.adapter.session import ENV_ENFORCE, ENV_SANDBOX - from hermit.sandbox import framework_env - env = framework_env({ENV_SANDBOX: "1", ENV_ENFORCE: enforce or os.environ.get(ENV_ENFORCE, "required")}) - flags = [*(["--context-file", str(context_file)] if context_file is not None else []), - *(["--context-limit", str(int(context_limit))] if host == "codex" and context_limit else [])] - run = subprocess.run([python, "-m", "hermit.harbor.staging", "--host", host, "--adapter", str(adapter), - "--workspace", str(deploy), "--python", python, *flags], capture_output=True, text=True, - env=env) - if run.returncode != 0: - raise RuntimeError(f"deploying the adapter failed: {run.stderr.strip()[-1500:]}") - return json.loads(run.stdout) - - -def deploy_here(host: str, adapter: Path, workspace: Path, python: str, *, context: str | None = None, - context_limit: int | None = None) -> dict: - """The host's `prepare` in this process (prepare_workspace's child): the adapter's tools not served (the runner - serves them, hermit.harbor.tools), the task's context when it is per task, and for Codex the session's limit on - its AGENTS.md (None: the host's default).""" - from importlib import import_module - module = import_module(HOSTS[host]) - extra = {"context_limit": context_limit} if host == "codex" and context_limit else {} - return module.prepare(Path(adapter), Path(workspace), python, context=context, serve_tools=False, **extra) - - -def main(argv=None) -> int: - """`python -m hermit.harbor.staging --host --adapter --workspace --python - [--context-file ] [--context-limit ]`: deploy_here, its report as JSON on stdout.""" - parser = argparse.ArgumentParser(description="One Harbor trial's adapter deployment (hermit.harbor.staging).") - parser.add_argument("--host", required=True, choices=sorted(HOSTS)) - parser.add_argument("--adapter", type=Path, required=True) - parser.add_argument("--workspace", type=Path, required=True) - parser.add_argument("--python", default=sys.executable) - parser.add_argument("--context-file", type=Path) - parser.add_argument("--context-limit", type=int) - args = parser.parse_args(argv) - context = args.context_file.read_text(encoding="utf-8") if args.context_file is not None else None - # the interpreter's own path: resolve() would follow a venv's link out of it, as the hosts' own CLIs keep it - report = deploy_here(args.host, args.adapter, args.workspace, str(Path(args.python).absolute()), context=context, - context_limit=args.context_limit) - print(json.dumps(report, ensure_ascii=False, default=str)) - return 0 - - -def container_hooks(adapter: Path, manifest: dict, deploy: Path, host: str = "codex") -> bool: - """Rewrite the deployed hooks for the container: the ones the host's deployment connects (the hook types it runs, - its HOOK_TYPES: Codex's command hooks, Claude Code's command and prompt hooks; the deployment puts a hook of - another type in the context), every command run from the adapter's copy by its workspace-relative path (Codex's - `.codex/hooks.json`; the `hooks` of Claude Code's `.claude/settings.json`, the file's other settings kept). True - when the adapter has hooks the CLI runs (Codex's and Claude Code's: the other CLIs have none, and their deployment - names the hooks' files in the context).""" - from importlib import import_module - from hermit.adapter.loader import hook_definitions - from hermit.hosts.inputs import ADAPTER_DIR, hook_commands, native_hooks - if host not in HOOK_HOSTS: - return False - hooks = native_hooks(hook_definitions(adapter, manifest), import_module(HOSTS[host]).HOOK_TYPES) - relative = hook_commands(hooks, Path(ADAPTER_DIR) / manifest["id"]) if hooks else None - if host == "claude": - # Claude Code runs a hook in the shell's current directory, which follows the agent's `cd`: start from the - # copy under the project's directory, which Claude Code names in $CLAUDE_PROJECT_DIR for its hooks - if relative is not None: - relative = _from_project(relative, f"{ADAPTER_DIR}/{manifest['id']}") - path = deploy / ".claude" / "settings.json" - settings = json.loads(path.read_text()) if path.is_file() else {} - if relative is None: - if settings.pop("hooks", None) is not None: - path.write_text(json.dumps(settings, indent=2) + "\n") - return False - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(json.dumps({**settings, "hooks": relative["hooks"]}, indent=2) + "\n") - return True - path = deploy / ".codex" / "hooks.json" - if relative is None: - path.unlink(missing_ok=True) - return False - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(json.dumps(relative, indent=2) + "\n") - return True - - -def _from_project(hooks: dict, folder: str) -> dict: - """The hooks with each command's `cd ` made `cd "${CLAUDE_PROJECT_DIR:-.}"/`.""" - import shlex - out = json.loads(json.dumps(hooks)) - plain = f"cd {shlex.quote(folder)} || exit 1\n" - for groups in out["hooks"].values(): - for hook in (h for g in groups for h in g["hooks"]): - if hook.get("type") == "command" and hook["command"].startswith(plain): - hook["command"] = f'cd "${{CLAUDE_PROJECT_DIR:-.}}"/{shlex.quote(folder)} || exit 1\n' + \ - hook["command"][len(plain):] - return out - - -def per_task(manifest: dict) -> bool: - """Whether the adapter's context is per task: a context tool, or an adapter only Hermit runs (not plugin-portable), - which no installed workspace holds (`hermit deploy` refuses it).""" - from hermit.adapter.manifest import PORTABLE - return bool(manifest.get("context_tool")) or not manifest.get(PORTABLE, True) - - -def stage(adapter: Path | None, staging: Path, python: str, enforce: str | None = None, - host: str = "codex", *, task: dict | None = None, context_query: str | None = None, - context_limit: int | None = None) -> dict | None: - """The adapter deployed for the container under `staging/deploy/`, for the acting CLI `host` (a HOSTS key, or - answer: the context alone); what was deployed, or None without an adapter. `task` is the row: an adapter whose - context is per task gets the task's (`context_query`: the row's field it retrieves for, else its instruction). - `context_limit` is Codex's cap on its AGENTS.md (project_doc_max_bytes).""" - if adapter is None: - return None - from hermit.adapter.loader import agent_tools, load_manifest - from hermit.harbor.context import compute - if host not in HOSTS and host != "answer": - raise ValueError(f"no host deploys an adapter for {host!r}: one of {', '.join(HOSTS)} or answer") - adapter = Path(adapter).resolve() - manifest = load_manifest(adapter) - tools = sorted(t["name"] for t in agent_tools(manifest)) - out: dict = {"host": host, "adapter_id": manifest["id"], "version": manifest.get("version"), - "adapter_dir": "hermit/" + str(manifest["id"]), "tools": tools} - if host == "answer": - # one model call, no workspace: the context is the whole deployment (ADAPTER.md, and the context tool's text) - text, record = compute(adapter, dict(task or {}), staging, python=python, enforce=enforce, - query_field=context_query) - (staging / CONTEXT_FILE).write_text(text, encoding="utf-8") - out.update(context_file=str(staging / CONTEXT_FILE), context_tool=record, roles=[], skills=[], hooks=False, - tools=[], tools_unused=tools, deploy=None) - (staging / STAGING_INFO).write_text(json.dumps(out, indent=1, default=str) + "\n") - return out - context_file, record = None, None - if per_task(manifest): - text, record = compute(adapter, dict(task or {}), staging, python=python, enforce=enforce, - query_field=context_query) - context_file = staging / CONTEXT_FILE - context_file.write_text(text, encoding="utf-8") - deploy = staging / "deploy" - info = prepare_workspace(adapter, deploy, python, enforce, host, context_file=context_file, - context_limit=context_limit) - hooks = container_hooks(adapter, manifest, deploy, host) - out.update(adapter_id=info.get("adapter_id") or manifest["id"], version=info.get("version") or out["version"], - roles=sorted(info.get("roles") or {}), skills=list(info.get("skills") or []), hooks=hooks, - patch=info.get("patch"), server_name=info.get("server_name"), - context_file=str(context_file) if context_file else None, context_tool=record, deploy=str(deploy)) - (staging / STAGING_INFO).write_text(json.dumps(out, indent=1, default=str) + "\n") - return out - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/hermit/harbor/stream.py b/hermit/harbor/stream.py deleted file mode 100644 index 0c60f1ca..00000000 --- a/hermit/harbor/stream.py +++ /dev/null @@ -1,375 +0,0 @@ -"""The host's side of a stream: one driver process per multi-step task, asked for one step at a time. - -The driver (`harborside/driver.py`, under Harbor's interpreter) holds the task's environment open between steps -and answers on a Unix socket in the stream's directory; this client starts it when there is none (under a file -lock, so concurrent rollouts of one stream start one driver), waits for its turn (the driver runs steps in order), -asks for its step and reads the step's result back. A driver that answers no stop (hung in a step) is ended with its -process group, and its trial's compose projects are taken down by name, as an expired trial's are. The stream's -directory: - - /streams// - driver.json {pid, socket, name, steps, started, trial_dir}: the live driver (name: Harbor's trial; - steps: the task's step names, which name each step's separate verifier environment) - driver.log its log (Harbor's trial log is in trial_dir) - .lock the start lock - steps/--/ a rollout of this step: its staging (deploy/), launch plan (launch/) and curated - log, each rollout in a directory of its own - trials// Harbor's trial directory: steps//{agent,verifier,result.json}, result.json at the end - replies/.json the driver's answer for each step it ran - earlier// an earlier driver's trials/ and replies/, set aside when a new driver starts here (Harbor's - trial of the same name would reuse its step directories: a step that leaves no output of - its own would be read with the earlier run's); driver.log goes on, each start headed - -Protocol: one JSON object per line, one request per connection. - {"op": "status"} -> {"ok": true, "ready": bool, "next_index": i, "running": name|null, - "finished": bool, "aborted": reason|null, "error": msg|null} - {"op": "step", "name", "index", "plan", "agent_kwargs", "allowed_hosts"} (plan: the step's launch plan, - hermit.harbor.launch; allowed_hosts: the hosts the step's agent phase lets - the CLI reach, Harbor's extra_allowed_hosts, set for each step) - -> {"ok": true, "step": , "paths": {...}, "aborted": ...} - | {"ok": false, "out_of_order": true, "next_index": i} | {"ok": false, "error"} - {"op": "result", "index": i} -> the stored reply of a finished step, or {"ok": false, "error"} - {"op": "stop"} -> {"ok": true} -""" -from __future__ import annotations - -from datetime import datetime -import fcntl -import json -import os -from pathlib import Path -import signal -import socket -import subprocess -import sys -import time - -DRIVER_INFO = "driver.json" -DRIVER_LOG = "driver.log" -EARLIER = "earlier" # earlier drivers' trials and replies, set aside -POLL_S = 5.0 -TURN_GRACE_POLLS = 2 # a missing earlier step is waited for this many polls more -START_TIMEOUT_S = 3600.0 # the environment's build and the agent's install -ENDING_WAIT_S = 30.0 # an ending driver's exit, waited for before the next starts -STOP_TIMEOUT_S = 900.0 # a stop's answer: the environment down, the result written -HARBORSIDE = Path(__file__).resolve().parent / "harborside" - - -class StreamError(RuntimeError): - pass - - -class StreamOrderError(StreamError): - """The step's turn did not come: an earlier step is missing, or this one was already run.""" - - -class StepAlreadyRunError(StreamOrderError): - """The step's turn has passed: the driver is past it (a second rollout of a step the stream already ran, as an - Evolver's run_agent of a training step whose stream went on). The stream is not this rollout's to give up.""" - - -class TurnLostError(StreamOrderError): - """The step's turn cannot come: no rollout runs the step its driver waits for, or the stream has no driver and no - rollout of its first step is running to start one (`why`).""" - - def __init__(self, index: int, why: str): - self.why = why - super().__init__(f"step {index}'s turn cannot come: {why}; a stream's steps run in order, each after the one " - "before it") - - -def socket_path(stream_dir: Path) -> Path: - """Where the driver listens: a short path (AF_UNIX allows 107 bytes) under the user's runtime directory, named - by a digest of the stream directory. Never /tmp: a sandboxed caller has a private one.""" - import hashlib - name = hashlib.sha1(str(Path(stream_dir).resolve()).encode()).hexdigest()[:16] + ".sock" - bases = [] - if os.environ.get("XDG_RUNTIME_DIR") and Path(os.environ["XDG_RUNTIME_DIR"]).is_dir(): - bases.append(Path(os.environ["XDG_RUNTIME_DIR"])) - bases += [Path.home() / ".cache" / "hermit" / "harbor", Path("/tmp") / f"hermit-harbor-{os.getuid()}"] - for base in bases: - path = base / "hermit-harbor-sockets" / name - if len(str(path).encode()) <= 100: - path.parent.mkdir(parents=True, exist_ok=True) - return path - raise StreamError(f"no short enough path for the stream's socket (tried {[str(b) for b in bases]})") - - -def alive(pid: int) -> bool: - try: - os.kill(pid, 0) - except ProcessLookupError: - return False - except PermissionError: - return True - return True - - -class StreamClient: - def __init__(self, stream_dir: Path, *, python: str, log=None): - self.dir = Path(stream_dir) - self.python = python # Harbor's venv interpreter - self.log = log or (lambda m: print(m, file=sys.stderr, flush=True)) - - # --- the driver process ----------------------------------------------------------------------------------- - @property - def info_path(self) -> Path: - return self.dir / DRIVER_INFO - - def info(self) -> dict | None: - try: - return json.loads(self.info_path.read_text()) - except (OSError, ValueError): - return None - - def running(self) -> dict | None: - """The live driver's record, or None when there is none: no record, its process gone (a zombie too: gone), its - socket gone, or a driver that is ending (its record's `ending`: its step loop over, its socket about to close), - which no request may use any more and whose stream's next pass starts a new driver (ensure waits it out).""" - info = self.info() - if info and not info.get("ending") and alive(int(info.get("pid", 0))) and \ - Path(info.get("socket", "")).exists() and not self.gone(): - return info - return None - - def gone(self) -> bool: - """Whether the driver's process has ended: no record, no such process, or a zombie (a driver this process - started and that died stays one until it is reaped, and `alive` still finds it): no request reaches it again.""" - info = self.info() - if not info: - return True - pid = int(info.get("pid") or 0) - if pid <= 0: - return False - if not alive(pid): - return True - try: - return Path(f"/proc/{pid}/stat").read_text().rsplit(")", 1)[1].split()[0] == "Z" - except (OSError, IndexError): - return False # no /proc here: alive's answer - - def ensure(self, *, task_dir: Path, name: str, agent: dict, environment: dict, verifier: dict | None = None, - idle_s: float = 4 * 3600, env: dict | None = None, start_timeout_s: float = START_TIMEOUT_S, - trial: dict | None = None) -> dict: - """The live driver, started if there is none, ready (environment up, agent installed) or raising with the - error it reported. A driver it starts runs with `env` as it is (this process's environment when none is - given), and `trial` holds the settings of Harbor's trial config (its timeout multipliers).""" - self.dir.mkdir(parents=True, exist_ok=True) - if os.environ.get("HA_ADAPTER_SANDBOX") == "1" and self.running() is None: - # A driver started from a sandboxed worker would die with it (bubblewrap's --die-with-parent), and its - # socket would sit in a private /tmp. A Harbor task set's pipeline runs on the host (its environment is - # Harbor's), so this is a task's own agent driving a stream from inside the task sandbox. - raise StreamError("a stream's driver cannot be started from inside the task sandbox: run the task set " - "with agent.name: harbor, whose pipeline runs on the host") - with open(self.dir / ".lock", "w") as lock: - fcntl.flock(lock, fcntl.LOCK_EX) - try: - info = self.running() - if info is None: - self._wait_out() - info = self._start(task_dir=task_dir, name=name, agent=agent, environment=environment, - verifier=verifier or {}, idle_s=idle_s, env=env, trial=trial or {}) - finally: - fcntl.flock(lock, fcntl.LOCK_UN) - self._wait_ready(info, start_timeout_s) - return info - - def _wait_out(self, timeout_s: float | None = None) -> None: - """An earlier driver of this directory that `running` no longer takes but whose process is still there (one - ending: its step loop over, its socket still open; one whose socket is gone) waited for until its process has - ended (ENDING_WAIT_S at most), then killed if it has not: a new driver never binds the socket's path while the - old one is still to remove it at its exit, nor shares its trial's name with a process still using it.""" - if self.gone(): - return - deadline = time.time() + (ENDING_WAIT_S if timeout_s is None else timeout_s) - while time.time() < deadline and not self.gone(): - time.sleep(0.1) - if not self.gone(): - self.log(f"the stream's earlier driver had not ended {ENDING_WAIT_S:.0f} s after it stopped answering: " - "killing it") - self.kill() - - def _start(self, *, task_dir: Path, name: str, agent: dict, environment: dict, verifier: dict, idle_s: float, - env: dict | None, trial: dict) -> dict: - sock = socket_path(self.dir) - sock.unlink(missing_ok=True) - self.info_path.unlink(missing_ok=True) - self._set_aside() - cmd = [self.python, "-m", "harborside.driver", "--task-dir", str(task_dir), "--stream-dir", str(self.dir), - "--name", name, "--socket", str(sock), "--agent-json", json.dumps(agent), - "--environment-json", json.dumps(environment), "--verifier-json", json.dumps(verifier), - "--trial-json", json.dumps(trial), "--idle-s", str(idle_s)] - # The environment as given (harbor_env: what Harbor's processes may have), never this process's back under it - run_env = dict(os.environ if env is None else env) - run_env["PYTHONPATH"] = str(HARBORSIDE.parent) + (os.pathsep + run_env["PYTHONPATH"] - if run_env.get("PYTHONPATH") else "") - with open(self.dir / DRIVER_LOG, "a") as log: - log.write(f"=== {datetime.now().isoformat(timespec='seconds')} start: {' '.join(cmd)}\n") - log.flush() - proc = subprocess.Popen(cmd, env=run_env, stdout=log, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL, - start_new_session=True, cwd=str(self.dir)) - deadline = time.time() + 120 - while time.time() < deadline: - info = self.info() - if info and Path(info.get("socket", "")).exists(): - return info - if proc.poll() is not None: - raise StreamError(f"the stream driver exited with {proc.returncode} before it listened; see " - f"{self.dir / DRIVER_LOG}") - time.sleep(0.5) - raise StreamError(f"the stream driver did not listen within 120 s; see {self.dir / DRIVER_LOG}") - - def _set_aside(self) -> None: - """What an earlier driver of this directory left, moved to earlier// before a new one starts (under the start - lock): its trial directory, whose step directories Harbor's next trial of the same name would reuse (a step - that leaves no output of its own would be read with the earlier run's: its usage billed again, its reply and - outputs taken for the step's), and its replies. driver.log, which every step's curated log copies, stays: each - driver's part of it begins with the line its start writes.""" - left = [self.dir / name for name in ("trials", "replies") if os.path.lexists(self.dir / name)] - if not left: - return - taken = [int(p.name) for p in (self.dir / EARLIER).glob("*") if p.name.isdigit()] - place = self.dir / EARLIER / str(max(taken, default=0) + 1) - place.mkdir(parents=True) - for path in left: - path.rename(place / path.name) - - def _wait_ready(self, info: dict, timeout_s: float) -> None: - deadline = time.time() + timeout_s - while True: - status = self.request({"op": "status"}, timeout=60) - if status.get("error"): - raise StreamError(f"the stream's environment failed: {status['error']}") - if status.get("ready"): - return - if status.get("finished"): - raise StreamError("the stream driver finished before this step") - if time.time() >= deadline or not alive(int(info["pid"])): - raise StreamError(f"the stream's environment was not ready within {timeout_s:.0f} s; see " - f"{self.dir / DRIVER_LOG}") - time.sleep(POLL_S) - - # --- requests --------------------------------------------------------------------------------------------- - def request(self, obj: dict, *, timeout: float | None) -> dict: - info = self.info() - if not info: - raise StreamError("no stream driver record") - with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as s: - s.settimeout(30) - s.connect(info["socket"]) - s.sendall((json.dumps(obj) + "\n").encode()) - s.settimeout(timeout) - chunks = [] - while True: - chunk = s.recv(1 << 16) - if not chunk: - break - chunks.append(chunk) - if chunk.endswith(b"\n"): - break - data = b"".join(chunks).decode("utf-8", errors="replace").strip() - if not data: - raise StreamError("the stream driver closed the connection without an answer") - try: - return json.loads(data) - except ValueError: # cut off: a driver killed while it wrote its answer - raise StreamError("the stream driver closed the connection before its answer was whole") from None - - def status(self) -> dict: - return self.request({"op": "status"}, timeout=60) - - def wait_turn(self, index: int, *, timeout_s: float, missing=None) -> dict: - """Block until the driver's next step is `index` (earlier steps of the stream, running in other rollouts, - come first); the status then. Raises when the stream finished, aborted or died first (an aborted stream's - driver refuses every step, the step at its turn too, so that step is never asked for), and when `missing` - (called with the step the driver waits for, while it is idle at an earlier one) says why no rollout will ask - for that step, on TURN_GRACE_POLLS polls more in a row (by then a rollout submitted just before this one has - come): a turn that cannot come ends at once (TurnLostError), not at the deadline.""" - deadline = time.time() + timeout_s - missed = 0 - while True: - status = self.status() - if status.get("error"): - raise StreamError(f"the stream's environment failed: {status['error']}") - nxt = int(status.get("next_index") or 0) - if nxt > index: - raise StepAlreadyRunError(f"step {index} was already run (the driver is at step {nxt})") - if status.get("finished") or status.get("aborted"): # before its turn: the driver would refuse the step - raise StreamError(f"the stream {'aborted: ' + str(status.get('aborted')) if status.get('aborted') else 'finished'} " - f"before step {index}") - if nxt == index and not status.get("running"): - return status - why = missing(nxt) if missing is not None and not status.get("running") else None - missed = missed + 1 if why else 0 - if missed > TURN_GRACE_POLLS: - raise TurnLostError(index, why) - if time.time() >= deadline: - raise StreamOrderError(f"waited {timeout_s:.0f} s for step {index}'s turn (the driver is at step {nxt}): " - "a stream's steps run in order, each after the one before it") - time.sleep(POLL_S) - - def step(self, *, name: str, index: int, plan: str | None, agent_kwargs: dict | None, - timeout_s: float, allowed_hosts: list[str] | None = None) -> dict: - try: - return self.request({"op": "step", "name": name, "index": index, "plan": plan, - "agent_kwargs": agent_kwargs or {}, "allowed_hosts": list(allowed_hosts or [])}, - timeout=timeout_s) - except (socket.timeout, TimeoutError): - return {"ok": False, "expired": True, "error": f"no answer from the stream driver within {timeout_s:.0f} s"} - - def result(self, index: int) -> dict: - return self.request({"op": "result", "index": index}, timeout=60) - - def stop(self, *, timeout_s: float | None = None) -> None: - """Ask the driver to stop (its step loop ends, the environment goes down, the trial's result is written) and - wait for its answer. A driver that gives none within `timeout_s` (STOP_TIMEOUT_S) is hung in a step, which the - stop would wait behind (one whose request expired), and neither its idle stop nor a later stop would end it: - it is killed (kill).""" - try: - self.request({"op": "stop"}, timeout=STOP_TIMEOUT_S if timeout_s is None else timeout_s) - except TimeoutError as exc: - killed = self.kill() - self.log(f"stopping the stream driver: no answer ({exc}); " - + ("killed it, its compose projects taken down" if killed else "no process of it to kill")) - except (OSError, StreamError, ValueError) as exc: - self.log(f"stopping the stream driver: {exc}") - - def kill(self) -> bool: - """End the driver's process group (it leads one: started in a session of its own), SIGTERM then SIGKILL, as - an expired trial's `harbor run` is ended, and take its trial's compose projects down by name (its environment, - which its own stop would have taken down: docker.compose_down); whether it did. Only a live process whose - command line names this stream's directory is signalled: the pid its record gives may since be another - program's.""" - pid = self._driver_pid() - if pid is None: - return False - for sig, grace in ((signal.SIGTERM, 30.0), (signal.SIGKILL, 10.0)): - try: - os.killpg(pid, sig) - except (ProcessLookupError, PermissionError): - break - deadline = time.time() + grace - while time.time() < deadline and not self.gone(): - time.sleep(0.2) - if self.gone(): # its socket, which a killed driver leaves (its own exit removes it) - socket_path(self.dir).unlink(missing_ok=True) - info = self.info() or {} - if info.get("name"): - from hermit.harbor import docker - docker.compose_down(str(info["name"]), keys=[str(s) for s in info.get("steps") or []]) - return True - - def _driver_pid(self) -> int | None: - """The pid of this directory's live driver, when its command line (/proc) says it is one and it leads its own - process group; None otherwise (no record, a process gone, one of another program, no /proc to tell).""" - pid = int((self.info() or {}).get("pid") or 0) - if pid <= 0 or self.gone(): - return None - try: - argv = Path(f"/proc/{pid}/cmdline").read_bytes().split(b"\0") - leads = os.getpgid(pid) == pid - except OSError: - return None - where = argv.index(b"--stream-dir") + 1 if b"--stream-dir" in argv else len(argv) - named = where < len(argv) and Path(os.fsdecode(argv[where])).resolve() == self.dir.resolve() - return pid if leads and named and b"harborside.driver" in argv else None diff --git a/hermit/harbor/tools.py b/hermit/harbor/tools.py deleted file mode 100644 index dbe3bfcc..00000000 --- a/hermit/harbor/tools.py +++ /dev/null @@ -1,449 +0,0 @@ -"""What a trial's agent reaches on this machine: the adapter's tools, and the answer-only agent's model. - -`AdapterTools`: an adapter's agent tools for a CLI in a task container, served from here as Hermit's own sessions in -a container get them (hermit.agent.base.cli: a ToolServer on the container network's gateway): a Session of the -adapter in the strict adapter sandbox (the adapter's code never runs in this process, nor with its keys or network), -its tools behind a ToolServer listening on the Docker bridge's gateway address (`docker network inspect bridge`; -a container on a fresh compose network reaches the host there, checked 2026-11-11), the URL under a random token, each -call in the audit log (`tool-calls.jsonl` in the trial's job directory, which the curated log keeps). The launch plan -names the URL in the CLI's own MCP entry. One server per trial, closed after it; a stream keeps one server for the -stream's life and swaps the Session behind it at each step, whose adapter may have changed (`swap`). - -`ModelEndpoint`: the answer-only agent's model, a ChatEndpoint on 127.0.0.1 (the agent runs in Harbor's own process on -this machine) over one model call each (Direct, Layer 1) of the model's spec: a relay that sends the agent's request on -with the backend's key, which never leaves this process; or, for a model only a CLI reaches that the runner serves (the -benchmark file's `model: {cli: ...}`), over that CLI's sessions (cli_endpoint's Base Agent, each session in a sandbox of -its own), each request answered here as the endpoint answers it (cli_answer), with three things of the relay's own: on -the model's login it waits while the login's plan holds sessions back before the session starts, and records why -(`plan`: the call's `waited`, which the trial's facts read); what a session cannot take (contract_problems: a figure, -tool calls, another model; settings it cannot start with) is refused with the relay's own code (UNSUPPORTED), which the -trial's facts read as the run's configuration refused; and what a call used is read from its reply, a failed one's error -body included (a CLI session's failure reports its usage there: endpoint._no_reply). It counts each call from its -arrival, what it used (`usage`, priced at the spec's price, an explicit price included) and how it was answered -(`statuses`, `facts`). Closed after the trial (or the step), it gives up the calls still in flight (an agent Harbor -stopped leaves its request running), whose tokens would otherwise be spent after the trial's usage was read, and waits -for them up to CLOSE_WAIT_S. -""" -from __future__ import annotations - -import dataclasses -import json -import os -import subprocess -import threading -from pathlib import Path - -from hermit.agent.base.endpoint import ChatEndpoint - -_GATEWAY: dict[str, str] = {} -_LOCK = threading.Lock() -CLOSE_WAIT_S = 15.0 # how long a closing ModelEndpoint waits for the calls it gave up to end -# the error code of a request the relay refused because a CLI session cannot take it (ModelEndpoint), which a trial's -# facts read as the run's configuration refused (hermit.harbor.runner.config_refused) -UNSUPPORTED = "hermit_unsupported" - - -def docker_gateway(network: str = "bridge") -> str: - """The gateway address of a Docker network (the host's address on it), asked once per process.""" - with _LOCK: - if network not in _GATEWAY: - try: - out = subprocess.run(["docker", "network", "inspect", network, "--format", - "{{range .IPAM.Config}}{{.Gateway}} {{end}}"], capture_output=True, text=True, - timeout=60) - except (OSError, subprocess.SubprocessError) as exc: - raise RuntimeError(f"hermit: the Docker network {network} could not be inspected for its gateway " - f"({exc}): the adapter's tools are served on it") from None - found = [a for a in out.stdout.split() if a.count(".") == 3] - if out.returncode != 0 or not found: - raise RuntimeError(f"hermit: the Docker network {network} has no IPv4 gateway " - f"({(out.stderr or out.stdout).strip()[-300:]}): the adapter's tools are served on " - "it (or set agent.params.tools_host)") - _GATEWAY[network] = found[0] - return _GATEWAY[network] - - -def adapter_sandbox(enforce: str | None = None): - """The strict adapter sandbox of a Session (hermit.adapter.session), with the run's sandbox.enforce.""" - from hermit.adapter.session import _adapter_sandbox - box = _adapter_sandbox() - return dataclasses.replace(box, enforce=enforce) if enforce else box - - -class AdapterTools: - """An adapter's agent tools served to a trial's container (the module's docstring).""" - - def __init__(self, adapter: Path, *, audit_log: Path, host: str | None = None, enforce: str | None = None): - self.host = host or docker_gateway() - self.enforce = enforce - self.server = None - self.session = self.sandbox = None - self.adapter, self.audit_log = Path(adapter), Path(audit_log) - - def _open(self, adapter: Path, audit_log: Path): - from hermit.adapter.session import Session - box = adapter_sandbox(self.enforce) - try: - session = Session(adapter, sandbox=box, audit_log=audit_log) - except BaseException: - box.close() - raise - return session, box - - def start(self) -> "AdapterTools": - from hermit.agent.base.tool_server import ToolServer - self.session, self.sandbox = self._open(self.adapter, self.audit_log) - try: - self.server = ToolServer(self.session.as_tools(), host=self.host).start() - except BaseException: - self._close_session(self.session, self.sandbox) - raise - return self - - @property - def mcp_url(self) -> str: - return self.server.mcp_url - - @property - def names(self) -> list[str]: - return sorted(self.server.tools) - - def swap(self, adapter: Path, audit_log: Path) -> None: - """The tools of another adapter (a stream's next step) behind the same URL: the server's tools replaced in - place (its handlers read them at each call), the old Session closed.""" - if self.server is None: - raise RuntimeError("hermit: the stream's tool server was closed (its stream stopped)") - session, box = self._open(Path(adapter), Path(audit_log)) - tools = {t.name: t for t in session.as_tools()} - old = (self.session, self.sandbox) - self.server.tools.clear() - self.server.tools.update(tools) - self.server.mcp_tools.clear() - self.server.mcp_tools.update(tools) - self.session, self.sandbox, self.adapter, self.audit_log = session, box, Path(adapter), Path(audit_log) - self._close_session(*old) - - @staticmethod - def _close_session(session, box) -> None: - try: - if session is not None: - session.close() - finally: - if box is not None: - box.close() - - def close(self) -> None: - try: - if self.server is not None: - self.server.close() - finally: - self.server = None - self._close_session(self.session, self.sandbox) - self.session = self.sandbox = None - - -def _add(total, used: list[dict]) -> None: - """The chat-completions usage of each reply in `used` added into `total` (hermit.agent.model.usage.Usage).""" - for u in used: - total.input_tokens += int(u.get("prompt_tokens") or 0) - total.output_tokens += int(u.get("completion_tokens") or 0) - total.cached_input_tokens += int((u.get("prompt_tokens_details") or {}).get("cached_tokens") or 0) - - -class ModelEndpoint: - """The answer-only agent's model on 127.0.0.1 (the module's docstring). `spec` is a model spec (a Binding's: the - a model call (Direct) relays each request to the backend, with the spec's client `timeout` and `max_retries`), or a - model only a CLI reaches on its login (`{cli: codex | claude, model, login, ...}`: each request one session of - that CLI, in a sandbox of its own, hermit.isolate.cli_sandbox, as a run's `llm: {cli: ...}`; its event streams kept - under `records`). `plan`: (the plan, whether its sessions may draw usage credits) of the login that CLI's sessions - run on (hermit.harbor.runner.HarborAgent.model_plan), which the relay waits for before each session (`facts`).""" - - def __init__(self, spec: dict, *, host: str = "127.0.0.1", records: Path | None = None, - enforce: str | None = None, plan: tuple | None = None): - self.spec, self.host = dict(spec), host - self.records = Path(records) if records is not None else None - self.enforce = enforce - self.plan = tuple(plan) if plan is not None else None - self.plan_notes = self.plan is not None - self.client = self.endpoint = None - self.lock = threading.Condition() # the calls' records; notified as each call ends - self.running: set = set() # the calls in flight, by their own abort event - self.aborted = 0 # calls given up before their reply - self.closing = False - self.statuses: list = [] - self.codes: list = [] - self.messages: list = [] # each call's error message (its last 1000 characters) - self.used: list[dict] = [] # each call's usage its reply reported (any status) - self.failed_used: list[dict] = [] # ... of a relayed call that failed (not in the client's) - self.notes: list[dict] = [] # each call's waits for its plan, by arrival (plan) - - def start(self) -> "ModelEndpoint": - if self.spec.get("cli"): - from hermit.agent.base.endpoint import cli_endpoint - from hermit.isolate import HOME_SECRETS, cli_sandbox - # the isolation check proves the user's own secrets absent, as for the pipeline's sessions of such a model - # (hermit.isolate.agent_pipeline); the run's own secret paths are the config's, which the runner never gets - box, proxy = cli_sandbox(self.spec, records=self.records, enforce=self.enforce or "required", - role="agent-model", probe_absent=HOME_SECRETS) - try: - # a session the plan refused comes back at once (429: `refused`, the trial infra, run again once the - # plan allows: HarborAgent._model_plan_wait), never run again inside the request (main's CliAgent.run - # does so by default, plan_rerun), which would wait for the plan's window inside the agent's time and - # end the trial as the agent's timeout - # a Codex session asks for web_search set either way; the answer agent's takes none unless given - endpoint = cli_endpoint({"web_search": False, **self.spec, "plan_rerun": False}, - self.records / "model-sessions" if self.records else None, sandbox=box) - except BaseException: - for thing in (box, proxy): - if thing is not None: - thing.close() - raise - endpoint.closing += [t for t in (box, proxy) if t is not None] - answer = self._cli_answer(endpoint.agent, endpoint.place) - else: - from hermit.agent.model.direct import Direct - from hermit.agent.model.llm import ChatClient - # its own retries for a provider that could not serve (the spec's max_retries, when it gives them) - given = self.spec.get("max_retries") - self.client = ChatClient.from_spec(self.spec, **({} if given is None else {"max_retries": int(given)})) - endpoint = ChatEndpoint(Direct({}, self.client), host=self.host).start() - relayed = endpoint.answer - answer = lambda request, authorization, abort, note: relayed(request, authorization, abort) # noqa: E731 - - def metered(request, authorization, abort=None): - # each call from its arrival (in flight under its own abort event, which close sets), then its status, a - # failed one's error code and message; with a plan, the relay's record of its wait for the plan - abort = abort if abort is not None else threading.Event() - mine = {} if self.plan_notes else None - with self.lock: - self.running.add(abort) - if mine is not None: - self.notes.append(mine) - if self.closing: # it came as the endpoint closed: given up at once - abort.set() - reply = None - try: - reply = answer(request, authorization, abort, mine) - finally: - self._note(reply, abort) - return reply - endpoint.answer = metered - self.endpoint = endpoint - return self - - def _cli_answer(self, agent, place=None): - """The answer of each request to a CLI's model (hermit.agent.base.endpoint.cli_answer, as the endpoint's own - answer gives it) with the relay's three things: the plan's wait first (`plan`; `note` records why it waited, - `waited`, and while it waits, `waiting`; a call given up while it waits, the endpoint closing, gets no reply - and stays `waiting`, as a session cancelled gets none: endpoint._DROP), a request a session cannot take refused - with UNSUPPORTED, and the settings a session cannot start with (ValueError, Unsupported) refused the same way; - any other failure of the session's start is a 502, as the endpoint's. A Codex session that gave no reply and - whose stream says its sign-in or key was refused (signin.codex_signin_lost, which the Base Agent's Codex - reading does not report) is a 401 with LOGIN_FAILED, as the endpoint says a Claude session's: the runner then - ends the run (no later call on that sign-in can be served). One whose stream says the provider refused the - request for good (signin.codex_rejected: a 400, 402 or 404 Codex does not retry, a provider's error body, - Codex's words for a spent quota or a plan that refuses every session), which that reading leaves a 502 (the - provider's, which the runner runs again), is a 400 whose message ends with Codex's words, as a trial's Codex - session is read: a refusal of the run's configuration or bill, or else the agent's result (a prompt too - long).""" - from hermit.agent.base.endpoint import _DROP, cli_answer, contract_problems - from hermit.agent.base.interface import LOGIN_FAILED, Unsupported - from hermit.harbor import signin - folder = agent.settings.get("log_dir") if getattr(agent, "backend", None) == "codex" else None - - def refused(why: str): - return 400, json.dumps({"error": {"code": UNSUPPORTED, "message": why}}).encode(), {} - - def streams() -> set: - return set(os.listdir(folder)) if folder and os.path.isdir(folder) else set() - - def said(before: set) -> tuple[bool, str | None]: - # (a lost sign-in, a refusal for good) in the streams this call's sessions left in the agent's log_dir - # (endpoint.cli_answer keeps each there) - refusal = None - for name in sorted(streams() - before): - path = Path(folder) / name - if not name.endswith(".jsonl") or path.is_symlink() or not path.is_file(): - continue - try: - raw = path.read_text(encoding="utf-8", errors="replace") - except OSError: - continue - raw = raw.replace("\r\n", "\n").replace("\r", "\n") - if signin.codex_signin_lost(raw): - return True, None - refusal = signin.codex_rejected(raw) or refusal - return False, refusal - - def checked(reply, before: set): - if not isinstance(reply, tuple) or reply[0] == 200 or folder is None: - return reply - lost, refusal = said(before) - if not lost and not (refusal and reply[0] == 502): - return reply - try: - body = json.loads(reply[1]) - except (ValueError, TypeError, RecursionError): - body = {} - body = body if isinstance(body, dict) else {} - error = body.get("error") if isinstance(body.get("error"), dict) else {} - if lost: - # the code first, as the endpoint writes it: a client's error text keeps only the start of the body - body["error"] = {"code": LOGIN_FAILED, **{k: v for k, v in error.items() if k != "code"}} - return 401, json.dumps(body).encode(), reply[2] - # no retry changes it: the message ends with Codex's words, which the runner reads (_note keeps the end) - why = str(error.get("message") or "the codex session gave no reply") - body["error"] = {**{k: v for k, v in error.items() if k not in ("code", "message")}, - "message": f"{why}: {refusal}"} - return 400, json.dumps(body).encode(), reply[2] - - def answer(request, authorization, abort, note): - problems, _ = contract_problems(agent, request) - if problems: - return refused("; ".join(problems)) - if self.plan is not None: - from hermit.agent.base import login - plan, credits = self.plan[0], bool(self.plan[1]) - why = login.held(plan, credits) - if why: - note.update(waited=why, waiting=True) - max_s = agent.settings.get("plan_max_wait_s", login.MAX_WAIT_S) - got = login.wait(plan, abort, None, max_s, credits) - if got == "cancelled": # given up as it waited (the endpoint closed) - return _DROP - note["waiting"] = False - if got: # past plan_max_wait_s - return 503, json.dumps({"error": {"message": f"the login's plan held the session back " - f"({why}): {got}"}}).encode(), {} - before = streams() - try: - return checked(cli_answer(agent, request, abort, place=place), before) - except (ValueError, Unsupported) as exc: # the session's settings: no session can take them - return refused(f"{type(exc).__name__}: {exc}") - except Exception as exc: # noqa: BLE001 - the session failed to run - return 502, json.dumps({"error": {"message": f"{type(exc).__name__}: {exc}"}}).encode(), {} - return answer - - def _note(self, reply, call: threading.Event | None = None) -> None: - """A call's end: its reply, and the call (its abort event) no longer in flight; one given up before its reply - (no reply while its abort was set) is counted apart, its use unknown.""" - status = reply[0] if isinstance(reply, tuple) else None # None: no answer (the upstream's) - code, used, message = None, None, None - if isinstance(reply, tuple): - try: - body = json.loads(reply[1]) - except (ValueError, TypeError, RecursionError): # no JSON, or nested past the parser's depth - body = {} - body = body if isinstance(body, dict) else {} - error = body.get("error") - code = error.get("code") if status != 200 and isinstance(error, dict) else None - if code is None and status != 200 and isinstance(error, dict) and error.get("type") == "insufficient_quota": - code = "insufficient_quota" # a spent quota some providers name by its type alone - if status != 200: # why it failed: a CLI session's 400 says it only here (endpoint._no_reply); its - # last 1000 characters: the whole of what that reply carries ("the session gave no reply (exit - # 1): " and the last 600 characters of the CLI's failure, whose leading status stays readable), the - # end of a longer message. A body whose error has no message may say it at its top level (Mistral's - # `message`, a FastAPI server's `detail`) - said = error.get("message") if isinstance(error, dict) else error - if not said: - said = next((body[k] for k in ("message", "detail") if isinstance(body.get(k), str) and body[k]), - None) - message = str(said)[-1000:] if said else None - used = body.get("usage") if isinstance(body.get("usage"), dict) else None - with self.lock: - self.statuses.append(status) - self.codes.append(code) - self.messages.append(message) - if used is not None: - self.used.append(used) - if status != 200: # the relay's client counts a served call's alone - self.failed_used.append(used) - if call is not None: - self.running.discard(call) - self.aborted += int(call.is_set() and not isinstance(reply, tuple)) - self.lock.notify_all() - - @property - def url(self) -> str: - return self.endpoint.url - - @property - def usage(self): - """What the calls used (hermit.agent.model.usage.Usage): the relay's client's total (its served calls), with what - a failed call's error body reported, each such call counted (a CLI session's failure behind the pipeline's - endpoint reports its usage there; of a call the client retried, the last attempt's alone: the client keeps no - failed attempt's), priced at the spec's price; a CLI's sessions' replies, - a failed one's too, priced the same way (none on a login). A call given up before its reply, or still in flight - when the endpoint closed, is among its calls as one whose tokens no reply reported (`unreported_calls`): the - cost is then unknown (None), never what the other calls cost.""" - from hermit.agent.base.records import price_usage - from hermit.agent.model.usage import Usage - with self.lock: - used, failed, uncounted = list(self.used), list(self.failed_used), self.aborted + len(self.running) - if self.client is not None: - total = dataclasses.replace(self.client.total) - if failed: # calls the client counted none of - total.calls += len(failed) - spent = Usage(model=total.model) - _add(spent, failed) - price_usage([spent], self.spec) - total.input_tokens += spent.input_tokens - total.output_tokens += spent.output_tokens - total.cached_input_tokens += spent.cached_input_tokens - total.cost_usd = None if total.cost_usd is None or spent.cost_usd is None else \ - total.cost_usd + spent.cost_usd - else: - total = Usage(model=str(self.spec.get("model") or self.spec.get("cli"))) - _add(total, used) - total.calls = len(used) - total.calls += uncounted - total.unreported_calls += uncounted - if self.client is None: - price_usage([total], self.spec) - if uncounted: - total.cost_usd = None - return total - - def facts(self) -> dict: - """The session's facts, as a CLI's stream gives them: served (a call answered), unavailable (the last call got - no answer, or a status a retry may change), refused (a CLI session the plan refused: 429), login_failed (a - CLI-reached model's sign-in lost: the endpoint's code), rejected (a status no retry changes), code and failure - (the last call's error code and message); started: a call was made, counted from its arrival (a request still - in flight when the agent's time ran out was made); usage_incomplete: a call was given up before its reply, or - was still in flight when the endpoint closed, so what it used is not in `usage`; with a plan, plan_waited (why - the plan held the last call back, as the relay met it before its session: login.held; "" when it never waited) - and plan_waiting (it was still waiting when it was read: the call cut off, or given up, while it waited).""" - from hermit.agent.base.interface import LOGIN_FAILED - with self.lock: - statuses, codes, messages = list(self.statuses), list(self.codes), list(self.messages) - running, aborted = len(self.running), self.aborted - note = dict(self.notes[-1]) if self.notes else {} - last = statuses[-1] if statuses else 0 - served = 200 in statuses - lost = not served and bool(codes) and codes[-1] == LOGIN_FAILED - refused = not served and bool(self.spec.get("cli")) and last == 429 - busy = not refused and (last is None or last in (408, 409, 429) or (isinstance(last, int) and last >= 500)) - calls = len(statuses) + running - waits = {"plan_waited": str(note.get("waited") or ""), "plan_waiting": bool(note.get("waiting"))} \ - if self.plan_notes else {} - return {"served": served, "unavailable": not served and bool(statuses) and busy and not lost, - "refused": refused, "login_failed": lost, "started": calls > 0, "calls": calls, - "rejected": not served and not lost and isinstance(last, int) and 400 <= last < 500 and not busy - and not refused, "statuses": statuses, "code": codes[-1] if codes else None, - "failure": (messages[-1] if messages else None) or "", "usage_incomplete": bool(running or aborted), - **waits} - - def close(self) -> None: - """The endpoint closed, after the calls still in flight were given up (each one's abort: a relay's call is - dropped upstream, ChatAborted; a CLI's session is cancelled) and waited for, up to CLOSE_WAIT_S, so that what - they recorded is in the facts and the usage read after it; then its listener, and what goes with it (a CLI's - sandbox).""" - if self.endpoint is None: - return - with self.lock: - self.closing = True - for call in self.running: - call.set() - self.lock.wait_for(lambda: not self.running, timeout=CLOSE_WAIT_S) - self.endpoint.close() - self.endpoint = None diff --git a/hermit/hosts/inputs.py b/hermit/hosts/inputs.py index 351f9487..dc105005 100644 --- a/hermit/hosts/inputs.py +++ b/hermit/hosts/inputs.py @@ -28,10 +28,10 @@ def render_exports(adapter, python): agent's worker sets HA_ADAPTER_SANDBOX), the adapter's modules (the learner's work) are imported in a strict sandbox: the adapter read-only, no network, none of the agent's keys, so they never run with the agent's login.""" command = [python, "-m", "hermit.adapter.render_context", "--adapter", str(adapter)] - if os.environ.get("HA_ADAPTER_SANDBOX") == "1": - from hermit.adapter.session import ENV_ENFORCE - result = run_in_strict_sandbox(command, role="render", readable=[Path(adapter)], - enforce=os.environ.get(ENV_ENFORCE, "required")) + from hermit.adapter.session import confined + enforce = confined() # an acting agent's worker, or its call in the run's process (Harbor's) + if enforce: + result = run_in_strict_sandbox(command, role="render", readable=[Path(adapter)], enforce=enforce) else: result = subprocess.run(command, capture_output=True, text=True, env=framework_env()) if result.returncode != 0: diff --git a/hermit/isolate.py b/hermit/isolate.py index 44afb412..5df32362 100644 --- a/hermit/isolate.py +++ b/hermit/isolate.py @@ -17,7 +17,9 @@ The data and split files, the judge, the runs and trees directories may not lie in a declared path: the config is refused, and the isolation check inside proves them absent. A task with `environment.image` runs its CLI agent in a container of its own (the Base Agent's container isolation), started from a worker on the host that keeps the -adapter's code in the strict sandbox; a Harbor task set (`agent.name: harbor`) runs in Harbor's, from this process. +adapter's code in the strict sandbox. A task set in Harbor's environments (`environment.harbor`) gives each task its +own environment (hermit.environment.harbor, kept in this process with its driver and Docker), where every command +and CLI of the agent runs: the agent's loop runs in this process with that environment (host_pipeline), no worker. The judge runs where `judge.mode` says: `host` in this process on the downloaded answer, `shared` in the task sandbox after the agent (a new exec over the same work directory, with `judge.tests` shown to it alone), `separate` @@ -43,13 +45,19 @@ import uuid from hermit.agent.base import login as codex_login +from hermit.agent.base.credentials import held from hermit.agent.base.cli import CLIS from hermit.agent.base.endpoint import cli_endpoint, endpoint_spec from hermit.redact import redact_name, redact_tree from hermit.sandbox import (LIMITS, NETWORK_MODES, PACKAGE, AllowlistProxy, Bind, Sandbox, framework_env, gpu_list, hidden_paths, model_access, network_mode, owner_readable, package_closure, - program_paths, sanitized_copy, set_hidden) + program_paths, sandbox_fault, sanitized_copy, set_hidden) +from hermit.environment.bwrap import task_sandbox +from hermit.environment.spec import EnvironmentSpec from hermit.task import RunOutput, load_callable, output_record +from hermit.task.verify import LOGS, SANDBOX_ENV, TESTS + +SCRIPT_VERIFIER = ("hermit.task.verify", "script") # judge: {path, name} of a task directory's own verifier WORKER = PACKAGE / "worker.py" CLI_HOMES = {c.backend: (c.user_folder, c.home_var, c.home_files[0]) for c in CLIS.values()} # folder, var, sign-in @@ -75,12 +83,12 @@ def host_pipeline(config) -> str | None: d = config.data if not d["sandbox"]["enabled"]: return "sandbox.enabled is false" - if str(d["agent"]["name"]) == "harbor": - return "a Harbor task set runs in Harbor's own environment" if d["environment"].get("runner") == "host": return "environment.runner: host (the task's own code isolates the adapter's code it runs, in containers)" if d["environment"].get("image"): return "environment.image: the agent runs in a container of its own" + if d["environment"].get("harbor"): + return "environment.harbor: every command and CLI of the agent runs in the task's own Harbor environment" return None @@ -205,7 +213,7 @@ def validate(config) -> None: "environment.image the image holds the agent's world") if env.get("resources"): raise ValueError("environment.resources are the bubblewrap sandbox's limits; a container " - "(environment.image) keeps its own (hermit/agent/environment/docker.py)") + "(environment.image) keeps its own (hermit/environment/docker.py)") if env.get("gpu"): raise ValueError("environment.gpu shows GPUs in the bubblewrap sandbox; a container (environment.image) " "has none") @@ -229,6 +237,17 @@ def _secrets(config) -> tuple[list[Path], Path | None]: if judge.get("path") and str(judge["path"]).partition(":")[0].endswith(".py"): out.append(config.path(judge["path"])) out += [config.path(t) for t in judge.get("tests") or []] + harbor = (d.get("environment") or {}).get("harbor") + if harbor: # a task set in Harbor's environments: its task directories hold the tests and solutions + try: + from hermit.environment.harbor import settings_of + settings, folder = settings_of(harbor, Path(config.base)) + if settings.get("tasks_dir"): + tasks = Path(os.path.expandvars(str(settings["tasks_dir"]))).expanduser() + tasks = tasks if tasks.is_absolute() else folder / tasks + out += [tasks.resolve(), Path(os.path.realpath(tasks))] + except (OSError, ValueError): # its own check reports a benchmark file it cannot read + pass runs = None try: out.append(config.run_dir()) @@ -444,7 +463,12 @@ def read(): out.close() payload = json.loads(data) if not payload.get("ok"): - raise RuntimeError(f"{payload.get('error')}\n{payload.get('traceback', '')}".strip()) + failed = RuntimeError(f"{payload.get('error')}\n{payload.get('traceback', '')}".strip()) + spent = payload.get("spent") + if isinstance(spent, dict): # what the failed call billed (an endpoint's error usage) + failed.spent = RunOutput(output=None, cost_usd=spent.get("cost_usd"), tokens=spent.get("tokens"), + api_speed={"model_calls": spent.get("model_calls")}) + raise failed return payload @@ -470,6 +494,19 @@ def _safe(text: str) -> str: return re.sub(r"[^A-Za-z0-9._-]+", "_", str(text))[:60] or "task" +STARTUP_S = 60 # beyond a task's own timeout: the sandbox and the worker start, the result comes back + + +def _deadline(run_timeout: float | None, task: dict) -> float | None: + """A call's deadline: the run's (agent.timeout_min), and a task directory's own `agent_timeout_sec` (task.toml's + [agent] timeout_sec, which Harbor enforces) when it is stricter, with the worker's start-up beside it.""" + own = task.get("agent_timeout_sec") + if not own: + return run_timeout + own = float(own) + STARTUP_S + return own if run_timeout is None else min(run_timeout, own) + + class SandboxedAgent: """The run's agent as a pipeline, every call in a worker inside the task sandbox.""" @@ -482,8 +519,7 @@ def __init__(self, config, role: str, *, records: Path | None, work_root: Path | missing = [p for p in envc.get("packages") or [] if p != "*" and not package_closure([p])] if missing: raise ValueError(f"environment.packages names distributions that are not installed: {missing}") - readable = config_paths(config, "environment.read", envc.get("read")) - writable = config_paths(config, "environment.write", envc.get("write")) + self.spec = EnvironmentSpec.from_section(envc, lambda where, values: config_paths(config, where, values)) self.params = dict(agent.get("params") or {}) if model_endpoint is not None: # a model only a CLI reaches: its endpoint is out here, with the sign-in; self.params["model"] = endpoint_spec(model_endpoint, self.params["model"]) # the call gets a URL @@ -494,12 +530,14 @@ def __init__(self, config, role: str, *, records: Path | None, work_root: Path | self.cli = _cli_kind(self.name, self.params) self.timeout = float(agent["timeout_min"]) * 60 if agent.get("timeout_min") else None self.enforce = d["sandbox"].get("enforce", "required") + # A task's own verifier runs in a sandbox like this one, so its network is checked with the agent's + judge = d.get("judge") or {} + self.verifies = (str(judge.get("path") or ""), judge.get("name")) == SCRIPT_VERIFIER self.logs = _agent_logs(self.name, self.base) for log_dir in self.logs: # the mount point of each call's own log directory log_dir.mkdir(exist_ok=True) - self.others: list[Sandbox] = [] # the judge's sandboxes, closed with this one - if self.cli: # the CLI and the node it may run on, as installed - readable += program_paths(CLIS[self.cli].program) + self.others: list = [] # the judge's sandboxes (a verifier's proxy), closed with it + program = program_paths(CLIS[self.cli].program) if self.cli else [] # a CLI and its node, as installed # A CLI agent on a login waits for the login's plan with every session on the machine: the plan's directory # (its PAUSE, MAX_USED, SLOTS and report) is the one outside, shared into every call; the worker inside runs # the Base Agent's wait, slot and holds there. A ChatGPT login's file is copied in; a Claude login signs in @@ -513,25 +551,35 @@ def __init__(self, config, role: str, *, records: Path | None, work_root: Path | if self.cli and codex_login.LOGINS.get(login) == self.cli else None if self.plan is not None: self.plan.mkdir(parents=True, exist_ok=True) - network, allow = envc.get("network") or "no-network", list(envc.get("allowed_hosts") or []) - wanted = [*model_hosts, *(codex_login.hosts(login) if self.plan is not None else [])] # and a plan login's - if network != "public" and wanted: - network, allow = "allowlist", [*allow, *[h for h in wanted if h not in allow]] - isolated, wants_proxy = network_mode(network) - self.allow_proxy = AllowlistProxy(allow, log=records / f"network-{role}.jsonl" if records else None) \ - if (isolated == "isolated" and wants_proxy and allow) else None - self.sandbox = Sandbox(enabled=True, network=network, allow=allow, readable=readable, - writable=[*writable, *([self.plan] if self.plan is not None else [])], - proxy_url=self.allow_proxy.url if self.allow_proxy else None, - env_allow=[*(envc.get("env") or []), *model_keys], enforce=self.enforce, - packages=list(envc.get("packages") or []), framework=True, record_dir=records, - probe_absent=[*HOME_SECRETS, *run_secrets(config)], - limits=dict(envc.get("resources") or {}), gpu=gpu_list(envc.get("gpu")), role=role) + # The task sandbox is the bwrap backend's (hermit.environment.bwrap), with what is the agent's own added: its + # model's hosts and key variable, a login's plan and its hosts, the CLI's installation. + self.records = records + self.sandbox, self.allow_proxy = task_sandbox( + self.spec, enforce=self.enforce, records=records, role=role, + probe_absent=[*HOME_SECRETS, *run_secrets(config)], extra_readable=program, + extra_writable=[self.plan] if self.plan is not None else [], extra_env=model_keys, + extra_hosts=[*model_hosts, *(codex_login.hosts(login) if self.plan is not None else [])]) # What the config declares (not the CLI's installation), searched again for run records before calls. - self._recheck = Recheck([*config_paths(config, "environment.read", envc.get("read")), - *config_paths(config, "environment.write", envc.get("write"))]) - - def __call__(self, task: dict, adapter: Path | None) -> RunOutput: + self._recheck = Recheck([*self.spec.read, *self.spec.write]) + + def refuses(self, needs: dict) -> list[str]: + """What a task asks of its environment (a task directory's `needs`) that this run's does not give: such a task + is never run with less (hermit.trainer.rollout asks before each call, unless the task has an environment of + its own: a Harbor one gives what it asks).""" + return self.spec.problems_for(needs, verifier=self.verifies) + + def __call__(self, task: dict, adapter: Path | None, needs: dict | None = None) -> RunOutput: + """`needs`: what the task asks of its environment (its private half's; hermit.trainer.rollout passes it), of + which its agent's timeout bounds the call. (A task in an environment of its own, a Harbor one, never comes + here: its agent runs in this process, host_pipeline.)""" + return self._call(task, adapter, needs) + + def _login_root(self) -> Path: + """The ChatGPT login's shared copy this run's preflight checked: the agent's `login_root`, else the default.""" + root = self.params.get("login_root") + return Path(os.path.expanduser(str(root))) if root else codex_login.ROOT + + def _call(self, task: dict, adapter: Path | None, needs: dict | None = None) -> RunOutput: self._recheck() work = self.work_root / f"{_safe(task.get('id', 'task'))}-{uuid.uuid4().hex[:8]}" home = work / "home" @@ -553,31 +601,42 @@ def __call__(self, task: dict, adapter: Path | None) -> RunOutput: folder, var, _ = CLI_HOMES[self.cli] agent_home = home / folder agent_home.mkdir(parents=True) # a folder of more than one level (Pi's .pi/agent) - if self.chatgpt: - codex_login.prepare(codex_login.ROOT, agent_home) # the shared copy, refreshed by earlier calls env[var] = str(agent_home) keep.append(var) binds = _log_binds(work, self.logs, [work]) job = {"kind": "agent", "name": self.name, "params": params, "base": str(self.base), "task": task, "adapter": str(adapter) if adapter else None} + forward = list(self.forward) clean = work.parent / f"{work.name}.log" + left: list = [] + + def provision(): # the shared copy, refreshed by earlier calls + if self.cli and self.chatgpt: + codex_login.prepare(self._login_root(), home / CLI_HOMES[self.cli][0]) + return [] + + def take_back(_): + # Every copy the call left: its run copy, and a session's own when the worker was stopped before it could + # hand that back (a timeout kills it). The latest refresh goes back to the shared copy. + if self.chatgpt: + codex_login.remove(home / CLI_HOMES["codex"][0] / CLI_HOMES["codex"][2], within=work) + left[:] = [work / "workspaces" / "login" / codex_login.FILE, + *work.glob(f"workspaces/*/home/{codex_login.FILE}")] + for path in left: + codex_login.take_back(self._login_root(), path, within=work) + + def remove(_): # then none of them is kept, however the take-back went + if self.chatgpt: + for path in left: + codex_login.remove(path, within=work) + shutil.rmtree(work / "workspaces" / "login", ignore_errors=True) + try: - try: + with held(provision, take_back, remove): payload = _run_worker(self.sandbox, job, work, home=home, readable=[adapter] if adapter else [], - env=env, keep=keep, forward=self.forward, timeout=self.timeout, binds=binds) - finally: - if self.chatgpt: - # Every copy the call left: its run copy, and a session's own when the worker was stopped - # before it could hand that back (a timeout kills it). The latest refresh goes back to the - # shared copy; then none of them is kept. - codex_login.remove(home / CLI_HOMES["codex"][0] / CLI_HOMES["codex"][2], within=work) - left = [work / "workspaces" / "login" / codex_login.FILE, - *work.glob(f"workspaces/*/home/{codex_login.FILE}")] - for path in left: - codex_login.take_back(codex_login.ROOT, path, within=work) - for path in left: - codex_login.remove(path, within=work) - shutil.rmtree(work / "workspaces" / "login", ignore_errors=True) + env=env, keep=keep, forward=forward, + timeout=_deadline(self.timeout, needs or task), + binds=binds) rec = payload["output"] # The log is copied into the learner's material outside any sandbox: it must lie where this call could # write (its work directory, its logs, the declared write paths), never a file it merely names. @@ -635,8 +694,11 @@ def __init__(self, config, role: str, params: dict, *, work_root: Path | None = self.enforce = d["sandbox"].get("enforce", "required") self.box = Sandbox(enabled=False, role=role) # the worker is on the host; the container is the boundary - def __call__(self, task: dict, adapter: Path | None) -> RunOutput: + def __call__(self, task: dict, adapter: Path | None, environment=None) -> RunOutput: from hermit.adapter.session import ENV_ENFORCE + if environment is not None: + raise ValueError("environment.image runs the agent in a container of its own; a task that gives its own " + "environment (environment.harbor) runs in that one: give one of them") work = self.work_root / f"{_safe(task.get('id', 'task'))}-{uuid.uuid4().hex[:8]}" work.mkdir(parents=True) # The agent's logs go to this call's own directory; its workspaces stay where the task puts them @@ -699,10 +761,28 @@ def cli_sandbox(spec: dict, *, records: Path | None = None, enabled: bool = True return box, proxy +def confined_run(run, enforce: str): + """A Harbor task set's agent run in this process with the adapter's own code confined as an acting worker confined + it (the runner's too): its context tool's Session and its exports' render in the strict adapter sandbox + (hermit.adapter.session.CONFINED, for this call only), the agent's loop here.""" + import functools + from hermit.adapter.session import CONFINED + + @functools.wraps(run) + def call(*args, **kwargs): + token = CONFINED.set(enforce) + try: + return run(*args, **kwargs) + finally: + CONFINED.reset(token) + return call + + def agent_pipeline(config, role: str, *, records: Path | None, work_root: Path | None = None): """(pipeline, close): the agent in the task sandbox; in its own container from a worker on the host (`environment.image`); or in this process when the pipeline runs on the host (`host_pipeline`: the sandbox off, - a Harbor task set, `environment.runner: host`). An agent whose model only a CLI reaches (`agent.params.model: + `environment.runner: host`; a Harbor task set's agent, whose commands run in the task's container, with the + adapter's own code confined: confined_run). An agent whose model only a CLI reaches (`agent.params.model: {cli: ...}`) gets it as a model endpoint started here, each request one session in a sandbox of its own built as the Trainer builds the llm's (cli_sandbox, role agent-model); the pipeline sees a URL and an ordinary model spec, which is never written to the config.""" @@ -733,7 +813,18 @@ def agent_pipeline(config, role: str, *, records: Path | None, work_root: Path | from hermit.agent.acting.agent import load_agent if endpoint is not None: params["model"] = endpoint_spec(endpoint, params["model"]) - return load_agent(d["agent"]["name"], params, config.base).run, stop + run = load_agent(d["agent"]["name"], params, config.base).run + if d["environment"].get("harbor") and d["sandbox"]["enabled"]: + enforce = d["sandbox"].get("enforce", "required") + fault = sandbox_fault(enforce, fresh=False) # the runner's preflight: every deployment renders there + if fault is not None: + stop() + raise ValueError(f"a task in Harbor's environments deploys its adapter by rendering it in the " + f"strict adapter sandbox, and {fault}: every rollout would fail; let bubblewrap " + "make its namespaces (user namespaces on), or set sandbox.enforce: best_effort " + "(unconfined, recorded)") + return confined_run(run, enforce), stop + return run, stop agent = SandboxedAgent(config, role, records=records, work_root=work_root, model_endpoint=endpoint) except BaseException: stop() # the endpoint started for a pipeline that never came @@ -757,16 +848,41 @@ def __init__(self, config, agent: SandboxedAgent): self.readable = [*([config.path(module)] if module.endswith(".py") else []), # the file, not its dir *config_paths(config, "judge.tests", d["judge"].get("tests"))] self.collect = list(d["judge"].get("collect") or []) + # A task directory's own verifier (hermit.task.verify.script): each call shows the task's tests at /tests and a + # fresh log directory at /logs/verifier, as Harbor's verifier sees them. + self.script = self.spec == f"{SCRIPT_VERIFIER[0]}:{SCRIPT_VERIFIER[1]}" # The judge's own sandbox: the task sandbox's settings, and its probe checks only the home's secrets (the # judge, its tests and the data are its to read, and so is what sandbox.hide covers for everyone else). A # separate judge gets none of what the agent could write (environment.write): only what is collected. Nor # does any judge get a login's plan, which steers every session on the machine: its probe proves it absent. plan = [agent.plan] if agent.plan is not None else [] own = [w for w in agent.sandbox.writable if Path(w) not in plan] - self.box = dataclasses.replace(agent.sandbox, probe_absent=[*HOME_SECRETS, *plan], role="judge", hide=[], - writable=[] if self.mode == "separate" else own) + self.forward = agent.forward + if self.script: + # The task author's verifier gets the task's environment only (environment.read, write, env, network and + # allowed_hosts, as Harbor's verifier gets its task's), never what is the acting agent's own: its model's + # key variable, its provider's hosts, a model server on the loopback, a login's plan, the CLI's install. + self.box, proxy = task_sandbox(agent.spec, enforce=agent.enforce, records=agent.records, + role="judge", probe_absent=[*HOME_SECRETS, *plan]) + # sandbox.hide is what the agent may not read (a scoring module, say): a verifier reads it, as any judge + self.box = dataclasses.replace(self.box, hide=[], writable=[] if self.mode == "separate" else + self.box.writable) + self.forward = [] + if proxy is not None: + agent.others.append(proxy) # closed with the agent's sandboxes + else: + self.box = dataclasses.replace(agent.sandbox, probe_absent=[*HOME_SECRETS, *plan], role="judge", hide=[], + writable=[] if self.mode == "separate" else own) agent.others.append(self.box) + def _timeout(self, task: dict) -> float | None: + """The judge's deadline: the agent's; a task's verifier script gets at least its own timeout_sec and the + worker's start-up (its script stops itself at timeout_sec).""" + own = ((task.get("_private") or {}).get("verifier") or {}).get("timeout_sec") if self.script else None + if not own or self.agent.timeout is None: + return self.agent.timeout + return max(self.agent.timeout, float(own) + STARTUP_S) + def __call__(self, task: dict, out: RunOutput) -> dict: work = out.extra.get("sandbox_work") if not work: @@ -777,6 +893,8 @@ def __call__(self, task: dict, out: RunOutput) -> dict: home.mkdir(parents=True) record = output_record(out) record["extra"] = {k: v for k, v in (record.get("extra") or {}).items() if k != "sandbox_work"} + if self.script and not record["extra"].get("workspace"): + record["extra"]["workspace"] = str(work) # where the agent worked: the verifier's workspace job = {"kind": "judge", "spec": self.spec, "base": str(self.base), "task": task, "output": record} # Shared: the agent's work as the agent saw it, its per-call `.logs` at the names the agent wrote them under. binds = _log_binds(work, self.agent.logs, [work, place], existing=True, @@ -792,16 +910,120 @@ def __call__(self, task: dict, out: RunOutput) -> dict: binds, writable = [Bind(fresh, str(work), True)], [] # The log the agent returned is a sanitized copy beside its work directory: the judge reads it there. log_copy = [out.log.parent] if out.log and Path(out.log).parent.name == f"{work.name}.log" else [] + env, keep = framework_env(), [] + if self.script: + private = task.get("_private") or {} + if not private.get("tests"): + raise RuntimeError(f"task {task.get('id')!r} has no tests/ for its verifier script") + if (private.get("verifier") or {}).get("environment_mode") == "separate" and self.mode != "separate": + raise RuntimeError(f"task {task.get('id')!r} asks for a separate verifier (task.toml " + "[verifier] environment_mode); set judge.mode: separate") + logs = place / "verifier" + logs.mkdir() + tests = Path(private["tests"]) + if private.get("step_tests"): # a step's own tests laid over the task's, as Harbor does + tests = place / "tests" + shutil.copytree(private["tests"], tests, symlinks=True) + shutil.copytree(private["step_tests"], tests, symlinks=True, dirs_exist_ok=True) + binds = [*binds, Bind(tests, TESTS), Bind(logs, LOGS, True)] + env[SANDBOX_ENV], keep = "1", [SANDBOX_ENV] try: payload = _run_worker(self.box, job, place, home=home, readable=[*self.readable, *log_copy], - writable=writable, env=framework_env(), - forward=self.agent.forward, timeout=self.agent.timeout, binds=binds) + writable=writable, env=env, keep=keep, + forward=self.forward, timeout=self._timeout(task), binds=binds) finally: for kept in (place, *writable): # the judge holds the task's keys too: none kept redact_tree(kept) return payload["result"] +def task_environments(config, *, log=None): + """The run's task environments: Harbor's (hermit.environment.harbor.HarborEnvironments), one for each task, when + the config's `environment.harbor` names them; else None (the agent works in its own workspace). A task whose agent + phase has no public network lets the agent reach its model's hosts too.""" + d = config.data + section = d["environment"].get("harbor") + if not section: + return None + if d["environment"].get("image"): + raise ValueError("environment.harbor and environment.image: a task in Harbor's environment runs its agent " + "there, not in a container of the agent's own; give one of them") + from hermit.environment.harbor import HarborEnvironments + params = dict(d["agent"].get("params") or {}) + model = params.get("model") + if isinstance(model, dict) and model.get("cli"): + raise ValueError("agent.params.model: a model a CLI reaches (`cli: ...`) is not supported for a task in " + "Harbor's environments: its login's plan would be waited for inside the task's time; give the " + "agent a / reference, or run the CLI itself as the agent (agent.base)") + backend = str(_cli_kind(d["agent"].get("name"), params) or "") # the CLI the agent runs, `cli` too + binding = params.get("binding") + if backend in ("codex", "claude") and not binding and not params.get("login"): + # the runner's refusal: a CLI in a task's container signs in by the run's model binding or a named login, never + # by a sign-in of this machine's own it would carry there + raise ValueError(f"agent {backend} in Harbor's environments needs a model binding (agent.model: /" + ") or a login (login: chatgpt | claude)") + if backend in ("codex", "claude") and isinstance(binding, dict): + # the runner's check: a CLI runs in the task's container, which cannot reach this machine's loopback + from urllib.parse import urlsplit + from hermit.agent.model.backends import Binding + bound = Binding.from_spec(binding) + url = bound.base_url if backend == "codex" else bound.backend.messages_root + if (urlsplit(str(url)).hostname or "") in ("127.0.0.1", "localhost", "::1", "0.0.0.0"): + raise ValueError(f"the model's endpoint {url} is on this machine's loopback, which a task container cannot " + "reach: serve it on an address the container reaches (_BASE_URL); the answer and " + "ReAct agents, which call the model from this machine, may use the loopback") + _, hosts, _ = model_access([params.get("binding"), params.get("model")]) + login = params.get("login") + hosts = [*hosts, *(h for h in codex_login.hosts(login) if h not in hosts)] # a subscription's own hosts too + # the driver (and Harbor's compose, which takes its environment as a base) never sees an ambient credential the + # agent's commands do not pass themselves (the runner's harbor_env): the switches to Codex's own ChatGPT file, the + # Claude subscription's token, and for Claude Code another bill's or a parent session's variables + from hermit.agent.base.cli import CLAUDE_BILLING, CLAUDE_INHERITED + drop = ["CODEX_FORCE_AUTH_JSON", "CODEX_AUTH_JSON_PATH", codex_login.TOKEN_ENV] + if params.get("backend") in ("claude", "claude-code") or d["agent"].get("name") in ("claude-code", "claude"): + drop += [*CLAUDE_BILLING, *CLAUDE_INHERITED, "CLAUDE_FORCE_OAUTH"] + environments = HarborEnvironments(section, base=Path(config.base), allowed_hosts=hosts, log=log, drop_env=drop) + # what the agent needs installed before its phase (Agent.installs), which Harbor's setup installs: a built-in + # agent's (a task's own class is not loaded on this machine; one in Harbor's environment brings what it needs) + from hermit.agent.acting.agent import builtin + kind = builtin(str(d["agent"].get("name") or "")) + if kind is not None: + declared = kind(params, Path(config.base)) + if callable(getattr(declared, "installs", None)): + environments.installs_for = declared.installs + if callable(getattr(declared, "usage_of", None)): + environments.usage_of = declared.usage_of + if codex_login.LOGINS.get(login): + environments.admission = _plan_admission(codex_login.plan_dir(params.get("plan_dir"), login), + codex_login.opted_in(params.get("usage_credits")), + environments.stopping, log) + return environments + + +def _plan_admission(plan: Path, credits: bool, stopping, log=None): + """A task's admission to its login's plan, as the Harbor runner held a trial's: the wait while the plan holds + sessions back, then one of the machine's session slots, held around the whole of the task's environment (the CLI + in it takes none of its own: `plan_admitted`), never inside the agent's time. A run that ended meanwhile + (`stopping`) ends the rollout as a lost sign-in does.""" + import contextlib + from hermit.agent.base.interface import LoginFailed + say = log or (lambda m: print(m, file=sys.stderr, flush=True)) + + @contextlib.contextmanager + def admitted(): + while True: + if codex_login.wait(plan, stopping, say, None, credits): + raise LoginFailed("the run ended while the task waited for its login's plan") + with codex_login.slot(plan, stopping, say, None) as got: + if got is not True: + raise LoginFailed("the run ended while the task waited for its login's plan") + if codex_login.held(plan, credits): # held while it waited for the slot: wait again + continue + yield + return + return admitted + + def judge_for(config, pipeline): """The judge as `run_tasks` calls it: in this process (`host`, or any mode with the sandbox off), or sandboxed beside the task sandbox.""" diff --git a/hermit/sandbox.py b/hermit/sandbox.py index 8849f690..3fb667f0 100644 --- a/hermit/sandbox.py +++ b/hermit/sandbox.py @@ -277,6 +277,32 @@ def run_in_strict_sandbox(command: list[str], *, role: str, readable, enforce: s box.close() +_SANDBOX_STARTED: list[bool] = [] # a strict adapter sandbox started in this process (a run's startup) + + +def sandbox_fault(enforce: str | None, *, fresh: bool = True) -> str | None: + """Why the strict adapter sandbox that renders each adapter does not start on this machine (bubblewrap that cannot + make its namespaces, say), seen from this process (the Harbor runner's, ported): None when it starts, or when it + is not required (`enforce` not "required"). `fresh` false (a run's startup): once it started in this process, it + is not tried again. Bubblewrap not installed is a fault too (the runner refused it at startup).""" + if enforce != "required": + return None + if not shutil.which("bwrap"): + return "bubblewrap (bwrap) is not installed" + if not fresh and _SANDBOX_STARTED: + return None + try: + run = run_in_strict_sandbox([sys.executable, "-c", "pass"], role="render", readable=[], enforce="required", + timeout=120) + except (OSError, subprocess.SubprocessError) as exc: + return f"the strict adapter sandbox did not start: {type(exc).__name__}: {exc}" + if run.returncode != 0: + return (f"the strict adapter sandbox did not start (exit {run.returncode}): " + f"{(run.stderr or run.stdout or '').strip()[-300:]}") + _SANDBOX_STARTED.append(True) + return None + + NETWORK_MODES = ("no-network", "allowlist", "public") # what a run config may name NETWORKS = {"no-network": ("isolated", False), "allowlist": ("isolated", True), "public": ("open", False), "isolated": ("isolated", True), "open": ("open", False)} # the last two: the framework's own callers diff --git a/hermit/settings_check.py b/hermit/settings_check.py index f2787aa0..80aa1d41 100644 --- a/hermit/settings_check.py +++ b/hermit/settings_check.py @@ -17,8 +17,9 @@ from hermit.agent.evolver.loader import find, instantiate, load_class from hermit.spec import check_spec from hermit.agent.acting.agent import load_agent -from hermit.isolate import agent_pipeline, config_paths, host_pipeline, judge_for, validate as validate_sandboxes -from hermit.task import TaskData, load_callable +from hermit.isolate import agent_pipeline, config_paths, host_pipeline, judge_for, task_environments, \ + validate as validate_sandboxes +from hermit.task import Benchmark, load_callable from hermit.trainer.rollout import run_one from hermit.trainer.trainer import needs_errors, warnings_for @@ -100,8 +101,7 @@ def report(item: str, problems: list[str], note: str = ""): problems.append("task_info/README.md is missing") data = None try: - data = TaskData(config.path(d["data"]["tasks"]), config.path(d["data"]["train"]), - config.path(d["data"]["val"]), config.path(d["data"]["test"])) + data = Benchmark.from_config(config) problems += data.overlaps() if not data.splits["train"]: problems.append("the training split is empty") @@ -159,6 +159,7 @@ def report(item: str, problems: list[str], note: str = ""): try: probe_pipeline, close = agent_pipeline(config, "check", records=None) judge = judge_for(config, probe_pipeline) + environments = task_environments(config) except Exception as exc: close() # what was started before the error probe_pipeline, close = None, (lambda: None) @@ -166,14 +167,14 @@ def report(item: str, problems: list[str], note: str = ""): "api_speed": None, "cost_usd": None} if probe_pipeline is not None: try: - record = run_one(task, None, probe_pipeline, judge) + record = run_one(task, None, probe_pipeline, judge, environments) except LoginFailed as exc: record = {"status": "error", "error": str(exc), "judge": {"score": 0.0}, "api_speed": None, "cost_usd": None} finally: close() problems = [record["error"]] if record["status"] != "ok" else [] - report("agent probe", problems, f"task {task['id']}: score {record['judge']['score']}, " + report("agent probe", problems, f"task {task.id}: score {record['judge']['score']}, " f"{time.time() - t0:.0f} s, api speed {json.dumps(record.get('api_speed'))}, cost {record['cost_usd']}") for w in warnings_for(config, val_given=True): log(f"[warning] {w}") diff --git a/hermit/task/__init__.py b/hermit/task/__init__.py new file mode 100644 index 00000000..5dbc6ba1 --- /dev/null +++ b/hermit/task/__init__.py @@ -0,0 +1,21 @@ +"""Tasks: what a task set is, what each task gives the agent and keeps for the judge, and how an answer is judged +(docs/framework/task.md). A task knows nothing of the agent that answers it (hermit.agent), and the agent nothing of +the task set or the judge: the run (hermit.trainer.rollout) brings the two together, in an environment +(hermit.environment) both may use. + + task.py Task: one task, its public half (what the agent is given) and its private half (the judge's own) + benchmark.py Benchmark: a task set, its splits + verify.py the judge's verdict on one answer + data.py the rows of tasks.jsonl and the split files, the records a run keeps + +The names code imports from here before the package existed (RunOutput, JudgeResult, TaskData, load_callable, ...) +are all still importable from `hermit.task`. +""" +from hermit.common import RunOutput, load_callable +from hermit.task.benchmark import Benchmark +from hermit.task.data import JudgeResult, TaskData, load_pipeline, normalise_judge, output_record, read_ids +from hermit.task.task import Task, as_task +from hermit.task.verify import Verifier, judge_verifier + +__all__ = ["Benchmark", "JudgeResult", "RunOutput", "Task", "TaskData", "Verifier", "as_task", "judge_verifier", + "load_callable", "load_pipeline", "normalise_judge", "output_record", "read_ids"] diff --git a/hermit/task/benchmark.py b/hermit/task/benchmark.py new file mode 100644 index 00000000..4f5e7ff0 --- /dev/null +++ b/hermit/task/benchmark.py @@ -0,0 +1,60 @@ +"""A task set and its splits, as the run config names them (`data.tasks`, `data.train`, `data.val`, `data.test`). + + bench = Benchmark.from_config(config) + bench.split("test") # [Task, ...] in the split file's order + bench.tasks["q1"] # one Task by id + +`data.tasks` is a `tasks.jsonl` (one row per task, every field public) or a directory of task directories in +Harbor's format (hermit.task.directory: each task's instruction and settings public, its tests private); +`data.rollouts: k` makes k rows of each directory task (`#1..k`); `data.view: :` turns a +directory task's row into the row the task set's agents and learners read (what its tasks.jsonl held, say). +""" +from __future__ import annotations + +from pathlib import Path +from typing import Callable + +from hermit.common import load_callable +from hermit.task.data import TaskData, read_ids +from hermit.task.directory import tasks_of +from hermit.task.task import Task + +SPLITS = ("train", "val", "test") + + +class Benchmark: + def __init__(self, tasks: dict[str, Task], splits: dict[str, list[str]]): + self.tasks = tasks + self.splits = {name: list(splits.get(name) or []) for name in SPLITS} + for name, ids in self.splits.items(): + missing = [i for i in ids if i not in self.tasks] + if missing: + raise ValueError(f"{name} split names unknown tasks: {missing[:5]}") + + @classmethod + def load(cls, tasks: Path, train: Path | None = None, val: Path | None = None, test: Path | None = None, *, + rollouts: int = 1, view: Callable[[dict], dict] | None = None) -> "Benchmark": + splits = {"train": read_ids(train), "val": read_ids(val), "test": read_ids(test)} + if Path(tasks).is_dir(): + return cls(tasks_of(Path(tasks), rollouts=int(rollouts), view=view), splits) + data = TaskData(tasks, None, None, None) + return cls({tid: Task.from_row(row) for tid, row in data.tasks.items()}, splits) + + @classmethod + def from_config(cls, config) -> "Benchmark": + d = config.data["data"] + view = load_callable(str(d["view"]), config.base) if d.get("view") else None + return cls.load(config.path(d["tasks"]), config.path(d.get("train")), config.path(d.get("val")), + config.path(d.get("test")), rollouts=int(d.get("rollouts") or 1), view=view) + + def split(self, name: str) -> list[Task]: + return [self.tasks[i] for i in self.splits[name]] + + def overlaps(self) -> list[str]: + out = [] + for i, a in enumerate(SPLITS): + for b in SPLITS[i + 1:]: + common = set(self.splits[a]) & set(self.splits[b]) + if common: + out.append(f"{a} and {b} share {len(common)} tasks (e.g. {sorted(common)[:3]})") + return out diff --git a/hermit/task.py b/hermit/task/data.py similarity index 64% rename from hermit/task.py rename to hermit/task/data.py index 175b4f0f..6a8dfdf7 100644 --- a/hermit/task.py +++ b/hermit/task/data.py @@ -1,4 +1,4 @@ -"""The evolving task: data and splits, the user's pipeline (how one task is run) and judge (how it is scored). +"""A task set's data: its rows (`tasks.jsonl`) and splits, the judge's verdict and the records a run keeps. def run(task: dict, adapter: Path | None) -> RunOutput # pipeline; adapter=None = no adapter at all def score(task: dict, output: RunOutput) -> JudgeResult | dict # judge @@ -6,24 +6,11 @@ def score(task: dict, output: RunOutput) -> JudgeResult | dict # judge from __future__ import annotations from dataclasses import asdict, dataclass, field -import importlib -import importlib.util import json from pathlib import Path -import sys -from typing import Any, Callable +from typing import Callable - -@dataclass -class RunOutput: - output: Any # what the judge scores - log: Path | None = None # the host's original log (the trajectory), a file or a directory - cost_usd: float | None = None # None = unknown - wall_s: float = 0.0 - api_speed: dict | None = None # for example output tokens per second - messages: list | None = None # optional normalised message list of the trajectory - tokens: int | None = None # input + output tokens, if known - extra: dict = field(default_factory=dict) +from hermit.common import RunOutput, load_callable @dataclass @@ -33,28 +20,6 @@ class JudgeResult: extra: dict = field(default_factory=dict) -def load_callable(spec: str, base: Path) -> Callable: - """'file.py:function' relative to `base`, or 'package.module:function'.""" - module_part, _, name = spec.partition(":") - if not name: - raise ValueError(f"Expected 'file.py:function' or 'module:function', got {spec!r}") - if module_part.endswith(".py"): - path = (Path(base) / module_part).resolve() - mod_name = f"_task_{path.stem}_{abs(hash(path))}" - mod_spec = importlib.util.spec_from_file_location(mod_name, path) - module = importlib.util.module_from_spec(mod_spec) - sys.modules[mod_name] = module - if str(path.parent) not in sys.path: - sys.path.insert(0, str(path.parent)) - mod_spec.loader.exec_module(module) - else: - module = importlib.import_module(module_part) - fn = getattr(module, name, None) - if not callable(fn): - raise ValueError(f"{spec} is not callable") - return fn - - def load_pipeline(spec: str, base: Path, params: dict | None = None) -> Callable: """A `run(task, adapter)` callable with `agent.params` and the config directory bound when its signature asks for them (keyword parameters named `params` and `base`); `hermit.agent.acting.FunctionAgent` is the Agent form.""" diff --git a/hermit/task/directory.py b/hermit/task/directory.py new file mode 100644 index 00000000..8b0594fc --- /dev/null +++ b/hermit/task/directory.py @@ -0,0 +1,276 @@ +"""Tasks as directories in Harbor's format (https://harborframework.com/docs/tasks), read without Harbor: + + / + task.toml [metadata], [agent] timeout_sec, [verifier] timeout_sec / environment_mode / env, + [environment] resources and network (network_mode, allowed_hosts, allow_internet) + instruction.md what the agent is asked + environment/ what the task needs of its environment (a Dockerfile for a container; a task Hermit runs in + its own sandbox may have none) + tests/test.sh the verifier: it writes /logs/verifier/reward.txt (a number) or reward.json (named numbers) + +A multi-step task adds `[[steps]]` to task.toml and `steps//{instruction.md, tests/}`. A task's row is its +public half (the instruction, its timeouts, resources, network modes and metadata, and `harbor_task`, the +directory's name); where its tests and its directory are is the private half (hermit.task.task), which only the +verifier reads. +""" +from __future__ import annotations + +import copy +import json +from pathlib import Path +from typing import Callable +import tomllib + +from hermit.task.task import Task + +TASK_FILE = "task.toml" + + +def is_task(path: Path) -> bool: + return path.is_dir() and (path / TASK_FILE).is_file() + + +def task_dirs(tasks_dir: Path) -> list[Path]: + """The task directories under `tasks_dir`, by name.""" + tasks_dir = Path(tasks_dir) + if is_task(tasks_dir): + return [tasks_dir] + return sorted(p for p in tasks_dir.iterdir() if is_task(p)) if tasks_dir.is_dir() else [] + + +def task_config(task_dir: Path) -> dict: + return tomllib.loads((Path(task_dir) / TASK_FILE).read_text()) + + +def step_names(config: dict) -> list[str]: + return [str(s["name"]) for s in (config.get("steps") or [])] + + +def _baseline(env) -> str: + """An environment's network mode (Harbor's baseline): its network_mode, else the deprecated allow_internet (true: + public, false: no-network), else public.""" + env = env if isinstance(env, dict) else {} + if env.get("network_mode"): + return str(env["network_mode"]) + if env.get("allow_internet") is not None and env.get("allowed_hosts") is None: + return "public" if env["allow_internet"] else "no-network" + return "public" + + +def _step(config: dict, step: str | None) -> dict: + return (next((s for s in config.get("steps") or [] if str(s.get("name")) == str(step)), None) or {}) if step \ + else {} + + +def verifier_environment(config: dict, step: str | None = None) -> tuple[str, dict | None]: + """(mode, the verifier's own environment or None) as Harbor resolves them (harbor/models/task/verifier_mode.py): + a level's explicit environment_mode, else `separate` when it defines an environment; the step's level before the + task's; `shared` when neither says. Separate: the step's environment, else the task verifier's, else [environment] + again; shared: none (the verifier runs in the agent's environment).""" + def level(verifier: dict): + if verifier.get("environment_mode"): + return str(verifier["environment_mode"]) + return "separate" if verifier.get("environment") is not None else None + step_v, task_v = (_step(config, step).get("verifier") or {}), (config.get("verifier") or {}) + mode = (level(step_v) if step else None) or level(task_v) or "shared" + if mode != "separate": + return mode, None + for env in (step_v.get("environment") if step else None, task_v.get("environment")): + if env is not None: + return mode, env + return mode, dict(config.get("environment") or {}) + + +def network_modes(config: dict, step: str | None = None) -> dict[str, str]: + """{"agent": mode, "verifier": mode} of a task (or of its step `step`), as Harbor resolves them + (harbor/trial/network_policy.py): a phase's own `network_mode` ([agent], [verifier], a step's first), else the + environment's baseline (the verifier's own environment's when it has one); modes public, no-network, allowlist.""" + step_cfg = next((s for s in config.get("steps") or [] if str(s.get("name")) == str(step)), None) if step else None + step_cfg = step_cfg or {} + + def explicit(role: str): + return ((step_cfg.get(role) or {}).get("network_mode")) or ((config.get(role) or {}).get("network_mode")) + base = _baseline(config.get("environment")) + verifier_env = verifier_environment(config, step)[1] + return {"agent": str(explicit("agent") or base), + "verifier": str(explicit("verifier") or (_baseline(verifier_env) if verifier_env is not None else base))} + + +def allowed_hosts(config: dict, step: str | None = None) -> dict[str, list[str]]: + """{"agent": hosts, "verifier": hosts}: the hosts each phase's allowlist names, resolved as network_modes resolves + the modes (a phase's own policy, a step's first, else the baseline: the verifier's own environment's when it has + one, else [environment]'s).""" + step_cfg = next((s for s in config.get("steps") or [] if str(s.get("name")) == str(step)), None) if step else None + step_cfg = step_cfg or {} + + def own(role: str): + for section in (step_cfg.get(role) or {}, config.get(role) or {}): + if section.get("network_mode"): + return list(section.get("allowed_hosts") or []) + return None + base = list((config.get("environment") or {}).get("allowed_hosts") or []) + verifier_env = verifier_environment(config, step)[1] + verifier_base = list(verifier_env.get("allowed_hosts") or []) if verifier_env is not None else base + agent, verifier = own("agent"), own("verifier") + return {"agent": base if agent is None else agent, "verifier": verifier_base if verifier is None else verifier} + + +def _size_mb(value) -> int | None: + """A size as Harbor reads the deprecated `memory`/`storage` fields ("2G", "512M"), in MB; None when unreadable.""" + text = str(value).strip().upper() + try: + if text.endswith("G"): + return int(float(text[:-1]) * 1024) + if text.endswith("M"): + return int(float(text[:-1])) + if text.endswith("K"): + return int(float(text[:-1]) / 1024) + except ValueError: + return None + return None + + +def resources(config: dict) -> dict: + """The task's resources ([environment]): cpus, memory_mb, storage_mb, gpus (None where it sets none).""" + env = config.get("environment") or {} + memory = env.get("memory_mb") if env.get("memory_mb") is not None else \ + (_size_mb(env["memory"]) if env.get("memory") is not None else None) + storage = env.get("storage_mb") if env.get("storage_mb") is not None else \ + (_size_mb(env["storage"]) if env.get("storage") is not None else None) + return {"cpus": env.get("cpus"), "memory_mb": memory, "storage_mb": storage, "gpus": env.get("gpus")} + + +def _instruction(path: Path) -> str: + """The instruction as the file holds it (its line endings kept: a `\r\n` in a question is part of it), trimmed.""" + try: + with open(path, encoding="utf-8", newline="") as f: + return f.read().strip() + except OSError: + return "" + + +def rows_of(task_dir: Path, *, rollouts: int = 1) -> list[dict]: + """The Hermit rows of one task: by rollout, and by step for a multi-step task.""" + task_dir = Path(task_dir) + config = task_config(task_dir) + env, agent = (config.get("environment") or {}), (config.get("agent") or {}) + verifier = config.get("verifier") or {} + base = {"harbor_task": task_dir.name, "agent_timeout_sec": agent.get("timeout_sec"), + "verifier_timeout_sec": verifier.get("timeout_sec"), "build_timeout_sec": env.get("build_timeout_sec"), + "docker_image": env.get("docker_image"), "metadata": config.get("metadata") or {}, **resources(config), + "network": network_modes(config)} + steps = step_names(config) + out = [] + for k in range(1, rollouts + 1): + suffix = f"#{k}" if rollouts > 1 else "" + if not steps: + out.append({"id": f"{task_dir.name}{suffix}", **base, "rollout": k, + "instruction": _instruction(task_dir / "instruction.md")}) + continue + for i, step in enumerate(steps, start=1): + step_cfg = next(s for s in config["steps"] if str(s["name"]) == step) + out.append({"id": f"{task_dir.name}:{step}{suffix}", **base, "rollout": k, "step": step, "step_index": i, + "steps": len(steps), "instruction": _instruction(task_dir / "steps" / step / "instruction.md"), + "network": network_modes(config, step), + "agent_timeout_sec": ((step_cfg.get("agent") or {}).get("timeout_sec") + or base["agent_timeout_sec"]), + "verifier_timeout_sec": ((step_cfg.get("verifier") or {}).get("timeout_sec") + or base["verifier_timeout_sec"])}) + return out + + +def toml_value(value) -> str: + """A value as TOML writes it, read back by tomllib as the same value (key order kept): a string (JSON's escapes + are TOML's; DEL escaped too), a boolean, an integer, a float, a list, a mapping (an inline table). TOML has no + null: None is refused.""" + if isinstance(value, bool): + return "true" if value else "false" + if isinstance(value, int): + return str(value) + if isinstance(value, float): + if value != value or value in (float("inf"), float("-inf")): + return "nan" if value != value else "inf" if value > 0 else "-inf" + return repr(value) + if isinstance(value, str): + return json.dumps(value, ensure_ascii=False).replace("\x7f", "\\u007F") + if isinstance(value, (list, tuple)): + return "[" + ", ".join(toml_value(v) for v in value) + "]" + if isinstance(value, dict): + return "{" + ", ".join(f"{toml_key(k)} = {toml_value(v)}" for k, v in value.items()) + "}" + raise ValueError(f"TOML cannot hold {value!r}") + + +def toml_key(key) -> str: + """A key as TOML writes it: bare when it can be, else quoted.""" + key = str(key) + return key if key and all(c.isascii() and (c.isalnum() or c in "-_") for c in key) else toml_value(key) + + +def write_task(task_dir: Path, instruction: str, *, metadata: dict | None = None, **sections: dict) -> Path: + """One task directory in Harbor's format: instruction.md (the instruction, as rows_of reads it back: trimmed, so + it must be trimmed already) and task.toml (`version`, [metadata], then each of `sections` as a table: + environment={"network_mode": "no-network"}, agent=..., verifier=...). tasks_of reads it back with every value as + given, in its order; a value TOML cannot hold (None) is refused.""" + if instruction != instruction.strip(): + raise ValueError(f"{task_dir}: the instruction is read back trimmed; trim it first") + lines = ['version = "1.0"'] + for name, table in {"metadata": metadata or {}, **sections}.items(): + if not table: + continue + lines += ["", f"[{name}]"] + for key, value in table.items(): + try: + lines.append(f"{toml_key(key)} = {toml_value(value)}") + except ValueError as exc: + raise ValueError(f"{task_dir}: [{name}] {key}: {exc}") from None + task_dir = Path(task_dir) + task_dir.mkdir(parents=True, exist_ok=True) + with open(task_dir / "instruction.md", "w", encoding="utf-8", newline="") as f: # its line endings kept + f.write(instruction + "\n") + (task_dir / TASK_FILE).write_text("\n".join(lines) + "\n", encoding="utf-8") + return task_dir + + +VERIFIER_KEYS = ("environment_mode", "timeout_sec", "env") # what the judge reads of [verifier] +VERIFIER_TIMEOUT_SEC = 600.0 # Harbor's default [verifier] timeout_sec + + +def tasks_of(tasks_dir: Path, *, rollouts: int = 1, view: Callable[[dict], dict] | None = None) -> dict[str, Task]: + """Every task under `tasks_dir` (one directory is a set of one), by id: each row's public half (or `view(row)`: + the row the task set's agents and learners read, `data.view`), and its private half naming its directory, its + tests (the task's, and a step's own laid over them), its verifier's settings, and `needs`: what it asks of its + environment (network modes, allowlist hosts, memory, its agent's timeout), applied to each call.""" + out: dict[str, Task] = {} + for task_dir in task_dirs(tasks_dir): + for row in rows_of(task_dir, rollouts=rollouts): + if row["id"] in out: + raise ValueError(f"two tasks under {tasks_dir} have the id {row['id']!r}") + step = row.get("step") + # The tests as Harbor shows them: the task's, a step's own laid over them (its test.sh first) + tests, step_tests = task_dir / "tests", (task_dir / "steps" / step / "tests") if step else None + config = task_config(task_dir) + step_cfg = _step(config, step) + verifier = {**(config.get("verifier") or {}), **(step_cfg.get("verifier") or {})} # a step's own first + verifier.setdefault("timeout_sec", VERIFIER_TIMEOUT_SEC) + verifier["environment_mode"] = verifier_environment(config, step)[0] + verifier["env"] = {**((config.get("verifier") or {}).get("env") or {}), + **((step_cfg.get("verifier") or {}).get("env") or {})} + private = {"task_dir": str(task_dir.resolve()), "tests": str(tests.resolve()) if tests.is_dir() else None, + "step_tests": str(step_tests.resolve()) if step_tests and step_tests.is_dir() else None, + "verifier": {k: v for k, v in verifier.items() if k in VERIFIER_KEYS}} + if private["tests"] is None and private["step_tests"]: + private["tests"], private["step_tests"] = private["step_tests"], None + verifier_env = verifier_environment(config, step)[1] + private["needs"] = {"network": row["network"], "allowed_hosts": allowed_hosts(config, step), + "memory_mb": row.get("memory_mb"), "agent_timeout_sec": row.get("agent_timeout_sec"), + "verifier_memory_mb": resources({"environment": verifier_env})["memory_mb"] + if verifier_env is not None else None, + "steps": row.get("steps")} + public = view(copy.deepcopy(row)) if view else row # a view never changes the task it reads + if public.get("id", row["id"]) != row["id"]: # the run's every record, split and key is the task's id + raise ValueError(f"data.view gave the task {row['id']!r} the id {public['id']!r}: a view keeps each " + "task's id (its directory's name), or gives none") + out[row["id"]] = Task(id=row["id"], public=public, private=private) + if not out: + raise ValueError(f"no task directories (with {TASK_FILE}) under {tasks_dir}") + return out diff --git a/hermit/task/task.py b/hermit/task/task.py new file mode 100644 index 00000000..d1fd38a9 --- /dev/null +++ b/hermit/task/task.py @@ -0,0 +1,55 @@ +"""One task: what the agent is given and what only the judge reads. + + task = Task.from_row({"id": "q1", "question": "...", "_private": {...}}) + task.public # {"id": "q1", "question": "..."}: what the acting agent gets, and what a run records as its input + task.private # what the judge alone may read (answers, tests, held-out data); never given to the agent + +A row of tasks.jsonl is all public: a task set keeps its answers apart (the judge reads them from where the agent's +sandbox cannot see, `.private/` beside the data), so `private` is empty for those rows. The split is a type, not a +convention: what an agent is handed is `task.public`, which has no field for the judge's own. +""" +from __future__ import annotations + +from dataclasses import dataclass, field +from types import MappingProxyType +from typing import Any, Mapping + +PRIVATE = "_private" # a row's key for what only the judge reads (a task directory's tests are private too) + + +@dataclass(frozen=True) +class Task: + id: str + public: Mapping[str, Any] + private: Mapping[str, Any] = field(default_factory=dict) + + def __post_init__(self): + if PRIVATE in self.public: + raise ValueError(f"task {self.id}: its public half holds {PRIVATE!r}, the judge's own") + object.__setattr__(self, "public", MappingProxyType(dict(self.public))) + object.__setattr__(self, "private", MappingProxyType(dict(self.private))) + + @classmethod + def from_row(cls, row: Mapping[str, Any]) -> "Task": + """A row with an `id` (or `task_id`); its `_private` mapping, when it has one, is the judge's own.""" + row = dict(row) + private = row.pop(PRIVATE, None) or {} + tid = str(row.get("id", row.get("task_id"))) + if tid in ("None", ""): + raise ValueError("every task needs an id") + row.setdefault("id", tid) + return cls(id=tid, public=row, private=private) + + def given(self) -> dict: + """What the acting agent is handed: a copy of the public half.""" + return dict(self.public) + + def row(self) -> dict: + """The task as a row again, what a judge is called with: the public half and, if any, the private one under + `_private`.""" + return {**self.public, PRIVATE: dict(self.private)} if self.private else dict(self.public) + + +def as_task(value) -> Task: + """A Task as it is, or one made from a row (what callers held before tasks were typed).""" + return value if isinstance(value, Task) else Task.from_row(value) diff --git a/hermit/task/verify.py b/hermit/task/verify.py new file mode 100644 index 00000000..4e1765e2 --- /dev/null +++ b/hermit/task/verify.py @@ -0,0 +1,173 @@ +"""The judge's verdict on one answer: a score in [0, 1], feedback a learning method can read, and anything else the +judge records (JudgeResult). How a task is judged is the task set's own, not the experiment's: a judge function in +this process on the answer it reads from where the agent could not (`judge.mode: host`), or in a sandbox beside the +agent's (`shared` over the agent's work, `separate` over what is collected; hermit.isolate builds those). + + verifier = judge_verifier(score) # score(task_row, output) -> JudgeResult | dict | float + verdict = verifier(task, output) # JudgeResult + +A task directory's verifier is its own script (`script` below, `judge: {path: hermit.task.verify, name: script, +mode: shared | separate}`), as Harbor runs it: `tests/test.sh` with the tests at /tests, in the agent's workspace +(its cwd; HERMIT_WORKSPACE) or a fresh copy of what is collected, the agent's output in the file HERMIT_OUTPUT names, +writing /logs/verifier/reward.txt (one number) or reward.json (named numbers; `reward` is the score, else their +mean), and, beyond Harbor, /logs/verifier/feedback.txt: what a learning method reads (else the script's output's +tail). It runs only in the judge's sandbox, which shows the task's tests and a fresh log directory at those names. + +A task in its own Harbor environment (`environment.harbor`: hermit.environment.harbor) is verified by Harbor's +verifier in that environment, after the agent, by the run (hermit.trainer.rollout); `environment` below is the judge +that reads that verdict (`judge: {path: hermit.task.verify, name: environment}`): the reward is the score, whatever the +session's end (its timeout, a CLI that exited with an error: Harbor verifies all of them, and its own job mean scores +them), 0 when the verifier left none (`verifier_failed`) or ran out of its time on the agent's solution +(`solution_timed_out`); a rollout that stayed infra after the run's runs again (its environment failed, its verifier +ran no test, its session never started: `unscored`) is not the agent's result: no score (Unscored), so the run records +a failed run (judge_error), as Hermit's Harbor runner's judge refused one. `passed` says whether the reward reached +the task set's pass reward. +""" +from __future__ import annotations + +import json +import math +import os +import re +from pathlib import Path +import subprocess +from typing import Callable, Protocol + +from hermit.common import RunOutput +from hermit.task.data import JudgeResult, normalise_judge +from hermit.task.task import Task + + +class Verifier(Protocol): + """Called with the whole Task. `takes_task = True` tells hermit.trainer.rollout so (a callable without it is taken + for a bare judge and wrapped by judge_verifier).""" + takes_task: bool + + def __call__(self, task: Task, output: RunOutput) -> JudgeResult: ... + + +TESTS = "/tests" # where the judge's sandbox shows a task's tests (Harbor's path) +LOGS = "/logs/verifier" # ... and the directory the script writes its reward to +SANDBOX_ENV = "HERMIT_VERIFIER_SANDBOX" # set by the judge's sandbox that shows them (hermit.isolate.SandboxedJudge) +FEEDBACK_CHARS = 4000 +TAIL = 1500 # characters of the verifier's output in the feedback of a task that did not pass + + +class Unscored(RuntimeError): + """A run that is not the agent's result (its environment failed): no score is the agent's.""" + + +def judge_verifier(judge: Callable) -> Verifier: + """A verifier over a judge as a task set writes it: called with the task's row (Task.row: its public half and, under + `_private`, what only the judge reads) and the agent's output; whatever it returns, normalised.""" + def verify(task: Task, output: RunOutput) -> JudgeResult: + return normalise_judge(judge(task.row(), output)) + verify.judge = judge # the judge it wraps (a sandboxed one keeps its own state) + verify.takes_task = True # called with a Task (hermit.trainer.rollout tells it from a bare judge) + return verify + + +def read_reward(logs: Path, output_tail: str = "") -> JudgeResult: + """The verdict a verifier script left in `logs`: reward.json (named numbers: `reward` is the score, else their + mean) over reward.txt (one number); feedback.txt, else the script's output's tail. No reward file: an error.""" + logs = Path(logs) + rewards: dict = {} + if (logs / "reward.json").is_file(): + value = json.loads((logs / "reward.json").read_text()) + # As Harbor reads it (harbor/verifier/verifier.py): finite numbers, a boolean among them (Python's bool is an + # int, so Harbor takes `true` for 1) + if not isinstance(value, dict) or not value or \ + not all(isinstance(v, (int, float)) and math.isfinite(v) for v in value.values()): + raise ValueError("reward.json must be an object of named finite numbers") + rewards = {str(k): float(v) for k, v in value.items()} + score = rewards.get("reward", sum(rewards.values()) / len(rewards)) + elif (logs / "reward.txt").is_file(): + score = float((logs / "reward.txt").read_text().strip()) + if not math.isfinite(score): # as Harbor: a NaN or infinite reward is no reward + raise ValueError(f"reward.txt holds no finite number: {score}") + else: + raise RuntimeError(f"the verifier wrote no reward ({LOGS}/reward.txt or reward.json): " + f"{output_tail[-FEEDBACK_CHARS:]}") + feedback = (logs / "feedback.txt").read_text(errors="replace") if (logs / "feedback.txt").is_file() \ + else output_tail + extra = {"rewards": rewards} if rewards else {} + return JudgeResult(float(score), feedback[-FEEDBACK_CHARS:], extra) + + +def resolve_env(env: dict) -> dict[str, str]: + """task.toml's [verifier.env] as Harbor resolves it (harbor/utils/env.py): `${NAME}` and `${NAME:-default}` from + this process's environment (the judge sandbox's: what environment.env lets in), anything else as written.""" + out = {} + for key, value in env.items(): + match = re.fullmatch(r"\$\{([A-Za-z_][A-Za-z0-9_]*)(?::-(.*))?\}", str(value)) + if not match: + out[str(key)] = str(value) + elif match.group(1) in os.environ: + out[str(key)] = os.environ[match.group(1)] + elif match.group(2) is not None: + out[str(key)] = match.group(2) + else: + raise RuntimeError(f"[verifier.env] {key} needs {match.group(1)}, which the judge's sandbox does not have " + "(environment.env)") + return out + + +def script(task: dict, output: RunOutput) -> JudgeResult: + """A task directory's own verifier, `tests/test.sh`, run in the judge's sandbox (module docstring).""" + if os.environ.get(SANDBOX_ENV) != "1" or not Path(TESTS, "test.sh").is_file(): + raise RuntimeError("the script verifier runs in the judge's sandbox, which shows the task's tests at /tests " + "(judge.mode: shared or separate, with the run's sandbox on)") + private = task.get("_private") or {} + timeout = (private.get("verifier") or {}).get("timeout_sec") + workspace = Path(str((output.extra or {}).get("workspace") or os.getcwd())) + answer = Path(LOGS) / "agent_output.json" + answer.write_text(json.dumps(output.output, ensure_ascii=False, default=str)) + env = {**os.environ, **resolve_env((private.get("verifier") or {}).get("env") or {}), + "HERMIT_OUTPUT": str(answer), "HERMIT_WORKSPACE": str(workspace)} + try: + done = subprocess.run(["bash", f"{TESTS}/test.sh"], cwd=str(workspace if workspace.is_dir() else Path.cwd()), + env=env, capture_output=True, text=True, timeout=float(timeout) if timeout else None) + tail = (done.stdout + done.stderr)[-FEEDBACK_CHARS:] + except subprocess.TimeoutExpired as exc: + out = (exc.stdout or b"") + (exc.stderr or b"") + tail = out.decode(errors="replace") if isinstance(out, bytes) else str(out) + raise RuntimeError(f"the verifier ran longer than its timeout_sec={timeout}: " + f"{tail[-FEEDBACK_CHARS:]}") from None + verdict = read_reward(Path(LOGS), tail) + verdict.extra["exit_code"] = done.returncode + return verdict + + +def environment(task: dict, output: RunOutput) -> JudgeResult: + """The verdict of the task's own environment's verifier (module docstring), which the run put in + `output.extra["verified"]` (hermit.environment.harbor.HarborEnvironments.verdict).""" + verdict = (output.extra or {}).get("verified") + if not isinstance(verdict, dict): + raise RuntimeError("the output holds no verdict of the task's environment: the environment judge is for a task " + "set in its own environments (environment.harbor)") + status, cause = verdict.get("status") or "infra", verdict.get("failure_cause") + exception = verdict.get("exception") or {} + why = (output.extra or {}).get("unscored") + if status == "infra" or why: + said = str(verdict.get("exception_message") or exception.get("message") or "")[:600] + raise Unscored(f"the rollout is not the agent's result ({why or cause or 'infra'}" + f"{': ' + said if said else ''}): it has no score") + reward = verdict.get("reward") + passed = verdict.get("passed") + extra = output.extra or {} + if status == "ok": + feedback = "passed" if passed else f"reward {reward:g}" if reward is not None else "no reward" + else: + feedback = f"{status} ({cause}{': ' + exception['type'] if exception.get('type') else ''})" + if cause == "solution_timed_out": + feedback += ("; the verifier ran out of its time on the agent's solution (too slow to finish its tests, " + "or a process it left that hung them)") + if extra.get("timed_out"): + feedback += "; the agent ran out of its time budget" + if not passed and verdict.get("feedback"): + text = str(verdict["feedback"]).strip() + feedback += "\nverifier output (tail):\n" + (text[-TAIL:] if len(text) > TAIL else text) + return JudgeResult(float(reward) if reward is not None else 0.0, feedback, + {"status": status, "passed": passed, "rewards": verdict.get("rewards"), + **({"failure_cause": cause} if cause else {}), + **({"trial_dir": verdict["trial_dir"]} if verdict.get("trial_dir") else {})}) diff --git a/hermit/trainer/rollout.py b/hermit/trainer/rollout.py index 86aa4ab3..54692c11 100644 --- a/hermit/trainer/rollout.py +++ b/hermit/trainer/rollout.py @@ -1,57 +1,357 @@ -"""Run tasks through the user's pipeline and score them with the judge; shared by training rollouts, eval and the -acting-agent runner offered to Evolvers.""" +"""Run tasks through the acting agent and judge each answer; shared by training rollouts, eval and the acting-agent +runner offered to Evolvers. This is where a task (hermit.task) meets the agent that answers it (hermit.agent): the +agent is handed the task's public half, the verifier reads the whole task, and neither knows the other.""" from __future__ import annotations from concurrent.futures import ThreadPoolExecutor, as_completed from dataclasses import asdict +import sys from pathlib import Path import threading import time import traceback from hermit.agent.base.interface import LoginFailed, login_failure -from hermit.task import RunOutput, normalise_judge +from hermit.common import RunOutput +from hermit.task import Task, as_task, judge_verifier from hermit.trainer.budget import Budget, BudgetExceeded -def run_one(task: dict, adapter: Path | None, pipeline, judge) -> dict: +def _given(pipeline, given: dict, adapter: Path | None, environment, needs: dict | None = None) -> RunOutput: + """The agent's run: handed the task's environment when it has its own, and a sandboxed agent (one that + `refuses`) the task's needs, whose agent timeout bounds the call.""" + kwargs = {"environment": environment} if environment is not None else {} + if getattr(pipeline, "refuses", None) is not None and needs is not None: + kwargs["needs"] = needs + out = pipeline(dict(given), adapter, **kwargs) + if not isinstance(out, RunOutput): + raise TypeError(f"the pipeline returned {type(out).__name__}, not RunOutput") + return out + + +def _failed_record(task: Task, given: dict, started: float, exc: Exception, what: str) -> dict: + spent = getattr(exc, "spent", None) # what the failed call billed, when the agent said + return {"id": given.get("id", task.id), "input": given, "status": "error", "error": f"{type(exc).__name__}: {exc}", + "traceback": traceback.format_exc()[-3000:], "output": None, + "judge": {"score": 0.0, "feedback": f"{what}: {type(exc).__name__}: {exc}"}, + "cost_usd": spent.cost_usd if spent is not None else None, "wall_s": round(time.time() - started, 2), + "api_speed": {"model_calls": (spent.api_speed or {}).get("model_calls")} if spent is not None else None, + "tokens": spent.tokens if spent is not None else None, "log": None, "messages": None} + + +def run_one(task: Task | dict, adapter: Path | None, pipeline, judge, environments=None) -> dict: + """One task: the agent (`pipeline(given, adapter) -> RunOutput`) on the task's public half, then the judge on + its answer: a verifier (`verifier(task, output) -> JudgeResult`, hermit.task.verify) or a bare judge + (`score(row, output)`); the record the run keeps. A record keeps the row's own id (7 stays 7). + + With `environments` (a task set whose tasks give their own environment: hermit.environment.harbor), the task's + environment is started first (it may wait for the disk gate), the agent is handed it (`pipeline(given, adapter, + environment=env)`), the environment's own verifier runs on the agent's work, whatever the session's end, and its + verdict is the output's `extra["verified"]`, which the judge reads (hermit.task.verify.environment); then the + environment is stopped. A rollout that is not the agent's result (infra: the environment did not start, failed + under the verifier, the session never started: hermit.environment.harbor_errors) runs again in a fresh environment + (environment_rollout); one that stays so is marked `unscored`, and the judge refuses to score it: a failed run + (judge_error), as Hermit's Harbor runner recorded one.""" + task = as_task(task) + verify = judge if getattr(judge, "takes_task", False) else judge_verifier(judge) + given = task.given() # what the agent is handed, and what the record shows started = time.time() + refuses = getattr(pipeline, "refuses", None) # a sandboxed agent: what the task asks that the run lacks + problems = refuses(task.private.get("needs") or {}) if refuses and environments is None else [] + if problems: # (a task's own environment gives what it asks itself) + exc = ValueError(f"task {task.id!r} cannot run in this run's environment: {'; '.join(problems)}") + return _failed_record(task, given, started, exc, "the pipeline failed") + if environments is not None: + out = environment_rollout(task, given, adapter, pipeline, environments, started) + return out if isinstance(out, dict) else _judged(task, given, out, verify, started) try: - out = pipeline(dict(task), adapter) - if not isinstance(out, RunOutput): - raise TypeError(f"the pipeline returned {type(out).__name__}, not RunOutput") + out = _given(pipeline, given, adapter, None, task.private.get("needs") or {}) except Exception as exc: why = login_failure(exc) # as text from a sandbox's worker, or from a chat endpoint if why is not None: raise LoginFailed(why) from None - return {"id": task["id"], "input": task, "status": "error", "error": f"{type(exc).__name__}: {exc}", - "traceback": traceback.format_exc()[-3000:], "output": None, - "judge": {"score": 0.0, "feedback": f"the pipeline failed: {type(exc).__name__}: {exc}"}, - "cost_usd": None, "wall_s": round(time.time() - started, 2), "api_speed": None, "tokens": None, - "log": None, "messages": None} + return _failed_record(task, given, started, exc, "the pipeline failed") failed = (out.extra or {}).get("session_failed") if failed: # no answer to judge; what it spent and its log still count - return {"id": task["id"], "input": task, "status": "error", "error": f"the agent's session failed: {failed}", + return {"id": given.get("id", task.id), "input": given, "status": "error", + "error": f"the agent's session failed: {failed}", "output": None, "judge": {"score": 0.0, "feedback": f"the agent's session failed: {failed}"}, "cost_usd": out.cost_usd, "wall_s": round(out.wall_s or (time.time() - started), 2), "api_speed": out.api_speed, "tokens": out.tokens, "log": str(out.log) if out.log else None, "messages": out.messages, "pipeline_extra": out.extra} + return _judged(task, given, out, verify, started) + + +def _read(out: RunOutput, env, task: Task, begun: float) -> RunOutput: + """The agent's output with its environment's verdict: the verifier's (verify), or for an adapter that failed before + any session the agent's result scored 0 with its message (the runner's adapter_failed, context_over_limit), or the + agent's own word that the run is not its result (`extra.infra`, read by the caller).""" + extra = out.extra or {} + if extra.get("infra"): + return out + if extra.get("adapter_failed"): + bar = getattr(getattr(env, "owner", None), "pass_reward", 1.0) # no pass reward: no pass rate + verdict = {"reward": None, "rewards": None, "status": "agent_error", "failure_cause": extra["adapter_failed"], + "passed": None if bar is None else False, "exception": None, + "exception_message": extra.get("exception_message"), + "feedback": str(extra.get("exception_message") or "")} + out.extra = {**extra, "verified": verdict, "status": "agent_error", "failure_cause": extra["adapter_failed"], + "verifier_reward_raw": None} + return out + logs = Path(out.log) if out.log and Path(out.log).is_dir() else None + verdict = env.verify(agent_logs=logs) + out.extra = {**extra, "verified": verdict, **_status(extra, verdict)} + if verdict["status"] == "infra": + out.extra["unscored"] = verdict["failure_cause"] or "infra" + return out + + +def _status(extra: dict, verdict: dict) -> dict: + """The rollout's status as Hermit's Harbor runner wrote it beside the verdict (`status`, `failure_cause`), which + the methods' readers take (AHE's classification): infra (not the agent's result), timeout (the agent's session ran + out of its time, or the verifier on its solution), agent_error (the CLI exited with an error or its session failed; + a verifier that left no reward), else the verifier's own, ok.""" + status, cause = verdict.get("status") or "infra", verdict.get("failure_cause") + if status != "infra": + if extra.get("timed_out"): + status, cause = "timeout", "agent_exhausted_budget" + elif extra.get("session_failed") or extra.get("exit_code") not in (None, 0): + status, cause = "agent_error", cause or "unclassified" + # and the verifier's own reward beside it (the runner's verifier_reward_raw: AHE asks whether a timed-out trial's + # work passed) + return {"status": status, "failure_cause": cause, "verifier_reward_raw": verdict.get("reward")} + + +def _infra_output(task: Task, cause: str, message: str, started: float, *, ran: bool, + spent: RunOutput | None = None) -> RunOutput: + """A rollout that ended infra before its environment's verdict (the environment did not start, the session never + did): no verdict but this one, which the judge refuses (hermit.task.verify.environment). Nothing ran when its + environment never came up (its cost 0); a session that may have run leaves its cost unknown.""" + verdict = {"status": "infra", "failure_cause": cause, "reward": None, "rewards": None, "passed": None, + "exception": {"type": None, "message": message[-2000:]}, "exception_message": message[-2000:], + "feedback": ""} + # what its sessions spent, when the agent said (`spent`: its output); unknown when one may have run unsaid + cost = spent.cost_usd if spent is not None else (None if ran else 0.0) + tokens = spent.tokens if spent is not None else (None if ran else 0) + calls = (spent.api_speed or {}).get("model_calls") if spent is not None else (None if ran else 0) + return RunOutput(output={"harbor_task": task.public.get("harbor_task"), "reward": None, "status": "infra", + "failure_cause": cause}, + log=spent.log if spent is not None else None, + cost_usd=cost, tokens=tokens, wall_s=round(time.time() - started, 1), + api_speed={"model_calls": calls, "status": "infra"}, + extra={"status": "infra", "failure_cause": cause, "exception_message": message[-2000:], + "verified": verdict, "unscored": cause}) + + +def _sleep(seconds: float) -> None: + """The pause before a rollout runs again (a test's stand-in replaces it).""" + time.sleep(seconds) + + +# The rollout's retry and admission states (the runner's _run_trial), after each run: the agent's result or a cause +# that runs no better again (DONE), not the agent's result run again in a fresh environment after infra_wait_s (INFRA, +# counted in infra_retries), a session the provider could not serve run again after n x busy_wait_s (PROVIDER_BUSY, +# its own allowance, busy_retries), a session the login's plan refused run again once the plan allows (PLAN_WAIT, not +# counted: the admission waits for the window; a hold that could not be written pauses HOLD_UNWRITTEN_S first), and +# the run ended meanwhile (STOP: this rollout as it stands). +DONE, INFRA, PROVIDER_BUSY, PLAN_WAIT, STOP = "done", "infra", "provider_busy", "plan_wait", "stop" + + +class Retries: + """The one retry and admission state machine of a rollout in its own environment: `next(out, final, said)`, the + state after a run and how long to pause before the next.""" + + def __init__(self, environments): + self.retries, self.counted = int(environments.infra_retries), 0 + self.busy, self.busy_retries = 0, int(getattr(environments, "busy_retries", 3)) + self.busy_wait_s = float(getattr(environments, "busy_wait_s", 120.0)) + self.infra_wait_s = float(environments.infra_wait_s) + self.stopping = getattr(environments, "stopping", None) + + def next(self, out: RunOutput, final: bool, said_by_agent: bool) -> tuple[str, float]: + from hermit.environment.harbor_errors import NO_RETRY + cause = out.extra.get("unscored") + if cause is None: + return DONE, 0.0 + provider = cause == "model_provider_failed" and said_by_agent and not final # a CLI session's provider + if provider: + self.busy += 1 # the provider's own allowance (busy_retries), not infra's + if self.busy > self.busy_retries: + return DONE, 0.0 + elif cause != "usage_limit": # the plan's refusal runs again once it allows: not counted + self.counted += 1 + if cause in NO_RETRY or final or self.counted > self.retries: + return DONE, 0.0 + if self.stopping is not None and self.stopping.is_set(): + return STOP, 0.0 + if cause == "usage_limit": + from hermit.agent.base.login import HOLD_UNWRITTEN_S + return PLAN_WAIT, HOLD_UNWRITTEN_S if out.extra.get("hold_unwritten") else 0.0 + if provider: + return PROVIDER_BUSY, self.busy * self.busy_wait_s # the n-th after n x busy_wait_s + return INFRA, self.infra_wait_s + + +def _scrub_log(out: RunOutput) -> None: + """The agent's log, scrubbed of this process's key values (hermit.redact) before a record names it: an agent in a + task's own environment runs in this process (no worker scrubs what it returns).""" + from hermit.redact import redact_text, redact_tree + log = Path(out.log) if out.log else None + if log is None or log.is_symlink(): + return + if log.is_dir(): + redact_tree(log) + elif log.is_file(): + try: + text = log.read_text(encoding="utf-8") + except (OSError, UnicodeDecodeError): + return + clean = redact_text(text) + if clean != text: + log.write_text(clean, encoding="utf-8") + + +def environment_rollout(task: Task, given: dict, adapter: Path | None, pipeline, environments, started: float, + *, pause=None): + """One task in its own environment, run again while it ends infra (not the agent's result: `unscored`) as the + Retries state machine says (the runner's `_run_trial`: infra_retries after infra_wait_s, never for a cause that + runs no better again; the provider's busy_retries; the plan's refusal once it allows); on a login the run's + admission to the plan (its wait and one of its slots, environments.admission) is held around the whole + environment, as the runner held it around a trial. What every run used is the rollout's (Attempts). Returns the + output, or the record of a pipeline that failed on its own (no environment's fault: the run's failed record, as + for any task).""" + from hermit.agent.model.usage import Attempts + attempts, retries = Attempts(), Retries(environments) + stopping = retries.stopping + while True: + if stopping is not None and stopping.is_set(): # the run ended: no environment starts for this task + raise LoginFailed("the run ended (a lost sign-in or the account's limit) before the task started") + run = _environment_run(task, given, adapter, pipeline, environments, started) + if isinstance(run, dict): # the pipeline's own failure: the run's failed record, with + if attempts.runs: # what the runs before it spent, and this one's (when it said) + attempts.add(RunOutput(output=None, cost_usd=run.get("cost_usd"), tokens=run.get("tokens"), + api_speed={"model_calls": (run.get("api_speed") or {}).get("model_calls")})) + total = attempts.output() + run = {**run, "tokens": total.tokens, "cost_usd": total.cost_usd, + "api_speed": {"model_calls": (total.api_speed or {}).get("model_calls")}, + "pipeline_extra": {**(run.get("pipeline_extra") or {}), "attempts": total.extra["attempts"]}} + return run + out, final, said_by_agent = run + _scrub_log(out) # the log the record names, scrubbed of the run's keys + attempts.add(out) + state, wait = retries.next(out, final, said_by_agent) + if state in (DONE, STOP): + return attempts.output() + message = str(out.extra.get("exception_message") or (out.extra.get("verified") or {}).get("exception_message") + or "") + print(f"hermit {task.id}: not the agent's result ({out.extra.get('unscored')}: {message[:300]}); again in " + f"{wait:.0f} s ({retries.counted}/{retries.retries})", file=sys.stderr, flush=True) + if not wait: + continue + if pause is not None: + pause(wait) + elif stopping is not None: + if stopping.wait(wait): # the run ended meanwhile: this rollout as it stands + return attempts.output() + else: + _sleep(wait) + + +def _environment_run(task: Task, given: dict, adapter: Path | None, pipeline, environments, started: float): + """One run of the task in a fresh environment: (its output, whether it runs no better again, whether the agent + itself said it is not its result), or the pipeline's failed record.""" + import contextlib + from hermit.environment.harbor import EnvironmentUnavailable + from hermit.environment.harbor_errors import AGENT_PHASE_OVER, pipeline_cause + stopping = getattr(environments, "stopping", None) + begun, final, out, said_by_agent = time.time(), False, None, False + admission = getattr(environments, "admission", None) + try: + with (admission() if admission is not None else contextlib.nullcontext()): + installs = getattr(environments, "installs_for", None) + with environments.session(environments.task_dir(task), task.id, + installs(adapter) if installs is not None else None) as env: + env.plan_admitted = admission is not None + try: + out = _given(pipeline, given, adapter, env, task.private.get("needs") or {}) + except LoginFailed: + if stopping is not None: # the run ends now: no task admitted while this one stops + stopping.set() + raise + except Exception as exc: + why = login_failure(exc) + if why is not None: + if stopping is not None: + stopping.set() + raise LoginFailed(why) from None + if AGENT_PHASE_OVER in str(exc): # Harbor's agent timeout ended the phase: its verdict + out = _read(RunOutput(output=None, extra={"timed_out": True, "session_failed": + f"{type(exc).__name__}: {exc}"[-600:]}), + env, task, begun) + kept = (out.extra.get("verified") or {}).get("agent_dir") + if out.log is None and kept and Path(kept).is_dir(): + out.log = Path(kept) # the trial's agent logs: the stream the CLI wrote there + usage_of = getattr(environments, "usage_of", None) + reported = usage_of(Path(kept)) if usage_of is not None else [] + if reported: # what the session had reported before it was cut off + from hermit.agent.model.usage import Ledger + ledger = Ledger().add(reported) + out.cost_usd, out.tokens = ledger.cost(None), ledger.tokens(None) + out.api_speed = {**(out.api_speed or {}), "model_calls": ledger.calls()} + elif isinstance(exc, OSError): # this machine's (the runner's host_failed): never run again + out = _infra_output(task, "host_failed", f"{type(exc).__name__}: {exc}", begun, ran=True) + final = True + else: + read = pipeline_cause(f"{type(exc).__name__}: {exc}") + if read is None: + return _failed_record(task, given, started, exc, "the pipeline failed") + cause, again = read + final = not again + out = _infra_output(task, cause, f"{type(exc).__name__}: {exc}", begun, ran=True, + spent=getattr(exc, "spent", None)) # what the agent said it spent + else: + out = _read(out, env, task, begun) + if out.extra.get("infra"): + final, said_by_agent = bool(out.extra.get("infra_final")), True + unwritten = bool(out.extra.get("hold_unwritten")) + out = _infra_output(task, out.extra["infra"], str(out.extra.get("exception_message")), + begun, ran=True, spent=out) + if unwritten: + out.extra["hold_unwritten"] = True + except EnvironmentUnavailable as exc: + if exc.cause == "stopped": # the run ended (another task's lost sign-in) + raise LoginFailed(str(exc)) from None + out = _infra_output(task, exc.cause, str(exc), begun, ran=False) + except ValueError as exc: # no task directory: the task set's, not the environment's + return _failed_record(task, given, started, exc, "the task's environment could not be given") + except OSError as exc: # this machine's, the environment's teardown among it + if out is None or out.extra.get("unscored") is None and "verified" not in out.extra: + out = _infra_output(task, "host_failed", f"{type(exc).__name__}: {exc}", begun, ran=out is not None, + spent=out) + final = True + print(f"WARNING hermit {task.id}: this machine failed after the task's verdict: {exc}", file=sys.stderr, + flush=True) + return out, final, said_by_agent + + +def _judged(task: Task, given: dict, out: RunOutput, verify, started: float) -> dict: try: - verdict = normalise_judge(judge(dict(task), out)) + verdict = verify(task, out) judged = asdict(verdict) status, error = "ok", None except Exception as exc: judged = {"score": 0.0, "feedback": f"the judge failed: {type(exc).__name__}: {exc}"} status, error = "judge_error", f"{type(exc).__name__}: {exc}" - return {"id": task["id"], "input": task, "status": status, "error": error, "output": out.output, + return {"id": given.get("id", task.id), "input": given, "status": status, "error": error, "output": out.output, "judge": judged, "cost_usd": out.cost_usd, "wall_s": round(out.wall_s or (time.time() - started), 2), "api_speed": out.api_speed, "tokens": out.tokens, "log": str(out.log) if out.log else None, "messages": out.messages, "pipeline_extra": out.extra} -def run_tasks(tasks: list[dict], adapter: Path | None, pipeline, judge, *, concurrency: int, - budget: Budget | None = None, part: str = "rollout") -> list[dict]: - """All tasks, in the given order. Raises BudgetExceeded (after in-flight tasks end) when a hard budget is hit.""" +def run_tasks(tasks: list[Task | dict], adapter: Path | None, pipeline, judge, *, concurrency: int, + budget: Budget | None = None, part: str = "rollout", environments=None) -> list[dict]: + """All tasks, in the given order. Raises BudgetExceeded (after in-flight tasks end) when a hard budget is hit. + `environments`: the task set's own environments (run_one).""" results: dict[str, dict] = {} stop = threading.Event() @@ -59,9 +359,11 @@ def work(task): if stop.is_set(): return None try: - record = run_one(task, adapter, pipeline, judge) + record = run_one(task, adapter, pipeline, judge, environments) except LoginFailed: stop.set() # the tasks waiting would fail the same way + if environments is not None and getattr(environments, "stopping", None) is not None: + environments.stopping.set() # and no disk gate or pause before a run again waits on raise if budget is not None: budget.add(part, usd=record["cost_usd"] or 0.0, tokens=record["tokens"] or 0) @@ -72,12 +374,13 @@ def work(task): stop.set() return record + tasks = [as_task(t) for t in tasks] with ThreadPoolExecutor(max_workers=max(1, int(concurrency))) as pool: - futures = [pool.submit(work, t) for t in tasks] + futures = {pool.submit(work, t): t.id for t in tasks} for future in as_completed(futures): record = future.result() if record is not None: - results[record["id"]] = record + results[futures[future]] = record # by the task's id: a record keeps the row's own if stop.is_set() and budget is not None: raise BudgetExceeded(budget.exceeded() or "budget reached") - return [results[t["id"]] for t in tasks if t["id"] in results] + return [results[t.id] for t in tasks if t.id in results] diff --git a/hermit/trainer/trainer.py b/hermit/trainer/trainer.py index ab00cc0a..869c2c75 100644 --- a/hermit/trainer/trainer.py +++ b/hermit/trainer/trainer.py @@ -42,12 +42,12 @@ from hermit.agent.evolver.loader import find, instantiate from hermit.agent.model.llm import ChatClient from hermit.isolate import CHATGPT_HOSTS, HOME_SECRETS, Recheck, ToolRunner, _cli_kind, _safe, agent_pipeline, \ - cli_sandbox, config_paths, host_pipeline, judge_for, run_secrets, sanitized_copy + cli_sandbox, config_paths, host_pipeline, judge_for, run_secrets, sanitized_copy, task_environments from hermit.redact import redact_name, redact_path, redact_tree, redact_value from hermit.sandbox import (AllowlistProxy, Bind, Sandbox, drop_unsafe, hidden_paths, inside_tools, model_access, set_hidden) from hermit.spec import check_spec, repair -from hermit.task import TaskData +from hermit.task import Benchmark from hermit.tools import FunctionTool, ToolServer, assemble, evolver_agent from hermit.trainer import card from hermit.trainer.budget import Budget, BudgetExceeded @@ -208,13 +208,13 @@ def __init__(self, config: Config, *, python: str | None = None, log=print): self.run_name = config.run_name() self.run_dir = config.run_dir() self.branch = self.run_name - self.data = TaskData(config.path(d["data"]["tasks"]), config.path(d["data"]["train"]), - config.path(d["data"]["val"]), config.path(d["data"]["test"])) + self.data = Benchmark.from_config(config) # The spec checks run the adapter's modules in a strict sandbox, as strictly as the run asks; None: the run's # sandbox is off and they run unconfined like everything else. self.enforce = d["sandbox"].get("enforce", "required") if d["sandbox"]["enabled"] else None self.pipeline = None # set when the run starts: the agent runs in its sandbox, never imported here self.judge = None # set when the run starts (judge_for): a sandboxed judge is never imported here + self.environments = None # ... and the task set's own environments (task_environments), if any self._login_failed: LoginFailed | None = None # a sign-in that failed inside a learner's tool call self.spec = find(d["evolver"]["name"], config.base) self.evolver, self.params = instantiate(self.spec, d["evolver"]["params"], @@ -293,8 +293,8 @@ def _prepare(self) -> None: val = self.run_dir / "access" / "validation" val.mkdir(parents=True, exist_ok=True) with open(val / "tasks.jsonl", "w") as f: - for row in self.data.rows("val"): - f.write(json.dumps(row, ensure_ascii=False) + "\n") + for task in self.data.split("val"): # each task's public half, never the judge's own + f.write(json.dumps(task.given(), ensure_ascii=False) + "\n") env = tuple(config_paths(self.config, "evolver_access.environment", d["evolver_access"]["environment"])) self.access = Access(validation=val, environment=env) self._recheck_grants = Recheck([*env, self.ws.task_info], own=self.run_dir) @@ -322,6 +322,7 @@ def _prepare(self) -> None: # for the Evolver), its judge when sandboxed, and the tools a template or task declares. self.pipeline, self._close_pipeline = agent_pipeline(self.config, "rollout", records=records) self.judge = judge_for(self.config, self.pipeline) + self.environments = task_environments(self.config, log=self.log) # a task set's own environments self.tool_runner = ToolRunner(self.config, records=records, work_root=self.run_dir / "tool_calls") (self.run_dir / "agent_logs").mkdir(exist_ok=True) # run_agent's log copies, shown read-only binds = [Bind(self.ws.adapter, "/w/adapter", True), Bind(self.ws.learning_material, "/w/material"), @@ -561,7 +562,7 @@ def run_agent(task_ids: list, adapter: str | None = None, _snapshot=None) -> dic try: results = run_tasks(tasks, snapshot, self.pipeline, self.judge, concurrency=int(self.config["agent"]["concurrency"]), budget=self.budget, - part="evolver") + part="evolver", environments=self.environments) finally: shutil.rmtree(snapshot, ignore_errors=True) # A training task's run also names its log (the trajectory, when the pipeline keeps one), copied into the @@ -572,7 +573,10 @@ def run_agent(task_ids: list, adapter: str | None = None, _snapshot=None) -> dic | ({"log": self._visible_log(r)} if r["id"] in allowed["train"] else {}) | ({"error": r.get("error")} if r.get("status") == "error" else {}) | {k: (r.get("pipeline_extra") or {})[k] for k in ("exit_code", "timed_out") - if k in (r.get("pipeline_extra") or {})} for r in results] + if k in (r.get("pipeline_extra") or {})} + # a task in its own environment: the rollout's outcome, as the Harbor runner's output carried it + | ({"outcome": {k: (r.get("pipeline_extra") or {}).get(k) for k in ("status", "failure_cause")}} + if "verified" in (r.get("pipeline_extra") or {}) else {}) for r in results] return {"results": redact_value(shown)} return FunctionTool("run_agent", "Run an adapter (default: your working copy) on training or validation " "tasks through the task's pipeline and judge; returns outputs and scores, and for " @@ -678,7 +682,8 @@ def _step(self, progress: Progress, chunk: list[str], feedback: list | None, ses # 1–2. rollouts with the current adapter (exported fresh from the commit). rollout_adapter = self.tree.export(parent, self.run_dir / "rollout_adapter") results = run_tasks([self.data.tasks[t] for t in chunk], rollout_adapter, self.pipeline, self.judge, - concurrency=int(d["agent"]["concurrency"]), budget=self.budget) + concurrency=int(d["agent"]["concurrency"]), budget=self.budget, + environments=self.environments) rollout_s = time.time() - t0 changed_since_last = self._last_rollout_commit is not None and parent != self._last_rollout_commit self._last_rollout_commit = parent diff --git a/hermit/worker.py b/hermit/worker.py index 1c5db1d6..982ec656 100644 --- a/hermit/worker.py +++ b/hermit/worker.py @@ -4,10 +4,10 @@ python worker.py # the job arrives as JSON on stdin; HA_RESULT_FD names the pipe the result leaves by A job is {"kind": "agent", "name", "params", "base", "task", "adapter"} (what `load_agent(...).run(task, adapter)` -takes), {"kind": "judge", "spec", "base", "task", "output"} (the judge on an output record, in a sandboxed judge -mode) or {"kind": "tool", "entrypoint", "base", "arguments", "bound"} (a template's or task's declared tool). The -result is {"ok": true, "output": } / {"ok": true, "result": ...} or {"ok": false, "error", -"traceback"}. +takes), {"kind": "judge", "spec", "base", "task", "output"} +(the judge on an output record, in a sandboxed judge mode) or {"kind": "tool", "entrypoint", "base", "arguments", +"bound"} (a template's or task's declared tool). The result is {"ok": true, "output": } / {"ok": true, +"result": ...} or {"ok": false, "error", "traceback"}. The result pipe is made non-inheritable before anything else runs, so a program the agent starts cannot write a result of its own (a score it did not earn, a cost it did not spend). Whatever the agent prints goes to its own @@ -31,7 +31,8 @@ def _run(job: dict) -> dict: os.environ[ENV_SANDBOX] = "1" os.environ[ENV_PYTHON] = sys.executable agent = load_agent(job["name"], job.get("params"), Path(job["base"])) - out = agent.run(job["task"], Path(job["adapter"]) if job.get("adapter") else None) + adapter = Path(job["adapter"]) if job.get("adapter") else None + out = agent.run(job["task"], adapter) if not isinstance(out, RunOutput): raise TypeError(f"the pipeline returned {type(out).__name__}, not RunOutput") return {"ok": True, "output": output_record(out)} @@ -73,6 +74,10 @@ def main() -> None: except BaseException as exc: # the host turns this into the task's error record payload = {"ok": False, "error": f"{type(exc).__name__}: {exc}", "traceback": traceback.format_exc()[-3000:]} + spent = getattr(exc, "spent", None) # what the failed call billed, counted on the host's record + if spent is not None: + payload["spent"] = {"cost_usd": spent.cost_usd, "tokens": spent.tokens, + "model_calls": (spent.api_speed or {}).get("model_calls")} result.write(json.dumps(payload, default=str)) diff --git a/tests/core/test_base_agents.py b/tests/core/test_base_agents.py index 5454a986..39eed84c 100644 --- a/tests/core/test_base_agents.py +++ b/tests/core/test_base_agents.py @@ -10,7 +10,7 @@ from hermit.agent.base.interface import Limits, Mount, Unsupported, make from hermit.agent.base.tool import Tool from hermit.agent.base.tool_server import ToolServer -from hermit.agent.environment import Container, Local, Sandboxed +from hermit.environment import Container, Local, Sandboxed from hermit.agent.model.direct import Direct diff --git a/tests/core/test_claude_subscription.py b/tests/core/test_claude_subscription.py index b997d15f..105610dc 100644 --- a/tests/core/test_claude_subscription.py +++ b/tests/core/test_claude_subscription.py @@ -28,7 +28,7 @@ from hermit.sandbox import Sandbox from claude_stand import INDEX, ROOT, STREAMS, TOKEN, bwrap_works, stand # noqa: F401 (stand: the fixture) -from hermit.agent.environment import Container, Local, Sandboxed +from hermit.environment import Container, Local, Sandboxed EXAMPLE = ROOT / "examples" / "claude_subscription" @@ -492,7 +492,7 @@ def test_a_container_session_reaches_its_tool_server_directly(tmp_path, monkeypa def test_the_agent_checks_the_images_claude_code_and_its_sandbox_before_a_session(tmp_path, monkeypatch): """Before the first session: Claude Code in the image, recent enough for the settings checked here, and its sandbox able to start there (else every command would fail and the session still count as served).""" - from hermit.agent.environment import docker as container + from hermit.environment import docker as container bin_dir = tmp_path / "bin" bin_dir.mkdir() (bin_dir / "docker").write_text(FAKE_DOCKER) @@ -514,7 +514,7 @@ def test_the_agent_checks_the_images_claude_code_and_its_sandbox_before_a_sessio def test_a_secret_by_descriptor_is_one_env_passes_and_named_plainly(tmp_path): - from hermit.agent.environment import docker as container + from hermit.environment import docker as container kw = dict(conf={"image": "i", "network": "n", "proxy": "p"}, work=tmp_path, home=tmp_path, mounts=[], timeout=1, stdout=tmp_path / "out") with pytest.raises(ValueError, match="does not pass"): @@ -1981,7 +1981,7 @@ def real_container(real_cli): container settings, the network's gateway, where a scripted model listens for the container). The model's URL is plain HTTP on the gateway, as the tools' server is, so the sessions are told to reach it directly (NO_PROXY), and to try once (the host's variables do not reach a container).""" - from hermit.agent.environment import docker as container + from hermit.environment import docker as container conf = container.settings({}, "claude") if not shutil.which("docker") or container.check(conf, "claude"): pytest.skip("needs docker, Claude Code's image and its network (HERMIT_CLAUDE_IMAGE, HERMIT_CLAUDE_NETWORK)") diff --git a/tests/core/test_cli_agent.py b/tests/core/test_cli_agent.py index d366f443..27c7e6ad 100644 --- a/tests/core/test_cli_agent.py +++ b/tests/core/test_cli_agent.py @@ -15,8 +15,8 @@ from hermit.agent.base import cli as base_cli from hermit.agent.base.interface import Limits as LIMITS from hermit.agent.base import login -from hermit.agent.environment import docker as container -from hermit.agent.environment import Sandboxed +from hermit.environment import docker as container +from hermit.environment import Sandboxed from hermit.agent.base.interface import make ROOT = Path(__file__).resolve().parents[2] @@ -1201,7 +1201,7 @@ def test_a_stopped_session_takes_a_descendant_in_a_session_of_its_own(tmp_path, import subprocess import time if not proc_fs: - from hermit.agent.environment import base as environment_base # where _parents reads /proc + from hermit.environment import base as environment_base # where _parents reads /proc monkeypatch.setattr(environment_base, "Path", lambda p: Path("/no-proc") if p == "/proc/self/stat" else Path(p)) pid_file = tmp_path / "pid" diff --git a/tests/core/test_dsh_agent.py b/tests/core/test_dsh_agent.py index 08883d8b..3508ef9d 100644 --- a/tests/core/test_dsh_agent.py +++ b/tests/core/test_dsh_agent.py @@ -22,7 +22,7 @@ from hermit.agent.base import trajectory from hermit.agent.base.interface import Unsupported, make from hermit.agent.base.tool import Tool -from hermit.agent.environment import Container, Local, Sandboxed, placed +from hermit.environment import Container, Local, Sandboxed, placed from hermit.agent.model.backends import bind_agent_params, registry ROOT = Path(__file__).resolve().parents[2] diff --git a/tests/core/test_environment.py b/tests/core/test_environment.py index dfd74b5f..0a58c0c4 100644 --- a/tests/core/test_environment.py +++ b/tests/core/test_environment.py @@ -9,7 +9,7 @@ import pytest from hermit.agent.base.interface import Unsupported, make -from hermit.agent.environment import Container, Limits, Local, Sandboxed, as_environment, placed +from hermit.environment import Container, Limits, Local, Sandboxed, as_environment, placed from hermit.agent.model.usage import Usage @@ -83,7 +83,7 @@ def test_a_sandboxed_environment_writes_only_its_workspace(tmp_path): @pytest.mark.skipif(shutil.which("bwrap") is None, reason="needs bubblewrap") def test_a_sandboxed_command_reads_its_mounts_read_only_within_its_clock(tmp_path): - from hermit.agent.environment import Mount + from hermit.environment import Mount from hermit.sandbox import Sandbox data, work = tmp_path / "data", tmp_path / "w" data.mkdir() @@ -342,15 +342,12 @@ def test_no_shipped_config_runs_a_method_on_an_agent_that_lacks_what_it_needs(): assert checked >= 10 # MWM, ASPIRE, ReHarness and AHE's examples -def test_harbor_takes_what_its_acting_agent_takes(tmp_path): - from hermit.agent.acting.agent import supports_of - tasks = tmp_path / "tasks" - tasks.mkdir() - for acting, expected in (({"cli": "codex"}, "all"), ({"cli": "answer"}, ("context", "context_tool")), - ({"name": "oracle"}, ())): - supports = supports_of("harbor", tmp_path, {"tasks_dir": str(tasks), "acting_agent": acting}, - may_import=False) - assert (len(supports) == 6 and "hooks" in supports) if expected == "all" else supports == expected, acting +def test_the_answer_agent_takes_the_context_and_harbor_is_no_agent_any_more(tmp_path): + import pytest + from hermit.agent.acting.agent import load_agent, supports_of + assert supports_of("answer", tmp_path, {}, may_import=False) == ("context", "context_tool") + with pytest.raises(ValueError, match="agent.name: harbor is gone"): + load_agent("harbor", {}, tmp_path) def test_a_download_keeps_a_link_as_the_link_it_is(tmp_path): @@ -363,7 +360,7 @@ def test_a_download_keeps_a_link_as_the_link_it_is(tmp_path): def test_a_local_command_stays_within_the_environments_limits(tmp_path): - from hermit.agent.environment import Mount, Unsupported + from hermit.environment import Mount, Unsupported data = tmp_path / "data" data.mkdir() (data / "fact.txt").write_text("42") @@ -419,7 +416,7 @@ def test_a_cli_endpoints_sessions_keep_their_isolation_whether_or_not_they_are_l def test_a_sandboxed_command_is_refused_a_limit_its_sandbox_cannot_keep(tmp_path): """What the run's sandbox grants (a readable path, an allowed host, a forwarded port) it shows to every command, and a sandbox that fell back to running unisolated keeps no limit: such a limit is refused, never ignored.""" - from hermit.agent.environment import Unsupported + from hermit.environment import Unsupported class Box: enabled, network, allow, forward, sockets, binds = True, "isolated", (), (), {}, () @@ -444,7 +441,7 @@ def wrap(self, command, **kw): def test_a_local_mount_replaces_an_earlier_link_and_refuses_the_workspaces_own_file(tmp_path): - from hermit.agent.environment import Mount, Unsupported + from hermit.environment import Mount, Unsupported old, new, work = tmp_path / "old", tmp_path / "new", tmp_path / "w" for d, text in ((old, "old"), (new, "new")): d.mkdir() @@ -482,7 +479,7 @@ def test_an_upload_keeps_a_link_to_a_folder_as_the_link_it_is(tmp_path): def test_a_limited_network_is_refused_where_the_sandbox_has_a_proxy(tmp_path): - from hermit.agent.environment import Unsupported + from hermit.environment import Unsupported class Box: enabled, network, allow, forward, sockets, binds, readable, writable = True, "isolated", (), (), {}, (), (), () @@ -503,15 +500,6 @@ def lookup(paths: list[str], ids: Optional[list[int]] = None, raw: list = (), la "raw": {"type": "array", "items": {}}, "later": {"type": "array", "items": {"type": "string"}}} -def test_harbor_reads_its_acting_agent_as_the_runner_merges_it(tmp_path): - from hermit.agent.acting.agent import supports_of - (tmp_path / "tasks").mkdir() - (tmp_path / "benchmark.yaml").write_text("acting_agent: {cli: answer}\n") - given = {"tasks_dir": str(tmp_path / "tasks"), "benchmark": "benchmark.yaml"} - assert supports_of("harbor", tmp_path, given, may_import=False) == ("context", "context_tool") - assert "hooks" in supports_of("harbor", tmp_path, {**given, "acting_agent": None}, may_import=False) # the default - - def test_an_endpoint_entered_after_it_was_started_serves_once_and_stops_on_close(): """cli_endpoint starts its endpoint and callers enter it with `with`: a second serving loop would outlive close() and spin on the closed socket, slowing the whole process.""" @@ -572,7 +560,7 @@ def test_a_folder_upload_never_writes_through_a_link_in_the_workspace(tmp_path): @pytest.mark.skipif(shutil.which("bwrap") is None, reason="needs bubblewrap") def test_a_sandboxed_command_can_start_in_a_folder_only_the_sandbox_shows(tmp_path): - from hermit.agent.environment import Mount + from hermit.environment import Mount from hermit.sandbox import Sandbox data, work = tmp_path / "data", tmp_path / "w" data.mkdir() diff --git a/tests/core/test_harbor_dataset.py b/tests/core/test_harbor_dataset.py index b2cba223..c5c98619 100644 --- a/tests/core/test_harbor_dataset.py +++ b/tests/core/test_harbor_dataset.py @@ -1,7 +1,7 @@ """A benchmark's task set in Harbor's format made Hermit data by `python -m hermit.harbor.dataset prepare` (its benchmark file's `dataset:` block), without Harbor or a registry: the download (a stand-in), the patch, the rows with their resources and network modes, the lock and its check on reuse, the splits, the smoke rows and the images' -sizes; and, without Docker, the teardown of a trial's compose projects (hermit.harbor.docker).""" +sizes; and, without Docker, the teardown of a trial's compose projects (hermit.environment.harbor_docker).""" import errno import json import os @@ -718,9 +718,9 @@ def test_a_relative_data_directory_is_the_benchmark_files_not_the_working_direct @pytest.mark.skipif(not shutil.which("git"), reason="needs git") def test_a_trials_compose_projects_are_taken_down_from_an_empty_directory_of_their_own(tmp_path, monkeypatch): - """hermit.harbor.docker.compose_down names the trial's projects alone (`-p`): it runs where no compose file or - .env of this process's working directory can be read, whose volumes `down --volumes` would remove too.""" - from hermit.harbor import docker + """hermit.environment.harbor_docker.compose_down names the trial's projects alone (`-p`): it runs where no compose + file or .env of this process's working directory can be read, whose volumes `down --volumes` would remove too.""" + from hermit.environment import harbor_docker as docker (tmp_path / "compose.yaml").write_text("services: {}\nvolumes: {mine: {}}\n") (tmp_path / ".env").write_text("COMPOSE_FILE=compose.yaml\n") monkeypatch.chdir(tmp_path) @@ -744,7 +744,7 @@ def test_a_trials_compose_projects_are_taken_down_with_none_of_this_processs_com """A compose file this process's environment selects (COMPOSE_FILE, and its profiles, env files, project directory) is not read by the teardown either, whose `down --volumes` would remove its volumes too: each `down` runs with none of the COMPOSE_* variables, and with every other one (the Docker daemon it reaches).""" - from hermit.harbor import docker + from hermit.environment import harbor_docker as docker (tmp_path / "compose.yaml").write_text("services: {}\nvolumes: {mine: {name: mine}}\n") for name, value in (("COMPOSE_FILE", str(tmp_path / "compose.yaml")), ("COMPOSE_PROFILES", "all"), ("COMPOSE_ENV_FILES", str(tmp_path / ".env")), ("COMPOSE_PROJECT_DIRECTORY", str(tmp_path)), @@ -770,7 +770,7 @@ def run(cmd, **kwargs): def test_with_no_room_for_an_empty_directory_a_trials_projects_are_taken_down_from_the_root(tmp_path, monkeypatch): """With no room for an empty directory (a full /tmp) the teardown still takes the trial's projects down, from `/`, not from this process's working directory and its compose file.""" - from hermit.harbor import docker + from hermit.environment import harbor_docker as docker (tmp_path / "compose.yaml").write_text("services: {}\nvolumes: {mine: {}}\n") monkeypatch.chdir(tmp_path) @@ -807,7 +807,7 @@ def test_a_verifier_sessions_name_is_harbors_own_and_a_cut_one_is_taken_down_by_ 63 characters, past them cut and named by its digest, which no prefix of the trial's name finds. compose_down takes the one of each key down by that exact name (a single-step trial's "trial", a stream's step names), and any other Harbor kept whole by the prefix, each project once.""" - from hermit.harbor import docker + from hermit.environment import harbor_docker as docker for (trial, key), name in VERIFIER_SESSIONS.items(): assert docker.verifier_session(trial, key) == name trial = "long-task-name-for-a-stream-of-steps-x__s1__0a1b2c3d" # a stream's trial: 52 characters diff --git a/tests/core/test_harbor_environment.py b/tests/core/test_harbor_environment.py new file mode 100644 index 00000000..9745412a --- /dev/null +++ b/tests/core/test_harbor_environment.py @@ -0,0 +1,2908 @@ +"""A task's Harbor environment (hermit/environment/harbor.py) and its driver, a CLI Base Agent in a remote +environment, the answer-only agent, and the run that +starts each task's environment, hands it to the agent and verifies with it (hermit/trainer/rollout.py). + +Stand-ins only at the edge: Harbor's driver process is a small script speaking the driver's protocol (JSON lines), +which plays the task's container with a directory of this machine; a remote environment is an Environment whose exec +records the commands and answers with a CLI's stream; Docker's daemon is not asked (the disk gate and the image +cleanup are recorded). The last test runs the real driver against Docker and Harbor's venv, and skips where either is +absent.""" +from __future__ import annotations + +import json +import os +from pathlib import Path, PurePosixPath +import shlex +import shutil +import subprocess +import sys +import textwrap + +import pytest + +from hermit.common import RunOutput +from hermit.environment import harbor as harbor_mod +from hermit.environment.base import Environment, ExecResult, Limits, Unsupported +from hermit.environment.harbor import EnvironmentUnavailable, HarborEnvironments + +# The driver's stand-in: the container is the folder FAKE_ROOT, its workdir FAKE_ROOT/app; `done` writes the reward +# a file /app/reward-to-give names (0 when there is none), or fails as FAKE_VERIFY says. +FAKE_DRIVER = textwrap.dedent(''' + import json, os, shutil, subprocess, sys + from pathlib import Path + root = Path(os.environ["FAKE_ROOT"]) + out = os.fdopen(os.dup(1), "w"); os.dup2(2, 1) + def local(p): return root / str(p).lstrip("/") + seen = root.parent / "requests.jsonl" + for line in sys.stdin: + req = json.loads(line) + with open(seen, "a") as f: + f.write(json.dumps({k: v for k, v in req.items() if k != "stdin"}) + "\\n") + op, rid = req["op"], req["id"] + reply = {"id": rid, "ok": True} + if op == "start": + (root / "app").mkdir(parents=True, exist_ok=True) + if os.environ.get("FAKE_START") == "fail": + reply = {"id": rid, "ok": False, "error": "RuntimeError: the image did not pull"} + else: + reply.update(workdir="/app", agent_timeout_sec=900.0, trial_dir=str(root.parent / "trial")) + if os.environ.get("FAKE_ELAPSED"): # the phase began this long before the reply + import time + reply["deadline_epoch"] = time.time() + 900.0 - float(os.environ["FAKE_ELAPSED"]) + elif op == "exec": + env = {**os.environ, **req.get("env", {})} + done = subprocess.run(["bash", "-c", req["command"]], cwd=local(req["cwd"]), env=env, input=None, + capture_output=True, text=True) + reply.update(code=done.returncode, output=done.stdout + done.stderr, timed_out=False) + elif op == "upload": + src, dst = Path(req["source"]), local(req["target"]) + dst.parent.mkdir(parents=True, exist_ok=True) + shutil.copytree(src, dst, dirs_exist_ok=True) if src.is_dir() else shutil.copy2(src, dst) + elif op == "download": + src, dst = local(req["source"]), Path(req["target"]) + dst.parent.mkdir(parents=True, exist_ok=True) + shutil.copytree(src, dst, dirs_exist_ok=True) if src.is_dir() else shutil.copy2(src, dst) + elif op == "done": + logs = root.parent / "verifier"; logs.mkdir(exist_ok=True) + how = os.environ.get("FAKE_VERIFY", "") + if how: + reply.update(rewards=None, exception={"type": how, "message": "it failed"}, verifier_dir=str(logs)) + else: + given = local("/app/reward-to-give") + reward = float(given.read_text()) if given.is_file() else 0.0 + (logs / "test-stdout.txt").write_text("1 failed" if reward < 1 else "all passed") + reply.update(rewards={"reward": reward}, exception=None, verifier_dir=str(logs)) + elif op == "stop": + reply.update(result=str(root.parent / "result.json")) + out.write(json.dumps(reply) + "\\n"); out.flush() + break + out.write(json.dumps(reply) + "\\n"); out.flush() +''') + + +@pytest.fixture +def harbor(tmp_path, monkeypatch): + """HarborEnvironments over the stand-in driver: (environments, the container's folder, the gate's and cleanup's + calls).""" + driver = tmp_path / "fake_driver.py" + driver.write_text(FAKE_DRIVER) + monkeypatch.setattr(harbor_mod, "DRIVER", driver) + root = tmp_path / "container" + monkeypatch.setenv("FAKE_ROOT", str(root)) + calls = {"gate": [], "cleanup": [], "down": []} + monkeypatch.setattr(harbor_mod.docker, "wait_for_disk", lambda where, gate, wait, **kw: + calls["gate"].append((str(where), gate, kw.get("image"))) or True) + monkeypatch.setattr(harbor_mod.docker, "data_root", lambda: Path("/var/lib/docker")) + monkeypatch.setattr(harbor_mod.docker, "cleanup_images", lambda mode, task_dir, lock_dir, threshold_gb=1.0: + calls["cleanup"].append((mode, task_dir.name))) + monkeypatch.setattr(harbor_mod.docker, "compose_down", lambda name, **kw: calls["down"].append(name)) + tasks = tmp_path / "tasks" / "demo" + tasks.mkdir(parents=True) + (tasks / "task.toml").write_text('[environment]\ndocker_image = "demo:1"\n') + environments = HarborEnvironments({"tasks_dir": str(tmp_path / "tasks"), "harbor": sys.executable, + "jobs_dir": str(tmp_path / "jobs"), "docker_cleanup": "rmi_task_image", + "disk_gate_gb": 7}, base=tmp_path) + return environments, root, calls + + +def test_an_environment_starts_behind_the_disk_gate_serves_the_agent_verifies_and_stops(harbor, tmp_path): + environments, root, calls = harbor + with environments.session(environments.task_dir({"id": "demo#1", "harbor_task": "demo"}), "demo#1") as env: + assert env.workspace == PurePosixPath("/app") and env.limits.timeout_s == 840.0 # the task's, less the margin + assert env.exec(["sh", "-c", "echo $WHO > who.txt; cat who.txt"], env={"WHO": "agent"}).output == "agent\n" + (tmp_path / "local.txt").write_text("1\n") + env.upload(tmp_path / "local.txt", "reward-to-give") # relative: in the workspace + assert (root / "app" / "reward-to-give").read_text() == "1\n" + env.download("who.txt", tmp_path / "back" / "who.txt") + assert (tmp_path / "back" / "who.txt").read_text() == "agent\n" + with pytest.raises(ValueError, match="outside the workspace"): + env.path("../etc/passwd") + assert env.path("/logs/agent") == PurePosixPath("/logs/agent") # absolute: anywhere there + verdict = env.verify() + assert (verdict["reward"], verdict["status"], verdict["passed"]) == (1.0, "ok", True) + assert verdict["feedback"] == "all passed" + with pytest.raises(Unsupported, match="verified"): # the agent's phase is over + env.exec(["true"]) + # the warm start: the host's /tmp, then Docker's data root with the task's image; the image removed after + assert calls["gate"] == [("/tmp", 1.0, None), ("/var/lib/docker", 7.0, "demo:1")] + assert calls["cleanup"] == [("rmi_task_image", "demo")] and calls["down"] == [] + sent = [json.loads(line) for line in (tmp_path / "requests.jsonl").read_text().splitlines()] + start = sent[0] + assert start["op"] == "start" and start["environment"] == {"delete": True} + assert start["trial_name"].startswith("demo_1__") and Path(start["task_dir"]).name == "demo" + assert sent[-1]["op"] == "stop" and not environments.live + + +def test_an_environment_that_does_not_start_is_unavailable_and_taken_down(harbor, monkeypatch): + environments, _, calls = harbor + monkeypatch.setenv("FAKE_START", "fail") + with pytest.raises(EnvironmentUnavailable, match="did not start.*did not pull") as raised: + with environments.session(environments.task_dir({"harbor_task": "demo"})): + pass + assert raised.value.cause == "sandbox_failed" # infra: run again + assert calls["cleanup"] == [("rmi_task_image", "demo")] and not environments.live + + +def test_a_disk_gate_that_waits_its_time_out_starts_nothing(harbor, monkeypatch): + environments, _, calls = harbor + monkeypatch.setattr(harbor_mod.docker, "wait_for_disk", lambda *a, **k: False) + monkeypatch.setattr(harbor_mod.docker, "free_gb", lambda root: 0.5) + with pytest.raises(EnvironmentUnavailable, match="/tmp stayed under 1 GB free: no environment started") as raised: + environments.open(environments.task_dir({"harbor_task": "demo"})).start() + assert raised.value.cause == "disk" # one that runs no better again + + +@pytest.mark.parametrize("raised, status, cause", [ + ("VerifierTimeoutError", "timeout", "solution_timed_out"), # Harbor's grading: the agent's solution, 0 + ("RewardFileNotFoundError", "agent_error", "verifier_failed"), # no reward: 0, as Harbor grades it + ("EnvironmentStartTimeoutError", "infra", "sandbox_failed"), # not the agent's result +]) +def test_a_verifier_that_fails_is_read_as_harbor_grades_it(harbor, monkeypatch, raised, status, cause): + environments, _, _ = harbor + monkeypatch.setenv("FAKE_VERIFY", raised) + with environments.session(environments.task_dir({"harbor_task": "demo"})) as env: + verdict = env.verify() + assert (verdict["reward"], verdict["status"], verdict["failure_cause"]) == (None, status, cause) + + +def test_the_settings_merge_over_the_benchmark_file_and_refuse_what_they_do_not_know(tmp_path): + (tmp_path / "tasks").mkdir() + (tmp_path / "benchmark.yaml").write_text("dataset: {ref: x@1}\ntasks_dir: tasks\nenvironment: {type: docker, " + "env: {A: '1'}}\ndocker_cleanup: images_keep\n") + environments = HarborEnvironments({"benchmark": "benchmark.yaml", "harbor": sys.executable, + "environment": {"force_build": True}}, base=tmp_path) + assert environments.environment_config() == {"delete": True, "type": "docker", "env": {"A": "1"}, + "force_build": True} + for bad, match in (({"web_search": False}, "unknown keys"), ({"docker_cleanup": "prune"}, "docker_cleanup"), + ({"verifier_timeout": "never"}, "verifier_timeout"), + ({"verifier_timeout_multiplier": 0.5}, "at least 1")): + with pytest.raises(ValueError, match=match): + HarborEnvironments({"benchmark": "benchmark.yaml", "harbor": sys.executable, **bad}, base=tmp_path) + + +class Recorded(Environment): + """A remote environment of the test's: its exec records each command and answers with `answer(command)`.""" + isolation = "remote" + + def __init__(self, tmp: Path, answer=None): + super().__init__(PurePosixPath("/"), Limits(timeout_s=600)) + self.workspace = PurePosixPath("/app") + self.tmp, self.answer, self.calls, self.uploads = tmp, answer or (lambda c: ExecResult(0, "")), [], {} + + def path(self, relative="", *, follow=True): + rel = PurePosixPath(str(relative)) + return rel if rel.is_absolute() else self.workspace / rel + + def exec(self, command, *, cwd=None, env=None, timeout=None, stdin=None, by_fd=None): + self.calls.append({"command": list(command), "cwd": cwd, "env": dict(env or {}), "timeout": timeout, + "stdin": stdin, "by_fd": dict(by_fd or {})}) + if list(command[:2]) == ["mv", "-f"] and command[2] in self.uploads: # an upload put in place + self.uploads[command[3]] = self.uploads.pop(command[2]) + return self.answer(list(command)) + + def upload(self, source, target): + source = Path(source) + self.uploads[str(self.path(target))] = {p.name: p.read_text() if p.is_file() else None + for p in sorted(source.iterdir())} if source.is_dir() else \ + source.read_text() + return self.path(target) + + def download(self, source, target): + raise FileNotFoundError(source) + + def install(self, cli, version=None): + self.calls.append({"install": cli, "version": version}) + return version + + +def session_of(command) -> bool: + """Whether a command an environment ran is a CLI's session (the CLI after the remote prologue), not a probe.""" + return command[:2] == ["bash", "-c"] and len(command) > 4 and command[3] == "bash" + + +def probe_of(command) -> bool: + return command[:2] == ["bash", "-c"] and len(command) == 3 and "command -v" in command[2] + + +ROTATED = json.dumps({"tokens": {"access_token": "a" * 20, "refresh_token": "r" * 20, "account_id": "acc"}, + "last_refresh": "2026-01-01T00:00:00Z"}) # a ChatGPT sign-in Codex refreshed +CODEX_STREAM = "\n".join(json.dumps(e) for e in [ + {"type": "thread.started", "thread_id": "t-1"}, {"type": "turn.started"}, + {"type": "item.completed", "item": {"id": "i", "type": "agent_message", "text": "fixed it"}}, + {"type": "turn.completed", "usage": {"input_tokens": 10, "cached_input_tokens": 0, "output_tokens": 5}}]) + "\n" + + +def test_codex_runs_where_a_remote_environment_runs_its_commands(tmp_path, monkeypatch): + from hermit.agent.base.interface import make + monkeypatch.setenv("FAKE_KEY", "the-key") + monkeypatch.setenv("HERMIT_HOMES", str(tmp_path / "homes")) + binding = {"ref": "fake/m", "backend": "fake", "model": "m", "base_url": "https://models.example/v1", + "api_key_env": "FAKE_KEY", "key_required": True, "apis": ["responses"]} + env = Recorded(tmp_path, lambda c: ExecResult(0, CODEX_STREAM) if session_of(c) else + ExecResult(1, "") if probe_of(c) else ExecResult(0, "")) + agent = make("codex", {"web_search": False, "binding": binding, "model": "m", + "log": str(tmp_path / "log" / "agent-codex.jsonl")}) + result = agent.run("fix the repository", environment=env) + assert result.reply == "fixed it" and result.extra["served"] and result.extra["isolation"] == "remote" + assert (tmp_path / "log" / "agent-codex.jsonl").read_text() == CODEX_STREAM # the stream is its output + install = next(c for c in env.calls if "install" in c) + assert install == {"install": "codex", "version": None} # not there: Harbor installs it + session = next(c for c in env.calls if session_of(c.get("command", []))) + prologue, command = session["command"][2], session["command"][4:] + assert 'nvm.sh' in prologue and '$HOME/.local/bin' in prologue # the installs' places on PATH + assert probe_of(next(c["command"] for c in env.calls if probe_of(c.get("command", [])))) + assert command[-1] == "-" and session["stdin"] == "fix the repository" # the prompt on stdin + assert command[command.index("-C") + 1] == "/app" and command[command.index("-s") + 1] == "danger-full-access" + assert command[command.index("-o") + 1] == "/installed-agent/hermit-home/last_message.txt" + assert session["cwd"] == "/app" and session["timeout"] == 600 + assert session["env"]["FAKE_KEY"] == "the-key" # by name, never in a file + assert session["env"]["CODEX_HOME"] == "/installed-agent/hermit-home" + assert "config.toml" in env.uploads["/installed-agent/hermit-home"] + assert "the-key" not in json.dumps(env.uploads) + + +def test_a_remote_session_refuses_what_the_environment_cannot_keep(tmp_path): + from hermit.agent.base.interface import make + from hermit.agent.base.tool import Tool + env = Recorded(tmp_path) + agent = make("codex", {"web_search": False}) + with pytest.raises(Unsupported, match="tool server"): + agent.run("x", environment=env, tools=[Tool("t", "d", lambda: 1)]) + with pytest.raises(Unsupported, match="takes isolation"): + make("pi", {"web_search": False}).run("x", environment=env) + + +def test_claude_codes_token_reaches_a_remote_session_on_a_descriptor(tmp_path, monkeypatch): + from hermit.agent.base import login + from hermit.agent.base.interface import make + monkeypatch.setenv(login.TOKEN_ENV, "the-token") + monkeypatch.setenv("HERMIT_HOMES", str(tmp_path / "homes")) + stream = json.dumps({"type": "result", "subtype": "success", "is_error": False, "result": "done", + "usage": {"input_tokens": 1, "output_tokens": 1}}) + "\n" + env = Recorded(tmp_path, lambda c: ExecResult(0, stream) if session_of(c) else ExecResult(0, "/usr/bin/claude")) + agent = make("claude", {"web_search": False, "login": "claude", "model": "claude-haiku-5-5", + "plan_dir": str(tmp_path / "plan"), "bin": "claude"}) + result = agent.run("hi", environment=env) + assert result.reply == "done" + session = next(c for c in env.calls if session_of(c["command"])) + assert session["by_fd"] == {login.TOKEN_ENV: "CLAUDE_CODE_OAUTH_TOKEN_FILE_DESCRIPTOR"} # the environment's move + assert session["command"][4] == "claude" + assert session["env"][login.TOKEN_ENV] == "the-token" and session["env"]["IS_SANDBOX"] == "1" + assert session["stdin"] == "hi" + + +def test_the_cli_agent_uploads_its_staging_into_a_remote_workspace_and_refuses_mounts(tmp_path, monkeypatch): + from hermit.agent.acting.workspace import CliAgent + monkeypatch.setenv("HERMIT_HOMES", str(tmp_path / "homes")) + env = Recorded(tmp_path, lambda c: ExecResult(0, CODEX_STREAM) if session_of(c) else ExecResult(0, "")) + (tmp_path / "fixtures").mkdir() + (tmp_path / "fixtures" / "a.txt").write_text("a") + agent = CliAgent({"backend": "codex", "web_search": False, "bin": "/opt/codex", "files": ["fixtures"], + "workspaces": str(tmp_path / "ws")}, tmp_path) + out = agent.run({"id": "t#1", "instruction": "do it"}, None, environment=env) + assert out.output == "fixed it" and out.extra["isolation"] == "remote" and out.extra["workspace"] == "/app" + assert env.uploads["/app/fixtures"] == {"a.txt": "a"} + session = next(c for c in env.calls if session_of(c["command"])) + assert session["command"][4] == "/opt/codex" and session["stdin"] == "do it" + assert session["timeout"] == 600 # the environment's budget is the cap + with pytest.raises(Unsupported, match="mounts"): + CliAgent({"backend": "codex", "web_search": False, "mounts": [{"source": ".", "at": "d"}]}, + tmp_path).run({"id": "t"}, None, environment=env) + + +def test_the_answer_agent_answers_once_and_writes_the_answer_where_the_verifier_reads_it(tmp_path, monkeypatch): + from hermit.agent.acting import answer as answer_mod + from hermit.agent.model.usage import Usage + asked = [] + + class Model: + model = "m" + + def chat(self, messages, **kwargs): + asked.append(messages) + return {"role": "assistant", "content": "Thinking... B"}, Usage(model="m", input_tokens=3, + output_tokens=2, calls=1, + cost_usd=0.001) + + monkeypatch.setattr(answer_mod.ChatClient, "from_spec", staticmethod(lambda spec: Model())) + env = Recorded(tmp_path, lambda c: ExecResult(0, "Which panel? A or B\n") if c[0] == "cat" else ExecResult(0, "")) + agent = answer_mod.AnswerAgent({"model": {"base_url": "https://x", "model": "m"}, "instruction_file": + "/app/instruction.md", "answer_file": "/app/answer.txt", + "answer_pattern": "(.*?)", "logs": str(tmp_path / "logs")}, + tmp_path) + out = agent.run({"id": "q1", "instruction": "unused"}, None, environment=env) + assert out.output == {"answer": "B", "reply": "Thinking... B"} and out.cost_usd == 0.001 + assert asked == [[{"role": "user", "content": "Which panel? A or B"}]] # no adapter: no system message + assert env.uploads["/app/answer.txt"] == "B" + + +class Environments: + """The run's task environments, the test's: each task's environment records what happened to it. `fails` names + what each start (in turn) does: None starts, a cause raises EnvironmentUnavailable with it (the last one repeats); + `verdicts` the verifier's verdict of each run in turn (the last one repeats).""" + infra_wait_s = 7.0 + + def __init__(self, tmp: Path, fails=(None,), verdicts=None, infra_retries: int = 2): + self.tmp, self.fails, self.events, self.infra_retries = tmp, list(fails), [], infra_retries + self.verdicts = list(verdicts or [{"reward": 1.0, "rewards": {"reward": 1.0}, "status": "ok", + "passed": True, "feedback": ""}]) + self.starts = 0 + + def task_dir(self, task): + return self.tmp / str(task.public["harbor_task"]) + + def session(self, task_dir, name=None, installs=None): + import contextlib + outer = self + + class Env(Recorded): + def verify(self, agent_logs=None): + outer.events.append(("verify", agent_logs)) + done = sum(event[0] == "verify" for event in outer.events) + return dict(outer.verdicts[min(done, len(outer.verdicts)) - 1]) + + @contextlib.contextmanager + def cm(): + fail = outer.fails[min(outer.starts, len(outer.fails) - 1)] + outer.starts += 1 + if fail is not None: + raise EnvironmentUnavailable(f"the image did not pull ({fail})", fail) + outer.events.append(("start", Path(task_dir).name)) + try: + yield Env(outer.tmp) + finally: + outer.events.append(("stop", Path(task_dir).name)) + return cm() + + +def test_the_run_hands_the_agent_the_tasks_environment_and_judges_its_verifiers_verdict(tmp_path): + from hermit.task import Task + from hermit.task.verify import environment as judge + from hermit.trainer.rollout import run_one + environments, given = Environments(tmp_path), [] + logs = tmp_path / "log" + logs.mkdir() + + def pipeline(task, adapter, environment=None): + given.append((task, environment.workspace)) + return RunOutput(output="done", log=logs, extra={"session_failed": "the CLI failed"}) # verified all the same + + task = Task(id="demo#1", public={"id": "demo#1", "harbor_task": "demo"}, private={"task_dir": "/secret/demo"}) + record = run_one(task, None, pipeline, judge, environments) + assert given == [({"id": "demo#1", "harbor_task": "demo"}, PurePosixPath("/app"))] # the public half only + assert environments.events == [("start", "demo"), ("verify", logs), ("stop", "demo")] + assert record["status"] == "ok" and record["judge"]["score"] == 1.0 and record["judge"]["feedback"] == "passed" + assert "attempts" not in record["pipeline_extra"] # one run: no attempts recorded + + +def test_the_environment_judge_scores_the_reward_and_refuses_an_environment_that_failed(): + from hermit.task.verify import Unscored, environment as judge + out = lambda verdict, **extra: RunOutput(output=None, extra={"verified": verdict, **extra}) + timed = judge({}, out({"reward": 0.0, "status": "timeout", "failure_cause": "solution_timed_out", + "exception": {"type": "VerifierTimeoutError"}, "passed": False, "feedback": "slow"})) + assert timed.score == 0.0 and "solution_timed_out" in timed.feedback and "slow" in timed.feedback + late = judge({}, out({"reward": 1.0, "status": "ok", "passed": True}, timed_out=True)) + assert late.score == 1.0 and "ran out of its time" in late.feedback # its work still scored + with pytest.raises(Unscored): + judge({}, out({"reward": None, "status": "infra", "failure_cause": "verifier_failed"})) + with pytest.raises(RuntimeError, match="environment.harbor"): + judge({}, RunOutput(output="x")) + + +HARBOR_VENV = Path(os.path.expanduser("~/.cache/hermit/harbor-venv/bin/harbor")) +AGENT_KEY = "sk-agent-synthetic-0123456789abcdef" # an agent's model credential, the test's own + + +def _docker_up() -> bool: + try: + return shutil.which("docker") is not None and \ + subprocess.run(["docker", "info"], capture_output=True, timeout=30).returncode == 0 + except (OSError, subprocess.SubprocessError): + return False + + +@pytest.mark.skipif(not HARBOR_VENV.exists() or not _docker_up(), reason="needs Docker and Harbor's venv " + "(~/.cache/hermit/harbor-venv)") +def test_a_real_harbor_environment_runs_the_agents_commands_and_harbors_verifier(tmp_path): + """A task of the test's own (an image Docker already has, no pull): what the agent writes in the environment is what + Harbor's verifier scores, 1 for the right answer and 0 for none, the tests uploaded only for the verifier. A command + of the agent's still running when the verifier starts is ended first; what an ended command left running (a server + the task asked for) stays.""" + import threading + image = next((i for i in ("python:3.12-slim", "python:3.11-slim", "python:3.10-slim") + if subprocess.run(["docker", "image", "inspect", i], capture_output=True).returncode == 0), None) + if image is None: + pytest.skip("no python slim image on this machine (the test pulls none)") + task = tmp_path / "tasks" / "answer-42" + (task / "environment").mkdir(parents=True) + (task / "tests").mkdir() + (task / "task.toml").write_text(f'[agent]\ntimeout_sec = 120\n[verifier]\ntimeout_sec = 120\n' + f'[environment]\ndocker_image = "{image}"\n') + (task / "instruction.md").write_text("Write 42 to /app/answer.txt.\n") + (task / "tests" / "test.sh").write_text( + '#!/bin/bash\nrunning() { for p in /proc/[0-9]*; do tr "\\0" " " < $p/cmdline 2>/dev/null; echo; done | ' + 'grep -q "^sleep $1"; }\n' + 'echo "grader key: $GRADER_API_KEY"\n' + 'test -e /tmp/hermit-sign-in && echo "secret-present"\n' + 'running 777 && echo "777-alive"; stat -c "home-owner:%u" /installed-agent/hermit-home 2>/dev/null\n' + 'echo "stream: $(cat /logs/agent/agent-codex.jsonl 2>&1)"\n' + 'if [ "$(cat /app/answer.txt 2>/dev/null)" = 42 ] && ! running 777 && ! running 666 && running 888 && ' + '! grep -rqs sk-agent-synthetic /logs/agent; then ' + 'echo 1 > /logs/verifier/reward.txt; else echo 0 > /logs/verifier/reward.txt; fi\n') + environments = HarborEnvironments({"tasks_dir": str(tmp_path / "tasks"), "jobs_dir": str(tmp_path / "jobs"), + "disk_gate_gb": 0, "tmp_gate_gb": 0, + "verifier": {"env": {"GRADER_API_KEY": "grader-literal-credential"}}}, + base=tmp_path) + rewards = [] + for script in ("mkdir -p /app && echo 42 > /app/answer.txt && (nohup sleep 888 >/dev/null 2>&1 &)", "true"): + with environments.session(task) as env: + assert env.instruction == "Write 42 to /app/answer.txt.\n" # Harbor's, byte for byte + assert env.exec(["sh", "-c", "test -e /tests/test.sh"]).code != 0 # no tests while the agent works + assert env.exec(["sh", "-c", script], timeout=60).code == 0 + # the agent's variables reach the container only: a DOCKER_HOST of its own leaves this machine's compose + # client as it was (the next command still runs), and a secret on a descriptor is in no environment there + said = env.exec(["sh", "-c", 'echo "host=$DOCKER_HOST"; cat <&"$SECRET_FD"; echo; ' + 'grep -l s3cr3t-value /proc/[0-9]*/environ 2>/dev/null | wc -l'], + env={"DOCKER_HOST": "tcp://127.0.0.1:9", "SECRET": "s3cr3t-value"}, + by_fd={"SECRET": "SECRET_FD"}, timeout=60) + assert said.output.split() == ["host=tcp://127.0.0.1:9", "s3cr3t-value", "0"], said.output + assert env.exec(["true"], env={"FAKE_AGENT_API_KEY": AGENT_KEY}, timeout=60).code == 0 + if script != "true": + # an image whose AGENTS.md is a link to its CLAUDE.md: the adapter's text goes first in a file of its + # own there, the image's text (read through the link there) after it, CLAUDE.md untouched + from hermit.agent.acting import workspace + assert env.exec(["sh", "-c", "echo IMAGE-CLAUDE > CLAUDE.md && ln -sf CLAUDE.md AGENTS.md"]).code == 0 + staging = tmp_path / "staging" + staging.mkdir(exist_ok=True) + (staging / "AGENTS.md").write_text("ADAPTER") + workspace._upload_staging(env, staging, "codex", {"adapter_id": "a"}, {}, {}) + said = env.exec(["sh", "-c", "test -L AGENTS.md && echo link; cat AGENTS.md; echo ---; cat CLAUDE.md"]) + assert said.output == "ADAPTER\n\nIMAGE-CLAUDE\n---\nIMAGE-CLAUDE\n", said.output + ended = [] + + def still_running(): + try: # one marked, one below it with a cleared environment (no mark): both ended + env.exec(["bash", "-c", "env -i /bin/sleep 666 & sleep 777"], timeout=100) + except RuntimeError as exc: # ended by the verifier's start + ended.append(str(exc)) + + late = threading.Thread(target=still_running, daemon=True) + late.start() # still running at the verifier + import time + time.sleep(3) + logs = tmp_path / "agent-logs" # the session's log holding the agent's credential + logs.mkdir(exist_ok=True) + (logs / "agent-codex.jsonl").write_text(f"error: bad key {AGENT_KEY}\n") + verdict = env.verify(agent_logs=logs) + rewards.append(verdict["reward"]) + # Harbor's scrub of the verifier's credential: neither the feedback nor the trial's files hold it + assert "grader-literal-credential" not in verdict["feedback"] and "grader key" in verdict["feedback"] + late.join(timeout=30) + assert ended and "the agent's phase ended" in ended[0] + assert rewards == [1.0, 0.0] # the agent's credential never reached the verifier + kept = [p for p in (tmp_path / "jobs").rglob("*") if p.is_file() and not p.is_symlink()] + assert not [p for p in kept if b"grader-literal-credential" in p.read_bytes()] + assert not [p for p in kept if AGENT_KEY.encode() in p.read_bytes()] # nor stays in the trial's files + # a task whose agent user is not root: what Hermit provisions is that user's (its uid:gid, as the runner's + # _owner), its home among it + user_task = tmp_path / "tasks" / "as-nobody" + shutil.copytree(task, user_task) + (user_task / "task.toml").write_text(f'[agent]\ntimeout_sec = 120\nuser = "nobody"\n[verifier]\ntimeout_sec = 120\n' + f'[environment]\ndocker_image = "{image}"\n') + home = tmp_path / "home" + home.mkdir() + (home / "config.toml").write_text("x = 1\n") + with environments.session(user_task) as env: + assert env.exec(["id", "-u"]).output.strip() == "65534" + env.upload(home, "/installed-agent/hermit-home") # Hermit's own: given to the user + assert env.exec(["stat", "-c", "%u", "/installed-agent/hermit-home/config.toml"]).output.strip() == "65534" + def sleeps(): + try: + env.exec(["sleep", "777"], timeout=100) + except RuntimeError: # ended at the phase's end + pass + + late = threading.Thread(target=sleeps, daemon=True) + late.start() # a command of the non-root agent's, running at the phase's end + import time + time.sleep(3) + verdict = env.verify() + late.join(timeout=30) + # its process group ended at the phase's end (killed as root), its home its own + assert "777-alive" not in verdict["feedback"] and "home-owner:65534" in verdict["feedback"], verdict["feedback"] + with environments.session(user_task) as env: # a credential path the agent cannot remove: no verifier + env.secret(["/etc/hostname"]) + verdict = env.verify() + assert verdict["status"] == "infra" and (verdict["exception"] or {}).get("type") == "HarborDriverError" + # Harbor's own agent timeout ends the phase: the command still running is ended (AgentPhaseOver), and the trial + # runs on to its verifier, its verdict the agent's timeout + timed = HarborEnvironments({"tasks_dir": str(tmp_path / "tasks"), "jobs_dir": str(tmp_path / "jobs2"), + "disk_gate_gb": 0, "tmp_gate_gb": 0, "agent_timeout_sec": 15}, base=tmp_path) + with timed.session(task) as env: + assert env.limits.timeout_s == 13.5 # its own budget ends first + env.deadline_at = None # this side's own cap off: Harbor's timeout first, as on an abnormal path + env.secret(["/tmp/hermit-sign-in"]) # a sign-in, registered first + assert env.exec(["sh", "-c", "echo token > /tmp/hermit-sign-in"]).code == 0 + with pytest.raises(RuntimeError, match="AgentPhaseOver"): # a CLI still streaming at the end + env.exec(["sh", "-c", "echo '{\"type\": \"answer\"}'; sleep 100"], timeout=100, + stream="agent-codex.jsonl") + verdict = env.verify() + assert (verdict["status"], (verdict["exception"] or {}).get("type")) == ("timeout", "AgentTimeoutError") + # before the verifier: the sign-in removed, the stream so far kept in the agent logs under its name + assert "secret-present" not in verdict["feedback"] and 'stream: {"type": "answer"}' in verdict["feedback"] + + +def test_a_react_agent_works_in_the_tasks_environment_its_tools_run_there(tmp_path, monkeypatch): + """A ReactAgent subclass's tools use the task's environment (`self.environment`, the run's by default); the loop, + which cannot stop a model call at the environment's budget, is given it without that budget.""" + from hermit.agent.acting.react import ReactAgent, tool + from hermit.agent.model import llm + from hermit.agent.model.usage import Usage + replies = iter([{"role": "assistant", "content": "", "tool_calls": [ + {"id": "c1", "type": "function", "function": {"name": "bash", "arguments": json.dumps({"command": "ls"})}}]}, + {"role": "assistant", "content": "listed"}]) + + class Model: + model = "m" + usage = [] + + def chat(self, messages, **kwargs): + return next(replies), Usage(model="m", input_tokens=1, output_tokens=1, calls=1, cost_usd=0.0) + + monkeypatch.setattr(llm.ChatClient, "from_spec", staticmethod(lambda spec: Model())) + + class Terminal(ReactAgent): + @tool + def bash(self, command: str) -> str: + """Run a shell command in the task's environment.""" + return self.environment.exec(["bash", "-c", command]).output + + env = Recorded(tmp_path, lambda c: ExecResult(0, "README.md\n")) + out = Terminal({"model": {"base_url": "https://x", "model": "m"}, "logs": str(tmp_path / "logs")}, + tmp_path).run({"id": "t", "instruction": "list the files"}, None, environment=env) + assert out.output == "listed" and env.calls[0]["command"] == ["bash", "-c", "ls"] + + +def _infra_run(tmp_path, environments, pauses, monkeypatch): + """A task run through run_one with the test's environments, its pauses recorded instead of slept.""" + from hermit.task import Task + from hermit.task.verify import environment as judge + from hermit.trainer import rollout + calls = [] + + def pipeline(task, adapter, environment=None): + calls.append(task["id"]) + return RunOutput(output="done", cost_usd=0.01, tokens=10, api_speed={"model_calls": 1}) + + monkeypatch.setattr(rollout, "_sleep", pauses.append) + task = Task(id="demo#1", public={"id": "demo#1", "harbor_task": "demo"}) + return rollout.run_one(task, None, pipeline, judge, environments), calls + + +def test_an_environment_that_fails_to_start_once_runs_again_and_is_scored(tmp_path, monkeypatch): + """The old runner's infra semantics: a rollout that is not the agent's result runs again in a fresh environment, + after infra_wait_s; the run that works is the one scored, and its record carries every run (attempts).""" + environments, pauses = Environments(tmp_path, fails=["sandbox_failed", None]), [] + record, calls = _infra_run(tmp_path, environments, pauses, monkeypatch) + assert pauses == [7.0] and calls == ["demo#1"] and environments.starts == 2 + assert record["status"] == "ok" and record["judge"]["score"] == 1.0 + attempts = record["pipeline_extra"]["attempts"] + assert [a["unscored"] for a in attempts] == ["sandbox_failed", None] + assert attempts[0]["cost_usd"] == 0.0 and record["cost_usd"] == 0.01 # the environment that never came up: 0 + + +def test_a_rollout_that_stays_infra_is_not_scored_as_the_agents_zero(tmp_path, monkeypatch): + """Retried infra_retries times; then the judge refuses to score it (Unscored): a failed run, judge_error, which + `hermit eval` counts among its errors and the report as infra, never a verdict of the agent's 0.""" + from hermit.harbor.report import kind + infra = {"reward": None, "rewards": None, "status": "infra", "failure_cause": "verifier_failed", "passed": None, + "exception_message": "the verifier's tools never arrived", "feedback": ""} + environments, pauses = Environments(tmp_path, verdicts=[infra]), [] + record, calls = _infra_run(tmp_path, environments, pauses, monkeypatch) + assert calls == ["demo#1"] * 3 and pauses == [7.0, 7.0] # 1 + infra_retries runs + assert record["status"] == "judge_error" and "Unscored" in record["error"] + assert "not the agent's result (verifier_failed" in record["judge"]["feedback"] + assert record["pipeline_extra"]["unscored"] == "verifier_failed" and len(record["pipeline_extra"]["attempts"]) == 3 + assert record["cost_usd"] == pytest.approx(0.03) and record["tokens"] == 30 # every run's usage is the rollout's + assert kind(record) == "infra" + never = Environments(tmp_path, fails=["harbor_failed"]) + record, calls = _infra_run(tmp_path, never, [], monkeypatch) + assert calls == [] and never.starts == 3 and record["status"] == "judge_error" + assert record["output"]["status"] == "infra" and kind(record) == "infra" and record["cost_usd"] == 0.0 + + +def test_a_cause_that_runs_no_better_again_is_not_run_again(tmp_path, monkeypatch): + for cause in ("disk", "model_rejected"): # NO_RETRY, the runner's + environments, pauses = Environments(tmp_path, fails=[cause, None]), [] + record, calls = _infra_run(tmp_path, environments, pauses, monkeypatch) + assert environments.starts == 1 and pauses == [] and calls == [] + assert record["status"] == "judge_error" and f"({cause}" in record["judge"]["feedback"] + + +def test_a_session_the_provider_could_not_serve_is_infra_and_any_other_pipeline_failure_the_pipelines(tmp_path): + from hermit.task import Task + from hermit.task.verify import environment as judge + from hermit.trainer import rollout + seen = [] + + def pipeline(task, adapter, environment=None): + seen.append(1) + raise RuntimeError("agent codex could not be served after 3 attempts: 503") + + task = Task(id="demo#1", public={"id": "demo#1", "harbor_task": "demo"}) + out = rollout.environment_rollout(task, task.given(), None, pipeline, Environments(tmp_path), 0.0, + pause=lambda s: None) + # the CLI agent's own runs again are the rollout's one provider allowance (the runner's busy_retries): spent + assert len(seen) == 1 and out.extra["unscored"] == "model_provider_failed" and out.cost_usd is None + for said, cause, runs in (("chat/completions HTTP 503: overloaded", "model_provider_failed", 3), + ("EnvironmentUnavailable: the Harbor driver exited (-9)", "harbor_failed", 3), + ("agent codex: model_rejected: unexpected status 404 Not Found", "model_rejected", 1), + ("chat/completions HTTP 401: invalid key", "model_rejected", 1)): + seen.clear() + + def failing(task, adapter, environment=None, said=said): + seen.append(1) + raise RuntimeError(said) + + out = rollout.environment_rollout(task, task.given(), None, failing, Environments(tmp_path), 0.0, + pause=lambda s: None) + assert (len(seen), out.extra["unscored"]) == (runs, cause), said + def phase_over(task, adapter, environment=None): # Harbor's agent timeout ended the phase first + raise RuntimeError("AgentPhaseOver: exec: the agent's phase ended") + + environments = Environments(tmp_path, verdicts=[{"reward": 0.0, "rewards": {"reward": 0.0}, "status": "timeout", + "failure_cause": "agent_exhausted_budget", "passed": False, + "feedback": ""}]) + out = rollout.environment_rollout(task, task.given(), None, phase_over, environments, 0.0, pause=lambda s: None) + assert out.extra["verified"]["status"] == "timeout" and out.extra["timed_out"] is True # the trial's verdict + + def broken(task, adapter, environment=None): + raise KeyError("a bug of the agent's own") + + record = rollout.run_one(task, None, broken, judge, Environments(tmp_path)) + assert record["status"] == "error" and "the pipeline failed" in record["judge"]["feedback"] + + +@pytest.mark.parametrize("verified, status, cause", [ + ({"rewards": {"reward": 0.0}, "exception": None, "stdout": "failed to download https://github.com/astral-sh/uv/" + "releases/download/x\nuvx: command not found"}, "infra", "verifier_failed"), # its tools never arrived + ({"rewards": {"reward": 0.0}, "exception": None, "stdout": "1 failed"}, "ok", None), # the agent's 0 + ({"rewards": None, "exception": None}, "infra", "verifier_failed"), # no exception and no reward + ({"rewards": None, "exception": {"type": "RuntimeError", "message": "Docker compose command failed for " + "environment x"}}, "infra", "sandbox_failed"), # the host's (HOST_FAILURE) + ({"rewards": None, "exception": {"type": "HarborDriverError", "message": "gone"}}, "infra", "harbor_failed"), +]) +def test_the_verifiers_answer_is_classified_as_the_runner_classified_a_trial(harbor, tmp_path, verified, status, + cause): + environments, _, _ = harbor + logs = tmp_path / "verifier" + logs.mkdir() + if "stdout" in verified: + (logs / "test-stdout.txt").write_text(verified.pop("stdout")) + verdict = environments.verdict({**verified, "verifier_dir": str(logs)}) + assert (verdict["status"], verdict["failure_cause"]) == (status, cause) + + +def test_infra_retries_and_infra_wait_s_are_read_and_checked(tmp_path): + (tmp_path / "tasks").mkdir() + base = {"tasks_dir": "tasks", "harbor": sys.executable} + environments = HarborEnvironments(base, base=tmp_path) + assert (environments.infra_retries, environments.infra_wait_s) == (2, 300.0) # the runner's defaults + assert HarborEnvironments({**base, "infra_retries": 0, "infra_wait_s": 1}, base=tmp_path).infra_retries == 0 + with pytest.raises(ValueError, match="infra_retries"): + HarborEnvironments({**base, "infra_retries": -1}, base=tmp_path) + + +# --- review round 1: the old runner's behavior, each restored --------------------------------------------------------- + +def test_a_remote_session_refuses_to_resume_and_turns_web_search_off_on_a_restricted_network(tmp_path, monkeypatch): + from hermit.agent.base.interface import make + monkeypatch.setenv("HERMIT_HOMES", str(tmp_path / "homes")) + env = Recorded(tmp_path, lambda c: ExecResult(0, CODEX_STREAM) if session_of(c) else ExecResult(0, "/x")) + with pytest.raises(Unsupported, match="resume"): + make("codex", {"web_search": True, "bin": "codex"}).run("x", environment=env, session="t-1") + env.network_public = False # the task's agent phase has no public network + make("codex", {"web_search": True, "bin": "codex"}).run("x", environment=env) + config = env.uploads["/installed-agent/hermit-home"]["config.toml"] + assert 'web_search = "disabled"' in config + + +def test_every_sign_in_file_leaves_the_environment_and_an_unconfirmed_removal_fails(tmp_path, monkeypatch): + from hermit.agent.base import login + from hermit.agent.base.interface import make + monkeypatch.setenv("HERMIT_HOMES", str(tmp_path / "homes")) + monkeypatch.setattr(login, "prepare", lambda root, home: (Path(home) / login.FILE).write_text("{}")) + monkeypatch.setattr(login, "keep_refreshed", lambda *a: None) + monkeypatch.setattr(login, "note_usage", lambda *a: None) + params = {"web_search": False, "bin": "codex", "login": "chatgpt", "model": "m", "plan_dir": str(tmp_path / "p")} + env = Recorded(tmp_path, lambda c: ExecResult(0, CODEX_STREAM) if session_of(c) else ExecResult(0, "")) + env.plan_admitted = True + make("codex", params).run("x", environment=env) + removal = next(c for c in env.calls if c["command"][:2] == ["rm", "-f"]) + assert removal["command"][2:] == ["/installed-agent/hermit-home/auth.json"] and removal["cwd"] == "/" + stuck = Recorded(tmp_path, lambda c: ExecResult(1, "busy") if c[:2] == ["rm", "-f"] else + ExecResult(0, CODEX_STREAM) if session_of(c) else ExecResult(0, "")) + stuck.plan_admitted = True + with pytest.raises(RuntimeError, match="sign-in could not be removed"): + make("codex", params).run("x", environment=stuck) + + +def test_codexs_lost_sign_in_and_refused_configuration_are_read_from_its_notices(tmp_path, monkeypatch): + from hermit.agent.acting.workspace import CliAgent + from hermit.agent.base.interface import LoginFailed + monkeypatch.setenv("HERMIT_HOMES", str(tmp_path / "homes")) + lost = json.dumps({"type": "turn.failed", "error": {"message": "unexpected status 401 Unauthorized: bad"}}) + "\n" + model = json.dumps({"type": "turn.failed", "error": {"message": "unexpected status 404 Not Found: model " + "gpt-x does not exist"}}) + "\n" + params = {"backend": "codex", "web_search": False, "bin": "codex", "workspaces": str(tmp_path / "ws"), + "retries": 1} + agent = CliAgent(params, tmp_path) + answer = lambda stream: (lambda c: ExecResult(1, stream) if session_of(c) else ExecResult(0, "")) + with pytest.raises(LoginFailed, match="401"): # a lost sign-in ends the run + agent.run({"id": "t", "instruction": "x"}, None, environment=Recorded(tmp_path, answer(lost))) + out = agent.run({"id": "t", "instruction": "x"}, None, environment=Recorded(tmp_path, answer(model))) + assert out.extra["infra"] == "model_rejected" and out.extra["infra_final"] is True # never run again + billed = json.dumps({"type": "turn.failed", "error": {"message": "unexpected status 402 Payment Required: " + "insufficient credits"}}) + "\n" + with pytest.raises(LoginFailed, match="usage_limit"): # nothing later can be served + agent.run({"id": "t", "instruction": "x"}, None, environment=Recorded(tmp_path, answer(billed))) + long = json.dumps({"type": "turn.failed", "error": {"message": "unexpected status 400 Bad Request: the prompt " + "is too long"}}) + "\n" + out = agent.run({"id": "t", "instruction": "x"}, None, environment=Recorded(tmp_path, answer(long))) + assert "infra" not in out.extra # the input's: the agent's result + + +def test_a_refusal_is_the_agents_unless_the_configuration_or_the_bill_refused_it(): + """The runner's session_verdict for every CLI: an oversized prompt and the model's own refusal are the agent's + result; a configuration refused before any token, model_rejected; a bill or a spent quota, usage_limit (a halt).""" + from hermit.agent.base.codex_stream import session_verdict + base = {"served": False, "rejected": True} + assert session_verdict({**base, "failure": "API Error: 400 prompt is too long"}, 0) is None + assert session_verdict({**base, "failure": "API Error: 404 model not found", "model_refusal": True}, 0) is None + assert session_verdict({**base, "failure": "API Error: 404 model not found"}, 0) == "model_rejected" + assert session_verdict({**base, "failure": "API Error: 404 model not found"}, 12) is None # it had served + assert session_verdict({**base, "failure": "API Error: 402 credit balance is too low"}, 0) == "usage_limit" + assert session_verdict({"served": False, "unavailable": True, "failure": "429 insufficient_quota"}, 0) == \ + "usage_limit" + + +def test_the_session_takes_harbors_instruction_and_the_tasks_budget_and_counts_every_attempt(tmp_path, monkeypatch): + """Harbor's instruction byte for byte (what the runner gave the CLI), the task's own agent budget (8 hours for + Terminal-Bench-Science) unless the agent's timeout_s says less, and the usage of a run again counted too.""" + from hermit.agent.acting.workspace import CliAgent + from hermit.agent.base import cli + from hermit.agent.base.interface import Result + from hermit.agent.model.usage import Usage + monkeypatch.setenv("HERMIT_HOMES", str(tmp_path / "homes")) + monkeypatch.setattr("time.sleep", lambda s: None) + seen = [] + + def run(self, prompt, *, environment=None, tools=(), session=None, roles=()): + seen.append((prompt, environment.limits.timeout_s)) + served = len(seen) > 1 # the first: the provider could not serve it, after 100 tokens + return Result(reply="done" if served else "", usage=[Usage(model="m", input_tokens=100 if not served else 10, + output_tokens=5, calls=1, cost_usd=0.5)], + exit_code=0, extra={"served": served, "unavailable": not served, "started": True, + "failed": False, "output_tail": ""}) + + monkeypatch.setattr(cli.Codex, "run", run) + env = Recorded(tmp_path) + env.limits = Limits(timeout_s=28800.0) + env.instruction = "Fix the repo.\n\n keep this \n" + out = CliAgent({"backend": "codex", "web_search": False, "bin": "codex", "workspaces": str(tmp_path / "ws")}, + tmp_path).run({"id": "t", "question": "Fix?", "instruction": "Fix the repo."}, None, + environment=env) + assert seen == [("Fix the repo.\n\n keep this \n", 28800.0)] + # the provider could not serve it: never run again in this environment, the run runs it in a fresh one, its + # spend kept + assert out.extra["infra"] == "model_provider_failed" and out.extra["infra_final"] is False + assert out.tokens == 105 and out.cost_usd == 0.5 + capped = CliAgent({"backend": "codex", "web_search": False, "bin": "codex", "timeout_s": 600, + "workspaces": str(tmp_path / "ws")}, tmp_path) + seen.clear() + capped.run({"id": "t"}, None, environment=env) + assert seen[0][1] == 600.0 # the agent's own cap, when less + + +def test_hooks_get_python3_their_audit_comes_back_and_the_image_context_is_checked(tmp_path, monkeypatch): + from hermit.agent.acting import workspace + log = tmp_path / "log" + log.mkdir() + calls = [] + + class Env(Recorded): + def ensure(self, packages): + calls.append(("ensure", list(packages))) + + def download(self, source, target): + if source == ".hermit-hooks": + (Path(target) / "hook-state").mkdir(parents=True) + (Path(target) / "hook-calls.jsonl").write_text("{}\n") + return Path(target) + raise OSError("docker cp failed") + + env = Env(tmp_path, lambda c: ExecResult(1, "") if c[:2] == ["sh", "-c"] else ExecResult(0, "")) + assert workspace._ensure_python3(env) is False and calls == [] # nothing installs during the agent phase + workspace._collect_hooks(env, log) + assert (log / "hook-calls.jsonl").is_file() and (log / "hook-state").is_dir() + work = tmp_path / "work" + work.mkdir() + (work / "AGENTS.md").write_text("adapter") + with pytest.raises(RuntimeError, match="could not be read"): # there, but unread: never overwritten + workspace._upload_staging(env, work, "codex", {"adapter_id": "a"}, {}) + from hermit.environment.harbor_errors import pipeline_cause + assert pipeline_cause(f"RuntimeError: {workspace.IMAGE_CONTEXT_UNREAD} (AGENTS.md)") == ("harbor_failed", True) + + big = Recorded(tmp_path, lambda c: ExecResult(0, workspace.IMAGE_MARK + "x" * 40000) if c[:2] == ["sh", "-c"] + else ExecResult(0, "")) + with pytest.raises(workspace.AdapterContextTooLong, match="over Codex's limit"): # the merge within the cap + workspace._upload_staging(big, work, "codex", {"adapter_id": "a"}, {}) + + +def test_the_answer_agent_writes_to_harbors_default_file_unless_told_none(tmp_path, monkeypatch): + from hermit.agent.acting import answer as answer_mod + from hermit.agent.model.usage import Usage + + class Model: + model = "m" + + def chat(self, messages, **kwargs): + return {"role": "assistant", "content": "C"}, Usage(model="m", calls=1) + + monkeypatch.setattr(answer_mod.ChatClient, "from_spec", staticmethod(lambda spec: Model())) + for given, written in (({}, {"/app/answer.txt": "C"}), ({"answer_file": None}, {})): + env = Recorded(tmp_path) + env.instruction = "Which? A, B or C" + answer_mod.AnswerAgent({"model": {"base_url": "https://x", "model": "m"}, "logs": str(tmp_path / "logs"), + **given}, tmp_path).run({"id": "q"}, None, environment=env) + assert env.uploads == written + + +def test_the_rollouts_status_is_the_runners_and_a_login_plan_is_waited_for_before_the_environment(tmp_path): + """`status` beside the verdict as the runner wrote it (AHE classifies by it): a CLI that exited with an error is + agent_error, a session that ran out of its time a timeout, whatever the verifier then scored.""" + from hermit.task import Task + from hermit.trainer import rollout + task = Task(id="demo#1", public={"id": "demo#1", "harbor_task": "demo"}) + import contextlib + environments = Environments(tmp_path) + + @contextlib.contextmanager + def admission(): + environments.events.append(("plan",)) + yield + environments.events.append(("plan done",)) + + environments.admission = admission + admitted = [] + for extra, status in (({"exit_code": 1}, "agent_error"), ({"timed_out": True}, "timeout"), ({}, "ok")): + def pipeline(t, a, environment=None, extra=extra): + admitted.append(environment.plan_admitted) + return RunOutput(output="x", extra=dict(extra)) + out = rollout.environment_rollout(task, task.given(), None, pipeline, environments, 0.0) + assert out.extra["status"] == status and out.extra["verifier_reward_raw"] == 1.0 + assert environments.events[:2] == [("plan",), ("start", "demo")] and environments.events[4] == ("plan done",) + assert admitted == [True] * 3 # the CLI in it takes no wait or slot of its own + + +def test_a_run_that_ends_stops_the_pause_before_a_run_again(tmp_path): + import threading + from hermit.task import Task + from hermit.trainer import rollout + environments = Environments(tmp_path, fails=["sandbox_failed"]) + environments.stopping = threading.Event() + environments.infra_wait_s = 3600.0 + task = Task(id="demo#1", public={"id": "demo#1", "harbor_task": "demo"}) + session = environments.session + + def failing(task_dir, name=None, installs=None): # another task ends the run while this one fails + environments.stopping.set() + return session(task_dir, name) + + environments.session = failing + out = rollout.environment_rollout(task, task.given(), None, lambda *a, **k: None, environments, 0.0) + assert environments.starts == 1 and out.extra["unscored"] == "sandbox_failed" + from hermit.agent.base.interface import LoginFailed + with pytest.raises(LoginFailed): # and no environment starts for a task after it + rollout.environment_rollout(task, task.given(), None, lambda *a, **k: None, environments, 0.0) + assert environments.starts == 1 + + +def test_the_harbor_task_directories_are_run_secrets_and_a_login_reaches_its_hosts(tmp_path): + from types import SimpleNamespace + from hermit import isolate + (tmp_path / "tasks").mkdir() + data = {"data": {"tasks": None}, "judge": {}, "environment": {"harbor": {"tasks_dir": "tasks", + "harbor": sys.executable}}, + "agent": {"params": {"model": "gpt-5.6-luna", "login": "chatgpt"}}} + config = SimpleNamespace(data=data, base=tmp_path, path=lambda v: None if v is None else tmp_path / v, + run_dir=lambda: (_ for _ in ()).throw(ValueError("none"))) + secrets, _ = isolate._secrets(config) + assert (tmp_path / "tasks").resolve() in secrets + environments = isolate.task_environments(config) + assert "chatgpt.com" in environments.allowed_hosts and environments.admission is not None + + +def test_a_request_to_a_driver_that_reads_nothing_times_out(tmp_path): + """The request's write is the writer thread's: a driver whose pipe is full holds no caller past its timeout.""" + import time + from hermit.environment.harbor import _Driver + script = tmp_path / "deaf.py" + script.write_text("import time\ntime.sleep(60)\n") + original = harbor_mod.DRIVER + harbor_mod.DRIVER = script + try: + driver = _Driver(sys.executable, tmp_path / "driver.log", dict(os.environ)) + started = time.time() + with pytest.raises(TimeoutError): + driver.request("exec", 0.5, command="x" * (4 << 20)) + assert time.time() - started < 5 + started = time.time() + assert driver.close(1) is False # its writer stuck: killed, not waited on + assert time.time() - started < 30 + finally: + harbor_mod.DRIVER = original + + +def _driver_function(name: str, **space): + """One function of the driver, loaded alone (the driver imports harbor, which this interpreter has not).""" + import ast + source = (Path(harbor_mod.__file__).with_name("harbor_driver.py")).read_text() + node = next(n for n in ast.parse(source).body if isinstance(n, ast.FunctionDef) and n.name == name) + exec(compile(ast.Module([node], []), name, "exec"), space) + return space[name] + + +def test_the_agents_logs_are_copied_for_the_verifier_never_through_a_link_left_there(tmp_path): + """safe_copy: a link the agent put in /logs/agent, a folder it shares with this machine, is replaced, never written + through, and so is one it puts there while the copy runs (a background process of the agent's): a file where the + copy is writing, a folder the copy is in swapped for a link to this machine's files.""" + import contextlib + import shutil as shutil_mod + import uuid + victim, host_dir = tmp_path / "host-file", tmp_path / "host-dir" + victim.write_text("precious") + host_dir.mkdir() + logs, shared = tmp_path / "logs", tmp_path / "agent-dir" + (logs / "sub").mkdir(parents=True) + (logs / "linked").mkdir() + shared.mkdir() + (logs / "agent-codex.jsonl").write_text("stream") + (logs / "sub" / "a.txt").write_text("a") + (logs / "linked" / "b.txt").write_text("b") + (shared / "agent-codex.jsonl").symlink_to(victim) # there before the copy + (shared / "linked").symlink_to(host_dir) + (shared / "sub").mkdir() + + class Racing: # shutil, but the agent moves while a file is being written + def __getattr__(self, name): + return getattr(shutil_mod, name) + + @staticmethod + def copyfileobj(given, out): + if os.readlink(f"/proc/self/fd/{given.fileno()}").endswith("a.txt"): + (shared / "sub").rename(shared / "moved") # the folder the copy writes in, swapped + (shared / "sub").symlink_to(host_dir) + (shared / "moved" / "a.txt").symlink_to(victim) # and a link where its file goes + shutil_mod.copyfileobj(given, out) + + import stat as stat_mod + copy = _driver_function("safe_copy", os=os, shutil=Racing(), Path=Path, contextlib=contextlib, uuid=uuid, + stat=stat_mod) + copy(logs, shared) + assert victim.read_text() == "precious" and list(host_dir.iterdir()) == [] # nothing of this machine's touched + assert (shared / "agent-codex.jsonl").read_text() == "stream" + assert not (shared / "agent-codex.jsonl").is_symlink() + assert (shared / "linked" / "b.txt").read_text() == "b" and not (shared / "linked").is_symlink() + assert (shared / "moved" / "a.txt").read_text() == "a" # written where the copy held the folder + assert not (shared / "moved" / "a.txt").is_symlink() + + +def _own_pid_namespace() -> bool: + try: + return subprocess.run(["unshare", "--user", "--map-root-user", "--pid", "--fork", "--mount-proc", "true"], + capture_output=True, timeout=30).returncode == 0 + except (OSError, subprocess.SubprocessError): + return False + + +HARBOR_PYTHON = HARBOR_VENV.with_name("python") +DRIVER_FILE = Path(harbor_mod.__file__).with_name("harbor_driver.py") + + +def test_an_admitted_session_takes_no_slot_and_a_refused_one_holds_the_plan_and_comes_back(tmp_path, monkeypatch): + from hermit.agent.base import cli, login + from hermit.agent.base.interface import Result + from hermit.agent.model.usage import Usage + monkeypatch.setattr(login, "slot", lambda *a, **k: (_ for _ in ()).throw(AssertionError("no slot inside"))) + monkeypatch.setattr(login, "wait", lambda *a, **k: (_ for _ in ()).throw(AssertionError("no wait inside"))) + held = [] + monkeypatch.setattr(login, "hold", lambda plan, until=None, **k: held.append(until)) + monkeypatch.setattr(cli.CliAgent, "_session", lambda self, prompt, **k: Result( + reply="", usage=[Usage(model="m", input_tokens=5, calls=1)], + extra={"served": False, "refused": True, "held_until": 123.0})) + env = Recorded(tmp_path) + env.plan_admitted = True + out = cli.Codex({"web_search": False, "login": "chatgpt", "model": "m", + "plan_dir": str(tmp_path / "plan")}).run("x", environment=env) + assert out.extra["refused"] and held == [123.0] + + +def test_a_session_the_plan_refused_and_then_ran_counts_both_sessions_usage(tmp_path, monkeypatch): + from hermit.agent.base import cli, login + from hermit.agent.base.interface import Result + from hermit.agent.model.usage import Usage + import contextlib + monkeypatch.setattr(login, "wait", lambda *a, **k: "") + monkeypatch.setattr(login, "slot", lambda *a, **k: contextlib.nullcontext(True)) + monkeypatch.setattr(login, "held", lambda *a, **k: "") + monkeypatch.setattr(login, "hold", lambda *a, **k: None) + runs = iter([False, True]) + monkeypatch.setattr(cli.CliAgent, "_session", lambda self, prompt, **k: (lambda served: Result( + reply="ok" if served else "", usage=[Usage(model="m", input_tokens=10, cost_usd=0.01, calls=1)], + extra={"served": served, "refused": not served}))(next(runs))) + out = cli.Codex({"web_search": False, "login": "chatgpt", "model": "m", + "plan_dir": str(tmp_path / "plan")}).run("x", environment=tmp_path) + assert out.reply == "ok" and sum(u.input_tokens for u in out.usage) == 20 + + +def test_a_provider_retry_reuses_the_images_own_context_and_a_failed_attempt_keeps_its_usage(tmp_path, monkeypatch): + from hermit.agent.acting import workspace + reads = [] + + def answer(c): + if c[:2] == ["sh", "-c"] and "cat" in c[2]: + reads.append(c[2]) + return ExecResult(0, workspace.IMAGE_MARK + "IMAGE TEXT") + return ExecResult(0, "") + + work = tmp_path / "work" + work.mkdir() + env, cache = Recorded(tmp_path, answer), {} + for _ in range(3): # three deployments into one environment + (work / "CLAUDE.md").write_text("ADAPTER") + workspace._upload_staging(env, work, "claude", {"adapter_id": "a"}, {}, cache) + assert env.uploads["/app/CLAUDE.md"] == "ADAPTER\n\nIMAGE TEXT" + assert len(reads) == 1 # read once, never the merge back + removal = [c["command"] for c in env.calls if c["command"][:2] == ["sh", "-c"] and "-L" in c["command"][2]] + assert removal and "rm -f -- CLAUDE.md" in removal[0][2] # a link there replaced, never written through + + +def test_harbor_settings_override_on_the_command_line_and_eval_rows_keep_the_outcome(tmp_path): + from hermit.config import Config + examples = Path(__file__).resolve().parents[2] / "examples" + config = Config.load(examples / "terminal_bench_2_0" / "baseline" / "codex.yaml", + ["environment.harbor.infra_retries=0", "environment.harbor.agent_timeout_sec=900", + "environment.harbor.disk_gate_gb=0"]) + harbor = config.data["environment"]["harbor"] + assert (harbor["infra_retries"], harbor["agent_timeout_sec"], harbor["disk_gate_gb"]) == (0, 900, 0) + from hermit.harbor.report import kind + assert kind({"status": "judge_error", "outcome": {"status": "infra"}}) == "infra" + + +def test_a_host_pipelines_answer_agent_keeps_the_tasks_budget(tmp_path, monkeypatch): + """With the pipeline on the host (no worker deadline), the task's agent budget still ends the model's call.""" + from hermit.agent.acting import answer as answer_mod + from hermit.agent.model.llm import ChatAborted + + class Slow: + model = "m" + + def chat(self, messages, abort=None, **kwargs): + if abort is None or not abort.wait(10): + raise AssertionError("never given up") + raise ChatAborted("given up") + + monkeypatch.setattr(answer_mod.ChatClient, "from_spec", staticmethod(lambda spec: Slow())) + env = Recorded(tmp_path) + env.limits = Limits(timeout_s=0.3) + env.instruction = "Q?" + out = answer_mod.AnswerAgent({"model": {"base_url": "https://x", "model": "m"}, "logs": str(tmp_path / "l")}, + tmp_path).run({"id": "q"}, None, environment=env) + assert out.extra["timed_out"] is True and env.uploads == {} # no answer written past the budget + + +def test_a_rotated_sign_in_comes_back_though_the_sessions_exec_failed(tmp_path, monkeypatch): + from hermit.agent.base import login + from hermit.agent.base.interface import make + monkeypatch.setenv("HERMIT_HOMES", str(tmp_path / "homes")) + seen = [] + monkeypatch.setattr(login, "prepare", lambda root, home: (Path(home) / login.FILE).write_text('{"old": 1}')) + monkeypatch.setattr(login, "keep_refreshed", lambda root, home: seen.append(("keep", (Path(home) / login.FILE) + .read_text()))) + monkeypatch.setattr(login, "note_usage", lambda *a, **k: None) + + class Env(Recorded): + def exec(self, command, **kw): + if session_of(list(command)): + raise RuntimeError("EnvironmentUnavailable: the transport broke") + if "wc -c" in str(command[-1]): # the sign-in measured there first + return ExecResult(0, "200\n") + return super().exec(command, **kw) + + def download(self, source, target): + if source.endswith(login.FILE): + seen.append("download") + Path(target).write_text(ROTATED) + return Path(target) + raise FileNotFoundError(source) + + env = Env(tmp_path) + env.plan_admitted = True + with pytest.raises(RuntimeError, match="transport"): + make("codex", {"web_search": False, "login": "chatgpt", "model": "m", "bin": "codex", + "plan_dir": str(tmp_path / "plan"), "login_root": str(tmp_path / "root")}).run( + "x", environment=env) + assert seen == ["download", ("keep", ROTATED)] # brought back before the removal + + +def test_an_environment_failing_under_the_agent_or_this_machine_is_infra(harbor, monkeypatch): + environments, root, _ = harbor + with environments.session(environments.task_dir({"harbor_task": "demo"})) as env: + with pytest.raises(EnvironmentUnavailable): # a copy Docker failed: the environment's, not a 0 + env.download("/no/such/file", root.parent / "x") + monkeypatch.setattr(harbor_mod, "_Driver", lambda *a, **k: (_ for _ in ()).throw(OSError(24, "Too many files"))) + with pytest.raises(EnvironmentUnavailable) as raised: + environments.open(environments.task_dir({"harbor_task": "demo"})).start() + assert raised.value.cause == "host_failed" + from hermit.environment.harbor_errors import NO_RETRY + assert "host_failed" in NO_RETRY + + +# --- review round 3 --------------------------------------------------------------------------------------------------- + +def test_a_cli_sessions_provider_failure_runs_again_in_a_fresh_environment_on_its_own_allowance(tmp_path): + """The runner's busy_retries: the provider's allowance over the task's runs (not infra_retries), the n-th after + n x busy_wait_s, each run in a fresh environment; every run's spend kept.""" + from hermit.task import Task + from hermit.trainer import rollout + environments, pauses = Environments(tmp_path, infra_retries=0), [] + environments.busy_retries, environments.busy_wait_s = 2, 10.0 + task = Task(id="demo#1", public={"id": "demo#1", "harbor_task": "demo"}) + down = lambda t, a, environment=None: RunOutput(output=None, cost_usd=0.01, tokens=5, api_speed={"model_calls": 1}, + extra={"infra": "model_provider_failed", "infra_final": False}) + out = rollout.environment_rollout(task, task.given(), None, down, environments, 0.0, pause=pauses.append) + assert environments.starts == 3 and pauses == [10.0, 20.0] + assert out.extra["unscored"] == "model_provider_failed" and out.cost_usd == pytest.approx(0.03) + + +def test_an_adapter_that_failed_is_scored_zero_not_left_out(tmp_path, monkeypatch): + """The runner's adapter_failed and context_over_limit: the agent's result, scored 0 with its message (no + verifier), so a candidate's broken adapter stays in pass@1's denominator.""" + from hermit.agent.acting import workspace + from hermit.harbor.report import kind + from hermit.task import Task + from hermit.task.verify import environment as judge + from hermit.trainer import rollout + monkeypatch.setattr(workspace, "_deploy", lambda *a, **k: (_ for _ in ()).throw(KeyError("the context tool broke"))) + agent = workspace.CliAgent({"backend": "codex", "web_search": False, "bin": "codex", + "workspaces": str(tmp_path / "ws")}, tmp_path) + environments = Environments(tmp_path) + task = Task(id="demo#1", public={"id": "demo#1", "harbor_task": "demo", "instruction": "x"}) + record = rollout.run_one(task, tmp_path, agent.run, judge, environments) + assert record["status"] == "ok" and record["judge"]["score"] == 0.0 and kind(record) == "scored" + assert "the context tool broke" in record["judge"]["feedback"] + assert not [e for e in environments.events if e[0] == "verify"] # no verifier for it + + +def test_a_refused_session_without_a_plan_ends_the_run(tmp_path, monkeypatch): + from hermit.agent.acting.workspace import CliAgent + from hermit.agent.base import cli + from hermit.agent.base.interface import LoginFailed, Result + monkeypatch.setenv("HERMIT_HOMES", str(tmp_path / "homes")) + monkeypatch.setattr(cli.Codex, "run", lambda self, prompt, **k: Result( + reply="", exit_code=1, extra={"served": False, "refused": True, "unavailable": False, "started": True, + "failed": True, "output_tail": "", "failure": "You've hit your usage limit"})) + with pytest.raises(LoginFailed, match="no plan"): + CliAgent({"backend": "codex", "web_search": False, "bin": "codex", "workspaces": str(tmp_path / "ws")}, + tmp_path).run({"id": "t", "instruction": "x"}, None, environment=Recorded(tmp_path)) + + +def test_an_admitted_claude_session_that_drew_credits_holds_the_plan(tmp_path, monkeypatch): + from hermit.agent.base import cli, login + from hermit.agent.base.interface import Result + held, noted = [], [] + monkeypatch.setattr(login, "hold", lambda plan, until=None, why="session_refused", **k: held.append(why)) + monkeypatch.setattr(login, "note_usage", lambda *a, **k: noted.append(1)) + monkeypatch.setattr(cli.CliAgent, "_session", lambda self, prompt, **k: Result( + reply="done", extra={"served": True, "plan_report": {"using_credits": True, "resets_at": 99.0}})) + env = Recorded(tmp_path) + env.plan_admitted = True + monkeypatch.setenv(login.TOKEN_ENV, "t") + cli.Claude({"web_search": False, "login": "claude", "model": "claude-haiku-5-5", + "plan_dir": str(tmp_path / "plan")}).run("x", environment=env) + assert noted and held == ["usage_credits"] + + +def test_a_local_sign_in_copy_goes_though_its_take_back_failed(tmp_path, monkeypatch): + from hermit.agent.base import login + from hermit.agent.base.interface import make + monkeypatch.setenv("HERMIT_HOMES", str(tmp_path / "homes")) + copies = [] + + def prepare(root, home): + (Path(home) / login.FILE).write_text("{}") + copies.append(Path(home) / login.FILE) + + monkeypatch.setattr(login, "prepare", prepare) + monkeypatch.setattr(login, "keep_refreshed", lambda *a: (_ for _ in ()).throw(OSError(28, "No space left"))) + env = Recorded(tmp_path, lambda c: ExecResult(0, CODEX_STREAM) if session_of(c) else ExecResult(0, "")) + env.plan_admitted = True + out = make("codex", {"web_search": False, "login": "chatgpt", "model": "m", "bin": "codex", + "home": str(tmp_path / "kept"), "plan_dir": str(tmp_path / "plan")}).run("x", environment=env) + assert out.reply == "fixed it" # the finished session stands (the runner's) + assert copies and not copies[0].exists() # removed however the take-back went + + +def test_a_react_timeout_keeps_what_the_loop_spent_and_asks_no_follow_up(tmp_path, monkeypatch): + import time + from hermit.agent.acting.react import ReactAgent, tool + from hermit.agent.model import llm + from hermit.agent.model.usage import Usage + calls = [] + + class Model: + model = "m" + + def chat(self, messages, abort=None, **kwargs): + calls.append(1) + return ({"role": "assistant", "content": "", "tool_calls": [ + {"id": "c", "type": "function", "function": {"name": "slow", "arguments": "{}"}}]}, + Usage(model="m", input_tokens=100, output_tokens=10, calls=1, cost_usd=0.5)) + + monkeypatch.setattr(llm.ChatClient, "from_spec", staticmethod(lambda spec: Model())) + + class Agent(ReactAgent): + @tool + def slow(self) -> str: + """Takes longer than the budget.""" + time.sleep(0.5) + return "done" + + env = Recorded(tmp_path) + env.limits = Limits(timeout_s=0.2) + out = Agent({"model": {"base_url": "https://x", "model": "m"}, "logs": str(tmp_path / "l"), + "answer_tag": "FINAL"}, tmp_path).run({"id": "t", "instruction": "go"}, None, environment=env) + assert out.extra["timed_out"] is True and out.tokens == 110 and out.cost_usd == 0.5 and len(calls) == 1 + + +def test_the_answer_agents_fd_retries_attachments_and_http_refusals(tmp_path, monkeypatch): + import os as os_mod + from hermit.agent.acting import answer as answer_mod + from hermit.agent.base.interface import LoginFailed + from hermit.agent.model.llm import ChatHTTPError + from hermit.agent.model.usage import Usage + made, sent = [], [] + + class Model: + model, max_retries = "m", 4 + + def __init__(self): + made.append(self) + + def chat(self, messages, **kwargs): + sent.append(messages) + return {"role": "assistant", "content": "A"}, Usage(model="m", calls=1) + + monkeypatch.setattr(answer_mod.ChatClient, "from_spec", staticmethod(lambda spec: Model())) + fds = len(os_mod.listdir("/proc/self/fd")) + + class Env(Recorded): + def download(self, source, target): + Path(target).write_bytes(b"\x89PNG") + return Path(target) + + def answer(c): # the listing, then the image measured and read there + if "wc -c" in c[-1]: + return ExecResult(0, "4\n") + if "base64" in c[-1]: # framed, a client's warning beside it + return ExecResult(0, "WARN: compose\nhermit-attachment-begin\niVBORw==\nhermit-attachment-end\n") + return ExecResult(0, "/app/a.png\n/app/a.png\n") + + env = Env(tmp_path, answer) + agent = answer_mod.AnswerAgent({"model": {"base_url": "https://x", "model": "m", "max_retries": 0}, + "attachments": ["/app/*.png", "/app/a.png"], "logs": str(tmp_path / "l")}, + tmp_path) + for _ in range(3): + agent.run({"id": "q", "instruction": "Which?"}, None, environment=env) + assert len(os_mod.listdir("/proc/self/fd")) == fds # no descriptor left per task + assert made[0].max_retries == 0 # the model's own setting honoured + assert sum(1 for part in sent[0][-1]["content"] if part["type"] == "image_url") == 1 # each image once + + def refused(code, body): + class Refusing(Model): + def chat(self, messages, **kwargs): + raise ChatHTTPError(code, body.encode(), {}) + monkeypatch.setattr(answer_mod.ChatClient, "from_spec", staticmethod(lambda spec: Refusing())) + return answer_mod.AnswerAgent({"model": {"base_url": "https://x", "model": "m"}, "logs": str(tmp_path / "l")}, + tmp_path).run({"id": "q", "instruction": "Which?"}, None, + environment=Recorded(tmp_path)) + with pytest.raises(LoginFailed, match="usage_limit"): + refused(402, "payment required") + with pytest.raises(LoginFailed, match="usage_limit"): + refused(429, '{"error": {"code": "insufficient_quota"}}') + with pytest.raises(RuntimeError, match="model_rejected"): + refused(400, '{"error": {"message": "Unsupported parameter: max_tokens"}}') + assert refused(400, '{"error": {"message": "prompt is too long"}}').output["answer"] == "" # the agent's + + +def test_the_verifier_watchdog_follows_harbors_own_timeout(harbor, monkeypatch): + environments, _, _ = harbor + asked = [] + with environments.session(environments.task_dir({"harbor_task": "demo"})) as env: + env.started["verifier_timeout_sec"] = 14400.0 + original = env.driver.request + env.driver.request = lambda op, timeout, **f: asked.append((op, timeout)) or original(op, timeout, **f) + env.verify() + assert ("done", 14400.0 + harbor_mod.VERIFY_ALLOWANCE_S) in asked + + +def test_a_teardown_failure_keeps_the_tasks_verdict_and_a_host_failure_is_not_run_again(tmp_path): + import contextlib + from hermit.task import Task + from hermit.trainer import rollout + environments = Environments(tmp_path) + session = environments.session + + @contextlib.contextmanager + def breaking(task_dir, name=None, installs=None): + with session(task_dir, name) as env: + yield env + raise OSError(28, "No space left on device: the cleanup lock") + + environments.session = breaking + task = Task(id="demo#1", public={"id": "demo#1", "harbor_task": "demo"}) + out = rollout.environment_rollout(task, task.given(), None, lambda t, a, environment=None: RunOutput( + output="x", cost_usd=0.5, tokens=100), environments, 0.0) + assert out.extra["verified"]["reward"] == 1.0 and out.cost_usd == 0.5 # the verdict and its spend kept + + def host(t, a, environment=None): + raise OSError(24, "Too many open files") + + out = rollout.environment_rollout(task, task.given(), None, host, Environments(tmp_path), 0.0, + pause=lambda s: None) + assert out.extra["unscored"] == "host_failed" and len(out.extra.get("attempts", [1])) == 1 + + +def test_a_lost_sign_in_stops_new_work_before_the_tasks_teardown(tmp_path): + import contextlib + import threading + from hermit.agent.base.interface import LoginFailed + from hermit.task import Task + from hermit.trainer import rollout + environments = Environments(tmp_path) + environments.stopping = threading.Event() + seen = [] + session = environments.session + + @contextlib.contextmanager + def watched(task_dir, name=None, installs=None): + try: + with session(task_dir, name) as env: + yield env + finally: + seen.append(environments.stopping.is_set()) # at the teardown: the run already ended + + environments.session = watched + + def lost(t, a, environment=None): + raise LoginFailed("agent codex: 401") + + task = Task(id="demo#1", public={"id": "demo#1", "harbor_task": "demo"}) + with pytest.raises(LoginFailed): + rollout.environment_rollout(task, task.given(), None, lost, environments, 0.0) + assert seen == [True] + + +def test_a_minimal_rows_context_is_computed_from_harbors_instruction(tmp_path, monkeypatch): + from hermit.agent.acting import workspace + asked = [] + monkeypatch.setattr(workspace, "_deploy", lambda task, *a, **k: asked.append(task) or + (_ for _ in ()).throw(KeyError("stop here"))) + env = Recorded(tmp_path) + env.instruction = "What is 6 x 7?" + workspace.CliAgent({"backend": "codex", "web_search": False, "bin": "codex", "workspaces": str(tmp_path / "w")}, + tmp_path).run({"id": "t#1", "harbor_task": "t"}, tmp_path, environment=env) + assert asked[0]["instruction"] == "What is 6 x 7?" + + +def test_rrsi_sees_a_harbor_rollouts_failure_and_a_cli_model_is_refused_for_harbor(tmp_path): + from types import SimpleNamespace + from hermit import isolate + from hermit.agent.evolver.methods.rrsi_adapter.domain import AdapterDomain + domain = AdapterDomain.__new__(AdapterDomain) + assert domain.smoke_failure({"output": "done", "outcome": {"status": "agent_error"}}) is True + assert domain.smoke_failure({"output": "done", "outcome": {"status": "ok"}}) is False + (tmp_path / "tasks").mkdir() + data = {"environment": {"harbor": {"tasks_dir": "tasks", "harbor": sys.executable}}, + "agent": {"params": {"model": {"cli": "codex", "model": "m", "login": "chatgpt"}}}} + with pytest.raises(ValueError, match="a model a CLI reaches"): + isolate.task_environments(SimpleNamespace(data=data, base=tmp_path)) + + +def test_a_run_that_ended_while_the_disk_gate_held_a_task_starts_nothing(harbor, monkeypatch): + """The run stopped (another task's lost sign-in) while this task waited at the disk gate, or between its gates: + no driver, no container; the rollout ends as the stopped run's (LoginFailed), not as an infra failure.""" + environments, _, calls = harbor + + def gate(where, gate_gb, wait, **kw): + calls["gate"].append(str(where)) + environments.stopping.set() # the stop arrives while it waits; room is found anyway + return True + + monkeypatch.setattr(harbor_mod.docker, "wait_for_disk", gate) + with pytest.raises(EnvironmentUnavailable, match="run ended") as raised: + environments.open(environments.task_dir({"harbor_task": "demo"})).start() + assert raised.value.cause == "stopped" and calls["gate"] == ["/tmp"] # the second gate never entered + assert not (environments.jobs_dir).exists() or not any(environments.jobs_dir.rglob("requests.jsonl")) + assert not environments.live + + +def _prompt_adapter(tmp_path, text): + a = tmp_path / "adapter" + (a / ".hermit").mkdir(parents=True) + (a / ".hermit" / "adapter.yaml").write_text("template: {id: big, version: '1'}\n" + "structure: {learnable: [ADAPTER.md]}\n") + (a / "exports.yaml").write_text("{context: ADAPTER.md}\n") + (a / "ADAPTER.md").write_text(text) + return a + + +def test_codex_in_a_tasks_environment_keeps_the_runners_256k_instruction_allowance(tmp_path, monkeypatch): + """A 35,000-byte prompt-only adapter (the template allows 40,000) runs as it ran in the Harbor runner, whose Codex + took project_doc_max_bytes = 262144: deployed, checked against that cap, and the session's config says it.""" + from hermit.agent.acting.workspace import CliAgent + monkeypatch.setenv("HERMIT_HOMES", str(tmp_path / "homes")) + env = Recorded(tmp_path, lambda c: ExecResult(0, CODEX_STREAM) if session_of(c) else + ExecResult(1, "") if c[:2] == ["test", "-e"] else ExecResult(0, "")) + adapter = _prompt_adapter(tmp_path, "Be careful.\n" * 2917) # 35,004 bytes + out = CliAgent({"backend": "codex", "web_search": False, "bin": "codex", "workspaces": str(tmp_path / "ws")}, + tmp_path).run({"id": "t", "instruction": "do it"}, adapter, environment=env) + assert out.output == "fixed it" and "adapter_failed" not in out.extra + assert len(env.uploads["/app/AGENTS.md"].encode()) > 32768 + assert "project_doc_max_bytes = 262144" in env.uploads["/installed-agent/hermit-home"]["config.toml"] + + +def test_a_hook_audit_that_cannot_be_read_keeps_the_finished_session_and_its_usage(tmp_path, monkeypatch): + from hermit.agent.acting import workspace + from hermit.agent.acting.workspace import CliAgent + monkeypatch.setenv("HERMIT_HOMES", str(tmp_path / "homes")) + real = workspace._deploy + monkeypatch.setattr(workspace, "_deploy", lambda *a, **k: ({**real(*a, **k)[0], "hooks": True}, None)) + monkeypatch.setattr(workspace, "_ensure_python3", lambda env: True) + monkeypatch.setattr(workspace, "_collect_hooks", lambda env, log: (_ for _ in ()).throw( + RuntimeError("docker cp failed"))) + env = Recorded(tmp_path, lambda c: ExecResult(0, CODEX_STREAM) if session_of(c) else ExecResult(0, "")) + out = CliAgent({"backend": "codex", "web_search": False, "bin": "codex", "workspaces": str(tmp_path / "ws")}, + tmp_path).run({"id": "t", "instruction": "do it"}, None, environment=env) + assert out.output == "fixed it" and out.tokens == 15 and "docker cp failed" in out.extra["hooks_audit_missing"] + assert "infra" not in out.extra # the verifier decides the rest + + +def test_the_benchmark_file_form_takes_nested_overrides_and_a_remote_cli_is_not_the_hosts(tmp_path, monkeypatch): + from hermit.agent.base import login + from hermit.config import Config + import yaml + folder = Path(__file__).resolve().parents[2] / "examples" / "terminal_bench_2_0" / "baseline" + run = yaml.safe_load((folder / "codex.yaml").read_text()) + run["data"] = {k: str((folder / v).resolve()) for k, v in run["data"].items()} + run["task_info"] = str((folder / run["task_info"]).resolve()) + benchmark = str((folder / "../utils/benchmark.yaml").resolve()) + run["environment"] = {"harbor": benchmark} # the file alone + (tmp_path / "run.yaml").write_text(yaml.safe_dump(run)) + config = Config.load(tmp_path / "run.yaml", ["environment.harbor.infra_retries=0"]) + assert config.data["environment"]["harbor"] == {"benchmark": benchmark, "infra_retries": 0} + data = {"environment": {"harbor": {"benchmark": "b.yaml"}}, + "agent": {"params": {"login": "claude", "bin": "/opt/hermit-claude/claude"}}} + ((who, _, program, contained, _),) = login.logins_of(data) + assert who == "agent" and program == "/opt/hermit-claude/claude" and contained is True + monkeypatch.setenv(login.TOKEN_ENV, "t") + assert login.problems_of("claude", program, contained) == [] # not looked for on this machine + + +def test_the_answer_agent_scores_its_adapters_failure_retries_its_environments_and_delivers_in_time(tmp_path, + monkeypatch): + import time + from hermit.agent.acting import answer as answer_mod + from hermit.agent.model.usage import Usage + from hermit.environment.harbor_errors import pipeline_cause + + class Model: + model = "m" + + def chat(self, messages, abort=None, **kwargs): + return {"role": "assistant", "content": "B"}, Usage(model="m", calls=1, cost_usd=0.01) + + monkeypatch.setattr(answer_mod.ChatClient, "from_spec", staticmethod(lambda spec: Model())) + params = {"model": {"base_url": "https://x", "model": "m"}, "logs": str(tmp_path / "l")} + + class Failing: + last_context = None + + def context(self, **kw): + raise ValueError("the context tool broke") + + def close(self): + pass + + agent = answer_mod.AnswerAgent(params, tmp_path) + monkeypatch.setattr(agent, "session", lambda adapter: Failing()) + env = Recorded(tmp_path) + env.instruction = "Which?" + out = agent.run({"id": "q"}, tmp_path, environment=env) + assert out.extra["adapter_failed"] == "adapter_failed" and out.output is None # the agent's result: scored 0 + gone = Recorded(tmp_path, lambda c: ExecResult(1, "no such file")) + with pytest.raises(RuntimeError) as raised: + answer_mod.AnswerAgent({**params, "instruction_file": "/app/q.txt", "wait_s": 0.01}, tmp_path).run( + {"id": "q"}, None, environment=gone) + assert pipeline_cause(f"RuntimeError: {raised.value}") == ("harbor_failed", True) # run again, a fresh one + + class Late(Recorded): # the answer's upload outlasts the budget + def upload(self, source, target): + time.sleep(0.4) + return super().upload(source, target) + + late = Late(tmp_path) + late.limits, late.instruction = Limits(timeout_s=0.2), "Which?" + out = answer_mod.AnswerAgent(params, tmp_path).run({"id": "q"}, None, environment=late) + assert out.extra["timed_out"] is True and "/app/answer.txt" not in late.uploads + on_time = Recorded(tmp_path) + on_time.instruction = "Which?" + answer_mod.AnswerAgent(params, tmp_path).run({"id": "q"}, None, environment=on_time) + assert on_time.uploads == {"/app/answer.txt": "B"} # moved into place + + +def test_react_keeps_the_budget_through_the_answer_tag_follow_up_and_counts_an_abandoned_call(tmp_path, monkeypatch): + from hermit.agent.acting.react import ReactAgent + from hermit.agent.model import llm + from hermit.agent.model.llm import ChatAborted + from hermit.agent.model.usage import Usage + calls = [] + + class Model: + model = "m" + + def chat(self, messages, abort=None, **kwargs): + calls.append(abort) + if len(calls) == 1: # an answer without the tag: the follow-up is asked + return {"role": "assistant", "content": "B"}, Usage(model="m", input_tokens=10, calls=1, cost_usd=0.1) + if abort is None or not abort.wait(10): + raise AssertionError("the follow-up was not bounded by the task's budget") + raise ChatAborted("given up") + + monkeypatch.setattr(llm.ChatClient, "from_spec", staticmethod(lambda spec: Model())) + env = Recorded(tmp_path) + env.limits = Limits(timeout_s=0.3) + out = ReactAgent({"model": {"base_url": "https://x", "model": "m"}, "logs": str(tmp_path / "l"), + "answer_tag": "FINAL"}, tmp_path).run({"id": "t", "instruction": "go"}, None, environment=env) + assert len(calls) == 2 and out.extra["timed_out"] is True + assert out.cost_usd is None and out.api_speed["model_calls"] == 2 and out.tokens == 10 # its cost not known + + +def test_a_cli_agent_declares_what_harbors_setup_installs_and_the_phase_installs_nothing(harbor, tmp_path): + """Agent.installs: the CLI (unless `bin` names one there; probed first unless a version is pinned) and python3 + for an adapter's hooks, which the run hands the environment's setup; during the phase install() only reports + what setup installed, and refuses anything else.""" + from hermit.agent.acting.workspace import CliAgent + from hermit.agent.acting.agent import Agent + assert Agent().installs(None) == [] # an agent that needs nothing + codex = CliAgent({"backend": "codex", "version": "0.162.0"}, tmp_path).installs(None) + assert codex[0]["agent"] == "codex" and codex[0]["version"] == "0.162.0" and "command -v codex" in codex[0]["probe"] + mounted = CliAgent({"backend": "claude", "bin": "/opt/claude"}, tmp_path).installs(None) + assert mounted == [{"agent": "claude-code", "version": None, "bin": "/opt/claude", "program": "claude"}] + environments, _, _ = harbor + with environments.session(environments.task_dir({"harbor_task": "demo"}), installs=codex) as env: + sent = [json.loads(line) for line in (tmp_path / "requests.jsonl").read_text().splitlines()] + assert sent[0]["installs"] == codex + env.started["installed"] = {"codex": "0.162.0"} + assert env.install("codex", "0.162.0") == "0.162.0" + with pytest.raises(Unsupported, match="not installed at the environment's setup"): + env.install("claude-code") + + +def test_the_driver_scrubs_the_agents_credentials_from_the_logs_it_hands_the_verifier(tmp_path): + import contextlib + import shutil as shutil_mod + import uuid + import stat as stat_mod + copy = _driver_function("safe_copy", os=os, shutil=shutil_mod, Path=Path, contextlib=contextlib, uuid=uuid, + stat=stat_mod, + scrubbed=_driver_function("scrubbed")) + logs, shared = tmp_path / "logs", tmp_path / "agent" + logs.mkdir() + shared.mkdir() + (logs / "agent-codex.jsonl").write_text(f"401: bad key {AGENT_KEY}\n") + copy(logs, shared, {AGENT_KEY}) + assert (shared / "agent-codex.jsonl").read_text() == "401: bad key [REDACTED]\n" + + +def test_harbor_settings_redact_the_verifier_and_take_the_named_reward_only(harbor, tmp_path): + environments, _, _ = harbor + with pytest.raises(ValueError, match="verifier_redact"): + HarborEnvironments({"tasks_dir": str(tmp_path / "tasks"), "harbor": sys.executable, + "verifier_redact": ["("]}, base=tmp_path) + redacting = HarborEnvironments({"tasks_dir": str(tmp_path / "tasks"), "harbor": sys.executable, + "reward_key": "accuracy", "verifier_redact": [r"expected \d+"]}, base=tmp_path) + out = tmp_path / "verifier" + out.mkdir() + (out / "test-stdout.txt").write_text("FAILED: expected 44463, got 12\n") + verdict = redacting.verdict({"rewards": {"accuracy": 0.0}, "verifier_dir": str(out)}) + assert "44463" not in verdict["feedback"] and "[redacted]" in verdict["feedback"] + missing = redacting.verdict({"rewards": {"f1": 0.85, "answered": 1}, "verifier_dir": str(out)}) + assert (missing["status"], missing["failure_cause"], missing["reward"]) == ("infra", "verifier_failed", None) + + +def test_an_adapter_failure_on_a_scale_benchmark_has_no_pass(tmp_path): + from hermit.trainer.rollout import _read + from hermit.task import Task + env = Recorded(tmp_path) + env.owner = type("Owner", (), {"pass_reward": None})() + out = _read(RunOutput(output=None, extra={"adapter_failed": "adapter_failed", "exception_message": "x"}), env, + Task(id="t", public={}), 0.0) + assert out.extra["verified"]["passed"] is None + + +def test_the_sandboxed_agent_uses_the_configured_chatgpt_login_root(tmp_path): + from hermit import isolate + from hermit.agent.base import login as codex_login + agent = isolate.SandboxedAgent.__new__(isolate.SandboxedAgent) + agent.params = {"login_root": str(tmp_path / "root")} + assert agent._login_root() == tmp_path / "root" + agent.params = {} + assert agent._login_root() == codex_login.ROOT + + +def test_a_refreshed_sign_in_that_is_no_valid_one_never_replaces_the_shared_copy(tmp_path, monkeypatch): + from hermit.agent.base import codex_stream, login + from hermit.agent.base.interface import make + monkeypatch.setenv("HERMIT_HOMES", str(tmp_path / "homes")) + kept = [] + monkeypatch.setattr(login, "prepare", lambda root, home: (Path(home) / login.FILE).write_text(ROTATED)) + monkeypatch.setattr(login, "keep_refreshed", lambda root, home: kept.append((Path(home) / login.FILE).read_text())) + monkeypatch.setattr(login, "note_usage", lambda *a, **k: None) + no_tokens = json.dumps({"tokens": {"account_id": "acc"}, "last_refresh": "2026-11-19T00:00:00Z"}) + + class Env(Recorded): + def download(self, source, target): + if source.endswith(login.FILE): + Path(target).write_text(no_tokens) + return Path(target) + raise FileNotFoundError(source) + + env = Env(tmp_path, lambda c: ExecResult(0, CODEX_STREAM) if session_of(c) else + ExecResult(0, "100\n") if "wc -c" in str(c[-1]) else ExecResult(0, "")) + env.plan_admitted = True + make("codex", {"web_search": False, "login": "chatgpt", "model": "m", "bin": "codex", + "plan_dir": str(tmp_path / "plan"), "login_root": str(tmp_path / "root")}).run("x", environment=env) + assert kept == [ROTATED] # the session's own copy, not the file it handed back + assert not codex_stream.signed_in(b"x" * 10, 60) and codex_stream.regular_bytes(tmp_path / "none", 10) is None + big = tmp_path / "big.json" + big.write_bytes(b" " * (codex_stream.SIGNIN_MAX_BYTES + 1)) + assert codex_stream.regular_bytes(big, codex_stream.SIGNIN_MAX_BYTES) is None + + +def test_a_long_codex_rejection_keeps_its_status_for_the_verdict(tmp_path): + from hermit.agent.base.cli import Codex + from hermit.agent.base.codex_stream import session_verdict + raw = json.dumps({"type": "turn.failed", "error": {"message": "unexpected status 402 Payment Required: " + + "proxy said " * 200}}) + "\n" + agent = Codex({"web_search": False}) + agent.environment = Recorded(tmp_path) + stream = tmp_path / "s.jsonl" + stream.write_text(raw) + said = agent.outcome(raw, stream, tmp_path / "none") + assert said["failure"].startswith("unexpected status 402") or "402" in said["failure"][:100] + assert session_verdict({**said, "served": False}, 0) == "usage_limit" + + +def test_a_session_cut_off_before_its_usage_reports_an_unknown_cost(tmp_path, monkeypatch): + from hermit.agent.acting.workspace import CliAgent + monkeypatch.setenv("HERMIT_HOMES", str(tmp_path / "homes")) + started = json.dumps({"type": "thread.started", "thread_id": "t"}) + "\n" + json.dumps({"type": "turn.started"}) + env = Recorded(tmp_path, lambda c: ExecResult(1, started) if session_of(c) else ExecResult(0, "")) + out = CliAgent({"backend": "codex", "web_search": False, "bin": "codex", "workspaces": str(tmp_path / "ws")}, + tmp_path).run({"id": "t", "instruction": "x"}, None, environment=env) + assert out.cost_usd is None # never a known 0 + + +def test_a_refused_session_whose_hold_cannot_be_written_pauses_before_the_next_environment(tmp_path, monkeypatch): + from hermit.agent.base import login + monkeypatch.setattr(login, "_hold", lambda *a, **k: (_ for _ in ()).throw(OSError(28, "No space left"))) + assert login.hold(tmp_path / "plan", max_s=0.0) is False + from hermit.task import Task + from hermit.task.verify import environment as judge + from hermit.trainer import rollout + pauses, runs = [], [] + + def pipeline(task, adapter, environment=None): + runs.append(1) + if len(runs) == 1: + return RunOutput(output=None, extra={"infra": "usage_limit", "infra_final": False, + "exception_message": "refused", "hold_unwritten": True}) + return RunOutput(output="done") + + monkeypatch.setattr(rollout, "_sleep", pauses.append) + environments = Environments(tmp_path) + task = Task(id="demo#1", public={"id": "demo#1", "harbor_task": "demo"}) + record = rollout.run_one(task, None, pipeline, judge, environments) + assert len(runs) == 2 and pauses == [login.HOLD_UNWRITTEN_S] and record["judge"]["score"] == 1.0 + + +def test_the_answer_agent_reads_patterns_and_token_limits_as_the_runner(tmp_path, monkeypatch): + from hermit.agent.acting import answer as answer_mod + from hermit.agent.model.usage import Usage + sent = [] + + class Model: + model = "m" + + def chat(self, messages, abort=None, **kwargs): + sent.append(kwargs) + return {"role": "assistant", "content": "yes"}, Usage(model="m", calls=1, cost_usd=0.01) + + monkeypatch.setattr(answer_mod.ChatClient, "from_spec", staticmethod(lambda spec: Model())) + params = {"model": {"base_url": "https://x", "model": "m"}, "logs": str(tmp_path / "l")} + + env = Recorded(tmp_path) + env.instruction = "Which?" + answer_mod.AnswerAgent(params, tmp_path).run({"id": "q"}, None, environment=env) + assert "max_tokens" not in sent[0] # none given: none sent (the runner's) + agent = answer_mod.AnswerAgent({**params, "answer_pattern": r"yes(?:=(\w+))?"}, tmp_path) + assert agent.answer("yes") == "yes" and agent.answer("yes=B") == "B" + + +def test_a_dead_driver_wakes_every_request_at_once(tmp_path): + """The reader publishes the death under the lock a request registers under: no request waits its watchdog.""" + import contextlib + import threading + import time + from hermit.environment.harbor import _Driver + script = tmp_path / "dies.py" + script.write_text("import sys, time\nsys.stdin.readline()\ntime.sleep(0.2)\n") + original = harbor_mod.DRIVER + harbor_mod.DRIVER = script + try: + driver = _Driver(sys.executable, tmp_path / "driver.log", dict(os.environ)) + ends = [] + + def ask(): + started = time.time() + with contextlib.suppress(EnvironmentUnavailable, RuntimeError): + driver.request("exec", 30, command="x") + ends.append(time.time() - started) + + threads = [threading.Thread(target=ask) for _ in range(20)] + for t in threads: + t.start() + time.sleep(0.02) + for t in threads: + t.join(timeout=40) + assert len(ends) == 20 and max(ends) < 10 + driver.close(5) + finally: + harbor_mod.DRIVER = original + + +def test_the_cli_agent_refuses_limits_given_with_the_environment(tmp_path): + from hermit.agent.acting.workspace import CliAgent + env = Recorded(tmp_path) + env.limits = Limits(timeout_s=600, network=[]) + with pytest.raises(Unsupported, match="network limit"): + CliAgent({"backend": "codex", "web_search": False, "bin": "codex", "workspaces": str(tmp_path / "ws")}, + tmp_path).run({"id": "t", "instruction": "x"}, None, environment=env) + + +# ---- the single implementations ---------------------------------------------------------------------------------- + +def test_the_usage_ledger_sums_once_with_unknown_costs_and_empty_ledgers_as_the_caller_says(): + from hermit.agent.model.usage import Attempts, Ledger, Usage + ledger = Ledger("m").add([Usage(model="m", input_tokens=10, output_tokens=5, calls=1, cost_usd=0.1)]) + assert (ledger.cost(), ledger.tokens(), ledger.calls()) == (0.1, 15, 1) + ledger.unreported(1) # a call given up in flight: counted, its cost unknown + assert ledger.cost() is None and ledger.calls() == 2 and ledger.total().unreported_calls == 1 + assert Ledger().cost(0.0) == 0.0 and Ledger().tokens(None) is None # nothing ran / nothing known + cut = Ledger("m").unreported(0, sessions=1) # a session cut off before any usage: unknown, never 0 + assert (cut.tokens(0), cut.cost(0.0)) == (None, None) + attempts = Attempts() + for tokens, cost, cause in ((10, 0.1, "sandbox_failed"), (None, None, None)): + attempts.add(RunOutput(output=None, tokens=tokens, cost_usd=cost, api_speed={"model_calls": 1}, + extra={"unscored": cause})) + out = attempts.output() + assert (out.tokens, out.cost_usd, out.api_speed["model_calls"]) == (10, None, 2) + assert [a["unscored"] for a in out.extra["attempts"]] == ["sandbox_failed", None] + + +def test_a_credential_is_taken_back_and_removed_however_the_session_ends(): + from hermit.agent.base.credentials import held + seen = [] + with pytest.raises(KeyError): + with held(lambda: ["copy"], lambda c: seen.append(("back", c)), + lambda c: (seen.append(("removed", c)), (_ for _ in ()).throw(RuntimeError("rm failed")))): + raise KeyError("the session failed") # its error stands; the removal's is printed + assert seen == [("back", ["copy"]), ("removed", ["copy"])] + with pytest.raises(RuntimeError, match="rm failed"): # a session that ended well: the removal's error raised + with held(lambda: [], None, lambda c: (_ for _ in ()).throw(RuntimeError("rm failed"))): + pass + seen.clear() + with pytest.raises(OSError): # a take-back that fails never keeps the copies + with held(lambda: ["c"], lambda c: (_ for _ in ()).throw(OSError("full")), lambda c: seen.append(c)): + pass + assert seen == [["c"]] + + +def test_one_deadline_cuts_limits_and_ends_model_calls(): + import time + from hermit.agent.base.deadline import Deadline + from hermit.agent.model.llm import ChatAborted + assert Deadline().cut(30) == 30 and Deadline().remaining() is None + assert Deadline(time.time() - 1).cut(30) is False + assert 9 < Deadline.within(10).cut(30) <= 10 + + class Slow: + model = "m" + + def chat(self, messages, abort=None, **kw): + abort.wait(5) + raise ChatAborted("given up") + + bound = Deadline.within(0.2).bound(Slow()) + with pytest.raises(ChatAborted): + bound.chat([]) + assert bound.expired and bound.abandoned == 1 + bound.close() + + +def test_the_retry_state_machine_names_each_state(): + from types import SimpleNamespace + from hermit.trainer import rollout + envs = SimpleNamespace(infra_retries=1, busy_retries=1, busy_wait_s=120.0, infra_wait_s=300.0, stopping=None) + out = lambda cause, **extra: RunOutput(output=None, extra={"unscored": cause, **extra}) + machine = rollout.Retries(envs) + assert machine.next(out(None), False, False) == (rollout.DONE, 0.0) + assert machine.next(out("model_provider_failed"), False, True) == (rollout.PROVIDER_BUSY, 120.0) + assert machine.next(out("model_provider_failed"), False, True) == (rollout.DONE, 0.0) # its allowance spent + assert machine.next(out("usage_limit", hold_unwritten=True), False, True)[0] == rollout.PLAN_WAIT + assert machine.next(out("sandbox_failed"), False, False) == (rollout.INFRA, 300.0) + assert machine.next(out("sandbox_failed"), False, False) == (rollout.DONE, 0.0) # infra_retries spent + assert rollout.Retries(envs).next(out("disk"), False, False) == (rollout.DONE, 0.0) # runs no better again + import threading + stopped = threading.Event() + stopped.set() + assert rollout.Retries(SimpleNamespace(**{**vars(envs), "stopping": stopped})).next( + out("sandbox_failed"), False, False) == (rollout.STOP, 0.0) + + +@pytest.mark.skipif(not HARBOR_PYTHON.exists(), reason="needs Harbor's venv (~/.cache/hermit/harbor-venv)") +def test_the_driver_stops_unblocked_kills_cancelled_clients_and_keeps_a_verdict_it_could_not_write(tmp_path): + """In Harbor's interpreter: a stop is never behind a start waiting on its trial; a client whose wait is cancelled + is killed (a compose up never outlives the teardown); a hook package that fails to install leaves the setup + going; a verdict whose result.json could not be written comes from the trial itself; Harbor's refusal of a + Windows task at the trial's creation is UnsupportedTask.""" + script = tmp_path / "probe.py" + script.write_text(textwrap.dedent(f""" + import asyncio, importlib.util, json, sys + from pathlib import Path + from types import SimpleNamespace + spec = importlib.util.spec_from_file_location("drv", {str(DRIVER_FILE)!r}) + drv = importlib.util.module_from_spec(spec); sys.modules["drv"] = drv; spec.loader.exec_module(drv) + replies = [] + + class Out: + def write(self, s): replies.append(json.loads(s)) + def flush(self): pass + + async def main(): + out = [] + d = drv.DRIVER = drv.Driver(Out()) + async with d.lock: # a start holds the lock, waiting on its trial + await asyncio.wait_for(d.handle({{"id": 7, "op": "stop"}}), 5) + out.append(replies[-1]["id"] == 7 and replies[-1]["ok"]) + live = d.run_task = asyncio.ensure_future(asyncio.sleep(30)); await asyncio.sleep(0) + await asyncio.wait_for(d.stop(), 5) # a live trial (a start watchdog): cancelled, nothing raised + out.append(live.cancelled()) + d.run_task = None + async def waits(): + p = await asyncio.create_subprocess_exec("sleep", "30", stdout=asyncio.subprocess.PIPE) + held.append(p) + await p.communicate() + held = [] + t = asyncio.create_task(waits()); await asyncio.sleep(0.3); t.cancel() + await asyncio.gather(t, return_exceptions=True) + await asyncio.sleep(0.2) + out.append(held[0].returncode is not None) + class Failing: + async def ensure_system_dependencies(self, env, packages): raise RuntimeError("mirror down") + d.installs, d.repaired = [{{"packages": ["python3"]}}], True + d._installer = lambda name, version=None: Failing() + await d.setup(SimpleNamespace()) + out.append("python3" not in d.installed) + class Result: + verifier_result = None + def model_dump_json(self): + return json.dumps({{"verifier_result": {{"rewards": {{"reward": 1.0}}}}, "exception_info": None}}) + async def raising(): raise OSError(28, "No space left on device") + d.run_task = asyncio.ensure_future(raising()); await asyncio.sleep(0) + base = Path({str(tmp_path)!r}) + paths = SimpleNamespace(trial_dir=base / "t", result_path=base / "none", agent_dir=base / "t" / "agent", + verifier_dir=Path({str(tmp_path)!r}) / "v") + paths.trial_dir.mkdir() + d.trial = SimpleNamespace(paths=paths, result=Result(), task=SimpleNamespace(config=None)) + drv.resolve_task_verifier_mode = lambda config: SimpleNamespace(value="shared") + verdict = await d.done_op({{}}) + out.append(verdict["rewards"]) + async def refuse(config): raise ValueError("Windows containers are not supported by e2b") + drv.Trial = SimpleNamespace(create=refuse) + e = drv.Driver(Out()) + try: + await e.start({{"task_dir": "/x", "trials_dir": "/y", "trial_name": "z"}}) + except drv.Failed as exc: + out.append(exc.kind) + print("RESULT", out) + asyncio.run(main()) + """)) + done = subprocess.run([str(HARBOR_PYTHON), "-I", str(script)], capture_output=True, text=True, timeout=120) + line = next((x for x in done.stdout.splitlines() if x.startswith("RESULT")), done.stdout + done.stderr) + assert line == "RESULT " + str([True, True, True, True, {"reward": 1.0}, "UnsupportedTask"]), line + + +def test_round_6_hermit_side(tmp_path, monkeypatch): + """A verifier that failed after the agent's timeout is read by its infra policy; a disabled verifier is refused; + the claude-code alias installs Claude Code; an adapter whose exports do not parse still declares its CLI (the + deployment scores it); a deployment Harbor's phase end cut short is not the adapter's failure.""" + from hermit.agent.acting.workspace import CliAgent, installs + harbor_settings = {"tasks_dir": str(tmp_path), "harbor": sys.executable} + envs = HarborEnvironments({**harbor_settings, "verifier_timeout": "infra"}, base=tmp_path) + verdict = envs.verdict({"rewards": {"reward": 0.0}, "exception": {"type": "AgentTimeoutError", "message": "t"}, + "later": [{"type": "VerifierTimeoutError", "message": "slow"}]}) + assert verdict["status"] == "infra" + kept = envs.verdict({"rewards": {"reward": 0.0}, "exception": {"type": "AgentTimeoutError", "message": "t"}, + "later": []}) + assert kept["status"] == "timeout" + with pytest.raises(ValueError, match="verifier.disable"): + HarborEnvironments({**harbor_settings, "verifier": {"disable": True}}, base=tmp_path) + assert installs({"backend": "claude-code"}, None)[0]["agent"] == "claude-code" + bad = tmp_path / "bad" + (bad / ".hermit").mkdir(parents=True) + (bad / ".hermit" / "adapter.yaml").write_text("template: {id: bad, version: '1'}\nstructure: {}\n") + (bad / "exports.yaml").write_text("context: [unclosed\n") + assert installs({"backend": "codex"}, bad)[0]["agent"] == "codex" + monkeypatch.setenv("HERMIT_HOMES", str(tmp_path / "homes")) + + class Cut(Recorded): + def upload(self, source, target): + raise RuntimeError("AgentPhaseOver: upload: the agent's phase ended") + + with pytest.raises(RuntimeError, match="AgentPhaseOver"): + CliAgent({"backend": "codex", "web_search": False, "bin": "codex", "workspaces": str(tmp_path / "ws")}, + tmp_path).run({"id": "t", "instruction": "x"}, _prompt_adapter(tmp_path, "be careful\n"), + environment=Cut(tmp_path, lambda c: ExecResult(1, "") if c[:2] == ["test", "-e"] + else ExecResult(0, ""))) + + +def test_the_answer_agent_publishes_answer_json_in_its_log(tmp_path, monkeypatch): + from hermit.agent.acting import answer as answer_mod + from hermit.agent.model.usage import Usage + + class Model: + model = "m" + + def chat(self, messages, **kwargs): + return {"role": "assistant", "content": "B"}, Usage(model="m", input_tokens=3, output_tokens=1, calls=1) + + monkeypatch.setattr(answer_mod.ChatClient, "from_spec", staticmethod(lambda spec: Model())) + env = Recorded(tmp_path) + env.instruction = "Which?" + out = answer_mod.AnswerAgent({"model": {"base_url": "https://x", "model": "m"}, "logs": str(tmp_path / "l"), + "answer_file": None}, tmp_path).run({"id": "q"}, None, environment=env) + published = json.loads((Path(out.log) / "answer.json").read_text()) + assert Path(out.log).is_dir() and published["answer"] == "B" and published["reply"] == "B" + + +def test_eval_checks_the_sign_in_before_it_replaces_results(tmp_path, monkeypatch): + """The runner's check_agent: a Harbor eval on a Claude login with no token refuses before results.jsonl is + opened (replaced) or any trial starts.""" + import yaml + from hermit import evaluate as evaluate_mod + from hermit.agent.base import login + from hermit.config import Config + folder = Path(__file__).resolve().parents[2] / "examples" / "terminal_bench_2_0" / "baseline" + run = yaml.safe_load((folder / "claude.yaml").read_text()) # the migrated config, its data the test's + (tmp_path / "tasks" / "t1").mkdir(parents=True) + (tmp_path / "tasks.jsonl").write_text('{"id": "t1#1", "harbor_task": "t1"}\n') + (tmp_path / "split.txt").write_text("t1#1\n") + (tmp_path / "info").mkdir() + (tmp_path / "info" / "README.md").write_text("A task set of the test's.\n") + run["data"] = {"tasks": str(tmp_path / "tasks.jsonl"), "train": str(tmp_path / "split.txt"), + "test": str(tmp_path / "split.txt")} + run["task_info"] = str(tmp_path / "info") + run["environment"]["harbor"] = {"tasks_dir": str(tmp_path / "tasks")} + run["trainer"] = {**run.get("trainer", {}), "output_dir": str(tmp_path / "runs")} + (tmp_path / "run.yaml").write_text(yaml.safe_dump(run)) + config = Config.load(tmp_path / "run.yaml", []) + monkeypatch.delenv(login.TOKEN_ENV, raising=False) + monkeypatch.setattr(evaluate_mod, "agent_pipeline", lambda *a, **k: (None, lambda: None)) + monkeypatch.setattr(evaluate_mod, "judge_for", lambda *a, **k: None) + monkeypatch.setattr(evaluate_mod, "task_environments", lambda *a, **k: object()) + with pytest.raises(SystemExit, match="eval refuses to start"): + evaluate_mod.evaluate(config, "none", log=lambda m: None) + assert not list((tmp_path / "runs").rglob("results.jsonl")) + + +@pytest.mark.skipif(not HARBOR_PYTHON.exists(), reason="needs Harbor's venv (~/.cache/hermit/harbor-venv)") +def test_a_mounted_codex_serves_only_as_the_asked_version_and_gets_rg_and_ca_roots(tmp_path): + """The runner's _use_mounted and _after_mount, in the bridge's setup: a mounted binary that answers another + version is not used (the CLI is installed instead); one that serves is linked, with the release's rg and this + machine's CA bundle for an image that has none.""" + script = tmp_path / "probe.py" + script.write_text(textwrap.dedent(f""" + import asyncio, importlib.util, sys + from types import SimpleNamespace + spec = importlib.util.spec_from_file_location("drv", {str(DRIVER_FILE)!r}) + drv = importlib.util.module_from_spec(spec); sys.modules["drv"] = drv; spec.loader.exec_module(drv) + + class Env: + def __init__(self, version): self.version, self.calls, self.uploads = version, [], [] + async def exec(self, command, user=None, **kw): + self.calls.append(command) + if "--version" in command: + return SimpleNamespace(return_code=0, stdout=f"hermit-version-begin\\ncodex-cli {{self.version}}" + f"\\nhermit-version-end\\n", stderr="") + if command.startswith("[ -s"): # the image has no CA bundle + return SimpleNamespace(return_code=1, stdout="", stderr="") + return SimpleNamespace(return_code=0, stdout="", stderr="") + async def upload_file(self, source, target): self.uploads.append(target) + + class Installer: + def parse_version(self, text): return text.split()[-1] + + async def main(): + d = drv.Driver(None) + d._installer = lambda name, version=None: Installer() + item = {{"agent": "codex", "version": "0.162.0", "bin": "/opt/hermit-codex/codex", "program": "codex"}} + old = Env("0.170.0") + served_old = await d._use_mounted(old, item) + new = Env("0.162.0") + served_new = await d._use_mounted(new, item) + print("RESULT", [served_old, served_new, any("codex-path/rg" in c for c in new.calls), + new.uploads == ["/etc/ssl/certs/ca-certificates.crt"], d.installed.get("codex")]) + asyncio.run(main()) + """)) + done = subprocess.run([str(HARBOR_PYTHON), "-I", str(script)], capture_output=True, text=True, timeout=120) + line = next((x for x in done.stdout.splitlines() if x.startswith("RESULT")), done.stdout + done.stderr) + assert line == "RESULT " + str([False, True, True, True, "0.162.0"]), line + + +def test_the_cli_runs_the_installed_program_when_the_mounted_one_did_not_serve_and_claude_gets_no_raw_key( + tmp_path, monkeypatch): + from hermit.agent.base.interface import make + monkeypatch.setenv("HERMIT_HOMES", str(tmp_path / "homes")) + env = Recorded(tmp_path, lambda c: ExecResult(0, CODEX_STREAM) if session_of(c) else ExecResult(0, "/usr/bin/x")) + env.bin_served = {"codex": False} + make("codex", {"web_search": False, "bin": "/opt/hermit-codex/codex"}).run("x", environment=env) + session = next(c for c in env.calls if session_of(c["command"])) + assert session["command"][4] == "codex" # the installed program, not the mount + monkeypatch.setenv("FAKE_KEY", "the-key") + binding = {"ref": "fake/m", "backend": "fake", "model": "m", "base_url": "https://models.example/v1", + "api_key_env": "FAKE_KEY", "key_required": True, "apis": ["messages"]} + stream = json.dumps({"type": "result", "subtype": "success", "is_error": False, "result": "done", + "usage": {"input_tokens": 1, "output_tokens": 1}}) + "\n" + claude_env = Recorded(tmp_path, lambda c: ExecResult(0, stream) if session_of(c) else ExecResult(0, "/x")) + make("claude", {"web_search": False, "binding": binding, "model": "m", "bin": "claude"}).run( + "hi", environment=claude_env) + session = next(c for c in claude_env.calls if session_of(c["command"])) + assert "FAKE_KEY" not in session["env"] # its key in its own variable (session_env) + + +def test_round_7_hermit_side(harbor, tmp_path, monkeypatch): + """The driver never sees an ambient credential the run dropped; a stop Harbor acknowledged with its projects + still there takes them down by name; every command's time comes off the phase's absolute deadline; the start's + and a separate verifier's watchdogs follow Harbor's own budgets; eval checks the acting agent's sign-in only.""" + import time + environments, root, calls = harbor + seen = {} + original = harbor_mod._Driver + + def capture(python, log_path, env): + seen.update(env) + return original(python, log_path, env) + + monkeypatch.setattr(harbor_mod, "_Driver", capture) + monkeypatch.setenv("CLAUDE_CODE_OAUTH_TOKEN", "ambient-token") + environments.drop_env = {"CLAUDE_CODE_OAUTH_TOKEN"} + monkeypatch.setattr(harbor_mod.docker, "compose_projects", lambda prefix: [prefix + "__env"]) + task = environments.task_dir({"harbor_task": "demo"}) + with environments.session(task) as env: + env.deadline_at = time.time() + 5 + env.exec(["true"], timeout=100) + assert "CLAUDE_CODE_OAUTH_TOKEN" not in seen + sent = [json.loads(line) for line in (tmp_path / "requests.jsonl").read_text().splitlines()] + assert next(r for r in sent if r["op"] == "exec")["timeout_sec"] <= 5 + assert calls["down"] # Harbor's down failed: by name + (task / "task.toml").write_text('[environment]\ndocker_image = "demo:1"\nbuild_timeout_sec = 1800\n') + scaled = HarborEnvironments({"tasks_dir": str(tmp_path / "tasks"), "harbor": sys.executable, + "environment_build_timeout_multiplier": 3}, base=tmp_path) + assert scaled.start_timeout(task) == 1800 * 3 + harbor_mod.SETUP_TIMEOUT_S + harbor_mod.START_ALLOWANCE_S + from hermit import evaluate as evaluate_mod + from hermit.agent.base import login + monkeypatch.setattr(login, "logins_of", lambda d: [("llm", "claude", None, False, None), + ("agent", "chatgpt", None, True, None)]) + monkeypatch.setattr(login, "problems_of", lambda *a: [f"no {a[0]}"]) + assert evaluate_mod.sign_in_problems({}) == ["agent: no chatgpt"] # never the learner's + + +def test_a_separate_verifiers_watchdog_includes_its_start_and_a_timed_out_rollout_keeps_its_stream(harbor, tmp_path): + from hermit.task import Task + from hermit.trainer import rollout + environments, _, _ = harbor + asked = [] + with environments.session(environments.task_dir({"harbor_task": "demo"})) as env: + env.started.update(verifier_timeout_sec=60.0, verifier_mode="separate", build_timeout_sec=2700.0) + original = env.driver.request + env.driver.request = lambda op, timeout, **f: asked.append((op, timeout)) or original(op, timeout, **f) + env.verify() + assert ("done", 60.0 + 2700.0 + harbor_mod.VERIFY_ALLOWANCE_S) in asked + kept = tmp_path / "trial-agent" + kept.mkdir() + (kept / "agent-codex.jsonl").write_text(json.dumps({"type": "turn.completed", "usage": { + "input_tokens": 1000, "cached_input_tokens": 0, "output_tokens": 200}}) + "\n") + timed = Environments(tmp_path, verdicts=[{"reward": 0.0, "rewards": {"reward": 0.0}, "status": "timeout", + "failure_cause": "agent_exhausted_budget", "passed": False, + "feedback": "", "agent_dir": str(kept)}]) + + def phase_over(task, adapter, environment=None): + raise RuntimeError("AgentPhaseOver: exec: the agent's phase ended") + + task = Task(id="demo#1", public={"id": "demo#1", "harbor_task": "demo"}) + from hermit.agent.acting.workspace import CliAgent + timed.usage_of = CliAgent({"backend": "codex", "model": "m", "web_search": False, + "price": {"input": 1.0, "output": 2.0}}, + tmp_path).usage_of + out = rollout.environment_rollout(task, task.given(), None, phase_over, timed, 0.0, pause=lambda s: None) + assert Path(out.log) == kept # the stream the bridge kept, for the learner + # what the CLI had reported before Harbor's timeout cut it off is counted (the runner parsed the kept stream) + assert out.tokens == 1200 and out.api_speed["model_calls"] == 1 and out.cost_usd == pytest.approx(0.0014) + + +def test_a_pipeline_error_on_a_run_again_keeps_what_the_runs_before_it_spent(tmp_path): + from hermit.task import Task + from hermit.trainer import rollout + runs = [] + + def pipeline(task, adapter, environment=None): + runs.append(1) + if len(runs) == 1: + return RunOutput(output=None, tokens=2200, cost_usd=0.1, api_speed={"model_calls": 1}, + extra={"infra": "model_provider_failed", "infra_final": False, "exception_message": "x"}) + raise KeyError("a bug of the agent's own") + + task = Task(id="demo#1", public={"id": "demo#1", "harbor_task": "demo"}) + record = rollout.environment_rollout(task, task.given(), None, pipeline, Environments(tmp_path), 0.0, + pause=lambda s: None) + assert record["status"] == "error" and record["tokens"] == 2200 and record["cost_usd"] is None + assert record["api_speed"]["model_calls"] == 1 and record["pipeline_extra"]["attempts"] + + +def test_an_attachment_over_the_limit_is_measured_there_and_never_read(tmp_path, monkeypatch): + from hermit.agent.acting import answer as answer_mod + from hermit.agent.model.usage import Usage + asked = [] + + class Model: + model = "m" + + def chat(self, messages, **kwargs): + asked.append(messages) + return {"role": "assistant", "content": "B"}, Usage(model="m", calls=1) + + monkeypatch.setattr(answer_mod.ChatClient, "from_spec", staticmethod(lambda spec: Model())) + + def answer(c): + if "wc -c" in c[-1]: + return ExecResult(0, f"{answer_mod.MAX_ATTACHMENT_BYTES + 1}\n") + if "base64" in c[-1]: + raise AssertionError("an image over the limit is never read") + return ExecResult(0, "/app/big.png\n") + + env = Recorded(tmp_path, answer) + env.instruction = "Which?" + answer_mod.AnswerAgent({"model": {"base_url": "https://x", "model": "m"}, "logs": str(tmp_path / "l"), + "attachments": ["/app/*.png"]}, tmp_path).run({"id": "q"}, None, environment=env) + assert isinstance(asked[0][-1]["content"], str) # sent without it + + +def test_round_8_hermit_side(harbor, tmp_path, monkeypatch): + environments, root, calls = harbor + # 6: the binding's key variable, whatever its name, reaches the driver as a credential to scrub + with environments.session(environments.task_dir({"harbor_task": "demo"})) as env: + env.exec(["true"], env={"LLM_BEARER": "a-credential-of-any-name"}, secret_env=["LLM_BEARER"]) + sent = [json.loads(line) for line in (tmp_path / "requests.jsonl").read_text().splitlines()] + assert next(r for r in sent if r["op"] == "exec")["secret_env"] == ["LLM_BEARER"] + # 11: the deadline is the phase's as the bridge saw it begin + monkeypatch.setenv("FAKE_ELAPSED", "100") + import time + with environments.session(environments.task_dir({"harbor_task": "demo"})) as env: + assert abs(env.deadline_at - (time.time() + 900 - 100 - 60)) < 5 + # 15: Harbor told to keep its containers: never taken down by name + kept = HarborEnvironments({"tasks_dir": str(tmp_path / "tasks"), "harbor": sys.executable, + "environment": {"kwargs": {"keep_containers": True}}}, base=tmp_path) + assert kept.local_docker is False + # 7: the exception's own message redacted too + redacting = HarborEnvironments({"tasks_dir": str(tmp_path / "tasks"), "harbor": sys.executable, + "verifier_redact": [r"expected: \d+"]}, base=tmp_path) + out = tmp_path / "v" + out.mkdir() + verdict = redacting.verdict({"rewards": None, "verifier_dir": str(out), + "exception": {"type": "ValueError", "message": "bad reward, expected: 42"}}) + assert "42" not in verdict["exception"]["message"] and "42" not in verdict["exception_message"] + + +def test_round_8_agents(tmp_path, monkeypatch): + from hermit.agent.acting import answer as answer_mod + from hermit.agent.model.usage import Usage + + class Model: + model = "m" + + def chat(self, messages, **kwargs): + return {"role": "assistant", "content": "B"}, Usage(model="m", input_tokens=12, output_tokens=5, calls=1, + cost_usd=0.1) + + monkeypatch.setattr(answer_mod.ChatClient, "from_spec", staticmethod(lambda spec: Model())) + + class Failing(Recorded): + def upload(self, source, target): + raise RuntimeError("EnvironmentUnavailable: upload failed in the task's environment") + + env = Failing(tmp_path) + env.instruction = "Which?" + out = answer_mod.AnswerAgent({"model": {"base_url": "https://x", "model": "m"}, "logs": str(tmp_path / "l")}, + tmp_path).run({"id": "q"}, None, environment=env) + assert (out.tokens, out.cost_usd, out.extra["infra"]) == (17, 0.1, "harbor_failed") # its spend kept + # 13: an image whose bytes do not come back whole is the environment's failure, never an empty image sent + broken = Recorded(tmp_path, lambda c: ExecResult(0, "4\n") if "wc -c" in c[-1] else + ExecResult(0, "") if "base64" in c[-1] else ExecResult(0, "/app/a.png\n")) + broken.instruction = "Which?" + with pytest.raises(RuntimeError, match="did not provide the attachment"): + answer_mod.AnswerAgent({"model": {"base_url": "https://x", "model": "m"}, "logs": str(tmp_path / "l"), + "attachments": ["/app/*.png"]}, tmp_path).run({"id": "q"}, None, environment=broken) + # 14: a sign-in larger than the limit there is never brought here + from hermit.agent.base.cli import CliAgent as Base + downloads = [] + + class Big(Recorded): + def download(self, source, target): + downloads.append(source) + raise AssertionError("never downloaded") + + Base._signin_back(Big(tmp_path, lambda c: ExecResult(0, "10000000\n")), PurePosixPath("/h"), tmp_path) + assert downloads == [] + # 16: a CLI's model on this machine's loopback, which the task's container cannot reach, is refused + from types import SimpleNamespace + from hermit import isolate + binding = {"ref": "local/m", "backend": "local", "model": "m", "base_url": "http://127.0.0.1:8000/v1", + "apis": ["responses"]} + config = SimpleNamespace(base=tmp_path, data={"environment": {"harbor": {"tasks_dir": str(tmp_path)}}, + "agent": {"name": "codex", "params": {"binding": binding}}}) + with pytest.raises(ValueError, match="loopback"): + isolate.task_environments(config) + + +def test_round_9_hermit_side(harbor, tmp_path, monkeypatch): + from hermit.agent.acting import answer as answer_mod + from hermit.agent.model.usage import Usage + + class Model: + model = "m" + + def chat(self, messages, **kwargs): + return {"role": "assistant", "content": "B"}, Usage(model="m", input_tokens=12, output_tokens=5, calls=1, + cost_usd=0.1) + + monkeypatch.setattr(answer_mod.ChatClient, "from_spec", staticmethod(lambda spec: Model())) + + class Closed(Recorded): + def upload(self, source, target): + raise RuntimeError("AgentPhaseOver: upload: the agent's phase ended") + + env = Closed(tmp_path) + env.instruction = "Which?" + out = answer_mod.AnswerAgent({"model": {"base_url": "https://x", "model": "m"}, "logs": str(tmp_path / "l")}, + tmp_path).run({"id": "q"}, None, environment=env) + assert out.extra["timed_out"] is True and (out.tokens, out.cost_usd) == (17, 0.1) # Harbor's verdict, its spend + environments, root, calls = harbor + kept = HarborEnvironments({"tasks_dir": str(tmp_path / "tasks"), "harbor": sys.executable, + "jobs_dir": str(tmp_path / "jobs"), + "environment": {"kwargs": {"keep_containers": True}}}, base=tmp_path) + assert kept.keep_containers and not kept.local_docker + from types import SimpleNamespace + from hermit import isolate + binding = {"ref": "local/m", "backend": "local", "model": "m", "base_url": "http://127.0.0.1:8000/v1", + "apis": ["responses"]} + config = SimpleNamespace(base=tmp_path, data={"environment": {"harbor": {"tasks_dir": str(tmp_path)}}, + "agent": {"name": "cli", "params": {"binding": binding}}}) + with pytest.raises(ValueError, match="loopback"): # `cli` is Codex by default + isolate.task_environments(config) + + + +# ---- the simple end of the agent phase ------------------------------------------------------------------------- + +def _kill_program(shell: str, mark: str, *, stuck: bool = False) -> str: + """A PID 1 of its own (Python, which never reaps an orphan, as a container's `sleep infinity` does not): an agent + command's group started under setsid as the bridge starts it, then kill_groups run by `shell` (a stuck kill first + when `stuck`), then the count of that group's processes left alive (a zombie is not).""" + pidfile = f"/tmp/hermit-pid-{mark}" + kill = _driver_function("kill_groups", TOKEN="hermit-mark-", PIDS="/tmp/hermit-pid-")(pidfile) + scenario = (f'setsid bash -c "sleep 60 & sleep 60; exit" hermit-mark-{mark} >/dev/null 2>&1 & echo $! > {pidfile}; ' + "echo leader=$!; sleep 0.5; ") + if stuck: # a group the kill does not end + scenario += (f"{shell} -c {shlex.quote(kill.replace('kill -KILL', ': kill -KILL'))}; echo stuck=$?; " + f"[ -f {pidfile} ] && echo kept; ") + scenario += f"{shell} -c {shlex.quote(kill)}; echo killed=$?; [ -f {pidfile} ] || echo removed" + return textwrap.dedent(f""" + import os, subprocess + lines = subprocess.run(["sh", "-c", {scenario!r}], capture_output=True, text=True).stdout.split() + leader = next(x for x in lines if x.startswith("leader="))[len("leader="):] + left = 0 + for pid in filter(str.isdigit, os.listdir("/proc")): + try: + state, _, group = open(f"/proc/{{pid}}/stat").read().rsplit(") ", 1)[1].split()[:3] + except (OSError, ValueError): + continue + left += group == leader and state not in "ZX" + print(*[x for x in lines if not x.startswith("leader=")], f"left={{left}}") + """) + + +@pytest.mark.skipif(not _own_pid_namespace(), reason="needs unshare (a PID namespace of its own, as a container's)") +@pytest.mark.parametrize("shell", ["sh", "bash"]) +def test_a_commands_process_group_is_killed_from_its_pid_file_and_checked_gone(tmp_path, shell): + """kill_groups, as the runner's _stop_session: the group of the command whose leader's pid file names it (its + word on its command line confirming it), what it started included, then one check that none is left; under a + PID 1 that never reaps (its zombies are gone) and in POSIX sh (dash on Debian and Ubuntu) as in bash.""" + mark = "m" + os.urandom(4).hex() + kill = _driver_function("kill_groups", TOKEN="hermit-mark-", PIDS="/tmp/hermit-pid-") + done = subprocess.run(["unshare", "--user", "--map-root-user", "--pid", "--fork", "--mount-proc", sys.executable, + "-c", _kill_program(shell, mark)], capture_output=True, text=True, timeout=60) + assert done.stdout.split() == ["killed=0", "removed", "left=0"], done.stdout + done.stderr + assert not os.path.exists(f"/tmp/hermit-pid-{mark}") + fake = tmp_path / "fake" # a pid file naming a process without the word: left alone + keep = subprocess.Popen(["sleep", "30"]) + try: + fake_file = f"/tmp/hermit-pid-{mark}x" + Path(fake_file).write_text(str(keep.pid)) + subprocess.run(["sh", "-c", kill(fake_file)], capture_output=True, timeout=30) + assert keep.poll() is None + finally: + keep.kill() + fake.touch() + + +@pytest.mark.skipif(not HARBOR_PYTHON.exists(), reason="needs Harbor's venv (~/.cache/hermit/harbor-venv)") +def test_the_bridge_ends_its_phase_at_the_runners_level(tmp_path): + """The bridge's run: the phase starts with the instruction; a relayed command is refused outside it; at `done` + and at Harbor's timeout the groups are killed (one root exec), the registered credentials removed, Hermit's logs + copied scrubbed; a kill or a removal that fails is HarborDriverError (no verifier); a cancelled client's whole + group is killed (docker's compose plugin with it).""" + script = tmp_path / "probe.py" + logs = tmp_path / "logs" + logs.mkdir() + (logs / "agent-codex.jsonl").write_text(f"key {AGENT_KEY}\n") + script.write_text(textwrap.dedent(f""" + import asyncio, importlib.util, sys + from pathlib import Path + from types import SimpleNamespace + spec = importlib.util.spec_from_file_location("drv", {str(DRIVER_FILE)!r}) + drv = importlib.util.module_from_spec(spec); sys.modules["drv"] = drv; spec.loader.exec_module(drv) + drv.resolve_task_verifier_mode = lambda config: SimpleNamespace(value="shared") + + class Env: + def __init__(self, kill_ok=True, rm_ok=True): self.calls, self.kill_ok, self.rm_ok = [], kill_ok, rm_ok + async def exec(self, command, cwd=None, user=None, **kw): + self.calls.append((command, user)) + if command.startswith("printf"): + return SimpleNamespace(return_code=0, stdout="hermit-workdir:/app\\n", stderr="") + if "hermit-owner:" in command: + return SimpleNamespace(return_code=0, stdout="hermit-owner:0:0\\n", stderr="") + ok = self.kill_ok if "hermit-pid-" in command else self.rm_ok if command.startswith("rm -f") else True + if ok == "raise": + raise ConnectionError("docker's transport") + return SimpleNamespace(return_code=0 if ok else 1, stdout="", stderr="") + + def trial(tmp): + plan = SimpleNamespace(agent_phase=SimpleNamespace(network_mode="no-network")) + paths = SimpleNamespace(trial_dir=Path(tmp), agent_dir=Path(tmp) / "agent") + config = SimpleNamespace(agent=SimpleNamespace(user=None), environment=SimpleNamespace( + os=SimpleNamespace(value="linux"))) + return SimpleNamespace(task=SimpleNamespace(config=config), paths=paths, agent_timeout_sec=60, + agent_setup_timeout_sec=60, verifier_timeout_sec=60, + environment_build_timeout_sec=600, _network_plan=lambda step: plan) + + class Out: + def write(self, s): pass + def flush(self): pass + + async def main(): + out = [] + for how, kill_ok, rm_ok in (("done", True, True), ("timeout", True, True), ("kill fails", False, True), + ("rm fails", True, False), ("kill raises", "raise", True)): + d = drv.DRIVER = drv.Driver(Out()) + d.trial = trial({str(tmp_path)!r} + "/" + how.replace(" ", "-")) + env = Env(kill_ok, rm_ok) + await d.handle({{"id": 1, "op": "exec", "command": "true"}}) # before the phase: refused + run = asyncio.create_task(d.phase("Do it.", env)) + started = await d.phase_started + d.secret_files.add("/installed-agent/hermit-home/auth.json") + try: + if how == "timeout": + run.cancel() + r = await asyncio.gather(run, return_exceptions=True) + if isinstance(r[0], drv.HarborDriverError): + raise r[0] + else: + d.agent_logs, d.secrets = {str(logs)!r}, {{{AGENT_KEY!r}}} + d.done.set() + await run + ended = "returned" + except drv.HarborDriverError: + ended = "raised" + killed = [c for c, u in env.calls if "hermit-pid-" in c and u == "root"] + removed = [c for c, u in env.calls if c.startswith("rm -f") and "auth.json" in c] + kept = Path({str(tmp_path)!r}) / how.replace(" ", "-") / "agent" / "agent-codex.jsonl" + out.append([how, started["instruction"], ended, len(killed), len(removed), + kept.read_text() if kept.exists() else None]) + held = [] + async def waits(): + p = await asyncio.create_subprocess_exec("sh", "-c", "sleep 30 & echo $!; wait", + stdout=asyncio.subprocess.PIPE) + held.append(p) + await p.communicate() + t = asyncio.create_task(waits()); await asyncio.sleep(0.5); t.cancel() + await asyncio.gather(t, return_exceptions=True) + await asyncio.sleep(0.3) + out.append(not drv.group_alive(held[0])) + print("RESULT", out) + asyncio.run(main()) + """)) + done = subprocess.run([str(HARBOR_PYTHON), "-I", str(script)], capture_output=True, text=True, timeout=120) + line = next((x for x in done.stdout.splitlines() if x.startswith("RESULT")), done.stdout + done.stderr) + assert line == "RESULT " + str([["done", "Do it.", "returned", 1, 1, "key [REDACTED]\n"], + ["timeout", "Do it.", "returned", 1, 1, None], + ["kill fails", "Do it.", "raised", 1, 1, None], + ["rm fails", "Do it.", "raised", 1, 1, None], + ["kill raises", "Do it.", "raised", 1, 1, None], True]), line + + +def test_a_harbor_task_sets_agent_runs_in_this_process(tmp_path): + """No worker and no proxy for a task in Harbor's environment: its commands and CLIs run in the task's own + container, the agent's loop here (host_pipeline).""" + from types import SimpleNamespace + from hermit.isolate import host_pipeline + config = SimpleNamespace(data={"sandbox": {"enabled": True}, "environment": {"harbor": {"tasks_dir": "x"}}}) + assert "environment.harbor" in host_pipeline(config) + + +def test_the_agents_setup_defaults_to_twenty_minutes_and_a_setting_wins(harbor, tmp_path): + environments, _, _ = harbor + task = environments.task_dir({"harbor_task": "demo"}) + assert environments.start_request(task, tmp_path, "t")["agent"]["override_setup_timeout_sec"] == 1200.0 + assert environments.start_timeout(task) >= 600 + 1200 + harbor_mod.START_ALLOWANCE_S # the watchdog follows + own = HarborEnvironments({"tasks_dir": str(tmp_path / "tasks"), "harbor": sys.executable, + "setup_timeout_sec": 300}, base=tmp_path) + assert own.start_request(task, tmp_path, "t")["agent"]["override_setup_timeout_sec"] == 300.0 + + +@pytest.mark.skipif(not HARBOR_PYTHON.exists(), reason="needs Harbor's venv (~/.cache/hermit/harbor-venv)") +def test_any_failure_of_the_phases_end_is_infra(tmp_path): + """A failure of the phase's end of any kind (its bound, a log folder it cannot write) is HarborDriverError: no + verifier, infra, never the agent's zero.""" + script = tmp_path / "probe.py" + script.write_text(textwrap.dedent(f""" + import asyncio, importlib.util, sys + spec = importlib.util.spec_from_file_location("drv", {str(DRIVER_FILE)!r}) + drv = importlib.util.module_from_spec(spec); sys.modules["drv"] = drv; spec.loader.exec_module(drv) + + async def main(): + out = [] + for failing in ("hangs", "denied"): + d = drv.Driver(None) + async def end(how=failing): + if how == "hangs": + await asyncio.sleep(60) + raise PermissionError("the agent logs' folder") + d._end_phase = end + drv.END_PHASE_S = 0.2 + try: + await d._close_phase(False); out.append("returned") + except drv.HarborDriverError: + out.append("infra") + print("RESULT", out) + asyncio.run(main()) + """)) + done = subprocess.run([str(HARBOR_PYTHON), "-I", str(script)], capture_output=True, text=True, timeout=60) + line = next((x for x in done.stdout.splitlines() if x.startswith("RESULT")), done.stdout + done.stderr) + assert line == "RESULT " + str(["infra", "infra"]), line + + +def test_a_harbor_agents_adapter_code_runs_in_the_strict_adapter_sandbox(tmp_path, monkeypatch): + """No worker for a Harbor task set's agent: its call in this process still confines the adapter's own code as the + worker did (the runner's adapter sandbox): a context tool's Session is sandboxed, its exports' render goes through + the strict sandbox; outside the call nothing is.""" + from types import SimpleNamespace + from hermit import Session + from hermit import isolate + from hermit.hosts import inputs + monkeypatch.delenv("HA_ADAPTER_SANDBOX", raising=False) + adapter = tmp_path / "adapter" + shutil.copytree(Path(__file__).resolve().parents[2] / "hermit/adapter/templates/prompt_only", adapter) + rendered = [] + monkeypatch.setattr(inputs, "run_in_strict_sandbox", lambda command, **kw: rendered.append(("strict", kw)) or + SimpleNamespace(returncode=0, stdout="{}", stderr="")) + monkeypatch.setattr(inputs.subprocess, "run", lambda command, **kw: rendered.append(("host", kw)) or + SimpleNamespace(returncode=0, stdout="{}", stderr="")) + + def run(task, adapter_dir, environment=None): + inputs.render_exports(adapter_dir, sys.executable) + return Session(adapter_dir).sandboxed + + config = SimpleNamespace(data={"sandbox": {"enabled": True, "enforce": "best_effort", "hide": []}, + "environment": {"harbor": {"tasks_dir": "x"}}, + "agent": {"name": "codex", "params": {}}}, base=tmp_path) + monkeypatch.setattr("hermit.agent.acting.agent.load_agent", + lambda name, params, base: SimpleNamespace(run=run)) + pipeline, close = isolate.agent_pipeline(config, "rollout", records=None) + assert pipeline({"id": "q"}, adapter) is True + assert rendered == [("strict", {"role": "render", "readable": [adapter], "enforce": "best_effort"})] + assert Session(adapter).sandboxed is False # outside the call: as before + close() + + +def test_a_harbor_cli_signs_in_by_its_binding_or_a_named_login_only(tmp_path, monkeypatch): + """The runner's refusal: a CLI in a task's container without a model binding or a login is refused before any + trial; and a session in a remote environment never carries this machine's own CLI sign-in there.""" + from types import SimpleNamespace + from hermit import isolate + from hermit.agent.base import login + from hermit.agent.base.interface import make + (tmp_path / "tasks").mkdir() + data = {"data": {"tasks": None}, "judge": {}, "environment": {"harbor": {"tasks_dir": "tasks", + "harbor": sys.executable}}, + "agent": {"name": "codex", "params": {"model": "gpt-5.6-luna", "web_search": False}}} + config = SimpleNamespace(data=data, base=tmp_path) + with pytest.raises(ValueError, match="needs a model binding"): + isolate.task_environments(config) + own = tmp_path / "user-codex" + own.mkdir() + (own / "auth.json").write_text('{"tokens": "mine"}') + monkeypatch.setattr(login, "source", lambda: own / "auth.json") + env = Recorded(tmp_path, lambda c: ExecResult(0, CODEX_STREAM) if session_of(c) else ExecResult(0, "")) + out = make("codex", {"web_search": False, "model": "m", "bin": "codex", + "home": str(tmp_path / "kept")}).run("x", environment=env) + assert out.reply == "fixed it" and env.uploads + assert not any("auth.json" in (files or {}) for files in env.uploads.values()) + + +def test_the_log_a_record_names_is_scrubbed_of_the_runs_keys(tmp_path, monkeypatch): + """An agent in a task's own environment runs in this process: the log its record names (results.jsonl points at + it) is scrubbed of the run's key values, as the worker scrubbed what it returned.""" + from hermit.task import Task + from hermit.trainer import rollout + monkeypatch.setenv("FIREWORKS_API_KEY", AGENT_KEY) + log = tmp_path / "agent-codex.jsonl" + log.write_text(f"provider said: bad key {AGENT_KEY}\n") + + def pipeline(task, adapter, environment=None): + return RunOutput(output="done", log=log) + + task = Task(id="demo#1", public={"id": "demo#1", "harbor_task": "demo"}) + rollout.environment_rollout(task, task.given(), None, pipeline, Environments(tmp_path), 0.0, pause=lambda s: None) + assert AGENT_KEY not in log.read_text() and "bad key" in log.read_text() + + +@pytest.mark.skipif(not _own_pid_namespace(), reason="needs unshare (a PID namespace of its own, as a container's)") +@pytest.mark.parametrize("shell", ["sh", "bash"]) +def test_a_group_not_confirmed_gone_keeps_its_pid_file_for_the_phases_end(shell): + """kill_groups removes a command's pid file only once its group is confirmed gone: one that survives keeps its + record, so the phase's end tries again and fails the trial (infra) rather than letting the verifier run.""" + mark = "m" + os.urandom(4).hex() + try: + done = subprocess.run(["unshare", "--user", "--map-root-user", "--pid", "--fork", "--mount-proc", + sys.executable, "-c", _kill_program(shell, mark, stuck=True)], + capture_output=True, text=True, timeout=60) + finally: + Path(f"/tmp/hermit-pid-{mark}").unlink(missing_ok=True) + assert done.stdout.split() == ["stuck=1", "kept", "killed=0", "removed", "left=0"], done.stdout + done.stderr + + +@pytest.mark.skipif(not _docker_up() or subprocess.run(["docker", "image", "inspect", "python:3.12-slim"], + capture_output=True).returncode != 0, + reason="needs Docker and python:3.12-slim (Debian: dash is its sh)") +def test_kill_groups_in_a_debian_container_whose_pid_1_never_reaps(): + """The same in a container: Debian's sh (dash, whose kill reads `--` as a pid) and a PID 1 that never reaps.""" + mark = "m" + os.urandom(4).hex() + done = subprocess.run(["docker", "run", "--rm", "--name", f"hermit-test-kill-{mark}", "python:3.12-slim", + "python3", "-c", _kill_program("sh", mark)], capture_output=True, text=True, timeout=180) + assert done.stdout.split() == ["killed=0", "removed", "left=0"], done.stdout + done.stderr + + +@pytest.mark.skipif(not HARBOR_PYTHON.exists(), reason="needs Harbor's venv (~/.cache/hermit/harbor-venv)") +def test_a_download_of_a_link_the_agent_left_is_refused(tmp_path): + """The runner refused a link at the notes' folder: a download whose source is a link (a hook audit or a notes + folder the agent replaced) is never copied out with Harbor's privileged copy.""" + script = tmp_path / "probe.py" + script.write_text(textwrap.dedent(f""" + import asyncio, importlib.util, sys + from types import SimpleNamespace + spec = importlib.util.spec_from_file_location("drv", {str(DRIVER_FILE)!r}) + drv = importlib.util.module_from_spec(spec); sys.modules["drv"] = drv; spec.loader.exec_module(drv) + + class Env: + copied = [] + async def exec(self, command, user=None, **kw): + return SimpleNamespace(return_code=0 if "/app/.hermit-hooks" in command else 1, stdout="", stderr="") + async def is_dir(self, path, user=None): return True + async def download_dir(self, source, target): self.copied.append(source) + async def download_file(self, source, target): self.copied.append(source) + + async def main(): + d = drv.Driver(None) + d.env = Env() + out = [] + for source in ("/app/.hermit-hooks", "/app/notes"): + try: + await d.download({{"source": source, "target": {str(tmp_path / "got")!r} + source}}) + out.append("copied") + except drv.Failed as exc: + out.append(exc.kind) + print("RESULT", out, Env.copied) + asyncio.run(main()) + """)) + done = subprocess.run([str(HARBOR_PYTHON), "-I", str(script)], capture_output=True, text=True, timeout=60) + line = next((x for x in done.stdout.splitlines() if x.startswith("RESULT")), done.stdout + done.stderr) + assert line == "RESULT " + str(["PermissionError", "copied"]) + " " + str(["/app/notes"]), line + + +def test_a_forced_teardown_signals_only_the_drivers_own_group(tmp_path): + """Killing the driver (its watchdog) signals the driver's own process group, never a group recorded earlier whose + pid this busy machine may since have given to another program.""" + from hermit.environment.harbor import _Driver + script = tmp_path / "driver.py" + script.write_text("import time\ntime.sleep(60)\n") + other = subprocess.Popen(["sleep", "60"], start_new_session=True) # not the driver's: left alone + original = harbor_mod.DRIVER + harbor_mod.DRIVER = script + try: + (tmp_path / "client-groups").write_text(f"{other.pid}\n") # what the driver once recorded there + driver = _Driver(sys.executable, tmp_path / "driver.log", dict(os.environ)) + driver._kill() + assert driver.proc.poll() is not None + with pytest.raises(subprocess.TimeoutExpired): # still running + other.wait(timeout=1) + finally: + harbor_mod.DRIVER = original + other.kill() + other.wait() + + +def test_a_sandbox_this_machine_cannot_start_is_infra_never_the_adapters_zero(tmp_path, monkeypatch): + """The runner's preflight and staging_failed: a run whose strict adapter sandbox does not start here is refused at + startup, and a deployment or context that fails because it stopped starting is infra (run again), not the + adapter's scored zero.""" + from types import SimpleNamespace + from hermit import isolate, sandbox + from hermit.adapter.session import CONFINED + from hermit.agent.acting import answer as answer_mod + from hermit.agent.model.usage import Usage + fault = "the strict adapter sandbox did not start (exit 1): no user namespaces" + monkeypatch.setattr(sandbox, "sandbox_fault", lambda enforce, fresh=True: fault) + monkeypatch.setattr(isolate, "sandbox_fault", sandbox.sandbox_fault) + monkeypatch.setattr("hermit.agent.acting.agent.load_agent", + lambda name, params, base: SimpleNamespace(run=lambda *a, **k: None)) + config = SimpleNamespace(data={"sandbox": {"enabled": True, "enforce": "required", "hide": []}, + "environment": {"harbor": {"tasks_dir": "x"}}, + "agent": {"name": "codex", "params": {}}}, base=tmp_path) + with pytest.raises(ValueError, match="did not start"): + isolate.agent_pipeline(config, "rollout", records=None) + + class Model: + model = "m" + + def chat(self, messages, abort=None, **kwargs): + return {"role": "assistant", "content": "B"}, Usage(model="m", calls=1) + + class Failing: + last_context = None + + def context(self, **kw): + raise RuntimeError("the sandbox's server did not start") + + def close(self): + pass + + def server_fails(adapter): # the context tool's server, started in its sandbox, did not start + raise RuntimeError("bwrap: setting up uid map: Permission denied") + + monkeypatch.setattr(answer_mod.ChatClient, "from_spec", staticmethod(lambda spec: Model())) + agent = answer_mod.AnswerAgent({"model": {"base_url": "https://x", "model": "m"}, "logs": str(tmp_path / "l")}, + tmp_path) + env = Recorded(tmp_path) + env.instruction = "Which?" + token = CONFINED.set("required") + try: + for session in (lambda adapter: Failing(), server_fails): + monkeypatch.setattr(agent, "session", session) + out = agent.run({"id": "q"}, tmp_path, environment=env) + assert out.extra["infra"] == "staging_failed" and not out.extra["infra_final"] # run again + assert (out.cost_usd, out.tokens) == (0.0, 0) # nothing spent + assert "adapter_failed" not in out.extra + finally: + CONFINED.reset(token) + def disk_full(adapter): # the server's files not made: this machine's, whatever the sandbox + raise OSError(28, "No space left on device") + + monkeypatch.setattr(agent, "session", disk_full) + out = agent.run({"id": "q"}, tmp_path, environment=env) + assert out.extra["infra"] == "staging_failed" and out.cost_usd == 0.0 and "adapter_failed" not in out.extra + monkeypatch.undo() + monkeypatch.setattr(sandbox.shutil, "which", lambda name: None) # bubblewrap not installed + assert "not installed" in sandbox.sandbox_fault("required") and sandbox.sandbox_fault("best_effort") is None + + +def test_the_usage_an_error_response_reports_is_counted(tmp_path, monkeypatch): + """A terminal HTTP error whose body reports usage (a relay's or provider's): counted on the agent's scored result + (an input refused) and kept on the infra end (a provider that could not serve), as the runner's relay priced it.""" + from hermit.agent.acting import answer as answer_mod + from hermit.agent.model.llm import ChatHTTPError + from hermit.task import Task + from hermit.trainer import rollout + body = json.dumps({"error": {"message": "nope"}, "usage": {"prompt_tokens": 120, "completion_tokens": 20}}) + + def model_failing(code): + class Model: + model = "m" + + def cost(self, prompt, cached, completion): + return (prompt + completion) / 1e6 + + def chat(self, messages, abort=None, **kwargs): + raise ChatHTTPError(code, body.encode(), {}) + return Model() + + params = {"model": {"base_url": "https://x", "model": "m"}, "logs": str(tmp_path / "l")} + monkeypatch.setattr(answer_mod.ChatClient, "from_spec", staticmethod(lambda spec: model_failing(400))) + env = Recorded(tmp_path) + env.instruction = "Which?" + out = answer_mod.AnswerAgent(params, tmp_path).run({"id": "q"}, None, environment=env) + assert out.tokens == 140 and out.api_speed["model_calls"] == 1 and out.cost_usd == pytest.approx(1.4e-4) + monkeypatch.setattr(answer_mod.ChatClient, "from_spec", staticmethod(lambda spec: model_failing(503))) + agent = answer_mod.AnswerAgent(params, tmp_path) + task = Task(id="demo#1", public={"id": "demo#1", "harbor_task": "demo", "question": "Which?"}) + record = rollout.environment_rollout(task, task.given(), None, lambda t, a, environment=None: agent.run( + t, a, environment=environment), Environments(tmp_path), 0.0, pause=lambda s: None) + calls = record.api_speed["model_calls"] # each run again's spend kept, its call counted + assert calls >= 2 and record.tokens == 140 * calls and record.extra["unscored"] == "model_provider_failed" + assert record.cost_usd == pytest.approx(1.4e-4 * calls) + from hermit.trainer.rollout import run_one # and outside a task's own environment: the record + failed = run_one(task, None, lambda t, a: agent.run(t, a), lambda row, out: {"score": 0.0}) + assert failed["status"] == "error" and failed["tokens"] == 140 and failed["cost_usd"] == pytest.approx(1.4e-4) + + +def test_an_unreadable_verifier_log_leaves_the_reward(tmp_path, monkeypatch): + """feedback.txt is optional (the runner's log reader tolerated one it could not read): its output's tail is read + instead, and a reward the verifier wrote stands.""" + logs = tmp_path / "verifier" + logs.mkdir() + (logs / "feedback.txt").write_text("hidden") + (logs / "test-stdout.txt").write_text("1 passed") + original = Path.read_text + + def read_text(self, *a, **k): + if self.name == "feedback.txt": + raise PermissionError(13, "Permission denied", str(self)) + return original(self, *a, **k) + + monkeypatch.setattr(Path, "read_text", read_text) + envs = HarborEnvironments({"tasks_dir": str(tmp_path), "harbor": sys.executable}, base=tmp_path) + verdict = envs.verdict({"rewards": {"reward": 1.0}, "verifier_dir": str(logs)}) + assert verdict["reward"] == 1.0 and verdict["status"] == "ok" and verdict["feedback"] == "1 passed" + + +def test_a_staging_failure_is_run_again_and_spends_nothing(tmp_path): + """The runner's staging_failed (this machine's, before any session): infra, run again, its attempt's spend 0, so + the run that then passes keeps a known cost.""" + from hermit.adapter.session import staging_failed + from hermit.task import Task + from hermit.trainer import rollout + runs = [] + + def pipeline(task, adapter, environment=None): + runs.append(1) + if len(runs) == 1: + return staging_failed("staging_failed: OSError: [Errno 28] No space left on device", 0.0) + return RunOutput(output="done", cost_usd=0.25, tokens=100, api_speed={"model_calls": 1}) + + task = Task(id="demo#1", public={"id": "demo#1", "harbor_task": "demo"}) + out = rollout.environment_rollout(task, task.given(), None, pipeline, Environments(tmp_path), 0.0, + pause=lambda s: None) + assert len(runs) == 2 and out.cost_usd == pytest.approx(0.25) and out.tokens == 100 + assert [a["unscored"] for a in out.extra["attempts"]] == ["staging_failed", None] + + +def test_a_cli_agents_staging_folders_on_a_full_disk_are_staging_failed(tmp_path, monkeypatch): + """The runner guarded its job's folders with the staging: a full disk making them is staging_failed (run again, + nothing spent), never this machine's final host_failed.""" + from hermit.agent.acting.workspace import CliAgent + monkeypatch.setenv("HERMIT_HOMES", str(tmp_path / "homes")) + original = Path.mkdir + + def mkdir(self, *a, **k): + if self.name == "work": + raise OSError(28, "No space left on device", str(self)) + return original(self, *a, **k) + + monkeypatch.setattr(Path, "mkdir", mkdir) + env = Recorded(tmp_path, lambda c: ExecResult(0, CODEX_STREAM) if session_of(c) else ExecResult(0, "")) + out = CliAgent({"backend": "codex", "web_search": False, "bin": "codex", "workspaces": str(tmp_path / "ws")}, + tmp_path).run({"id": "t", "instruction": "do it"}, None, environment=env) + assert out.extra["infra"] == "staging_failed" and not out.extra["infra_final"] + assert (out.cost_usd, out.tokens, out.api_speed["model_calls"]) == (0.0, 0, 0) + + +def test_a_terminal_failure_after_a_run_again_keeps_every_attempts_spend(tmp_path): + """A pipeline's own failure after an infra run again: the record sums the earlier attempts' spend and the failed + one's (what it said it billed), its cost known when each is.""" + from hermit.task import Task + from hermit.trainer import rollout + runs = [] + + def pipeline(task, adapter, environment=None): + runs.append(1) + if len(runs) == 1: + return RunOutput(output=None, tokens=140, cost_usd=0.014, api_speed={"model_calls": 1}, + extra={"infra": "model_provider_failed", "infra_final": False, "exception_message": "x"}) + failed = KeyError("a bug of the agent's own") + failed.spent = RunOutput(output=None, tokens=200, cost_usd=0.020, api_speed={"model_calls": 1}) + raise failed + + task = Task(id="demo#1", public={"id": "demo#1", "harbor_task": "demo"}) + record = rollout.environment_rollout(task, task.given(), None, pipeline, Environments(tmp_path), 0.0, + pause=lambda s: None) + assert record["status"] == "error" and record["tokens"] == 340 and record["cost_usd"] == pytest.approx(0.034) + assert record["api_speed"]["model_calls"] == 2 and len(record["pipeline_extra"]["attempts"]) == 2 + + +def test_a_sessions_placeholder_usage_in_an_error_is_no_report(tmp_path, monkeypatch): + """A CLI endpoint's error carries all-zero usage when its session reported none: unknown, never a measured 0 + (hermit.tools.failed_usage's rule).""" + from hermit.agent.acting.answer import _error_usage + from hermit.agent.model.llm import ChatHTTPError + zero = json.dumps({"error": {"message": "timed out"}, "usage": {"prompt_tokens": 0, "completion_tokens": 0, + "prompt_tokens_details": {"cached_tokens": 0}}}) + + class Client: + model = "m" + + def cost(self, prompt, cached, completion): + return 0.0 + + assert _error_usage(Client(), ChatHTTPError(504, zero.encode(), {})) == [] diff --git a/tests/core/test_harbor_examples.py b/tests/core/test_harbor_examples.py index 8cadc12a..ca7533f8 100644 --- a/tests/core/test_harbor_examples.py +++ b/tests/core/test_harbor_examples.py @@ -5,13 +5,12 @@ trains and validates on no row it is tested on, tests on the rows of the baseline beside it, shares its profile's jobs_dir (the smoke keeps its own, on rows no run takes) and names its profile's CLI, model, login and effort for the acting agent and every learner, whose wait for the plan's window has no limit of its own (AHE's clocks count it); -every learner runs in its -sandbox and no agent searches the web; SkillOpt sorts a rollout as its benchmark scores it; a baseline is a config for -`hermit eval` alone (the keys every config names, a training split, a template and an Evolver, are placeholders, and -no Evolver has the name: `hermit train` refuses it) and a method's a training config; every config loads as `hermit` -loads it, prepared or not, which refuses a key it does not know (a block of its own for how a config runs, say); and -the params every config hands the Harbor runner pass the runner's own checks of them at its start (no trial cache, -say).""" +every learner runs in its sandbox and no agent searches the web; SkillOpt sorts a rollout as its benchmark scores it; +a baseline is a config for `hermit eval` alone (the keys every config names, a training split, a template and an +Evolver, are placeholders, and no Evolver has the name: `hermit train` refuses it) and a method's a training config; +every config loads as `hermit` loads it, prepared or not, which refuses a key it does not know (a block of its own for +how a config runs, say); and the Harbor settings every config gives (its own over its benchmark file's) pass the +Harbor environments' own checks of them (hermit.environment.harbor).""" from __future__ import annotations import importlib.util @@ -23,11 +22,9 @@ import yaml from hermit.adapter.manifest import read_manifest -from hermit.agent.acting.agent import load_agent from hermit.agent.evolver.loader import find from hermit.config import Config from hermit.harbor.dataset import split -from hermit.harbor.runner import INFRA_RETRIES, INFRA_WAIT_S EXAMPLES = Path(__file__).resolve().parents[2] / "examples" IN_SAMPLE = ["terminal_bench_2_0", "terminal_bench_science"] @@ -92,11 +89,11 @@ def configs(bench: str, p: str) -> dict[str, dict]: def test_a_method_tests_on_its_baselines_rows_with_its_profiles_models(bench, p): profile, got = PROFILES[p], configs(bench, p) for name, config in got.items(): - params = config["agent"]["params"] - assert params["acting_agent"] == {"cli": profile["cli"]}, name - assert (params["model"], params["login"], params["effort"]) == (profile["model"], profile["login"], - "medium"), name - assert params["benchmark"] == "../utils/benchmark.yaml", name + agent = config["agent"] + assert agent["base"] == profile["cli"], name + assert agent["model"] == {"name": profile["model"], "login": profile["login"], "effort": "medium"}, name + assert agent[profile["cli"]]["web_search"] is False, name + assert config["environment"]["harbor"]["benchmark"] == "../utils/benchmark.yaml", name if name != "baseline": # every learner on the acting model if "llm" in config: # SkillOpt's learner llm = config["llm"] @@ -118,8 +115,9 @@ def test_a_method_tests_on_its_baselines_rows_with_its_profiles_models(bench, p) run, smoke = got["run"], got["smoke"] assert run["data"]["test"] == base["data"]["test"] assert run["judge"] == base["judge"] - assert run["agent"]["params"]["jobs_dir"] == base["agent"]["params"]["jobs_dir"] # the profile's trials - assert smoke["agent"]["params"]["jobs_dir"] != run["agent"]["params"]["jobs_dir"] + jobs = lambda config: config["environment"]["harbor"]["jobs_dir"] + assert jobs(run) == jobs(base) # the profile's environments + assert jobs(smoke) != jobs(run) assert smoke["data"]["train"] == smoke["data"]["test"] == "../utils/splits/smoke.txt" where = EXAMPLES / bench / METHODS[bench] taken = {row for part in ("train", "val", "test") if run["data"].get(part) @@ -148,12 +146,13 @@ def every_config(): @pytest.mark.parametrize("bench", METHODS) def test_every_learner_is_sandboxed_and_no_agent_searches_the_web(bench): - assert read(EXAMPLES / bench / "utils" / "benchmark.yaml")["web_search"] is False # the acting agent's, shared + assert "web_search" not in read(EXAMPLES / bench / "utils" / "benchmark.yaml") # the agent's, not the tasks' for path in sorted((EXAMPLES / bench).glob("*/*.yaml")): if path.parent.name == "utils": continue config = read(path) - assert config["agent"]["params"].get("web_search", False) is False, path.name # no config turns it on + agent = config["agent"] + assert agent[agent["base"]]["web_search"] is False, path.name # every config sets it off if "evolver" in config and path.parent.name != "baseline": # a baseline's Evolver: a placeholder sandbox = config["sandbox"] assert sandbox["enabled"] is True and sandbox.get("enforce", "required") == "required", path.name @@ -196,7 +195,9 @@ def test_a_config_loads_as_hermit_loads_it(path): # no download needed: a config names its benchmark's files, not its tasks config = Config.load(path) # a key hermit does not know, a `run:` block say, fails here d = config.data - assert d["agent"]["name"] == "harbor" + assert d["agent"]["name"] in ("codex", "claude-code") and d["environment"]["harbor"] + assert (d["judge"]["path"], d["judge"]["name"]) in (("hermit.task.verify", "environment"), + ("../utils/judge.py", "score")) read_manifest(config.template_dir()) # hermit eval reads the template's id (its tree's folder) if path.parent.name == "baseline": # the Trainer's lookup: hermit train refuses the config with pytest.raises(ValueError, match="No Evolver named 'eval-only'"): @@ -206,24 +207,14 @@ def test_a_config_loads_as_hermit_loads_it(path): @pytest.mark.parametrize("path", every_config(), ids=config_id) -def test_a_configs_params_pass_the_runners_own_checks(path, tmp_path, monkeypatch): - """The runner built from a config's params (its own over its benchmark file's) as hermit builds it, its startup - checks (check_agent) included: what the runner refuses of its params at its start (a trial cache, a count or a - cleanup mode it cannot read, a benchmark file's model reference, a login for another CLI, no web_search, a setting - the CLI's Base Agent refuses as it renders the launch plan) is refused here. This machine's own are stood in for, - as the run's and hermit check's: the sign-in (its check, and a stand-in for a Claude Code session's token - variable), bubblewrap and its start of the strict adapter sandbox, and Harbor's own `harbor` (this interpreter). - The runner's own infra retries apply: the configs set none.""" - import shutil - from hermit.agent.base import login as subscription - from hermit.harbor import runner - which = shutil.which - monkeypatch.setattr(subscription, "problems_of", lambda *args, **kwargs: []) - monkeypatch.setattr(shutil, "which", lambda name, *args, **kwargs: - "/usr/bin/bwrap" if name == "bwrap" else which(name, *args, **kwargs)) - monkeypatch.setattr(runner, "sandbox_fault", lambda enforce, *, fresh=True: None) - monkeypatch.setenv("CLAUDE_CODE_OAUTH_TOKEN", "stand-in-token-of-no-account") +def test_a_configs_harbor_settings_pass_the_environments_own_checks(path, tmp_path): + """The run's Harbor environments built from a config's settings (its own over its benchmark file's) as a run builds + them: what they refuse at the start (an unknown key, a cleanup mode or a verifier_timeout they do not know, a + multiplier below 1, no tasks_dir) is refused here. Harbor's own command is stood in for (this interpreter, with a + python beside it), and the jobs directory is a temporary one; no environment is started.""" + from hermit.environment.harbor import HarborEnvironments config = Config.load(path) - params = {**config.data["agent"]["params"], "harbor": sys.executable, "jobs_dir": str(tmp_path / "jobs")} - agent = load_agent(config.data["agent"]["name"], params, config.base) - assert (agent.infra_retries, agent.infra_wait_s) == (INFRA_RETRIES, INFRA_WAIT_S) == (2, 300.0) + settings = {**config.data["environment"]["harbor"], "harbor": sys.executable, "jobs_dir": str(tmp_path / "jobs")} + environments = HarborEnvironments(settings, base=config.base) + assert environments.tasks_dir.name and environments.cleanup in ("images_keep", "rmi_task_image", "rmi_over_gb") + assert environments.environment_config()["delete"] is True diff --git a/tests/core/test_harbor_harborside.py b/tests/core/test_harbor_harborside.py deleted file mode 100644 index 8c210b5d..00000000 --- a/tests/core/test_harbor_harborside.py +++ /dev/null @@ -1,3118 +0,0 @@ -"""The Harbor-side half (hermit/harbor/harborside): the launch plan's executor (harborside.hermit) under every -HermitXXX, each class's install, the answer agent, the scripted agent and the stream driver's requests. - -harbor is not importable from Hermit's venv, so the checks run under the Harbor venv's Python: this module is that -script too (`python test_harbor_harborside.py [,...] ` prints one JSON report per check), and -the tests share one such process (the `checks` fixture runs every check once); they skip when that venv is absent. -Stand-ins, at the edges only: - -- FakeEnv: Harbor's environment as the agents see it, recording every upload, exec, compose call and download, and - answering the probes the classes and Harbor's own installs make (the agent user, versions, the package manager, the - image's own context file, mini-swe-agent's Python, a session that hangs until Harbor's timeout cancels it); a - mounted binary's probe runs with bash, a stand-in in the binary's place (one only root may run refuses another - user, and fails that user's `test -x`, as a mount of mode 700 does; one that warns prints Codex's warning before - its version to a user other than root, whose HOME it cannot write); an exec may raise as one whose compose client - this machine could not spawn. -- ShellEnv: the container's /installed-agent, /logs, /app and /opt (where a CLI's binary is mounted) mapped under the - scratch directory and every command run with bash on this machine, so the generated launch script, the session's - process group and its kill, the descriptor that carries a token, the copy-back and the files returned to the host - run for real (no Docker). It refuses any command that names a path outside its root, answers an exec whose working - directory is gone as Docker does (exit 126, runc's words), and may run with a PATH of its own (an image without - base64 or wc, or whose cp or cat fails part-way, as on a full disk) and a HOME of its own (the agent user's, with an - nvm) and in the C locale (where a test matches what bash says, in the same words whatever this machine's LANG). -- either with `noise`: other lines before and after every exec's output, as Harbor's docker exec merges stderr into - stdout (ShellEnv's from a BASH_ENV of the image's that echoes). -- a stand-in CLI (a bash script first on the plan's PATH, or in the program's place), and a stand-in chat endpoint and - tool server on 127.0.0.1. -- the launch plans Hermit's side renders (hermit.harbor.launch) for every CLI on every built-in backend: one test - renders them, its keys fakes, then runs its check (`launch_env`) alone in a run of this module of its own. - -No Docker, no network, no model, no secret: the key and the token are fakes set in the check's own process.""" -import errno -import json -import os -from pathlib import Path -import subprocess -import sys -import traceback - -ROOT = Path(__file__).resolve().parents[2] -HARBORSIDE_ROOT = ROOT / "hermit" / "harbor" -HARBOR_VENV = Path(os.environ.get("HERMIT_HARBOR_VENV", "~/.cache/hermit/harbor-venv")).expanduser() -HARBOR_PYTHON = HARBOR_VENV / "bin" / "python" -FAKE_KEY = "fw-fake-key-never-real-0123456789" -FAKE_TOKEN = "sk-ant-oat01-fake-token-never-real" -INSTRUCTION = "Fix the failing test.\nIt's \"quoted\", $HOME stays text, and so does `this`.\n" -SUFFIX = "\n\n`hermit/demo/memory/NOTES.md` is your memory across questions." -IMAGE_CONTEXT = "# Project\n\nAlways run the tests before you stop.\n" -TOOL_PYTHON = "/root/.local/share/uv/tools/mini-swe-agent/bin/python" -# What Harbor's docker exec merges into a command's stdout besides its output: an echo of the image's BASH_ENV before -# it, a warning on stderr after it (the noisy environments' BASH_ENV prints both) -NOISE_BEFORE = "BASH_ENV of the image: hello there\n" -NOISE_AFTER = "WARN[0000] a compose warning after the output\n" -BASH_ENV_NOISE = f"echo \"{NOISE_BEFORE.strip()}\"\ntrap 'echo \"{NOISE_AFTER.strip()}\" >&2' EXIT\n" -# What Codex 0.164.0 says to --version before its version under a HOME it cannot write (HOME=/: a uid without a passwd -# entry, a home never created) -CODEX_HOME_WARNING = ("WARNING: proceeding, even though we could not create PATH aliases: Permission denied " - "(os error 13)") -CONTEXT_FILE = {"codex": "AGENTS.md", "claude": "CLAUDE.md", "pi": "AGENTS.md", "opencode": "AGENTS.md", - "dsh": "AGENTS.md", "mini-swe-agent": None} -PROGRAM = {"codex": "codex", "claude": "claude", "pi": "pi", "opencode": "opencode", "dsh": "dsh", - "mini-swe-agent": "@@HERMIT_PYTHON@@"} -CLASSES = {"codex": ("harborside.codex", "HermitCodex"), "claude": ("harborside.claude", "HermitClaude"), - "pi": ("harborside.pi", "HermitPi"), "opencode": ("harborside.opencode", "HermitOpenCode"), - "dsh": ("harborside.dsh", "HermitDsh"), - "mini-swe-agent": ("harborside.mini_swe_agent", "HermitMiniSweAgent")} -# What each host writes beside the adapter's copy (hermit.hosts.), reduced to a file or two each. -HOST_FILES = {"codex": {".codex/hooks.json": '{"hooks": {}}', ".codex/config.toml": "# roles\n"}, - "claude": {".claude/settings.json": '{"hooks": {}}', ".mcp.json": '{"mcpServers": {}}'}, - "pi": {".pi/mcp.json": '{"mcpServers": {}}'}, "opencode": {"opencode.json": '{"mcp": {}}'}, - "dsh": {".dsh/hermit.patch.yml": "[]\n"}, "mini-swe-agent": {"mini-swe-agent.yaml": "agent: {}\n"}} -STUB_CLI = r"""#!/bin/bash -# A stand-in CLI: it reports what it was given, writes what a session writes, and may run on in the background. -printf 'argv:'; printf ' <%s>' "$@"; echo -echo "stdin: <$(cat)>" -echo "pwd: $PWD" -echo "hook-logs: $HERMIT_HOOK_LOGS" -echo "key-in-env: ${FIREWORKS_API_KEY:+yes}" -echo "renamed-key: ${OPENAI_API_KEY:+yes}" -echo "token-in-env: ${CLAUDE_CODE_OAUTH_TOKEN:+yes}" -if [ -n "$CLAUDE_CODE_OAUTH_TOKEN_FILE_DESCRIPTOR" ]; then - echo "fd-sha: $(cat <&"$CLAUDE_CODE_OAUTH_TOKEN_FILE_DESCRIPTOR" | sha256sum | cut -c1-16)" - echo "parent-environ-token: $(tr '\0' '\n' < /proc/$PPID/environ | grep -c '^CLAUDE_CODE_OAUTH_TOKEN=')" -fi -echo "pgid-is-own: $([ "$(ps -o pgid= -p $$ | tr -d ' ')" = "$$" ] && echo yes || echo no)" -[ -n "$CODEX_HOME" ] && echo "auth-mode: $(stat -c %a "$CODEX_HOME/auth.json" 2>/dev/null)" -mkdir -p hermit/demo/memory && echo "learned something" >> hermit/demo/memory/NOTES.md -echo "the answer" > "$CODEX_HOME/last_message.txt" -echo '{"refreshed": true}' > "$CODEX_HOME/auth.json" -if [ -n "$STUB_SLEEP" ]; then sleep "$STUB_SLEEP" & echo "$!" > "$HERMIT_HOOK_LOGS/child.pid"; wait; fi -exit "${STUB_EXIT:-0}" -""" -# A cp and a cat that fail part-way, as on a full disk (first on a ShellEnv's PATH): what they write is the first bytes -# of what they copy (cp's files mode 600), then they say so and exit 1. The cp fails only copying an adapter's memory, -# a last_message.txt or a sessions folder: a source of another name it copies as cp does -FAILING_CP = r"""#!/bin/bash -src="${@: -2:1}" dst="${@: -1}" -case "${src##*/}" in - memory|last_message.txt|sessions) ;; - *) command -p cp "$@"; exit ;; -esac -if [ -d "$src" ]; then - mkdir -p "$dst" && for f in "$src"/*; do head -c 4 "$f" > "$dst/${f##*/}"; chmod 600 "$dst/${f##*/}"; done -else - head -c 4 "$src" > "$dst" && chmod 600 "$dst" -fi -echo "cp: error writing '$dst': No space left on device" >&2 -exit 1 -""" -FAILING_CAT = r"""#!/bin/bash -head -c 8 "$1" -echo "cat: write error: No space left on device" >&2 -exit 1 -""" - - -def make_home(base: Path) -> Path: - """A CLI home as hermit.harbor.launch renders it: a config naming the workdir by its placeholder, a sign-in.""" - home = base / "launch" / "home" - if not home.is_dir(): - (home / "agents").mkdir(parents=True) - (home / "config.toml").write_text('[projects."@@HERMIT_WORKDIR@@"]\ntrust_level = "trusted"\n') - (home / "agents" / "reviewer.toml").write_text('name = "reviewer"\n') - (home / "auth.json").write_text('{"tokens": "a sign-in, @@HERMIT_WORKDIR@@ left as it is"}') - return home - - -def make_deploy(base: Path, cli: str, *, context: str = "Hermit context.\n") -> Path: - """A staged deployment (hermit.harbor.staging): the adapter's copy, the host's files, the context, the record.""" - work = base / "deploy" - if work.is_dir(): - return work - (work / "hermit" / "demo" / "memory").mkdir(parents=True) - (work / "hermit" / "demo" / "ADAPTER.md").write_text("The adapter.\n") - (work / "hermit" / "demo" / "memory" / "NOTES.md").write_text("# Notes\n") - for rel, text in HOST_FILES[cli].items(): - (work / rel).parent.mkdir(parents=True, exist_ok=True) - (work / rel).write_text(text) - if CONTEXT_FILE[cli]: - (work / CONTEXT_FILE[cli]).write_text(context) - (work / ".hermit-deploy.json").write_text(json.dumps({"host": cli, "adapter_id": "demo", "hooks": True})) - return work - - -def make_bundle(base: Path) -> Path: - """Hermit's modules for mini-swe-agent's runner, as the runner bundles them (reduced to the runner's file).""" - (base / "hermit" / "agent" / "base").mkdir(parents=True, exist_ok=True) - (base / "hermit" / "agent" / "base" / "mini_swe_agent_run.py").write_text("# Hermit's runner\n") - return base - - -def write_plan(base: Path, cli: str, *, deploy=True, name="plan.json", **fields) -> Path: - """A launch plan (schema version 1) for `cli` under `base`, with the fields given replacing the defaults.""" - base.mkdir(parents=True, exist_ok=True) - home = make_home(base) - work = make_deploy(base, cli) if deploy else None - plan = {"version": 1, "cli": cli, "label": cli, "workdir": "/app", - "deploy": {"work_dir": str(work), "adapter_id": "demo", "context_file": CONTEXT_FILE[cli], - "context_limit": None, "hooks": True, "copy_back": ["hermit/demo/memory"]} if work else None, - "home": {"host_dir": str(home), "container_dir": "/installed-agent/hermit-home", - "secret_files": ["auth.json"]}, - "bundle": None, "command": [PROGRAM[cli], "exec", "-C", "@@HERMIT_WORKDIR@@", "@@HERMIT_PROMPT@@"], - "stdin": False, "prompt_suffix": SUFFIX, - "env": {"CODEX_HOME": "/installed-agent/hermit-home", "PROJECT": "@@HERMIT_WORKDIR@@/src"}, - "env_from_host": {}, "env_by_fd": {}, "nvm": False, "path_prepend": [], "stream": f"agent-{cli}.jsonl", - "collect": [{"from": "/installed-agent/hermit-home/last_message.txt", "to": "last_message.txt"}], - "return_files": [], "tools_script": None, "kill_grace_s": 2, "metadata": {"plan_digest": "abc123"}} - plan.update(fields) - path = base / name - path.write_text(json.dumps(plan)) - return path - - -# ----- the Harbor side (runs under the Harbor venv) ----------------------------------------------------------------- - -def harbor_side(check: str, scratch: Path) -> dict: - scratch = scratch / check # one directory per check - scratch.mkdir(parents=True, exist_ok=True) - import asyncio - import base64 - import hashlib - import http.server - import importlib - import re - import shlex - import shutil - import threading - import time - from types import SimpleNamespace - from harbor.agents.installed.base import NonZeroAgentExitCodeError - from harbor.environments.base import ExecResult - from harbor.models.agent.context import AgentContext - from harborside import hermit as executor - - os.environ["FIREWORKS_API_KEY"] = FAKE_KEY # Harbor's environment holds the secrets by value - os.environ["CLAUDE_CODE_OAUTH_TOKEN"] = FAKE_TOKEN - - def agent_class(cli: str): - module, name = CLASSES[cli] - return getattr(importlib.import_module(module), name) - - def make(cli: str, plan: Path | None = None, **kwargs): - logs = scratch / "logs" - logs.mkdir(exist_ok=True) - return agent_class(cli)(logs_dir=logs, **({"plan": str(plan)} if plan else {}), **kwargs) - - class FakeEnv: - """Records every call; answers the probes. `files` are the container's files `test -f` and `base64` find.""" - - def __init__(self, *, workdir="/app", image_context="", python3=True, missing=(), launch_code=0, hang=False, - mounted=None, tool_python=TOOL_PYTHON, bundle_ok=True, files=None, misses=0, listing="", - failing=(), answers=None, pwd="/app", noise=False, hanging=(), hang_s=3600, hanging_once=(), - root_only=(), warns=(), raising=None): - self.calls, self.default_user, self.pwd = [], None, pwd - self.task_env_config = SimpleNamespace(workdir=workdir) - self.image_context, self.python3, self.missing = image_context, python3, set(missing) - self.launch_code, self.hang, self.running = launch_code, hang, False - # commands holding one of these fail, saying what the mapping gives (given names alone: a full disk) - self.failing = dict(failing) if isinstance(failing, dict) else \ - dict.fromkeys(failing, "cp: cannot create directory: No space left on device\n") - # ... and these exit as the mapping gives (code, output): an exec that exits 0 without what it prints - self.answers = dict(answers or {}) - self.hanging, self.hang_s = tuple(hanging), hang_s # ... hang until Harbor's timeout (or for hang_s) - self.hanging_once = list(hanging_once) # ... the first such exec only (the deadline landed in it) - # ... and these raise OSError(errno) from the exec itself: its compose client not spawned (EMFILE, EAGAIN) - self.raising = dict(raising or {}) - self.mounted = dict(mounted or {}) # a mounted binary's path -> what its --version says - self.root_only = set(root_only) # ... those only root may run (mode 700): another user's - # probe exits 126 and its `test -x` 1 (no user: the - # image's own, root) - self.warns = set(warns) # ... those that, run by a user other than root (whose - # HOME they cannot write), warn before their version - self.tool_python, self.bundle_ok = tool_python, bundle_ok - self.files, self.misses, self.listing = dict(files or {}), misses, listing - self.noise = noise # every exec's output between NOISE_BEFORE and _AFTER - - def _merge_env(self, env): - return dict(env or {}) - - def _resolve_user(self, user): - return user if user is not None else self.default_user - - def _output_callback(self): - return None - - async def _run_docker_compose_command(self, command, check=True, timeout_sec=None, stdin_data=None, - on_output=None): - names = [command[i + 1] for i, a in enumerate(command) if a == "-e" and "=" not in command[i + 1]] - self.calls.append(("compose", list(command), {n: bool(os.environ.get(n)) for n in names})) - return await self._answer(command[-1], None) - - async def exec(self, command, cwd=None, env=None, timeout_sec=None, user=None): - command = command.removeprefix("set -o pipefail; ") # BaseInstalledAgent._exec's prefix - self.calls.append(("exec", user, command, dict(env or {}))) - result = await self._answer(command, user) - if self.noise: - result = ExecResult(stdout=NOISE_BEFORE + (result.stdout or "") + NOISE_AFTER, stderr=result.stderr, - return_code=result.return_code) - return result - - async def _answer(self, c, user): - def ok(out=""): - return ExecResult(stdout=out, stderr="", return_code=0) - - def code(n, out=""): - return ExecResult(stdout=out, stderr="", return_code=n) - - def b64(data) -> str: - return base64.b64encode(data.encode() if isinstance(data, str) else data).decode() - - if any(part in c for part in self.hanging): - await asyncio.sleep(self.hang_s) - for part in [p for p in self.hanging_once if p in c]: - self.hanging_once.remove(part) # run again, it answers - await asyncio.sleep(self.hang_s) - for part, number in self.raising.items(): - if part in c: - raise OSError(number, os.strerror(number)) - for part, said in self.failing.items(): - if part in c: - return code(1, said) - for part, (n, said) in self.answers.items(): - if part in c: - return code(n, said) - if c.startswith('echo "hermit-owner:$(id -u):$(id -g)"'): - return ok("hermit-owner:1000:1000\n") - if c == 'echo "hermit-pwd:$(pwd)"': # where Harbor's execs start: the image's WORKDIR - return ok(f"hermit-pwd:{self.pwd}\n") - if "/installed-agent/hermit/original-" in c and c.startswith("mkdir -p"): # the context's capture - return ok(f"hermit-context:{b64(self.image_context)}\n") # empty: the image has none - for binary, said in self.mounted.items(): - other = self._resolve_user(user) not in (None, "root", 0) - if c == f"test -x {shlex.quote(binary)}": # whether the exec's user may execute the mount - return code(1 if other and binary in self.root_only else 0) - if f"{binary} --version" in c: # the mount's probe, with bash: a stand-in says `said` - stand_in = f"printf %s {shlex.quote(said)}" if said else "(exit 127)" - if other and binary in self.warns: - stand_in = f"{{ echo {shlex.quote(CODEX_HOME_WARNING)} >&2; {stand_in}; }}" - if other and binary in self.root_only: - refused = shlex.quote(f"bash: {binary}: Permission denied") - stand_in = f"{{ echo {refused} >&2; (exit 126); }}" - got = subprocess.run(["bash", "-c", c.replace(f"{binary} --version", stand_in)], text=True, - capture_output=True, env={"PATH": os.environ["PATH"]}, timeout=60) - return code(got.returncode, got.stdout) - if "hermit-data:" in c: # an attachment's bytes, in base64 - path = c.split("base64 < ", 1)[1].split(" |")[0].strip() - return ok(f"hermit-data:{b64(self.files.get(path, b''))}\n") - if re.search(r"(codex|claude|pi|opencode|dsh) --version$", c) and "npm" not in c and "curl" not in c: - return code(127) # Harbor's probe of an installed CLI: none yet - if c.startswith("command -v") and "||" in c: # any of them (OpenCode's downloaders) - return code(0 if any(n not in self.missing for n in re.findall(r"command -v (\S+)", c)) else 1) - if c.startswith("command -v python3"): - return code(0 if self.python3 else 1) - if c.startswith("command -v"): - return code(0 if all(n not in self.missing for n in re.findall(r"command -v (\S+)", c)) else 1) - if c.startswith("for manager in apt-get"): - return ok("apt-get") - if "uv tool dir" in c: - return code(0, f"hermit-python:{self.tool_python}\n") if self.tool_python else code(1) - if "import hermit.agent.base.mini_swe_agent_run" in c: - return code(0, "") if self.bundle_ok else code(1, "ModuleNotFoundError: No module named 'yaml'\n") - if c.startswith("test -f "): - path = c.split()[2].strip("'") - if executor.RETURN_LINE in c: # the return's check: the file's size, if the session left it - return ok(f"{executor.RETURN_LINE}{len(self.files[path]) if path in self.files else 'no'}\n") - if path not in self.files: - return code(1) - if self.misses > 0 and "base64" in c: # the task's entrypoint has not written it yet - self.misses -= 1 - return code(1) - return ok(f"hermit-instruction:{b64(self.files[path])}\n" if "base64" in c else "") - if c.startswith("shopt -s nullglob"): - return ok("".join(f"hermit-file:{path}\n" for path in self.listing.splitlines())) - if c.startswith("n=$(wc -c < "): - return ok(f"hermit-size:{len(self.files.get(c.split('< ', 1)[1].split(')')[0].strip(), b''))}\n") - if c.endswith("bash /installed-agent/hermit/run/launch.sh"): - if self.hang: - self.running = True - await asyncio.sleep(3600) # until Harbor's timeout cancels the agent - return code(self.launch_code) - if "session.pid" in c and "kill -TERM" in c: - killed, self.running = self.running, False - return ok("hermit-killed\n" if killed else "") - return ok() - - async def upload_file(self, source_path, target_path): - self.calls.append(("upload_file", target_path, Path(source_path).read_bytes().decode("utf-8", "replace"))) - - async def upload_dir(self, source_dir, target_dir): - files = {} # as `docker cp` takes them: a link as the link it is - for p in sorted(Path(source_dir).rglob("*")): - if p.is_symlink(): - files[p.relative_to(source_dir).as_posix()] = f"" - elif p.is_file(): - files[p.relative_to(source_dir).as_posix()] = p.read_bytes().decode("utf-8", "replace") - self.calls.append(("upload_dir", target_dir, files)) - - async def download_file(self, source_path, target_path): - self.calls.append(("download_file", source_path, str(target_path))) - Path(target_path).write_text(f"downloaded {source_path}") - - async def download_dir(self, source_dir, target_dir): - self.calls.append(("download_dir", source_dir, str(target_dir))) - - async def is_dir(self, path, user=None): - return True - - async def is_file(self, path, user=None): - return path in self.files - - class PlainEnv(FakeEnv): - """An environment that cannot pass a variable by name (not a local Docker one).""" - _run_docker_compose_command = None - - class ShellEnv: - """The container's paths under `root`, its commands run here with bash (the module's docstring).""" - PREFIXES = re.compile(r"(?(;|&'])|(?<=[\s=<>(]"))/[^\s"';|&()<>]*""", re.M) - SAFE = ("/dev/null", "/proc/") - - def __init__(self, root: Path, workdir="/app", noise=False, path=None, locale=None, home=None): - self.root, self.calls, self.procs, self.default_user = root, [], [], None - for d in ("installed-agent", "logs/agent", "app"): - (root / d).mkdir(parents=True, exist_ok=True) - self.task_env_config = SimpleNamespace(workdir=workdir) - self.path = path # the container's PATH, when not this machine's - self.home = home # the agent user's HOME, when not this machine's - self.locale = locale # its LC_ALL ("C": bash's own words, whatever the LANG) - self.bash_env = None # noise: every bash prints NOISE_BEFORE and NOISE_AFTER - if noise: - self.bash_env = root / "bash_env.sh" - self.bash_env.write_text(BASH_ENV_NOISE) - - def map(self, text: str) -> str: - return self.PREFIXES.sub(lambda m: str(self.root) + m.group(1), str(text)) - - def _check(self, command: str) -> None: - for path in self.PATHS.findall(command): - if not (path.startswith(str(self.root)) or path.startswith(self.SAFE)): - raise AssertionError(f"ShellEnv refuses a command naming {path} outside its root: {command[:300]}") - - def _merge_env(self, env): - return dict(env or {}) - - def _resolve_user(self, user): - return user if user is not None else self.default_user - - def _output_callback(self): - return None - - async def _bash(self, command, env, cwd, names=()): - mapped = self.map(command) - self._check(mapped) - full = {k: v for k, v in os.environ.items() if k in ("PATH", "HOME", "LANG", "USER", "LOGNAME")} - if self.path is not None: - full["PATH"] = self.path - if self.home is not None: - full["HOME"] = str(self.home) - if self.locale is not None: - full["LC_ALL"] = self.locale - full.update({k: self.map(v) for k, v in env.items()}) - full.update({n: os.environ[n] for n in names if n in os.environ}) # by name, as compose does - if self.bash_env is not None: - full["BASH_ENV"] = str(self.bash_env) - where = cwd or self.task_env_config.workdir - if not Path(self.map(where)).is_dir(): # as Docker refuses it: runc cannot enter a workdir gone - return ExecResult(stdout=f"OCI runtime exec failed: exec failed: unable to start container process: " - f'chdir to cwd ("{where}") set in config.json failed: no such file or ' - "directory: unknown\n", stderr="", return_code=126) - proc = await asyncio.create_subprocess_exec( - "bash", "-c", mapped, cwd=self.map(where), env=full, - stdin=asyncio.subprocess.DEVNULL, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.STDOUT) - self.procs.append(proc) - out, _ = await proc.communicate() # if cancelled, the process goes on, as docker exec's does - return ExecResult(stdout=out.decode(errors="replace"), stderr="", return_code=proc.returncode) - - async def exec(self, command, cwd=None, env=None, timeout_sec=None, user=None): - command = command.removeprefix("set -o pipefail; ") - self.calls.append(("exec", user, command, dict(env or {}))) - return await self._bash(command, dict(env or {}), cwd) - - async def _run_docker_compose_command(self, command, check=True, timeout_sec=None, stdin_data=None, - on_output=None): - self.calls.append(("compose", list(command), {})) - args, env, names, cwd = list(command[1:]), {}, [], None - while args and args[0] != "main": - flag, value = args.pop(0), args.pop(0) - if flag == "-w": - cwd = value - elif flag == "-e" and "=" in value: - env[value.split("=", 1)[0]] = value.split("=", 1)[1] - elif flag == "-e": - names.append(value) - return await self._bash(args[-1], env, cwd, names) - - async def upload_file(self, source_path, target_path): - target = Path(self.map(target_path)) - target.parent.mkdir(parents=True, exist_ok=True) - data = Path(source_path).read_bytes() - self.calls.append(("upload_file", target_path, data.decode("utf-8", "replace"))) - target.write_bytes(self.map(data.decode()).encode() if target.name == "launch.sh" else data) - - async def upload_dir(self, source_dir, target_dir): - self.calls.append(("upload_dir", target_dir, {})) - shutil.copytree(source_dir, self.map(target_dir), dirs_exist_ok=True) - - async def download_file(self, source_path, target_path): - self.calls.append(("download_file", source_path, str(target_path))) - shutil.copy(self.map(source_path), target_path) - - async def settle(self): - for proc in self.procs: - if proc.returncode is None: - await asyncio.wait_for(proc.wait(), 30) - - def calls(env, kind): - return [c for c in env.calls if c[0] == kind] - - def uploaded(env) -> dict: - """target -> text, for every file uploaded (a directory's files under it).""" - out = {} - for c in env.calls: - if c[0] == "upload_file": - out[c[1]] = c[2] - elif c[0] == "upload_dir": - out.update({f"{c[1]}/{rel}": text for rel, text in c[2].items()}) - return out - - def leaks(env) -> list: - """Every recorded call (exec, upload, compose, download) that holds the key's or the token's value.""" - return [json.dumps(c)[:300] for c in env.calls if FAKE_KEY in json.dumps(c) or FAKE_TOKEN in json.dumps(c)] - - def order(env) -> list: - return [c[0] + ":" + (c[2][:60] if c[0] == "exec" else c[1][-1][:60] if c[0] == "compose" else c[1]) - for c in env.calls] - - def run(agent, env, instruction=INSTRUCTION, timeout=None, cancel_when=None): - """The agent's run as Harbor calls it, under its timeout if any, or cancelled (what that timeout does to the - run) once `cancel_when()` holds; the error raised, or None.""" - async def go(): - task = asyncio.ensure_future(agent.run(instruction, env, AgentContext())) - try: - if cancel_when is not None: - deadline = time.monotonic() + 60 - while not cancel_when() and not task.done() and time.monotonic() < deadline: - await asyncio.sleep(0.05) - task.cancel() - await (task if timeout is None else asyncio.wait_for(task, timeout)) - finally: - if isinstance(env, ShellEnv): - await env.settle() - try: - asyncio.run(go()) - return None - except BaseException as exc: # noqa: BLE001 - reported to the test - return [type(exc).__name__, str(exc)] - - def metadata(agent) -> dict: - context = AgentContext() - agent.populate_context_post_run(context) - return context.model_dump() - - report: dict = {} - - if check == "codex_plan": - plan = write_plan(scratch, "codex", env_from_host={"FIREWORKS_API_KEY": "FIREWORKS_API_KEY", - "OPENAI_API_KEY": "FIREWORKS_API_KEY"}, - path_prepend=["/opt/hermit-codex/bin"], - collect=[{"from": "/installed-agent/hermit-home/last_message.txt", "to": "last_message.txt"}, - {"from": "/installed-agent/hermit-home/auth.json", "to": "auth.json"}], - return_files=[{"from": "/installed-agent/hermit-home/auth.json", - "to": str(scratch / "return" / "auth.json")}]) - canary = scratch / "host-file.txt" # a file of this host that links in the staging point at - canary.write_text("a host file, @@HERMIT_WORKDIR@@\n") - (scratch / "deploy" / ".codex" / "linked.txt").symlink_to(canary) - (scratch / "deploy" / "linked-dir").symlink_to(scratch) - (scratch / "launch" / "home" / "linked.toml").symlink_to(canary) - report["canary"], report["scratch"] = str(canary), str(scratch) - env = FakeEnv(image_context=IMAGE_CONTEXT, files={"/installed-agent/hermit-home/auth.json": "{}"}) - agent = make("codex", plan) - report["raised"] = run(agent, env) - report["order"] = order(env) - report["uploaded"] = uploaded(env) - report["execs"] = [(c[1], c[2]) for c in calls(env, "exec")] - report["compose"] = [c[1] for c in calls(env, "compose")] - report["compose_names_resolved"] = [c[2] for c in calls(env, "compose")] - report["leaks"] = leaks(env) - report["returned"] = (scratch / "return" / "auth.json").read_text() - report["canary_after"] = canary.read_text() - report["context"] = metadata(agent) - - elif check == "context_limit": - for name, limit in (("over", 60), ("within", 4096)): - plan = write_plan(scratch / name, "dsh") - data = json.loads(plan.read_text()) - data["deploy"]["context_limit"] = limit - plan.write_text(json.dumps(data)) - env = FakeEnv(image_context=IMAGE_CONTEXT) - report[name] = {"raised": run(make("dsh", plan), env), "uploaded": uploaded(env), "order": order(env)} - # the image's own text as dsh counts it, each `` a byte longer, when the plan says it escapes - for name, escaped in (("escaped", ""), ("unescaped", None)): - plan = write_plan(scratch / name, "dsh") - data = json.loads(plan.read_text()) - data["deploy"].update(context_limit=124, context_escaped=escaped) - plan.write_text(json.dumps(data)) - env = FakeEnv(image_context=IMAGE_CONTEXT + "\n" * 3) - report[name] = {"raised": run(make("dsh", plan), env), "uploaded": uploaded(env)} - - elif check == "every_cli": - bundle = make_bundle(scratch / "bundle") - for cli in CLASSES: - extra = {"bundle": {"host_dir": str(bundle), "container_dir": "/installed-agent/bundle"}} \ - if cli == "mini-swe-agent" else {} - plan = write_plan(scratch / cli, cli, stdin=cli in ("claude", "opencode", "mini-swe-agent"), **extra) - env = FakeEnv(image_context=IMAGE_CONTEXT) - raised = run(make(cli, plan), env) - files = uploaded(env) - report[cli] = {"raised": raised, "record": files.get("/app/.hermit-deploy.json"), - "launch": files.get("/installed-agent/hermit/run/launch.sh"), - "prompt": files.get("/installed-agent/hermit/run/prompt.txt"), - "context": files.get(f"/app/{CONTEXT_FILE[cli]}") if CONTEXT_FILE[cli] else None, - "workdir_files": sorted(k for k in files if k.startswith("/app/"))} - - elif check == "workdir_pwd": - plan = write_plan(scratch, "codex", workdir=None) - env = FakeEnv(workdir=None, pwd="/testbed") - report["raised"] = run(make("codex", plan), env) - report["uploaded"] = uploaded(env) - report["pwd_user"] = [c[1] for c in calls(env, "exec") if c[2] == 'echo "hermit-pwd:$(pwd)"'] - # ... and an image with no WORKDIR either: Harbor's execs start in / - env = FakeEnv(workdir=None, pwd="/") - report["root_raised"] = run(make("codex", write_plan(scratch / "root", "codex", workdir=None)), env) - report["root_dirs"] = [c[1] for c in calls(env, "upload_dir")] - report["root_files"] = sorted(f"{c[1]} {c[2]}" for c in calls(env, "upload_file")) - - elif check == "shell_noise": - # Harbor's docker exec merges stderr into stdout: every value read from the container (the agent user, the - # workdir, the image's own context file) comes with other lines before and after it, for real with bash - env = ShellEnv(scratch / "c", noise=True) - fakebin = env.root / "installed-agent" / "fakebin" - fakebin.mkdir(parents=True) - (fakebin / "codex").write_text(STUB_CLI) - (fakebin / "codex").chmod(0o755) - (env.root / "app" / "AGENTS.md").write_text(IMAGE_CONTEXT) # the image's own context file - plan = write_plan(scratch, "codex", workdir=None, path_prepend=["/installed-agent/fakebin"]) - report["raised"] = run(make("codex", plan), env, timeout=120) - report["root"], report["owner"] = str(env.root), f"{os.getuid()}:{os.getgid()}" - report["chowns"] = [c[2] for c in calls(env, "exec") if c[2].startswith("chown")] - report["context"] = (env.root / "app" / "AGENTS.md").read_text() - report["launch"] = next((c[2] for c in calls(env, "upload_file") if c[1].endswith("/launch.sh")), None) - stream = env.root / "logs" / "agent" / "agent-codex.jsonl" - report["stream"] = stream.read_text() if stream.is_file() else None - - elif check == "slash_prompt": - for name, prefix in (("guarded", "\n"), ("plain", None)): - plan = write_plan(scratch / name, "claude", deploy=False, stdin=True, prompt_slash_prefix=prefix) - env = FakeEnv() - report[name] = {"raised": run(make("claude", plan), env, instruction="/app/data.csv holds the rows."), - "prompt": uploaded(env).get("/installed-agent/hermit/run/prompt.txt")} - - elif check == "stream_steps": - first = write_plan(scratch / "one", "codex") - make_deploy(scratch / "two", "codex", context="Hermit context, revised.\n") - (scratch / "two" / "deploy" / ".codex" / "hooks.json").unlink() # this step deploys no hooks - second = write_plan(scratch / "two", "codex") - third = write_plan(scratch / "three", "codex", deploy=False) - env = FakeEnv(image_context=IMAGE_CONTEXT) - agent = make("codex", first) - for name, plan in (("one", first), ("two", second), ("three", third)): - agent.set_plan(str(plan)) - env.calls.clear() - report[name] = {"raised": run(agent, env), "uploaded": uploaded(env), - "root_execs": [c[2] for c in calls(env, "exec") if c[1] == "root"]} - - elif check == "deploy_failed_step": - # One container, two steps: the first deploys an adapter whose hooks are active; the second deploys none, and - # the removal of what the first one deployed fails - env = FakeEnv(image_context=IMAGE_CONTEXT) - agent = make("codex", write_plan(scratch / "one", "codex")) - report["one"] = {"raised": run(agent, env), "metadata": metadata(agent)["metadata"]} - agent.set_plan(str(write_plan(scratch / "two", "codex", deploy=False))) - env.failing = {"rm -rf /app/.codex": "rm: cannot remove '/app/.codex/hooks.json': Device or resource busy\n"} - report["two"] = {"raised": run(agent, env), "metadata": metadata(agent)["metadata"]} - - elif check in ("cancel_fake", "nonzero_exit"): - plan = write_plan(scratch, "pi", deploy=check == "nonzero_exit", nvm=True) - env = FakeEnv(hang=check == "cancel_fake", launch_code=3, - failing=("hermit-copy",) if check == "nonzero_exit" else ()) - agent = make("pi", plan) - report["raised"] = run(agent, env, timeout=1.5 if check == "cancel_fake" else None) - report["order"] = order(env) - report["kill"] = [(c[1], c[2]) for c in calls(env, "exec") if "kill -TERM" in c[2]] - report["metadata"] = metadata(agent)["metadata"] - - elif check == "launch_failures": - # harborside's own steps before the session's exec failing (the deployment's chown, the probe of the task's - # agent user), and a CLI that exits 97 itself (no launcher's words with it) - # (and an image without base64, whose context file must not be taken for none and overwritten) - for name, failing, exit_code in (("chown", ("chown -R 1000:1000 /app/hermit/demo",), 0), - ("owner", ("id -u",), 0), ("own_97", (), 97), - ("no_base64", ("command -v base64",), 0)): - env = FakeEnv(failing=failing, launch_code=exit_code, image_context=IMAGE_CONTEXT) - report[name] = {"raised": run(make("codex", write_plan(scratch / name, "codex")), env), - "order": order(env)} - # the launch script cannot enter the workdir, for real with bash: its exit 97 and its own words - env = ShellEnv(scratch / "c") - plan = write_plan(scratch / "enter", "codex", deploy=False, workdir="/app/missing") - report["cannot_enter"] = {"raised": run(make("codex", plan), env, timeout=120), "order": order(env)} - # the image's own context file whose read fails, for real with bash (a base64 first on the container's PATH - # that fails reading it); a stand-in CLI first on the plan's PATH, were the session to start - failing_bin = scratch / "failing-base64-bin" - failing_bin.mkdir() - (failing_bin / "base64").write_text("#!/bin/sh\necho 'base64: read error: Input/output error' >&2\nexit 1\n") - (failing_bin / "base64").chmod(0o755) - env = ShellEnv(scratch / "read-c", path=f"{failing_bin}:{os.environ['PATH']}") - (env.root / "app" / "AGENTS.md").write_text(IMAGE_CONTEXT) - (env.root / "installed-agent" / "fakebin").mkdir(parents=True) - (env.root / "installed-agent" / "fakebin" / "codex").write_text("#!/bin/bash\nexit 0\n") - (env.root / "installed-agent" / "fakebin" / "codex").chmod(0o755) - plan = write_plan(scratch / "read_fails", "codex", path_prepend=["/installed-agent/fakebin"]) - report["read_fails"] = {"raised": run(make("codex", plan), env, timeout=120), "order": order(env), - "context": (env.root / "app" / "AGENTS.md").read_text()} - - elif check == "capture_fails": - # For real with bash, one container and three steps of a stream: the first step's capture of the image's own - # AGENTS.md meets a cat first on the container's PATH that fails part-way (FAILING_CAT); the second runs with - # the container's cat working again; the third deploys nothing, so the image's own file is put back. A - # stand-in CLI first on the plan's PATH, were a session to start. - failing_bin = scratch / "failing-cat-bin" - failing_bin.mkdir() - (failing_bin / "cat").write_text(FAILING_CAT) - (failing_bin / "cat").chmod(0o755) - env = ShellEnv(scratch / "c") - (env.root / "app" / "AGENTS.md").write_text(IMAGE_CONTEXT) - (env.root / "installed-agent" / "fakebin").mkdir(parents=True) - (env.root / "installed-agent" / "fakebin" / "codex").write_text("#!/bin/bash\nexit 0\n") - (env.root / "installed-agent" / "fakebin" / "codex").chmod(0o755) - deployed = write_plan(scratch / "deployed", "codex", path_prepend=["/installed-agent/fakebin"]) - bare = write_plan(scratch / "bare", "codex", deploy=False, path_prepend=["/installed-agent/fakebin"]) - agent = make("codex", deployed) - kept = env.root / "installed-agent" / "hermit" / "original-AGENTS.md" - for name, plan, path in (("failed", deployed, f"{failing_bin}:{os.environ['PATH']}"), - ("again", deployed, None), ("restored", bare, None)): - agent.set_plan(str(plan)) - env.path = path - env.calls.clear() - report[name] = {"raised": run(agent, env, timeout=120), - "launched": any(step.startswith("exec:exec bash") for step in order(env)), - "context": (env.root / "app" / "AGENTS.md").read_text(), - "kept": kept.read_text() if kept.is_file() else None} - - elif check == "launch_exec_failed": - # The session's exec failing in Docker before the launch script runs: runc could not start it (the host's /tmp - # full: exit 126 and runc's words, the script not run; every other command for real with bash), and compose's - # service had stopped (the probe of the stream fails as well). Then, for real with bash, a launch script that - # ran a stand-in CLI which exits 3 at once and prints nothing: its stream is there, empty. - oci = ("OCI runtime exec failed: exec failed: unable to start container process: open /tmp/runc-process1: no " - "space left on device: unknown\n") - - class RuncFailedEnv(ShellEnv): - """ShellEnv whose launch exec fails as Docker's does when runc cannot start it: the script never runs - (once the first `runs` launches have run for real).""" - runs = 0 - - async def exec(self, command, cwd=None, env=None, timeout_sec=None, user=None): - if command.endswith("bash /installed-agent/hermit/run/launch.sh") and self.runs > 0: - self.runs -= 1 - elif command.endswith("bash /installed-agent/hermit/run/launch.sh"): - self.calls.append(("exec", user, command, dict(env or {}))) - return ExecResult(stdout=oci, stderr=None, return_code=126) - return await super().exec(command, cwd=cwd, env=env, timeout_sec=timeout_sec, user=user) - env = RuncFailedEnv(scratch / "runc-c") - plan = write_plan(scratch / "runc", "codex", deploy=False) - report["runc"] = {"raised": run(make("codex", plan), env, timeout=120), "order": order(env), - "stream": (env.root / "logs" / "agent" / "agent-codex.jsonl").exists()} - stopped = 'service "main" is not running\n' - env = FakeEnv(failing={"bash /installed-agent/hermit/run/launch.sh": stopped, - "test -e /logs/agent/agent-codex.jsonl": stopped}) - report["stopped"] = {"raised": run(make("codex", write_plan(scratch / "stopped", "codex", deploy=False)), env), - "order": order(env)} - env = ShellEnv(scratch / "empty-c") - (env.root / "installed-agent" / "fakebin").mkdir(parents=True) - (env.root / "installed-agent" / "fakebin" / "codex").write_text("#!/bin/bash\nexit 3\n") - (env.root / "installed-agent" / "fakebin" / "codex").chmod(0o755) - plan = write_plan(scratch / "empty", "codex", deploy=False, path_prepend=["/installed-agent/fakebin"]) - stream = env.root / "logs" / "agent" / "agent-codex.jsonl" - report["empty"] = {"raised": run(make("codex", plan), env, timeout=120), "order": order(env), - "stream": stream.read_text() if stream.is_file() else None} - # ... and one whose stand-in CLI removed its (empty) workdir before it exits 3: an exec in that workdir now - # fails (ShellEnv refuses it as Docker does: exit 126) - env = ShellEnv(scratch / "gone-c") - (env.root / "installed-agent" / "fakebin").mkdir(parents=True) - (env.root / "installed-agent" / "fakebin" / "codex").write_text('#!/bin/bash\nrmdir -- "$PWD" && exit 3\n' - 'exit 4\n') - (env.root / "installed-agent" / "fakebin" / "codex").chmod(0o755) - plan = write_plan(scratch / "gone", "codex", deploy=False, path_prepend=["/installed-agent/fakebin"]) - report["workdir_gone"] = {"raised": run(make("codex", plan), env, timeout=120), - "workdir": (env.root / "app").exists()} - # A stream's later step whose exec fails as runc's does, in the container where the step before it ran for real - # (a stand-in CLI that prints a line and exits 0) and left its stream there - env = RuncFailedEnv(scratch / "step-c") - env.runs = 1 - (env.root / "installed-agent" / "fakebin").mkdir(parents=True) - (env.root / "installed-agent" / "fakebin" / "codex").write_text("#!/bin/bash\necho 'the first step'\n") - (env.root / "installed-agent" / "fakebin" / "codex").chmod(0o755) - agent = make("codex", write_plan(scratch / "step-one", "codex", deploy=False, - path_prepend=["/installed-agent/fakebin"])) - first = run(agent, env, timeout=120) - stream = env.root / "logs" / "agent" / "agent-codex.jsonl" - left = stream.read_text() if stream.is_file() else None - agent.set_plan(str(write_plan(scratch / "step-two", "codex", deploy=False, - path_prepend=["/installed-agent/fakebin"]))) - report["later_step"] = {"first": first, "left": left, "raised": run(agent, env, timeout=120), - "stream": stream.exists()} - # The session's exec, or the look for its stream after a non-zero exit, whose compose client this machine could - # not spawn (an OSError from the exec itself: EMFILE, EAGAIN on a loaded host); the exec by name as well - for name, raising, code, fields in ( - ("unspawned", {"bash /installed-agent/hermit/run/launch.sh": errno.EMFILE}, 0, {}), - ("unspawned_by_name", {"bash /installed-agent/hermit/run/launch.sh": errno.EAGAIN}, 0, - {"env_from_host": {"FIREWORKS_API_KEY": "FIREWORKS_API_KEY"}}), - ("probe_unspawned", {"test -e /logs/agent/agent-codex.jsonl": errno.EAGAIN}, 3, {})): - env = FakeEnv(raising=raising, launch_code=code) - plan = write_plan(scratch / name, "codex", deploy=False, **fields) - report[name] = {"raised": run(make("codex", plan), env), "order": order(env)} - - elif check == "after_cancelled": - # Harbor's timeout lands while the copy-back runs (the session has just ended by itself), whose exec answers - # when run again; and, in another trial, while the refreshed sign-in is returned (its check of the file) - returned = scratch / "return" / "auth.json" - plan = write_plan(scratch, "codex", return_files=[{"from": "/installed-agent/hermit-home/auth.json", - "to": str(returned)}]) - env = FakeEnv(hanging_once=("hermit-copy",), files={"/installed-agent/hermit-home/auth.json": "{}"}) - agent = make("codex", plan) - report["raised"] = run(agent, env, timeout=1.5) - report["order"] = order(env) - report["returned"] = returned.read_text() if returned.is_file() else None - report["metadata"] = metadata(agent)["metadata"] - returned = scratch / "in_return" / "return" / "auth.json" - plan = write_plan(scratch / "in_return", "codex", return_files=[ - {"from": "/installed-agent/hermit-home/auth.json", "to": str(returned)}]) - env = FakeEnv(hanging_once=("test -f /installed-agent/hermit-home/auth.json",), - files={"/installed-agent/hermit-home/auth.json": "{}"}) - agent = make("codex", plan) - report["in_return"] = {"raised": run(agent, env, timeout=1.5), "order": order(env), - "returned": returned.read_text() if returned.is_file() else None, - "metadata": metadata(agent)["metadata"]} - - elif check == "after_own_error": - # Harbor's timeout lands in an after-step (the collect hangs into it) of a session that raised its own error: a - # launch that left no stream (exit 126: runc could not start the exec), and a CLI that exited 3 with its stream - for name, code, failing in (("launch_failed", 126, {"test -e /logs/agent/agent-codex.jsonl": ""}), - ("non_zero", 3, {})): - returned = scratch / name / "return" / "auth.json" - plan = write_plan(scratch / name, "codex", deploy=False, return_files=[ - {"from": "/installed-agent/hermit-home/auth.json", "to": str(returned)}]) - env = FakeEnv(launch_code=code, failing=failing, hanging=("hermit-collect",), - files={"/installed-agent/hermit-home/auth.json": "{}"}) - agent = make("codex", plan) - report[name] = {"raised": run(agent, env, timeout=1.5), "order": order(env), - "returned": returned.read_text() if returned.is_file() else None, - "metadata": metadata(agent)["metadata"]} - - elif check == "after_bounded": - # Harbor's timeout cancels the session and the container hangs after it (the copy-back and the collect do not - # answer for 5 s): each step left has AFTER_CANCEL_S, cut to 0.3 s here. Then a second cancellation (the job's, - # after Harbor's timeout) while the copy-back hangs. - returned = scratch / "return" / "auth.json" - plan = write_plan(scratch, "codex", return_files=[{"from": "/installed-agent/hermit-home/auth.json", - "to": str(returned)}]) - files = {"/installed-agent/hermit-home/auth.json": "{}"} - bound, executor.AFTER_CANCEL_S = executor.AFTER_CANCEL_S, 0.3 - try: - env = FakeEnv(hang=True, hanging=("hermit-copy", "hermit-collect"), hang_s=5, files=files) - agent = make("codex", plan) - started = time.monotonic() - raised = run(agent, env, timeout=0.5) - report["bounded"] = {"raised": raised, "seconds": round(time.monotonic() - started, 1), - "order": order(env), "returned": returned.read_text() if returned.is_file() else None, - "metadata": metadata(agent)["metadata"]} - finally: - executor.AFTER_CANCEL_S = bound - env = FakeEnv(hang=True, hanging=("hermit-copy",), files=files) - agent = make("codex", plan) - - async def cancelled_twice(): - task = asyncio.ensure_future(agent.run(INSTRUCTION, env, AgentContext())) - deadline = time.monotonic() + 30 - while not env.running and time.monotonic() < deadline: # the session runs - await asyncio.sleep(0.02) - task.cancel() # Harbor's timeout - while not any("hermit-copy" in c[2] for c in calls(env, "exec")) and time.monotonic() < deadline: - await asyncio.sleep(0.02) # the copy-back hangs - task.cancel() # the job's cancellation - started = time.monotonic() - try: - await asyncio.wait_for(task, 30) - raised = None - except BaseException as exc: # noqa: BLE001 - reported to the test - raised = [type(exc).__name__, str(exc)] - return raised, round(time.monotonic() - started, 1) - raised, seconds = asyncio.run(cancelled_twice()) - report["twice"] = {"raised": raised, "seconds": seconds, "order": order(env), - "metadata": metadata(agent)["metadata"]} - - elif check == "return_check": - # The check of a file to return in a container whose compose service had stopped (its exec fails, saying so), - # and in one where the session left no such file - stopped = {"test -f /installed-agent/hermit-home/auth.json": 'service "main" is not running\n'} - for name, env in (("failed", FakeEnv(failing=stopped, files={"/installed-agent/hermit-home/auth.json": "{}"})), - ("absent", FakeEnv())): - returned = scratch / name / "return" / "auth.json" - agent = make("codex", write_plan(scratch / name, "codex", deploy=False, return_files=[ - {"from": "/installed-agent/hermit-home/auth.json", "to": str(returned)}])) - report[name] = {"raised": run(agent, env), "downloads": calls(env, "download_file"), - "returned": returned.is_file(), "metadata": metadata(agent)["metadata"]} - - elif check == "return_limits": - # For real with bash, a stand-in CLI that leaves in its home's auth.json a file of exactly the plan's max_bytes, - # one a byte over it, or a link to a sign-in beside the home; and one that leaves a sign-in in an image whose wc - # fails (first on its PATH: as one without wc) - writer = ('#!/bin/bash\ncase "$STUB_AUTH" in\n' - ' at_limit) head -c 4096 /dev/zero | tr "\\0" x > "$CODEX_HOME/auth.json";;\n' - ' over) head -c 4097 /dev/zero | tr "\\0" x > "$CODEX_HOME/auth.json";;\n' - ' link) echo \'{"refreshed": true}\' > "$CODEX_HOME/../beside.json" && ' - 'ln -sf ../beside.json "$CODEX_HOME/auth.json";;\n' - ' *) echo \'{"refreshed": true}\' > "$CODEX_HOME/auth.json";;\nesac\n') - no_wc = scratch / "no-wc-bin" - no_wc.mkdir() - (no_wc / "wc").write_text('#!/bin/bash\necho "bash: wc: command not found" >&2\nexit 127\n') - (no_wc / "wc").chmod(0o755) - for name, path in (("at_limit", None), ("over", None), ("link", None), - ("no_wc", f"{no_wc}:{os.environ['PATH']}")): - env = ShellEnv(scratch / f"{name}-c", path=path) - (env.root / "installed-agent" / "fakebin").mkdir(parents=True) - (env.root / "installed-agent" / "fakebin" / "codex").write_text(writer) - (env.root / "installed-agent" / "fakebin" / "codex").chmod(0o755) - returned = scratch / name / "returned" / "auth.json" - plan = write_plan(scratch / name, "codex", deploy=False, path_prepend=["/installed-agent/fakebin"], - env={"CODEX_HOME": "/installed-agent/hermit-home", "STUB_AUTH": name}, - return_files=[{"from": "/installed-agent/hermit-home/auth.json", "to": str(returned), - "max_bytes": 4096}]) - agent = make("codex", plan) - report[name] = {"raised": run(agent, env, timeout=120), "downloads": calls(env, "download_file"), - "returned": returned.stat().st_size if returned.is_file() else None, - "metadata": metadata(agent)["metadata"]} - - elif check == "workdir_removed": - # A session that writes a refreshed sign-in and its last message, then removes its workdir with the deployment - # in it, whose copy_back (the adapter's memory) goes with it: the deployment's files, then its folders and the - # workdir with rmdir, which refuses one holding anything else. For real with bash: it ends by itself, or runs - # on (a child of its own) until Harbor's timeout cancels the run. Docker refuses every exec in that workdir - # from then on, and so does ShellEnv (exit 126, runc's words). - remover = ('#!/bin/bash\necho \'{"refreshed": true}\' > "$CODEX_HOME/auth.json"\n' - 'echo "the answer" > "$CODEX_HOME/last_message.txt"\n' - 'rm -f -- AGENTS.md .hermit-deploy.json .codex/hooks.json .codex/config.toml hermit/demo/ADAPTER.md ' - 'hermit/demo/memory/NOTES.md && rmdir -- .codex hermit/demo/memory hermit/demo hermit "$PWD" || ' - 'exit 9\n' - 'if [ -n "$STUB_SLEEP" ]; then sleep "$STUB_SLEEP" & echo "$!" > "$HERMIT_HOOK_LOGS/child.pid"; ' - "wait; fi\n") - - def alive(pid: int) -> bool: # the session's child, not a zombie - try: - status = Path(f"/proc/{pid}/status").read_text() - cmdline = Path(f"/proc/{pid}/cmdline").read_bytes() - except OSError: - return False - return b"sleep" in cmdline and not re.search(r"^State:\s+Z", status, re.M) - for name, sleep in (("ended", None), ("running", "120")): - env = ShellEnv(scratch / f"{name}-c") - (env.root / "installed-agent" / "fakebin").mkdir(parents=True) - (env.root / "installed-agent" / "fakebin" / "codex").write_text(remover) - (env.root / "installed-agent" / "fakebin" / "codex").chmod(0o755) - returned = scratch / name / "returned" / "auth.json" - plan = write_plan(scratch / name, "codex", path_prepend=["/installed-agent/fakebin"], - env={"CODEX_HOME": "/installed-agent/hermit-home", **({"STUB_SLEEP": sleep} if sleep - else {})}, - return_files=[{"from": "/installed-agent/hermit-home/auth.json", "to": str(returned)}]) - agent, logs = make("codex", plan), env.root / "logs" / "agent" - raised = run(agent, env, cancel_when=(logs / "child.pid").is_file) if sleep else \ - run(agent, env, timeout=120) - child, deadline = (logs / "child.pid"), time.monotonic() + 10 - while child.is_file() and alive(int(child.read_text())) and time.monotonic() < deadline: - time.sleep(0.1) - report[name] = {"raised": raised, "workdir": (env.root / "app").exists(), - "returned": returned.read_text() if returned.is_file() else None, - "copied": sorted(p.name for p in (logs / "hermit-copy").glob("*")) - if (logs / "hermit-copy").is_dir() else None, - "collected": sorted(p.name for p in (logs / "hermit-collect").glob("*")), - "child_alive": child.is_file() and alive(int(child.read_text())), - "metadata": metadata(agent)["metadata"]} - - elif check == "copy_fails": - # For real with bash, after a session of the stand-in CLI that ended by itself: a cp first on the container's - # PATH that fails part-way (FAILING_CP) in the copy-back of the adapter's memory and in the collect of - # last_message.txt and of a sessions folder, and copies the part after them in each (a folder of the - # adapter's own, the home's agents folder) as cp does: a file of mode 600 in each (the home is private), which - # only the step's chmod after the failed part makes readable in the copy - failing_bin = scratch / "failing-cp-bin" - failing_bin.mkdir() - (failing_bin / "cp").write_text(FAILING_CP) - (failing_bin / "cp").chmod(0o755) - env = ShellEnv(scratch / "c", path=f"{failing_bin}:{os.environ['PATH']}") - (env.root / "installed-agent" / "fakebin").mkdir(parents=True) - (env.root / "installed-agent" / "fakebin" / "codex").write_text(STUB_CLI) - (env.root / "installed-agent" / "fakebin" / "codex").chmod(0o755) - journal = make_deploy(scratch, "codex") / "hermit" / "demo" / "journal" - journal.mkdir() - (journal / "JOURNAL.md").write_text("# Journal\n") - (journal / "JOURNAL.md").chmod(0o600) - sessions = make_home(scratch) / "sessions" - sessions.mkdir() - (sessions / "rollout-1.jsonl").write_text('{"type": "session_meta"}\n') - home = "/installed-agent/hermit-home" - plan = write_plan(scratch, "codex", path_prepend=["/installed-agent/fakebin"], collect=[ - {"from": f"{home}/last_message.txt", "to": "last_message.txt"}, - {"from": f"{home}/sessions", "to": "sessions"}, {"from": f"{home}/agents", "to": "agents"}]) - data = json.loads(plan.read_text()) - data["deploy"]["copy_back"] = ["hermit/demo/memory", "hermit/demo/journal"] - plan.write_text(json.dumps(data)) - agent = make("codex", plan) - report["raised"] = run(agent, env, timeout=120) - logs = env.root / "logs" / "agent" - report["entries"] = sorted(p.relative_to(logs).as_posix() for d in ("hermit-copy", "hermit-collect") - for p in (logs / d).rglob("*")) - report["kept"] = {p.relative_to(logs).as_posix(): [p.read_text(), oct(p.stat().st_mode & 0o777)] - for d in ("hermit-copy", "hermit-collect") for p in sorted((logs / d).rglob("*")) - if p.is_file()} - sources = {"hermit-copy/journal/JOURNAL.md": "app/hermit/demo/journal/JOURNAL.md", # in the container - "hermit-collect/agents/reviewer.toml": "installed-agent/hermit-home/agents/reviewer.toml"} - report["source_modes"] = {rel: oct((env.root / src).stat().st_mode & 0o777) for rel, src in sources.items()} - report["metadata"] = metadata(agent)["metadata"] - - elif check == "links_planted": - # For real with bash: the agent's user (/logs/agent is writable by every user) planted links where the - # copy-back and the collect write, to folders of root's only (700, a file of 600 in each) - env = ShellEnv(scratch / "c") - fakebin = env.root / "installed-agent" / "fakebin" - fakebin.mkdir(parents=True) - (fakebin / "codex").write_text(STUB_CLI) - (fakebin / "codex").chmod(0o755) - logs = env.root / "logs" / "agent" - private = {} - for name in ("hermit-copy", "hermit-collect"): - folder = env.root / f"root-only-{name}" - folder.mkdir() - (folder / "secret.txt").write_text("root's\n") - (folder / "secret.txt").chmod(0o600) - folder.chmod(0o700) - (logs / name).symlink_to(folder, target_is_directory=True) - private[name] = folder - plan = write_plan(scratch, "codex", path_prepend=["/installed-agent/fakebin"], - collect=[{"from": "/installed-agent/hermit-home/last_message.txt", "to": "last_message.txt"}]) - agent = make("codex", plan) - report["raised"] = run(agent, env, timeout=120) - report["links"] = {name: (logs / name).is_symlink() for name in private} - report["copied_notes"] = (logs / "hermit-copy" / "memory" / "NOTES.md").read_text() \ - if (logs / "hermit-copy" / "memory" / "NOTES.md").is_file() else None - report["collected"] = sorted(p.name for p in (logs / "hermit-collect").iterdir()) \ - if (logs / "hermit-collect").is_dir() else None - report["private"] = {name: [oct(folder.stat().st_mode & 0o777), oct((folder / "secret.txt").stat().st_mode - & 0o777), sorted(os.listdir(folder))] - for name, folder in private.items()} - report["metadata"] = metadata(agent)["metadata"] - - elif check in ("shell_session", "shell_cancel", "shell_token"): - env = ShellEnv(scratch / "c") - report["root"] = str(env.root) - fakebin = env.root / "installed-agent" / "fakebin" - fakebin.mkdir(parents=True) - cli = "claude" if check == "shell_token" else "codex" - (fakebin / PROGRAM[cli]).write_text(STUB_CLI) - (fakebin / PROGRAM[cli]).chmod(0o755) - fields = {"path_prepend": ["/installed-agent/fakebin"], - "return_files": [{"from": "/installed-agent/hermit-home/auth.json", - "to": str(scratch / "returned" / "auth.json")}], - "collect": [{"from": "/installed-agent/hermit-home/last_message.txt", "to": "last_message.txt"}, - {"from": "/installed-agent/hermit-home/auth.json", "to": "auth.json"}]} - if check == "shell_session": - fields["env_from_host"] = {"FIREWORKS_API_KEY": "FIREWORKS_API_KEY", "OPENAI_API_KEY": "FIREWORKS_API_KEY"} - if check == "shell_cancel": - fields["env"] = {"CODEX_HOME": "/installed-agent/hermit-home", "STUB_SLEEP": "120"} - if check == "shell_token": - fields.update(stdin=True, command=["claude", "-p", "--output-format", "stream-json"], - env_by_fd={"CLAUDE_CODE_OAUTH_TOKEN": "CLAUDE_CODE_OAUTH_TOKEN_FILE_DESCRIPTOR"}) - plan = write_plan(scratch, cli, deploy=check != "shell_cancel", **fields) - agent = make(cli, plan) - logs = env.root / "logs" / "agent" - started = time.monotonic() - if check == "shell_cancel": # once the session has started something of its own - report["raised"] = run(agent, env, cancel_when=(logs / "child.pid").is_file) - else: - report["raised"] = run(agent, env, timeout=120) - report["seconds"] = round(time.monotonic() - started, 1) - - def listing(d: Path) -> list: - return sorted(str(p.relative_to(d)) for p in d.rglob("*")) if d.is_dir() else [] - - def text(p: Path): - return p.read_text() if p.is_file() else None - report["stream"] = text(logs / f"agent-{cli}.jsonl") - report["copied"] = listing(logs / "hermit-copy") - report["copied_notes"] = text(logs / "hermit-copy" / "memory" / "NOTES.md") - report["collected"] = listing(logs / "hermit-collect") - report["returned"] = text(scratch / "returned" / "auth.json") - report["pidfile_left"] = (env.root / "installed-agent" / "hermit" / "run" / "session.pid").exists() - if (logs / "child.pid").is_file(): # what the session started in the background - - def alive(pid: int) -> bool: - try: - status = Path(f"/proc/{pid}/status").read_text() - cmdline = Path(f"/proc/{pid}/cmdline").read_bytes() - except OSError: - return False - return b"sleep" in cmdline and not re.search(r"^State:\s+Z", status, re.M) - pid, deadline = int((logs / "child.pid").read_text()), time.monotonic() + 10 - while alive(pid) and time.monotonic() < deadline: - time.sleep(0.1) - report["child_alive"] = alive(pid) - home = env.root / "installed-agent" / "hermit-home" - report["home_modes"] = {p.name: oct(p.stat().st_mode & 0o777) for p in home.iterdir() if p.is_file()} - report["home_config"] = (home / "config.toml").read_text() - report["leaks"] = leaks(env) - report["token_sha"] = hashlib.sha256((FAKE_TOKEN + "\n").encode()).hexdigest()[:16] - report["metadata"] = metadata(agent)["metadata"] - - elif check == "secrets_refused": - plan = write_plan(scratch, "codex", deploy=False, env_from_host={"FIREWORKS_API_KEY": "FIREWORKS_API_KEY"}) - report["plain_env"] = run(make("codex", plan), PlainEnv()) - os.environ.pop("FIREWORKS_API_KEY") - env = FakeEnv() - report["missing"] = run(make("codex", plan), env) - report["missing_compose"] = calls(env, "compose") - - elif check == "launch_env": - # The plans the test rendered (jobs//launch/plan.json), each one's launch script run for real with bash - # as the exec runs it: the plan's sources by name (keys.json: fakes), as `docker compose exec -e NAME` passes - # them, a HOME without nvm, and a stand-in in the program's place that prints its environment, NUL-separated - # (the class's own session otherwise: mini-swe-agent's probes answered by FakeEnv). What the stand-in saw is - # reported with the root ShellEnv maps the container's paths under taken off. - keys = json.loads((scratch / "keys.json").read_text()) - for path in sorted(scratch.glob("jobs/*/launch/plan.json")): - name = path.parent.parent.name - env = ShellEnv(scratch / "roots" / name) - stand_in = env.root / "installed-agent" / "fakebin" / "print-env" - stand_in.parent.mkdir(parents=True) - stand_in.write_text("#!/bin/bash\nenv -0\n") - stand_in.chmod(0o755) - agent = make(json.loads(path.read_text())["cli"], path) - session = asyncio.run(agent.session(FakeEnv(), agent._plan, "/app")) - session["argv"] = ["/installed-agent/fakebin/print-env", *session["argv"][1:]] - script = env.root / "installed-agent" / "hermit" / "run" / "launch.sh" - script.parent.mkdir(parents=True) - script.write_text(env.map(agent.launch_script(agent._plan, session, "/app"))) - (script.parent / "prompt.txt").write_text(INSTRUCTION) - (env.root / "home").mkdir() - passed = {n: keys[n] for n in (agent._plan.get("env_from_host") or {}).values()} - done = subprocess.run(["bash", str(script)], capture_output=True, text=True, timeout=60, - env={"PATH": os.environ["PATH"], "HOME": str(env.root / "home"), **passed}) - stream = env.root / "logs" / "agent" / agent._plan["stream"] - seen = [e.split("=", 1) for e in (stream.read_text() if stream.is_file() else "").split("\0") if "=" in e] - report[name] = {"code": done.returncode, "said": (done.stdout + done.stderr)[-1000:], - "session_env": session["env"], - "seen": {k: v.replace(str(env.root), "") for k, v in seen}} - - elif check == "installs": - def install(cli, env, **kwargs): - agent = make(cli, **kwargs) - raised = None - try: - asyncio.run(agent.setup(env)) - except BaseException as exc: # noqa: BLE001 - reported to the test - raised = [type(exc).__name__, str(exc)] - return {"raised": raised, "mounted": agent._mounted, - "execs": [(c[1], c[2]) for c in env.calls if c[0] == "exec"], - "uploads": [c[1] for c in env.calls if c[0] == "upload_file"]} - - codex_bin = {"codex_bin": "/opt/hermit-codex/bin/codex", "version": "0.164.0"} - report["codex_mounted"] = install("codex", FakeEnv(mounted={codex_bin["codex_bin"]: "codex-cli 0.164.0\n"}), - **codex_bin) - bare = FakeEnv(mounted={codex_bin["codex_bin"]: "codex-cli 0.164.0\n"}) - answer = bare._answer - - async def no_ca_bundle(c, user): # TB2.0's bare ubuntu:24.04 has none at all - if c.startswith("[ -s /etc/ssl/certs/ca-certificates.crt ]"): - return ExecResult(stdout="", stderr="", return_code=1) - return await answer(c, user) - bare._answer = no_ca_bundle - report["codex_bare"] = install("codex", bare, **codex_bin) - report["codex_other"] = install("codex", FakeEnv(mounted={codex_bin["codex_bin"]: "codex-cli 0.160.0\n"}), - **codex_bin) - report["claude_mounted"] = install("claude", FakeEnv(mounted={"/opt/hermit-claude/claude": - "2.1.318 (Claude Code)\n"}), - version="2.1.318", claude_bin="/opt/hermit-claude/claude") - report["claude"] = install("claude", FakeEnv(), version="2.1.318") - report["pi"] = install("pi", FakeEnv(), version="1.5.0") - report["pi_mounted"] = install("pi", FakeEnv(mounted={"/opt/hermit-pi/pi": "1.5.0\n"}), version="1.5.0", - pi_bin="/opt/hermit-pi/pi") - # ... among other lines before and after its own (Codex's parse takes the first line, Pi's the last) - report["codex_mounted_noisy"] = install("codex", FakeEnv(noise=True, mounted={ - codex_bin["codex_bin"]: "codex-cli 0.164.0\n"}), **codex_bin) - report["pi_mounted_noisy"] = install("pi", FakeEnv(noise=True, mounted={"/opt/hermit-pi/pi": "1.5.0\n"}), - version="1.5.0", pi_bin="/opt/hermit-pi/pi") - report["opencode"] = install("opencode", FakeEnv(), version="2.2.6") - report["opencode_bare"] = install("opencode", FakeEnv(missing={"curl", "wget", "python3"}), version="2.2.6") - report["opencode_mounted"] = install("opencode", FakeEnv(mounted={"/opt/hermit-opencode/opencode": "2.2.6\n"}), - version="2.2.6", opencode_bin="/opt/hermit-opencode/opencode") - report["dsh"] = install("dsh", FakeEnv(), version="0.2.2-rc.1") - report["dsh_mounted"] = install("dsh", FakeEnv(mounted={"/opt/hermit-dsh/dsh": "dsh 0.2.2-rc.1\n"}), - version="0.2.2-rc.1", dsh_bin="/opt/hermit-dsh/dsh") - report["mini"] = install("mini-swe-agent", FakeEnv(), version="2.4.6") - report["setup_timeouts"] = {cli: agent_class(cli).DEFAULT_SETUP_TIMEOUT_SEC for cli in CLASSES} - # The image tests of the OpenCode and dsh scripts, run with bash under the pipefail Harbor's exec_as_root and - # exec_as_agent put first, a stand-in ldd as each libc's answers (musl's on stderr, then exit 1) and a curl - # that fails. - fakebin = scratch / "fakebin" - fakebin.mkdir() - (fakebin / "curl").write_text(f"#!/bin/sh\necho \"curl $*\" >> {scratch}/curl.log\nexit 1\n") - (fakebin / "curl").chmod(0o755) - opencode = make("opencode", version="2.2.6").install_command() - url_only = opencode[:opencode.index('rm -rf "$d"')] + 'echo "$url"\n' # no download: the URL picked - dsh = make("dsh", version="0.2.2-rc.1").install_command() - report["musl_test"] = {} - for libc, ldd in (("musl", "echo 'musl libc (x86_64)\nVersion 1.2.4' >&2; exit 1"), - ("glibc", "echo 'ldd (GNU libc) 2.36'")): - (fakebin / "ldd").write_text(f"#!/bin/sh\n{ldd}\n") - (fakebin / "ldd").chmod(0o755) - env = {"PATH": f"{fakebin}:/usr/bin:/bin", "HOME": str(scratch / f"home-{libc}")} - got = subprocess.run(["bash", "-c", f"set -o pipefail; if {executor.MUSL}; then echo musl; else echo " - "glibc; fi"], capture_output=True, text=True, env=env, timeout=60) - report["musl_test"][libc] = got.stdout.strip() - got = subprocess.run(["bash", "-c", "set -o pipefail; " + url_only], capture_output=True, text=True, - env=env, timeout=60) - report[f"opencode_url_{libc}"] = got.stdout.strip() - (scratch / "curl.log").unlink(missing_ok=True) - got = subprocess.run(["bash", "-c", "set -o pipefail; " + dsh], capture_output=True, text=True, env=env, - timeout=60) - curl_log = scratch / "curl.log" - report[f"dsh_{libc}"] = {"code": got.returncode, "stderr": got.stderr[-400:], - "curl": curl_log.read_text() if curl_log.exists() else ""} - # A mount whose binary did not serve is left off PATH, so that the installed CLI is the one found. - plan = write_plan(scratch / "mount", "codex", deploy=False, path_prepend=["/opt/hermit-codex/bin", "/x/bin"]) - agent = make("codex", plan, **codex_bin) - session = {"argv": ["codex"], "stream": "agent-codex.jsonl"} - report["path_unmounted"] = [line for line in agent.launch_script(agent._plan, session, "/app").splitlines() - if line.startswith("export PATH")] - agent._mounted = True - report["path_mounted"] = [line for line in agent.launch_script(agent._plan, session, "/app").splitlines() - if line.startswith("export PATH")] - - elif check == "install_failures": - # The setup Harbor runs before the trial's agent phase: dsh refusing a musl image, Pi's npm install failing, - # and Pi's npm install hanging until Harbor's setup timeout cancels it - musl = "hermit: dsh needs Node 24.9 or later from nvm, whose builds need glibc, and this image is musl\n" - npm = ("npm error code ETARGET\nnpm error notarget No matching version found for " - "@earendil-works/pi-coding-agent@1.5.0.\n") - for name, cli, version, env, timeout in ( - ("dsh_musl", "dsh", "0.2.2-rc.1", FakeEnv(failing={"@deepseek-ai/dsh": musl}), None), - ("pi_npm", "pi", "1.5.0", FakeEnv(failing={"pi-coding-agent": npm}), None), - ("pi_setup_timeout", "pi", "1.5.0", FakeEnv(hanging=("pi-coding-agent",)), 0.5)): - work = make(cli, version=version).setup(env) - try: - asyncio.run(work if timeout is None else asyncio.wait_for(work, timeout)) - report[name] = None - except BaseException as exc: # noqa: BLE001 - reported to the test - report[name] = [type(exc).__name__, str(exc)] - - elif check == "mount_rejected": - # The plan as hermit.harbor.launch renders it for a mounted codex_bin: the mount's path is the command's - # program and its folder first on PATH. The mount answers another version here, or is the asked-for one; or is - # the asked-for one that only root may run (mode 700, a uid of this machine's) in a task whose agent user is - # not root, which Harbor makes the default user for the install as for the session (with_default_user); or is - # the asked-for one, which that user may run, under a HOME that user cannot write (Codex warns first there) - codex_bin = "/opt/hermit-codex/bin/codex" - for name, said, user in (("rejected", "codex-cli 0.160.0\n", None), ("served", "codex-cli 0.164.0\n", None), - ("root_only", "codex-cli 0.164.0\n", "agent"), - ("home_unwritable", "codex-cli 0.164.0\n", "agent")): - plan = write_plan(scratch / name, "codex", deploy=False, nvm=True, path_prepend=["/opt/hermit-codex/bin"], - command=[codex_bin, "exec", "-C", "@@HERMIT_WORKDIR@@", "@@HERMIT_PROMPT@@"]) - env = FakeEnv(mounted={codex_bin: said}, root_only=[codex_bin] if name == "root_only" else (), - warns=[codex_bin] if name == "home_unwritable" else ()) - env.default_user = user - agent = make("codex", plan, codex_bin=codex_bin, version="0.164.0") - asyncio.run(agent.setup(env)) - installed = any("npm install -g @openai/codex@0.164.0" in c[2] for c in calls(env, "exec")) - # what the mount was asked, and as whom (None: Harbor's default user) - asked = [[c[1], "version" if "--version" in c[2] else "test -x"] for c in calls(env, "exec") - if f"{codex_bin} --version" in c[2] or c[2] == f"test -x {codex_bin}"] - report[name] = {"raised": run(agent, env), "mounted": agent._mounted, "installed": installed, - "asked": asked, "launch": uploaded(env).get("/installed-agent/hermit/run/launch.sh")} - - elif check == "codex_nvm": - # Harbor's Codex install on a glibc image: codex under the agent user's nvm, none on the image's PATH (its - # link into /usr/local/bin, made without nvm, finds none). For real with bash: that user's HOME, whose nvm.sh - # puts its node's bin (a stand-in codex there) on PATH, a container PATH without codex, a plan that loads nvm - # (as hermit.harbor.launch renders Codex's); and then a mounted codex as well (the plan's program, its folder - # on the plan's PATH, as launch renders a mounted codex_bin's), the mount having served. - stand_in = '#!/bin/bash\necho "ran: $0"\necho "codex on PATH: $(command -v codex)"\n' - mount = "/opt/hermit-codex/bin/codex" - for name in ("installed", "mounted"): - env = ShellEnv(scratch / f"{name}-c", path="/usr/bin:/bin", home=scratch / f"{name}-c" / "home" / "agent") - (env.home / ".nvm").mkdir(parents=True) - (env.home / ".nvm" / "nvm.sh").write_text('export PATH="$NVM_DIR/versions/node/v22.12.0/bin:$PATH"\n') - programs = [env.home / ".nvm" / "versions" / "node" / "v22.12.0" / "bin" / "codex"] - fields, kwargs = {}, {} - if name == "mounted": - programs.append(Path(env.map(mount))) - fields = {"command": [mount, "exec", "-C", "@@HERMIT_WORKDIR@@", "@@HERMIT_PROMPT@@"], - "path_prepend": ["/opt/hermit-codex/bin"]} - kwargs = {"codex_bin": mount, "version": "0.164.0"} - for program in programs: - program.parent.mkdir(parents=True, exist_ok=True) - program.write_text(stand_in) - program.chmod(0o755) - agent = make("codex", write_plan(scratch / name, "codex", deploy=False, nvm=True, **fields), **kwargs) - agent._mounted = name == "mounted" # as the install leaves it when the mount served - stream = env.root / "logs" / "agent" / "agent-codex.jsonl" - report[name] = {"raised": run(agent, env, timeout=120), "programs": [str(p) for p in programs], - "stream": stream.read_text() if stream.is_file() else None} - - elif check == "mini_swe_agent": - bundle = make_bundle(scratch / "bundle") - settings = {"model": "accounts/fireworks/models/kimi", "effort": "medium", - "binding": {"litellm": "fireworks_ai/accounts/fireworks/models/kimi", - "base_url": "https://api.fireworks.ai/inference/v1", - "api_key_env": "FIREWORKS_API_KEY"}} - for name, imports, noise in (("bundle", True, False), ("no_import", False, False), ("noisy", True, True)): - base = scratch / name - plan = write_plan(base, "mini-swe-agent", stdin=True, - command=["python3", "-P", "-m", # hermit.harbor.launch's `bin` for it - "hermit.agent.base.mini_swe_agent_run", "--settings", - "/installed-agent/hermit-home/mini-swe-agent.json", "--cwd", - "@@HERMIT_WORKDIR@@", "--last", "/installed-agent/hermit-home/last_message.txt", - "--stdin"], - bundle={"host_dir": str(bundle), "container_dir": "/installed-agent/bundle"}, - env={"MSWEA_GLOBAL_CONFIG_DIR": "/installed-agent/hermit-home", - "PYTHONPATH": "/installed-agent/bundle:@@HERMIT_WORKDIR@@/lib", - "HERMIT_TOOLS_URL": "http://10.100.0.1:5000/tok"}, - env_from_host={"FIREWORKS_API_KEY": "FIREWORKS_API_KEY"}, - tools_script={"url_env": "HERMIT_TOOLS_URL"}) - (base / "launch" / "home" / "mini-swe-agent.json").write_text(json.dumps(settings)) - env = FakeEnv(bundle_ok=imports, noise=noise) - agent = make("mini-swe-agent", plan) - raised = run(agent, env) - files = uploaded(env) - report[name] = {"raised": raised, "launch": files.get("/installed-agent/hermit/run/launch.sh"), - "tool": files.get("/installed-agent/hermit/bin/hermit-tool"), - "bundle_upload": [c[1] for c in calls(env, "upload_dir")], - "probe": [(c[2], c[3]) for c in calls(env, "exec") if "import hermit" in c[2]], - "leaks": leaks(env)} - - elif check == "tool_script": - seen = [] - # What GET //tools is answered with: an HTTP error, as a JSON object saying so (its "error", FastAPI's - # "detail"), a JSON string, a number; never the tools' list - not_tools = {"gone": ({"error": "no such path"}, 404), "detail": ({"detail": "Not Found"}, 404), - "text": ("Bad Gateway", 502), "number": (5, 500)} - # ... or with a list that holds no tools: items that are not objects, an object without a name - not_tool_items = {"strings": ["lookup"], "nameless": [{"description": "Finds a row."}]} - - class Tools(http.server.BaseHTTPRequestHandler): - def log_message(self, *args): - pass - - def _send(self, body, status=200): - data = json.dumps(body).encode() - self.send_response(status) - self.send_header("Content-Length", str(len(data))) - self.end_headers() - self.wfile.write(data) - - def do_GET(self): - name = self.path.split("/")[1] - if self.path == "/tok/tools": - self._send([{"name": "lookup", "description": "Finds a row."}]) - elif name in not_tool_items: # //tools: not_tool_items[name], with a 200 - self._send(not_tool_items[name]) - else: # //tools: not_tools[name] - self._send(*not_tools[name]) - - def do_POST(self): - body = json.loads(self.rfile.read(int(self.headers["Content-Length"]))) - seen.append((self.path, body)) - self._send({"result": {"text": f"row {body.get('id')}"}}) - - server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Tools) - threading.Thread(target=server.serve_forever, daemon=True).start() - script = scratch / "hermit-tool" - script.write_text(executor.TOOL_SCRIPT.replace("__PYTHON__", sys.executable) - .replace("__URL_ENV__", "HERMIT_TOOLS_URL")) - script.chmod(0o755) - env = {"PATH": "/usr/bin:/bin", "HERMIT_TOOLS_URL": f"http://127.0.0.1:{server.server_address[1]}/tok"} - for name, args in (("list", []), ("call", ["lookup", '{"id": 7}']), ("unknown", ["nope", "{}"]), - ("bad_json", ["lookup", "{oops"]), ("too_many", ["lookup", "{}", "x"])): - got = subprocess.run([str(script), *args], capture_output=True, text=True, env=env, timeout=60) - report[name] = [got.returncode, got.stdout.strip()] - got = subprocess.run([str(script)], capture_output=True, text=True, env={"PATH": "/usr/bin:/bin"}, timeout=60) - report["no_url"] = [got.returncode, got.stdout.strip()] - # URLs whose GET /tools the server answers with an HTTP error, each in a shape of not_tools - report["list_error"] = {} - for name in not_tools: - got = subprocess.run([str(script)], capture_output=True, text=True, timeout=60, env={ - **env, "HERMIT_TOOLS_URL": f"http://127.0.0.1:{server.server_address[1]}/{name}"}) - report["list_error"][name] = [got.returncode, got.stdout.strip(), got.stderr.strip()[-300:]] - report["list_not_tools"] = {} - for name in not_tool_items: - got = subprocess.run([str(script)], capture_output=True, text=True, timeout=60, env={ - **env, "HERMIT_TOOLS_URL": f"http://127.0.0.1:{server.server_address[1]}/{name}"}) - report["list_not_tools"][name] = [got.returncode, got.stdout.strip(), got.stderr.strip()[-300:]] - report["seen"] = seen - server.shutdown() - server.server_close() - - elif check == "answer": - import logging - from harbor.agents.capabilities import AgentCapabilities - from harborside import answer as answer_module - HermitAnswer = answer_module.HermitAnswer - answer_module.POLL_S = 0.05 - for name in ("http_proxy", "HTTP_PROXY", "https_proxy", "HTTPS_PROXY"): - os.environ[name] = "http://127.0.0.1:9" # a proxy that is not there: an endpoint here is reached - # directly - os.environ.pop("no_proxy", None) - os.environ.pop("NO_PROXY", None) - requests, replies = [], [] - - class Model(http.server.BaseHTTPRequestHandler): - def log_message(self, *args): - pass - - def do_POST(self): - requests.append((self.path, json.loads(self.rfile.read(int(self.headers["Content-Length"]))))) - status, payload = replies.pop(0) - data = json.dumps(payload).encode() - self.send_response(status) - self.send_header("Content-Length", str(len(data))) - self.end_headers() - self.wfile.write(data) - - server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Model) - threading.Thread(target=server.serve_forever, daemon=True).start() - endpoint = f"http://127.0.0.1:{server.server_address[1]}/endpoint-token-1f2e3d" - - def answer_agent(name, **spec): - base = scratch / name - base.mkdir(parents=True, exist_ok=True) - plan = {"version": 1, "cli": "answer", "label": "answer", "metadata": {"plan_digest": "d1"}, - "answer": {"endpoint": endpoint, "model": "kimi", "system": "Context: tables are in /app/data.", - "instruction_file": None, "wait_s": 5, "attachments": [], "answer_file": None, - "answer_pattern": None, "answer_format": "{answer}", "max_tokens": None, - "effort": None, **spec}} - (base / "plan.json").write_text(json.dumps(plan)) - return HermitAnswer(logs_dir=base / "logs", plan=str(base / "plan.json")) - - def answer_run(agent, env, instruction="What is 6 x 7?"): - context = AgentContext() - try: - asyncio.run(agent.run(instruction, env, context)) - except BaseException as exc: # noqa: BLE001 - reported to the test - return [type(exc).__name__, str(exc)], context.model_dump() - return None, context.model_dump() - - png = b"\x89PNG\r\n\x1a\nfake" - env = FakeEnv(files={"/app/instruction.md": "Which panel shows the knockdown?\n", "/app/fig.png": png, - "/app/notes.txt": b"not an image"}, - misses=2, listing="/app/fig.png\n/app/notes.txt\n/app/fig.png\n") - replies.append((200, {"model": "kimi-served", "choices": [{"message": {"role": "assistant", "content": - "First [ANSWER]A[/ANSWER], then on reflection [ANSWER] C [/ANSWER]."}}], - "usage": {"prompt_tokens": 120, "completion_tokens": 9, - "prompt_tokens_details": {"cached_tokens": 100}}})) - full = answer_agent("full", instruction_file="/app/instruction.md", attachments=["/app/*.png", "/app/*.txt"], - answer_file="/app/answer.txt", answer_pattern=r"\[ANSWER\](.*?)\[/ANSWER\]", - answer_format="[ANSWER]{answer}[/ANSWER]", max_tokens=256, effort="high") - report["raised"], report["context"] = answer_run(full, env, "Read /app/instruction.md and answer.") - report["request"] = requests[-1] - writes = [c[2] for c in calls(env, "exec") if "base64 -d" in c[2]] - report["writes"] = writes - report["written"] = base64.b64decode(re.search(r"printf '%s' (\S+) \|", writes[0]).group(1).strip("'") - ).decode() if writes else None - report["write_users"] = [c[1] for c in calls(env, "exec") if "base64 -d" in c[2]] - report["answer_json"] = json.loads((scratch / "full" / "logs" / "answer.json").read_text()) - report["polls"] = sum(1 for c in calls(env, "exec") if c[2].startswith("test -f /app/instruction.md")) - report["png_url"] = "data:image/png;base64," + base64.b64encode(png).decode() - # No match of the pattern: the whole reply; no answer file: nothing written; Harbor's instruction asked. - replies.append((200, {"choices": [{"message": {"content": [{"type": "text", "text": "Just 42."}]}}]})) - plain, env = answer_agent("plain", answer_pattern=r"\[ANSWER\](.*)"), FakeEnv() - report["plain_raised"], _ = answer_run(plain, env) - report["plain_request"] = requests[-1][1] - report["plain_answer"] = json.loads((scratch / "plain" / "logs" / "answer.json").read_text()) - report["plain_execs"] = [c[2] for c in calls(env, "exec")] - # The model failed: NonZeroAgentExitCodeError (the verifier still runs), the endpoint's token in no message. - replies.append((500, {"error": {"message": "upstream exploded"}})) - report["failed"], _ = answer_run(plain, FakeEnv()) - report["late"], _ = answer_run(answer_agent("late", instruction_file="/app/instruction.md", wait_s=0.2), - FakeEnv()) - # What is read from the container (the question, a figure's size and bytes) comes with other lines before and - # after it, as Harbor's docker exec merges stderr into stdout: for real with bash - noisy = ShellEnv(scratch / "noisy-c", noise=True) - (noisy.root / "app" / "instruction.md").write_text("Which panel shows the knockdown?\n") - (noisy.root / "app" / "fig.png").write_bytes(png) - replies.append((200, {"choices": [{"message": {"content": "[ANSWER]B[/ANSWER]"}}]})) - report["noisy_raised"], _ = answer_run(answer_agent( - "noisy", instruction_file="/app/instruction.md", attachments=["/app/*.png"], answer_file="/app/answer.txt", - answer_pattern=r"\[ANSWER\](.*?)\[/ANSWER\]"), noisy) - report["noisy_request"] = requests[-1][1] if report["noisy_raised"] is None else None - answered = noisy.root / "app" / "answer.txt" - report["noisy_written"] = answered.read_text() if answered.is_file() else None - # An image without base64 (for real with bash, its PATH without one): the question and a figure - nobin = scratch / "no-base64-bin" - nobin.mkdir() - for tool in ("bash", "tr", "wc"): - (nobin / tool).symlink_to(shutil.which(tool)) - bare = ShellEnv(scratch / "no-base64-c", path=str(nobin)) - (bare.root / "app" / "instruction.md").write_text("Which panel shows the knockdown?\n") - (bare.root / "app" / "fig.png").write_bytes(png) - asked = len(requests) - replies.extend([(200, {"choices": [{"message": {"content": "[ANSWER]B[/ANSWER]"}}]})] * 2) # if it asks - report["no_base64_question"], _ = answer_run(answer_agent( - "no_base64_question", instruction_file="/app/instruction.md"), bare) - report["no_base64_figure"], _ = answer_run(answer_agent("no_base64_figure", attachments=["/app/*.png"]), bare) - report["no_base64_asked"] = len(requests) - asked - - def figure_run(name, env): - """A run whose one figure cannot be listed, measured or read: what it raised, and how often the model was - asked.""" - asked = len(requests) - replies.append((200, {"choices": [{"message": {"content": "[ANSWER]B[/ANSWER]"}}]})) # if it asks - raised, _ = answer_run(answer_agent(name, attachments=["/app/*.png"]), env) - return {"raised": raised, "asked": len(requests) - asked} - - # A figure that cannot be measured or read is never left out: for real with bash (in the C locale, as the test - # matches what bash says), an image without wc (its PATH without one) and a figure of mode 000; on FakeEnv's - # stand-ins, a read whose exec failed otherwise (compose's service stopped), a measure that exits 0 without its - # line or with one whose value is no number, and a read that exits 0 without its line - nowc = scratch / "no-wc-bin" - nowc.mkdir() - for tool in ("bash", "base64", "tr"): - (nowc / tool).symlink_to(shutil.which(tool)) - unmeasured = ShellEnv(scratch / "no-wc-c", path=str(nowc), locale="C") - (unmeasured.root / "app" / "fig.png").write_bytes(png) - report["no_wc_figure"] = figure_run("no_wc_figure", unmeasured) - locked = ShellEnv(scratch / "locked-c", locale="C") - (locked.root / "app" / "fig.png").write_bytes(png) - (locked.root / "app" / "fig.png").chmod(0) - report["locked_figure"] = figure_run("locked_figure", locked) - (locked.root / "app" / "fig.png").chmod(0o644) - stopped = FakeEnv(files={"/app/fig.png": png}, listing="/app/fig.png\n", - failing={"hermit-data:": 'service "main" is not running\n'}) # docker compose exec - report["failed_read_figure"] = figure_run("failed_read_figure", stopped) - for name, part, said in (("unmarked_size_figure", "hermit-size:", "something else\n"), - ("odd_size_figure", "hermit-size:", "hermit-size:12 abc\n"), - ("unmarked_data_figure", "hermit-data:", "something else\n")): - report[name] = figure_run(name, FakeEnv(files={"/app/fig.png": png}, listing="/app/fig.png\n", - answers={part: (0, said)})) - # Nor are the figures left out when the exec that lists the globs failed (compose's service stopped) - unlisted = FakeEnv(files={"/app/fig.png": png}, listing="/app/fig.png\n", - failing={"shopt -s nullglob": 'service "main" is not running\n'}) - report["failed_listing"] = figure_run("failed_listing", unlisted) - # One over MAX_ATTACHMENT_BYTES (a sparse file a byte over) is left out with its warning, and the model is - # asked with the others: for real with bash - sized = ShellEnv(scratch / "over-limit-c") - (sized.root / "app" / "fig.png").write_bytes(png) - with open(sized.root / "app" / "huge.png", "wb") as huge: - huge.truncate(answer_module.MAX_ATTACHMENT_BYTES + 1) - over, warned, catch = answer_agent("over_limit", attachments=["/app/*.png"]), [], logging.Handler() - catch.emit = lambda record: warned.append(record.getMessage().replace(str(sized.root), "")) - over.logger.addHandler(catch) - replies.append((200, {"choices": [{"message": {"content": "[ANSWER]B[/ANSWER]"}}]})) - report["over_limit_raised"], _ = answer_run(over, sized) - over.logger.removeHandler(catch) - report["over_limit_content"] = requests[-1][1]["messages"][-1]["content"] - report["over_limit_warned"], report["max_bytes"] = warned, answer_module.MAX_ATTACHMENT_BYTES - # A glob that matches no file is no failure: the model is asked once, with the plain question (for real with - # bash, an /app that holds no figure) - unmatched = ShellEnv(scratch / "no-match-c") - asked = len(requests) - replies.append((200, {"choices": [{"message": {"content": "[ANSWER]B[/ANSWER]"}}]})) - report["no_match_raised"], _ = answer_run(answer_agent("no_match", attachments=["/app/*.png"]), unmatched) - report["no_match_asked"] = len(requests) - asked - report["no_match_content"] = requests[-1][1]["messages"][-1]["content"] if report["no_match_asked"] else None - # The model answered, and this machine could not then run the exec that writes the answer into the task (its - # compose client not spawned: an OSError from the exec itself, EMFILE, EAGAIN on a loaded host), or could not - # keep the answer in answer.json (its disk quota reached); and, for contrast, a write the container refused - refused = "mkdir: cannot create directory '/app': Permission denied\n" - for name, env in (("write_unspawned", FakeEnv(raising={"base64 -d": errno.EMFILE})), - ("write_unspawned_eagain", FakeEnv(raising={"base64 -d": errno.EAGAIN})), - ("write_refused", FakeEnv(failing={"base64 -d": refused}))): - replies.append((200, {"choices": [{"message": {"content": "[ANSWER]B[/ANSWER]"}}]})) - raised, _ = answer_run(answer_agent(name, answer_file="/app/answer.txt"), env) - report[name] = {"raised": raised, "kept": (scratch / name / "logs" / "answer.json").is_file()} - - class QuotaReached: - """A host whose disk quota is reached: the answer's own file in the logs directory cannot be made.""" - - @staticmethod - def mkstemp(*args, **kwargs): - raise OSError(errno.EDQUOT, os.strerror(errno.EDQUOT)) - unkept, env = answer_agent("unkept", answer_file="/app/answer.txt"), FakeEnv() - kept_tempfile, answer_module.tempfile = answer_module.tempfile, QuotaReached - replies.append((200, {"choices": [{"message": {"content": "[ANSWER]B[/ANSWER]"}}]})) - try: - raised, _ = answer_run(unkept, env) - finally: - answer_module.tempfile = kept_tempfile - report["unkept"] = {"raised": raised, "logs": str(unkept.logs_dir), - "writes": [c[2] for c in calls(env, "exec") if "base64 -d" in c[2]]} - # a link the task's container left at answer.json (its logs folder is the container's /logs/agent) - planted, env = answer_agent("planted"), FakeEnv() - outside = scratch / "planted-outside.txt" - outside.write_text("a file of this machine\n") - planted.logs_dir.mkdir(parents=True, exist_ok=True) - (planted.logs_dir / "answer.json").symlink_to(outside) - replies.clear() # (an earlier case's spare reply, queued in case it asked) - replies.append((200, {"choices": [{"message": {"content": "B"}}]})) - raised, _ = answer_run(planted, env) - kept = planted.logs_dir / "answer.json" - report["planted"] = {"raised": raised, "outside": outside.read_text(), "link": kept.is_symlink(), - "answer": json.loads(kept.read_text()).get("answer") if kept.is_file() else None, - "left": sorted(p.name for p in planted.logs_dir.iterdir())} - replies.clear() - server.shutdown() - server.server_close() - report["unreached"], _ = answer_run(plain, FakeEnv()) - report["error_class"] = NonZeroAgentExitCodeError.__name__ - report["capabilities"] = HermitAnswer.capabilities == AgentCapabilities() - report["name"] = HermitAnswer.name() - bad = scratch / "bad" - bad.mkdir() - (bad / "plan.json").write_text(json.dumps({"version": 1, "cli": "answer", "answer": { - "endpoint": "ftp://x", "model": "", "attachments": ["/app/a b.png"], "answer_format": "no slot", - "answer_pattern": "(", "max_tokens": 0}})) - try: - HermitAnswer(logs_dir=bad / "logs", plan=str(bad / "plan.json")) - report["bad"] = "accepted" - except ValueError as exc: - report["bad"] = str(exc) - - elif check == "options": - from harbor.agents.factory import AgentFactory - from harbor.models.trial.config import AgentConfig - plan = write_plan(scratch, "codex", deploy=False) - report["classes"] = {} - for cli in CLASSES: - cls = agent_class(cli) - refused = {} - for key, kwargs in (("bogus", {"bogus": 1}), ("deploy_dir", {"deploy_dir": "/x"}), - ("missing_plan", {"plan": str(scratch / "nowhere.json")})): - try: - cls.parse_options(kwargs, {}) - refused[key] = "accepted" - except ValueError as exc: - refused[key] = str(exc) - report["classes"][cli] = {"refused": refused, "name": cls.name(), "import_path": cls.import_path(), - "bin_option": bool(cls.BIN_OPTION) and cls.BIN_OPTION in - cls.options_model.model_fields} - for name, change in (("version", {"version": 2}), ("cli", {"cli": "claude"}), - ("stream", {"stream": "../escape.jsonl"}), - ("secret_path", {"home": {"host_dir": str(scratch / "launch" / "home"), - "secret_files": ["../auth.json"]}}), - ("env_name", {"env_from_host": {"NOT A NAME": "X"}})): - (scratch / f"{name}.json").write_text(json.dumps({**json.loads(plan.read_text()), **change})) - try: - make("codex", scratch / f"{name}.json") - report[name] = "accepted" - except ValueError as exc: - report[name] = str(exc) - try: - make("codex", plan, base_url="https://api.example.com/v1") - report["base_url"] = "accepted" - except ValueError as exc: - report["base_url"] = str(exc) - try: # no key in Harbor's environment for this provider - os.environ.pop("FIREWORKS_API_KEY") - AgentFactory.run_preflight(AgentConfig(import_path="harborside.codex:HermitCodex", - model_name="fireworks_ai/accounts/fireworks/models/kimi", - kwargs={"plan": str(plan)})) - report["preflight"] = "passed" - except Exception as exc: # noqa: BLE001 - reported to the test - report["preflight"] = f"{type(exc).__name__}: {exc}" - connection = make("codex", plan).model_connection - report["connection"] = [connection.provider, connection.inferred_base_url, connection.api_key] - report["no_plan"] = run(make("codex"), FakeEnv()) - - elif check == "scripted": - from harborside.scripted import ScriptedAgent - logs = scratch / "slogs" - agent = ScriptedAgent(logs_dir=logs, script="echo hi > /app/answer.txt", timeout_sec=5) - env = FakeEnv() - answer = env._answer - - async def script_ran(c, user): - if c.startswith("bash -c"): - return ExecResult(stdout="script ran\n", stderr="", return_code=3) - return await answer(c, user) - env._answer = script_ran - context = AgentContext() - asyncio.run(agent.setup(env)) - asyncio.run(agent.run("Write hi.", env, context)) - report["uploads"] = [(c[1], c[2]) for c in calls(env, "upload_file")] - report["execs"] = [(c[1], c[2], c[3]) for c in env.calls if c[0] == "exec"] - report["script_txt"] = (logs / "script.txt").read_text() - report["context"] = context.model_dump() - report["name"], report["exit_code"] = agent.name(), agent.exit_code - try: - ScriptedAgent(logs_dir=logs, script="x", bogus=1) - report["unknown"] = "accepted" - except ValueError as exc: - report["unknown"] = str(exc) - - elif check == "driver_protocol": - from harborside.driver import StepDriverTrial - trial = StepDriverTrial.__new__(StepDriverTrial) # the request handling alone, no Harbor trial behind it - trial.commands = asyncio.Queue() - trial.prepared, trial.done = asyncio.Event(), asyncio.Event() - trial.error, trial.running, trial.next_index = None, None, 1 - trial.aborted, trial.loop_done, trial.finished = None, False, False - trial.replies_dir = scratch / "replies" - trial.replies_dir.mkdir() - trial.paths = SimpleNamespace(trial_dir=scratch / "trial") - - async def scenario(): - out = {"before_ready": await trial.handle({"op": "status"})} - trial.prepared.set() - out["ready"] = await trial.handle({"op": "status"}) - out["out_of_order"] = await trial.handle({"op": "step", "name": "s2", "index": 2}) - trial.running = "s1" - out["busy"] = await trial.handle({"op": "step", "name": "s1", "index": 1}) - trial.running = None - (trial.replies_dir / "1.json").write_text(json.dumps({"ok": True, "index": 1})) - out["stored"] = await trial.handle({"op": "result", "index": 1}) - out["missing"] = await trial.handle({"op": "result", "index": 7}) - # A step request is queued for the loop; the loop answers through the future. - task = asyncio.ensure_future(trial.handle({"op": "step", "name": "s1", "index": 1})) - await asyncio.sleep(0) - command, future = await trial.commands.get() - future.set_result({"ok": True, "index": 1, "step": {}}) - out["queued"] = (command["name"], await task) - trial.error = "SandboxBuildFailedError: no image" - out["errored"] = await trial.handle({"op": "step", "name": "s1", "index": 1}) - trial.done.set() - out["stop_after_error"] = await trial.handle({"op": "stop"}) - trial.error, trial.loop_done = None, True - out["finished"] = await trial.handle({"op": "step", "name": "s1", "index": 1}) - out["unknown"] = await trial.handle({"op": "nope"}) - return out - report = asyncio.run(scenario()) - - elif check == "driver_plan": - from harborside.driver import StepDriverTrial - first = write_plan(scratch / "one", "codex", deploy=False) - second = write_plan(scratch / "two", "codex", deploy=False, command=["codex", "exec", "second"]) - trial = StepDriverTrial.__new__(StepDriverTrial) # the step's _apply alone, no Harbor trial behind it - trial.agent = make("codex", first) - trial._apply({"name": "s1", "agent_kwargs": {"plan": str(second)}}) # what the stream client sends - report["from_kwargs"] = trial.agent._plan["command"] - trial._apply({"name": "s2", "plan": str(first)}) - report["from_request"] = trial.agent._plan["command"] - trial._apply({"name": "s3", "agent_kwargs": {}}) - report["kept"] = trial.agent._plan["command"] - trial.agent = SimpleNamespace() # an agent with no plan: the step runs as configured - trial._apply({"name": "s4", "plan": str(first)}) - report["no_plan_agent"] = "ran" - # the hosts each step's agent phase may reach (Harbor's extra_allowed_hosts, normalised as Harbor does): a - # request that names none keeps them, one that names an empty list has none; a host Harbor refuses fails it - trial.config = SimpleNamespace(agent=SimpleNamespace(extra_allowed_hosts=None)) - trial._apply({"name": "s5", "allowed_hosts": ["API.Example.com", "10.0.0.0/8"]}) - report["hosts"] = trial.config.agent.extra_allowed_hosts - trial._apply({"name": "s6"}) - report["hosts_kept"] = trial.config.agent.extra_allowed_hosts - trial._apply({"name": "s7", "allowed_hosts": []}) - report["hosts_none"] = trial.config.agent.extra_allowed_hosts - try: - trial._apply({"name": "s8", "allowed_hosts": ["https://models.example/v1"]}) - except Exception as exc: # noqa: BLE001 - its type is what is reported - report["bad_hosts"] = f"{type(exc).__name__}: {exc}" - - elif check == "driver_loop": - import logging - from harborside.driver import StepDriverTrial - good = write_plan(scratch / "good", "codex", deploy=False) - bad = scratch / "bad.json" # a plan this agent cannot take: another CLI's - bad.write_text(json.dumps({**json.loads(good.read_text()), "cli": "claude"})) - trial = StepDriverTrial.__new__(StepDriverTrial) # the step loop alone; Harbor's step stands in - trial.commands, trial.logger = asyncio.Queue(), logging.getLogger("driver_loop") - trial.running, trial.next_index, trial.aborted, trial.loop_done, trial.finished = None, 1, None, False, False - trial.replies_dir = scratch / "replies" - trial.replies_dir.mkdir() - trial.task = SimpleNamespace(name="demo", config=SimpleNamespace( - steps=[SimpleNamespace(name="s1", min_reward=None), SimpleNamespace(name="s2", min_reward=None)])) - trial._result = SimpleNamespace(step_results=None, verifier_result=None) - trial.paths = SimpleNamespace(trial_dir=scratch / "trial", step_dir=lambda name: scratch / "trial" / name, - cleanup_empty_mount_dirs=lambda: None) - trial.agent = make("codex", good) - ran = [] - - async def run_step(step, step_result, *, index, total): - ran.append(step.name) - - async def stop_environment(): - pass - trial._run_step, trial._stop_agent_environment = run_step, stop_environment - trial._move_agent_dir_to_step, trial._select_multi_step_reward = (lambda step: None), (lambda: None) - - async def scenario(): - loop = asyncio.ensure_future(trial._run()) - - async def ask(command): - future = asyncio.get_running_loop().create_future() - await trial.commands.put((command, future)) - try: - return await asyncio.wait_for(future, 5) - except asyncio.TimeoutError: - return "no reply" - out = {"unknown": await ask({"op": "step", "name": "nope", "index": 1})} - out["next"] = await ask({"op": "step", "name": "s1", "index": 1, "plan": str(good)}) - out["bad_plan"] = await ask({"op": "step", "name": "s2", "index": 2, "plan": str(bad)}) - try: - await asyncio.wait_for(loop, 5) - except BaseException as exc: # noqa: BLE001 - reported to the test - out["loop_error"] = f"{type(exc).__name__}: {exc}" - out["ran"], out["loop_done"] = ran, trial.loop_done - return out - report = asyncio.run(scenario()) - - elif check == "driver_config": - import contextlib - import io - from harborside import driver - seen, create = [], driver.StepDriverTrial.create - - async def capture(cls, config): # the trial serve() would run: its config, and no more - seen.append(config) - raise RuntimeError("no trial here") - args = ["--task-dir", str(scratch / "task"), "--stream-dir", str(scratch), "--name", "demo__s1", - "--socket", str(scratch / "driver.sock"), - "--agent-json", json.dumps({"import_path": "harborside.codex:HermitCodex"})] - given = {"verifier_timeout_multiplier": 2.5, "environment_build_timeout_multiplier": 3.0} - driver.StepDriverTrial.create = classmethod(capture) - try: - for name, extra in (("given", ["--trial-json", json.dumps(given)]), ("default", []), - ("unknown", ["--trial-json", json.dumps({"agent_timeout_multiplier": 2.0})])): - seen.clear() - said = io.StringIO() - try: - with contextlib.redirect_stderr(said): - code = driver.main(args + extra) - except SystemExit as exc: # argparse refused an argument - code = f"SystemExit {exc.code}" - multipliers = [[c.verifier_timeout_multiplier, c.environment_build_timeout_multiplier] for c in seen] - report[name] = {"code": code, "log": said.getvalue()[-1000:], - "multipliers": multipliers[0] if multipliers else None} - finally: - driver.StepDriverTrial.create = create - - elif check == "driver_full_disk": - # serve() end to end on a one-step stream, the driver's own step loop on a stand-in of Harbor's trial (its - # run(), as Harbor's, records an exception out of the loop and returns the trial's result), whose disk fills - # up: the driver's stderr refuses its lines once the step runs (the step fails, and the loop logs it), the - # replies/ folder cannot take the step's reply (a file is there), or the failed step's agent/ cannot be moved - # under the step. A client asks for the step and reads the answer, then waits for the driver to end. - import argparse - import io - import logging - import tempfile - from harborside import driver - - class FullDisk(io.TextIOBase): - def write(self, text): - raise OSError(errno.ENOSPC, os.strerror(errno.ENOSPC)) - - def full_disk(step): - raise OSError(errno.ENOSPC, os.strerror(errno.ENOSPC)) - - def stand_in(case: str, where: Path): - trial = driver.StepDriverTrial.__new__(driver.StepDriverTrial) - trial.commands, trial.logger = asyncio.Queue(), logging.getLogger("driver_full_disk") - trial.prepared, trial.done, trial.error, trial.recorded = asyncio.Event(), asyncio.Event(), None, None - trial.running, trial.next_index, trial.aborted = None, 1, None - trial.loop_done, trial.finished = False, False - trial.replies_dir = where / "replies" - if case == "reply": - trial.replies_dir.write_text("a file where the replies' folder would be\n") - else: - trial.replies_dir.mkdir() - trial.task = SimpleNamespace(name="demo", config=SimpleNamespace(steps=[SimpleNamespace(name="s1", - min_reward=None)])) - trial._result = SimpleNamespace(step_results=None, verifier_result=None) - trial.paths = SimpleNamespace(trial_dir=where / "trial", step_dir=lambda name: where / "trial" / name, - cleanup_empty_mount_dirs=lambda: None) - trial.agent = SimpleNamespace() # an agent that takes no launch plan - - async def run_step(step, step_result, *, index, total): - if case == "log": - sys.stderr = FullDisk() # the disk fills up while the step runs - if case != "reply": - raise RuntimeError("the step's agent failed") - - async def stop_environment(): - pass - - async def run(): # Harbor's Trial.run, as far as serve() sees it - try: - await trial._run() - except Exception as exc: # noqa: BLE001 - recorded, as Harbor records it - trial.recorded = f"{type(exc).__name__}: {exc}" - return trial._result - trial._run_step, trial._stop_agent_environment, trial.run = run_step, stop_environment, run - trial._move_agent_dir_to_step = full_disk if case == "move" else (lambda step: None) - trial._select_multi_step_reward = lambda: None - return trial - - async def ask(args, where: Path): - served = asyncio.ensure_future(driver.serve(args)) - deadline = time.monotonic() + 10 - while not (where / "driver.json").is_file() and not served.done() and time.monotonic() < deadline: - await asyncio.sleep(0.02) - reader, writer = await asyncio.open_unix_connection(args.socket) - writer.write((json.dumps({"op": "step", "name": "s1", "index": 1}) + "\n").encode()) - await writer.drain() - try: - line = await asyncio.wait_for(reader.readline(), 5) - answer = json.loads(line) if line.strip() else "the connection closed without an answer" - except asyncio.TimeoutError: - answer = "no answer within 5 s" - writer.close() - try: - code = await asyncio.wait_for(served, 5) - except asyncio.TimeoutError: - code = "still running 5 s after the answer" - return answer, code - - create, stderr = vars(driver.StepDriverTrial)["create"], sys.stderr - for case in ("log", "reply", "move"): - where = scratch / case - where.mkdir() - # AF_UNIX takes 107 bytes: the socket beside the stream when that path is short enough - short = where if len(str(where / "d.sock").encode()) <= 100 else Path(tempfile.mkdtemp(prefix="hd-", - dir="/tmp")) - trial = stand_in(case, where) - - async def made(cls, config, trial=trial): - return trial - args = argparse.Namespace(task_dir=str(where / "task"), stream_dir=str(where), name="demo", - socket=str(short / "d.sock"), - agent_json=json.dumps({"import_path": "harborside.codex:HermitCodex"}), - environment_json="{}", verifier_json="{}", trial_json="{}", idle_s=3600.0) - driver.StepDriverTrial.create = classmethod(made) - try: - answer, code = asyncio.run(ask(args, where)) - finally: - sys.stderr, driver.StepDriverTrial.create = stderr, create - (short / "d.sock").unlink(missing_ok=True) - if short != where: - short.rmdir() - report[case] = {"answer": answer, "code": code, "recorded": trial.recorded, "loop_done": trial.loop_done, - "finished": trial.finished, - "ending": json.loads((where / "driver.json").read_text()).get("ending")} - - else: - raise ValueError(f"unknown check {check}") - return report - - -CHECKS = ("codex_plan", "context_limit", "every_cli", "workdir_pwd", "shell_noise", "slash_prompt", "stream_steps", - "deploy_failed_step", "cancel_fake", "nonzero_exit", "launch_failures", "capture_fails", - "launch_exec_failed", "after_cancelled", "after_own_error", "after_bounded", "return_check", "return_limits", - "workdir_removed", "copy_fails", "links_planted", "shell_session", "shell_cancel", "shell_token", - "secrets_refused", - "installs", "install_failures", "mount_rejected", "codex_nvm", "mini_swe_agent", "tool_script", "answer", - "options", "scripted", "driver_protocol", "driver_plan", "driver_loop", "driver_config", "driver_full_disk") - -if __name__ == "__main__": - sys.path.insert(0, str(HARBORSIDE_ROOT)) - reports = {} - for name in sys.argv[1].split(","): - environ = dict(os.environ) - try: - reports[name] = harbor_side(name, Path(sys.argv[2])) - except BaseException: # noqa: BLE001 - the test that reads it fails, saying why - reports[name] = {"check_failed": traceback.format_exc()[-4000:]} - finally: - os.environ.clear() - os.environ.update(environ) - print(json.dumps(reports, default=str)) - raise SystemExit(0) - - -# ----- the pytest side ---------------------------------------------------------------------------------------------- - -import pytest # noqa: E402 (the Harbor venv has no pytest; the script half above must not need it) - -from hermit.harbor import errors # noqa: E402 (Hermit's side: how it counts what harborside raised) - -pytestmark = pytest.mark.skipif(not HARBOR_PYTHON.is_file(), reason=f"needs the Harbor venv at {HARBOR_PYTHON}") - - -def harbor_checks(names, scratch: Path) -> dict: - """The reports of the checks `names`, from one run of this module under the Harbor venv's Python.""" - proc = subprocess.run([str(HARBOR_PYTHON), "-I", str(Path(__file__).resolve()), ",".join(names), str(scratch)], - capture_output=True, text=True, timeout=600) - assert proc.returncode == 0, proc.stderr[-3000:] - return json.loads(proc.stdout.strip().splitlines()[-1]) - - -@pytest.fixture(scope="module") -def checks(tmp_path_factory): - """Every check's report, from one run of this module under the Harbor venv's Python.""" - return harbor_checks(CHECKS, tmp_path_factory.mktemp("harborside")) - - -def report(checks: dict, name: str) -> dict: - got = checks[name] - assert "check_failed" not in got, got["check_failed"] - return got - - -def verdict(raised: list) -> tuple: - """How Hermit's side counts a trial whose agent raised `raised` ([type, message]): (status, failure cause).""" - got = errors.classify({"exception_info": {"exception_type": raised[0], "exception_message": raised[1]}}) - return got["status"], got["failure_cause"] - - -# the sign-in a trial returns: downloaded, then removed from the container -RETURNED = ["download_file:/installed-agent/hermit-home/auth.json", "exec:rm -f /installed-agent/hermit-home/auth.json"] - - -def test_a_codex_plan_is_carried_out_in_order_with_its_files_where_the_cli_reads_them(checks): - """The deployment (copy, host files, context, record), the home, the prompt and the launch script, the launch; - afterwards the kill, the copy-back, the collected files and the returned sign-in. Placeholders are filled with - the workdir in the plan's command and variables and in the home's files, not in its sign-in file.""" - r = report(checks, "codex_plan") - assert r["raised"] is None - order = r["order"] - assert order.index("upload_dir:/app/hermit") < order.index("upload_dir:/app") \ - < order.index("upload_file:/app/AGENTS.md") < order.index("upload_file:/app/.hermit-deploy.json") \ - < order.index("upload_dir:/installed-agent/hermit-home") \ - < order.index("upload_file:/installed-agent/hermit/run/prompt.txt") \ - < order.index("upload_file:/installed-agent/hermit/run/launch.sh") \ - < order.index("compose:exec bash /installed-agent/hermit/run/launch.sh") - after = order[order.index("compose:exec bash /installed-agent/hermit/run/launch.sh") + 1:] - assert after[0].startswith("exec:p=$(cat /installed-agent/hermit/run/session.pid") - assert after[1].startswith("exec:rm -rf /logs/agent/hermit-copy && mkdir -m 755") - assert after[2].startswith("exec:rm -rf /logs/agent/hermit-collect && mkdir -m 755") - assert after[-2:] == ["download_file:/installed-agent/hermit-home/auth.json", - "exec:rm -f /installed-agent/hermit-home/auth.json"] # the sign-in leaves with its copy - files = r["uploaded"] - assert files["/app/hermit/demo/ADAPTER.md"] == "The adapter.\n" - assert files["/app/.codex/hooks.json"] == '{"hooks": {}}' and files["/app/.codex/config.toml"] == "# roles\n" - assert files["/app/AGENTS.md"] == "Hermit context.\n\n" + IMAGE_CONTEXT # Hermit's first, the image's after - assert json.loads(files["/app/.hermit-deploy.json"])["adapter_id"] == "demo" - home = "/installed-agent/hermit-home" - assert files[f"{home}/config.toml"] == '[projects."/app"]\ntrust_level = "trusted"\n' - assert "@@HERMIT_WORKDIR@@ left as it is" in files[f"{home}/auth.json"] - assert files["/installed-agent/hermit/run/prompt.txt"] == INSTRUCTION + SUFFIX - script = files["/installed-agent/hermit/run/launch.sh"] - for line in ("cd -- /app || { echo 'hermit: cannot enter the workdir /app' >&2; exit 97; }", - "export CODEX_HOME=/installed-agent/hermit-home", "export PROJECT=/app/src", - 'export PATH=/opt/hermit-codex/bin:"$PATH"', "export HERMIT_HOOK_LOGS=/logs/agent", - 'export OPENAI_API_KEY="${FIREWORKS_API_KEY-}"', - "IFS= read -r -d '' HERMIT_PROMPT_TEXT < /installed-agent/hermit/run/prompt.txt || true", - ' setsid codex exec -C /app "$HERMIT_PROMPT_TEXT" > /logs/agent/agent-codex.jsonl 2>&1 < /dev/null &', - 'echo "$pid" > /installed-agent/hermit/run/session.pid'): - assert line in script.splitlines(), line - assert "unset FIREWORKS_API_KEY" not in script # also passed under its own name - roots = [e[1] for e in r["execs"] if e[0] == "root"] - assert "rm -rf /app/hermit/demo && mkdir -p /app/hermit" in roots # a replaced copy - assert ("chown -R 1000:1000 /app/hermit/demo /app/.codex /app/AGENTS.md /app/.hermit-deploy.json && " - "chmod -R u+rwX /app/hermit/demo /app/.codex /app/AGENTS.md /app/.hermit-deploy.json") in roots - assert (f"chown -R 1000:1000 {home} && chmod -R u+rwX,go-rwx {home} && chmod 600 {home}/auth.json") in roots - copy = next(c for c in roots if "hermit-copy" in c) - assert copy.startswith('rm -rf /logs/agent/hermit-copy && mkdir -m 755 /logs/agent/hermit-copy && cd -P ' - '/logs/agent/hermit-copy && [ "$(pwd -P)" = "$(cd -P /logs/agent && pwd -P)/hermit-copy" ] ' - '|| exit 1; f=0; ') # a folder of root's, entered, checked - assert "then cp -a /app/hermit/demo/memory ./memory || { f=1; rm -rf ./memory; }; fi" in copy - collect = next(c for c in roots if "hermit-collect" in c) - assert (f"then cp {home}/last_message.txt ./last_message.txt || {{ f=1; rm -rf ./last_message.txt; }}; " - "fi") in collect - assert "auth.json" not in collect # a sign-in never goes to logs - assert r["returned"] == f"downloaded {home}/auth.json" - assert r["context"]["metadata"] == { - "plan_digest": "abc123", "hermit_cli": "codex", "adapter_deployed": True, "adapter_id": "demo", - "hooks_active": True, "python3": True, "mounted": False, "exit_code": 0, "killed": False, - "stream": "agent-codex.jsonl"} - assert r["context"]["n_input_tokens"] is None # Hermit reads the stream - - -def test_a_link_in_the_staging_is_uploaded_as_a_link_never_followed_to_a_host_file(checks): - """An evolved adapter's deployment or a rendered home could hold a link to any file of this host: it reaches the - container as the link it is (as `docker cp` keeps the adapter's own copy), its target is neither read nor - written (no placeholder filled through it), and a link at the top of the workdir is not chown'd as root.""" - r = report(checks, "codex_plan") - files = r["uploaded"] - assert files["/app/.codex/linked.txt"] == f"" - assert files["/app/linked-dir"] == f"" - assert files["/installed-agent/hermit-home/linked.toml"] == f"" - assert not any("a host file" in text for text in files.values()) - assert r["canary_after"] == "a host file, @@HERMIT_WORKDIR@@\n" - assert not any("/app/linked-dir" in c for user, c in r["execs"] if user == "root" and c.startswith("chown")) - - -def test_a_secret_goes_to_compose_by_name_and_is_in_no_argument_file_or_log(checks): - """FIREWORKS_API_KEY's value is in Harbor's environment only: the compose call names it, the compose client - resolves it, and no recorded exec, upload, compose argument or download holds the value.""" - r = report(checks, "codex_plan") - assert r["leaks"] == [] - assert r["compose"] == [["exec", "-w", "/app", "-e", "FIREWORKS_API_KEY", "main", "bash", "-c", - "exec bash /installed-agent/hermit/run/launch.sh"]] - assert r["compose_names_resolved"] == [{"FIREWORKS_API_KEY": True}] - - -def test_secrets_by_name_are_refused_clearly_outside_docker_or_when_harbor_lacks_them(checks): - r = report(checks, "secrets_refused") - assert r["plain_env"][0] == "HermitLaunchError" and "only a local Docker environment does" in r["plain_env"][1] - assert r["missing"][0] == "HermitLaunchError" - assert "FIREWORKS_API_KEY is not in Harbor's environment" in r["missing"][1] - assert r["missing_compose"] == [] # nothing was started - - -def test_a_context_over_the_clis_limit_fails_the_trial_before_the_session_and_one_within_is_merged(checks): - """dsh drops AGENTS.md past its limit without saying so: the merged file (Hermit's, then the image's own, its text - counted as dsh escapes it) is checked against the plan's context_limit before it is written, and the after-steps - still run.""" - over, within = report(checks, "context_limit")["over"], report(checks, "context_limit")["within"] - assert over["raised"] == ["HermitContextLimitError", - "hermit: AGENTS.md for dsh would be 66 bytes (Hermit's context 16, then the image's own " - "49), over the CLI's limit of 60 bytes: shorten the adapter's context"] - assert "/app/AGENTS.md" not in over["uploaded"] - assert not any("launch.sh" in step and step.startswith("exec:exec") for step in over["order"]) - kill, copy, collect = over["order"][-3:] # the after-steps still ran - assert kill.startswith("exec:p=$(cat /installed-agent/hermit/run/session.pid") - assert copy.startswith("exec:rm -rf /logs/agent/hermit-copy && mkdir -m 755") - assert collect.startswith("exec:rm -rf /logs/agent/hermit-collect && mkdir -m 755") - assert within["raised"] is None and within["uploaded"]["/app/AGENTS.md"] == "Hermit context.\n\n" + IMAGE_CONTEXT - # 123 bytes merged, within the limit of 124, but 126 as dsh counts them (the image's three tags escaped) - escaped, unescaped = report(checks, "context_limit")["escaped"], report(checks, "context_limit")["unescaped"] - assert escaped["raised"] == ["HermitContextLimitError", - "hermit: AGENTS.md for dsh would be 126 bytes (Hermit's context 16, then the image's " - "own 109), over the CLI's limit of 124 bytes: shorten the adapter's context"] - assert unescaped["raised"] is None and len(unescaped["uploaded"]["/app/AGENTS.md"].encode()) == 123 - - -def test_every_cli_gets_the_deployment_record_hook_logs_and_its_prompt_where_the_plan_says(checks): - """The record (an adapter's hook runner finds its host there) is uploaded for every CLI, not only Claude Code's, - and so is HERMIT_HOOK_LOGS set; the prompt goes on stdin or into the argument the plan marks.""" - r = report(checks, "every_cli") - for cli, got in r.items(): - assert got["raised"] is None, (cli, got["raised"]) - assert json.loads(got["record"])["adapter_id"] == "demo", cli - assert "export HERMIT_HOOK_LOGS=/logs/agent" in got["launch"].splitlines(), cli - assert got["prompt"] == INSTRUCTION + SUFFIX, cli - stdin = "< /installed-agent/hermit/run/prompt.txt &" in got["launch"] - assert stdin == (cli in ("claude", "opencode", "mini-swe-agent")), cli - assert f"> /logs/agent/agent-{cli}.jsonl 2>&1" in got["launch"], cli - for rel in HOST_FILES[cli]: - assert f"/app/{rel}" in got["workdir_files"], (cli, rel) - if CONTEXT_FILE[cli]: - assert got["context"] == "Hermit context.\n\n" + IMAGE_CONTEXT, cli - assert f" setsid {TOOL_PYTHON} exec -C /app" in r["mini-swe-agent"]["launch"] # its own Python - - -def test_a_plan_without_a_workdir_runs_where_harbors_execs_start(checks): - """A task.toml that names no workdir: the session, the deployment and the placeholders use `pwd` as the task's - agent user (the image's WORKDIR).""" - r = report(checks, "workdir_pwd") - assert r["raised"] is None and r["pwd_user"] == [None] - files = r["uploaded"] - assert "/testbed/hermit/demo/ADAPTER.md" in files and "/testbed/.hermit-deploy.json" in files - assert files["/testbed/AGENTS.md"] == "Hermit context.\n" # the image had none - assert files["/installed-agent/hermit-home/config.toml"].startswith('[projects."/testbed"]') - script = files["/installed-agent/hermit/run/launch.sh"].splitlines() - assert "cd -- /testbed || { echo 'hermit: cannot enter the workdir /testbed' >&2; exit 97; }" in script - assert "export PROJECT=/testbed/src" in script - - -def test_a_workdir_of_the_root_gets_the_deployment_in_the_root(checks): - """No workdir in the task and none in the image: Harbor's execs start in /, and the deployment's files go to /, - not to an empty container path.""" - r = report(checks, "workdir_pwd") - assert r["root_raised"] is None - assert r["root_dirs"] == ["/hermit", "/", "/installed-agent/hermit-home"] - assert any(f.startswith("/AGENTS.md Hermit context.") for f in r["root_files"]) - assert any(f.startswith("/.hermit-deploy.json ") for f in r["root_files"]) - - -def test_the_agent_user_the_workdir_and_the_images_context_are_read_among_other_lines_of_the_exec(checks): - """Harbor's docker exec merges stderr into stdout, and an image's BASH_ENV may echo: each value read from the - container is found on a marked line of its own among other lines before and after it, for real with bash (the - trial is not refused for its agent user, and the image's context file is merged as it is, without the noise).""" - r = report(checks, "shell_noise") - assert r["raised"] is None - assert r["context"] == "Hermit context.\n\n" + IMAGE_CONTEXT - assert any(c.startswith(f"chown -R {r['owner']} ") for c in r["chowns"]) - assert any(line.startswith(f"cd -- {r['root']}/app ||") for line in r["launch"].splitlines()) - assert f"pwd: {r['root']}/app" in r["stream"].splitlines() # where the CLI ran - - -def test_a_prompt_that_starts_with_a_slash_gets_the_plans_prefix(checks): - """Claude Code runs a prompt that starts with `/` as one of its commands: the plan's prompt_slash_prefix (its Base - Agent's line break) goes before such an instruction; a plan without one passes it as it is.""" - r = report(checks, "slash_prompt") - assert r["guarded"]["raised"] is None and r["plain"]["raised"] is None - assert r["guarded"]["prompt"] == "\n/app/data.csv holds the rows." + SUFFIX - assert r["plain"]["prompt"] == "/app/data.csv holds the rows." + SUFFIX - - -def test_a_streams_later_step_replaces_the_deployment_and_one_without_restores_the_image(checks): - """One container, three steps: the second's context is merged with the image's own file (captured before the - first step), not with the first step's; what the first deployed and the second does not is removed; a step with - no deployment puts the image's context file back and removes the rest.""" - r = report(checks, "stream_steps") - one, two, three = r["one"], r["two"], r["three"] - assert one["raised"] is None and two["raised"] is None and three["raised"] is None - assert one["uploaded"]["/app/AGENTS.md"] == "Hermit context.\n\n" + IMAGE_CONTEXT - assert two["uploaded"]["/app/AGENTS.md"] == "Hermit context, revised.\n\n" + IMAGE_CONTEXT - assert "rm -rf /app/.codex/hooks.json" in two["root_execs"] - assert "/app/.codex/hooks.json" not in two["uploaded"] - assert ("rm -rf /app/.codex/config.toml /app/hermit/demo /app/.hermit-deploy.json && if [ -f " - "/installed-agent/hermit/original-AGENTS.md ]; then cat /installed-agent/hermit/original-AGENTS.md > " - "/app/AGENTS.md; else rm -f /app/AGENTS.md; fi") in three["root_execs"] - assert not any(k.startswith("/app/") for k in three["uploaded"]) - - -def test_a_streams_step_whose_deployment_failed_does_not_report_the_hooks_of_the_step_before(checks): - """One container, two steps: the first deploys an adapter whose hooks are active; the second deploys none, and - removing what the first one deployed fails (HermitLaunchError). The second step's metadata says that no hooks were - active, not the first step's value.""" - r = report(checks, "deploy_failed_step") - one, two = r["one"], r["two"] - assert one["raised"] is None and one["metadata"]["hooks_active"] is True - assert two["raised"][0] == "HermitLaunchError" - assert two["raised"][1].startswith("hermit: deploying the adapter failed: ") - assert two["metadata"]["hooks_active"] is False and two["metadata"]["adapter_deployed"] is False - - -def test_a_session_harbors_timeout_cancels_is_killed_even_without_an_adapter(checks): - """Harbor's timeout cancels only its wait on the compose client, and the CLI goes on in the container: the - session's process group is killed in the finally (TERM, the plan's grace, KILL) as root from its pid file, with no - adapter deployed; the metadata says it was killed.""" - r = report(checks, "cancel_fake") - assert r["raised"][0] == "TimeoutError" - assert len(r["kill"]) == 1 and r["kill"][0][0] == "root" - assert 'kill -TERM -- -"$p"' in r["kill"][0][1] and 'kill -KILL -- -"$p"' in r["kill"][0][1] - assert 'while [ "$i" -lt 2 ]' in r["kill"][0][1] # the plan's grace - assert r["metadata"]["killed"] is True and r["metadata"]["exit_code"] is None - assert r["metadata"]["adapter_deployed"] is False and r["metadata"]["hermit_cli"] == "pi" - - -def test_a_cli_that_exits_non_zero_raises_harbors_agent_error_and_a_failed_copy_back_does_not_hide_it(checks): - """NonZeroAgentExitCodeError lets the verifier run (Hermit's side reads the verdict from the stream, which the - launch script made); the copy-back that failed after it is recorded in the metadata, and the steps after it still - ran.""" - r = report(checks, "nonzero_exit") - assert r["raised"] == ["NonZeroAgentExitCodeError", "hermit: pi exited with 3 (its stream: " - "/logs/agent/agent-pi.jsonl)"] - after = r["order"][r["order"].index("exec:exec bash /installed-agent/hermit/run/launch.sh") + 1:] - assert after[0] == "exec:test -e /logs/agent/agent-pi.jsonl" # the stream is there - assert after[1].startswith("exec:p=$(cat /installed-agent/hermit/run/session.pid") - assert after[2].startswith("exec:rm -rf /logs/agent/hermit-copy && mkdir -m 755") # it failed - assert after[3].startswith("exec:rm -rf /logs/agent/hermit-collect && mkdir -m 755") # and the next one ran - assert r["metadata"]["exit_code"] == 3 and r["metadata"]["killed"] is False - assert [e.split(":")[0] for e in r["metadata"]["after_session_errors"]] == ["copy_back"] - - -def test_harborsides_own_failures_before_the_session_are_launch_failures_not_the_agents_zero(checks): - """A chown of the deployment or the probe of the task's agent user failing, or the launch script unable to enter - the workdir (its exit 97 with its own words, for real with bash), is harborside's, not the agent's: it raises - HermitLaunchError saying what failed, which Hermit's side counts as infra (launch_failed), and the after-steps - still run. So does an image's context file that cannot be read (no base64 there): it is never taken for none and - overwritten with the adapter's alone. A CLI that exits 97 by itself stays the agent's - (NonZeroAgentExitCodeError).""" - r = report(checks, "launch_failures") - chown, owner, enter, own = r["chown"], r["owner"], r["cannot_enter"], r["own_97"] - assert r["no_base64"]["raised"][0] == "HermitLaunchError" - assert r["no_base64"]["raised"][1].startswith("hermit: deploying the adapter failed: NonZeroAgentExitCodeError: ") - assert verdict(r["no_base64"]["raised"]) == ("infra", "launch_failed") - assert not any(step.startswith("exec:exec bash") for step in r["no_base64"]["order"]) - assert chown["raised"][0] == "HermitLaunchError" - assert chown["raised"][1].startswith("hermit: deploying the adapter failed: NonZeroAgentExitCodeError: Command " - "failed (exit 1): chown -R 1000:1000 /app/hermit/demo ") - assert owner["raised"][0] == "HermitLaunchError" - assert owner["raised"][1].startswith("hermit: reading the task's agent user failed: NonZeroAgentExitCodeError: " - 'Command failed (exit 1): echo "hermit-owner:$(id -u):$(id -g)"') - assert enter["raised"][0] == "HermitLaunchError" - assert enter["raised"][1].startswith("hermit: codex did not start: ") - assert "hermit: cannot enter the workdir " in enter["raised"][1] - for got in (chown, owner, enter): - assert verdict(got["raised"]) == ("infra", "launch_failed") - for got in (chown, owner): # no session; the after-steps ran - assert not any(step.startswith("exec:exec bash") for step in got["order"]) - kill, copy, collect = got["order"][-3:] - assert kill.startswith("exec:p=$(cat /installed-agent/hermit/run/session.pid") - assert copy.startswith("exec:rm -rf /logs/agent/hermit-copy && mkdir -m 755") - assert collect.startswith("exec:rm -rf /logs/agent/hermit-collect && mkdir -m 755") - assert own["raised"] == ["NonZeroAgentExitCodeError", "hermit: codex exited with 97 (its stream: " - "/logs/agent/agent-codex.jsonl)"] - assert verdict(own["raised"]) == ("agent_error", "unclassified") - - -def test_an_images_context_file_whose_read_fails_is_a_launch_failure_never_taken_for_none_and_overwritten(checks): - """For real with bash, the image's own AGENTS.md and a base64 that fails reading it (base64 and tr are both - there): the capture exits saying so instead of printing an empty value, so the trial is HermitLaunchError (infra, - launch_failed) before any session, and the image's file is left as it was, not replaced by the adapter's alone.""" - got = report(checks, "launch_failures")["read_fails"] - raised = got["raised"] - assert raised is not None and raised[0] == "HermitLaunchError", raised - assert raised[1].startswith("hermit: deploying the adapter failed: NonZeroAgentExitCodeError: Command failed " - "(exit 3): ") - assert "base64: read error: Input/output error" in raised[1] - assert verdict(raised) == ("infra", "launch_failed") - assert got["context"] == IMAGE_CONTEXT - assert not any(step.startswith("exec:exec bash") for step in got["order"]) - - -def test_an_images_context_file_whose_capture_failed_part_way_is_a_launch_failure_never_merged_or_put_back(checks): - """For real with bash, one container and three steps of a stream: the first step's capture of the image's own - AGENTS.md meets a cat that fails part-way, as on a full disk. That step is HermitLaunchError saying why (infra, - launch_failed) before any session, the image's file left as it was and no part of it kept as the image's own. The - next step, the cat working again, captures and merges the whole file, and a step without a deployment puts the - whole file back.""" - r = report(checks, "capture_fails") - failed, again, restored = r["failed"], r["again"], r["restored"] - raised = failed["raised"] - assert raised is not None and raised[0] == "HermitLaunchError", raised - assert raised[1].startswith("hermit: deploying the adapter failed: NonZeroAgentExitCodeError: Command failed " - "(exit 3): ") - assert "cat: write error: No space left on device" in raised[1] - assert verdict(raised) == ("infra", "launch_failed") - assert failed["launched"] is False and failed["context"] == IMAGE_CONTEXT and failed["kept"] is None - assert again["raised"] is None and again["launched"] is True - assert again["context"] == "Hermit context.\n\n" + IMAGE_CONTEXT and again["kept"] == IMAGE_CONTEXT - assert restored["raised"] is None and restored["context"] == IMAGE_CONTEXT - - -def test_a_launch_that_left_no_stream_is_a_launch_failure_and_one_that_left_a_stream_even_empty_is_the_agents(checks): - """The launch script's redirect makes the stream before the CLI starts. A non-zero exit with no stream is an exec - that failed in Docker before the script ran (runc could not start it: exit 126, the stream really absent, for real - with bash; compose's service had stopped, its probe failing as well): HermitLaunchError, which Hermit's side - counts as infra (launch_failed), not the agent's 0 on an untouched workspace. A stream that is there, even an - empty one (for real with bash, a CLI that exits 3 at once and prints nothing), keeps the exit the agent's - (NonZeroAgentExitCodeError), for Hermit's side to read; so it does when the CLI removed its workdir (the stream is - looked for from /).""" - r = report(checks, "launch_exec_failed") - runc, stopped, empty, gone = r["runc"], r["stopped"], r["empty"], r["workdir_gone"] - assert runc["raised"] == ["HermitLaunchError", "hermit: codex did not start: its exec exited with 126 and left no " - "stream (/logs/agent/agent-codex.jsonl)\nOCI runtime exec failed: exec failed: unable " - "to start container process: open /tmp/runc-process1: no space left on device: unknown"] - assert runc["stream"] is False - after = runc["order"][runc["order"].index("exec:exec bash /installed-agent/hermit/run/launch.sh") + 1:] - assert after[0] == "exec:test -e /logs/agent/agent-codex.jsonl" - assert after[1].startswith("exec:p=$(cat /installed-agent/hermit/run/session.pid") # the after-steps still ran - assert stopped["raised"] == ["HermitLaunchError", "hermit: codex did not start: its exec exited with 1 and left no " - 'stream (/logs/agent/agent-codex.jsonl)\nservice "main" is not running'] - for got in (runc, stopped): - assert verdict(got["raised"]) == ("infra", "launch_failed") - assert empty["raised"] == ["NonZeroAgentExitCodeError", "hermit: codex exited with 3 (its stream: " - "/logs/agent/agent-codex.jsonl)"] - assert empty["stream"] == "" - assert verdict(empty["raised"]) == ("agent_error", "unclassified") - assert gone["workdir"] is False - assert gone["raised"] == ["NonZeroAgentExitCodeError", "hermit: codex exited with 3 (its stream: " - "/logs/agent/agent-codex.jsonl)"] - - -def test_a_streams_later_step_whose_exec_failed_in_docker_is_a_launch_failure_though_the_step_before_left_a_stream( - checks): - """One container, two steps of a stream: the first runs for real with bash and leaves its stream in /logs/agent; - the second's exec fails as runc's does (exit 126, its launch script not run). The stream an earlier step left is - removed before the exec, so this step's exec left none: HermitLaunchError, which Hermit's side counts as infra - (launch_failed), not the agent's exit read from the earlier step's stream.""" - r = report(checks, "launch_exec_failed")["later_step"] - assert r["first"] is None and r["left"] == "the first step\n" - assert r["raised"] == ["HermitLaunchError", "hermit: codex did not start: its exec exited with 126 and left no " - "stream (/logs/agent/agent-codex.jsonl)\nOCI runtime exec failed: exec failed: unable to " - "start container process: open /tmp/runc-process1: no space left on device: unknown"] - assert verdict(r["raised"]) == ("infra", "launch_failed") and r["stream"] is False - - -def test_a_session_exec_or_a_look_for_its_stream_this_machine_could_not_spawn_is_a_launch_failure(checks): - """The compose client of the session's exec (or, after a non-zero exit, of the look for its stream) that this - machine could not spawn (an OSError from the exec itself: EMFILE, EAGAIN on a loaded host), the exec by name too: - HermitLaunchError saying what failed, which Hermit's side counts as infra (launch_failed), not an unclassified - agent error scored 0; the after-steps still run.""" - r = report(checks, "launch_exec_failed") - for name, what, number in ( - ("unspawned", "codex did not start: its exec could not be run here", errno.EMFILE), - ("unspawned_by_name", "codex did not start: its exec could not be run here", errno.EAGAIN), - ("probe_unspawned", "codex exited with 3, and its stream (/logs/agent/agent-codex.jsonl) could not be " - "looked for here", errno.EAGAIN)): - got, raised = r[name], OSError(number, os.strerror(number)) # as Python raises it: EAGAIN's BlockingIOError - assert got["raised"] == ["HermitLaunchError", f"hermit: {what}: {type(raised).__name__}: {raised}"], \ - (name, got["raised"]) - assert verdict(got["raised"]) == ("infra", "launch_failed"), name - kill, collect = got["order"][-2:] - assert kill.startswith("exec:p=$(cat /installed-agent/hermit/run/session.pid"), name - assert collect.startswith("exec:rm -rf /logs/agent/hermit-collect && mkdir -m 755"), name - - -def test_harbors_timeout_landing_in_an_after_step_still_runs_the_rest_and_then_cancels(checks): - """The timeout cancels the copy-back of a session that has just ended: the copy-back is not run again (a second - `rm -rf` then `cp -a` could nest its copy inside the cancelled one's, still running in the container), the - collected files and the refreshed sign-in still come back (ChatGPT rotates its refresh tokens), the cancellation is - recorded, and it goes on after the last step, so Harbor still sees its timeout.""" - r = report(checks, "after_cancelled") - assert r["raised"][0] == "TimeoutError" - after = r["order"][r["order"].index("exec:exec bash /installed-agent/hermit/run/launch.sh") + 1:] - assert after[0].startswith("exec:p=$(cat /installed-agent/hermit/run/session.pid") - assert after[1].startswith("exec:rm -rf /logs/agent/hermit-copy && mkdir -m 755") # cancelled, not run again - assert after[2].startswith("exec:rm -rf /logs/agent/hermit-collect && mkdir -m 755") # and the rest still ran - assert after[-2:] == RETURNED - assert r["returned"] == "downloaded /installed-agent/hermit-home/auth.json" - assert r["metadata"]["after_session_errors"] == ["copy_back: cancelled"] - - -def test_harbors_timeout_landing_in_the_return_of_the_files_still_brings_the_refreshed_sign_in_back(checks): - """The timeout lands while the refreshed sign-in of a session that has just ended is returned (its check of the - file, in the last after-step): that step runs once more, so the sign-in still comes back to the host (ChatGPT - rotates its refresh tokens: the old one would be stale), and the cancellation goes on after it, so Harbor still - sees its timeout.""" - r = report(checks, "after_cancelled")["in_return"] - assert r["raised"][0] == "TimeoutError" - after = r["order"][r["order"].index("exec:exec bash /installed-agent/hermit/run/launch.sh") + 1:] - check = "exec:test -f /installed-agent/hermit-home/auth.json && test ! -L " - assert after[2].startswith("exec:rm -rf /logs/agent/hermit-collect && mkdir -m 755") - assert [step.startswith(check) for step in after[3:5]] == [True, True], after # cancelled, then once more - assert after[5:] == RETURNED - assert r["returned"] == "downloaded /installed-agent/hermit-home/auth.json" - assert r["metadata"]["after_session_errors"] == ["return_files: cancelled"] - - -def test_a_launch_failure_or_a_non_zero_exit_keeps_its_own_error_when_harbors_timeout_lands_in_an_after_step(checks): - """A launch that left no stream (exit 126: runc could not start the exec) and a CLI that exited 3, each followed by - a collect that hangs until Harbor's timeout: the cancellation is recorded, the refreshed sign-in still comes back, - and the session's own error goes on, not the timeout (Harbor's wait_for lets it through). So the launch failure is - still infra (run again), not the agent's timeout scored on a workspace nobody touched, and the non-zero exit still - the agent's error read from its stream, not a run that ran out of time.""" - r = report(checks, "after_own_error") - launch, non_zero = r["launch_failed"], r["non_zero"] - assert launch["raised"] == ["HermitLaunchError", "hermit: codex did not start: its exec exited with 126 and left " - "no stream (/logs/agent/agent-codex.jsonl)"] - assert verdict(launch["raised"]) == ("infra", "launch_failed") - assert non_zero["raised"] == ["NonZeroAgentExitCodeError", "hermit: codex exited with 3 (its stream: " - "/logs/agent/agent-codex.jsonl)"] - assert verdict(non_zero["raised"]) == ("agent_error", "unclassified") - for name, got in r.items(): - after = got["order"][got["order"].index("exec:exec bash /installed-agent/hermit/run/launch.sh") + 1:] - assert after[2].startswith("exec:rm -rf /logs/agent/hermit-collect && mkdir -m 755"), name # it hung - assert after[-2:] == RETURNED, name # the rest ran - assert got["returned"] == "downloaded /installed-agent/hermit-home/auth.json", name - assert got["metadata"]["after_session_errors"] == ["collect: cancelled"], name - - -def test_once_the_run_is_cancelled_each_after_step_has_a_bounded_time_and_the_files_still_come_back(checks): - """Harbor's timeout cancels the session, and the container hangs after it: the copy-back and the collect are each - cut short at AFTER_CANCEL_S (0.3 s here) and recorded so, the refreshed sign-in still comes back, and Harbor sees - its timeout in a few seconds, not after every step has waited as long as the container hangs (5 s each here).""" - r = report(checks, "after_bounded")["bounded"] - assert r["raised"][0] == "TimeoutError" and r["seconds"] < 4 - assert r["metadata"]["after_session_errors"] == ["copy_back: not done within 0.3 s of the run's cancellation", - "collect: not done within 0.3 s of the run's cancellation"] - assert r["metadata"]["killed"] is True - assert r["order"][-2:] == RETURNED - assert r["returned"] == "downloaded /installed-agent/hermit-home/auth.json" - - -def test_a_second_cancellation_ends_the_run_at_once_without_the_after_steps_left(checks): - """The job cancelled after Harbor's timeout (a second cancellation, while the copy-back hangs): the run ends at - once with it, and the steps after the copy-back (the collected files, the returned ones) are not run.""" - r = report(checks, "after_bounded")["twice"] - assert r["raised"][0] == "CancelledError" and r["seconds"] < 2 - after = r["order"][r["order"].index("exec:exec bash /installed-agent/hermit/run/launch.sh") + 1:] - assert len(after) == 2, after - assert after[0].startswith("exec:p=$(cat /installed-agent/hermit/run/session.pid") - assert after[1].startswith("exec:rm -rf /logs/agent/hermit-copy && mkdir -m 755") - - -def test_after_a_session_removed_its_workdir_the_after_steps_still_kill_it_collect_and_return_the_sign_in(checks): - """Docker refuses an exec in a workdir the session removed (exit 126; ShellEnv, for real with bash, likewise): - the kill, the copy-back, the collect and the return run from /, so a session that removed its workdir (and the - deployment in it) and ran on until Harbor's timeout is killed with the child it started, and whether it ended by - itself or was cancelled, the copy-back comes back (its folder, empty: the memory went with the workdir), its last - message is collected and its refreshed sign-in comes back to the host (ChatGPT rotates its refresh tokens), none - of it failing.""" - r = report(checks, "workdir_removed") - ended, running = r["ended"], r["running"] - for name, got in r.items(): - assert got["workdir"] is False, name - assert got["returned"] == '{"refreshed": true}\n', name - assert got["copied"] == [], name - assert got["collected"] == ["last_message.txt"], name - assert "after_session_errors" not in got["metadata"], (name, got["metadata"]) - assert ended["raised"] is None and ended["metadata"]["killed"] is False - assert running["raised"][0] == "CancelledError" - assert running["metadata"]["killed"] is True and running["child_alive"] is False - - -def test_a_return_whose_check_failed_is_recorded_and_a_file_the_session_did_not_leave_is_skipped(checks): - """The check of a file to return fails in a container whose compose service had stopped: the failure is recorded - with the after-steps' errors, saying so, not taken for a file the session did not leave (a refreshed sign-in that - never came back, unseen). A file the session did not leave is skipped, and nothing is recorded.""" - r = report(checks, "return_check") - failed, absent = r["failed"], r["absent"] - assert failed["raised"] is None and failed["downloads"] == [] and failed["returned"] is False - assert failed["metadata"]["after_session_errors"] == [ - "return_files: whether the session left /installed-agent/hermit-home/auth.json could not be told (exit 1): " - 'service "main" is not running'] - assert absent["raised"] is None and absent["downloads"] == [] and absent["returned"] is False - assert "after_session_errors" not in absent["metadata"] - - -def test_a_file_to_return_over_its_limit_or_left_as_a_link_is_not_downloaded_nor_one_whose_size_cannot_be_told(checks): - """For real with bash: the session can write its home's auth.json (the ChatGPT sign-in that comes back to the - host) as any file. One of the plan's max_bytes is downloaded; one a byte over is not, and the step's failure says - so (a sign-in is a few KB: the host would read the larger file whole); a link is skipped, as a file the session did - not leave is; and one whose size cannot be told (an image whose wc fails) is not downloaded either, its failure - recorded, never taken for none.""" - r = report(checks, "return_limits") - source = "/installed-agent/hermit-home/auth.json" - for name, got in r.items(): - assert got["raised"] is None, (name, got["raised"]) - assert [c[1] for c in got["downloads"]] == ([source] if name == "at_limit" else []), name - assert r["at_limit"]["returned"] == 4096 and "after_session_errors" not in r["at_limit"]["metadata"] - assert r["over"]["returned"] is None and r["over"]["metadata"]["after_session_errors"] == [ - f"return_files: {source} is 4097 bytes, over the 4096 it may hold: not returned"] - assert r["link"]["returned"] is None and "after_session_errors" not in r["link"]["metadata"] - failed = r["no_wc"]["metadata"]["after_session_errors"] - assert r["no_wc"]["returned"] is None and len(failed) == 1 - assert failed[0].startswith(f"return_files: whether the session left {source} could not be told (exit 3): "), failed - - -def test_a_copy_back_or_collect_that_failed_part_way_is_recorded_and_removed_and_the_parts_after_it_come_back(checks): - """For real with bash, a cp that fails part-way, as on a full disk (the first bytes of each file written, mode - 600, then exit 1), copying the adapter's memory in the copy-back and last_message.txt and a sessions folder in the - collect: each copy that failed is removed, never kept in part (Hermit's side would take a cut last message for the - reply, where without the file it reads the stream's, and a cut memory for the adapter's), and its step fails, - recorded with the after-steps' errors saying why, the session's own outcome unchanged. The part after them in each - step (a folder of the adapter's own, the home's agents folder) is still copied, whole, and the chmod after them - still ran, so what was copied is readable: each file there, mode 600 where it was copied from, comes back 644.""" - r = report(checks, "copy_fails") - assert r["raised"] is None - failed = r["metadata"]["after_session_errors"] - assert [e.split(":")[0] for e in failed] == ["copy_back", "collect"], failed - for error in failed: - assert "Command failed (exit 1): " in error and "No space left on device" in error, error - assert r["entries"] == ["hermit-collect/agents", "hermit-collect/agents/reviewer.toml", "hermit-copy/journal", - "hermit-copy/journal/JOURNAL.md"] - assert r["source_modes"] == {"hermit-copy/journal/JOURNAL.md": "0o600", - "hermit-collect/agents/reviewer.toml": "0o600"} - assert r["kept"] == {"hermit-copy/journal/JOURNAL.md": ["# Journal\n", "0o644"], - "hermit-collect/agents/reviewer.toml": ['name = "reviewer"\n', "0o644"]} - - -def test_the_after_steps_follow_no_link_the_agents_user_planted_where_they_write(checks): - """/logs/agent is writable by every user, so the agent's user can put a link where the copy-back and the collect - write (hermit-copy, hermit-collect), to a folder of root's only: each step makes its folder anew, root's, and - works in it, so nothing is copied there and its modes stay (before, root copied into it and made it readable to - all).""" - r = report(checks, "links_planted") - assert r["raised"] is None and not r["metadata"].get("after_session_errors"), r["metadata"] - assert r["links"] == {"hermit-copy": False, "hermit-collect": False} - assert r["copied_notes"] == "# Notes\nlearned something\n" and r["collected"] == ["last_message.txt"] - assert r["private"] == {name: ["0o700", "0o600", ["secret.txt"]] for name in ("hermit-copy", "hermit-collect")} - - -def test_the_launch_script_runs_the_cli_with_its_prompt_variables_and_secrets_and_brings_its_files_back(checks): - """For real, with bash and a stand-in CLI: the prompt arrives byte for byte (quotes, $HOME, newlines), the - workdir, the hook log directory, the key by name (also under its second name), a process group of the session's - own; afterwards the memory is copied back, last_message.txt collected (the sign-in is not), the refreshed sign-in - returned to the host, the home's files private, and the pid file gone.""" - r = report(checks, "shell_session") - root = r["root"] - assert r["raised"] is None - lines = r["stream"].split("\n") - assert r["stream"].startswith(f"argv: <-C> <{root}/app> <{INSTRUCTION + SUFFIX}>\n") - for line in ("stdin: <>", f"pwd: {root}/app", f"hook-logs: {root}/logs/agent", "key-in-env: yes", - "renamed-key: yes", "token-in-env: ", "pgid-is-own: yes"): - assert line in lines, line - assert r["copied_notes"] == "# Notes\nlearned something\n" - assert r["collected"] == ["last_message.txt"] - assert r["returned"] == '{"refreshed": true}\n' - assert "auth-mode: 600" in lines # the session's sign-in, its owner's alone - assert "auth.json" not in r["home_modes"] and r["home_modes"]["config.toml"] == "0o600" # gone once returned - assert r["home_config"].startswith('[projects."/app"]') - assert r["pidfile_left"] is False and r["leaks"] == [] - assert r["metadata"]["exit_code"] == 0 and r["metadata"]["killed"] is False - - -def test_harbors_timeout_kills_the_real_session_and_what_it_started_and_the_files_still_come_back(checks): - """The run cancelled as Harbor's timeout cancels it, while the stand-in CLI waits on a child of its own: the - kill reaches the child (the group, not just the CLI), and the collected and returned files still come back.""" - r = report(checks, "shell_cancel") - assert r["raised"][0] == "CancelledError" and r["seconds"] < 30 - assert r["child_alive"] is False # the group, not just the CLI - assert r["metadata"]["killed"] is True and r["pidfile_left"] is False - assert r["collected"] == ["last_message.txt"] and r["returned"] == '{"refreshed": true}\n' - - -def test_the_subscription_token_reaches_the_cli_on_its_descriptor_and_in_no_environment(checks): - """Claude Code's env_by_fd token: by name to compose, into a pipe on descriptor 3, out of the environment of the - CLI and of the shell that started it (the one that had it was replaced), never in an argument or a file.""" - r = report(checks, "shell_token") - assert r["raised"] is None - lines = r["stream"].split("\n") - for line in (f"fd-sha: {r['token_sha']}", "token-in-env: ", "parent-environ-token: 0", - "argv: <-p> <--output-format> "): - assert line in lines, line - assert f"stdin: <{INSTRUCTION + SUFFIX}>" in r["stream"] - assert r["leaks"] == [] - - -def test_every_clis_launch_script_gives_the_session_the_variables_its_plan_names_on_every_backend(tmp_path, - monkeypatch): - """For real with bash, each CLI's plan as Hermit's side renders it on each built-in backend whose dialect the CLI - speaks (every key a fake, passed by name as compose passes it), and a plan whose by-name entries name each other: - the session sees each by-name variable with its source's value as Harbor passed it, each of the plan's own - variables with its value, and no source the launch script unset. Claude Code on Anthropic's own API ran every - session without a token: the plan's own ANTHROPIC_API_KEY (emptied) was exported before ANTHROPIC_AUTH_TOKEN read - the key from it.""" - from hermit.agent.model import backends - from hermit.harbor import launch - registry = backends.registry(user_file=None, environ={}) # the built-ins, nothing of this machine's - own = ("SECOND_TOKEN", "THIRD_TOKEN", "OWN_KEY", "ONLY_SOURCE_KEY") # the chained plan's sources - keys = {name: f"fake-{name.lower()}-0123456789" for name in (*(b.api_key_env for b in registry.values()), *own)} - for name, value in keys.items(): - monkeypatch.setenv(name, value) - scratch = tmp_path / "launch_env" - for cli in CLASSES: - for backend in registry.values(): - binding = backends.resolve(f"{backend.name}/test-model", registry) - if backends.needs(cli, binding) in binding.backend.apis: - launch.render(cli, launch.settings_for(cli, binding=binding, web_search=False), deploy=None, - tools_url=None, job=scratch / "jobs" / f"{cli}__{backend.name}", workdir="/app") - # A source another entry sets (read before it is set), so a target that is a source, one kept under its own name, - # a source the plan's own variables empty, and one no entry sets (unset) - write_plan(scratch / "jobs" / "chained", "codex", deploy=False, name="launch/plan.json", - env={"CODEX_HOME": "/installed-agent/hermit-home", "PROJECT": "@@HERMIT_WORKDIR@@/src", - "THIRD_TOKEN": ""}, - env_from_host={"FIRST_TOKEN": "SECOND_TOKEN", "SECOND_TOKEN": "THIRD_TOKEN", "OWN_KEY": "OWN_KEY", - "RENAMED_KEY": "ONLY_SOURCE_KEY"}) - (scratch / "keys.json").write_text(json.dumps(keys)) - got = harbor_checks(["launch_env"], tmp_path)["launch_env"] - assert "check_failed" not in got, got.get("check_failed") - plans = {p.parent.parent.name: json.loads(p.read_text()) for p in scratch.glob("jobs/*/launch/plan.json")} - assert set(got) == set(plans) and {name.split("__")[0] for name in plans} == {*CLASSES, "chained"} - assert {"claude__anthropic", "chained"} <= {n for n, p in plans.items() if set(p["env_from_host"].values()) & - set(p["env"])} # a source that is a plain variable too - for name, plan in plans.items(): - case, by_name = got[name], plan["env_from_host"] - seen = case["seen"] - assert case["code"] == 0 and seen.get("HERMIT_HOOK_LOGS") == "/logs/agent", (name, case["said"]) - assert {var: seen.get(var) for var in by_name} == {var: keys[src] for var, src in by_name.items()}, name - plain = {k: v.replace("@@HERMIT_WORKDIR@@", "/app") for k, v in plan["env"].items()} | case["session_env"] - assert {k: seen.get(k) for k in plain} == plain, name - assert not (set(by_name.values()) - set(by_name) - set(plain)) & set(seen), name - - -def test_each_class_installs_its_cli_or_uses_a_mounted_binary_of_the_asked_for_version(checks): - r = report(checks, "installs") - for name, got in r.items(): - if isinstance(got, dict) and "raised" in got: - assert got["raised"] is None, (name, got["raised"]) - - def root(name): - return [e[1] for e in r[name]["execs"] if e[0] == "root"] - - def agent(name): - return [e[1] for e in r[name]["execs"] if e[0] is None] - - assert r["codex_mounted"]["mounted"] is True - assert "mkdir -p /usr/local/bin && ln -sf /opt/hermit-codex/bin/codex /usr/local/bin/codex" in root("codex_mounted") - assert any("[ -x /opt/hermit-codex/codex-path/rg ]" in c for c in root("codex_mounted")) - assert r["codex_mounted"]["uploads"] == [] and not any("npm" in c for c in agent("codex_mounted")) - assert r["codex_bare"]["uploads"] == ["/etc/ssl/certs/ca-certificates.crt"] # the host's CA bundle - assert r["codex_other"]["mounted"] is False - assert any("npm install -g @openai/codex@0.164.0" in c for c in agent("codex_other")) - assert r["claude_mounted"]["mounted"] is True - assert "mkdir -p /usr/local/bin && ln -sf /opt/hermit-claude/claude /usr/local/bin/claude" in root("claude_mounted") - assert any("bootstrap.sh | bash -s -- 2.1.318" in c for c in agent("claude")) - assert any("npm install -g --ignore-scripts @earendil-works/pi-coding-agent@1.5.0" in c for c in agent("pi")) - assert r["pi_mounted"]["mounted"] is True - assert "mkdir -p /usr/local/bin && ln -sf /opt/hermit-pi/pi /usr/local/bin/pi" in root("pi_mounted") - opencode = next(c for c in root("opencode") if "opencode-linux" in c) - assert 'url="https://github.com/anomalyco/opencode/releases/download/v2.2.6/$asset"' in opencode - assert not any("apt-get install" in c for c in root("opencode")) - assert "apt-get update && apt-get install -y curl ca-certificates" in root("opencode_bare") - assert r["opencode_mounted"]["mounted"] is True - pattern = "https://github.com/anomalyco/opencode/releases/download/v2.2.6/opencode-linux-{}.tar.gz" - assert r["opencode_url_musl"] in (pattern.format("x64-musl"), pattern.format("arm64-musl")) - assert r["opencode_url_glibc"] in (pattern.format("x64"), pattern.format("arm64")) - dsh = next(c for c in agent("dsh") if "@deepseek-ai/dsh" in c) - assert "nvm install 24 && nvm alias default 24" in dsh and "npm install -g @deepseek-ai/dsh@0.2.2-rc.1" in dsh - assert "if (a < 24 || (a === 24 && b < 9))" in dsh # Node 24.9 or later - assert r["dsh_musl"]["code"] == 1 and "this image is musl" in r["dsh_musl"]["stderr"] - assert r["dsh_musl"]["curl"] == "" # refused before any download - assert "https://raw.githubusercontent.com/nvm-sh/nvm/" in r["dsh_glibc"]["curl"] # glibc: on to nvm's installer - assert r["dsh_mounted"]["mounted"] is True - assert any("uv tool install --python 3.12 mini-swe-agent==2.4.6" in c for c in agent("mini")) - assert r["setup_timeouts"]["dsh"] == 600.0 - assert r["path_unmounted"] == ['export PATH=/x/bin:"$PATH"'] - assert r["path_mounted"] == ['export PATH=/x/bin:"$PATH"', 'export PATH=/opt/hermit-codex/bin:"$PATH"'] - - -def test_a_mounted_binarys_version_is_read_among_other_lines_of_its_exec(checks): - """Harbor's docker exec merges stderr into stdout, and an image's BASH_ENV may echo: lines before and after what - the mount says to --version (Codex's parse takes the first line, Pi's the last) do not reject a mount of the - asked-for version, whose own lines are read between marks of their own.""" - r = report(checks, "installs") - for name in ("codex_mounted_noisy", "pi_mounted_noisy"): - assert r[name]["raised"] is None and r[name]["mounted"] is True, name - - -def test_a_musl_image_is_told_by_its_ldd_under_harbors_pipefail(checks): - """musl's ldd says what it is on stderr and exits 1: under the pipefail Harbor's exec_as_root and exec_as_agent - put first (dsh's install sets its own), an image that is musl without being Alpine is still found to be musl, so - OpenCode's musl build is the one downloaded and dsh refuses the image before any download.""" - r = report(checks, "installs") - assert r["musl_test"] == {"musl": "musl", "glibc": "glibc"} - assert r["opencode_url_musl"].endswith("-musl.tar.gz") and not r["opencode_url_glibc"].endswith("-musl.tar.gz") - assert r["dsh_musl"]["code"] == 1 and "this image is musl" in r["dsh_musl"]["stderr"] - assert r["dsh_musl"]["curl"] == "" - - -def test_an_install_that_fails_is_a_launch_failure_not_the_agents_zero(checks): - """What Harbor's setup runs before the agent phase (dsh refusing a musl image, Pi's npm install failing) is not the - agent's doing: it raises HermitLaunchError naming the install, which Hermit's side counts as infra (launch_failed), - not as the agent's 0 that a run would keep; Harbor's setup timeout cancelling the install stays its timeout.""" - r = report(checks, "install_failures") - for name, cli, said in (("dsh_musl", "dsh", "this image is musl"), ("pi_npm", "pi", "No matching version found")): - assert r[name] is not None and r[name][0] == "HermitLaunchError", (name, r[name]) - assert r[name][1].startswith(f"hermit: installing {cli} failed: NonZeroAgentExitCodeError: Command failed " - "(exit 1): "), name - assert said in r[name][1], name - assert verdict(r[name]) == ("infra", "launch_failed"), name - assert r["pi_setup_timeout"][0] == "TimeoutError" - - -def test_a_mounted_binary_that_did_not_serve_is_not_the_sessions_program_the_installed_cli_is(checks): - """launch.render makes the mounted binary's path the command's program: when the mount is another version here - (the class's install ran instead), the session runs the installed CLI found on PATH, not the mount; a mount that - served stays the program.""" - r = report(checks, "mount_rejected") - rejected, served = r["rejected"], r["served"] - assert rejected["raised"] is None and rejected["mounted"] is False - assert ' setsid codex exec -C /app "$HERMIT_PROMPT_TEXT" > /logs/agent/agent-codex.jsonl 2>&1 < /dev/null &' \ - in rejected["launch"].splitlines() - assert "/opt/hermit-codex" not in rejected["launch"] - assert served["raised"] is None and served["mounted"] is True - assert (' setsid /opt/hermit-codex/bin/codex exec -C /app "$HERMIT_PROMPT_TEXT" > /logs/agent/agent-codex.jsonl ' - '2>&1 < /dev/null &') in served["launch"].splitlines() - assert 'export PATH=/opt/hermit-codex/bin:"$PATH"' in served["launch"].splitlines() - - -def test_a_mounted_binary_the_sessions_user_may_not_run_is_not_used_and_the_class_installs_the_cli(checks): - """A mount of the asked-for version that only root may run (mode 700, a uid of this machine's), in a task whose - agent user is not root (Harbor's default user for the install as for the session): its version, asked as root, is - the asked-for one, but that user's `test -x` fails, so the mount is not used, the class's install runs and the - session's program is the installed CLI found on PATH, where the mount would have made every session exit 126.""" - got = report(checks, "mount_rejected")["root_only"] - assert got["asked"] == [["root", "version"], [None, "test -x"]] - assert got["raised"] is None and got["mounted"] is False and got["installed"] is True - assert ' setsid codex exec -C /app "$HERMIT_PROMPT_TEXT" > /logs/agent/agent-codex.jsonl 2>&1 < /dev/null &' \ - in got["launch"].splitlines() - assert "/opt/hermit-codex" not in got["launch"] - - -def test_a_mounted_codex_is_used_though_it_warns_before_its_version_under_an_agent_users_unwritable_home(checks): - """A mount of the asked-for version that the task's agent user may run, in a task whose agent user's HOME is not - writable (HOME=/: a uid without a passwd entry, a home never created): Codex run by that user warns before its - version, a line Harbor's parse_version would take for the version. Its version is asked as root, and only - `test -x` as that user, so the mount is used, nothing is installed (Harbor's install would run under that HOME - too) and the session's program is the mount.""" - got = report(checks, "mount_rejected")["home_unwritable"] - assert got["asked"] == [["root", "version"], [None, "test -x"]] - assert got["raised"] is None and got["mounted"] is True and got["installed"] is False - assert (' setsid /opt/hermit-codex/bin/codex exec -C /app "$HERMIT_PROMPT_TEXT" > /logs/agent/agent-codex.jsonl ' - '2>&1 < /dev/null &') in got["launch"].splitlines() - - -def test_the_launch_script_runs_the_codex_harbors_install_put_under_the_agent_users_nvm(checks): - """Harbor's Codex install on a glibc image puts codex under the agent user's nvm, and none on the image's PATH: - for real with bash, with that user's HOME and a container PATH without codex, the launch script of a plan that - loads nvm (Codex's, as hermit.harbor.launch renders it) runs that codex, and it is the codex on the session's - PATH.""" - r = report(checks, "codex_nvm")["installed"] - [installed] = r["programs"] - assert r["raised"] is None, r["raised"] - assert r["stream"].splitlines() == [f"ran: {installed}", f"codex on PATH: {installed}"] - - -def test_a_mounted_codex_stays_the_sessions_program_and_first_on_its_path_with_nvm_loaded(checks): - """For real with bash, a mounted codex that served, with the agent user's nvm loaded too (its codex installed - there): the session runs the mount, and the mount's folder, put on PATH after nvm's, comes first on it.""" - r = report(checks, "codex_nvm")["mounted"] - mount = r["programs"][1] # after nvm's own codex - assert r["raised"] is None, r["raised"] - assert r["stream"].splitlines() == [f"ran: {mount}", f"codex on PATH: {mount}"] - - -def test_mini_swe_agent_runs_hermits_runner_from_the_bundle_in_its_own_python_or_not_at_all(checks): - """The runner's interpreter (a host path in the plan) becomes mini-swe-agent's uv tool Python, with the bundle - first on PYTHONPATH; when the runner does not import there, the trial is a launch failure saying why (infra on - Hermit's side), and no other agent runs in its place.""" - r = report(checks, "mini_swe_agent") - bundle, no_import = r["bundle"], r["no_import"] - assert bundle["raised"] is None - assert (f" setsid {TOOL_PYTHON} -P -m hermit.agent.base.mini_swe_agent_run --settings " - "/installed-agent/hermit-home/mini-swe-agent.json --cwd /app --last " - "/installed-agent/hermit-home/last_message.txt --stdin > /logs/agent/agent-mini-swe-agent.jsonl 2>&1 < " - "/installed-agent/hermit/run/prompt.txt &") in bundle["launch"].splitlines() - assert "export PYTHONPATH=/installed-agent/bundle:/app/lib" in bundle["launch"].splitlines() - assert 'export PATH=/installed-agent/hermit/bin:"$PATH"' in bundle["launch"].splitlines() - assert "/installed-agent/bundle" in bundle["bundle_upload"] - assert bundle["probe"][0][1] == {"PYTHONPATH": "/installed-agent/bundle:/app/lib"} - assert bundle["tool"].startswith(f"#!{TOOL_PYTHON}\n") and 'os.environ.get("HERMIT_TOOLS_URL"' in bundle["tool"] - assert no_import["raised"][0] == "HermitLaunchError" - assert no_import["raised"][1].startswith("hermit: Hermit's mini-swe-agent runner cannot run in this container: " - f"hermit.agent.base.mini_swe_agent_run does not import in {TOOL_PYTHON}: " - "ModuleNotFoundError") - assert no_import["launch"] is None # no session of anything else - assert bundle["leaks"] == [] and no_import["leaks"] == [] - - -def test_mini_swe_agents_python_is_found_among_other_lines_of_its_exec(checks): - """Harbor's docker exec merges stderr into stdout: mini-swe-agent's Python is read from a marked line, so a line - after it (a warning on stderr) does not keep Hermit's runner from running.""" - noisy = report(checks, "mini_swe_agent")["noisy"] - assert noisy["raised"] is None - assert f" setsid {TOOL_PYTHON} -P -m hermit.agent.base.mini_swe_agent_run --settings " in noisy["launch"] - - -def test_the_hermit_tool_script_lists_and_calls_the_tool_servers_tools(checks): - """The command an adapter's tools are for a CLI that only runs bash (mini-swe-agent), against a stand-in of - Hermit's tool server: the list, a call, and the errors the model sees.""" - r = report(checks, "tool_script") - assert r["list"] == [0, "lookup: Finds a row."] - assert r["call"] == [0, "row 7"] and r["seen"] == [["/tok/call/lookup", {"id": 7}]] - assert r["unknown"] == [1, "Error: ValueError: Unknown tool: nope"] - assert r["bad_json"][0] == 1 and r["bad_json"][1].startswith("Error: JSONDecodeError") - assert r["too_many"][0] == 1 and "as a command of its own" in r["too_many"][1] - assert r["no_url"] == [1, "Error: no tools are available here"] - - -def test_the_hermit_tool_listing_says_error_when_the_tool_server_answers_one(checks): - """GET /tools answered with an HTTP error, in any shape but a list (a JSON object saying so by its "error" or - by FastAPI's "detail", a JSON string, a number): the model reads `Error: ...` saying what was answered, as for any - other error (exit 1), not a traceback from listing the answer as tools.""" - r = report(checks, "tool_script") - assert r["list_error"] == {"gone": [1, "Error: RuntimeError: no such path", ""], - "detail": [1, "Error: RuntimeError: Not Found", ""], - "text": [1, "Error: RuntimeError: Bad Gateway", ""], - "number": [1, "Error: RuntimeError: 5", ""]} - - -def test_the_hermit_tool_listing_says_error_when_the_tool_servers_list_holds_no_tools(checks): - """GET /tools answered with a list whose items are not tools (not objects; an object without a name): the - model reads `Error: ...` saying what was answered (exit 1), as for any other answer that is not the tools, not a - traceback from listing it.""" - r = report(checks, "tool_script") - assert r["list_not_tools"] == {"strings": [1, "Error: RuntimeError: ['lookup']", ""], - "nameless": [1, "Error: RuntimeError: [{'description': 'Finds a row.'}]", ""]} - - -def test_the_answer_agent_asks_once_and_writes_the_formatted_answer_into_the_task(checks): - """LAB-Bench's shape: the question is a file the task's entrypoint writes (waited for), a figure goes with it as - an image data URL (a text file does not), the answer is the reply's last tagged match, written in the tagged - form as the task's agent user; answer.json keeps no image bytes and no endpoint token. Harbor's environment has - an http(s)_proxy that is not there: the endpoint on this machine is reached directly.""" - r = report(checks, "answer") - assert r["raised"] is None and r["polls"] == 3 - path, body = r["request"] - assert path == "/endpoint-token-1f2e3d/chat/completions" - assert body["model"] == "kimi" and body["max_tokens"] == 256 and body["reasoning_effort"] == "high" - assert body["messages"] == [ - {"role": "system", "content": "Context: tables are in /app/data."}, - {"role": "user", "content": [{"type": "text", "text": "Which panel shows the knockdown?\n"}, - {"type": "image_url", "image_url": {"url": r["png_url"]}}]}] - assert r["written"] == "[ANSWER]C[/ANSWER]" and r["writes"][0].endswith("| base64 -d > /app/answer.txt") - assert r["write_users"] == [None] # the task's agent user - assert r["answer_json"] == { - "reply": "First [ANSWER]A[/ANSWER], then on reflection [ANSWER] C [/ANSWER].", "answer": "C", - "usage": {"prompt_tokens": 120, "completion_tokens": 9, "prompt_tokens_details": {"cached_tokens": 100}}, - "model": "kimi-served", "attachments": ["/app/fig.png"]} - context = r["context"] - assert [context["n_input_tokens"], context["n_output_tokens"], context["n_cache_tokens"]] == [120, 9, 100] - assert context["metadata"] == {"plan_digest": "d1", "hermit_cli": "answer", "answer_file": "/app/answer.txt", - "attachments": 1, "instruction_from": "file"} - - -def test_the_answer_agent_without_a_match_or_a_file_and_when_its_model_or_question_is_missing(checks): - """No match: the whole reply is the answer; no answer file: nothing is written. A model that failed or was not - reached raises NonZeroAgentExitCodeError (the verifier still runs), naming no endpoint token; an instruction - file that never comes raises HermitLaunchError after wait_s; a bad answer block is refused at construction.""" - r = report(checks, "answer") - assert r["plain_raised"] is None - assert r["plain_request"]["messages"][1] == {"role": "user", "content": "What is 6 x 7?"} - assert "max_tokens" not in r["plain_request"] and r["plain_answer"]["answer"] == "Just 42." - assert r["plain_execs"] == [] - assert r["failed"][0] == r["error_class"] - assert r["failed"][1].startswith("hermit: the answer's model failed: HTTP 500: ") - assert "upstream exploded" in r["failed"][1] - assert r["unreached"][0] == r["error_class"] - assert r["unreached"][1].startswith("hermit: the answer's model was not reached: ") - assert "endpoint-token" not in r["failed"][1] + r["unreached"][1] - assert r["late"][0] == "HermitLaunchError" and "/app/instruction.md was not there after 0.2 s" in r["late"][1] - assert r["capabilities"] is True and r["name"] == "hermit-answer" - for field in ("endpoint", "model", "attachments", "answer_format", "answer_pattern", "max_tokens"): - assert f"answer.{field}" in r["bad"], field - - -def test_the_answer_agent_reads_the_question_and_a_figure_among_other_lines_of_its_execs(checks): - """Harbor's docker exec merges stderr into stdout, and an image's BASH_ENV may echo: the question, a figure's size - and its bytes are each found on a marked line of their own, for real with bash; the answer is written.""" - r = report(checks, "answer") - assert r["noisy_raised"] is None - assert r["noisy_request"]["messages"][-1]["content"] == [ - {"type": "text", "text": "Which panel shows the knockdown?\n"}, - {"type": "image_url", "image_url": {"url": r["png_url"]}}] - assert r["noisy_written"] == "B" - - -def test_the_answer_agent_in_an_image_without_base64_fails_to_launch_rather_than_ask_an_empty_question(checks): - """For real with bash, no base64 on the container's PATH: without pipefail, `base64 < f | tr -d '\\n'` succeeds - with nothing, so the question and a figure would be read as empty ones. They cannot be read: HermitLaunchError - saying so, at once (not after wait_s), which Hermit's side counts as infra (launch_failed); the model is not - asked.""" - r = report(checks, "answer") - for name in ("no_base64_question", "no_base64_figure"): - assert r[name] is not None and r[name][0] == "HermitLaunchError", (name, r[name]) - assert "no base64 or tr in the image" in r[name][1], name - assert verdict(r[name]) == ("infra", "launch_failed"), name - assert r["no_base64_question"][1].startswith("hermit: the task's instruction file /app/instruction.md could not " - "be read: ") - assert r["no_base64_figure"][1].startswith("hermit: the attachment ") # its path as ShellEnv maps it - assert "/app/fig.png could not be read: hermit: no base64 or tr in the image" in r["no_base64_figure"][1] - assert r["no_base64_asked"] == 0 - - -def test_the_answer_agent_fails_to_launch_rather_than_ask_without_a_figure_it_cannot_measure_or_read(checks): - """A figure the task names goes with the question, or the trial fails to launch (such a figure was left out with - a warning, and the model asked without it): an image without wc (for real with bash, in the C locale: the - figure's size cannot be measured) and, on FakeEnv's stand-ins, a read whose exec failed otherwise (compose's - service had stopped), a measure that exits 0 without its hermit-size line or with one whose value is no number, - and a read that exits 0 without its data line, are HermitLaunchError saying why, which Hermit's side counts as - infra; the model is not asked.""" - r = report(checks, "answer") - stand_ins = ("failed_read_figure", "unmarked_size_figure", "odd_size_figure", "unmarked_data_figure") - for name in ("no_wc_figure", *stand_ins): - raised = r[name]["raised"] - assert raised is not None and raised[0] == "HermitLaunchError", (name, raised) - assert verdict(raised) == ("infra", "launch_failed") and r[name]["asked"] == 0, name - assert "/app/fig.png could not be measured: " in r["no_wc_figure"]["raised"][1] # its path as ShellEnv maps it - assert "wc: command not found" in r["no_wc_figure"]["raised"][1] - assert {name: r[name]["raised"][1] for name in stand_ins} == { - "failed_read_figure": 'hermit: the attachment /app/fig.png could not be read: service "main" is not running', - "unmarked_size_figure": "hermit: the attachment /app/fig.png could not be measured: something else", - "odd_size_figure": "hermit: the attachment /app/fig.png could not be measured: hermit-size:12 abc", - "unmarked_data_figure": "hermit: the attachment /app/fig.png could not be read: something else"} - - -def test_the_answer_agent_fails_to_launch_rather_than_ask_without_any_figure_when_the_listing_exec_failed(checks): - """The globs are listed in one exec whose command cannot fail (it ends in `true`): when that exec failed itself - (compose's service had stopped, on FakeEnv's stand-in), which figures the task has is not known. HermitLaunchError - saying so (infra), where the model was asked without any figure; the model is not asked.""" - got = report(checks, "answer")["failed_listing"] - raised = got["raised"] - assert raised == ["HermitLaunchError", - 'hermit: the attachments /app/*.png could not be listed: service "main" is not running'], raised - assert verdict(raised) == ("infra", "launch_failed") and got["asked"] == 0 - - -@pytest.mark.skipif(os.geteuid() == 0, reason="root opens a file of mode 000") -def test_the_answer_agent_fails_to_launch_rather_than_ask_without_a_figure_its_agent_user_may_not_open(checks): - """For real with bash in the C locale, a figure of mode 000: the task's agent user cannot open it, so its size - cannot be measured. HermitLaunchError saying so (infra), where the figure was left out with a warning and the - model asked without it; the model is not asked.""" - got = report(checks, "answer")["locked_figure"] - raised = got["raised"] - assert raised is not None and raised[0] == "HermitLaunchError", raised - assert "/app/fig.png could not be measured: " in raised[1] and "Permission denied" in raised[1] - assert verdict(raised) == ("infra", "launch_failed") and got["asked"] == 0 - - -def test_the_answer_agent_leaves_out_a_figure_over_the_size_limit_with_a_warning_and_asks_with_the_others(checks): - """For real with bash, a figure a byte over MAX_ATTACHMENT_BYTES (each is sent inline) is left out with a - warning, not a failure: the model is asked with the figure within the limit.""" - r = report(checks, "answer") - assert r["over_limit_raised"] is None - assert r["over_limit_content"] == [{"type": "text", "text": "What is 6 x 7?"}, - {"type": "image_url", "image_url": {"url": r["png_url"]}}] - [warned] = r["over_limit_warned"] - assert "/app/huge.png is " in warned and f"over {r['max_bytes']} bytes: it is not sent" in warned - - -def test_the_answer_agent_asks_once_with_the_plain_question_when_a_glob_matches_no_file(checks): - """For real with bash, a glob of the attachments that matches no file (an /app that holds no figure) is no - failure, where a listing whose exec failed is one: nothing goes with the question, and the model is asked once, - with the plain question as the message's content.""" - r = report(checks, "answer") - assert r["no_match_raised"] is None and r["no_match_asked"] == 1 - assert r["no_match_content"] == "What is 6 x 7?" - - -def test_the_answer_agents_answer_this_machine_could_not_write_or_keep_is_a_launch_failure_not_the_agents_zero(checks): - """The model answered, and then the exec that writes the answer into the task could not be run here (its compose - client not spawned: an OSError from the exec itself, EMFILE, EAGAIN on a loaded host), or answer.json could not be - kept on this machine (its disk quota reached, the answer then not written): HermitLaunchError saying what failed, - which Hermit's side counts as infra (launch_failed), as it does a write the container refused, not an unclassified - agent error scored 0. A link the task's container left at answer.json (the folder is its /logs/agent) is - replaced, never followed onto this machine.""" - r = report(checks, "answer") - for name, number in (("write_unspawned", errno.EMFILE), ("write_unspawned_eagain", errno.EAGAIN)): - got, raised = r[name], OSError(number, os.strerror(number)) # as Python raises it: EAGAIN's BlockingIOError - assert got["raised"] == ["HermitLaunchError", "hermit: the answer could not be written to /app/answer.txt: its " - f"exec could not be run here: {type(raised).__name__}: {raised}"], (name, got) - assert verdict(got["raised"]) == ("infra", "launch_failed") and got["kept"] is True, name - refused = r["write_refused"] - assert refused["raised"] == ["HermitLaunchError", "hermit: the answer could not be written to /app/answer.txt as " - "the task's agent user: mkdir: cannot create directory '/app': Permission denied"] - assert verdict(refused["raised"]) == ("infra", "launch_failed") and refused["kept"] is True - unkept, raised = r["unkept"], OSError(errno.EDQUOT, os.strerror(errno.EDQUOT)) - assert unkept["raised"] == ["HermitLaunchError", f"hermit: the answer could not be kept in {unkept['logs']}/" - f"answer.json: OSError: {raised}"], unkept - assert verdict(unkept["raised"]) == ("infra", "launch_failed") and unkept["writes"] == [] - planted = r["planted"] # the link replaced, never followed: this machine's file untouched - assert (planted["raised"], planted["outside"], planted["link"], planted["answer"], planted["left"]) == \ - (None, "a file of this machine\n", False, "B", ["answer.json"]) - - -def test_every_class_refuses_unknown_kwargs_a_bad_plan_and_harbors_model_overrides(checks): - """The old deploy_dir is gone (the runner writes `plan`); a plan of another version or CLI, or with a path that - escapes, is refused when the agent is made; Harbor's own model connection is off (no base_url, no credential - check, nothing inferred) because the model is the plan's.""" - r = report(checks, "options") - names = {"codex": "codex", "claude": "claude-code", "pi": "pi", "opencode": "opencode", "dsh": "dsh", - "mini-swe-agent": "mini-swe-agent"} - for cli, got in r["classes"].items(): - assert "Unknown option 'bogus'" in got["refused"]["bogus"], cli - assert "Unknown option 'deploy_dir'" in got["refused"]["deploy_dir"], cli - assert "is not a file" in got["refused"]["missing_plan"], cli - assert got["name"] == names[cli] and got["import_path"] == ":".join(CLASSES[cli]), cli - assert got["bin_option"] is (cli != "mini-swe-agent"), cli - assert "is version 2 and this harborside reads version 1" in r["version"] - assert "is for claude, and this agent runs codex" in r["cli"] - assert "stream must be a file name in /logs/agent" in r["stream"] - assert "home.secret_files must be paths relative to the home" in r["secret_path"] - assert "env_from_host has a variable name that is not one" in r["env_name"] - assert r["base_url"] == "Agent 'codex' does not support inference connection overrides" - assert r["preflight"] == "passed" and r["connection"] == [None, None, None] - assert r["no_plan"][0] == "HermitLaunchError" and r["no_plan"][1].endswith("(kwargs.plan), and it was given none") - - -def test_scripted_agent_runs_the_script(checks): - r = report(checks, "scripted") - assert r["uploads"] == [["/logs/agent/instruction.md", "Write hi."]] - run = [e for e in r["execs"] if e[1].startswith("bash -c")] - assert run and run[0][1] == "bash -c 'echo hi > /app/answer.txt'" - assert run[0][2]["HERMIT_INSTRUCTION_FILE"] == "/logs/agent/instruction.md" - assert r["exit_code"] == 3 and "[exit 3]" in r["script_txt"] and "script ran" in r["script_txt"] - assert r["context"]["n_input_tokens"] == 0 and r["context"]["metadata"]["exit_code"] == 3 - assert r["name"] == "scripted" - assert "bogus" in r["unknown"] - - -def test_driver_answers_requests_in_order(checks): - r = report(checks, "driver_protocol") - assert r["before_ready"]["ready"] is False and r["ready"]["ready"] is True and r["ready"]["next_index"] == 1 - assert r["out_of_order"] == {"ok": False, "out_of_order": True, "next_index": 1} - assert r["busy"] == {"ok": False, "busy": "s1", "next_index": 1} - assert r["stored"]["index"] == 1 and r["missing"]["ok"] is False - assert r["queued"] == ["s1", {"ok": True, "index": 1, "step": {}}] - assert r["errored"] == {"ok": False, "error": "SandboxBuildFailedError: no image"} - assert r["stop_after_error"]["stopped"] is True - assert r["finished"]["ok"] is False and "finished" in r["finished"]["error"] - assert "unknown op" in r["unknown"]["error"] - - -def test_the_driver_gives_each_step_its_own_launch_plan(checks): - """A stream keeps one agent: each step's plan (in the request, or in the agent kwargs the stream client sends) - replaces the last one before the step runs; a step that gives none keeps it.""" - r = report(checks, "driver_plan") - assert r["from_kwargs"] == ["codex", "exec", "second"] - assert r["from_request"] == ["codex", "exec", "-C", "@@HERMIT_WORKDIR@@", "@@HERMIT_PROMPT@@"] - assert r["kept"] == r["from_request"] and r["no_plan_agent"] == "ran" - assert r["hosts"] == ["api.example.com", "10.0.0.0/8"] and r["hosts_kept"] == r["hosts"] and r["hosts_none"] == [] - assert r["bad_hosts"].startswith("HermitLaunchError: the step's allowed hosts: ") - - -def test_a_request_for_a_step_the_task_has_not_gets_an_error_reply_and_the_driver_goes_on(checks): - """An unknown step name is answered (not left for the runner to wait out) and the loop serves the next request.""" - r = report(checks, "driver_loop") - assert r["unknown"]["ok"] is False and "nope" in r["unknown"]["error"] - assert r["next"]["ok"] is True and r["next"]["index"] == 1 and r["ran"] == ["s1"] - - -def test_a_steps_launch_plan_the_agent_cannot_take_is_a_launch_failure_of_the_step(checks): - """A plan the agent refuses (another CLI's) fails the step as HermitLaunchError, which Hermit's side counts as - infra (launch_failed), not as the agent's 0; the step is not run.""" - r = report(checks, "driver_loop") - step = r["bad_plan"]["step"] - assert step["exception_info"]["exception_type"] == "HermitLaunchError" - assert "is for claude, and this agent runs codex" in step["exception_info"]["exception_message"] - assert (errors.classify(step)["status"], errors.classify(step)["failure_cause"]) == ("infra", "launch_failed") - assert r["ran"] == ["s1"] and r["loop_done"] is True and "loop_error" not in r - - -def test_the_driver_gives_the_streams_trial_the_runs_timeout_multipliers(checks): - """The runner starts the driver with --trial-json: the verifier's and the environment build's timeout multipliers - reach the stream's TrialConfig, as a job's trials get them; none given, Harbor's defaults; another key is refused, - saying which ones it takes, before any trial.""" - r = report(checks, "driver_config") - assert r["given"]["multipliers"] == [2.5, 3.0] - assert r["default"]["multipliers"] == [None, None] - unknown = r["unknown"] - assert unknown["multipliers"] is None and unknown["code"] == 2 - assert "agent_timeout_multiplier" in unknown["log"] - assert "verifier_timeout_multiplier, environment_build_timeout_multiplier" in unknown["log"] - - -def test_a_full_disk_refusing_the_drivers_log_a_steps_reply_or_its_move_still_answers_the_step_and_ends_the_driver( - checks): - """serve() end to end on a one-step stream whose disk fills up. A log line the driver's stderr refuses is lost: the - failed step is answered with its result. A step whose reply cannot be kept in replies/, or whose agent/ cannot be - moved under the failed step, is answered with an error saying so (infra on Hermit's side). Either way the request - is answered, not left for the client to wait out its time, and the driver ends after its last step, as a stream's - driver does, instead of waiting for good on the open connection, its record saying it is ending.""" - r = report(checks, "driver_full_disk") - for name, got in r.items(): - assert isinstance(got["answer"], dict), (name, got) # answered, not left to wait - log, reply, move = r["log"]["answer"], r["reply"]["answer"], r["move"]["answer"] - assert log["ok"] is True and log["aborted"] == "step s1 failed" and log["last"] is True, log - assert log["step"]["exception_info"]["exception_type"] == "RuntimeError" - assert reply["ok"] is False and set(reply) == {"ok", "error"}, reply - assert reply["error"].startswith("step s1's result could not be kept: NotADirectoryError: [Errno 20] Not a " - "directory: "), reply - assert move == {"ok": False, "error": "step s1's result could not be kept: OSError: [Errno 28] No space left on " - "device"} - for name, got in r.items(): - assert got["code"] == 0 and got["recorded"] is None, (name, got) - assert got["loop_done"] is True and got["finished"] is True, (name, got) - # its record said it was ending before its last answers (_mark_ending): no client takes it for a live driver - # in the moment its socket stays open, and the stream's next driver waits for this one to have exited - assert got["ending"] is True, (name, got) - - -def test_harborside_is_harbor_and_stdlib_only(): - """harborside runs in Harbor's venv: it imports harbor, its own modules and the standard library, nothing of - Hermit's. pydantic is Harbor's API for an agent's options; certifi is an optional fallback (its ImportError is - handled). Every module keeps its docstring.""" - import ast - allowed = set(sys.stdlib_module_names) | {"harbor", "harborside", "pydantic", "certifi"} - for module in sorted((HARBORSIDE_ROOT / "harborside").glob("*.py")): - tree = ast.parse(module.read_text()) - imported = set() - for node in ast.walk(tree): - if isinstance(node, ast.Import): - imported |= {alias.name.split(".")[0] for alias in node.names} - elif isinstance(node, ast.ImportFrom): # a relative import is harborside's own - imported.add("harborside" if node.level else node.module.split(".")[0]) - assert imported <= allowed, (module.name, sorted(imported - allowed)) - assert ast.get_docstring(tree), module.name diff --git a/tests/core/test_harbor_launch.py b/tests/core/test_harbor_launch.py deleted file mode 100644 index 2373e032..00000000 --- a/tests/core/test_harbor_launch.py +++ /dev/null @@ -1,474 +0,0 @@ -"""A Harbor trial's launch plan (hermit.harbor.launch) and the per-CLI table (hermit.harbor.clis), without Docker or -Harbor: what each CLI's plan holds as Hermit's own CLI Base Agents render it, that no secret's value and no path of -this machine reaches the container, the sign-ins (ChatGPT's file that comes back, Claude's token on a descriptor), -the memory note, the mini-swe-agent bundle's imports, the answer agent's plan, and what the Base Agents' run() refuses -before a session (check_settings).""" -import ast -import json -import os -from pathlib import Path -import stat -import sys - -import pytest - -from hermit.harbor import clis, launch - -KEY_ENV, KEY = "FAKE_MODEL_KEY", "fake-model-key-0123456789abcdef" -TOKEN = "fake-setup-token-never-real" -BINDING = {"ref": "fake/glm-5", "backend": "fake", "model": "glm-5", "base_url": "https://models.example/v1", - "api_key_env": KEY_ENV, "key_required": True, "apis": ["chat", "responses", "messages"], - "price": {"input": 1.0, "output": 2.0}} -CLIS = ("codex", "claude", "pi", "opencode", "dsh", "mini-swe-agent") -URL = "http://172.17.0.1:41234/" + "ab" * 16 + "/mcp" # a tool server's MCP URL (its token: 32 hex characters) - - -@pytest.fixture(autouse=True) -def key(monkeypatch): - """The model's key is set: a plan names its variable, never its value.""" - monkeypatch.setenv(KEY_ENV, KEY) - - -def settings(cli: str, **kwargs) -> dict: - from hermit.agent.model.backends import Binding - return launch.settings_for(cli, binding=Binding.from_spec(BINDING), web_search=False, **kwargs) - - -def plan_of(path: Path) -> dict: - return json.loads(Path(path).read_text()) - - -def launch_texts(job: Path) -> str: - """Every file the launch left under the job (the plan, the home, the bundle), joined.""" - return "\n".join(p.read_text(errors="replace") for p in (job / "launch").rglob("*") if p.is_file()) - - -def staged(root: Path, adapter_id: str = "demo", memory: bool = False) -> dict: - """A staging record (hermit.harbor.staging.stage) of a deployment with the adapter's copy, and its notes.""" - deploy = root / "deploy" - (deploy / "hermit" / adapter_id).mkdir(parents=True) - (deploy / "AGENTS.md").write_text("# demo\n") - if memory: - (deploy / "hermit" / adapter_id / "memory").mkdir() - (deploy / "hermit" / adapter_id / "memory" / "NOTES.md").write_text("- notes\n") - return {"host": "codex", "adapter_id": adapter_id, "deploy": str(deploy), "hooks": False, "skills": [], - "tools": ["calc"]} - - -# ---- the table --------------------------------------------------------------------------------------------------- - -def test_the_table_names_one_class_per_cli(): - assert clis.cli_of({}) == "codex" and clis.cli_of(None) == "codex" - assert clis.cli_of({"cli": "dsh"}) == "dsh" - assert clis.cli_of({"import_path": "harborside.claude:HermitClaude"}) == "claude" - assert clis.cli_of({"import_path": "harborside.pi:HermitPi"}) == "pi" - assert clis.cli_of({"name": "terminus-2"}) is None - assert clis.cli_of({"import_path": "my_agents:Agent"}) is None # another class: Harbor's way, no plan - with pytest.raises(ValueError, match="give one"): - clis.cli_of({"cli": "codex", "name": "codex"}) - with pytest.raises(ValueError, match="one of"): - clis.cli_of({"cli": "aider"}) - assert clis.bin_mount("opencode") == "/opt/hermit-opencode" - assert all(clis.entry(c)["context_file"] for c in ("codex", "claude", "pi", "opencode", "dsh")) - - -# ---- every CLI's plan -------------------------------------------------------------------------------------------- - -@pytest.mark.parametrize("cli", CLIS) -def test_every_cli_renders_a_plan_for_the_container(tmp_path, cli): - job = tmp_path / "job" - plan = plan_of(launch.render(cli, settings(cli), deploy=None, tools_url=URL, job=job, tools_name="demo")) - backend = clis.entry(cli)["backend"] - assert (plan["version"], plan["cli"], plan["stream"]) == (1, cli, f"agent-{backend}.jsonl") - assert plan["metadata"] == {"hermit_cli": cli, "backend": backend} - # The prompt goes on stdin; the home is the container's - assert plan["stdin"] is True and launch.PROMPT not in json.dumps(plan["command"]) - # Claude Code runs a prompt that starts with `/` as one of its commands: its Base Agent's line break goes first - assert plan["prompt_slash_prefix"] == ("\n" if cli == "claude" else None) - assert plan["home"] == {"host_dir": str((job / "launch" / "home").resolve()), "container_dir": clis.CONTAINER_HOME, - "secret_files": []} - # Nothing of this machine's job where the container reads it, and the key by its name only - assert str(job.resolve()) not in json.dumps([plan["command"], plan["env"]]) - assert KEY not in launch_texts(job) - expected = {"ANTHROPIC_AUTH_TOKEN": KEY_ENV} if cli == "claude" else {KEY_ENV: KEY_ENV} - assert plan["env_from_host"] == expected and plan["env_by_fd"] == {} - assert plan["nvm"] is clis.entry(cli)["nvm"] and plan["kill_grace_s"] == launch.KILL_GRACE_S - # The tools' server is named where the CLI looks for it: its MCP entry, or for mini-swe-agent's shell the JSON one - home = Path(plan["home"]["host_dir"]) - named = "\n".join(p.read_text(errors="replace") for p in home.rglob("*") if p.is_file()) - url = URL.removesuffix("/mcp") if cli == "mini-swe-agent" else URL - assert url in named + json.dumps(plan["command"]) + json.dumps(plan["env"]) - assert plan["tools_script"] == ({"url_env": "HERMIT_TOOLS_URL"} if cli == "mini-swe-agent" else None) - - -def test_the_table_settings_and_the_runs_own_reach_the_cli(tmp_path): - codex = plan_of(launch.render("codex", settings("codex", effort="high", extra={"args": ["-c", "x=1"]}), - deploy=None, tools_url=None, job=tmp_path / "c")) - command = codex["command"] - assert command[:2] == ["codex", "exec"] and command[-1] == "-" and "-o" in command - assert command[command.index("-s") + 1] == "danger-full-access" # the container is the boundary - assert 'model_reasoning_effort="high"' in command and "x=1" in command and 'web_search="disabled"' in command - assert codex["collect"] == [{"from": f"{clis.CONTAINER_HOME}/last_message.txt", "to": "last_message.txt"}] - assert command[command.index("-o") + 1] == f"{clis.CONTAINER_HOME}/last_message.txt" - assert codex["env"]["CODEX_HOME"] == clis.CONTAINER_HOME - claude = plan_of(launch.render("claude", settings("claude"), deploy=None, tools_url=None, job=tmp_path / "cl")) - assert claude["env"]["IS_SANDBOX"] == "1" and claude["env"]["ANTHROPIC_BASE_URL"] == "https://models.example" - opencode = plan_of(launch.render("opencode", settings("opencode"), deploy=None, tools_url=None, - job=tmp_path / "o")) - assert opencode["collect"] == [{"from": f"{clis.CONTAINER_HOME}/data/opencode/{name}", "to": name} # WAL mode - for name in ("opencode.db", "opencode.db-wal", "opencode.db-shm")] - dsh = plan_of(launch.render("dsh", settings("dsh"), deploy=None, tools_url=None, job=tmp_path / "d")) - assert "danger-full-access" in json.dumps(dsh) and dsh["nvm"] is True - assert launch.context_limit("dsh", {}) == clis.entry("dsh")["context_limit"] - assert launch.context_limit("codex", settings("codex")) == 262144 and launch.context_limit("pi", {}) is None - - -def test_codexs_plan_loads_the_agent_users_nvm_where_harbors_install_puts_codex(tmp_path): - """Harbor's Codex install puts codex under the agent user's nvm on a glibc image and links none into - /usr/local/bin: the plan loads nvm, and so does a mounted codex_bin's (its program and folder the mount's).""" - installed = plan_of(launch.render("codex", settings("codex"), deploy=None, tools_url=None, job=tmp_path / "i")) - assert installed["nvm"] is True and installed["command"][0] == "codex" and installed["path_prepend"] == [] - mounted = plan_of(launch.render("codex", settings("codex"), deploy=None, tools_url=None, job=tmp_path / "m", - bin_path="/opt/hermit-codex/bin/codex", path_prepend=["/opt/hermit-codex/bin"])) - assert mounted["nvm"] is True and mounted["command"][0] == "/opt/hermit-codex/bin/codex" - assert mounted["path_prepend"] == ["/opt/hermit-codex/bin"] - - -def test_mini_swe_agent_runs_hermits_runner_from_its_bundle(tmp_path): - plan = plan_of(launch.render("mini-swe-agent", settings("mini-swe-agent"), deploy=None, tools_url=None, - job=tmp_path / "job")) - assert plan["command"][:4] == ["python3", "-P", "-m", "hermit.agent.base.mini_swe_agent_run"] - assert plan["bundle"]["container_dir"] == launch.BUNDLE_DIR and plan["env"]["PYTHONPATH"] == launch.BUNDLE_DIR - bundle = Path(plan["bundle"]["host_dir"]) - files = sorted(p.relative_to(bundle).as_posix() for p in bundle.rglob("*") if p.is_file()) - assert files == sorted(launch.MINI_BUNDLE) - # The runner imports nothing of Hermit the bundle lacks; the rest is the standard library, PyYAML, mini-swe-agent - bundled = {".".join(Path(rel).with_suffix("").parts).removesuffix(".__init__") - for rel in launch.MINI_BUNDLE if rel.endswith(".py")} - allowed = {*sys.stdlib_module_names, "yaml", "minisweagent", "litellm"} - for rel in (r for r in launch.MINI_BUNDLE if r.endswith(".py")): - tree = ast.parse((bundle / rel).read_text()) - lazy = {id(n) for f in ast.walk(tree) if isinstance(f, ast.FunctionDef) and f.name == "__getattr__" - for n in ast.walk(f)} # a package's attribute hook (hermit.Session): not run - for node in ast.walk(tree): - if id(node) in lazy: - continue - if isinstance(node, ast.Import): - names = [a.name for a in node.names] - elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: - names = [node.module] - else: - continue - for name in names: - assert (name in bundled) if name.startswith("hermit") else (name.split(".")[0] in allowed), (rel, name) - - -# ---- what never reaches the container ---------------------------------------------------------------------------- - -def test_a_secret_reaches_the_container_by_its_name_only(tmp_path, monkeypatch): - monkeypatch.setenv("SERVICE_TOKEN", "svc-token-0123456789abcdef") - plan = plan_of(launch.render("codex", settings("codex", extra={"env": {"SERVICE_TOKEN": "$SERVICE_TOKEN", - "PLAIN": "$HOME_LIKE", "N": "8"}}), - deploy=None, tools_url=None, job=tmp_path / "job")) - assert plan["env_from_host"]["SERVICE_TOKEN"] == "SERVICE_TOKEN" and plan["env"]["N"] == "8" - assert "svc-token-0123456789abcdef" not in launch_texts(tmp_path / "job") - with pytest.raises(ValueError, match="only whole"): # a secret inside other text would be written out - launch.render("codex", settings("codex", extra={"env": {"HEADER": "Bearer $SERVICE_TOKEN"}}), deploy=None, - tools_url=None, job=tmp_path / "bad") - # A setting that writes a value or a path of this machine where the container reads it: the plan is refused - with pytest.raises(RuntimeError, match="value of SERVICE_TOKEN"): - launch.render("codex", settings("codex", extra={"args": ["-c", "token=svc-token-0123456789abcdef"]}), - deploy=None, tools_url=None, job=tmp_path / "leak") - with pytest.raises(RuntimeError, match="path of this machine"): - launch.render("codex", settings("codex", extra={"args": ["-c", f"x={tmp_path / 'path' / 'launch' / 'x'}"]}), - deploy=None, tools_url=None, job=tmp_path / "path") - - -def test_a_chatgpt_sign_in_is_a_secret_file_that_comes_back(tmp_path): - from hermit.agent.base import login - from hermit.harbor import signin - user = Path(os.environ["CODEX_HOME"]) / "auth.json" # the test's own (tests/conftest.py) - user.parent.mkdir(parents=True, exist_ok=True) - user.write_text(json.dumps({"last_refresh": "2026-01-01T00:00:00Z", "tokens": {"refresh_token": "rt-stand-in"}})) - job = tmp_path / "job" - plan = plan_of(launch.render("codex", settings("codex", login="chatgpt"), deploy=None, tools_url=None, job=job, - login="chatgpt")) - home = Path(plan["home"]["host_dir"]) - assert plan["home"]["secret_files"] == ["auth.json"] and "rt-stand-in" in (home / "auth.json").read_text() - assert plan["return_files"] == [{"from": f"{clis.CONTAINER_HOME}/auth.json", # no larger than a sign-in - "to": str((job / "launch" / "return" / "auth.json").resolve()), - "max_bytes": signin.MAX_BYTES}] - assert "rt-stand-in" not in (job / "launch" / "plan.json").read_text() - assert login.master(login.ROOT).is_file() # the shared copy the home got - # without the sign-in (a check): no secret file at all - dry = plan_of(launch.render("codex", settings("codex", login="chatgpt"), deploy=None, tools_url=None, - job=tmp_path / "dry", login="chatgpt", sign_in=False)) - assert dry["home"]["secret_files"] == [] and not (tmp_path / "dry" / "launch" / "home" / "auth.json").exists() - # a plan refused after the sign-in was copied leaves no copy of it - with pytest.raises(RuntimeError): - launch.render("codex", settings("codex", login="chatgpt", extra={"args": [str(tmp_path / "bad" / "x")]}), - deploy=None, tools_url=None, job=tmp_path / "bad", login="chatgpt") - assert not (tmp_path / "bad" / "launch" / "home" / "auth.json").exists() - - -def test_a_claude_subscription_token_goes_on_a_descriptor(tmp_path, monkeypatch): - from hermit.agent.base import cli - with pytest.raises(ValueError, match="setup-token"): - launch.render("claude", launch.settings_for("claude", model="claude-haiku-5-5", login="claude", - web_search=False), - deploy=None, tools_url=None, job=tmp_path / "none", login="claude") - monkeypatch.setenv("CLAUDE_CODE_OAUTH_TOKEN", TOKEN) - job = tmp_path / "job" - plan = plan_of(launch.render("claude", launch.settings_for("claude", model="claude-haiku-5-5", login="claude", - web_search=False, effort="medium"), - deploy=None, tools_url=None, job=job, login="claude")) - assert plan["env_by_fd"] == {"CLAUDE_CODE_OAUTH_TOKEN": cli.CLAUDE_TOKEN_FD} - assert "CLAUDE_CODE_OAUTH_TOKEN" not in plan["env"] and plan["env_from_host"] == {} - assert TOKEN not in launch_texts(job) and plan["command"][-2:] == ["--settings", cli.CLAUDE_LOCK] - assert "--model" in plan["command"] and "claude-haiku-5-5" in plan["command"] - - -def test_the_memory_note_goes_with_the_prompt_and_the_notes_come_back(tmp_path): - from hermit.agent.acting.workspace import MEMORY_NOTE - job = tmp_path / "job" - plan = plan_of(launch.render("codex", settings("codex"), deploy=staged(job, memory=True), tools_url=None, job=job)) - assert plan["prompt_suffix"] == "\n" + MEMORY_NOTE.format(path="hermit/demo/memory/NOTES.md") - assert plan["deploy"] == {"work_dir": str((job / "deploy").resolve()), "adapter_id": "demo", - "context_file": "AGENTS.md", "context_limit": 262144, "context_escaped": None, - "hooks": False, "copy_back": ["hermit/demo/memory"]} - plain = plan_of(launch.render("codex", settings("codex"), deploy=staged(tmp_path / "p"), tools_url=None, - job=tmp_path / "p")) - assert plain["prompt_suffix"] == "" and plain["deploy"]["copy_back"] == [] - - -def test_dshs_context_limit_counts_the_byte_dsh_adds_for_each_closing_tag(tmp_path): - """dsh gives the model the project's instructions in a frame within its cap, each `` of them - escaped by one more byte (hermit.hosts.dsh counts it so): the limit a dsh deployment's context gets, with - Hermit's text's tags, is a text dsh takes whole and not a byte more, and the plan names the tag, so the image's - own file merged in the container is counted so too (harborside).""" - from hermit.agent.base.cli import DSH_INSTRUCTION_BYTES, DSH_INSTRUCTION_FRAME - job = tmp_path / "job" - deploy = staged(job) - ours = "# demo\na and b\n" - (Path(deploy["deploy"]) / "AGENTS.md").write_text(ours) - given = plan_of(launch.render("dsh", settings("dsh"), deploy=deploy, tools_url=None, job=job))["deploy"] - limit = given["context_limit"] - assert given["context_escaped"] == "" - - def fits(size: int) -> bool: # hermit.hosts.dsh's rule, for a text of `size` bytes holding Hermit's tags - return size + DSH_INSTRUCTION_FRAME + ours.count("") <= DSH_INSTRUCTION_BYTES - assert fits(limit) and not fits(limit + 1) - assert launch.context_limit("dsh", {}) == DSH_INSTRUCTION_BYTES - DSH_INSTRUCTION_FRAME # no tag: the frame - - -# ---- the answer agent -------------------------------------------------------------------------------------------- - -def test_the_answer_agents_plan_has_no_command(tmp_path): - plan = plan_of(launch.render_answer(settings("answer", effort="low"), endpoint="http://127.0.0.1:5/" + "0" * 32, - job=tmp_path / "job", system="# context\n", - options={"answer_pattern": r"ANSWER: (\w+)", "answer_format": "{answer}\n", - "attachments": ["/app/*.png"], "wait_s": 5})) - assert plan["command"] is None and plan["home"] is None and plan["stream"] == "answer.json" - assert plan["answer"] == {"endpoint": "http://127.0.0.1:5/" + "0" * 32, "model": "glm-5", "system": "# context\n", - "instruction_file": None, "wait_s": 5.0, "attachments": ["/app/*.png"], - "answer_file": "/app/answer.txt", "answer_pattern": r"ANSWER: (\w+)", - "answer_format": "{answer}\n", "max_tokens": None, "effort": "low"} - with pytest.raises(ValueError, match="answer_format"): - launch.render_answer({}, endpoint="x", job=tmp_path / "b", options={"answer_format": "no slot"}) - with pytest.raises(ValueError, match="takes instruction_file"): - launch.render_answer({}, endpoint="x", job=tmp_path / "b", options={"temperature": 0}) - with pytest.raises(ValueError, match="no CLI to launch"): - launch.render("answer", {}, deploy=None, tools_url=None, job=tmp_path / "b") - - -def test_the_answer_agents_null_options_keep_their_meaning(tmp_path): - """answer_file: null is an answer written nowhere (harborside's answer agent writes none), not the default file; - wait_s: null is the default wait, as an option left out is.""" - def answer(**options) -> dict: - return plan_of(launch.render_answer(settings("answer"), endpoint="http://127.0.0.1:5/" + "0" * 32, - job=tmp_path / "job", options=options))["answer"] - assert answer(answer_file=None)["answer_file"] is None - assert answer(wait_s=None)["wait_s"] == 30.0 - assert (answer()["answer_file"], answer()["wait_s"]) == ("/app/answer.txt", 30.0) - assert answer(answer_file="")["answer_file"] == "/app/answer.txt" # an empty one is none given - - -# ---- a stream's step: the sign-in at its turn ---------------------------------------------------------------------- - -def test_a_steps_plan_lists_the_sign_in_and_takes_the_shared_copy_as_it_is_at_its_turn(tmp_path): - """`sign_in="later"` (a stream's step, rendered when its rollout begins) lists the sign-in as a secret file that - comes back, as a trial's plan does, but copies none into the home; `sign_in` copies the shared copy in when the - step's turn comes, as it is then (an earlier step's session may have refreshed it since): never the one there was - when the plan was rendered.""" - from hermit.agent.base import login - user = Path(os.environ["CODEX_HOME"]) / "auth.json" - user.parent.mkdir(parents=True, exist_ok=True) - user.write_text(json.dumps({"last_refresh": "2026-01-01T00:00:00Z", "tokens": {"refresh_token": "rt-first"}})) - path = launch.render("codex", settings("codex", login="chatgpt"), deploy=None, tools_url=None, job=tmp_path / "s", - login="chatgpt", sign_in="later") - plan = plan_of(path) - home = Path(plan["home"]["host_dir"]) - assert plan["home"]["secret_files"] == ["auth.json"] and not (home / "auth.json").exists() - assert plan["return_files"][0]["to"] == str((tmp_path / "s" / "launch" / "return" / "auth.json").resolve()) - # meanwhile an earlier step's session refreshed the shared copy - login.master(login.ROOT).parent.mkdir(parents=True, exist_ok=True) - login.master(login.ROOT).write_text(json.dumps({"last_refresh": "2026-01-03T00:00:00Z", - "tokens": {"refresh_token": "rt-refreshed"}})) - launch.sign_in(path) - assert "rt-refreshed" in (home / "auth.json").read_text() - # a plan without a sign-in gets none - plain = launch.render("codex", settings("codex"), deploy=None, tools_url=None, job=tmp_path / "p") - launch.sign_in(plain) - assert not (Path(plan_of(plain)["home"]["host_dir"]) / "auth.json").exists() - - -# ---- what the Base Agents' run() refuses --------------------------------------------------------------------------- - -def test_a_plan_is_refused_for_what_the_base_agents_run_would_refuse_before_a_session(tmp_path): - """harborside carries a plan out without the Base Agent's run(), so what run() refuses before any session is - refused when the plan is rendered (launch.check_settings, the same checks and messages): a login of another CLI, - and mini-swe-agent's settings (no login, no web search, a model, a model class whose messages Hermit reads).""" - from hermit.agent.base.interface import Unsupported, make - with pytest.raises(Unsupported, match="does not sign in with login: claude"): - launch.check_settings(make("codex", {"login": "claude"})) - launch.check_settings(make("codex", {"login": "chatgpt"})) - for extra, match in (({"web_search": True}, "no web search"), ({"model_class": "portkey"}, "calls its own service")): - with pytest.raises(Unsupported, match=match): - launch.render("mini-swe-agent", {**settings("mini-swe-agent"), **extra}, deploy=None, tools_url=None, - job=tmp_path / f"m{len(extra)}{match[:3]}") - with pytest.raises(ValueError, match="needs a model"): - launch.check_settings(make("mini-swe-agent", {})) - with pytest.raises(Unsupported, match="Hermit reads litellm"): - launch.check_settings(make("mini-swe-agent", {"model": "openai/gpt-x", "model_class": "openrouter"})) - - -# ---- what main's own sessions get (the round-10 review) ------------------------------------------------------------- - -@pytest.mark.parametrize("cli", ["pi", "dsh"]) -def test_a_trials_provider_settings_are_what_mains_own_session_on_the_same_model_gets(cli, monkeypatch): - """Pi's and dsh's provider settings in a trial's plan are those `main` binds for its own session of the same model - (bind_agent_params over the backends registry): the spec the config's loading keeps drops the backend's - developer_role (Binding.spec), and a backend whose deployments refuse the role (Fireworks: developer_role system) - keeps its `compat` in the trial, as in Hermit's own sessions.""" - from hermit.agent.model.backends import as_binding, bind_agent_params, registry - monkeypatch.setenv("FIREWORKS_API_KEY", "fw-key-0123456789abcdef") - model = "fireworks/accounts/fireworks/models/glm-5p3" - own = bind_agent_params({"model": model}, cli, registry()) - spec = as_binding(model, registry()).spec() # what agent.params.binding holds once loaded - assert "developer_role" not in spec - trial = launch.settings_for(cli, binding=launch.binding_of({"binding": spec}), web_search=False) - assert trial["provider"] == own["provider"] - assert json.dumps(own["provider"]).count("supportsDeveloperRole") == 1 # the role the deployment refuses - - -def test_a_trials_developer_role_is_the_registrys_only_for_its_backend_at_its_url(monkeypatch): - """The developer_role a trial's spec dropped is the backends registry's only for the backend the registry has at - the spec's URL (launch._with_role): the same backend moved to another URL (a proxy, a run config's own - `base_url`), or one only the run config declares, keeps the spec's `keep` (hermit/harbor/FRAMEWORK_SUPPORT.md, - candidate 14).""" - from hermit.agent.model.backends import as_binding, registry - monkeypatch.setenv("FIREWORKS_API_KEY", "fw-key-0123456789abcdef") - spec = as_binding("fireworks/accounts/fireworks/models/glm-5p3", registry()).spec() - moved = {**spec, "base_url": "http://127.0.0.1:9/v1"} - own = {**spec, "backend": "myfw", "ref": "myfw/accounts/fireworks/models/glm-5p3"} - for key in ("binding", "model"): # the spec as `binding`, or as a loaded config's `model` - assert launch.binding_of({key: spec}).backend.developer_role == "system", key - assert launch.binding_of({key: moved}).backend.developer_role == "keep", key - assert launch.binding_of({key: own}).backend.developer_role == "keep", key - - -def test_effort_off_as_yaml_reads_it_is_no_effort_flag_as_in_mains_own_sessions(tmp_path, monkeypatch): - """`effort: off` (YAML's false) is no effort at all, as main's Base Agents take it: Codex gets no - model_reasoning_effort, Claude Code no --effort (and no refusal), the answer agent none.""" - monkeypatch.setenv("CLAUDE_CODE_OAUTH_TOKEN", TOKEN) - codex = launch.settings_for("codex", model="gpt-5.6-luna", effort=False, web_search=False, login="chatgpt") - assert codex["effort"] is False - plan = plan_of(launch.render("codex", settings("codex", effort=False), deploy=None, tools_url=None, - job=tmp_path / "codex")) - assert "model_reasoning_effort" not in json.dumps(plan["command"]) - claude = launch.settings_for("claude", model="claude-haiku-5-5", effort=False, web_search=False, login="claude") - plan = plan_of(launch.render("claude", claude, deploy=None, tools_url=None, job=tmp_path / "claude")) - assert "--effort" not in plan["command"] - answer = launch.render_answer(launch.settings_for("answer", binding=None, model="m", effort=False), - endpoint="http://127.0.0.1:1/v1", job=tmp_path / "answer") - assert plan_of(answer)["answer"]["effort"] is None - - -def test_a_settings_env_secret_is_passed_by_name_only_whatever_its_names_rule(tmp_path, monkeypatch): - """acting_agent.settings.env: a variable inside other text that names a secret by any of the rules is refused - (Harbor's among them: OAUTH_CLIENT_ID, which only its AUTH names one; PASSWD, which only launch.SECRET_VAR does), - as is a value given as it is to a variable named as a secret, a word of its name being a secret's wherever it - stands (HF_TOKEN_READ, OPENAI_API_KEY_2, SECRET_ID); a count named like one (MAX_OUTPUT_TOKENS), or a word that - only holds one (GIT_AUTHOR_NAME, TOKENIZERS_PARALLELISM), is no secret, and another variable inside other text is - this machine's value.""" - monkeypatch.setenv("EXTRA_DIR", "/opt/extra") - for env in ({"NPMRC": "//registry/:_auth=$NPM_AUTH"}, {"DSN": "postgres://u:$DB_PASSWD@db/x"}, - {"HEADER": "client=$OAUTH_CLIENT_ID"}, - {"OPENAI_API_KEY": "sk-local-0123456789"}, {"DB_PASSWD": "hunter2"}, {"NPM_AUTH": "abc"}, - {"HF_TOKEN_READ": "hf_local0123456789"}, {"OPENAI_API_KEY_2": "sk-local-0123456789"}, - {"SECRET_ID": "abc"}): - with pytest.raises(ValueError, match="only whole|named as a secret"): - launch.render("codex", settings("codex", extra={"env": env}), deploy=None, tools_url=None, - job=tmp_path / "bad") - plan = plan_of(launch.render("codex", settings("codex", extra={"env": { - "MAX_OUTPUT_TOKENS": "4096", "LIBS": "$EXTRA_DIR/lib", "GH_TOKEN": "$GH_TOKEN", "GIT_AUTHOR_NAME": "Hermit", - "TOKENIZERS_PARALLELISM": "false"}}), deploy=None, tools_url=None, job=tmp_path / "ok")) - assert (plan["env"]["MAX_OUTPUT_TOKENS"], plan["env"]["LIBS"], plan["env_from_host"]["GH_TOKEN"]) == \ - ("4096", "/opt/extra/lib", "GH_TOKEN") - assert (plan["env"]["GIT_AUTHOR_NAME"], plan["env"]["TOKENIZERS_PARALLELISM"]) == ("Hermit", "false") - - -def test_a_plans_folder_is_this_users_only(tmp_path): - """The launch folder holds the plan and the CLI's home, which name the trial's tool URL or the answer agent's relay - URL with its token: it is made this user's only, whatever the umask or the jobs directory's mode.""" - old = os.umask(0o022) - try: - cli = launch.render("codex", settings("codex"), deploy=None, tools_url=None, job=tmp_path / "codex") - answer = launch.render_answer(launch.settings_for("answer", binding=None, model="m"), - endpoint="http://127.0.0.1:1/t0ken/v1", job=tmp_path / "answer") - finally: - os.umask(old) - assert [stat.S_IMODE(Path(p).parent.stat().st_mode) for p in (cli, answer)] == [0o700, 0o700] - - -def test_claude_on_a_backend_that_needs_no_key_gets_the_placeholder_token(tmp_path, monkeypatch): - """A backend with no key (vLLM's, key_required false): Claude Code is given the placeholder token main gives its own - sessions (Binding.claude_env), which is no secret, so the plan is rendered, not refused as an inline secret.""" - from hermit.agent.model.backends import PLACEHOLDER_KEY, as_binding, registry - monkeypatch.delenv("VLLM_API_KEY", raising=False) - spec = as_binding("vllm/Qwen/Qwen3-30B", registry()).spec() - claude = launch.settings_for("claude", binding=launch.binding_of({"binding": spec}), web_search=False) - plan = plan_of(launch.render("claude", claude, deploy=None, tools_url=None, job=tmp_path / "claude")) - assert plan["env"]["ANTHROPIC_AUTH_TOKEN"] == PLACEHOLDER_KEY - - -def test_a_trials_env_on_a_login_is_what_mains_session_on_it_gets(tmp_path, monkeypatch): - """A trial's container gets none of this process's variables but those its settings name, so main's dropped_env - (this process's variables a session must not get) has nothing to drop there; and as main's Claude Code session on - login: claude (ClaudeCode.session_env), a trial's is refused an `env` that names what would pay instead - (ANTHROPIC_API_KEY: it would bill before the subscription), never handed to the CLI.""" - from hermit.agent.base.interface import Unsupported - monkeypatch.setenv("CLAUDE_CODE_OAUTH_TOKEN", TOKEN) - monkeypatch.setenv("ANTHROPIC_API_KEY", "sk-ant-0123456789abcdef") - claude = launch.settings_for("claude", model="claude-haiku-5-5", web_search=False, login="claude") - plan = plan_of(launch.render("claude", claude, deploy=None, tools_url=None, job=tmp_path / "plain")) - assert "ANTHROPIC_API_KEY" not in {**plan["env"], **plan["env_from_host"]} # (main's lock blanks it in settings) - assert "sk-ant-0123456789abcdef" not in json.dumps(plan) - paying = launch.settings_for("claude", model="claude-haiku-5-5", web_search=False, login="claude", - extra={"env": {"ANTHROPIC_API_KEY": "$ANTHROPIC_API_KEY"}}) - with pytest.raises(Unsupported, match="what would pay instead"): - launch.render("claude", paying, deploy=None, tools_url=None, job=tmp_path / "claude") - - -def test_text_only_with_the_adapters_tools_and_a_dsh_server_name_too_long_are_refused_as_mains_run_refuses_them( - tmp_path): - """What main's sessions refuse is refused for a trial's plan: text_only with the adapter's tools - (CliAgent._session), and for dsh a tools server name that is not one of its MCP server names (32 characters at most: an adapter's id - too long, prepare_home).""" - from hermit.agent.base.interface import Unsupported - with pytest.raises(Unsupported, match="text_only has no tools"): - launch.render("codex", settings("codex", extra={"text_only": True}), deploy=None, tools_url=URL, - tools_name="calc", job=tmp_path / "text") - with pytest.raises(ValueError, match="32 characters at most"): - launch.render("dsh", settings("dsh"), deploy=None, tools_url=URL, tools_name="a" * 33, job=tmp_path / "dsh") - launch.render("dsh", settings("dsh"), deploy=None, tools_url=URL, tools_name="a" * 32, job=tmp_path / "dsh-ok") diff --git a/tests/core/test_harbor_runner.py b/tests/core/test_harbor_runner.py deleted file mode 100644 index 409b066d..00000000 --- a/tests/core/test_harbor_runner.py +++ /dev/null @@ -1,5356 +0,0 @@ -"""The Harbor runner's host side without Docker or Harbor: the verdicts of a trial's result and of the CLI's stream, -the curated log, the dataset rows, lock and splits, the judge, the adapter's staging, the startup checks, one trial -through a fake `harbor` command that carries out the launch plan as harborside would, the answer agent's model on -this machine, the adapter's tools served for a container, and stream steps through a fake driver.""" -import contextlib -from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer -import json -import os -from pathlib import Path -import select -import shutil -import socket -import stat -import sys -import textwrap -import threading -import time -import urllib.request -import uuid - -import pytest -import yaml - -from hermit.agent.base.interface import LoginFailed -from hermit.harbor import curate, dataset, docker, errors, judge, stream -from hermit.harbor import runner as runner_module -from hermit.harbor.runner import HarborAgent -from hermit.task import RunOutput - -ROOT = Path(__file__).resolve().parents[2] -TEMPLATES = ROOT / "hermit" / "adapter" / "templates" -KEY_ENV = "FAKE_MODEL_KEY" -KEY = "fake-model-key-0123456789abcdef" # never a real key: what a plan must never hold -# A model bound as the config's loading binds it (hermit.agent.model.backends.bind_agent_params): its id, and the -# whole spec under `binding` -BINDING = {"ref": "fake/glm-5", "backend": "fake", "model": "glm-5", "base_url": "https://models.example/v1", - "api_key_env": KEY_ENV, "key_required": True, "apis": ["chat", "responses"], - "price": {"input": 1.0, "cached_input": 0.5, "output": 2.0}} - - -# ---- helpers ----------------------------------------------------------------------------------------------------- - -def write(path: Path, text: str, exe: bool = False) -> Path: - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(text) - if exe: - path.chmod(path.stat().st_mode | stat.S_IXUSR) - return path - - -def make_task(root: Path, name: str, *, steps: tuple[str, ...] = (), image: str | None = None, - network: str | None = None) -> Path: - toml = 'schema_version = "1.4"\n[metadata]\ndifficulty = "easy"\n[verifier]\ntimeout_sec = 30.0\n[agent]\ntimeout_sec = 45.0\n' \ - '[environment]\nbuild_timeout_sec = 120.0\n' - if image: - toml += f'docker_image = "{image}"\n' - if network: - toml += f'network_mode = "{network}"\n' - toml += "".join(f'\n[[steps]]\nname = "{s}"\n' for s in steps) - task = root / name - write(task / "task.toml", toml) - write(task / "instruction.md", f"Do {name}.\n") - write(task / "environment" / "Dockerfile", "FROM scratch\n") - write(task / "tests" / "test.sh", "#!/bin/bash\necho 1 > /logs/verifier/reward.txt\n", exe=True) - for s in steps: - write(task / "steps" / s / "instruction.md", f"Step {s} of {name}.\n") - write(task / "steps" / s / "tests" / "test.sh", "#!/bin/bash\necho 1 > /logs/verifier/reward.txt\n", exe=True) - return task - - -def trial_result(reward=1.0, *, exc=None, tokens=(100, 20), metadata=None) -> dict: - result = {"task_name": "t", "trial_name": "t__abc1234", "started_at": "2026-01-01T00:00:00Z", - "finished_at": "2026-01-01T00:01:00Z", - "agent_execution": {"started_at": "2026-01-01T00:00:10Z", "finished_at": "2026-01-01T00:00:50Z"}, - "agent_result": {"n_input_tokens": tokens[0], "n_output_tokens": tokens[1], "n_cache_tokens": 5, - "metadata": metadata or {}}, - "verifier_result": {"rewards": {"reward": reward}} if reward is not None else None, - "exception_info": None} - if exc: - result["exception_info"] = {"exception_type": exc, "exception_message": f"{exc} happened", - "exception_traceback": "Traceback..."} - return result - - -# A Codex session's stream (hermit.agent.base.cli.Codex reads it): one turn served, its usage -CODEX_STREAM = ('{"type":"thread.started","thread_id":"th-1"}\n' - '{"type":"item.completed","item":{"id":"i1","type":"agent_message","text":"all done"}}\n' - '{"type":"turn.completed","usage":{"input_tokens":100,"cached_input_tokens":40,"output_tokens":20}}\n') - -FAKE_HARBOR = textwrap.dedent('''\ - #!/usr/bin/env python3 - """A stand-in for `harbor run -c job.yaml`: the trial directory a job leaves, with the agent's outputs as harborside - leaves them from the launch plan (the CLI's stream, the files it collects, a sign-in handed back to this machine; - the answer agent's one call to the endpoint its plan names).""" - import json, os, sys, time, urllib.error, urllib.request, yaml - from datetime import datetime, timedelta, timezone - from pathlib import Path - args = sys.argv[1:] - assert args[:3] == ["run", "-c", args[2]], args - job = yaml.safe_load(Path(args[2]).read_text()) - jobs_dir, name = Path(job["jobs_dir"]), job["job_name"] - task = Path(job["tasks"][0]["path"]) - trial = jobs_dir / name / f"{task.name[:32].rstrip('_-')}__abc1234" # as Harbor names a trial - (trial / "agent").mkdir(parents=True) - (trial / "verifier").mkdir() - agent = job["agents"][0] - plan_path = (agent.get("kwargs") or {}).get("plan") - plan = json.loads(Path(plan_path).read_text()) if plan_path else None - (trial / "agent" / "codex.txt").write_text("the agent's output\\n") - (trial / "agent" / "sessions").mkdir() - (trial / "agent" / "sessions" / "rollout.jsonl").write_text("secret transcript\\n") - (trial / "agent" / "trajectory.json").write_text(json.dumps({"steps": [{"metrics": {"a": 1}}, {"metrics": {}}, {}]})) - (trial / "verifier" / "test-stdout.txt").write_text("PASS\\n") - (trial / "verifier" / "reward.txt").write_text("1\\n") - names = list((plan or {}).get("env_from_host", {}).values()) + list((plan or {}).get("env_by_fd", {})) - (trial / "job-seen.json").write_text(json.dumps({ - "env": {k: v for k, v in os.environ.items() if k in ("PYTHONPATH", "TMPDIR")}, - "token": "CLAUDE_CODE_OAUTH_TOKEN" in os.environ, - "by_name": sorted(n for n in names if os.environ.get(n))})) - mode = os.environ.get("FAKE_HARBOR_MODE", "ok") - if plan is not None and plan.get("command") and mode != "no_environment": - streams = [x for x in os.environ.get("FAKE_STREAMS", "").split(",") if x] - if streams: # the session's stream, one per call (the last repeats) - count = jobs_dir / ".fake-calls" - n = int(count.read_text()) if count.exists() else 0 - count.write_text(str(n + 1)) - text = Path(streams[min(n, len(streams) - 1)]).read_text() - else: - text = os.environ["FAKE_DEFAULT_STREAM"] if mode != "silent" else "bash: codex: command not found\\n" - (trial / "agent" / plan["stream"]).write_text(text) - if os.environ.get("FAKE_LINK_STREAM_TO"): # the session left a link in its stream's place - (trial / "agent" / plan["stream"]).unlink() - (trial / "agent" / plan["stream"]).symlink_to(os.environ["FAKE_LINK_STREAM_TO"]) - collect = trial / "agent" / "hermit-collect" - collect.mkdir() - for item in plan["collect"]: - if item["to"] == "last_message.txt" and mode != "silent": - (collect / item["to"]).write_text("all done\\n") - elif item["to"] == "sessions" and os.environ.get("FAKE_RATE_LIMITS"): # Codex's session log - day = collect / "sessions" / "2026" / "11" / "11" - day.mkdir(parents=True) - limits = json.loads(os.environ["FAKE_RATE_LIMITS"]) - (day / "rollout-1.jsonl").write_text(json.dumps({ - "timestamp": datetime.now(timezone.utc).isoformat(), "type": "event_msg", - "payload": {"type": "token_count", "rate_limits": limits}}) + "\\n") - for item in plan["return_files"]: # what the container hands back: the sign-in Codex refreshed - refreshed = (datetime.now(timezone.utc) - timedelta(minutes=5)).isoformat() # (stand-ins), or another file - Path(item["to"]).write_text(os.environ.get("FAKE_RETURNED") or json.dumps({ - "last_refresh": refreshed, "tokens": {"account_id": "acct-1", "access_token": "at-stand-in", - "refresh_token": "rt-rotated-stand-in"}})) - elif plan is not None and plan.get("answer"): # the answer agent: one call to the endpoint its plan names - block = plan["answer"] - body = json.dumps({"model": block["model"], "messages": [{"role": "system", "content": block["system"]}, - {"role": "user", "content": "What is 6 x 7?"}]}) - request = urllib.request.Request(block["endpoint"] + "/chat/completions", data=body.encode(), - headers={"Content-Type": "application/json"}) - try: - reply = json.loads(urllib.request.urlopen( - request, timeout=float(os.environ.get("FAKE_ANSWER_WAIT_S", "30"))).read()) - except TimeoutError: # no answer within the agent's time (FAKE_HARBOR_MODE timeout) - reply = None - except urllib.error.HTTPError: # the model failed: the agent exits non-zero (harborside.answer) - reply, mode = None, "agent_exit" - if reply is not None: - said = reply["choices"][0]["message"]["content"] - (trial / "agent" / "answer.json").write_text(json.dumps({"reply": said, "answer": said.split()[-1], - "usage": reply.get("usage"), - "model": block["model"]})) - deploy = (plan or {}).get("deploy") or {} - result = {"task_name": task.name, "trial_name": trial.name, "started_at": "2026-01-01T00:00:00Z", - "finished_at": "2026-01-01T00:01:00Z", - "agent_execution": {"started_at": "2026-01-01T00:00:10Z", "finished_at": "2026-01-01T00:00:50Z"}, - "agent_result": {"n_input_tokens": 100, "n_output_tokens": 20, "n_cache_tokens": 5, - "metadata": {"hooks_active": bool(deploy.get("hooks")), - "adapter_deployed": bool(deploy)}}, - "verifier_result": {"rewards": {"reward": 1.0}}, "exception_info": None} - if mode == "usage_limit": - result["exception_info"] = {"exception_type": "ApiUsageLimitError", "exception_message": "limit", "exception_traceback": "tb"} - result["verifier_result"] = None - elif mode == "hang": - time.sleep(30) - elif mode == "no_tools": # the verifier's uv download failed (GitHub unreachable): no test ran - (trial / "verifier" / "test-stdout.txt").write_text( - "curl: (22) The requested URL returned error: 504\\n" - "failed to download https://github.com/astral-sh/uv/releases/download/0.9.5/" - "uv-x86_64-unknown-linux-gnu.tar.gz\\n" - "/tests/test.sh: line 9: /root/.local/bin/env: No such file or directory\\n" - "/tests/test.sh: line 18: uvx: command not found\\n") - result["verifier_result"] = {"rewards": {"reward": 0.0}} - elif mode in ("silent", "agent_exit"): # the CLI exited non-zero: Harbor's error, the verifier still ran - result["exception_info"] = {"exception_type": "NonZeroAgentExitCodeError", "exception_message": "exit 1", - "exception_traceback": "tb"} - result["verifier_result"] = {"rewards": {"reward": 0.5}} - elif mode == "no_environment": # the environment never started: no agent ran, no verifier - result.pop("agent_execution") - result["agent_result"] = result["verifier_result"] = None - result["exception_info"] = {"exception_type": "EnvironmentStartTimeoutError", - "exception_message": "start timed out", "exception_traceback": "tb"} - elif mode == "timeout_no_tools": # the agent ran out of its time; the verifier's uv download failed - (trial / "verifier" / "test-stdout.txt").write_text( - "curl: (22) The requested URL returned error: 504\\n" - "failed to download https://github.com/astral-sh/uv/releases/download/0.9.5/" - "uv-x86_64-unknown-linux-gnu.tar.gz\\n" - "/tests/test.sh: line 9: /root/.local/bin/env: No such file or directory\\n" - "/tests/test.sh: line 18: uvx: command not found\\n") - result["verifier_result"] = {"rewards": {"reward": 0.0}} - result["exception_info"] = {"exception_type": "AgentTimeoutError", "exception_message": "Agent timed out", - "exception_traceback": "tb"} - elif mode == "timeout": # the agent ran out of its time; the verifier still ran, and passed - result["exception_info"] = {"exception_type": "AgentTimeoutError", "exception_message": "Agent timed out", - "exception_traceback": "tb"} - (trial / "result.json").write_text(json.dumps(result)) - ''') - - -REAL_SANDBOX_FAULT = runner_module.sandbox_fault - - -@pytest.fixture(autouse=True) -def sandbox_starts(monkeypatch): - """This machine's strict adapter sandbox taken as starting (sandbox_fault), so that a runner's startup and the - verdict of a staging that failed need no working bubblewrap, as main's tests skip what needs one; the tests of that - check run the real one (REAL_SANDBOX_FAULT).""" - monkeypatch.setattr(runner_module, "sandbox_fault", lambda enforce, *, fresh=True: None) - - -@pytest.fixture -def fake_harbor(tmp_path, monkeypatch): - """A `harbor` command and a `python` beside it (the runner derives the driver's interpreter from it); the model's - key set (a plan holds its name, never its value).""" - bin_dir = tmp_path / "harbor-venv" / "bin" - harbor = write(bin_dir / "harbor", FAKE_HARBOR, exe=True) - python = bin_dir / "python" - python.symlink_to(sys.executable) - monkeypatch.setenv(KEY_ENV, KEY) - monkeypatch.setenv("FAKE_DEFAULT_STREAM", CODEX_STREAM) - return harbor - - -def make_agent(tmp_path, fake_harbor, tasks, **params) -> HarborAgent: - """A runner for HermitCodex on the bound model, web search off, an infra trial not run again (a config's params - override these).""" - return HarborAgent({"tasks_dir": str(tasks), "jobs_dir": str(tmp_path / "jobs"), "harbor": str(fake_harbor), - "disk_gate_gb": 0, "tmp_gate_gb": 0, "model": BINDING["model"], "binding": BINDING, - "web_search": False, "infra_retries": 0, **params}, base=tmp_path) - - -def cli_answer_agent(tmp_path, fake_harbor, tasks, model: dict, **params) -> HarborAgent: - """A runner for the answer agent on a model a CLI reaches, given where it must be, in the benchmark file (the - runner serves it itself; a run config's own is the pipeline's to serve, refused).""" - bench = write(tmp_path / "bench" / "benchmark.yaml", yaml.safe_dump({"model": model})) - return HarborAgent({"tasks_dir": str(tasks), "jobs_dir": str(tmp_path / "jobs"), "harbor": str(fake_harbor), - "disk_gate_gb": 0, "tmp_gate_gb": 0, "benchmark": str(bench), "acting_agent": {"cli": "answer"}, - "web_search": None, "infra_retries": 0, **params}, base=tmp_path) - - -def chatgpt_login(root: Path, account: str = "acct-1") -> Path: - """The user's Codex sign-in where the test's CODEX_HOME points (tests/conftest.py): a stand-in, no real token, - refreshed two days ago.""" - from datetime import datetime, timedelta, timezone - when = (datetime.now(timezone.utc) - timedelta(days=2)).isoformat() - return write(Path(os.environ["CODEX_HOME"]) / "auth.json", - json.dumps({"last_refresh": when, "tokens": {"account_id": account, "refresh_token": "rt-stand-in"}})) - - -def stream_name(agent: HarborAgent, task: str = "flow", rollout: int = 1) -> str: - """The name of a multi-step task's stream in `agent`'s runs: its task, its rollout and the runner's own id (one - runner's streams are its own, never another's on the same jobs directory).""" - return f"{task}__s{rollout}__{agent.run_id}" - - -def step_dirs(stream_dir: Path, index: int, name: str) -> list[Path]: - """The directories of a stream's step `index` (`name`), one per rollout of it, oldest first.""" - return sorted((Path(stream_dir) / "steps").glob(f"{index:03d}-{name}-*"), key=lambda d: d.stat().st_mtime_ns) - - -def unscored_record(record: dict, cause: str) -> None: - """A rollout's record (hermit.trainer.rollout.run_one) of a trial that is not the agent's result, for `cause`: the - judge refused to score it (hermit.harbor.judge.Unscored), so the run records a failed run (judge_error, scored 0, - as `hermit eval` and the Trainer count one), never a 0 the agent earned.""" - assert (record["status"], record["judge"]["score"]) == ("judge_error", 0.0), record["status"] - assert record["error"].startswith(f"Unscored: the trial is not the agent's result ({cause}"), record["error"] - assert record["pipeline_extra"]["unscored"] == cause - - -def job_files(job: Path) -> str: - """Every text file of a job directory, joined: what must never hold a secret.""" - return "\n".join(p.read_text(errors="replace") for p in Path(job).rglob("*") if p.is_file() and not p.is_symlink()) - - -# ---- errors ------------------------------------------------------------------------------------------------------ - -def test_verifier_unreached_needs_the_failed_download_and_the_missing_runner(): - failed = ("curl: (22) The requested URL returned error: 504\n" - "failed to download https://github.com/astral-sh/uv/releases/download/0.9.5/uv.tar.gz\n" - "/tests/test.sh: line 9: /root/.local/bin/env: No such file or directory\n" - "/tests/test.sh: line 18: uvx: command not found\n") - assert errors.verifier_unreached(failed) - # A test's own failed request is the agent's; a failed download with the runner already there ran the tests. - assert not errors.verifier_unreached("curl: (7) Failed to connect to localhost port 8080\nFAILED test_server\n") - assert not errors.verifier_unreached("failed to download https://github.com/x/y.tar.gz\n=== 3 passed in 1.2s ===\n") - # The host's disk full while the verifier ran. - assert errors.verifier_disk_full(" Caused by: No space left on device (os error 28)\n") - assert not errors.verifier_disk_full("=== 3 passed in 1.2s ===\n") - - -def test_classify_reads_the_result(): - ok = errors.classify(trial_result(0.5)) - assert (ok["status"], ok["reward"], ok["failure_cause"], ok["halt"]) == ("ok", 0.5, None, False) - timeout = errors.classify(trial_result(1.0, exc="AgentTimeoutError")) - assert (timeout["status"], timeout["timed_out"], timeout["reward"]) == ("timeout", True, 1.0) - infra = errors.classify(trial_result(None, exc="ApiUsageLimitError")) - assert (infra["status"], infra["failure_cause"], infra["halt"]) == ("infra", "model_provider_failed", True) - agent = errors.classify(trial_result(None, exc="NonZeroAgentExitCodeError")) - assert (agent["status"], agent["failure_cause"]) == ("agent_error", "unclassified") - unknown = errors.classify(trial_result(None, exc="SomethingNewError")) - assert (unknown["status"], unknown["failure_cause"]) == ("agent_error", "unclassified") - assert errors.classify(trial_result(None))["failure_cause"] == "verifier_failed" - # the host failing under an unlisted exception: infra (rerun, never kept), not the agent's zero - full = trial_result(None, exc="RuntimeError") - full["exception_info"]["exception_message"] = ("Agent install failed: 'OCI runtime exec failed: open " - "/tmp/runc-process870598827: no space left on device'") - assert (errors.classify(full)["status"], errors.classify(full)["failure_cause"]) == ("infra", "sandbox_failed") - # the task's environment never came up (an image build whose dependencies no longer resolve): not the agent's zero - build = trial_result(None, exc="RuntimeError") - build["exception_info"]["exception_message"] = ( - "Docker compose command failed for environment animal-reid. Command: docker compose --project-name " - "animal-reid__weecvwf__env build. Return code: 1. Stdout: Failed to resolve dependencies") - assert (errors.classify(build)["status"], errors.classify(build)["failure_cause"]) == ("infra", "sandbox_failed") - own = trial_result(None, exc="NonZeroAgentExitCodeError") - own["exception_info"]["exception_message"] = "the agent wrote: no space left on device" - assert errors.classify(own)["status"] == "agent_error" # the agent's own exit stays the agent's - assert errors.classify(None, expired=True)["failure_cause"] == "harbor_timeout" - assert errors.classify(None, harbor_exit=3)["exception_message"].startswith("harbor run exited 3") - assert errors.classify(trial_result(0.0), reward_key="f1")["failure_cause"] == "verifier_failed" - # what the runner reads from the session is infra too - assert {"session_failed", "tools_failed", "login_failed", "usage_limit"} <= errors.INFRA_CAUSES - # harborside's launch failing for a reason not the agent's (a key missing, no agent user): infra; the adapter's - # context too long beside the image's own: the agent's result - launch_failed = errors.classify(trial_result(None, exc="HermitLaunchError")) - assert (launch_failed["status"], launch_failed["failure_cause"]) == ("infra", "launch_failed") - too_long = errors.classify(trial_result(None, exc="HermitContextLimitError")) - assert (too_long["status"], too_long["failure_cause"]) == ("agent_error", "context_over_limit") - - -def test_a_verifier_that_could_not_fetch_its_tests_is_infra(tmp_path): - """Terminal-Bench's tests.sh downloads uv before pytest: when GitHub fails the download (a 5xx), the tests never run - and the reward is 0. That trial is infra (run again, never kept), not the agent's 0, and so is one whose agent ran - out of its time or exited with an error (Harbor verifies those too and grades their reward); a test run that failed - stays the agent's (its timeout too), and so does a pass, and so does a setup that fails the same way every time (a - mirror's 404).""" - trial = tmp_path / "t__abc1234" - said = ("downloading uv 0.9.5 x86_64-unknown-linux-gnu\ncurl: (22) The requested URL returned error: 504\n" - "failed to download https://github.com/astral-sh/uv/releases/download/0.9.5/uv-x86_64-unknown-linux-gnu" - ".tar.gz\n/tests/test.sh: line 10: /root/.local/bin/env: No such file or directory\n" - "/tests/test.sh: line 19: uvx: command not found\n") - write(trial / "verifier" / "test-stdout.txt", said) - verdict = HarborAgent.verifier_verdict(errors.classify(trial_result(0.0)), trial) - assert (verdict["status"], verdict["failure_cause"]) == ("infra", "verifier_failed") - assert "uvx: command not found" in verdict["exception_message"] - assert HarborAgent.verifier_verdict(errors.classify(trial_result(1.0)), trial)["status"] == "ok" - for exc in ("AgentTimeoutError", "NonZeroAgentExitCodeError"): # the agent's time ran out, - failed = errors.classify(trial_result(0.0, exc=exc)) # or it failed: no test ran - assert HarborAgent.verifier_verdict(failed, trial)["status"] == "infra", exc - write(trial / "verifier" / "ctrf.json", "{}") # the tests ran and failed - assert HarborAgent.verifier_verdict(errors.classify(trial_result(0.0)), trial)["status"] == "ok" - timeout = errors.classify(trial_result(0.0, exc="AgentTimeoutError")) - assert HarborAgent.verifier_verdict(timeout, trial)["status"] == "timeout" - (trial / "verifier" / "ctrf.json").unlink() - write(trial / "verifier" / "test-stdout.txt", "FAILED test_outputs.py::test_answer - assert 3 == 4\n") - assert HarborAgent.verifier_verdict(errors.classify(trial_result(0.0)), trial)["status"] == "ok" - assert HarborAgent.verifier_verdict(errors.classify(trial_result(0.0)), None)["status"] == "ok" - write(trial / "verifier" / "test-stdout.txt", # GitHub's connection died - "Setting up curl (8.5.0-2ubuntu10.16) ...\ndownloading uv 0.9.5 x86_64-unknown-linux-gnu\n" - "curl: (56) Connection died, tried 5 times before giving up\nfailed to download " - "https://github.com/astral-sh/uv/releases/download/0.9.5/uv-x86_64-unknown-linux-gnu.tar.gz\n" - "/tests/test.sh: line 19: uvx: command not found\n") - assert HarborAgent.verifier_verdict(errors.classify(trial_result(0.0)), trial)["status"] == "infra" - write(trial / "verifier" / "test-stdout.txt", # qemu-startup's, every run - "Err:6 http://deb.debian.org/debian-security bullseye-security/main amd64 curl amd64 7.74.0-1.3+deb11u16\n" - " 404 Not Found [IP: 146.75.38.132 80]\n/tests/test.sh: line 8: curl: command not found\n" - "/tests/test.sh: line 10: /root/.local/bin/env: No such file or directory\n" - "/tests/test.sh: line 19: uvx: command not found\n") - assert HarborAgent.verifier_verdict(errors.classify(trial_result(0.0)), trial)["status"] == "ok" - - -def test_a_verifier_that_left_no_reward_harbor_could_read_is_the_agents_result_as_harbor_grades_it(): - """Harbor grades its verifier group as the agent's result (its error policy: retry=False, grade=True), and its job - mean counts the trial's missing reward as 0: a verifier that wrote no reward file, an empty one or one that does - not parse is an agent_error the judge scores 0 and the mean keeps, not infra run again.""" - from hermit.trainer.rollout import run_one - passed = run_one({"id": "t#1"}, None, lambda task, adapter: RunOutput(output={"status": "ok", "reward": 1.0}), - judge.score) - for exc in ("RewardFileNotFoundError", "RewardFileEmptyError", "VerifierOutputParseError"): - verdict = errors.classify(trial_result(None, exc=exc)) - assert (verdict["status"], verdict["failure_cause"], verdict["reward"]) == \ - ("agent_error", "verifier_failed", None), exc - out = RunOutput(output={"status": verdict["status"], "failure_cause": verdict["failure_cause"], - "exception_type": exc, "reward": verdict["reward"]}) - record = run_one({"id": "t#2"}, None, lambda task, adapter: out, judge.score) - assert record["status"] == "ok" and record["judge"]["score"] == 0.0, exc - assert (passed["judge"]["score"] + record["judge"]["score"]) / 2 == 0.5, exc # hermit eval's mean - - -def test_a_request_harbors_own_agent_made_that_the_provider_refused_for_what_it_held_is_the_agents_result(): - """litellm's BadRequestError and UnprocessableEntityError (a request one of Harbor's own agents made that the - provider refused for what it held: a prompt too long, an invalid request) are the agent's result, its work scored, - as Harbor grades them and as a Hermit CLI's first request refused for its input is. A provider that could not - serve (a rate limit, a lost connection, a server error) or that refused the run's configuration (its key, its - model, a parameter, a budget) stays infra and runs again, which Harbor grades but for the key and the model.""" - for exc in ("BadRequestError", "UnprocessableEntityError"): - verdict = errors.classify(trial_result(0.0, exc=exc)) - assert (verdict["status"], verdict["failure_cause"], verdict["reward"]) == \ - ("agent_error", "model_provider_failed", 0.0), exc - for exc in ("ApiRateLimitError", "APIConnectionError", "InternalServerError", "ServiceUnavailableError", - "APIResponseValidationError", "UnsupportedParamsError", "BudgetExceededError", "AuthenticationError", - "NotFoundError"): - verdict = errors.classify(trial_result(None, exc=exc)) - assert (verdict["status"], verdict["failure_cause"]) == ("infra", "model_provider_failed"), exc - - -def test_a_verifier_the_host_kept_from_its_tests_is_infra_whatever_harbor_raised_for_it(tmp_path, fake_harbor): - """Hermit's checks of the verifier's output (its tools never arrived; the host's disk was full) find the host behind - a verifier that left no reward or ran out of its time, as behind one that wrote 0, and behind the verifier of an - agent that ran out of its own time or exited with an error (Harbor verifies those too): infra, run again. Without - that evidence such a verifier is the agent's result, and the agent's own timeout or error stays its own.""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - agent = make_agent(tmp_path, fake_harbor, tasks) - trial = tmp_path / "t__abc1234" - - def verdict(exc): - classified = agent.verifier_timeout_verdict(errors.classify(trial_result(None, exc=exc))) - return HarborAgent.verifier_verdict(classified, trial) - write(trial / "verifier" / "test-stdout.txt", "FAILED test_outputs.py::test_answer - assert 3 == 4\n") - assert verdict("RewardFileNotFoundError")["status"] == "agent_error" - assert verdict("VerifierTimeoutError")["status"] == "timeout" - assert verdict("AgentTimeoutError")["status"] == "timeout" # the agent's own time ran out - assert verdict("NonZeroAgentExitCodeError")["status"] == "agent_error" - write(trial / "verifier" / "test-stdout.txt", - "failed to download https://github.com/astral-sh/uv/releases/download/0.9.5/uv.tar.gz\n" - "/tests/test.sh: line 18: uvx: command not found\n") - for exc in ("RewardFileNotFoundError", "RewardFileEmptyError", "VerifierTimeoutError"): - assert (verdict(exc)["status"], verdict(exc)["failure_cause"]) == ("infra", "verifier_failed"), exc - write(trial / "verifier" / "test-stdout.txt", "OSError: [Errno 28] No space left on device: '/logs/verifier'\n") - for exc in ("RewardFileNotFoundError", "VerifierOutputParseError", "VerifierTimeoutError", "AgentTimeoutError", - "NonZeroAgentExitCodeError"): - assert verdict(exc)["status"] == "infra", exc - - -def test_an_infra_verifier_verdict_quotes_the_verifiers_output_with_verifier_redact_applied(tmp_path): - """The verifier's output an infra verdict quotes (the tail behind its cause) has `verifier_redact`'s patterns - removed, as the curated test-stdout.txt has: the test's expected answer reaches neither the verdict's message - (the rollout's extra, the learning material) nor the log's exception.txt.""" - job = tmp_path / "job" - trial = job / "t__abc1234" - write(trial / "verifier" / "test-stdout.txt", - "FAILED test_outputs.py::test_answer - AssertionError: expected GOLD-ANSWER-42, got 41\n" - "OSError: [Errno 28] No space left on device: '/tmp/pytest-cache'\n") - write(trial / "verifier" / "reward.txt", "0\n") - redact = [r"GOLD-ANSWER-\d+"] - verdict = HarborAgent.verifier_verdict(errors.classify(trial_result(0.0)), trial, redact) - assert (verdict["status"], verdict["failure_cause"]) == ("infra", "verifier_failed") - assert "GOLD-ANSWER-42" not in verdict["exception_message"] - assert f"expected {curate.REDACTED}, got 41" in verdict["exception_message"] - log = curate.curate(trial, job, verdict, redact=redact) - assert "GOLD-ANSWER-42" not in (log / "exception.txt").read_text() - assert "GOLD-ANSWER-42" not in (log / "verifier" / "test-stdout.txt").read_text() - - -def test_teardown_takes_the_trials_verifier_projects_down_too(monkeypatch): - """A killed trial leaves its environment's compose project and each separate verifier's - (`__verifier__`, harbor/trial/trial.py): the teardown downs them all, a single-step trial's by its - exact name (its key "trial", whatever its length: docker.verifier_session) and any other by the prefix, and no - other project.""" - calls = [] - - class Done: - stdout = "alpha__abc1234__env\nalpha__abc1234__verifier__main\nbeta__xyz9876__verifier__main\n" - - def run(cmd, **kwargs): - calls.append(cmd) - return Done() - monkeypatch.setattr(docker.subprocess, "run", run) - docker.compose_down("alpha__abc1234") - downs = [c[c.index("-p") + 1] for c in calls if "down" in c] - assert downs == ["alpha__abc1234__env", "alpha__abc1234__verifier__trial", "alpha__abc1234__verifier__main"] - assert docker.verifier_prefix("Mixed.Case__abc1234") == "mixed-case__abc1234__verifier" - - -# ---- curate ------------------------------------------------------------------------------------------------------ - -def test_curate_keeps_outputs_and_drops_sessions(tmp_path, monkeypatch): - monkeypatch.setenv(KEY_ENV, KEY) - job = tmp_path / "job" - trial = job / "t__abc1234" - write(trial / "agent" / "codex.txt", "out\n") - write(trial / "agent" / "agent-codex.jsonl", '{"type":"thread.started"}\n') - write(trial / "agent" / "sessions" / "rollout.jsonl", "secret\n") - write(trial / "agent" / "hook-state" / "state.json", "{}") - write(trial / "agent" / "hermit-collect" / "last_message.txt", f"done; the key was {KEY}\n") - write(trial / "agent" / "hermit-collect" / "opencode.db", "SQLite format 3\0") - write(trial / "agent" / "hermit-copy" / "memory" / "NOTES.md", "- keep the cache warm\n") - (trial / "agent" / "hook-state" / "linked.txt").symlink_to(tmp_path / "outside.txt") # the container's link - write(tmp_path / "outside.txt", "a file of this machine\n") - write(trial / "verifier" / "test-stdout.txt", "x" * 25_000 + "\nexpected answer: 42\n") - write(trial / "verifier" / "reward.txt", "1\n") - write(trial / "result.json", json.dumps(trial_result())) - write(job / "harbor.log", "harbor\n") - write(job / "job.yaml", "job_name: j\n") - write(job / "tool-calls.jsonl", '{"tool": "calc"}\n') - write(job / "launch" / "home" / "auth.json", "{}") # never curated - log = curate.curate(trial, job, errors.classify(trial_result(None, exc="VerifierTimeoutError")), - redact=[r"expected answer: \d+"], audit=job / "tool-calls.jsonl") - names = sorted(str(p.relative_to(log)) for p in log.rglob("*") if p.is_file() or p.is_symlink()) - assert names == ["agent-codex.jsonl", "agent/codex.txt", "agent/hermit-collect/last_message.txt", - "exception.txt", "harbor.log", "hook-state/state.json", "job.yaml", "memory/NOTES.md", - "result.json", "tool-calls.jsonl", "verifier/reward.txt", "verifier/test-stdout.txt"] - text = (log / "verifier" / "test-stdout.txt").read_text() - assert text.startswith("[... ") and "characters dropped ...]" in text and len(text) < 20_100 - assert "expected answer" not in text and text.rstrip().endswith(curate.REDACTED) - assert KEY not in (log / "agent" / "hermit-collect" / "last_message.txt").read_text() # scrubbed: - assert "VerifierTimeoutError" in (log / "exception.txt").read_text() - assert curate.seconds(trial_result()["agent_execution"]) == 40.0 - write(trial / "agent" / "trajectory.json", json.dumps({"steps": [{"metrics": {"a": 1}}, {}]})) - assert curate.model_calls(trial / "agent" / "trajectory.json") == 1 - - -def test_curate_lays_the_log_out_where_hermits_readers_look(tmp_path): - """The curated log is laid out as a Hermit session's log, where the methods' readers look (the trajectory's - agent-.jsonl at the top, AHE's hook-calls.jsonl, prompt.txt and task.md, RRSI's task text): the CLI's stream - at the top under its Hermit name (Harbor's own codex's codex.txt and claude-code's claude-code.txt too), the - hooks' audit and state beside it; the task's instruction (task.md), the prompt the session was given - (prompt.txt: the instruction with the plan's prompt suffix, and a slash command's prefix) and the CLI's reply - (last_message.txt).""" - job = tmp_path / "job" - trial = job / "t__abc1234" - write(trial / "agent" / "claude-code.txt", '{"type":"system"}\n') - write(trial / "agent" / "hook-calls.jsonl", '{"hook": "stop"}\n') - plan = write(job / "launch" / "plan.json", json.dumps({"prompt_suffix": "\n\nKeep notes."})) - log = curate.curate(trial, job, errors.classify(trial_result()), instruction="Do alpha.\n", plan=plan, - reply="all done\n") - assert (log / "agent-claude.jsonl").read_text() == '{"type":"system"}\n' - assert not (log / "agent" / "claude-code.txt").exists() and not (log / "agent" / "hook-calls.jsonl").exists() - assert (log / "hook-calls.jsonl").is_file() and (log / "task.md").read_text() == "Do alpha.\n" - assert (log / "prompt.txt").read_text().startswith("Do alpha.") and "Keep notes." in (log / "prompt.txt").read_text() - assert (log / "last_message.txt").read_text() == "all done\n" - - -def test_a_verifier_output_the_container_left_as_a_link_is_never_read(tmp_path): - """The container can write the trial's verifier/ folder: a test-stdout.txt that is a link (to a file of this - machine) is read into neither the curated log, the learner's feedback nor the verdict.""" - host_file = write(tmp_path / "host-only" / "notes.txt", "failed to download https://github.com/x/uv.tar.gz\n" - "uvx: command not found\ndb_password=not-a-real-one\n") - job = tmp_path / "job" - trial = job / "t__abc1234" - write(trial / "agent" / "agent-codex.jsonl", '{"type":"thread.started"}\n') - (trial / "verifier").mkdir() - (trial / "verifier" / "test-stdout.txt").symlink_to(host_file) - verdict = errors.classify(trial_result(0.0)) - log = curate.curate(trial, job, verdict) - assert not (log / "verifier" / "test-stdout.txt").exists() - feedback = judge.score({}, RunOutput(output={"status": "ok", "reward": 0.0}, log=log))["feedback"] - assert "not-a-real-one" not in feedback and "uvx" not in feedback - assert HarborAgent.verifier_verdict(verdict, trial) == verdict # its words are not the verifier's - - -# ---- dataset ----------------------------------------------------------------------------------------------------- - -def test_dataset_rows_lock_and_splits(tmp_path): - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - make_task(tasks, "beta", steps=("s1", "s2")) - rows = dataset.index(tasks) - assert [r["id"] for r in rows] == ["alpha", "beta:s1", "beta:s2"] - alpha, s2 = rows[0], rows[2] - assert alpha["harbor_task"] == "alpha" and alpha["docker_image"] == "example/alpha:1" and alpha["rollout"] == 1 - assert (alpha["agent_timeout_sec"], alpha["verifier_timeout_sec"], alpha["build_timeout_sec"]) == (45.0, 30.0, 120.0) - assert alpha["instruction"] == "Do alpha." and "step" not in alpha - assert (s2["step"], s2["step_index"], s2["steps"], s2["instruction"]) == ("s2", 2, 2, "Step s2 of beta.") - two = dataset.index(tasks, rollouts=2) - assert [r["id"] for r in two][:4] == ["alpha#1", "alpha#2", "beta:s1#1", "beta:s2#1"] - assert dataset.index(tasks, include=["beta"])[0]["id"] == "beta:s1" - with pytest.raises(ValueError): - dataset.index(tasks, include=["gamma"]) - lock = dataset.lock_of(tasks, source={"ref": "local"}) - assert set(lock["tasks"]) == {"alpha", "beta"} and lock["source"] == {"ref": "local"} - lock_path = tmp_path / "dataset.lock" - lock_path.write_text(json.dumps(lock)) - assert dataset.check_lock(tasks, lock_path) == [] - (tasks / "alpha" / "tests" / "test.sh").write_text("#!/bin/bash\necho 0 > /logs/verifier/reward.txt\n") - make_task(tasks, "gamma") - problems = dataset.check_lock(tasks, lock_path) - assert problems == ["task not in the lock: gamma", "task changed since the lock: alpha"] - parts = dataset.split(tasks, train=2, test=1, seed=3) - assert sorted(parts["train"] + parts["test"]) == ["alpha", "beta", "gamma"] - assert dataset.ids_for(dataset.index(tasks), ["beta"]) == ["beta:s1", "beta:s2"] - out = tmp_path / "tasks.jsonl" - assert dataset.main(["index", str(tasks), "--out", str(out), "--rollouts", "2"]) == 0 - assert len(out.read_text().splitlines()) == 8 - assert dataset.main(["check", str(tasks), "--lock", str(lock_path)]) == 1 - - -# ---- judge ------------------------------------------------------------------------------------------------------- - -def test_judge_scores_the_reward_whenever_the_verifier_gave_one(tmp_path): - passed = RunOutput(output={"status": "ok", "reward": 1.0}) - assert judge.score({}, passed) == {"score": 1.0, "feedback": "passed", "status": "ok", "passed": True, - "rewards": None} - log = tmp_path / "log" - write(log / "verifier" / "test-stdout.txt", "FAIL: expected 3 got 2\n") - failed = RunOutput(output={"status": "ok", "reward": 0.0, "rewards": {"reward": 0.0, "f1": 0.3}}, log=log) - verdict = judge.score({}, failed) - assert verdict["score"] == 0.0 and "FAIL: expected 3 got 2" in verdict["feedback"] - assert verdict["rewards"] == {"reward": 0.0, "f1": 0.3} - # A trial that ran out of the agent's time still has its work scored, as Harbor's job mean scores it - timeout = RunOutput(output={"status": "timeout", "failure_cause": "agent_exhausted_budget", - "exception_type": "AgentTimeoutError", "reward": 1.0}, log=log) - verdict = judge.score({}, timeout) - assert verdict["score"] == 1.0 and "time budget" in verdict["feedback"] - assert "passed the verifier" in verdict["feedback"] - assert "FAIL" not in verdict["feedback"] # a pass needs no tail - exited = RunOutput(output={"status": "agent_error", "failure_cause": "unclassified", "reward": 0.5, - "exception_type": "NonZeroAgentExitCodeError", "rewards": {"reward": 0.5}}, log=log) - verdict = judge.score({}, exited) - assert verdict["score"] == 0.5 and "scored its work 0.5" in verdict["feedback"] and "FAIL" in verdict["feedback"] - assert verdict["rewards"] == {"reward": 0.5} - slow = RunOutput(output={"status": "timeout", "failure_cause": "solution_timed_out", - "exception_type": "VerifierTimeoutError", "reward": None}) - assert judge.score({}, slow)["score"] == 0.0 and "solution" in judge.score({}, slow)["feedback"] - # Not the agent's result: no score at all, which the run records as a failed run (judge_error), never a 0 it - # earned - infra = RunOutput(output={"status": "infra", "failure_cause": "verifier_failed", "reward": 0.0}, - extra={"unscored": "verifier_failed", "exception_message": "the verifier's tools never arrived"}) - with pytest.raises(judge.Unscored, match=r"\(verifier_failed: the verifier's tools never arrived\)"): - judge.score({}, infra) - with pytest.raises(judge.Unscored, match=r"\(infra\)"): - judge.score({}, RunOutput(output=None)) - # The benchmark's pass reward: a reward that reaches it passed; with none, nothing is said to pass - half = RunOutput(output={"status": "ok", "reward": 0.8}, extra={"judge": {"pass_reward": 0.75}}) - assert judge.score({}, half)["feedback"] == "passed" and judge.score({}, half)["passed"] is True - scale = RunOutput(output={"status": "ok", "reward": 1.38}, log=log, extra={"judge": {"pass_reward": None}}) - verdict = judge.score({}, scale) - assert verdict["score"] == 1.38 and verdict["feedback"].startswith("reward 1.38") and "FAIL" in verdict["feedback"] - assert verdict["passed"] is None # a score scale: no pass - assert judge.score({}, failed)["passed"] is False - # The adapter's context too long beside the task's own: its 0, and the learner is told by how much - too_long = RunOutput(output={"status": "agent_error", "failure_cause": "context_over_limit", "reward": None, - "exception_type": "HermitContextLimitError"}, - extra={"exception_message": "hermit: AGENTS.md for Codex would be 300000 bytes (Hermit's " - "context 290000, then the image's own 10000), over the CLI's " - "limit of 262144 bytes: shorten the adapter's context"}) - verdict = judge.score({}, too_long) - assert verdict["score"] == 0.0 and "over the CLI's limit of 262144 bytes" in verdict["feedback"] - assert "hermit: " not in verdict["feedback"] - - -# ---- staging ----------------------------------------------------------------------------------------------------- - -def simple_template() -> Path: - """A built-in template without tools or a context tool: an adapter the staging can deploy.""" - for candidate in sorted(TEMPLATES.iterdir()): - exports = candidate / "exports.yaml" - if not exports.is_file(): - continue - declared = yaml.safe_load(exports.read_text()) or {} - if not declared.get("tools") and not declared.get("context_tool") and not declared.get("hooks"): - return candidate - pytest.skip("no built-in template without tools") - - -TOOL_META = """\ -template: {id: calcdemo, version: '1', contract: '2'} -structure: - learnable: [{path: ADAPTER.md, format: {text: true}}] - fixed: [interface.py, exports.yaml] -""" -TOOL_EXPORTS = """\ -context: ADAPTER.md -tools: - - name: calc - description: a tool for the agent - entrypoint: interface.py:calc - input_schema: {type: object, properties: {x: {type: number}}} - output_schema: {type: object} -""" -CONTEXT_EXPORTS = TOOL_EXPORTS + """\ - - name: lookup - description: the harness's per-task context - entrypoint: interface.py:lookup - input_schema: {type: object, properties: {query: {type: string}, task: {type: object}}} - output_schema: {type: object} -context_tool: lookup -""" -TOOL_INTERFACE = '''\ -def calc(x=0): - return {"result": FACTOR * x} - - -def lookup(query="", task=None): - return {"text": f"[NOTES FOR {(task or {}).get('id', '?')}]", "retrieved": [(task or {}).get("id")]} - - -FACTOR = %d -''' - - -def tool_adapter(root: Path, *, factor: int = 2, context_tool: bool = False) -> Path: - """A minimal adapter with one agent tool (calc), and with `context_tool` a per-task context (lookup).""" - write(root / ".hermit" / "adapter.yaml", TOOL_META + ("plugin-portable: false\n" if context_tool else "")) - write(root / "exports.yaml", CONTEXT_EXPORTS if context_tool else TOOL_EXPORTS) - write(root / "ADAPTER.md", "# Static instructions\n") - write(root / "interface.py", TOOL_INTERFACE % factor) - return root - - -def test_stage_deploys_the_adapter_for_the_container(tmp_path): - from hermit.harbor.staging import stage - adapter = simple_template() - info = stage(adapter, tmp_path / "job", sys.executable, "best_effort") - deploy = tmp_path / "job" / "deploy" - assert info["adapter_id"] and (deploy / "AGENTS.md").is_file() and info["deploy"] == str(deploy) - assert (deploy / "hermit" / info["adapter_id"]).is_dir() and info["adapter_dir"] == f"hermit/{info['adapter_id']}" - assert info["hooks"] is False and not (deploy / ".codex" / "hooks.json").exists() - assert info["tools"] == [] and info["context_tool"] is None - assert json.loads((tmp_path / "job" / "staging.json").read_text())["adapter_id"] == info["adapter_id"] - assert stage(None, tmp_path / "none", sys.executable) is None - - -def test_stage_names_the_tools_and_gives_a_per_task_context(tmp_path): - """The staging record names the adapter's agent tools (their names: the server is this machine's, not the - workspace's), and an adapter whose context is per task gets the task's, computed in the adapter sandbox, in its - host's context file; the answer agent's deployment is that context alone.""" - from hermit.harbor.staging import stage - adapter = tool_adapter(tmp_path / "adapter", context_tool=True) - row = {"id": "alpha", "instruction": "Do alpha."} - info = stage(adapter, tmp_path / "job", sys.executable, "best_effort", task=row) - assert info["tools"] == ["calc"] and info["context_tool"] == {"text": "[NOTES FOR alpha]", "retrieved": ["alpha"]} - agents = (tmp_path / "job" / "deploy" / "AGENTS.md").read_text() - assert "[NOTES FOR alpha]" in agents and "Static instructions" in agents - answer = stage(adapter, tmp_path / "ajob", sys.executable, "best_effort", "answer", task=row) - assert answer["deploy"] is None and answer["tools"] == [] and answer["tools_unused"] == ["calc"] - assert "[NOTES FOR alpha]" in Path(answer["context_file"]).read_text() - - -def test_stage_rewrites_hooks_relative_to_the_workspace(tmp_path, monkeypatch, fake_harbor): - import shutil - from hermit.harbor.staging import stage - adapter = tmp_path / "adapter" - shutil.copytree(simple_template(), adapter) - exports = adapter / "exports.yaml" - declared = yaml.safe_load(exports.read_text()) or {} - (adapter / "hooks.json").write_text(json.dumps({"hooks": {"SessionStart": [{"hooks": [ - {"type": "command", "command": "python3 scripts/start.py"}]}]}})) - declared["hooks"] = "hooks.json" - exports.write_text(yaml.safe_dump(declared)) - info = stage(adapter, tmp_path / "job", sys.executable, "best_effort") - hooks = json.loads((tmp_path / "job" / "deploy" / ".codex" / "hooks.json").read_text()) - command = hooks["hooks"]["SessionStart"][0]["hooks"][0]["command"] - assert info["hooks"] is True and command.startswith(f"cd hermit/{info['adapter_id']} || exit 1\n") - assert "python3 scripts/start.py" in command and str(tmp_path) not in command - # Claude Code runs a hook in the shell's current directory: its commands start under $CLAUDE_PROJECT_DIR, and the - # deployment's record is kept for the container (a hook runner reads its host there) - info = stage(adapter, tmp_path / "cjob", sys.executable, "best_effort", "claude") - settings = json.loads((tmp_path / "cjob" / "deploy" / ".claude" / "settings.json").read_text()) - command = settings["hooks"]["SessionStart"][0]["hooks"][0]["command"] - assert info["host"] == "claude" and info["hooks"] is True - assert command.startswith(f'cd "${{CLAUDE_PROJECT_DIR:-.}}"/hermit/{info["adapter_id"]} || exit 1\n') - assert "python3 scripts/start.py" in command and str(tmp_path) not in command - record = json.loads((tmp_path / "cjob" / "deploy" / ".hermit-deploy.json").read_text()) - assert record["host"] == "claude-code" and Path(record["adapter_dir"]).name == info["adapter_id"] - # The launch plan names the deployment the container gets, with its hooks - monkeypatch.setenv("CLAUDE_CODE_OAUTH_TOKEN", TOKEN) - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - agent = HarborAgent({"tasks_dir": str(tasks), "infra_retries": 0, "jobs_dir": str(tmp_path / "jobs"), "web_search": False, - "acting_agent": {"import_path": "harborside.claude:HermitClaude"}, "harbor": str(fake_harbor), - "model": "claude-haiku-5-5", "login": "claude"}, base=tmp_path) - plan_path = agent._launch(row, tmp_path / "cjob", info, None) - kwargs = agent.agent_config(row, plan_path, info)["kwargs"] - plan = json.loads(Path(kwargs["plan"]).read_text()) - assert plan["deploy"]["work_dir"] == str((tmp_path / "cjob" / "deploy").resolve()) - assert plan["deploy"]["hooks"] is True - assert plan["deploy"]["context_file"] == "CLAUDE.md" and "deploy_dir" not in kwargs - - -# ---- the runner's params, checked at startup ------------------------------------------------------------------------- - -def test_runner_is_a_builtin_and_validates_params(tmp_path, fake_harbor): - from hermit.agent.acting.agent import BUILTIN_NAMES, load_agent - assert "harbor" in BUILTIN_NAMES - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - agent = load_agent("harbor", {"tasks_dir": str(tasks), "harbor": str(fake_harbor), "jobs_dir": str(tmp_path / "j"), - "model": "glm-5", "binding": BINDING, "web_search": False}, tmp_path) - assert isinstance(agent, HarborAgent) and agent.tasks_dir == tasks and agent.cli == "codex" - with pytest.raises(ValueError, match="tasks_dir"): - HarborAgent({}, tmp_path) - with pytest.raises(ValueError, match="docker_cleanup"): - make_agent(tmp_path, fake_harbor, tasks, docker_cleanup="prune") - with pytest.raises(ValueError, match="both a name and an import_path"): - make_agent(tmp_path, fake_harbor, tasks, acting_agent={"name": "codex", "import_path": "x:Y"}) - with pytest.raises(ValueError, match="one of codex"): - make_agent(tmp_path, fake_harbor, tasks, acting_agent={"cli": "aider"}) - with pytest.raises(ValueError, match="pass_reward"): - make_agent(tmp_path, fake_harbor, tasks, pass_reward="high") - with pytest.raises(ValueError, match="verifier_redact"): - make_agent(tmp_path, fake_harbor, tasks, verifier_redact=["(unclosed"]) - - -def test_a_cli_without_its_sign_in_is_refused_before_any_trial(tmp_path, fake_harbor, monkeypatch): - """A HermitXXX runs on a / reference or on its own CLI's login, never on neither (whatever the - container finds would pay) nor on both; web search is set either way; a model on this machine's loopback is out - of a container's reach.""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - with pytest.raises(ValueError, match="no / reference"): - make_agent(tmp_path, fake_harbor, tasks, model="gpt-5.6-luna", binding=None) - with pytest.raises(ValueError, match="two bills"): - make_agent(tmp_path, fake_harbor, tasks, login="chatgpt") - with pytest.raises(ValueError, match="takes login: claude"): - make_agent(tmp_path, fake_harbor, tasks, acting_agent={"cli": "claude"}, binding=None, model="haiku", - login="chatgpt") - with pytest.raises(ValueError, match="takes no login"): - make_agent(tmp_path, fake_harbor, tasks, acting_agent={"cli": "pi"}, binding=None, model="m", login="claude") - with pytest.raises(ValueError, match="web_search must be set"): - make_agent(tmp_path, fake_harbor, tasks, web_search=None) - with pytest.raises(ValueError, match="loopback"): - make_agent(tmp_path, fake_harbor, tasks, binding={**BINDING, "base_url": "http://127.0.0.1:8000/v1"}) - with pytest.raises(ValueError, match="codex login"): # no ChatGPT sign-in on this machine - make_agent(tmp_path, fake_harbor, tasks, binding=None, model="gpt-5.6-luna", login="chatgpt") - with pytest.raises(ValueError, match="setup-token"): # nor Claude's token - make_agent(tmp_path, fake_harbor, tasks, acting_agent={"cli": "claude"}, binding=None, model="haiku", - login="claude") - chatgpt_login(tmp_path) - assert make_agent(tmp_path, fake_harbor, tasks, binding=None, model="gpt-5.6-luna", login="chatgpt").cli == "codex" - # Harbor's own agents keep their own sign-in and need none of this - assert make_agent(tmp_path, fake_harbor, tasks, acting_agent={"name": "terminus-2"}, binding=None, - web_search=None).cli is None - # The model's key goes to the session by name: unset here, every trial would fail, so none starts; nor in an - # environment other than Harbor's local Docker one, which cannot pass a variable by name - with pytest.raises(ValueError, match="docker environment"): - make_agent(tmp_path, fake_harbor, tasks, environment={"type": "daytona"}) - monkeypatch.delenv(KEY_ENV) - with pytest.raises(ValueError, match=f"{KEY_ENV} is not set"): - make_agent(tmp_path, fake_harbor, tasks) - - -def test_the_answer_agent_without_its_models_key_is_refused_at_startup(tmp_path, fake_harbor, monkeypatch): - """The answer agent's endpoint on this machine sends each request on with the model's key: without it set, every - trial's call would fail (each an infra rollout), so none starts.""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - monkeypatch.delenv(KEY_ENV) - with pytest.raises(ValueError, match=f"{KEY_ENV} is not set"): - make_agent(tmp_path, fake_harbor, tasks, acting_agent={"cli": "answer"}, web_search=None) - - -# The spec hermit.isolate.agent_pipeline gives the runner for a model a CLI reaches, made by main's own -# endpoint.endpoint_spec (so that a key main adds to it breaks the runner's check of it here, PIPELINE_CLI_SPEC): its -# endpoint's URL and, on a login, a timeout that waits out the plan (3600 s and plan_max_wait_s); it names neither the -# CLI nor its login -def _served() -> dict: - from types import SimpleNamespace - from hermit.agent.base.endpoint import endpoint_spec - endpoint = SimpleNamespace(url="http://127.0.0.1:9/" + "0" * 32, - agent=SimpleNamespace(settings={"model": "gpt-5.6-luna", "login": "chatgpt"})) - return endpoint_spec(endpoint, {"cli": "codex", "model": "gpt-5.6-luna", "login": "chatgpt"}) - - -SERVED = _served() - - -def test_the_answer_agents_attachments_on_a_model_a_cli_reaches_are_refused_at_startup(tmp_path, fake_harbor): - """A session of a model a CLI reaches takes text only: the answer agent's attachments (the task's figures) on such - a model the runner serves (model: {cli: ...}, the benchmark file's) would be refused in every trial, so the run is - refused before the first one. Without attachments, or on a / reference, it starts. A model a CLI - reaches that the pipeline serves (a run config's own `{cli}`: hermit.isolate.agent_pipeline gives the runner its - endpoint's spec alone, which names no CLI; hermit check, the config's params as written) is refused with or - without them: its session waits for its login's plan inside the request, which the runner cannot tell from the - model's own time (the benchmark file's is the runner's own to serve, and wait for).""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - figures = {"attachments": ["/app/figure*.png"]} - answer = {"acting_agent": {"cli": "answer"}, "binding": None, "web_search": None} - reached = {"cli": "codex", "model": "gpt-5.6-luna", "login": "chatgpt"} - with pytest.raises(ValueError, match="takes text only"): - cli_answer_agent(tmp_path, fake_harbor, tasks, reached, answer=figures) - assert make_agent(tmp_path, fake_harbor, tasks, acting_agent={"cli": "answer"}, web_search=None, - answer=figures).cli == "answer" - from hermit.harbor.runner import PIPELINE_CLI_SPEC - assert set(SERVED) == PIPELINE_CLI_SPEC and SERVED["timeout"] == 3600.0 + 12 * 3600 - for given in ({}, {"answer": figures}): - with pytest.raises(ValueError, match="Give it in the benchmark file"): # as hermit eval hands it - make_agent(tmp_path, fake_harbor, tasks, **answer, model=SERVED, **given) - with pytest.raises(ValueError, match="Give it in the benchmark file"): # as hermit check hands it - make_agent(tmp_path, fake_harbor, tasks, **answer, model=reached, **given) - # a model with more than the pipeline endpoint's keys (a loaded config's binding) is the config's own - assert make_agent(tmp_path, fake_harbor, tasks, **answer, model={**SERVED, "ref": "local/gpt-5.6-luna"}).cli \ - == "answer" - - -def test_a_task_harbor_runs_for_none_of_hermits_agents_is_infra_and_never_started(tmp_path, fake_harbor): - """Harbor refuses a task that declares MCP servers or a skills folder, or targets a Windows container, for an - agent whose capabilities lack them (Trial._validate_agent_capabilities), as harborside's agents' do: its refusal - leaves no stream, so such a trial would be an agent error, scored 0. For a table CLI and the answer agent its - rollout is infra (`unsupported_task`, naming why), never handed to Harbor, while the task set's other tasks run (a - split may leave such tasks out, so the run is not refused); one of Harbor's own agents takes them.""" - tasks = tmp_path / "tasks" - make_task(tasks, "plain") - for name, extra in (("mcp", '\n[[environment.mcp_servers]]\nname = "bank"\ntransport = "sse"\n' - 'url = "http://bank:8000/sse"\n'), - ("skills", 'skills_dir = "/skills"\n'), ("windows", 'os = "windows"\n')): - task = make_task(tasks, name) - toml = (task / "task.toml").read_text() - (task / "task.toml").write_text(toml.replace("[environment]\n", "[environment]\n" + extra, 1) - if not extra.startswith("\n[[") else toml + extra) - rows = {r["harbor_task"]: r for r in dataset.index(tasks)} - agent = make_agent(tmp_path, fake_harbor, tasks) - jobs = tmp_path / "jobs" - for name, why in (("mcp", "MCP servers"), ("skills", "a skills folder"), ("windows", "a Windows container")): - out = agent.run(rows[name], None) - assert out.output["status"] == "infra" and out.extra["unscored"] == "unsupported_task", name - assert f"it declares {why}" in out.extra["exception_message"] and out.log is None, name - assert not jobs.exists() or not any(jobs.glob("*/job.yaml")) # none was handed to Harbor - assert agent.run(rows["plain"], None).output["status"] == "ok" - answer = make_agent(tmp_path, fake_harbor, tasks, acting_agent={"cli": "answer"}, web_search=None) - assert answer.run(rows["mcp"], None).extra["unscored"] == "unsupported_task" - own = make_agent(tmp_path, fake_harbor, tasks, acting_agent={"name": "codex"}, binding=None, model="gpt-5.6-luna") - assert own.unsupported(rows["mcp"]) == ["MCP servers"] and own.run(rows["mcp"], None).extra.get("unscored") != \ - "unsupported_task" # Harbor's own Codex takes them - - -def test_the_answer_agents_model_keeps_the_client_timeout_and_retries_its_spec_gives(tmp_path, fake_harbor): - """A model with a client timeout beside it (one that waits out a slow model) keeps it in each trial's relay, which - would otherwise cut the request at its client's 600 s and send it again; so does a bound model with a timeout or - max_retries beside it (the binding's spec keeps them).""" - from hermit.harbor.tools import ModelEndpoint - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - answer = {"acting_agent": {"cli": "answer"}, "web_search": None} - served = make_agent(tmp_path, fake_harbor, tasks, **answer, binding=None, model={**SERVED, "ref": "local/luna"}) - bound = make_agent(tmp_path, fake_harbor, tasks, **answer, binding={**BINDING, "timeout": 1200, "max_retries": 1}) - for agent, client in ((served, (SERVED["timeout"], 4)), (bound, (1200.0, 1))): - relay = ModelEndpoint(agent.answer_model()).start() - try: - assert (relay.client.timeout, relay.client.max_retries) == client - finally: - relay.close() - assert served.answer_model()["timeout"] == SERVED["timeout"] and "max_retries" not in served.answer_model() - - -def test_a_mini_swe_agent_trial_whose_deploying_interpreter_lacks_the_package_is_refused_at_startup( - tmp_path, fake_harbor, monkeypatch): - """A mini-swe-agent trial's adapter is deployed on this machine (hermit.hosts.mini_swe_agent, run with `python`), - which reads the package's own config: where that interpreter has no package, every trial with an adapter would - fail, so the run is refused before the first one, with the install hint; this interpreter is asked in this - process, another in a short child of it. Another CLI's deployment needs no such package.""" - import importlib.util - from types import SimpleNamespace - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - without = write(tmp_path / "bare" / "python", "#!/bin/sh\nexit 1\n", exe=True) # finds no such package - within = write(tmp_path / "full" / "python", "#!/bin/sh\nexit 0\n", exe=True) # finds it - mini = {"acting_agent": {"cli": "mini-swe-agent"}} - with pytest.raises(ValueError, match="mini-swe-agent is not installed"): - make_agent(tmp_path, fake_harbor, tasks, python=str(without), **mini) - assert make_agent(tmp_path, fake_harbor, tasks, python=str(within), **mini).cli == "mini-swe-agent" - assert make_agent(tmp_path, fake_harbor, tasks, python=str(without)).cli == "codex" - real = importlib.util.find_spec - package = SimpleNamespace(submodule_search_locations=[str(tmp_path / "minisweagent")]) - for found in (None, package): # this interpreter, asked in this process - monkeypatch.setattr(importlib.util, "find_spec", - lambda name, *args, found=found: found if name == "minisweagent" else real(name, *args)) - if found is None: - with pytest.raises(ValueError, match="mini-swe-agent is not installed"): - make_agent(tmp_path, fake_harbor, tasks, **mini) - else: - assert make_agent(tmp_path, fake_harbor, tasks, **mini).cli == "mini-swe-agent" - - -def test_a_runner_without_bubblewrap_is_refused_at_startup_unless_its_adapter_sandbox_is_best_effort( - tmp_path, fake_harbor, monkeypatch): - """Staging renders every adapter a trial deploys (its exports, its per-task context) in the strict adapter sandbox, - which, its enforce required (`sandbox_enforce`, the default), refuses to run without bubblewrap: each such trial - would end in that error, scored 0, so the run is refused before the first, with what to do, the answer agent's as - a CLI's. The runner is not told whether the run deploys an adapter, so one that deploys none (hermit eval --adapter - none) is refused too; with the adapter sandbox best_effort it starts.""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - real = shutil.which - with monkeypatch.context() as no_bwrap: - no_bwrap.setattr(shutil, "which", lambda name, *a, **k: None if name == "bwrap" else real(name, *a, **k)) - for params in ({}, {"acting_agent": {"cli": "answer"}}): - cli = (params.get("acting_agent") or {}).get("cli", "codex") - with pytest.raises(ValueError, match=r"bubblewrap \(bwrap\) is not installed: install it, or set " - r"agent.params.sandbox_enforce: best_effort"): - make_agent(tmp_path, fake_harbor, tasks, **params) - assert make_agent(tmp_path, fake_harbor, tasks, sandbox_enforce="best_effort", **params).cli == cli - assert make_agent(tmp_path, fake_harbor, tasks).cli == "codex" # bubblewrap here: as before - - -def test_a_multi_step_task_set_whose_harbor_has_no_python_beside_it_is_refused_at_startup(tmp_path, fake_harbor): - """A multi-step task's steps run in a stream whose driver runs under the python of Harbor's venv, beside `harbor` - (or beside the file a link of that name points to): a `harbor` with none beside it (a wrapper script elsewhere) - would lose every step of every stream, so a task set with a multi-step task is refused before the first one. One - without runs every trial through `harbor` alone, and starts.""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - wrapper = write(tmp_path / "wrapper" / "harbor", f'#!/bin/sh\nexec "{fake_harbor}" "$@"\n', exe=True) - assert make_agent(tmp_path, wrapper, tasks).harbor == str(wrapper) - make_task(tasks, "flow", steps=("s1", "s2")) - with pytest.raises(ValueError, match="multi-step task's driver runs under"): - make_agent(tmp_path, wrapper, tasks) - assert make_agent(tmp_path, fake_harbor, tasks).multi_step() is True - - -def test_a_harbor_command_that_is_not_there_is_refused_at_startup(tmp_path, fake_harbor): - """Every rollout starts `harbor`: one that is not there, or not executable, would make each an error scored 0.""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - with pytest.raises(ValueError, match="not an executable file"): - make_agent(tmp_path, tmp_path / "no-such-venv" / "bin" / "harbor", tasks) - with pytest.raises(ValueError, match="not an executable file"): - make_agent(tmp_path, write(tmp_path / "plain" / "harbor", "#!/bin/sh\n"), tasks) - with pytest.raises(ValueError, match="not an executable file"): # Harbor's own agents run it too - make_agent(tmp_path, tmp_path / "nowhere", tasks, acting_agent={"name": "terminus-2"}, binding=None) - - -def test_an_acting_agent_cli_with_no_class_of_the_table_is_refused_at_startup(tmp_path, fake_harbor): - """`acting_agent.cli` names a CLI of the table: Hermes Agent, a CLI Base Agent of main's (#107), runs in Hermit's - own workspaces only, and a CLI with no HermitXXX class would make every trial an error.""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - for cli in ("hermes", "nope"): - with pytest.raises(ValueError, match=f"acting_agent.cli is one of codex, .*, not '{cli}'"): - make_agent(tmp_path, fake_harbor, tasks, acting_agent={"cli": cli}) - - -def test_a_harbor_named_on_path_or_by_a_relative_path_is_the_one_every_trial_starts(tmp_path, fake_harbor, monkeypatch): - """`harbor: harbor` is the command on PATH, and a relative path is the working directory's: each is resolved once, - so the startup check and every trial (whose harbor runs in its job's directory) start the same one.""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - monkeypatch.setenv("PATH", f"{fake_harbor.parent}{os.pathsep}{os.environ.get('PATH', '')}") - on_path = make_agent(tmp_path, fake_harbor, tasks, harbor="harbor") - assert on_path.harbor == str(fake_harbor) and on_path.run(row, None).output["status"] == "ok" - monkeypatch.chdir(tmp_path) - relative = make_agent(tmp_path, fake_harbor, tasks, harbor="harbor-venv/bin/harbor") - assert os.path.isabs(relative.harbor) and os.path.samefile(relative.harbor, fake_harbor) - assert relative.run(row, None).output["status"] == "ok" - with pytest.raises(ValueError, match="nor a command on PATH"): - make_agent(tmp_path, fake_harbor, tasks, harbor="no-such-harbor") - - -def test_a_harbor_builtins_login_or_effort_is_refused_at_startup(tmp_path, fake_harbor): - """What Harbor's own agents would refuse in every trial is refused before the first one: a login (Harbor's own - agent would put the sign-in on a command line), an effort Claude Code does not take.""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - with pytest.raises(ValueError, match="command line"): - make_agent(tmp_path, fake_harbor, tasks, acting_agent={"name": "codex"}, binding=None, model="m", - login="chatgpt") - with pytest.raises(ValueError, match="command line"): - make_agent(tmp_path, fake_harbor, tasks, acting_agent={"name": "terminus-2"}, binding=None, login="claude") - with pytest.raises(ValueError, match="takes effort"): - make_agent(tmp_path, fake_harbor, tasks, acting_agent={"name": "claude-code"}, binding=None, model="m", - effort="extreme") - assert make_agent(tmp_path, fake_harbor, tasks, acting_agent={"name": "claude-code"}, binding=None, model="m", - effort="high").cli is None - - -def test_harbors_own_claude_code_keeps_harbors_web_rule_per_task_unless_web_search_is_off(tmp_path, fake_harbor): - """Any `disable_web_search` Harbor's agent is given overrides Harbor's own rule (WebSearch and WebFetch off where - the task's agent network is not public): `web_search: false` turns them off on every task, and true or unset gives - Harbor nothing, so a task with no public network never has them; the tools that outlive the session stay denied.""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - row = dataset.index(tasks)[0] - - def kwargs(**params): - return make_agent(tmp_path, fake_harbor, tasks, acting_agent={"name": "claude-code"}, binding=None, model="m", - **params).agent_config(row)["kwargs"] - assert kwargs(web_search=False)["disable_web_search"] is True - for web in (True, None): - given = kwargs(web_search=web) - assert "disable_web_search" not in given and "RemoteTrigger" in given["disallowed_tools"], web - with pytest.raises(ValueError, match="web_search is true or false"): - kwargs(web_search="no") - - -def test_harbors_own_claude_code_is_refused_a_sign_in_it_would_never_be_handed(tmp_path, fake_harbor, monkeypatch): - """Harbor's own claude-code is never handed the subscription's token (it would put it on a command line): - CLAUDE_FORCE_OAUTH, which asks for it, would bill the key in every trial, and the token as the only sign-in of a - model at Anthropic's API would run each one signed out: both refused at startup. A key bills the key, the token - beside it or not, as Harbor's claude-code always did; a model elsewhere takes no subscription.""" - from hermit.agent.base.cli import CLAUDE_BILLING - for name in (*CLAUDE_BILLING, "CLAUDE_FORCE_OAUTH", "CLAUDE_CODE_OAUTH_TOKEN"): - monkeypatch.delenv(name, raising=False) - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - - def builtin(model="claude-haiku-5-5"): - return make_agent(tmp_path, fake_harbor, tasks, acting_agent={"name": "claude-code"}, binding=None, - model=model) - monkeypatch.setenv("CLAUDE_CODE_OAUTH_TOKEN", TOKEN) - for model in ("claude-haiku-5-5", "anthropic/claude-haiku-5-5", normalized("anthropic/claude-haiku-5-5", tasks)): - with pytest.raises(ValueError, match="the only sign-in here.*login: claude"): - builtin(model) - builtin("openrouter/anthropic/claude-haiku-5-5") - make_agent(tmp_path, fake_harbor, tasks, binding=None, model="claude-haiku-5-5", # a bill in the agent's env - acting_agent={"name": "claude-code", "env": {"CLAUDE_CODE_USE_BEDROCK": "1"}}) - own = {"model": "claude-haiku-5-5", "base_url": "https://api.anthropic.com/v1", "api_key_env": "MY_ANTHROPIC_KEY", - "messages_url": "https://api.anthropic.com"} # a backend with a key variable of its own - with pytest.raises(ValueError, match="the only sign-in here"): - builtin(own) - monkeypatch.setenv("MY_ANTHROPIC_KEY", "sk-ant-mine-0123456789abcdef") - builtin(own) - monkeypatch.setenv("ANTHROPIC_API_KEY", "sk-ant-fake-0123456789abcdef") - env = builtin().harbor_env() - assert env["ANTHROPIC_API_KEY"] == "sk-ant-fake-0123456789abcdef" and "CLAUDE_CODE_OAUTH_TOKEN" not in env - for on in ("1", "true", "YES"): - monkeypatch.setenv("CLAUDE_FORCE_OAUTH", on) - with pytest.raises(ValueError, match="CLAUDE_FORCE_OAUTH asks .* bill ANTHROPIC_API_KEY instead"): - builtin() - monkeypatch.setenv("CLAUDE_FORCE_OAUTH", "0") - assert "CLAUDE_FORCE_OAUTH" not in builtin().harbor_env() - with pytest.raises(ValueError, match="CLAUDE_FORCE_OAUTH asks"): # the agent's own env, which Harbor reads - make_agent(tmp_path, fake_harbor, tasks, acting_agent={"name": "claude-code", "env": {"CLAUDE_FORCE_OAUTH": 1}}, - binding=None, model="claude-haiku-5-5") - - -def normalized(ref: str, tasks: Path, acting: dict | None = None): - """The model a loaded config gives one of Harbor's own agents for `ref` (backends.normalize_config): its spec.""" - from hermit.agent.model.backends import normalize_config - data = {"agent": {"name": "harbor", "params": {"tasks_dir": str(tasks), "web_search": False, "model": ref, - "acting_agent": acting or {"name": "claude-code"}}}} - normalize_config(data) - return data["agent"]["params"]["model"] - - -def test_harbors_own_claude_code_is_given_the_messages_root_and_anthropics_own_api_by_name(tmp_path, fake_harbor, - monkeypatch): - """Claude Code appends /v1/messages to its base URL: Harbor's own claude-code on a model a loaded config bound is - given the backend's Messages API root (Hermit's own Claude sessions are given it too), and a model at Anthropic's - own API by name alone, which Harbor signs in to with its key (no base URL, so no skip_model_parsing); Harbor's - own codex is given the endpoint as it is bound.""" - from hermit.agent.base.cli import CLAUDE_BILLING - for name in (*CLAUDE_BILLING, "CLAUDE_FORCE_OAUTH", "CLAUDE_CODE_OAUTH_TOKEN"): - monkeypatch.delenv(name, raising=False) - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - - def agent(ref, acting): - return make_agent(tmp_path, fake_harbor, tasks, acting_agent=acting, binding=None, - model=normalized(ref, tasks, acting)) - claude = agent("anthropic/claude-haiku-5-5", {"name": "claude-code"}) - assert claude.model_fields() == {"model_name": "anthropic/claude-haiku-5-5", - "model_api_key_name": "ANTHROPIC_API_KEY"} - assert "skip_model_parsing" not in claude.agent_config(dataset.index(tasks)[0]) - deepseek = agent("deepseek/deepseek-chat", {"name": "claude-code"}).model_fields() - assert deepseek["base_url"] == "https://api.deepseek.com/anthropic" and deepseek["model_name"] == "deepseek-chat" - assert agent("openai/gpt-4o", {"name": "codex"}).model_fields()["base_url"] == "https://api.openai.com/v1" - - -def test_a_harbor_mini_swe_agent_trial_is_refused_at_startup_what_its_base_agent_refuses(tmp_path, fake_harbor): - """A Harbor trial's launch plan is rendered without the Base Agent's run, whose checks mini-swe-agent's are: they - run on the render too, so web search (mini-swe-agent has none) or a model class whose replies Hermit does not - read is refused at startup, not lost in the container.""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - with pytest.raises(ValueError, match="mini-swe-agent has no web search"): - make_agent(tmp_path, fake_harbor, tasks, acting_agent={"cli": "mini-swe-agent"}, web_search=True) - with pytest.raises(ValueError, match="model_class 'litellm_response'"): - make_agent(tmp_path, fake_harbor, tasks, - acting_agent={"cli": "mini-swe-agent", "settings": {"model_class": "litellm_response"}}) - make_agent(tmp_path, fake_harbor, tasks, acting_agent={"cli": "mini-swe-agent"}) - - -def test_a_verifier_that_ran_out_of_its_time_is_the_agents_result_unless_verifier_timeout_says_infra(tmp_path, - fake_harbor): - """A verifier that ran out of its time (VerifierTimeoutError) is the agent's result by default (`verifier_timeout: - agent`), as Harbor grades it: a timeout on the agent's solution, scored 0 and kept. `verifier_timeout: infra`, for - a host whose load makes verifiers time out, makes it infra (run again); any other value is refused at startup.""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - ran_out = errors.classify(trial_result(None, exc="VerifierTimeoutError")) - timed = make_agent(tmp_path, fake_harbor, tasks).verifier_timeout_verdict(ran_out) - # the session did not run out of its time: timed_out stays false (Continual Harness reads it as the agent's) - assert (timed["status"], timed["failure_cause"], timed["timed_out"]) == ("timeout", "solution_timed_out", False) - assert make_agent(tmp_path, fake_harbor, tasks, verifier_timeout="agent").verifier_timeout_verdict(ran_out) == timed - loaded = make_agent(tmp_path, fake_harbor, tasks, verifier_timeout="infra") - assert (loaded.verifier_timeout_verdict(ran_out)["status"], loaded.verifier_timeout_verdict(ran_out)["failure_cause"], - loaded.verifier_timeout_verdict(ran_out)["timed_out"]) == ("infra", "verifier_failed", False) - other = errors.classify(trial_result(None, exc="RewardFileNotFoundError")) # only the timeout is the option's - assert loaded.verifier_timeout_verdict(other) == other and other["status"] == "agent_error" - with pytest.raises(ValueError, match="verifier_timeout is agent or infra"): - make_agent(tmp_path, fake_harbor, tasks, verifier_timeout="host") - - -def test_runner_merges_a_benchmark_file_under_params(tmp_path, fake_harbor): - tasks = tmp_path / "utils" / "tasks" - make_task(tasks, "alpha") - write(tmp_path / "utils" / "benchmark.yaml", yaml.safe_dump({ - "tasks_dir": "tasks", "model": "glm-5", "binding": BINDING, "effort": "medium", "docker_cleanup": "rmi_over_gb", - "web_search": False, "dataset": {"ref": "local@1"}, # prepare's block, not a param - "acting_agent": {"kwargs": {"version": "0.164.0"}, "env": {"A": "1"}}})) - agent = HarborAgent({"benchmark": "../utils/benchmark.yaml", "effort": "high", "harbor": str(fake_harbor), - "infra_retries": 0, - "jobs_dir": str(tmp_path / "jobs"), "acting_agent": {"env": {"B": "2"}}}, - base=tmp_path / "method") - assert agent.tasks_dir == tasks.resolve() and agent.params["model"] == "glm-5" and "dataset" not in agent.params - assert agent.params["effort"] == "high" and agent.docker_cleanup == "rmi_over_gb" - assert agent.acting_agent == {"kwargs": {"version": "0.164.0"}, "env": {"B": "2"}, "cli": "codex"} - - -def test_job_config_runs_the_cli_class_on_its_launch_plan(tmp_path, fake_harbor): - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - chatgpt_login(tmp_path) - signed_in = {"binding": None, "model": "gpt-5.6-luna", "login": "chatgpt"} - # Harbor's own retries of a trial would each start from the sign-in copied before the first, which the first may - # have refreshed (and so spent): refused on login: chatgpt - with pytest.raises(ValueError, match=r"retry.max_retries > 0 with login: chatgpt"): - make_agent(tmp_path, fake_harbor, tasks, **signed_in, retry={"max_retries": 1}) - agent = make_agent(tmp_path, fake_harbor, tasks, **signed_in, - effort="medium", codex_version="0.164.0", agent_timeout_sec=3600, retry={"max_retries": 0}, - verifier={"override_timeout_sec": 900}, setup_timeout_sec=900, - acting_agent={"cli": "codex", "env": {"OMP_NUM_THREADS": "2"}}) - plan = tmp_path / "job" / "launch" / "plan.json" - config = agent.job_config(row, "alpha-1", agent.agent_config(row, plan, None)) - harbor_agent = config["agents"][0] - # The table's class on the plan; no model for Harbor to check credentials for, no sign-in variable - assert harbor_agent["import_path"] == "harborside.codex:HermitCodex" and "model_name" not in harbor_agent - assert harbor_agent["kwargs"] == {"plan": str(plan), "version": "0.164.0"} - assert harbor_agent["env"] == {"OMP_NUM_THREADS": "2"} and "CODEX_FORCE_AUTH_JSON" not in json.dumps(config) - assert harbor_agent["override_timeout_sec"] == 3600 and harbor_agent["override_setup_timeout_sec"] == 900 - assert "extra_allowed_hosts" not in harbor_agent # a public task: Harbor's own rule holds - assert config["tasks"] == [{"path": str(tasks / "alpha")}] and config["n_concurrent_trials"] == 1 - assert config["retry"] == {"max_retries": 0} and config["verifier"] == {"override_timeout_sec": 900} - retried = make_agent(tmp_path, fake_harbor, tasks, retry={"max_retries": 1}) # a key's: Harbor's retries - assert retried.job_config(row, "alpha-1", retried.agent_config(row, plan, None))["retry"] == {"max_retries": 1} - assert config["environment"] == {"delete": True} - # A table CLI's own default version, when the config names none - assert make_agent(tmp_path, fake_harbor, tasks, acting_agent={"cli": "pi"}).agent_config(row)["kwargs"] == \ - {"version": "1.5.0"} - # A Harbor built-in keeps its model fields: the bound backend's endpoint and the key's variable - bound = make_agent(tmp_path, fake_harbor, tasks, acting_agent={"name": "terminus-2"}, binding=None, - model={"model": "glm", "base_url": "https://x/v1", "api_key_env": "X_KEY"}) - cfg = bound.agent_config(row, None, None) - assert (cfg["model_name"], cfg["base_url"], cfg["model_api_key_name"]) == ("glm", "https://x/v1", "X_KEY") - assert "env" not in cfg - # The wait budget: agent + setup (setup_timeout_sec over Harbor's 360 s) + verifier (its override over the task's - # 30 s) + teardown, plus the build for a Dockerfile task. - assert agent.wait_budget(row) == 3600 + 900 + 900 + 600 - make_task(tasks, "built") - built = dataset.index(tasks, include=["built"])[0] - assert make_agent(tmp_path, fake_harbor, tasks).wait_budget(built) == 45 + 360 + 30 + 120 + 600 - # Harbor's timeout multipliers: into the job, and into the wait budget (the verifier's 30 s doubled; a pulled - # image's start gets 2 x its 120 s limit more, a Dockerfile's build 3 x 120 s); unset, Harbor's default. - assert not {"verifier_timeout_multiplier", "environment_build_timeout_multiplier"} & set(config) - slow = make_agent(tmp_path, fake_harbor, tasks, verifier_timeout_multiplier=2, - environment_build_timeout_multiplier=3) - scaled = slow.job_config(row, "alpha-1", slow.agent_config(row, None, None)) - assert (scaled["verifier_timeout_multiplier"], scaled["environment_build_timeout_multiplier"]) == (2.0, 3.0) - assert slow.wait_budget(row) == 45 + 360 + 60 + 240 + 600 - assert slow.wait_budget(built) == 45 + 360 + 60 + 360 + 600 - with pytest.raises(ValueError, match="verifier_timeout_multiplier must be at least 1"): - make_agent(tmp_path, fake_harbor, tasks, verifier_timeout_multiplier=0.5) - # Another acting agent, no adapter possible. - other = make_agent(tmp_path, fake_harbor, tasks, acting_agent={"name": "mini-swe-agent", "kwargs": {"k": 1}}, - binding=None, model=None) - assert other.agent_config(row, None, None) == {"name": "mini-swe-agent", "kwargs": {"k": 1}} - with pytest.raises(ValueError, match="takes no adapter"): - other.agent_config(row, tmp_path, {"hooks": False}) - - -def test_a_task_without_public_network_names_the_hosts_its_cli_reaches(tmp_path, fake_harbor): - """Harbor's agent phase without the public network (task.toml): the job names the model's host and the address of - the adapter's tools (extra_allowed_hosts, which replaces Harbor's own pick), and the CLI's web search is off - whatever the config says.""" - tasks = tmp_path / "tasks" - make_task(tasks, "closed", network="no-network") - make_task(tasks, "open") - rows = {r["id"]: r for r in dataset.index(tasks)} - agent = make_agent(tmp_path, fake_harbor, tasks, web_search=True) - assert agent.restricted(rows["closed"]) and not agent.restricted(rows["open"]) - hosts = agent.agent_config(rows["closed"], None, None, tools_host="172.17.0.1")["extra_allowed_hosts"] - assert hosts == ["models.example", "172.17.0.1"] - assert "extra_allowed_hosts" not in agent.agent_config(rows["open"], None, None, tools_host="172.17.0.1") - assert agent.settings(rows["closed"])["web_search"] is False and agent.settings(rows["open"])["web_search"] is True - chatgpt_login(tmp_path) - login = make_agent(tmp_path, fake_harbor, tasks, binding=None, model="gpt-5.6-luna", login="chatgpt") - assert login.agent_config(rows["closed"], None, None)["extra_allowed_hosts"] == ["chatgpt.com", "auth.openai.com"] - - -def test_a_mounted_binary_is_the_containers_program(tmp_path, fake_harbor): - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - release = tmp_path / "releases" / "0.164.0-x86_64-unknown-linux-musl" - binary = write(release / "bin" / "codex", "#!/bin/sh\necho codex-cli 0.164.0\n", exe=True) - write(release / "codex-package.json", '{"entrypoint": "bin/codex", "pathDir": "codex-path"}') - agent = make_agent(tmp_path, fake_harbor, tasks, codex_bin=str(binary), codex_version="0.164.0") - config = agent.job_config(row, "alpha-1", agent.agent_config(row, None, None)) - assert config["agents"][0]["kwargs"]["codex_bin"] == "/opt/hermit-codex/bin/codex" - assert {"type": "bind", "source": str(release), "target": "/opt/hermit-codex", "read_only": True} in \ - config["environment"]["mounts"] - plan = json.loads(agent._launch(row, tmp_path / "job", None, None).read_text()) - assert plan["command"][0] == "/opt/hermit-codex/bin/codex" and plan["path_prepend"] == ["/opt/hermit-codex/bin"] - lone = write(tmp_path / "pin" / "dsh", "#!/bin/sh\necho 0.2.2\n", exe=True) # a bare binary, of another CLI - agent = make_agent(tmp_path, fake_harbor, tasks, acting_agent={"cli": "dsh"}, dsh_bin=str(lone)) - config = agent.job_config(row, "alpha-1", agent.agent_config(row, None, None)) - assert config["agents"][0]["kwargs"]["dsh_bin"] == "/opt/hermit-dsh/dsh" - assert config["environment"]["mounts"][-1] == {"type": "bind", "source": str(lone.parent), - "target": "/opt/hermit-dsh", "read_only": True} - with pytest.raises(ValueError, match="not an executable file"): - make_agent(tmp_path, fake_harbor, tasks, codex_bin=str(tmp_path / "nowhere")) - - -# ---- the runner: one trial through the fake harbor ------------------------------------------------------------------ - -def test_a_task_name_longer_than_harbors_trial_name_is_found(tmp_path, fake_harbor): - # Harbor names the trial directory after the task's first 32 characters, '_' and '-' stripped from the end - # (harbor/models/trial/config.py generate_trial_name): a longer name (AlgoTune, GSO, tau3-bench, OSWorld) must - # still find its trial, or every one of its rollouts is an infra zero - tasks = tmp_path / "tasks" - name = "algotune-generalized-eigenvalues-complex" # 41 characters; the 32nd is '-' - make_task(tasks, name, image="example/long:1") - row = dataset.index(tasks)[0] - out = make_agent(tmp_path, fake_harbor, tasks).run(row, None) - assert out.output["status"] == "ok" and out.output["reward"] == 1.0 - assert out.output["trial"] == f"{name[:32].rstrip('_-')}__abc1234" == "algotune-generalized-eigenvalues__abc1234" - - -def test_one_trial_through_the_fake_harbor(tmp_path, fake_harbor, monkeypatch): - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - agent = make_agent(tmp_path, fake_harbor, tasks) - out = agent.run(row, None) - assert out.output["status"] == "ok" and out.output["reward"] == 1.0 and out.output["trial"] == "alpha__abc1234" - # Hermit's reading of the CLI's stream: its tokens and calls, priced at the backend's price, and its reply - assert out.tokens == 120 and out.api_speed == {"model_calls": 1, "status": "ok"} and out.wall_s == 40.0 - assert out.cost_usd == pytest.approx((60 * 1.0 + 40 * 0.5 + 20 * 2.0) / 1e6) - assert out.output["reply"] == "all done\n" and out.extra["session"]["served"] is True - assert out.output["hooks_active"] is False and out.extra["harbor_exit"] == 0 and out.extra["deployed"] is None - assert out.extra["judge"] == {"pass_reward": 1.0} and "unscored" not in out.extra - log = Path(out.log) - assert (log / "agent-codex.jsonl").is_file() and not (log / "agent" / "sessions").exists() - job = Path(out.extra["harbor_job"]) - seen = json.loads(next(job.glob("alpha__*/job-seen.json")).read_text()) - assert seen["env"]["PYTHONPATH"].split(os.pathsep)[0] == str(ROOT / "hermit" / "harbor") - assert seen["env"]["TMPDIR"] == str(tmp_path / "jobs" / "tmp") - assert seen["by_name"] == [KEY_ENV] and seen["token"] is False # the key by its name; no Claude token - harbor_agent = yaml.safe_load((job / "job.yaml").read_text())["agents"][0] - plan = json.loads(Path(harbor_agent["kwargs"]["plan"]).read_text()) - assert plan["env_from_host"] == {KEY_ENV: KEY_ENV} and plan["stream"] == "agent-codex.jsonl" - assert KEY not in job_files(job) # neither the plan, the job nor the logs - assert judge.score(row, out)["score"] == 1.0 - # With an adapter: staged, and the plan names the deployment - with_adapter = agent.run(row, simple_template()) - kwargs = yaml.safe_load((Path(with_adapter.extra["harbor_job"]) / "job.yaml").read_text())["agents"][0]["kwargs"] - deployed = json.loads(Path(kwargs["plan"]).read_text())["deploy"] - assert Path(deployed["work_dir"], "AGENTS.md").is_file() and deployed["context_file"] == "AGENTS.md" - assert with_adapter.output["hooks_active"] is False and with_adapter.extra["deployed"]["adapter_id"] - assert with_adapter.extra["agent_metadata"]["adapter_deployed"] is True - # A verifier whose tools never arrived: infra (run again), not a zero the agent earned. - monkeypatch.setenv("FAKE_HARBOR_MODE", "no_tools") - unreached = agent.run(row, None) - assert unreached.output["status"] == "infra" and unreached.output["failure_cause"] == "verifier_failed" - # A CLI that never started (its stream holds no event): infra, whatever the verifier found - monkeypatch.setenv("FAKE_HARBOR_MODE", "silent") - silent = agent.run(row, None) - assert (silent.output["status"], silent.output["failure_cause"]) == ("infra", "session_failed") - assert silent.extra["unscored"] == "session_failed" and "never started" in silent.extra["exception_message"] - # A CLI that exited non-zero after it worked: the agent's own error, its work scored - monkeypatch.setenv("FAKE_HARBOR_MODE", "agent_exit") - exited = agent.run(row, None) - assert exited.output["status"] == "agent_error" and exited.output["reward"] == 0.5 - assert judge.score(row, exited)["score"] == 0.5 and exited.tokens == 120 - # The wait budget kills a hung harbor. - monkeypatch.setenv("FAKE_HARBOR_MODE", "hang") - monkeypatch.setattr("hermit.harbor.docker.compose_down", lambda name, **k: None) - hung = make_agent(tmp_path, fake_harbor, tasks, wait_budget_s=2).run(row, None) - assert hung.output["status"] == "infra" and hung.output["failure_cause"] == "harbor_timeout" and hung.extra["timed_out"] - # An API account's usage limit ends the run (LoginFailed), the trial's log kept; no later rollout of the runner - # starts a trial - monkeypatch.setenv("FAKE_HARBOR_MODE", "usage_limit") - jobs = set((tmp_path / "jobs").glob("alpha-*")) - with pytest.raises(LoginFailed, match=r"model_provider_failed \(ApiUsageLimitError\): limit; nothing later"): - agent.run(row, None) - [halted] = set((tmp_path / "jobs").glob("alpha-*")) - jobs - assert (halted / "hermit-log" / "result.json").is_file() - monkeypatch.setenv("FAKE_HARBOR_MODE", "ok") - with pytest.raises(LoginFailed, match="nothing later in the run can succeed"): - agent.run(row, None) - assert set((tmp_path / "jobs").glob("alpha-*")) - jobs == {halted} # no job after it - - -def rewrite_trials(monkeypatch, change) -> None: - """Each trial the fake harbor leaves changed by `change(trial)` before the runner reads it.""" - run = HarborAgent.harbor_run - - def harbor_run(self, job, config, task, keep=()): - got = run(self, job, config, task, keep) - for trial in self.trial_dirs(job, task): - change(trial) - return got - monkeypatch.setattr(HarborAgent, "harbor_run", harbor_run) - - -def verifier_said(exc: str | None, stdout: str, reward: float | None = None): - """A change for rewrite_trials: the verifier wrote `stdout` and `reward` (None: no reward file), and Harbor recorded - `exc` (None: no exception).""" - def change(trial: Path) -> None: - result = json.loads((trial / "result.json").read_text()) - result["verifier_result"] = {"rewards": {"reward": reward}} if reward is not None else None - result["exception_info"] = {"exception_type": exc, "exception_message": f"{exc} happened", - "exception_traceback": "tb"} if exc else None - (trial / "result.json").write_text(json.dumps(result)) - (trial / "verifier" / "test-stdout.txt").write_text(stdout) - if reward is None: - (trial / "verifier" / "reward.txt").unlink() - return change - - -def test_a_trial_whose_verifier_failed_on_the_agents_work_is_judged_and_kept_as_harbor_grades_it(tmp_path, fake_harbor, - monkeypatch): - """Through the runner: a verifier that left no reward is the agent's result, judged 0 and kept (in the mean); one - that ran out of its time is the agent's timeout, judged 0, unless `verifier_timeout: infra` makes it infra - (unscored: run again, and recorded as a failed run when it stays so); and one the host's full disk stopped is - infra, its quoted output redacted.""" - from hermit.trainer.rollout import run_one - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - row = dataset.index(tasks)[0] - failing = "FAILED test_outputs.py::test_answer - AssertionError: expected GOLD-ANSWER-42, got 41\n" - with monkeypatch.context() as m: - rewrite_trials(m, verifier_said("RewardFileNotFoundError", failing)) - record = run_one(row, None, make_agent(tmp_path, fake_harbor, tasks).run, judge.score) - assert (record["status"], record["output"]["status"], record["output"]["failure_cause"]) == \ - ("ok", "agent_error", "verifier_failed") - assert record["judge"]["score"] == 0.0 and "unscored" not in record["pipeline_extra"] - with monkeypatch.context() as m: - rewrite_trials(m, verifier_said("VerifierTimeoutError", failing)) - slow = run_one(row, None, make_agent(tmp_path, fake_harbor, tasks).run, judge.score) - loaded = run_one(row, None, make_agent(tmp_path, fake_harbor, tasks, verifier_timeout="infra").run, judge.score) - assert (slow["status"], slow["output"]["status"], slow["output"]["failure_cause"]) == \ - ("ok", "timeout", "solution_timed_out") - assert slow["judge"]["score"] == 0.0 and "solution" in slow["judge"]["feedback"] - unscored_record(loaded, "verifier_failed") - with monkeypatch.context() as m: - rewrite_trials(m, verifier_said(None, failing + "OSError: [Errno 28] No space left on device\n", reward=0.0)) - full = make_agent(tmp_path, fake_harbor, tasks, verifier_redact=[r"GOLD-ANSWER-\d+"]).run(row, None) - assert (full.output["status"], full.extra["unscored"]) == ("infra", "verifier_failed") - assert "No space left on device" in full.extra["exception_message"] - assert "GOLD-ANSWER-42" not in full.extra["exception_message"] - assert "GOLD-ANSWER-42" not in (Path(full.log) / "exception.txt").read_text() - - -def test_an_adapter_this_machine_could_not_stage_is_infra_and_one_whose_own_deployment_failed_the_agents_0( - tmp_path, fake_harbor, monkeypatch): - """Staging the adapter (its host's deployment, its per-task context) runs on this machine before the trial: an - OSError there (a full disk) is the host's, infra (run again), not the agent's 0, for a trial and for a stream's step - alike, and neither a trial nor a stream's driver is started; a RuntimeError, the adapter's deployment or context - failing in its child process, is the agent's result (what the child printed could be the adapter's own words): - agent_error (adapter_failed), scored 0 with its message as the feedback, a trial python -m hermit.harbor.report - counts among the scored ones (its pass@1 a fail), as hermit eval's mean does.""" - from hermit.harbor import report - from hermit.trainer.rollout import run_one - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - make_task(tasks, "flow", steps=("s1", "s2")) - rows = {r["id"]: r for r in dataset.index(tasks)} - agent = make_agent(tmp_path, fake_harbor, tasks) - for failure in (OSError(28, "No space left on device"), - RuntimeError("deploying the adapter failed: No space left on device")): # the adapter could say it - def stage(*args, **kwargs): - raise failure - monkeypatch.setattr("hermit.harbor.runner.stage", stage) - for row in (rows["alpha"], rows["flow:s1"]): - record = run_one(row, simple_template(), agent.run, judge.score) - if isinstance(failure, OSError): - unscored_record(record, "staging_failed") - assert record["output"]["failure_cause"] == "staging_failed", row["id"] - assert str(failure) in record["pipeline_extra"]["exception_message"] - else: # the agent's result: scored 0, its message the feedback - assert (record["status"], record["output"]["status"], record["output"]["failure_cause"]) == \ - ("ok", "agent_error", "adapter_failed"), row["id"] - assert record["judge"]["score"] == 0.0 and "deploying the adapter failed" in record["judge"]["feedback"] - assert (report.kind(record), report.passed(record)) == ("scored", False), row["id"] - assert not list((tmp_path / "jobs").glob("alpha-*/alpha__*")) # no trial ran - assert not list((tmp_path / "jobs" / "streams").glob("flow__s1__*/driver.json")) # no driver started - - - -def test_this_machines_failure_while_a_trials_job_or_launch_is_written_is_infra_and_starts_no_trial( - tmp_path, fake_harbor, monkeypatch): - """Before its container a trial writes its job directory, its launch plan and its job file on this machine: an - OSError there (a full disk, a job directory that exists) is the host's, infra (run again), never the agent's 0.""" - from types import SimpleNamespace - from hermit.trainer.rollout import run_one - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - row = dataset.index(tasks)[0] - agent = make_agent(tmp_path, fake_harbor, tasks) - - def render(*args, **kwargs): - raise OSError(28, "No space left on device") - with monkeypatch.context() as patch: - patch.setattr("hermit.harbor.launch.render", render) - record = run_one(row, simple_template(), agent.run, judge.score) - unscored_record(record, "staging_failed") - assert "No space left on device" in record["pipeline_extra"]["exception_message"] - (tmp_path / "jobs" / "alpha-00000000").write_text("") # where the job directory goes - monkeypatch.setattr("hermit.harbor.runner.uuid.uuid4", lambda: SimpleNamespace(hex="0" * 32)) - record = run_one(row, simple_template(), agent.run, judge.score) - unscored_record(record, "staging_failed") - assert not list((tmp_path / "jobs").glob("alpha-*/alpha__*")) # no trial ran - - -def test_a_chatgpt_trial_hands_the_refreshed_sign_in_back(tmp_path, fake_harbor): - """On `login: chatgpt` the shared sign-in goes into the CLI's home as a secret file, comes back from the - container after the trial and, a later refresh of it, becomes the shared copy; no copy is left in the job.""" - from hermit.agent.base import login - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - chatgpt_login(tmp_path) - agent = make_agent(tmp_path, fake_harbor, tasks, binding=None, model="gpt-5.6-luna", login="chatgpt") - out = agent.run(row, None) - assert out.output["status"] == "ok" and out.cost_usd is None and out.tokens == 120 # the plan's: no price - shared = json.loads(login.master(login.ROOT).read_text()) - assert shared["tokens"]["refresh_token"] == "rt-rotated-stand-in" # the container's later refresh - job = Path(out.extra["harbor_job"]) - assert not list(job.rglob("auth.json")) and "rt-stand-in" not in job_files(job) - plan = json.loads(next(job.glob("launch/plan.json")).read_text()) - assert plan["home"]["secret_files"] == ["auth.json"] and plan["return_files"][0]["from"].endswith("/auth.json") - assert out.extra["session"]["served"] is True - - -def test_a_chatgpt_trials_sign_in_replaces_the_copy_only_as_a_refresh_stamped_on_its_environments_clock( - tmp_path, fake_harbor, monkeypatch): - """The container's session can write the sign-in it hands back (its commands run as the file's owner, with no - Codex sandbox there): a trial whose container left one without its tokens leaves the shared copy as it was (it - replaced the copy before, and every session after it started signed out) and no copy in the job. A refresh stamped - ahead of this machine's clock is refused from Harbor's local Docker environment, whose containers share the clock, - and taken from another environment, whose clock may run up to an hour ahead (a cloud sandbox's).""" - from datetime import datetime, timedelta, timezone - from hermit.agent.base import login - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - chatgpt_login(tmp_path) - monkeypatch.setenv("FAKE_RETURNED", json.dumps({"last_refresh": datetime.now(timezone.utc).isoformat(), - "tokens": {"account_id": "acct-1"}})) - out = make_agent(tmp_path, fake_harbor, tasks, binding=None, model="gpt-5.6-luna", login="chatgpt").run(row, None) - assert out.output["status"] == "ok" and not list(Path(out.extra["harbor_job"]).rglob("auth.json")) - assert json.loads(login.master(login.ROOT).read_text())["tokens"]["refresh_token"] == "rt-stand-in" # the user's - ahead = json.dumps({"last_refresh": (datetime.now(timezone.utc) + timedelta(minutes=30)).isoformat(), - "tokens": {"account_id": "acct-1", "access_token": "at-stand-in", "refresh_token": "rt-ahead"}}) - for kind, taken in ((None, False), ("docker", False), ("modal", True)): - job = tmp_path / "ahead" / str(kind) - (job / "launch" / "return").mkdir(parents=True) - (job / "launch" / "return" / "auth.json").write_text(ahead) - agent = make_agent(tmp_path, fake_harbor, tasks, binding=None, model="gpt-5.6-luna", login="chatgpt", - **({"environment": {"type": kind}} if kind else {})) - agent.sign_in_back(job) - shared = json.loads(login.master(login.ROOT).read_text()) - assert (shared["tokens"]["refresh_token"] == "rt-ahead") is taken, kind - assert not (job / "launch" / "return" / "auth.json").exists(), kind - - -def test_a_codex_trial_on_chatgpt_notes_the_plans_usage_from_its_session_logs(tmp_path, fake_harbor, monkeypatch): - """On `login: chatgpt` the launch plan collects the Codex home's session logs, whose rate-limit reports say how - much of the plan is used: the trial notes the report (as a Hermit session of Codex does), so that the plan's - MAX_USED holds back the trials after it; the logs (the session's transcripts) stay out of the curated log.""" - from hermit.agent.base import login - from hermit.harbor import clis - resets = __import__("time").time() + 3600 - monkeypatch.setenv("FAKE_RATE_LIMITS", json.dumps({"primary": {"used_percent": 97.0, "resets_at": resets}})) - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - chatgpt_login(tmp_path) - out = make_agent(tmp_path, fake_harbor, tasks, binding=None, model="gpt-5.6-luna", login="chatgpt").run(row, None) - plan = json.loads((Path(out.extra["harbor_job"]) / "launch" / "plan.json").read_text()) - assert {"from": f"{clis.CONTAINER_HOME}/sessions", "to": "sessions"} in plan["collect"] - assert out.output["status"] == "ok" and out.extra["session"]["plan_report"]["primary"]["used_percent"] == 97.0 - (login.PLAN / "MAX_USED").write_text("90") - assert "97%" in login.held(login.PLAN) # the trials after it wait for the window - assert not (Path(out.log) / "agent" / "hermit-collect" / "sessions").exists() - - -def test_a_collected_codex_session_log_that_cannot_be_read_gives_no_plan_report(tmp_path, fake_harbor, monkeypatch): - """The container wrote the collected session logs: one that cannot be read (usage_of raises on it) is no report - of the plan, and the session is still read (the trial does not fail).""" - from hermit.agent.base import login - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - row = dataset.index(tasks)[0] - chatgpt_login(tmp_path) - agent = make_agent(tmp_path, fake_harbor, tasks, binding=None, model="gpt-5.6-luna", login="chatgpt") - job, trial = tmp_path / "job", tmp_path / "trial" - plan = agent._launch(row, job, None, None) - write(trial / "agent" / "agent-codex.jsonl", CODEX_STREAM) - log = write(trial / "agent" / "hermit-collect" / "sessions" / "2026" / "11" / "11" / "rollout-1.jsonl", - '{"timestamp": "t", "payload": {"rate_limits": {"primary": {"used_percent": 97.0}}}}\n') - seen = [] - - def unreadable(home): - seen.append(Path(home)) - raise PermissionError(13, "Permission denied", str(log)) - monkeypatch.setattr(login, "usage_of", unreadable) - facts = agent.session(trial, plan, None, job, row)[0] - assert seen == [trial / "agent" / "hermit-collect"] # the collected home was read, and it raised - assert facts["served"] is True and "plan_report" not in facts - - -def test_a_trial_is_priced_at_the_runs_explicit_price(tmp_path, fake_harbor): - """A price given beside the model (agent.params.price, which the config's loading keeps over the backend's) is - the one each trial's usage is priced at, and the one the answer agent's endpoint serves the model at.""" - price = {"input": 10.0, "cached_input": 5.0, "output": 20.0} - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - out = make_agent(tmp_path, fake_harbor, tasks, price=price).run(row, None) - assert out.output["status"] == "ok" and out.cost_usd == pytest.approx((60 * 10.0 + 40 * 5.0 + 20 * 20.0) / 1e6) - answer = make_agent(tmp_path, fake_harbor, tasks, acting_agent={"cli": "answer"}, web_search=None, price=price) - assert answer.answer_model()["price"] == price - - -def test_a_stream_the_session_left_as_a_link_is_no_stream(tmp_path, fake_harbor, monkeypatch): - """The container can write the trial's agent/ folder: a link the session left in its stream's place (to a file - of this machine) is never read, so nothing of that file reaches the rollout's output or its log.""" - host_file = write(tmp_path / "host-only" / "notes.txt", "db_url=postgres://u:not-a-real-password@db/x\n") - monkeypatch.setenv("FAKE_LINK_STREAM_TO", str(host_file)) - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - out = make_agent(tmp_path, fake_harbor, tasks).run(row, None) - assert out.extra.get("session") is None and "reply" not in out.output # no session to read - assert "not-a-real-password" not in json.dumps([out.output, out.extra], default=str) - assert "not-a-real-password" not in job_files(Path(out.extra["harbor_job"])) # its curated log included - - -def test_json_the_container_wrote_nested_past_the_parsers_depth_is_read_as_json_that_does_not_parse( - tmp_path, fake_harbor): - """What the container wrote may be nested past the JSON parser's depth (a RecursionError, not a ValueError): the - runner reads a line of the CLI's stream so nested as no event, as it reads a line that does not parse; the answer - agent's answer.json so nested as no reply and no answer; a Harbor agent's trajectory.json as no model call: never - an error that fails the trial.""" - from hermit.harbor import runner - nested = '{"a": ' + "[" * 100_000 + "]" * 100_000 + "}" - init = {"type": "system", "subtype": "init", "mcp_servers": [{"name": "calcdemo", "status": "connected"}]} - assert list(runner._events(f"{nested}\n{json.dumps(init)}\n")) == [init] - assert curate.model_calls(write(tmp_path / "trajectory.json", nested)) == 0 - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - row = dataset.index(tasks)[0] - agent = make_agent(tmp_path, fake_harbor, tasks, acting_agent={"cli": "answer"}, web_search=None) - plan = write(tmp_path / "job" / "launch" / "plan.json", "{}") - write(tmp_path / "trial" / "agent" / "answer.json", nested) - assert agent.session(tmp_path / "trial", plan, None, tmp_path / "job", row) == \ - ({"started": False}, None, {"reply": None, "answer": None}) - - -def test_an_answer_file_that_is_json_but_no_object_is_no_answer(tmp_path, fake_harbor): - """The answer agent's answer.json is the container's to write: one that is JSON but no object (a list, a string, a - number, null) is read as no reply and no answer, never an error that fails the trial.""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - row = dataset.index(tasks)[0] - agent = make_agent(tmp_path, fake_harbor, tasks, acting_agent={"cli": "answer"}, web_search=None) - plan = write(tmp_path / "job" / "launch" / "plan.json", "{}") - for text in ('["42"]', '"42"', "42", "null"): - write(tmp_path / "trial" / "agent" / "answer.json", text) - assert agent.session(tmp_path / "trial", plan, None, tmp_path / "job", row) == \ - ({"started": False}, None, {"reply": None, "answer": None}), text - - -def test_a_cli_whose_only_line_holds_a_unicode_line_separator_before_a_brace_never_started(tmp_path, fake_harbor): - """A trial's stream is read as Hermit's own run reads one: U+0085, U+2028 and U+2029 end no line, so a CLI's own - line that holds one before a `{` (a value it quotes) starts no event, and a trial whose CLI wrote that line alone - never started a session: infra (session_failed), not the agent's 0.""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - row = dataset.index(tasks)[0] - agent = make_agent(tmp_path, fake_harbor, tasks) - job, trial = tmp_path / "job", tmp_path / "trial" - plan = agent._launch(row, job, None, None) - exited = errors.classify(trial_result(0.0, exc="NonZeroAgentExitCodeError")) - stream = trial / "agent" / "agent-codex.jsonl" - stream.parent.mkdir(parents=True) - for sep in ("\x85", "\u2028", "\u2029"): - stream.write_bytes(f"Error loading config.toml: unknown key{sep}{{ model = 1 }}\n".encode("utf-8")) - facts = agent.session(trial, plan, None, job, row)[0] - assert facts["started"] is False, repr(sep) - verdict = HarborAgent.session_verdict(exited, facts) - assert (verdict["status"], verdict["failure_cause"]) == ("infra", "session_failed"), repr(sep) - - -def test_a_stream_whose_progress_line_ends_at_a_carriage_return_before_an_event_reads_the_event(tmp_path, fake_harbor): - """A trial's stream is read as Hermit's own run reads one (Path.read_text): a "\\r" ends a line, as "\\r\\n" and - "\\n" do. So the CLI's own progress line ending at "\\r" ("working 50%\\r") before an event on the same line leaves - the event whole: a command's output that names a lost sign-in is the session's work, not a notice of the CLI's, - and a turn that completed is read as served.""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - row = dataset.index(tasks)[0] - agent = make_agent(tmp_path, fake_harbor, tasks) - job, trial = tmp_path / "job", tmp_path / "trial" - plan = agent._launch(row, job, None, None) - command = {"type": "item.completed", "item": {"id": "i1", "type": "command_execution", "command": "cat auth.log", - "aggregated_output": "Not logged in", "exit_code": 0, - "status": "completed"}} - done = {"type": "turn.completed", "usage": {"input_tokens": 100, "cached_input_tokens": 40, "output_tokens": 20}} - start = '{"type":"thread.started","thread_id":"th-1"}\r\n{"type":"turn.started"}\n' - cut = start + f"working 50%\r{json.dumps(command)}\n" # no turn completed: cut off - stream = trial / "agent" / "agent-codex.jsonl" - stream.parent.mkdir(parents=True) - stream.write_bytes(cut.encode("utf-8")) - facts = agent.session(trial, plan, None, job, row)[0] - assert (facts["served"], facts["login_failed"]) == (False, False) - stream.write_bytes((cut + f"working 90%\r{json.dumps(done)}\n").encode("utf-8")) - facts = agent.session(trial, plan, None, job, row)[0] - assert (facts["served"], facts["login_failed"], facts["tokens"]) == (True, False, 120) - - -# A Codex session that ran out of the agent's time: the provider's notice it recovered from, its work, no turn completed -TIMED_OUT = ('{"type":"thread.started","thread_id":"th-1"}\n{"type":"turn.started"}\n' - '{"type":"error","message":"Reconnecting... 1/5 (stream disconnected before completion: error sending ' - 'request)"}\n' - '{"type":"item.completed","item":{"id":"i1","type":"agent_message","text":"working"}}\n') - - -def test_a_trial_that_ran_out_of_its_time_after_a_notice_it_recovered_from_is_the_agents_result( - tmp_path, fake_harbor, monkeypatch): - """A session that ran out of the agent's time has no completed turn: a provider's notice earlier in its stream, - which the CLI recovered from, does not make it infra. The trial is the agent's result, its work scored, and on a - login it is not run again, as Hermit's own sessions keep a timed-out session.""" - monkeypatch.setenv("FAKE_STREAMS", str(write(tmp_path / "timed-out.jsonl", TIMED_OUT))) - monkeypatch.setenv("FAKE_HARBOR_MODE", "timeout") - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - chatgpt_login(tmp_path) - out = make_agent(tmp_path, fake_harbor, tasks, binding=None, model="gpt-5.6-luna", login="chatgpt", - busy_retries=1, busy_wait_s=0).run(row, None) - assert (out.output["status"], out.output["failure_cause"], out.output["reward"]) == \ - ("timeout", "agent_exhausted_budget", 1.0) - assert "unscored" not in out.extra and judge.score(row, out)["score"] == 1.0 - assert (tmp_path / "jobs" / ".fake-calls").read_text() == "1" # not run again - - -# A Codex session that met a 401 it reconnected from (Codex retries it), went on working and ran out of the agent's time -RECOVERED_401 = ('{"type":"thread.started","thread_id":"th-1"}\n{"type":"turn.started"}\n' - '{"type":"error","message":"Reconnecting... 1/5 (unexpected status 401 Unauthorized: Missing bearer ' - 'or basic authentication in header, url: https://api.openai.com/v1/responses)"}\n' - '{"type":"item.completed","item":{"id":"i1","type":"agent_message","text":"working"}}\n') - - -def test_a_codex_trial_that_ran_out_of_its_time_after_a_401_it_reconnected_from_is_the_agents_timeout( - tmp_path, fake_harbor, monkeypatch): - """Codex retries a 401 it meets (its reconnect notice), as it does a 403; one that lasts fails the turn. A session - that recovered from one and then ran out of the agent's time is the agent's timeout, its work scored, never a lost - sign-in that halts the run.""" - monkeypatch.setenv("FAKE_DEFAULT_STREAM", RECOVERED_401) - monkeypatch.setenv("FAKE_HARBOR_MODE", "timeout") - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - agent = make_agent(tmp_path, fake_harbor, tasks) - out = agent.run(row, None) - assert (out.output["status"], out.output["reward"], out.extra["halt"]) == ("timeout", 1.0, False) - assert out.extra["session"]["login_failed"] is False and "unscored" not in out.extra - assert not agent._stop.is_set() - - -# A Codex session cut off by the agent's time after Codex logged a patch whose lines it did not find (harborside's -# `> stream 2>&1` puts its stderr in the stream): the record, then those lines, the model's own text -QUOTED_SIGN_IN = ('{"type":"thread.started","thread_id":"th-1"}\n{"type":"turn.started"}\n' - '{"type":"item.completed","item":{"id":"i1","type":"agent_message","text":"fixing the handler"}}\n' - '2026-11-12T01:02:03.456789Z ERROR codex_core::tools::router: error=apply_patch verification failed: ' - 'Failed to find expected lines in /app/server/auth.py:\n' - ' return jsonify(error="Invalid API key"), 401\n' - '{"type":"item.started","item":{"id":"i2","type":"command_execution","command":"pytest -q",' - '"aggregated_output":"","exit_code":null,"status":"in_progress"}}\n') - - -def test_a_codex_trial_that_ran_out_of_its_time_after_its_log_quoted_a_lost_sign_in_is_the_agents_timeout( - tmp_path, fake_harbor, monkeypatch): - """What Codex's log quotes of the session's tool calls is the session's own work: a trial that ran out of the - agent's time after the log quoted the words of a lost sign-in is the agent's timeout, its work scored, never a - lost sign-in that halts the run (the trial after it runs).""" - monkeypatch.setenv("FAKE_DEFAULT_STREAM", QUOTED_SIGN_IN) - monkeypatch.setenv("FAKE_HARBOR_MODE", "timeout") - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - make_task(tasks, "beta", image="example/beta:1") - rows = dataset.index(tasks) - agent = make_agent(tmp_path, fake_harbor, tasks) - for row in rows: - out = agent.run(row, None) - assert (out.output["status"], out.output["reward"], out.extra["halt"]) == ("timeout", 1.0, False), row["id"] - assert out.extra["session"]["login_failed"] is False and "unscored" not in out.extra - assert not agent._stop.is_set() - - -# A Codex session the provider refused for its sign-in (a revoked ChatGPT login, a key it does not take) -SIGN_IN_REFUSED = ('{"type":"thread.started","thread_id":"th-1"}\n{"type":"turn.started"}\n' - '{"type":"error","message":"unexpected status 401 Unauthorized: Your authentication token has ' - 'been invalidated. Please try signing in again."}\n' - '{"type":"turn.failed","error":{"message":"unexpected status 401 Unauthorized"}}\n') - - -def test_a_codex_session_whose_sign_in_the_provider_refused_ends_the_run(tmp_path, fake_harbor, monkeypatch, capsys): - """A 401 for the session's sign-in is no answer of the agent's, and no wait restores it: the run ends (LoginFailed, - which hermit.trainer.rollout passes up, as for a Hermit session's lost sign-in; Harbor's own Codex agent's - AgentAuthenticationError did so too), naming the cause and the exit Harbor kept, the trial's log kept; no later - rollout of the runner starts a trial.""" - monkeypatch.setenv("FAKE_DEFAULT_STREAM", SIGN_IN_REFUSED) - monkeypatch.setenv("FAKE_HARBOR_MODE", "agent_exit") # the CLI exited 1; the verifier ran - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - agent = make_agent(tmp_path, fake_harbor, tasks) - lost = r"a Harbor trial \(alpha-\w+\): login_failed \(NonZeroAgentExitCodeError\): " - with pytest.raises(LoginFailed, match=lost) as ended: - agent.run(row, None) - assert "nothing later in the run can succeed" in str(ended.value) and agent._stop.is_set() - [job] = (tmp_path / "jobs").glob("alpha-*") - assert (job / "hermit-log" / "result.json").is_file() and f"(the trial's job: {job})" in capsys.readouterr().err - with pytest.raises(LoginFailed): - agent.run(row, None) - assert len(list((tmp_path / "jobs").glob("alpha-*"))) == 1 # no trial after it - # through the rollouts of hermit train and hermit eval (hermit.trainer.rollout.run_tasks): the run ends there, the - # row no record of a failed run - from hermit.trainer.rollout import run_tasks - with pytest.raises(LoginFailed, match="login_failed"): - run_tasks([row], None, make_agent(tmp_path, fake_harbor, tasks).run, judge.score, concurrency=1) - - -def test_a_codex_session_an_api_account_refused_for_its_bill_ends_the_run_and_for_its_model_is_model_rejected( - tmp_path, fake_harbor, monkeypatch): - """Codex on an API account (a key, no login): a session the provider refused for the account's bill (its quota - spent, a 402) ends the run, as an API account's usage limit does (usage_limit, a halt: LoginFailed), since nothing - later can be served; one it refused for the run's configuration (a model not there) is infra (model_rejected) and - never run again, as the other CLIs' are, never the agent's 0; the run goes on. The plan's usage-limit words that - Codex's log quotes of a tool call (a file the session patched) are no refusal: the trial is the agent's result, and - the run goes on (the Base Agent's reading of the whole stream takes them for the account's limit, which on an API - account ends the run).""" - monkeypatch.setenv("FAKE_HARBOR_MODE", "agent_exit") # the CLI exited 1; the verifier ran - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - start = '{"type":"thread.started","thread_id":"th-1"}\n{"type":"turn.started"}\n' - failed = '{"type":"turn.failed","error":{"message":"%s"}}\n' - for n, why in enumerate(("Quota exceeded. Check your plan and billing details.", - "unexpected status 402 Payment Required: {\\\"detail\\\":\\\"payment required\\\"}")): - monkeypatch.setenv("FAKE_DEFAULT_STREAM", start + '{"type":"error","message":"%s"}\n' % why + failed % why) - jobs = tmp_path / f"jobs-bill-{n}" - agent = make_agent(tmp_path, fake_harbor, tasks, infra_retries=2, infra_wait_s=0, jobs_dir=str(jobs)) - with pytest.raises(LoginFailed, match="the provider does not bill the account"): - agent.run(row, None) - assert agent._stop.is_set(), why - assert len(list(jobs.glob("alpha-*"))) == 1, why # never run again - why = "unexpected status 404 Not Found: The model `gpt-x` does not exist" - monkeypatch.setenv("FAKE_DEFAULT_STREAM", start + '{"type":"error","message":"%s"}\n' % why + failed % why) - jobs = tmp_path / "jobs-model" - agent = make_agent(tmp_path, fake_harbor, tasks, infra_retries=2, infra_wait_s=0, jobs_dir=str(jobs)) - out = agent.run(row, None) - assert (out.output["status"], out.output["failure_cause"], out.extra["halt"]) == ("infra", "model_rejected", False) - assert out.extra["unscored"] == "model_rejected" and not agent._stop.is_set() - assert len(list(jobs.glob("alpha-*"))) == 1 # never run again - quoted = (start + "2026-11-12T01:02:03.456789Z ERROR codex_core::tools::router: error=apply_patch verification " - "failed: Failed to find expected lines in /app/limits.py:\n" - '{"note": "a JSON line of the patch"}\n' - " raise Exhausted(\"You've hit your usage limit. Try again in 2 hours.\")\n" + failed % "exit") - monkeypatch.setenv("FAKE_DEFAULT_STREAM", quoted) - agent = make_agent(tmp_path, fake_harbor, tasks, jobs_dir=str(tmp_path / "jobs-quoted")) - out = agent.run(row, None) - assert (out.output["status"], out.extra["halt"], out.extra["session"]["refused"]) == ("agent_error", False, False) - assert not agent._stop.is_set() - - -def test_a_login_trial_the_plan_refused_holds_the_plan_with_the_runs_stop(tmp_path, fake_harbor, monkeypatch): - """On a login, a trial whose session the plan refused for its usage limit (Codex's own words) holds every session - until the window resets and runs again then; the hold gets the run's stop, so that a hold that cannot be written - (main's hold then waits five minutes itself, in the trial's slot) ends as soon as the run ends.""" - from hermit.agent.base import login as subscription - monkeypatch.setenv("FAKE_HARBOR_MODE", "agent_exit") - limit = "You've hit your usage limit. Upgrade to Pro or try again in 2 hours." - start = '{"type":"thread.started","thread_id":"th-1"}\n{"type":"turn.started"}\n' - events = start + ('{"type":"error","message":"%s"}\n' % limit - + '{"type":"turn.failed","error":{"message":"%s"}}\n' % limit) - monkeypatch.setenv("FAKE_STREAMS", str(write(tmp_path / "limit.jsonl", events))) - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - chatgpt_login(tmp_path) - agent = make_agent(tmp_path, fake_harbor, tasks, binding=None, model="gpt-5.6-luna", login="chatgpt", - plan_dir=str(tmp_path / "plan")) - holds = [] - - def hold(plan, until=None, why="session_refused", *, cancel=None, max_s=None): - holds.append(cancel) - agent._stop.set() # the run ends meanwhile (another rollout's lost sign-in) - monkeypatch.setattr(subscription, "hold", hold) - with pytest.raises(LoginFailed): - agent.run(row, None) - assert holds == [agent._stop] and (tmp_path / "jobs" / ".fake-calls").read_text() == "1" - - -# A Codex session that met a provider's notice (a reconnect), then lost its sign-in: the turn failed with a 401 -LOST_AFTER_NOTICE = ('{"type":"thread.started","thread_id":"th-1"}\n{"type":"turn.started"}\n' - '{"type":"error","message":"Reconnecting... 1/5 (stream disconnected before completion: error ' - 'sending request)"}\n' - '{"type":"turn.failed","error":{"message":"unexpected status 401 Unauthorized: Your ' - 'authentication token has been invalidated."}}\n') -# ... and one that reported the lost sign-in, then failed its turn on a dropped stream: lost, and unserved too -LOST_THEN_DROPPED = ('{"type":"thread.started","thread_id":"th-1"}\n{"type":"turn.started"}\n' - '{"type":"error","message":"unexpected status 401 Unauthorized: Your authentication token has ' - 'been invalidated."}\n' - '{"type":"turn.failed","error":{"message":"stream disconnected before completion: error sending ' - 'request"}}\n') - - -def test_a_login_trial_that_lost_its_sign_in_after_a_providers_notice_is_never_run_again(tmp_path, fake_harbor, - monkeypatch): - """On a login, a session whose stream holds a provider's notice and a lost sign-in is the lost sign-in: it ends the - run (LoginFailed), never run again as one the provider could not serve; whether the notice is one it recovered - from (a reconnect) or the failure its turn ended with.""" - monkeypatch.setenv("FAKE_HARBOR_MODE", "agent_exit") - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - chatgpt_login(tmp_path) - for n, events in enumerate((LOST_AFTER_NOTICE, LOST_THEN_DROPPED)): - monkeypatch.setenv("FAKE_STREAMS", str(write(tmp_path / f"lost-{n}.jsonl", events))) - jobs = tmp_path / f"jobs-{n}" - agent = make_agent(tmp_path, fake_harbor, tasks, binding=None, model="gpt-5.6-luna", login="chatgpt", - busy_retries=2, busy_wait_s=0, jobs_dir=str(jobs)) - with pytest.raises(LoginFailed, match="login_failed"): - agent.run(row, None) - assert agent._stop.is_set() and (jobs / ".fake-calls").read_text() == "1", n # not run again - -# A Codex 0.164.0 session (the standalone release, Linux; its stderr merged in, as harborside keeps it) on a ChatGPT -# login whose requests to chatgpt.com all failed, from a recorded TB2.0 trial on gpt-5.6-luna (2026-11-08), its ids and -# times replaced and its repeated lines cut, every message as Codex wrote it: its refresh of its models and its MCP -# client failed ("error sending request"), it reconnected over WebSockets, fell back to HTTPS (an `error` item), -# reconnected again, then failed its turn with "workspace routing discovery failed" and exited 1 -UNREACHABLE = ( - "Reading additional input from stdin...\n" - "2026-11-08T05:13:56.000000Z ERROR codex_models_manager::manager: failed to refresh available mod" - "els: Connection failed: error sending request\n" - "{\"type\":\"thread.started\",\"thread_id\":\"00000000-0000-7000-8000-000000000001\"}\n" - "2026-11-08T05:13:57.000000Z ERROR rmcp::transport::worker: worker quit with fatal: Transport cha" - "nnel closed, when Client(HttpRequest(HttpRequest(\"http/request failed: error sending request for" - " url (https://chatgpt.com/backend-api/ps/mcp)\")))\n" - "{\"type\":\"turn.started\"}\n" - "{\"type\":\"error\",\"message\":\"Reconnecting... 4/5 (workspace routing discovery failed)\"}\n" - "{\"type\":\"error\",\"message\":\"Reconnecting... 5/5 (workspace routing discovery failed)\"}\n" - "{\"type\":\"item.completed\",\"item\":{\"id\":\"item_0\",\"type\":\"error\",\"message\":\"Falling back from WebSo" - "ckets to HTTPS transport. workspace routing discovery failed\"}}\n" - "{\"type\":\"error\",\"message\":\"Reconnecting... 4/5 (workspace routing discovery failed)\"}\n" - "{\"type\":\"error\",\"message\":\"Reconnecting... 5/5 (workspace routing discovery failed)\"}\n" - "{\"type\":\"error\",\"message\":\"workspace routing discovery failed\"}\n" - "{\"type\":\"turn.failed\",\"error\":{\"message\":\"workspace routing discovery failed\"}}\n" -) - - -def test_a_login_trial_whose_codex_could_not_reach_its_provider_is_run_again_and_never_the_agents_result( - tmp_path, fake_harbor, monkeypatch): - """Codex 0.164 on a ChatGPT login whose requests to chatgpt.com fail retries them, fails its turn with "workspace - routing discovery failed" and exits 1 (UNREACHABLE): the provider could not serve the trial, which runs again - after a pause, busy_retries times over the whole trial (infra_retries, here 2, adds none: one count, never their - product); when every run met the same it is infra (`model_provider_failed`, unscored, no halt), never the agent's - result scored as Harbor scored it.""" - monkeypatch.setenv("FAKE_STREAMS", str(write(tmp_path / "unreachable.jsonl", UNREACHABLE))) - monkeypatch.setenv("FAKE_HARBOR_MODE", "agent_exit") # the CLI exited 1; the verifier ran - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - chatgpt_login(tmp_path) - out = make_agent(tmp_path, fake_harbor, tasks, binding=None, model="gpt-5.6-luna", login="chatgpt", - busy_retries=1, busy_wait_s=0, infra_retries=2, infra_wait_s=0).run(row, None) - assert (out.output["status"], out.output["failure_cause"], out.extra["unscored"], out.extra["halt"]) == \ - ("infra", "model_provider_failed", "model_provider_failed", False) - assert out.extra["session"]["unavailable"] is True and out.extra["session"]["login_failed"] is False - assert (tmp_path / "jobs" / ".fake-calls").read_text() == "2" # run again once (busy_retries) ... - assert [a["unscored"] for a in out.extra["attempts"]] == ["model_provider_failed"] * 2 # infra_retries: no more - - -def test_the_answer_agent_asks_the_model_through_an_endpoint_of_this_machine(tmp_path, fake_harbor): - """`cli: answer`: one model call from Harbor's process to an endpoint the trial serves on 127.0.0.1, which relays - it to the backend with the key (never in the plan or the job); the answer, the reply and the usage are Hermit's.""" - seen = [] - - class Model(BaseHTTPRequestHandler): - def log_message(self, *a): - pass - - def do_POST(self): - seen.append((self.path, self.headers.get("Authorization"), - json.loads(self.rfile.read(int(self.headers["Content-Length"]))))) - body = json.dumps({"id": "c1", "object": "chat.completion", "model": "glm-5", - "choices": [{"index": 0, "message": {"role": "assistant", "content": "The answer is 42"}, - "finish_reason": "stop"}], - "usage": {"prompt_tokens": 30, "completion_tokens": 5}}).encode() - self.send_response(200) - self.send_header("Content-Type", "application/json") - self.send_header("Content-Length", str(len(body))) - self.end_headers() - self.wfile.write(body) - - server = ThreadingHTTPServer(("127.0.0.1", 0), Model) - threading.Thread(target=server.serve_forever, daemon=True).start() - try: - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - local = {**BINDING, "base_url": f"http://127.0.0.1:{server.server_address[1]}/v1"} - agent = make_agent(tmp_path, fake_harbor, tasks, acting_agent={"cli": "answer"}, binding=local, - web_search=None, answer={"answer_file": "/app/answer.txt"}) - out = agent.run(row, simple_template()) - finally: - server.shutdown() - assert out.output["status"] == "ok" and out.output["answer"] == "42" and out.output["reply"] == "The answer is 42" - assert out.tokens == 35 and out.api_speed["model_calls"] == 1 - assert out.cost_usd == pytest.approx((30 * 1.0 + 5 * 2.0) / 1e6) - path, authorization, request = seen[0] - assert path == "/v1/chat/completions" and authorization == f"Bearer {KEY}" and request["model"] == "glm-5" - context = Path(out.extra["deployed"]["context_file"]).read_text() - assert context and request["messages"][0] == {"role": "system", "content": context} # the adapter's context - job = Path(out.extra["harbor_job"]) - plan = json.loads((job / "launch" / "plan.json").read_text()) - assert plan["command"] is None and plan["answer"]["answer_file"] == "/app/answer.txt" - assert plan["answer"]["endpoint"].startswith("http://127.0.0.1:") and KEY not in job_files(job) - assert out.extra["session"]["served"] is True and out.extra["deployed"]["deploy"] is None - with pytest.raises(ValueError, match="answer_format"): - make_agent(tmp_path, fake_harbor, tasks, acting_agent={"cli": "answer"}, answer={"answer_format": "x"}) - with pytest.raises(ValueError, match="takes no login"): - make_agent(tmp_path, fake_harbor, tasks, acting_agent={"cli": "answer"}, binding=None, model="m", - login="chatgpt") - - -def test_the_answer_agents_endpoint_says_how_its_calls_went(): - """The answer agent's session facts are its endpoint's calls, as a CLI's stream gives them: a relay's served, - unavailable (a retry may help) or rejected; a CLI session's refused (the plan's 429) or lost sign-in; its usage - from the sessions' replies, unpriced on a login.""" - from hermit.agent.base.interface import LOGIN_FAILED - from hermit.harbor.tools import ModelEndpoint - relay = ModelEndpoint(BINDING) - relay._note((503, b'{"error": {"message": "busy"}}', {})) - relay._note((200, b'{"choices": []}', {})) - assert relay.facts()["served"] is True and relay.facts()["calls"] == 2 - relay.statuses, relay.codes = [None], [None] # no answer at all - assert relay.facts()["unavailable"] is True and relay.facts()["started"] is True - relay.statuses, relay.codes = [400], [None] - assert relay.facts()["rejected"] is True and not relay.facts()["unavailable"] - session = ModelEndpoint({"cli": "claude", "model": "claude-haiku-5-5", "login": "claude"}) - session._note((429, b'{"error": {"message": "the plan refused"}}', {})) - assert session.facts()["refused"] is True and not session.facts()["unavailable"] - session._note((401, json.dumps({"error": {"code": LOGIN_FAILED, "message": "lost"}}).encode(), {})) - assert session.facts()["login_failed"] is True and not session.facts()["rejected"] - session._note((200, b'{"usage": {"prompt_tokens": 10, "completion_tokens": 2}}', {})) - assert (session.usage.input_tokens, session.usage.output_tokens, session.usage.calls) == (10, 2, 1) - assert session.usage.cost_usd is None and ModelEndpoint({"cli": "pi"}).facts()["started"] is False - - -def test_the_answer_agents_endpoint_keeps_what_a_refusal_says_in_a_long_message_or_at_a_bodys_top_level(tmp_path): - """A failed call's message is kept to its last 1000 characters: the whole of Hermit's endpoint's reply to a CLI's - failure (endpoint._no_reply: its prefix and the last 600 characters of the failure, as much as the runner reads of - a container CLI's), so that a Pi failure of up to 600 characters keeps its leading `401 {` behind the prefix; and - of a longer provider message its end, where the refusal's words are. A body whose error has no message says it at - its top level, as a `message` (Mistral's) or a `detail` string (a FastAPI server's). Either way, a refusal of the - configuration is model_rejected.""" - from hermit.agent.base.cli import Pi - from hermit.agent.base.endpoint import _no_reply - from hermit.agent.base.interface import Result - from hermit.harbor.tools import ModelEndpoint - exited = errors.classify(trial_result(0.5, exc="NonZeroAgentExitCodeError")) - head = '401 {"error":{"message":"bad key","request":"' - for n in (500, 598, 599, 600): # a Pi failure of n characters (Pi keeps its last 600) - failure = head + "x" * (n - len(head) - 3) + '"}}' - said = outcome(Pi({}), pi_failed(failure), tmp_path) - session = ModelEndpoint({"cli": "pi", "model": "m"}) - session._note(_no_reply(Pi({}), Result(reply="", exit_code=1, extra=said))) - facts = session.facts() - assert facts["statuses"] == [400] and facts["failure"] == f"the pi session gave no reply (exit 1): {failure}", n - assert HarborAgent.session_verdict(exited, facts)["failure_cause"] == "model_rejected", n - longer = "The request could not be completed: " + "details " * 150 + "The model `gpt-nope` does not exist." - relay = ModelEndpoint(BINDING) - relay._note((400, json.dumps({"error": {"message": longer}}).encode(), {})) - tail = relay.facts()["failure"] - assert len(longer) > 1000 and longer.endswith(tail) and tail.endswith("does not exist.") - assert HarborAgent.session_verdict(exited, relay.facts())["failure_cause"] == "model_rejected" - for body, kept in (({"object": "error", "message": "Invalid model: mistral-nope", "type": "invalid_model", - "param": None, "code": "1500"}, "Invalid model: mistral-nope"), - ({"detail": "Model not found"}, "Model not found"), - ({"error": {"message": "refused"}, "message": "elsewhere"}, "refused"), - ({"detail": [{"loc": ["body", "model"], "msg": "field required"}]}, "")): # no string - relay = ModelEndpoint(BINDING) - relay._note((400, json.dumps(body).encode(), {})) - assert relay.facts()["failure"] == kept, body - relay = ModelEndpoint(BINDING) - relay._note((400, b'{"detail": "Model not found"}', {})) - assert HarborAgent.session_verdict(exited, relay.facts())["failure_cause"] == "model_rejected" - - -def test_a_model_reply_nested_past_the_parsers_depth_is_read_as_one_that_is_no_json(): - """A reply body nested past the JSON parser's depth is a reply without a body the endpoint can read, as one that - does not parse: its status counted, no error raised in the call's thread (which would leave it uncounted).""" - from hermit.harbor.tools import ModelEndpoint - relay = ModelEndpoint(BINDING) - relay._note((400, b"[" * 100_000 + b"]" * 100_000, {})) - facts = relay.facts() - assert facts["statuses"] == [400] and facts["rejected"] is True and facts["failure"] == "" - - -@contextlib.contextmanager -def held_model(): - """A model endpoint on 127.0.0.1 that holds each request unanswered until it is released or its caller hangs up: - (its base URL, an event set once a request arrived, one set once its caller hung up, the release).""" - arrived, dropped, release = threading.Event(), threading.Event(), threading.Event() - - class Held(BaseHTTPRequestHandler): - def log_message(self, *a): - pass - - def do_POST(self): - self.rfile.read(int(self.headers["Content-Length"])) - arrived.set() - while not release.wait(0.05): - ready, _, _ = select.select([self.connection], [], [], 0) - if ready and self.connection.recv(1, socket.MSG_PEEK) == b"": - dropped.set() - return - server = ThreadingHTTPServer(("127.0.0.1", 0), Held) - server.daemon_threads = True - threading.Thread(target=server.serve_forever, daemon=True).start() - try: - yield f"http://127.0.0.1:{server.server_address[1]}/v1", arrived, dropped, release - finally: - release.set() - server.shutdown() - server.server_close() - - -def ask_in_background(url: str) -> threading.Thread: - """One chat request to `url` from a thread of its own that keeps its connection until it is answered or closed, - as the answer agent's request thread does once Harbor's timeout stopped the agent.""" - def ask(): - try: - urllib.request.urlopen(urllib.request.Request(url + "/chat/completions", data=json.dumps( - {"model": "glm-5", "messages": [{"role": "user", "content": "What is 6 x 7?"}]}).encode()), timeout=60) - except OSError: - pass - thread = threading.Thread(target=ask, daemon=True) - thread.start() - return thread - - -def test_a_first_request_still_in_flight_when_the_agent_runs_out_of_time_is_the_agents_timeout(monkeypatch): - """The answer agent's endpoint counts a call from its arrival: a first request still in flight when Harbor - stopped the agent at its timeout was made, so the trial is the agent's timeout, scored as Harbor scores it, not - a session that never started (infra, run again); and so it stays once the endpoint has closed.""" - from hermit.harbor.tools import ModelEndpoint - monkeypatch.setenv(KEY_ENV, KEY) - timed_out = errors.classify(trial_result(1.0, exc="AgentTimeoutError")) - with held_model() as (url, arrived, _, _): - relay = ModelEndpoint({**BINDING, "base_url": url}).start() - try: - ask_in_background(relay.url) - assert arrived.wait(10) - facts = relay.facts() - finally: - relay.close() - assert (facts["started"], facts["calls"], facts["statuses"]) == (True, 1, []) - for seen in (facts, relay.facts()): # in flight, then given up at the endpoint's close - verdict = HarborAgent.session_verdict(timed_out, seen) - assert (verdict["status"], verdict["failure_cause"], verdict["reward"]) == \ - ("timeout", timed_out["failure_cause"], 1.0) - - -def test_an_answer_trial_whose_agent_sent_no_request_cost_nothing(tmp_path, fake_harbor): - """The answer agent failed before its request (its instruction file never came, an attachment it could not read): - the relay saw no call, so the trial's usage is known, none, not unknown, which would make the whole eval's cost - unknown.""" - from types import SimpleNamespace - from hermit.agent.model.usage import Usage - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - agent = make_agent(tmp_path, fake_harbor, tasks, acting_agent={"cli": "answer"}) - served = SimpleNamespace(facts=lambda: {"started": False, "calls": 0, "statuses": []}, usage=Usage(model="m")) - facts, usage, _ = agent.session(None, write(tmp_path / "plan.json", "{}"), served, tmp_path) - assert (usage.calls, usage.input_tokens + usage.output_tokens, usage.cost_usd) == (0, 0, 0.0) - # a request that reached the relay and had no reply (its upstream dropped it) may have been billed: unknown - served = SimpleNamespace(facts=lambda: {"started": True, "calls": 1, "statuses": [None]}, usage=Usage(model="m")) - assert agent.session(None, write(tmp_path / "plan.json", "{}"), served, tmp_path)[1] is None - - -def test_a_quota_spent_ends_the_run_even_when_the_cli_retried_it_until_the_agents_time_ran_out(): - """A session the provider never served because the account's quota is spent (a 429 insufficient_quota, which a CLI - such as Pi retries until the agent's time runs out) ends the run (usage_limit, a halt), not the agent's timeout.""" - facts = {"started": True, "served": False, "unavailable": True, "code": runner_module.QUOTA_SPENT} - verdict = errors.classify(trial_result(None, exc="AgentTimeoutError")) - out = HarborAgent.session_verdict(verdict, facts) - assert (out["status"], out["failure_cause"], out["halt"]) == ("infra", "usage_limit", True) - - -def test_closing_the_answer_agents_endpoint_gives_its_calls_in_flight_up_and_leaves_their_cost_unknown(monkeypatch): - """Harbor's timeout stops the answer agent, not its request (its thread keeps the connection; a stream's driver - keeps running): closed after the trial, the endpoint gives each call still in flight up, so the relay drops it - upstream rather than spend on after the trial's usage was read, and waits for it. What the call used is unknown, - and so is the trial's cost (None, never the other calls' alone), which its facts say.""" - from hermit.harbor import tools - monkeypatch.setenv(KEY_ENV, KEY) - with held_model() as (url, arrived, dropped, _): - relay = tools.ModelEndpoint({**BINDING, "base_url": url}).start() - ask_in_background(relay.url) - assert arrived.wait(10) - began = time.time() - relay.close() - took = time.time() - began - assert dropped.wait(10) # the upstream's caller, the relay, hung up - facts, usage = relay.facts(), relay.usage - assert took < tools.CLOSE_WAIT_S and (facts["statuses"], facts["usage_incomplete"]) == ([None], True) - assert (usage.calls, usage.unreported_calls, usage.cost_usd) == (1, 1, None) - - -def test_closing_the_answer_agents_endpoint_cancels_its_sessions_and_waits_for_them_only_so_long(monkeypatch): - """A model a CLI reaches: each request is one session, which the endpoint's close cancels and waits for, so that - what it recorded is in the facts read after the close; a session still running past the wait leaves the usage - incomplete (the cost unknown), and the endpoint is closed all the same.""" - from types import SimpleNamespace - from hermit import isolate - from hermit.agent.base import endpoint as endpoint_mod - from hermit.harbor import tools - cancelled, release = [], threading.Event() - - def sessions(spec, log_dir=None, *, sandbox=None): # the CLI's endpoint, over a stand-in of its agent - served = endpoint_mod.ChatEndpoint(None).start() - served.agent = SimpleNamespace(settings={"model": spec["model"]}) - return served - - def session(agent, request, abort=None, place=None): # each request one stand-in session (cli_answer) - if agent.settings["model"] == "stuck": # a session that does not hear its cancel - release.wait(30) - return endpoint_mod._DROP - cancelled.append(abort.wait(30)) - time.sleep(0.3) # its process stopped, its files cleaned up - return endpoint_mod._DROP # as cli_answer gives a cancelled session - monkeypatch.setattr(endpoint_mod, "cli_endpoint", sessions) - monkeypatch.setattr(endpoint_mod, "cli_answer", session) - monkeypatch.setattr(endpoint_mod, "contract_problems", lambda agent, request: ([], [])) - monkeypatch.setattr(isolate, "cli_sandbox", lambda spec, **kwargs: (None, None)) - try: - for model, wait_s in (("m", 10.0), ("stuck", 0.5)): - endpoint = tools.ModelEndpoint({"cli": "pi", "model": model}).start() - ask_in_background(endpoint.url) - deadline = time.time() + 10 - while not endpoint.facts()["calls"] and time.time() < deadline: - time.sleep(0.05) - monkeypatch.setattr(tools, "CLOSE_WAIT_S", wait_s, raising=False) - began = time.time() - endpoint.close() - took, facts, usage = time.time() - began, endpoint.facts(), endpoint.usage - assert cancelled == [True], model # the first one heard its cancel - assert facts["statuses"] == ([None] if model == "m" else []), model # recorded, or still running - assert (facts["calls"], facts["usage_incomplete"], usage.cost_usd) == (1, True, None), model - assert took < wait_s + 2 and endpoint.endpoint is None, model - finally: - release.set() - - -def test_an_answer_model_a_cli_reaches_on_a_login_with_no_sign_in_is_refused_at_startup(tmp_path, fake_harbor): - """The answer agent's model reached through a CLI on a login (the benchmark file's `model: {cli, login}`): with no - sign-in there (no Codex login, or no Claude token), the runner refuses the run at startup, as it does the acting - CLI's, instead of every trial going unserved (model_provider_failed), run again and then a failed run.""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - spec = {"cli": "codex", "model": "gpt-5.6-luna", "login": "chatgpt", "plan_dir": str(tmp_path / "plan")} - with pytest.raises(ValueError, match="login: chatgpt, but there is no Codex login"): - cli_answer_agent(tmp_path, fake_harbor, tasks, spec) - chatgpt_login(tmp_path) - assert cli_answer_agent(tmp_path, fake_harbor, tasks, spec).model_plan() is not None - - -def test_the_relays_cli_sessions_return_a_refusal_never_wait_for_the_window_inside_the_request(monkeypatch): - """The answer agent's model reached through a CLI (the benchmark file's `model: {cli, ...}`): each request's session - runs with plan_rerun off, whatever the spec says, so that a session the plan refused comes back at once (429, the - relay's `refused`: the trial infra, run again once the plan allows) instead of waiting for the plan's window inside - the request, as main's CliAgent.run does by default, which would end the trial as the agent's timeout.""" - from types import SimpleNamespace - from hermit import isolate - from hermit.agent.base import endpoint as endpoint_mod - from hermit.harbor import tools - specs = [] - - def sessions(spec, log_dir=None, *, sandbox=None): # the CLI's endpoint, over a stand-in of its agent - specs.append(dict(spec)) - served = endpoint_mod.ChatEndpoint(None).start() - served.agent = SimpleNamespace(settings={"model": spec["model"]}) - return served - monkeypatch.setattr(endpoint_mod, "cli_endpoint", sessions) - monkeypatch.setattr(isolate, "cli_sandbox", lambda spec, **kwargs: (None, None)) - for given in ({}, {"plan_rerun": True}): - endpoint = tools.ModelEndpoint({"cli": "codex", "model": "gpt-5.6-luna", "login": "chatgpt", **given}).start() - endpoint.close() - assert [spec["plan_rerun"] for spec in specs] == [False, False] - assert all(spec["model"] == "gpt-5.6-luna" and spec["login"] == "chatgpt" for spec in specs) - - -def test_the_answer_agents_codex_model_has_web_search_off_unless_given(monkeypatch): - """The answer agent's model reached through Codex (`model: {cli: codex, model, login}`, the docs' minimal form): - each session gets web_search off unless the spec sets it, since Codex refuses a session without it (main's - Codex.run), which would make every request the trial's model_rejected.""" - from types import SimpleNamespace - from hermit import isolate - from hermit.agent.base import endpoint as endpoint_mod - from hermit.harbor import tools - specs = [] - - def sessions(spec, log_dir=None, *, sandbox=None): - specs.append(dict(spec)) - served = endpoint_mod.ChatEndpoint(None).start() - served.agent = SimpleNamespace(settings={"model": spec["model"]}) - return served - monkeypatch.setattr(endpoint_mod, "cli_endpoint", sessions) - monkeypatch.setattr(isolate, "cli_sandbox", lambda spec, **kwargs: (None, None)) - for given in ({}, {"web_search": True}): - tools.ModelEndpoint({"cli": "codex", "model": "gpt-5.6-luna", "login": "chatgpt", **given}).start().close() - assert [spec["web_search"] for spec in specs] == [False, True] - - -def test_a_rows_context_is_computed_for_harbors_instruction_when_the_row_gives_none(tmp_path, fake_harbor, - monkeypatch): - """An adapter whose context is per task gets the instruction the agent is given (the task's instruction.md) when - the row carries none (`{"id", "harbor_task"}`), not the row's JSON.""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - staged = [] - monkeypatch.setattr(runner_module, "stage", lambda *args, task, **kwargs: staged.append(task) or {}) - agent = make_agent(tmp_path, fake_harbor, tasks) - agent._stage({"id": "alpha", "harbor_task": "alpha"}, tmp_path / "adapter", tmp_path) - agent._stage({"id": "alpha", "harbor_task": "alpha", "instruction": "the row's"}, tmp_path / "adapter", tmp_path) - assert [t["instruction"] for t in staged] == ["Do alpha.", "the row's"] - - -def test_an_answer_trial_on_a_login_waits_for_its_plan_and_one_whose_request_the_plan_held_runs_again_once_it_allows( - tmp_path, fake_harbor, monkeypatch): - """The answer agent's model reached through a CLI on a login, which the runner serves (model: {cli, login}, the - benchmark file's): each request is one session on the login, and each trial waits for the plan before it starts, - so that its request does not wait the plan out in the agent's time. A trial whose request waited for the plan all - the same (a hold that came once the trial had started) until the agent's time ran out is infra (usage_limit, no - halt), not the agent's timeout, as the relay recorded its wait; it runs again once the plan allows, as a login's - own refusal does, not counted among infra_retries (0 here). (A model the pipeline serves, a run config's own - `{cli}`, reaches the runner as its endpoint's spec alone: neither wait is the runner's.)""" - out, agent, waits = answer_trial_on_a_login(tmp_path / "own", fake_harbor, monkeypatch, held=True) - assert [a["unscored"] for a in out.extra["attempts"]] == ["usage_limit", None] # held, then run again - assert waits == [(agent.model_plan()[0], False)] * 2 # each run waited for the plan before it started - assert (out.output["status"], out.output["failure_cause"]) == ("timeout", "agent_exhausted_budget") - assert "unscored" not in out.extra and not agent._stop.is_set() # the run again's own result: its session ran - - -def test_an_answer_request_that_ran_while_another_sessions_report_set_a_hold_is_the_agents_timeout( - tmp_path, fake_harbor, monkeypatch): - """A request whose session started (the plan allowed it) and was working when another session's report put the - plan past MAX_USED never waited for the plan: a trial that ran out of the agent's time with it unanswered is the - agent's timeout, scored, though the plan holds sessions back when the trial is read.""" - from hermit.agent.base import login - plan = tmp_path / "trial" / "plan" - - def report(): # another session's: its plan at 95%, past MAX_USED 90 - write(plan / "MAX_USED", "90\n") - login.note_usage(plan, None, {"time": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), - "primary": {"used_percent": 95, "resets_at": time.time() + 3600}}) - out, _, _ = answer_trial_on_a_login(tmp_path / "trial", fake_harbor, monkeypatch, held=False, on_start=report) - assert (out.output["status"], out.output["failure_cause"], out.output["reward"]) == \ - ("timeout", "agent_exhausted_budget", 1.0) - row = dataset.index(tmp_path / "trial" / "tasks")[0] - assert "unscored" not in out.extra and judge.score(row, out)["score"] == 1.0 - facts = out.extra["session"] - assert (facts["calls"], facts["statuses"], facts["plan_waited"], "plan_held" in facts) == (1, [None], "", False) - assert login.held(plan).startswith("plan usage 95%") # the plan holds sessions back as it is read - - -def test_an_answer_request_that_waited_for_a_hold_that_ended_before_its_trial_was_read_is_infra( - tmp_path, fake_harbor, monkeypatch): - """A request whose session waited for its login's plan, and was unanswered when the agent's time ran out, is infra - (usage_limit), however the plan stands when the trial is read: here its hold ended as the session waited (which - then started, and was working when the time ran out), or only once Harbor's run had ended (it still waited). The - trial then runs again once the plan allows (its first run's cause in `attempts`).""" - from hermit.agent.base import login - for released in (True, False): - plan, started = tmp_path / f"released-{released}" / "plan", [] - out, _, _ = answer_trial_on_a_login(plan.parent, fake_harbor, monkeypatch, held=True, released=released, - on_start=lambda: started.append(True), - after_harbor=lambda: (plan / "PAUSE").unlink(missing_ok=True)) - assert [a["unscored"] for a in out.extra["attempts"]] == ["usage_limit", None], released # then again - assert started == ([True, True] if released else [True]), released # the run again's session started - assert login.held(plan) == "", released # no hold as the trials are read - - -def planned_sessions(plan: Path, on_start=lambda: None): - """A chat endpoint (started) over Codex sessions on the ChatGPT login whose plan is `plan`, as the pipeline serves a - model a CLI reaches: each request is one session that waits for the plan as Codex's Base Agent does (CliAgent.run), - whose own work is a stand-in: `on_start` once the plan let it start, then it works until it is cancelled (no - reply).""" - from hermit.agent.base.cli import Codex - from hermit.agent.base.endpoint import ChatEndpoint - from hermit.agent.base.interface import Result - - class Session(Codex): - def _session(self, prompt, **kwargs): - on_start() - self.settings["cancel"].wait(30) - return Result(reply="", exit_code=-1, extra={"served": False, "unavailable": False, "refused": False, - "started": True, "cancelled": True}) - return ChatEndpoint(Session({"model": "gpt-5.6-luna", "login": "chatgpt", "plan_dir": str(plan), - "web_search": False})).start() - - -def answer_trial_on_a_login(root: Path, fake_harbor, monkeypatch, *, held: bool, released: bool = False, - on_start=lambda: None, after_harbor=lambda: None): - """One answer trial whose model a CLI reaches on the ChatGPT login (its plan /plan), served by the runner - (model: {cli, login}), that runs out of the agent's time (1 s) with its one request unanswered: (its output, the - runner, and the trial's waits for the plan before it started). The plan allows the trial to start, then holds - sessions back when `held`; with `released` the hold ends as the request waits on it (its session then starts). - The request is one session of planned_sessions; `after_harbor` runs once Harbor's run ended, before the session is - read. A trial run again (usage_limit) waited for the plan's window: the hold is gone when it starts.""" - from hermit import isolate - from hermit.agent.base import endpoint as endpoint_mod - from hermit.agent.base import login - tasks, plan = root / "tasks", root / "plan" - make_task(tasks, "alpha", image="example/alpha:1") - plan.mkdir(parents=True) - real_wait, waits, runner = getattr(login.wait, "__wrapped__", login.wait), [], {} # never an earlier stand-in - - class Ended: # the request's cancel, as it waits on the plan with it: the hold ends then - def __init__(self, cancel): - self.cancel = cancel - - def wait(self, timeout=None): - (plan / "PAUSE").unlink(missing_ok=True) - return self.cancel.is_set() - - def wait(where, cancel=None, log=None, max_s=None, credits=False, **kwargs): - if cancel is not runner["agent"]._stop: # the request's (the relay's, its session's): the plan's own - return real_wait(where, Ended(cancel) if released else cancel, log, max_s, credits, **kwargs) - waits.append((Path(where), credits)) # the trial's, before it starts: the plan allows it - if held and len(waits) == 1: - write(Path(where) / "PAUSE", "held\n") - else: # a run again: it waited for the window, which came - (Path(where) / "PAUSE").unlink(missing_ok=True) - return "" - wait.__wrapped__ = real_wait - monkeypatch.setattr(login, "wait", wait) - monkeypatch.setenv("FAKE_HARBOR_MODE", "timeout") - monkeypatch.setenv("FAKE_ANSWER_WAIT_S", "1") # the agent's time, as the fake's request waits it out - made = [] # each trial's relay its own endpoint, as cli_endpoint makes - - def endpoint(spec, log_dir=None, *, sandbox=None): - made.append(planned_sessions(plan, on_start)) - return made[-1] - chatgpt_login(root) # the sign-in its startup check looks for - try: - monkeypatch.setattr(endpoint_mod, "cli_endpoint", endpoint) - monkeypatch.setattr(isolate, "cli_sandbox", lambda spec, **kwargs: (None, None)) - agent = cli_answer_agent(root, fake_harbor, tasks, - {"cli": "codex", "model": "gpt-5.6-luna", "login": "chatgpt", "plan_dir": str(plan)}) - runner["agent"] = agent - ran = agent.harbor_run - agent.harbor_run = lambda *a, **k: (ran(*a, **k), after_harbor())[0] - return agent.run(dataset.index(tasks)[0], None), agent, waits - finally: - for sessions in made: - sessions.close() - - -def test_the_adapters_tools_are_served_from_this_machine(tmp_path): - """An adapter's agent tools for a CLI in a container: a Session in the strict adapter sandbox behind a tool server - on the address the container reaches (here the loopback), each call in the audit log; a stream's next step swaps - the adapter behind the same URL.""" - from hermit.harbor.tools import AdapterTools - - def call(url, x): - request = urllib.request.Request(f"{url.removesuffix('/mcp')}/call/calc", data=json.dumps({"x": x}).encode(), - headers={"Content-Type": "application/json"}) - return json.loads(urllib.request.urlopen(request, timeout=60).read()) - - served = AdapterTools(tool_adapter(tmp_path / "one"), audit_log=tmp_path / "a" / "tool-calls.jsonl", - host="127.0.0.1", enforce="best_effort").start() - try: - assert served.names == ["calc"] and served.mcp_url.startswith("http://127.0.0.1:") - assert call(served.mcp_url, 4) == {"result": {"result": 8}} - url = served.mcp_url - served.swap(tool_adapter(tmp_path / "two", factor=3), tmp_path / "b" / "tool-calls.jsonl") - assert served.mcp_url == url and call(url, 4) == {"result": {"result": 12}} - finally: - served.close() - first = [json.loads(line) for line in (tmp_path / "a" / "tool-calls.jsonl").read_text().splitlines()] - second = [json.loads(line) for line in (tmp_path / "b" / "tool-calls.jsonl").read_text().splitlines()] - assert [(r["tool"], r["arguments"]) for r in first] == [("calc", {"x": 4})] and len(second) == 1 - with pytest.raises(RuntimeError, match="closed"): - served.swap(tmp_path / "one", tmp_path / "c.jsonl") - - -def test_a_trial_with_tools_names_their_server_in_the_cli_home(tmp_path, fake_harbor, monkeypatch): - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - agent = make_agent(tmp_path, fake_harbor, tasks, tools_host="127.0.0.1", sandbox_enforce="best_effort") - out = agent.run(row, tool_adapter(tmp_path / "adapter")) - assert out.output["status"] == "ok" and out.extra["deployed"]["tools"] == ["calc"] - plan = json.loads((Path(out.extra["harbor_job"]) / "launch" / "plan.json").read_text()) - config = (Path(plan["home"]["host_dir"]) / "config.toml").read_text() - assert "calcdemo" in config and "http://127.0.0.1:" in config and plan["metadata"]["tools"] == ["calc"] - # A server that could not start: no trial, and not the agent's result - from hermit.harbor import tools - monkeypatch.setattr(tools.AdapterTools, "start", lambda self: (_ for _ in ()).throw(RuntimeError("no gateway"))) - failed = agent.run(row, tool_adapter(tmp_path / "adapter")) - assert failed.output["failure_cause"] == "tools_failed" and failed.extra["unscored"] == "tools_failed" - - -# ---- the runner: one stream step through a fake driver --------------------------------------------------------------- - -FAKE_DRIVER = textwrap.dedent('''\ - #!/usr/bin/env python3 - """A stand-in for harborside.driver: the protocol, no Harbor. Steps succeed; step "s2" aborts the stream. Each step - leaves the CLI's stream as harborside would, from the step's launch plan; FAKE_STEP_VERIFIER makes each other - step's verifier one whose tools never arrived (no_tools) or one that ran out of its time (timeout). The step - FAKE_STEP_FAILS names (`:`) fails with that exception before its agent ran (no stream, no - verifier), which aborts the stream; the step FAKE_STEP_HANGS names never answers (a driver hung in a step).""" - import argparse, json, os, socket, sys, threading, time - from pathlib import Path - parser = argparse.ArgumentParser() - parser.add_argument("-m"); parser.add_argument("--task-dir"); parser.add_argument("--stream-dir") - parser.add_argument("--name"); parser.add_argument("--socket"); parser.add_argument("--agent-json") - parser.add_argument("--environment-json"); parser.add_argument("--verifier-json"); parser.add_argument("--idle-s") - parser.add_argument("--trial-json") - args, _ = parser.parse_known_args() - stream_dir, sock_path = Path(args.stream_dir), args.socket - agent = json.loads(args.agent_json) - state = {"next": 1, "finished": False, "aborted": None} - (stream_dir / "seen-agent.json").write_text(json.dumps(agent)) - (stream_dir / "seen-driver.json").write_text(json.dumps({ - "trial": json.loads(args.trial_json) if args.trial_json else None, - "token": "CLAUDE_CODE_OAUTH_TOKEN" in os.environ})) - verifier_mode = os.environ.get("FAKE_STEP_VERIFIER", "") - verifier_out = ("failed to download https://github.com/astral-sh/uv/releases/download/0.9.5/uv.tar.gz\\n" - "/tests/test.sh: line 18: uvx: command not found\\n" if verifier_mode == "no_tools" else "PASS\\n") - srv = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) - srv.bind(sock_path); srv.listen(8) - (stream_dir / "driver.json").write_text(json.dumps({"pid": os.getpid(), "socket": sock_path, "name": args.name, - "trial_dir": str(stream_dir / "trials" / args.name)})) - hangs, (fails, failure) = os.environ.get("FAKE_STEP_HANGS"), os.environ.get("FAKE_STEP_FAILS", ":").split(":", 1) - def status(): - return {"ok": True, "ready": True, "error": None, "next_index": state["next"], "running": None, - "finished": state["finished"], "aborted": state["aborted"]} - while not state["finished"]: - conn, _ = srv.accept() - req = json.loads(conn.makefile().readline() or "{}") - if req["op"] == "status": - reply = status() - elif req["op"] == "stop": - state["finished"] = True; reply = {"ok": True, "stopped": True} - # as harborside.driver ends (_mark_ending): its record says so before the answer, its socket open a moment - record = json.loads((stream_dir / "driver.json").read_text()); record["ending"] = True - (stream_dir / "driver.json.part").write_text(json.dumps(record)) - os.replace(stream_dir / "driver.json.part", stream_dir / "driver.json") - elif req["op"] == "step": - if req["index"] != state["next"]: - reply = {"ok": False, "out_of_order": True, "next_index": state["next"]} - else: - if req["name"] == hangs: - time.sleep(600) - failed = failure if req["name"] == fails else None - step_dir = stream_dir / "trials" / args.name / "steps" / req["name"] - (step_dir / "agent").mkdir(parents=True, exist_ok=True); (step_dir / "verifier").mkdir(exist_ok=True) - if not failed: - (step_dir / "verifier" / "test-stdout.txt").write_text(verifier_out) - plan_path = (req.get("agent_kwargs") or {}).get("plan") - (step_dir / "agent" / "plan-seen.txt").write_text(str(plan_path)) - plan = json.loads(Path(plan_path).read_text()) - if not failed: # one that failed before its agent ran left no stream - (step_dir / "agent" / plan["stream"]).write_text( - '{"type":"thread.started","thread_id":"th-%s"}\\n{"type":"turn.completed","usage":' - '{"input_tokens":10,"cached_input_tokens":0,"output_tokens":2}}\\n' % req["name"]) - aborted = req["name"] == "s2" or bool(failed) - why = ({"exception_type": failed, "exception_message": f"{failed} in step {req['name']}"} if failed - else {"exception_type": "RuntimeError", "exception_message": "boom"}) - reward = 0.0 if verifier_mode == "no_tools" else 1.0 - result = {"step_name": req["name"], "agent_result": {"n_input_tokens": 10, "n_output_tokens": 2, "metadata": {}}, - "verifier_result": None if aborted else {"rewards": {"reward": reward}}, - "exception_info": why if aborted else None, - "agent_execution": {"started_at": "2026-01-01T00:00:00Z", "finished_at": "2026-01-01T00:00:05Z"}} - if verifier_mode == "timeout" and not aborted: - result.update(verifier_result=None, exception_info={"exception_type": "VerifierTimeoutError", - "exception_message": "verifier timed out"}) - state["next"] = req["index"] + 1 - if aborted: state["aborted"] = f"step {req['name']} failed" - reply = {"ok": True, "index": req["index"], "step": result, "aborted": state["aborted"], - "paths": {"trial_dir": str(stream_dir / "trials" / args.name), "step_dir": str(step_dir)}, - "last": req["index"] >= 3} - else: - reply = {"ok": False, "error": "unknown op"} - conn.sendall((json.dumps(reply) + "\\n").encode()); conn.close() - time.sleep(0.5) # as harborside.driver's last answers: its socket there until it exits - os.unlink(sock_path) - ''') - - -def test_stream_steps_through_a_fake_driver(tmp_path, fake_harbor, monkeypatch): - bin_dir = fake_harbor.parent - (bin_dir / "python").unlink() - write(bin_dir / "python", FAKE_DRIVER, exe=True) # the "interpreter" the driver is started with - monkeypatch.setattr(stream, "POLL_S", 0.1) - tasks = tmp_path / "tasks" - make_task(tasks, "flow", steps=("s1", "s2", "s3")) - rows = {r["id"]: r for r in dataset.index(tasks)} - agent = make_agent(tmp_path, fake_harbor, tasks, stream={"idle_s": 60, "start_timeout_s": 30, "turn_wait_s": 6}) - try: - first = agent.run(rows["flow:s1"], simple_template()) - assert first.output["status"] == "ok" and first.output["reward"] == 1.0 and first.output["step"] == "s1" - assert first.output["trial"] == stream_name(agent) and first.tokens == 12 and first.wall_s == 5.0 - stream_dir = tmp_path / "jobs" / "streams" / stream_name(agent) - seen = json.loads((stream_dir / "seen-agent.json").read_text()) - [step] = step_dirs(stream_dir, 1, "s1") - step_plan = step / "launch" / "plan.json" - assert seen["kwargs"]["plan"] == str(step_plan) and "model_name" not in seen - work = json.loads(step_plan.read_text())["deploy"]["work_dir"] - assert work == str((step / "deploy").resolve()) - assert (Path(first.log) / "verifier" / "test-stdout.txt").read_text() == "PASS\n" - assert (Path(first.log) / "agent" / "plan-seen.txt").read_text() == str(step_plan) - client = stream.StreamClient(stream_dir, python=str(bin_dir / "python")) - assert client.running() is not None - - def stopped(): - deadline = time.time() + 10 - while client.running() is not None and time.time() < deadline: - time.sleep(0.2) - return client.running() is None - # Out of order: step 3 while no rollout runs step 2, whose turn it would wait for, ends at once; the stream's - # driver, of no use to any step after it, is stopped. - skipped = agent.run(rows["flow:s3"], None) - assert skipped.output["status"] == "infra" and skipped.output["failure_cause"] == "stream_order" - assert "in order" in skipped.extra["exception_message"] and stopped() - # Run again from step 1 (a new driver), step 2 fails: the stream aborts and the driver is stopped. - assert agent.run(rows["flow:s1"], None).output["status"] == "ok" - second = agent.run(rows["flow:s2"], None) - assert second.output["status"] == "agent_error" and second.extra["aborted"] is True - assert "RuntimeError" in (Path(second.log) / "exception.txt").read_text() - assert stopped() - # A sandboxed worker may not start a driver. - monkeypatch.setenv("HA_ADAPTER_SANDBOX", "1") - blocked = agent.run(rows["flow:s1"], None) - assert blocked.output["failure_cause"] == "stream_lost" and \ - "inside the task sandbox" in blocked.extra["exception_message"] - finally: - stop_streams(tmp_path) - - -def fake_driver(fake_harbor: Path) -> None: - """FAKE_DRIVER as the interpreter beside `harbor` (the one a stream's driver is started with).""" - (fake_harbor.parent / "python").unlink() - write(fake_harbor.parent / "python", FAKE_DRIVER, exe=True) - - -def stop_streams(tmp_path: Path) -> None: - """Every fake driver of the test's streams stopped (one a step left running).""" - for stream_dir in (tmp_path / "jobs" / "streams").glob("*"): - stream.StreamClient(stream_dir, python="unused", log=lambda m: None).stop(timeout_s=10) - - -@pytest.fixture -def fake_tools(monkeypatch): - """The adapter's tools as a stream serves them, without a server or a sandbox (the surface of - hermit.harbor.tools.AdapterTools): each one made, with the adapter and the log it serves, its swaps, its close.""" - made = [] - - class Tools: - def __init__(self, adapter, *, audit_log, host=None, enforce=None): - self.adapter, self.audit_log, self.host = Path(adapter), Path(audit_log), host - self.mcp_url, self.names = "http://127.0.0.1:9/" + "ab" * 16 + "/mcp", ["calc"] - self.swaps, self.closed = [], False - made.append(self) - - def start(self): - return self - - def swap(self, adapter, audit_log): - self.swaps.append((Path(adapter), Path(audit_log))) - self.adapter, self.audit_log = Path(adapter), Path(audit_log) - - def close(self): - self.closed = True - from hermit.harbor import tools - monkeypatch.setattr(tools, "AdapterTools", Tools) - return made - - -def test_a_stream_steps_verifier_verdicts_are_a_trials(tmp_path, fake_harbor, monkeypatch): - """A step's result is judged as a trial's: a verifier whose tools never arrived is infra (run again), its quoted - output with `verifier_redact` applied, and a verifier that ran out of its time on the agent's solution is the - agent's result (`verifier_timeout`'s default).""" - fake_driver(fake_harbor) - tasks = tmp_path / "tasks" - make_task(tasks, "unreached", steps=("s1", "s2")) - make_task(tasks, "slow", steps=("s1", "s2")) - rows = {r["id"]: r for r in dataset.index(tasks)} - agent = make_agent(tmp_path, fake_harbor, tasks, verifier_redact=[r"releases/download/\S+"], - stream={"idle_s": 60, "start_timeout_s": 30, "turn_wait_s": 6}) - try: - monkeypatch.setenv("FAKE_STEP_VERIFIER", "no_tools") # each stream's driver: the mode it started with - unreached = agent.run(rows["unreached:s1"], None) - monkeypatch.setenv("FAKE_STEP_VERIFIER", "timeout") - slow = agent.run(rows["slow:s1"], None) - finally: - stop_streams(tmp_path) - assert (unreached.output["status"], unreached.output["failure_cause"]) == ("infra", "verifier_failed") - assert unreached.extra["unscored"] == "verifier_failed" - assert "uvx: command not found" in unreached.extra["exception_message"] - assert "releases/download" not in unreached.extra["exception_message"] - assert f"github.com/astral-sh/uv/{curate.REDACTED}" in unreached.extra["exception_message"] - assert (slow.output["status"], slow.output["failure_cause"]) == ("timeout", "solution_timed_out") - - -def test_a_streams_driver_runs_in_harbors_environment_with_the_timeout_multipliers(tmp_path, fake_harbor, - monkeypatch): - """A stream's driver is Harbor's process for the stream: it runs with the environment a trial's harbor gets (no - subscription token outside a Claude login) and gets the job's timeout multipliers for its trial config.""" - fake_driver(fake_harbor) - monkeypatch.setenv("CLAUDE_CODE_OAUTH_TOKEN", TOKEN) - tasks = tmp_path / "tasks" - make_task(tasks, "flow", steps=("s1", "s2")) - rows = {r["id"]: r for r in dataset.index(tasks)} - agent = make_agent(tmp_path, fake_harbor, tasks, verifier_timeout_multiplier=2, - stream={"idle_s": 60, "start_timeout_s": 30, "turn_wait_s": 6}) - try: - assert agent.run(rows["flow:s1"], None).output["status"] == "ok" - finally: - stop_streams(tmp_path) - seen = json.loads((tmp_path / "jobs" / "streams" / stream_name(agent) / "seen-driver.json").read_text()) - assert seen == {"trial": {"verifier_timeout_multiplier": 2.0, "environment_build_timeout_multiplier": 1.0}, - "token": False} - - -def test_a_streams_settings_given_as_null_take_their_defaults(tmp_path, fake_harbor, monkeypatch): - """stream.idle_s, start_timeout_s and turn_wait_s left empty (null) are their defaults (the driver stops after - 4 h without a step, its environment has an hour to start), not an error in every step of the task.""" - fake_driver(fake_harbor) - started = [] - ensure = stream.StreamClient.ensure - monkeypatch.setattr(stream.StreamClient, "ensure", lambda self, **kw: started.append(kw) or ensure(self, **kw)) - tasks = tmp_path / "tasks" - make_task(tasks, "flow", steps=("one", "two")) - rows = {r["id"]: r for r in dataset.index(tasks)} - agent = make_agent(tmp_path, fake_harbor, tasks, - stream={"idle_s": None, "start_timeout_s": None, "turn_wait_s": None}) - try: - assert [agent.run(rows[f"flow:{s}"], None).output["status"] for s in ("one", "two")] == ["ok", "ok"] - finally: - stop_streams(tmp_path) - assert [(kw["idle_s"], kw["start_timeout_s"]) for kw in started] == [(14400.0, 3600.0)] * 2 - - -def test_a_step_whose_driver_is_gone_is_infra_and_closes_the_streams_tools(tmp_path, fake_harbor, monkeypatch, - fake_tools): - """A driver killed while a step waits for its turn (its socket refuses): the step is infra (stream_lost), not a - pipeline error scored 0, and the stream's tool server is closed. A driver only silent past a request's time is - infra too; that step was never asked for, so its stream is given up: once none of its rollouts runs (none here), - its driver is stopped and its tools closed, no later step's turn being able to come.""" - fake_driver(fake_harbor) - tasks = tmp_path / "tasks" - make_task(tasks, "flow", steps=("s1", "s2", "s3")) - rows = {r["id"]: r for r in dataset.index(tasks)} - settings = {"tools_host": "127.0.0.1", "stream": {"idle_s": 60, "start_timeout_s": 30, "turn_wait_s": 6}} - agent, other = (make_agent(tmp_path, fake_harbor, tasks, **settings) for _ in range(2)) # a stream each - adapter = tool_adapter(tmp_path / "adapter") - try: - assert agent.run(rows["flow:s1"], adapter).output["status"] == "ok" - - def silent(self, index, *, timeout_s, missing=None): - raise TimeoutError("timed out") - with monkeypatch.context() as patched: - patched.setattr(stream.StreamClient, "wait_turn", silent) - slow = agent.run(rows["flow:s2"], adapter) - client = stream.StreamClient(tmp_path / "jobs" / "streams" / stream_name(agent), python="unused") - given_up = stream_name(agent) not in agent._streams and fake_tools[0].closed and client.running() is None - assert other.run(rows["flow:s1"], adapter).output["status"] == "ok" - - def gone(self, index, *, timeout_s, missing=None): - raise ConnectionRefusedError(111, "Connection refused") - with monkeypatch.context() as patched: - patched.setattr(stream.StreamClient, "wait_turn", gone) - lost = other.run(rows["flow:s2"], adapter) - finally: - stop_streams(tmp_path) - assert (slow.output["status"], slow.output["failure_cause"]) == ("infra", "stream_lost") and given_up - assert (lost.output["status"], lost.output["failure_cause"], lost.extra["unscored"]) == \ - ("infra", "stream_lost", "stream_lost") - assert len(fake_tools) == 2 and all(served.closed for served in fake_tools) - assert stream_name(other) not in other._streams - - -def test_a_driver_that_died_while_it_answered_closes_the_streams_tools(tmp_path, fake_harbor, monkeypatch, fake_tools): - """A driver killed while it answered closed the connection without an answer (a StreamError) and is gone: no step - after it reaches it, so the stream's tool server is closed, as for a driver whose socket refuses. A driver still - running that reports its environment failed has its step never asked for either: its stream is given up, its - driver stopped and its tools closed once none of its rollouts runs.""" - import signal - import time - fake_driver(fake_harbor) - tasks = tmp_path / "tasks" - make_task(tasks, "flow", steps=("s1", "s2", "s3")) - rows = {r["id"]: r for r in dataset.index(tasks)} - settings = {"tools_host": "127.0.0.1", "stream": {"idle_s": 60, "start_timeout_s": 30, "turn_wait_s": 6}} - agent, other = (make_agent(tmp_path, fake_harbor, tasks, **settings) for _ in range(2)) # a stream each - adapter = tool_adapter(tmp_path / "adapter") - try: - assert agent.run(rows["flow:s1"], adapter).output["status"] == "ok" - - def failed(self, index, *, timeout_s, missing=None): - raise stream.StreamError("the stream's environment failed: a step's container exited") - with monkeypatch.context() as patched: - patched.setattr(stream.StreamClient, "wait_turn", failed) - running = agent.run(rows["flow:s2"], adapter) - client = stream.StreamClient(tmp_path / "jobs" / "streams" / stream_name(agent), python="unused") - given_up = stream_name(agent) not in agent._streams and fake_tools[0].closed and client.running() is None - assert other.run(rows["flow:s1"], adapter).output["status"] == "ok" - - def dies(self, index, *, timeout_s, missing=None): # killed while it answered (its zombie is not reaped) - pid = int(self.info()["pid"]) - os.kill(pid, signal.SIGKILL) - state, deadline = Path(f"/proc/{pid}/stat"), time.time() + 10 - while time.time() < deadline and state.exists() and " Z " not in state.read_text(): - time.sleep(0.05) - raise stream.StreamError("the stream driver closed the connection without an answer") - with monkeypatch.context() as patched: - patched.setattr(stream.StreamClient, "wait_turn", dies) - lost = other.run(rows["flow:s2"], adapter) - finally: - stop_streams(tmp_path) - assert (running.output["status"], running.output["failure_cause"]) == ("infra", "stream_lost") and given_up - assert (lost.output["status"], lost.output["failure_cause"]) == ("infra", "stream_lost") - assert len(fake_tools) == 2 and all(served.closed for served in fake_tools) - assert stream_name(other) not in other._streams - - -def test_a_step_lost_before_its_driver_was_asked_for_it_costs_0_and_one_lost_after_keeps_its_cost_unknown( - tmp_path, fake_harbor, monkeypatch): - """A stream step lost before its driver was asked for it ran no session: it costs 0, as a rollout that never reached - a container does. One lost after its request was sent may have run its session, whose usage no one read: its cost - and tokens are unknown, never 0.""" - fake_driver(fake_harbor) - monkeypatch.setattr(stream, "POLL_S", 0.1) - tasks = tmp_path / "tasks" - make_task(tasks, "flow", steps=("s1", "s2", "s3")) - rows = {r["id"]: r for r in dataset.index(tasks)} - settings = {"idle_s": 60, "start_timeout_s": 30, "turn_wait_s": 6} - # two runners, each its own stream of the task: the first's is given up as its step 2 is lost before its driver - agent, other = (make_agent(tmp_path, fake_harbor, tasks, stream=settings) for _ in range(2)) - try: - assert agent.run(rows["flow:s1"], simple_template()).output["status"] == "ok" - - def unreached(self, index, *, timeout_s, missing=None): - raise stream.StreamError("the stream's environment failed: a step's container exited") - with monkeypatch.context() as patched: - patched.setattr(stream.StreamClient, "wait_turn", unreached) - before = agent.run(rows["flow:s2"], simple_template()) - assert other.run(rows["flow:s1"], simple_template()).output["status"] == "ok" - - def unanswered(self, **request): # sent, and the connection closed without an answer - raise stream.StreamError("the stream driver closed the connection without an answer") - with monkeypatch.context() as patched: - patched.setattr(stream.StreamClient, "step", unanswered) - after = other.run(rows["flow:s2"], simple_template()) - finally: - stop_streams(tmp_path) - assert (before.output["failure_cause"], before.cost_usd, before.tokens) == ("stream_lost", 0.0, 0) - assert (after.output["failure_cause"], after.cost_usd, after.tokens) == ("stream_lost", None, None) - - -def test_a_driver_killed_while_it_wrote_its_answer_is_a_stream_error(tmp_path): - """A driver killed while it wrote its answer leaves part of one: a StreamError, as for a driver that closed the - connection without an answer (the runner's lost-driver path: the step is infra, stream_lost), never a ValueError - that fails the rollout.""" - path = tmp_path / "d.sock" - listener = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) - listener.bind(str(path)) - listener.listen(1) - - def answer(): # its answer cut off where the driver died - conn, _ = listener.accept() - with conn: - conn.makefile().readline() - conn.sendall(b'{"ok": true, "index": 1, "step": {"verifier_result": {"rewards"') - thread = threading.Thread(target=answer, daemon=True) - thread.start() - write(tmp_path / "stream" / stream.DRIVER_INFO, json.dumps({"pid": os.getpid(), "socket": str(path)})) - try: - with pytest.raises(stream.StreamError, match="before its answer was whole"): - stream.StreamClient(tmp_path / "stream", python="unused").request({"op": "status"}, timeout=10) - finally: - thread.join(10) - listener.close() - - -def test_each_stream_step_serves_its_own_adapter_once_its_turn_comes(tmp_path, fake_harbor, monkeypatch, fake_tools): - """A later step's rollout may start the stream's tool server before the step before it has run (that step's rollout - begun first, as a batch's rows are submitted in order, but slower to stage): each step, once its turn comes, serves - its own adapter behind the stream's URL and logs its calls in its own log (each rollout of a step has a directory - of its own).""" - fake_driver(fake_harbor) - monkeypatch.setattr(stream, "POLL_S", 0.1) - tasks = tmp_path / "tasks" - make_task(tasks, "flow", steps=("s1", "s2", "s3")) - rows = {r["id"]: r for r in dataset.index(tasks)} - agent = make_agent(tmp_path, fake_harbor, tasks, tools_host="127.0.0.1", - stream={"idle_s": 60, "start_timeout_s": 30, "turn_wait_s": 120}) - one, two = tool_adapter(tmp_path / "one"), tool_adapter(tmp_path / "two", factor=3) - real_stage, staging, held = HarborAgent._stage, threading.Event(), threading.Event() - - def stage(self, task, adapter, where): # step 1's staging held until step 2 served the tools - if task["step"] == "s1": - staging.set() - held.wait(120) - return real_stage(self, task, adapter, where) - monkeypatch.setattr(HarborAgent, "_stage", stage) - earlier, later = {}, {} - first = threading.Thread(target=lambda: earlier.update(out=agent.run(rows["flow:s1"], one))) - second = threading.Thread(target=lambda: later.update(out=agent.run(rows["flow:s2"], two))) - first.start() - try: - staging.wait(120) - second.start() - deadline = time.time() + 120 - while not fake_tools and time.time() < deadline: # step 2's rollout started the stream's server - time.sleep(0.05) - held.set() - first.join(180) - second.join(180) - finally: - held.set() - stop_streams(tmp_path) - stream_dir = tmp_path / "jobs" / "streams" / stream_name(agent) - [first_step], [second_step] = step_dirs(stream_dir, 1, "s1"), step_dirs(stream_dir, 2, "s2") - [served] = fake_tools - assert served.swaps == [(one, first_step / curate.TOOL_AUDIT), (two, second_step / curate.TOOL_AUDIT)] - assert earlier["out"].output["status"] == "ok" and later["out"].output["step"] == "s2" - - -def driver_starts(monkeypatch) -> list[str]: - """The stream drivers started from now on (StreamClient._start), by their stream's name.""" - started, start = [], stream.StreamClient._start - - def counted(self, **kwargs): - started.append(self.dir.name) - return start(self, **kwargs) - monkeypatch.setattr(stream.StreamClient, "_start", counted) - return started - - -def test_a_streams_last_step_that_never_reached_its_driver_stops_it_and_its_next_pass_starts_anew( - tmp_path, fake_harbor, monkeypatch): - """A stream's last step that ends before it reaches the driver (this machine could not stage its adapter) gives - the stream up too, though no step comes after it: its rollout stops the driver as it ends, so that the stream's - next pass in the same runner (an eval's next repeat, a later epoch) runs from its first step in a new environment. - Before, the driver stayed up, waiting for that step: the next pass's first step ended stream_order (already run), - and its last step ran on the environment the pass before had left, scored as the agent's result.""" - fake_driver(fake_harbor) - monkeypatch.setattr(stream, "POLL_S", 0.1) - started = driver_starts(monkeypatch) - tasks = tmp_path / "tasks" - make_task(tasks, "flow", steps=("one", "two")) - rows = {r["id"]: r for r in dataset.index(tasks)} - agent = make_agent(tmp_path, fake_harbor, tasks, stream={"idle_s": 60, "start_timeout_s": 30, "turn_wait_s": 60}) - real_stage, failing = HarborAgent._stage, {"two"} - - def stage(self, task, adapter, where): - if task["step"] in failing: - raise OSError(28, "No space left on device") - return real_stage(self, task, adapter, where) - monkeypatch.setattr(HarborAgent, "_stage", stage) - client = stream.StreamClient(tmp_path / "jobs" / "streams" / stream_name(agent), python="unused") - try: - first = [agent.run(rows[f"flow:{step}"], None) for step in ("one", "two")] - deadline = time.time() + 10 - while client.running() is not None and time.time() < deadline: - time.sleep(0.1) - stopped = client.running() is None - failing.clear() - again = [agent.run(rows[f"flow:{step}"], None) for step in ("one", "two")] - finally: - stop_streams(tmp_path) - assert [(out.output["status"], out.output["failure_cause"]) for out in first] == \ - [("ok", None), ("infra", "staging_failed")] - assert stopped # as its last step's rollout ended - assert [(out.output["status"], out.output["failure_cause"]) for out in again] == [("ok", None), ("ok", None)] - assert started == [stream_name(agent)] * 2 # the next pass in a new driver - - -@pytest.mark.parametrize("end", ["deadline", "swap", "config", "sign_in", "plan"]) -def test_a_streams_last_step_that_ends_in_any_way_before_its_driver_is_asked_gives_the_stream_up( - tmp_path, fake_harbor, monkeypatch, fake_tools, end): - """Every way a stream's step can end before its driver is asked for it gives the stream up, not only a staging - that failed: its turn's wait running out (a plain StreamOrderError), the adapter's tools failing to swap in at its - turn, an exception of its launch's (a ValueError of agent_config), its turn's copy of the ChatGPT sign-in failing - (staging_failed), its launch plan refused for its adapter (adapter_failed). The driver, waiting for that step, is - stopped as the last rollout ends, and the stream's next pass runs from its first step in a new driver. Before, each - of these left the driver up for stream.idle_s, and the next pass's steps met it (already run, or scored on the old - environment).""" - from hermit.harbor import launch as launch_mod - fake_driver(fake_harbor) - monkeypatch.setattr(stream, "POLL_S", 0.1) - started = driver_starts(monkeypatch) - tasks = tmp_path / "tasks" - make_task(tasks, "flow", steps=("one", "two")) - rows = {r["id"]: r for r in dataset.index(tasks)} - login = {"binding": None, "model": "gpt-5.6-luna", "login": "chatgpt"} if end == "sign_in" else {} - if login: - chatgpt_login(tmp_path) - agent = make_agent(tmp_path, fake_harbor, tasks, tools_host="127.0.0.1", - stream={"idle_s": 60, "start_timeout_s": 30, "turn_wait_s": 60}, **login) - adapter = tool_adapter(tmp_path / "adapter") - client = stream.StreamClient(tmp_path / "jobs" / "streams" / stream_name(agent), python="unused") - real_wait, real_config = stream.StreamClient.wait_turn, HarborAgent.agent_config - failing = {"two"} - - def wait_turn(self, index, *, timeout_s, missing=None): - if index == 2 and failing: - raise stream.StreamOrderError(f"waited {timeout_s:.0f} s for step {index}'s turn (the driver is at step 1)") - return real_wait(self, index, timeout_s=timeout_s, missing=missing) - - def agent_config(self, task, *args, **kwargs): - if task["step"] in failing: - raise ValueError("an agent config the launch cannot take") - return real_config(self, task, *args, **kwargs) - real_sign_in, real_launch, signed = launch_mod.sign_in, HarborAgent._launch, [] - - def sign_in(plan, root): - signed.append(plan) - if failing and len(signed) == 2: - raise OSError(28, "No space left on device") - return real_sign_in(plan, root) - - def plan_of(self, task, *args, **kwargs): - if task["step"] in failing: - raise ValueError("dsh takes an MCP server name of 32 characters at most ([A-Za-z0-9_-]), not 'a.b'") - return real_launch(self, task, *args, **kwargs) - if end == "deadline": - monkeypatch.setattr(stream.StreamClient, "wait_turn", wait_turn) - elif end == "config": - monkeypatch.setattr(HarborAgent, "agent_config", agent_config) - elif end == "sign_in": - monkeypatch.setattr(launch_mod, "sign_in", sign_in) - elif end == "plan": - monkeypatch.setattr(HarborAgent, "_launch", plan_of) - try: - first = [agent.run(rows["flow:one"], adapter)] - if end == "swap": - def swap(adapter, audit_log): - raise OSError(28, "No space left on device") - fake_tools[0].swap = swap - if end == "config": - with pytest.raises(ValueError, match="cannot take"): - agent.run(rows["flow:two"], adapter) - else: - first.append(agent.run(rows["flow:two"], adapter)) - deadline = time.time() + 10 - while client.running() is not None and time.time() < deadline: - time.sleep(0.1) - stopped = client.running() is None and fake_tools[0].closed - failing.clear() - again = [agent.run(rows[f"flow:{step}"], adapter) for step in ("one", "two")] - finally: - stop_streams(tmp_path) - causes = {"deadline": ("infra", "stream_order"), "swap": ("infra", "tools_failed"), - "sign_in": ("infra", "staging_failed"), "plan": ("agent_error", "adapter_failed")} - assert [(out.output["status"], out.output["failure_cause"]) for out in first] == \ - [("ok", None), *([causes[end]] if end in causes else [])] - assert stopped # as its last step's rollout ended - assert [(out.output["status"], out.output["failure_cause"]) for out in again] == [("ok", None), ("ok", None)] - assert started == [stream_name(agent)] * 2 # the next pass in a new driver - assert not agent._given_up and not agent._in_flight - - -def test_a_second_rollout_of_a_step_the_stream_already_ran_gives_nothing_up(tmp_path, fake_harbor, monkeypatch): - """A rollout of a step its stream's driver is already past (an Evolver's run_agent of a training step whose stream - went on) ends stream_order and gives nothing up: the stream goes on in the same driver, its next step's turn - intact.""" - fake_driver(fake_harbor) - monkeypatch.setattr(stream, "POLL_S", 0.1) - started = driver_starts(monkeypatch) - tasks = tmp_path / "tasks" - make_task(tasks, "flow", steps=("one", "two", "three")) - rows = {r["id"]: r for r in dataset.index(tasks)} - agent = make_agent(tmp_path, fake_harbor, tasks, stream={"idle_s": 60, "start_timeout_s": 30, "turn_wait_s": 60}) - try: - outs = [agent.run(rows[f"flow:{step}"], None) for step in ("one", "two", "one", "three")] - finally: - stop_streams(tmp_path) - assert [(out.output["status"], out.output["failure_cause"]) for out in outs] == \ - [("ok", None), ("ok", None), ("infra", "stream_order"), ("ok", None)] - assert "already run" in outs[2].extra["exception_message"] - assert started == [stream_name(agent)] # one driver, the stream never given up - - -def test_a_step_after_one_that_never_reached_the_driver_ends_at_once_and_stops_the_stream(tmp_path, fake_harbor, - monkeypatch, fake_tools): - """A step that ends before it reaches its stream's driver (this machine could not stage its adapter) gives the - stream up: its driver and tool server, of use to no later step, are stopped and closed as it ends, and a later step - cannot have its turn: it ends at once, infra (stream_order), starting no driver. Before, each later step held a - rollout worker for its whole turn's wait (by default over an hour, more for each later step), then ended so.""" - fake_driver(fake_harbor) - monkeypatch.setattr(stream, "POLL_S", 0.1) - started = driver_starts(monkeypatch) - tasks = tmp_path / "tasks" - make_task(tasks, "flow", steps=("one", "two", "three")) - rows = {r["id"]: r for r in dataset.index(tasks)} - agent = make_agent(tmp_path, fake_harbor, tasks, tools_host="127.0.0.1", - stream={"idle_s": 60, "start_timeout_s": 30, "turn_wait_s": 60}) - real_stage = HarborAgent._stage - - def stage(self, task, adapter, where): - if task["step"] == "two": - raise OSError(28, "No space left on device") - return real_stage(self, task, adapter, where) - monkeypatch.setattr(HarborAgent, "_stage", stage) - adapter = tool_adapter(tmp_path / "adapter") - client = stream.StreamClient(tmp_path / "jobs" / "streams" / stream_name(agent), python="unused") - try: - first, second = (agent.run(rows[f"flow:{step}"], adapter) for step in ("one", "two")) - t0 = time.time() - third = agent.run(rows["flow:three"], adapter) - took = time.time() - t0 - deadline = time.time() + 10 - while client.running() is not None and time.time() < deadline: - time.sleep(0.1) - stopped = client.running() is None - finally: - stop_streams(tmp_path) - assert (first.output["status"], second.output["failure_cause"]) == ("ok", "staging_failed") - assert (third.output["status"], third.output["failure_cause"], third.extra["unscored"]) == \ - ("infra", "stream_order", "stream_order") - assert "step 3's turn cannot come: the stream has no driver" in third.extra["exception_message"] - assert took < 20 # its turn's wait is 60 s - assert started == [stream_name(agent)] and stopped - [served] = fake_tools - assert served.closed and stream_name(agent) not in agent._streams and stream_name(agent) not in agent._drivers - - -def test_a_later_step_of_a_stream_whose_driver_is_gone_ends_at_once_and_starts_none(tmp_path, fake_harbor, - monkeypatch): - """A later step whose stream has no driver (it idled out after stream.idle_s, or was stopped; or the run was - resumed, whose runner names its streams anew) and no rollout of its first step to start one cannot have its turn: - it ends at once, infra (stream_order), starts no driver, and its curated log holds no tool call. Before, it started - one (a new environment) at the stream's first step and waited its whole turn's wait for a turn that never came.""" - fake_driver(fake_harbor) - monkeypatch.setattr(stream, "POLL_S", 0.1) - started = driver_starts(monkeypatch) - tasks = tmp_path / "tasks" - make_task(tasks, "flow", steps=("one", "two", "three")) - rows = {r["id"]: r for r in dataset.index(tasks)} - settings = {"stream": {"idle_s": 60, "start_timeout_s": 30, "turn_wait_s": 60}} - try: - first = make_agent(tmp_path, fake_harbor, tasks, **settings).run(rows["flow:one"], None) - client = stream.StreamClient(Path(first.extra["stream_dir"]), python="unused") - client.stop(timeout_s=10) # as at its idle time - deadline = time.time() + 10 - while client.running() is not None and time.time() < deadline: - time.sleep(0.1) - resumed = make_agent(tmp_path, fake_harbor, tasks, **settings) - t0 = time.time() - second = resumed.run(rows["flow:two"], None) - took = time.time() - t0 - finally: - stop_streams(tmp_path) - assert first.output["status"] == "ok" and took < 20 # its turn's wait is 60 s - assert (second.output["status"], second.output["failure_cause"]) == ("infra", "stream_order") - assert "the stream has no driver" in second.extra["exception_message"] - assert started == [first.output["trial"]] and not (Path(second.log) / curate.TOOL_AUDIT).exists() - - -def test_a_steps_halt_stops_its_streams_driver_and_ends_the_run(tmp_path, fake_harbor, monkeypatch): - """A halt in a step (an API account's usage limit) stops its stream's driver, keeps the step's log and ends the - run (LoginFailed): no later rollout of the runner starts a step or a driver. Resumed (a new runner, which names its - streams anew), that stream's later steps end at once, infra (stream_order: its first step is in no rollout), and - the stream runs anew from its first step.""" - from hermit.trainer.rollout import run_tasks - fake_driver(fake_harbor) - monkeypatch.setattr(stream, "POLL_S", 0.1) - started = driver_starts(monkeypatch) - tasks = tmp_path / "tasks" - make_task(tasks, "flow", steps=("one", "two", "three")) - rows = {r["id"]: r for r in dataset.index(tasks)} - settings = {"stream": {"idle_s": 60, "start_timeout_s": 30, "turn_wait_s": 60}} - monkeypatch.setenv("FAKE_STEP_FAILS", "two:ApiUsageLimitError") # each driver reads it as it starts - try: - first = make_agent(tmp_path, fake_harbor, tasks, **settings) - done = first.run(rows["flow:one"], None) - with pytest.raises(LoginFailed, match=r"model_provider_failed \(ApiUsageLimitError\)"): - first.run(rows["flow:two"], None) - client = stream.StreamClient(tmp_path / "jobs" / "streams" / stream_name(first), python="unused") - deadline = time.time() + 10 - while client.running() is not None and time.time() < deadline: - time.sleep(0.1) - stopped = client.running() is None - [halted] = step_dirs(client.dir, 2, "two") - with pytest.raises(LoginFailed): - first.run(rows["flow:three"], None) - monkeypatch.delenv("FAKE_STEP_FAILS") - resumed = make_agent(tmp_path, fake_harbor, tasks, **settings) - again = run_tasks([rows[t] for t in ("flow:two", "flow:three")], None, resumed.run, judge.score, - concurrency=2) - anew = resumed.run(rows["flow:one"], None) - finally: - stop_streams(tmp_path) - assert done.output["status"] == "ok" and stopped and (halted / "hermit-log").is_dir() - assert not step_dirs(client.dir, 3, "three") # no rollout after the halt - for record in again: - unscored_record(record, "stream_order") - assert anew.output["status"] == "ok" and started == [stream_name(first), stream_name(resumed)] - - -def test_a_batch_runs_a_streams_steps_in_order_while_the_first_is_still_to_reach_the_driver(tmp_path, fake_harbor, - monkeypatch): - """run_tasks submits a batch's rows in order, all at once: a stream's later steps wait for their turns while the - step before is still in flight (here the first, slower to stage than the later steps' grace), none ends for a turn - taken for lost, and one driver runs them all.""" - from hermit.trainer.rollout import run_tasks - fake_driver(fake_harbor) - monkeypatch.setattr(stream, "POLL_S", 0.1) # the shortest grace: the strictest check - started = driver_starts(monkeypatch) - tasks = tmp_path / "tasks" - make_task(tasks, "flow", steps=("one", "two", "three")) - agent = make_agent(tmp_path, fake_harbor, tasks, stream={"idle_s": 60, "start_timeout_s": 30, "turn_wait_s": 60}) - real_stage = HarborAgent._stage - - def stage(self, task, adapter, where): - if task["step"] == "one": - time.sleep(1.0) - return real_stage(self, task, adapter, where) - monkeypatch.setattr(HarborAgent, "_stage", stage) - try: - records = run_tasks(dataset.index(tasks), None, agent.run, judge.score, concurrency=3) - finally: - stop_streams(tmp_path) - assert [(r["id"], r["status"]) for r in records] == [("flow:one", "ok"), ("flow:two", "ok"), ("flow:three", "ok")] - assert started == [stream_name(agent)] - - -def test_a_steps_launch_plan_this_machine_could_not_write_is_infra_not_the_agents_0(tmp_path, fake_harbor, - monkeypatch): - """A host OSError while a step's launch plan is rendered (a full disk) is this machine's, before the step: infra - (staging_failed), as one while its adapter is staged, and no driver is started. Before, it failed the rollout, - scored 0 as the agent's.""" - fake_driver(fake_harbor) - started = driver_starts(monkeypatch) - tasks = tmp_path / "tasks" - make_task(tasks, "flow", steps=("one", "two")) - row = {r["id"]: r for r in dataset.index(tasks)}["flow:one"] - agent = make_agent(tmp_path, fake_harbor, tasks, stream={"idle_s": 60, "start_timeout_s": 30, "turn_wait_s": 6}) - monkeypatch.setattr(HarborAgent, "_launch", - lambda self, *a, **k: (_ for _ in ()).throw(OSError(28, "No space left on device"))) - try: - out = agent.run(row, None) - finally: - stop_streams(tmp_path) - assert (out.output["status"], out.output["failure_cause"], out.extra["unscored"]) == \ - ("infra", "staging_failed", "staging_failed") - assert "No space left on device" in out.extra["exception_message"] and started == [] - - -@pytest.mark.skipif(not Path("/proc/self/cmdline").is_file(), reason="a driver is told by its command line in /proc") -def test_a_driver_that_answers_no_stop_is_killed_and_its_environment_taken_down(tmp_path, fake_harbor, monkeypatch): - """A driver hung in a step never answers it (the step expires), nor the stop then asked of it, which queues - behind the step; neither its idle stop nor the pipeline's close would end it: it is killed with its process group, - and its trial's compose projects are taken down by name, as an expired trial's are. Before, it stayed up with its - environment.""" - import signal - fake_driver(fake_harbor) - monkeypatch.setattr(stream, "STOP_TIMEOUT_S", 1.0) - downs = [] - monkeypatch.setattr(docker, "compose_down", lambda name, **kwargs: downs.append(name)) - monkeypatch.setenv("FAKE_STEP_HANGS", "one") - tasks = tmp_path / "tasks" - make_task(tasks, "flow", steps=("one", "two")) - row = {r["id"]: r for r in dataset.index(tasks)}["flow:one"] - agent = make_agent(tmp_path, fake_harbor, tasks, wait_budget_s=1, - stream={"idle_s": 60, "start_timeout_s": 30, "turn_wait_s": 6}) - client = stream.StreamClient(tmp_path / "jobs" / "streams" / stream_name(agent), python="unused") - try: - t0 = time.time() - out = agent.run(row, None) - took = time.time() - t0 - gone, socket_left = client.gone(), stream.socket_path(client.dir).exists() - finally: - pid = client._driver_pid() # this test's own driver, were it not killed - if pid: - os.killpg(pid, signal.SIGKILL) - stop_streams(tmp_path) - assert (out.output["status"], out.output["failure_cause"]) == ("infra", "harbor_timeout") - assert gone and not socket_left and downs == [stream_name(agent)] and took < 30 - - -def test_each_repeat_of_a_multi_step_task_runs_its_stream_anew_and_keeps_its_step_logs(tmp_path, fake_harbor, - monkeypatch): - """`hermit eval` runs its repeats one after the other, each over every row: a multi-step task's stream finishes in - one repeat, and the next runs it anew in a driver of its own (its environment new), in the same stream directory, - each rollout of a step in a directory of its own, so that no repeat overwrites another's step logs.""" - from hermit.trainer.rollout import run_tasks - fake_driver(fake_harbor) - started = driver_starts(monkeypatch) - tasks = tmp_path / "tasks" - make_task(tasks, "flow", steps=("one", "two")) # not "s2": the fake driver aborts the stream there - agent = make_agent(tmp_path, fake_harbor, tasks, stream={"idle_s": 60, "start_timeout_s": 30, "turn_wait_s": 6}) - try: - repeats = [run_tasks(dataset.index(tasks), None, agent.run, judge.score, concurrency=1) for _ in (1, 2)] - finally: - stop_streams(tmp_path) - assert [[r["status"] for r in records] for records in repeats] == [["ok", "ok"]] * 2 - assert started == [stream_name(agent)] * 2 - logs = [r["log"] for records in repeats for r in records] - assert len(set(logs)) == 4 and all((Path(log) / "verifier" / "test-stdout.txt").is_file() for log in logs) - - -def test_each_runners_streams_are_its_own_and_a_resumed_run_starts_its_streams_anew(tmp_path, fake_harbor): - """A multi-step task's stream is named for its runner too (HarborAgent.run_id, made as the runner is): two runs of - one task on one jobs directory, at once or one after the other, never share its driver, its environment or its - step logs. A resumed run is another runner: a stream its earlier steps ran in is not met again, so its later step - ends at once (stream_order: its first step is in no rollout).""" - fake_driver(fake_harbor) - tasks = tmp_path / "tasks" - make_task(tasks, "flow", steps=("one", "two")) # not "s2": the fake driver aborts the stream there - rows = {r["id"]: r for r in dataset.index(tasks)} - agents = [make_agent(tmp_path, fake_harbor, tasks, stream={"idle_s": 60, "start_timeout_s": 30, "turn_wait_s": 6}) - for _ in range(3)] - try: - first, apart = (agent.run(rows["flow:one"], None) for agent in agents[:2]) - resumed = agents[2].run(rows["flow:two"], None) - finally: - stop_streams(tmp_path) - assert len({agent.run_id for agent in agents}) == 3 - assert [o.output["status"] for o in (first, apart)] == ["ok", "ok"] - assert [o.output["trial"] for o in (first, apart)] == [stream_name(agents[0]), stream_name(agents[1])] - assert first.log != apart.log and first.extra["stream_dir"] != apart.extra["stream_dir"] - assert (resumed.output["status"], resumed.output["failure_cause"]) == ("infra", "stream_order") - - -def test_a_stream_run_again_after_its_last_step_keeps_its_step_logs_and_one_still_going_refuses_its_steps( - tmp_path, fake_harbor): - """A stream is its runner's. Run again after its last step (a second epoch of a split that lists it whole), it runs - in a new driver and environment in the same stream directory, each step's rollout in a directory of its own (the - first run's step logs kept); run again while its driver waits for its next step (a split that lists it in part), - its steps' turns are taken (stream_order: infra, which the run records as failed rollouts).""" - fake_driver(fake_harbor) - tasks = tmp_path / "tasks" - make_task(tasks, "whole", steps=("one", "two")) # not "s2": the fake driver aborts the stream there - make_task(tasks, "part", steps=("one", "two")) - rows = {r["id"]: r for r in dataset.index(tasks)} - agent = make_agent(tmp_path, fake_harbor, tasks, stream={"idle_s": 60, "start_timeout_s": 30, "turn_wait_s": 6}) - try: - first, second = ([agent.run(rows[t], None) for t in ("whole:one", "whole:two", "part:one")] for _ in (1, 2)) - finally: - stop_streams(tmp_path) - assert [o.output["status"] for o in first] == ["ok"] * 3 and [o.output["status"] for o in second[:2]] == ["ok"] * 2 - assert [o.extra["stream_dir"] for o in second[:2]] == [o.extra["stream_dir"] for o in first[:2]] - assert not {o.log for o in second[:2]} & {o.log for o in first[:2]} - assert all(Path(o.log).is_dir() for o in first + second[:2]) - assert (second[2].output["status"], second[2].output["failure_cause"]) == ("infra", "stream_order") - - -def test_a_stream_run_again_after_its_last_step_reads_none_of_the_first_runs_outputs(tmp_path, fake_harbor, - monkeypatch): - """A stream run again after its last step (a validation task run_agent runs in each call) starts a new driver in - its directory, whose trial Harbor names as the first one's and would reuse step by step: the first run's trial - directory and replies are set aside first (earlier/1), so a step of the second run that leaves no output of its own - (its setup failed before its agent ran) is read with none of the first run's. Before, it was billed the first run's - usage again and curated with its stream.""" - fake_driver(fake_harbor) - tasks = tmp_path / "tasks" - make_task(tasks, "flow", steps=("one", "two")) - rows = {r["id"]: r for r in dataset.index(tasks)} - agent = make_agent(tmp_path, fake_harbor, tasks, stream={"idle_s": 60, "start_timeout_s": 30, "turn_wait_s": 6}) - try: - call = [agent.run(rows[t], None) for t in ("flow:one", "flow:two")] # the stream finished - monkeypatch.setenv("FAKE_STEP_FAILS", "one:RuntimeError") # the next driver's step one - again = agent.run(rows["flow:one"], None) - finally: - stop_streams(tmp_path) - stream_dir, trial = Path(again.extra["stream_dir"]), again.output["trial"] - name = json.loads((step_dirs(stream_dir, 1, "one")[-1] / "launch" / "plan.json").read_text())["stream"] - assert [o.output["status"] for o in call] == ["ok", "ok"] and call[0].tokens == 12 and call[0].cost_usd - assert (again.output["status"], again.tokens, again.cost_usd) == ("agent_error", None, None) - assert not (Path(again.log) / "agent" / name).exists() - assert (stream_dir / "earlier" / "1" / "trials" / trial / "steps" / "one" / "agent" / name).is_file() - - -def test_closing_the_runner_stops_the_drivers_of_its_unfinished_streams(tmp_path, fake_harbor, monkeypatch): - """A runner's streams are met by no later runner (a resumed run names its own anew): its close stops the driver of - each stream it stepped that did not finish, which would otherwise hold its task's environment for stream.idle_s; - so does its finalizer for a runner never closed (hermit.isolate.agent_pipeline's close does not close it): at exit - for one still held then, and as soon as one is freed. Before, a weak set of the runners was closed at exit, and a - runner freed before then (`hermit check`'s probe, `hermit eval` once it returned) left its drivers up. A stream that - finished was stopped by its last step, and is not stopped again; nor is a driver that ended by itself (its idle - time) asked to stop.""" - import gc - fake_driver(fake_harbor) - tasks = tmp_path / "tasks" - make_task(tasks, "flow", steps=("one", "two")) # not "s2": the fake driver aborts the stream there - rows = {r["id"]: r for r in dataset.index(tasks)} - settings = {"stream": {"idle_s": 60, "start_timeout_s": 30, "turn_wait_s": 6}} - stopped = [] # each stop asked of a driver, by its stream's name - real_stop = stream.StreamClient.stop - - timeouts = [] # ... and the time each was given before its kill - - def counted(self, **kwargs): - stopped.append(self.dir.name) - timeouts.append(kwargs.get("timeout_s")) - return real_stop(self, **kwargs) - monkeypatch.setattr(stream.StreamClient, "stop", counted) - - def driver(out, *, stopping=False): - """The live driver of the stream a step ran in, or None; one being stopped is waited for (the driver removes - its socket as it ends).""" - client = stream.StreamClient(Path(out.extra["stream_dir"]), python="unused") - deadline = time.time() + 10 - while stopping and client.running() is not None and time.time() < deadline: - time.sleep(0.1) - return client.running() - try: - agent = make_agent(tmp_path, fake_harbor, tasks, **settings) - done = [agent.run(rows[f"flow:{step}"], None) for step in ("one", "two")] # the stream finished - assert [o.output["status"] for o in done] == ["ok", "ok"] and stopped == [done[1].output["trial"]] - assert done[1].output["trial"] not in agent._drivers # no longer its to stop - agent.close() - assert stopped == [done[1].output["trial"]] - other = make_agent(tmp_path, fake_harbor, tasks, **settings) - left = other.run(rows["flow:one"], None) # this one did not finish - assert driver(left) is not None - other.close() - assert stopped[1:] == [left.output["trial"]] and driver(left, stopping=True) is None - interrupted = make_agent(tmp_path, fake_harbor, tasks, **settings) # never closed, held at exit - cut = interrupted.run(rows["flow:one"], None) - assert interrupted._finalizer.atexit and interrupted._finalizer.alive - interrupted._finalizer() # as at exit - assert stopped[2:] == [cut.output["trial"]] and driver(cut, stopping=True) is None - freed = make_agent(tmp_path, fake_harbor, tasks, **settings) # never closed, and let go - dropped = freed.run(rows["flow:one"], None) - assert driver(dropped) is not None - del freed - gc.collect() - assert stopped[3:] == [dropped.output["trial"]] and driver(dropped, stopping=True) is None - ended = make_agent(tmp_path, fake_harbor, tasks, **settings) - idle = ended.run(rows["flow:one"], None) - real_stop(stream.StreamClient(Path(idle.extra["stream_dir"]), python="unused"), timeout_s=10) # by itself - assert driver(idle, stopping=True) is None # (as at its idle time) - ended.close() - assert stopped[4:] == [] - # the last step's own stop waits as long as a stop does; a close's, its finalizer's, CLOSE_STOP_S each - assert timeouts == [None] + [runner_module.CLOSE_STOP_S] * 3 - finally: - stop_streams(tmp_path) - - -def test_a_harbor_command_that_is_a_link_starts_a_streams_driver_with_its_venvs_python(tmp_path, fake_harbor): - """`harbor` on PATH may be a link into its tool's venv (uv tool's and pipx's ~/.local/bin/harbor): a stream's - driver runs under that venv's interpreter, beside the link's target, not a `python` beside the link.""" - fake_driver(fake_harbor) - tasks = tmp_path / "tasks" - make_task(tasks, "flow", steps=("s1", "s2")) - row = {r["id"]: r for r in dataset.index(tasks)}["flow:s1"] - link = tmp_path / "local" / "bin" / "harbor" - link.parent.mkdir(parents=True) - link.symlink_to(fake_harbor) - agent = make_agent(tmp_path, fake_harbor, tasks, harbor=str(link), - stream={"idle_s": 60, "start_timeout_s": 30, "turn_wait_s": 6}) - assert agent.harbor == str(link) and os.path.samefile(Path(agent.harbor_python).parent, fake_harbor.parent) - try: - out = agent.run(row, None) - finally: - stop_streams(tmp_path) - assert (out.output["status"], out.output["trial"]) == ("ok", stream_name(agent)) - - -def test_socket_path_is_short(tmp_path, monkeypatch): - short = tmp_path / "r" - short.mkdir() - monkeypatch.setenv("XDG_RUNTIME_DIR", str(short)) - path = stream.socket_path(tmp_path / ("x" * 200)) - assert len(str(path)) <= 100 and path.parent.is_dir() - fits = len(str(short / "hermit-harbor-sockets" / path.name).encode()) <= 100 # tmp_path's length varies - assert path.is_relative_to(short) == fits - monkeypatch.setenv("XDG_RUNTIME_DIR", str(tmp_path / ("y" * 120))) # too long: another base is taken - (tmp_path / ("y" * 120)).mkdir() - assert len(str(stream.socket_path(tmp_path))) <= 100 - - -# ---- Claude Code on the subscription ------------------------------------------------------------------------------- - -STREAMS = ROOT / "tests" / "fixtures" / "claude_streams" -TOKEN = "fake-setup-token-never-real" - - -def claude_agent(tmp_path, fake_harbor, tasks, **params) -> HarborAgent: - return make_agent(tmp_path, fake_harbor, tasks, - **{"acting_agent": {"import_path": "harborside.claude:HermitClaude"}, - "model": "claude-haiku-5-5", "binding": None, "effort": "medium", "login": "claude", - "claude_version": "2.1.318", **params}) - - -def test_a_compose_overlay_is_found_beside_the_benchmark_file(tmp_path, fake_harbor): - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - utils = tmp_path / "utils" - overlay = write(utils / "no-core-dumps.yaml", "services:\n main:\n ulimits:\n core: 0\n") - write(utils / "benchmark.yaml", f"tasks_dir: {tasks}\nenvironment:\n env: {{OMP_NUM_THREADS: '8'}}\n" - " extra_docker_compose: [no-core-dumps.yaml]\n") - agent = make_agent(tmp_path, fake_harbor, tasks, benchmark=str(utils / "benchmark.yaml")) - environment = agent.job_config(row, "alpha-1", agent.agent_config(row, None, None))["environment"] - assert environment["extra_docker_compose"] == [str(overlay)] # absolute: Harbor resolves it from its own cwd - assert environment["env"] == {"OMP_NUM_THREADS": "8"} and environment["delete"] is True - - -def test_claude_plan_carries_the_login_rules_and_never_the_token(tmp_path, fake_harbor, monkeypatch): - from hermit.agent.base import cli - monkeypatch.setenv("CLAUDE_CODE_OAUTH_TOKEN", TOKEN) - for name in ("ANTHROPIC_API_KEY", "CLAUDECODE", "CLAUDE_CODE_SESSION_ID"): - monkeypatch.setenv(name, "from-the-parent") - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - binary = write(tmp_path / "pin" / "claude", "#!/bin/sh\necho 2.1.318\n", exe=True) - agent = claude_agent(tmp_path, fake_harbor, tasks, claude_bin=str(binary)) - job = tmp_path / "job" - plan_path = agent._launch(row, job, None, None) - config = agent.job_config(row, "alpha-1", agent.agent_config(row, plan_path, None)) - kwargs = config["agents"][0]["kwargs"] - assert kwargs == {"plan": str(plan_path), "version": "2.1.318", "claude_bin": "/opt/hermit-claude/claude"} - assert {"type": "bind", "source": str(binary.parent), "target": "/opt/hermit-claude", "read_only": True} in \ - config["environment"]["mounts"] - plan = json.loads(plan_path.read_text()) - # The token on a descriptor, the login's lock last on the command line, the tools that outlive a session denied - assert plan["env_by_fd"] == {"CLAUDE_CODE_OAUTH_TOKEN": "CLAUDE_CODE_OAUTH_TOKEN_FILE_DESCRIPTOR"} - assert plan["command"][-2:] == ["--settings", cli.CLAUDE_LOCK] and plan["stdin"] is True - denied = plan["command"][plan["command"].index("--disallowedTools") + 1].split(",") - assert {"WebSearch", "WebFetch", *cli.CLAUDE_BEYOND_SESSION} <= set(denied) - assert {k: plan["env"][k] for k in cli.CLAUDE_LOGIN_ENV} == cli.CLAUDE_LOGIN_ENV - assert plan["env"]["IS_SANDBOX"] == "1" - assert TOKEN not in yaml.safe_dump(config) and TOKEN not in job_files(job) # the job and the plan never hold it - env = agent.harbor_env(set(plan["env_from_host"].values())) - assert env["CLAUDE_CODE_OAUTH_TOKEN"] == TOKEN # Harbor's environment does, by name - assert not {"ANTHROPIC_API_KEY", "CLAUDECODE", "CLAUDE_CODE_SESSION_ID"} & set(env) - monkeypatch.delenv("CLAUDE_CODE_OAUTH_TOKEN") - with pytest.raises(ValueError, match="setup-token"): - agent.harbor_env() - monkeypatch.setenv("CLAUDE_CODE_OAUTH_TOKEN", TOKEN) - with pytest.raises(ValueError, match="takes effort"): - claude_agent(tmp_path, fake_harbor, tasks, effort="extreme") - with pytest.raises(ValueError, match="command line"): # Harbor's own agent puts the token in argv - make_agent(tmp_path, fake_harbor, tasks, acting_agent={"name": "claude-code"}, binding=None, model="m", - login="claude") - with pytest.raises(ValueError, match="model name"): - claude_agent(tmp_path, fake_harbor, tasks, model=None) - # A Codex trial's Harbor never gets the token at all - assert "CLAUDE_CODE_OAUTH_TOKEN" not in make_agent(tmp_path, fake_harbor, tasks).harbor_env() - - -def test_harbors_own_claude_code_keeps_the_key_it_signs_in_with(tmp_path, fake_harbor, monkeypatch): - """Harbor's own Claude Code signs in with this process's key and endpoint: its harbor gets them, and nothing of a - parent Claude Code session; HermitClaude's harbor gets no bill but the one its plan names.""" - parent = {"ANTHROPIC_API_KEY": "sk-ant-fake-0123456789abcdef", "ANTHROPIC_BASE_URL": "https://api.example", - "CLAUDECODE": "1", "CLAUDE_CODE_SESSION_ID": "the-parents"} - for name, value in parent.items(): - monkeypatch.setenv(name, value) - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - env = make_agent(tmp_path, fake_harbor, tasks, acting_agent={"name": "claude-code"}, binding=None, - model="claude-haiku-5-5").harbor_env() - assert {k: env.get(k) for k in ("ANTHROPIC_API_KEY", "ANTHROPIC_BASE_URL")} == \ - {k: parent[k] for k in ("ANTHROPIC_API_KEY", "ANTHROPIC_BASE_URL")} - assert not {"CLAUDECODE", "CLAUDE_CODE_SESSION_ID"} & set(env) - hermits = make_agent(tmp_path, fake_harbor, tasks, acting_agent={"cli": "claude"}).harbor_env() - assert not set(parent) & set(hermits) - - -def test_harbor_never_gets_the_switches_that_sign_its_own_codex_in_with_the_users_chatgpt_file(tmp_path, fake_harbor, - monkeypatch): - """CODEX_FORCE_AUTH_JSON or CODEX_AUTH_JSON_PATH in this process's environment has Harbor's own codex agent upload - the user's ChatGPT sign-in (~/.codex/auth.json, or the file named) into every trial's container: the plan used - outside its slots and MAX_USED. Neither reaches Harbor, for its own codex or for a trial of Hermit's table.""" - monkeypatch.setenv("CODEX_FORCE_AUTH_JSON", "1") - monkeypatch.setenv("CODEX_AUTH_JSON_PATH", str(tmp_path / "auth.json")) - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - for spec in ({"name": "codex"}, {"cli": "codex"}): - env = make_agent(tmp_path, fake_harbor, tasks, acting_agent=spec).harbor_env() - assert not {"CODEX_FORCE_AUTH_JSON", "CODEX_AUTH_JSON_PATH"} & set(env), spec - assert env[KEY_ENV] == KEY # the rest of this process's environment - - -def test_a_session_is_read_from_its_stream(tmp_path, fake_harbor, monkeypatch): - monkeypatch.setenv("CLAUDE_CODE_OAUTH_TOKEN", TOKEN) - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - row = dataset.index(tasks)[0] - agent = claude_agent(tmp_path, fake_harbor, tasks) - job = tmp_path / "job" - plan = agent._launch(row, job, None, None) - trial = tmp_path / "trial" - - def facts(case): - write(trial / "agent" / "agent-claude.jsonl", (STREAMS / f"{case}.jsonl").read_text()) - return agent.session(trial, plan, None, job, row)[0] - - assert facts("reply")["served"] is True and facts("reply")["refused"] is False and facts("reply")["started"] - limit = facts("limit_seven_day") - assert limit["refused"] is True and limit["held_until"] == 1793856468 and limit["plan_report"]["windows"] - assert facts("overloaded")["unavailable"] is True and facts("not_logged_in")["login_failed"] is True - assert agent.session(tmp_path / "nowhere", plan, None, job, row) == (None, None, {}) # the CLI never ran - _, usage, said = agent.session(trial, plan, None, job, row) - assert usage is None or usage.cost_usd is None # a subscription: no price - ok = errors.classify(trial_result()) - verdict = HarborAgent.session_verdict(ok, limit, "claude") - assert verdict["status"] == "infra" and verdict["failure_cause"] == "usage_limit" and not verdict["halt"] - assert HarborAgent.session_verdict(ok, limit)["halt"] is True # an API account's limit: nothing later runs - lost = HarborAgent.session_verdict(ok, facts("not_logged_in"), "claude") - assert lost["halt"] is True and lost["failure_cause"] == "login_failed" - usage_limit = errors.classify(trial_result(None, exc="ApiUsageLimitError")) - assert usage_limit["halt"] is True - assert HarborAgent.session_verdict(usage_limit, facts("reply"), "claude")["halt"] is False - # the model's safeguards ended the session: Harbor raises UnknownApiError (a provider failure, infra), but it is - # the agent's own end (agent_refused_task), kept as its 0 and not run again - refusal = facts("model_refusal") - assert refusal["model_refusal"] is True - flagged = errors.classify(trial_result(None, exc="UnknownApiError")) - assert flagged["status"] == "infra" - kept = HarborAgent.session_verdict(flagged, refusal, "claude") - assert (kept["status"], kept["failure_cause"], kept["halt"]) == ("agent_error", "agent_refused_task", False) - assert HarborAgent.session_verdict(ok, refusal, "claude")["status"] == "ok" # a scored trial stays scored - # the adapter's tools never came up (Claude Code's init names the server failed): infra, not the agent's 0 - init = {"type": "system", "subtype": "init", "session_id": "s", "mcp_servers": [{"name": "calcdemo", - "status": "failed"}]} - write(trial / "agent" / "agent-claude.jsonl", json.dumps(init) + "\n" + - (STREAMS / "reply.jsonl").read_text().split("\n", 1)[1]) - data = json.loads(plan.read_text()) - data["metadata"].update(adapter_id="calcdemo", tools=["calc"]) - plan.write_text(json.dumps(data)) - down = agent.session(trial, plan, None, job, row)[0] - assert down["tools_down"] is True and "calcdemo: failed" in down["failure"] - assert HarborAgent.session_verdict(ok, down, "claude")["failure_cause"] == "session_failed" - - -def test_claude_codes_init_event_whose_strings_hold_a_unicode_line_separator_still_names_its_tools_down( - tmp_path, fake_harbor, monkeypatch): - """Claude Code's init event names each tool server's status, and JSON.stringify leaves U+0085, U+2028 and U+2029 - unescaped in its strings: the event is read whole, so the adapter's tools that never came up are still infra.""" - monkeypatch.setenv("CLAUDE_CODE_OAUTH_TOKEN", TOKEN) - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - row = dataset.index(tasks)[0] - agent = claude_agent(tmp_path, fake_harbor, tasks) - job, trial = tmp_path / "job", tmp_path / "trial" - plan = agent._launch(row, job, None, None) - data = json.loads(plan.read_text()) - data["metadata"].update(adapter_id="calcdemo", tools=["calc"]) - plan.write_text(json.dumps(data)) - stream = trial / "agent" / "agent-claude.jsonl" - stream.parent.mkdir(parents=True) - for sep in ("\x85", "\u2028", "\u2029"): - init = {"type": "system", "subtype": "init", "session_id": "s", "cwd": f"/app/notes{sep}2026", - "mcp_servers": [{"name": "calcdemo", "status": "failed"}]} - rest = (STREAMS / "reply.jsonl").read_text(encoding="utf-8").split("\n", 1)[1] - stream.write_bytes((json.dumps(init, ensure_ascii=False) + "\n" + rest).encode("utf-8")) - down = agent.session(trial, plan, None, job, row)[0] - assert down["tools_down"] is True and "calcdemo: failed" in down["failure"], repr(sep) - - -def test_a_session_whose_key_the_provider_refused_is_infra_without_a_halt(tmp_path, fake_harbor): - """A provider that refused the session's requests before it served any (Claude Code on a key the provider does - not take: `rejected`) is not the agent's result: infra (model_rejected), and the run goes on (there is no login - to restore). A session the provider served before its last request was refused stays the agent's.""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - row = dataset.index(tasks)[0] - agent = make_agent(tmp_path, fake_harbor, tasks, acting_agent={"cli": "claude"}) - job, trial = tmp_path / "job", tmp_path / "trial" - plan = agent._launch(row, job, None, None) - write(trial / "agent" / "agent-claude.jsonl", (STREAMS / "bad_token.jsonl").read_text()) - facts = agent.session(trial, plan, None, job, row)[0] - assert facts["rejected"] is True and facts["login_failed"] is False - ok = errors.classify(trial_result()) - verdict = HarborAgent.session_verdict(ok, facts) - assert (verdict["status"], verdict["failure_cause"], verdict["halt"]) == ("infra", "model_rejected", False) - assert "model_rejected" in errors.INFRA_CAUSES and facts["tokens"] == 0 - assert HarborAgent.session_verdict(ok, {**facts, "tokens": 1200})["status"] == "ok" # it had been served - - -def test_a_first_request_refused_for_what_the_input_made_it_is_the_agents_result(tmp_path, fake_harbor): - """Only a refusal of the run's configuration (a key or token, a permission, a model the provider does not have) - is model_rejected: a first request the adapter's or the task's input made too long or invalid is the agent's - result, its work scored as Harbor scores it. The answer agent's 400 is its result too, unless its message names a - configuration refusal; its 401, the configuration's.""" - from hermit.harbor.tools import ModelEndpoint - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - row = dataset.index(tasks)[0] - agent = make_agent(tmp_path, fake_harbor, tasks, acting_agent={"cli": "claude"}) - job, trial = tmp_path / "job", tmp_path / "trial" - plan = agent._launch(row, job, None, None) - exited = errors.classify(trial_result(0.5, exc="NonZeroAgentExitCodeError")) # the CLI exited 1; verified - - def verdict(case): - write(trial / "agent" / "agent-claude.jsonl", (STREAMS / f"{case}.jsonl").read_text()) - facts = agent.session(trial, plan, None, job, row)[0] - assert facts["rejected"] is True and not facts["served"] and facts["tokens"] == 0, case - return HarborAgent.session_verdict(exited, facts) - too_long = verdict("too_long") # "Prompt is too long", a 400 - assert (too_long["status"], too_long["failure_cause"], too_long["reward"]) == ("agent_error", "unclassified", 0.5) - assert judge.score(row, RunOutput(output={"status": too_long["status"], "reward": too_long["reward"]}))["score"] \ - == 0.5 - for case in ("bad_token", "no_model"): # a 401, a model that may not exist (404) - assert verdict(case)["failure_cause"] == "model_rejected", case - for status, cause in ((400, "unclassified"), (401, "model_rejected")): - relay = ModelEndpoint(BINDING) - relay._note((status, b'{"error": {"message": "refused"}}', {})) - assert HarborAgent.session_verdict(exited, relay.facts())["failure_cause"] == cause, status - - -def test_the_answer_agents_cli_session_refused_for_its_key_is_model_rejected_by_what_its_400_says(): - """Hermit's endpoint answers every session of a model a CLI reaches that the provider rejected with a 400 (the - refusal named only in its message, endpoint._no_reply): one refused for its key (Claude Code's 401 for a bad - token) is the configuration's, model_rejected, its message the session's failure where the trial has none; one - whose prompt was too long stays the agent's result.""" - from hermit.agent.base.cli import Claude - from hermit.agent.base.endpoint import _no_reply - from hermit.agent.base.interface import Result - from hermit.harbor.tools import ModelEndpoint - exited = errors.classify(trial_result(0.5, exc="NonZeroAgentExitCodeError")) - - def facts(case): - claude = Claude({"model": "claude-haiku-5-5"}) - said = claude.outcome((STREAMS / f"{case}.jsonl").read_text(), None, None) - endpoint = ModelEndpoint({"cli": "claude", "model": "claude-haiku-5-5"}) - endpoint._note(_no_reply(claude, Result(reply="", exit_code=1, extra=said))) # the endpoint's own reply - out = endpoint.facts() - assert out["statuses"] == [400] and out["rejected"] is True, case - return out - bad_key = facts("bad_token") - refused = HarborAgent.session_verdict(exited, bad_key) - assert (refused["status"], refused["failure_cause"], refused["halt"]) == ("infra", "model_rejected", False) - assert bad_key["failure"].endswith("Failed to authenticate. API Error: 401 Invalid bearer token") - unexplained = HarborAgent.session_verdict(errors.classify(trial_result(0.0)), bad_key) # no exception of its own - assert unexplained["failure_cause"] == "model_rejected" and unexplained["exception_message"] == bad_key["failure"] - too_long = HarborAgent.session_verdict(exited, facts("too_long")) - assert (too_long["status"], too_long["failure_cause"], too_long["reward"]) == ("agent_error", "unclassified", 0.5) - - -def pi_failed(error_message: str) -> str: - """A Pi session whose one request failed with `error_message` (as Pi writes the provider's status and body).""" - events = [{"type": "session", "version": 3, "id": "s"}, {"type": "agent_start"}, {"type": "turn_start"}, - {"type": "message_end", "message": {"role": "assistant", "stopReason": "error", - "errorMessage": error_message, "usage": {"input": 0, "output": 0}}}, - {"type": "agent_settled"}] - return "".join(json.dumps(e) + "\n" for e in events) - - -def outcome(agent, raw: str, where: Path) -> dict: - """What `agent` (a CLI Base Agent) reads of a session whose stream is `raw`, given as a file, as the runner gives a - trial's stream to it (its own copy under launch/read/).""" - where.mkdir(parents=True, exist_ok=True) - stream = where / f"stream-{uuid.uuid4().hex[:8]}.jsonl" - stream.write_text(raw, encoding="utf-8") - return agent.outcome(raw, stream, None) - - -def test_a_refusal_of_the_key_or_model_in_a_providers_own_wording_is_model_rejected_one_of_the_bill_ends_the_run( - tmp_path, fake_harbor): - """A provider's refusal of the run's configuration in its own wording (DeepSeek's model that does not exist, - Zhipu GLM's unknown model, Mistral's invalid model, OpenRouter's model that is not a valid model ID, Gemini's key - not valid) is model_rejected. Its refusal of the account's bill (a 402, a balance or credit too low, an - insufficient quota, billing; the answer agent's relay's 402) ends the run (usage_limit, a halt), as an API - account's usage limit does: nothing later can be served. A rate limit is the provider's, `unavailable` before any - rejection is read, and its advice (billing details, a link to the billing page) is no such wording; nor is a prompt - too long, a content filter or an overload. (An OpenCode error whose words name none of it is the agent's result: - OpenCode's Base Agent gives its failure without its status.)""" - from hermit.agent.base.cli import Claude, OpenCode, Pi - from hermit.harbor.runner import CONFIG_REFUSED - exited = errors.classify(trial_result(0.5, exc="NonZeroAgentExitCodeError")) - - def verdict(raw, cli=Pi): - said = outcome(cli({}), raw, tmp_path / "read") - facts = {k: said.get(k) for k in ("served", "unavailable", "refused", "login_failed", "rejected", - "model_refusal")} - return HarborAgent.session_verdict(exited, {**facts, "failure": said["failure"], "started": True, "tokens": 0}) - refusals = ['400 {"error":{"message":"Model Not Exist","type":"invalid_request_error","param":null,' - '"code":"invalid_request_error"}}', - '400 {"error":{"code":"1211","message":"Unknown Model, please check the model code."}}', - '400 {"object":"error","message":"Invalid model: mistral-nope","type":"invalid_model","param":null,' - '"code":"1500"}', - '400 {"error":{"message":"openai/gpt-nope is not a valid model ID","code":400}}', - '400 {"error":{"code":400,"message":"API key not valid. Please pass a valid API key.",' - '"status":"INVALID_ARGUMENT"}}'] - for text in refusals: - assert (verdict(pi_failed(text))["failure_cause"], verdict(pi_failed(text))["halt"]) == \ - ("model_rejected", False), text - bills = ['402 {"error":{"message":"Insufficient Balance","type":"unknown_error"}}', - '400 {"type":"error","error":{"type":"invalid_request_error","message":"Your credit balance is too low ' - 'to access the Anthropic API. Please go to Plans & Billing to upgrade or purchase credits."}}', - '400 {"error":{"message":"You exceeded your current quota.","type":"insufficient_quota"}}', - '400 {"error":{"code":400,"message":"Gemini API free tier is not available in your country. Please ' - 'enable billing on your project in Google AI Studio.","status":"FAILED_PRECONDITION"}}'] - for text in bills: - assert (verdict(pi_failed(text))["failure_cause"], verdict(pi_failed(text))["halt"]) == \ - ("usage_limit", True), text - credit = verdict((STREAMS / "credit_balance.jsonl").read_text(), Claude) - assert (credit["failure_cause"], credit["halt"]) == ("usage_limit", True) - from hermit.harbor.tools import ModelEndpoint - relay = ModelEndpoint(BINDING) # the answer agent's relay: the bill's own status - relay._note((402, b'{"error": {"message": "Insufficient Balance"}}', {})) - relayed = HarborAgent.session_verdict(exited, relay.facts()) - assert (relayed["failure_cause"], relayed["halt"]) == ("usage_limit", True) - relay = ModelEndpoint(BINDING) # ... its 402 whatever its words - relay._note((402, b'{"error": {"message": "This account is not active"}}', {})) - relayed = HarborAgent.session_verdict(exited, relay.facts()) - assert (relayed["failure_cause"], relayed["halt"]) == ("usage_limit", True) - # a 429 for a spent quota (OpenAI's insufficient_quota, its code or its type alone), which no pause ends, ends the - # run too: the relay's, and a CLI's that quotes the reply; any other 429 is the provider's, a pause away - spent = b'{"error": {"message": "You exceeded your current quota, please check your plan and billing details.", ' - for body in (spent + b'"type": "insufficient_quota", "code": "insufficient_quota"}}', spent + b'"type": ' - b'"insufficient_quota"}}'): - relay = ModelEndpoint(BINDING) - relay._note((429, body, {})) - relayed = HarborAgent.session_verdict(exited, relay.facts()) - assert (relayed["failure_cause"], relayed["halt"]) == ("usage_limit", True), body - relay = ModelEndpoint(BINDING) - relay._note((429, b'{"error": {"message": "Rate limit reached", "code": "rate_limit_exceeded"}}', {})) - assert HarborAgent.session_verdict(exited, relay.facts())["failure_cause"] == "model_provider_failed" - quoted = verdict(pi_failed('429 {"error":{"message":"You exceeded your current quota, please check your plan and ' - 'billing details.","type":"insufficient_quota","code":"insufficient_quota"}}')) - assert (quoted["failure_cause"], quoted["halt"]) == ("usage_limit", True) - relay = ModelEndpoint(BINDING) # ... and its refusal of the model - relay._note((404, b'{"error": {"message": "The model does not exist"}}', {})) - relayed = HarborAgent.session_verdict(exited, relay.facts()) - assert (relayed["failure_cause"], relayed["halt"]) == ("model_rejected", False) - rate_limits = ['429 {"error":{"message":"Rate limit reached for gpt-5 in organization org-x on requests per min ' - '(RPM): Limit 3, Used 3, Requested 1. Please try again in 20s. Visit ' - 'https://platform.openai.com/account/billing to add a payment method.",' - '"code":"rate_limit_exceeded"}}', - '429 {"error":{"code":429,"message":"You exceeded your current quota, please check your plan and ' - 'billing details.","status":"RESOURCE_EXHAUSTED"}}'] - for text in rate_limits: - assert verdict(pi_failed(text))["failure_cause"] == "model_provider_failed", text - assert not CONFIG_REFUSED.search(text), text # nor in a rejection's earlier notices - others = [outcome(Claude({}), (STREAMS / f"{case}.jsonl").read_text(), tmp_path / "read")["failure"] - for case in ("too_long", "model_refusal", "throttled", "overloaded", "server_error")] - others += [outcome(OpenCode({}), (OPENCODE / f"{case}.jsonl").read_text(), tmp_path / "read")["failure"] - for case in ("failure_429", "failure_500")] - others.append("400 The response was filtered due to the prompt triggering Azure OpenAI's content management " - "policy. Please modify your prompt and retry.") - others.append('400 {"error":{"message":"Invalid prompt: your prompt was flagged as potentially violating our usage ' - 'policy."}}') - assert [text for text in others if CONFIG_REFUSED.search(text)] == [] - - -def test_a_rejections_status_is_read_only_where_one_is_written_never_from_a_number_in_its_words(tmp_path): - """A refusal's status is read where a status is written: leading the text or a line of it (Pi's `401 {...}`, - `403: {...}`), after the prefix Hermit's endpoint gives a CLI's failure in its reply (`...(exit 1): 401 {...}`, - `(timed out): `), after `API Error: ` (Claude Code's) or `Error code: ` (OpenAI's SDK), in `(HTTP 402)` - (OpenCode's) or `API error (404)` (Pi's), after `status` or a JSON `"code"`. A number in the words is none: - OpenAI's refusal of a prompt over the context length names token counts (`you requested 4402 tokens (402 in the - messages, ...)`), a refused image's message the image host's status (`(HTTP 403 from the image host)`), and a - provider's own `(...): 403` is no endpoint's prefix, each the agent's result however it is read: a Pi session's - stream, the answer agent's relay's 400, a Pi session the answer agent's model is.""" - from hermit.agent.base.cli import Pi - from hermit.agent.base.endpoint import _no_reply - from hermit.agent.base.interface import Result - from hermit.harbor.runner import CONFIG_REFUSED - from hermit.harbor.tools import ModelEndpoint - exited = errors.classify(trial_result(0.5, exc="NonZeroAgentExitCodeError")) - - def read(text): - """(as a Pi trial's stream is read, as the answer agent's relay reads a provider's 400 saying `text`, as the - answer agent's endpoint reads a Pi session that failed with it): each verdict's failure cause.""" - said = outcome(Pi({}), pi_failed(text), tmp_path) - facts = {k: said.get(k) for k in ("served", "unavailable", "refused", "login_failed", "rejected")} - relay = ModelEndpoint(BINDING) - relay._note((400, json.dumps({"error": {"message": text}}).encode(), {})) - session = ModelEndpoint({"cli": "pi", "model": "m"}) - session._note(_no_reply(Pi({}), Result(reply="", exit_code=1, extra=said))) - return [HarborAgent.session_verdict(exited, f)["failure_cause"] for f in - ({**facts, "failure": said["failure"], "started": True, "tokens": 0}, relay.facts(), session.facts())] - for text in ('401 {"error":{"message":"bad key"}}', '403: {"message":"Your account has been suspended."}', - 'hermit API error (404): {"error":{"message":"no such model"}}'): # Pi's own forms - assert read(text) == ["model_rejected"] * 3, text - written = ["API Error: 402 Insufficient credits", "Error code: 401 - {'error': {'message': 'bad key'}}", - "provider.quota: the model is down (HTTP 402)", "request failed with status code 403", - '{"error": {"code": 402, "message": "Insufficient credits"}}', "the model is down\n404 no such model", - "the codex session gave no reply (exit -9): 401 {}", "the pi session gave no reply (timed out): 404 {}"] - assert [text for text in written if not CONFIG_REFUSED.search(text)] == [] - numbers = ["400 This model's maximum context length is 4097 tokens. However, you requested 4402 tokens (402 in the " - "messages, 4000 in the completion). Please reduce the length of the messages or completion.", - '400 {"error":{"message":"This model\'s maximum context length is 131072 tokens. However, you requested ' - '402 tokens (1 in the messages, 401 in the completion)."}}', - '400 {"error":{"message":"The input (403 tokens) is longer than the model\'s context length (402 ' - 'tokens)."}}', - '400 {"type":"error","error":{"type":"invalid_request_error","message":"messages.0.content.0.image.' - 'source: Unable to download the file. Please verify the URL and try again (HTTP 403 from the image ' - 'host)."}}', - '400 {"error":{"message":"input tokens (402): 403 over the limit"}}', - '400 {"error":{"message":"(1 of 2): 403 tokens exceed the limit"}}'] - for text in numbers: - assert not CONFIG_REFUSED.search(text), text - assert read(text) == ["unclassified"] * 3, text - - -def test_a_request_the_models_session_or_its_provider_cannot_take_refused_the_configuration(tmp_path): - """What Hermit's endpoint refuses a session of a model a CLI reaches cannot take (a figure, tool calls, another - model, more than one reply; settings the session cannot start with: a 400 with Hermit's own code, - endpoint.UNSUPPORTED), and a provider's refusal of a request setting its model does not take (OpenAI's unsupported - parameter or value, Azure's unrecognized request argument, OpenCode's unsupported operation) refused the run's - configuration, which no input changes: model_rejected (unscored, no halt), never the agent's 0 in every trial. A 400 - without that code or such words stays the agent's. (The code is the runner's relay's, hermit.harbor.tools: Hermit's - endpoint's own 400 names the problem in its words alone.)""" - from hermit.agent.base.cli import OpenCode - from hermit.agent.base.interface import make - from hermit.harbor.tools import ModelEndpoint - exited = errors.classify(trial_result(0.0, exc="NonZeroAgentExitCodeError")) - codex = make("codex", {"model": "gpt-5.6-luna", "login": "chatgpt", "web_search": False}) - figure = {"model": "gpt-5.6-luna", "messages": [{"role": "user", "content": [ - {"type": "text", "text": "What does the figure show?"}, - {"type": "image_url", "image_url": {"url": "data:image/png;base64,AAAA"}}]}]} - text = {"model": "gpt-5.6-luna", "messages": [{"role": "user", "content": "What is 6 x 7?"}]} - for request in (figure, {**text, "model": "gpt-other"}, {**text, "n": 2}, - {**text, "tools": [{"type": "function", "function": {"name": "f", "parameters": {}}}]}): - session = ModelEndpoint({"cli": "codex", "model": "gpt-5.6-luna", "login": "chatgpt"}) - session._note(session._cli_answer(codex)(request, None, None, {})) # refused before any session - verdict = HarborAgent.session_verdict(exited, session.facts()) - assert (session.facts()["statuses"], verdict["failure_cause"], verdict["halt"]) == \ - ([400], "model_rejected", False), request - session = ModelEndpoint({"cli": "codex", "model": "gpt-5.6-luna"}) # web_search unset: it never starts - unset = session._cli_answer(make("codex", {"model": "gpt-5.6-luna"})) - session._note(unset(text, None, None, {})) # the session's settings refused, by its code - assert HarborAgent.session_verdict(exited, session.facts())["failure_cause"] == "model_rejected" - for said in ("Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' " - "instead.", "Unsupported value: 'reasoning_effort' does not support 'minimal' with this model.", - "Unrecognized request argument supplied: reasoning_effort"): - relay = ModelEndpoint(BINDING) - relay._note((400, json.dumps({"error": {"message": said, "type": "invalid_request_error", "param": None, - "code": None}}).encode(), {})) - assert HarborAgent.session_verdict(exited, relay.facts())["failure_cause"] == "model_rejected", said - unsupported = outcome(OpenCode({}), '{"type":"error","error":{"type":"provider.unsupported-operation","message":' - '"the model takes no images","status":400}}\n', tmp_path) - facts = {k: unsupported.get(k) for k in ("served", "unavailable", "refused", "login_failed", "rejected")} - assert HarborAgent.session_verdict(exited, {**facts, "failure": unsupported["failure"], "started": True, - "tokens": 0})["failure_cause"] == "model_rejected" - relay = ModelEndpoint(BINDING) # another code, and words that name no setting: the agent's - relay._note((400, b'{"error": {"code": "invalid_request", "message": "Invalid prompt"}}', {})) - assert HarborAgent.session_verdict(exited, relay.facts())["failure_cause"] == "unclassified" - - -OPENCODE = ROOT / "tests" / "fixtures" / "opencode" -OPENCODE_SESSION = "ses_e40d3f481ffePQb8uaj39iK60v" # success.jsonl's session - - -def opencode_db(path: Path, *, wal: bool = False, compactions: int = 1, **tokens): - """OpenCode's session database as 2.2.6 keeps a session's totals (tests/core/test_opencode_agent.py's - keep_totals), at `path`; with `wal` in WAL mode with its rows still in the write-ahead log, as a session cut off - leaves them. The connection is returned open (closing it would merge them in).""" - import sqlite3 - path.parent.mkdir(parents=True, exist_ok=True) - c = sqlite3.connect(path) - if wal: - c.execute("PRAGMA journal_mode = WAL") - c.execute("PRAGMA wal_autocheckpoint = 0") - c.execute("CREATE TABLE session_v2 (id text PRIMARY KEY, tokens_input integer, tokens_output integer, " - "tokens_reasoning integer, tokens_cache_read integer, tokens_cache_write integer)") - c.execute("CREATE TABLE session_message (id text PRIMARY KEY, session_id text, type text)") - c.execute("INSERT INTO session_v2 VALUES (?, ?, ?, ?, ?, ?)", (OPENCODE_SESSION, *( - tokens.get(k, 0) for k in ("input", "output", "reasoning", "cache_read", "cache_write")))) - c.executemany("INSERT INTO session_message VALUES (?, ?, 'compaction')", - [(f"msg_{i}", OPENCODE_SESSION) for i in range(compactions)]) - c.commit() - return c - - -def opencode_trial(tmp_path: Path, fake_harbor: Path): - """An OpenCode runner, its launch plan, and a trial whose agent/ holds the recorded session's stream (four requests - of 1000 prompt tokens, 400 cached, and 20 output tokens each).""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - row = dataset.index(tasks)[0] - agent = make_agent(tmp_path, fake_harbor, tasks, acting_agent={"cli": "opencode"}) - job, trial = tmp_path / "job", tmp_path / "trial" - plan = agent._launch(row, job, None, None) - write(trial / "agent" / "agent-opencode.jsonl", (OPENCODE / "success.jsonl").read_text()) - (trial / "agent" / "hermit-collect").mkdir() - return agent, row, job, trial, plan - - -def test_an_opencode_sessions_last_rows_in_its_write_ahead_log_are_counted(tmp_path, fake_harbor): - """OpenCode keeps its database in WAL mode: the plan collects its -wal and -shm beside it, and what a session cut - off left only in the -wal (its compaction's model call here) counts in its usage.""" - agent, row, job, trial, plan = opencode_trial(tmp_path, fake_harbor) - assert [c["to"] for c in json.loads(plan.read_text())["collect"]] == \ - ["opencode.db", "opencode.db-wal", "opencode.db-shm"] - live = opencode_db(tmp_path / "live" / "opencode.db", wal=True, input=4400, cache_read=2600, output=90, - reasoning=30) - try: - for name in ("opencode.db", "opencode.db-wal"): # as harborside collects them, the session cut off - shutil.copy(tmp_path / "live" / name, trial / "agent" / "hermit-collect" / name) - finally: - live.close() - usage = agent.session(trial, plan, None, job, row)[1] - assert (usage.input_tokens, usage.cached_input_tokens, usage.output_tokens, usage.calls) == (7000, 2600, 120, 5) - - -def test_an_opencode_database_the_container_left_as_a_link_is_never_copied(tmp_path, fake_harbor): - """A collected opencode.db that is a link (to a database of this machine) is not read: the usage is the - stream's.""" - agent, row, job, trial, plan = opencode_trial(tmp_path, fake_harbor) - opencode_db(tmp_path / "host-only" / "opencode.db", input=999_000).close() - (trial / "agent" / "hermit-collect" / "opencode.db").symlink_to(tmp_path / "host-only" / "opencode.db") - usage = agent.session(trial, plan, None, job, row)[1] - assert (usage.input_tokens, usage.calls) == (4000, 4) - assert not list((job / "launch" / "read").rglob("opencode.db")) - - -def test_a_refused_trial_waits_for_the_plan_and_runs_again(tmp_path, fake_harbor, monkeypatch): - from hermit.agent.base import login - monkeypatch.setenv("CLAUDE_CODE_OAUTH_TOKEN", TOKEN) - monkeypatch.setenv("FAKE_STREAMS", f"{STREAMS / 'limit_seven_day.jsonl'},{STREAMS / 'reply.jsonl'}") - waits, holds = [], [] - monkeypatch.setattr(login, "wait", lambda plan, cancel, log, max_s, credits: waits.append(max_s) or "") - monkeypatch.setattr(login, "hold", lambda plan, until=None, why="session_refused", **kw: holds.append((until, why))) - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - agent = claude_agent(tmp_path, fake_harbor, tasks) - out = agent.run(row, None) - assert out.output["status"] == "ok" and out.extra["session"]["served"] is True - assert holds == [(1793856468, "session_refused")] and waits == [None, None] # no limit on the wait - assert (tmp_path / "jobs" / ".fake-calls").read_text() == "2" and not agent._stop.is_set() - noted = (login.CLAUDE_PLAN / "plan.jsonl").read_text().splitlines() # both sessions' reports are kept - assert len(noted) == 2 and json.loads(noted[0])["used_percent"] == 100.0 - # a lost sign-in ends the run instead - monkeypatch.setenv("FAKE_STREAMS", str(STREAMS / "not_logged_in.jsonl")) - lost = claude_agent(tmp_path, fake_harbor, tasks) - with pytest.raises(LoginFailed, match=r"login_failed \(AgentAuthenticationError\): Not logged in"): - lost.run(row, None) - assert lost._stop.is_set() - - -def test_a_login_trials_report_and_holds_are_written_before_its_slot_is_given_back(tmp_path, fake_harbor, - monkeypatch): - """A trial waiting for one of the plan's slots reads the plan the moment it has one: what the session before it - said of the plan (its report, a refusal's hold) is written while that session still holds its slot, as - CliAgent.run writes it, so that with SLOTS: 1 no trial starts on a plan the one before it found exhausted.""" - from contextlib import contextmanager - from hermit.agent.base import login - monkeypatch.setenv("CLAUDE_CODE_OAUTH_TOKEN", TOKEN) - monkeypatch.setenv("FAKE_STREAMS", f"{STREAMS / 'limit_seven_day.jsonl'},{STREAMS / 'reply.jsonl'}") - seen = [] - - @contextmanager - def slot(plan, cancel=None, log=None, deadline=None): - seen.append("slot") - try: - yield True - finally: - seen.append("given back") - monkeypatch.setattr(login, "slot", slot) - monkeypatch.setattr(login, "wait", lambda *a, **k: "") - monkeypatch.setattr(login, "note_usage", lambda plan, home, seen_=None: seen.append("report")) - monkeypatch.setattr(login, "hold", lambda plan, until=None, why="session_refused", **kw: seen.append(f"hold {why}")) - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - assert claude_agent(tmp_path, fake_harbor, tasks).run(row, None).output["status"] == "ok" - assert seen == ["slot", "report", "hold session_refused", "given back", "slot", "report", "given back"] - - -def claude_stream(case: str, path: Path, input_tokens: int, output_tokens: int) -> Path: - """A Claude Code stream of the fixtures whose result reports this usage (none of it cached), written to `path`.""" - lines = [] - for line in (STREAMS / f"{case}.jsonl").read_text().splitlines(): - event = json.loads(line) if line.lstrip().startswith("{") else None - if isinstance(event, dict) and event.get("type") == "result": - event.pop("model_usage_since_last_result", None) - event["usage"] = {"input_tokens": input_tokens, "output_tokens": output_tokens, - "cache_creation_input_tokens": 0, "cache_read_input_tokens": 0} - line = json.dumps(event) - lines.append(line) - return write(path, "\n".join(lines) + "\n") - - -def test_a_login_trials_usage_is_what_every_run_of_it_used(tmp_path, fake_harbor, monkeypatch): - """A login trial run again (its provider could not serve the first session) used what each run used: a first - session that worked on, used 5,100 tokens and then met an overload is in the trial's 5,300 tokens, in the - rollout and in the run's budget, each run's own in `extra.attempts`; its cost is unknown when any run's is.""" - from dataclasses import replace - from hermit.trainer.budget import Budget - from hermit.trainer.rollout import run_tasks - monkeypatch.setenv("CLAUDE_CODE_OAUTH_TOKEN", TOKEN) - busy = claude_stream("overloaded", tmp_path / "busy.jsonl", 5000, 100) - served = claude_stream("reply", tmp_path / "served.jsonl", 150, 50) - monkeypatch.setenv("FAKE_STREAMS", f"{busy},{served}") - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - budget = Budget() - [record] = run_tasks([row], None, claude_agent(tmp_path, fake_harbor, tasks, busy_wait_s=0).run, judge.score, - concurrency=1, budget=budget) - assert (record["status"], record["tokens"], budget.tokens["rollout"]) == ("ok", 5300, 5300) - assert [a["tokens"] for a in record["pipeline_extra"]["attempts"]] == [5100, 200] - assert record["api_speed"]["model_calls"] == sum(a["model_calls"] for a in record["pipeline_extra"]["attempts"]) - assert (tmp_path / "jobs" / ".fake-calls").read_text() == "2" - runs = [RunOutput(output={"status": "infra"}, cost_usd=0.5, tokens=10, api_speed={"model_calls": 1}), - RunOutput(output={"status": "ok"}, cost_usd=0.25, tokens=20, api_speed={"model_calls": 2})] - total = HarborAgent._attempts(runs) - assert (total.tokens, total.cost_usd, total.api_speed["model_calls"], len(total.extra["attempts"])) == \ - (30, 0.75, 3, 2) - assert HarborAgent._attempts([runs[0], replace(runs[1], cost_usd=None)]).cost_usd is None - assert HarborAgent._attempts(runs[1:]) is runs[1] # one run: as it is - - -def relayed_sessions(*streams: Path): - """A chat endpoint (started) over Claude Code sessions, as hermit.isolate.agent_pipeline serves the answer agent's - model a CLI reaches (cli_endpoint: each request one session): each session plays the next of `streams` (the last - one repeats), read as Claude Code's Base Agent reads a session's stream; (the endpoint, the streams played).""" - from hermit.agent.base.cli import Claude - from hermit.agent.base.endpoint import ChatEndpoint - from hermit.agent.base.interface import Result - played: list[Path] = [] - - class Session(Claude): - def _session(self, prompt, **kwargs): - stream = streams[min(len(played), len(streams) - 1)] - played.append(stream) - said = self.outcome(stream.read_text(), stream, None) - said.pop("started", None) - usage = self.parse(stream, self.settings["model"])[1] - return Result(reply=said.pop("reply"), finish_reason=said.pop("finish_reason", None), usage=usage, - exit_code=0 if said.get("served") else 1, - extra={"failed": False, "login_failed": False, "rejected": False, **said}) - return ChatEndpoint(Session({"model": "claude-haiku-5-5", "web_search": False})).start(), played - - -def test_a_relayed_cli_session_that_failed_gives_the_trial_the_tokens_it_used_and_the_budget_their_cost( - tmp_path, fake_harbor, monkeypatch): - """The answer agent's model behind an endpoint of CLI sessions (as endpoint.cli_answer serves them), which the - trial's endpoint relays each request to (a client of its own): a session that failed after it used tokens (Claude Code's model refusal, whose - 400 reports them: endpoint._no_reply) gives the trial those tokens, priced at the spec's price, and the run's budget - is charged them. Of a session the relay's client ran again (the provider could not serve it: a 503), the trial has - the served one's alone: the failed attempt's usage is in the error reply the client read and did not keep.""" - from types import SimpleNamespace - from hermit.agent.model import llm - from hermit.trainer.budget import Budget - from hermit.trainer.rollout import run_tasks - waits = [] - monkeypatch.setattr(llm, "time", SimpleNamespace(sleep=waits.append)) # the relay's client: no wait - price = {"input": 1.0, "cached_input": 0.5, "output": 2.0} - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - busy = claude_stream("overloaded", tmp_path / "busy.jsonl", 5000, 100) - served = claude_stream("text_session", tmp_path / "served.jsonl", 150, 50) # a session with no tools - seen = {} - for case, streams in (("refused", [STREAMS / "model_refusal.jsonl"]), ("retried", [busy, served])): - sessions, played = relayed_sessions(*streams) - (tmp_path / case).mkdir() - try: - agent = make_agent(tmp_path / case, fake_harbor, tasks, acting_agent={"cli": "answer"}, binding=None, - web_search=None, - model={**SERVED, "model": "claude-haiku-5-5", "base_url": sessions.url, "price": price, - "ref": "local/claude-haiku-5-5"}) - budget = Budget() - [record] = run_tasks([row], None, agent.run, judge.score, concurrency=1, budget=budget) - finally: - sessions.close() - seen[case] = (record, budget, played) - record, budget, played = seen["refused"] - cost = ((558 - 178) * 1.0 + 178 * 0.5 + 90 * 2.0) / 1e6 # the refusal's tokens, 178 of them cached - assert record["pipeline_extra"]["session"]["statuses"] == [400] and len(played) == 1 - assert (record["tokens"], record["api_speed"]["model_calls"], budget.tokens["rollout"]) == (648, 1, 648) - assert record["cost_usd"] == pytest.approx(cost) and budget.usd["rollout"] == pytest.approx(cost) - assert record["pipeline_extra"]["n_cache_tokens"] == 178 and budget.unknown_cost_calls == 0 - record, budget, played = seen["retried"] - assert (record["status"], record["pipeline_extra"]["session"]["statuses"], played) == ("ok", [200], [busy, served]) - assert waits == [2.0] # the relay's client ran the session again - assert (record["tokens"], record["api_speed"]["model_calls"], budget.tokens["rollout"]) == (200, 1, 200) - cost = (150 * 1.0 + 50 * 2.0) / 1e6 - assert record["cost_usd"] == pytest.approx(cost) and budget.usd["rollout"] == pytest.approx(cost) - - -# A Codex session whose turn failed on a dropped stream: a provider that could not serve it -UNSERVED = ('{"type":"thread.started","thread_id":"th-1"}\n{"type":"turn.started"}\n' - '{"type":"turn.failed","error":{"message":"stream disconnected before completion: error sending ' - 'request"}}\n') - - -def test_a_relayed_codex_session_that_lost_its_sign_in_is_a_401_whose_login_failed_ends_the_run(tmp_path, monkeypatch): - """The answer agent's model reached through Codex (`{cli: codex}`): a session that gave no reply and whose stream, - kept in the agent's log_dir, says its sign-in was refused (signin.codex_signin_lost, which the Base Agent's Codex - reading does not report) is a 401 with LOGIN_FAILED, as the endpoint says a Claude session's, so the trial's facts - say login_failed (the runner ends the run); one the provider refused for good (signin.codex_rejected) is a 400 - ending with Codex's words, read as a trial's Codex session is; one that failed otherwise keeps its reply (a 502: - unavailable).""" - import threading - from types import SimpleNamespace - from hermit.agent.base import endpoint - from hermit.agent.base.interface import LOGIN_FAILED - from hermit.harbor.tools import ModelEndpoint - logs = tmp_path / "model-sessions" - logs.mkdir() - refused = ('{"type":"thread.started","thread_id":"th-1"}\n{"type":"turn.started"}\n' - '{"type":"turn.failed","error":{"message":"unexpected status 401 Unauthorized: Your token has been ' - 'invalidated."}}\n') - streams = iter([refused, UNSERVED]) - - def answer(agent, request, abort=None, place=None): - (logs / f"t{len(list(logs.iterdir()))}-codex.jsonl").write_text(next(streams)) - return 502, json.dumps({"error": {"message": "the codex session gave no reply (exit 1)"}}).encode(), {} - monkeypatch.setattr(endpoint, "cli_answer", answer) - monkeypatch.setattr(endpoint, "contract_problems", lambda agent, request: ([], [])) - relay = ModelEndpoint({"cli": "codex", "model": "gpt-5.6-luna"}) - ask = relay._cli_answer(SimpleNamespace(backend="codex", settings={"log_dir": str(logs)})) - request = {"messages": [{"role": "user", "content": "2+2?"}]} - status, body, _ = reply = ask(request, None, threading.Event(), {}) - assert (status, json.loads(body)["error"]["code"]) == (401, LOGIN_FAILED) - relay._note(reply) - assert relay.facts()["login_failed"] is True - status, body, _ = reply = ask(request, None, threading.Event(), {}) - assert (status, json.loads(body)["error"].get("code")) == (502, None) - relay._note(reply) - facts = relay.facts() - assert (facts["login_failed"], facts["unavailable"]) == (False, True) - # a session the provider refused for good (signin.codex_rejected), which the Base Agent's Codex reading leaves a - # 502, is a 400 ending with Codex's words: the run's configuration (model_rejected), its bill (a halt), or the - # agent's result (a prompt too long), never the provider's outage run again - exited = errors.classify(trial_result(0.0, exc="NonZeroAgentExitCodeError")) - start = '{"type":"thread.started","thread_id":"th-1"}\n{"type":"turn.started"}\n' - for said, want in (("unexpected status 400 Bad Request: Unsupported parameter: 'max_tokens'", "model_rejected"), - ("Your workspace is out of credits. Add credits to continue.", "usage_limit"), - ('{"error":{"message":"Your input exceeds the context window of this model.",' - '"type":"invalid_request_error","code":"context_length_exceeded"}}', None)): - streams = iter([start + json.dumps({"type": "turn.failed", "error": {"message": said}}) + "\n"]) - relay = ModelEndpoint({"cli": "codex", "model": "gpt-5.6-luna"}) - ask = relay._cli_answer(SimpleNamespace(backend="codex", settings={"log_dir": str(logs)})) - status, body, _ = reply = ask(request, None, threading.Event(), {}) - assert status == 400 and json.loads(body)["error"]["message"].endswith(said), (said, body) - relay._note(reply) - facts = relay.facts() - assert (facts["rejected"], facts["unavailable"]) == (True, False), said - got = HarborAgent.session_verdict(exited, facts) - if want is None: - assert got["status"] == "agent_error" and got["failure_cause"] != "model_rejected", (said, got) - else: - assert (got["failure_cause"], bool(got.get("halt"))) == (want, want == "usage_limit"), (said, got) - - -def test_a_login_trial_starts_no_container_once_another_trial_ended_the_run_while_it_waited(tmp_path, fake_harbor, - monkeypatch): - """A login trial may wait long for the plan, or pause before it runs again: another rollout that ended the run - meanwhile (a lost sign-in: HarborAgent.end_run) ends it too (LoginFailed) before it starts a container, as it ends - every rollout after it.""" - from hermit.agent.base import login - monkeypatch.setenv("FAKE_STREAMS", str(write(tmp_path / "unserved.jsonl", UNSERVED))) # each run's, counted - monkeypatch.setenv("FAKE_HARBOR_MODE", "agent_exit") - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - chatgpt_login(tmp_path) - lost = {"failure_cause": "login_failed", "exception_type": "AgentAuthenticationError", - "exception_message": "the sign-in was lost"} - for ended_at in (1, 2): # the wait before the first run, the one after the pause before another - jobs, waits = tmp_path / f"jobs-{ended_at}", [] - agent = make_agent(tmp_path, fake_harbor, tasks, binding=None, model="gpt-5.6-luna", login="chatgpt", - busy_retries=3, busy_wait_s=0, jobs_dir=str(jobs)) - - def wait(plan, cancel, log, max_s, credits, agent=agent, waits=waits, ended_at=ended_at): - waits.append(plan) - if len(waits) == ended_at: # another rollout's lost sign-in, as this one waits - with contextlib.suppress(LoginFailed): - agent.end_run(lost, jobs / "another") - return "" - monkeypatch.setattr(login, "wait", wait) - with pytest.raises(LoginFailed, match="the sign-in was lost"): - agent.run(row, None) - ran = (jobs / ".fake-calls").read_text() if (jobs / ".fake-calls").exists() else "0" - assert (ran, len(waits)) == (str(ended_at - 1), ended_at), ended_at - - -def test_a_login_trials_busy_settings_given_as_null_take_their_defaults(tmp_path, fake_harbor, monkeypatch): - """busy_retries and busy_wait_s left empty (null) are their defaults (the session runs again, 120 s later), not an - error in every rollout.""" - slept = [] - monkeypatch.setattr(HarborAgent, "_pause", lambda self, seconds: slept.append(seconds)) - monkeypatch.setenv("FAKE_STREAMS", f"{write(tmp_path / 'unserved.jsonl', UNSERVED)}," - f"{write(tmp_path / 'served.jsonl', CODEX_STREAM)}") - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - chatgpt_login(tmp_path) - out = make_agent(tmp_path, fake_harbor, tasks, binding=None, model="gpt-5.6-luna", login="chatgpt", - busy_retries=None, busy_wait_s=None).run(row, None) - assert out.output["status"] == "ok" and slept == [120.0] - assert (tmp_path / "jobs" / ".fake-calls").read_text() == "2" - - -def test_the_disk_gate_waits_for_the_hosts_tmp_too(tmp_path, monkeypatch): - from hermit.harbor import runner as runner_mod - free = {runner_mod.HOST_TMP: 0.5, tmp_path: 100.0} - monkeypatch.setattr(docker, "free_gb", lambda root: free[root]) - monkeypatch.setattr(docker.time, "sleep", lambda s: None) - r = runner_mod.HarborAgent.__new__(runner_mod.HarborAgent) - r.params, r._root, r.pulls, r.image_sizes = {"disk_gate_wait_s": 0.01}, tmp_path, docker.Pulls(), {} - r._stop, r._stop_why = threading.Event(), "" - assert not r.wait_for_disk() # /tmp under 1 GB: no trial starts - assert r._disk_held() == "the host's /tmp stayed under 1 GB free" # the trial's verdict says which gate - free[runner_mod.HOST_TMP] = 2.0 - assert r.wait_for_disk() - free[tmp_path] = 5.0 # the data root under its 20 GB - assert not r.wait_for_disk() and r._disk_held() == f"the Docker data root {tmp_path} stayed under 20 GB free" - free[tmp_path] = 100.0 - r.params["tmp_gate_gb"] = 0 # off - free[runner_mod.HOST_TMP] = 0.0 - assert r.wait_for_disk() - r._stop_why = "a Harbor trial (alpha-1): login_failed" # another rollout ended the run - r._stop.set() - with pytest.raises(LoginFailed, match="login_failed"): - r.wait_for_disk() - - -def test_the_disk_gate_keeps_room_for_images_still_to_be_pulled(monkeypatch): - present = set() - monkeypatch.setattr(docker, "free_gb", lambda root: 30.0) - monkeypatch.setattr(docker, "image_size", lambda image: 1 if image in present else None) - pulls, first, second = docker.Pulls(), [], [] - gate = dict(gate_gb=5, wait_s=0, poll_s=0, log=lambda m: None, pulls=pulls) - assert docker.wait_for_disk(Path("/"), image="big:1", image_gb=20, admitted=first, **gate) - assert pulls.taken_gb() == 20 and len(first) == 1 - assert not docker.wait_for_disk(Path("/"), image="big:2", image_gb=20, admitted=second, **gate) # 10 left - assert docker.wait_for_disk(Path("/"), image="small:1", image_gb=2, admitted=second, **gate) # 10 >= 7 - present.add("big:1") # pulled: the free space shows it now, not the book - assert pulls.taken_gb() == 2 - pulls.done(first[0]) - pulls.done(second[0]) - assert pulls.taken_gb() == 0 and docker.wait_for_disk(Path("/"), image="big:2", image_gb=20, **gate) - assert docker.wait_for_disk(Path("/"), image="big:1", image_gb=20, **gate) # here already: the gate alone - - -def test_a_trial_whose_agent_ran_out_of_its_time_and_whose_verifier_ran_no_test_is_infra(tmp_path, fake_harbor, - monkeypatch): - """Harbor verifies a trial whose agent ran out of its time and grades its reward: when the verifier's tools never - arrived (GitHub's 504), its tests never ran, and the trial is infra (run again), as one the agent finished is; not - the agent's 0.""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - monkeypatch.setenv("FAKE_HARBOR_MODE", "timeout_no_tools") - out = make_agent(tmp_path, fake_harbor, tasks).run(row, None) - assert (out.output["status"], out.output["failure_cause"], out.extra["unscored"]) == \ - ("infra", "verifier_failed", "verifier_failed") - - -def test_a_trial_whose_environment_never_started_spent_nothing(tmp_path, fake_harbor, monkeypatch): - """Harbor records no agent start (`agent_execution`) for a trial whose environment or agent setup failed first: no - session ran, so the trial costs 0, never an unknown cost that would leave an eval's total unknown.""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - monkeypatch.setenv("FAKE_HARBOR_MODE", "no_environment") - out = make_agent(tmp_path, fake_harbor, tasks).run(row, None) - assert (out.output["status"], out.output["failure_cause"]) == ("infra", "sandbox_failed") - assert (out.cost_usd, out.tokens) == (0.0, 0) - - -def test_a_host_failure_is_infra(tmp_path, fake_harbor, monkeypatch): - """This machine's failure past the steps that guard it (no process left to start `harbor`: EAGAIN) is infra - (host_failed: run again, its cost unknown), not the agent's error scored 0.""" - import subprocess - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - agent = make_agent(tmp_path, fake_harbor, tasks) - - def eagain(*args, **kwargs): - raise BlockingIOError(11, "Resource temporarily unavailable") - with monkeypatch.context() as patched: - patched.setattr(subprocess, "Popen", eagain) - out = agent.run(row, None) - assert (out.output["status"], out.output["failure_cause"], out.cost_usd) == ("infra", "host_failed", None) - assert out.extra.get("unscored") and agent.run(row, None).output["status"] == "ok" - - -def test_a_step_whose_stream_aborted_before_its_turn_is_never_asked_for(tmp_path, monkeypatch): - """An aborted stream's driver refuses every step, the one at its turn too: the wait ends before the step is asked - for (a step never asked for ran no session: it costs 0), never on the turn it would only be refused at.""" - client = stream.StreamClient(tmp_path / "flow__s1", python="unused") - monkeypatch.setattr(stream.StreamClient, "status", - lambda self: {"next_index": 2, "running": False, "aborted": "step 1 missed min_reward"}) - with pytest.raises(stream.StreamError, match="aborted"): - client.wait_turn(2, timeout_s=5) - - -def test_a_halt_the_rollout_met_ends_the_run_when_this_machines_failure_follows_it(tmp_path, fake_harbor, - monkeypatch): - """A rollout that met a halt (an API account's limit) and then this machine's OSError (its curated log not - written: a full disk) still ends the run (LoginFailed, through hermit.trainer.rollout.run_one), rather than every - later trial starting on the dead account. A jobs directory this machine cannot make is infra (host_failed) before - anything ran: it costs 0.""" - from hermit.harbor import runner as runner_mod - from hermit.trainer.rollout import run_one - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - row = dataset.index(tasks)[0] - monkeypatch.setenv("FAKE_HARBOR_MODE", "usage_limit") - - def full(*args, **kwargs): - raise OSError(28, "No space left on device") - with monkeypatch.context() as patched: - patched.setattr(runner_mod.curate_mod, "curate", full) - agent = make_agent(tmp_path, fake_harbor, tasks) - with pytest.raises(LoginFailed, match=r"model_provider_failed \(ApiUsageLimitError\)"): - run_one(row, None, agent.run, judge.score) - assert agent._stop.is_set() - blocked = tmp_path / "a-file" - blocked.write_text("") - agent = make_agent(tmp_path, fake_harbor, tasks) - agent.params["jobs_dir"] = str(blocked / "jobs") - out = agent.run(row, None) - assert (out.output["status"], out.output["failure_cause"], out.cost_usd) == ("infra", "host_failed", 0.0) - - -def test_the_words_of_a_full_disk_in_tests_that_ran_are_their_output_not_the_hosts(tmp_path): - """A verifier's output that says the disk was full is the host's failure (infra) only before its tests wrote a - report: with ctrf.json there the tests ran, and the words are a test's (an agent's program that printed them, a - task about a full disk), so the trial stays the agent's result, timed out or failed alike.""" - assert errors.verifier_disk_full("OSError: [Errno 28] No space left on device\n") - assert not errors.verifier_disk_full("OSError: [Errno 28] No space left on device\n", report_exists=True) - trial = tmp_path / "t__abc1234" - write(trial / "verifier" / "test-stdout.txt", "FAILED test_writes - OSError: [Errno 28] No space left on device\n") - write(trial / "verifier" / "reward.txt", "0\n") - for verdict in (errors.classify(trial_result(0.0)), errors.classify(trial_result(0.5, exc="AgentTimeoutError"))): - assert HarborAgent.verifier_verdict(verdict, trial)["status"] == "infra" - write(trial / "verifier" / "ctrf.json", '{"results": {"summary": {"tests": 1, "failed": 1}}}') - assert HarborAgent.verifier_verdict(verdict, trial) == verdict - (trial / "verifier" / "ctrf.json").unlink() - - -def test_the_task_sets_link_is_followed_once_so_a_prepare_mid_run_changes_no_trials_set(tmp_path, fake_harbor): - """`tasks_dir` names the link prepare writes (utils/tasks): the runner follows it once, so a prepare that points it - at another set mid-run changes none of the tasks the later trials run.""" - first, second = tmp_path / "set-1", tmp_path / "set-2" - for root, image in ((first, "example/alpha:1"), (second, "example/alpha:2")): - make_task(root, "alpha", image=image) - link = tmp_path / "utils" / "tasks" - link.parent.mkdir() - link.symlink_to(first, target_is_directory=True) - row = dataset.index(link)[0] - agent = make_agent(tmp_path, fake_harbor, link) - link.unlink() - link.symlink_to(second, target_is_directory=True) # another prepare, mid-run - assert agent.tasks_dir == first.resolve() and agent.task_dir(row) == first.resolve() / "alpha" - assert make_agent(tmp_path, fake_harbor, link).task_dir(row) == second.resolve() / "alpha" - - -def test_a_secret_in_the_acting_agents_env_is_refused_at_startup(tmp_path, fake_harbor): - """acting_agent.env is written into each trial's job and onto `docker compose exec`'s command line: a variable of - a secret's name, or a value naming one, is refused at startup, for a table CLI and Harbor's own agents alike: as - `${NAME}` or `${NAME:-default}`, which Harbor fills in from this process's environment, or as `$NAME`, which it - hands on as that text. A secret's name is one Harbor treats as one (KEY, SECRET, TOKEN but not TOKENS, PASSWORD, - CREDENTIAL or AUTH anywhere in it: TOKENIZERS_PARALLELISM and GIT_AUTHOR_NAME too) or one Hermit does (DB_PASSWD); - a count named like one (MAX_OUTPUT_TOKENS) is none, nor is a value naming one.""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - for spec in ({"cli": "codex", "env": {"OPENAI_API_KEY": "sk-stand-in"}}, - {"cli": "codex", "env": {"GITHUB": "${GH_TOKEN}"}}, - {"cli": "codex", "env": {"GITHUB": "${GH_TOKEN:-}"}}, - {"cli": "codex", "env": {"GITHUB": "$GH_TOKEN"}}, - {"cli": "codex", "env": {"AUTH": "Bearer ${HF_TOKEN:-none}"}}, - {"cli": "codex", "env": {"PARALLEL": "${TOKENIZERS_PARALLELISM:-false}"}}, - {"cli": "codex", "env": {"GIT_AUTHOR_NAME": "me"}}, - {"cli": "codex", "env": {"DB": "${DB_PASSWD}"}}, - {"name": "claude-code", "env": {"ANTHROPIC_API_KEY": "stand-in"}}): - with pytest.raises(ValueError, match=r"acting_agent.env \w+ holds a secret"): - make_agent(tmp_path, fake_harbor, tasks, acting_agent=spec) - env = {"CLAUDE_CODE_MAX_OUTPUT_TOKENS": "8000", "A": "1", "LIMIT": "$MAX_OUTPUT_TOKENS"} - agent = make_agent(tmp_path, fake_harbor, tasks, acting_agent={"cli": "codex", "env": env}) - assert agent.agent_config(dataset.index(tasks)[0])["env"] == env - - -def test_a_table_clis_trial_of_a_model_a_cli_reaches_is_refused_for_what_it_is(tmp_path, fake_harbor): - """A model a CLI reaches is the answer agent's: a codex trial of one the runner is given (model: {cli: ...}) is - refused saying so; one a run config's loading served (its model the endpoint's spec, on the loopback) is refused - for the loopback, which a container cannot reach, the message naming such a model too.""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - with pytest.raises(ValueError, match="is a model the answer agent reaches through a CLI"): - make_agent(tmp_path, fake_harbor, tasks, binding=None, model={"cli": "codex", "model": "gpt-5.6-luna"}) - with pytest.raises(ValueError, match=r"loopback.*a model a CLI reaches \(model: \{cli: \.\.\.\}"): - make_agent(tmp_path, fake_harbor, tasks, binding={**BINDING, "base_url": "http://127.0.0.1:8123/v1"}) - - -def test_a_sign_in_that_cannot_be_taken_back_leaves_the_trial_and_no_copy_in_the_job(tmp_path, monkeypatch, capsys): - """The ChatGPT sign-in a trial's Codex refreshed, when the shared copy's folder cannot take it (an OSError: a full - disk), is lost with a warning: the trial stands, and no copy of a sign-in is left in its job.""" - from datetime import datetime, timedelta, timezone - from types import SimpleNamespace - from hermit.agent.base import login as subscription - from hermit.harbor import launch - job = tmp_path / "job" - when = (datetime.now(timezone.utc) - timedelta(hours=1)).isoformat() - refreshed = json.dumps({"last_refresh": when, "tokens": {"refresh_token": "rt-stand-in", "access_token": "at"}}) - for folder in (launch.RETURN_DIR, launch.HOME_DIR): - write(job / launch.LAUNCH_DIR / folder / subscription.FILE, refreshed) - - def full(*args, **kwargs): - raise OSError(28, "No space left on device") - monkeypatch.setattr(subscription, "take_back", full) - agent = SimpleNamespace(login="chatgpt", login_root=tmp_path / "login", params={}) - HarborAgent.sign_in_back(agent, job) - assert "could not be taken back" in capsys.readouterr().err - assert not list(job.rglob(subscription.FILE)) - - -# ---- a trial's sign-in: what Hermit's side reads of it (hermit.harbor.signin) ------------------------------------ - -def test_a_codex_streams_lost_sign_in_is_codexs_own_word_since_the_session_last_worked(): - """signin.codex_signin_lost: Codex's report of a sign-in or a key the provider refused (CODEX_AUTH) in a line of - its own or an error since the session last worked, or a 403 in the error its turn failed with; never a recovery - notice Codex went on from (its reconnect), a 403 anywhere else, a refusal the session worked after, what Codex's - log quotes of the session's tool calls, or a line another program's log record that follows a record of Codex's.""" - from hermit.harbor import signin - start = '{"type":"thread.started","thread_id":"th-1"}\n{"type":"turn.started"}\n' - error = '{"type":"error","message":"%s"}\n' - failed = '{"type":"turn.failed","error":{"message":"%s"}}\n' - worked = '{"type":"item.completed","item":{"id":"i1","type":"agent_message","text":"working"}}\n' - lost = (start + "Error: Not logged in\n", # codex exec's own line - start + failed % "unexpected status 401 Unauthorized: Your token has been invalidated.", - start + error % "Your access token could not be refreshed. Please log out and sign in again." - + failed % "stream disconnected before completion", - start + failed % "unexpected status 403 Forbidden", - start + "2026-11-12T01:02:03.456789Z ERROR codex_core::client: Invalid API key\n" + failed % "exit") - kept = (start + error % "Reconnecting... 1/5 (unexpected status 401 Unauthorized: Missing bearer)" + worked, - start + error % "Reconnecting... 2/5 (unexpected status 403 Forbidden)" + failed % "stream disconnected", - start + error % "unexpected status 403 Forbidden" + failed % "stream disconnected", - start + error % "unexpected status 401 Unauthorized" + worked + failed % "the context window ran out", - start + "2026-11-12T01:02:03.456789Z ERROR codex_core::tools::router: error=apply_patch verification " - "failed: Failed to find expected lines in /app/auth.py:\n return jsonify(error=\"Invalid API key\"), 401\n" - + failed % "exit", - start + "2026-11-12T01:02:03.456789Z ERROR codex_exec: a step failed\n" - "2026-11-12T01:02:03Z ERROR server::auth: Invalid API key\n" + failed % "exit") - assert [signin.codex_signin_lost(raw) for raw in lost] == [True] * len(lost) - assert [signin.codex_signin_lost(raw) for raw in kept] == [False] * len(kept) - - -def test_a_codex_streams_provider_outage_is_its_last_failure_or_a_retry_it_was_still_making(tmp_path): - """signin.codex_unavailable: the provider could not serve a Codex session that served no turn when its last - failure says so (the error its turn failed with, else its last error since it last worked), when its turn failed - with the error Codex was retrying (its retries run out) or the stream ended while it retried, or when no session - began and a line says so (docker's own error); never a reconnect Codex went on from before the session failed on - its own (its retried request got through: the session's own error came after it), which the Base Agent's reading - would take for an outage, nor a response the provider's content filter ended (Codex writes it as a disconnected - stream): the agent's result.""" - from hermit.harbor import signin - start = '{"type":"thread.started","thread_id":"th-1"}\n{"type":"turn.started"}\n' - error = '{"type":"error","message":"%s"}\n' - failed = '{"type":"turn.failed","error":{"message":"%s"}}\n' - worked = '{"type":"item.completed","item":{"id":"i1","type":"agent_message","text":"working"}}\n' - routing = "workspace routing discovery failed" - out = (UNREACHABLE, - start + failed % "stream disconnected before completion: error sending request", - start + worked + error % "Reconnecting... 3/5 (stream disconnected before completion)", - start + error % f"Reconnecting... 5/5 ({routing})" + error % routing + failed % routing, - start + error % "Reconnecting... 2/5 (unexpected status 401 Unauthorized)", # it ended as Codex retried - start + error % "Reconnecting... 2/5" + failed % "exit", # no word of what it retried - "docker: Error response from daemon: no such image.\nSee 'docker run --help'.\n") - own = (start + error % "Reconnecting... 1/5 (stream disconnected before completion)" + worked - + failed % "Your input exceeds the context window of this model.", - start + error % "Reconnecting... 1/5 (stream disconnected before completion)" - + error % "Your input exceeds the context window of this model." - + failed % "Your input exceeds the context window of this model.", - start + error % f"Reconnecting... 5/5 ({routing})" + failed % "the agent stopped", - start + failed % ("stream disconnected before completion: Incomplete response returned, reason: " - "content_filter"), - start + error % "Reconnecting... 1/5 (stream disconnected before completion: Incomplete response returned, " - "reason: content_filter)" + failed % "stream disconnected before completion: Incomplete response returned, " - "reason: content_filter", - start + worked + failed % "the agent stopped") - assert [signin.codex_unavailable(raw) for raw in out] == [True] * len(out) - assert [signin.codex_unavailable(raw) for raw in own] == [False] * len(own) - # the Base Agent's own reading takes the first of `own` for an outage: why the runner reads it here - from hermit.agent.base.cli import Codex - stream = write(tmp_path / "agent-codex.jsonl", own[0]) - assert Codex({}).outcome(own[0], stream, None)["unavailable"] is True - - -def test_a_codex_streams_usage_limit_and_refused_configuration_are_codexs_own_words(tmp_path): - """signin.codex_refused and signin.codex_rejected read Codex's own notices only: the plan's usage limit (the Base - Agent's words) in a line of Codex's or an error since the session last worked, and a refusal of the run's - configuration or bill in its last failure (a 400, 402 or 404 Codex does not retry, an API account's quota); never - the words in what Codex's log quotes of the session's tool calls, a line of which may be a JSON object, which the - Base Agent's reading takes for its notices (and its JSON line for an event that ends the record).""" - from hermit.harbor import signin - start = '{"type":"thread.started","thread_id":"th-1"}\n{"type":"turn.started"}\n' - error = '{"type":"error","message":"%s"}\n' - failed = '{"type":"turn.failed","error":{"message":"%s"}}\n' - worked = '{"type":"item.completed","item":{"id":"i1","type":"agent_message","text":"working"}}\n' - record = "2026-11-12T01:02:03.456789Z ERROR codex_core::tools::router: error=apply_patch verification failed:\n" - limit = "You've hit your usage limit. Upgrade to Pro or try again in 2 hours." - quota = "Quota exceeded. Check your plan and billing details." - assert signin.codex_refused(start + error % limit + failed % limit) - assert signin.codex_refused(start + "Error: " + limit + "\n") - assert not signin.codex_refused(start + error % limit + worked + failed % "the context window ran out") - quoted = start + record + '{"note": "a JSON line of the patch"}\n' + f" say('{limit}')\n" + failed % "exit" - assert not signin.codex_refused(quoted) - assert signin.codex_rejected(start + failed % quota) == quota - for status in ("400 Bad Request: Unsupported parameter: 'max_tokens'", "402 Payment Required", - "404 Not Found: The model `gpt-x` does not exist"): - raw = start + error % f"unexpected status {status}" + failed % f"unexpected status {status}" - assert signin.codex_rejected(raw) == f"unexpected status {status}" - for raw in (start + failed % "unexpected status 401 Unauthorized", - start + failed % "unexpected status 429 Too Many", - start + error % "unexpected status 404 Not Found" + worked + failed % "the agent stopped", - start + record + f" raise SystemExit('{quota}')\n" + failed % "exit"): - assert signin.codex_rejected(raw) is None - # a provider's error body Codex gives as it is (a 400 it reads as an invalid request), and its words for a plan - # that refuses every session until someone acts: refusals too, of the configuration (model_rejected) or of the - # bill, which end the run (usage_limit, a halt); a body that refused the request's input is the agent's result - exited = errors.classify(trial_result(0.0, exc="NonZeroAgentExitCodeError")) - cases = {'{"error":{"message":"Unsupported value: \'xhigh\' is not supported with the \'gpt-5-mini\' model.",' - '"type":"invalid_request_error","param":"reasoning.effort","code":"unsupported_value"}}': "model_rejected", - '{"detail":"The \'gpt-x\' model is not supported when using Codex with a ChatGPT account."}': - "model_rejected", - "Your workspace is out of credits. Add credits to continue.": "usage_limit", - "You hit your spend cap set in your workspace. Increase your spend cap to continue.": "usage_limit", - "To use Codex with your ChatGPT plan, upgrade to Plus: https://chatgpt.com/explore/plus.": "usage_limit", - '{"error":{"message":"Your input exceeds the context window of this model.",' - '"type":"invalid_request_error","code":"context_length_exceeded"}}': "agent"} - for said, want in cases.items(): - raw = start + json.dumps({"type": "turn.failed", "error": {"message": said}}) + "\n" - refusal = signin.codex_rejected(raw) - assert refusal == said, said - got = HarborAgent.session_verdict(exited, {"served": False, "rejected": True, "failure": refusal, - "started": True, "tokens": 0}) - if want == "agent": - assert got["status"] == "agent_error" and got["failure_cause"] != "model_rejected", (said, got) - else: - assert (got["failure_cause"], got["halt"]) == (want, want == "usage_limit"), (said, got) - assert not signin.codex_refused(start + failed % "Your workspace is out of credits. Add credits to continue.") - # the Base Agent's own reading takes the quoted limit for the plan's refusal - from hermit.agent.base.cli import Codex - stream = write(tmp_path / "agent-codex.jsonl", quoted) - assert Codex({}).outcome(quoted, stream, None)["refused"] is True - - -def test_a_returned_sign_in_becomes_the_shared_copy_only_as_a_small_regular_chatgpt_sign_in_on_a_clock_it_trusts( - tmp_path, monkeypatch): - """signin.take_back offers the sign-in a container handed back to login.take_back only as a regular file (never - through a link) of at most MAX_BYTES holding a refresh and an access token, stamped at most SKEW past this - machine's clock in Harbor's local Docker environment (OTHER_CLOCK in another), read once into a private copy that - is removed after.""" - from datetime import datetime, timedelta, timezone - from hermit.agent.base import login - from hermit.harbor import signin - offered = [] - monkeypatch.setattr(login, "take_back", lambda where, path, within=None: offered.append( - json.loads(Path(path).read_text())["tokens"]["refresh_token"])) - now = datetime.now(timezone.utc) - - def sign_in(when, refresh, access="at-stand-in"): - return json.dumps({"last_refresh": when.isoformat(), "tokens": {"refresh_token": refresh, - "access_token": access}}) - - def back(name, text=None, link=None, local=True): - folder = tmp_path / name / "launch" / "return" - folder.mkdir(parents=True) - if link is not None: - (folder / signin.FILE).symlink_to(link) - else: - (folder / signin.FILE).write_text(text) - return signin.take_back(tmp_path / "login", folder, local=local) - assert back("refreshed", sign_in(now - timedelta(hours=1), "rt-refreshed")) - assert not back("ahead", sign_in(now + timedelta(minutes=10), "rt-ahead")) - assert back("elsewhere", sign_in(now + timedelta(minutes=10), "rt-elsewhere"), local=False) - assert not back("no-access", sign_in(now, "rt-no-access", access="")) - assert not back("large", sign_in(now, "rt-" + "x" * signin.MAX_BYTES)) - assert not back("link", link=write(tmp_path / "host" / signin.FILE, sign_in(now, "rt-host"))) - assert offered == ["rt-refreshed", "rt-elsewhere"] and not list(tmp_path.glob("*/launch/" + signin.TAKEN_DIR)) - - -# ---- a trial that is not the agent's result ------------------------------------------------------------------------ - -def test_a_trial_that_ended_infra_runs_again_and_one_that_stays_so_is_a_failed_run(tmp_path, fake_harbor, monkeypatch): - """A trial that is not the agent's result runs again, infra_retries times at most, each after infra_wait_s, what - every run used the trial's (extra.attempts). One that ends so every time is returned so, and the judge refuses to - score it: the run records a failed run (judge_error), never a 0 the agent earned. A run again that is the agent's - result stands; a cause no run again helps (a task Harbor runs for none of Hermit's agents) is not run again.""" - from hermit.trainer.rollout import run_one - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - mcp = make_task(tasks, "mcp") - (mcp / "task.toml").write_text((mcp / "task.toml").read_text() + '\n[[environment.mcp_servers]]\nname = "bank"\n' - 'transport = "sse"\nurl = "http://bank:8000/sse"\n') - rows = {r["harbor_task"]: r for r in dataset.index(tasks)} - paused = [] - monkeypatch.setattr(HarborAgent, "_pause", lambda self, seconds: paused.append(seconds)) - monkeypatch.setenv("FAKE_HARBOR_MODE", "no_tools") # every verifier's tools never arrive - agent = make_agent(tmp_path, fake_harbor, tasks, infra_retries=2, infra_wait_s=7) - record = run_one(rows["alpha"], None, agent.run, judge.score) - unscored_record(record, "verifier_failed") - assert paused == [7.0, 7.0] and len(record["pipeline_extra"]["attempts"]) == 3 - assert len(list((tmp_path / "jobs").glob("alpha-*"))) == 3 - - def recovered(self, seconds): # the verifier's tools arrive the second time - monkeypatch.setenv("FAKE_HARBOR_MODE", "ok") - monkeypatch.setattr(HarborAgent, "_pause", recovered) - again = run_one(rows["alpha"], None, make_agent(tmp_path / "again", fake_harbor, tasks, infra_retries=2, - infra_wait_s=0).run, judge.score) - assert (again["status"], again["judge"]["score"], len(again["pipeline_extra"]["attempts"])) == ("ok", 1.0, 2) - paused.clear() - monkeypatch.setattr(HarborAgent, "_pause", lambda self, seconds: paused.append(seconds)) - once = make_agent(tmp_path / "once", fake_harbor, tasks, infra_retries=2).run(rows["mcp"], None) - assert once.extra["unscored"] == "unsupported_task" and paused == [] and "attempts" not in once.extra - - -def test_a_cause_no_run_again_helps_ends_at_its_first_run(tmp_path, fake_harbor, monkeypatch): - """With runs again allowed (infra_retries 2), a trial whose cause runs no better again is not run again - (runner.NO_RETRY): a disk gate that waited its time out, and a request the provider refused for the run's - configuration (model_rejected: its key, its permission, its model, its bill); nor is a host failure past the - trial's staging (host_failed, whose session may have run: HarborAgent.run catches it outside the trial's runs). - Each is a failed run at once, its first run's; another infra cause (a session that never started) runs again.""" - from hermit.trainer.rollout import run_one - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - row = dataset.index(tasks)[0] - paused = [] - monkeypatch.setattr(HarborAgent, "_pause", lambda self, seconds: paused.append(seconds)) - agent = make_agent(tmp_path, fake_harbor, tasks, infra_retries=2, infra_wait_s=0) - with monkeypatch.context() as patch: - patch.setattr(HarborAgent, "wait_for_disk", lambda self, *args, **kwargs: False) - record = run_one(row, None, agent.run, judge.score) - unscored_record(record, "disk") - assert paused == [] and "attempts" not in record["pipeline_extra"] - runs = [] - - def once(self, task, adapter, started): # a trial's verdict, as the runner reads its session - runs.append(cause) - return self._output(task, errors.no_trial(cause, f"{cause}: a stand-in"), None, None, started) - monkeypatch.setattr(HarborAgent, "_trial_once", once) - for cause, times in (("model_rejected", 1), ("session_failed", 3)): - runs.clear() - record = run_one(row, None, agent.run, judge.score) - unscored_record(record, cause) - assert len(runs) == times, cause - assert paused == [0.0, 0.0] # session_failed's two runs again - launched = [] - - def no_fork(self, task, adapter, started): # a host failure past the staging: its session may have run - launched.append(task["id"]) - raise BlockingIOError(11, "Resource temporarily unavailable") - monkeypatch.setattr(HarborAgent, "_trial_once", no_fork) - record = run_one(row, None, agent.run, judge.score) - unscored_record(record, "host_failed") - assert launched == [row["id"]] and paused == [0.0, 0.0] # never run again - - -def test_the_runner_refuses_a_trial_cache_and_infra_settings_it_cannot_take(tmp_path, fake_harbor): - """There is no trial cache (no trial is taken from an earlier run's): `trial_cache` is refused, saying what a - resumed run runs instead, as is an infra retry count or wait that is no count or no number of seconds.""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - with pytest.raises(ValueError, match="there is no trial cache any more"): - make_agent(tmp_path, fake_harbor, tasks, trial_cache=True) - for bad, match in (({"infra_retries": -1}, "infra_retries is a count"), ({"infra_retries": "two"}, "a count"), - ({"infra_wait_s": -5}, "infra_wait_s is a number of seconds")): - with pytest.raises(ValueError, match=match): - make_agent(tmp_path, fake_harbor, tasks, **bad) - agent = make_agent(tmp_path, fake_harbor, tasks, infra_retries=None, infra_wait_s=None) - assert (agent.infra_retries, agent.infra_wait_s) == (2, 300.0) - - -def test_a_benchmark_files_model_is_a_name_or_one_a_cli_reaches(tmp_path, fake_harbor): - """The benchmark file names the methods' configs' model by name (the CLI's own, with a login; one of Harbor's own - agents') or as one a CLI reaches (model: {cli: ...}, which the runner serves itself, its login's plan the trials' - to wait for: plan_login); a / reference or another model mapping there is refused, as it belongs - in the run config's agent.params.model, which the config's loading binds and hermit check sees. A run config that - gives its own model never reads the file's.""" - tasks = tmp_path / "utils" / "tasks" - make_task(tasks, "alpha") - - def bench(model): - write(tmp_path / "utils" / "benchmark.yaml", yaml.safe_dump({"tasks_dir": "tasks", "web_search": False, - "model": model})) - return {"benchmark": "../utils/benchmark.yaml", "harbor": str(fake_harbor), "jobs_dir": str(tmp_path / "jobs"), - "infra_retries": 0, "disk_gate_gb": 0, "tmp_gate_gb": 0} - for model in ("openai/gpt-5.6-luna", {"model": "glm-5", "base_url": "https://models.example/v1"}): - with pytest.raises(ValueError, match="give it in the run config's own agent.params.model"): - HarborAgent(bench(model), base=tmp_path / "method") - given = HarborAgent({**bench("openai/gpt-5.6-luna"), "model": BINDING["model"], - "binding": BINDING}, base=tmp_path / "method") - assert given.params["model"] == BINDING["model"] and given.plan_login() is None - chatgpt_login(tmp_path) - signed_in = HarborAgent({**bench("gpt-5.6-luna"), "login": "chatgpt"}, base=tmp_path / "method") - assert signed_in.plan_login() == "chatgpt" - reached = {"cli": "codex", "model": "gpt-5.6-luna", "login": "chatgpt", "plan_dir": str(tmp_path / "plan")} - served = HarborAgent({**bench(reached), "acting_agent": {"cli": "answer"}, "web_search": None}, - base=tmp_path / "method") - assert served.cli_model["cli"] == "codex" and served.plan_login() == "chatgpt" - assert served.model_plan() == (tmp_path / "plan", False) - - -# ---- the report: pass@1 over the agent's results -------------------------------------------------------------------- - -def test_the_report_gives_pass_at_1_over_the_trials_that_are_the_agents_result(tmp_path, capsys): - """python -m hermit.harbor.report reads an eval's results.jsonl: of its trials, the ones the judge scored (the - agent's results), the ones that are not the agent's result (infra: the judge refused to score them) and the other - failed runs; pass@1 and the mean of the scored ones, and hermit eval's mean (every trial, the others as 0); by - repeat too.""" - from dataclasses import asdict - from hermit.harbor import report - from hermit.task import RunOutput, normalise_judge - - def judged(reward, status="ok"): - # the judge's record as hermit eval keeps it (hermit.trainer.rollout: asdict of normalise_judge) - output = RunOutput(output={"status": status, "reward": reward}, - extra={"judge": {"pass_reward": 1.0}}) - return asdict(normalise_judge(judge.score({"id": "t"}, output))) - failed = {"score": 0.0, "feedback": "the judge failed: Unscored: ..."} - rows = [{"repeat": 1, "id": "a", "status": "ok", "output": {"status": "ok"}, "judge": judged(1.0)}, - {"repeat": 1, "id": "b", "status": "ok", "output": {"status": "agent_error"}, - "judge": judged(0.0, "agent_error")}, - {"repeat": 1, "id": "c", "status": "judge_error", "output": {"status": "infra"}, "judge": failed}, - {"repeat": 2, "id": "a", "status": "error", "output": None, "judge": {"score": 0.0}}, - {"repeat": 2, "id": "b", "status": "ok", "output": {"status": "ok"}, "judge": judged(1.0)}] - assert [report.passed(r) for r in rows] == [True, False, None, None, True] - results = write(tmp_path / "eval" / "final" / "results.jsonl", "".join(json.dumps(r) + "\n" for r in rows)) - figures = report.report(results.parent) - assert [figures[k] for k in ("trials", "scored", "infra", "failed")] == [5, 3, 1, 1] - assert figures["pass_at_1"] == pytest.approx(2 / 3) and figures["mean_scored"] == pytest.approx(2 / 3) - assert figures["mean_all"] == pytest.approx(2 / 5) - assert figures["by_repeat"]["1"]["pass_at_1"] == 0.5 and figures["by_repeat"]["2"]["failed"] == 1 - assert report.main([str(results), "--json"]) == 0 - assert json.loads(capsys.readouterr().out) == {**figures, "results": str(results)} - assert report.main([str(results.parent)]) == 0 - out = capsys.readouterr().out - assert "5 trials: 3 scored, 1 infra, 1 failed; pass@1 66.7% of the scored" in out and "\n repeat 2: " in out - assert report.main([str(tmp_path / "none")]) == 1 - - -def test_the_report_says_when_an_eval_did_not_finish(tmp_path, monkeypatch, capsys): - """Through main's own `hermit eval` (hermit.evaluate.evaluate, on the toy task): a finished eval reads as finished, - and so does its copy whatever the files' times (a copy changes them). An eval of the same folder that a lost - sign-in stopped (LoginFailed) wrote no row of the repeat it stopped in, run_tasks handing back a repeat's records - once all are done, beside the summary of the eval before: the report says it did not finish (`finished` false, - `unfinished` why), as it does for rows with no summary beside them, for a summary that is not of its rows, and for - rows whose last one is cut off (an eval stopped as it wrote it), which is left out instead of failing the report.""" - import shutil - from hermit.agent.base.interface import LoginFailed - from hermit.config import Config - from hermit.evaluate import evaluate - from hermit.harbor import report - from hermit.trainer import rollout - from toy_task import make_task - cfg = Config.load(make_task(tmp_path), []) - evaluate(cfg, "none", log=lambda *_: None) - folder = cfg.run_dir() / "eval" / "none" - rows, summary = report.rows_of(folder), (folder / "summary.json").read_text() - finished = report.report(folder) - assert (finished["finished"], finished["trials"]) == (True, 3) and "unfinished" not in finished - assert "note" not in finished # in place: the summary written after the rows - copy = shutil.copytree(folder, tmp_path / "handed-back") - os.utime(copy / "summary.json", (time.time() - 3600, time.time() - 3600)) # an hour older than the rows - handed = report.report(copy) - assert handed["finished"] is True and "unless the folder was copied" in handed["note"] # noted, never unfinished - assert report.main([str(copy)]) == 0 and "\n note: its results.jsonl was written" in capsys.readouterr().out - (copy / "summary.json").unlink() - assert "no summary.json beside its rows" in report.report(copy)["unfinished"] - other = write(tmp_path / "other" / "results.jsonl", "".join( - json.dumps({**r, "judge": {**r["judge"], "score": 1.0 - r["judge"]["score"]}}) + "\n" for r in rows)) - write(other.parent / "summary.json", summary) # three trials, but another mean score - assert "is not of its rows" in report.report(other)["unfinished"] - failed = write(tmp_path / "failed" / "results.jsonl", "".join( # the same scores, but a failed run among them - json.dumps({**r, "status": "error" if n == 0 else r["status"]}) + "\n" for n, r in enumerate(rows))) - write(failed.parent / "summary.json", summary) - assert "another count of failed runs" in report.report(failed)["unfinished"] - text = (folder / "results.jsonl").read_text() # an eval stopped as it wrote its last row - cut = write(tmp_path / "cut" / "results.jsonl", text[:len(text) - 40]) - write(cut.parent / "summary.json", summary) - figures = report.report(cut.parent) - assert (figures["finished"], figures["trials"]) == (False, 2) and "is cut off" in figures["unfinished"] - assert report.main([str(cut.parent)]) == 0 and " unfinished: the last row" in capsys.readouterr().out - broken = write(tmp_path / "broken" / "results.jsonl", text.replace("\n", "\n{not json\n", 1)) - with pytest.raises(ValueError, match="line 2 of .* is no JSON row"): # a row of its own that is none: no cut - report.read_rows(broken) - - def lost(task, adapter, pipeline, judge): - raise LoginFailed("the sign-in was lost (a stand-in)") - monkeypatch.setattr(rollout, "run_one", lost) - with pytest.raises(LoginFailed): - evaluate(cfg, "none", log=lambda *_: None) - stopped = report.report(folder) - assert (stopped["finished"], stopped["trials"]) == (False, 0) # beside the eval before's summary - assert "counts 3 trials and its results.jsonl holds 0" in stopped["unfinished"] - assert report.main([str(folder)]) == 0 - assert "\n unfinished: its summary.json counts 3 trials" in capsys.readouterr().out - - -# ---- staging's child: the host's deployment with the tools the runner serves --------------------------------------- - -@pytest.mark.parametrize("host", ["pi", "dsh"]) -def test_staging_deploys_pi_and_dsh_without_the_tools_the_runner_serves(tmp_path, host, capsys): - """`python -m hermit.harbor.staging` (prepare_workspace's child, in the strict adapter sandbox) deploys the adapter - as the host's own deployment does, but for its tools, which the runner serves to the container itself: Pi's - .pi/mcp.json names no server, dsh's patch layer holds no server row (serve_tools=False), and its report is JSON.""" - from hermit.harbor import staging - adapter = tool_adapter(tmp_path / "adapter") - workspace = tmp_path / "w" - assert staging.main(["--host", host, "--adapter", str(adapter), "--workspace", str(workspace), - "--python", sys.executable]) == 0 - reported = json.loads(capsys.readouterr().out) - assert reported["context_injected_via"] == "AGENTS.md" and (workspace / "AGENTS.md").is_file() - if host == "pi": - assert json.loads((workspace / ".pi" / "mcp.json").read_text()) == {"mcpServers": {}} - else: - assert not [r for r in json.loads((workspace / ".dsh" / "hermit.patch.yml").read_text()) if "insert" in r] - - -# ---- the round-10 review: what tells the host's failure from the adapter's, and Codex's readings --------------------- - -def test_a_staging_child_that_failed_beside_this_machines_own_failure_is_infra_never_the_adapters_0( - tmp_path, fake_harbor, monkeypatch): - """A staging child's failure (its RuntimeError) is the adapter's unless this process sees this machine fail any - adapter's staging there: a job folder it cannot write, less than STAGING_FREE_MB free beside the trial's job, or the - strict adapter sandbox not starting (bubblewrap without its namespaces: sandbox_fault). Then the trial, and a stream's step, are infra - (staging_failed: run again), the child's own words kept in the message; the same failure beside room and a sandbox - that starts is the adapter's (adapter_failed, scored 0).""" - from collections import namedtuple - import shutil as shutil_mod - from hermit.harbor import runner as runner_mod - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - make_task(tasks, "flow", steps=("s1", "s2")) - rows = {r["id"]: r for r in dataset.index(tasks)} - agent = make_agent(tmp_path, fake_harbor, tasks) - - def stage(*args, **kwargs): - raise RuntimeError("deploying the adapter failed: OSError: [Errno 28] No space left on device") - monkeypatch.setattr("hermit.harbor.runner.stage", stage) - usage, real_usage, real_access = namedtuple("usage", "total used free"), shutil_mod.disk_usage, os.access - jobs = tmp_path / "jobs" - - def access(path, mode, **kwargs): # the job folder this process may not write - return False if Path(path).is_relative_to(jobs) and mode & os.W_OK else real_access(path, mode, **kwargs) - bwrap = "the strict adapter sandbox did not start (exit 1): bwrap: setting up uid map: Permission denied" - says = {"full": "10 MB free", "sandbox": "bwrap: setting up uid map", "unwritable": "cannot be written"} - for case, low, fault in (("full", True, None), ("sandbox", False, bwrap), ("unwritable", False, None), - ("adapter", False, None)): - monkeypatch.setattr(runner_mod.shutil, "disk_usage", - (lambda path: usage(1 << 40, 1 << 40, 10 << 20)) if low else real_usage) - monkeypatch.setattr(runner_mod, "sandbox_fault", lambda enforce, fresh=True, fault=fault: fault) - monkeypatch.setattr(runner_mod.os, "access", access if case == "unwritable" else real_access) - for row in (rows["alpha"], rows["flow:s1"]): - out = agent.run(row, simple_template()) - want = ("agent_error", "adapter_failed") if case == "adapter" else ("infra", "staging_failed") - assert (out.output["status"], out.output["failure_cause"]) == want, (case, row["id"]) - if case != "adapter": - message = out.extra["exception_message"] - assert says[case] in message, (case, message) - assert "No space left on device" in message, (case, message) # the child's own words, kept - assert not agent._in_flight - - -def test_a_runner_on_a_machine_whose_bubblewrap_cannot_start_its_sandbox_is_refused_at_startup( - tmp_path, fake_harbor, monkeypatch): - """bubblewrap installed but unable to start the strict adapter sandbox (no user namespaces, an AppArmor rule) - would fail every trial's staging: the runner is refused at startup, saying why and the way out (sandbox_enforce: - best_effort, with which it starts); once a sandbox started in this process, a later runner does not try again.""" - import shutil as shutil_mod - import subprocess as subprocess_mod - from hermit import sandbox as sandbox_mod - from hermit.harbor import runner as runner_mod - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - real_which, tries = shutil_mod.which, [] - monkeypatch.setattr(runner_mod.shutil, "which", - lambda name, *a, **k: "/usr/bin/bwrap" if name == "bwrap" else real_which(name, *a, **k)) - monkeypatch.setattr(runner_mod, "_SANDBOX_STARTED", []) - monkeypatch.setattr(runner_mod, "sandbox_fault", REAL_SANDBOX_FAULT) - - def refused(command, **kwargs): - tries.append(command) - return subprocess_mod.CompletedProcess(command, 1, "", "bwrap: setting up uid map: Permission denied\n") - monkeypatch.setattr(sandbox_mod, "run_in_strict_sandbox", refused) - with pytest.raises(ValueError, match=r"the strict adapter sandbox did not start \(exit 1\): bwrap: setting up uid " - r"map: Permission denied: every trial's staging would fail"): - make_agent(tmp_path, fake_harbor, tasks) - assert make_agent(tmp_path, fake_harbor, tasks, sandbox_enforce="best_effort").cli == "codex" - monkeypatch.setattr(sandbox_mod, "run_in_strict_sandbox", - lambda command, **kwargs: (tries.append(command), subprocess_mod.CompletedProcess( - command, 0, "", ""))[1]) - make_agent(tmp_path, fake_harbor, tasks) - make_agent(tmp_path, fake_harbor, tasks) - assert len(tries) == 2 # the refused try, then one that started: no more - - -def test_a_trials_launch_plan_refused_for_its_adapter_is_the_adapters_0_and_with_no_adapter_the_configs_error( - tmp_path, fake_harbor, monkeypatch): - """A launch plan refused at a trial for what its adapter is (dsh's MCP server name made of the adapter's id, over 32 - characters or with a '.') is the adapter's result, as a deployment of it that failed: adapter_failed, scored 0 with - the message as its feedback, for a trial and for a stream's step alike. With no adapter, the same plan rendered at - startup (check_agent), so its refusal then is the config's: a pipeline error, as before.""" - from hermit.trainer.rollout import run_one - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - make_task(tasks, "flow", steps=("s1", "s2")) - rows = {r["id"]: r for r in dataset.index(tasks)} - agent = make_agent(tmp_path, fake_harbor, tasks) - - def refuse(self, *args, **kwargs): - raise ValueError("dsh takes an MCP server name of 32 characters at most ([A-Za-z0-9_-]), not 'a.b'") - monkeypatch.setattr(HarborAgent, "_launch", refuse) - for row in (rows["alpha"], rows["flow:s1"]): - record = run_one(row, simple_template(), agent.run, judge.score) - assert (record["status"], record["output"]["status"], record["output"]["failure_cause"]) == \ - ("ok", "agent_error", "adapter_failed"), row["id"] - assert record["judge"]["score"] == 0.0 and "32 characters at most" in record["judge"]["feedback"] - assert not agent._in_flight and not list((tmp_path / "jobs").glob("alpha-*/alpha__*")) # no trial ran - with pytest.raises(ValueError, match="32 characters at most"): - agent.run(rows["alpha"], None) - - -def test_a_codex_session_that_failed_on_its_own_after_a_reconnect_it_went_on_from_is_the_agents_result( - tmp_path, fake_harbor, monkeypatch): - """A Codex session that met a provider's notice, reconnected (Codex went on: an item after it) and then failed on - its own (its turn failed, not on the provider) is the agent's result, scored and never run again: the runner reads - Codex's last failure (signin.codex_unavailable), where the Base Agent's reading of the whole stream (main's Codex - outcome) takes the reconnect's notice for an outage.""" - from hermit.agent.base.cli import Codex - raw = TIMED_OUT + '{"type":"turn.failed","error":{"message":"exit"}}\n' - assert outcome(Codex({}), raw, tmp_path / "read")["unavailable"] is True # main's reading: an outage - monkeypatch.setenv("FAKE_DEFAULT_STREAM", raw) - monkeypatch.setenv("FAKE_HARBOR_MODE", "agent_exit") # the CLI exited 1; the verifier ran - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - out = make_agent(tmp_path, fake_harbor, tasks, infra_retries=1, infra_wait_s=0).run(row, None) - assert (out.output["status"], "unscored" in out.extra, out.extra["session"]["unavailable"]) == \ - ("agent_error", False, False) - assert len(list((tmp_path / "jobs").glob("alpha-*"))) == 1 # not run again - - -def test_a_stream_given_up_twice_keeps_the_first_reason(tmp_path, fake_harbor): - """A stream two of its steps gave up (one ending before its driver, another as it waited) keeps the first reason, - which the steps after it report.""" - tasks = tmp_path / "tasks" - make_task(tasks, "flow", steps=("s1", "s2", "s3")) - agent = make_agent(tmp_path, fake_harbor, tasks) - agent._give_up("flow__s1__x", 2, "staging_failed") - agent._give_up("flow__s1__x", 3, "tools_failed") - assert agent._given_up["flow__s1__x"] == \ - "its step 2 ended before it reached the stream's driver (staging_failed)" - - -def test_curate_puts_at_the_top_the_stream_the_plan_names_never_another_clis_the_agent_wrote(tmp_path): - """The log's top stream is the one the trial's launch plan names (a Pi trial's agent-pi.jsonl): a file of another - CLI's name the session wrote in its agent folder (an agent-codex.jsonl) is never taken for it; a plan of the answer - agent, whose answer.json is no CLI's stream, puts none at the top.""" - trial = tmp_path / "trial" - write(trial / "agent" / "agent-pi.jsonl", '{"type":"session"}\n') - write(trial / "agent" / "agent-codex.jsonl", '{"type":"thread.started"}\n') # the session's own file - plan = write(tmp_path / "job" / "launch" / "plan.json", json.dumps({"stream": "agent-pi.jsonl"})) - log = curate.curate(trial, tmp_path / "job", errors.classify(trial_result()), plan=plan) - assert (log / "agent-pi.jsonl").read_text() == '{"type":"session"}\n' and not (log / "agent-codex.jsonl").exists() - answer = write(tmp_path / "answer" / "launch" / "plan.json", json.dumps({"stream": "answer.json"})) - log = curate.curate(trial, tmp_path / "answer", errors.classify(trial_result()), plan=answer) - assert not (log / "agent-pi.jsonl").exists() and not (log / "agent-codex.jsonl").exists() - - -# ---- the round-11 review: which disk gate held a trial back --------------------------------------------------------- - -def test_a_trial_or_step_the_disk_gate_held_back_names_the_gate_that_held_it(tmp_path, fake_harbor, monkeypatch): - """The verdict of a trial, or of a stream's first step, that a disk gate held back past its wait (disk: infra, not - run again) names the gate that held it: the host's /tmp (its own gate, waited for first) or Docker's data root. - Before, it named the data root whichever gate held it.""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - make_task(tasks, "flow", steps=("one", "two")) - rows = {r["id"]: r for r in dataset.index(tasks)} - root = tmp_path / "docker-root" - free = {} - monkeypatch.setattr(docker, "free_gb", lambda path: free.get(path, 100.0)) - monkeypatch.setattr(docker.time, "sleep", lambda s: None) - for case, (short, says) in enumerate(((runner_module.HOST_TMP, "the host's /tmp stayed under 1 GB free"), - (root, f"the Docker data root {root} stayed under 20 GB free"))): - free.clear() - free[short] = 0.5 - agent = make_agent(tmp_path / f"case-{case}", fake_harbor, tasks, disk_gate_gb=20, tmp_gate_gb=1, - disk_gate_wait_s=0.01) - agent._root = root - for row, then in ((rows["alpha"], "no container started"), - (rows["flow:one"], "the stream's environment was not started")): - out = agent.run(row, None) - assert (out.output["status"], out.extra["unscored"]) == ("infra", "disk"), row["id"] - assert out.extra["exception_message"] == f"{says}: {then}", (row["id"], out.extra["exception_message"]) - - -# ---- the round-12 review: a stream's next pass, a curated log's links, the providers' refusals ---------------------- - -def test_a_driver_that_is_ending_or_died_unreaped_is_no_live_driver_and_the_next_waits_for_its_exit(tmp_path, - monkeypatch): - """StreamClient.running takes no driver whose record says it is ending (harborside.driver's _mark_ending: its - step loop over, its socket open a moment more), nor one killed and not yet reaped (a zombie, which `alive` still - finds, beside the socket it left): the stream's next step starts a new driver instead of reusing it (a repeat's - whole pass lost: stream_order) or failing on it (stream_lost). Before a new driver starts, an earlier one still - there is waited for until it has exited (ENDING_WAIT_S), then killed.""" - import subprocess as subprocess_mod - if not Path("/proc/self/stat").exists(): - pytest.skip("no /proc to tell a zombie or a driver's command line") - sock = write(tmp_path / "d.sock", "") - # a process named as this directory's driver would be, leading its own process group - proc = subprocess_mod.Popen([sys.executable, "-c", "import time; time.sleep(60)", "harborside.driver", - "--stream-dir", str(tmp_path)], start_new_session=True) - try: - client = stream.StreamClient(tmp_path, python="unused", log=lambda m: None) - write(tmp_path / "driver.json", json.dumps({"pid": proc.pid, "socket": str(sock)})) - assert client.running() is not None # alive, its socket there: live - write(tmp_path / "driver.json", json.dumps({"pid": proc.pid, "socket": str(sock), "ending": True})) - assert client.running() is None and not client.gone() # ending: no live driver, its process still there - monkeypatch.setattr(stream, "ENDING_WAIT_S", 0.5) - client._wait_out() # never exits by itself: killed - assert proc.wait(timeout=30) is not None and client.gone() - finally: - proc.kill() - proc.wait() - zombie = subprocess_mod.Popen([sys.executable, "-c", "import time; time.sleep(60)"]) - try: - write(tmp_path / "driver.json", json.dumps({"pid": zombie.pid, "socket": str(sock)})) - assert client.running() is not None - zombie.kill() # not reaped: a zombie - deadline = time.time() + 10 - while not client.gone() and time.time() < deadline: - time.sleep(0.05) - assert client.gone() and client.running() is None - finally: - zombie.wait() - - -def test_a_step_whose_folder_cannot_be_made_gives_its_stream_up_and_the_next_pass_starts_anew(tmp_path, fake_harbor, - monkeypatch): - """A step whose own folder this machine cannot make (a full disk) ends host_failed before it reaches the driver, - which waits for it: the stream is given up and its driver stopped, so the stream's next pass (an eval's next - repeat) starts anew in an environment of its own, never running that step on the earlier pass's. Before, the - folder was made outside what gives a stream up: the next pass's first step met the old driver (stream_order) and - its last ran on the old environment, scored as the agent's.""" - import errno - fake_driver(fake_harbor) - started = driver_starts(monkeypatch) - tasks = tmp_path / "tasks" - make_task(tasks, "flow", steps=("one", "two")) - rows = {r["id"]: r for r in dataset.index(tasks)} - agent = make_agent(tmp_path, fake_harbor, tasks, stream={"idle_s": 60, "start_timeout_s": 30, "turn_wait_s": 6}) - real_mkdir = Path.mkdir - - def full(self, *args, **kwargs): - if self.parent.name == "steps" and "-two-" in self.name: - raise OSError(errno.ENOSPC, os.strerror(errno.ENOSPC), str(self)) - return real_mkdir(self, *args, **kwargs) - try: - assert agent.run(rows["flow:one"], None).output["status"] == "ok" - with monkeypatch.context() as patch: - patch.setattr(Path, "mkdir", full) - failed = agent.run(rows["flow:two"], None) - assert (failed.output["status"], failed.output["failure_cause"]) == ("infra", "host_failed") - again = [agent.run(rows[f"flow:{step}"], None) for step in ("one", "two")] # the stream's next pass - assert [o.output["status"] for o in again] == ["ok", "ok"] - assert started == [stream_name(agent)] * 2 # in a driver of its own - finally: - stop_streams(tmp_path) - - -def test_a_reward_or_the_copys_folder_swapped_for_a_link_after_its_checks_is_never_copied_through(tmp_path, monkeypatch): - """A verifier's reward file, and the agent's hermit-copy folder above its memory/, swapped for links to this - machine's files after the curation's checks (a process the agent left running): neither link is followed, the - reward opened at its own name (open_regular) and memory/ through the copy's folder (curate._subdir).""" - secret = write(tmp_path / "host" / "auth.json", "the machine's own secret\n") - write(tmp_path / "host" / "memory" / "notes.md", "the machine's own notes\n") - outputs = tmp_path / "step" - (outputs / "verifier").mkdir(parents=True) - os.symlink(secret, outputs / "verifier" / "reward.txt") - (outputs / "agent").mkdir() - os.symlink(tmp_path / "host", outputs / "agent" / "hermit-copy") - job = tmp_path / "job" - job.mkdir() - monkeypatch.setattr(Path, "is_symlink", lambda self: False) # the checks, made before the swap - log = Path(curate.curate(outputs, job, errors.classify(trial_result()))) - monkeypatch.undo() - assert not [p for p in log.rglob("*") if p.is_file() and "machine's own" in p.read_text()] - - -@pytest.mark.parametrize("swap,after", [("link", 1), ("gone", 1), ("gone", 2), ("gone", 3)]) -def test_a_curated_log_never_copies_through_a_link_swapped_in_while_it_walks(tmp_path, monkeypatch, swap, after): - """A process the agent left running (a stream's environment stays up between steps) may swap a file of the step's - agent/ folders for a link to a file of this machine, or remove it, while the step's log is curated: each entry is - opened through its folder, never through a link at its name (curate._files, open_regular), so a link swapped in - after the walk's checks is left out, not copied through, and a file gone meanwhile (after any of the checks made of - it) is left out, the curation going on (no OSError that would make the step this machine's failure). Before, the - checks of each file were followed by a copy that followed a link, and a file gone between them raised.""" - secret = write(tmp_path / "host" / "auth.json", "the machine's own secret\n") - outputs = tmp_path / "step" - victims = [write(outputs / "agent" / folder / f"a{n}.json", '{"ok": 1}\n') - for n, folder in enumerate(("hook-state", "hermit-collect", ".hermit", "hermit-copy/memory"))] - victims.append(write(outputs / "agent" / "hook-calls.jsonl", "{}\n")) - for folder in ("hook-state", "hermit-collect", ".hermit", "hermit-copy/memory"): - write(outputs / "agent" / folder / "kept.json", '{"kept": 1}\n') - real_stat, checks = os.stat, {str(v): 0 for v in victims} - - def stat_then_move(path, *args, **kwargs): # the container's move, right after a check of the file - got = real_stat(path, *args, **kwargs) - if isinstance(path, int): - return got - name = os.fsdecode(path) - if kwargs.get("dir_fd") is not None: - name = os.path.join(os.readlink(f"/proc/self/fd/{kwargs['dir_fd']}"), name) - name = os.path.abspath(name) - if name in checks: - checks[name] += 1 - if checks[name] == after: - os.unlink(name) - if swap == "link": - os.symlink(secret, name) - return got - if not Path("/proc/self/fd").is_dir(): - pytest.skip("no /proc to name a folder by its descriptor") - monkeypatch.setattr(os, "stat", stat_then_move) - try: - log = curate.curate(outputs, tmp_path / "job", errors.classify(trial_result())) - finally: - monkeypatch.setattr(os, "stat", real_stat) - assert any(checks.values()) # the curation checked the files (moved after that) - copied = {p.relative_to(log).as_posix(): p.read_text() for p in log.rglob("*") if p.is_file()} - assert not any("own secret" in text for text in copied.values()), copied - assert {"hook-state/kept.json", "agent/hermit-collect/kept.json", "agent/.hermit/kept.json", - "memory/kept.json"} <= set(copied) # the walk went on - - -def test_a_benchmark_files_cli_model_without_a_login_is_refused_at_startup(tmp_path, fake_harbor): - """The benchmark file's `model: {cli: ...}` is served as it is written (tools.ModelEndpoint): no config's loading - binds a `/` reference there, as it binds `llm`'s, so one without a login (Pi's, which has none, - among them) would run sessions with no key, endpoint or host they may reach; it is refused at startup. On the - CLI's own login it runs.""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - for model in ({"cli": "codex", "model": "openai/gpt-5.6-luna"}, {"cli": "pi", "model": "openai/gpt-5.6-luna"}, - {"cli": "claude", "model": "anthropic/claude-haiku-5-5"}): - with pytest.raises(ValueError, match="with no login"): - cli_answer_agent(tmp_path, fake_harbor, tasks, model) - chatgpt_login(tmp_path) - assert cli_answer_agent(tmp_path, fake_harbor, tasks, {"cli": "codex", "model": "gpt-5.6-luna", - "login": "chatgpt"}).cli_model["login"] == "chatgpt" - - -def test_a_login_in_the_acting_agents_settings_is_refused_and_a_relative_config_is_the_params_path(tmp_path, - fake_harbor): - """`acting_agent.settings.login` would sign the session in (launch.render_plan) where the runner, which reads - agent.params.login alone, neither checks it nor hands the trial its sign-in, plan and refusals: refused at startup. - mini-swe-agent's settings.config, when relative, is read beside the params (the runner's other paths), as main - reads agent.params.config beside the run config, never from the working directory.""" - tasks = tmp_path / "tasks" - make_task(tasks, "alpha") - with pytest.raises(ValueError, match="acting_agent.settings.login"): - make_agent(tmp_path, fake_harbor, tasks, acting_agent={"cli": "codex", "settings": {"login": "chatgpt"}}, - model="gpt-5.6-luna", binding=None).check_agent() - write(tmp_path / "mini.yaml", "agent: {}\n") # beside the params, not in the working directory - agent = make_agent(tmp_path, fake_harbor, tasks, acting_agent={"cli": "mini-swe-agent", - "settings": {"config": "mini.yaml"}}) - assert agent.settings()["config"] == str((tmp_path / "mini.yaml").resolve()) - agent.check_agent() # its dry render reads it there - absolute = write(tmp_path / "elsewhere" / "m.yaml", "agent: {}\n") - agent = make_agent(tmp_path, fake_harbor, tasks, acting_agent={"cli": "mini-swe-agent", - "settings": {"config": str(absolute)}}) - assert agent.settings()["config"] == str(absolute) - - -def test_a_held_trials_verdict_names_the_room_its_image_and_other_pulls_took(tmp_path, fake_harbor, monkeypatch): - """_disk_held names what held a trial back: the data root's gate, with the room the task's image takes when it is - not here yet (`image_sizes`) and the room other trials hold for their pulls, which the gate counts as taken.""" - from hermit.harbor import docker as docker_mod - tasks = tmp_path / "tasks" - make_task(tasks, "alpha", image="example/alpha:1") - row = dataset.index(tasks)[0] - agent = make_agent(tmp_path, fake_harbor, tasks, disk_gate_gb=5) - agent._root = tmp_path - monkeypatch.setattr(docker_mod, "image_size", lambda image: None) - assert agent._disk_held(row) == f"the Docker data root {tmp_path} stayed under 5 GB free" - agent.image_sizes = {"example/alpha:1": 2 * docker_mod.GB} - agent.pulls.add("example/beta:1", 3.0) - held = agent._disk_held(row) - assert "GB more for the task's image example/alpha:1, not here yet" in held, held - assert held.endswith("beside 3.0 GB held for the images other trials are still to pull"), held - - -def test_a_streams_first_step_holds_room_for_its_image_until_its_environment_is_up(tmp_path, fake_harbor, - monkeypatch): - """The disk gate before a stream's environment starts is a single trial's: it keeps room for the task's image - (wait_for_disk with the task), held for the image's pull until the driver is up, then released.""" - fake_driver(fake_harbor) - tasks = tmp_path / "tasks" - make_task(tasks, "flow", steps=("one", "two")) - rows = {r["id"]: r for r in dataset.index(tasks)} - agent = make_agent(tmp_path, fake_harbor, tasks, stream={"idle_s": 60, "start_timeout_s": 30, "turn_wait_s": 6}) - calls = [] - - def gate(task=None, admitted=None): - calls.append(task["id"] if task else None) - if admitted is not None: - admitted.append(agent.pulls.add("example/flow:1", 1.0)) - return True - monkeypatch.setattr(agent, "wait_for_disk", gate) - try: - assert [agent.run(rows[f"flow:{s}"], None).output["status"] for s in ("one", "two")] == ["ok", "ok"] - finally: - stop_streams(tmp_path) - assert calls == ["flow:one"] and not agent.pulls.pending # the first step's, its room released - - -def test_a_streams_step_waits_for_its_turn_over_the_budgets_of_the_steps_before_it(tmp_path, fake_harbor): - """A step's wait for its turn (stream.turn_wait_s left empty) is the budgets of the steps before it, each its - own (a step may set its own agent and verifier timeouts), and the driver's start; never this step's budget times - their number.""" - tasks = tmp_path / "tasks" - make_task(tasks, "flow", steps=("one", "two", "three")) - toml = tasks / "flow" / "task.toml" - text = toml.read_text().replace('name = "one"\n', 'name = "one"\n[steps.agent]\ntimeout_sec = 100.0\n') \ - .replace('name = "two"\n', 'name = "two"\n[steps.agent]\ntimeout_sec = 900.0\n') - toml.write_text(text) - rows = {r["id"]: r for r in dataset.index(tasks)} - agent = make_agent(tmp_path, fake_harbor, tasks) - settings = {"start_timeout_s": 30, "turn_wait_s": None} - budget = {s: agent.wait_budget(rows[f"flow:{s}"], build=False) for s in ("one", "two", "three")} - assert len(set(budget.values())) == 3 - assert agent._turn_wait(rows["flow:three"], 3, settings) == budget["one"] + budget["two"] + 30 - assert agent._turn_wait(rows["flow:one"], 1, settings) == budget["one"] + 30 - assert agent._turn_wait(rows["flow:three"], 3, {**settings, "turn_wait_s": 7}) == 7 diff --git a/tests/core/test_harbor_smoke.py b/tests/core/test_harbor_smoke.py deleted file mode 100644 index 26d2825d..00000000 --- a/tests/core/test_harbor_smoke.py +++ /dev/null @@ -1,129 +0,0 @@ -"""The Harbor runner against a real Docker daemon: a single-step task as one trial, a three-step task as a stream -(one container through the Trainer's steps), with the scripted agent and no model. - -Needs Docker, Harbor's venv (~/.cache/hermit/harbor-venv or HERMIT_HARBOR_VENV) and HERMIT_HARBOR_SMOKE=1; skipped -otherwise. The two tasks are built here from `python:3.11-slim`; everything they leave is under a temporary -directory, and the stream's socket under the user's runtime directory. -""" -import json -import os -from pathlib import Path -import shutil -import stat -import subprocess - -import pytest - -from hermit.harbor import dataset - -HARBOR_VENV = Path(os.environ.get("HERMIT_HARBOR_VENV", "~/.cache/hermit/harbor-venv")).expanduser() -TOML = '''schema_version = "1.4" - -[metadata] -author_name = "hermit" -difficulty = "easy" -category = "smoke" -tags = ["smoke"] - -[verifier] -timeout_sec = 60.0 - -[agent] -timeout_sec = 90.0 - -[environment] -build_timeout_sec = 300.0 -cpus = 1 -memory_mb = 512 -''' -DOCKERFILE = "FROM python:3.11-slim\nWORKDIR /app\n" - - -def _write(path: Path, text: str, exe: bool = False) -> None: - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(text) - if exe: - path.chmod(path.stat().st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH) - - -def make_tasks(root: Path) -> None: - """`echo-answer` (one step: 42 in /app/answer.txt) and `counter` (three steps: one more line in /app/log.txt - each; a step's verifier counts the lines, so it passes only when the earlier steps' work is still there).""" - t = root / "echo-answer" - _write(t / "task.toml", TOML) - _write(t / "instruction.md", "Write the number 42 to /app/answer.txt.\n") - _write(t / "environment" / "Dockerfile", DOCKERFILE) - _write(t / "tests" / "test.sh", '#!/bin/bash\nmkdir -p /logs/verifier\n' - 'if [ "$(cat /app/answer.txt 2>/dev/null | tr -d "[:space:]")" = "42" ]; then echo 1 > /logs/verifier/reward.txt; ' - 'echo PASS; else echo 0 > /logs/verifier/reward.txt; echo "FAIL: answer.txt is $(cat /app/answer.txt 2>/dev/null)"; fi\n', - exe=True) - s = root / "counter" - _write(s / "task.toml", TOML + "".join(f'\n[[steps]]\nname = "s{i}"\n' for i in (1, 2, 3))) - _write(s / "instruction.md", "A three-step stream: each step appends one line to /app/log.txt.\n") - _write(s / "environment" / "Dockerfile", DOCKERFILE) - _write(s / "tests" / "test.sh", "#!/bin/bash\nmkdir -p /logs/verifier; echo 0 > /logs/verifier/reward.txt\n", exe=True) - for i in (1, 2, 3): - _write(s / "steps" / f"s{i}" / "instruction.md", f"Step {i}: append the line 'step {i}' to /app/log.txt.\n") - _write(s / "steps" / f"s{i}" / "tests" / "test.sh", "#!/bin/bash\nmkdir -p /logs/verifier\n" - f'n=$(wc -l < /app/log.txt 2>/dev/null || echo 0); if [ "$n" -eq {i} ]; then echo 1 > /logs/verifier/reward.txt; ' - f'echo PASS $n; else echo 0 > /logs/verifier/reward.txt; echo "FAIL: $n lines"; fi\n', exe=True) - - -def _docker_ok() -> bool: - try: - return subprocess.run(["docker", "info"], capture_output=True, timeout=30).returncode == 0 - except (OSError, subprocess.TimeoutExpired): - return False - - -pytestmark = pytest.mark.skipif( - os.environ.get("HERMIT_HARBOR_SMOKE") != "1" or not (HARBOR_VENV / "bin" / "harbor").is_file() or not _docker_ok(), - reason="HERMIT_HARBOR_SMOKE=1, Harbor's venv and Docker are needed") - - -@pytest.fixture(scope="module") -def tasks(tmp_path_factory): - root = tmp_path_factory.mktemp("harbor-smoke") - make_tasks(root / "tasks") - rows = {r["id"]: r for r in dataset.index(root / "tasks")} - yield root, rows - shutil.rmtree(root, ignore_errors=True) - - -def _agent(root: Path, script: str, jobs: str, **params): - from hermit.harbor.runner import HarborAgent - return HarborAgent({"tasks_dir": str(root / "tasks"), "jobs_dir": str(root / jobs), "disk_gate_gb": 5, - "harbor": str(HARBOR_VENV / "bin" / "harbor"), - "acting_agent": {"import_path": "harborside.scripted:ScriptedAgent", "kwargs": {"script": script}}, - **params}, base=root) - - -def test_single_step_trial_scores_the_verifier_reward(tasks): - from hermit.harbor.judge import score - root, rows = tasks - out = _agent(root, "echo 42 > /app/answer.txt", "jobs").run(rows["echo-answer"], None) - assert out.output["status"] == "ok" and out.output["reward"] == 1.0, out.extra - assert out.tokens == 0 and out.log and (Path(out.log) / "verifier" / "reward.txt").read_text().strip() == "1" - assert (Path(out.log) / "agent" / "script.txt").is_file() and (Path(out.log) / "result.json").is_file() - assert score(rows["echo-answer"], out)["score"] == 1.0 - wrong = _agent(root, "echo 41 > /app/answer.txt", "jobs").run(rows["echo-answer"], None) - assert wrong.output["status"] == "ok" and wrong.output["reward"] == 0.0 - verdict = score(rows["echo-answer"], wrong) - assert verdict["score"] == 0.0 and "FAIL: answer.txt is 41" in verdict["feedback"] - - -def test_multi_step_task_runs_as_one_stream(tasks): - root, rows = tasks - agent = _agent(root, "echo step >> /app/log.txt", "jobs-stream", stream={"idle_s": 600, "start_timeout_s": 600}) - results = [agent.run(rows[f"counter:s{i}"], None) for i in (1, 2, 3)] - name = f"counter__s1__{agent.run_id}" # the stream is this runner's own - for i, out in enumerate(results, start=1): - assert out.output["status"] == "ok" and out.output["reward"] == 1.0, (i, out.extra) - assert out.output["step"] == f"s{i}" and out.output["trial"] == name - assert (Path(out.log) / "verifier" / "test-stdout.txt").read_text().strip() == f"PASS {i}" - stream_dir = root / "jobs-stream" / "streams" / name - result = json.loads((stream_dir / "trials" / name / "result.json").read_text()) - assert [s["step_name"] for s in result["step_results"]] == ["s1", "s2", "s3"] - assert result["verifier_result"]["rewards"]["reward"] == 1.0 # the mean over the steps - containers = subprocess.run(["docker", "ps", "-a", "--format", "{{.Names}}"], capture_output=True, text=True).stdout - assert name not in containers # the stream's environment is down diff --git a/tests/core/test_hermes_agent.py b/tests/core/test_hermes_agent.py index 76a2fbaf..cd1834e3 100644 --- a/tests/core/test_hermes_agent.py +++ b/tests/core/test_hermes_agent.py @@ -18,7 +18,7 @@ import pytest -from hermit.agent.environment import Container, placed +from hermit.environment import Container, placed import yaml from hermit.agent.base import cli as base_cli diff --git a/tests/core/test_layers.py b/tests/core/test_layers.py index 9a3665f2..171ea056 100644 --- a/tests/core/test_layers.py +++ b/tests/core/test_layers.py @@ -1,17 +1,20 @@ -"""The agent layers depend downward only (docs/framework/agent-layers.md): Layer 1 (hermit.agent.model) imports no -other layer, the environments (hermit.agent.environment) import neither Layer 2 nor Layer 3, Layer 2 -(hermit.agent.base) imports Layer 1 and the environments only, and none imports Layer 3 (hermit.agent.acting, -hermit.agent.evolver) or the framework around them.""" +"""The layers depend downward only (docs/framework/agent-layers.md): Layer 1 (hermit.agent.model) imports no other +layer; the environments (hermit.environment) import no agent and no task; Layer 2 (hermit.agent.base) imports Layer +1 and the environments only, and none imports Layer 3 (hermit.agent.acting, hermit.agent.evolver) or the framework +around them. Tasks (hermit.task) and agents (hermit.agent) never import each other: the run (hermit.trainer) brings +them together, and both may use the environments and hermit.common.""" import ast from pathlib import Path -ROOT = Path(__file__).resolve().parents[2] / "hermit" / "agent" -ABOVE = {"model": ("hermit.agent.base", "hermit.agent.environment", "hermit.agent.acting", "hermit.agent.evolver", - "hermit.trainer", "hermit.adapter", "hermit.tools", "hermit.task"), - "environment": ("hermit.agent.base", "hermit.agent.acting", "hermit.agent.evolver", "hermit.trainer", - "hermit.adapter", "hermit.tools", "hermit.task"), - "base": ("hermit.agent.acting", "hermit.agent.evolver", "hermit.trainer", "hermit.adapter", "hermit.tools", - "hermit.task")} +HERMIT = Path(__file__).resolve().parents[2] / "hermit" +FRAMEWORK = ("hermit.trainer", "hermit.adapter", "hermit.tools", "hermit.isolate", "hermit.harbor") +ABOVE = {"agent/model": ("hermit.agent.base", "hermit.environment", "hermit.agent.acting", "hermit.agent.evolver", + "hermit.task", *FRAMEWORK), + "environment": ("hermit.agent", "hermit.task", *FRAMEWORK), + "agent/base": ("hermit.agent.acting", "hermit.agent.evolver", "hermit.task", *FRAMEWORK), + "agent/acting": ("hermit.task", "hermit.trainer", "hermit.isolate"), + "agent/evolver": ("hermit.task",), + "task": ("hermit.agent", *FRAMEWORK)} def imports(path: Path) -> set[str]: @@ -27,8 +30,8 @@ def imports(path: Path) -> set[str]: def test_each_layer_imports_only_the_layers_below(): found = [] for layer, banned in ABOVE.items(): - for path in sorted((ROOT / layer).rglob("*.py")): - found += [f"{path.relative_to(ROOT)} imports {name}" for name in imports(path) + for path in sorted((HERMIT / layer).rglob("*.py")): + found += [f"{path.relative_to(HERMIT)} imports {name}" for name in imports(path) if any(name == b or name.startswith(b + ".") for b in banned)] assert found == [] diff --git a/tests/core/test_mini_swe_agent.py b/tests/core/test_mini_swe_agent.py index 4f47c5f9..ffe81718 100644 --- a/tests/core/test_mini_swe_agent.py +++ b/tests/core/test_mini_swe_agent.py @@ -21,7 +21,7 @@ StrictUndefined = Template = None import pytest -from hermit.agent.environment import Container, Sandboxed +from hermit.environment import Container, Sandboxed import yaml from hermit.agent.acting.agent import load_agent diff --git a/tests/core/test_opencode_agent.py b/tests/core/test_opencode_agent.py index becee5ce..212cbf63 100644 --- a/tests/core/test_opencode_agent.py +++ b/tests/core/test_opencode_agent.py @@ -15,7 +15,7 @@ import pytest -from hermit.agent.environment import Container, placed +from hermit.environment import Container, placed from hermit.agent.base import cli as base_cli from hermit.agent.base import trajectory diff --git a/tests/core/test_pi_agent.py b/tests/core/test_pi_agent.py index 26f8409a..b3b9484f 100644 --- a/tests/core/test_pi_agent.py +++ b/tests/core/test_pi_agent.py @@ -14,7 +14,7 @@ import pytest -from hermit.agent.environment import Container, placed +from hermit.environment import Container, placed from hermit.agent.base import cli as base_cli from hermit.agent.base import trajectory diff --git a/tests/core/test_selection.py b/tests/core/test_selection.py index 668b0b6d..941e695b 100644 --- a/tests/core/test_selection.py +++ b/tests/core/test_selection.py @@ -24,7 +24,8 @@ def test_a_methods_own_files_run_its_group_and_core(): assert groups_for("docs/methods/ahe_adapter.md") == {"ahe", "core"} # its capability table is held # a pattern's `*` is one directory: a benchmark's shared folder named like a method, or a method's own folder # named like another, is not that method's - assert groups_for("examples/clbench_database/utils/task-info/mwm/host.json") >= {"mwm", "reharness", "core"} + assert groups_for("examples/clbench_database/utils/benchmark/task-info/mwm/host.json") >= {"mwm", "reharness", + "core"} assert groups_for("examples/officeqa/rrsi_adapter/texts/ace/SKILL.md") == {"rrsi_adapter", "core"} assert groups_for("examples/officeqa/gepa/run.yaml") == {"optimizers", "core", "ahe"} # read by AHE's assert groups_for("hermit/adapter/templates/harness2_codex/exports.yaml") == {"harness2", "core", diff --git a/tests/core/test_task.py b/tests/core/test_task.py new file mode 100644 index 00000000..0e62cd33 --- /dev/null +++ b/tests/core/test_task.py @@ -0,0 +1,107 @@ +"""Tasks (hermit.task): the public half an agent is handed, the private half only the judge reads, the task set and +its splits, and the run that brings a task and an agent together (hermit.trainer.rollout).""" +import json +import warnings + +import pytest + +from hermit.common import RunOutput +from hermit.task import Benchmark, JudgeResult, Task, as_task, judge_verifier +from hermit.trainer.rollout import run_one, run_tasks + + +def test_a_task_keeps_the_judges_own_out_of_what_the_agent_is_handed(): + task = Task.from_row({"id": "q1", "question": "2+2?", "_private": {"answer": "4"}}) + assert task.id == "q1" and task.private == {"answer": "4"} + given = task.given() + assert given == {"id": "q1", "question": "2+2?"} and "_private" not in given + given["question"] = "changed" # a copy: the task is not changed through it + assert task.public["question"] == "2+2?" + with pytest.raises(TypeError): + task.public["question"] = "x" # read-only + with pytest.raises(ValueError, match="judge's own"): + Task(id="q2", public={"_private": {}}) + assert Task.from_row({"task_id": 7}).id == "7" and as_task(task) is task + with pytest.raises(ValueError, match="needs an id"): + Task.from_row({"question": "no id"}) + + +def test_a_benchmark_reads_its_task_set_and_splits_in_order(tmp_path): + rows = [{"id": f"t{i}", "q": i} for i in range(4)] + (tmp_path / "tasks.jsonl").write_text("\n".join(json.dumps(r) for r in rows)) + (tmp_path / "train.txt").write_text("t2\nt0\n") + (tmp_path / "test.txt").write_text("# a comment\nt3\n") + bench = Benchmark.load(tmp_path / "tasks.jsonl", tmp_path / "train.txt", None, tmp_path / "test.txt") + assert [t.id for t in bench.split("train")] == ["t2", "t0"] and [t.id for t in bench.split("test")] == ["t3"] + assert bench.split("val") == [] and bench.tasks["t1"].given() == {"id": "t1", "q": 1} + + +def test_the_run_hands_the_agent_the_public_half_and_the_verifier_the_whole_task(): + seen = {} + + def agent(given, adapter): + seen["given"] = given + return RunOutput(output="4", cost_usd=0.01, tokens=10) + + def verify(task, output): + seen["task"] = task + return JudgeResult(float(output.output == task.private["answer"]), "checked") + verify.takes_task = True + record = run_one(Task.from_row({"id": "q1", "question": "2+2?", "_private": {"answer": "4"}}), None, agent, verify) + assert seen["given"] == {"id": "q1", "question": "2+2?"} and seen["task"].private == {"answer": "4"} + assert record["status"] == "ok" and record["judge"] == {"score": 1.0, "feedback": "checked", "extra": {}} + assert record["input"] == {"id": "q1", "question": "2+2?"} # the record shows what the agent was given + + +def test_a_bare_judge_and_a_row_still_run_as_before(): + """Judges written as `score(task_row, output)` and callers that pass rows keep their records unchanged.""" + def agent(given, adapter): + return RunOutput(output=given["q"] * 2) + + def score(row, output): + return {"score": float(output.output == 4), "feedback": "ok", "seen": sorted(row)} + records = run_tasks([{"id": "a", "q": 2}, Task.from_row({"id": "b", "q": 3})], None, agent, score, concurrency=2) + assert [r["id"] for r in records] == ["a", "b"] + assert records[0]["judge"] == {"score": 1.0, "feedback": "ok", "extra": {"seen": ["id", "q"]}} + assert records[1]["judge"]["score"] == 0.0 and records[0]["input"] == {"id": "a", "q": 2} + wrapped = judge_verifier(score) + assert wrapped(as_task({"id": "c", "q": 2}), RunOutput(output=4)).score == 1.0 + + +def test_the_environments_moved_to_the_top_level_and_the_old_path_forwards(): + import importlib + import hermit.agent.environment as forwarded + from hermit.environment import Local + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + importlib.reload(forwarded) # the warning comes with the import, once + from hermit.agent.environment import Local as old + from hermit.agent.environment.base import Sandboxed as old_sandboxed + from hermit.environment.base import Sandboxed + assert old is Local and old_sandboxed is Sandboxed + assert any("hermit.environment" in str(w.message) for w in caught) + + +def test_the_names_imported_from_hermit_task_before_the_package_still_import(): + from hermit.task import (JudgeResult, RunOutput, TaskData, load_callable, load_pipeline, # noqa: F401 + normalise_judge, output_record, read_ids) + from hermit import common + assert RunOutput is common.RunOutput and load_callable is common.load_callable + + +def test_a_bare_judge_still_reads_the_private_half_and_records_keep_the_rows_id(): + """As before tasks were typed: a judge `score(row, output)` is called with the whole row, `_private` included (the + agent is not); a numeric id stays a number in the record; `judge=` is still the keyword.""" + seen = {} + + def agent(given, adapter): + seen["agent"] = given + return RunOutput(output="4") + + def score(row, output): + return float(output.output == row["_private"]["answer"]) + row = {"id": 7, "question": "2+2?", "_private": {"answer": "4"}} + record = run_one(row, None, agent, judge=score) + assert record["judge"]["score"] == 1.0 and "_private" not in seen["agent"] + assert record["id"] == 7 and json.loads(json.dumps(record))["id"] == 7 + assert run_tasks([row], None, agent, judge=score, concurrency=1)[0]["judge"]["score"] == 1.0 diff --git a/tests/core/test_task_directory.py b/tests/core/test_task_directory.py new file mode 100644 index 00000000..18d02a32 --- /dev/null +++ b/tests/core/test_task_directory.py @@ -0,0 +1,385 @@ +"""Tasks as directories in Harbor's format (hermit.task.directory), read by Benchmark, and their own verifier script +(hermit.task.verify.script) run in the judge's sandbox as Harbor runs it.""" +import json +from pathlib import Path +import subprocess +import shutil + +import pytest + +from hermit.common import RunOutput +from hermit.config import Config +from hermit.task import Benchmark +from hermit.task.directory import tasks_of +from hermit.environment.spec import EnvironmentSpec +from hermit.isolate import _deadline +from hermit.task.verify import read_reward, resolve_env, script +from toy_task import make_task + + +def a_task(root: Path, name: str, *, instruction="Write 42 to answer.txt.", test_sh=None, toml_extra="", + steps=False) -> Path: + d = root / name + (d / "tests").mkdir(parents=True) + (d / "environment").mkdir() + (d / "instruction.md").write_text(instruction + "\n") + (d / "task.toml").write_text('version = "1.0"\n[metadata]\ndifficulty = "easy"\n[agent]\ntimeout_sec = 60\n' + '[verifier]\ntimeout_sec = 30\n' + toml_extra + + '[environment]\ncpus = 1\nmemory_mb = 512\nnetwork_mode = "no-network"\n') + (d / "tests" / "test.sh").write_text(test_sh or "echo 1 > /logs/verifier/reward.txt\n") + (d / "tests" / "expected.txt").write_text("42") + if steps: + with open(d / "task.toml", "a") as f: + f.write('[[steps]]\nname = "one"\n[[steps]]\nname = "two"\n') + for step in ("one", "two"): + (d / "steps" / step / "tests").mkdir(parents=True) + (d / "steps" / step / "instruction.md").write_text(f"step {step}\n") + return d + + +def test_a_directory_of_tasks_is_a_task_set_with_the_tests_kept_private(tmp_path): + a_task(tmp_path / "tasks", "write-42") + a_task(tmp_path / "tasks", "flow", steps=True) + (tmp_path / "tasks" / "notes").mkdir() # a folder without task.toml is no task + (tmp_path / "train.txt").write_text("write-42\nflow:one\n") + bench = Benchmark.load(tmp_path / "tasks", tmp_path / "train.txt") + assert sorted(bench.tasks) == ["flow:one", "flow:two", "write-42"] + task = bench.tasks["write-42"] + given = task.given() + assert given["instruction"] == "Write 42 to answer.txt." and given["harbor_task"] == "write-42" + assert given["metadata"] == {"difficulty": "easy"} and given["network"] == {"agent": "no-network", + "verifier": "no-network"} + assert (given["cpus"], given["memory_mb"], given["agent_timeout_sec"]) == (1, 512, 60) + assert not any("tests" in str(v) for v in given.values()) # nothing of the tests reaches the agent + assert task.private["tests"] == str((tmp_path / "tasks" / "write-42" / "tests").resolve()) + assert task.private["verifier"] == {"timeout_sec": 30, "environment_mode": "shared", "env": {}} + assert task.private["needs"] == {"network": {"agent": "no-network", "verifier": "no-network"}, + "allowed_hosts": {"agent": [], "verifier": []}, "memory_mb": 512, + "agent_timeout_sec": 60, "verifier_memory_mb": None, "steps": None} + flow = bench.tasks["flow:one"].private["needs"] + assert flow["steps"] == 2 and "multi-step" in " ".join(EnvironmentSpec().problems_for(flow)) + assert task.row()["_private"]["tests"] == task.private["tests"] # what the judge is called with + assert bench.tasks["flow:two"].private["step_tests"].endswith("flow/steps/two/tests") # over the task's + assert [t.id for t in bench.split("train")] == ["write-42", "flow:one"] + assert sorted(tasks_of(tmp_path / "tasks", rollouts=2)) == ["flow:one#1", "flow:one#2", "flow:two#1", + "flow:two#2", "write-42#1", "write-42#2"] + with pytest.raises(ValueError, match="no task directories"): + tasks_of(tmp_path / "tasks" / "notes") + + +def test_a_task_written_by_write_task_reads_back_as_written(tmp_path): + """write_task's task.toml holds every value tomllib reads back the same, in its order (a benchmark's rows ride in + [metadata], data.view turns them back); a value TOML cannot hold is refused.""" + from hermit.task.directory import write_task + meta = {"zeta": 1, "alpha": "a \"q\"\\ \u2019\x7f\t\x00", "seeds": [1, 2], "on": False, "rate": 0.1, + "big": 1e100, "nested": {"vector b": "x\ny", "a": [{"k": 1}]}, "empty": [], "neg": -3} + write_task(tmp_path / "t", "Line one\r\nline two", metadata=meta, environment={"network_mode": "no-network"}) + given = tasks_of(tmp_path / "t")["t"].given() + assert given["metadata"] == meta and list(given["metadata"]) == list(meta) + assert list(given["metadata"]["nested"]) == ["vector b", "a"] + assert given["instruction"] == "Line one\r\nline two" and given["network"]["agent"] == "no-network" + with pytest.raises(ValueError, match="none_here"): + write_task(tmp_path / "u", "x", metadata={"none_here": None}) + with pytest.raises(ValueError, match="trimmed"): + write_task(tmp_path / "v", "x\n") + + +def test_a_reward_is_read_as_harbor_writes_it(tmp_path): + (tmp_path / "reward.txt").write_text("0.5\n") + assert read_reward(tmp_path, "the script said hi").score == 0.5 + assert read_reward(tmp_path, "the script said hi").feedback == "the script said hi" + (tmp_path / "feedback.txt").write_text("expected 42, found 41") + (tmp_path / "reward.json").write_text(json.dumps({"reward": 1, "style": 0.25})) + verdict = read_reward(tmp_path) + assert (verdict.score, verdict.feedback, verdict.extra) == (1.0, "expected 42, found 41", + {"rewards": {"reward": 1.0, "style": 0.25}}) + (tmp_path / "reward.json").write_text(json.dumps({"a": 1, "b": 0})) + assert read_reward(tmp_path).score == 0.5 # no `reward`: the mean of the named numbers + (tmp_path / "reward.json").write_text(json.dumps({"reward": 1, "f1": 1, "answered": True})) + assert read_reward(tmp_path).score == 1.0 # a boolean beside them, as Harbor takes it + (tmp_path / "reward.json").write_text(json.dumps({"a": "yes"})) + (tmp_path / "nan").mkdir() + (tmp_path / "nan" / "reward.txt").write_text("nan\n") + with pytest.raises(ValueError, match="finite"): + read_reward(tmp_path / "nan") + with pytest.raises(ValueError, match="named finite numbers"): + read_reward(tmp_path) + with pytest.raises(RuntimeError, match="wrote no reward"): + read_reward(tmp_path / "missing", "tail") + + +def test_the_verifier_script_runs_only_in_the_judges_sandbox(): + with pytest.raises(RuntimeError, match="judge's sandbox"): + script({"id": "t", "_private": {"tests": "/nowhere"}}, RunOutput(output="42")) + + +def _bwrap_works() -> bool: + probe = ["bwrap", "--unshare-user", "--unshare-net", "--ro-bind", "/", "/", "true"] + return bool(shutil.which("bwrap")) and subprocess.run(probe, capture_output=True).returncode == 0 + + +needs_bwrap = pytest.mark.skipif(not _bwrap_works(), reason="needs a working bubblewrap") + +AGENT = """ +import json, os +from pathlib import Path +from hermit.task import RunOutput + +def run(task, adapter): + work = os.getcwd() + Path(work, "answer.txt").write_text("42" if "42" in task["instruction"] else "0") + return RunOutput(output="done", extra={"sees_tests": os.path.exists("/tests"), + "fields": sorted(task)}) +""" + +TEST_SH = """set -u +expected=$(cat /tests/expected.txt) +found=$(cat "$HERMIT_WORKSPACE/answer.txt" 2>/dev/null || echo missing) +cat "$HERMIT_OUTPUT" > /logs/verifier/seen_output.txt +if [ "$found" = "$expected" ]; then echo 1 > /logs/verifier/reward.txt; else echo 0 > /logs/verifier/reward.txt; fi +echo "expected $expected, found $found" > /logs/verifier/feedback.txt +""" + + +@needs_bwrap +def test_a_task_directory_runs_with_its_own_verifier_in_the_judges_sandbox(tmp_path): + from hermit.isolate import SandboxedAgent, judge_for + from hermit.trainer.rollout import run_one + root = tmp_path / "run" + root.mkdir() + path = make_task(root, sandbox={"enabled": True}, agent={"name": "agent/pipeline.py:run", "concurrency": 1}, + environment={"read": ["agent"]}) + (root / "agent").mkdir() + (root / "agent" / "pipeline.py").write_text(AGENT) + a_task(root / "tasks", "write-42", test_sh=TEST_SH) + a_task(root / "tasks", "write-0", instruction="Write nothing.", test_sh=TEST_SH) + a_task(root / "tasks", "apart", test_sh=TEST_SH, toml_extra='environment_mode = "separate"\n') + (root / "splits.txt").write_text("write-42\nwrite-0\napart\n") + config = Config.load(path, ["data.tasks=tasks", "data.train=splits.txt", "data.val=null", "data.test=null", + "judge.path=hermit.task.verify", "judge.name=script", "judge.mode=shared"]) + bench = Benchmark.from_config(config) + agent = SandboxedAgent(config, "rollout", records=None) + try: + verifier = judge_for(config, agent) + right = run_one(bench.tasks["write-42"], None, agent, verifier) + wrong = run_one(bench.tasks["write-0"], None, agent, verifier) + apart = run_one(bench.tasks["apart"], None, agent, verifier) + finally: + agent.close() + assert right["status"] == "ok" and right["judge"]["score"] == 1.0, right + assert right["judge"]["feedback"] == "expected 42, found 42\n" + assert right["pipeline_extra"]["sees_tests"] is False # the tests are the judge's, not the agent's + assert "_private" not in right["pipeline_extra"]["fields"] # nor anything else of the private half + assert wrong["judge"]["score"] == 0.0 and "found 0" in wrong["judge"]["feedback"] + assert apart["status"] == "judge_error" and "separate verifier" in apart["error"] + assert "_private" not in right["input"] # the record keeps what the agent was given + + +def test_a_step_keeps_its_own_verifier_settings_and_a_task_asks_for_what_it_needs(tmp_path, monkeypatch): + d = a_task(tmp_path / "tasks", "flow", steps=True, + toml_extra='environment_mode = "shared"\n[verifier.env]\nEXPECTED = "42"\n' + 'KEY = "${HERMIT_T_KEY:-none}"\n') + text = (d / "task.toml").read_text().replace('[[steps]]\nname = "two"\n', + '[[steps]]\nname = "two"\n[steps.verifier]\ntimeout_sec = 2\n' + 'environment_mode = "separate"\n') + (d / "task.toml").write_text(text.replace('network_mode = "no-network"', + 'network_mode = "allowlist"\nallowed_hosts = ["data.example.com"]')) + tasks = tasks_of(tmp_path / "tasks") + assert tasks["flow:two"].private["verifier"]["timeout_sec"] == 2 + assert tasks["flow:two"].private["verifier"]["environment_mode"] == "separate" + assert tasks["flow:one"].private["verifier"]["environment_mode"] == "shared" + env = tasks["flow:one"].private["verifier"]["env"] + monkeypatch.delenv("HERMIT_T_KEY", raising=False) + assert resolve_env(env) == {"EXPECTED": "42", "KEY": "none"} + monkeypatch.setenv("HERMIT_T_KEY", "k") + assert resolve_env(env)["KEY"] == "k" + with pytest.raises(RuntimeError, match="HERMIT_T_MISSING"): + resolve_env({"X": "${HERMIT_T_MISSING}"}) + needs = {**tasks["flow:one"].private["needs"], "steps": None} # its hosts, apart from its steps + assert needs["allowed_hosts"] == {"agent": ["data.example.com"], "verifier": ["data.example.com"]} + other = EnvironmentSpec(network="allowlist", allowed_hosts=("api.example.com",)) + assert "data.example.com" in " ".join(other.problems_for(needs)) + assert EnvironmentSpec(network="allowlist", allowed_hosts=("data.example.com",)).problems_for(needs) == [] + wild = {**needs, "allowed_hosts": {"agent": ["*.example.com"]}} + assert "wildcard" in " ".join(EnvironmentSpec(network="allowlist", allowed_hosts=("*.example.com",)) + .problems_for(wild)) + loud = {**needs, "network": {"agent": "no-network", "verifier": "public"}} + assert EnvironmentSpec().problems_for(loud) == [] # judged on the host: the agent's only + assert "verifier needs network public" in " ".join(EnvironmentSpec().problems_for(loud, verifier=True)) + + +def test_a_task_directorys_own_agent_timeout_bounds_a_call(): + assert _deadline(None, {"agent_timeout_sec": 60}) == 120 # its own, with the worker's start-up + assert _deadline(90.0, {"agent_timeout_sec": 60}) == 90.0 # the run's when stricter + assert _deadline(600.0, {"agent_timeout_sec": 60}) == 120 + assert _deadline(600.0, {"id": "row"}) == 600.0 and _deadline(None, {}) is None + + +def test_harbors_defaults_and_phase_allowlists_and_a_view_of_the_rows(tmp_path): + d = a_task(tmp_path / "tasks", "own", toml_extra='[verifier.environment]\ncpus = 1\n' + '[agent]\nnetwork_mode = "allowlist"\nallowed_hosts = ["data.example.com"]\n') + text = (d / "task.toml").read_text().replace("[verifier]\ntimeout_sec = 30\n", "[verifier]\n") + (d / "task.toml").write_text(text.replace("[agent]\ntimeout_sec = 60\n", "")) + task = tasks_of(tmp_path / "tasks")["own"] + assert task.private["verifier"]["timeout_sec"] == 600.0 # Harbor's default + assert task.private["verifier"]["environment_mode"] == "separate" # implied by its own environment + assert task.private["needs"]["allowed_hosts"]["agent"] == ["data.example.com"] + assert "data.example.com" in " ".join(EnvironmentSpec(network="allowlist").problems_for(task.private["needs"])) + (tmp_path / "split.txt").write_text("own\n") + view = Benchmark.load(tmp_path / "tasks", tmp_path / "split.txt", + view=lambda row: {"id": row["id"], "question": row["instruction"]}) + viewed = view.tasks["own"] + assert viewed.given() == {"id": "own", "question": "Write 42 to answer.txt."} + assert viewed.private["needs"] == task.private["needs"] # still checked, though not shown + with pytest.raises(ValueError, match="keeps each task's id"): # a legacy id would key no record + tasks_of(tmp_path / "tasks", view=lambda row: {"id": 7, "question": row["instruction"]}) + a_task(tmp_path / "tasks", "other") + with pytest.raises(ValueError, match="keeps each task's id"): # one that rewrites its input row too + tasks_of(tmp_path / "tasks", view=lambda row: row.update(id="same") or row) + + +def test_the_run_refuses_a_task_its_environment_cannot_give_before_the_agent_starts(): + from hermit.task import Task + from hermit.trainer.rollout import run_one + calls = [] + + class Agent: + def refuses(self, needs): + return ["the task's agent needs network public"] if needs.get("network") else [] + + def __call__(self, given, adapter, needs=None): + calls.append((given, needs)) + return RunOutput(output="x") + task = Task(id="t", public={"id": "t", "q": 1}, private={"needs": {"network": {"agent": "public"}}}) + record = run_one(task, None, Agent(), lambda row, out: 1.0) + assert record["status"] == "error" and "cannot run in this run's environment" in record["error"] and not calls + assert run_one(Task.from_row({"id": "r"}), None, Agent(), lambda row, out: 1.0)["status"] == "ok" + assert calls == [({"id": "r"}, {})] # handed its needs (none) + + +def test_a_steps_verifier_as_harbor_resolves_it(tmp_path): + from hermit.task.directory import network_modes, verifier_environment + config = {"environment": {"network_mode": "no-network"}, + "verifier": {"environment_mode": "shared", "environment": {"network_mode": "public"}}, + "steps": [{"name": "own", "verifier": {"environment": {}}}, + {"name": "back", "verifier": {"environment_mode": "shared"}}, {"name": "plain"}]} + assert verifier_environment(config, "own") == ("separate", {}) # a step's environment, even empty + assert verifier_environment(config, "back") == ("shared", None) + assert verifier_environment(config, "plain") == ("shared", None) # the task's explicit mode + assert network_modes(config, "back")["verifier"] == "no-network" # shared: the agent's environment + task_level = {"environment": {"network_mode": "no-network"}, + "verifier": {"environment": {"network_mode": "public"}}} + assert verifier_environment(task_level) == ("separate", {"network_mode": "public"}) + assert network_modes(task_level)["verifier"] == "public" + d = a_task(tmp_path / "tasks", "flow", steps=True) # task tests/test.sh, steps' own tests/ + (d / "steps" / "one" / "tests" / "expected.txt").write_text("41") + tasks = tasks_of(tmp_path / "tasks") + one = tasks["flow:one"].private + assert one["tests"].endswith("flow/tests") and one["step_tests"].endswith("flow/steps/one/tests") + + +def test_a_viewed_tasks_agent_timeout_still_bounds_its_call(tmp_path): + a_task(tmp_path / "tasks", "t") # [agent] timeout_sec = 60 + (tmp_path / "s.txt").write_text("t\n") + task = Benchmark.load(tmp_path / "tasks", tmp_path / "s.txt", view=lambda row: {"id": row["id"]}).tasks["t"] + assert task.given() == {"id": "t"} and task.private["needs"]["agent_timeout_sec"] == 60 + assert _deadline(None, task.private["needs"]) == 120 + + +def test_a_separate_verifiers_own_memory_is_checked(tmp_path): + a_task(tmp_path / "tasks", "big", toml_extra='[verifier.environment]\nmemory_mb = 8192\n') + needs = tasks_of(tmp_path / "tasks")["big"].private["needs"] + assert needs["verifier_memory_mb"] == 8192 + small = EnvironmentSpec(resources={"memory_mb": 1024}) + assert small.problems_for(needs) == [] # judged on the host: not asked + assert "verifier needs 8192 MB" in " ".join(small.problems_for(needs, verifier=True)) + + +AGENT_SEES_KEY = """ +import os +from pathlib import Path +from hermit.task import RunOutput + +def run(task, adapter): + Path(os.getcwd(), "answer.txt").write_text("42") + try: + import pytest + hidden = "readable" if hasattr(pytest, "main") else "covered" + except ImportError: + hidden = "covered" + return RunOutput(output="done", extra={"key": os.environ.get("HERMIT_TEST_MODEL_KEY"), "hide": hidden}) +""" + +TEST_SH_SPIES = """echo 1 > /logs/verifier/reward.txt +hidden=$(python3 -c 'import pytest; print("readable" if hasattr(pytest, "main") else "covered")' 2>&1 | tail -1) +echo "key=${HERMIT_TEST_MODEL_KEY:-absent} proxy=${HTTPS_PROXY:-${https_proxy:-none}} hide=$hidden" \ + > /logs/verifier/feedback.txt +""" + + +@needs_bwrap +@pytest.mark.parametrize("mode", ["shared", "separate"]) +def test_a_tasks_own_verifier_never_gets_the_acting_models_key_or_hosts(tmp_path, monkeypatch, mode): + """The task author's tests/test.sh runs with the task's environment, not the acting agent's: its model's key + variable is not passed, nor is a proxy to its provider given (Harbor's verifier never had the agent's key); what + sandbox.hide covers for the agent it reads, as a trusted judge does.""" + from hermit.isolate import SandboxedAgent, judge_for + from hermit.trainer.rollout import run_one + from hermit.sandbox import ENV_HIDE, hidden_paths + monkeypatch.setenv("HERMIT_TEST_MODEL_KEY", "sk-model-synthetic-0123456789") + monkeypatch.setenv(ENV_HIDE, ":".join(map(str, hidden_paths(["pytest"])))) # the run's sandbox.hide (set_hidden) + root = tmp_path / "run" + root.mkdir() + binding = {"ref": "x/m", "backend": "x", "model": "m", "base_url": "https://model.example.invalid/v1", + "api_key_env": "HERMIT_TEST_MODEL_KEY"} + path = make_task(root, sandbox={"enabled": True, "hide": ["pytest"]}, environment={"read": ["agent"]}, + agent={"name": "agent/pipeline.py:run", "concurrency": 1, "params": {"binding": binding}}) + (root / "agent").mkdir() + (root / "agent" / "pipeline.py").write_text(AGENT_SEES_KEY) + a_task(root / "tasks", "spy", test_sh=TEST_SH_SPIES) + (root / "splits.txt").write_text("spy\n") + config = Config.load(path, ["data.tasks=tasks", "data.train=splits.txt", "data.val=null", "data.test=null", + "judge.path=hermit.task.verify", "judge.name=script", f"judge.mode={mode}"]) + bench = Benchmark.from_config(config) + agent = SandboxedAgent(config, "rollout", records=None) + try: + record = run_one(bench.tasks["spy"], None, agent, judge_for(config, agent)) + finally: + agent.close() + assert record["status"] == "ok", record + assert record["pipeline_extra"]["key"] == "sk-model-synthetic-0123456789" # the agent reaches its model + assert record["pipeline_extra"]["hide"] == "covered" # and cannot read the scoring module + # a scoring module sandbox.hide keeps from the agent stays the verifier's to read, as any judge's + assert record["judge"]["feedback"] == "key=absent proxy=none hide=readable\n", record["judge"] + + +AGENT_BILLED_AND_FAILED = """ +from hermit.task import RunOutput + +def run(task, adapter): + failed = RuntimeError("chat/completions HTTP 503: busy") + failed.spent = RunOutput(output=None, cost_usd=0.002, tokens=140, api_speed={"model_calls": 1}) + raise failed +""" + + +@needs_bwrap +def test_what_a_failed_call_billed_crosses_the_sandboxes_worker(tmp_path): + """A call that failed after its endpoint billed it (its error's usage): counted on the failed record, from a + worker in the task sandbox too (its exception crosses as text, its spend beside it).""" + from hermit.isolate import SandboxedAgent + from hermit.trainer.rollout import run_one + root = tmp_path / "run" + root.mkdir() + path = make_task(root, sandbox={"enabled": True}, environment={"read": ["agent"]}, + agent={"name": "agent/pipeline.py:run", "concurrency": 1}) + (root / "agent").mkdir() + (root / "agent" / "pipeline.py").write_text(AGENT_BILLED_AND_FAILED) + config = Config.load(path) + agent = SandboxedAgent(config, "rollout", records=None) + try: + record = run_one({"id": "t0", "prompt": "x"}, None, agent, lambda row, out: {"score": 0.0}) + finally: + agent.close() + assert record["status"] == "error" and record["tokens"] == 140 and record["cost_usd"] == 0.002, record + assert record["api_speed"] == {"model_calls": 1} diff --git a/tests/core/test_training.py b/tests/core/test_training.py index 496c16df..5bebeb5f 100644 --- a/tests/core/test_training.py +++ b/tests/core/test_training.py @@ -837,3 +837,17 @@ def test_a_link_follows_its_directory_through_a_name_collision(tmp_path, monkeyp redact_tree(tmp_path) assert (tmp_path / "current" / "mine.txt").read_text() == "mine" assert key not in os.readlink(tmp_path / "current") + + +def test_eval_scores_a_task_set_with_numeric_ids(tmp_path): + """Rows whose ids are numbers (as a tasks.jsonl may hold them) are evaluated and recorded with the same ids.""" + path = make_task(tmp_path) + rows = [json.loads(l) for l in (tmp_path / "tasks.jsonl").read_text().splitlines()] + (tmp_path / "tasks.jsonl").write_text("".join(json.dumps({**r, "id": int(r["id"][1:])}) + "\n" for r in rows)) + for split in ("train", "val", "test"): + ids = (tmp_path / f"{split}.txt").read_text().split() + (tmp_path / f"{split}.txt").write_text("".join(i[1:] + "\n" for i in ids)) + result = evaluate(load(path), "none", log=lambda *_: None) + assert result["tasks"] == 3 and result["errors"] == 0 + results = tmp_path / "runs" / "toy-run" / "eval" / "none" / "results.jsonl" + assert [json.loads(l)["id"] for l in results.read_text().splitlines()] == [7, 8, 9] diff --git a/tests/methods/ahe/test_ahe_adapter_example.py b/tests/methods/ahe/test_ahe_adapter_example.py index a7ed74a3..1892b384 100644 --- a/tests/methods/ahe/test_ahe_adapter_example.py +++ b/tests/methods/ahe/test_ahe_adapter_example.py @@ -38,12 +38,12 @@ def trials(): def prepared() -> dict: - """../utils's data.json once utils/prepare.py has run (tasks.jsonl, splits/, the answers, the corpus it names), and - skillopt; the test is skipped without them.""" + """utils/benchmark/data.json once utils/benchmark/prepare.py has run (tasks/, splits/, the answers, the corpus it + names), and skillopt; the test is skipped without them.""" pytest.importorskip("skillopt") - if not all((UTILS / name).exists() for name in ("tasks.jsonl", "splits", "data.json")): - pytest.skip("OfficeQA is not prepared here (examples/officeqa/utils/prepare.py)") - data = json.loads((UTILS / "data.json").read_text()) + if not all((UTILS / "benchmark" / name).exists() for name in ("tasks", "splits", "data.json")): + pytest.skip("OfficeQA is not prepared here (examples/officeqa/utils/benchmark/prepare.py)") + data = json.loads((UTILS / "benchmark" / "data.json").read_text()) if not all((Path(data["docs"]) / sub).is_dir() for sub in ("transformed", "jsons")): pytest.skip(f"the OfficeQA corpus is not at {data['docs']}") return data @@ -61,8 +61,8 @@ def context(tmp: Path, params: dict) -> StepContext: # ---------- the configs ---------- @pytest.mark.parametrize("name, train, test, step_size, concurrency, target", [ - ("run_codex_luna.yaml", "rows/train_k2.txt", "../utils/splits/test.txt", 100, 8, 0.95), - ("smoke_codex_luna.yaml", "rows/smoke_k2.txt", "../utils/splits/smoke_test.txt", 8, 4, 1.01)]) + ("run_codex_luna.yaml", "rows/train_k2.txt", "../utils/benchmark/splits/test.txt", 100, 8, 0.95), + ("smoke_codex_luna.yaml", "rows/smoke_k2.txt", "../utils/benchmark/splits/smoke_test.txt", 8, 4, 1.01)]) def test_the_configs_resolve_the_template_and_the_evolver_on_the_luna_row(tmp_path, name, train, test, step_size, concurrency, target): config = Config.load(EXAMPLE / name) @@ -152,7 +152,7 @@ def test_a_trial_row_is_the_questions_own_row(): def test_prepare_writes_the_questions_own_task_md(tmp_path): prepared() - assert make_rows(UTILS / "tasks.jsonl", UTILS / "splits", tmp_path / "rows").returncode == 0 + assert make_rows(UTILS / "benchmark" / "tasks", UTILS / "benchmark" / "splits", tmp_path / "rows").returncode == 0 rows = {r["id"]: r for r in map(json.loads, (tmp_path / "rows" / "tasks.jsonl").read_text().splitlines())} uid = (tmp_path / "rows" / "train_k2.txt").read_text().split()[0].split("#")[0] trial_prepare = load_callable("trials.py:prepare", EXAMPLE) # as the configs name them @@ -173,8 +173,8 @@ def test_the_judge_scores_the_question(monkeypatch): monkeypatch.setattr(t, "utils", lambda spec: lambda task, output: seen.append((spec, task)) or {"score": 1.0}) t.score({"id": "UID0002#1", "uid": "UID0002", "rollout": 1, "question": "q?"}, RunOutput(output={})) t.score({"id": "UID0003", "question": "r?"}, RunOutput(output={})) - assert seen == [("judge.py:score", {"id": "UID0002", "question": "q?"}), - ("judge.py:score", {"id": "UID0003", "question": "r?"})] + assert seen == [("benchmark/judge.py:score", {"id": "UID0002", "question": "q?"}), + ("benchmark/judge.py:score", {"id": "UID0003", "question": "r?"})] def test_the_judge_scores_a_trial_against_the_questions_answer(): diff --git a/tests/methods/aspire/test_aspire.py b/tests/methods/aspire/test_aspire.py index cf799aef..516e9cf6 100644 --- a/tests/methods/aspire/test_aspire.py +++ b/tests/methods/aspire/test_aspire.py @@ -600,19 +600,27 @@ def recorded(cmd, **_): def test_the_libero_tasks_are_liberos_five_suites_and_liberos_pros_eighty_with_aspires_seeds(tmp_path, monkeypatch): - """The rows and splits prepare.py writes: what the simulator service, the judge and every method's configs read.""" + """The tasks and splits prepare.py writes, read as their rows (aspire_fields.py): what the simulator service, the + judge and every method's configs read.""" from hermit.config import Config - P = _load("libero_prepare", LIBERO / "utils" / "prepare.py") + from hermit.task import Benchmark + P = _load("libero_prepare", LIBERO / "utils" / "benchmark" / "prepare.py") + F = _load("aspire_fields", EXAMPLE / "agent" / "aspire_fields.py") assert sum(P.SUITES.values()) == 130 and P.SUITES["libero_90"] == 90 pro = [f"libero_{family}_{kind}" for family in ("object", "spatial", "goal", "10") for kind in ("swap", "task")] assert list(P.PRO_SUITES) == pro and set(P.PRO_SUITES.values()) == {10} monkeypatch.setattr(P, "HERE", tmp_path) monkeypatch.setattr(sys, "argv", ["prepare.py"]) P.main() - rows = [json.loads(line) for line in (tmp_path / "tasks.jsonl").read_text().splitlines()] - tasks = [r for r in rows if "role" not in r] + found = F.task_rows(tmp_path / "tasks") # each task directory as its row + rows = list(found.values()) + tasks = [found[i] for i in (tmp_path / "splits" / "all.txt").read_text().split()] assert len(tasks) == 210 and len(rows) == 630 and len({r["id"] for r in rows}) == 630 # a suite and its assert all(r["benchmark"] == "libero-pro" for r in rows) # perturbations share their task ids + assert list(found["libero_goal-t03"]) == ["id", "benchmark", "suite", "task_id", "debug_seeds", "unseen_seeds", + "eval_seeds"] # the row's keys, in its order + assert found["verify-libero_goal-t03"]["findings"] is False and found["rerun-libero_goal-t03"]["role"] == "rerun" + assert (tmp_path / "tasks" / "libero_goal-t03" / "instruction.md").is_file() assert all((r["debug_seeds"], r["unseen_seeds"], r["eval_seeds"]) == ("51-65", "66-80", "1-50") for r in tasks) split = lambda name: (tmp_path / "splits" / name).read_text().split() by = lambda *suites: [r["id"] for r in tasks if r["suite"] in suites] @@ -624,10 +632,15 @@ def test_the_libero_tasks_are_liberos_five_suites_and_liberos_pros_eighty_with_a assert split("libero_sample.txt") == [f"{s}-t{n:02d}" for s in ("libero_object", "libero_spatial", "libero_goal") for n in (0, 5)] for path in sorted((LIBERO / "aspire").glob("*.yaml")) + sorted((LIBERO / "prompt_only").glob("*.yaml")): - data = Config.load(path).data["data"] - rows_file = Path(data["tasks"]) - rows_file = rows_file if rows_file.is_absolute() else path.parent / rows_file - assert rows_file.resolve() == (LIBERO / "utils" / "tasks.jsonl").resolve(), path # the benchmark's rows + config = Config.load(path) + data = config.data["data"] + assert config.path(data["tasks"]) == (LIBERO / "utils" / "benchmark" / "tasks").resolve(), path + assert config.path(data["view"].split(":")[0]) == (EXAMPLE / "agent" / "aspire_fields.py").resolve(), path + monkeypatch.setitem(data, "tasks", str(tmp_path / "tasks")) # the tasks prepared here + for key in ("train", "val", "test"): + if data.get(key): + monkeypatch.setitem(data, key, str(tmp_path / "splits" / Path(data[key]).name)) + assert {t.id: t.given() for t in Benchmark.from_config(config).tasks.values()} == found, path train, test = split(Path(data["train"]).name), split(Path(data["test"]).name) assert train and set(train) <= {r["id"] for r in tasks}, path assert test == [f"rerun-{i}" for i in train], path # the reruns are of the tasks it trained on @@ -1030,7 +1043,7 @@ def test_a_run_takes_the_registrations_max_steps_and_stops_at_the_timeout(tmp_pa def test_the_judge_runs_stage_two_on_the_rows_seeds_and_a_verifier_only_when_verified(tmp_path, monkeypatch): - J = _load("aspire_judge", EXAMPLE / "judge.py") + J = _load("aspire_judge", EXAMPLE / "benchmark" / "judge.py") runs, engines = [], [] class Engine: @@ -1070,9 +1083,43 @@ def run(self, program, *, suite, seeds, out, task_id=0): assert J.score(ROW, out({"role": "actor"}))["score"] == 0.0 # no program: not scored +def test_the_robosuite_tasks_are_task_directories_read_back_as_the_rows_they_were(tmp_path, monkeypatch): + """prepare.py writes each task, verifier and rerun as a task directory (Harbor's format, no network, judged on the + host); aspire_fields.py, every config's data.view, reads each back as exactly the row tasks.jsonl held.""" + from hermit.config import Config + from hermit.task import Benchmark + P = _load("robosuite_prepare", EXAMPLE / "benchmark" / "prepare.py") + F = _load("aspire_fields", EXAMPLE / "agent" / "aspire_fields.py") + monkeypatch.setattr(P, "HERE", tmp_path) + monkeypatch.setattr(sys, "argv", ["prepare.py", "--preset", "smoke"]) + P.main() + found = F.task_rows(tmp_path / "tasks") + seeds = {"debug_seeds": "101-103", "unseen_seeds": "126", "eval_seeds": "1-3"} + task = {"id": "robosuite-cube_stack", "suite": "cube_stack", "task_id": 0} + assert json.dumps(found["robosuite-cube_stack"]) == json.dumps({**task, **seeds}) # its keys in order, types + assert json.dumps(found["verify-robosuite-cube_stack"]) == json.dumps( + {**task, "id": "verify-robosuite-cube_stack", "role": "verifier", "debug_seeds": "101-105", + "eval_seeds": "none", "max_versions": 2, "search": "debug", "findings": False}) + assert json.dumps(found["rerun-robosuite-cube_stack"]) == json.dumps( + {**task, "id": "rerun-robosuite-cube_stack", "role": "rerun", "findings": False, **seeds}) + assert len(found) == 21 and (tmp_path / "splits" / "smoke.txt").read_text().split() == \ + ["robosuite-cube_lifting", "robosuite-cube_stack"] + loaded = Benchmark.load(tmp_path / "tasks", view=F.aspire_task).tasks["robosuite-cube_stack"] + assert loaded.private["tests"] is None and loaded.private["needs"]["network"]["agent"] == "no-network" + for path in sorted((EXAMPLE.parent / "aspire").glob("*.yaml")): + config = Config.load(path) + data = config.data["data"] + assert config.path(data["tasks"]) == (EXAMPLE / "benchmark" / "tasks").resolve(), path + monkeypatch.setitem(data, "tasks", str(tmp_path / "tasks")) + for key in ("train", "val", "test"): + monkeypatch.setitem(data, key, str(tmp_path / "splits" / Path(data[key]).name)) + assert {t.id: t.given() for t in Benchmark.from_config(config).tasks.values()} == found, path + + def test_prepare_refuses_held_out_seeds_a_session_could_run(): for args in (["--eval", "1-200"], ["--unseen", "120-130"], ["--eval", "1-100", "--verify", "50-54"]): - done = subprocess.run([sys.executable, str(EXAMPLE / "prepare.py"), *args], capture_output=True, text=True) + done = subprocess.run([sys.executable, str(EXAMPLE / "benchmark" / "prepare.py"), *args], capture_output=True, + text=True) assert done.returncode == 2 and "overlap" in done.stderr, args diff --git a/tests/methods/evosafe/test_agentdyn.py b/tests/methods/evosafe/test_agentdyn.py index f8f8b0e2..b19e6244 100644 --- a/tests/methods/evosafe/test_agentdyn.py +++ b/tests/methods/evosafe/test_agentdyn.py @@ -92,7 +92,8 @@ def test_metrics_direction_and_separate_denominators(): def test_split_groups_and_no_answer_leakage(tmp_path): - spec = importlib.util.spec_from_file_location("prepare_expanded", ROOT / "examples/agentdyn/utils/prepare_expanded.py") + spec = importlib.util.spec_from_file_location("prepare_expanded", + ROOT / "examples/agentdyn/utils/benchmark/prepare_expanded.py") module = importlib.util.module_from_spec(spec) spec.loader.exec_module(module) for suite in module.SUITES: @@ -380,7 +381,8 @@ def test_native_three_suite_initialization(): get_suite("v1.2.2", "shopping") except KeyError: pytest.skip("upstream AgentDojo installed, not AgentDyn") - spec = importlib.util.spec_from_file_location("prepare_expanded", ROOT / "examples/agentdyn/utils/prepare_expanded.py") + spec = importlib.util.spec_from_file_location("prepare_expanded", + ROOT / "examples/agentdyn/utils/benchmark/prepare_expanded.py") module = importlib.util.module_from_spec(spec) spec.loader.exec_module(module) cases = {} @@ -416,8 +418,16 @@ def test_report_rejects_missing_and_separates_goal_namespaces(): module.compare(tasks,['a','b','c'],{'baseline':rows[:2]}) +def _cases(): + spec = importlib.util.spec_from_file_location("agentdyn_cases", ROOT / "examples/agentdyn/utils/benchmark/cases.py") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + def test_a_clean_case_with_empty_injection_fields_is_the_same_case(tmp_path): - spec = importlib.util.spec_from_file_location("prepare_cases", ROOT / "examples/agentdyn/utils/prepare.py") + spec = importlib.util.spec_from_file_location("prepare_cases", + ROOT / "examples/agentdyn/utils/benchmark/prepare.py") module = importlib.util.module_from_spec(spec) spec.loader.exec_module(module) cases = [{"suite": "shopping", "user_task_id": f"user_task_{n}", "injection_task_id": "injection_task_0", @@ -425,13 +435,55 @@ def test_a_clean_case_with_empty_injection_fields_is_the_same_case(tmp_path): cases += [{"suite": "shopping", "user_task_id": "user_task_0"}, {"suite": "shopping", "user_task_id": "user_task_0", "injection_task_id": None, "injections": {}}] module.prepare(cases, tmp_path / "out") - rows = [json.loads(s) for s in (tmp_path / "out/tasks.jsonl").read_text().splitlines()] + rows = _cases().read_cases(tmp_path / "out") # the cases' task directories, as their rows assert sum(r["user_task_id"] == "user_task_0" and r["kind"] == "clean" for r in rows) == 1 +def test_cases_are_task_directories_read_back_as_the_rows_tasks_jsonl_held(tmp_path, monkeypatch): + """write_cases writes each case as a task directory and the rows' order; read_cases and every AgentDyn and AgentDojo + config (data.view: cases.py:case_row) read each back as exactly its row, the injections' key order and the id's + place among sorted keys kept; the hash is the JSON lines'.""" + import hashlib + from hermit.config import Config + from hermit.task import Benchmark + cases = _cases() + rows = [{"suite": "shopping", "user_task_id": "user_task_3", "injection_task_id": "injection_task_1", + "injections": {"z_vector": "frozen \u2019payload\" \\ \n", "a_vector": "x"}, "id": "case-1", + "kind": "injected", "group": "shopping/user_task_3", "domain": "shopping"}, + {"id": "shopping-user_task_3-clean", "suite": "shopping", "user_task_id": "user_task_3", "kind": "clean", + "group": "shopping/user_task_3", "domain": "shopping"}] + for sort_keys in (False, True): + out = tmp_path / f"sorted-{sort_keys}" + digest = cases.write_cases(out, rows, benchmark="agentdyn", sort_keys=sort_keys) + lines = "".join(json.dumps(r, sort_keys=sort_keys) + "\n" for r in rows) + assert digest == hashlib.sha256(lines.encode()).hexdigest() + assert [json.dumps(r) for r in cases.read_cases(out)] == lines.splitlines() + out = tmp_path / "sorted-True" + (out / "split.txt").write_text("".join(r["id"] + "\n" for r in rows)) + configs = [*sorted((ROOT / "examples/agentdyn").glob("*/run*.yaml")), + *sorted((ROOT / "examples/agentdojo").glob("*/run*.yaml"))] + assert len(configs) == 11 + for path in configs: + config = Config.load(path) + data = config.data["data"] + assert config.path(data["view"].split(":")[0]) == ROOT / "examples/agentdyn/utils/benchmark/cases.py", path + monkeypatch.setitem(data, "tasks", str(out / "tasks")) + for key in ("train", "val", "test"): + if data.get(key): + monkeypatch.setitem(data, key, str(out / "split.txt")) + bench = Benchmark.from_config(config) + assert [json.dumps(t.given()) for t in bench.split("train")] == \ + [json.dumps(r, sort_keys=True) for r in rows], path + assert bench.split("train")[0].private["tests"] is None # judged by judge.py + assert cases.rows(out / "tasks") == cases.read_cases(out) # what report.py --tasks reads + legacy = [{**rows[1], "injection_task_id": None, "injections": {}}] # a clean case of an old tasks.jsonl + cases.write_cases(tmp_path / "none", legacy, benchmark="agentdyn") + assert cases.read_cases(tmp_path / "none") == legacy # null kept, not dropped + + @pytest.mark.parametrize("goals", [0, -1]) def test_all_groups_needs_a_positive_goal_count(tmp_path, goals): - path = ROOT / "examples/agentdyn/utils/prepare_all_groups.py" + path = ROOT / "examples/agentdyn/utils/benchmark/prepare_all_groups.py" spec = importlib.util.spec_from_file_location("prepare_groups", path) module = importlib.util.module_from_spec(spec) spec.loader.exec_module(module) @@ -440,7 +492,7 @@ def test_all_groups_needs_a_positive_goal_count(tmp_path, goals): def test_full_test_keeps_search_cases_and_only_adds_test_owned_variants(tmp_path): - path = ROOT / "examples/agentdyn/utils/prepare_all_groups.py" + path = ROOT / "examples/agentdyn/utils/benchmark/prepare_all_groups.py" spec = importlib.util.spec_from_file_location("prepare_groups", path) module = importlib.util.module_from_spec(spec) spec.loader.exec_module(module) @@ -450,7 +502,7 @@ def test_full_test_keeps_search_cases_and_only_adds_test_owned_variants(tmp_path rows, groups = [], [] for split in module.SPLITS: groups.append({"suite": "github", "user_task_id": split, "split": split}) - members = [dict(id=f"{split}-{n}", suite="github", user_task_id=split, + members = [dict(id=f"{split}-{n}", suite="github", user_task_id=split, # a clean case: null, {} injection_task_id=f"injection_task_{n}" if n else None, injections={"vector": f"frozen-{n}"} if n else {}) for n in range(4)] rows.extend(members) @@ -465,8 +517,8 @@ def test_full_test_keeps_search_cases_and_only_adds_test_owned_variants(tmp_path for split in ("train", "val"): assert (expanded / f"{split}.txt").read_bytes() == (original / f"{split}.txt").read_bytes() assert (expanded / "test.txt").read_text() == (original / "test.txt").read_text() + "test-3\n" - old_rows = {r["id"]: r for r in map(json.loads, (original / "tasks.jsonl").read_text().splitlines())} - new_rows = {r["id"]: r for r in map(json.loads, (expanded / "tasks.jsonl").read_text().splitlines())} + old_rows = {r["id"]: r for r in _cases().read_cases(original)} + new_rows = {r["id"]: r for r in _cases().read_cases(expanded)} assert {i: new_rows[i] for i in old_rows} == old_rows assert new_rows.keys() - old_rows.keys() == {"test-3"} assert new_rows["test-3"]["injections"] == {"vector": "frozen-3"} diff --git a/tests/methods/evosafe/test_agentdyn_experiment_runtime.py b/tests/methods/evosafe/test_agentdyn_experiment_runtime.py index 482709e2..da343dc6 100644 --- a/tests/methods/evosafe/test_agentdyn_experiment_runtime.py +++ b/tests/methods/evosafe/test_agentdyn_experiment_runtime.py @@ -17,6 +17,45 @@ from experiment_runtime import snapshot, verify_snapshot +def test_a_task_set_of_task_directories_is_frozen_and_hashed_whole(tmp_path, monkeypatch): + """A protocol's tasks/ (task directories) is copied into the snapshot, not read as a file, and digest() hashes + every file of it: a changed case changes the digest.""" + from experiment_runtime import digest + root = tmp_path / 'repo' + for name in ['template', 'task-info']: + (root / name).mkdir(parents=True) + (root / name / 'info').write_text('committed context') + (root / 'worker.py').write_text('def run(task, adapter):\n pass\n') + import yaml + config_data = {'data': {'tasks': 'data/tasks', **{k: 'data/split.txt' for k in ('train', 'val', 'test')}}, + 'agent': {'name': 'worker.py:run', 'params': {}}, 'judge': {'path': 'worker.py'}, + 'trainer': {}, 'task_info': 'task-info', 'template': 'template', 'evolver': {'name': 'she'}} + (root / 'run.yaml').write_text(yaml.safe_dump(config_data)) + (root / '.gitignore').write_text('data/\n') + subprocess.run(['git', 'init', '-q', str(root)], check=True) + subprocess.run(['git', '-C', str(root), 'add', '.'], check=True) + subprocess.run(['git', '-C', str(root), '-c', 'user.name=Fixture', '-c', 'user.email=fixture@example.invalid', + 'commit', '-qm', 'fixture'], check=True) + case = root / 'data/tasks/case-1' + case.mkdir(parents=True) + (case / 'task.toml').write_text('[metadata]\nid = "case-1"\n') + (case / 'instruction.md').write_text('A case.\n') + (root / 'data/split.txt').write_text('case-1\n') + before = digest(root / 'data/tasks') + config = SimpleNamespace(data=config_data, run_dir=lambda: tmp_path / 'output/run', + run_name=lambda: 'run', path=lambda value: (root / value).resolve(), + template_dir=lambda: root / 'template', tree_dir=lambda method: tmp_path / ('tree-' + method)) + monkeypatch.setenv('HA_BACKENDS', str(tmp_path / 'none.yaml')) + meta = snapshot(root, root / 'run.yaml', config) + frozen_tasks = Path(yaml.safe_load(Path(meta['config']).read_text())['data']['tasks']) + assert frozen_tasks.is_dir() and (frozen_tasks / 'case-1/task.toml').read_text() == '[metadata]\nid = "case-1"\n' + assert digest(frozen_tasks) == before + (case / 'task.toml').write_text('[metadata]\nid = "case-2"\n') + assert digest(root / 'data/tasks') != before + with pytest.raises(FileNotFoundError): + digest(root / 'data/missing.jsonl') + + @pytest.mark.parametrize('with_validation', [True, False]) @pytest.mark.parametrize('disable_sandbox', [True, False]) def test_snapshot_survives_changes_to_original_code_config_and_data(tmp_path, monkeypatch, with_validation, disable_sandbox): @@ -210,3 +249,46 @@ def test_a_frozen_run_cannot_receive_an_unrecorded_sandbox_override(tmp_path): capture_output=True, text=True, timeout=15) assert result.returncode == 2 assert '--no-sandbox must be applied before source freezing' in result.stderr + + +def test_snapshot_freezes_the_view_and_refuses_one_outside_the_repository(tmp_path, monkeypatch): + """data.view is read code like the agent and the judge: the snapshot's own copy, never a live file elsewhere.""" + import yaml + root = tmp_path / 'repo' + for name in ['template', 'task-info']: + (root / name).mkdir(parents=True) + (root / name / 'info').write_text('committed') + (root / 'worker.py').write_text('') + (root / 'view.py').write_text('def row(task):\n return task\n') + outside = tmp_path / 'outside.py' + outside.write_text('def row(task):\n return {}\n') + config_data = {'data': {**{k: 'cases.txt' for k in ('tasks', 'train', 'test')}, 'view': 'view.py:row'}, + 'agent': {'name': 'worker.py:run', 'params': {}}, 'judge': {'path': 'worker.py'}, + 'trainer': {}, 'task_info': 'task-info', 'template': 'template'} + config_path = root / 'run.yaml' + config_path.write_text(yaml.safe_dump(config_data)) + subprocess.run(['git', 'init', '-q', str(root)], check=True) + subprocess.run(['git', '-C', str(root), 'add', '.'], check=True) + subprocess.run(['git', '-C', str(root), '-c', 'user.name=Fixture', '-c', 'user.email=fixture@example.invalid', + 'commit', '-qm', 'fixture'], check=True) + (root / 'cases.txt').write_text('cases') + config = SimpleNamespace(data=config_data, run_dir=lambda: tmp_path / 'output/run', + run_name=lambda: 'run', path=lambda value: (root / value).resolve(), + template_dir=lambda: root / 'template', tree_dir=lambda _: tmp_path / 'tree') + monkeypatch.setenv('HA_BACKENDS', str(tmp_path / 'none.yaml')) + meta = snapshot(root, config_path, config) + view = yaml.safe_load(Path(meta['config']).read_text())['data']['view'] + assert view == str(Path(meta['snapshot_root']) / 'view.py') + ':row' and 'view.py' in meta['file_sha256'] + config_data['data']['view'] = f'{outside}:row' + config_path.chmod(0o644) + config_path.write_text(yaml.safe_dump(config_data)) + subprocess.run(['git', '-C', str(root), '-c', 'user.name=Fixture', '-c', 'user.email=fixture@example.invalid', + 'commit', '-qam', 'outside'], check=True) + with pytest.raises(ValueError, match='data.view must be a committed entry point'): + snapshot(root, config_path, config) + config_data['data']['view'] = 'some_installed_benchmark.fields:row' # a module, read wherever it is installed + config_path.write_text(yaml.safe_dump(config_data)) + subprocess.run(['git', '-C', str(root), '-c', 'user.name=Fixture', '-c', 'user.email=fixture@example.invalid', + 'commit', '-qam', 'module'], check=True) + with pytest.raises(ValueError, match='data.view must be a committed file entry point'): + snapshot(root, config_path, config) diff --git a/tests/methods/mwm/test_mwm_evolver.py b/tests/methods/mwm/test_mwm_evolver.py index 79c85b9c..e356f08d 100644 --- a/tests/methods/mwm/test_mwm_evolver.py +++ b/tests/methods/mwm/test_mwm_evolver.py @@ -123,7 +123,7 @@ def make(tmp: Path, monkeypatch, steps: int = 2, deployment_param: bool = False) (tmp / "agent" / "pipeline.py").write_text(PIPELINE) # the task sandbox's one declared path (tmp / "judge.py").write_text("def score(task, output):\n return 1.0\n") (tmp / "echo.py").write_text("def echo(text):\n return {'text': text}\n") - deployment = Path(__file__).resolve().parents[3] / "examples/clbench_database/utils/task-info/mwm" + deployment = Path(__file__).resolve().parents[3] / "examples/clbench_database/utils/benchmark/task-info/mwm" (tmp / "task-info").mkdir() params = {"evidence": str(evidence), "retry_wait_s": 0, "session_timeout_s": 120} if deployment_param: # the venue texts beside the config, named by params.deployment diff --git a/tests/selection.py b/tests/selection.py index 8b06039f..1cf62140 100644 --- a/tests/selection.py +++ b/tests/selection.py @@ -58,7 +58,8 @@ def fnmatch(path: str, pattern: str) -> bool: "harness2": {"continual_harness"}} # Continual Harness deploys with the harness2_codex template # what every method is built on: a change here runs every group (tests/core/test_selection.py checks that every # framework module these import, or a method's tests import, is here too) -SHARED = ["hermit/agent/__init__.py", "hermit/agent/base/*", "hermit/agent/environment/*", "hermit/agent/acting/*", +SHARED = ["hermit/agent/__init__.py", "hermit/agent/base/*", "hermit/environment/*", "hermit/agent/environment/*", + "hermit/agent/acting/*", "hermit/harbor/*", "hermit/agent/evolver/*.py", "hermit/agent/evolver/methods/__init__.py", @@ -67,7 +68,7 @@ def fnmatch(path: str, pattern: str) -> bool: "hermit/redact.py", "hermit/_moved.py", # the moved modules' forwards in hermit/agent/base "hermit/hosts/*", "hermit/runtime/*", "hermit/adapter/*.py", "hermit/config.py", "hermit/defaults.yaml", "hermit/tools.py", - "hermit/task.py", "hermit/__init__.py", "hermit/adapter/templates/prompt_only/*", # the toy task's + "hermit/task/*", "hermit/common.py", "hermit/__init__.py", "hermit/adapter/templates/prompt_only/*", # toy "pyproject.toml", "conftest.py", "tests/conftest.py", "tests/fakes.py", "tests/toy_task.py", "tests/selection.py", "tests/pi_stand.py", "tests/fixtures/pi/*", # the Pi stand-in: the methods'