Conversation
Quantify Podman/Docker sandbox failures from a training run's logs (Beaker experiment or local files): concurrency saturation, host rotation/cooldown, disk exhaustion, OOM, step/reset timeouts, and the "running hot" regime (Ray actor/node death + resource exhaustion). Emits counts and per-rollout rates with FATAL vs recoverable-INFRA totals; each signature cites the emitting code site. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Broaden node_marked_dead to catch Ray node-death phrasings ("actor's node
was terminated", "node has died", missed-heartbeats), which the ps768
crash surfaced -- FATAL was undercounting node/raylet OOM deaths.
- Add --compare to print 2+ logs side-by-side (count + rate/rollout per
signature, with FATAL/INFRA/rollout totals), so the throttled-vs-running-hot
profile shift is visible at a glance. Sources are file paths or exp:<id>.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a third SandboxBackend that runs sandboxes as modal.Sandbox containers instead of on-node Podman/Docker. Selected via "backend": "modal" in the env config; defaults to the agent-training Modal environment. File I/O is exec-piped like ApptainerBackend so exception semantics match the other backends; sandboxes get a hard lifetime plus an atexit reaper to bound the cost of leaks. Verified end-to-end against real Modal (ai2-reviz), including an egress check from ai2/jupiter (scripts/modal/check_modal_egress.sh). Includes a Modal variant of the tmax 2-node toy launch script and promotes modal to a core dependency so the training image gets it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
SWERLVanilluxSandboxEnvConfig is a fixed-field dataclass, so "sandbox_lifetime" in tool_configs fails validation at startup. Use SWERL_MODAL_SANDBOX_LIFETIME_S instead, which ModalBackend already reads. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
modal.Image.from_registry requires Python + pip inside the image and its build fails otherwise, which is common for task images (the tmax images are bare Ubuntu 22.04). This bricked the first Modal training run: every sandbox create failed and the trainer waited on step 0 forever. Try the image as-is first so images that ship their own Python aren't shadowed by a standalone one, then retry with add_python (version from SWERL_MODAL_ADD_PYTHON_VERSION, default 3.12) and remember the outcome per tag. Verified against the exact task image that failed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Modal warns on every blocking call made from an async context; with hundreds of env actors making sandbox calls this floods training logs (most of the log volume in the first Modal run). The blocking usage is deliberate since SandboxBackend is a sync interface. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
add_python drops a standalone interpreter into /usr/local/bin, which shadows the image's own /usr/bin/python3. tmax task verification runs `python3 -m pytest`, which then hits the bare standalone interpreter (no pytest, no task deps) and unconditionally writes reward 0 — every rollout group had zero reward variance, so active sampling discarded all of them and training never produced a batch (258 rollouts, 0 accepted; the podman baseline accepts its first group within ~74). After a fallback start, if the image has its own python3 outside /usr/local/bin, remove the standalone's interpreter/pip links so resolution returns to the image's python. Verified on the failing image: python3 -> /usr/bin/python3 (3.10) with pytest 9.0.3. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The first Modal toy run billed $1,219 vs a ~$15 estimate; the overage was sandbox-hours nobody was using. Fixes: - Close the sandbox at episode end (submit and OOM paths) in both SWERL envs instead of holding it idle until the env's next reset. Also removes most end-of-run leakage since envs finish their last episode with no live sandbox. - ModalBackend.close(): retry terminate once and log failures loudly instead of suppressing them (a failed terminate bills until the lifetime cap). - scripts/modal/cleanup_modal_sandboxes.sh: janitor to terminate all live sandboxes for an app after killing a job. - Toy launch script: sandbox lifetime 4h -> 1h (caps worst-case leak 4x), per-run Modal app name so the janitor can target it. - Note the measured toy-run cost in the podman-vs-modal doc. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Episodes in flight when training completes never reach the submit-path close, so the pool's sandboxes (122 of 128 in the last run) leak until their lifetime cap. Chain the janitor after grpo_fast.py so a graceful job end cleans them up; hard kills still need the janitor run manually. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…service New SandboxBackend that runs each sandbox as a pod on a self-hosted OpenSandbox deployment (GKE Autopilot). Unlike Modal, the OCI image is pulled as-is, so there is no rebuild/fidelity gap. Includes the same leak protections as ModalBackend (hard lifetime cap, loud kill retry, atexit reaper, end-of-job janitor), an egress-check feasibility script, a toy launch script, and unit tests with a fake SDK. Verified live against the AI2 GKE deployment: warm start ~5.3s, ~1.05s/exec, all six backend methods and error semantics pass. Known issue documented: cold-start creates can 504 at the ingress's 30s timeout while still creating the pod server-side (orphan reclaimed by the janitor via the open_instruct_app metadata tag). Also renames docs/sandbox_modal_vs_podman.md to docs/sandbox_management.md, rewrites it to document all four backends as implemented (present tense, no stale line numbers), and updates all references. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When a create outlasts the ingress's upstream timeout, the client gets HTTP 504 but the sandbox still comes up server-side (sometimes more than once), so every cold start both errored AND leaked billed orphan pods. start() now tags each create with a unique open_instruct_create_id; on a gateway-timeout error it polls the management API for a sandbox carrying that tag (bounded by ready_timeout), adopts it via SandboxSync.connect, and kills any duplicates. If nothing appears, the original error is re-raised. Also fixes _sandbox_is_alive comparing against "RUNNING": the SDK's SandboxState.RUNNING is "Running" (mixed case), so the liveness probe never matched and any transient exec error would have restarted the sandbox, wiping episode state mid-rollout. Live verification against the AI2 GKE deployment: after the LB backend timeout was raised server-side, a 101s cold create (golang image + node provisioning) completed on the normal path; the adoption path is covered by unit tests with the fake SDK. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The toy launch script defaulted SWERL_OPENSANDBOX_PROTOCOL to http when unset in the launching shell, so training jobs spoke plain HTTP to the HTTPS-only endpoint. The load balancer resets those connections immediately, which made every sandbox create fail within seconds with "[Errno 104] Connection reset by peer" / "Server disconnected" and deadlocked two toy runs at step 0 (identical at pool_size 128 and 16 -- initially misread as a concurrency ceiling). Default to https, and log an explicit hint from OpenSandboxBackend.start() when a create dies with a connection reset while protocol is http. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pairs OpenSandboxBackend's "sandbox started"/"Closing" log lines into per-sandbox lifetimes and prices total sandbox-hours at GKE Autopilot per-request rates (cpu/memory/lifetime auto-detected from the log). Handles Ray log dedup by counting "[repeated Nx]" multipliers and pricing hidden events at the mean observed lifetime, and prices unclosed sandboxes up to end-of-log bounded by the lifetime cap. Cross-check against the BigQuery billing export for ground truth. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ray log dedup hides Closing lines, so per-id pairing misread most closed sandboxes as unclosed and priced each up to the lifetime cap (2.7x overestimate on the first pool-128 run: $44.54 vs $13.84). Reconcile starts vs closes globally (including dedup multipliers and end-of-job janitor kills) so only the true start/close gap is priced as leaked; other unpaired sandboxes get the mean paired lifetime. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
EnvironmentPool now counts in-use actors, blocked acquirers, and acquire wait times; stats() snapshots and resets them. run_training polls every pool once per training step and logs pool/<name>/in_use, in_use_peak, waiting, acquires, and acquire_wait_s_mean, so pool_size saturation (in_use pinned at size with nonzero waiting/wait times) is visible on wandb instead of requiring log archaeology. Analysis of the first pool-128 OpenSandbox run showed the pool pinned at 128 for essentially the whole run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A pool-768 run collapsed at startup: ~768 concurrent creates overwhelmed the control plane (mass 504s) and forced Autopilot node provisioning in bulk, so pods sat Pending past the 180s ready_timeout and nearly all post-504 adoptions gave up on pods that were about to come up -- env retries then amplified the load (99 sandbox starts in 2h vs 3,205 at pool 128). Wrap creates (including the adoption poll) in a per-node file-slot semaphore, SWERL_OPENSANDBOX_START_CONCURRENCY (default 64), so large pools ramp instead of stampeding -- running sandboxes hold no slot, so steady-state still reaches full pool size. Raise the ready_timeout default from 180s to 600s (SWERL_OPENSANDBOX_READY_TIMEOUT_S) so creates and adoptions survive node-provisioning waves. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
SWERL_RESET_FAILURE_ZERO_REWARD defaults to off and the toy-script lineage (podman -> modal -> opensandbox) never set it, so a single env reset that exhausted its retries (create 504 + failed adoption) crashed the whole pool-768 training job during step 2. With the flag on, such resets become zero-reward rollouts and the job survives. Proper infra-failure semantics (exclude from training instead of reward 0) remains follow-up work. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sandbox image pulls happen on the GKE nodes, not the training job, so the Podman-style local docker login does nothing for this backend. Pass credentials through the SDK's SandboxImageSpec/SandboxImageAuth instead: OpenSandboxBackend reads DOCKERHUB_USERNAME / DOCKER_PAT (the same convention as the Podman setup) and attaches auth to the create call when both are set. The toy launch script wires the same Beaker secret used by the Podman script. Anonymous Docker Hub pulls are rate-limited per NAT IP, which mass pod scale-out trips. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When a sandbox vanished during an exec (Spot preemption, crash, lifetime expiry), OpenSandboxBackend and ModalBackend restarted a fresh sandbox and retried the command. The replacement is a blank container -- no task seed files, no agent-built state -- so the episode continued against a corrupted environment and scored garbage rewards silently (~66 episodes in the first Spot-backed toy run). Raise SandboxDiedError (mirroring the SandboxOOMError contract) from both backends instead, and end the episode in the SWERL envs with reward 0 and metadata.sandbox_died=True. OpenSandboxBackend.close() now also skips the kill-retry-and-alarm path when the sandbox is already gone. This makes Spot sandbox pods safe: a preemption costs one episode, not signal integrity. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The SWERL envs now track sandbox_died and oom_killed per episode (set by the terminal handlers, cleared on reset) and report them from get_metrics(). The existing rollout-state aggregation averages these over each training step's batch, so wandb shows env/<env_name>/sandbox_died and env/<env_name>/oom_killed as the fraction of rollouts lost to Spot preemption/expiry and OOM kills. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…aker A spot-backed toy run silently stalled for 12.5 hours between training steps 3 and 4 with zero log output, leaving no evidence of where the main thread was blocked (the step-4 weight sync completed in 0.75s once whatever it waited on released). Two defenses: - MainLoopStallWatchdog (utils.py): the training loop heartbeats at each phase; a daemon thread dumps all thread stacks via faulthandler when no beat arrives for SWERL_MAIN_LOOP_STALL_DUMP_S (default 1800), repeating until the loop advances. health_check_fn's silent weight-sync handshake poll now also logs every 120s while waiting. - Circuit breaker (vllm_utils.py): a sandbox on a preempted Spot node blackholes the exec stream instead of erroring, so every tool step times out while the env actor sits blocked -- observed as hour-long zombie rollouts that collapsed generation throughput and likely triggered the sync hang. After SWERL_MAX_CONSECUTIVE_TOOL_TIMEOUTS (default 3) consecutive tool-step timeouts the rollout ends instead of grinding through max_steps x tool_call_timeout. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The create-throttle change added ", semaphore_wait=Xs" to the sandbox started log line, which the estimator's regex did not match, so it found zero lifecycle events in newer logs. Match both formats and subtract the semaphore queue time when backdating the billing start (billing begins after the slot is acquired, not when start() is entered). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Records the planned design (valid_mask in compute_group_advantages + the existing keep_idxes drop machinery + mask_infra_failed_completions flag) and why it matters at observed Spot death rates, plus the retry-in-process_request follow-up. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The estimator priced everything at on-demand rates. Sandbox capacity
type is a deployment property invisible to the client, but the run's
endpoint domain (in the Starting log lines) identifies which server it
used. Add --pricing {auto,on-demand,spot}: auto selects spot when the
domain contains a --spot-domain-markers substring, spot applies
--spot-discount (default 70%) to the on-demand rates, and spot output
carries the caveats (preempted pods stop billing before the lifetime
cap; realized discounts vary -- BigQuery Spot SKUs are ground truth).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When a post-504 adoption gives up, the training log only said 'no sandbox appeared', hiding WHY the pod never became Running -- a ps768 run burned an hour on what was likely capacity exhaustion with no identifiable cause in the log. The failure path now distinguishes never-admitted creates from stuck-Pending pods, logs each stuck pod's status state/reason/message, and relays the server's diagnostics API content (Kubernetes events: quota names, IP_SPACE_EXHAUSTED, GCE_STOCKOUT, ...). Servers without the diagnostics API (current AI2 deployment answers DIAGNOSTICS_NOT_IMPLEMENTED) get one warning and no repeated fetch attempts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Clone of qwen35_4b_base_tmax_10k.sh with the on-node Podman plumbing (subcontainer perms, podman services, registry mirror, docker_login) replaced by the OpenSandbox config from the 2-node toy script: endpoint env vars, registry auth via SandboxImageSpec, create throttle, reset zero-reward flag, and the end-of-job janitor. pool_size is 512 rather than 1024: the Spot-backed server's observed capacity ceiling is ~450-520 sandboxes and oversubscribing it slows the run (see the ps768 postmortem notes in docs/sandbox_management.md). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
OpenSandbox's built-in TTL reaper runs inside the control plane it protects; during the 2026-08-12 overload it stopped reaping and ~2,300 expired-but-running pods plus ~2,300 finished BatchSandbox records accumulated, exhausting Spot capacity and stalling a production run for 19 hours in a create-fail/retry spiral. This CronJob is scheduled by Kubernetes and deletes through the Kubernetes API directly, so it keeps working while the OpenSandbox server/controller are unhealthy: finished (Succeed/Failed) CRs older than 30 min and any CR older than 2 h are removed every 10 minutes. Apply once per sandbox cluster with kubectl. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
bitnami/kubectl:1.35 does not exist (Bitnami's public catalog was discontinued); use gcr.io/cloud-builders/kubectl, which also pulls in-GCP without touching Docker Hub. Scope the Role to the CRD's actual group (sandbox.opensandbox.io) instead of a wildcard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Drives OpenSandboxBackend (the production code path: create throttle, adopt-on-504, SandboxDiedError handling, registry auth) with N concurrent workers looping realistic episodes (create -> spaced execs -> close) to answer whether the deployment sustains multi-run load without paying for GPUs. Reports create/exec latencies, failure taxonomy, preemption deaths, and judges pass/fail on steady-state create p95 and failure rate (cold-cluster ramp creates excluded). Smoke-tested live: 8 episodes, 22/22 execs, steady create p50 4.5s, zero sandboxes leaked. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Prod script: add the WANDB_API_KEY Beaker secret (required in the oe-agents workspace) and point at that workspace. Toy script: reflect its current use as the concurrency acceptance test (start concurrency 128, distinct exp_name, description). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… stall training wandb.log hands its payload to the wandb service process over a local socket and waits on a future with no timeout. When the service wedges, the caller blocks indefinitely: three long GRPO runs stalled this way, each with the main thread stuck inside wandb.log at the end of one_training_step for 3-12 hours while every GPU idled (stack traces captured by the stall watchdog were identical across incidents). Add AsyncWandbLogger (utils.py): log() enqueues to a bounded queue and returns immediately; a daemon thread is the only wandb.log caller, preserving step ordering. If the queue fills because the SDK stopped draining, entries are dropped with a loud warning - losing metrics is acceptable, losing training time is not. flush() (atexit-registered) counts in-flight entries so a healthy run's final metrics still land. Route the two training-loop call sites (per-step metrics in grpo_fast.one_training_step, eval metrics in grpo_utils.maybe_evaluate) through it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
SWERL_OPENSANDBOX_IMAGE_PREFIX (or image_prefix kwarg) rewrites bare Docker Hub references to pull through a pull-through mirror -- an Artifact Registry remote repository in-region with the sandbox cluster. user/img:tag gains the prefix, official images additionally gain Docker Hub's implicit library/ namespace, and already registry-qualified references pass through unchanged. Rewritten pulls skip the Docker Hub username/PAT (mirror access uses the cluster's own credentials). Each unique tag is fetched from Docker Hub once ever (by the mirror); node pulls stay in-region, removing multi-minute cold pulls on fresh nodes and Docker Hub rate-limit exposure. Verified live against the AI2 mirror with an official and a task image (create + exec both paths). Launch scripts default the prefix to the AI2 mirror, overridable or blankable from the launching shell. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Explain what SWERL_OPENSANDBOX_IMAGE_PREFIX does (pull-through cache of Docker Hub, in-region node pulls, no Hub rate limits) and how to bypass it; also drop the stale "Autopilot" and "no registry mirror" phrasing now that the cluster is GKE Standard and mirror pulls exist. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
checkpoint_state_freq was set but checkpoint_state_dir was not, so the save block never ran and the 2026-08-21 run lost all 84 steps (21.5h on 32 GPUs) when Beaker preempted one node; point it at weka so a node death now costs at most 10 steps. SWERL_OPENSANDBOX_CPU=2 (overridable): at cpu=1, ~3 sandboxes per 4-vCPU e2 spot node ran nodes at 77-85% CPU and stretched rollouts ~2.4x with ~500 exec timeouts/hr; doubling the reservation is the contention experiment. Header documents the packing consequence (one sandbox per e2-standard-4 -> node autoscaler max must clear ~520 nodes or the machine type must grow). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Workers check stop_event at the loop top, then can spend minutes queued in the create semaphore inside backend.start(); after --duration-s expired, queued workers emerged in waves and each ran a full episode, keeping the test creating sandboxes long past its deadline (observed: +900 CRs ten minutes after the 1024-worker run ended). Close the fresh sandbox and exit instead when the stop arrived mid-create. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
launch_load_test_beaker.sh submits a CPU-only mason job (ai2/hammond) that runs run_load_test_beaker.sh: two side-by-side 1024-worker load tests with distinct app names, simulating two concurrent training runs' sandbox churn on the production network path. The job prints both final summaries, janitor-sweeps both app tags, and exits nonzero if either process misses the success criteria. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…issing Docker 20.10 without the buildx plugin reports "unknown flag: --platform" for the buildx invocation. Detect the plugin and fall back to DOCKER_BUILDKIT=1 docker build; the platform flag is unnecessary on linux/amd64 hosts and the registry cache-to/from is buildx-only, so the fallback builds without the shared cache. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
At full CPU quota there is no warm-node headroom, so every Spot death routes replacement creates onto freshly provisioned nodes (~2-3 min GCE boot) and the default 30s create-p95 criterion fails even when the service is healthy (2026-08-25 run: p50 5.2s, p95 ~140s, failure rate 0.7-0.8%, zero control-plane restarts). Note in the launcher header how to read the verdict at this scale. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… script mason gains --min_runtime (Beaker duration string, e.g. '90m'): the scheduler will not preempt the job before it has run this long. Needs beaker-py >= 2.7.2 (BeakerTaskContext.min_runtime), so bump the pin. Pass the string through verbatim — a bare int is interpreted as nanoseconds by the client. The 4-node OpenSandbox prod script sets BEAKER_MIN_RUNTIME (default 2h): enough to clear the ~25-40 min first step and reach the first checkpoint_state save at step 10, so a preemption never sends a retry back to step 1. Both recent prod attempts died to node preemption. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ompletions) Spot-preempted sandboxes end episodes with a fake zero reward (~5-15% of episodes, 30% in spikes), and GRPO's group-relative advantages then push the policy away from trajectories that were fine until the pod vanished. The rollout loop now flags info.infra_failed on mid-episode sandbox death, environment reset failure, and consecutive-timeout circuit breaker trips. OOM kills are deliberately NOT flagged: the policy's own commands caused them, so their zero reward is legitimate feedback. With --mask_infra_failed_completions (default false), group advantage statistics are computed over surviving samples only via a valid_mask in compute_group_advantages (masked mean/std; advantage 0 for failed samples; all-invalid groups get all zeros), and the failed rollouts are then dropped from the batch through the existing keep_idxes machinery. val/infra_failed_count|rate are reported on the unfiltered batch whether or not masking is on. The OpenSandbox prod script enables it (MASK_INFRA_FAILED=false to opt out); other launch configs unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Google dropped the transitional google-cloud-sdk apt package from packages.cloud.google.com; installs now fail with "Package 'google-cloud-sdk' has no installation candidate ... replaced by google-cloud-cli". Same binaries (gcloud, gsutil), new package name. Breaks every image build from any commit, so this needs to reach main too. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ur override --pricing auto detected spot from a "spot" substring in the endpoint domain — a relic of the two-cluster era. The current single-domain deployment has no marker, so a Spot ps1024 run was silently priced on-demand (3.3x too high, $1918 vs $575). auto now prints a loud warning when it falls back to on-demand. --sandbox-hourly-cost prices sandbox-hours at a flat rate, bypassing the vCPU/GiB pod model: on GKE Standard each cpu=2 sandbox occupies a whole e2-standard-4 spot node (~$0.040/hr), which the pod model undercounts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A killed Beaker job never reaches the trailing janitor; at pool_size 1024 that leaves ~1,000 live sandboxes holding the whole node pool and CPU quota for up to one lifetime, and an immediately relaunched run ramps into leftovers (creates 504 while the pool is full — observed 2026-09-09). Sweep with the janitor before relaunching, or wait one sandbox lifetime. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…not resume its predecessor The checkpoint dir was a hardcoded literal while exp_name was edited per-experiment; the loss-exclusion run silently auto-resumed the previous run's step-320 state because the trainer treats an existing checkpoint dir as its own retry. One EXP_NAME variable now feeds both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Experiment config inherited verbatim from omni_agent's qwen35_9b_dppo_repro_4node_64k.sh (DPPO tv@0.1, liger loss, 64k response, 8x32 rollouts, 16 engines / 8+8 learners); only the sandbox plumbing differs: OpenSandboxBackend with the AR pull-through mirror, cpu=2, create throttle, loss exclusion (MASK_INFRA_FAILED, default on), checkpoint_state_dir derived from EXP_NAME, Beaker min_runtime, and a distinct app tag (swerl-tmax-9b-opensandbox) so the janitor and metrics never collide with the 4B run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
BEAKER_CLUSTER / BEAKER_WORKSPACE env overrides (defaults unchanged: ai2/jupiter, ai2/oe-agents) so the same script launches on the B300 cluster with BEAKER_CLUSTER=ai2/holmes BEAKER_WORKSPACE=ai2/oe-agents-holmes. Add ai2/holmes to WEKA_CLUSTERS (matching main) so mason mounts weka there — the checkpoint_state and rollouts paths need it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
OpenSandboxBackend— RL sandboxes on a self-hosted OpenSandbox service on GKE — as an alternative to on-node Podman and Modal, plus the trainer-side robustness work that a month of production runs on it demanded.Sandbox backend
OpenSandboxBackend(all sixSandboxBackendmethods), verified live; selected viabackend: opensandboxin tool_configsSWERL_OPENSANDBOX_START_CONCURRENCY), terminalSandboxDiedErroron mid-episode death (no more silent blank-container restarts — ModalBackend too), registry auth, adoption-failure diagnosticsSWERL_OPENSANDBOX_IMAGE_PREFIX): one Docker Hub fetch per tag ever, in-region node pullsTrainer robustness
--mask_infra_failed_completions): sandbox deaths / reset failures / breaker trips are excluded from GRPO group mean/std (newvalid_maskincompute_group_advantages) and dropped from the batch;val/infra_failed_ratereported always. Without it, Spot preemption's fake zeros (~5–15% of episodes) bias group advantagesSWERL_MAX_CONSECUTIVE_TOOL_TIMEOUTS): ends rollouts on unresponsive envs instead of grinding through max_steps × timeoutMainLoopStallWatchdog): faulthandler stack dumps when the trainer stops advancing — this is what caught the wandb.log hang (Publish wandb metrics from a background thread so a wedged SDK cannot stall training #1841, cherry-picked here)wandb.logmoved off the main thread; a wedged wandb SDK now costs dropped metrics, not stalled training (three multi-hour production stalls traced to this)pool/<name>/in_use|waiting|acquire_wait_s_mean)--min_runtime(Beaker preemption-protection window; needs beaker-py ≥ 2.7.2, pin bumped)Ops
scripts/opensandbox/: egress check, janitor (by app tag), cost estimator (spot/on-demand), GC CronJob manifest, synthetic load test + Beaker batch-job wrapper (validated the service at 2,048 concurrent sandboxes / four ps512 runs' churn)docs/sandbox_management.md(renamed from sandbox_modal_vs_podman.md): all four backends, tuning, incident postmortemsgoogle-cloud-sdk→google-cloud-cli(upstream package removed; unbreaks all image builds); build_image_and_launch buildx fallback for Docker 20.10 hostsValidation
Validated by extended production GRPO runs (4 nodes × 8 GPUs, 512–1024 concurrent sandboxes, 100+ training steps across runs) and a 2×1024-worker synthetic scale test against the live service; 42 unit tests for the backend, 11 for masked advantages, plus watchdog/pool suites.
GPU_TESTS=bypass (infra/backend branch validated by the production runs above; the GRPO loss-path change is unit-tested and flag-gated, default off)
Note for the merger: two known conflicts vs omni_agent (
pyproject.toml,uv.lock) from the beaker-py2.5.7 → 2.7.2bump — keep>=2.7.2and re-lock.🤖 Generated with Claude Code