feat(evals): group trajectories by run and cross-link Braintrust experiments

[miguelg719]

What

Stacked on #2138.

• Persist trajectories under <root>/<experiment>__<model>/<task.id>/<runId>/ instead of scattered per-task timestamp dirs, so a run's trajectories live in one folder and concurrent multi-model runs of the same suite never interleave on disk.
• Write a metadata.json into every trajectory dir (experiment, model, provider, environment, task, runId, status) so a trajectory's run never has to be reverse-engineered after the fact.
• Write an experiment.json at the group root cross-linking the local trajectories to the resolved Braintrust experiment (hashed name, id, project, URLs) once Eval() resolves.
• New framework/trajectoryGroup.ts owns the slugging/layout/metadata helpers; TrajectoryRecorder, the external-harness persister, and both eval entrypoints (index.eval.ts, framework/runner.ts) consume it.

Why

Mapping on-disk trajectories back to their Braintrust experiment previously required guessing by timestamp, and concurrent runs of different models wrote into the same root with no on-disk marker — making post-hoc analysis error-prone.

Notes

• Local persistence only; Braintrust experiment naming/metadata is unchanged.
• Tooling that globs .trajectories/* keeps working (one extra path level); the trajectoryRecorder unit test asserts the new layout + metadata.json.

---

Summary by cubic

Groups eval trajectories by experiment, unambiguous model, and a run-unique token, and cross-links each run to its Braintrust experiment. Adds atomic dir reservation, single-flight caching, and env hygiene so runs never overwrite, mislabel, or leak; the run token uses 64‑bit entropy and result persistence is order-free (also trims internal comments; no logic changes).

• New Features

  • Store trajectories at <root>/<experiment>[__<model>]__<runToken>/<task.id>/<runId>/; write metadata.json per trajectory and an experiment.json at the group root linking to the resolved Braintrust experiment; implemented in framework/trajectoryGroup.ts and used by the recorder, adapter persister, runner, and index.eval.ts.
  • Generate one run token per run (64‑bit entropy); derive the group/metadata model from the actual testcase matrix when unambiguous (EVAL_TRAJECTORY_MODEL), not the requested override; stamp EVAL_EXPERIMENT_NAME/EVAL_TRAJECTORY_GROUP; writeExperimentLink() takes an explicit group, respects EVAL_TRAJECTORY_ROOT, and only writes if persistence is on and the group dir exists.
  • Bench hardening: add taskDates.ts for safe rolling dates (WebMD); update Healthline to stop before final subscribe; clarify Google Maps rubric; KFC requires pickup scheduling; SFPL uses the live application URL; default maxSteps: agent/sf_library_card 1→10, agent/sf_library_card_multiple 20→25. TUI verify help and docs reflect the new layout.

• Bug Fixes

  • Prevent clobbering by atomically reserving trajectory dirs (reserveTrajectoryDir adds -2/-3 on collision) with single-flight caching in the recorder; finish() is idempotent; result persistence is order-free (now covered by tests); metadata.json records runDir and attempt.
  • Hardened slugging: reject pure-dot values (e.g., ".", "..") to avoid path escapes; fall back to the "default" group.
  • REPL env hygiene: withEnvOverrides() snapshots/restores run-stamped env (EVAL_TRAJECTORY_GROUP, EVAL_EXPERIMENT_NAME, EVAL_TRAJECTORY_MODEL) so runs don’t leak between commands; recorder and adapter both honor EVAL_TRAJECTORY_ROOT.
  • Trimmed internal comments and moved a misplaced docstring; no logic changes.

^{Written for commit 2d17642. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 2d17642

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

1 issue found across 6 files

Confidence score: 4/5

• This PR is likely safe to merge, with one moderate-risk edge case rather than a broad functional regression.
• In packages/evals/index.eval.ts, experiment.json may be written under a different root than trajectory outputs when EVAL_TRAJECTORY_ROOT is set, which can cause confusing or inconsistent eval artifacts for users relying on aligned output locations.
• Because the issue is scoped to a specific environment-variable path configuration (severity 5/10, high confidence) and not a clear break of core flows, the overall merge risk stays moderate-low.
• Pay close attention to packages/evals/index.eval.ts - ensure experiment.json and trajectory files resolve to the same root when EVAL_TRAJECTORY_ROOT is enabled.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/evals/index.eval.ts">

<violation number="1" location="packages/evals/index.eval.ts:567">
P2: `experiment.json` can be written to a different root than the actual trajectories when `EVAL_TRAJECTORY_ROOT` is set.</violation>
</file>

Architecture diagram

sequenceDiagram
    participant Runner as Eval Runner (index.eval.ts / runner.ts)
    participant TG as trajectoryGroup.ts
    participant TR as TrajectoryRecorder
    participant PH as Persist Adapter (persistTrajectory.ts)
    participant FS as Local Filesystem
    participant BT as Braintrust SDK (Eval())

    Note over Runner,BT: Eval run setup — stamp env vars before tasks execute

    Runner->>TG: buildTrajectoryGroupSlug(experimentName, model?)
    TG-->>Runner: "experimentName__model" (or just "experimentName")
    Runner->>Runner: set process.env.EVAL_TRAJECTORY_GROUP = slug
    Runner->>Runner: set process.env.EVAL_EXPERIMENT_NAME / MODEL / PROVIDER

    Note over TR,FS: Per-task trajectory persistence

    Runner->>BT: Eval() — execute all tasks
    BT-->>Runner: EvaluationResult (hashed experiment name/id/URLs)

    alt For each task
        alt TrajectoryRecorder path (framework runner)
            Runner->>TR: recordTrajectory(taskSpec, runId)
            TR->>TG: resolveTrajectoryDir(root, taskId, runId)
            TG-->>TR: "<root>/<group>/<taskId>/<runId>"
            TR->>FS: write trajectory files to computed dir
            TR->>TG: writeTrajectoryMetadata(dir, {task, runId, status})
            TG->>FS: write metadata.json (experiment, model, provider, env, task, runId, status)
        else PersistAdapter path (external harness)
            Runner->>PH: persistAdapterTrajectory(opts)
            PH->>TG: resolveTrajectoryDir(root, taskId, runId)
            TG-->>PH: "<root>/<group>/<taskId>/<runId>"
            PH->>FS: write trajectory files
            PH->>TG: writeTrajectoryMetadata(dir, {task, runId, status})
            TG->>FS: write metadata.json
        end
    end

    Note over Runner,FS: Post-run — Braintrust experiment link

    Runner->>TG: resolveTrajectoryRoot()
    TG-->>Runner: "<root>" (default ./.trajectories)
    Runner->>TG: writeExperimentLink(root, resolvedBraintrustMetadata)
    TG->>FS: mkdir <root>/<group>/
    TG->>FS: write experiment.json (hashed name, id, URLs, requested name)
    Note over TG,FS: Best-effort — errors silently ignored

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 7 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/evals/framework/trajectoryRecorder.ts">

<violation number="1" location="packages/evals/framework/trajectoryRecorder.ts:281">
P2: The new order-free flow still loses data when `persistResult()` runs before `finish()`: `finish()` overwrites `task_data.json` and drops `result`. If this order is supported, preserve/merge existing `task_data.json` during `finish()` or re-apply `result` after writing trajectory files.</violation>
</file>

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: The new order-free flow still loses data when persistResult() runs before finish(): finish() overwrites task_data.json and drops result. If this order is supported, preserve/merge existing task_data.json during finish() or re-apply result after writing trajectory files.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/framework/trajectoryRecorder.ts, line 281:

<comment>The new order-free flow still loses data when `persistResult()` runs before `finish()`: `finish()` overwrites `task_data.json` and drops `result`. If this order is supported, preserve/merge existing `task_data.json` during `finish()` or re-apply `result` after writing trajectory files.</comment>

<file context>
@@ -225,13 +248,38 @@ export class TrajectoryRecorder {
-    const scoresDir = path.join(this.outputDir, "scores");
+    // Route through the shared reservation so scores land in the same dir as
+    // the trajectory regardless of finish()/persistResult() call order.
+    const { directory } = await this.ensureReserved();
+    const scoresDir = path.join(directory, "scores");
     await fs.mkdir(scoresDir, { recursive: true });
</file context>

[miguelg719]

Does not reproduce — leaving as-is. persistResult() writes <dir>/scores/result.json, while finish() → writeTrajectoryDir() writes task_data.json, trajectory.json and screenshots/ and never touches scores/. The result keys writeTrajectoryDir does write are step.toolOutput.result (per-step tool output), which is unrelated to the EvaluationResult that persistResult() persists — so a persistResult()-before-finish() ordering would not drop it.

Independently, the order is never reversed at the only call site (verifierAdapter.ts: await recorder.finish(...) then await recorder.persistResult(...)). The related real issue was the shared reservation, which is fixed in f3e4824.

[miguelg719]

Re-checked this with a test rather than leaving it at my earlier reading — pushed in the latest commit.

The test runs persistResult() before finish() (the reverse of the adapter's order) and asserts the score survives, every trajectory file is present, and the reservation did not split. It passes, so the reported data loss does not reproduce:

• persistResult() writes <dir>/scores/result.json.
• finish() → writeTrajectoryDir() writes task_data.json, trajectory.json, screenshots/ — and never touches scores/.
• The result keys writeTrajectoryDir does write are step.toolOutput.result (per-step tool output), unrelated to the EvaluationResult that persistResult() persists.

So there is no merge/re-apply needed. Keeping the test regardless of the disagreement: the PR advertises that result persistence is order-free, and an advertised ordering guarantee with no test is one refactor away from being false. Happy to be shown a concrete failing sequence if you think I have missed one.

[cubic-dev-ai]

All reported issues were addressed across 9 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[miguelg719]

@cubic-dev-ai

[cubic-dev-ai]

@cubic-dev-ai

@miguelg719 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

All reported issues were addressed across 19 files

Confidence score: 5/5

• Safe to merge after the addressed issues were fixed.

Architecture diagram

sequenceDiagram
    participant Entrypoint as "Eval Entrypoint (runner.ts / index.eval.ts)"
    participant TG as trajectoryGroup.ts
    participant Recorder as TrajectoryRecorder
    participant Adapter as persistAdapterTrajectory
    participant FS as "Filesystem (disk)"
    participant Braintrust as "Braintrust SDK"

    Note over Entrypoint,FS: NEW: Run-scoped trajectory grouping

    Entrypoint->>TG: generateRunToken()
    TG-->>Entrypoint: "20260715-110342"
    Entrypoint->>TG: buildTrajectoryGroupSlug(experiment, model, runToken)
    TG-->>Entrypoint: "agent__openai_gpt-4.1-mini__20260715-110342"
    Entrypoint->>Entrypoint: set process.env.EVAL_TRAJECTORY_GROUP = group
    Note over Entrypoint: Also sets EVAL_EXPERIMENT_NAME, EVAL_MODEL_OVERRIDE, EVAL_PROVIDER

    par For each task
        Entrypoint->>Recorder: new TrajectoryRecorder(taskSpec)
        Note over Recorder: Captures group from env at construction (snapshot)
        Recorder->>Recorder: ensureReserved() - single-flight caching

        Recorder->>TG: reserveTrajectoryDir(root, taskId, runId, group)
        critical Atomic directory creation
            TG->>FS: mkdir <root>/<group>/<taskId>/<runId> (non-recursive)
            alt Success
                FS-->>TG: OK
            else Directory exists (EEXIST)
                loop attempt=2..N
                    TG->>FS: mkdir <root>/<group>/<taskId>/<runId>-<attempt>
                    alt Success
                        FS-->>TG: OK
                    else EEXIST
                        Note over TG,FS: EEXIST, retry
                    end
                end
            end
        end
        TG-->>Recorder: { directory, attempt }

        Recorder->>FS: writeTrajectoryDir(directory, trajectory)
        Recorder->>TG: writeTrajectoryMetadata(directory, { task, runId, runDir, attempt, status })
        TG-->>FS: writes metadata.json

        Recorder->>FS: persistResult(...) - also uses same reservation

        opt adapter/harness persist
            Adapter->>TG: resolveTrajectoryDir(root, taskId, runId)
            Adapter->>TG: reserveTrajectoryDir(...)
            Adapter->>FS: writeTrajectoryDir + writeTrajectoryMetadata
        end
    end

    Entrypoint->>Braintrust: Eval() - resolves experiment name (hashed)
    Braintrust-->>Entrypoint: summary (experimentName, experimentId, URLs)

    Entrypoint->>TG: writeExperimentLink(root, group, {...})
    Note over Entrypoint,TG: Cross-link local trajectories to Braintrust experiment
    alt Persistence enabled (shouldPersistTrajectory)
        TG->>FS: mkdir <root>/<group>
        TG->>FS: write <root>/<group>/experiment.json
    else Persistence disabled (CI, core-tier)
        TG-->>Entrypoint: No-op (no stray .trajectories/)
    end

    Note over Recorder,FS: Collision avoidance: re-run with same runId -> `run-1-2` dir
    Note over Entrypoint: Generated run token ensures runs never clobber each other’s experiment.json

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 7 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>