Fix terminal evaluation recovery and SFT data filtering for the KDA checkpoint

[abhishekraok]

Codex · Codex task

Pradeep’s continued-SFT KDA checkpoint scores 1/100 on TBLite and 0/89 on Terminal Bench 2.1. Inspection of all 189 saved trajectories found failed training demonstrations, long no-action reasoning loops, recovery prompts followed by premature submission, and two harness bugs. Fix these before deciding whether more training is needed; their score improvements have not been measured in an A/B.

Parent checkpoint: 1-epoch-dolci-think (#1859).

Runs:

1. TBLite, pass@1 1%: Beaker
2. Terminal Bench 2.1, pass@1 0%: Beaker

Run provenance and outcomes

Both evaluate /weka/oe-adapt-default/allennlp/deletable_checkpoint_states/ecppxpon/hf_step604, continued from the durable step23607 supplied above. Training: one epoch, 604 steps, LR 2.5e-5, seq 65,536; W&B run c08ewfs2. Serving: vLLM 0.19.1, qwen3_xml + olmo3 parsers, 65,536 context, two data-parallel replicas and eight concurrent trials; tmax revision 595caabfeaec.

Mutually exclusive trial ending	TBLite	TB 2.1
Agent wall-clock timeout	50	40
Submit command executed	40	39
Context limit reached	6	7
64-step limit reached	3	2
Individual bash command timeout, 120 seconds	1	1
Total	100	89

Both Beaker jobs exited successfully. Trial outcomes and verifier results explain the scores.

Findings

1. Failed demonstrations were included in SFT. The matching cache, /weka/oe-adapt-default/allennlp/deletable_open_instruct_dataset_cache/numpy_sft/bb1d94703a-6068a350, contains 633M tokens from SimFC and TMAX. Its manifest records TMAX revision 00b28cd071976a2cadcde3f5c5201f772eba6d6c, whose default config is all: 16,728 trajectories, versus 10,029 in successful. The cache confirms all 16,728 were loaded and ordinary tokenization/filtering retained 16,037. Even if all 691 dropped rows were unsuccessful, at least 6,008 unsuccessful trajectories remained. TMAX supplies 53.2% of trainable tokens. The exact retained success count was not recomputed.

2. Long reasoning without actions consumes much of the budget. With the exported tokenizer, at least 47 TBLite tasks and 44 TB tasks contain a completed no-tool reply retokenizing to 16,380–16,384 tokens. These counts overlap the timeouts above. The 114 near-cap replies consume about 57% of logged completed LLM-call latency, summed across concurrent requests. Timed-out in-flight replies are absent from saved trajectories.

3. Recovery frequently leads directly to submission. After a missing tool call, the harness says:

If you want to end the task, please issue the command echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT

Then 37/40 TBLite submissions and 38/39 TB submissions occur immediately after the correction. Only one submitted task passes. This is an observed sequence; whether the wording causes the exits needs an A/B.

For bash-log-processor-fix, the model reads files, produces two 16K-token no-action replies, and submits without editing the requested script. The final sequence is:

Model:    [second ~16K-token discussion; no bash call]
Harness:  [format error, including the submit-command suggestion]
Model:    bash(command="echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT")
Harness:  [ends the agent attempt and runs the verifier]
Verifier: processed.log was not created. Score: 0.0

echo prints a marker; the agent recognizes it and breaks the action loop. The verifier grades the files/environment produced so far. Submission does not establish completion or correctness.

4. Context budgeting and multiple-call handling are broken. The agent always requests 16,384 output tokens, so vLLM rejects prompts above 49,152 tokens; the agent catches the exception and stops. This accounts for the 13 context exits. Separately, the parser takes only tool_calls[0], while the agent retains the entire assistant message. 43 additional calls across 21 tasks receive no execution/result.

Proposed follow-up

• Train on load_dataset("allenai/tmax-sft-glm-52", "successful", split="train"). The inspected mixer does not expose HF config selection; a local Parquet export is a supported input route.
• Remove the submit example from format-error recovery. Request the next concrete action, e.g. “Your response contained no bash tool call. Take one concrete action toward completing the task. Describing a solution does not create or modify the required files.” Keep normal finishing instructions in the original task prompt.
• Pair that recovery change with a 4K per-action completion cap as a diagnostic. Set the agent limit, e.g. tmax runner EXTRA_AGENT_KWARGS=max_tokens=4096; its explicit request overrides the server default. A lower cap alone may just truncate the same loop earlier.
• Budget output against actual rendered prompt length and compact history when necessary.
• Execute all returned tool calls with matching responses, or explicitly reject multiple calls and repair the history; avoid silently dropping them.
• Run a controlled small-task comparison before another full benchmark or increasing training duration. Track score, no-action cap hits, timeout rate and recovery-to-submit transitions.

Checks and limits

Cached training and exported evaluation templates are byte-identical; both tool smoke tests passed, and 2,765 tool-result messages were recorded. No literal <tool_call> blocks were stranded in saved assistant content. Only three cached full-window rows ended on trainable non-EOS tokens, making truncation-tail repair a low-priority explanation here.

Real capability errors remain: kv-store-grpc passes 5/7 verifier tests but declares request field val where value is required. These traces do not distinguish looping inherited from the parent checkpoint from behavior intensified by continued SFT.

Three local diagnostic tests and Ruff passed: parser behavior, reconciliation of all trial endings/context errors, and the 75 recovery-to-submit transitions. No new inference/training or numerical export-parity check was run. No new GPU jobs were launched.