Case timeout kills custom engine before it can flush its session-result, losing all evaluation data for that case

[BiBoyang]

Summary

When a case times out, skill-up kills the custom engine process outright. The engine never gets a chance to write its ${output_file} session-result, so all evaluation data for that case is lost — including the early-turn events that are already available.

Minimal reproduction

1. Configure a custom engine (transport: local, response_format: session_result) whose adapter writes the SessionResult JSON to ${output_file} after the child CLI process exits (the documented contract).
2. Create a case whose prompt makes the agent do real work (e.g. a heavy research task), with cases.defaults.timeout_seconds: 180.
3. skill-up run evals/eval.yaml

[ERROR] case trigger-pos-digest-repo: agent execution failed: custom engine run failed:
context deadline exceeded (case timeout 180s via cases.defaults.timeout_seconds)

Post-mortem — only the input was persisted:

$ ls <output-dir>/iteration-1/trigger-pos-digest-repo/with_skill/outputs/agent/run/
messages.json
$ # session-result.json absent — the agent's transcript died with the killed process

Expected vs actual

• Expected: on case timeout the engine gets a chance to flush a partial/timeout-marked result (e.g. SIGTERM with a grace window), or the harness synthesizes a SessionResult with exit_code: 124. At minimum the docs should state that a timed-out custom-engine case yields no transcript at all.
• Actual: the engine process is killed outright; nothing downstream can grade the case. For trigger-style evaluation this is especially costly: skill-activation events happen in the first turns and are already available, but they die with the process.

Environment

• skill-up: local build dev-fix253 (upstream main + a fix for Skill source behind a symlink silently installs 0 files (filepath.Walk does not follow symlinked root) #253), darwin/arm64
• engine: custom local transport (reproduced with both a kimi adapter and a Claude Code adapter)

Workaround we use

Action-free prompts for heavy tasks ("assume the repo is already cloned, describe your process") plus per-case timeout_seconds bumps. Fragile — any prompt that slips through silently loses its data point.

Fix ownership (PR sketch, no code)

skill-up side, runtime/process management:

• In the custom-engine local runner, on deadline expiry send SIGTERM first, wait a short configurable grace period (5–10s), then SIGKILL. Most adapter scripts can write a timeout-marked result in that window.
• Smaller alternative: when the deadline fires and no output file exists, synthesize a SessionResult with exit_code: 124 and stderr: "case timeout" so downstream tooling at least has a record.
• Touch points: the internal/agent/custom_local.go exec path plus wherever the case-level context.WithTimeout is enforced; needs verification against both local and http transports.

Happy to turn this into a PR if the approach sounds right.