Skip to content

fix(frontierchallenge): unify scoring and repair task-cache upgrades - #30

Open
fzp0424 wants to merge 11 commits into
ApodexAI:mainfrom
fzp0424:fix/frontierchallenge-open-track
Open

fzp0424 wants to merge 11 commits into
ApodexAI:mainfrom
fzp0424:fix/frontierchallenge-open-track

Conversation

@fzp0424

@fzp0424 fzp0424 commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #27 and fixes #28. This PR is scoped to benchmarks/frontierchallenge/.

1. One official scoring rule

  • A task passes only when evaluation completes and its valid, unrounded task_score > 0.999. Exactly 0.999 does not pass. Native per-task pass thresholds are not used; partial-credit arithmetic is unchanged.
  • Rewards and summaries expose only the official passed field. Native pass decisions are removed from published verifier diagnostics; raw native logs are confined to temporary storage and cleaned up on success/failure.
  • Refuse incompatible legacy job resumes before staging. Use a fresh job name when upgrading; historical rewards/logs are not silently rewritten or mixed into new-policy jobs.
  • Use the fixed 97-task denominator for headline metrics, flag missing/incomplete work, and identify the policy as score-gt-0.999. Separate subset reports must declare their denominator.
  • Align the benchmark README, scoring/running guides, and both HF dataset cards with this policy. Historical site numbers are labeled rather than silently re-scored.

2. Effective task selection and old-cache upgrades (#27)

  • Resolve one registry-backed task selection before staging, ORCA preflight, unsealing and Harbor. Stale or excluded directories cannot add tasks to a run.
  • task_098_orca_claisen_thermochemistry analyzes supplied outputs without executing ORCA and remains open-track.
  • Dereference HF cache symlinks when staging. Version .frontier-source as dereferenced-v2 to force a one-time restage of old cp -a caches, addressing @zhanghanduo's upgrade-path comment.
  • Regression tests cover stale licensed directories, include/exclude selection, broken cached symlinks and reuse after migration.

HF verification and supporting setup fixes (#28)

  • Only the mutable root dataset card is outside payload checksums; task-level README files and all payload commitments remain checked.
  • Pin the verified solve/reference revisions and require their registries to match the runtime. The HF updates change cards only, not task payloads, reference archives, image bytes or access settings.
  • Keep the HF task-image integrity checks working on Docker 29/containerd by validating the loaded OCI manifest's link to the published config digest inside the SHA-256-verified archive.
  • Correct the evaluator Python requirement to 3.12+, matching pinned Harbor 0.20.0; synchronize installation guidance and add a regression test. Frozen scientific task images are unchanged.

HF pins:

  • Solve: 6306f5d07fcd23c911242a36c6c923e9b2ad19bf
  • Reference: e3e6719c43de28cc0f0116cf1f1cd95e05825aab

Validation

  • 114 FrontierChallenge tests, including scoring boundaries, log sanitation, legacy job rejection, cache migration, task selection and image-identity tampering checks.
  • All 97 published reference adapters patched and compiled; 582 injected grading cases preserve partial scores and verify the official pass/log contract. These are adapter tests, not model E2E.
  • On Linux ECS: both HF packages verify all 97 tasks/archives; full open setup loads/verifies the HF image and reports ready: 81 tasks.
  • Real Claude Code / claude-opus-5 E2E: task_011 scores 0.65 and task_098 scores 0.98685; both have evaluation_complete=1, passed=0, no trial exceptions, and runner exit 0. Frozen judge configuration is retained. The fixed-97 summary correctly labels this two-task run incomplete.
  • Actual ECS upgrade checks: an unselected licensed directory does not block the open run; three broken links in a real legacy HF cache are repaired; old jobs are rejected before stage changes. Published native diagnostic files contain no alternate pass decisions.
  • Benchmark scripts are unchanged by the scope cleanup, so the completed E2E evidence remains applicable. Remote CI is rerun on the cleanup commit.

Licensed ORCA calculation E2E remains unverified without an authorized ORCA 6.0.1 installation. task_098 reads precomputed output and is not an ORCA runtime test.

Both HF cards state that the runtime rollout is pending this PR's merge.

@zhanghanduo

Copy link
Copy Markdown
Collaborator

Small upgrade-path edge case: the stage cache identity is unchanged from the previous runner, so an existing default stage created with cp -a can still pass the reuse check and skip the new cp -aL. If that cached copy contains Hugging Face cache-relative symlinks, they may remain broken after upgrading unless the evaluator uses --force-restage or a fresh stage directory.

It may be worth adding a staging-format/version value to .frontier-source, checking cached tasks for symlinks before reuse, or otherwise forcing a one-time restage.

@fzp0424 fzp0424 changed the title fix(frontierchallenge): make open-track selection reproducible fix: align FrontierChallenge release, scoring, and public Docker setup Sep 23, 2026
@fzp0424 fzp0424 changed the title fix: align FrontierChallenge release, scoring, and public Docker setup fix(frontierchallenge): unify scoring and repair task-cache upgrades Sep 23, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants