Conversation
Collaborator
|
Small upgrade-path edge case: the stage cache identity is unchanged from the previous runner, so an existing default stage created with It may be worth adding a staging-format/version value to |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #27 and fixes #28. This PR is scoped to
benchmarks/frontierchallenge/.1. One official scoring rule
task_score > 0.999. Exactly 0.999 does not pass. Native per-task pass thresholds are not used; partial-credit arithmetic is unchanged.passedfield. Native pass decisions are removed from published verifier diagnostics; raw native logs are confined to temporary storage and cleaned up on success/failure.score-gt-0.999. Separate subset reports must declare their denominator.2. Effective task selection and old-cache upgrades (#27)
task_098_orca_claisen_thermochemistryanalyzes supplied outputs without executing ORCA and remains open-track..frontier-sourceasdereferenced-v2to force a one-time restage of oldcp -acaches, addressing @zhanghanduo's upgrade-path comment.HF verification and supporting setup fixes (#28)
HF pins:
6306f5d07fcd23c911242a36c6c923e9b2ad19bfe3e6719c43de28cc0f0116cf1f1cd95e05825aabValidation
ready: 81 tasks.claude-opus-5E2E: task_011 scores 0.65 and task_098 scores 0.98685; both haveevaluation_complete=1,passed=0, no trial exceptions, and runner exit 0. Frozen judge configuration is retained. The fixed-97 summary correctly labels this two-task run incomplete.Licensed ORCA calculation E2E remains unverified without an authorized ORCA 6.0.1 installation. task_098 reads precomputed output and is not an ORCA runtime test.
Both HF cards state that the runtime rollout is pending this PR's merge.