chore(pipeline): persist run progress [skip ci]

Author: github-actions[bot]

projects/PROJ-581-https-arxiv-org-abs-2605-13301/paper/reviews/paper_reviewer_claim_accuracy__2026-06-06__paper.md

@@ -0,0 +1,29 @@
+---
+action_items:
+- id: 7637034d8ff8
+  severity: writing
+  text: Fix broken citation key 'Math-Verify' in text to match existing bib entry
+    'chen2025xverify' or add correct entry.
+- id: 986489a3cf94
+  severity: science
+  text: Clarify the status of IMO 2025/USAMO 2026 results; future competition data
+    cannot be presented as established fact without verification.
+- id: 3e02e8d1e14f
+  severity: science
+  text: Verify existence and specifications of baseline models (GPT-5.5, Gemini 3.1
+    Pro) cited in Table 2.
+artifact_hash: 6b23039f76721ac00eaa6c408647f026893a62ad0f423ddd12fdde82e2327635
+artifact_path: projects/PROJ-581-https-arxiv-org-abs-2605-13301/paper/metadata.json
+backend: dartmouth
+feedback: ''
+github_authenticated: false
+model_name: qwen.qwen3.5-122b
+prompt_version: 1.1.0
+reviewed_at: '2026-06-06T07:27:05.906186Z'
+reviewer_kind: llm
+reviewer_name: paper_reviewer_claim_accuracy
+score: 0.0
+verdict: full_revision
+---
+
+The paper makes several factual claims that are currently unverifiable or unsupported by the provided bibliography, significantly impacting the claim accuracy. First, the central claim of achieving gold-medal-level scores on IMO 2025 and USAMO 2026 (Section Experimental Results, Table 3) refers to competitions that have not yet occurred in the real world. Presenting future competition results as established empirical facts undermines the scientific validity of the accuracy claims; these should be clearly labeled as projections or simulations if not actual historical data. Second, there are critical citation mismatches. The text cites `\citep{Math-Verify}` multiple times (e.g., Section 4.1, Appendix Evaluation Details), but the provided `iclr2026_conference.bib` does not contain an entry with this key (only `chen2025xverify` is visible). This breaks the link between the claim and its source, hindering verification. Third, specific parameter claims, such as DeepSeek-V3.2 using "> 943.7B tokens of sparse pre-training" (Section Cost Analysis), rely on citations (`deepseekai2025deepseekv32`) that cannot be externally verified in this context and appear to be future-dated. Finally, claims regarding "GPT-5.5" and "Gemini 3.1 Pro" as baselines (Table 2) reference models that are not publicly available or documented in standard repositories, making the comparative accuracy claims speculative. Without verifiable sources for these baselines and competition results, the paper's core accuracy claims cannot be validated. Please revise to ensure all data points are sourced from accessible, existing literature or clearly distinguished as hypothetical to maintain scientific rigor and reproducibility.

projects/PROJ-581-https-arxiv-org-abs-2605-13301/paper/reviews/paper_reviewer_code_quality_paper__2026-06-06__paper.md

@@ -0,0 +1,49 @@
+---
+action_items:
+- id: 771a3c1a892f
+  severity: science
+  text: Code artifacts (training scripts, evaluation harness) are not included in
+    the submission package. Reviewer cannot assess modularity, tests, or dependency
+    hygiene.
+- id: 127db961bf1f
+  severity: science
+  text: Appendix provides hyperparameters but lacks implementation details (e.g.,
+    custom RL modifiers, data loader structure) required for reproducibility from
+    scratch.
+artifact_hash: 6b23039f76721ac00eaa6c408647f026893a62ad0f423ddd12fdde82e2327635
+artifact_path: projects/PROJ-581-https-arxiv-org-abs-2605-13301/paper/metadata.json
+backend: dartmouth
+feedback: ''
+github_authenticated: false
+model_name: qwen.qwen3.5-122b
+prompt_version: 1.1.0
+reviewed_at: '2026-06-06T07:32:38.469535Z'
+reviewer_kind: llm
+reviewer_name: paper_reviewer_code_quality_paper
+score: 0.0
+verdict: minor_revision
+---
+
+## Re-Review: Code Quality Assessment
+
+This re-review follows the prior bar set by the previous code_quality_paper review. My assessment focuses strictly on whether the two prior action items have been adequately addressed in the current revision.
+
+### Prior Item Status
+
+**Item 771a3c1a892f (Code artifacts not included):** NOT ADDRESSED. The paper submission still contains only LaTeX source and references to external repositories (GitHub: https://github.com/Simplified-Reasoning/SU-01, HuggingFace: https://huggingface.co/Simplified-Reasoning/SU-01). No actual training scripts, evaluation harness, or test files are included in the submission package. Without access to the actual code, I cannot assess modularity, test coverage, or dependency hygiene.
+
+**Item 127db961bf1f (Missing implementation details):** NOT ADDRESSED. While the appendix now provides hyperparameters (learning rates, batch sizes, optimizer settings in `app:rl-training-details`, `app:sft-training-details`), critical implementation details remain absent:
+- Custom RL modifiers (GSPO advantage estimator implementation)
+- Data loader structure for the 338K SFT trajectories
+- Experience replay admission/retirement logic
+- Self-refinement prompt templates
+
+### New Issues
+
+None identified in this re-review.
+
+### Recommendation
+
+The paper remains in a state where code quality cannot be assessed because the artifacts are external to the submission. For reproducibility claims to be verifiable, either:
+1. Include code artifacts in the submission package, or
+2. Provide complete implementation specifications in the appendix sufficient to re-implement the pipeline from scratch

projects/PROJ-581-https-arxiv-org-abs-2605-13301/paper/reviews/paper_reviewer_data_quality_paper__2026-06-06__paper.md

@@ -0,0 +1,44 @@
+---
+action_items:
+- id: c8f5785a602c
+  severity: writing
+  text: Explicitly declare licenses for all training data sources (e.g., DeepMath,
+    NaturalReasoning) in a data card appendix to ensure compliance and reproducibility.
+- id: 667e8b44e469
+  severity: writing
+  text: Provide version commit hashes or specific release tags for external datasets
+    (e.g., HuggingFace URLs) to prevent reproducibility drift due to dataset updates.
+- id: 78e138deddfa
+  severity: science
+  text: Detail the schema and exact methodology for 'filtering contaminated problems'
+    to verify that test sets (IMO 2025, USAMO 2026) were not included in the 338K
+    SFT trajectories.
+- id: f71b80352868
+  severity: writing
+  text: Archive or replace external links (e.g., Evan Chen, AoPS) with persistent
+    identifiers (DOIs or Wayback Machine snapshots) to mitigate link rot risks.
+artifact_hash: 6b23039f76721ac00eaa6c408647f026893a62ad0f423ddd12fdde82e2327635
+artifact_path: projects/PROJ-581-https-arxiv-org-abs-2605-13301/paper/metadata.json
+backend: dartmouth
+feedback: ''
+github_authenticated: false
+model_name: qwen.qwen3.5-122b
+prompt_version: 1.1.0
+reviewed_at: '2026-06-06T07:33:46.366177Z'
+reviewer_kind: llm
+reviewer_name: paper_reviewer_data_quality_paper
+score: 0.0
+verdict: full_revision
+---
+
+This re-review confirms that none of the four data quality action items from the prior review have been addressed in the current revision.
+
+1. **Licenses (Item c8f5785a602c):** Section 3.1 ("SFT Data Curation") lists datasets like DeepMath, NaturalReasoning, and Nemotron but does not declare their licenses. The Appendix lacks a data card with compliance information.
+
+2. **Versioning (Item 667e8b44e469):** HuggingFace URLs (e.g., `nvidia/Nemotron-Instruction-Following-Chat-v1`) are provided without commit hashes, release tags, or snapshot versions. This risks reproducibility drift if datasets are updated.
+
+3. **Contamination (Item 78e138deddfa):** Section 3.1 states "we first filter contaminated problems" but provides no schema, code, or methodology for verifying that test sets (IMO 2025, USAMO 2026) were excluded from the 338K SFT trajectories. Given the claim of gold-medal performance on these specific competitions, leakage is a critical science risk.
+
+4. **Link Rot (Item f71b80352868):** Footnotes reference external URLs (e.g., `web.evanchen.cc`, `artofproblemsolving.com`) without persistent identifiers (DOIs/Wayback). These links may become inaccessible over time.
+
+All prior items remain open. Please address these data provenance and integrity gaps to ensure the results are verifiable and reproducible.

projects/PROJ-581-https-arxiv-org-abs-2605-13301/paper/reviews/paper_reviewer_jargon_police__2026-06-06__paper.md

@@ -0,0 +1,64 @@
+---
+action_items:
+- id: 1659502cb402
+  severity: writing
+  text: Define all acronyms (SFT, RL, TTS, RLVR, GSPO, PPL) at first use in Abstract
+    or Introduction.
+- id: 0217a15613b9
+  severity: writing
+  text: Replace dense jargon (e.g., "post-trained backbone", "trajectory", "rollouts")
+    with plain English equivalents in Methods.
+- id: d11ed40a4199
+  severity: writing
+  text: Define niche acronyms (MoE, CoT, KL, API, GPU, LLM, AoPS, STEM, IF) in Related
+    Work or Appendix.
+artifact_hash: 6b23039f76721ac00eaa6c408647f026893a62ad0f423ddd12fdde82e2327635
+artifact_path: projects/PROJ-581-https-arxiv-org-abs-2605-13301/paper/metadata.json
+backend: dartmouth
+feedback: ''
+github_authenticated: false
+model_name: qwen.qwen3.5-122b
+prompt_version: 1.1.0
+reviewed_at: '2026-06-06T07:36:20.199673Z'
+reviewer_kind: llm
+reviewer_name: paper_reviewer_jargon_police
+score: 0.0
+verdict: minor_revision
+---
+
+## Jargon Overuse Assessment — Re-Review
+
+This re-review confirms that **none of the three prior action items from the previous jargon_police review have been adequately addressed** in the current revision. The manuscript continues to use field-specific terminology without sufficient definition for non-specialist readers.
+
+### Item 1: Undefined Acronyms in Abstract/Introduction (ID: 1659502cb402)
+**Status: UNADDRESSED**
+
+The Abstract (e000) introduces SFT, RL, TTS, RLVR, GSPO, and PPL without any definitions. For example:
+- "reverse-perplexity SFT" — SFT is undefined
+- "two-stage RL (coarse RLVR → refined proof-level RL)" — RL, RLVR undefined
+- "test-time scaling (TTS)" — TTS is defined in parentheses, but this appears late
+- "30B-A3B backbone with 200 RL steps" — RL undefined
+
+The Introduction similarly uses "post-trained backbone" without defining what a backbone is in this context.
+
+### Item 2: Dense Jargon in Methods (ID: 0217a15613b9)
+**Status: UNADDRESSED**
+
+The Methods sections continue to use opaque terminology:
+- "338 K filtered trajectories" (Section 3) — "trajectory" is never explained as "reasoning traces" or "solution paths"
+- "sub-8K-token trajectories" — no plain English equivalent
+- "rollouts" (Appendix RL Training Details) — undefined
+- "post-trained backbone" (Introduction, Section 3) — should be "model that has already undergone initial training"
+
+### Item 3: Niche Acronyms in Related Work/Appendix (ID: d11ed40a4199)
+**Status: UNADDRESSED**
+
+These acronyms appear without definition:
+- MoE (Related Work) — Mixture of Experts
+- CoT (Related Work) — Chain of Thought
+- KL (RL Training Details) — Kullback-Leibler divergence
+- API, GPU, LLM (Appendix) — undefined
+- AoPS, STEM, IF (Abstract/Related Work) — undefined
+
+### Recommendation
+All three items remain unresolved. The paper would benefit from a glossary in the Appendix or inline definitions at first use. Without these changes, the manuscript excludes readers outside the immediate LLM-research community.

projects/PROJ-581-https-arxiv-org-abs-2605-13301/paper/reviews/paper_reviewer_logical_consistency__2026-06-06__paper.md

@@ -0,0 +1,42 @@
+---
+action_items: []
+artifact_hash: 6b23039f76721ac00eaa6c408647f026893a62ad0f423ddd12fdde82e2327635
+artifact_path: projects/PROJ-581-https-arxiv-org-abs-2605-13301/paper/metadata.json
+backend: dartmouth
+feedback: ''
+github_authenticated: false
+model_name: qwen.qwen3.5-122b
+prompt_version: 1.1.0
+reviewed_at: '2026-06-06T07:25:31.614246Z'
+reviewer_kind: llm
+reviewer_name: paper_reviewer_logical_consistency
+score: 0.5
+verdict: accept
+---
+
+## Re-Review: Logical Consistency Assessment
+
+This re-review follows the prior bar set by the previous logical_consistency review (verdict: accept, no action items). Per protocol, I assessed: (a) whether all prior action items were addressed (none existed), and (b) whether the revision introduced new logical consistency issues.
+
+### Assessment of Prior Bar
+
+The prior review found robust internal logical consistency with no action items required. This is a re-review with an empty prior action item list, so condition (a) is vacuously satisfied.
+
+### New Issues Analysis
+
+I examined the manuscript for new logical consistency concerns:
+
+1. **Core claims and evidence alignment**: The gold-medal performance claim (IMO 2025: 35 pts, USAMO 2026: 35 pts) is directly supported by Table 3. The medal line thresholds (IMO: 35/28/19, USAMO: 25/18/11) are correctly cited in the caption. ✓
+
+2. **Training pipeline consistency**: The 200 RL steps claim is internally consistent (96 coarse + 104 refined = 200, Appendix). The 338K SFT trajectories claim matches the abstract's "≈340K" (minor rounding, not a logical flaw). ✓
+
+3. **Stage-wise performance progression**: Figure 5 shows SFT → AnswerBench 69.2% → 59.8%, then RL recovery to 77.2%. The paper acknowledges the SFT drop and frames it as behavioral specialization. This is logically consistent as a staged process. ✓
+
+4. **Minor presentation gaps (not logical inconsistencies)**:
+   - The "simple and unified" label vs. multi-stage pipeline complexity is a rhetorical choice, not a logical contradiction.
+   - Cross-domain generalization (math/physics RL → chemistry/biology) lacks mechanistic explanation but is empirically supported by Table 1 (FrontierScience-Olympiad: 61.5%).
+   - The "compact" model descriptor (30B-A3B) is relative; this is a claim framing issue, not a logical inconsistency.
+
+### Conclusion
+
+No new logical consistency issues were introduced. The paper's conclusions follow from its stated premises and evidence. The prior "accept" verdict stands.

projects/PROJ-581-https-arxiv-org-abs-2605-13301/paper/reviews/paper_reviewer_safety_ethics__2026-06-06__paper.md

@@ -0,0 +1,32 @@
+---
+action_items:
+- id: c000d926a058
+  severity: writing
+  text: Add explicit ethics statement regarding IRB approval or informed consent for
+    the three human gold-medal experts used in IMO/USAMO evaluation.
+- id: f28f296a1e2f
+  severity: writing
+  text: Include a discussion on dual-use risks and responsible release policies for
+    the SU-01 model, particularly given its generalization to Chemistry and Biology
+    domains.
+artifact_hash: 6b23039f76721ac00eaa6c408647f026893a62ad0f423ddd12fdde82e2327635
+artifact_path: projects/PROJ-581-https-arxiv-org-abs-2605-13301/paper/metadata.json
+backend: dartmouth
+feedback: ''
+github_authenticated: false
+model_name: qwen.qwen3.5-122b
+prompt_version: 1.1.0
+reviewed_at: '2026-06-06T07:30:57.954317Z'
+reviewer_kind: llm
+reviewer_name: paper_reviewer_safety_ethics
+score: 0.0
+verdict: minor_revision
+---
+
+This re-review assesses whether the prior safety and ethics action items were addressed in the current revision. My assessment is that both items remain unaddressed in the provided manuscript text.
+
+Regarding the human evaluation component, the manuscript explicitly states in Table 3 (caption) and Appendix Section "Evaluation Details" that "three human gold-medal experts" were used to evaluate IMO/USAMO solutions. However, there is no corresponding ethics statement confirming IRB approval, informed consent, or ethical oversight for these human subjects. Standard safety protocols require explicit disclosure of how human evaluators were recruited, compensated, and protected, particularly when their expertise is leveraged for model benchmarking. This omission persists from the prior review.
+
+Regarding dual-use risks, the Abstract and Table 1 highlight "strong cross-domain scientific generalisation," specifically citing performance in Chemistry (69.4%) and Biology (25.0%) within the FrontierScience-Olympiad benchmarks. Given the potential for advanced reasoning models to assist in generating chemical or biological pathways, a discussion on dual-use risks and responsible release policies is necessary. The current text lacks a dedicated section or paragraph addressing these risks, model release constraints, or mitigation strategies for potential misuse in scientific domains.
+
+As neither item has been adequately addressed, the verdict remains `minor_revision`. Please add a formal ethics statement regarding the human experts and include a risk discussion section before the Conclusion or in the Appendix.

projects/PROJ-581-https-arxiv-org-abs-2605-13301/paper/reviews/paper_reviewer_scientific_evidence__2026-06-06__paper.md

@@ -0,0 +1,53 @@
+---
+action_items:
+- id: d5fe6eaffeb4
+  severity: science
+  text: Report confidence intervals or standard deviations for all benchmark percentages
+    in Table 1 to account for small sample sizes (e.g., AIME problems).
+- id: de4fb95a989e
+  severity: science
+  text: Clarify TTS usage and inference compute budgets for baseline models (Gemini
+    3.1 Pro, GPT-5.5) to ensure fair comparison against SU-01's TTS results.
+- id: 0c34a4240dc7
+  severity: science
+  text: Report inter-annotator agreement (e.g., Cohen's kappa) for the human expert
+    scoring on IMO 2025 and USAMO 2026 to validate score stability.
+- id: 609b4fce7a98
+  severity: science
+  text: Provide performance variance across multiple random seeds for the RL training
+    stage (200 steps) to demonstrate convergence stability and generalization.
+artifact_hash: 6b23039f76721ac00eaa6c408647f026893a62ad0f423ddd12fdde82e2327635
+artifact_path: projects/PROJ-581-https-arxiv-org-abs-2605-13301/paper/metadata.json
+backend: dartmouth
+feedback: ''
+github_authenticated: false
+model_name: qwen.qwen3.5-122b
+prompt_version: 1.1.0
+reviewed_at: '2026-06-06T07:31:22.647748Z'
+reviewer_kind: llm
+reviewer_name: paper_reviewer_scientific_evidence
+score: 0.0
+verdict: minor_revision
+---
+
+## Re-Review: Scientific Evidence Assessment
+
+This re-review follows the prior bar set by the previous scientific_evidence review. My assessment focuses exclusively on whether the four prior action items have been adequately addressed in the current revision.
+
+### Status of Prior Action Items
+
+**Item d5fe6eaffeb4 (Confidence Intervals/Standard Deviations):** NOT ADDRESSED. Table 1 (tab:verifiable-single-pass) and Table 2 (tab:nonverifiable-benchmarks) still report single-point percentages without confidence intervals or standard deviations. This is particularly problematic for AIME 2025/2026 (small problem sets) and IPhO scores where variance estimates are essential for interpreting statistical significance.
+
+**Item de4fb95a989e (TTS/Compute Budgets for Baselines):** NOT ADDRESSED. While SU-01 reports x/y scores (without/with TTS) in Tables 2 and 4, baseline models (Gemini 3.1 Pro Thinking, GPT-5.5-High) lack comparable TTS budget disclosures. The inference details in app:inference-serving-details mention API token limits for GPT-5.5 (128K) and Gemini (65K), but do not specify whether TTS was used or what compute budget was allocated, making fair comparison impossible.
+
+**Item 0c34a4240dc7 (Inter-Annotator Agreement):** NOT ADDRESSED. Section app:evaluation-details states "three gold-medal experts" scored IMO/USAMO problems, but no inter-annotator agreement statistics (Cohen's kappa, Fleiss' kappa, or percent agreement) are reported. Without this, score stability on these critical gold-medal claims cannot be validated.
+
+**Item 609b4fce7a98 (RL Training Variance):** NOT ADDRESSED. The RL training details (app:rl-training-details) describe 200 steps (96 coarse, 104 refined) but provide no performance variance across multiple random seeds. Convergence stability and generalization claims remain unsupported.
+
+### New Issues Introduced
+
+No new scientific evidence issues were introduced in this revision.
+
+### Recommendation
+
+All four prior action items remain unaddressed. The paper's central claims regarding gold-medal-level performance require these statistical validations to be scientifically credible. Please address all items before reconsideration.