Skip to content

🐛 Align RL rewards and terminal outcomes - #830

Draft
flowerthrower wants to merge 5 commits into
munich-quantum-toolkit:v3/669-gnn-rlfrom
flowerthrower:fix/rl-horizon-termination
Draft

flowerthrower wants to merge 5 commits into
munich-quantum-toolkit:v3/669-gnn-rlfrom
flowerthrower:fix/rl-horizon-termination

Conversation

@flowerthrower

@flowerthrower flowerthrower commented Sep 22, 2026 •

Copy link
Copy Markdown
Member

🤖 AI text below 🤖

Description

A compilation that exhausts its pass budget now receives the same final reward as explicit termination: the circuit score once if synthesis, layout, and routing are complete, otherwise the existing no_effect_penalty. This terminal reward replaces the last pass's intermediate reward; earlier improvement rewards and no-effect penalties remain unchanged. Inference raises for incomplete terminal results.

Hard-budget endings, pass failures, and pass timeouts terminate without PPO value bootstrapping. External truncations retain their reward and bootstrap from the final observation. Both flat and GNN observations include the normalized remaining pass budget; GNN global features increase from 36 to 37. Models must be retrained. The upgrade guide documents the observation and error-info changes.

Depends on #800 and remains the final draft in the paper stack:
#831 → #755 → #758 → #794 → #795 → #796 → #797 → #798 → #799 → #800 → #830.
Only #831 targets main. Keeping this policy in #830 lets its tests exercise #799's rewards and #800's GNN together.

Validation: all 117 non-training compilation tests and full lint pass, including all 68 RL/GNN tests. Coverage includes complete/incomplete horizons, explicit termination, failure/timeout penalties, unchanged intermediate rewards, remaining-budget observations, traces, flat/GNN PPO bootstrap behavior, and saved GNN inference. Fresh CI remains pending.

AI assistance: Codex implemented the policy, extended the existing tests, and updated the changelog, upgrade guide, and PR description.

Checklist

  • The pull request only contains commits that are focused and relevant to this change.
  • I have added appropriate tests that cover the new/changed functionality.
  • I have updated the documentation to reflect these changes.
  • I have added entries to the changelog for any noteworthy additions, changes, fixes, or removals.
  • I have added migration instructions to the upgrade guide (if needed).
  • The changes follow the project's style guidelines and introduce no new warnings.
  • The changes are fully tested and pass the CI checks.
  • I have reviewed my own code changes.

If PR contains AI-assisted content:

  • Any agent that created, edited, or submitted GitHub content was explicitly authorized for that scope, as required by our AI Usage Guidelines.
  • Every agent-authored or agent-edited public text body begins with the visible disclosure 🤖 *AI text below* 🤖 (titles are exempt).
  • I have disclosed AI assistance in the PR description.
  • I confirm that I have personally reviewed and understood all AI-generated content, and accept full responsibility for it.

Assisted-by: GPT-5.6 via Codex

Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Assisted-by: GPT-5.6 via Codex

Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Exercise intermediate rewards, explicit termination, timeouts, traces, and flat/GNN rollout bootstrapping on the combined paper stack.

Assisted-by: GPT-6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
@flowerthrower
flowerthrower changed the base branch from main to v3/669-gnn-rl September 23, 2026 12:07
@flowerthrower
flowerthrower force-pushed the fix/rl-horizon-termination branch from 52d3e49 to 2ce69cd Compare September 23, 2026 12:07
@codecov

codecov Bot commented Sep 23, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Score complete compilations once at termination or the hard horizon; penalize incomplete and aborted attempts without bootstrapping. Expose the remaining pass budget to flat and GNN policies.

Assisted-by: GPT-6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
@flowerthrower flowerthrower changed the title 🐛 End RL episodes at the hard pass horizon 🐛 Align RL rewards and terminal outcomes Sep 23, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant