evidence report: regenerate on main by bot, advisory in PRs, strict at release (stop per-PR report churn) #4703

Owner decision (2026-10-08), revising #4601's "CI runs --check" for PRs. The rendered evidence report (docs/language_status/evidence_report/*/report.{json,md}) is derived output. Today every PR must commit it byte-current (tests/tools/test_evidence_report.py::test_the_committed_reports_are_current). That forces parallel PRs into merge conflicts on the same generated files, and it makes every open PR stale again after each merge. On 2026-10-07/08 it was the main source of CI churn (#4660, #4661, #4663, #4693, ...), while protecting nothing a buyer sees between releases.

Plan (land in this order)

1. Reliable bot refresh on main. Run .github/workflows/evidence-refresh.yml on every push to main (plus the nightly). It runs evidence.py refresh --stale, det_coverage_ledger.py update and evidence_report.py --refresh --live, and commits through an auto-merged bot PR from auto/evidence-refresh. Make it robust first; earlier bot PRs were closed dirty or opened against the wrong branch (evidence: nightly re-proof of stale records #4640, evidence: nightly re-proof of stale records #4649):
   • always branch from the current main;
   • skip the PR when nothing changed;
   • use a concurrency group, so a newer push cancels an older refresh;
   • stamp the report "as of commit ".
2. Advisory report check in PRs. test_the_committed_reports_are_current no longer fails PRs. CI still runs evidence_report.py --check and writes the per-program level changes the PR would cause to the job summary, but does not fail. PRs stop committing regenerated report files. test_committed_report_validates (schema) stays blocking.
3. Release gate. A release or tag workflow requires evidence_report.py --check --live to be clean, so anything published is current.

Unchanged

• Evidence records rule (Q1) stays as is: stale on a program's own port, case, corpus pin or declared differences fails CI and is re-proven in the PR (test_every_committed_record_is_current). Harness, oracle and generator staleness stays scheduled.
• The det-sweep baseline (tests/equivalence/det_sweep_baseline.json) stays a blocking two-way ratchet.

Acceptance

• A PR that changes a case or ledger and doesn't touch the report files passes CI, and its job summary shows the level deltas.
• After it merges, the bot PR refreshes the report and merges itself.
• Two parallel PRs no longer conflict on report files.
• The release gate fails on a stale report.
• Docs explain the new flow: docs/language_status/evidence_report/README.md, the evidence docs, and the gitgalaxy-pr-worker brief (drop "refresh the report in your PR").

🤖 Generated with Claude Code