Review CI-passing PRs without manual requests

[neubig]

Problem

The scheduled GitHub PR reviewer in #654 retries outstanding all-hands-bot requests, and #656 bounds how many new conversations one scan may launch. Once those requests are exhausted, the automation stops making progress even while open, non-draft PRs with passing CI still lack a review on their current head. A one-off batch of review requests would not cover later PRs.

Desired behavior

The existing scheduled reviewer continuously discovers eligible CI-passing PRs in its configured repositories and reviews them at a bounded pace. Explicit all-hands-bot requests continue to work. Keep this in the Extensions reviewer automation, reusing its scan, CI gate, keyed conversation delivery, native review, and maintainer handoff; do not add a separate host daemon or another automation.

Acceptance criteria

• A scheduled scan considers every open, non-draft PR without a current-head all-hands-bot review, even if nobody requested the bot or applied a trigger label.
• Before starting an agent, the scan requires the current head's applicable GitHub check runs and Actions workflow runs to be complete and non-failing. A failed or pending workflow with zero jobs is not mistaken for green. Explicit requests retain their existing gate behavior.
• Unrequested PRs use a stable repository/PR/head delivery key so repeated scans create neither duplicate conversations nor duplicate native reviews. A new head is eligible again.
• Use Bound scheduled PR review intake across OSS repositories #656's per-scan new-conversation limit across all configured repositories. Order candidates deterministically; explicit requests take priority, then the oldest eligible unreviewed PRs. Ineligible PRs do not consume slots.
• A completed review uses the existing verdict and maintainer-handoff path. Handle bot-authored PRs through the reviewer's supported non-approval verdict path so they are not silently skipped because GitHub ignores self-review requests.
• Focused tests cover an unrequested green PR, a failed zero-job workflow, a pending workflow, deduplication across scans, the cap, and a changed head. Live evidence shows a scheduled scan starting a bounded number of reviews across at least two of the four OSS repositories.

Dependencies

Stack after #654 and #656, or incorporate only the minimal shared code needed from them. #651's exact-head gate must include the zero-job workflow-run case reported on that PR. Do not merge automatically.

---

OpenHands AI triage

The following comments and acceptance criteria were added by the OpenHands AI agent.

Triage

This is bounded scheduling work in the existing github-pr-reviewer automation, so no product or design decision is outstanding. It belongs in skills/github-pr-reviewer/scripts/worker.py, reusing the machinery the stacked PRs already add: the scheduled scan (run() with no event payload), the AgentConversationDispatcher.deliver(subject, delivery, prompt) keyed delivery that dedupes a {event-or-key}:{sha} delivery per subject, _finish_completed_review, the exact-head check gate (_classify_check_runs / _latest_check_runs), and run_repositories. No second automation record, host daemon, queue, or runtime layer is added.

Concrete inferred decisions, drawn from the adjacent code rather than invented:

• Eligibility is "open, non-draft, and no completed native review by the reviewer account on the current head." The completion handler in worker.py already reads GET /pulls/{n}/reviews and filters by commit_id == head_sha and reviewer login, so the negative of that set is the candidate filter. The trigger is not a label or an outstanding request; it is the absence of a current-head review, so an unrequested PR is picked up without any label.
• The delivery key is the existing deliver(subject="{repo_id}:pr:{n}", delivery="{key}:{sha}") mechanism. For an unrequested PR there is no reviewer-request event to key on (that is fix(review): resume requested reviews on a scheduled check scan #654's path for outstanding requests), so the natural stable key is the repository/PR identity plus the head SHA. fix(review): resume requested reviews on a scheduled check scan #654's keyed path and the explicit-request event path are unchanged.
• Ordering and the cap follow Bound scheduled PR review intake across OSS repositories #656, whose merged behavior (per the stacked negotiation on that PR) is a configurable max_new_per_run defaulting to 2, keyed as max_new_per_run, shared across run_repositories' per-repository loop. Explicit all-hands-bot requests keep priority, then eligible unrequested PRs oldest-first; candidates whose gate is blocked/waiting post their disposition and consume no slot. Configure the maximum through the worker's rendered config, following the max_new_per_run key that github-issue-to-pr, gitlab-issue-to-mr, and github-agents-md-maintainer already use.
• The "I review all current-head CI" rule is the existing check gate extended to Actions workflow runs. The precedent for the read is skills/github-delivery-watchdog/scripts/worker.py, which pages GET /actions/runs?head_sha={sha}, ignores runs whose head_sha differs, groups by (workflow_id, event, head_branch), takes the latest run_number per group, and requires every remaining run to be status == "completed" and conclusion == "success". This matches fix(review): gate agent launch on exact-head checks #651's stated goal on that PR (a head whose commit check-run rollup can even read SUCCESS while a pull_request workflow run failed with zero jobs) and the fix(review): gate agent launch on exact-head checks #651 comment requesting workflow-run coverage with the same blocked/waiting behavior, obsolete-head filtering, and no duplicated gate machinery. The gate stays single-pass: check-run and workflow-run findings combine into one disposition and one managed gate comment.

Stacking: base is #654 (which is itself stacked on #651), and the cap comes from #656. #651, #654, and #656 are all open and unmerged, so this issue is blocked on them; the implementation should target the stacked head rather than main. No merge is automatic and no OSS deployment change belongs here.

Non-goals: no repository-specific policy, VM/memory tuning, or deployment control in OpenHands/extensions; no second automation record, host daemon, or runtime layer; no new webhook event types, check_run parsing in the host, or GitHub permission scopes; no change to the review prompt, verdict parsing, maintainer handoff, secret selection, or the explicit review_requested / trigger-label paths beyond the shared gate. The live canary depends on the OSS deployment accepting a scheduled dispatch; the PR should state the live evidence it actually obtained.

Acceptance Criteria

• With no trigger label and no all-hands-bot request, a scheduled scan considers an open, non-draft PR whose current head has no completed native review by the configured reviewer account, and starts a review for it. Drafts and PRs already carrying a current-head reviewer review are skipped.
• Before starting an agent, the scan requires the exact head's applicable GitHub check runs to be complete and non-failing, with the existing policy: obsolete-head runs ignored, only the latest run of each logical check counted, and unknown conclusions failing closed.
• Before starting an agent, the scan also requires the exact head's Actions workflow runs to be complete and non-failing: a workflow run whose status is not completed, or whose conclusion is not a non-blocking value, prevents the launch even when the commit's check-run rollup reports success. A workflow with zero jobs that failed or is pending is not treated as green.
• A non-green head posts the existing managed gate comment (blocked or waiting) and starts no conversation; the deterministic comment is upserted, not duplicated, and the behavior is identical whether the finding came from a check run or a workflow run.
• An explicit all-hands-bot request keeps its current trigger behavior and gate behavior, and is not regressed by the new unrequested-PR path.
• Unrequested PRs use a stable repository/PR/head delivery key, so repeated scheduled scans create neither a second conversation nor a second native review for the same head. A PR whose head changes becomes eligible again for the new head and is reviewed exactly once under the new key.
• A per-scan limit on new conversations applies across all configured repositories, defaulting to 2 and configurable via the max_new_per_run config key. The limit bounds conversations started, so deliveries that only deduplicate or report an already-running conversation do not consume it.
• When more candidates are eligible than the limit allows, explicit requests are ordered before unrequested candidates, and unrequested candidates are ordered by PR age (oldest first) deterministically with stable tie-breakers; candidates whose gate is blocked or waiting consume no slot and do not prevent later eligible candidates from being considered in the same run.
• When the limit is reached mid-scan, the run still completes its non-dispatching work for the remaining candidates (gate evaluation and gate comments, completed-review reconciliation, maintainer handoff) and does not abort; a per-PR or per-repository error is reported without aborting the other repositories.
• A completed review uses the existing verdict parsing and maintainer-handoff path unchanged. A PR authored by the configured bot account is reviewed through the reviewer's supported non-approval verdict path (a COMMENT-event review that keeps the approved verdict) rather than being skipped because GitHub ignores a self-review request.
• Focused automated tests driving the shipped reviewer entrypoint through the existing catalog-bundle test helper cover: an unrequested green PR that starts a review; a failed zero-job workflow run that blocks; a pending workflow run that waits; no duplicate conversation or review across repeated scans; the per-run cap limiting conversations across repositories; and a changed head becoming eligible again.
• The reviewer SKILL.md and README.md document the unrequested-PR scan, the check-run plus workflow-run gate, the delivery key, and the per-run maximum with its default; the catalog entry's bundle and entry versions are bumped and the generated catalogs are regenerated so python scripts/sync_extensions.py --check and npm run build report no drift.
• No new secret, token scope, or GitHub permission is required beyond what the current reviewer uses, and no secret value appears in logs, comments, or the PR.
• Live evidence in the PR's AGENT section shows a scheduled scan starting a bounded number of reviews across at least two of the four OSS repositories, with links to the reviewed pull requests and their conversations.

[neubig]

@all-hands-bot Please implement this ready-for-dev issue as a focused Extensions PR. Reuse the existing scheduled reviewer and the bounded-intake branch from #656; stack after #656 once its PR is open. Do not create a host-local script or a second automation. Keep the current-head CI gate, deduplication, and maintainer handoff. Run focused tests and provide live evidence on at least two OSS repos before requesting review. Do not merge.

[all-hands-bot]

Implemented as a focused Extensions PR: #660 — feat(review): review unrequested green PRs on the scheduled scan (#660).

It is stacked on openhands/issue-656 (the head branch of #656, now PR #659) and retargets to main when the stack merges. No host-local script and no second automation were added; the change is entirely inside the existing skills/github-pr-reviewer worker.

What changed

• Unrequested scan. In scheduled mode the scan now also considers every open, non-draft PR whose current head carries no completed review by the reviewer account, even with no trigger label and no all-hands-bot request. The candidate filter is the exact negative of the completion handler's predicate, so a review the scan starts and one it finds already present are the same set; an already-reviewed head is reconciled (verdict parsing + maintainer handoff) instead of restarted.
• Stable delivery key. Unrequested heads use scan:{repository}:{number}:{head}, so repeated scans create neither a duplicate conversation nor a duplicate native review, and a changed head is eligible again under its new SHA.
• Bounded intake, deterministic order. max_new_per_run (default 2, configurable) now orders candidates explicitly: an explicit request or trigger label first, then the oldest candidate — request time for a requested PR, the PR's own creation time for an unrequested one — with repository and PR number as stable tie-breakers. Ineligible candidates consume no slot and do not block later ones, and reaching the cap never aborts the scan's non-dispatching work.
• Current-head CI gate. The exact-head gate now also reads Actions workflow runs, so a workflow that failed before creating any check run (a zero-job pull_request run) blocks even when the commit's check-run rollup is green. Workflow runs whose suite already reported check runs are left to those runs (never double-counted), obsolete-head runs are ignored, and the latest run of each workflow supersedes an earlier attempt. The existing blocked/waiting comment upsert is reused unchanged, so behavior is identical whether the finding came from a check run or a workflow run.
• Self-authored PRs. A PR authored by the configured bot is directed to publish the clean review as an event: COMMENT review that keeps the approved verdict, rather than being skipped because GitHub ignores a self-review request.
• Docs (SKILL.md, README.md), the catalog manifest, and the fixture document all of the above; bundle/entry versions bumped 1.7.0 → 1.8.0; npm run build regenerated the catalogs.

Validation

• uv run --group test pytest -q tests/test_github_reviewer_delivery.py tests/test_github_automation_foundation.py — all pass (72 in the reviewer file).
• uv run --group test pytest -q — 1042 passed, 23 skipped.
• uv run --group test python scripts/sync_extensions.py --check — clean apart from the pre-existing non-blocking issue-duplicate-checker coverage warning.
• npm run build — no drift.

Focused tests drive the real shipped worker.py entrypoint and run_scan control flow: an unrequested green PR that starts a review; a failed zero-job workflow that blocks; a pending workflow that waits; no duplicate across repeated scans; a changed head reviewed again under its new key; the cap spanning repositories with explicit-request priority; the oldest unrequested PRs first; a blocked PR past the cap that still gates without consuming a slot; reconciliation + maintainer handoff of an already-reviewed head; and the self-authored COMMENT-verdict path.

Live evidence (read-only)

With GITHUB_TOKEN_REVIEW authenticated as all-hands-bot, I loaded the shipped bundle and drove the real PullRequestReviewer.run() scheduled path and the real shared-intake drain against the live GitHub API. No agent was started and no comment posted — the dispatcher recorded what would have been delivered and the gate-comment upsert was recorded instead of written.

OpenHands/extensions          open=84   non_draft=51
OpenHands/OpenHands           open=429  non_draft=339
OpenHands/software-agent-sdk  open=262  non_draft=197
OpenHands/automation          open=48   non_draft=33

registered green unrequested candidates: 1
conversations started: 1
  OpenHands/software-agent-sdk PR #2495
    delivery scan:OpenHands/software-agent-sdk:2495:72d2f217a7054fb3c51e6ac66d061a700bbe5013

real drain at the bound on live data (gate forced eligible; keys/order/drain shipped):
  max_new_per_run=2 -> 2 started: software-agent-sdk #1780 (oldest), extensions #63
  max_new_per_run=1 -> 1 started: software-agent-sdk #1780 (oldest)

The shipped gate against live heads, including the #426 zero-job reproduction:

OpenHands/extensions 41ffb9d4 -> ('blocked', ['Check Extensions', 'Deprecation deadlines', 'Tests'])
OpenHands/software-agent-sdk 72d2f217 -> ('green', [])

The read-only canary is committed at .pr/canary_unrequested_scan.py.

CI

Validate PR description, pr-title / Lint PR title (conventional), and pr-title / Apply conventional-type label all pass on head 4b6b827. The Tests, Check Extensions, and Deprecation deadlines workflows are on: pull_request with branches: ["*"]/["**"], which does not match a slashed stacked base branch — the same behavior seen on #659 and #654, not a failure of this change. Those run once the stack is retargeted to main.

Two notes: the live canary deliberately started no agent and posted no comment, so the "links to the reviewed pull requests and their conversations" part of the acceptance criteria depends on the OSS deployment switching the reviewer automation to this bundle. And I did not merge anything.

Generated by OpenHands AI on behalf of the user.

[all-hands-bot]

Follow-up on #660: the base branch openhands/issue-656 was force-pushed while this PR was open (it dropped the unrelated Granola integration files), which briefly left the stacked PR CONFLICTING. I rebased my two commits onto the new base tip 23ee3e7 and force-pushed my own branch only.

Current state: head c01bac7, mergeable=MERGEABLE, mergeStateStatus=CLEAN. CI on the new head is green — Validate PR description, Deprecation deadlines (check), pr-title / Lint PR title (conventional), and pr-title / Apply conventional-type label all pass, 0 failing, 0 pending. Tests and Check Extensions still do not fire because their on: pull_request branches filter does not match a slashed stacked base; they run once the stack retargets to main.

Re-validated after the rebase: uv run --group test pytest -q — 1041 passed, 23 skipped; npm run build — no drift; scripts/sync_extensions.py --check — clean apart from the pre-existing non-blocking issue-duplicate-checker warning. No behavior changed in the rebase; the diff against the new base is the same 11 files. Nothing was merged.

Generated by OpenHands AI on behalf of the user.

[neubig]

Live canary correction (2026-09-23): the acceptance criteria above that say to inspect every open PR and post a managed gate comment for every non-green unrequested head are superseded. That behavior scanned hundreds of PRs and posted 190 comments before dispatching any review. A scheduled run must instead process a bounded, fairly rotating candidate window using Automation KV state, prioritize explicit requests, avoid gate comments for unrequested red/pending PRs, and finish within the scan interval while eventually covering the backlog. Keep native review and maintainer handoff unchanged. See #660 for the live failure and repair.