Skip to content

refactor(review): replace agentic reviews with /review guidance #974

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ko3n1g wants to merge 1 commit into
base: main
Choose a base branch
from
+78 −278
Open

refactor(review): replace agentic reviews with /review guidance #974

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions .agents/recipes/pr-review/recipe.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,10 @@ permissions:

# PR Review

This is the legacy local Agentic CI recipe. The GitHub workflow now posts
`/review` guidance; the review runner loads the existing `review-code` skill
and owns review publication. The procedure below applies only to local use.

Review pull request #{{pr_number}} using the `review-code` skill.

## Instructions
Expand All @@ -34,8 +38,7 @@ Review pull request #{{pr_number}} using the `review-code` skill.

## Constraints

- Do NOT post the review to GitHub yourself. The workflow handles posting via
`gh pr comment --body-file`.
- Do NOT post the review to GitHub yourself. Save the review locally; this recipe no longer has a GitHub publisher.
- Do NOT approve or request changes on the PR.
- If the diff is extremely large (>100 changed files), focus on the most
critical files and note that a full review was not feasible in a single pass.
Expand Down
349 changes: 73 additions & 276 deletions .github/workflows/agentic-ci-pr-review.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,298 +1,95 @@
name: "Agentic CI: PR Review"

on:
name: 'Agentic CI: Review guidance'
'on':
pull_request_target:
types: [opened, ready_for_review, labeled]
branches: [main]
types:
- opened
- ready_for_review
- labeled
branches:
- main
workflow_dispatch:
inputs:
pr_number:
description: "PR number to review"
description: PR number to receive /review guidance
required: true
timeout_minutes:
description: "Review timeout in minutes. Defaults to AGENTIC_CI_TIMEOUT_MINUTES or 30."
required: false

permissions:
checks: write
contents: read
pull-requests: write

concurrency:
group: agentic-ci-pr-review-${{ github.event.pull_request.number || github.event.inputs.pr_number }}
cancel-in-progress: true

jobs:
gate:
# Decide whether the review job should run. Uses the collaborator API
# instead of author_association (which is unreliable when org membership
# is private).
runs-on: ubuntu-latest
outputs:
allowed: ${{ steps.check.outputs.allowed }}
timeout_minutes: ${{ steps.timeout.outputs.minutes }}
steps:
- name: Check permissions
id: check
env:
GH_TOKEN: ${{ github.token }}
EVENT_NAME: ${{ github.event_name }}
EVENT_ACTION: ${{ github.event.action }}
LABEL_NAME: ${{ github.event.label.name }}
IS_DRAFT: ${{ github.event.pull_request.draft }}
SENDER_LOGIN: ${{ github.event.sender.login }}
PR_AUTHOR: ${{ github.event.pull_request.user.login }}
REPO: ${{ github.repository }}
run: |
# workflow_dispatch callers already have write access.
if [ "$EVENT_NAME" = "workflow_dispatch" ]; then
echo "allowed=true" >> "$GITHUB_OUTPUT"
exit 0
fi

# Only the agent-review label should trigger a run.
if [ "$EVENT_ACTION" = "labeled" ] && [ "$LABEL_NAME" != "agent-review" ]; then
echo "Skipping: labeled event but not agent-review"
- name: Check permissions
id: check
env:
GH_TOKEN: ${{ github.token }}
EVENT_NAME: ${{ github.event_name }}
EVENT_ACTION: ${{ github.event.action }}
LABEL_NAME: ${{ github.event.label.name }}
IS_DRAFT: ${{ github.event.pull_request.draft }}
SENDER_LOGIN: ${{ github.event.sender.login }}
PR_AUTHOR: ${{ github.event.pull_request.user.login }}
REPO: ${{ github.repository }}
run: |
# workflow_dispatch callers already have write access.
if [ "$EVENT_NAME" = "workflow_dispatch" ]; then
echo "allowed=true" >> "$GITHUB_OUTPUT"
exit 0
fi

# Only the agent-review label should trigger a run.
if [ "$EVENT_ACTION" = "labeled" ] && [ "$LABEL_NAME" != "agent-review" ]; then
echo "Skipping: labeled event but not agent-review"
echo "allowed=false" >> "$GITHUB_OUTPUT"
exit 0
fi

# Skip drafts unless agent-review label is being added.
if [ "$IS_DRAFT" = "true" ]; then
if [ "$EVENT_ACTION" != "labeled" ] || [ "$LABEL_NAME" != "agent-review" ]; then
echo "Skipping: draft PR"
echo "allowed=false" >> "$GITHUB_OUTPUT"
exit 0
fi

# Skip drafts unless agent-review label is being added.
if [ "$IS_DRAFT" = "true" ]; then
if [ "$EVENT_ACTION" != "labeled" ] || [ "$LABEL_NAME" != "agent-review" ]; then
echo "Skipping: draft PR"
echo "allowed=false" >> "$GITHUB_OUTPUT"
exit 0
fi
fi

# For labeled events, check the sender (who added the label) so
# maintainers can authorize reviews on external PRs.
# For other events, check the PR author.
if [ "$EVENT_ACTION" = "labeled" ]; then
USER="$SENDER_LOGIN"
echo "Checking sender (labeler): ${USER}"
else
USER="$PR_AUTHOR"
echo "Checking PR author: ${USER}"
fi

PERMISSION=$(gh api "repos/${REPO}/collaborators/${USER}/permission" --jq '.permission' 2>/dev/null || echo "none")
echo "permission=${PERMISSION}"

if [ "$PERMISSION" = "admin" ] || [ "$PERMISSION" = "write" ]; then
echo "allowed=true" >> "$GITHUB_OUTPUT"
else
echo "Skipping: ${USER} does not have write access (permission=${PERMISSION})"
echo "allowed=false" >> "$GITHUB_OUTPUT"
fi

- name: Resolve review timeout
id: timeout
if: steps.check.outputs.allowed == 'true'
env:
INPUT_TIMEOUT_MINUTES: ${{ inputs.timeout_minutes }}
CONFIG_TIMEOUT_MINUTES: ${{ vars.AGENTIC_CI_TIMEOUT_MINUTES }}
run: |
TIMEOUT_MINUTES="${INPUT_TIMEOUT_MINUTES:-${CONFIG_TIMEOUT_MINUTES:-30}}"
if ! [[ "$TIMEOUT_MINUTES" =~ ^[0-9]+$ ]] ||
(( TIMEOUT_MINUTES < 1 || TIMEOUT_MINUTES > 45 )); then
echo "::error::Review timeout must be an integer from 1 to 45 minutes."
exit 1
fi
echo "minutes=${TIMEOUT_MINUTES}" >> "$GITHUB_OUTPUT"

review:
name: Agentic review (advisory)
fi

# For labeled events, check the sender (who added the label) so
# maintainers can authorize reviews on external PRs.
# For other events, check the PR author.
if [ "$EVENT_ACTION" = "labeled" ]; then
USER="$SENDER_LOGIN"
echo "Checking sender (labeler): ${USER}"
else
USER="$PR_AUTHOR"
echo "Checking PR author: ${USER}"
fi

PERMISSION=$(gh api "repos/${REPO}/collaborators/${USER}/permission" --jq '.permission')

@greptile-apps greptile-apps Bot Oct 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Permission lookup fails the gate

When a non-collaborator opens a PR, this lookup can return 404. Without the previous fallback, the runner’s default bash -e stops the gate before it writes allowed=false. External contributors then get a failed workflow run instead of a clean skip.

Suggested change
PERMISSION=$(gh api "repos/${REPO}/collaborators/${USER}/permission" --jq '.permission')
PERMISSION=$(gh api "repos/${REPO}/collaborators/${USER}/permission" --jq '.permission' 2>/dev/null || echo "none")
Prompt To Fix With AI 
This is a comment left during a code review.
Path: .github/workflows/agentic-ci-pr-review.yml
Line: 71

Comment:
**Permission lookup fails the gate**

When a non-collaborator opens a PR, this lookup can return 404. Without the previous fallback, the runner’s default `bash -e` stops the gate before it writes `allowed=false`. External contributors then get a failed workflow run instead of a clean skip.

```suggestion
        PERMISSION=$(gh api "repos/${REPO}/collaborators/${USER}/permission" --jq '.permission' 2>/dev/null || echo "none")
```

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

echo "permission=${PERMISSION}"

if [ "$PERMISSION" = "admin" ] || [ "$PERMISSION" = "write" ]; then
echo "allowed=true" >> "$GITHUB_OUTPUT"
else
echo "Skipping: ${USER} does not have write access (permission=${PERMISSION})"
echo "allowed=false" >> "$GITHUB_OUTPUT"
fi
timeout-minutes: 5
review-guidance:
name: Post /review guidance
needs: gate
if: needs.gate.outputs.allowed == 'true'
runs-on: [self-hosted, agentic-ci]
environment: agentic-ci
timeout-minutes: 60
runs-on: ubuntu-latest
timeout-minutes: 5
env:
GH_TOKEN: ${{ github.token }}
REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number || inputs.pr_number }}
steps:
- name: Determine PR number
id: pr
env:
EVENT_NAME: ${{ github.event_name }}
INPUT_PR_NUMBER: ${{ github.event.inputs.pr_number }}
PR_NUMBER: ${{ github.event.pull_request.number }}
run: |
if [ "$EVENT_NAME" = "workflow_dispatch" ]; then
echo "number=${INPUT_PR_NUMBER}" >> "$GITHUB_OUTPUT"
else
echo "number=${PR_NUMBER}" >> "$GITHUB_OUTPUT"
fi

- name: Validate PR number
env:
PR_NUMBER: ${{ steps.pr.outputs.number }}
run: |
if ! [[ "$PR_NUMBER" =~ ^[0-9]+$ ]]; then
echo "::error::Invalid PR number: ${PR_NUMBER}"
exit 1
fi

- name: Check required config
env:
AGENTIC_CI_MODEL: ${{ vars.AGENTIC_CI_MODEL }}
run: |
if [ -z "$AGENTIC_CI_MODEL" ]; then
echo "::error::AGENTIC_CI_MODEL variable is not set. Configure it in repo settings."
exit 1
fi

- name: Resolve head SHA
id: head
env:
GH_TOKEN: ${{ github.token }}
EVENT_NAME: ${{ github.event_name }}
PR_NUMBER: ${{ steps.pr.outputs.number }}
PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
run: |
if [ "$EVENT_NAME" = "workflow_dispatch" ]; then
SHA=$(gh pr view "$PR_NUMBER" --json headRefOid -q '.headRefOid')
else
SHA="$PR_HEAD_SHA"
fi
echo "sha=$SHA" >> "$GITHUB_OUTPUT"

- name: Checkout PR branch
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
# Preserve maintainer-approved fork reviews while #804 hardens isolation.
allow-unsafe-pr-checkout: true
ref: ${{ steps.head.outputs.sha }}
fetch-depth: 0

# SECURITY: Recipe and tool files define the agent's prompt and run
# with secrets in scope. Always read them from the base branch so a
# fork PR cannot inject malicious instructions or code.
- name: Checkout base branch agent files
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ github.event.pull_request.base.sha || 'main' }}
sparse-checkout: |
.agents/recipes
.agents/tools
path: base-agents

- name: List changed Python files
id: changed-py
env:
PR_NUMBER: ${{ steps.pr.outputs.number }}
GH_TOKEN: ${{ github.token }}
run: |
gh pr diff "$PR_NUMBER" --name-only | grep '\.py$' > /tmp/changed-py.txt || true
echo "count=$(wc -l < /tmp/changed-py.txt | tr -d ' ')" >> "$GITHUB_OUTPUT"

- name: Structural impact analysis
if: steps.changed-py.outputs.count != '0'
run: |
rm -f "/tmp/structural-impact-${{ steps.pr.outputs.number }}.md"
mapfile -t CHANGED_PY < /tmp/changed-py.txt
python -m venv /tmp/graphify-venv
/tmp/graphify-venv/bin/python -m pip install graphifyy==0.4.23 --quiet 2>&1 | tail -3
/tmp/graphify-venv/bin/python base-agents/.agents/tools/structural_impact.py \
--repo-root "${{ github.workspace }}" \
--changed-files "${CHANGED_PY[@]}" \
--output "/tmp/structural-impact-${{ steps.pr.outputs.number }}.md"
echo "Structural impact analysis complete:"
cat "/tmp/structural-impact-${{ steps.pr.outputs.number }}.md"
continue-on-error: true

- name: Pre-flight checks
env:
ANTHROPIC_BASE_URL: ${{ secrets.AGENTIC_CI_API_BASE_URL }}
ANTHROPIC_API_KEY: ${{ secrets.AGENTIC_CI_API_KEY }}
AGENTIC_CI_MODEL: ${{ vars.AGENTIC_CI_MODEL }}
run: |
if ! command -v claude &> /dev/null; then
echo "::error::claude CLI not found in PATH"
exit 1
fi
echo "Claude CLI version: $(claude --version 2>&1 || true)"

# Quick API check (custom endpoint only)
if [ -n "$ANTHROPIC_BASE_URL" ] && [ -n "$ANTHROPIC_API_KEY" ]; then
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" \
--max-time 30 \
-X POST "${ANTHROPIC_BASE_URL}/v1/messages" \
-H "Content-Type: application/json" \
-H "x-api-key: ${ANTHROPIC_API_KEY}" \
-H "anthropic-version: 2023-06-01" \
-d "{\"model\":\"${AGENTIC_CI_MODEL}\",\"max_tokens\":5,\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}")
if [ "$HTTP_CODE" -lt 200 ] || [ "$HTTP_CODE" -ge 300 ]; then
echo "::error::API pre-flight failed with HTTP ${HTTP_CODE}"
exit 1
fi
echo "API pre-flight passed (HTTP ${HTTP_CODE})"
fi

- name: Run PR review recipe
id: review
timeout-minutes: ${{ fromJSON(needs.gate.outputs.timeout_minutes) }}
env:
ANTHROPIC_BASE_URL: ${{ secrets.AGENTIC_CI_API_BASE_URL }}
ANTHROPIC_API_KEY: ${{ secrets.AGENTIC_CI_API_KEY }}
AGENTIC_CI_MODEL: ${{ vars.AGENTIC_CI_MODEL }}
DISABLE_PROMPT_CACHING: "1"
GH_TOKEN: ${{ github.token }}
PR_NUMBER: ${{ steps.pr.outputs.number }}
run: |
set -o pipefail

# Build the prompt from _runner.md + recipe, substituting template vars.
# Read from base-agents/ (checked out from the base branch) so fork
# PRs cannot tamper with the agent prompt.
RUNNER_CTX=$(cat base-agents/.agents/recipes/_runner.md)
RECIPE_BODY=$(cat base-agents/.agents/recipes/pr-review/recipe.md \
| sed '1,/^---$/{ /^---$/,/^---$/d }')

PROMPT=$(printf '%s\n\n%s\n' "${RUNNER_CTX}" "${RECIPE_BODY}" \
| sed "s/{{pr_number}}/${PR_NUMBER}/g")

claude \
--model "$AGENTIC_CI_MODEL" \
-p "$PROMPT" \
--max-turns 50 \
--output-format text \
--verbose \
2>&1 | tee /tmp/claude-review-log.txt
continue-on-error: true

- name: Report incomplete review
if: steps.review.outcome != 'success'
env:
GH_TOKEN: ${{ github.token }}
PR_NUMBER: ${{ steps.pr.outputs.number }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
TIMEOUT_MINUTES: ${{ needs.gate.outputs.timeout_minutes }}
run: |
echo "::warning::Agentic review failed or exceeded the ${TIMEOUT_MINUTES}-minute limit."
{
echo "## Agentic review incomplete"
echo
echo "The advisory review failed or timed out. This does not block merging."
} >> "$GITHUB_STEP_SUMMARY"
gh pr comment "$PR_NUMBER" --body "Agentic review did not complete. [View the workflow run]($RUN_URL) for details." || \
echo "::warning::Could not post incomplete review comment."

- name: Post review comment
if: steps.review.outcome == 'success'
env:
GH_TOKEN: ${{ github.token }}
PR_NUMBER: ${{ steps.pr.outputs.number }}
run: |
if [ -s "/tmp/review-${PR_NUMBER}.md" ]; then
gh pr comment "$PR_NUMBER" --body-file "/tmp/review-${PR_NUMBER}.md"
else
echo "::warning::Review file not created by agent."
fi

- name: Remove agent-review label
if: always() && github.event.action == 'labeled' && github.event.label.name == 'agent-review'
env:
GH_TOKEN: ${{ github.token }}
PR_NUMBER: ${{ steps.pr.outputs.number }}
run: |
gh pr edit "$PR_NUMBER" --remove-label "agent-review"
- name: Explain the replacement command
env:
NOTICE: Automatic Agentic CI reviews have been retired. Request a review by commenting `/review` on this pull request. Use `/review mode=strict` for a deeper review, or `/review model=claude` to select Claude. Comment `/review help` for all options.
run: gh pr comment "$PR_NUMBER" --repo "$REPO" --body-file - <<< "$NOTICE"
Loading