fix(#2253): pass human review comments to intent-coherence sub-agent#2257
fix(#2253): pass human review comments to intent-coherence sub-agent#2257fullsend-ai-coder[bot] wants to merge 1 commit into
Conversation
The intent-coherence sub-agent evaluated scope authorization solely against the linked issue, with no visibility into human review comments from earlier PR review cycles. When a human reviewer explicitly requested a change that deviated from the issue spec (e.g., renaming a category), the sub-agent flagged it as unauthorized scope creep — a false positive. Changes: - Add step 2b to the review orchestrator (SKILL.md) to fetch human-authored review comments from the GitHub PR reviews API, filtering out bot reviews - Include human_review_comments in the context package sent to sub-agents (step 3d) and in the dispatch template (step 4) - Add a "Human-authorized scope amendments" section to the intent-coherence sub-agent with guidance on identifying human-authorized deviations and reporting them as info-level findings instead of medium+ findings - Handle ambiguous cases: vague comments default to normal severity, conflicting reviewer feedback flags for human resolution Note: pre-commit could not run due to sandbox network restrictions (shellcheck install failed). The post-script runs pre-commit authoritatively on the runner. Closes #2253
E2E tests did not runE2E tests run automatically for org/repo members and collaborators on pull requests. For other contributors, a maintainer must add the See E2E testing guide for details. |
Site previewPreview: https://076bd3a1-site.fullsend-ai.workers.dev Commit: |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
🤖 Finished Review · ✅ Success · Started 6:57 PM UTC · Completed 7:08 PM UTC |
ReviewFindingsMedium
Low
Info
|
![fullsend-ai-review[bot]](https://avatars.githubusercontent.com/in/3478501?s=60&v=4){kind=small}
| avoid flagging human-directed changes as unauthorized scope creep. | ||
|
|
||
| ```bash | ||
| # Fetch all review comments (not inline diff comments) |
There was a problem hiding this comment.
[medium] prompt-injection
Human review comments are passed as context to sub-agents without being identified as untrusted input. The meta-prompt warns that diff and PR metadata are untrusted, but human_review_comments is a new input channel not covered by this warning. Any GitHub user with permission to leave PR reviews can craft review comments containing prompt injection payloads to manipulate the intent-coherence sub-agent into suppressing legitimate scope-creep findings.
Suggested fix: Update meta-prompt.md to explicitly list human_review_comments as untrusted input alongside diff and PR metadata. Additionally, in SKILL.md step 4 context package template, add a warning label to the Human review comments section.
| **Identifying human-authorized deviations:** | ||
|
|
||
| A deviation is human-authorized when a human reviewer (not a bot) has | ||
| explicitly requested it in a review comment with state |
There was a problem hiding this comment.
[low] edge-case
The Human-authorized scope amendments section specifies that deviations authorized by CHANGES_REQUESTED or COMMENTED review states should be treated as info-level, but APPROVED is excluded. A reviewer who approves with a comment authorizing a deviation has given stronger authorization than a COMMENTED review. The data is already available but the sub-agent does not act on it.
| scope. If there is no linked issue, flag a `missing-authorization` | ||
| finding — non-trivial changes require explicit authorization. | ||
|
|
||
| ## Human-authorized scope amendments |
There was a problem hiding this comment.
[low] prompt-injection
The section instructs the sub-agent to treat review comments as authoritative scope amendments that suppress medium+ findings to info-level. This creates a trust elevation: PR review comments (which any collaborator can post) are granted the same authority as linked issues without verifying the commenter authority level.
| avoid flagging human-directed changes as unauthorized scope creep. | ||
|
|
||
| ```bash | ||
| # Fetch all review comments (not inline diff comments) |
There was a problem hiding this comment.
[info] edge-case
The gh api call filters reviews where .user.type != Bot. Organization and Mannequin account types would pass this filter, though in practice neither type can submit PR reviews on GitHub.
| @@ -42,6 +42,48 @@ Calibrate investigation to the diff size and nature. | |||
| scope. If there is no linked issue, flag a `missing-authorization` | |||
| finding — non-trivial changes require explicit authorization. | |||
|
|
|||
There was a problem hiding this comment.
[info] design-smell
Scope authorization is becoming a multi-source reconciliation problem (issue spec + human review comments + PR description) relying on LLM interpretation of natural language. Consider whether structured representation would be more robust.
The intent-coherence sub-agent evaluated scope authorization solely against the linked issue, with no visibility into human review comments from earlier PR review cycles. When a human reviewer explicitly requested a change that deviated from the issue spec (e.g., renaming a category), the sub-agent flagged it as unauthorized scope creep — a false positive.
Changes:
human-authored review comments from the GitHub PR reviews API,
filtering out bot reviews
sub-agents (step 3d) and in the dispatch template (step 4)
intent-coherence sub-agent with guidance on identifying
human-authorized deviations and reporting them as info-level
findings instead of medium+ findings
severity, conflicting reviewer feedback flags for human
resolution
Note: pre-commit could not run due to sandbox network restrictions (shellcheck install failed). The post-script runs pre-commit authoritatively on the runner.
Closes #2253
Post-script verification
agent/2253-human-review-scope-auth)0f53834f72372426c24dc0ede0030486a3e510ea..HEAD)