What happened
The review agent approved PR #4474 with 2 low-severity stylistic findings (import ordering, commit-prefix mismatch), spending $7.80 on Claude Opus at high effort (run 32971799794).
Five days later, human reviewer gustavolira identified 7 substantive issues that the agent missed, including: an unbounded JQL query that can silently truncate incident results and report 0% CFR for failing services; an epoch-based lookback that pages through all in-window deployments before reaching a usable row (5+ extra GraphQL pages per metric per entity per run for repos with 500 deployments); an unbounded leading interval that makes CFR read 100% for quarterly deployers; and a missing time-range filter on merged deployments that allows custom collectors to introduce out-of-range rows.
All of these findings required understanding the DORA data collection pipeline: how collectors feed providers, how pagination interacts with time windows, and how the from/to boundaries affect both the number of API requests and the accuracy of metric calculations.
What could go better
The scorecard workspace AGENTS.md is thorough on metric IDs, naming conventions, and threshold resolution (lines 14–217), but contains no information about the data collection pipeline — the path from collector input schemas through API calls to provider metric calculations. Without this context, the review agent treated the PR as a straightforward feature addition and focused on surface-level style.
The human reviewer's findings all required reasoning about data flow across API boundaries: how from/to parameters map to pagination behavior, the difference between "rows returned" and "requests made" in GitHub's GraphQL API, how Jira's lack of ORDER BY in JQL affects which items survive truncation, and how time-window boundaries affect denominator counts in DORA metrics.
I am confident this is the root cause. The review agent had access to the AGENTS.md and used it to verify naming conventions (it correctly flagged the import ordering issue), but the file gave it no basis for reasoning about the collection pipeline. The human reviewer's domain expertise in these areas is what enabled the substantive findings.
Uncertainty: Even with pipeline context in AGENTS.md, the review agent may not catch all edge cases that require deep API-specific knowledge (e.g., Jira's pagination truncation behavior). However, documenting the key invariants — especially around time-window boundaries and pagination limits — would give the agent the conceptual framework to reason about them.
Proposed change
Add a new section to workspaces/scorecard/AGENTS.md after the existing "MetricProvider Architecture" section, titled "DORA Collector Pipeline" or similar, covering:
-
Data flow overview: Collectors (GitHub deployments, GitHub workflow runs, Jira incidents) → DefaultScorecardCollectorsService.collect() → DORA providers (CFR, lead time, DF, MTTR) → metric calculation. Each collector has an input schema (deploymentsCollectorOutputSchema, etc.) and pagination limits.
-
Time-window invariants: DORA metrics operate over a 30-day sliding window (windowFrom to windowTo). Key invariants to verify in reviews:
- Every API query must bound both
from and to to avoid unbounded history scans
- The distinction between rows returned (controlled by
fetchItemsLimit) and API requests made (controlled by pagination stopping conditions in getDeployments, getWorkflowRuns, etc.)
- Merged arrays (e.g., in-window + pre-window deployments) must be filtered to the expected time range before calculation
-
Pagination boundaries: Document that GitHub GraphQL pagination stops when reachedOlderThanWindow flips or deployments.length hits the limit, but rows newer than to are skipped without incrementing deployments.length. Jira's sendPaginatedRequest stops at DEFAULT_PAGINATED_FETCH_ITEMS_LIMIT (1000) with only a warning. Flag any query that sets from to epoch or unbounded as high-risk.
-
Review checklist for DORA provider changes: When reviewing changes to DORA providers or collectors, verify: (a) all API queries have bounded from/to; (b) merged deployment arrays are filtered to the expected range; (c) pagination limits match the intended scope; (d) error handling distinguishes collection failures from data-insufficiency; (e) metric documentation matches the actual calculation window.
Validation criteria
On the next 3 DORA-related PRs in the scorecard workspace that modify collector or provider logic, the review agent should flag at least one of the following when applicable: unbounded time-range queries, pagination limit mismatches, or unfiltered merged arrays. The agent should not approve DORA pipeline changes without addressing time-window boundary conditions.
As a concrete test: if PR #4732 (the successor to #4474) still uses an epoch-based lookback or unbounded incident query, the review agent should flag it.
Generated by retro agent from #4474
What happened
The review agent approved PR #4474 with 2 low-severity stylistic findings (import ordering, commit-prefix mismatch), spending $7.80 on Claude Opus at high effort (run 32971799794).
Five days later, human reviewer gustavolira identified 7 substantive issues that the agent missed, including: an unbounded JQL query that can silently truncate incident results and report 0% CFR for failing services; an epoch-based lookback that pages through all in-window deployments before reaching a usable row (5+ extra GraphQL pages per metric per entity per run for repos with 500 deployments); an unbounded leading interval that makes CFR read 100% for quarterly deployers; and a missing time-range filter on merged deployments that allows custom collectors to introduce out-of-range rows.
All of these findings required understanding the DORA data collection pipeline: how collectors feed providers, how pagination interacts with time windows, and how the
from/toboundaries affect both the number of API requests and the accuracy of metric calculations.What could go better
The scorecard workspace AGENTS.md is thorough on metric IDs, naming conventions, and threshold resolution (lines 14–217), but contains no information about the data collection pipeline — the path from collector input schemas through API calls to provider metric calculations. Without this context, the review agent treated the PR as a straightforward feature addition and focused on surface-level style.
The human reviewer's findings all required reasoning about data flow across API boundaries: how
from/toparameters map to pagination behavior, the difference between "rows returned" and "requests made" in GitHub's GraphQL API, how Jira's lack ofORDER BYin JQL affects which items survive truncation, and how time-window boundaries affect denominator counts in DORA metrics.I am confident this is the root cause. The review agent had access to the AGENTS.md and used it to verify naming conventions (it correctly flagged the import ordering issue), but the file gave it no basis for reasoning about the collection pipeline. The human reviewer's domain expertise in these areas is what enabled the substantive findings.
Uncertainty: Even with pipeline context in AGENTS.md, the review agent may not catch all edge cases that require deep API-specific knowledge (e.g., Jira's pagination truncation behavior). However, documenting the key invariants — especially around time-window boundaries and pagination limits — would give the agent the conceptual framework to reason about them.
Proposed change
Add a new section to
workspaces/scorecard/AGENTS.mdafter the existing "MetricProvider Architecture" section, titled "DORA Collector Pipeline" or similar, covering:Data flow overview: Collectors (GitHub deployments, GitHub workflow runs, Jira incidents) →
DefaultScorecardCollectorsService.collect()→ DORA providers (CFR, lead time, DF, MTTR) → metric calculation. Each collector has an input schema (deploymentsCollectorOutputSchema, etc.) and pagination limits.Time-window invariants: DORA metrics operate over a 30-day sliding window (
windowFromtowindowTo). Key invariants to verify in reviews:fromandtoto avoid unbounded history scansfetchItemsLimit) and API requests made (controlled by pagination stopping conditions ingetDeployments,getWorkflowRuns, etc.)Pagination boundaries: Document that GitHub GraphQL pagination stops when
reachedOlderThanWindowflips ordeployments.lengthhits the limit, but rows newer thantoare skipped without incrementingdeployments.length. Jira'ssendPaginatedRequeststops atDEFAULT_PAGINATED_FETCH_ITEMS_LIMIT(1000) with only a warning. Flag any query that setsfromto epoch or unbounded as high-risk.Review checklist for DORA provider changes: When reviewing changes to DORA providers or collectors, verify: (a) all API queries have bounded
from/to; (b) merged deployment arrays are filtered to the expected range; (c) pagination limits match the intended scope; (d) error handling distinguishes collection failures from data-insufficiency; (e) metric documentation matches the actual calculation window.Validation criteria
On the next 3 DORA-related PRs in the scorecard workspace that modify collector or provider logic, the review agent should flag at least one of the following when applicable: unbounded time-range queries, pagination limit mismatches, or unfiltered merged arrays. The agent should not approve DORA pipeline changes without addressing time-window boundary conditions.
As a concrete test: if PR #4732 (the successor to #4474) still uses an epoch-based lookback or unbounded incident query, the review agent should flag it.
Generated by retro agent from #4474