Repository navigation
ci(evals): narrow native eval trigger to paths read from the PR - #340
Conversation
The native Fullsend run reads only the triage-security skill folder and the two triage-security schemas from the PR plugin; the eval suite, fixtures, host policy/config and host scripts come from the trusted base checkout. Triggering on those paths re-tested main and sent untrusted PRs to the eval-protected gate for nothing. Trust/approval gating is unchanged. Add a pytest tripwire pinning the Step 0.6/0.7 gate commands of triage-security SKILL.md (GATE_COMMANDS_SHA256): native valid_case_tools diagnostics match against main's commands, so a PR changing them reads unrecognized and needs a manual transcript check. Implements TC-6919 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assisted-by: Claude Code
Reviewer's GuideThe PR reduces native eval execution to changes in the skill and schemas actually read from the PR, leaving trusted base-checkout eval assets and host configuration changes out of the trigger path. It expands discovery coverage for both accepted and rejected paths and adds a hash-based guard to flag gate-command drift that could invalidate native evidence interpretation. Flow diagram for narrowed native eval triggeringflowchart LR
PR["PR changed files"] --> discover["discover"]
discover --> relevant{"Matches triage-security skill or input/result schema?"}
relevant -->|yes| native["Run native eval"]
relevant -->|no| skip["Skip native eval"]
base["Trusted base checkout\neval suite, fixtures, policy, providers, profiles, env, host scripts"] --> native
Flow diagram for gate command drift protectionflowchart LR
skill["triage-security SKILL.md\nStep 0.6/0.7 gate blocks"] --> parse["Parse gate commands"]
parse --> hash["Compare GATE_COMMANDS_SHA256"]
hash -->|match| pass["Gate test passes"]
hash -->|changed| fail["Gate test fails with update guidance"]
fail --> evidence["Prevent unrecognized native\npresence_gate_* / input_validation_*"]
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 4 issues
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path=".github/workflows/eval-pr-run.yml" line_range="232-234" />
<code_context>
const skills = confirmed.join(',');
const nativeRelevant = files.some(f =>
- /^evals\/fullsend\//.test(f.filename) ||
- /^evals\/triage-security\//.test(f.filename) ||
/^plugins\/sdlc-workflow\/skills\/triage-security\//.test(f.filename) ||
- /^plugins\/sdlc-workflow\/(policies\/triage-security\.yaml|providers\/vertex-ai\.yaml|profiles\/fullsend-vertex-ai\.yaml|env\/gcp-vertex\.env|schemas\/triage-security-(input|result)\.schema\.json|scripts\/(validate-output-schema\.sh|strip_extra_properties\.py|test_(fullsend_gate_eval|native_fullsend_eval_ci|native_fullsend_execution)\.py))$/.test(f.filename)
+ /^plugins\/sdlc-workflow\/schemas\/triage-security-(input|result)\.schema\.json$/.test(f.filename)
);
core.setOutput('native', String(nativeRelevant));
</code_context>
<issue_to_address>
**Renamed removals escape native evaluation**
When a triage-security skill file or schema is renamed outside the watched paths, `pulls.listFiles` reports the destination in `filename` and the old path in `previous_filename`, but discovery checks only `filename`. The rename leaves `nativeRelevant` false, so the native run skips evaluation of the removal.
Check `previous_filename` as well as `filename` when determining whether a changed file matches the watched paths.
</issue_to_address>
### Comment 2
<location path=".github/workflows/eval-pr-run.yml" line_range="234" />
<code_context>
- /^evals\/triage-security\//.test(f.filename) ||
/^plugins\/sdlc-workflow\/skills\/triage-security\//.test(f.filename) ||
- /^plugins\/sdlc-workflow\/(policies\/triage-security\.yaml|providers\/vertex-ai\.yaml|profiles\/fullsend-vertex-ai\.yaml|env\/gcp-vertex\.env|schemas\/triage-security-(input|result)\.schema\.json|scripts\/(validate-output-schema\.sh|strip_extra_properties\.py|test_(fullsend_gate_eval|native_fullsend_eval_ci|native_fullsend_execution)\.py))$/.test(f.filename)
+ /^plugins\/sdlc-workflow\/schemas\/triage-security-(input|result)\.schema\.json$/.test(f.filename)
);
core.setOutput('native', String(nativeRelevant));
</code_context>
<issue_to_address>
**Changed plugin scripts skip native evaluation**
When a PR changes a plugin script used by the sandbox, such as `scripts/sha256-digest.py`, without changing the triage-security skill directory or either schema, `run-fullsend.py` copies the entire PR plugin into `tested-plugin`, which is supplied to the sandbox, and triage-security remediation templates invoke `scripts/sha256-digest.py`. A PR changing a plugin script outside the matched skill or schema paths therefore changes sandbox behavior without triggering the native eval, so its regression goes untested.
Include PR plugin files consumed by the sandbox in native discovery, or stage only the specific PR files the native run is intended to test.
</issue_to_address>
### Comment 3
<location path=".github/workflows/eval-pr-run.yml" line_range="234" />
<code_context>
- /^evals\/triage-security\//.test(f.filename) ||
/^plugins\/sdlc-workflow\/skills\/triage-security\//.test(f.filename) ||
- /^plugins\/sdlc-workflow\/(policies\/triage-security\.yaml|providers\/vertex-ai\.yaml|profiles\/fullsend-vertex-ai\.yaml|env\/gcp-vertex\.env|schemas\/triage-security-(input|result)\.schema\.json|scripts\/(validate-output-schema\.sh|strip_extra_properties\.py|test_(fullsend_gate_eval|native_fullsend_eval_ci|native_fullsend_execution)\.py))$/.test(f.filename)
+ /^plugins\/sdlc-workflow\/schemas\/triage-security-(input|result)\.schema\.json$/.test(f.filename)
);
core.setOutput('native', String(nativeRelevant));
</code_context>
<issue_to_address>
**Result schema edits launch native evals**
When a PR changes only `plugins/sdlc-workflow/schemas/triage-security-result.schema.json`, the discovery regex marks a result-schema change as native-relevant, so the approval gate and native job run. `run-fullsend.py` replaces that PR schema with the trusted base version when staging, so the change cannot affect the run and still incurs the gate and runtime.
Exclude the result schema from native discovery and its expected-true test case unless the native runner is changed to consume the PR version.
Also at `plugins/sdlc-workflow/scripts/test_native_fullsend_eval_ci.py:132`.
</issue_to_address>
### Comment 4
<location path=".github/workflows/eval-pr-run.yml" line_range="234" />
<code_context>
- /^evals\/triage-security\//.test(f.filename) ||
/^plugins\/sdlc-workflow\/skills\/triage-security\//.test(f.filename) ||
- /^plugins\/sdlc-workflow\/(policies\/triage-security\.yaml|providers\/vertex-ai\.yaml|profiles\/fullsend-vertex-ai\.yaml|env\/gcp-vertex\.env|schemas\/triage-security-(input|result)\.schema\.json|scripts\/(validate-output-schema\.sh|strip_extra_properties\.py|test_(fullsend_gate_eval|native_fullsend_eval_ci|native_fullsend_execution)\.py))$/.test(f.filename)
+ /^plugins\/sdlc-workflow\/schemas\/triage-security-(input|result)\.schema\.json$/.test(f.filename)
);
core.setOutput('native', String(nativeRelevant));
</code_context>
<issue_to_address>
**Fixture edits still require approval**
When an untrusted PR changes only a file under `evals/triage-security/files/`, `evalPattern` still classifies changes under `evals/triage-security/files/` as the `triage-security` skill, and the existing `evals/triage-security/evals.json` confirms it. `skills` therefore remains nonempty, so an untrusted PR still reaches the `eval-protected` gate and ordinary eval job for these fixture-only changes.
Exclude native-only fixture changes from ordinary skill discovery, or narrow the claim to say only the native run is avoided for these paths.
</issue_to_address>Sourcery assessment
Approval pending. 4 findings to address first.
Blocking findings: .github/workflows/eval-pr-run.yml:234, .github/workflows/eval-pr-run.yml:234, .github/workflows/eval-pr-run.yml:234, .github/workflows/eval-pr-run.yml:234
| const nativeRelevant = files.some(f => | ||
| /^evals\/fullsend\//.test(f.filename) || | ||
| /^evals\/triage-security\//.test(f.filename) || | ||
| /^plugins\/sdlc-workflow\/skills\/triage-security\//.test(f.filename) || | ||
| /^plugins\/sdlc-workflow\/(policies\/triage-security\.yaml|providers\/vertex-ai\.yaml|profiles\/fullsend-vertex-ai\.yaml|env\/gcp-vertex\.env|schemas\/triage-security-(input|result)\.schema\.json|scripts\/(validate-output-schema\.sh|strip_extra_properties\.py|test_(fullsend_gate_eval|native_fullsend_eval_ci|native_fullsend_execution)\.py))$/.test(f.filename) | ||
| /^plugins\/sdlc-workflow\/schemas\/triage-security-(input|result)\.schema\.json$/.test(f.filename) |
There was a problem hiding this comment.
🟡 Medium · Renamed removals escape native evaluation
When a triage-security skill file or schema is renamed outside the watched paths, pulls.listFiles reports the destination in filename and the old path in previous_filename, but discovery checks only filename. The rename leaves nativeRelevant false, so the native run skips evaluation of the removal.
Check previous_filename as well as filename when determining whether a changed file matches the watched paths.
Prompt for AI agents
In `.github/workflows/eval-pr-run.yml` at lines 232-234:
**Renamed removals escape native evaluation**
When a triage-security skill file or schema is renamed outside the watched paths, `pulls.listFiles` reports the destination in `filename` and the old path in `previous_filename`, but discovery checks only `filename`. The rename leaves `nativeRelevant` false, so the native run skips evaluation of the removal.
Check `previous_filename` as well as `filename` when determining whether a changed file matches the watched paths.
There was a problem hiding this comment.
[sdlc-workflow/verify-pr] Classified as suggestion - valid robustness enhancement, but it describes long-standing behavior (the discovery expression has only ever matched filename, both before and after this PR) and is outside the scope of TC-6919, which narrows which paths trigger the native run rather than how renames are tracked. It is not documented in CONVENTIONS.md and no established codebase pattern uses previous_filename. No sub-task created.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Native Fullsend Eval Results
Head: 39c4079
Merge: 8022f1d
Trusted workflow: 66692a5
Reviewed eval source: 66692a5
Native job: failure; complete: false.
Execution evidence: invalid; quality score (advisory): unavailable (grading incomplete; 21/28 judgments available, 20 passed / 1 failed).
- 037-release / assertion_1: scorer-error (phase-exit)
- 037-release / assertion_2: scorer-error (phase-exit)
- 037-release / assertion_3: scorer-error (phase-exit)
- 037-release / assertion_4: scorer-error (phase-exit)
- 037-release / assertion_5: scorer-error (phase-exit)
- 037-release / assertion_6: scorer-error (phase-exit)
- 037-release / assertion_7: scorer-error (phase-exit)
| Case | Execution evidence | Quality score (advisory) |
|---|---|---|
| 033-absent | valid | 4/4 |
| 034-empty | valid | 5/5 |
| 035-malformed | valid | 4/5 |
| 036-valid | valid | 7/7 |
| 037-release | valid | unavailable (0/7 judgments, 0 passed / 0 failed) |
|
🤖 Finished Verify Pr · ✅ Success · Started 1:54 PM UTC · Completed 2:07 PM UTC Commit: Runtime: claude · Model: claude-opus-4-8 · Effort: high · Cost: $6.50 |
Verification Report for TC-6919 (commit 39c4079)
Overall: PASSAll verification checks pass. The PR narrows the Review feedback: sourcery-ai raised 4 suggestions, all on Advisory (non-blocking): the separate "Native Fullsend Eval Results" review reported a scorer-error (phase-exit) on case This report is informational - a human reviewer decides whether to merge. This comment was AI-generated by sdlc-workflow/verify-pr v0.13.10. |
1 participant
Summary
discoverineval-pr-run.yml:nativeRelevantnow matches onlyplugins/sdlc-workflow/skills/triage-security/**andschemas/triage-security-(input|result).schema.json— the only paths the native run reads from the PR. Eval suite, fixtures, host policy/providers/profiles/env and host scripts come from the trusted base checkout, so changes there no longer trigger the ~18 min native run (or theeval-protectedgate for untrusted PRs). Trust/approval conditions and default skill discovery are unchanged.test_native_fullsend_eval_ci.py: discovery parametrization updated (3 live paths →true; eval suite, fixture, policy, provider, host script, test file →false).test_fullsend_gate_eval.py: tripwiretest_gate_commands_match_valid_case_tools_pinpins the Step 0.6/0.7 gate blocks of triage-securitySKILL.md(GATE_COMMANDS_SHA256), parsed likevalid_case_tool_observationsinevals/fullsend/run.py. On change it fails explaining that nativepresence_gate_*/input_validation_*will read unrecognized and how to update the pin.Test plan
python3 -m pytest plugins/sdlc-workflow/scripts/test_native_fullsend_eval_ci.py -q— 134 passedpython3 -m pytest plugins/sdlc-workflow/scripts/ -q— 688 passed, 2 skippeduvx skillsaw(0 errors),claude plugin validate plugins/sdlc-workflow✔Implements TC-6919
🤖 Generated with Claude Code
Summary by Sourcery
Limit native triage-security evaluations to pull-request changes that can affect their behavior while guarding against unnoticed gate-command drift.
Enhancements:
CI:
Tests: