feat(ci): extend daily scan to full BOM image set at Medium+ with auto-generated VEX drafts (#1473)

[mohityadav8]

Summary

Extends vuln-scan-images.yaml to cover the full ~82-image upstream set (sourced from make bom) at Medium+ severity, and adds a gen-vex-drafts.py script that auto-generates under_investigation OpenVEX draft statements for new findings and opens a PR for human triage.

Closes #1473.
Parent epic: #739.

Changes

.github/workflows/vuln-scan-images.yaml

Three new jobs added after the existing scan job; notify and cleanup updated.

Job	Depends on	What it does
bom-list	(independent)	Runs make bom, extracts all container image refs from bom.cdx.json via jq, outputs a JSON array. New components are picked up automatically — no hand-maintained list.
scan-bom	bom-list	Dynamic matrix over all BOM images. severity-cutoff: medium, only-fixed: false (upstream images — escalation/VEX feeds rather than in-tree fixes). max-parallel: 10 to cap bandwidth and registry rate-limit risk. Per-job upload: .txt summary + .json (full Grype output for VEX) + .ref (raw image ref).
vex-draft	scan-bom	Downloads all bom-scan-* artifacts, runs gen-vex-drafts.py, opens a PR against .openvex.json when new (vuln-id, product-purl) pairs are found. Skipped cleanly when bom-list fails or produces an empty list.
notify (updated)	scan + scan-bom	Two-tier Slack: NVIDIA-built images → Critical/High with full CVE IDs (unchanged); BOM upstream images → Critical/High with IDs always, Medium as a one-line count digest so the channel stays actionable. scan-bom being skipped does not block the NVIDIA notification.
cleanup (updated)	notify + vex-draft	Waits for the VEX draft PR to open before deleting scan images.

.github/scripts/gen-vex-drafts.py (new)

• Reads existing (vuln-name, product-purl) pairs from .openvex.json — idempotent, never duplicates.
• Reads each *.ref / *.json pair from bom-scan-results/.
• For each Medium+ match not already covered, emits an under_investigation statement with an OCI PURL keyed to the exact image ref.
• Does not auto-suppress. under_investigation leaves Grype reporting findings; a human must update status to not_affected/affected/fixed with evidence before suppression takes effect.
• Appends to .openvex.json, increments version, stamps timestamp and tooling.
• Writes new_count=N to $GITHUB_OUTPUT; the workflow gates the create-pull-request step on new_count > 0.

Success criteria (from #1473)

• Daily scan covers the full BOM image set sourced from make bom — no hand-maintained image list
• Severity cutoff is Medium+; counts and IDs reported per image
• New Medium+ findings produce under_investigation OpenVEX draft statements without auto-suppressing
• Re-runs apply existing VEX so notifications surface only new/changed findings (idempotency tested locally)
• Notification volume tuned: Critical/High always, Medium as a count digest
• Registry auth for gated sources — see open question below

Testing

• yamllint -c .yamllint.yaml — 0 warnings
• python3 -m py_compile gen-vex-drafts.py — clean
• Idempotency: second run on same scan data outputs new_count=0, .openvex.json unchanged
• Job dependency graph verified via python3 -c "import yaml; ..." — all needs: chains correct

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

📝 Walkthrough

Walkthrough

A new Python script .github/scripts/gen-vex-drafts.py generates idempotent OpenVEX under_investigation draft statements from Grype BOM scan results. It discovers *.ref/*.json pairs, reads existing (vulnerability, product purl) keys from .openvex.json to skip duplicates, filters Grype matches to Medium+ severity, constructs statement dicts, then updates .openvex.json with incremented version and timestamp. The workflow gains three new jobs: bom-list (parses dist/bom/bom.cdx.json into a matrix), scan-bom (runs anchore/scan-action per image at medium cutoff), and vex-draft (downloads artifacts, runs the script, opens a PR for new statements). Slack notification aggregation is split into NVIDIA and BOM sections, and cleanup waits on both notify and vex-draft.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main CI change: full BOM scanning at Medium+ with auto-generated VEX drafts.
Description check	✅ Passed	The description is directly aligned with the workflow and script changes in this PR.
Linked Issues check	✅ Passed	The PR implements the core #1473 goals: full BOM scanning from make bom, Medium+ severity, and auto-generated under_investigation VEX drafts.
Out of Scope Changes check	✅ Passed	The changes stay focused on BOM scanning, VEX draft generation, and related notification/cleanup wiring.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/scripts/gen-vex-drafts.py:
- Around line 223-226: The empty scan-pairs branch in gen-vex-drafts.py
currently exits successfully, which masks a broken scan-bom artifact/output
contract. Update the scan_pairs check in the main flow so that when no pairs are
found it records the zero count if needed but fails the job with a non-zero exit
instead of returning 0, keeping the failure tied to the vex-draft execution
path.

In @.github/workflows/vuln-scan-images.yaml:
- Around line 483-494: The bom-scan artifact filenames are keyed only by
SHORT_NAME, so different IMAGE references with the same basename can overwrite
each other in bom-scan-results and lose findings. Update the naming logic in the
bom-scan steps that write OUT, JSON_OUT, and the .ref file so each artifact uses
a collision-resistant identifier derived from the full image reference (for
example, include a sanitized registry/namespace or a hash) while keeping the
existing bom-scan-results flow intact. Make the same change wherever the same
basename-based naming is used in the related bom-scan sections to ensure every
image produces unique .txt, .json, and .ref outputs.
- Around line 695-699: The Slack/log text in the BOM medium summary is
overstating the result by saying a VEX draft PR was opened whenever
BOM_TOTAL_MEDIUM is greater than zero. Update the logic around BOM_MEDIUM_LINE
so it only mentions an opened VEX draft PR when the same condition used by
vex-draft indicates a new PR was actually created (for example, new_count > 0),
and otherwise keep the message limited to the upstream medium total. Use the
BOM_TOTAL_MEDIUM/BOM_MEDIUM_LINE block and the vex-draft PR-opening condition as
the key symbols to align the message with actual behavior.
- Around line 460-531: The BOM scan workflow currently skips artifact creation
when anchore/scan-action fails before JSON output exists, so failed images are
lost from downstream Slack/VEX processing. Update the extraction step that reads
steps.scan.outputs.json to run under always(), and make the Upload BOM scan
result step run always() as well so bom-scan-results is still published even on
failures. In the extraction logic, keep the existing “scan output missing” path
but safely handle an empty JSON_FILE/output situation without failing the job.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 261f144d-9b3e-4e17-8b87-abea7b1c84c5

📥 Commits

Reviewing files that changed from the base of the PR and between 4b24b7d and 17e96c5.

📒 Files selected for processing (2)

• .github/scripts/gen-vex-drafts.py
• .github/workflows/vuln-scan-images.yaml

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

.github/workflows/vuln-scan-images.yaml (1)

511-517: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Add the missing + before the related CVE suffix. The jq expression is syntactically invalid as written, so this step exits before generating the .txt summary for high/critical matches.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/vuln-scan-images.yaml around lines 511 - 517, The jq
expression used in the high/critical image scan summary is missing the
concatenation operator before the related CVE suffix, which makes the `HIGH_IDS`
assignment invalid. Update the jq filter in the `vuln-scan-images` workflow to
concatenate `.artifact.name`, `.vulnerability.id`, and the optional related
vulnerabilities string correctly, so the `HIGH_IDS` step can complete and
produce the `.txt` summary.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In @.github/workflows/vuln-scan-images.yaml:
- Around line 511-517: The jq expression used in the high/critical image scan
summary is missing the concatenation operator before the related CVE suffix,
which makes the `HIGH_IDS` assignment invalid. Update the jq filter in the
`vuln-scan-images` workflow to concatenate `.artifact.name`,
`.vulnerability.id`, and the optional related vulnerabilities string correctly,
so the `HIGH_IDS` step can complete and produce the `.txt` summary.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 961d25a7-79ff-47cc-802e-9dd8a3cc41dd

📥 Commits

Reviewing files that changed from the base of the PR and between 17e96c5 and 8085003.

📒 Files selected for processing (2)

• .github/scripts/gen-vex-drafts.py
• .github/workflows/vuln-scan-images.yaml

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

.github/workflows/vuln-scan-images.yaml (1)

699-709: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Send the BOM Medium digest when Medium is the highest severity.

Line 706 skips Slack whenever Critical/High totals are zero, so BOM_MEDIUM_LINE is never posted for Medium-only BOM findings despite the Medium+ reporting objective.

Proposed fix

-          # Only notify when critical or high vulnerabilities are found
-          if [[ "${TOTAL_CRITICAL}" -eq 0 && "${TOTAL_HIGH}" -eq 0 ]]; then
+          # Notify for Critical/High findings, or for the BOM Medium digest.
+          if [[ "${TOTAL_CRITICAL}" -eq 0 && "${TOTAL_HIGH}" -eq 0 && "${BOM_TOTAL_MEDIUM}" -eq 0 ]]; then
             echo "No critical or high vulnerabilities found, skipping Slack notification"
             echo -e "Vulnerability Scan: ${TIMESTAMP} (${SHORT_SHA})${NVIDIA_RESULTS}${BOM_MEDIUM_LINE}"
             exit 0
           fi

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/vuln-scan-images.yaml around lines 699 - 709, The Slack
notification gate in the vuln scan workflow is too narrow: the current check in
the notification block only sends when Critical or High findings exist, so
`BOM_MEDIUM_LINE` from the BOM digest is skipped for Medium-only cases. Update
the logic around the existing `TOTAL_CRITICAL`, `TOTAL_HIGH`, and
`BOM_MEDIUM_LINE` handling so Slack is still notified when Medium is the highest
severity for BOM findings, while preserving the existing skip behavior for truly
clean results. Use the notification section in `vuln-scan-images.yaml` and the
surrounding severity totals to locate the fix.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In @.github/workflows/vuln-scan-images.yaml:
- Around line 699-709: The Slack notification gate in the vuln scan workflow is
too narrow: the current check in the notification block only sends when Critical
or High findings exist, so `BOM_MEDIUM_LINE` from the BOM digest is skipped for
Medium-only cases. Update the logic around the existing `TOTAL_CRITICAL`,
`TOTAL_HIGH`, and `BOM_MEDIUM_LINE` handling so Slack is still notified when
Medium is the highest severity for BOM findings, while preserving the existing
skip behavior for truly clean results. Use the notification section in
`vuln-scan-images.yaml` and the surrounding severity totals to locate the fix.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 0fffeead-b317-4b17-a418-72db37ad7bc8

📥 Commits

Reviewing files that changed from the base of the PR and between 6898eb8 and 3e0eabc.

📒 Files selected for processing (1)

• .github/workflows/vuln-scan-images.yaml