ci: skip CI on the generated eval results commit by claude[bot] · Pull Request #332 · supabase/evals

claude · 2026-09-24T13:29:43Z

Requested by Matt Rossman · Slack thread

The eval-refresh workflow's GITHUB_TOKEN push of chore: refresh eval results onto a PR branch still fires pull_request: synchronize, so Refresh eval results and Code Quality re-run as github-actions[bot] and sit in "approval required" (seen on #281). This appends [skip ci] to that direct-push commit so GitHub creates no runs for it, leaving the create-pull-request path and its CI untouched (AI-1254).

🤖 Generated with Claude Code

https://claude.ai/code/session_01ECba2eYo5iJPLnVPT8JvkV

Closes AI-1254

Generated by Claude Code

vercel · 2026-09-24T13:29:49Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
evals	Ignored	Preview	Sep 24, 2026 2:06pm UTC

The eval-refresh workflow pushes "chore: refresh eval results" back onto the PR head branch with GITHUB_TOKEN. That push does not fire `push` workflows, but it does fire `pull_request: synchronize`, so Refresh eval results and Code Quality re-run as github-actions[bot] and sit in "approval required" for a maintainer (see #281). Because the refresh is label-gated, the re-run would also refresh again. Append the skip-ci marker to the direct-push results commit so GitHub creates no runs for it. The scheduled/dispatch path that opens a fresh results PR through create-pull-request is unchanged and still gets normal CI. main has no required status checks, so a skipped head commit does not block merging. Refs: AI-1254 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ECba2eYo5iJPLnVPT8JvkV

…ecovery to exercise the refresh commit in CI Adds a behaviour-neutral comment to the eval's scorer so the run-evals-changed path resolves to a non-empty pairs list and the workflow's direct-push results commit executes. This commit will be reverted before merge. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ECba2eYo5iJPLnVPT8JvkV

Reverts 48b346e, the results commit the workflow pushed while exercising the run-evals-changed path on this PR. The results were only generated to prove the skip-ci behaviour; they are not part of this change. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ECba2eYo5iJPLnVPT8JvkV

…roject-recovery Reverts f5f68c4 now that the refresh commit has been exercised in CI. The PR's diff is back to the workflow and CONTRIBUTING changes only. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ECba2eYo5iJPLnVPT8JvkV

claude · 2026-09-24T13:52:44Z

Proof the [skip ci] results commit creates no workflow runs

Pushed a behaviour-neutral comment to evals/regression/resolve-reliability-001-unhealthy-project-recovery/EVAL.ts (f5f68c4) so the run-evals-changed path had something to run.
Run 36007634037 resolved 2 pairs (codex-gpt-6-luna, codex-gpt-6-luna-no-skills x that eval, 3 runs each); prepare, run-evals, publish-results all succeeded and pushed chore: refresh eval results [skip ci] as 48b346e (touching apps/web/src/data/regression-eval-results.json).
GET /actions/runs?head_sha=48b346e3... -> total_count: 0, and /commits/48b346e3.../check-suites has no github-actions suite at all (only vercel, netlify, github-pages, sentry, blacksmith-sh, claude). No Refresh eval results or Code Quality run was created for the bot commit.
Contrast, feat(evals): add build-database-002-stack-lifecycle CLI eval (CLI-2398) #281's results commit 955d9b9 (no [skip ci]) produced two pull_request runs, both action_required: 35980017936 (Refresh eval results) and 35980018118 (Code Quality).
Reverted both the temporary change and the generated results (ff3c936, c78e114); the PR diff is back to .github/workflows/eval-refresh.yml + CONTRIBUTING.md only.

Generated by Claude Code

Replace the six-line comment above the results commit with a one-line explainer plus the GitHub docs link, and restore CONTRIBUTING.md to main's version, per review feedback on #332. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ECba2eYo5iJPLnVPT8JvkV

## Motivation We have unit / e2e tests across `apps/*` and `packages/*` but don't run them or type checks in CI, so they quietly regress (e.g. #351 broke a Codex parser test that #366 added the same day). ## Changes - Code Quality gets a `typecheck` job next to `format` - A new Tests workflow runs `test`, `test-platform-lite` (which is slow enough for its own parallel job), and `test-docker`. The Docker tests take ~2.5 min, so they only run when a PR touches relevant files ([skipped example](https://github.com/supabase/evals/actions/runs/37680288402/job/112994246780)). - Refresh results commits drop `[skip ci]` (added in #332) and push with our GH App token instead, so required checks run on them without approval. A `run-evals-changed` refresh's [results commit](b57f997) ran CI as the app with no approval prompt and didn't start another refresh. - Upgraded to Vitest v5 so we get per-package labels on the job summary (compare [before](https://github.com/supabase/evals/actions/runs/37533147821) and [after](https://github.com/supabase/evals/actions/runs/37630783292)). | [Tests](https://github.com/supabase/evals/actions/runs/37679563185) | [Code Quality](https://github.com/supabase/evals/actions/runs/37679562944) | | --- | --- | | <img alt="CleanShot 2026-10-07 at 4 16 00 PM@2x" src="https://github.com/user-attachments/assets/8593f06b-d45a-4576-b346-163caec4689f" /> | <img alt="CleanShot 2026-10-07 at 3 37 37 PM@2x" src="https://github.com/user-attachments/assets/5600cda5-208d-432c-8b18-534c6f06f7f5" /> | Example with tests failing ([run](https://github.com/supabase/evals/actions/runs/37666155427)): <img alt="CleanShot 2026-10-07 at 2 24 36 PM@2x" src="https://github.com/user-attachments/assets/47b7d635-616c-47bc-806a-428fe851c78d" /> I added `OPENAI_API_KEY` to [Dependabot secrets](https://github.com/supabase/evals/settings/secrets/dependabot) so Tests runs on Dependabot PRs. Fork PRs don't get secrets as is the case on [supabase/mcp](https://github.com/supabase/mcp). ## Follow-ups - After merge, require `format`, `typecheck`, `test`, `test-platform-lite`, and `test-docker` on the [`main: review required`](https://github.com/supabase/evals/settings/rules/18775100) ruleset. - AI-934: replace the framework [smoke tests](https://github.com/supabase/evals/blob/5f8de50a5e8c8ef40d983d9aff9030dd25080a89/apps/framework/scripts/smoke-framework.ts) with per-scorer Vitest files. - AI-1295: cut platform-lite setup time with PGlite snapshots. Closes AI-1121 --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

claude Bot added the run-evals-changed Add to a PR to refresh only the benchmark evals that have had changes label Sep 24, 2026

claude Bot force-pushed the claude/skip-ci-on-refresh-commit branch from 02877ea to 2525034 Compare September 24, 2026 13:34

claude and others added 2 commits September 24, 2026 13:43

chore: refresh eval results [skip ci]

48b346e

vercel Bot deployed to Preview September 24, 2026 13:47 View deployment

claude added 2 commits September 24, 2026 13:52

mattrossman reviewed Sep 24, 2026

View reviewed changes

Comment thread .github/workflows/eval-refresh.yml

mattrossman approved these changes Sep 24, 2026

View reviewed changes

mattrossman marked this pull request as ready for review September 24, 2026 14:14

mattrossman requested a review from a team as a code owner September 24, 2026 14:14

Rodriguespn approved these changes Sep 24, 2026

View reviewed changes

mattrossman merged commit dacee2b into main Sep 24, 2026
7 checks passed

mattrossman mentioned this pull request Oct 7, 2026

feat(ci): run tests and type checks on pull requests #371

Merged

Navigation Menu

Repository navigation

ci: skip CI on the generated eval results commit - #332

ci: skip CI on the generated eval results commit#332
mattrossman merged 6 commits into
mainfrom
claude/skip-ci-on-refresh-commit

claude Bot commented Sep 24, 2026 •

edited by mattrossman

Loading

Uh oh!

vercel Bot commented Sep 24, 2026 •

edited

Loading

Uh oh!

claude Bot commented Sep 24, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

claude Bot commented Sep 24, 2026 • edited by mattrossman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Closes AI-1254

Uh oh!

vercel Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

claude Bot commented Sep 24, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

claude Bot commented Sep 24, 2026 •

edited by mattrossman

Loading

vercel Bot commented Sep 24, 2026 •

edited

Loading