Skip to content

fix: reject runs on Depot CI with an explicit error - #554

Draft
fargito wants to merge 1 commit into
mainfrom
fix/reject-depot-ci
Draft

fargito wants to merge 1 commit into
mainfrom
fix/reject-depot-ci

Conversation

@fargito

@fargito fargito commented Sep 28, 2026

Copy link
Copy Markdown
Member

Fail with an explicit error when the runner runs on Depot CI.

Depot CI is a drop-in replacement for GitHub Actions: it sets GITHUB_ACTIONS=true and the usual GITHUB_* variables, so the runner detected its jobs as GitHub Actions ones. Its run ids (e.g. 271476942194171) do not exist on GitHub Actions, so CodSpeed could not process these runs, and users got no feedback on why.

The GitHub Actions provider now bails before any benchmark runs when DEPOT_JOB_URL is set, pointing to support@codspeed.io and Discord to request Depot CI support. DEPOT_JOB_URL is only set on Depot CI, not on Depot's GitHub Actions runners (Depot environment variables), so those keep working.

$ GITHUB_ACTIONS=true DEPOT_JOB_URL=… codspeed run --mode walltime --skip-upload -- echo hi
Error: Depot CI is not supported yet by CodSpeed.
Reach out at support@codspeed.io or on Discord (https://codspeed.io/discord) to request support for it.

Closes COD-3635

Depot CI mimics GitHub Actions: it sets `GITHUB_ACTIONS=true` and the usual
`GITHUB_*` variables, so its runs were detected as GitHub Actions ones. Its run
ids do not exist on GitHub Actions, so CodSpeed could not process them.

Fail before running the benchmarks when `DEPOT_JOB_URL` is set, which Depot
only sets on Depot CI and not on its GitHub Actions runners, with a message
pointing to support to request Depot CI support.

Closes COD-3635
@codspeed

codspeed Bot commented Sep 28, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 13.99%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

❌ 1 regressed benchmark
✅ 32 untouched benchmarks
⏩ 4 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ WallTime memtrack track tar 8 s 9.3 s -13.99%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing fix/reject-depot-ci (66db707) with main (7c135cc)

Open in CodSpeed

Footnotes

  1. 4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant