ci: disable earthly TLS in bot workflows (settings-hash parity with CI)#1693
ci: disable earthly TLS in bot workflows (settings-hash parity with CI)#1693skylar-simoncelli wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1d61c237c0
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| # cancelling every in-flight build on the box. | ||
| - name: Disable earthly TLS (self-hosted runner has no certs) | ||
| run: | | ||
| echo " tls_enabled: false" >> .earthly/config.yml |
There was a problem hiding this comment.
Use the same self-hosted Earthly config as CI
On self-hosted runners, the other Earthly workflows source .envrc before invoking Earthly (for example .github/workflows/continuous-integration.yml:184), and .envrc:29-30 switches them to .earthly/config.selfhosted.yml, which includes both tls_enabled: false and buildkit_additional_args: ["-m", "180Gb"] (.earthly/config.selfhosted.yml:15-20). This bot still runs with EARTHLY_CONFIG: .earthly/config.yml, so appending only TLS leaves its BuildKit settings hash different from CI and the metadata bot can still force-recreate the shared earthly-buildkitd on each run. Use the same self-hosted config (or source .envrc) for the bot invocation instead of patching the default config.
Useful? React with 👍 / 👎.
| # cancelling every in-flight build on the box. | ||
| - name: Disable earthly TLS (self-hosted runner has no certs) | ||
| run: | | ||
| echo " tls_enabled: false" >> .earthly/config.yml |
There was a problem hiding this comment.
Use the same self-hosted Earthly config as CI
On self-hosted runners, the other Earthly workflows source .envrc before invoking Earthly (for example .github/workflows/continuous-integration.yml:648), and .envrc:29-30 switches them to .earthly/config.selfhosted.yml, which includes both tls_enabled: false and buildkit_additional_args: ["-m", "180Gb"] (.earthly/config.selfhosted.yml:15-20). This bot still runs with EARTHLY_CONFIG: .earthly/config.yml, so appending only TLS leaves its BuildKit settings hash different from CI and the chainspec bot can still force-recreate the shared earthly-buildkitd on each run. Use the same self-hosted config (or source .envrc) for the bot invocation instead of patching the default config.
Useful? React with 👍 / 👎.
Problem
rebuild-metadata-bot.ymlandrebuild-chainspec-bot.ymlrunearthly -Pon the self-hosted pool without thetls_enabled: falseconfig append that every job incontinuous-integration.ymlapplies. Earthly derives a settings hash from its buildkit config; on mismatch itdocker rm -fs the sharedearthly-buildkitdand recreates it, cancelling every in-flight earthly build on the box.Observed on 2026-06-12 (fsn1-runner-01): one
/bot rebuild-metadatacomment caused 4 buildkitd recreations in 45s (dockerd journal, 08:48:22-08:49:07 UTC) and simultaneously killed three unrelated jobs (+teston #1677 withunlazy force execution: Canceled: context canceled, a Local Environment Tests, another Run tests). The bot jobs themselves also fail (TLS handshake against the non-TLS daemon, no certs provisioned), so users re-trigger and kill more CI each time. Repeated at 09:43 UTC.Fix
Add the same
tls_enabled: falseappend step the CI jobs use to both bot workflows, between GHCR login and the earthly invocation. This fixes the bots' own failures AND stops them restarting the shared daemon.Both workflows trigger on
issue_comment, so they execute the workflow file from the default branch — this takes effect for all open PRs immediately on merge, no rebases needed.Testing
After merge, comment
/bot rebuild-metadataon any PR and confirm (a) the job passes, (b)docker ps --filter name=earthly-buildkitdon fsn1-runner-01 shows the daemon NOT freshly recreated.