Skip to content

ci: disable earthly TLS in bot workflows (settings-hash parity with CI)#1693

Open
skylar-simoncelli wants to merge 3 commits into
mainfrom
skylar/bot-workflows-earthly-tls
Open

ci: disable earthly TLS in bot workflows (settings-hash parity with CI)#1693
skylar-simoncelli wants to merge 3 commits into
mainfrom
skylar/bot-workflows-earthly-tls

Conversation

@skylar-simoncelli

Copy link
Copy Markdown
Contributor

Problem

rebuild-metadata-bot.yml and rebuild-chainspec-bot.yml run earthly -P on the self-hosted pool without the tls_enabled: false config append that every job in continuous-integration.yml applies. Earthly derives a settings hash from its buildkit config; on mismatch it docker rm -fs the shared earthly-buildkitd and recreates it, cancelling every in-flight earthly build on the box.

Observed on 2026-06-12 (fsn1-runner-01): one /bot rebuild-metadata comment caused 4 buildkitd recreations in 45s (dockerd journal, 08:48:22-08:49:07 UTC) and simultaneously killed three unrelated jobs (+test on #1677 with unlazy force execution: Canceled: context canceled, a Local Environment Tests, another Run tests). The bot jobs themselves also fail (TLS handshake against the non-TLS daemon, no certs provisioned), so users re-trigger and kill more CI each time. Repeated at 09:43 UTC.

Fix

Add the same tls_enabled: false append step the CI jobs use to both bot workflows, between GHCR login and the earthly invocation. This fixes the bots' own failures AND stops them restarting the shared daemon.

Both workflows trigger on issue_comment, so they execute the workflow file from the default branch — this takes effect for all open PRs immediately on merge, no rebases needed.

Testing

After merge, comment /bot rebuild-metadata on any PR and confirm (a) the job passes, (b) docker ps --filter name=earthly-buildkitd on fsn1-runner-01 shows the daemon NOT freshly recreated.

@skylar-simoncelli skylar-simoncelli requested a review from a team as a code owner June 12, 2026 10:10
@skylar-simoncelli skylar-simoncelli added this pull request to the merge queue Jun 12, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 12, 2026
@gilescope gilescope added this pull request to the merge queue Jun 16, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 16, 2026
@justinfrevert justinfrevert enabled auto-merge June 22, 2026 22:26

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1d61c237c0

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

# cancelling every in-flight build on the box.
- name: Disable earthly TLS (self-hosted runner has no certs)
run: |
echo " tls_enabled: false" >> .earthly/config.yml

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Use the same self-hosted Earthly config as CI

On self-hosted runners, the other Earthly workflows source .envrc before invoking Earthly (for example .github/workflows/continuous-integration.yml:184), and .envrc:29-30 switches them to .earthly/config.selfhosted.yml, which includes both tls_enabled: false and buildkit_additional_args: ["-m", "180Gb"] (.earthly/config.selfhosted.yml:15-20). This bot still runs with EARTHLY_CONFIG: .earthly/config.yml, so appending only TLS leaves its BuildKit settings hash different from CI and the metadata bot can still force-recreate the shared earthly-buildkitd on each run. Use the same self-hosted config (or source .envrc) for the bot invocation instead of patching the default config.

Useful? React with 👍 / 👎.

# cancelling every in-flight build on the box.
- name: Disable earthly TLS (self-hosted runner has no certs)
run: |
echo " tls_enabled: false" >> .earthly/config.yml

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Use the same self-hosted Earthly config as CI

On self-hosted runners, the other Earthly workflows source .envrc before invoking Earthly (for example .github/workflows/continuous-integration.yml:648), and .envrc:29-30 switches them to .earthly/config.selfhosted.yml, which includes both tls_enabled: false and buildkit_additional_args: ["-m", "180Gb"] (.earthly/config.selfhosted.yml:15-20). This bot still runs with EARTHLY_CONFIG: .earthly/config.yml, so appending only TLS leaves its BuildKit settings hash different from CI and the chainspec bot can still force-recreate the shared earthly-buildkitd on each run. Use the same self-hosted config (or source .envrc) for the bot invocation instead of patching the default config.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants