Skip to content

fix(ssl): detect and rebuild stale ACME orders on verify/retry - #493

Open
mrrobot47 wants to merge 4 commits into
EasyEngine:developfrom
mrrobot47:fix/ssl-stale-order-self-heal
Open

mrrobot47 wants to merge 4 commits into
EasyEngine:developfrom
mrrobot47:fix/ssl-stale-order-self-heal

Conversation

@mrrobot47

@mrrobot47 mrrobot47 commented Jun 30, 2026 •

Copy link
Copy Markdown
Member

Problem

A one-time DNS hiccup or timeout could permanently break SSL for a site. On issuance failure the ACME order/challenge files are left on disk; the user fixes DNS and re-runs ee site ssl-verify → check() blindly reloads the on-disk order and reuses it. Within ~7 days while the order is still pending this works (and is preserved). But once Let's Encrypt expires the order, or marks an authorization invalid after a failed validation, the reloaded order is dead and every retry fails (for an expired or purged authorization, with an uncaught 404 MalformedServerException) — because the retry path never requests a fresh order (only init_le calls authorize()).

Fix

check() now detects a stale stored order and rebuilds it before validating. isCertificateOrderStale() reloads, for each domain, the challenge the active solver uses (the same challenge check() validates; once one challenge is attempted Let's Encrypt drops the others). The order is stale when that challenge is invalid, when Let's Encrypt answers 404 for it (an expired authorization returns 404 "Expired authorization", a purged one "No such challenge"), or when the order lacks a requested domain. pending, processing and valid are live. On staleness it revokes the stored challenges (best effort) and calls authorize() for a fresh order; the stale order is kept until authorize() overwrites it, so a failed rebuild (rate limit, network) is retried on the next run instead of dead-ending.

Safeguards:

  • A transient Let's Encrypt error (5xx/429/network) during the staleness check is inconclusive: the order is kept and no newOrder is spent. (The run itself still stops with the underlying error from the main validation loop, as before; that is a separate follow-up.)
  • The creation path (fresh order) and the normal "DNS not ready yet, retry later" path (pending order) do not trigger a rebuild — behavior there is unchanged.
  • For manual DNS-01 (no Cloudflare API key) a rebuild issues a new TXT value, so the command warns the user to publish the new record and re-run, instead of failing validation confusingly. HTTP-01 and Cloudflare-automated DNS continue to validation as before.

Known limitation

A user-run ee site ssl-verify <site> builds its domain list without www, so it only finds the stored order when that order is for the bare domain (sites whose www check failed at create time) or for DNS/wildcard sites. For HTTP-01 sites whose cert includes www, the stored order isn't found and this self-heal doesn't run; fixing ssl-verify's domain set is a separate follow-up. ssl-renew is unaffected (it always requests a fresh order).

Testing

Manual: on a site with a stored order for its domain set, point the stored challenge URLs at a nonexistent authorization (LE answers 404, like an expired one) and run ee site ssl-verify → "Stored ACME order is stale/expired; requesting a fresh order.", fresh challenge URLs in certificate_order.json, validation succeeds, and no new certificate while the current one is valid. Negative tests: a valid or pending order is not rebuilt (order file unchanged); with the ACME host unreachable the order is kept ("treating as inconclusive").

Tested on Ubuntu 26.04 with EasyEngine 4.12.0 against production Let's Encrypt (one certificate, then the stale-order, valid-order and unreachable cases on its stored order).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

The stale order was deleted before authorize() ran, so a failed rebuild (rate limit, network error, rejected identifier) left no order behind: the next ssl-verify fell into the order-less path and stopped with "not yet authorized" instead of retrying. authorize() overwrites the stored order on success, so the explicit removal isn't needed.

revokeAuthorizationChallenges() only catches revocation/CLI exceptions, while its newOrder request can throw ACME server or client exceptions, so the "best-effort" cleanup could abort check() before the rebuild. Catch and log those.
reloadAuthorization() fetches the challenge URL, and Let's Encrypt answers 404 ("Expired authorization") for a challenge once its authorization has expired; it never returns an "expired" or "deactivated" status there (challenge statuses are pending, processing, valid and invalid). The expiry case the stale-order check was written for was therefore treated as inconclusive, and check() then died on the same 404 in its own reload. Treat a 404 as stale, keep other errors inconclusive, and count "processing" as live.
isCertificateOrderStale() reloaded the first challenge of each authorization, which is often tls-alpn-01. Once any challenge has been attempted, Let's Encrypt drops the others and answers 404 "No such challenge" for them. So a valid order was treated as stale (and rebuilt on every ssl-verify) now that a 404 means stale, and an invalid one was never seen as invalid. Reload the challenge that the solver supports, the same one check() validates.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants