Skip to content

acme-configuration.md

Latest commit

 

History

History
218 lines (133 loc) · 11.5 KB

acme-configuration.md

File metadata and controls

218 lines (133 loc) · 11.5 KB
Raw

ACME Configuration

This document explains Vault's ACME configuration in depth, with particular attention to the parts that are not obvious from the Vault documentation — specifically the response header requirement, DNS resolution behaviour, and how to diagnose failures. The goal is to save you the debugging time we already spent.

What ACME is

ACME (Automatic Certificate Management Environment) is a protocol for automated certificate issuance and renewal. Let's Encrypt made it widely used — it is why certbot can renew a certificate for you automatically without any manual steps. Vault 1.14 and later implements the same ACME protocol on top of its PKI engine. This means any standard ACME client — certbot, acme.sh, Domino CertMgr, Caddy, Traefik — can request and automatically renew certificates from your private Vault CA using exactly the same tooling it uses with Let's Encrypt.

The response header problem

This is the most important section in this document. It is also the problem most likely to waste hours of your time if you do not know about it upfront.

Background

Vault has a security mechanism that filters HTTP response headers. When a backend engine (like the PKI engine) adds a custom header to an HTTP response, Vault's proxy layer strips it out before sending the response to the client. The intent is to prevent backends from setting arbitrary headers that could confuse clients or cause security issues.

The problem is that ACME depends on specific HTTP response headers to function. These headers are generated by the PKI engine, and Vault strips them — unless you explicitly configure the PKI mount to allow them.

The three headers ACME needs

Header Why ACME needs it
Replay-Nonce Every ACME operation requires a fresh nonce (a unique one-time token) to prevent replay attacks. Clients fetch a nonce from the new-nonce endpoint before each request. Without this header, the endpoint returns HTTP 200 with no nonce — clients silently retry forever and never make progress.
Location After creating an account or placing an order, the ACME server returns the URL of the newly created resource in the Location header. Without it, clients cannot find their account and fail with errors like "cannot find account ID URL".
Link Carries relations between resources: terms-of-service, up (pointing to the certificate chain), and index (pointing back to the directory). Without it, some clients cannot complete the full flow.

Why this is hard to debug

Several things make this problem difficult to identify:

  • Vault returns HTTP 200 for the nonce endpoint even when the nonce is missing. The endpoint is reachable, the response looks normal, but the Replay-Nonce header is absent. Most tools and clients interpret a 200 response as success and do not immediately surface the missing header.

  • There is no error in the Vault logs. Vault is not doing anything wrong from its own perspective — it served the request successfully. The header filtering is silent.

  • The Vault UI does not warn you. When you enable ACME through the Vault UI, nothing tells you that the headers also need to be configured. The ACME enable step succeeds, and the UI moves on.

  • vault write pki/config/acme enabled=true exits with no error. The command succeeds. ACME appears to be running. But it cannot work yet.

The result is that you can have ACME enabled, the directory endpoint reachable, and clients attempting to use it — and the failure is not an error anywhere, just an ACME client that silently retries the nonce fetch indefinitely.

The fix

After enabling ACME, you must explicitly tell Vault's PKI mount to allow these three headers through. Run this against your Vault server:

vault secrets tune \
  -allowed-response-headers="Replay-Nonce" \
  -allowed-response-headers="Link" \
  -allowed-response-headers="Location" \
  pki

Provisioner 03-pki-acme/setup.sh runs this automatically as part of the setup sequence. If you enabled ACME manually, through the Vault UI, or by running vault write pki/config/acme enabled=true directly, you must run secrets tune separately.

Verifying the headers are present

Once the mount tuning is in place, confirm the nonce endpoint is returning the header:

curl -s -D - https://your-vault/v1/pki/acme/new-nonce | head -10

The -D - flag prints response headers to stdout. You are looking for a line like:

replay-nonce: ABCDefghIJKLmnopQRST...

If that line is present, the header configuration is working. If the response returns 200 or 204 but no replay-nonce line appears, the secrets tune step has not been applied.

DNS for ACME challenge validation

How challenge validation works in this setup

In standard ACME with Let's Encrypt, the CA is a remote service run by Let's Encrypt. It validates your domain ownership by reaching out from Let's Encrypt's infrastructure.

With Vault, the CA is your Vault container. Vault itself reaches out to validate challenges — not some external service. This is an important difference that affects DNS configuration.

HTTP-01 challenge

With HTTP-01, the ACME client places a token file at a well-known URL on the target server:

http://<your-domain>/.well-known/acme-challenge/<token>

Vault then fetches that URL from inside the Docker container to verify that the domain resolves to a server the client controls.

For this to work from inside the container:

  • The domain must resolve to an IP address that Docker can reach
  • Port 80 on that IP must be accessible from the container
  • The ACME client must have a web server serving the token at that path

DNS-01 challenge

With DNS-01, the ACME client adds a TXT record to DNS:

_acme-challenge.<your-domain>  TXT  "<token>"

Vault then queries DNS for that TXT record from inside the Docker container to verify domain ownership.

For this to work from inside the container:

  • The domain must be in DNS that Docker can query
  • The DNS server must be reachable from the container

Docker's DNS resolver

Docker containers use Docker's internal DNS resolver, which by default runs at 127.0.0.11 inside the container. This resolver forwards queries to whatever DNS servers the host machine uses.

On a typical Linux host, that is whatever is in /etc/resolv.conf. On a Windows host with WSL, it is the DNS servers configured in Windows.

The problem with internal domains

If your target domain exists only on an internal or corporate DNS server — and is not registered in public DNS — Docker's default forwarding may not reach it. This means both HTTP-01 and DNS-01 validation fail with errors like "no such host" or "unable to resolve domain".

The fix is to configure DNS servers explicitly for the Vault container in server/docker-compose.yml:

dns:
  - 1.1.1.1          # public resolver — for internet-registered domains
  # - 192.168.1.1    # uncomment and set your internal DNS server if needed

With an internal DNS server listed, Docker will forward queries to it, and Vault can resolve your internal domains during challenge validation.

An advantage of private Vault ACME

Let's Encrypt validates domain ownership by reaching out from the public internet. If your domain is not in public DNS, Let's Encrypt cannot validate it and cannot issue a certificate for it.

Vault has no such restriction. Because Vault is doing the validation itself, you can point it at any DNS server — including an internal one that serves private domain names that have never appeared in public DNS. This means you can issue ACME-managed certificates for internal hostnames (app01.corp.example.com, vault.internal, etc.) that Let's Encrypt could never validate.

Challenge type comparison

Challenge How validation works When to use
HTTP-01 Vault fetches a token via HTTP from the domain being validated Server has DNS resolution from Vault and port 80 accessible from the container
DNS-01 Vault queries a DNS TXT record for the domain Any server, including ones without port 80; domain must be in DNS reachable from the container
TLS-ALPN-01 Vault connects to port 443 using a special TLS handshake extension Rarely used; requires control of TLS on the target server

DNS-01 is generally the more flexible choice for internal infrastructure because it does not require port 80 to be open on the servers being enrolled.

Diagnosing ACME failures

Work through these checks in order. Each one narrows down where the problem is.

1. Check the ACME directory is reachable

curl -s https://your-vault/v1/pki/acme/directory | jq .

This should return a JSON object containing URLs for newNonce, newAccount, newOrder, and keyChange. If this fails, ACME is not enabled or the PKI engine is not mounted correctly. Check ./server/vault.sh read pki/config/acme.

If you get a redirect (HTTP 301 or 307) instead of the directory JSON, the cluster_path in Vault does not match the URL you are using — see step 4.

2. Check the nonce endpoint returns a nonce

curl -s -D - https://your-vault/v1/pki/acme/new-nonce | head -10

Look for replay-nonce: in the response headers. A 200 or 204 without that header means secrets tune has not been applied — go back to the response header fix above.

3. Verify ACME is enabled

./server/vault.sh read pki/config/acme

The enabled field should be true. If it is not, run provisioner 03-pki-acme or enable it manually and apply the header tuning.

4. Verify the cluster path

./server/vault.sh read pki/config/cluster

The path field is the base URL that Vault uses for all ACME endpoint URLs. It must match exactly the URL that ACME clients use to reach Vault. If a client connects to https://vault.example.com but the cluster path is set to https://vault.example.com:8200, clients will follow redirects to the wrong port and fail.

If the cluster path is wrong, re-run provisioner 03-pki-acme with the correct VAULT_API_ADDR value.

5. Check DNS resolution from inside the container

docker inspect vault --format '{{json .HostConfig.Dns}}'

null means the container is using Docker's default DNS forwarding from the host. If your target domain is internal-only, add an internal DNS server to server/docker-compose.yml under the dns: key (see the DNS section above).

To test DNS resolution directly from inside the container without a shell, you can temporarily use a debug container on the same Docker network:

docker run --rm --network host alpine nslookup your-internal-domain.example.com 192.168.1.1

Replace 192.168.1.1 with your internal DNS server address. This tells you whether Docker can reach the DNS server and whether it has the record you need.

6. Check for certificate trust issues

ACME clients connecting to Vault over HTTPS must trust Vault's TLS certificate. If Vault is using the bootstrap self-signed certificate (from tls/bootstrap.sh), clients will reject it unless they are configured to trust it or to skip verification.

For acme.sh:

acme.sh --issue --server https://your-vault/v1/pki/acme/directory \
  -d host.example.com --standalone --insecure   # --insecure only for testing

For Domino CertMgr: import Vault's CA certificate into the Domino trust store, or use the internal CA certificate from provisioner 02-pki-internal-ca once Vault has issued its own TLS certificate through CertMgr.