diff --git a/.changeset/rotate-encryption-key.md b/.changeset/rotate-encryption-key.md new file mode 100644 index 0000000..9747284 --- /dev/null +++ b/.changeset/rotate-encryption-key.md @@ -0,0 +1,6 @@ +--- +'@crypt.fyi/core': minor +'@crypt.fyi/server': minor +--- + +Allow rotating the server's `ENCRYPTION_KEY` without losing unexpired secrets. Data encrypted or digested at rest now records which key wrote it, and the new `ENCRYPTION_PREVIOUS_KEYS` setting keeps old keys available for reads. A new `rotate-keys` command re-encrypts existing entries with the new key and reports when the old key can be removed. Core adds `machine.createKeyring` and keyring-based encrypt, decrypt and digest functions, and `machine.isEncrypted` and `machine.isDigest` also recognise the new formats. diff --git a/README.md b/README.md index 2a8d51b..8c7d9fa 100644 --- a/README.md +++ b/README.md @@ -85,7 +85,25 @@ API_URL=https://{your-domain-here} ENCRYPTION_KEY=$(openssl rand -base64 32) docker compose up --build ``` -`ENCRYPTION_KEY` protects verification hashes, delete tokens, IP restrictions and webhook URLs at rest in Redis. Keep it stable across restarts (e.g. in a `.env` file next to `docker-compose.yaml`); changing it makes all existing secrets unreadable until they expire. +`ENCRYPTION_KEY` protects verification hashes, delete tokens, IP restrictions and webhook URLs at rest in Redis. Keep it stable across restarts (e.g. in a `.env` file next to `docker-compose.yaml`). Replacing it outright makes all existing secrets unreadable until they expire; to change it without losing any, follow [Rotating `ENCRYPTION_KEY`](#rotating-encryption_key). + +#### Rotating `ENCRYPTION_KEY` + +The server holds a keyring: `ENCRYPTION_KEY` is the primary key, used for every new write, and `ENCRYPTION_PREVIOUS_KEYS` is a comma-separated list of older keys that are only used to read existing entries. Everything written at rest records which key wrote it, so the server picks the right key on each read. + +1. **Generate a new key:** `openssl rand -base64 32`. +2. **Multiple server instances only: make the new key known everywhere first.** Add the new key to `ENCRYPTION_PREVIOUS_KEYS` on every instance and roll them all out. Otherwise an instance that already writes with the new key can produce entries that an instance still on the old configuration can't read. A single instance can skip this step. +3. **Promote the new key.** Set `ENCRYPTION_KEY` to the new key and move the old key to `ENCRYPTION_PREVIOUS_KEYS`, then restart every instance. The startup log line `Loaded encryption keyring` shows how many previous keys each instance loaded. +4. **Re-encrypt existing entries.** With the same environment as the server, run: + + ```bash + docker compose exec server node packages/server/dist/rotate-keys.js + ``` + + (`pnpm --filter @crypt.fyi/server keys:rotate` outside Docker; add `--dry-run` to report without writing.) It re-encrypts IP restrictions, webhook URLs and delete tokens with the new key, keeping each entry's TTL and never overwriting a concurrent read or burn. It's safe to run repeatedly while the server takes traffic. Verification-hash digests can't be re-keyed, because the server never learns the hash itself, so entries created before step 3 keep needing the old key until they expire. The report's `previousKeysNeededUntil` says when that is (at most the max TTL, 7 days by default). +5. **Retire the old key.** After `previousKeysNeededUntil` has passed, run the command again. Once it reports `"safeToRemovePreviousKeys": true` (no entry or queued webhook depends on a previous key), remove the old key from `ENCRYPTION_PREVIOUS_KEYS` and restart. + +If the old key leaked, you can shorten step 5, but removing it before `previousKeysNeededUntil` makes the remaining older secrets unreadable. Entries already re-encrypted in step 4 no longer expose their IP restrictions, webhook URLs or delete tokens to the old key. A non-zero `unreadable` count in the report means an entry was written with a key that is no longer configured; its secret can't be recovered unless you add that key back. `API_URL` is read when the web container starts, so changing it only needs a restart (`docker compose up -d`), not a rebuild. It must be a plain `http(s)://host[:port][/path]` URL; the container refuses to start otherwise. @@ -95,7 +113,7 @@ A prebuilt API server image (`linux/amd64`, `linux/arm64`) is published to GHCR docker pull ghcr.io/osbytes/crypt.fyi-server:latest ``` -It runs with `NODE_ENV=production` and requires `REDIS_URL` and `ENCRYPTION_KEY`. +It runs with `NODE_ENV=production` and requires `REDIS_URL` and `ENCRYPTION_KEY` (plus `ENCRYPTION_PREVIOUS_KEYS` while [rotating it](#rotating-encryption_key)). The web client image is not published to GHCR yet; build it with `docker compose` as above. diff --git a/SPECIFICATION.md b/SPECIFICATION.md index f054cab..fb1358a 100644 --- a/SPECIFICATION.md +++ b/SPECIFICATION.md @@ -302,6 +302,20 @@ Abstract alone cannot be replayed to read or burn an entry - Delete tokens MUST NOT be stored in plaintext; servers MUST store them encrypted so they can be returned in webhook payloads + - The server-side key that protects these values MUST be rotatable without + making unexpired entries unreadable. Servers SHOULD hold a keyring of one + primary key, used for all new writes, and previous keys used only for + reads, and SHOULD tag each stored ciphertext and digest with the id of + the key that produced it. The reference implementation uses + `m2::` ciphertexts (AES-256-GCM with an HKDF-derived key, the header + bound as associated data) and `hm2::` HMAC-SHA-256 digests, where + `kid` is derived from the key itself. Encrypted values SHOULD be + re-encrypted with the primary key after a rotation. Digests cannot be + re-keyed without the original hash, so a previous key MUST stay loaded + until every entry digested with it has expired, which the maximum TTL + bounds + - A read MUST fail without consuming a failed attempt when the entry's key + is not loaded, so a misconfigured keyring cannot burn entries ### 6.3. Access Control @@ -378,7 +392,9 @@ Abstract Key configuration parameters: - `REDIS_URL`: Redis connection string - - `ENCRYPTION_KEY`: Server-side encryption key for metadata + - `ENCRYPTION_KEY`: Server-side encryption key for metadata (primary key) + - `ENCRYPTION_PREVIOUS_KEYS`: Comma-separated previous keys, accepted for + reads only while rotating `ENCRYPTION_KEY` - `RATE_LIMIT_MAX`: Maximum requests per time window - `BODY_LIMIT_BYTES`: Maximum request body size - `CORS_ORIGIN`: Allowed CORS origins diff --git a/docker-compose.yaml b/docker-compose.yaml index 4faf115..af4445f 100644 --- a/docker-compose.yaml +++ b/docker-compose.yaml @@ -39,6 +39,8 @@ services: - PORT=${SERVER_PORT:-4321} # Protects verification hashes, delete tokens, IP restrictions and webhook URLs at rest in Redis. - ENCRYPTION_KEY=${ENCRYPTION_KEY:?ENCRYPTION_KEY is required (e.g. openssl rand -base64 32)} + # Comma-separated keys kept for reading existing entries while rotating ENCRYPTION_KEY (see README). + - ENCRYPTION_PREVIOUS_KEYS=${ENCRYPTION_PREVIOUS_KEYS:-} ports: - '${SERVER_PORT:-4321}:${SERVER_PORT:-4321}' depends_on: diff --git a/packages/core/src/encryption/machine.test.ts b/packages/core/src/encryption/machine.test.ts index d34c710..41717c2 100644 --- a/packages/core/src/encryption/machine.test.ts +++ b/packages/core/src/encryption/machine.test.ts @@ -1,4 +1,20 @@ -import { encrypt, decrypt, digest, isDigest, isEncrypted } from './machine'; +import { + encrypt, + decrypt, + digest, + isDigest, + isEncrypted, + keyId, + keyIdOf, + createKeyring, + encryptWithKeyring, + decryptWithKeyring, + digestWithKeyring, + digestCandidates, + hasPreviousKeys, + isOnPrimaryKey, +} from './machine'; +import { DecryptError } from './encryption'; import { encrypt as gcmEncrypt } from './gcm'; describe('machine digest', () => { @@ -57,3 +73,105 @@ describe('machine cipher', () => { expect(await decrypt(legacy, key)).toBe(plaintext); }); }); + +describe('machine keyring', () => { + const oldKey = 'old-high-entropy-machine-key-0123456789'; + const newKey = 'new-high-entropy-machine-key-9876543210'; + const before = createKeyring(oldKey); + const rotating = createKeyring(newKey, [oldKey]); + const after = createKeyring(newKey); + + it('derives a stable, url-safe id that does not contain the key', () => { + expect(keyId(oldKey)).toBe(keyId(oldKey)); + expect(keyId(oldKey)).not.toBe(keyId(newKey)); + expect(keyId(oldKey)).toMatch(/^[A-Za-z0-9_-]{8}$/); + expect(oldKey).not.toContain(keyId(oldKey)); + }); + + it('uses the first key as primary and ignores duplicates and blanks', () => { + const keyring = createKeyring(newKey, [oldKey, newKey, '']); + expect(keyring.primaryId).toBe(keyId(newKey)); + expect([...keyring.keys.keys()]).toEqual([keyId(newKey), keyId(oldKey)]); + expect(() => createKeyring('')).toThrow(); + }); + + it('tags ciphertext with the primary key id', async () => { + const enc = await encryptWithKeyring('secret', rotating); + expect(enc.startsWith(`m2:${keyId(newKey)}:`)).toBe(true); + expect(isEncrypted(enc)).toBe(true); + expect(keyIdOf(enc)).toBe(keyId(newKey)); + expect(enc).not.toContain('secret'); + }); + + it('reads values written before a rotation while writing with the new key', async () => { + const written = await encryptWithKeyring('secret', before); + expect(await decryptWithKeyring(written, rotating)).toBe('secret'); + expect(keyIdOf(await encryptWithKeyring('secret', rotating))).toBe(keyId(newKey)); + }); + + it('fails once the key a value was written with is removed', async () => { + const written = await encryptWithKeyring('secret', before); + await expect(decryptWithKeyring(written, after)).rejects.toThrow(); + }); + + it('rejects a ciphertext relabelled with another key id', async () => { + const written = await encryptWithKeyring('secret', before); + const relabelled = written.replace(keyId(oldKey), keyId(newKey)); + await expect(decryptWithKeyring(relabelled, rotating)).rejects.toThrow(); + }); + + it('reads m1 and legacy PBKDF2 values under any key in the ring', async () => { + const m1 = await encrypt('m1 value', oldKey); + const legacy = await gcmEncrypt('legacy value', oldKey); + expect(await decryptWithKeyring(m1, rotating)).toBe('m1 value'); + expect(await decryptWithKeyring(legacy, rotating)).toBe('legacy value'); + await expect(decryptWithKeyring(m1, after)).rejects.toThrow(); + }); + + it('digests with the primary key and tags the digest', () => { + const d = digestWithKeyring('value', rotating, 'ctx'); + expect(d.startsWith(`hm2:${keyId(newKey)}:`)).toBe(true); + expect(isDigest(d)).toBe(true); + expect(d).toBe(digestWithKeyring('value', after, 'ctx')); + expect(d).not.toBe(digestWithKeyring('value', before, 'ctx')); + }); + + it('matches stored digests by the key they name', () => { + const stored = digestWithKeyring('value', before, 'ctx'); + expect(digestCandidates(stored, 'value', rotating, 'ctx')).toEqual([stored]); + expect(digestCandidates(stored, 'other', rotating, 'ctx')).not.toContain(stored); + expect(digestCandidates(stored, 'value', after, 'ctx')).toEqual([]); + }); + + it('matches hm1 digests under any key and passes pre-digest values through', () => { + const hm1 = digest('value', oldKey, 'ctx'); + expect(digestCandidates(hm1, 'value', rotating, 'ctx')).toContain(hm1); + expect(digestCandidates(hm1, 'value', after, 'ctx')).not.toContain(hm1); + expect(digestCandidates('abc123', 'abc123', rotating, 'ctx')).toEqual(['abc123']); + }); + + it('rejects values that cannot be ciphertext without trying each key', async () => { + await expect(decryptWithKeyring('192.168.1.1,10.0.0.0/24', rotating)).rejects.toThrow( + DecryptError, + ); + await expect(decryptWithKeyring('::1', rotating)).rejects.toThrow(DecryptError); + }); + + it('tells whether a value was written with the primary key', async () => { + expect(await isOnPrimaryKey(await encryptWithKeyring('x', rotating), rotating)).toBe(true); + expect(await isOnPrimaryKey(await encryptWithKeyring('x', before), rotating)).toBe(false); + expect(await isOnPrimaryKey(await encrypt('x', newKey), rotating)).toBe(true); + expect(await isOnPrimaryKey(await encrypt('x', oldKey), rotating)).toBe(false); + }); + + it('reports whether a keyring holds previous keys', () => { + expect(hasPreviousKeys(rotating)).toBe(true); + expect(hasPreviousKeys(after)).toBe(false); + expect(hasPreviousKeys(createKeyring(newKey, [newKey]))).toBe(false); + }); + + it('never accepts the stored digest itself as the value', () => { + const stored = digestWithKeyring('value', rotating, 'ctx'); + expect(digestCandidates(stored, stored, rotating, 'ctx')).not.toContain(stored); + }); +}); diff --git a/packages/core/src/encryption/machine.ts b/packages/core/src/encryption/machine.ts index 8c13ae7..0ea37a9 100644 --- a/packages/core/src/encryption/machine.ts +++ b/packages/core/src/encryption/machine.ts @@ -35,12 +35,27 @@ const DIGEST_PREFIX = 'hm1:'; // derived from the same machine key. const DIGEST_HKDF_INFO = utf8ToBytes('crypt.fyi/machine-digest'); +// Keyring formats, described in the keyring section below. +const KEYED_SCHEME_PREFIX = 'm2:'; +const KEYED_DIGEST_PREFIX = 'hm2:'; + const deriveKey = (key: string, salt: Uint8Array): Uint8Array => hkdf(sha256, utf8ToBytes(key), salt, HKDF_INFO, KEY_LENGTH); -export const isEncrypted = (value: string): boolean => value.startsWith(SCHEME_PREFIX); +const deriveDigestKey = (key: string): Uint8Array => + hkdf(sha256, utf8ToBytes(key), undefined, DIGEST_HKDF_INFO, KEY_LENGTH); + +export const isEncrypted = (value: string): boolean => + value.startsWith(SCHEME_PREFIX) || value.startsWith(KEYED_SCHEME_PREFIX); + +export const isDigest = (value: string): boolean => + value.startsWith(DIGEST_PREFIX) || value.startsWith(KEYED_DIGEST_PREFIX); -export const isDigest = (value: string): boolean => value.startsWith(DIGEST_PREFIX); +/** Raw base64 HMAC, shared by the hm1 and hm2 formats. */ +const rawDigest = (value: string, digestKey: Uint8Array, context: string): string => { + const message = concatBytes(utf8ToBytes(context), new Uint8Array([0]), utf8ToBytes(value)); + return Buffer.from(hmac(sha256, digestKey, message)).toString('base64'); +}; /** * Deterministic keyed digest, for values that must be compared where the machine @@ -48,11 +63,8 @@ export const isDigest = (value: string): boolean => value.startsWith(DIGEST_PREF * digest to one use and must not contain a NUL byte, which separates it from * `value` so the pair is encoded unambiguously. */ -export const digest = (value: string, key: string, context: string): string => { - const digestKey = hkdf(sha256, utf8ToBytes(key), undefined, DIGEST_HKDF_INFO, KEY_LENGTH); - const message = concatBytes(utf8ToBytes(context), new Uint8Array([0]), utf8ToBytes(value)); - return DIGEST_PREFIX + Buffer.from(hmac(sha256, digestKey, message)).toString('base64'); -}; +export const digest = (value: string, key: string, context: string): string => + DIGEST_PREFIX + rawDigest(value, deriveDigestKey(key), context); export const encrypt: Encrypt = async (content, key) => { try { @@ -89,3 +101,224 @@ export const decrypt: Decrypt = async (encryptedContent, key) => { throw new DecryptError(error); } }; + +// --------------------------------------------------------------------------- +// Keyrings: rotating the machine key without losing data +// --------------------------------------------------------------------------- +// +// A keyring holds one primary key, used for every new ciphertext and digest, +// and any number of previous keys that are only used to read existing values. +// This follows the keyset model of Google Tink and Rails Active Record +// Encryption: add the new key, make it primary, keep the old one for reads +// until nothing depends on it, then remove it. +// +// Values written through a keyring carry the id of the key that produced them, +// so a read goes straight to the right key, and a value whose key has been +// removed fails loudly instead of being tried against every key. +// +// Formats: +// "m2:" || kid || ":" || base64( salt[16] || iv[12] || aes-256-gcm-ciphertext ) +// "hm2:" || kid || ":" || base64( hmac-sha256 ) +// +// For m2 the "m2::" header is the GCM associated data, so relabelling a +// ciphertext with another key id makes it fail authentication. +// +// The key id is derived from the key itself (HKDF with its own info string), +// so operators never assign or track ids and every server computes the same id +// for the same key. 48 bits is plenty to tell a handful of live keys apart. + +const KEY_ID_HKDF_INFO = utf8ToBytes('crypt.fyi/machine-key-id'); +const KEY_ID_LENGTH = 6; +// Legacy PBKDF2 ciphertext is plain base64. Anything else (such as plaintext IP +// restrictions from before they were encrypted) can't be one, and skipping it +// avoids running the deliberately slow PBKDF2 once per key on a hot path. +const BASE64 = /^[A-Za-z0-9+/]+={0,2}$/; + +export interface Keyring { + /** Id of the key used for new ciphertexts and digests. */ + readonly primaryId: string; + /** Every key this keyring can read with, primary first. */ + readonly keys: ReadonlyMap; + /** HMAC key per key id, derived once because every vault read needs one. */ + readonly digestKeys: ReadonlyMap; +} + +const toBase64Url = (bytes: Uint8Array): string => + Buffer.from(bytes).toString('base64').replace(/\+/g, '-').replace(/\//g, '_').replace(/=+$/, ''); + +/** Stable, non-secret identifier for a machine key. */ +export const keyId = (key: string): string => + toBase64Url(hkdf(sha256, utf8ToBytes(key), undefined, KEY_ID_HKDF_INFO, KEY_ID_LENGTH)); + +/** + * Builds a keyring from the primary key and any previous keys that existing + * values may still be encrypted or digested with. Duplicates and empty entries + * are ignored, so listing the primary key among the previous ones is harmless. + */ +export const createKeyring = (primary: string, previous: readonly string[] = []): Keyring => { + if (!primary) { + throw new Error('machine keyring requires a primary key'); + } + const keys = new Map(); + for (const key of [primary, ...previous]) { + if (!key) { + continue; + } + const id = keyId(key); + const existing = keys.get(id); + if (existing !== undefined && existing !== key) { + // Practically impossible at 48 bits, but silently keeping one key would + // make everything written with the other unreadable. + throw new Error('machine keyring key id collision; generate a new key'); + } + keys.set(id, key); + } + const digestKeys = new Map([...keys].map(([id, key]) => [id, deriveDigestKey(key)])); + return { primaryId: keyId(primary), keys, digestKeys }; +}; + +/** Whether the keyring holds any key besides the primary. */ +export const hasPreviousKeys = (keyring: Keyring): boolean => keyring.keys.size > 1; + +const primaryKey = (keyring: Keyring): string => { + const key = keyring.keys.get(keyring.primaryId); + if (key === undefined) { + throw new Error('machine keyring is missing its primary key'); + } + return key; +}; + +/** Parses ":", or returns undefined when the value doesn't match. */ +const splitKeyed = (value: string, prefix: string) => { + if (!value.startsWith(prefix)) { + return undefined; + } + const rest = value.slice(prefix.length); + const separator = rest.indexOf(':'); + if (separator <= 0) { + return undefined; + } + return { kid: rest.slice(0, separator), payload: rest.slice(separator + 1) }; +}; + +/** + * The id of the key a keyring ciphertext or digest was made with, or undefined + * for values that predate key ids (m1, hm1 and legacy PBKDF2). + */ +export const keyIdOf = (value: string): string | undefined => + (splitKeyed(value, KEYED_SCHEME_PREFIX) ?? splitKeyed(value, KEYED_DIGEST_PREFIX))?.kid; + +/** Encrypts with the keyring's primary key. */ +export const encryptWithKeyring = async (content: string, keyring: Keyring): Promise => { + try { + const header = `${KEYED_SCHEME_PREFIX}${keyring.primaryId}:`; + const salt = randomBytes(SALT_LENGTH); + const iv = randomBytes(IV_LENGTH); + const cipher = aesGcm(deriveKey(primaryKey(keyring), salt), iv, utf8ToBytes(header)); + const encrypted = cipher.encrypt(utf8ToBytes(content)); + return header + Buffer.from(concatBytes(salt, iv, encrypted)).toString('base64'); + } catch (error) { + throw new EncryptError(error); + } +}; + +/** + * Decrypts any machine ciphertext the keyring holds a key for. Values without a + * key id (m1 and legacy PBKDF2) are tried against each key, primary first; + * AES-GCM authentication guarantees only the right key succeeds. + */ +export const decryptWithKeyring = async (value: string, keyring: Keyring): Promise => { + const keyed = splitKeyed(value, KEYED_SCHEME_PREFIX); + if (keyed) { + const key = keyring.keys.get(keyed.kid); + if (key === undefined) { + // The key was removed before every value written with it expired. + throw new DecryptError(new Error('unknown machine key id')); + } + try { + const data = Buffer.from(keyed.payload, 'base64'); + const salt = data.subarray(0, SALT_LENGTH); + const iv = data.subarray(SALT_LENGTH, SALT_LENGTH + IV_LENGTH); + const ciphertext = data.subarray(SALT_LENGTH + IV_LENGTH); + const header = utf8ToBytes(`${KEYED_SCHEME_PREFIX}${keyed.kid}:`); + const decrypted = aesGcm(deriveKey(key, salt), iv, header).decrypt(ciphertext); + return new TextDecoder().decode(decrypted); + } catch (error) { + throw new DecryptError(error); + } + } + + if (!value.startsWith(SCHEME_PREFIX) && !BASE64.test(value)) { + throw new DecryptError(new Error('not a machine ciphertext')); + } + + let lastError: unknown; + for (const key of keyring.keys.values()) { + try { + return await decrypt(value, key); + } catch (error) { + lastError = error; + } + } + throw lastError instanceof DecryptError ? lastError : new DecryptError(lastError); +}; + +/** + * Whether a machine ciphertext was written with the keyring's primary key. + * Values without a key id are checked by decrypting with the primary key. + */ +export const isOnPrimaryKey = async (value: string, keyring: Keyring): Promise => { + const kid = keyIdOf(value); + if (kid !== undefined) { + return kid === keyring.primaryId; + } + try { + await decrypt(value, primaryKey(keyring)); + return true; + } catch { + return false; + } +}; + +const keyedDigest = (value: string, digestKey: Uint8Array, kid: string, context: string) => + `${KEYED_DIGEST_PREFIX}${kid}:${rawDigest(value, digestKey, context)}`; + +const primaryDigestKey = (keyring: Keyring): Uint8Array => { + const digestKey = keyring.digestKeys.get(keyring.primaryId); + if (digestKey === undefined) { + throw new Error('machine keyring is missing its primary key'); + } + return digestKey; +}; + +/** Deterministic keyed digest with the keyring's primary key. See {@link digest}. */ +export const digestWithKeyring = (value: string, keyring: Keyring, context: string): string => + keyedDigest(value, primaryDigestKey(keyring), keyring.primaryId, context); + +/** + * The digests of `value` that `stored` could legitimately equal, computed in + * `stored`'s own format. Only the stored format decides this, never the + * caller, so a raw stored digest can't be submitted in place of `value`. + * + * - hm2: the digest under the key it names, or none if that key was removed. + * - hm1: one digest per key, because hm1 doesn't record which key made it. + * - anything else predates digests and is compared as-is. + */ +export const digestCandidates = ( + stored: string, + value: string, + keyring: Keyring, + context: string, +): string[] => { + const keyed = splitKeyed(stored, KEYED_DIGEST_PREFIX); + if (keyed) { + const digestKey = keyring.digestKeys.get(keyed.kid); + return digestKey === undefined ? [] : [keyedDigest(value, digestKey, keyed.kid, context)]; + } + if (isDigest(stored)) { + return [...keyring.digestKeys.values()].map( + (digestKey) => DIGEST_PREFIX + rawDigest(value, digestKey, context), + ); + } + return [value]; +}; diff --git a/packages/server/.env-example b/packages/server/.env-example index b51fe2a..c8868c8 100644 --- a/packages/server/.env-example +++ b/packages/server/.env-example @@ -18,6 +18,7 @@ CORS_ORIGIN=localhost:5173 CORS_METHODS=GET,HEAD,POST,DELETE,OPTIONS CORS_HEADERS=Content-Type,X-Client ENCRYPTION_KEY=foobar +ENCRYPTION_PREVIOUS_KEYS= SERVICE_NAME= SERVICE_VERSION= SERVICE_DESCRIPTION= diff --git a/packages/server/package.json b/packages/server/package.json index 4d0e7d9..acdcee7 100644 --- a/packages/server/package.json +++ b/packages/server/package.json @@ -17,6 +17,7 @@ "dev": "nodemon src/index.ts", "build": "tsc && copyfiles -u 2 \"src/assets/**/*\" dist/assets", "start": "node dist/index.js", + "keys:rotate": "node dist/rotate-keys.js", "test": "vitest run", "test:watch": "vitest watch", "test:coverage": "vitest run --coverage", diff --git a/packages/server/src/app.test.ts b/packages/server/src/app.test.ts index 89926d2..3158db4 100644 --- a/packages/server/src/app.test.ts +++ b/packages/server/src/app.test.ts @@ -26,7 +26,12 @@ const initAppTest = async () => { vaultEntryDeleteTokenLength: config.vaultEntryDeleteTokenLength, }); const redis = new Redis(); - const vault = createRedisVault(redis, tokenGenerator, createNopWebhookSender(), 'foobar'); + const vault = createRedisVault( + redis, + tokenGenerator, + createNopWebhookSender(), + machine.createKeyring('foobar'), + ); const app = await initApp(config, { logger, vault, @@ -395,8 +400,8 @@ describe('app', () => { expect(parsedValue.wh.u.length).toBeGreaterThan(0); const [decryptedIps, decryptedWhU] = await Promise.all([ - machine.decrypt(parsedValue.ips, testContext.config.encryptionKey), - machine.decrypt(parsedValue.wh.u, testContext.config.encryptionKey), + machine.decryptWithKeyring(parsedValue.ips, machine.createKeyring('foobar')), + machine.decryptWithKeyring(parsedValue.wh.u, machine.createKeyring('foobar')), ]); expect(decryptedIps).toBe('192.168.1.1,10.0.0.0/24'); @@ -493,7 +498,7 @@ describe('app', () => { testContext.redis, createTokenGenerator({ vaultEntryIdentifierLength: 20, vaultEntryDeleteTokenLength: 20 }), { send: async (message) => void messages.push(message) }, - 'foobar', + machine.createKeyring('foobar'), ); const { id, dt } = await vault.set({ c: 'c', @@ -516,7 +521,7 @@ describe('app', () => { testContext.redis, createTokenGenerator({ vaultEntryIdentifierLength: 20, vaultEntryDeleteTokenLength: 20 }), createNopWebhookSender(), - 'a-different-key', + machine.createKeyring('a-different-key'), ); await expect(rotated.get(id, 'abc123', '127.0.0.1')).rejects.toThrow(); diff --git a/packages/server/src/config.ts b/packages/server/src/config.ts index 1eface1..8483ce7 100644 --- a/packages/server/src/config.ts +++ b/packages/server/src/config.ts @@ -89,7 +89,16 @@ const configSchema = z.object({ .describe( 'allowed CORS headers (comma-separated) https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Headers', ), - encryptionKey: z.string().describe('encryption key'), + encryptionKey: z + .string() + .min(1) + .describe('primary key for metadata encrypted or digested at rest; used for all new writes'), + encryptionPreviousKeys: z + .array(z.string().min(1)) + .default([]) + .describe( + 'retired keys still accepted for reading existing entries during a key rotation (comma-separated)', + ), otelEnabled: z.coerce.boolean().default(false).describe('enable OpenTelemetry tracing'), otelExporterOtlpEndpoint: z.string().describe('OpenTelemetry collector endpoint').optional(), otelExporterOtlpHeaders: z @@ -182,6 +191,9 @@ export const config = (() => { corsMethods: process.env.CORS_METHODS, corsHeaders: process.env.CORS_HEADERS, encryptionKey, + encryptionPreviousKeys: process.env.ENCRYPTION_PREVIOUS_KEYS?.split(',') + .map((key) => key.trim()) + .filter(Boolean), otelEnabled: process.env.OTEL_ENABLED, otelExporterOtlpEndpoint: process.env.OTEL_EXPORTER_OTLP_ENDPOINT, otelExporterOtlpHeaders: process.env.OTEL_EXPORTER_OTLP_HEADERS diff --git a/packages/server/src/index.ts b/packages/server/src/index.ts index 441f626..da22e32 100644 --- a/packages/server/src/index.ts +++ b/packages/server/src/index.ts @@ -11,10 +11,14 @@ import { createHTTPWebhookSender, type WebhookSender, } from './webhook.js'; -import { createFetchRetryClient } from '@crypt.fyi/core'; +import { createFetchRetryClient, machine } from '@crypt.fyi/core'; const main = async () => { const logger = await initLogging(config); + const keyring = machine.createKeyring(config.encryptionKey, config.encryptionPreviousKeys); + // Only the count is logged: key ids are derived from the keys, so logging + // them would let anyone with log access check guesses of a weak key offline. + logger.info({ previousKeyCount: keyring.keys.size - 1 }, 'Loaded encryption keyring'); const redis = new Redis(config.redisUrl, { family: 0 }); await redis.ping(); const tokenGenerator = createTokenGenerator({ @@ -32,7 +36,7 @@ const main = async () => { createBullMQWebhookSender({ logger, redis: bullmqRedis, - encryptionKey: config.encryptionKey, + keyring, requestTimeoutMs: config.webhookRequestTimeoutMs, maxAttempts: config.webhookMaxAttempts, backoffType: config.webhookBackoffType, @@ -59,7 +63,7 @@ const main = async () => { throw new Error(`Invalid webhook sender: ${config.webhookSender}`); } - const vault = createRedisVault(redis, tokenGenerator, webhookSender, config.encryptionKey); + const vault = createRedisVault(redis, tokenGenerator, webhookSender, keyring); const app = await initApp(config, { logger, diff --git a/packages/server/src/rotate-keys.ts b/packages/server/src/rotate-keys.ts new file mode 100644 index 0000000..72baf45 --- /dev/null +++ b/packages/server/src/rotate-keys.ts @@ -0,0 +1,72 @@ +// Re-encrypts vault metadata with the primary ENCRYPTION_KEY and reports when +// the keys in ENCRYPTION_PREVIOUS_KEYS can be removed. See the "Rotating +// ENCRYPTION_KEY" section of the server README for the full procedure. +// +// node dist/rotate-keys.js [--dry-run] +// +// Run it with the same environment as the server, after every server node has +// been restarted with the new primary key. It is safe to run repeatedly and +// while the server is serving traffic. +import { Redis } from 'ioredis'; +import { Queue } from 'bullmq'; +import { machine } from '@crypt.fyi/core'; +import { config } from './config.js'; +import { rotateVaultKeys, countWebhookJobsOnPreviousKeys } from './vault/rotate.js'; +import { WEBHOOK_QUEUE_NAME } from './webhook.js'; + +const JOB_PAGE_SIZE = 500; + +const main = async () => { + const dryRun = process.argv.includes('--dry-run'); + const keyring = machine.createKeyring(config.encryptionKey, config.encryptionPreviousKeys); + const redis = new Redis(config.redisUrl, { family: 0 }); + + try { + const report = await rotateVaultKeys({ redis, keyring, dryRun }); + + let webhookJobsOnPreviousKeys = 0; + if (config.webhookSender === 'bullmq') { + const queue = new Queue(WEBHOOK_QUEUE_NAME, { connection: redis.options }); + try { + // Paged per state so a large backlog never has to fit in memory at once. + // Jobs moving between states mid-count can be missed or counted twice; + // rerunning before removing a key covers that. + for (const state of ['waiting', 'delayed', 'active', 'paused', 'prioritized'] as const) { + for (let start = 0; ; start += JOB_PAGE_SIZE) { + const jobs = await queue.getJobs([state], start, start + JOB_PAGE_SIZE - 1); + webhookJobsOnPreviousKeys += await countWebhookJobsOnPreviousKeys( + jobs.flatMap((job) => (typeof job?.data === 'string' ? [job.data] : [])), + keyring, + ); + if (jobs.length < JOB_PAGE_SIZE) { + break; + } + } + } + } finally { + await queue.close(); + } + } + + const safeToRemovePreviousKeys = + report.unreadable === 0 && + report.conflicts === 0 && + report.dependentOnPreviousKeys === 0 && + webhookJobsOnPreviousKeys === 0; + + // Only counts are printed, never keys, key ids or entry contents. + console.log( + JSON.stringify({ ...report, webhookJobsOnPreviousKeys, safeToRemovePreviousKeys }, null, 2), + ); + if (report.unreadable > 0) { + process.exitCode = 1; + } + } finally { + await redis.quit(); + } +}; + +main().catch((error) => { + console.error('key rotation failed:', error instanceof Error ? error.message : error); + process.exitCode = 1; +}); diff --git a/packages/server/src/vault/redis.ts b/packages/server/src/vault/redis.ts index 9ff6084..c878acf 100644 --- a/packages/server/src/vault/redis.ts +++ b/packages/server/src/vault/redis.ts @@ -11,14 +11,14 @@ import { import { isIpAllowed } from './ips.js'; import type { WebhookSender } from '../webhook.js'; -const parseResult = async (result: string, encryptionKey: string) => { +const parseResult = async (result: string, keyring: machine.Keyring) => { const jsonParsed = JSON.parse(result); if ('wh' in jsonParsed && jsonParsed.wh.u) { - jsonParsed.wh.u = await machine.decrypt(jsonParsed.wh.u, encryptionKey); + jsonParsed.wh.u = await machine.decryptWithKeyring(jsonParsed.wh.u, keyring); } if (jsonParsed.ips) { try { - const ips = await machine.decrypt(jsonParsed.ips, encryptionKey); + const ips = await machine.decryptWithKeyring(jsonParsed.ips, keyring); jsonParsed.ips = ips; } catch { // ignore - this maintains backward compatibility with existing vault entries that don't have ips encrypted @@ -27,35 +27,36 @@ const parseResult = async (result: string, encryptionKey: string) => { // Entries written before delete tokens were encrypted hold them in plaintext // and stay readable until they expire (at most the max TTL). if (typeof jsonParsed.dt === 'string' && machine.isEncrypted(jsonParsed.dt)) { - jsonParsed.dt = await machine.decrypt(jsonParsed.dt, encryptionKey); + jsonParsed.dt = await machine.decryptWithKeyring(jsonParsed.dt, keyring); } return vaultValueSchema.parse(jsonParsed); }; +export const VAULT_KEY_PREFIX = 'vault:'; +export const getVaultKey = (id: string) => `${VAULT_KEY_PREFIX}${id}`; +// Binding the digest to the entry id stops a digest being moved between +// entries and hides whether two entries share a verification hash. +export const vaultHashContext = (id: string) => `vault-h:${id}`; + export const createRedisVault = ( redis: Redis, tokenGenerator: TokenGenerator, webhookSender: WebhookSender, - encryptionKey: string, + keyring: machine.Keyring, ): Vault => { - const getKey = (id: string) => `vault:${id}`; - // Binding the digest to the entry id stops a digest being moved between - // entries and hides whether two entries share a verification hash. - const digestH = (id: string, h: string) => machine.digest(h, encryptionKey, `vault-h:${id}`); - return { async set(value) { const { c, h, b, ttl, ips, rc, wh, fc, m } = value; const { id, dt } = await tokenGenerator.generate(); - const key = getKey(id); + const key = getVaultKey(id); // The delete token is encrypted rather than digested because webhook // payloads must carry the original value back to the creator. const [encryptedIps, encryptedWhU, encryptedDt] = await Promise.all([ - ips ? machine.encrypt(ips, encryptionKey) : undefined, - wh ? machine.encrypt(wh.u, encryptionKey) : undefined, - machine.encrypt(dt, encryptionKey), + ips ? machine.encryptWithKeyring(ips, keyring) : undefined, + wh ? machine.encryptWithKeyring(wh.u, keyring) : undefined, + machine.encryptWithKeyring(dt, keyring), ]); // Single atomic SET with NX + PX: only creates the key if absent AND sets @@ -65,7 +66,7 @@ export const createRedisVault = ( key, JSON.stringify({ c, - h: digestH(id, h), + h: machine.digestWithKeyring(h, keyring, vaultHashContext(id)), b, fc, dt: encryptedDt, @@ -87,7 +88,7 @@ export const createRedisVault = ( return { id, dt }; }, async get(id, h, ip) { - const key = getKey(id); + const key = getVaultKey(id); // TODO: try to not pull the full redis entry into memory until the ip address is confirmed to be allowed const result = await redis.get(key); @@ -95,17 +96,7 @@ export const createRedisVault = ( return undefined; } - const { - h: storedH, - ips, - c, - b, - ttl, - cd, - dt, - wh, - m, - } = await parseResult(result, encryptionKey); + const { h: storedH, ips, c, b, ttl, cd, dt, wh, m } = await parseResult(result, keyring); if (!isIpAllowed(ip, ips)) { if (wh?.fip) { void webhookSender.send({ @@ -123,8 +114,15 @@ export const createRedisVault = ( // The stored format decides the comparison, never the request. Accepting // either form would let someone holding a Redis dump submit the stored // digest as `h`. Entries without a digest predate this scheme and expire - // within the max TTL. - const expectedH = machine.isDigest(storedH) ? digestH(id, h) : h; + // within the max TTL. During a key rotation an hm1 digest may have been + // made with any key in the ring, so every candidate is checked. + const expectedHs = machine.digestCandidates(storedH, h, keyring, vaultHashContext(id)); + if (expectedHs.length === 0) { + // The entry's key was removed from the keyring. Failing here keeps a + // server misconfiguration from counting as a wrong password and + // burning the entry through its failed-attempt limit. + throw new Error('vault entry digest was made with an encryption key that is not loaded'); + } const outcome = (await redis.eval( // https://redis.io/docs/latest/develop/interact/programmability/lua-api/ @@ -136,7 +134,14 @@ export const createRedisVault = ( end local data = cjson.decode(value) - if data.h ~= ARGV[1] then + local matched = false + for i = 1, #ARGV do + if data.h == ARGV[i] then + matched = true + break + end + end + if not matched then -- Handle failed attempts if data.fc then data.fc = (data.fc or 0) - 1 @@ -194,7 +199,7 @@ export const createRedisVault = ( `, 1, key, - expectedH, + ...expectedHs, )) as string | null; if (!outcome) { return undefined; @@ -265,13 +270,13 @@ export const createRedisVault = ( }; }, async del(id, dt) { - const key = getKey(id); + const key = getVaultKey(id); const result = await redis.get(key); if (!result) { return false; } - const { dt: actualDt } = await parseResult(result, encryptionKey); + const { dt: actualDt } = await parseResult(result, keyring); const dtBuf = Buffer.from(dt); const actualDtBuf = Buffer.from(actualDt); if (dtBuf.length !== actualDtBuf.length || !timingSafeEqual(dtBuf, actualDtBuf)) { @@ -283,7 +288,7 @@ export const createRedisVault = ( return delResult === 1; }, async exists(id) { - const key = getKey(id); + const key = getVaultKey(id); const result = await redis.exists(key); return result === 1; }, diff --git a/packages/server/src/vault/rotate.test.ts b/packages/server/src/vault/rotate.test.ts new file mode 100644 index 0000000..6b358e9 --- /dev/null +++ b/packages/server/src/vault/rotate.test.ts @@ -0,0 +1,212 @@ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { Redis } from 'ioredis'; +import { machine, type Vault } from '@crypt.fyi/core'; +import { createRedisVault, getVaultKey } from './redis.js'; +import { createTokenGenerator } from './tokens.js'; +import { createNopWebhookSender } from '../webhook.js'; +import { rotateVaultKeys, countWebhookJobsOnPreviousKeys } from './rotate.js'; + +const OLD_KEY = 'rotation-test-old-key-0123456789abcdef'; +const NEW_KEY = 'rotation-test-new-key-fedcba9876543210'; +const m = { encryption: { algorithm: 'aes-256-gcm' as const } }; +// A separate logical database keeps SCAN from seeing other tests' entries. +const TEST_DB = 9; + +describe('encryption key rotation', () => { + let redis: Redis; + + const before = machine.createKeyring(OLD_KEY); + const rotating = machine.createKeyring(NEW_KEY, [OLD_KEY]); + const after = machine.createKeyring(NEW_KEY); + + const vaultWith = (keyring: machine.Keyring): Vault => + createRedisVault( + redis, + createTokenGenerator({ vaultEntryIdentifierLength: 20, vaultEntryDeleteTokenLength: 20 }), + createNopWebhookSender(), + keyring, + ); + const stored = async (id: string) => + JSON.parse((await redis.get(getVaultKey(id)))!) as Record & { + wh: { u: string }; + }; + const create = (vault: Vault, fc?: number) => + vault.set({ + c: 'ciphertext', + h: 'abc123', + b: false, + ttl: 60_000, + m, + fc, + ips: '127.0.0.1', + wh: { u: 'https://example.com/hook', r: false, fpk: false, fip: false, b: false }, + }); + + beforeEach(async () => { + redis = new Redis({ db: TEST_DB }); + await redis.flushdb(); + }); + + afterEach(async () => { + await redis.flushdb(); + await redis.quit(); + }); + + it('keeps entries written with the previous key readable and deletable', async () => { + const a = await create(vaultWith(before)); + const b = await create(vaultWith(before)); + const vault = vaultWith(rotating); + + await expect(vault.get(a.id, 'abc123', '127.0.0.1')).resolves.toMatchObject({ + c: 'ciphertext', + }); + await expect(vault.del(b.id, b.dt)).resolves.toBe(true); + }); + + it('still rejects a wrong hash on an entry written with the previous key', async () => { + const { id } = await create(vaultWith(before), 2); + await expect(vaultWith(rotating).get(id, 'wrong', '127.0.0.1')).rejects.toThrow(); + expect((await stored(id)).fc).toBe(1); + }); + + it('writes new entries with the primary key', async () => { + const { id } = await create(vaultWith(rotating)); + const entry = await stored(id); + for (const value of [entry.h, entry.dt, entry.ips, entry.wh.u]) { + expect(machine.keyIdOf(value as string)).toBe(rotating.primaryId); + } + await expect(vaultWith(after).get(id, 'abc123', '127.0.0.1')).resolves.toBeDefined(); + }); + + it('reads hm1/m1 entries written before key ids existed', async () => { + const id = 'pre-keyring-entry'; + await redis.set( + getVaultKey(id), + JSON.stringify({ + c: 'ciphertext', + h: machine.digest('abc123', OLD_KEY, `vault-h:${id}`), + b: false, + dt: await machine.encrypt('the-delete-token', OLD_KEY), + ttl: 60_000, + cd: Date.now(), + m, + }), + 'PX', + 60_000, + ); + const vault = vaultWith(rotating); + + await expect(vault.get(id, 'wrong', '127.0.0.1')).rejects.toThrow(); + await expect(vault.get(id, 'abc123', '127.0.0.1')).resolves.toBeDefined(); + await expect(vault.del(id, 'the-delete-token')).resolves.toBe(true); + }); + + it('rewraps encrypted fields, keeps the TTL and reports the digest dependency', async () => { + const { id, dt } = await create(vaultWith(before), 3); + const ttlBefore = await redis.pttl(getVaultKey(id)); + const now = Date.now(); + + const report = await rotateVaultKeys({ redis, keyring: rotating, now }); + + expect(report).toMatchObject({ + previousKeyCount: 1, + scanned: 1, + rewrapped: 1, + unreadable: 0, + conflicts: 0, + dependentOnPreviousKeys: 1, + dependentWithoutTtl: 0, + }); + const neededUntil = Date.parse(report.previousKeysNeededUntil!); + expect(neededUntil).toBeGreaterThan(now); + expect(neededUntil).toBeLessThanOrEqual(now + ttlBefore); + + const entry = await stored(id); + expect(machine.keyIdOf(entry.dt as string)).toBe(rotating.primaryId); + expect(machine.keyIdOf(entry.ips as string)).toBe(rotating.primaryId); + expect(machine.keyIdOf(entry.wh.u)).toBe(rotating.primaryId); + // The digest can't be recomputed without the hash, so it keeps the old key. + expect(machine.keyIdOf(entry.h as string)).toBe(before.primaryId); + expect(entry.fc).toBe(3); + expect(await redis.pttl(getVaultKey(id))).toBeGreaterThan(ttlBefore - 5_000); + + const vault = vaultWith(rotating); + await expect(vault.get(id, 'abc123', '127.0.0.1')).resolves.toBeDefined(); + await expect(vault.del(id, dt)).resolves.toBe(true); + }); + + it('changes nothing in a dry run', async () => { + const { id } = await create(vaultWith(before)); + const raw = await redis.get(getVaultKey(id)); + + const report = await rotateVaultKeys({ redis, keyring: rotating, dryRun: true }); + + expect(report).toMatchObject({ dryRun: true, rewrapped: 1, dependentOnPreviousKeys: 1 }); + expect(await redis.get(getVaultKey(id))).toBe(raw); + }); + + it('reports nothing depending on previous keys once entries use the primary key', async () => { + await create(vaultWith(rotating)); + const report = await rotateVaultKeys({ redis, keyring: rotating }); + expect(report).toMatchObject({ + scanned: 1, + rewrapped: 0, + dependentOnPreviousKeys: 0, + previousKeysNeededUntil: null, + }); + }); + + it('does not report hm1/m1 entries as dependent when there are no previous keys', async () => { + const id = 'pre-keyring-primary'; + await redis.set( + getVaultKey(id), + JSON.stringify({ + c: 'ciphertext', + h: machine.digest('abc123', NEW_KEY, `vault-h:${id}`), + b: false, + dt: await machine.encrypt('the-delete-token', NEW_KEY), + ttl: 60_000, + cd: Date.now(), + m, + }), + 'PX', + 60_000, + ); + + const dry = await rotateVaultKeys({ redis, keyring: after, dryRun: true }); + expect(dry).toMatchObject({ previousKeyCount: 0, rewrapped: 1, dependentOnPreviousKeys: 0 }); + + const report = await rotateVaultKeys({ redis, keyring: after }); + expect(report).toMatchObject({ rewrapped: 1, dependentOnPreviousKeys: 0 }); + expect(machine.keyIdOf((await stored(id)).dt as string)).toBe(after.primaryId); + }); + + it('counts entries whose key was removed too early as unreadable and leaves them alone', async () => { + const { id } = await create(vaultWith(before)); + const raw = await redis.get(getVaultKey(id)); + + const report = await rotateVaultKeys({ redis, keyring: after }); + + expect(report.unreadable).toBe(1); + expect(await redis.get(getVaultKey(id))).toBe(raw); + }); + + it('fails a read without burning the entry when its key is not loaded', async () => { + const { id } = await create(vaultWith(before), 1); + await expect(vaultWith(after).get(id, 'abc123', '127.0.0.1')).rejects.toThrow(); + expect((await stored(id)).fc).toBe(1); + }); + + it('counts webhook jobs still encrypted with a previous key', async () => { + const jobs = [ + await machine.encryptWithKeyring('{}', before), + await machine.encryptWithKeyring('{}', rotating), + await machine.encrypt('{}', OLD_KEY), + ]; + expect(await countWebhookJobsOnPreviousKeys(jobs, rotating)).toBe(2); + // m1 jobs written with the primary key don't depend on a previous key. + expect( + await countWebhookJobsOnPreviousKeys([await machine.encrypt('{}', NEW_KEY)], rotating), + ).toBe(0); + }); +}); diff --git a/packages/server/src/vault/rotate.ts b/packages/server/src/vault/rotate.ts new file mode 100644 index 0000000..91e79b9 --- /dev/null +++ b/packages/server/src/vault/rotate.ts @@ -0,0 +1,251 @@ +import type { Redis } from 'ioredis'; +import { machine } from '@crypt.fyi/core'; +import { VAULT_KEY_PREFIX } from './redis.js'; + +// Re-encrypts vault metadata with the keyring's primary key and reports how +// long the previous keys are still needed. +// +// Encrypted fields (IP restrictions, webhook URL, delete token) are rewrapped: +// decrypted with whichever key wrote them and encrypted with the primary key. +// The verification-hash digest cannot be rewrapped, because the server never +// learns the hash itself, only a keyed digest of it. Entries whose digest uses +// a previous key keep depending on that key until they expire, which is +// bounded by the max TTL. The report gives the time after which no entry can +// depend on a previous key any more. + +type Entry = Record & { wh?: Record }; + +interface EncryptedField { + name: 'ips' | 'dt' | 'wh.u'; + get: (entry: Entry) => unknown; + set: (entry: Entry, value: string) => void; +} + +const ENCRYPTED_FIELDS: readonly EncryptedField[] = [ + { name: 'ips', get: (e) => e.ips, set: (e, v) => void (e.ips = v) }, + { name: 'dt', get: (e) => e.dt, set: (e, v) => void (e.dt = v) }, + { name: 'wh.u', get: (e) => e.wh?.u, set: (e, v) => void (e.wh!.u = v) }, +]; + +// Writes the new value only if the entry is unchanged since it was read, and +// keeps its remaining TTL. A concurrent read can decrement rc/fc or burn the +// entry; the CAS makes sure rotation never undoes that. +const COMPARE_AND_SET = ` +local current = redis.call('get', KEYS[1]) +if current ~= ARGV[1] then + return 0 +end +redis.call('set', KEYS[1], ARGV[2], 'KEEPTTL') +return 1 +`; + +const MAX_ATTEMPTS = 3; + +// Key ids are derived from the keys, so the report shows only how many keys +// are loaded: an id would let anyone reading the output check guesses of a +// weak key offline. +export interface RotationReport { + previousKeyCount: number; + dryRun: boolean; + /** Vault entries examined. */ + scanned: number; + /** Entries whose encrypted fields were (or, in a dry run, would be) re-encrypted. */ + rewrapped: number; + /** Entries that changed or expired while being rotated on every attempt; rerun to pick them up. */ + conflicts: number; + /** Entries holding a value no loaded key can decrypt. Their data is already lost. */ + unreadable: number; + /** Entries that still need a previous key after this run (digest or unrewrapped field). */ + dependentOnPreviousKeys: number; + /** + * When the last entry depending on a previous key expires. Previous keys can + * be removed after this time, once the webhook queue is also clear. Null when + * nothing depends on them. + */ + previousKeysNeededUntil: string | null; + /** Dependent entries without a TTL; previous keys stay needed until they are deleted. */ + dependentWithoutTtl: number; +} + +type EntryOutcome = + | { kind: 'gone' } + | { kind: 'conflict' } + | { kind: 'unreadable'; pttl: number } + | { kind: 'ok'; rewrapped: boolean; dependent: boolean; pttl: number }; + +const rotateEntry = async ( + redis: Redis, + key: string, + keyring: machine.Keyring, + dryRun: boolean, +): Promise => { + for (let attempt = 0; attempt < MAX_ATTEMPTS; attempt++) { + const [[, raw], [, pttl]] = (await redis.multi().get(key).pttl(key).exec()) as [ + [unknown, string | null], + [unknown, number], + ]; + if (raw === null) { + return { kind: 'gone' }; + } + + let entry: Entry; + try { + entry = JSON.parse(raw); + } catch { + return { kind: 'unreadable', pttl }; + } + + let changed = false; + let dependent = false; + let unreadable = false; + for (const field of ENCRYPTED_FIELDS) { + const value = field.get(entry); + if (typeof value !== 'string' || machine.keyIdOf(value) === keyring.primaryId) { + continue; + } + // Delete tokens written before they were encrypted are plaintext and + // carry no machine prefix; the server still reads them as-is. Unprefixed + // ips or webhook URLs may be legacy PBKDF2 ciphertext, so those are tried. + if (field.name === 'dt' && !machine.isEncrypted(value)) { + continue; + } + try { + const plaintext = await machine.decryptWithKeyring(value, keyring); + field.set(entry, await machine.encryptWithKeyring(plaintext, keyring)); + changed = true; + // A dry run rewrites nothing, so a field still on a previous key keeps + // depending on it. m1 values on the primary key are rewrapped only to + // gain a key id and depend on nothing else. + if (dryRun && !(await machine.isOnPrimaryKey(value, keyring))) { + dependent = true; + } + } catch { + // IP restrictions written before they were encrypted are plaintext, + // which the server also tolerates. + if (field.name === 'ips' && !machine.isEncrypted(value)) { + continue; + } + unreadable = true; + } + } + + // An hm1 digest doesn't record its key, and checking it needs the original + // hash, so it can only depend on a previous key if the keyring has one. + const h = entry.h; + if ( + typeof h === 'string' && + machine.isDigest(h) && + machine.keyIdOf(h) !== keyring.primaryId && + machine.hasPreviousKeys(keyring) + ) { + dependent = true; + } + + if (unreadable) { + return { kind: 'unreadable', pttl }; + } + if (!changed || dryRun) { + return { kind: 'ok', rewrapped: changed, dependent, pttl }; + } + + const written = await redis.eval(COMPARE_AND_SET, 1, key, raw, JSON.stringify(entry)); + if (written === 1) { + return { kind: 'ok', rewrapped: true, dependent, pttl }; + } + } + return { kind: 'conflict' }; +}; + +export const rotateVaultKeys = async ({ + redis, + keyring, + dryRun = false, + scanCount = 500, + now = Date.now(), +}: { + redis: Redis; + keyring: machine.Keyring; + dryRun?: boolean; + scanCount?: number; + now?: number; +}): Promise => { + const report: RotationReport = { + previousKeyCount: keyring.keys.size - 1, + dryRun, + scanned: 0, + rewrapped: 0, + conflicts: 0, + unreadable: 0, + dependentOnPreviousKeys: 0, + previousKeysNeededUntil: null, + dependentWithoutTtl: 0, + }; + let maxDependentPttl = -1; + // SCAN may return a key more than once; count each entry once. + const seen = new Set(); + + let cursor = '0'; + do { + const [next, keys] = await redis.scan( + cursor, + 'MATCH', + `${VAULT_KEY_PREFIX}*`, + 'COUNT', + scanCount, + ); + cursor = next; + for (const key of keys) { + if (seen.has(key)) { + continue; + } + seen.add(key); + const outcome = await rotateEntry(redis, key, keyring, dryRun); + if (outcome.kind === 'gone') { + continue; + } + report.scanned++; + if (outcome.kind === 'conflict') { + report.conflicts++; + continue; + } + if (outcome.kind === 'unreadable') { + report.unreadable++; + continue; + } + if (outcome.rewrapped) { + report.rewrapped++; + } + if (outcome.dependent) { + report.dependentOnPreviousKeys++; + if (outcome.pttl < 0) { + report.dependentWithoutTtl++; + } else { + maxDependentPttl = Math.max(maxDependentPttl, outcome.pttl); + } + } + } + } while (cursor !== '0'); + + if (maxDependentPttl >= 0) { + report.previousKeysNeededUntil = new Date(now + maxDependentPttl).toISOString(); + } + return report; +}; + +/** + * Counts queued webhook jobs that are still encrypted with a previous key. + * Jobs live for at most their retry window, so these are left to drain rather + * than rewritten inside BullMQ's own data structures. + */ +export const countWebhookJobsOnPreviousKeys = async ( + jobData: readonly string[], + keyring: machine.Keyring, +): Promise => { + let count = 0; + for (const data of jobData) { + if (!(await machine.isOnPrimaryKey(data, keyring))) { + count++; + } + } + return count; +}; diff --git a/packages/server/src/webhook.ts b/packages/server/src/webhook.ts index d134fbb..8402154 100644 --- a/packages/server/src/webhook.ts +++ b/packages/server/src/webhook.ts @@ -17,6 +17,9 @@ const messageSchema = z.object({ export type Message = z.infer; +export const WEBHOOK_QUEUE_NAME = 'webhooks'; +const JOB_NAME = 'webhook'; + export interface WebhookSender { send(message: Message): Promise; } @@ -36,7 +39,7 @@ interface BaseWebhookSenderOptions { type BullMQWebhookSenderOptions = BaseWebhookSenderOptions & { redis: Redis; - encryptionKey: string; + keyring: machine.Keyring; maxAttempts: number; backoffType: 'fixed' | 'exponential'; backoffDelayMs: number; @@ -50,7 +53,7 @@ type BullMQWebhookSenderOptions = BaseWebhookSenderOptions & { export const createBullMQWebhookSender = ({ logger, redis, - encryptionKey, + keyring, requestTimeoutMs, maxAttempts, backoffType, @@ -63,9 +66,7 @@ export const createBullMQWebhookSender = ({ fetchFn = fetch, shouldRetry = isRetryableFetchError, }: BullMQWebhookSenderOptions): { webhookSender: WebhookSender; cleanup: () => Promise } => { - const QUEUE_NAME = 'webhooks'; - const JOB_NAME = 'webhook'; - const queue = new Queue(QUEUE_NAME, { + const queue = new Queue(WEBHOOK_QUEUE_NAME, { connection: redis.options, streams: { events: { @@ -83,14 +84,14 @@ export const createBullMQWebhookSender = ({ }, }); const worker = new Worker( - QUEUE_NAME, + WEBHOOK_QUEUE_NAME, async (job) => { if (job.name !== JOB_NAME) { logger.warn({ job: job.name }, 'skipping unexpected job'); return; } - const dataString = await machine.decrypt(job.data, encryptionKey); + const dataString = await machine.decryptWithKeyring(job.data, keyring); const message = messageSchema.parse(JSON.parse(dataString)); const ac = new AbortController(); @@ -142,15 +143,25 @@ export const createBullMQWebhookSender = ({ worker.on('failed', async (job, error) => { if (job && job.attemptsMade >= maxAttempts) { - const dataString = await machine.decrypt(job.data, encryptionKey); - const message = messageSchema.parse(JSON.parse(dataString)); + // This listener is async, so a throw here would be an unhandled + // rejection. Decrypting fails when the job's key was removed from the + // keyring before the queue drained; log what's known without the payload. + let message: Message | undefined; + try { + message = messageSchema.parse( + JSON.parse(await machine.decryptWithKeyring(job.data, keyring)), + ); + } catch { + message = undefined; + } logger.info( { jobId: job.id, - id: message.id, - event: message.event, + id: message?.id, + event: message?.event, error, attempts: job.attemptsMade, + ...(message ? {} : { payloadUnreadable: true }), }, 'Webhook failed permanently after all retry attempts', ); @@ -160,7 +171,10 @@ export const createBullMQWebhookSender = ({ return { webhookSender: { send: async (message) => { - await queue.add(JOB_NAME, await machine.encrypt(JSON.stringify(message), encryptionKey)); + await queue.add( + JOB_NAME, + await machine.encryptWithKeyring(JSON.stringify(message), keyring), + ); }, }, cleanup: async () => {