diff --git a/.changeset/redis-aof-privacy-copy.md b/.changeset/redis-aof-privacy-copy.md new file mode 100644 index 0000000..3cd3064 --- /dev/null +++ b/.changeset/redis-aof-privacy-copy.md @@ -0,0 +1,5 @@ +--- +'@crypt.fyi/core': patch +--- + +Correct the security copy now that Redis persists data with an AOF: secrets reach disk only in encrypted form, and deleted entries can remain until the log is next compacted. diff --git a/README.md b/README.md index 067f461..9652ef7 100644 --- a/README.md +++ b/README.md @@ -96,6 +96,45 @@ It runs with `NODE_ENV=production` and requires `REDIS_URL` and `ENCRYPTION_KEY` The web client image is not published to GHCR yet; build it with `docker compose` as above. +#### Redis persistence + +The compose file stores Redis data in the `redis_data` volume using an append-only file (AOF) that is synced to disk before each write is acknowledged, so secrets and queued webhooks survive restarts and crashes. A secret that was burned stays burned, even after a crash. RDB snapshots stay off because restoring an older snapshot could bring a burned secret back. See the [specification](./SPECIFICATION.md) for the requirements. + +What this means for operators: + +- **The disk holds sensitive data.** The volume contains ciphertext, keyed digests of verification hashes and encrypted delete tokens, but never plaintext or decryption keys. The digests and tokens rely on `ENCRYPTION_KEY`, so keep that key off this disk where you can. Once a secret is deleted, the server never returns it, even if its ciphertext remains in the AOF until Redis next compacts it. Encrypt the disk. +- **Don't back up the volume.** Restoring a backup can bring back burned secrets. +- **Writes are slower** than without persistence, because each one waits for the disk. +- **`docker compose down -v` deletes the volume** and every unexpired secret with it. + +To run without persistence, set `--appendonly no` and remove the volume. A Redis restart then drops all unexpired secrets and queued webhooks. If you run Redis any other way (the GHCR image with your own Redis, a managed service, a hosting platform), apply these settings yourself: by default Redis takes snapshots and doesn't use AOF. + +##### Upgrading from RDB snapshots + +Deployments from before this change store data in a `dump.rdb` snapshot, which Redis ignores once AOF is on. **Upgrading without these steps loses all unexpired secrets**, and the old snapshot stays on disk. Before pulling the new compose file, turn on AOF in the running Redis: + +```bash +docker compose exec redis redis-cli config set appendonly yes +``` + +Wait until this shows `aof_rewrite_in_progress:0` and `aof_last_bgrewrite_status:ok`: + +```bash +docker compose exec redis redis-cli info persistence +``` + +Then pull the new compose file and restart: + +```bash +docker compose up -d +``` + +Once you've confirmed existing secrets still open, delete the old snapshot: + +```bash +docker compose exec redis rm /data/dump.rdb +``` + ### Railway [![Deploy on Railway](https://railway.com/button.svg)](https://railway.com/deploy/Pmkrsc?referralCode=ToZEjF&utm_medium=integration&utm_source=template&utm_campaign=generic) diff --git a/SPECIFICATION.md b/SPECIFICATION.md index dbd1b5f..f054cab 100644 --- a/SPECIFICATION.md +++ b/SPECIFICATION.md @@ -63,7 +63,7 @@ Abstract o Web Client (Browser-based interface) o Web Server (Static file server) o API Server - o Data Store (Redis for ephemeral storage) + o Data Store (currently Redis, holding entries until they are deleted) ### 2.2. Component Interaction Flow @@ -272,12 +272,31 @@ Abstract ### 6.2. Data Security - Data MUST have automatic expiration (TTL: 1 second to 7 days) - - Burn after reading MUST use atomic Redis operations + - Burn after reading MUST use atomic data store operations - Read count limits MUST NOT exceed 10 reads - Failed attempt limits MUST be between 1-10 attempts before burning - - Implementation MUST NOT use persistent storage - ephemeral Redis - storage only - - Data MUST be securely deleted upon expiration or burning + - Implementations MAY persist data to disk or keep it in memory only. A + memory-only data store loses all unexpired entries when it restarts + - Persisted data MUST NOT include plaintext secrets, encryption keys, or + passwords + - An entry is deleted when it expires, is burned, exhausts its failed + attempts, or is removed with its delete token. Deletion carries two + separate guarantees: + - Deleted entries MUST NOT be released. Deciding to release an entry + and deleting it MUST be a single atomic operation, so that concurrent + requests cannot race, and once an entry is deleted the server MUST NOT + release it through the API. This MUST also hold after a data store + restart or crash: an acknowledged deletion MUST be durable, and + recovery MUST NOT restore a deleted entry. Backups of persisted data + SHOULD NOT be taken, because restoring one can bring back deleted + entries + - Deleted entries MUST be purged. The data store MUST release any + snapshot, memory, or other reference to a deleted entry as soon as is + possible at scale. Data that remains on disk but cannot be released + through the API is acceptable until the data store is able to purge + it, which SHOULD happen as soon as it can. Implementations SHOULD + encrypt the underlying disk, because purging does not guarantee that + the physical blocks are erased - The verification hash MUST NOT be stored in plaintext; servers MUST store a keyed digest (HMAC-SHA-256) bound to the vault ID, so a datastore dump alone cannot be replayed to read or burn an entry @@ -321,9 +340,16 @@ Abstract ### 7.1. Storage Requirements -#### 7.1.1. Redis Configuration +#### 7.1.1. Data Store Configuration - - Implementations MUST use ephemeral storage (Redis recommended) + - Implementations MUST meet the requirements in Section 6.2 through a data + store adapter. Redis is the current adapter; others, such as PostgreSQL + or SQLite, MAY be added if they meet the same requirements + - The Redis adapter, when persisting data, MUST use `appendonly yes` and + `appendfsync always` so that a deletion is on disk before it is + acknowledged. RDB save points MUST be disabled, because restoring an + older snapshot can bring back a deleted entry. Deleted data is purged + when Redis rewrites the append-only file - TTL per entry MUST be configurable (1 second to 7 days) - Automatic expiration and cleanup MUST be implemented - Atomic operations MUST be used for critical functions: @@ -335,8 +361,9 @@ Abstract - Implementations MUST support concurrent access - Data consistency MUST be maintained using atomic transactions - - Atomic operations SHOULD use Lua scripts to prevent race conditions - - Concurrent read/delete operations MUST be handled properly + - Atomic operations SHOULD use the data store's native mechanism to prevent + race conditions (Lua scripts in Redis) + - Concurrent read/delete operations MUST meet the guarantees in Section 6.2 #### 7.1.3. Performance Requirements diff --git a/docker-compose.yaml b/docker-compose.yaml index b29689c..4faf115 100644 --- a/docker-compose.yaml +++ b/docker-compose.yaml @@ -5,6 +5,12 @@ services: # TTL'd and must never be silently dropped, so use noeviction: once the cap # is hit, new writes are rejected (surfacing pressure) rather than evicting # live secrets. Tune REDIS_MAXMEMORY to the host. + # + # Secrets survive restarts through an AOF synced before each write is + # acknowledged (SPECIFICATION.md 6.2), so a burn can't be undone by a crash. + # RDB save points stay off because restoring an older snapshot could bring a + # burned secret back. The low rewrite threshold limits how long deleted + # entries remain in the log on small instances. command: - redis-server - --maxmemory @@ -12,8 +18,13 @@ services: - --maxmemory-policy - noeviction - --save - - '900' - - '1' + - '' + - --appendonly + - 'yes' + - --appendfsync + - always + - --auto-aof-rewrite-min-size + - 1mb - --loglevel - warning volumes: diff --git a/package.json b/package.json index 938f82e..b1c510c 100644 --- a/package.json +++ b/package.json @@ -10,7 +10,7 @@ "format": "prettier --write .", "format:check": "prettier --check .", "test:e2e": "pnpm --filter @crypt.fyi/web test:e2e", - "redis": "docker run --rm -p 6379:6379 -v ./redis-data:/data redis:latest redis-server --save 1 1 --loglevel notice", + "redis": "docker run --rm -p 6379:6379 redis:latest redis-server --save \"\" --appendonly no --loglevel notice", "changeset": "changeset", "version-packages": "changeset version", "release": "node ./scripts/release.mjs", diff --git a/packages/core/src/i18n/locales/de.ts b/packages/core/src/i18n/locales/de.ts index 87f2646..5c9cb89 100644 --- a/packages/core/src/i18n/locales/de.ts +++ b/packages/core/src/i18n/locales/de.ts @@ -395,7 +395,8 @@ export const de: TranslationKeys = { tls: 'TLS-Verschlüsselung für alle API-Kommunikation', expiration: 'Automatischer Datenablauf mit konfigurierbarer Lebensdauer (TTL)', deletion: 'Sichere Datenlöschung nach Zugriff oder Ablauf', - storage: 'Keine dauerhafte Speicherung sensibler Informationen', + storage: + 'Geheimnisse gelangen nur verschlüsselt auf die Festplatte, und gelöschte Einträge können dort bis zur nächsten Log-Komprimierung verbleiben; Klartext und Schlüssel werden nie gespeichert', }, }, thirdParty: { diff --git a/packages/core/src/i18n/locales/en.ts b/packages/core/src/i18n/locales/en.ts index d631ceb..cce091f 100644 --- a/packages/core/src/i18n/locales/en.ts +++ b/packages/core/src/i18n/locales/en.ts @@ -387,7 +387,8 @@ export const en: TranslationKeys = { tls: 'TLS encryption for all API communications', expiration: 'Automatic data expiration with configurable Time-To-Live (TTL)', deletion: 'Secure deletion of data after access or expiration', - storage: 'No persistent storage of sensitive information', + storage: + 'Secrets reach disk only in encrypted form and deleted entries can remain there until the log is next compacted; plaintext and keys are never stored', }, }, thirdParty: { diff --git a/packages/core/src/i18n/locales/es.ts b/packages/core/src/i18n/locales/es.ts index 5b908dc..0e8d285 100644 --- a/packages/core/src/i18n/locales/es.ts +++ b/packages/core/src/i18n/locales/es.ts @@ -393,7 +393,8 @@ export const es: TranslationKeys = { tls: 'Cifrado TLS para todas las comunicaciones API', expiration: 'Vencimiento automático de datos con tiempo de vida configurable (TTL)', deletion: 'Eliminación segura de datos después del acceso o vencimiento', - storage: 'Sin almacenamiento persistente de información sensible', + storage: + 'Los secretos solo llegan al disco cifrados y las entradas eliminadas pueden permanecer allí hasta la siguiente compactación del registro; el texto plano y las claves nunca se almacenan', }, }, thirdParty: { diff --git a/packages/core/src/i18n/locales/fr.ts b/packages/core/src/i18n/locales/fr.ts index f75b298..3091dc9 100644 --- a/packages/core/src/i18n/locales/fr.ts +++ b/packages/core/src/i18n/locales/fr.ts @@ -394,7 +394,8 @@ export const fr: TranslationKeys = { tls: 'Chiffrement TLS pour toutes les communications API', expiration: 'Expiration automatique des données avec durée de vie configurable (TTL)', deletion: 'Suppression sécurisée des données après accès ou expiration', - storage: "Pas de stockage persistant d'informations sensibles", + storage: + "Les secrets n'atteignent le disque que chiffrés et les entrées supprimées peuvent y rester jusqu'au prochain compactage du journal ; le texte en clair et les clés ne sont jamais stockés", }, }, thirdParty: { diff --git a/packages/core/src/i18n/locales/zh.ts b/packages/core/src/i18n/locales/zh.ts index a9d6cfe..35af59c 100644 --- a/packages/core/src/i18n/locales/zh.ts +++ b/packages/core/src/i18n/locales/zh.ts @@ -371,7 +371,8 @@ export const zh: TranslationKeys = { tls: '所有 API 通信都使用 TLS 加密', expiration: '具有可配置生存时间 (TTL) 的自动数据过期', deletion: '访问或过期后安全删除数据', - storage: '不持久存储敏感信息', + storage: + '秘密内容仅以加密形式写入磁盘,已删除的条目可能保留到下一次日志压缩;绝不存储明文和密钥', }, }, thirdParty: { diff --git a/packages/server/docker-compose.yaml b/packages/server/docker-compose.yaml index e3ffaea..cd87c4e 100644 --- a/packages/server/docker-compose.yaml +++ b/packages/server/docker-compose.yaml @@ -1,10 +1,16 @@ services: redis: image: redis:latest + # Memory only: dev data is disposable. The image snapshots by default, so + # persistence has to be turned off explicitly. + command: + - redis-server + - --save + - '' + - --appendonly + - 'no' ports: - '6379:6379' - volumes: - - redis_data:/data jaeger: image: jaegertracing/all-in-one:latest environment: @@ -12,6 +18,3 @@ services: ports: - 16686:16686 - 4318:4318 - -volumes: - redis_data: