Skip to content

fix(cache): replace post-broadcast prefix-scan invalidation with precise deletes (CD-03) - #4065

Merged
ety001 merged 3 commits into
nextfrom
fix/cd-03-precise-cache-invalidation
Oct 19, 2026
Merged

ety001 merged 3 commits into
nextfrom
fix/cd-03-precise-cache-invalidation

Conversation

@ety001

@ety001 ety001 commented Oct 19, 2026

Copy link
Copy Markdown
Member

Problem

Audit finding CD-03 (MEDIUM): after a successful broadcast, the
invalidation path in app/api/steem/broadcast/route.ts still called
cacheDeleteByPrefix (Redis SCAN MATCH <prefix>* loop) for the
follow-list and communities-list cache families — a regression of
audit N-10
(2026-09-21), which had already moved this path to precise
deletes. SCAN iterates the entire keyspace regardless of match
count, so per-transaction cache work grew with the number of such
operations and the global key count — an amplified, shared-Redis-
consuming workload (bounded only by the 30/min/IP broadcast limit,
256KB body cap, op allowlist and on-chain signatures).

Fix

cacheDeleteByPrefix is removed outright from lib/cache/redis.ts
(compile-time protection against reintroduction), replaced by a
generation-counter mechanism:

  • Read side (lib/steem/client.ts): the four affected families
    embed getCacheGeneration(scope) as a #v{n} suffix in their cache
    key.
  • Write side (bumpCacheGeneration(scope)): one atomic
    MULTI INCR + EXPIRE(24h) — every key of the previous generation
    becomes unreachable immediately and expires on its own TTL. No SCAN,
    no DEL fan-out, O(1) work per write. It is also strictly safer than a
    DEL against the resurrection race: an in-flight read that fetched
    pre-write data lands under the old generation, where no later read
    ever looks.

Key-format inventory (former SCAN sites → what happens now)

Former prefix-scan call site Now Why the set is complete
steem:following:{follower}:* bump scope following:{follower} Read side (getFollowing) keys are steem:following:{account}:{type}:{start}#v{gen} with gen from exactly this scope — every {type}/{start} variant is covered by the one bump. {type} is route-whitelisted (`blog
steem:following-page:{follower}:* bump scope following-page:{follower} Keys are steem:following-page:{account}:{type}:{page}:{limit}#v{gen}; page spans [0,5000] × limit [1,100] — up to ~1M variants. Scope-embedded generation covers all of them.
steem:followers-page:{target}:* bump scope followers-page:{target} Same shape as above for the target's followers pages.
steem:communities:* bump scope communities (family-wide) Keys are steem:communities:{sort}:{query}:{limit}#v{gen} with query free text (64 chars) — unbounded fan-out; a subscribe shifts subscriber counts in every variant, so one family-wide counter is both complete and minimal.

The remaining invalidations were already precise exact-key DELs and are
unchanged: steem:profile:{account} (vote/comment/custom_json/
account_update2; lowercase + raw-case variants) and
steem:community-subscribers:{community} (exact key, 10-min TTL).
Defensive dual-case bumping (lowercase + trimmed raw-case) is preserved
for the follow families, mirroring the previous dual-variant sweep.

Counter-TTL invariant: the generation counter's TTL (24h, refreshed
on every bump) exceeds the longest versioned-family total TTL
(communities 600s+1800s = 40min) by 36×, so a counter can never expire
while entries from its current generation are still alive — an expired
counter restarting at 1 can never alias a previous cycle's generation.

Bound

MAX_INVALIDATION_OPS_PER_REQUEST = 64 (exact-key DELs + generation
bumps combined) in the broadcast route. A transaction may legally batch
many operations under the 256KB body cap; the budget bounds that batch
to constant Redis work. Fail-soft: on exhaustion it logs once and
skips the remaining invalidations — the response stays 200 (the
transaction is already on chain), skipped entries simply expire on their
own TTLs, and the pending overlay (write→read convergence) is never
budget-gated.

Correctness / convergence

The write→read convergence documented in docs/VOTE_STATE_FLOW.md
(VOTE_STATE_FLOW is updated in this PR) is preserved: pending-overlay
writes and reads are untouched, exact profile/subscriber deletes are
untouched, L1 X-Cache-Invalidate tokens are untouched. The only
observable change is that follow-list/communities entries of an old
generation linger unreachable until their own TTL (≤40min) instead of
being DELeted — at zero read/scan cost.

Test evidence

  • __tests__/api/steem/broadcast.test.ts: asserts the invalidation path
    issues ONLY precise DELs + bumps (a cacheDeleteByPrefix regression
    canary is supplied by the mock and asserted never called), exact keys /
    scopes per op type (vote, comment, follow variants incl. mute /
    unfollow-keep-muted / mixed-case / whitespace-dedupe / charset-guard,
    community subscribe/unsubscribe), and the per-request cap (80 wanted →
    exactly 64 executed, warn logged once, response 200, overlay intact).
  • __tests__/lib/steem/client.test.ts: pins the #v{n} key format and
    the exact scope strings the broadcaster bumps (scope mismatch = silent
    invalidation miss).
  • __tests__/lib/cache/redis.test.ts: pins the counter wire contract
    (namespaced key, MULTI INCR+EXPIRE 24h), no-op without REDIS_URL,
    fail-open reads, swallowed write errors.
  • Full suite: 85 files / 1070 tests pass; pnpm lint matches the
    pre-existing baseline exactly (1 pre-existing error in
    __tests__/proxy.test.ts:383 + 7 pre-existing warnings, zero new).

ety001 and others added 3 commits October 19, 2026 22:57
…ise deletes (CD-03)

The post-broadcast invalidation path still invalidated the follow-list and
communities-list cache families with cacheDeleteByPrefix (Redis SCAN), a
regression of audit N-10 whose per-transaction cost grew with the global
key count on the shared Redis. Those families now embed a generation
counter in their cache keys (lib/steem/client.ts) and the broadcast route
bumps the counter with a single O(1) INCR; cacheDeleteByPrefix is removed
outright so a SCAN-based regression cannot reappear. Invalidation work is
additionally capped at 64 ops per request, failing soft (log + skip) so an
already-broadcast transaction is never failed over cache housekeeping.
Write->read convergence per docs/VOTE_STATE_FLOW.md is preserved: exact
profile/community-subscriber deletes are unchanged and the pending overlay
is untouched.
…ire level (CD-03 review)

The broadcast route's SCAN-never-called canary lives in broadcast.test.ts,
which mocks lib/cache/redis wholesale — a SCAN reintroduced inside
lib/cache/redis.ts itself would be invisible to it. Extend the ioredis
double with del/scan spies and pin the wire contract: cacheDelete issues
exactly one namespaced DEL, and no module export (delete, bump, read)
ever issues a SCAN.
…ound-2 review)

The counter-TTL invariant comment claimed a counter 'can never expire
while entries from its current generation are still alive' — reads do
not refresh the counter TTL, so an entry can be cached under the
current generation right up to the counter's expiry, and a rebump
within a family total-TTL of that expiry restarts the counter at 1 and
can briefly serve the expired cycle's tail entries. Consequence is
bounded staleness equal to the Redis-less fallback (330s follow lists,
2400s communities) with no cross-scope leakage. Reword the comment in
lib/cache/redis.ts and the mirrored sentence in docs/VOTE_STATE_FLOW.md
to describe that residual accurately instead of overclaiming the 36x
margin as a proof of impossibility.
@ety001
ety001 merged commit d9937d4 into next Oct 19, 2026
0 of 2 checks passed
@ety001
ety001 deleted the fix/cd-03-precise-cache-invalidation branch October 19, 2026 15:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant