Skip to content

fix(security): drop middleware cache-control stamping; real 404 for nonexistent posts (CD-06) - #4069

Merged
ety001 merged 1 commit into
nextfrom
fix/cd-06-remove-stamp-real-404
Nov 3, 2026
Merged

ety001 merged 1 commit into
nextfrom
fix/cd-06-remove-stamp-real-404

Conversation

@ety001

@ety001 ety001 commented Nov 3, 2026

Copy link
Copy Markdown
Member

Summary

Audit finding CD-06 (steem-audit 2026-09-29, confirmed by live retesting on dev 2026-10-26): the anonymous-page Cache-Control stamping mechanism in proxy.ts was broken in three ways, all reproduced on a production build. This PR removes the stamp entirely and makes nonexistent posts a real 404 on both post-page forms.

Before: three reproduced bad forms

Case Before Root cause
GET /@user/<nonexistent> 404 + public, max-age=300 the overlay's response.status >= 300 guard runs at proxy time on the rewrite response (always ~200) and can never see render-time statuses; the browser cached the 404 for 5 min
GET /<tag>/@user/<nonexistent> 200 (soft-404) + public, max-age=300, edge-cached (X-Cache: HIT) the category post page rendered only a client-fetching skeleton; the body was fetched client-side so notFound() never ran server-side
GET <real post> + RSC: 1 307 + public, max-age=300 isRscFamilyRequest() never fires in production: Next 16's middleware adapter strips the flight markers (rsc header, _rsc query) before proxy() runs. Unit tests bypass the adapter, so they never caught it

After: what this PR does

1. No variant-aware Cache-Control at all (architectural). applyAnonymousPageCacheControl and everything feeding it is deleted. Every response keeps Next's dynamic-page default (private, no-cache, no-store, max-age=0, must-revalidate). Edge caching is fully owned by the openresty sidecar — openresty #23 (7a7e12c, deployed to dev 2026-09-28) added the RSC-family cache bypass, proxy_ignore_headers Cache-Control, and 200-only storage, so the edge ignores whatever the app sends. There is no page-level response-header API in Next 16 that could replace the stamp (headers() is read-only; generateMetadata emits HTML meta only), so no replacement mechanism is attempted. Removing the stamp also kills the browser-level poisoning (browsers cached stamped 404s/307s/soft-404s for 5 min per RFC 9111).

2. Real 404 for nonexistent posts (both post pages). The no-category form already had the server-side existence check; the category form is now unified on the same design:

  • The page resolves the discussion server-side via getDiscussion — the same fetch generateMetadata uses, deduped per request with React cache() (the L2 withCache has no in-flight coalescing, so a cold cache would otherwise fire the bridge.get_discussion RPC twice).
  • "Not found" vs "RPC failed" is the throw/no-throw boundary of getDiscussion: a definitive miss returns a content map without the author/permlink key (or null) without throwing → notFound() is thrown before the stream starts, so the status is a real 404. A transport/RPC error (hivemind transient outage, timeout, 5xx — withCache stale-while-error rethrows only when no stale copy exists) does not 404: the page falls back to the client-fetch skeleton, so a transient upstream error never becomes a fake 404.
  • generateMetadata keeps its degrade-to-bare-title behavior (metadata never 500s the page).

3. Reusable runtime verification script. pnpm verify:cache-headers (scripts/verify-page-cache-headers.ts) runs the CD-06 matrix against a running production server (BASE_URL, default http://127.0.0.1:8080) and exits non-zero on violation. Not wired into CI (it needs a server).

Runtime matrix (local run, pnpm build && pnpm start, STEEM_API_URL=https://api.steemitdev.com)

Case Status Cache-Control
real post, anonymous GET 200 private, no-cache, no-store, max-age=0, must-revalidate
nonexistent post with category 404 private, no-cache, no-store, max-age=0, must-revalidate
nonexistent post without category 404 private, no-cache, no-store, max-age=0, must-revalidate
real post + RSC: 1 307 (none)
real post, cookie-carrying GET 200 private, no-cache, no-store, max-age=0, must-revalidate

OK: all 5 cases conform (no public Cache-Control; real 404s) — exit 0.

Deleted from proxy.ts

applyAnonymousPageCacheControl, ANONYMOUS_PAGE_CACHE_CONTROL, COOKIED_PAGE_CACHE_CONTROL, ANON_POST_PAGE_GATE_RE, isRscFamilyRequest, RSC_FAMILY_REQUEST_HEADERS, RSC_FAMILY_QUERY_PARAM, rewriteTargetPathname, and the stamp/RSC rationale comment blocks. decodePathnameSafe is kept — it is still the %40-decode fallback for route resolution. proxy.ts's job is now URLs + CSP nonce only (route resolution/rewrites, CSP nonce generation+mirroring, security redirects, GDPR handling untouched).

Tests: the ANONYMOUS_PAGE_CACHE_CONTROL freeze, gate-regex/FLIGHT_HEADERS freeze guards, and the cache-eligibility suite in __tests__/proxy.test.ts are replaced by a proxy never stamps Cache-Control (CD-06) suite; new __tests__/app/post-category.test.tsx covers the category page's 404/fallback decision. docs/ROUTE_MAP.md rewritten accordingly (no app-side cache headers invariant; existence-check semantics; verify script).

Residuals

Gates

  • pnpm lint:ci: 0 errors (budget 0)
  • pnpm lint: 7 warnings / 0 errors (pre-existing baseline, unchanged)
  • pnpm test: 88 files, 1116 tests green
  • npx tsc --noEmit: clean

…onexistent posts (CD-06)

Remove the anonymous-page Cache-Control overlay from proxy.ts entirely:
every response now keeps Next's dynamic-page default
(private, no-cache, no-store, max-age=0, must-revalidate), and edge
caching is owned by the openresty sidecar (openresty #23: RSC-family
cache bypass + proxy_ignore_headers Cache-Control + 200-only storage).

The stamp was broken in three reproduced forms (live retest 2026-10-26):

1. Render-time 404 carried the public header: GET /@user/<missing>
   answered 404 + public, max-age=300. The status>=300 guard ran at
   proxy time on the rewrite response (always ~200) and can never see
   render-time statuses, so the requesting browser cached the 404 for
   5 minutes.
2. Soft-404: GET /<tag>/@user/<missing> answered 200 + the public
   header (the category post page was only a client-fetching skeleton),
   and openresty cached it (X-Cache: HIT reproduced).
3. RSC guard dead in runtime: RSC: 1 requests got 307 +
   public, max-age=300 — Next 16's middleware adapter strips the flight
   markers (rsc header, _rsc query) before proxy() runs, so
   isRscFamilyRequest can never fire in production. Unit tests bypass
   the adapter, so they never caught it.

There is no page-level response-header API in Next 16 that could
replace the stamp (headers() is read-only; generateMetadata emits HTML
meta only), so no replacement stamping mechanism is introduced.

Also make nonexistent posts a real 404 on both post pages: the
category-form page now resolves the discussion server-side (same
getDiscussion path as generateMetadata, deduped per request via React
cache()) and calls notFound() before the stream starts when the post
definitively does not exist. When the existence check itself fails
(RPC error, hivemind outage — withCache stale-while-error rethrowing
with no stale copy), the page falls back to the client-fetch skeleton
instead of 404ing: "asked and got an absent result" vs "the ask could
not be completed" is the throw/no-throw boundary of getDiscussion.

Adds scripts/verify-page-cache-headers.ts (pnpm verify:cache-headers)
to run the CD-06 runtime matrix against a running production server.
Local run against `pnpm start` (STEEM_API_URL=dev): real post 200 +
Next default; nonexistent with/without category 404 + Next default;
RSC: 1 307 with no Cache-Control; cookie'd GET 200 + Next default.

Residual: legacy production (#4032) has the same soft-404 shape —
tracked as a separate item there.
@ety001
ety001 merged commit 975d125 into next Nov 3, 2026
1 of 2 checks passed
@ety001
ety001 deleted the fix/cd-06-remove-stamp-real-404 branch November 3, 2026 22:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant