Repository navigation
fix(security): drop middleware cache-control stamping; real 404 for nonexistent posts (CD-06) - #4069
Merged
Merged
Conversation
…onexistent posts (CD-06) Remove the anonymous-page Cache-Control overlay from proxy.ts entirely: every response now keeps Next's dynamic-page default (private, no-cache, no-store, max-age=0, must-revalidate), and edge caching is owned by the openresty sidecar (openresty #23: RSC-family cache bypass + proxy_ignore_headers Cache-Control + 200-only storage). The stamp was broken in three reproduced forms (live retest 2026-10-26): 1. Render-time 404 carried the public header: GET /@user/<missing> answered 404 + public, max-age=300. The status>=300 guard ran at proxy time on the rewrite response (always ~200) and can never see render-time statuses, so the requesting browser cached the 404 for 5 minutes. 2. Soft-404: GET /<tag>/@user/<missing> answered 200 + the public header (the category post page was only a client-fetching skeleton), and openresty cached it (X-Cache: HIT reproduced). 3. RSC guard dead in runtime: RSC: 1 requests got 307 + public, max-age=300 — Next 16's middleware adapter strips the flight markers (rsc header, _rsc query) before proxy() runs, so isRscFamilyRequest can never fire in production. Unit tests bypass the adapter, so they never caught it. There is no page-level response-header API in Next 16 that could replace the stamp (headers() is read-only; generateMetadata emits HTML meta only), so no replacement stamping mechanism is introduced. Also make nonexistent posts a real 404 on both post pages: the category-form page now resolves the discussion server-side (same getDiscussion path as generateMetadata, deduped per request via React cache()) and calls notFound() before the stream starts when the post definitively does not exist. When the existence check itself fails (RPC error, hivemind outage — withCache stale-while-error rethrowing with no stale copy), the page falls back to the client-fetch skeleton instead of 404ing: "asked and got an absent result" vs "the ask could not be completed" is the throw/no-throw boundary of getDiscussion. Adds scripts/verify-page-cache-headers.ts (pnpm verify:cache-headers) to run the CD-06 runtime matrix against a running production server. Local run against `pnpm start` (STEEM_API_URL=dev): real post 200 + Next default; nonexistent with/without category 404 + Next default; RSC: 1 307 with no Cache-Control; cookie'd GET 200 + Next default. Residual: legacy production (#4032) has the same soft-404 shape — tracked as a separate item there.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Audit finding CD-06 (steem-audit 2026-09-29, confirmed by live retesting on dev 2026-10-26): the anonymous-page
Cache-Controlstamping mechanism inproxy.tswas broken in three ways, all reproduced on a production build. This PR removes the stamp entirely and makes nonexistent posts a real 404 on both post-page forms.Before: three reproduced bad forms
GET /@user/<nonexistent>public, max-age=300response.status >= 300guard runs at proxy time on the rewrite response (always ~200) and can never see render-time statuses; the browser cached the 404 for 5 minGET /<tag>/@user/<nonexistent>public, max-age=300, edge-cached (X-Cache: HIT)notFound()never ran server-sideGET <real post>+RSC: 1public, max-age=300isRscFamilyRequest()never fires in production: Next 16's middleware adapter strips the flight markers (rscheader,_rscquery) beforeproxy()runs. Unit tests bypass the adapter, so they never caught itAfter: what this PR does
1. No variant-aware
Cache-Controlat all (architectural).applyAnonymousPageCacheControland everything feeding it is deleted. Every response keeps Next's dynamic-page default (private, no-cache, no-store, max-age=0, must-revalidate). Edge caching is fully owned by the openresty sidecar — openresty #23 (7a7e12c, deployed to dev 2026-09-28) added the RSC-family cache bypass,proxy_ignore_headers Cache-Control, and 200-only storage, so the edge ignores whatever the app sends. There is no page-level response-header API in Next 16 that could replace the stamp (headers()is read-only;generateMetadataemits HTML meta only), so no replacement mechanism is attempted. Removing the stamp also kills the browser-level poisoning (browsers cached stamped 404s/307s/soft-404s for 5 min per RFC 9111).2. Real 404 for nonexistent posts (both post pages). The no-category form already had the server-side existence check; the category form is now unified on the same design:
getDiscussion— the same fetchgenerateMetadatauses, deduped per request with Reactcache()(the L2withCachehas no in-flight coalescing, so a cold cache would otherwise fire thebridge.get_discussionRPC twice).getDiscussion: a definitive miss returns a content map without theauthor/permlinkkey (or null) without throwing →notFound()is thrown before the stream starts, so the status is a real 404. A transport/RPC error (hivemind transient outage, timeout, 5xx —withCachestale-while-error rethrows only when no stale copy exists) does not 404: the page falls back to the client-fetch skeleton, so a transient upstream error never becomes a fake 404.generateMetadatakeeps its degrade-to-bare-title behavior (metadata never 500s the page).3. Reusable runtime verification script.
pnpm verify:cache-headers(scripts/verify-page-cache-headers.ts) runs the CD-06 matrix against a running production server (BASE_URL, defaulthttp://127.0.0.1:8080) and exits non-zero on violation. Not wired into CI (it needs a server).Runtime matrix (local run,
pnpm build && pnpm start,STEEM_API_URL=https://api.steemitdev.com)private, no-cache, no-store, max-age=0, must-revalidateprivate, no-cache, no-store, max-age=0, must-revalidateprivate, no-cache, no-store, max-age=0, must-revalidateRSC: 1private, no-cache, no-store, max-age=0, must-revalidateOK: all 5 cases conform (no public Cache-Control; real 404s)— exit 0.Deleted from proxy.ts
applyAnonymousPageCacheControl,ANONYMOUS_PAGE_CACHE_CONTROL,COOKIED_PAGE_CACHE_CONTROL,ANON_POST_PAGE_GATE_RE,isRscFamilyRequest,RSC_FAMILY_REQUEST_HEADERS,RSC_FAMILY_QUERY_PARAM,rewriteTargetPathname, and the stamp/RSC rationale comment blocks.decodePathnameSafeis kept — it is still the%40-decode fallback for route resolution. proxy.ts's job is now URLs + CSP nonce only (route resolution/rewrites, CSP nonce generation+mirroring, security redirects, GDPR handling untouched).Tests: the
ANONYMOUS_PAGE_CACHE_CONTROL freeze, gate-regex/FLIGHT_HEADERS freeze guards, and the cache-eligibility suite in__tests__/proxy.test.tsare replaced by aproxy never stamps Cache-Control (CD-06)suite; new__tests__/app/post-category.test.tsxcovers the category page's 404/fallback decision.docs/ROUTE_MAP.mdrewritten accordingly (no app-side cache headers invariant; existence-check semantics; verify script).Residuals
Gates
pnpm lint:ci: 0 errors (budget 0)pnpm lint: 7 warnings / 0 errors (pre-existing baseline, unchanged)pnpm test: 88 files, 1116 tests greennpx tsc --noEmit: clean