Skip to content

fix: hand Next.js the mount-relative request path - #62

Draft
kriszyp wants to merge 7 commits into
mainfrom
fix/mount-relative-request-path
Draft

kriszyp wants to merge 7 commits into
mainfrom
fix/mount-relative-request-path

Conversation

@kriszyp

@kriszyp kriszyp commented Aug 24, 2026 •

Copy link
Copy Markdown
Member

Harper's router strips an application's urlPath mount before a handler runs, but the plugin handed Next.js the un-stripped URL from request._nodeRequest, so an app mounted at /foo saw /foo/file.html for a path Harper had resolved to /file.html.

When middleware rewrote the URL (which mounting does), the request now goes through Request.withNodeAdapter(). Unmounted apps keep today's direct hand-off, and a Harper with no adapter falls back to it with a one-time warning.

Mounted apps still do not work on current Harper: the adapter 500s and stalls past 16 KB, so the mounted assertions are test.describe.fixme. Nothing that works today regresses (28 passed, 5 skipped).

🔧 Changes

Decisions and open questions (7)

For the human reviewer

  1. This is blocked on a harper change that is structural, not the patch I first estimated. I reported the harper side as four missing members; that was wrong. Those four (null-prototype request headers → {}, plus appendHeader, _implicitHeader, finished) get curl-level requests working — with them applied locally the fixture passes 6/6 — and still leave browser page loads hanging: chunks of 9.7 KB, 14.4 KB and 16.8 KB complete while 183 KB and 229 KB never finish, because res.write() returns false past the response PassThrough's 16 KB high-water mark, Node's Readable.pipe waits for 'drain', and compression's res.on override buffers that registration until a gzip stream that is never created. Making _implicitHeader() flush headers the way Node's ServerResponse does clears the stall and produces the next failure: raw gzip bytes with no Content-Encoding and truncated transfers. Root cause: the adapter's response is Object.assign(new EventEmitter(), {…}) rather than a real Writable, and Next's compression/on-headers/send stack depends on ServerResponse semantics. Decision needed: take the harper rewrite (response built on the PassThrough itself with header capture layered over it, plus a test that drives real middleware), or drop the adapter for entry 3. Reproducers, a README and the partial patch are at ~/dev/scripts/harper-node-adapter-repro.
  2. The adapter is gated on request.url !== request._nodeRequest.url rather than applied to every request. Routing everything through it moved unmounted apps — every existing user — onto that broken path, and charged a Proxy, a Promise, a PassThrough, a ResponseHeaders, an EventEmitter and a full header copy per request for a rewrite they never needed. The predicate is exact for URL rewrites: the two are equal precisely when nothing touched the URL. It is deliberately narrower than the adapter's purpose — a component that mutates only the Harper-level method or headers still takes the direct path, so Next.js reads the raw Node values. Making it "adapt whenever anything might have been mutated" means adapting always, which is the version this replaced.
  3. The alternative to the adapter entirely, verified working on today's harper, is urlParse(request.url, true). All three supported Next majors funnel through NextCustomServer.getRequestHandler, which does req.url = formatUrl(parsedUrl), so passing the mount-stripped parsed URL makes Next route mount-relatively with no adapter at all — measured against an unpatched harper: pages, nested pages, API route with query string and static assets all correct. It gives up the adapter's forwarding of middleware-mutated method and headers, which is why the issue prescribes the adapter. Switching costs one commit.
  4. On harper 5.1.x a mounted request now 500s or stalls where it used to 404. The capability guard only catches a Harper with no withNodeAdapter at all; where the method exists but is unfaithful there is nothing to detect. Mounted apps are non-functional either way, so this is a change in failure mode rather than a loss of function — but it is louder and, for the >16 KB stall, silent-and-hanging rather than a clean status code. Accepting it is what lets the fix land before the harper work; the alternative is holding this PR entirely. Mounted requests over Harper's HTTP/2 port take the same adapter path, which hands Next.js its own synthesized objects, so adaptHttp2ForNext (applied on the direct path only) cannot help there; the adapter rewrite covers it.
  5. The dev HMR upgrade handler is left on the un-stripped Node request, and the same defect does apply there. The issue asked me to leave it alone unless I could show that; I can. request.url === '/_next/webpack-hmr' compares the stripped URL so the branch is taken under a mount, but upgradeHandler(request._nodeRequest, …) then hands Next /mounted/_next/webpack-hmr. Not fixed here: withNodeAdapter is an HTTP adapter and does not apply to an upgrade, so the fix is a different mechanism needing its own design and test, and mounted dev mode cannot work anyway while Next generates the HMR client's URL at the root (entry 6). Dev-only; worth its own issue.
  6. basePath is out of scope and mounted apps are still incomplete even once harper is fixed. Inbound stripping makes Next route correctly under a mount, but Next still generates /_next/* asset URLs at the root, which 404 outside it. Only a build-time basePath fixes that. Whether the plugin should require, inject, or validate basePath against urlPath — or fail loudly when they disagree — is an open product decision left for a human, per the issue. The browser test asserts on server-rendered markup for exactly this reason.

    ❓ Your call: the rebase's pre-push review (harper-domain, round 1) sharpened this into an active regression rather than only incompleteness: before this PR, a user who set Next's basePath to match urlPath (e.g. both /mounted) got a fully working mount — routing and assets both — off the un-stripped direct hand-off, because Next's own basePath stripping handled it. This PR's gate on request.url !== request._nodeRequest.url is true for every mounted request regardless of a matching basePath, so that previously-working configuration now goes through the adapter unconditionally — a 500/hang on today's harper, and (per the domain review's code trace of Next 16's resolve-routes.js) a 404 even once the adapter is fixed, because Next then receives the already-stripped path and its own basePath-prefix check no longer matches. Not fixed here — it is the same open question as the rest of entry 6, now with a concrete regression case, so I left the gate as rebased rather than redesign it mid-rebase. A real fix likely means reading basePath from .next/routes-manifest.json and keeping the direct hand-off whenever _nodeRequest.url already carries it.

  7. Two review findings I did not act on. The stream-error listener logs at debug, which is off in production, so a truncated response and a completed one look alike — but Harper's own pipeBodyToResponse already warns on that error, and a second warn here would be the redundant error logging the guidelines rule out. And the listener is attached one microtask after the response resolves, so a handler that flushes headers and destroys the response in the same synchronous turn would still throw; Next.js's handler is async and does not, and harper's API offers no earlier hook — that one belongs upstream on withNodeAdapter rather than here.
Verification

Fails-on-base: the new next-16-mounted fixture (urlPath: /mounted) fails on origin/main — /mounted, /mounted/about and /mounted/api/echo all 404 because Next receives the un-stripped path — and passes 6 passed (9.2s) with this change against a harper carrying the partial adapter fix, including GET /mounted/api/echo?q=1 returning {"pathname":"/api/echo","search":"?q=1"} from inside the Next route handler and a chromium page load rendering the mounted home page. Those assertions are fixme on the branch as it stands; un-skipping them locally is how the run above was produced.

No regression: npm run test:integration on this branch against stock harper 5.1.23 is 28 passed, 5 skipped. An earlier revision that routed every request through the adapter scored 15 passed, 17 failed on the same suite, every failure a page.goto timeout on an unmounted fixture — integrationTests/next-16.pw.ts alone went from 5 passed (8.8s) on origin/main to 2 timeouts. That revision is what entry 2's gate replaced.

The adapter gaps were isolated in a standalone harness — plain Node http server, harper's Request class, Next's request handler, no Harper runtime — plus a chromium driver that reports which requests never finish; both are kept with a README at ~/dev/scripts/harper-node-adapter-repro. compress: false in the fixture's next.config.ts making the identical page load succeed is the experiment that isolates the 16 KB stall to Next's compression middleware rather than to Next or the harness.

Related PRs: #73 independent

Complexity: medium

Deferred findings

  • 717feccffacd6e52 minor correctness — src/plugin.ts — No-adapter fallback warns then hands Next.js the unstripped URL; suggestion to parse request.url — NOT FILED
  • 53983f2d938d6fc1 nit test-coverage — integrationTests/next-16-mounted.pw.ts — No test exercises the adapter branch, warning branch, or URL-mismatch selection — NOT FILED

Review-Coverage: authored=claude; ran=muse,gemini,codex; adjudicated=domain; declined=cursor-grok,cursor-composer,cursor-kimi,cursor-muse; rounds=3; full=1 @ 27eff90

Review-Attention: study ~9m (decisions: adapter-vs-url-rewrite, strip-vs-basepath, enabled-by-default-while-fixme, no-adapter-fallback; raised: open major) @ 27eff90

@kriszyp
kriszyp requested a review from Ethan-Arrowood August 24, 2026 14:18

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds support for running Next.js applications mounted under a subpath (urlPath) in Harper, updating the request handler in src/plugin.ts to use request.withNodeAdapter when a URL rewrite is detected. It also adds a new fixture and integration tests for this scenario. The review feedback correctly identifies a potential crash (TypeError) when response.body is null or undefined (such as in 204 or 304 responses) and suggests using optional chaining to safely attach the error listener.

Comment thread src/plugin.ts
Comment on lines +420 to +422
response.body.on('error', (error) =>
scope.logger.debug?.(`Next.js response stream error for ${request.pathname}: `, error)
);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If the Next.js response has no body (for example, with 204 No Content, 304 Not Modified, or HEAD requests), response.body will be null or undefined. Calling response.body.on('error', ...) directly will throw a TypeError: Cannot read properties of null (reading 'on') immediately when the promise resolves, causing these requests to fail with a 500 error.

Using optional chaining (response.body?.on) prevents this crash for empty responses.

Suggested change
response.body.on('error', (error) =>
scope.logger.debug?.(`Next.js response stream error for ${request.pathname}: `, error)
);
response.body?.on('error', (error) =>
scope.logger.debug?.(`Next.js response stream error for ${request.pathname}: `, error)
);

@kriszyp

kriszyp commented Sep 8, 2026

Copy link
Copy Markdown
Member Author

The upstream blocker in entry 1 now has an issue: Request.withNodeAdapter() cannot serve real Node middleware: Next.js 500s, then >16 KB responses hang silently (harper#2527, P1, Bug, v5.3, under [Epic] @harperdb/nextjs component integration). It carries the full six-symptom chain, the root cause, and the partial patch that closes 1–4 but not 5–6. A dev-agent task is queued against it.

Confirmed unfixed on main (6d725818c), v5.2.9 and v5.1.27, so this PR stays draft and its five mounted-app assertions stay test.describe.fixme until harper#2527 ships.

The design question in entry 1 is answered there as the rewrite, not dropping the adapter for entry 3 — the urlParse alternative gives up forwarding of middleware-mutated method and headers, which is the adapter's whole purpose.

— Claude (Opus 5)

@Ethan-Arrowood

Copy link
Copy Markdown
Member

Moving this back to draft. It isn't safe to land yet, for two reasons:

  1. It regresses the one mounted configuration that works today. An app built with a Next.js basePath equal to its Harper urlPath (both /mounted) currently serves correctly off the direct hand-off, because Next strips its own basePath. With this change, request.url !== request._nodeRequest.url is true for every mounted request, so that app goes through withNodeAdapter unconditionally. On Harper < 5.3 that means a 500 or a hang. On a fixed Harper it is still a 404, because Next receives the already-stripped path and its basePath prefix check fails. The gate needs to keep the direct hand-off whenever _nodeRequest.url already carries the app's basePath (readable from .next/routes-manifest.json at startup). This is entry 6 above, but it needs fixing here rather than leaving it as an open question.
  2. The capability check passes on Harpers where the adapter is broken. typeof request.withNodeAdapter === 'function' is true on every 5.1/5.2 release. The faithful adapter arrived in harper 5.3.0 (Rebuild withNodeAdapter's response on a real Writable so Node middleware (Next.js, compression, send) runs against it harper#2528), and the gzip fix for Node 26.11+ (Fix corrupt gzip from withNodeAdapter middleware on Node 26.11+, and return install_node_modules' npm report as an object on every npm harper#3108) is still unreleased in 5.4. Shipped as a fix: patch today, mounted users on current Harper would get a 500 or a silent hang instead of a 404. The repo still pins harper 5.1.23 for tests, so every mounted assertion is fixme and CI never exercises the new branch.

I think this is ready to come back once HarperFast/harper#2527 closes and the dev dependency moves to a Harper that carries the fix. At that point the fixme block can come off, along with the basePath-aware gate above. Two smaller cleanups for then: the new AGENTS.md entry points to ~/dev/scripts/harper-node-adapter-repro, which only exists on one machine, and it describes 5.1-era adapter failures that harper 5.3 has since fixed.

On the bot thread: response.body is always the adapter's stream (NodeAdapterResponse resolves with body: this), so the optional-chaining suggestion isn't needed.

🤖 Generated by Claude Opus 5.5; posted via @Ethan-Arrowood.

Kris Zyp and others added 7 commits October 19, 2026 15:52
Harper's router strips an application's mount from the Harper `Request` before the
plugin's handler runs, but the plugin was reaching past that proxy to
`request._nodeRequest`, which still carries the un-stripped URL. An app mounted at a
`urlPath` therefore handed Next.js a path it cannot route.

Route the request through `Request.withNodeAdapter()`, which presents the Harper
Request's own method/url/headers over the underlying Node request, and attach the
`error` listener the adapter's response body contract requires.

Adds a `next-16-mounted` fixture (mounted at `/mounted`) and integration coverage that
the mount root, a nested page, an API route and its query string all reach the right
Next.js route, and that paths outside the mount are not served by Next.js.

Refs #61

Co-Authored-By: Claude Opus <noreply@anthropic.com>
Note that routing through withNodeAdapter blocks on harper presenting a faithful Node
request/response, so a future agent does not chase the red page-based tests, and that the
Playwright browser binaries have to be installed for those tests to run at all.

Refs #61

Co-Authored-By: Claude Opus <noreply@anthropic.com>
Routing every request through withNodeAdapter moved apps with no urlPath — every existing
user — onto a path that today's harper cannot serve, and paid a proxy, a promise, a
PassThrough and a header copy per request for a rewrite they never needed. Take the adapter
only when middleware actually changed the URL; otherwise hand Next.js the Node request
directly, exactly as before.

Also guard on `_nodeResponse == null` rather than `=== undefined`, since harper's Bun and
uWS requests carry null and implement no Node adapter, and make the echo fixture return the
search string so the query-preservation test can fail for the reason it names.

Refs #61

Co-Authored-By: Claude Opus <noreply@anthropic.com>
Without a guard, a mounted application on a Harper predating Request.withNodeAdapter throws
per request from inside the HTTP chain. Fall back to the un-stripped hand-off — the behaviour
that Harper already had — and warn once so the operator learns why the mount does not route.

Skip the mounted assertions with test.describe.fixme rather than landing a permanently red
file: CI is disabled here, so a green local run is the repo's only regression signal, and a
file that is always red hides the next real failure inside it.

Refs #61

Co-Authored-By: Claude Opus <noreply@anthropic.com>
Refs #61

Co-Authored-By: Claude Opus <noreply@anthropic.com>
AGENTS.md already says CI runs integration-tests.yml on every PR (corrected on
main while this branch was rebased); this comment still claimed otherwise.
The "outside the fixme block" line narrated file structure a reader can see.

Dispatch-Task: pr-maint-fe1f4d8cb37821cbaefd78a61e6079ef
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…base

Dispatch-Task: pr-maint-fe1f4d8cb37821cbaefd78a61e6079ef

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@kriszyp
kriszyp force-pushed the fix/mount-relative-request-path branch from 33fe550 to 27eff90 Compare October 19, 2026 22:00

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants