feat(hub): the same file URL answers an agent with the file (BEA-186)

[ssowonny]

TL;DR

• The hub links our hook tells every agent to paste used to return 200 + an empty app shell, so an agent "read" nothing and said it had. Now the same URL returns the file.
• A browser still gets the viewer. Only a request whose Accept has no text/html (agents, curl) gets the file, plus an X-Bdrive-Provenance header.
• Honest failures: no credentials = 401, not a member = 403, missing file/project or hidden folder = 404. Before, all four got the same empty 200.
• Agent fetches count as agent reads on the heat map, never human ones, and never under an email.
• Known gap: Slack/Discord-style unfurlers are recognized by a User-Agent list so pasted links keep their titled card. A previewer that isn't on the list unfurls as a 401.

Closes BEA-186.

The one thing to check: the anonymous read

The SPA fallback runs outside authGate (auth.go leaves every non-/api/ path open so a browser can reach the login page). So the new branch has to authenticate itself, and the order matters:

flowchart LR
    A["GET /&lt;pid&gt;/&lt;path&gt;"] --> B{"Accept has text/html?<br/>view route? unfurler UA?<br/>project root?"}
    B -- yes --> Shell["app shell, unchanged"]
    B -- no --> C{"Authenticate"}
    C -- fail --> U401["401"]
    C -- ok --> D["serveProject(PermRead)"]
    D -- "no such project" --> N404["404"]
    D -- "not permitted" --> F403["403"]
    D -- ok --> E["serveFileAt<br/>(folder visibility -> 404)"]
    E --> OK["200 file + X-Bdrive-Provenance<br/>read recorded as agent"]

Loading

Authentication comes before serveProject. Otherwise its no such project 404 would tell an anonymous caller which project ids exist. TestAgentFetch_NegotiatedFileOnTheHumanURL covers the unauthenticated case first: 401, and no file bytes anywhere in the body.

What changed

Three seams, none of which change behavior on their own:

seam	where	why
proj() closure became the serveProject method	server.go	the SPA fallback is built outside Handler() and has no r.PathValue("project")
lookup split into lookup + lookupPath	server.go	the path comes from the URL, not ?path= (folder visibility is still applied)
recordRead learned the agent kind	reads.go	the actor is an ownsDevice-validated device id, or the fixed string "agent". It is never an email, because /heat?by=device shows agent actors to every member

The branch itself is agentFetchTarget + serveAgentFetch at the top of Server.frontend, above the shell's headers. A file served there carries exactly the headers /api/p/<id>/file does: ETag, sandbox CSP, nosniff, canonical-move header.

X-Bdrive-Provenance comes from the FileInfo already in hand (the newest journal op), so it costs no history query. Every value goes through strconv.Quote, because Author/User are client-written journal fields and a raw CR/LF would be response splitting.

Unchanged: view routes (/<pid>/history, dashboard, install, settings, legacy insights, which reservedViews mirrors from router.ts), project roots, single-volume mode, /s/<token> (out of scope per the spec), and the deliberate root-dotted-path 404 (/llms.txt).

Deviations from the plan

• One PR, not two. The build process ships one PR per issue. The refactor and the branch are in one commit, and the seams table above is the refactor half.
• Unfurler exemption (new). BEA-181 (feat(webapp): a pasted hub link unfurls with a real title (BEA-181) #205) landed after the plan and gave the shell per-URL <title>/og:* tags for unfurlers. Several unfurlers send no text/html, so without an exemption a pasted hub link in Slack would unfurl as a 401. unfurlerUAs (Slackbot, Discordbot, facebookexternalhit, Twitterbot, LinkedInBot, …) keeps those on the shell. TestShellOpenGraph now sends a Slackbot UA, which is the caller that test actually describes.
• Folder permissions (new since the plan). The agent door gets hidden-folder 404s for free through lookupPath. TestAgentFetch_HiddenFolderIsNotFound pins that.
• Branch …-with-2. An earlier build run left an unpushed commit on the original branch name (in another worktree). This branch cherry-picks that work onto current main and resolves conflicts with BEA-181, the Mac-app ReportRead hook and folder permissions. The agent branch sits above ReportRead, because the sidecar forwards whatever it gets as a human read.
• Three existing tests (TestFrontendRootDottedPathsAre404, TestSec_Router_TheShellIsServedForPathsTheClientMustSurvive, seccfgRaw) now send a browser Accept. They assert the shell a browser navigation gets. With no Accept at all, those requests now count as agent fetches.

What was run

• go test ./...: all green. New: internal/webapp/agentfetch_test.go (6 tests covering 401 without credentials and no bytes in the body, 403 for another org, 404 for a missing file/project and a hidden folder, 200 + provenance, a byte-safe browser shell, view routes, /llms.txt still 404, agent vs human heat, device-id actor, /store/* recording no read, and the unfurler exemption).
• npm run e2e: 236 passed, 7 failed, 1 skipped. The 7 are the admin/hub org specs (admin.spec.ts 15/42/202/230/246, hub.spec.ts 63/90), and a run on origin/main at 9b3eb19 fails the same 7. They're a known load-sensitive flake on main: admin.spec.ts:15 times out waiting for the account dropdown, leaves the org renamed, and the other six cascade from that. They don't touch the agent-fetch path (/orgs/*, /api/orgs). Caveat: this is 2 branch runs vs 1 main run, and the Go suite was running alongside some of them, which is the condition that makes this flake likely. Worth re-running on an idle machine before merge.
• go vet ./... is clean, and both mermaid blocks in architecture/webapp-server.md render with mmdc.
• Why no UI evaluation: the diff has no frontend changes. The browser still gets the same shell, and TestAgentFetch_BrowserAnswerIsUntouched plus the green SPA e2e specs cover that.

No frontend source changed, so there is no static/ rebuild and no screenshots. The change is a response body, not a UI.

Architecture changes

architecture/webapp-server.md: serveProject (the lifted proj closure) and the agentFetch branch of Server.frontend are new. agentFetch now reaches AuthProvider (401 first), serveProject (404, then 403) and ReadLedger (agent kind). The projectPerm note now names serveProject as the choke point.

✅ added · ❌ removed (strikethrough) · unmarked = unchanged

flowchart TB
    Server["Server"]
    AuthProvider["AuthProvider"]
    ReadLedger["ReadLedger"]
    projectPerm["<div style='text-align:left'><b>projectPerm</b><br/>org owner → admin<br/>explicit grant<br/>org member → project Default<br/>otherwise → none</div>"]
    serveProject["<div style='text-align:left'><b>serveProject</b><br/>projectVolume(id) → 404<br/>requirePermOn(level) → 403<br/>withProjectID(r, id)<br/>h(volume, w, r)</div>"]
    agentFetch["<div style='text-align:left'><b>agentFetch</b> (Server.frontend branch)<br/>Accept has no text/html<br/>&lt;project-id&gt;/&lt;path&gt;, rest non-empty<br/>head not in reservedViews<br/>UA not in unfurlerUAs<br/>Auth.Authenticate → 401<br/>serveProject(PermRead)<br/>serveFileAt + X-Bdrive-Provenance<br/>withAgentFetch(r) → agent-kind read</div>"]
    PermNote["perms.go — the one choke point:<br/><span style='background:#ef444455;padding:0 4px;border-radius:3px'>❌ <s>proj(level, h) closure in Handler()</s></span><br/><span style='background:#22c55e55;padding:0 4px;border-radius:3px'>✅ serveProject(w, r, id, level, h) method,<br/>shared by routes and the SPA fallback</span>"]
    Server -- "gates every per-project route" --> projectPerm
    Server -- "<span style='background:#22c55e55;padding:0 5px;border-radius:3px'>✅ one per-project resolver</span>" --> serveProject
    serveProject -. "<span style='background:#22c55e55;padding:0 5px;border-radius:3px'>✅ requirePermOn</span>" .-> projectPerm
    Server -- "<span style='background:#22c55e55;padding:0 5px;border-radius:3px'>✅ Server.frontend, above the shell</span>" --> agentFetch
    agentFetch -. "<span style='background:#22c55e55;padding:0 5px;border-radius:3px'>✅ Authenticate → 401 first</span>" .-> AuthProvider
    agentFetch -. "<span style='background:#22c55e55;padding:0 5px;border-radius:3px'>✅ 404 then 403, then the file</span>" .-> serveProject
    agentFetch -. "<span style='background:#22c55e55;padding:0 5px;border-radius:3px'>✅ ReadKindAgent, never an email</span>" .-> ReadLedger
    projectPerm -.- PermNote
    classDef added fill:#22c55e22,stroke:#22c55e,stroke-width:2px
    classDef noteBox fill:#88888822,stroke:#888888,stroke-dasharray:2 2
    class serveProject,agentFetch added
    class PermNote noteBox
    linkStyle 1 stroke:#22c55e,stroke-width:2px
    linkStyle 2 stroke:#22c55e,stroke-width:2px
    linkStyle 3 stroke:#22c55e,stroke-width:2px
    linkStyle 4 stroke:#22c55e,stroke-width:2px
    linkStyle 5 stroke:#22c55e,stroke-width:2px
    linkStyle 6 stroke:#22c55e,stroke-width:2px

Loading

Build session

cd $(git worktree list | grep bea-186-ph-idea-agent-fetchable-hub-urls-same-link-clean-text-with-2 | awk '{print $1}') && claude --resume c019b5bc-8723-4ba3-8380-ae756101a838

(only works on the build machine)

🤖 Generated with Claude Code