feat(api): serve retained document content and versions

[iuiu-py]

What

Adds a versioned read contract for retained originals:

• GET /api/v1/documents/{id}/content serves the current or requested ?version=N bytes from the document ledger.
• GET /api/v1/documents/{id}/versions lists version, sha256, size_bytes, and ingested_at.
• Purged documents answer 410 Gone; soft-deleted retained documents remain readable until purge.
• Responses carry the ledger MIME type, exact byte length, a quoted SHA-256 ETag, and RFC 5987-safe Content-Disposition.
• Authentication accepts browser sessions and Viewer-or-higher utp_pat_ personal access tokens scoped to the knowledge base; source ingest tokens are rejected.

The content transaction takes SELECT ... FOR NO KEY UPDATE and holds it through the blob read, so replacement or purge cannot move or remove the selected blob after the ledger check. Commit happens after the blob read is complete. A ledger entry whose blob is unavailable returns 500 as an invariant failure rather than masking it as a normal client error.

ADR 0052 records the API/lifecycle boundary and why the ledger is the source of truth.

Why

The document API currently exposes metadata while versions and original bytes stay internal. This makes auditable retention observable to clients, lets callers retrieve the exact recorded original, and closes #859.

Closes #859

Testing

Against Postgres 17 with UTOPIA_TEST_REQUIRE_DB=1 and UTOPIA_DATABASE_URL set:

cargo fmt --all
cargo clippy --locked --workspace --all-targets -- -D warnings
cargo test --locked -p utopia-server 'api::documents_routes::tests' -- --nocapture

Results:

• cargo fmt completed without changes.
• cargo clippy passed with warnings denied.
• 10/10 document-route tests passed, including version selection, deleted-versus-purged lifecycle, knowledge-base/PAT authorization, source-token rejection, transactional race protection, and missing-blob behavior.

[WaylandYang]

Read it end to end. CI approved — fork PRs need that here after every push.

This is careful work, and the parts I went looking for trouble in are the parts you got right.

Authentication. A new route that serves raw bytes is where I expected to find the hole. Instead there are two independent checks: access::require_kb(..., Role::Viewer) for the person, and pat.covers(kb_id) for the token, so a valid token for base A cannot read base B even if its owner is a member of both. Building a separate extractor rather than reusing AuthUser is the right call for the reason your comment gives — AuthUser would interpret any bearer string as a session and there would be nowhere to enforce scope.

Ingest tokens are rejected structurally rather than by a check that could be forgotten: utp_ tokens are Uuid::simple() hex, and pat_ is not hex, so an ingest token cannot be mistaken for a PAT and falls through to session decoding, which fails. That holds without anyone remembering to maintain it.

A scoped token gets the same 404 as an inaccessible document. Worth having written down in the record, because the tempting 403 leaks which document ids exist.

The transaction commits before the response is built. tx.commit() precedes (StatusCode::OK, headers, bytes).into_response(), so a slow client does not hold a row lock or a pool connection for the length of its download. That is the failure mode I went looking for after reviewing #833, and it is already avoided here.

version.size_bytes != bytes.len() is checked against the ledger and raised as an invariant failure rather than dressed up as a client error. Right instinct — a blob that disagrees with its ledger entry is not a 404.

Two things.

1. docs/decisions/README.md has no row for 0052. The record is added but appears in neither index table. Every other record this week (0048 through 0051) carries rows in both. 0052 is the correct next number — 0049 belongs to #839, which is open — so only the index rows are missing.

2. One sentence missing from the Limits section, which is otherwise the right section. You say the route "buffers within the existing upload cap", and MAX_UPLOAD_BYTES is 100 MB, so a single response is bounded. What is not bounded is how many of those happen at once: N concurrent downloads of large retained originals is N × up to 100 MB resident, and nothing in the route or the record caps N. Excluding Range requests and streaming is a defensible scope decision and I am not asking you to reopen it — but the concurrency multiplier is the thing whoever first serves a base full of large scans will meet, and the record is already the right place to say it.

Neither blocks. Once CI reports, and with the index rows added, this looks good to me.

[WaylandYang]

Cross-posting on #860 and #863: these two implement the same feature from #859 and
register byte-identical routes, so only one can land.

.route("/documents/{id}/content", get(documents_routes::content))
.route("/documents/{id}/versions", get(documents_routes::versions))

Neither exists on dev. They are independent single commits by different authors,
which is why GitHub reports both as mergeable — each is computed against dev
alone. Merging one and then the other conflicts:

$ git merge refs/pr/860   # clean
$ git merge refs/pr/863
CONFLICT (content): Merge conflict in crates/utopia-server/src/api/documents_routes.rs
CONFLICT (content): Merge conflict in crates/utopia-server/src/api/mod.rs

Worth a maintainer call on which one to keep before either gets more review
attention. One input: CONTRIBUTING asks for an ADR when public API changes, and
#860 carries docs/decisions/0052-document-content-is-a-read-contract.md (0052 is
free on dev) while #863 does not. #860 also touches sources_routes.rs and
tokens.rs; #863 touches utopia-core/src/models.rs instead.

[WaylandYang]

Reviewed against #859's spec and side by side with #863 (the other implementation
of the same issue — see my note there). This is the one that should land.

It covers the requirement the other branch misses: session or scoped PAT at
Viewer level, with ingest tokens rejected. The DocumentReader extractor is the
right shape — AuthUser can't do this because it treats every bearer as a JWT,
and the comment says exactly that. Holding FOR NO KEY UPDATE from the ledger read
through the blob read closes the replacement/purge window instead of asking the
client to retry, and re-checking purged_at under the lock is the detail that
makes it actually hold. Resolving the default version through document_versions
rather than trusting documents.sha256 directly, and treating a missing ledger row
as an invariant error rather than a 404, matches the issue's "500 not 404"
semantics. Header construction that fails loudly on a bad MIME or an unencodable
filename is preferable to silently emitting a wrong ETag. ADR 0052 says all of
this out loud, including the compatibility boundary on reusing current display
metadata for historical bytes.

Verified on a merge of this branch into current dev (after #833/#845/#864
landed), against pgvector/pgvector:pg16 on a fresh database:

cargo test -p utopia-server documents   →  12 passed, 0 failed
  (all five new cases ran, including
   content_reads_keep_viewer_access_pat_scope_and_reject_source_tokens)
cargo test -p utopia-store              →  329 passed, 0 failed
cargo clippy --workspace --all-targets -- -D warnings   →  clean

One thing for a follow-up rather than this PR: none of these tests run in CI.
documents_routes_tests gates on test_db::url(), so in the backend job (no
database) all of them skip and report green, and the migrations job's server
filters are api::mcp::tests, api::chat::, retrieval:: — documents isn't
among them. That's true of the existing documents tests on dev too, not just
these five. A one-line step in the migrations job, the way #845 added one for
api::chat::, would make the PAT/purge/invariant cases actually load-bearing.
Happy to open that separately.

One small thing worth a follow-up from the #863 side, which I've suggested there:
recomputing SHA-256 over the served bytes. This PR checks size_bytes against the
ledger; the digest check is the stronger guarantee for an auditor and is cheap to
add on top.

LGTM.