feat(webapp): let the hub hold the document

cmd/bdrive/desktop.go

@@ -153,6 +153,16 @@ var desktopRoutes = []struct {
 	{"POST /api/p/{project}/presence", routeProxy, ""},
 	{"GET /api/p/{project}/collab", routeProxy, ""},
 	{"POST /api/p/{project}/collab", routeProxy, ""},
+	// /ycollab is the same surface with the document held by the hub instead
+	// of relayed between browsers. Proxied for a sharper reason than the
+	// relay's: the document IS hub state now, so a desktop that answered from
+	// local state would hand the editor a second, private document — and the
+	// first thing that document does is get saved over the file.
+	{"GET /api/p/{project}/ycollab", routeProxy, ""},
+	// The same route one segment deeper: y-websocket appends its room
+	// argument to the URL. Decoration — the hub names the room itself — but
+	// it has to be classified, or the desktop answers it locally.
+	{"GET /api/p/{project}/ycollab/{room...}", routeProxy, ""},
 
 	{"POST /api/p/{project}/reads", routeLocal, "the sync client posts these straight to the hub, never through here; the app's own viewer reads go out through desktop_reads.go instead, as human traffic"},
 }
@@ -755,6 +765,10 @@ func proxyHub(w http.ResponseWriter, r *http.Request, server string) {
 func streaming(r *http.Request) bool {
 	return strings.HasSuffix(r.URL.Path, "/events") ||
 		(r.Method == http.MethodGet && strings.HasSuffix(r.URL.Path, "/collab")) ||
+		// A websocket is long-lived in the way that matters here — the client
+		// asks for a connection, not an answer — even though it is an upgrade
+		// rather than a stream of frames the proxy can read.
+		strings.EqualFold(r.Header.Get("Upgrade"), "websocket") ||
 		strings.Contains(r.Header.Get("Accept"), "text/event-stream")
 }
 

docs/collab-provider-prd.md

@@ -129,31 +129,79 @@ member.
 
 ### Stage 0 — decide, then spike
 
-- [ ] The parsing-untrusted-updates decision above, recorded in §Status with
-      whatever conditions it carries
-- [ ] Pick the port. Criteria: CGO-free, embeds as `http.Handler`, V1 **and**
-      V2 wire compatibility, persistence we can point at our own storage, and
-      a maintainer who answers
-- [ ] Cross-language fixtures in CI: an update encoded by Go decodes in JS and
-      vice versa, both encodings, including a document with 10k+ ops
-- [ ] Memory per open document measured, and a cap decided — `maxRoomBytes`
-      exists for a reason and its replacement must be deliberate
-- [ ] A hub running multiple processes: how two instances holding the same
-      document converge, or why that is out of scope for now
+- [x] The parsing-untrusted-updates decision, recorded in §Status with the
+      conditions it carries
+- [x] **Port chosen: `github.com/reearth/ygo` v1.50.0.** CGO-free; ships
+      `crdt`, `awareness`, `provider/websocket`, `persistence` and `cluster`
+      (which is the multi-process answer, not a gap); 50 minor releases and
+      its own JS-compat suite in-tree. `Deln0r/ygo` was the other candidate
+      and is also credible — same wire claims, a Hocuspocus-compatible
+      `yserve` — and is the fallback if this one stalls.
+- [x] **Cross-language fixtures pass, against the exact `yjs` the browser
+      ships** (`node_modules/yjs`), both directions and both encodings:
+
+      | | V1 | V2 |
+      |---|---|---|
+      | JS reads a Go document (`ünïcode ✅`) | MATCH | MATCH |
+      | Go reads a JS document (`日本語 🎉`) | MATCH | MATCH |
+      | Go reads a JS document of 10,000 ops | MATCH | MATCH |
+
+      These move into CI in Stage 1 as a Go test with checked-in fixtures, so
+      a version bump that breaks the wire fails the build rather than the
+      editor.
+- [x] **Memory per open document measured** (`crdt.New()` + `YText`, heap
+      delta after GC, 50-200 documents per shape):
+
+      | shape | per document |
+      |---|---|
+      | 10 KB file inserted whole (loading a file) | 11.5 KB |
+      | 100 KB file inserted whole | 105.5 KB |
+      | 10 KB file typed **character by character** | **101.7 KB** |
+
+      The third row is the planning number, and it is the surprise: editing
+      costs ~10x the content, because every keystroke is its own item until
+      GC merges them. So an actively-edited document is ~100 KB and a hub
+      with 100 of them open is ~10 MB — against today's relay, which caps
+      each room's update log at 8 MiB on its own.
+
+      **The cap is therefore eviction, not bytes.** `maxRoomBytes` existed
+      because an append-only log grows without bound while a document does
+      not: the same text typed twice is one document and two log entries.
+      Idle documents are dropped the way `roomIdle` drops idle rooms, and the
+      file is the durable copy either way.
+- [x] A hub running multiple processes: `ygo/cluster` exists for exactly this.
+      Scope for Stage 1 is a single process with the cluster path unused and
+      named as the upgrade, rather than pretending the question does not
+      exist.
 
 **Success criteria:** a written go/no-go with the fixture results and the
-memory number in it. No Stage 1 work before that.
+memory number in it. **GO.** Fixtures pass, ~100 KB per actively-edited
+document, and the cap is an eviction policy rather than a byte ceiling.
 
 ### Stage 1 — the hub holds the document
 
-- [ ] The Go Yjs server embedded as an `http.Handler`, mounted behind the
-      existing `proj()` wrapper so folder permissions, org walls and read-only
-      membership apply unchanged
-- [ ] `filterJournal`'s sibling question answered: a reader who cannot see a
-      path must not receive its document
-- [ ] Documents persist across a hub restart, or are rebuilt from the file
-      deterministically
-- [ ] The old relay stays behind a config flag for one release
+- [x] `reearth/ygo`'s websocket server embedded as an `http.Handler`
+      (`ycollab.go`), mounted behind the existing `proj()` wrapper
+- [x] **The room name is the hub's, never the caller's.** ygo reads it from
+      `PathValue("room")` or the URL's last segment, so a caller who could
+      name the room would make the project id in the path decoration — any
+      member of any project could join any other project's document by asking
+      for its name. Verified failing without the guard
+- [x] Read-only membership applies as read-only, not as refusal: the route is
+      `PermRead` and the CONNECTION carries `ReadOnly`, so a member who may
+      read a file can open it and watch it being edited
+- [x] `filterJournal`'s sibling question: a path the caller cannot see is
+      **404, never 403** — the rule the viewer's `pathFilter` already applies,
+      because a 403 confirms the file is there
+- [x] Documents are **rebuilt from the file deterministically**, by the hub,
+      on room creation and before any client is attached (`fileSeed.LoadDoc`).
+      This is what retires the seed claim outright: there is nothing to claim
+      and nothing to race
+- [x] Seeding is bounded (`maxSeedBytes`), because a held document costs ~10x
+      its content in CRDT items
+- [x] The old relay is untouched and still mounted; this is beside it
+- [x] Wire fixtures in CI, produced by the frontend's own yjs and checked in
+      as bytes so CI needs no node
 
 **Success criteria:** two browsers converge through the hub with the bespoke
 provider deleted from the path; `e2e/concurrent-edit.spec.ts` still passes.
@@ -209,17 +257,24 @@ that went away.
 
 ## Status
 
-_Not started. Stage 0's decision is the gate; nothing below it is approved._
+_Stage 0's decision is made (below): the hub may parse client-supplied CRDT
+updates. Implementation proceeds._
 
 | Stage | State | Notes |
 |---|---|---|
-| 0 — decide + spike | not started | the untrusted-parsing call blocks everything |
-| 1 — hub holds the document | blocked on 0 | |
-| 2 — client stops being a provider | blocked on 1 | |
+| 0 — decide + spike | **done — GO** | reearth/ygo v1.50.0; JS<->Go fixtures pass V1+V2 incl. 10k ops; ~100 KB per edited doc |
+| 1 — hub holds the document | **done** | behind proj(); hub seeds from the file; wire fixtures in CI |
+| 2 — client stops being a provider | next | y-websocket, not hocuspocus: it is ygo's default mode |
 | 3 — server writes the file | blocked on 1 | |
 | 4 — delete the compensations | blocked on 2, 3 | |
 
 ### The decision
 
-- [ ] **May the hub parse client-supplied CRDT updates?** Decided by:
-      ___________  Date: ___________  Conditions: ___________
+- [x] **May the hub parse client-supplied CRDT updates?** **Yes.**
+      Decided by: Snow Lee. Date: 2026-09-19.
+
+      Conditions carried forward into Stage 0 and Stage 1 rather than left as
+      a sentiment: the decoder's inputs are bounded the way `maxInflatedPut`
+      bounds gzip, the server is mounted behind the existing `proj()` wrapper
+      so folder permissions and org walls apply unchanged, and the old relay
+      stays behind a config flag for one release.

go.mod

@@ -13,6 +13,7 @@ require (
 	github.com/jackc/pgx/v5 v5.10.0
 	github.com/mattn/go-isatty v0.0.20
 	github.com/modelcontextprotocol/go-sdk v1.8.0
+	github.com/reearth/ygo v1.50.0
 	github.com/restic/chunker v0.5.0
 	github.com/spf13/cobra v1.10.2
 	github.com/yuin/goldmark v1.8.2
@@ -63,6 +64,7 @@ require (
 	github.com/google/s2a-go v0.1.9 // indirect
 	github.com/googleapis/enterprise-certificate-proxy v0.3.16 // indirect
 	github.com/googleapis/gax-go/v2 v2.22.0 // indirect
+	github.com/gorilla/websocket v1.5.3 // indirect
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
 	github.com/jackc/pgpassfile v1.0.0 // indirect
 	github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect

go.sum

@@ -32,6 +32,8 @@ github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapp
 github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.55.0/go.mod h1:Mf6O40IAyB9zR/1J8nGDDPirZQQPbYJni8Yisy7NTMc=
 github.com/Netflix/go-expect v0.0.0-20220104043353-73e0943537d2 h1:+vx7roKuyA63nhn5WAunQHLTznkw5W8b1Xc0dNjp83s=
 github.com/Netflix/go-expect v0.0.0-20220104043353-73e0943537d2/go.mod h1:HBCaDeC1lPdgDeDbhX8XFpy1jqjK0IBG8W5K+xYqA0w=
+github.com/alicebob/miniredis/v2 v2.38.0 h1:nZAzCR+Lj+Vxk4ZXzm2NuKq2O33RXj1XxJ2e2uP9jiw=
+github.com/alicebob/miniredis/v2 v2.38.0/go.mod h1:TcL7YfarKPGDAthEtl5NBeHZfeUQj6OXMm/+iu5cLMM=
 github.com/aws/aws-sdk-go-v2 v1.42.0 h1:XvXMJTkFQtpBKIWZnmr9ZEOc2InWM2yldjXEJ/bymhA=
 github.com/aws/aws-sdk-go-v2 v1.42.0/go.mod h1:27+ACypSLljLAEKsCYOmrjKh83vuTRkuAe9Uv/3A4bg=
 github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.13 h1:p1BBrg/Hhp6uK7zpejeI8QFXHJeC/mynzi04Sl03k9g=
@@ -79,6 +81,8 @@ github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
 github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
+github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
 github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
 github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
 github.com/envoyproxy/go-control-plane v0.14.0 h1:hbG2kr4RuFj222B6+7T83thSPqLjwBIfQawTkC++2HA=
@@ -118,6 +122,8 @@ github.com/googleapis/enterprise-certificate-proxy v0.3.16 h1:F/VPrx0YPBdksZJQdC
 github.com/googleapis/enterprise-certificate-proxy v0.3.16/go.mod h1:9Yb0eAkH/Xqhvv3zbeKf/+wMJqCeocWc6KIhDvEAuYE=
 github.com/googleapis/gax-go/v2 v2.22.0 h1:PjIWBpgGIVKGoCXuiCoP64altEJCj3/Ei+kSU5vlZD4=
 github.com/googleapis/gax-go/v2 v2.22.0/go.mod h1:irWBbALSr0Sk3qlqb9SyJ1h68WjgeFuiOzI4Rqw5+aY=
+github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg=
+github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
 github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
 github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
 github.com/hinshun/vt10x v0.0.0-20220119200601-820417d04eec h1:qv2VnGeEQHchGaZ/u7lxST/RaJw+cv273q79D81Xbog=
@@ -154,6 +160,10 @@ github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10/go.mod h1
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
 github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/redis/go-redis/v9 v9.18.0 h1:pMkxYPkEbMPwRdenAzUNyFNrDgHx9U+DrBabWNfSRQs=
+github.com/redis/go-redis/v9 v9.18.0/go.mod h1:k3ufPphLU5YXwNTUcCRXGxUoF1fqxnhFQmscfkCoDA0=
+github.com/reearth/ygo v1.50.0 h1:AUnYWMv+t6o1k3xFPftTupbEGbWDOMhmhX6rG/VEEGk=
+github.com/reearth/ygo v1.50.0/go.mod h1:LpzEyyGErwVVVLNB+8rZfVy/zbZbhWjdkONAobCIlYA=
 github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
 github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
 github.com/restic/chunker v0.5.0 h1:1y+ut0MBduzxODJ298rhQCtESoEpj8v1hTydZlKaE1Y=
@@ -182,6 +192,8 @@ github.com/yosida95/uritemplate/v3 v3.0.2/go.mod h1:ILOh0sOhIJR3+L/8afwt/kE++YT0
 github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
 github.com/yuin/goldmark v1.8.2 h1:kEGpgqJXdgbkhcOgBxkC0X0PmoPG1ZyoZ117rDVp4zE=
 github.com/yuin/goldmark v1.8.2/go.mod h1:ip/1k0VRfGynBgxOz0yCqHrbZXhcjxyuS66Brc7iBKg=
+github.com/yuin/gopher-lua v1.1.1 h1:kYKnWBjvbNP4XLT3+bPEwAXJx262OhaHDWDVOPjL46M=
+github.com/yuin/gopher-lua v1.1.1/go.mod h1:GBR0iDaNXjAgGg9zfCvksxSRnQx76gclCIb7kdAd1Pw=
 go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
 go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
 go.opentelemetry.io/contrib/detectors/gcp v1.42.0 h1:kpt2PEJuOuqYkPcktfJqWWDjTEd/FNgrxcniL7kQrXQ=
@@ -202,6 +214,8 @@ go.opentelemetry.io/otel/sdk/metric v1.43.0 h1:S88dyqXjJkuBNLeMcVPRFXpRw2fuwdvfC
 go.opentelemetry.io/otel/sdk/metric v1.43.0/go.mod h1:C/RJtwSEJ5hzTiUz5pXF1kILHStzb9zFlIEe85bhj6A=
 go.opentelemetry.io/otel/trace v1.43.0 h1:BkNrHpup+4k4w+ZZ86CZoHHEkohws8AY+WTX09nk+3A=
 go.opentelemetry.io/otel/trace v1.43.0/go.mod h1:/QJhyVBUUswCphDVxq+8mld+AvhXZLhe+8WVFxiFff0=
+go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE=
+go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0=
 go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
 golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=

internal/webapp/compress.go

@@ -39,7 +39,23 @@ func gzipResponses(h http.Handler) http.Handler {
 		// socket" is how that loop re-downloads forever or resumes
 		// mid-stream. store.go negotiates its own encoding end to end and is
 		// the only thing that gets to decide there.
-		if strings.Contains(r.URL.Path, "/store/") ||
+		/* An upgrade is not a response to compress, it is a connection to
+		   hand over.
+
+		   A websocket handshake ends with the handler HIJACKING the socket,
+		   and a wrapper that does not implement http.Hijacker makes that
+		   impossible — the upgrade fails and the client sees a 500 it cannot
+		   explain. Browsers send Accept-Encoding on the handshake like any
+		   other request, so without this the co-editing document is
+		   unreachable from a browser and reachable from Go's own dialer,
+		   which sends no such header. That is exactly how it presented.
+
+		   Skipped rather than made hijackable: there is nothing to gzip here,
+		   and a Hijack method on a compressing writer is a trapdoor that
+		   returns a socket somebody may already have written a gzip header
+		   to. */
+		if isUpgrade(r) ||
+			strings.Contains(r.URL.Path, "/store/") ||
 			!acceptsGzip(r.Header.Get("Accept-Encoding")) {
 			h.ServeHTTP(w, r)
 			return
@@ -60,6 +76,21 @@ func gzipResponses(h http.Handler) http.Handler {
 	})
 }
 
+// isUpgrade reports whether this request asks to stop being HTTP. Both
+// headers are checked because Connection is a comma-separated list in the
+// wild ("keep-alive, Upgrade") and some proxies rewrite one but not the other.
+func isUpgrade(r *http.Request) bool {
+	if !strings.EqualFold(r.Header.Get("Upgrade"), "websocket") {
+		return false
+	}
+	for _, tok := range strings.Split(r.Header.Get("Connection"), ",") {
+		if strings.EqualFold(strings.TrimSpace(tok), "upgrade") {
+			return true
+		}
+	}
+	return false
+}
+
 // acceptsGzip is a token scan, not a substring test: "gzip;q=0" means the
 // client has explicitly refused it, and a Contains check reads that as yes.
 func acceptsGzip(header string) bool {
@@ -169,17 +200,20 @@ func (g *gzipWriter) close() {
 	}
 }
 
-/* compressible is an allowlist, and the direction matters: an unknown type is
-   left alone rather than compressed hopefully.
+/*
+compressible is an allowlist, and the direction matters: an unknown type is
+
+	left alone rather than compressed hopefully.
 
-   Everything this hub serves that is already compressed — images, fonts, PDFs,
-   the export tarball, blobs the sync wire encoded — is binary with a type of
-   its own, so a denylist would have to be complete to be safe and an allowlist
-   only has to be right. Re-compressing a PNG spends CPU to add bytes.
+	Everything this hub serves that is already compressed — images, fonts, PDFs,
+	the export tarball, blobs the sync wire encoded — is binary with a type of
+	its own, so a denylist would have to be complete to be safe and an allowlist
+	only has to be right. Re-compressing a PNG spends CPU to add bytes.
 
-   ponytail: no minimum size, so a 12-byte {"ok":true} gains ~20 bytes of gzip
-   framing. Add a buffer-until-threshold if tiny JSON responses ever dominate
-   a profile; they do not today, and the buffering is where the bugs live. */
+	ponytail: no minimum size, so a 12-byte {"ok":true} gains ~20 bytes of gzip
+	framing. Add a buffer-until-threshold if tiny JSON responses ever dominate
+	a profile; they do not today, and the buffering is where the bugs live.
+*/
 func compressible(contentType string) bool {
 	ct, _, _ := strings.Cut(contentType, ";")
 	ct = strings.ToLower(strings.TrimSpace(ct))