Fix data preview "Server Error" for collections with more than 25 journals#2014
Open
jshearer wants to merge 1 commit into
Open
Fix data preview "Server Error" for collections with more than 25 journals#2014jshearer wants to merge 1 commit into
jshearer wants to merge 1 commit into
Conversation
…urnals The broker's journal `List` RPC is server-streaming and caps each `ListResponse` frame at 25 journals (gazette/core#466, down from 1000), so a collection split more than 25 ways returns its listing as several newline-delimited frames. `getJournals` parsed the body with a single `response.json()`, which failed on the second frame and surfaced as the generic "Server Error", leaving the preview with no journal to read. * Bump vendored `data-plane-gateway` to a build whose `JournalClient.list()` consumes the stream and merges every frame (estuary/data-plane-gateway#51). * Call `journalClient.list()` directly and remove the now-dead `getJournals` and `isNestedProtocolListResponse` helpers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Data Preview showed
Server Errorinstead of sample documents for any collection split into more than 25 journals.Root cause
Gazette's
ListRPC is server-streaming and returns the journal list as one or more newline-delimited{result: ListResponse}frames. gazette/core#466 lowered the per-frame cap (maxJournalsPerListResponse) from 1000 to 25, so once that build shipped, collections split more than 25 ways began coming back as multiple frames. The UI'sgetJournalsparsed the body with a singleresponse.json(), which handles only one frame and threw on the second. That surfaced as the genericFETCH_DEFAULT_ERROR("Server Error"), and because the list call failed the preview never selected a journal to read.Fix
estuary/data-plane-gateway#51 fixes
JournalClient.list()to drain the stream and merge every frame (matching whatread()already does), returning the fullProtocolListResponse. This PR vendors that build and migrates onto it:data-plane-gatewaytarball.useJournalsForCollectioncallsjournalClient.list()and reads.journalsdirectly.getJournalsandisNestedProtocolListResponsehelpers.Data preview still samples a single journal as before; handling multiple frames just lets the list parse.
Depends on estuary/data-plane-gateway#51.