Repository navigation
Conversation
…caller vouches its rows are fresh put_graph reads back every existing node id and (from, to) pair of a target table before copying, so rows that already exist take the upsert path. A caller that feeds one fresh graph as many chunks pays that full scan on every chunk after the first, quadratic in the chunk count: soc-Pokec in 5 M-edge chunks had not finished loading after 65 minutes on the reference host, and com-Orkut's 24 chunks would take days. set_bulk_load_trusts_fresh_rows(true) turns the read-back off for a caller that knows its chunks carry no stored key; every row is copied, and a duplicate key in the input is the caller's error. Off by default; single-row writes are unaffected. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sak2FQcdgs5pkL2ruKrSUZ
…rried endpoints A chunked load carries each edge batch's endpoint nodes again, so with the read-back off the node COPY failed on a duplicate primary key on the second chunk. The store now remembers, per table, the node ids it copied in this mode and skips them; relationship rows are still all copied. Test: nodes first, then four edge chunks each carrying their endpoints. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sak2FQcdgs5pkL2ruKrSUZ
…l edges The fresh-rows path still keyed the copy batch by (from, to), so a multigraph chunk collapsed its parallel edges to one per pair while put_graph reported every row written: on sx-stackoverflow the strain harness then found the hub's degree at 41,348 against the oracle's 104,863. With the read-back off there is nothing to merge against, so every row is copied in input order; the read-back path keeps its one-row-per-pair upsert. Test: two parallel edges with distinct ids land and read back as two. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sak2FQcdgs5pkL2ruKrSUZ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #6; one commit.
put_graphon the Ladybug adapter reads back every existing node id and(from, to)pair of a target table before copying, so rows that already exist take the per-row upsert path (kept in #7, skipped only when the table is empty). A caller that feeds one fresh graph as many chunks pays that full scan on every chunk after the first — quadratic in the chunk count. In the strain harness, which loads large graphs in 5 M-edge chunks, soc-Pokec had not finished loading after 65 minutes on the reference host (store file static, one thread iterating rows), and com-Orkut's 24 chunks would have taken days.LadybugGraphStore::set_bulk_load_trusts_fresh_rows(true)turns the read-back off for a caller that knows its chunks carry no key the store already holds: every row is copied, and a duplicate key in the input is the caller's error (a node COPY fails on it; a relationship table takes it as a parallel edge). Off by default; single-row writes and the non-bulk paths are unaffected. 19/19 crate tests pass (eigen).The harness sets it (
AG_LADYBUG_BULK=readbackrestores the read-back), tags such rowsladybug_bulk=freshso earlier rows are not superseded silently, and pins this head for the Ladybug ladder.🤖 Generated with Claude Code
https://claude.ai/code/session_01Sak2FQcdgs5pkL2ruKrSUZ