Repository navigation
Store conversion parts a unit a row, from Blob or the convert job - #824
Conversation
src_conversions gets a number from a sequence, added as a nullable column with its default so the seeded table is not rewritten under the lock, and src_conversion_units holds a parts conversion's units. The seed stores docling-parts 1 from Docling's output in Blob with --derivation docling-parts, and a convert job stores parts beside the pages it makes: Docling's from its output, a PDF's text layer as text-layer-parts 1. Storing parts ends no wait until a pipeline asks for them (#635).
…ement A bug in a parts derivation was stored as a permanent conversion_failed: now only output that is not a DoclingDocument, or not JSON, is a refusal. Any other error skips the file in a seed and stores no parts in a convert job, logged by the error's name, and the pages are stored all the same. replaceable matched stored conversions on converter only, so a file whose only Docling row held parts was never replaced; it now matches the derivation too. A PDF is known by its bytes, since an upload in Blob can be application/octet-stream. Tests cover the conversion number of a row stored again, parts stored before pages, and U+0000 left out before offsets are counted.
… fix the docs on them A PDF Docling refuses gets text-layer parts by its bytes too, and output that is not JSON is a NotDoclingDocument; both now have a test. The byte check's reason is a source that passes on a server's Content-Type, not an upload, and ADR 0042 says the Blob route alone needs no download.
|
Ready for review. Blind review, a claim-checking review and a check of the fixes are done and CI passes. Open: the acceptance run in test (parts seed of prefixes 00 to 06, bytes a page, migration time) follows the deploy; the convert job that asks for parts belongs to #635; #823 changes the same |
Model reviewNo findings. WalkthroughThe pull request starts storing conversion parts (ADR 0055, issue 634). A migration gives
Reviewed at 4e7262e with claude-opus-5-5 in 2.2 min. Pushes are checked against open findings, and fixed ones are resolved. Add the label |
…sion-parts-unit-row # Conflicts: # docs/adr/0055-conversion-parts.md # docs/adr/README.md
…sion-parts-unit-row # Conflicts: # apps/worker/src/knowledge-conversions.test.ts # apps/worker/src/knowledge-handlers.ts # docs/adr/0042-the-knowledge-database.md # packages/providers-pgvector/src/knowledge-conversions.ts # packages/providers-pgvector/src/partitions.ts
What and why
doclingPartswas a function nobody calledKnowledge server
src_conversions.id: a nullable column, then a sequence default, so no table rewriteidin the same migration transactionsrc_conversion_units, rows cascade with their conversionsrc_conversions_id_seqknowledge server migratedwith its millisecondsStored conversions
StoredConversion.units:storewrites them in its transaction,findreturns themWhere parts come from
knowledge.seedwithderivation: docling-partsderives them from Blob, with no fetch and no converter callnpm run conversions:seed -- --derivation docling-parts --prefixes 00,01sends only those batchesDoclingDocumentis stored as aconversion_failedrefusal; any other error skips the file, logged by its nametext-layer-parts1 (textLayerParts), one part a page; Word files get none%PDF-): a source can pass on a server's Content-Type, such asapplication/octet-streamreplaceablematches the derivation, so a file's parts never count as its Docling pagesDocs
docs/development.mddocs/architecture/modules/ingest.yamlandworker.yaml, and the System AtlasAdded beyond the issue
knowledge server migratedlog line, for the migration time the acceptance asks for--prefixeson the seed script andsendSeed, to ask for parts of about 1,100 filesBefore
After
More background
waithave no derivation and asking with today's key would loopdocling-parts1 is stored with printed labels (Give a page its printed number as the unit's label #633, merged before this PR) and without tablehtml(Build a table part's HTML from Docling's cells #632): adding html later makesdocling-parts2, derived again from Blob by a seed per converter version and settings (the script's defaults are the 2.123.1 export)id; one stored again gets a number (coalesce(id, nextval(...)))Part of #634: the acceptance run in test (parts for about 1,100 seeded Kudos files, bytes a page, migration duration) follows the deploy.
Risk
Sensitive: a migration on the knowledge server, inside its 5-second lock timeout. It adds a nullable column, a sequence default and a unique index on
src_conversions(no rewrite, checked by the table's file node), and a new table.Undo
Revert, then
delete from src_conversions where derivation like '%-parts'on the knowledge server: main'swaithas no derivation filter and would read a parts row as a stored conversion. The column, sequence, index andsrc_conversion_unitscan stay; dropping them rewrites nothing.How it was tested
npm run checklocally: everything but the tests passes; 12 tests outside the diff timed out at a load average near 30. CI passes on the build commit (test 1 to 4, static, image, CodeQL)The new tests, each red first (against main's code, or with its line removed):
packages/providers-pgvector/src/partitions.test.ts)packages/ingest/src/text-layer-parts.test.ts)apps/worker/src/knowledge-conversions.test.ts)packages/ingest/src/docling-output.test.ts)Existing conversion tests now read only pages rows through their helpers, since a converted PDF also has a parts row
Not tested: the runtime's wiring of the log line (one line; the engine callback is tested)
Run against Testcontainers Postgres 17 with pgvector, not yet against the test environment
npm run checkpasses (in CI; locally only timeouts under load, outside the diff)Checklist
packages/providers-*docs/architecture/modulesare updated if a contract, boundary or module changed