Skip to content

Store conversion parts a unit a row, from Blob or the convert job - #824

Merged
kentare merged 5 commits into
mainfrom
feat/634-store-conversion-parts-unit-row
Nov 19, 2026
Merged

kentare merged 5 commits into
mainfrom
feat/634-store-conversion-parts-unit-row

Conversation

@kentare

@kentare kentare commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

What and why

Knowledge server

  • src_conversions.id: a nullable column, then a sequence default, so no table rewrite
  • Unique index on id in the same migration transaction
  • New table src_conversion_units, rows cascade with their conversion
  • The writer is granted the table and src_conversions_id_seq
  • The worker logs knowledge server migrated with its milliseconds

Stored conversions

Where parts come from

  • knowledge.seed with derivation: docling-parts derives them from Blob, with no fetch and no converter call
  • npm run conversions:seed -- --derivation docling-parts --prefixes 00,01 sends only those batches
  • Output that is not a DoclingDocument is stored as a conversion_failed refusal; any other error skips the file, logged by its name
  • A convert job stores Docling's parts from the output that makes its pages, before the pages
  • A PDF's text layer gets text-layer-parts 1 (textLayerParts), one part a page; Word files get none
  • A PDF is known by its bytes (%PDF-): a source can pass on a server's Content-Type, such as application/octet-stream
  • A bug in deriving parts stores none; the pages are stored all the same
  • replaceable matches the derivation, so a file's parts never count as its Docling pages

Docs

  • ADR 0055 status and Consequences, ADR 0042 (status, stored conversions row, convert and seed queues), docs/development.md
  • docs/architecture/modules/ingest.yaml and worker.yaml, and the System Atlas

Added beyond the issue

  • The knowledge server migrated log line, for the migration time the acceptance asks for
  • --prefixes on the seed script and sendSeed, to ask for parts of about 1,100 files

Before

flowchart LR
  blob[(Blob: Docling output)] -->|knowledge.seed| pages[src_conversions<br/>docling-pages 1]
  convert[knowledge.convert] -->|Docling or text layer| pages
Loading

After

flowchart LR
  blob[(Blob: Docling output)] -->|knowledge.seed| pages[src_conversions<br/>docling-pages 1]
  blob -->|knowledge.seed<br/>derivation docling-parts| parts[src_conversions<br/>docling-parts 1]
  convert[knowledge.convert] -->|parts first| parts
  convert --> pages
  convert -->|PDF, text layer| tl[src_conversions<br/>text-layer-parts 1]
  parts --> units[(src_conversion_units)]
  tl --> units
  class parts,tl,units changed
  classDef changed stroke:#0969da,stroke-width:3px
Loading
More background
  • Decisions from the grilling: parts are made by the seed and at convert time; a convert job asked for parts (for files that have pages but no parts) comes with the pipeline setting in Chunk a PDF from its conversion parts, a page a chunk without furniture #635, since the convert key and wait have no derivation and asking with today's key would loop
  • docling-parts 1 is stored with printed labels (Give a page its printed number as the unit's label #633, merged before this PR) and without table html (Build a table part's HTML from Docling's cells #632): adding html later makes docling-parts 2, derived again from Blob by a seed per converter version and settings (the script's defaults are the 2.123.1 export)
  • The issue's run that asks for parts of 1,000 files is the parts seed: no ingestion run can ask for parts until Chunk a PDF from its conversion parts, a page a chunk without furniture #635
  • A failure to store parts (a database error) fails the convert job, which is tried again, as a failure to store pages does
  • A pre-migration row keeps a null id; one stored again gets a number (coalesce(id, nextval(...)))
  • Parts are stored before pages in a convert job, so the index jobs the pages send find both
  • A parts derivation error is logged with the file, the derivation and the error's name, never its message: a JSON parse error's message quotes the output
  • Docling output is read with U+0000 stripped before deriving, so offsets count the stored text
  • Units are inserted 200 a statement

Part of #634: the acceptance run in test (parts for about 1,100 seeded Kudos files, bytes a page, migration duration) follows the deploy.

Risk

Sensitive: a migration on the knowledge server, inside its 5-second lock timeout. It adds a nullable column, a sequence default and a unique index on src_conversions (no rewrite, checked by the table's file node), and a new table.

Undo

Revert, then delete from src_conversions where derivation like '%-parts' on the knowledge server: main's wait has no derivation filter and would read a parts row as a stored conversion. The column, sequence, index and src_conversion_units can stay; dropping them rewrites nothing.

How it was tested

  • npm run check locally: everything but the tests passes; 12 tests outside the diff timed out at a load average near 30. CI passes on the build commit (test 1 to 4, static, image, CodeQL)

  • The new tests, each red first (against main's code, or with its line removed):

    • "the migration adds the conversion number without rewriting src_conversions", "the migration runs twice without error", "the worker logs how long the knowledge server migration took" (packages/providers-pgvector/src/partitions.test.ts)
    • "text-layer-parts gives one text part a page, its region the whole page" (packages/ingest/src/text-layer-parts.test.ts)
    • "a conversion stored before the migration keeps no number and still reads", "a parts conversion reads back with its units in unit order", "storing a parts conversion twice leaves one set of unit rows", "deleting a stored conversion deletes its units", "U+0000 is left out of stored part text", "storing parts ends no wait, which is for pages", "parts are derived from the Blob copy with no fetch and no converter call", "an output that is not a DoclingDocument is stored as a parts refusal and not derived again", "a seed job sent before this release still stores pages", "a file with no Blob copy is converted once and gets both pages and parts", "text-layer-parts gives parts to PDFs only", "a conversion stored before the migration gets a number when stored again", "a parts derivation that throws on a DoclingDocument stores no refusal and logs its error name", "a convert job whose parts derivation throws stores the pages and no parts", "a convert job stores a file's parts before its pages", "a file whose only Docling conversion is its parts is still replaced", "a PDF fetched as application/octet-stream gets text-layer parts", "a PDF fetched as application/octet-stream that Docling refuses gets text-layer parts" (apps/worker/src/knowledge-conversions.test.ts)
    • "U+0000 is left out before offsets are counted", "output that is not JSON is not a DoclingDocument" (packages/ingest/src/docling-output.test.ts)
  • Existing conversion tests now read only pages rows through their helpers, since a converted PDF also has a parts row

  • Not tested: the runtime's wiring of the log line (one line; the engine callback is tested)

  • Run against Testcontainers Postgres 17 with pgvector, not yet against the test environment

  • npm run check passes (in CI; locally only timeouts under load, outside the diff)

Checklist

  • Every new rule has a test whose name states it
  • User-facing text is in the i18n catalogues, in bokmål and English, with no em-dashes
  • Vendor SDKs stay in packages/providers-*
  • ADRs, docs and docs/architecture/modules are updated if a contract, boundary or module changed
  • No secrets, personal data or local paths

src_conversions gets a number from a sequence, added as a nullable column
with its default so the seeded table is not rewritten under the lock, and
src_conversion_units holds a parts conversion's units. The seed stores
docling-parts 1 from Docling's output in Blob with --derivation
docling-parts, and a convert job stores parts beside the pages it makes:
Docling's from its output, a PDF's text layer as text-layer-parts 1.
Storing parts ends no wait until a pipeline asks for them (#635).
…ement

A bug in a parts derivation was stored as a permanent conversion_failed:
now only output that is not a DoclingDocument, or not JSON, is a refusal.
Any other error skips the file in a seed and stores no parts in a convert
job, logged by the error's name, and the pages are stored all the same.
replaceable matched stored conversions on converter only, so a file whose
only Docling row held parts was never replaced; it now matches the
derivation too. A PDF is known by its bytes, since an upload in Blob can
be application/octet-stream. Tests cover the conversion number of a row
stored again, parts stored before pages, and U+0000 left out before
offsets are counted.
… fix the docs on them

A PDF Docling refuses gets text-layer parts by its bytes too, and output that is not JSON is a NotDoclingDocument; both now have a test. The byte check's reason is a source that passes on a server's Content-Type, not an upload, and ADR 0042 says the Blob route alone needs no download.
@kentare

kentare commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

Ready for review. Blind review, a claim-checking review and a check of the fixes are done and CI passes. Open: the acceptance run in test (parts seed of prefixes 00 to 06, bytes a page, migration time) follows the deploy; the convert job that asks for parts belongs to #635; #823 changes the same replaceable query and ADR 0042 status line, so one of the two rebases on the other.

@kentare
kentare marked this pull request as ready for review November 19, 2026 13:48
@kentare
kentare added this pull request to the merge queue Nov 19, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 findings, 0 Important. The summary comment has the overview.

@github-actions

Copy link
Copy Markdown
Contributor

Model review

No findings.

Walkthrough

The pull request starts storing conversion parts (ADR 0055, issue 634). A migration gives src_conversions a numbered id and adds src_conversion_units, which holds one row per page. knowledge.seed gains a docling-parts batch that derives parts from the Docling output already in Blob storage, and knowledge.convert now stores Docling parts, or for a PDF its text-layer parts, before the pages. The wait, replacement and lookup queries filter by derivation so that a file's parts never count as its pages, and ADRs 0042 and 0055, the ADR index, the module atlas and docs/development.md are updated to match.

Area What changes
apps/worker The seed job takes an optional docling-parts derivation: it stores parts from Blob, stores output that is not a DoclingDocument as a conversion_failed refusal, and skips and logs other derivation errors. The convert job stores Docling or PDF text-layer parts before the pages, including after a Docling refusal, and logs a derivation bug by the error's name only.
packages/ingest Adds readDoclingOutput, asDoclingDocument and NotDoclingDocument, which strip U+0000 before offsets are counted. Adds the textLayerParts derivation and exports looksLikePdf.
packages/port-knowledge Adds textLayerPartsDerivation, partsDerivations, isPartsDerivation and an optional units field on StoredConversion, and updates the find and store contract docs.
packages/providers-pgvector The migration adds a nullable id with a sequence default and a unique index, the src_conversion_units table with a cascading delete, and grants for the writer. store replaces a conversion's units in its own transaction and ends no wait for parts, find returns units, wait ignores parts rows, replaceable matches by derivation, and the engine reports how long the migration took.
packages/jobs and scripts The seed payload becomes a union with an optional derivation: docling-parts, which also changes the job key. sendSeed and scripts/conversions-seed.ts accept a list of prefixes and a --derivation flag, and check both.
packages/composition Logs knowledge server migrated with the milliseconds the migration took.
docs ADR 0042 and ADR 0055 statuses, consequences, derivation text and test list, the ADR index row, the module YAMLs, the generated atlas and the seeding section of docs/development.md now describe the stored parts.

Reviewed at 4e7262e with claude-opus-5-5 in 2.2 min. Pushes are checked against open findings, and fixed ones are resolved. Add the label review again to review new commits.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 19, 2026
…sion-parts-unit-row

# Conflicts:
#	docs/adr/0055-conversion-parts.md
#	docs/adr/README.md
…sion-parts-unit-row

# Conflicts:
#	apps/worker/src/knowledge-conversions.test.ts
#	apps/worker/src/knowledge-handlers.ts
#	docs/adr/0042-the-knowledge-database.md
#	packages/providers-pgvector/src/knowledge-conversions.ts
#	packages/providers-pgvector/src/partitions.ts
@kentare
kentare enabled auto-merge November 19, 2026 17:07
@kentare
kentare added this pull request to the merge queue Nov 19, 2026
Merged via the queue into main with commit 4203b45 Nov 19, 2026
18 checks passed
@kentare
kentare deleted the feat/634-store-conversion-parts-unit-row branch November 19, 2026 17:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant