OCR Foundry

Synthetic OCR training data generator with a legacy Khmer/English path and opt-in multilingual whole-line rendering. Use the ocr-foundry command; existing khocr-gen commands keep working through the same CLI. The v0.2 distribution ocr-foundry installs both executables; Python imports remain khocr_gen.

See branding and compatibility for the name transition, version output, and compatibility contracts. Links to https://github.com/0x1d1e/ocr-foundry are current. v0.2.0 is the ocr-foundry release, and khocr-gen on PyPI is the deprecated redirect. See the v0.2 release plan.

It renders corpus text with configurable fonts and layouts, applies OCR-oriented degradation, and writes datasets as raw images, LMDB, or both.

Features

• Mixed Khmer/English rendering with per-span font selection.
• Variable and bucketed line heights without glyph clipping.
• Text decorations, visual effects, and 27 augmentation methods.
• Khmer text normalization with khmernormalizer.
• Multiprocess generation and deterministic dataset splitting.
• Optional Rust acceleration through khocr-gen-core.

Install

pip install ocr-foundry

khocr-gen on PyPI is the deprecated redirect: release 0.2.0 contains no code and depends on ocr-foundry, while khocr-gen==0.1.13 is the last standalone release. If 0.1.13 is installed, upgrade or remove it before installing ocr-foundry, because both distributions would otherwise provide the khocr-gen command.

Python 3.12+ is required.

To work from source:

git clone https://github.com/0x1d1e/ocr-foundry.git
cd ocr-foundry
uv sync --extra dev               # pure-Python dev tools
uv sync --extra dev --extra rust  # with optional native Rust extension

The rust extra builds the optional native acceleration extension.

Generate a dataset

Prepare fonts:

fonts/
├── khmer/
└── english/

Prepare a UTF-8 corpus with one sample per line:

សួស្តី
Hello World
ស្វាគមន៍ Welcome

Then run:

ocr-foundry generate \
  --corpus corpus/corpus.txt \
  --fonts fonts \
  --output data \
  --storage lmdb

For a seeded run with your own licensed corpus and fonts, use the calibrated multilingual recipe (run from the repository root):

ocr-foundry generate --config configs/multilingual-realistic.yml \
  --corpus /path/to/licensed-lines.txt --fonts /path/to/fonts --output /tmp/multilingual

The recipe sets seed: 42, 25% clean samples, a 12 px glyph-ink height guard, script intensity factors Khmr: 0.75 and Latn: 0.90, and mild rotation (0.05) and morphological (0.08) method weights. --seed N controls both splitting and per-sample rendering/augmentation, including variable heights; for fixed inputs, fonts, configuration, and software/runtime, output images and labels are reproducible across worker counts. Seeded generation uses Python augmentation fallbacks instead of native Rust methods. --seed none disables per-sample seeding; a seed alone cannot guarantee identical bytes across changed fonts or dependency versions. CLI flags override YAML values.

Multilingual generation and real evaluation

ocr-foundry evaluate --reference reference.jsonl --predictions predictions.jsonl \
  --normalization NFC
ocr-foundry benchmark verify --lock benchmarks/real_doc_v1/benchmark.lock.json
ocr-foundry benchmark compare --lock benchmarks/real_doc_v1/benchmark.lock.json \
  --baseline tesseract-5.5.4 --format markdown

Multilingual mode requires Pillow RAQM. Optional cluster-aware font fallback (--cluster-font-fallback) preserves combining marks and Khmer coeng sequences and aligns multi-run text to a shared baseline. The logical label follows the selected normalization policy (NFC in the calibrated recipe), without visual reordering; it need not equal the raw input bytes. Arbitrary font pools and RTL direction are supported. JSONL corpus manifests ingest up to 10 metadata fields (script, domain, source_uri, license, retrieved_at, source_sha256, normalization_policy, consent_or_restrictions), propagate complete provenance into provenance.jsonl sidecars, track skipped lines, and record font_sha256, runtime_version, and original_text in metadata.jsonl.

The realistic recipe disables rare oversampling and foreground crops. Its glyph-mask height guard measures ink rather than the whole-line bounding box; script intensity factors adjust degradation per ISO 15924 script.

The offline evaluator scores externally produced, ID-matched OCR predictions; it does not include a recognizer. The frozen real_doc_v1 suite has 103 lines from six documents (including one digital vector baseline), with a Tesseract 5.5.4 prediction baseline. Its tracked crop paths are small placeholder bytes, not runnable scan images. To freeze the suite again, use an unused temporary path and remove the lock only in the copy:

cp -a benchmarks/real_doc_v1 /tmp/real_doc_v1
rm /tmp/real_doc_v1/benchmark.lock.json
ocr-foundry benchmark freeze --manifest /tmp/real_doc_v1/manifest.json

Freeze refuses to overwrite the tracked lock.

The separately reported Model Version 6 experiment scored 757 fixed real crops with a pretrained-model baseline and synthetic continuation arms; it is not a run in real_doc_v1, and its CERs are not comparable to the Tesseract baseline across those disjoint cohorts. To evaluate your own OCR system, obtain rights-reviewed crops and run it externally to write predictions for every ID. See the evaluation protocol and suite commands.

Page layout composition (ocr-foundry layout) supports document-disjoint pages, multi-column/RTL flow, glyph-positioned whitespace word annotations, native RAQM vertical writing and paginated table/cell hierarchy. All geometry follows affine, perspective and elastic page transforms. New modes are opt-in; an explicit --config configs/layout.yml demonstrates their options.

See the capability matrix, page layout contract, evaluation protocol, and experiment details.

Commands

ocr-foundry evaluate   # score supplied OCR predictions offline
ocr-foundry benchmark  # freeze, verify and compare offline benchmark suites
ocr-foundry generate   # generate a dataset
ocr-foundry combine    # merge generated datasets
ocr-foundry layout     # compose document pages with geometry annotations
ocr-foundry verify     # render augmentation samples
ocr-foundry view       # inspect or extract LMDB data

Run ocr-foundry <command> --help for command-specific options.

Development

uv sync --extra dev               # pure-Python development
uv sync --extra dev --extra rust  # or with native Rust acceleration

uv run ruff check .
uv run pytest --cov=khocr_gen --cov-report=term-missing
uv run ty check

For the Rust extension:

cd rust
cargo fmt --check
cargo clippy --all-targets -- -D warnings
cargo build --release

CI tests Python 3.12, 3.13, and 3.14 with native Rust acceleration and pure-Python fallbacks. Ruff, ty, Rust formatting, Clippy, tests, and the Rust release build are enforced by the workflow. Repository rulesets and required-check settings must be verified separately before release.

Documentation

• Documentation index
• Repository transfer and v0.2 release plan
• Getting Started
• Configuration
• CLI Reference
• Offline evaluation and benchmark suites
• Augmentation
• Rust acceleration
• GitHub wiki

Wiki source lives in wiki/ and can be published with:

./scripts/publish_wiki.sh

License

MIT