Repository navigation
Description
Goal
Add a committed, deterministic vision-path integration fixture for the typed digital-export workflow (a real input shape per the plan's Context: "a digital form is filled in and exported to PDF"), plus the pytest -m integration harness that scores vision output against its golden — and iterate on fixes/changes to the extraction pipeline until those tests pass.
This is a new tier alongside the three in 0001 §Test data acquisition. It complements — does not replace — Tier 2 (real handwritten scans + REMS goldens, local) and Tier 3 (pseudonymized realistic scans, deferred). It is explicitly easier than handwriting; its job is baseline-quality stabilization and regression-catching for straightforward cases, not handwriting robustness.
Scope
docs/design/0002-synthetic-vision-fixtures.md— amends "Test data acquisition" to add the typed-scan tier and records why a clean typed scan is a legitimate production shape rather than the cautioned-against easy fake.- Generator: render the Faker-seeded form to a flattened (no-widget) image-only PDF, fill the currently-blank fields (
times_worked_position,mentor,level,successful) with a deterministic mix incl. signed-off / crossed-out / blank rows, and emit an exact golden JSON. - Harness:
pytest -m integrationscores vision output on the documented(official, position, successful)tuple contract (modulo position-name normalization), skipped unless Ollama is present; self-manages the daemon viaOllamaDaemon;--vision-model/--pull-modelsCLI options. - Iterate until green. The accuracy bar for this easy typed input is 100% (humans clear even the hardest handwritten forms at ~99%, so tolerance here is essentially zero). The harness is expected to fail on first run — that is its job. This issue stays open and tracks the prompt/pipeline/model fixes needed to make it pass honestly. Do not lower the floor to go green. If a fix turns out to need a major design change, spin that out into its own issue + design doc.
Acceptance
pytest -m integration passes at ACCURACY_FLOOR = 1.0 on the committed synthetic scan, with the daemon auto-managed and no manual setup beyond an installed Ollama + pulled vision model.
Relationship to #16
Re-scopes #16's harness onto a committable synthetic golden instead of the rems-sync-dependent real-scan path. #16 stays open for the Tier-2 real-scan path.
Activity
gavinbee commentedon May 30, 2026
First real-model run (qwen2.5vl:7b, 2-page synthetic scan)
Harness is wired and self-manages the daemon —
pytest -m integrationran the full vision pipeline + scoring in ~90s with no manual setup. As expected at the 100% bar, it fails, surfacing concrete quality issues to fix:test_tuple_contract_accuracy→ 0.77 (17/22):Chief Timer, wantTimer(likely vertical bleed from an adjacent "Chief Timer" row).successfulreadTrue, want False (struck-through row not honored — holistic-judgment miss).successfulreadTrue, want None (blank initials + mentor assigned read as signed off).test_meet_header→ competition_name came back'Osborn Open Open 2026', want'Osborn Open 2026'(doubled-token hallucination).test_finds_every_officialpasses (10/11 matched by name).Next: iterate on prompt/pipeline to close these. Decision recorded: keep
ACCURACY_FLOOR = 1.0; these tests merge red and serve as the quality gate until fixed. Major fixes that need a new design doc get their own issue.Commit blank template in-tree and use Arabic levels in the synthetic …
Move canonical blank template to top-level templates/swim_ontario_v1.pdf
Point both fixture generators at templates/swim_ontario_v1.pdf
gavinbee commentedon May 31, 2026
Handoff: picking up the quality iteration on another machine
The fixtures + harness half of this issue landed in PR #49 (squash
8a5afc4onmain). What remains is the expanded scope: iterate on the vision pipeline until the integration harness passes at its 100% accuracy floor. This comment is a self-contained brief for an agent starting cold on a different computer.Current state (all on
main)tests/test_integration_vision.py— the harness. 3 tests:test_finds_every_official,test_tuple_contract_accuracy,test_meet_header. Gated behind@pytest.mark.integration(skipped unless-m integration).tests/fixtures/vision_scan/make_synthetic_scan.py— deterministic generator (Faker seed 42) that flattens the blank form into an image-only PDF and writes a matching golden.tests/fixtures/vision_scan/session_1_evals_scan.pdf+session_1_evals_scan.golden.json— the committed fixture + golden (levels are Arabic 2–5).templates/swim_ontario_v1.pdf— the canonical blank Swim Ontario form (top-level, first-class asset; both generators resolve it via sharedTEMPLATE_ID).tests/conftest.py— adds pytest options--vision-model <tag>(defaultvision_extract.DEFAULT_VISION_MODEL=qwen2.5vl:7b) and--pull-models(allowollama pullof a missing model; off by default → skip with a hint).How to run it
The
parsedfixture self-manages the Ollama daemon viasrc.ollama_runtime.OllamaDaemon— it startsollama serveonly if one isn't already up and stops it on the way out. No manualollama serveneeded. On a box without Ollama / the model, the suite skips cleanly (with aollama pullhint) rather than failing. Requires a GPU box with the model pulled to actually exercise.The accuracy floor is deliberately 1.0 — do NOT lower it
This fixture is the easiest possible input: typed, clean, digital-export scan — no handwriting, skew, or smudges. Humans clear even the hardest handwritten forms at ~99%, so tolerance for error here is essentially zero. The tests are expected to fail until the underlying extraction-quality bugs are fixed — that is their job. The fix is in the model/prompt/pipeline, never in the threshold. (Rationale is documented at length in the module docstring and the
ACCURACY_FLOORcomment.)Known failures to close (representative — last real-model run, model output is not deterministic)
James MayosuccessfulgotTrue, wantFalse. The model isn't treating a crossed-out row as not-successful.Rodney TrujillosuccessfulgotTrue, wantNone. Should be ambiguous/null, not a positive sign-off.Daniel Adamsposition gotChief Timer, wantTimer. Adjacent-row label bleeding into the read.Angela Carternot found in output at all.competition_namegotOsborn Open Open 2026, wantOsborn Open 2026.The
successfulcontract (whole-row holistic judgment →true/false/null) lives insrc/vision_extract.py(_BASE_INSTRUCTIONS,build_prompt,_build_json_schema,_parse_successful). That's the first place to iterate for failures 1–2. Failures 3–5 are read/parse quality (prompt + JSON-schema constraints).Workflow notes
main(e.g.48-improve-vision-quality), one PR referencing the issue. Imperative commit messages, no conventional-commit prefixes,Co-Authored-Bytrailer..venv/Scripts/python.exe); project targets 3.12.make_synthetic_scan.pyinto a config-driven generator whose input is golden-shaped (rows + header config). Not part of this issue, but the two interact — keep the generator's golden output stable so the harness fixture doesn't churn.