Skip to content

Synthetic typed-scan vision fixtures + integration harness #48

Open

Description

@gavinbee

Goal

Add a committed, deterministic vision-path integration fixture for the typed digital-export workflow (a real input shape per the plan's Context: "a digital form is filled in and exported to PDF"), plus the pytest -m integration harness that scores vision output against its golden — and iterate on fixes/changes to the extraction pipeline until those tests pass.

This is a new tier alongside the three in 0001 §Test data acquisition. It complements — does not replace — Tier 2 (real handwritten scans + REMS goldens, local) and Tier 3 (pseudonymized realistic scans, deferred). It is explicitly easier than handwriting; its job is baseline-quality stabilization and regression-catching for straightforward cases, not handwriting robustness.

Scope

  • docs/design/0002-synthetic-vision-fixtures.md — amends "Test data acquisition" to add the typed-scan tier and records why a clean typed scan is a legitimate production shape rather than the cautioned-against easy fake.
  • Generator: render the Faker-seeded form to a flattened (no-widget) image-only PDF, fill the currently-blank fields (times_worked_position, mentor, level, successful) with a deterministic mix incl. signed-off / crossed-out / blank rows, and emit an exact golden JSON.
  • Harness: pytest -m integration scores vision output on the documented (official, position, successful) tuple contract (modulo position-name normalization), skipped unless Ollama is present; self-manages the daemon via OllamaDaemon; --vision-model / --pull-models CLI options.
  • Iterate until green. The accuracy bar for this easy typed input is 100% (humans clear even the hardest handwritten forms at ~99%, so tolerance here is essentially zero). The harness is expected to fail on first run — that is its job. This issue stays open and tracks the prompt/pipeline/model fixes needed to make it pass honestly. Do not lower the floor to go green. If a fix turns out to need a major design change, spin that out into its own issue + design doc.

Acceptance

pytest -m integration passes at ACCURACY_FLOOR = 1.0 on the committed synthetic scan, with the daemon auto-managed and no manual setup beyond an installed Ollama + pulled vision model.

Relationship to #16

Re-scopes #16's harness onto a committable synthetic golden instead of the rems-sync-dependent real-scan path. #16 stays open for the Tier-2 real-scan path.

Activity

  1. gavinbee commented on May 30, 2026

    @gavinbee
    CollaboratorAuthor

    First real-model run (qwen2.5vl:7b, 2-page synthetic scan)

    Harness is wired and self-manages the daemon — pytest -m integration ran the full vision pipeline + scoring in ~90s with no manual setup. As expected at the 100% bar, it fails, surfacing concrete quality issues to fix:

    test_tuple_contract_accuracy → 0.77 (17/22):

    • Angela Carter — row dropped / name not matched.
    • Daniel Adams — position Chief Timer, want Timer (likely vertical bleed from an adjacent "Chief Timer" row).
    • James Mayo — successful read True, want False (struck-through row not honored — holistic-judgment miss).
    • Rodney Trujillo — successful read True, want None (blank initials + mentor assigned read as signed off).

    test_meet_header → competition_name came back 'Osborn Open Open 2026', want 'Osborn Open 2026' (doubled-token hallucination).

    test_finds_every_official passes (10/11 matched by name).

    Next: iterate on prompt/pipeline to close these. Decision recorded: keep ACCURACY_FLOOR = 1.0; these tests merge red and serve as the quality gate until fixed. Major fixes that need a new design doc get their own issue.

  2. gavinbee commented on May 31, 2026

    @gavinbee
    CollaboratorAuthor

    Handoff: picking up the quality iteration on another machine

    The fixtures + harness half of this issue landed in PR #49 (squash 8a5afc4 on main). What remains is the expanded scope: iterate on the vision pipeline until the integration harness passes at its 100% accuracy floor. This comment is a self-contained brief for an agent starting cold on a different computer.

    Current state (all on main)

    • tests/test_integration_vision.py — the harness. 3 tests: test_finds_every_official, test_tuple_contract_accuracy, test_meet_header. Gated behind @pytest.mark.integration (skipped unless -m integration).
    • tests/fixtures/vision_scan/make_synthetic_scan.py — deterministic generator (Faker seed 42) that flattens the blank form into an image-only PDF and writes a matching golden.
    • tests/fixtures/vision_scan/session_1_evals_scan.pdf + session_1_evals_scan.golden.json — the committed fixture + golden (levels are Arabic 2–5).
    • templates/swim_ontario_v1.pdf — the canonical blank Swim Ontario form (top-level, first-class asset; both generators resolve it via shared TEMPLATE_ID).
    • tests/conftest.py — adds pytest options --vision-model <tag> (default vision_extract.DEFAULT_VISION_MODEL = qwen2.5vl:7b) and --pull-models (allow ollama pull of a missing model; off by default → skip with a hint).

    How to run it

    pytest -m integration tests/test_integration_vision.py
    # optionally: --vision-model qwen2.5vl:7b   --pull-models
    

    The parsed fixture self-manages the Ollama daemon via src.ollama_runtime.OllamaDaemon — it starts ollama serve only if one isn't already up and stops it on the way out. No manual ollama serve needed. On a box without Ollama / the model, the suite skips cleanly (with a ollama pull hint) rather than failing. Requires a GPU box with the model pulled to actually exercise.

    The accuracy floor is deliberately 1.0 — do NOT lower it

    This fixture is the easiest possible input: typed, clean, digital-export scan — no handwriting, skew, or smudges. Humans clear even the hardest handwritten forms at ~99%, so tolerance for error here is essentially zero. The tests are expected to fail until the underlying extraction-quality bugs are fixed — that is their job. The fix is in the model/prompt/pipeline, never in the threshold. (Rationale is documented at length in the module docstring and the ACCURACY_FLOOR comment.)

    Known failures to close (representative — last real-model run, model output is not deterministic)

    1. Struck-through row read as signed-off — e.g. James Mayo successful got True, want False. The model isn't treating a crossed-out row as not-successful.
    2. Blank-initials + mentor-present row — e.g. Rodney Trujillo successful got True, want None. Should be ambiguous/null, not a positive sign-off.
    3. Position vertical bleed — e.g. Daniel Adams position got Chief Timer, want Timer. Adjacent-row label bleeding into the read.
    4. Dropped row — e.g. Angela Carter not found in output at all.
    5. Doubled token in meet header — competition_name got Osborn Open Open 2026, want Osborn Open 2026.

    The successful contract (whole-row holistic judgment → true/false/null) lives in src/vision_extract.py (_BASE_INSTRUCTIONS, build_prompt, _build_json_schema, _parse_successful). That's the first place to iterate for failures 1–2. Failures 3–5 are read/parse quality (prompt + JSON-schema constraints).

    Workflow notes

    • Per CONTRIBUTING: this gets its own branch off main (e.g. 48-improve-vision-quality), one PR referencing the issue. Imperative commit messages, no conventional-commit prefixes, Co-Authored-By trailer.
    • The venv is Python 3.14 (.venv/Scripts/python.exe); project targets 3.12.
    • Default (non-integration) suite must stay green and runs with Ollama not running — all LLM/subprocess calls are mocked by the conftest autouse fixtures.
    • Related deferred work in Make the synthetic scan generator config-driven (golden-as-input) #50: turn make_synthetic_scan.py into a config-driven generator whose input is golden-shaped (rows + header config). Not part of this issue, but the two interact — keep the generator's golden output stable so the harness fixture doesn't churn.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions