Repository navigation
feat(ffp): form-fill provenance standard — marker, detection, print-path record (D189) #1142
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
8399b15
feat(ffp): form-fill provenance standard — marker, detection, print-p…
hyperpolymath 74efd7c
fix(ffp): keep /AP inside the widget dictionary — vectors must be par…
hyperpolymath 80bbff9
Update 1-formats/sub-specs/form-fill-provenance/spec/conformance/prob…
hyperpolymath 4db98e4
Update 1-formats/sub-specs/form-fill-provenance/spec/DETECTION.adoc
hyperpolymath 287cf0c
Update 1-formats/sub-specs/form-fill-provenance/spec/DETECTION.adoc
hyperpolymath File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
63 changes: 63 additions & 0 deletions
63
.machine_readable/scorecards/form-fill-provenance.scorecard.a2ml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,63 @@ | ||
| # SPDX-License-Identifier: CC-BY-SA-4.0 | ||
| # form-fill-provenance.scorecard.a2ml | ||
| # Hand-authored source. Regenerate the dashboard with: just scorecards | ||
| # Schema: .machine_readable/scorecards/scorecard.schema.json | ||
|
|
||
| [scorecard] | ||
| spec_id = "form-fill-provenance" | ||
| version = "1.0.0" | ||
| assessed_date = "2026-10-04" | ||
| assessor = "arena session 01a1046c (standards), on ruling D189" | ||
|
|
||
| [[must]] | ||
| id = "M1" | ||
| text = "The DETECTION rules MUST be executable and exact: a detector MUST reproduce every conformance vector's expected line, and the vectors MUST be byte-reproducible from their generator." | ||
| system = "1-formats/sub-specs/form-fill-provenance/spec/conformance/run-conformance.sh — 14 vectors, each diffed against expected/, plus a make-fixtures.sh --check reproducibility pass" | ||
| status = "pass" | ||
| evidence = "just ffp-conformance: 14 passed, 0 failed, 0 orphan expectation(s); run 2026-10-04" | ||
| check = "bash 1-formats/sub-specs/form-fill-provenance/spec/conformance/run-conformance.sh >/dev/null 2>&1" | ||
|
|
||
| [[must]] | ||
| id = "M2" | ||
| text = "The marker MUST be recognised by namespace, `filledBy=machine` MUST be the only defined value, and an unrecognised declaration MUST be recorded rather than trusted." | ||
| system = "vectors machine-filled-declared, machine-filled-declared-generated and declared-unknown-value, diffed by run-conformance.sh" | ||
| status = "pass" | ||
| evidence = "declared-unknown-value expects FFP-E-DECL-UNRECOGNISED with classification machine-filled-suspected — the unknown declaration does not promote the classification; machine-filled-declared-generated expects machine-filled with appearances=generated, i.e. declaration wins over the structural signature" | ||
| check = "grep -q 'FFP-E-DECL-UNRECOGNISED' 1-formats/sub-specs/form-fill-provenance/spec/conformance/expected/declared-unknown-value.expected && grep -q 'form-fill-provenance/1.0/' 1-formats/sub-specs/form-fill-provenance/spec/MARKER.adoc" | ||
|
|
||
| [[must]] | ||
| id = "M3" | ||
| text = "A print-path consumer MUST classify every entered document, persist the FFP record with the job, keep it in the audit trail, and make it visible/routable (FFP/3 P1-P4) — the D189 obligation in presswerk." | ||
| system = "none — presswerk has no FFP code; D189 ruled 2026-09-30 on presswerk#118, implementation issue filed 2026-10-04" | ||
| status = "fail" | ||
| effects = "A machine-filled form and a blank form to be completed by hand stay byte-indistinguishable in presswerk's queue and audit trail, so routing and audit cannot tell them apart and no record survives the print. The blank-print hazard (values present, appearance streams absent) is also unrecorded, so a job can print empty fields with nothing in the trail to explain it." | ||
|
|
||
| [[must]] | ||
| id = "M4" | ||
| text = "A form-filling producer MUST declare machine authorship via the FFP marker, and SHOULD generate widget appearance streams (FFP/1)." | ||
| system = "none — blocky-writer writes /V and NeedAppearances but no metadata marker; producer issue filed 2026-10-04" | ||
| status = "fail" | ||
| effects = "blocky-writer output can only ever classify as machine-filled-suspected (weaker evidence, no tool/version attribution), and its reliance on NeedAppearances leaves print output dependent on a flag PDF 2.0 deprecates and that some viewers and rasterisers ignore — the condition FFP/3 P5 exists to warn about." | ||
|
|
||
| [[should]] | ||
| id = "S1" | ||
| text = "The conformance suite SHOULD run in this repository's CI on every pull request, not only on a developer's workstation." | ||
| system = ".github/workflows/dyadt-verify.yml step 'FFP conformance vectors'" | ||
| status = "pass" | ||
| evidence = "step added 2026-10-04, running the same runner as just ffp-conformance; no new action references, so the actions.lock gate is untouched" | ||
| check = "grep -q 'form-fill-provenance/spec/conformance/run-conformance.sh' .github/workflows/dyadt-verify.yml" | ||
|
|
||
| [[should]] | ||
| id = "S2" | ||
| text = "A product detector SHOULD be testable against the same vectors without adding a runtime to the suite (no Python, no JS)." | ||
| system = "run-conformance.sh honours FFP_DETECTOR=<command>; the whole conformance tree is bash + awk" | ||
| status = "pass" | ||
| evidence = "the reference path itself is invoked as FFP_DETECTOR default ('awk -f probe.awk'); no .py/.js/.ts file exists under the spec home" | ||
| check = "test -z \"$(find 1-formats/sub-specs/form-fill-provenance -name '*.py' -o -name '*.js' -o -name '*.ts')\" && grep -q 'FFP_DETECTOR' 1-formats/sub-specs/form-fill-provenance/spec/conformance/run-conformance.sh" | ||
|
|
||
| [[could]] | ||
| id = "C1" | ||
| text = "A future revision COULD carry the appearance hazard into the submission gate (hold for confirmation) once a print-path consumer implements FFP/3, so that P5's SHOULD becomes mechanical." | ||
| system = "none — deliberately aspirational; P5 is a SHOULD in FFP/3 v1.0.0" | ||
| status = "aspirational" | ||
| effects = "Until a consumer implements it, the hazard is recorded and surfaced but not gated; a user can still send a form to the printer and receive blank fields." |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,233 @@ | ||
| // SPDX-License-Identifier: CC-BY-SA-4.0 | ||
| // (MPL-2.0 is automatic legal fallback until PMPL is formally recognised) | ||
| // | ||
| // FFP — Form-Fill Provenance. Normative sub-spec of 1-formats. | ||
| // Home: 1-formats/sub-specs/form-fill-provenance/ | ||
| // Version: 1.0.0 · Status: ACTIVE · Ratified: 2026-10-04 | ||
|
|
||
| = Form-Fill Provenance (FFP) — machine-filled vs hand-completed forms in the print path | ||
| :toc: preamble | ||
| :icons: font | ||
| :revdate: 2026-10-04 | ||
| :revnumber: 1.0.0 | ||
| :status: ACTIVE | ||
|
|
||
| [.lead] | ||
| FFP is the estate standard for *stating, detecting and recording how a PDF form | ||
| came to be filled* — by a machine before printing, or by hand after it. It | ||
| exists because a filled application form is a print job, and the print path | ||
| currently cannot tell the two apart. | ||
|
|
||
| == Why this document exists | ||
|
|
||
| `blocky-writer` fills fixed-layout PDF and application forms. Its `fill_blocks` | ||
| returns ordinary PDF bytes — an AcroForm with `/V` values written and | ||
| `NeedAppearances` set — with **no marker** distinguishing that output from a | ||
| form a human completed. It asked the question deliberately | ||
| (https://github.com/hyperpolymath/presswerk/issues/118[presswerk#118], from the | ||
| 2026-09-27 ecosystem recon in | ||
| https://github.com/hyperpolymath/blocky-writer/pull/73[blocky-writer#73]). | ||
|
|
||
| The owner answered it as **D189** (2026-09-30, `scope: repo`, basis | ||
| *applied-unless-struck*; surface | ||
| https://github.com/hyperpolymath/standards/issues/787#issuecomment-5913118060[standards#787]): | ||
|
|
||
| [quote] | ||
| ____ | ||
| Yes: record machine-filled provenance as metadata in the print path so | ||
| audit/routing can distinguish it. | ||
| ____ | ||
|
|
||
| A ruling that only one repository acts on cannot bind two. *How* the two sides | ||
| agree is a cross-repository matter — the marker a producer writes and the | ||
| evidence a print path validates — so the normative text lives here, and the | ||
| implementations live where the work happens: | ||
|
|
||
| [cols="1,2,2",options="header"] | ||
| |=== | ||
| | Class | Who | What they owe | ||
|
|
||
| | *Producer* | ||
| | `blocky-writer` (and any future form-filling producer) | ||
| | Writes the FFP marker (link:spec/MARKER.adoc[MARKER]) so its output is *declared*, not guessed. | ||
|
|
||
| | *Print-path consumer* | ||
| | `presswerk` (and any future print router/batch tool) | ||
| | Classifies every document that enters the print path (link:spec/DETECTION.adoc[DETECTION]), and | ||
| *records* the result on the job and in the audit trail (link:spec/PRINT-PATH.adoc[PRINT-PATH]). | ||
|
|
||
| | *Detector* | ||
| | any of the above, or a third party | ||
| | Implements link:spec/DETECTION.adoc[DETECTION] exactly; passes the | ||
| link:spec/conformance/README.adoc[conformance suite]. | ||
| |=== | ||
|
|
||
| [IMPORTANT] | ||
| ==== | ||
| This standard does **not** require a producer to write a marker before a | ||
| consumer may classify. It defines *both* legs: a declared marker that is | ||
| authoritative when present, and a structural signature that is honest when it | ||
| is not. A consumer that cannot see a marker MUST NOT invent one; see | ||
| <<honesty>>. | ||
| ==== | ||
|
|
||
| == Scope | ||
|
|
||
| *In scope:* PDF documents containing interactive form fields (AcroForm, | ||
| ISO 32000-1 §12.7), the provenance metadata a producer may attach to them, the | ||
| classification a print path derives from them, and the record it keeps. | ||
|
|
||
| *Out of scope:* XFA-only forms (PDF 2.0 deprecates XFA; XFA provenance is | ||
| undefined here), digital signatures, document authorship or identity of the | ||
| person who filled a form, and any rendering decision (FFP classifies; it never | ||
| re-renders). | ||
|
|
||
| == The spec set | ||
|
|
||
| [cols="1,3",options="header"] | ||
| |=== | ||
| | Document | What it defines | ||
|
|
||
| | link:spec/MARKER.adoc[MARKER] | ||
| | The `ffp:filledBy` XMP marker: namespace, properties, writer obligations, | ||
| forward-compatibility, and why XMP rather than `/Info`. | ||
|
|
||
| | link:spec/DETECTION.adoc[DETECTION] | ||
| | The classification algorithm: fillable-field counting, value meaningfulness, | ||
| appearance detection, the classification lattice, and the evidence codes. | ||
|
|
||
| | link:spec/PRINT-PATH.adoc[PRINT-PATH] | ||
| | The obligations of a print-path consumer: record, surface, never silently | ||
| normalise, and the blank-print hazard. | ||
|
|
||
| | link:spec/conformance/README.adoc[CONFORMANCE] | ||
| | Fourteen vectors + a reference probe + a runner. A conforming detector MUST | ||
| pass this suite. | ||
| |=== | ||
|
|
||
| == Definitions | ||
|
|
||
| machine-filled:: | ||
| A document whose field values were written programmatically, without an | ||
| interactive editing session — the `blocky-writer` case. | ||
|
|
||
| hand-completed:: | ||
| A document completed by a person on paper, after printing. A print path | ||
| usually sees this as a *blank form* (link:spec/DETECTION.adoc[DETECTION]): the values are added | ||
| later, by pen. | ||
|
|
||
| interactively-filled:: | ||
| A document completed by a person in a PDF viewer/editor. FFP deliberately does | ||
| **not** classify this case as `hand` — see <<honesty>>. | ||
|
|
||
| declared provenance:: | ||
| What the document says about itself, via the FFP marker (link:spec/MARKER.adoc[MARKER]). | ||
|
|
||
| inferred provenance:: | ||
| What the document's AcroForm structure shows, per the rules in | ||
| link:spec/DETECTION.adoc[DETECTION]. | ||
|
|
||
| appearances:: | ||
| The widget appearance streams (`/AP` with a normal appearance) that determine | ||
| what a viewer or printer actually draws. A field can hold a value and still | ||
| print blank when its appearance is missing — this is the reason FFP is a | ||
| *print-path* concern, not merely a labelling one. | ||
|
|
||
| == The two axes | ||
|
|
||
| FFP never collapses "is there a form?" with "how was it filled?" into one guess. | ||
| It reports both, and derives one classification from them. | ||
|
|
||
| *Axis 1 — form/fill state:* `absent` · `present`, with `filled_fields/total_fields`. | ||
| *Axis 2 — provenance evidence:* declared (marker) · inferred (structure) · none. | ||
|
|
||
| === The classification lattice | ||
|
|
||
| [cols="1,2,3",options="header"] | ||
| |=== | ||
| | Classification | Set when | Routing meaning | ||
|
|
||
| | `no-form` | ||
| | no interactive form, or no fillable terminal fields | ||
| | not a form; nothing to distinguish | ||
|
|
||
| | `blank-form` | ||
| | form present, *zero* meaningful values | ||
| | the print-and-hand-complete case — print blank, fill by hand | ||
|
|
||
| | `machine-filled` | ||
| | ≥1 meaningful value **and** the FFP marker declares `filledBy=machine` | ||
| | machine-filled, *declared* | ||
|
|
||
| | `machine-filled-suspected` | ||
| | ≥1 meaningful value **and** no machine marker **and** | ||
| `NeedAppearances=true` **and** appearances incomplete | ||
| | machine-filled by structural signature only — treat conservatively, record the weaker evidence | ||
|
|
||
| | `filled-unknown` | ||
| | ≥1 meaningful value, none of the above matched | ||
| | filled, provenance undetermined — never guess | ||
|
|
||
| | `unreadable` | ||
| | the document structure could not be parsed | ||
| | unknown; the job MUST still record that FFP ran and could not decide | ||
| |=== | ||
|
|
||
| `machine-filled-suspected` exists so a print path never has to choose between | ||
| lying and silence: the structural signature is strong evidence, and the | ||
| classification says exactly how strong. | ||
|
|
||
| === Precedence | ||
|
|
||
| 1. The declared marker wins over the structural signature when it is present | ||
| and recognised (`machine-filled`, even if appearances are complete and | ||
| `NeedAppearances` is absent). | ||
| 2. An unrecognised declaration does not fall back to "trusting" the producer: | ||
| it is recorded as evidence and the structural rules decide. | ||
| 3. `blank-form` is never promoted to `machine-filled*`: a form with zero | ||
| meaningful values is blank regardless of flags (a negative control for this | ||
| rule is vector `blank-form-need-appearances`). | ||
| 4. If the document cannot be parsed, the classification is `unreadable`. It is | ||
| never `blank-form`. | ||
|
|
||
| [[honesty]] | ||
| == The honesty rule (normative) | ||
|
|
||
| A consumer MUST NOT report a provenance the evidence does not support. | ||
|
|
||
| * FFP does not claim to detect "hand-filled in a viewer". `/AP` streams can be | ||
| generated by a program just as well as by an editor, so the honest | ||
| classification for that case is `filled-unknown`. | ||
| * `ffp:filledBy` is a *declaration*, not proof. Consumers MUST NOT present it as | ||
| verification, and MUST NOT infer authorship or identity from it: FFP records | ||
| **how** values were written, never **who** wrote them. | ||
| * A `machine-filled-suspected` result MUST be recorded with the evidence that | ||
| produced it, so a later reviewer can re-derive the classification. | ||
|
|
||
| == Relationship to other standards | ||
|
|
||
| * ISO 32000-1 §12.7 (interactive forms), §12.5.5 (`/AP` appearance streams), | ||
| §14.3.2 (metadata streams); ISO 16684-1 (XMP). `NeedAppearances` is | ||
| deprecated by ISO 32000-2 (PDF 2.0) and PDF/A requires appearances to be | ||
| present — which is *why* the structural signature in | ||
| link:spec/DETECTION.adoc[DETECTION] is a | ||
| reliable, if not sufficient, machine-fill signal. | ||
| * `docs/SEAMS-SPEC.adoc` — the blocky-writer → presswerk handoff is a declared | ||
| seam (`blocky-writer-presswerk-form-handoff`); the seam-register entry for | ||
| each side is in link:spec/PRINT-PATH.adoc[PRINT-PATH] §Seam. | ||
| * The precision-document suite's composition contract forbids any component | ||
| from silently normalising page geometry. FFP extends the same discipline to | ||
| content provenance: a print path may *report* that appearances are missing, | ||
| but it may not silently regenerate them. | ||
| * link:../../../docs/decisions/[ADR-004] (README discipline) is why this spec | ||
| is one README plus normative parts, and why the registry entry is a | ||
| `scripts/build-registry.sh` row rather than a hand-kept index. | ||
|
|
||
| == Versioning | ||
|
|
||
| This spec is versioned as a whole, `MAJOR.MINOR.PATCH`, and additionally carries | ||
| a *wire* version in the marker namespace (`…/1.0/`) and in the record | ||
| (`"ffp": "1.0"`). Adding a classification, an evidence code, or a marker | ||
| property is a MINOR change; repurposing an existing value is MAJOR. Consumers | ||
| MUST ignore unknown properties and MUST NOT fail a job because a document | ||
| declares a version they do not know. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
✅ Runtime observed
🏁 Script executed:
Repository: hyperpolymath/standards
Length of output: 306
🏁 Script executed:
Repository: hyperpolymath/standards
Length of output: 41710
🏁 Script executed:
Repository: hyperpolymath/standards
Length of output: 5117
Regenerate the registry after staging the FFP files.
The registry records the SHA-256 hash of empty input, but the tracked FFP files produce a different hash.
bash scripts/build-registry.sh --checkreports the registry as stale and exits with status 1. Stage the FFP files, runjust registry, and commit the regenerated.machine_readable/REGISTRY.a2ml.🤖 Prompt for AI Agents