Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 22 additions & 10 deletions EXPLAINME.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,18 @@
:toc:
:icons: font

[IMPORTANT]
====
Current evidence (2026-11-16): the line-ending algebra, header bridge and
pure streaming model now have checked Agda proofs and Lean/Rocq re-proofs.
See link:verification/proofs/SHOLES.adoc[the proof scope and checker evidence].
The runtime, classifier, Git driver, performance numbers and Creusot
verification described below are target design, not delivered or verified
implementation in this tree. The proof witnesses are hand re-proofs, not
automatically exported proof objects. Raw byte round trips require the
qualifications documented with the proofs.
====

The README makes claims. This file backs them up.

== Mechanized Quotient Canonicalization
Expand Down Expand Up @@ -31,11 +43,11 @@ The Agda formalization constructs a canonical projection function `canon : List

*Caveat:* Proofs in Agda operate over total inductive lists (`List (Fin 256)`). The operational mapping to unbounded streaming I/O and chunked ring buffers is proved separately via Creusot verification contracts on the Rust implementation.

- Agda core definitions: `proofs/agda/Sholes/Core.agda`
- Equivalence lemmas: `proofs/agda/Sholes/Equivalence.agda`
- Lens laws: `proofs/agda/Sholes/Lens.agda`
- Exported Lean 4 definitions: `proofs/lean/Sholes/Core.lean`
- Exported Coq/Rocq proofs: `proofs/coq/Sholes_Core.v`
- Agda core definitions: `verification/proofs/agda/Sholes/Core.agda`
- Equivalence lemmas: `verification/proofs/agda/Sholes/Equivalence.agda`
- Lens laws: `verification/proofs/agda/Sholes/Lens.agda`
- Exported Lean 4 definitions: `verification/proofs/lean4/Sholes/Core.lean`
- Exported Coq/Rocq proofs: `verification/proofs/coq/Sholes_Core.v`

== Zero-Allocation Creusot-Verified Rust Engine

Expand Down Expand Up @@ -159,11 +171,11 @@ A test corpus of 10,000 text files is subjected to arbitrary newline permutation
|===
| Path | Proves

| `proofs/agda/Sholes/Core.agda` | Formal quotient definitions over alphabet byte lists
| `proofs/agda/Sholes/Equivalence.agda` | Mechanized proofs of Idempotence and Invariance
| `proofs/agda/Sholes/Lens.agda` | Well-behaved lens theorems (GetPut and PutGet)
| `proofs/lean/Sholes/Core.lean` | Exported Lean 4 definitions for cross-prover replication
| `proofs/coq/Sholes_Core.v` | Exported Coq/Rocq proof objects
| `verification/proofs/agda/Sholes/Core.agda` | Formal quotient definitions over alphabet byte lists
| `verification/proofs/agda/Sholes/Equivalence.agda` | Mechanized proofs of Idempotence and Invariance
| `verification/proofs/agda/Sholes/Lens.agda` | Well-behaved lens theorems (GetPut and PutGet)
| `verification/proofs/lean4/Sholes/Core.lean` | Exported Lean 4 definitions for cross-prover replication
| `verification/proofs/coq/Sholes_Core.v` | Exported Coq/Rocq proof objects
| `crates/sholes-core/src/normalize.rs` | Zero-allocation streaming normalization engine
| `crates/sholes-core/src/contracts.rs` | Creusot formal verification contracts
| `crates/sholes-core/src/classifier.rs` | Three-tier binary non-interference classifier
Expand Down
4 changes: 4 additions & 0 deletions Justfile
Original file line number Diff line number Diff line change
Expand Up @@ -764,3 +764,7 @@ uncloak:
lock-sync-fix:
./scripts/lock-sync-fix.sh
./scripts/check-lock-sync.sh

# Strict line-ending proofs, Cubical consumer, replays and rejection controls.
check-line-endings:
bash scripts/check-line-endings.sh
Comment on lines +769 to +770

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Wire the strict line-ending gate into CI

This recipe is currently only a manual entry point. I checked .github/workflows, the ci recipe, and build/just/proofs.just; none invokes check-line-endings, while proof-check-all only scans the existing prover manifests and therefore never checks verification/proofs/cubical/SholesCubical.agda or runs the new assumption audits and rejection controls. A later regression in those checks can consequently merge without any automated failure, so this recipe should be added to the CI dependency chain or an equivalent workflow step.

Useful? React with 👍 / 👎.

16 changes: 14 additions & 2 deletions README.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,21 @@
:idprefix:
:idseparator: -

[IMPORTANT]
====
Current evidence (2026-11-16): the line-ending algebra, header bridge and
pure streaming model now have checked Agda proofs and Lean/Rocq re-proofs.
See link:verification/proofs/SHOLES.adoc[the proof scope and checker evidence].
The runtime, classifier, Git driver, performance numbers and Creusot
verification described below are target design, not delivered or verified
implementation in this tree. The proof witnesses are hand re-proofs, not
automatically exported proof objects. Raw byte round trips require the
qualifications documented with the proofs.
====

image:https://img.shields.io/badge/License-MPL--2.0-indigo.svg[License: MPL-2.0,link="https://github.com/hyperpolymath/sholes/blob/main/LICENSE"]
image:https://img.shields.io/badge/standard-RSR--compliant-green.svg[RSR Compliant]
image:https://img.shields.io/badge/verified-Agda%20%7C%20Creusot-9f55ff[Formal Verification,link="https://github.com/hyperpolymath/sholes/tree/main/proofs"]
image:https://img.shields.io/badge/verified-Agda%20%7C%20Creusot-9f55ff[Formal Verification,link="https://github.com/hyperpolymath/sholes/tree/main/verification/proofs"]
image:https://img.shields.io/badge/status-active-brightgreen.svg[Status]

**Sick of CRLF breaking your git diffs, shebangs, and CI caches? A formally verified quotient layer that ends the 50-year typewriter war forever. Agda proofs + verified Rust filter.**
Expand Down Expand Up @@ -110,7 +122,7 @@ To work on this repository directly:
just

# 2. Verify formal proofs (Agda / Coq)
just verify-proofs
just check-line-endings

# 3. Run Creusot verification conditions on Rust engine
just verify-code
Expand Down
81 changes: 81 additions & 0 deletions scripts/check-line-endings.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
#!/usr/bin/env bash
# SPDX-License-Identifier: MPL-2.0
# Copyright (c) 2026 Jonathan D.A. Jewell <j.d.a.jewell@open.ac.uk>
# Canonical line-ending proof check, including rejection controls.
set -euo pipefail
root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
evidence=${EVIDENCE_DIR:-$(mktemp -d -t sholes-evidence.XXXXXXXX)}
mkdir -p "$evidence"
evidence=$(cd "$evidence" && pwd)
work=$(mktemp -d -t sholes-proofs.XXXXXXXX)
trap 'rm -rf -- "$work"' EXIT
cp -R "$root/verification/proofs/agda" "$root/verification/proofs/coq" "$root/verification/proofs/cubical" "$work/"
cp -R "$root/verification/proofs/lean4" "$work/lean"
# Remove only generated files in the freshly allocated proof scratch directory.
find "$work" -type f \( -name '*.agdai' -o -name '*.olean' -o -name '*.vo' \) -delete
agda=${AGDA:-agda}
lean=${LEAN:-lean}
coqc=${COQC:-coqc}
coqchk=${COQCHK:-coqchk}
for tool in "$agda" "$lean" "$coqc" "$coqchk"; do
command -v "$tool" >/dev/null || { echo "FAIL: missing $tool" >&2; exit 1; }
done
{ "$agda" --version; "$lean" --version; "$coqc" --version; } > "$evidence/versions.log"
(cd "$root"; git rev-parse HEAD; git diff --stat; find verification/proofs -type f \( -name '*.agda' -o -name '*.lean' -o -name '*.v' \) -print0 | sort -z | xargs -0 sha256sum) > "$evidence/source-identity.log"
for module in Core Equivalence Lens HeaderBridge Wire Streaming; do
(cd "$work/agda"; "$agda" --safe --without-K --cubical-compatible --double-check --warning=error "Sholes/$module.agda") > "$evidence/agda-$module.log" 2>&1
done
(cd "$work"; "$agda" --safe --cubical --double-check --warning=error -i agda -i cubical cubical/SholesCubical.agda) > "$evidence/cubical.log" 2>&1
(cd "$work/lean"; "$lean" -o Sholes/Core.olean Sholes/Core.lean) > "$evidence/lean.log" 2>&1
{
printf 'import Sholes.Core\n'
awk 'BEGIN {ns="Sholes."} /^namespace Sholes.Wire$/ {ns="Sholes.Wire."} /^[[:space:]]*theorem / {print "#print axioms " ns $2}' "$work/lean/Sholes/Core.lean"
} > "$work/lean/Audit.lean"
(cd "$work/lean"; LEAN_PATH=. "$lean" Audit.lean) > "$evidence/lean-assumptions.log" 2>&1
if grep -Eq "depends on axioms:|declaration uses 'sorry'|error:" "$evidence/lean-assumptions.log" "$evidence/lean.log"; then
echo 'FAIL: Lean assumption audit' >&2; exit 1
fi
(cd "$work/coq"; "$coqc" Sholes_Core.v; "$coqc" Sholes_Wire.v; "$coqchk" -silent Sholes_Core Sholes_Wire) > "$evidence/rocq.log" 2>&1
{
printf 'Require Import Sholes_Core Sholes_Wire.\n'
sed -n 's/^Lemma \([A-Za-z0-9_]*\).*/Print Assumptions \1./p' "$work/coq/Sholes_Core.v"
sed -n 's/^Lemma \([A-Za-z0-9_]*\).*/Print Assumptions Wire.\1./p' "$work/coq/Sholes_Wire.v"
} > "$work/coq/Audit.v"
(cd "$work/coq"; "$coqc" Audit.v) > "$evidence/rocq-assumptions.log" 2>&1
if grep -q 'Axioms:' "$evidence/rocq-assumptions.log"; then
echo 'FAIL: Rocq assumption audit' >&2; exit 1
fi

# Each tool must reject the same false claim. Positive checks above establish
# that the tool, imports and source paths work before an exit failure counts.
cat > "$work/agda/Wrong.agda" <<'AGDA'
{-# OPTIONS --safe --without-K --cubical-compatible #-}
module Wrong where
open import Sholes.HeaderBridge
bad : hasBreak {Bool} (canon (cr ∷ [])) ≡ false
bad = refl
AGDA
if (cd "$work/agda"; "$agda" --safe --without-K --cubical-compatible --double-check Wrong.agda) > "$evidence/agda-negative.log" 2>&1; then
echo 'FAIL: Agda accepted false header claim' >&2; exit 1
fi
grep -q 'true != false' "$evidence/agda-negative.log"
cat > "$work/lean/Wrong.lean" <<'LEAN'
import Sholes.Core
open Sholes
theorem bad : hasBreak (canon ([Byte.cr] : List (Byte Bool))) = false := by rfl
LEAN
if (cd "$work/lean"; LEAN_PATH=. "$lean" Wrong.lean) > "$evidence/lean-negative.log" 2>&1; then
echo 'FAIL: Lean accepted false header claim' >&2; exit 1
fi
grep -q 'rfl' "$evidence/lean-negative.log"
cat > "$work/coq/Wrong.v" <<'COQ'
Require Import Sholes_Core.
Lemma bad : hasBreak (canon (cons (@cr bool) nil)) = false.
Proof. reflexivity. Qed.
COQ
if (cd "$work/coq"; "$coqc" Wrong.v) > "$evidence/rocq-negative.log" 2>&1; then
echo 'FAIL: Rocq accepted false header claim' >&2; exit 1
fi
grep -q 'Unable to unify' "$evidence/rocq-negative.log"
printf 'PASS: compatible Agda; full Cubical; Lean; Rocq/coqchk; assumption audits; three rejection controls.\nEvidence: %s\n' "$evidence"
printf 'NOT CHECKED: automatic witness transport; Lean4Lean; executable/compiler refinement; classifier accuracy.\n'
9 changes: 9 additions & 0 deletions verification/proofs/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# SPDX-License-Identifier: MPL-2.0
*.agdai
*.olean
*.ilean
*.vo
*.vos
*.vok
*.glob
.*.aux
128 changes: 128 additions & 0 deletions verification/proofs/SHOLES.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
// SPDX-License-Identifier: CC-BY-SA-4.0
// Copyright (c) 2026 Jonathan D.A. Jewell <j.d.a.jewell@open.ac.uk>
= Line-ending proofs

These are source proofs, not a verified Rust implementation. The initial
checked work is dated 2026-11-16. Run `bash scripts/check-line-endings.sh`
Comment on lines +5 to +6

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Add these proofs to the canonical proof census

The repository's verification/proofs/README.adoc procedure requires every new obligation to be recorded in docs/status/PROOF-NEEDS.adoc and every completed proof and recheck leg in docs/status/PROOF-STATUS.adoc, but this commit leaves the project-specific needs table empty and the status page reporting 0% proven with “No proofs completed yet.” Readers and tooling that use those canonical status files therefore receive the opposite result from this new checked-work claim, so both census files need entries for the Agda statements and their Lean/Rocq legs.

Useful? React with 👍 / 👎.

from the repository root. Every required checker must exist; missing tools
fail the command. The command prints the location of fresh logs and source
hashes and checks three negative controls after the positive proofs.

== Semantics

CRLF denotes one newline. A lone CR or LF also denotes one newline.
Other octets, repeated newlines, and a trailing newline are preserved.
The token stream distinguishes the empty file from a single newline;
it does not use a `lines` operation that discards the final terminator.
The text normal form renders each newline as LF.

`Byte A` separates CR, LF and ordinary payloads by construction.
`Sholes.Wire` instantiates the payload with a recursive finite type of size
254, with a checked bijection to all 256 numeric octets. CR is 13 and LF
is 10; the other values retain their numeric identities. `every-sound`
proves coverage of every finite inhabitant, so the closed computations
used for that bijection are finite proofs, not sampled runtime tests.
The list round trips lift the algebra to numeric octet streams.

== Checked statements and correspondence

[cols="2,3,3",options="header"]
|===
|Agda source |Meaning |Other witnesses
|`Sholes/Core.agda`
|Both renderers parse back to the original tokens; LF output has no CR;
explicit binary mode is identity; repeated and trailing breaks survive.
|Lean `Sholes/Core.lean`, Rocq `Sholes_Core.v`
|`Sholes/Equivalence.agda`
|Token equality is an equivalence; canon preserves and reflects it;
idempotence; congruence through any hash function; factorization.
|Same two witnesses
|`Sholes/Lens.agda`
|Exact round trips for canonical stores and environment-normal views;
raw views round-trip modulo token equality; canonical bytes alone cannot
recover every original spelling.
|Same two witnesses
|`Sholes/HeaderBridge.agda`
|Normalization preserves forbidden-break detection; LF detection after
normalization equals CR-or-LF detection before it; pair-only detection fails.
|Same two witnesses
|`Sholes/Streaming.agda`
|State and output compose across any split, including empty chunks;
streamed output equals the list specification; resetting at CR/LF is wrong.
|Same two witnesses
|`Sholes/Wire.agda`
|Finite octet bijection, list round trips, octet-level idempotence and
header-break preservation.
|Lean `Sholes/Core.lean` namespace `Sholes.Wire`; Rocq `Sholes_Wire.v`
|`cubical/SholesCubical.agda`
|Full Cubical consumer converts equality witnesses to paths for
idempotence, header preservation, octet round trip and chunk refinement.
|Separate Agda profile; no independent-kernel claim
|===

The Lean/Rocq files are *hand re-proofs*, not exported Agda proof objects.
The constructor map is `cr/cr`, `lf/lf`, `plain/plain`,
`newline/newline`, `literal/literal`; empty/cons lists and structural
equality map to the corresponding native constructs. All payload types
are arbitrary. Streaming states map `ready/ready` and
`suppressLF/suppressLF`; each step emits the same zero or one token.
Rocq's pair detector recurses through the tail rather than syntactically
skipping a plain byte after CR; its branch equations have the same meaning.
Agda hyphenated theorem names map to underscores in the other two files.
The finite type is recursive optional values over the empty type in all
three witnesses. Lean's `addLeft` makes its index arithmetic reduce in
the same direction as Agda/Rocq. Finite reflection computes all 256 values
under kernel reduction, without `native_decide` or an unchecked evaluator.

This correspondence is reviewed at source level here; it is not a checked
translation or an automatically certified statement-identity map. Kernel
acceptance in three systems does not by itself close that gap. No PWG
grade is asserted; receipt-based estate grading is a separate step.

== Scope and non-theorems

* Hash coherence is function congruence. It does not assert that hashes
are injective or collision-free.
* Binary preservation is conditional on explicit `binary` mode. There is
no proof that a content heuristic correctly classifies arbitrary files.
* The raw inverse impossibility theorem concerns reconstruction from
canonical bytes alone. A lens retaining the original source or explicit
spelling metadata can have stronger laws and needs its own definition.
* Streaming proofs concern a pure finite-state transducer. They do not
prove ring-buffer bounds, aliasing, allocation, syscalls or compiler correctness.
* The numeric-octet bijection is not a proof about Idris2 primitive strings,
RefC boxing, C layout, Zig pointers, or the Rust implementation.
* Full Cubical checking uses Agda's built-in path machinery. It does not
claim integration with a separately pinned Cubical library.

== Observed checking

Agda 2.6.4.3 accepted the compatible source with
`--safe --without-K --cubical-compatible --double-check --warning=error`.
The separate full-Cubical consumer passed with
`--safe --cubical --double-check --warning=error`.
Lean 4.15.0 (the repository pin) accepted the witness; the earlier core
also passed Lean 4.35.0. Rocq 9.2 accepted and `coqchk` replayed the witness.
The runner enumerates every named Lean theorem and Rocq lemma and audits
its assumptions. These declarations require no additional axioms.
Each prover rejected the false claim that canonicalizing lone CR removes
the forbidden break.

An installed Lean4Lean executable was also probed with both installed Lean
formats; it rejected the object headers as incompatible. That checker leg
is *not passed*. Its executable provenance and matching toolchain need
pinning before replay. Upstream also notes shared ancestry with Lean's C++
kernel; another implementation is not automatic independence of defects.

== Proven / Atoma application

At Proven `c13619f9a67a1b9e8234e22e22dae0940d2480cb`, the actual Idris
`Proven.SafeHeader.Types.hasCRLF`, compiled using RefC, returns true on
lone CR, lone LF, CRLF and embedded CR. The actual Zig export compiled
with Zig 0.15.2 returns false on the lone/embedded cases and true on CRLF.

The bridge proves what a correct normalizer would preserve; it cannot fix
that runtime mismatch. In fact every canonical LF-only stream defeats a
CRLF-pair-only check. Keep header rejection separate from storage
normalization. Atoma still needs an implementation-to-model relation,
ABI marshaling evidence and proof replay bound to the exact consumer source.
7 changes: 7 additions & 0 deletions verification/proofs/agda/MANIFEST
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,10 @@
# (Agda 2.6.3). 2026-11-12: Properties.agda made self-contained (no library
# imports; the pinned agda-stdlib does not build), rechecked with Agda 2.6.3.
verification/proofs/agda|Properties.agda|gated| compiles under `agda --safe --without-K --cubical-compatible`

verification/proofs/agda|Sholes/Core.agda|gated| line-token definitions and canonical renderers
verification/proofs/agda|Sholes/Equivalence.agda|gated| canonical quotient laws
verification/proofs/agda|Sholes/Lens.agda|gated| quotient round trips and no raw inverse
verification/proofs/agda|Sholes/HeaderBridge.agda|gated| header rejection bridge and counterexamples
verification/proofs/agda|Sholes/Wire.agda|gated| checked bijection with all 256 octets
verification/proofs/agda|Sholes/Streaming.agda|gated| stateful chunk composition and refinement
Loading
Loading