[Epic] E3 — Standardize and document the canonical normalization function

[jakebromberg]

Status (post-PR-279, 2026-05-11)

Rust/PyO3 side shipped; Postgres analog pending. See the pinned status comment for the current acceptance-criteria breakdown — and the companion wiki refresh in WXYC/wiki PR #55 for the §3.3.0 + §3.3.5 prose update. The remainder of this body captures the original epic framing.

---

Summary

Standardize and document the single canonical normalization function used by every consumer (LML, semantic-index, Backend-Service, all four caches) for artist / title / label normalization. Today, the full Homebrew Discogs cache is the only one with norm_name / norm_title columns; the function that produced them lives in Rust at load time, undocumented in-cache. LML, semantic-index, and Backend each apply different normalizations at query time. Cross-cache joins on artist/title strings silently misalign on diacritics, punctuation, the/a article, and word ordering.

This epic ships the locked §3.3.2 algorithm (NFKD-based, with seven-step pipeline, fallback semantics, and per-step opt-in for steps 6 + 8 if regression exceeds 2%), a Postgres analog (per-cache wxyc_identity_match_artist / wxyc_identity_match_title functions, mirroring the Rust naming), a parity test matrix (Rust ↔ Postgres ↔ Python), and a regression report comparing new function output against existing norm_* columns on the full Homebrew Discogs cache.

Sequenced workflow with decision gates (per §3.3.1)

Phase 0: Audit + spec + version decision

Step 1 — Normalization audit (BLOCKING; §3.3.1.1). No code changes. Produces wxyc-etl/docs/normalization-audit.md cataloging every existing normalization implementation across LML, semantic-index, Backend, all four caches, and tubafrenzy. Test-path verification scope (§8.6 hard gate): the audit also confirms test-path conventions in each consumer repo so §8.5's inventory can be corrected before any test PR opens. Resolves Q8 (NFC vs NFKD) and Q9 (empty-output) by escalating with documented options.

Step 2 — Algorithm spec (post-audit). Lock the seven-step pipeline per §3.3.2:

1. Unicode NFKD normalize
2. Casefold
3. Strip combining diacritics (\p{M})
4. Strip punctuation, collapse whitespace
5. Strip leading article (the / a / an / le / la / los / las)
6. Word-order canonicalization (opt-in per §3.3.4 if regression >2%)
7. Empty-output fallback (option B per §3.3.3 — re-run with step 6 omitted; flag norm_artist_fallback=true; CHECK constraint forbids empty norm)
8. Optional: ASCII-only collapse (opt-in per §3.3.4)

Step 3 — Version decision. Bump wxyc-etl to 0.2.0 with the new function exported. Captured in §3.3.4 regression report.

Phase 1: Implement + ship + consumer migration

Step 4 — Implement. Rust function in wxyc-etl/wxyc-etl/src/text/forms.rs (to_match_form baseline) + text/identity.rs (to_identity_match_form cross-cache-identity layer) + Postgres analog as wxyc_identity_match_artist / wxyc_identity_match_title SQL functions deployable per cache (tracked separately in WXYC/discogs-etl#194). Parity test fixture at wxyc-etl/tests/fixtures/normalization_cases.csv (≥600 rows, 250 source-divergent stratified). Vendoring + SHA-256 verification in each consumer repo per §8.5 path-verification process.

Step 5 — Consumer migration. PRs into LML, semantic-index, Backend, all four caches replacing local normalization with wxyc-etl 0.2.0 calls. Each PR includes the per-cache parity test referencing the canonical fixture.

Step 6 — Cache re-norm. Rebuild norm_* columns on the full Homebrew Discogs cache. Per-cache parity test gates promotion.

Dependencies

• Step 1 has no dependencies. Can start immediately, in parallel with E5 step 1.
• Steps 4-6 depend on step 1+2+3 merging.
• Blocks E1, E2-BS, E2-LML — every consumer of norm_* columns waits on §3.3.2 being shipped.

Existing issues folded in

• WXYC/library-metadata-lookup#194 — f_unaccent symmetry between Discogs and MB legs → the Postgres analog from §3.3 IS the symmetry.
• WXYC/library-metadata-lookup#168 — Greek lowercase sigma → covered by §3.3.3 test matrix.
• WXYC/library-metadata-lookup#211 (and sub-issues #214, #215) — Trigram + name preprocessing in DiscogsReconciler → coordinates; uses §3.3 normalization.
• WXYC/Backend-Service#521 — Normalize names in artist-identity ETL → child; uses canonical function.
• WXYC/semantic-index#231 — Review-to-artist matching via name normalization → child; uses canonical function.

Acceptance

• wxyc-etl 0.2.0 published to crates.io + PyPI with locked normalization function.
• Postgres analog deployed and parity-tested on every cache.
• Regression report (§3.3.4) shows ≤2% match shift on existing norm_* columns.
• All consumers migrated; no local normalization remains.
• §8.6 hard verification gate satisfied: every test PR cites the audit's confirmed test path.

Plan reference

plans/library-hook-canonicalization/plan.md §3.3 (full subsection), §3.3.1.1 (audit scope), §3.3.2 (algorithm), §3.3.3 (parity test matrix), §3.3.4 (regression report), §3.3.5 (Postgres analog).