dev: kdist<->divergence calibration harness (KDIST_CUTOFF study)

[cjfields]

Summary

Adds an ignored analysis test (src/kdist_calibrate.rs) that empirically re-derives the relationship between DADA2's k-mer distance (kmer_dist8, our port of the ESPRIT metric) and true alignment divergence (align_endsfree), so KDIST_CUTOFF (0.42, nominally ~10% divergence, calibrated on Illumina 16S) can be checked per dataset / platform / pooling regime.

Motivation: the k-mer distance definition traces to ESPRIT (Sun et al. 2009, reference implementation no longer available) and the 0.42/10% calibration to DADA2 (Callahan et al. 2016) — thinly documented. The Rosen 2012 DADA paper uses ESPRIT's kmerdist as a black box. So measuring the relationship directly is the only reliable way to know whether 0.42 still makes sense outside its original Illumina-16S regime.

What it does

Reads a derep JSON (uniques[].sequence) or seqtab JSON (sequences[] + counts), samples unique-sequence pairs (random-subsample above --maxpairs to bound the O(n²) cost), and emits per-pair: kdist, edits, core_len, pct_div, screened_in, ab_i, ab_j. Summary reports screened-in fraction and leakage (screened-in but >LEAK_PCT% divergent — too far to be an error copy). Pooling mode = which sequence set you feed it (per-sample derep vs pooled table).

First Illumina V4 result (sam1F, 449 uniques, k=5)

kdist bin	median divergence
0.00–0.10	1.2%
0.30–0.40	10.0%
0.40–0.42	14.2%

• kdist=0.42 ≈ 14.6% divergence here, not 10% (~10% sits at kdist≈0.29) — the cutoff is measurably looser than its nominal calibration.
• All ≤3%-divergent pairs are screened in (the screen is safe in the error-copy regime — 0 fall above 0.42).
• ~55% of screened-in pairs are >5% divergent — the wasted-alignment leakage, now quantified.

Notes

• Test-only module (#[cfg(test)] mod), no shipping-code or CLI surface; same idiom as the bench/wfa_diagnose harnesses.
• kdist↔%div is length- and k-dependent; long-read (PacBio) mapping is the next experiment.

Test plan

DADA2RS_KDIST_INPUT=derep.json cargo test --release --bins kdist_calibrate -- --ignored --nocapture → writes kdist.csv + summary. Validated on a sam1F derep.

🤖 Generated with Claude Code