|
| 1 | +# Plan: OBPI-0.0.26-03-clustering-chore |
| 2 | + |
| 3 | +**OBPI:** OBPI-0.0.26-03-clustering-chore |
| 4 | +**Parent ADR:** ADR-0.0.26-evaluation-feedback-loop-doctrine |
| 5 | +**Lane:** Heavy (foundation-kind parent) |
| 6 | +**Date:** 2026-05-03 |
| 7 | + |
| 8 | +## Context |
| 9 | + |
| 10 | +OBPI-0.0.26-01 (persist `adr-evaluation` ledger events) is ATTESTED COMPLETED — |
| 11 | +event shape confirmed stable. `AdrEvaluationEvent` fields: `artifact_id`, |
| 12 | +`artifact_type`, `dimensions: dict[str, float]`, `scores: dict[str, float]`, |
| 13 | +`weighted_total`, `red_team_challenges_fired: list[str]`, `evaluator_persona`, |
| 14 | +`timestamp`. Ledger stores as `event == "adr-evaluation"`. |
| 15 | + |
| 16 | +`gz-justify` artifacts live at `artifacts/justify/<slug>-<timestamp>.md` with |
| 17 | +YAML frontmatter (`anchor_id`, `anchor_kind`, `generated_at`, `scaffold_version`) |
| 18 | +followed by 8 markdown sections. |
| 19 | + |
| 20 | +`data/eval_feedback_thresholds.json` exists with `low_score_threshold` and |
| 21 | +`red_team_count_threshold`; needs `cluster_min_recurrence: 3` added. |
| 22 | + |
| 23 | +Chore two-surface layout (ADR-0.0.21): canonical package at |
| 24 | +`src/gzkit/chores/eval-feedback-cluster/`, project overlay at |
| 25 | +`.gzkit/chores/eval-feedback-cluster/`. `proofs/` is always project-local only. |
| 26 | + |
| 27 | +## Files |
| 28 | + |
| 29 | +### New files |
| 30 | +- `src/gzkit/chores/eval_feedback_cluster_lib.py` — clustering engine |
| 31 | +- `src/gzkit/chores/eval-feedback-cluster/CHORE.md` |
| 32 | +- `src/gzkit/chores/eval-feedback-cluster/acceptance.json` |
| 33 | +- `src/gzkit/chores/eval-feedback-cluster/README.md` |
| 34 | +- `.gzkit/chores/eval-feedback-cluster/proofs/.gitkeep` |
| 35 | +- `tests/chores/test_eval_feedback_cluster.py` |
| 36 | + |
| 37 | +### Modified files |
| 38 | +- `data/eval_feedback_thresholds.json` — add `cluster_min_recurrence: 3` |
| 39 | +- `src/gzkit/chores/registry.json` — add `eval-feedback-cluster` entry |
| 40 | + |
| 41 | +## Steps |
| 42 | + |
| 43 | +### Step 1 — TDD Red: write failing tests |
| 44 | + |
| 45 | +**File:** `tests/chores/test_eval_feedback_cluster.py` |
| 46 | + |
| 47 | +Write 6 test methods in class `TestEvalFeedbackCluster`, each decorated with |
| 48 | +`@covers("REQ-0.0.26-03-NN")`. All tests use `tempfile.TemporaryDirectory` for |
| 49 | +ledger and justify-artifact fixtures. No network, no real ledger. |
| 50 | + |
| 51 | +Helper: `_write_ledger(path, events)` — writes list of dicts as JSONL. |
| 52 | +Helper: `_write_justify(dir, anchor_id, sections)` — writes a minimal justify |
| 53 | +markdown with frontmatter + section content. |
| 54 | + |
| 55 | +Test list: |
| 56 | + |
| 57 | +1. `test_zero_evidence_no_proposals` — empty ledger + no justify artifacts → |
| 58 | + `run_cluster()` returns `[]`. Covers REQ-0.0.26-03-08 (zero-evidence run). |
| 59 | + |
| 60 | +2. `test_below_threshold_no_proposal` — 2 distinct artifacts with same |
| 61 | + dimension/score_band (below `cluster_min_recurrence=3`) → returns `[]`. |
| 62 | + Covers REQ-0.0.26-03-02 (below threshold). |
| 63 | + |
| 64 | +3. `test_at_threshold_emits_proposal` — 3 distinct `adr-evaluation` events |
| 65 | + sharing `(dimension_name="clarity", score_band="low")` → returns exactly one |
| 66 | + `ProposalRecord` with all schema fields populated and all 3 source IDs. |
| 67 | + Covers REQ-0.0.26-03-03, REQ-0.0.26-03-05 (at threshold + schema). |
| 68 | + |
| 69 | +4. `test_multiple_clusters_multiple_proposals` — two independent clusters each |
| 70 | + at threshold → returns exactly two proposals. |
| 71 | + Covers REQ-0.0.26-03-04 (multiple proposals). |
| 72 | + |
| 73 | +5. `test_idempotent_rerun` — run once, write proposals to proofs dir, run again |
| 74 | + with identical evidence → second run returns 0 new proposals written (dedup |
| 75 | + by content hash). |
| 76 | + Covers REQ-0.0.26-03-06 (idempotency). |
| 77 | + |
| 78 | +6. `test_readonly_constraint` — `run_cluster()` with a read-only ledger path |
| 79 | + succeeds (reads only) and never writes outside `proofs_dir`. |
| 80 | + Covers REQ-0.0.26-03-07 (read-only at ledger/ADR surfaces). |
| 81 | + |
| 82 | +Run: `uv run -m unittest tests/chores/test_eval_feedback_cluster.py -v` → |
| 83 | +expect 6 failures (ImportError or AttributeError — lib does not exist yet). |
| 84 | + |
| 85 | +### Step 2 — Implement `src/gzkit/chores/eval_feedback_cluster_lib.py` |
| 86 | + |
| 87 | +Keep module under 600 lines (pythonic.md). All models use |
| 88 | +`ConfigDict(frozen=True, extra="forbid")`. |
| 89 | + |
| 90 | +**`ProposalRecord` model** (REQ-0.0.26-03-05): |
| 91 | +```python |
| 92 | +class ProposalRecord(BaseModel): |
| 93 | + model_config = ConfigDict(frozen=True, extra="forbid") |
| 94 | + cluster_key: str |
| 95 | + recurrence_count: int |
| 96 | + source_artifact_ids: list[str] |
| 97 | + source_artifact_paths: list[str] |
| 98 | + summary: str |
| 99 | + proposed_rule_target: str |
| 100 | +``` |
| 101 | + |
| 102 | +**`_score_band(score: float) -> str`**: maps score to band: |
| 103 | +- `< 1.5` → `"critical"`, `< 2.5` → `"very_low"`, `< 3.0` → `"low"`. |
| 104 | + |
| 105 | +**`_read_adr_evaluation_events(ledger_path: Path) -> list[dict]`**: |
| 106 | +Read ledger.jsonl line-by-line; filter `event == "adr-evaluation"`. Return list. |
| 107 | +Pure read — no writes. |
| 108 | + |
| 109 | +**`_walk_justify_artifacts(justify_root: Path) -> list[dict]`**: |
| 110 | +Walk `justify_root` for `*.md` files. Parse YAML frontmatter (between `---` delimiters). |
| 111 | +Return list of `{"path": ..., "anchor_id": ..., "sections": ..., "raw": ...}`. |
| 112 | + |
| 113 | +**`_extract_confusion_keywords(sections_text: str) -> list[str]`**: |
| 114 | +Simple keyword scan over a predefined vocabulary set: |
| 115 | +`{"unclear", "ambiguous", "confusing", "scope drift", "boundary unclear", |
| 116 | +"not sure", "uncertain", "vague", "unresolved", "conflicting"}`. |
| 117 | +Returns sorted list of matched keywords present in the text. |
| 118 | + |
| 119 | +**`_build_buckets(events, justify_artifacts, score_threshold: float) -> dict[str, list[dict]]`**: |
| 120 | +Returns `{cluster_key: [{"artifact_id": ..., "artifact_path": ...}, ...]}`. |
| 121 | +Three bucket families: |
| 122 | +- `dim:{dimension_name}:{score_band}` — for each `adr-evaluation` event, for each |
| 123 | + dimension with score < `score_threshold`. |
| 124 | +- `rt:{challenge_id}` — for each entry in `red_team_challenges_fired`. |
| 125 | +- `jk:{keyword}` — for each justify artifact, for each confusion keyword found. |
| 126 | + |
| 127 | +**`_content_hash(cluster_key: str, artifact_ids: list[str]) -> str`**: |
| 128 | +`hashlib.sha256(json.dumps([cluster_key, sorted(artifact_ids)]).encode()).hexdigest()[:16]` |
| 129 | + |
| 130 | +**`_load_existing_hashes(proofs_dir: Path) -> set[str]`**: |
| 131 | +Read all `proposal-*.json` in `proofs_dir`. Return set of |
| 132 | +`content_hash` field values. |
| 133 | + |
| 134 | +**`run_cluster(project_root: Path, *, ledger_path: Path | None = None, justify_root: Path | None = None, proofs_dir: Path | None = None, cluster_min_recurrence: int = 3, score_threshold: float = 3.0) -> list[ProposalRecord]`**: |
| 135 | +1. Resolve defaults from `project_root`. |
| 136 | +2. Read events and artifacts. |
| 137 | +3. Build buckets. |
| 138 | +4. Load existing hashes. |
| 139 | +5. For each bucket with `len(members) >= cluster_min_recurrence`: |
| 140 | + - Compute `content_hash`. |
| 141 | + - Skip if hash already in existing hashes. |
| 142 | + - Build `ProposalRecord`. |
| 143 | + - Write to `proofs_dir/proposal-<utc_timestamp>.json`. |
| 144 | +6. Return list of new `ProposalRecord` instances written. |
| 145 | + |
| 146 | +Write proofs only to `proofs_dir` — never to `ledger_path` or `justify_root`. |
| 147 | + |
| 148 | +### Step 3 — TDD Green: run tests, iterate to passing |
| 149 | + |
| 150 | +```bash |
| 151 | +uv run -m unittest tests/chores/test_eval_feedback_cluster.py -v |
| 152 | +uv run ruff check . --fix && uv run ruff format . |
| 153 | +uvx ty check . --exclude 'features/**' |
| 154 | +``` |
| 155 | + |
| 156 | +All 6 tests must pass. |
| 157 | + |
| 158 | +### Step 4 — Add `cluster_min_recurrence` to `data/eval_feedback_thresholds.json` |
| 159 | + |
| 160 | +Add `"cluster_min_recurrence": 3` to the existing JSON object. |
| 161 | + |
| 162 | +### Step 5 — Create chore package `src/gzkit/chores/eval-feedback-cluster/` |
| 163 | + |
| 164 | +**`CHORE.md`** — slug: `eval-feedback-cluster`, lane: Medium (reads ledger, |
| 165 | +walks artifacts; no network), version 1.0.0. Include: overview, policy/guardrails |
| 166 | +(read-only at ledger/ADR), workflow (run → review proposals), acceptance criteria |
| 167 | +table, evidence commands referencing proofs dir. |
| 168 | + |
| 169 | +**`acceptance.json`** — two criteria: |
| 170 | +1. `exitCodeEquals`, `uv run -m unittest tests/chores/test_eval_feedback_cluster.py -q`, `0` |
| 171 | +2. `exitCodeEquals`, `uv run gz validate --chores-layout`, `0` |
| 172 | + |
| 173 | +**`README.md`** — one-paragraph summary, quick-start command. |
| 174 | + |
| 175 | +### Step 6 — Create project-local overlay `.gzkit/chores/eval-feedback-cluster/` |
| 176 | + |
| 177 | +Create directory + `proofs/.gitkeep`. |
| 178 | + |
| 179 | +### Step 7 — Register chore in `src/gzkit/chores/registry.json` |
| 180 | + |
| 181 | +Add entry at end of `"chores"` array: |
| 182 | +```json |
| 183 | +{ |
| 184 | + "slug": "eval-feedback-cluster", |
| 185 | + "title": "Evaluation Feedback Clustering (ADR-0.0.26-03)", |
| 186 | + "version": "1.0.0", |
| 187 | + "path": ".gzkit/chores/eval-feedback-cluster", |
| 188 | + "lane": "medium" |
| 189 | +} |
| 190 | +``` |
| 191 | + |
| 192 | +### Step 8 — Full quality sweep |
| 193 | + |
| 194 | +```bash |
| 195 | +uv run gz arb ruff |
| 196 | +uv run gz arb typecheck |
| 197 | +uv run gz arb step --name unittest -- uv run -m unittest -q |
| 198 | +uv run gz validate --chores-layout |
| 199 | +uv run gz chores show eval-feedback-cluster |
| 200 | +uv run gz chores run eval-feedback-cluster |
| 201 | +``` |
| 202 | + |
| 203 | +## Verification |
| 204 | + |
| 205 | +Per brief: |
| 206 | +```bash |
| 207 | +uv run gz lint |
| 208 | +uv run gz typecheck |
| 209 | +uv run gz test |
| 210 | +uv run gz arb step --name unittest -- uv run -m unittest tests/chores/test_eval_feedback_cluster.py -v |
| 211 | +uv run gz validate --chores-layout |
| 212 | +uv run gz chores show eval-feedback-cluster |
| 213 | +uv run gz chores run eval-feedback-cluster |
| 214 | +``` |
| 215 | + |
| 216 | +## Notes |
| 217 | + |
| 218 | +- `score_band` mapping is a derived heuristic; the threshold for what counts |
| 219 | + as "low" is `score_threshold` from `eval_feedback_thresholds.json` |
| 220 | + (`low_score_threshold: 3.0`). Scores below that threshold are bucketed by |
| 221 | + band subdivision. |
| 222 | +- Confusion-keyword vocabulary is a static set; no ML, no network. |
| 223 | +- `cluster_min_recurrence` (default 3) mirrors the ADR's "≥3 times across |
| 224 | + distinct artifacts" language verbatim. |
| 225 | +- Cross-platform: all paths via `pathlib.Path`, UTF-8 encoding explicit. |
| 226 | +- No `shell=True` in any subprocess; no subprocesses needed (pure Python). |
0 commit comments