|
2 | 2 |
|
3 | 3 | All notable changes to AutoDev. Format based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versioning per [SemVer](https://semver.org/spec/v2.0.0.html). |
4 | 4 |
|
5 | | -## [Unreleased] - v0.32.0 Phase 2 |
| 5 | +## [0.32.0] - 2026-05-16 |
| 6 | + |
| 7 | +Hardening release closing the soft-block loop. v0.31.0 fixed every |
| 8 | +v0.30.1-class failure but exposed a different layer (architect plan- |
| 9 | +phase failures, silent test-runner 0/0/0, soft-block→thin-evidence |
| 10 | +loop with no recovery action, invisible blocked-state). v0.32.0 |
| 11 | +attacks that layer AND the structural pattern of "every release |
| 12 | +fixes the most-recent surface and exposes the next one" via a |
| 13 | +standing retrospective discipline. |
| 14 | + |
| 15 | +Ten distinct phases shipped under this release. Per-phase summary |
| 16 | +follows; full details in the v0.32.0 plan at |
| 17 | +`~/.claude/plans/we-have-multiple-problems-transient-cray.md`. |
| 18 | + |
| 19 | +### Phase 0 — v0.31.1 hot-patch (already shipped) |
| 20 | +The Phase 1.1 empty-result dump silently skipped when the CLI |
| 21 | +emitted `is_error=true` alongside `result==""`. Dropped the |
| 22 | +`is_error` gate. Live as v0.31.1. |
| 23 | + |
| 24 | +### Phase 1 — Architect convergence (Gaps A + B + F) |
| 25 | +- **Architect feedback loop:** `src/orchestrator/retry_envelope.py` |
| 26 | + gains `most_recent_failures: list[PriorError]`. The architect |
| 27 | + prompt at `src/agents/prompts/architect.md` interpolates a |
| 28 | + `{rejection_history}` placeholder so retries see "you proposed |
| 29 | + these paths and they were rejected; do not propose them again". |
| 30 | +- **Generic budget escalation:** `BudgetEscalationTracker` (Phase 3 |
| 31 | + of v0.31.0) refactored to key on `(scope_id, role)` instead of |
| 32 | + per-task only. The architect retry loop now escalates `max_turns` |
| 33 | + + `timeout_s` + (optionally) sonnet → opus on repeated failures. |
| 34 | + New ledger op `plan_phase_budget_escalation`. |
| 35 | +- **Plan-time failure-class taxonomy:** new |
| 36 | + `src/state/failure_classes.py` with `FailureClass` enum |
| 37 | + distinguishing ExecuteTime / PlanTime / Infrastructure failures |
| 38 | + and a `classify(...)` dispatcher. |
| 39 | +- **Hard-fail recovery tiers:** new |
| 40 | + `src/orchestrator/plan_phase_recovery.py` with tier-4 scope |
| 41 | + degradation, tier-5 model escalation, tier-6 user escalation |
| 42 | + (populates `RecoveryHint`), tier-7 hard-fail with forensic |
| 43 | + summary referencing the archived `architect-failed-*.md` dumps. |
| 44 | + |
| 45 | +### Phase 2 — Autoreason A/B/AB reviewer pipeline (already documented above this entry; preserved verbatim) |
| 46 | +**Feature flag `cfg.tournaments.review_tournament_enabled` |
| 47 | +defaults to `False`** for v0.32.0 — opt-in for one cycle to gather |
| 48 | +real-world telemetry; v0.33.0 will flip the default. Includes the |
| 49 | +full A / B / AB candidate generation, three fresh judges with |
| 50 | +Borda voting, "do nothing" convergence, new |
| 51 | +`ReviewTournamentEvidence` schema, four new ledger ops, and full |
| 52 | +preservation of v0.31.0's reviewer instrumentation (chunked |
| 53 | +envelope, MALFORMED parsing, raw_response capture, empty-result |
| 54 | +dump). New roles: `adversarial_reviewer`, `merge_synthesizer`. |
| 55 | +Cost: ~4–5× per-review token spend in exchange for dramatically |
| 56 | +lower soft-block rate. |
| 57 | + |
| 58 | +### Phase 3 — Test runner self-diagnostic (Gap C) |
| 59 | +- `TestEvidence` schema gains five optional fields: |
| 60 | + `runner_returncode`, `tests_collected`, `collection_error`, |
| 61 | + `runner_stderr_tail`, `diagnosis: Literal[...]`. |
| 62 | +- New `src/orchestrator/test_result_classifier.py::classify_test_result()` |
| 63 | + with a 6-way diagnosis taxonomy: `ok`, `no_tests_found`, |
| 64 | + `collection_failed`, `runtime_crash`, `capture_failed`, |
| 65 | + `no_signal`. |
| 66 | +- Orchestrator-side handling now branches on diagnosis instead of |
| 67 | + the legacy `failed > 0 or not success` heuristic. `no_tests_found` |
| 68 | + no longer soft-blocks (legitimate); `collection_failed` retries |
| 69 | + once then hard-fails with `RecoveryHint`; etc. |
| 70 | +- Backward-compatible: pre-v0.32.0 evidence files load with |
| 71 | + `diagnosis=None`. |
| 72 | + |
| 73 | +### Phase 4 — Knowledge-aware escalation |
| 74 | +- **PRM as gate:** `src/orchestrator/escalation_ladder.py::next_step()` |
| 75 | + now accepts an optional `knowledge_context` dict. When PRM has |
| 76 | + detected `repetition_loop` for the current task, the ladder forces |
| 77 | + PIVOT instead of REFINE; `ping_pong` escalates further. |
| 78 | +- **KB lookup before refine:** new |
| 79 | + `src/orchestrator/knowledge_lookup.py::lookup_recent_failures()` |
| 80 | + with a 100ms timeout fallback. Past failures on the current task |
| 81 | + are injected into the critic's STUCK_CONTEXT so the critic sees |
| 82 | + "we've already tried X and Y." |
| 83 | +- **KnowledgeEntry metadata additions:** documented optional keys |
| 84 | + (`kb_entry_type`, `task_signature`, `task_id`, `tactic_tried`, |
| 85 | + `resolution`); new `compute_task_signature(task)` helper. |
| 86 | +- **Repetition-loop recovery taxonomy:** new |
| 87 | + `src/orchestrator/repetition_recovery.py::choose_recovery_action()` |
| 88 | + mapping per-task counters + PRM signal onto seven recovery |
| 89 | + actions (switch_tactic, increase_scope, decrease_scope, |
| 90 | + re_architect, kb_lookup, ask_human, do_nothing). |
| 91 | +- New ledger ops: `repetition_loop_detected`, |
| 92 | + `recovery_action_chosen`, `tactic_switch`. |
| 93 | + |
| 94 | +### Phase 5 — Soft-block surfacing (Gap G) |
| 95 | +- New `RecoveryHint` pydantic schema and `Task.recovery_hint` |
| 96 | + optional field. Six classes: `missing_test_output`, |
| 97 | + `thin_review_evidence`, `architect_unconvergent`, |
| 98 | + `model_capacity_exhausted`, `user_decision_required`, |
| 99 | + `network_transient`. Carries recommended user action, evidence |
| 100 | + files, debug files, and CLI commands to try. |
| 101 | +- Populated at every soft-block site: execute_phase exhaustion, |
| 102 | + plan_phase tier-6 user escalation, test gate `collection_failed` |
| 103 | + / `capture_failed` / `no_signal`. |
| 104 | +- New `autodev status --blocked` flag renders blocked tasks as |
| 105 | + rich panels with class, reason, evidence files, recommended |
| 106 | + action, and copyable commands. |
| 107 | +- `autodev doctor` reports blocked-task count. |
| 108 | +- `autodev plan` / `execute` / `resume` print a pre-flight banner |
| 109 | + when blocked tasks exist, directing users to |
| 110 | + `autodev status --blocked`. |
| 111 | + |
| 112 | +### Phase 6 — CI gate closure |
| 113 | +- New `tests/fixtures/fake_binaries/fake-pytest` covers |
| 114 | + `no_tests_collected`, `zero_pass_zero_fail`, `collection_error`, |
| 115 | + `runtime_crash`, `capture_failed` modes. |
| 116 | +- `fake-claude` extended with `architect_rejected_paths_{1,2,3}`, |
| 117 | + `repetition_loop`, `reviewer_returns_empty_silently` modes. |
| 118 | +- `fake-cursor` extended with `is_error_true_with_empty_result` |
| 119 | + mode (regression for the v0.31.1 dump-predicate fix). |
| 120 | +- Six new integration tests under |
| 121 | + `tests/integration/test_e2e_with_fake_binaries.py` covering: |
| 122 | + Phase 1.1 dump regression, architect-failed dumps, architect |
| 123 | + budget escalation, test-runner silent zero, repetition-loop |
| 124 | + recovery action, blocked-task RecoveryHint surfacing. |
| 125 | +- Four new `release.yml` preflight grep checks lock the integration |
| 126 | + anchors for Phase 1.1 (architect rejection_history), |
| 127 | + Phase 1.1 dump fix (pytest gate), Phase 3 (TestEvidence.diagnosis), |
| 128 | + Phase 5 (RecoveryHint + status.py integration). |
| 129 | + |
| 130 | +### Phase 7 — Real-task benchmark v1 |
| 131 | +- New `benchmarks/` tree with runner + scorer + 5 fully-synthetic |
| 132 | + MIT-licensed tasks across Python / TypeScript / Go covering |
| 133 | + type errors, null checks, off-by-one, resource cleanup, and |
| 134 | + string-allocation perf. |
| 135 | +- Runner produces `results.json` with primary pass/fail metric and |
| 136 | + secondaries (wall_time_s, autodev_calls, diff_size_delta_bytes). |
| 137 | +- Designed as the missing fitness signal for any future external |
| 138 | + optimiser (GEPA prompt evolution, RL training). Not yet wired |
| 139 | + into CI — that's a v0.32.1 follow-up. |
| 140 | + |
| 141 | +### Phase 8 — Standing "next-layer" retrospective discipline |
| 142 | +- New `docs/release-retrospective-template.md` with a REQUIRED |
| 143 | + Section 5: "What's the NEXT layer of failure?" — explicit |
| 144 | + speculation about what will surface AFTER this release's fixes |
| 145 | + ship. At least 3 candidates with severity guesses and falsifiable |
| 146 | + signals. |
| 147 | +- New `.github/pull_request_template.md` with a release-bump |
| 148 | + checklist that requires the prior version's retrospective have a |
| 149 | + populated Section 5. |
| 150 | +- New CI gate in `release.yml::preflight-checks` that parses the |
| 151 | + most-recent retrospective and fails if Section 5 has fewer than |
| 152 | + 3 bullets. |
| 153 | +- New `docs/retrospectives/v0.31.1.md` demonstrates the template |
| 154 | + and seeds the discipline; lists 4 candidates feeding v0.33.0 |
| 155 | + planning (autoreason cost ceiling at scale, KB recall on novel |
| 156 | + signatures, test-runner diagnosis on non-pytest/jest runners, |
| 157 | + RecoveryHint catalog drift). |
6 | 158 |
|
7 | | -### Added |
8 | | -- **Review tournament (autoreason A/B/AB pipeline)** — |
9 | | - `src/orchestrator/review_tournament_runner.py` introduces a |
10 | | - three-candidate refinement step for the reviewer: |
11 | | - - **A** is the unchanged developer patch + the original reviewer |
12 | | - verdict. |
13 | | - - **B** is an adversarial second-opinion review produced by the |
14 | | - new `adversarial_reviewer` role |
15 | | - (`src/agents/prompts/adversarial_reviewer.md`) — deliberately |
16 | | - framed to find the angle the original reviewer missed. |
17 | | - - **AB** is a synthesis produced by the new `merge_synthesizer` |
18 | | - role (`src/agents/prompts/merge_synthesizer.md`) that combines |
19 | | - A's strengths with B's improvements. |
20 | | - Three FRESH judges (default cohort: |
21 | | - `["judge", "minimality_judge", "judge_explorer"]`) blindly score |
22 | | - the candidates via Borda count. "Do nothing" (A wins |
23 | | - `convergence_k=2` rounds in a row) is a first-class verdict so |
24 | | - the loop converges on "the original was fine, stop" instead of |
25 | | - burning developer-refine cycles. |
26 | | -- New evidence type |
27 | | - `state.schemas.ReviewTournamentEvidence` carrying tournament_id, |
28 | | - candidates dict (A / B / AB), judge rankings, Borda scores, |
29 | | - winner, valid_judges count, converged flag, and rounds; written |
30 | | - to `.autodev/evidence/{task_id}-review_tournament.json`. |
31 | | -- New ledger ops `review_tournament_started`, |
32 | | - `review_tournament_judged`, `review_tournament_converged`, |
33 | | - `review_tournament_escalated` (audit-only — replay is a no-op). |
34 | | -- New config knobs on `TournamentsConfig`: |
35 | | - `review_tournament_enabled` (default `False`), |
36 | | - `review_num_judges`, `review_convergence_k`, `review_max_rounds`, |
37 | | - `review_judge_roles`. Mirror knobs added to the runtime |
38 | | - `tournament.core.TournamentConfig` dataclass. |
39 | | - |
40 | | -### Feature flag rollout (opt-in for v0.32.0) |
41 | | -- The review tournament ships **off by default** for one cycle. To |
42 | | - opt in, set `cfg.tournaments.review_tournament_enabled = true` |
43 | | - in `.autodev/config.json` (or programmatically before constructing |
44 | | - the orchestrator). The legacy single-shot reviewer path is |
45 | | - byte-identical when the flag is `false`. |
46 | | -- v0.33.0 will flip the default to `true` after one cycle of |
47 | | - real-world telemetry confirms the do-nothing convergence rate |
48 | | - matches the autoreason published technique (NousResearch). |
49 | | -- All v0.31.0 instrumentation is preserved by construction: |
50 | | - - each candidate is grounded against the same chunked review |
51 | | - envelope (Phase 1.4), |
52 | | - - each candidate's verdict parses through the existing |
53 | | - `_parse_review_verdict` so the Phase 1.3 MALFORMED-vs-content |
54 | | - distinction propagates, |
55 | | - - `raw_response` is captured on every candidate (Phase 1.2), |
56 | | - - the empty-result `*-empty.json` dump still fires from the |
57 | | - adapter layer. |
| 159 | +### Schema / migration |
| 160 | +- All new fields on `TestEvidence`, `Task`, `KnowledgeEntry` are |
| 161 | + optional with `None` defaults. v0.31.x evidence files load |
| 162 | + without migration. |
| 163 | +- `Task.recovery_hint`, `TestEvidence.diagnosis`, |
| 164 | + `ReviewTournamentEvidence` are additive. |
58 | 165 |
|
59 | | -### Tests added |
60 | | -- `tests/test_review_tournament_core.py` — Borda + tiebreak |
61 | | - invariants, `_no_progress` short-circuit, `_resolve_judge_cohort` |
62 | | - precedence (14 tests). |
63 | | -- `tests/test_review_tournament_integration.py` — full-flow |
64 | | - StubAdapter coverage of: B/AB-wins exit semantics, no-progress |
65 | | - short-circuit (judges never called), max-rounds escalation, |
66 | | - chunked-envelope reuse across all three candidates, empty-A |
67 | | - MALFORMED propagation, evidence + ledger breadcrumb writes (6 |
68 | | - tests). |
| 166 | +### Tests |
| 167 | +- 2955 passed, 2 skipped (vs ~2710 before v0.32.0; +245 new tests). |
| 168 | +- New test files: `test_architect_feedback_loop.py`, |
| 169 | + `test_plan_phase_budget_escalation.py`, `test_plan_phase_recovery.py`, |
| 170 | + `test_state_failure_classes.py`, `test_review_tournament_core.py`, |
| 171 | + `test_review_tournament_integration.py`, |
| 172 | + `test_orchestrator_test_result_classifier.py`, |
| 173 | + `test_orchestrator_test_handling.py`, |
| 174 | + `test_test_evidence_schema_backcompat.py`, |
| 175 | + `test_orchestrator_knowledge_lookup.py`, |
| 176 | + `test_orchestrator_repetition_recovery.py`, |
| 177 | + `test_state_knowledge_task_signature.py`, |
| 178 | + `test_recovery_hint_populated.py`, `test_status_blocked_flag.py`, |
| 179 | + `test_doctor_blocked_section.py`, `test_blocked_banner_on_resume.py`, |
| 180 | + `test_benchmark_runner.py`, `test_retrospective_discipline.py`, |
| 181 | + `test_orchestrator_plan_phase_failure_classes.py`. |
| 182 | + |
| 183 | +### Out of scope (deferred to v0.33+) |
| 184 | +- `agent-governance-toolkit` adoption (re-evaluate after one cycle |
| 185 | + of v0.32.0 in production). |
| 186 | +- `atropos` adoption (would build on Phase 7's benchmark). |
| 187 | +- `hermes-agent-self-evolution` GEPA prompt evolution (requires the |
| 188 | + benchmark + a stable prompt corpus). |
| 189 | +- Auto-routing for the new `repetition_recovery` actions |
| 190 | + (`increase_scope`, `re_architect`, `do_nothing`) — currently |
| 191 | + emitted as ledger telemetry; dispatch routing lands as a follow-up. |
69 | 192 |
|
70 | 193 | ## [0.31.1] - 2026-05-15 |
71 | 194 |
|
|
0 commit comments