Skip to content

Commit 2349279

Browse files
lhy0718lhy0718
committed
Add paper-readiness audit CLI
1 parent a10aaf8 commit 2349279

13 files changed

Lines changed: 1631 additions & 11 deletions

‎docs/design/README.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Design Notes
2+
3+
This directory contains bounded design reviews and implementation notes that support the canonical docs in `docs/`.
4+
5+
These notes are not claims that the described runtime behavior is already implemented or validated. Treat them as scoped contracts, design constraints, or future implementation plans unless the relevant checklist item and tests say otherwise.
6+
7+
## Current Notes
8+
9+
- `deep-reviewer-backend-integration-study.md`
10+
- `stage-evolution-design.md`
11+
- `exploration-knowledge-retention-review.md`
12+
- `multimodal-memory-layer-review.md`
13+
- `node-output-serialization-stability-audit.md`
14+
15+
## Placement Rule
16+
17+
Keep canonical user-facing and source-of-truth docs in `docs/`. Put longer design reviews, pressure tests, future implementation contracts, and P2/P3 planning notes here when they would otherwise crowd the docs root.

docs/deep-reviewer-backend-integration-study.md renamed to docs/design/deep-reviewer-backend-integration-study.md

File renamed without changes.

‎docs/design/exploration-knowledge-retention-review.md‎

Lines changed: 169 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,169 @@
1+
# ExplorationManager Knowledge-Retention Review
2+
3+
This note defines the P2-6 design review for autonomous knowledge retention inside `ExplorationManager`. It is not an implementation claim and does not report measured autonomous research improvement.
4+
5+
Knowledge retention means preserving branch, failure, evidence, and decision context so later node-internal exploration can avoid repeated mistakes and reuse defensible evidence traces. It does not mean persistent autonomous discovery is already solved, and it must not turn weak evidence into paper-ready claims.
6+
7+
## Boundary
8+
9+
The governed top-level workflow remains fixed:
10+
11+
`collect_papers -> analyze_papers -> generate_hypotheses -> design_experiments -> implement_experiments -> run_experiments -> analyze_results -> figure_audit -> review -> write_paper`
12+
13+
Knowledge retention is an internal sidecar for bounded exploration. It must not add workflow nodes, bypass `review`, weaken `claim ceiling`, or replace run-scoped artifacts as the source of truth.
14+
15+
Current relevant surfaces:
16+
17+
- `experiment_tree/research_tree.json`
18+
- `experiment_tree/manager_state.json`
19+
- `failure_memory.jsonl`
20+
- `result_table.json`
21+
- `figure_audit/figure_audit_summary.json`
22+
- `review/decision.json`
23+
- `paper/evidence_gate_decision.json`
24+
- `paper/paper_readiness.json`
25+
26+
## What May Be Retained
27+
28+
Allowed retained knowledge:
29+
30+
- branch identity, parentage, stage, and change set
31+
- baseline lock references and single-change checks
32+
- executed evidence manifests
33+
- reproducibility status and reproduction count
34+
- objective metrics and result-table links
35+
- failure fingerprints and retry policy
36+
- stage transition decisions and rollback reasons
37+
- promotion decisions and blocking reasons
38+
- figure audit summaries and severe mismatch counts
39+
- claim fingerprints that were blocked or downgraded
40+
41+
Disallowed retained knowledge:
42+
43+
- private absolute paths or machine-local source roots
44+
- raw credentials, environment secrets, or provider tokens
45+
- unverifiable natural-language summaries without artifact links
46+
- paper-ready labels not backed by result, review, and claim-evidence artifacts
47+
- hidden failed-run state
48+
- memories that override baseline/comparator requirements
49+
50+
## Proposed Retention Artifact
51+
52+
Future implementation should write a separate sidecar instead of overloading `manager_state.json`:
53+
54+
`experiment_tree/knowledge_retention.json`
55+
56+
Recommended fields:
57+
58+
- `run_id`
59+
- `generated_at`
60+
- `source_artifacts`
61+
- `retained_branch_summaries`
62+
- `blocked_failure_fingerprints`
63+
- `reusable_evidence_refs`
64+
- `non_reusable_evidence_refs`
65+
- `claim_ceiling_notes`
66+
- `resume_hints`
67+
- `redaction_status`
68+
- `schema_version`
69+
70+
The artifact should contain repo-relative or run-relative references only. It should not record external local note paths, temporary workspaces, or non-repository mirror roots.
71+
72+
## Retention Rules
73+
74+
Retention may guide future exploration only when the retained item is traceable.
75+
76+
Required traceability:
77+
78+
- branch summary links to a research-tree node id
79+
- metric claim links to `result_table.json`
80+
- baseline/comparator decision links to `baseline_lock.json` or result-table rows
81+
- failed branch memory links to a failure fingerprint
82+
- figure claim links to `figure_audit_summary.json`
83+
- paper-readiness note links to review and paper gate artifacts
84+
85+
If traceability is missing, the item may be retained as an operational hint but not as evidence.
86+
87+
## Reuse Rules
88+
89+
Retained knowledge may be reused for:
90+
91+
- avoiding a blocked failure fingerprint
92+
- prioritizing a branch family with reproduced evidence
93+
- warning that a stage transition previously failed
94+
- preloading baseline/comparator constraints
95+
- reminding review that a claim was previously downgraded
96+
- reducing repeated equivalent debug branches
97+
98+
Retained knowledge may not be reused for:
99+
100+
- claiming improvement without a fresh or traceable comparator
101+
- skipping result-table validation
102+
- skipping figure audit
103+
- treating fallback-only output as quantitative evidence
104+
- auto-promoting a branch to paper-ready
105+
- suppressing failed-run visibility
106+
107+
## Resume And Reload Contract
108+
109+
Knowledge retention must survive resume without changing meaning.
110+
111+
Resume checks should verify:
112+
113+
- retained branch ids still exist in `research_tree.json`
114+
- referenced artifacts still exist and are non-empty
115+
- blocked failure fingerprints still match `failure_memory.jsonl`
116+
- current stage and retained stage hints do not conflict
117+
- retained promotion notes do not contradict the latest review decision
118+
- failed run state remains visible
119+
120+
If a retained reference is stale, the system should downgrade it to a non-evidence resume hint and record a repair action. It should not silently drop the stale reference if that would hide a failure or blocker.
121+
122+
## Claim Ceiling Interaction
123+
124+
The strongest allowed claim from retained knowledge is capped by the weakest linked artifact.
125+
126+
Examples:
127+
128+
- retained branch summary without executed evidence: operational planning hint only
129+
- executed branch without baseline/comparator: descriptive result only
130+
- fallback-only evidence: system validation note only
131+
- result table without citation support for related work: related-work claim downgraded
132+
- figure mismatch present: no manuscript promotion
133+
- hidden failed run: blocked
134+
135+
`write_paper` completion remains only a workflow signal. It is not a retained proof of paper readiness.
136+
137+
## Audit Report Interaction
138+
139+
`autolabos audit` should be able to read retained knowledge later, but the audit verdict must remain controlled by artifact-level evidence:
140+
141+
- governance artifact contract
142+
- result table completeness
143+
- claim-evidence support
144+
- figure audit
145+
- review decision
146+
- paper readiness gate
147+
- failed-run visibility
148+
149+
Knowledge retention can add context to blockers and next actions. It cannot override blockers.
150+
151+
## Validation Plan
152+
153+
Before runtime implementation:
154+
155+
1. Add a schema test for `knowledge_retention.json`.
156+
2. Add stale-reference tests for missing branch ids and missing artifacts.
157+
3. Add resume tests proving retained hints do not change stage decisions without policy approval.
158+
4. Add audit tests proving retained knowledge cannot override missing baseline/comparator, fallback-only evidence, or figure mismatch blockers.
159+
5. Run `npm test -- tests/explorationManager.test.ts tests/explorationStatus.test.ts tests/paperReadinessAudit.test.ts`.
160+
6. Run `npm run validate:harness` and `npm run build` if runtime code changes.
161+
162+
## Completion Criteria For A Future Implementation
163+
164+
- Knowledge retention is run-scoped and inspectable.
165+
- Retained facts have artifact references or are explicitly marked as non-evidence hints.
166+
- Resume/reload preserves retained context without changing evidence meaning.
167+
- Failed branches and blocked claims remain visible.
168+
- Claim ceilings are never raised by memory alone.
169+
- Audit reports can use retention context only as explanatory support, not as a readiness override.

0 commit comments

Comments
 (0)