Merge pull request #275 from ContextLab/020-deterministic-claim-caching

spec 020: deterministic claim caching + planning reference-only verification

Author: jeremymanning

.claude/skills/speckit-clarify/SKILL.md

@@ -19,6 +19,12 @@ $ARGUMENTS
 
 You **MUST** consider the user input before proceeding (if not empty).
 
+> **Scope note**: Planning artifacts state the research question, method, and
+> references, and defer specific empirical values (counts, dataset sizes, measured
+> quantities) to the implementation/research phase. Do not introduce or verify
+> low-level empirical numbers while clarifying — clarify what will be measured and
+> against which reference, not the number itself.
+
 ## Pre-Execution Checks
 
 **Check for extension hooks (before clarification)**:

.claude/skills/speckit-plan/SKILL.md

@@ -19,6 +19,12 @@ $ARGUMENTS
 
 You **MUST** consider the user input before proceeding (if not empty).
 
+> **Scope note**: Planning artifacts state the research question, method, and
+> references, and defer specific empirical values (counts, dataset sizes, measured
+> quantities) to the implementation/research phase. Do not introduce or verify
+> low-level empirical numbers in the plan — cite the source/reference for any
+> quantity rather than asserting a measured value.
+
 ## Pre-Execution Checks
 
 **Check for extension hooks (before planning)**:

.claude/skills/speckit-specify/SKILL.md

@@ -308,9 +308,15 @@ When creating this spec from a user prompt:
 
 ### Success Criteria Guidelines
 
+A planning doc states the **research question**, **method**, and **references**,
+and defers specific empirical values (counts, dataset sizes, measured quantities,
+percentages) to the implementation/research phase. Express each success criterion
+as *what* is measured and the *reference/source* it is measured against, rather
+than a pre-asserted number.
+
 Success criteria must be:
 
-1. **Measurable**: Include specific metrics (time, percentage, count, rate)
+1. **Measurable**: Name the metric (time, rate, count, percentage) and the reference/source it is measured against; do not pre-assert the empirical value here
 2. **Technology-agnostic**: No mention of frameworks, languages, databases, or tools
 3. **User-focused**: Describe outcomes from user/business perspective, not system internals
 4. **Verifiable**: Can be tested/validated without knowing implementation details

.claude/skills/speckit-tasks/SKILL.md

@@ -19,6 +19,12 @@ $ARGUMENTS
 
 You **MUST** consider the user input before proceeding (if not empty).
 
+> **Scope note**: Planning artifacts state the research question, method, and
+> references, and defer specific empirical values (counts, dataset sizes, measured
+> quantities) to the implementation/research phase. Tasks should describe what to
+> measure and against which reference; do not introduce or verify low-level
+> empirical numbers in the task list.
+
 ## Pre-Execution Checks
 
 **Check for extension hooks (before tasks generation)**:

.specify/feature.json

@@ -1,3 +1,3 @@
 {
-  "feature_directory": "specs/019-semantic-substantiation"
+  "feature_directory": "specs/020-deterministic-claim-caching"
 }

.specify/templates/plan-template.md

@@ -27,6 +27,11 @@
 **Constraints**: [domain-specific, e.g., <200ms p95, <100MB memory, offline-capable or NEEDS CLARIFICATION]  
 **Scale/Scope**: [domain-specific, e.g., 10k users, 1M LOC, 50 screens or NEEDS CLARIFICATION]
 
+> Domain-specific empirical specifics (exact counts, dataset sizes, measured
+> quantities) are deferred to the research/implementation phase. For any
+> quantity stated here, cite its source/reference rather than asserting a
+> measured value.
+
 ## Constitution Check
 
 *GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*

.specify/templates/spec-template.md

@@ -109,10 +109,14 @@
 
 ### Measurable Outcomes
 
-- **SC-001**: [Measurable metric, e.g., "Users can complete account creation in under 2 minutes"]
-- **SC-002**: [Measurable metric, e.g., "System handles 1000 concurrent users without degradation"]
-- **SC-003**: [User satisfaction metric, e.g., "90% of users successfully complete primary task on first attempt"]
-- **SC-004**: [Business metric, e.g., "Reduce support tickets related to [X] by 50%"]
+> Planning docs state *what* will be measured and the *source/reference* it is
+> measured against; defer specific empirical values (counts, dataset sizes,
+> measured quantities, percentages) to the implementation/research phase.
+
+- **SC-001**: [What is measured + reference, e.g., "Task-completion time is measured against the current manual baseline"]
+- **SC-002**: [What is measured + reference, e.g., "Throughput under load is measured against the documented capacity requirement"]
+- **SC-003**: [What is measured + reference, e.g., "Primary-task success rate is measured against the prior workflow's observed rate"]
+- **SC-004**: [What is measured + reference, e.g., "Support-ticket volume for [X] is measured against the pre-change trailing average"]
 
 ## Assumptions
 

CLAUDE.md

@@ -70,5 +70,5 @@ Since this is primarily a research documentation repository without traditional
 <!-- SPECKIT START -->
 For additional context about technologies to be used, project structure,
 shell commands, and other important information, read the current plan:
-[specs/019-semantic-substantiation/plan.md](specs/019-semantic-substantiation/plan.md).
+[specs/020-deterministic-claim-caching/plan.md](specs/020-deterministic-claim-caching/plan.md).
 <!-- SPECKIT END -->

agents/prompts/panels/panel_plan_data_resources.md

@@ -16,6 +16,13 @@ the deterministic dataset-URL-reachability check.
 - **Compute / tooling fit**: the planned analysis must be runnable with the
   free-model + free-compute constraints llmXive imposes. Plans that assume
   GPT-4 / paid voyage embeddings / >100 GPU-hours are infeasible.
+- **Cite the source, don't pre-assert the number**: a planning doc must cite the
+  source/reference for any data resource or quantity rather than asserting a
+  specific empirical count/value. Do NOT flag or kick back a planning doc merely
+  for omitting a specific empirical number (exact counts, dataset sizes, measured
+  quantities) — those are established in the implementation/research phase, not in
+  planning. Flag only when a cited resource is wrong, inappropriate, or
+  unreferenced.
 
 You do NOT judge whether the method is sound (`methodology`), whether the
 spec is covered (`spec_coverage`), or whether internal docs cohere

notes/spec-015-review-status.md

@@ -543,3 +543,51 @@ The 3-part F-14/F-20-B design, built TDD, offline-gate green. Part A still OPEN.
 **Verification:** ruff + mypy clean. Offline gate **2140 passed** / 11 skipped / 2 deselected (baseline **2107** + 33 new; ZERO regression → SC-008; includes test_exact_count_no_regress, test_grounding_service, test_fill_extract_gate, test_claim_resolve_dispatch, test_llm_call_centralization all green). **Live real-call matrix 5/5 PASSED (54s, real Dartmouth):** Almoravid-block (SC-001), Paris-grounded (SC-004), Sydney-contradicted (SC-005), bound-preguard (SC-006), STRUCTURED-skip (SC-002/003 by construction). New tests: test_fill_channels_kind (14), test_fill_relevance (5), test_fill_extract_gate +TestAcceptGate (5), test_fill_claimkind_preguard (6), test_claim_canonical +TestSubjectKeywordsPublic (2), tests/real_call/test_semantic_substantiation (5).
 
 **The trustworthiness stack is now complete:** 016 detect → 017 fill → 018 verify-modes → **019 PROSE must be semantically substantiated (subject↔value asserted, not a coincidental digit match)**.
+
+## Spec 020 — Deterministic Claim Caching + Planning Reference-Only (IN PROGRESS) — 2026-06-04
+
+Two maintainer-identified problems: (1) claim WAFFLING (a value made→flagged→corrected→
+overwritten→re-flagged in a loop); (2) low-level claims BLOCKING planning (the PROJ-552
+"49 vs 9,988" kickback loop over a number that doesn't gate progress).
+
+Pipeline run via /speckit-execute: plan→tasks→analyze (7 findings, all fixed)→implement.
+
+**DONE + verified (offline gate 2198 passed, 0 fail; baseline was 2165 — no regression):**
+- **US1 (FR-001/002/003/004)** — planning = references-only + strip/smooth. `claims/stage.py`
+  `is_planning_stage()` (planning = spec/clarify/plan/tasks). `claims/smooth.py`
+  `strip_and_smooth()` (LLM rewrite → deterministic re-detect guard → clause-removal
+  fallback; idempotent, citations preserved). `process_document(stage_label=)` planning
+  branch: extract to FIND low-level claims, strip/smooth them, NO resolve/fill/ground/
+  marker/kickback. References still gated by the separate F-18/validate_artifact path.
+  Stage threaded via `SlashCommandAgent.claim_stage_label()` (NOT derivable from
+  slash_command_name — plan_cmd & paper_plan_cmd share "speckit.plan"). Fill-boundary
+  gate (channels_for/fill_claim stage_label) as defense-in-depth.
+- **US2 freeze (FR-009/010/011/013)** — `state/claims.load_verified_by_subject()` frozen
+  (kind, subject_key) index; `resolve_registered_claims` adopts a VERIFIED subject-twin
+  WITHOUT re-resolving (closes the PENDING-with-verified-twin re-resolution gap → no
+  re-open on transient failure). state/claims/ git-tracked (cross-run), grounding-cache
+  gitignored (within-run only).
+- **US2 cache key (FR-012, partial)** — dropped `resolved_value` from the fill verdict key
+  (the safe part). The stronger "different asserted values share a key" needs hardening the
+  shared `canonical._asserted_value` disambiguation (broad blast radius) — scoped, not done.
+- **US3 (FR-006)** — templates/skills/data-resources panel defer empirical specifics; PROJ-552
+  per-project template copies synced byte-identical.
+
+**FR-007/008, SC-007 — durable placeholder: IMPLEMENTED** (maintainer chose "implement now").
+`render(placeholder_verified=True)` keeps a durable `{{claim:id}}` in the canonical stored form;
+`render_view`/`render_artifact_view` substitute values for the human/published view from the frozen
+store; `strip_claim_artifacts(preserve_pointers=True)` keeps placeholders; `process_document`
+carries over prior-round placeholders + drops orphans (round-trip is a fixed point — SC-007). The
+proven-good value-correction path is PRESERVED: `render()`'s default mode is unchanged (all existing
+render tests green), and citation repair re-anchors on the placeholder. Per clarification Q4
+"convergence operates on the placeholder form", the LLM panel/reviser operate on placeholders (NOT
+render_view in the engine — that would re-persist baked values); the published paper prose does NOT
+pass through `process_document`, so it never carries placeholders. The reviser self-consistency claim
+pass was also threaded with the planning stage (completing US1 in the convergence loop). Only two
+offline tests needed updating to the deliberately-changed contract (test_double_run_is_stable,
+test_fill_render_repairs_citation); the proven-good render tests were untouched. Full offline gate
+2211 passed / 0 failed. ALL 35 tasks complete.
+
+Commits: c77de3fa (plan) 4b2cf2c4 (tasks+analyze) 4a06d0cc (US1) 16e44ee9 (US2 freeze+key)
+58877524 (US3+real-call tests) 8cb9771e+112eeb22 (real-call fixes) 2920cf1f (durable placeholder)
+081e74f4 (reviser stage threading). Real-call sign-off: free model = qwen.qwen3.5-122b.