[1.0 T11] Evaluate fresh-session design behavior and dogfood developer/designer journeys #187

Problem and outcome

Run full regression evaluation after the T1 pilot and integrated work. Verify better reasoning and human fluency at acceptable interruption cost, rather than validating only consistent agent output.

Interaction contract

Keep FluencyLoop. Proposed positioning: Stay fluent as people and agents evolve the product.

Use retained product reasoning automatically. Explain consequential choices at the right depth. Recommend a path. Continue within agreed intent. Ask when human judgment is needed. Update the shared model at natural boundaries.

The flagship goal/principles/precedents/tradeoffs reasoning is integrated into the existing design explanation at proportional depth. It is not a mandatory recital, proposal form, or new approval exchange. People must remain able to understand and challenge the design.

Scope

• Use T1 findings and agreed thresholds; this is the full evaluation, not the first evidence of product value.

• Observe fresh Claude Code/Codex sessions for prior-decision reuse, justified revision, human judgment, delegated action, routine edits, unknown familiarity, and requested guided learning.

• Include designer artifacts, absent brief, advisory stale/missing records, backfill uncertainty, overview drift, and concurrent successors.

• Measure forced pauses, repeated questions, duplicated rationale, unchanged-context reloads, record/context size, and attention/time against 0.3 and T1.

• Observe a person finding, understanding, and challenging a consequential design choice. Keep evidence person-neutral and do not infer learning or approval from silence.

• Publish release/iterate findings and remaining limitations; narrow or defer complexity that does not earn its cost.

• Evaluate all three product-conversation moments explicitly: initial framing at the first substantial plan/feature, a later novel product fork, and an evidence-backed revision recommendation at closure/review.

• Include first-work scenarios without a prewritten brief: partially specified intent requires a recommended question before dependent design; sufficient intent requires no questionnaire. Test later sessions reusing agreed direction.

• Count useful product questions separately from avoidable interruptions. Verify the agent writes agreed direction, does not ask users to fill templates, and distinguishes missing intent from missing documentation.

Acceptance criteria

• Grounded recommendations improve actual choices without extra confirmation for already-authorized work.

• Genuine unresolved decisions receive a recommendation and necessary question, while ordinary/missing-documentation cases continue.

• Requested learning works without quizzes and unknown familiarity alone creates no mandatory checkpoint.

• Human understanding/challenge evidence is reported alongside agent behavior and overhead.

• Failures and irrelevant citations remain visible in the report; full evaluation traces back to the early pilot.

• Evidence covers both first-plan and direct-feature framing, with a material product gap resolved before dependent design and no invented audience/outcome/exclusions.

• Fully specified requests, init, routine fixes, and new sessions with settled direction do not trigger redundant product framing or document approval.

• A later initiative questions only its novel fork; evidence at review prompts a specific direction recommendation, not a recurring strategy workshop.

• The report checks that authorized revisions are not re-asked, unagreed revisions remain proposals, and unrelated delivery continues while a direction recommendation is pending.

Validation

Combine contract suites with observed sessions and human assessment of relevance/understanding. Run the plan scenarios and compare both interruption count and explanation usefulness to T1/0.3. Passing keyword or plumbing tests alone is insufficient.

Trace each product question to a concrete design consequence and its triggering moment. Include paired incomplete/sufficient-intent fixtures, cross-session reuse, a later fork, and closure evidence. Human assessment should reject both omitted necessary questions and redundant framing; do not optimize interruption counts by skipping consequential product judgment.

Starting points

tests fixtures, all four workflow skills, site views; docs/fluencyloop evaluation report; representative mnemo-warmup-style project

Constraints

• Target dev; main distributes releases. No runtime implementation occurs in the planning PR.

• T1 is an observed pilot using existing artifacts before new infrastructure. Use its evidence to retain, narrow, or defer later scope.

• Recommend and continue within authorized intent; ask with a recommendation only for consequential unresolved judgment. Batch independent questions, reuse settled answers, and wait for dependent work. Silence is not approval.

• Absent briefs, historical link gaps, stale prose, and unreviewed metadata alone are advisory. Do not require onboarding, cleanup, or retrospective approvals to work.

• Unknown familiarity is not fluency and does not automatically require a blocking probe/check. Support requested guided learning without quizzes; keep calibration/preferences private. T5 explicitly updates inherited pause rules.

• Capture rationale once, link mechanical views, record outcome deviations at natural closure, distill once at wrap-up, and reuse unchanged context.

• Preserve append-only history/provenance; Bash and PowerShell implement the same contract. Core remains Node-optional; no new database, hosted service, or vector search.

• No brief prerequisite does not waive resolving product uncertainty that determines design: establish enough direction early, reuse agreements, and reopen only for a material fork or relevant new evidence.

Dependencies and size

• Task: T11
• Relative size: L (not a calendar estimate)
• Depends on: T6: [1.0 T6] Reconcile implementation and update review/backfill workflows #182, T9: [1.0 T9] Replace feature inventories with goal-linked changes and site-wide search #185, T10: [1.0 T10] Read existing project data without migration #186
• Milestone: https://github.com/baokhang83/fluencyloop/milestone/3
• Plan: draft source
• Planning PR: docs: propose FluencyLoop 1.0 shared product direction and design #189