Skip to content

feat: Add Assumptions section to research workflow planning documentation (/plan phase)#99

Open
gunisha30 wants to merge 6 commits into
uw-ssec:mainfrom
gunisha30:rse-plugins-GunishaC
Open

feat: Add Assumptions section to research workflow planning documentation (/plan phase)#99
gunisha30 wants to merge 6 commits into
uw-ssec:mainfrom
gunisha30:rse-plugins-GunishaC

Conversation

@gunisha30

Copy link
Copy Markdown

Changes

Added an"Assumptions" section to three documents to improve plan quality.

Files Modified

  1. research-workflow-orchestrator.md

    • Added a point on when to document assumptions in plans
  2. plan.md

    • Expanded Assumptions Guidelines section with detailed methodology
    • Includes categories to check (library/framework behavior, environment, concurrency, data characteristics, external contracts)
    • Explains the relationship between assumptions and other plan sections (Current State Analysis, Testing Strategy, Implementation Approach)
  3. plan-template.md

    • Added Assumptions section to the implementation plan template
    • Includes structured format - Description, Basis, Impact if wrong, How to verify
    • Emphasizes that assumptions are plan-critical but unverified beliefs

Impact

  • Reduces implementation surprises caused by invalid assumptions
  • Improves communication between planning and execution phases

@cdcore09 cdcore09 requested review from cdcore09 and lsetiawan May 5, 2026 22:34
@gunisha30

Copy link
Copy Markdown
Author

Hello @cdcore09 @lsetiawan, I believe this PR has a few gaps. While trying to add ‘assumptions’ section across these 3 files, I iteratively reviewed the md file that was created when I executed /plan on one of my projects. I tried to ensure redundant content is not being added inside assumptions that is already present in other sections of the doc.

However, I did not run an end-to-end test of the entire workflow till the /validate phase to see if adding assumptions is making any difference.
What is the recommended way to test enhancements in this plugin? Happy to raise a new modified PR if you can guide on this. Thanks!

@gunisha30

Copy link
Copy Markdown
Author

Hello @lsetiawan @cdcore09 , I was repeatedly facing 504 gateway timeout error when trying to use /plan for a lengthy/complex task. Attached screenshot below (1st image). I tried changing couple of settings on my corporate laptop but nothing worked. The model is Sonnet 4.6

So, I tried to alter the instructions - to create the plan markdown file in multiple steps. I have made corresponding changes to plan.md file. And that did work for me, I tested it 3-4 times. Attaching one of the screenshots below (2nd image). Please share your feedback whenever possible. Thank you!

image image

@cdcore09

Copy link
Copy Markdown
Member

Code review

Found 2 issues, both in the rewritten Step 4 of plan.md. The Assumptions additions themselves (template section, orchestrator checklist, plan checklist line) are coherent — these issues are specific to the Step 4 procedural rewrite.

  1. The 10-section write script silently drops 8 sections that the cited template still defines. Step 4 mandates exactly Section 1/10 through Section 10/10, ending at "References and Metadata". But plan-template.md (which Step 4 instructs the agent to read, and which the Quality Checklist at L463 still requires fidelity to with [ ] Plan uses the official template from assets/) contains 18 top-level ## sections. The script omits: Migration Strategy, Risk Assessment, Edge Cases and Error Handling, Performance Considerations, Documentation Updates, Timeline Estimate, Open Questions, and Review History. Following Step 4 verbatim will produce plans that fail the quality checklist on every run.

    Step 4 procedure:

    Write and save the plan in the following 10 sequential single-section writes:
    **Section 1 — Overview**
    - Write directly to `.agents/plan-<slug>.md` (creates the file)
    - Content: What, why, and high-level how (2-3 paragraphs max)
    - After saving: output `✓ Section 1/10 saved (Overview)`
    **Section 2 — Current State Analysis**
    - Append to `.agents/plan-<slug>.md`
    - Content: Existing code with file:line references, current behavior, current limitations
    - After saving: output `✓ Section 2/10 saved (Current State Analysis)`
    **Section 3 — Desired End State**
    - Append to `.agents/plan-<slug>.md`
    - Content: New behavior description, success observable outcomes
    - After saving: output `✓ Section 3/10 saved (Desired End State)`
    **Section 4 — What We're NOT Doing**
    - Append to `.agents/plan-<slug>.md`
    - Content: Explicit scope boundaries with rationale
    - After saving: output `✓ Section 4/10 saved (Scope Boundaries)`
    **Section 5 — Assumptions**
    - Append to `.agents/plan-<slug>.md`
    - Content: Genuinely unverified beliefs, each with Basis / Impact if wrong / How to verify
    - After saving: output `✓ Section 5/10 saved (Assumptions)`
    **Section 6 — Implementation Approach**
    - Append to `.agents/plan-<slug>.md`
    - Content: Technical strategy, key architectural decisions with rationale and trade-offs, patterns to follow
    - After saving: output `✓ Section 6/10 saved (Implementation Approach)`
    **Section 7 — Implementation Phases**
    - Append to `.agents/plan-<slug>.md`
    - Content: All phases with objectives, tasks (file:line references), dependencies, and verification steps
    - Write with specificity — not "Add authentication" but "Add JWT token validation in `api/middleware/auth.py:45` following the pattern from `api/middleware/session.py:23-34`"
    - After saving: output `✓ Section 7/10 saved (Implementation Phases)`
    **Section 8 — Success Criteria**
    - Append to `.agents/plan-<slug>.md`
    - Content: Split into Automated Verification (runnable commands) and Manual Verification (human-tested steps)
    - After saving: output `✓ Section 8/10 saved (Success Criteria)`
    **Section 9 — Testing Strategy**
    - Append to `.agents/plan-<slug>.md`
    - Content: Unit tests, integration tests, manual testing, test data requirements
    - After saving: output `✓ Section 9/10 saved (Testing Strategy)`
    **Section 10 — References and Metadata**
    - Append to `.agents/plan-<slug>.md`
    - Content: Research docs, experiment reports, files analyzed, external docs, review history
    - After saving: output `✓ Section 10/10 saved (References)`

    Template sections that get dropped:

    - [How to set up test environment]
    ## Migration Strategy
    [If applicable, describe how to migrate from the old implementation to the new one]
    **Migration Steps:**
    1. [Step 1]
    2. [Step 2]
    **Rollback Plan:**
    [How to revert if something goes wrong]
    **Backward Compatibility:**
    [How existing functionality is preserved]
    ## Risk Assessment
    **Potential Risks:**
    1. **Risk:** [Description]
    - **Likelihood:** High | Medium | Low
    - **Impact:** High | Medium | Low
    - **Mitigation:** [How to reduce or handle this risk]
    2. **Risk:** [Description]
    - **Likelihood:** High | Medium | Low
    - **Impact:** High | Medium | Low
    - **Mitigation:** [How to reduce or handle this risk]
    ## Edge Cases and Error Handling
    [Document edge cases and how they should be handled]
    **Edge Cases:**
    1. **Case:** [Description]
    - **Expected Behavior:** [How system should respond]
    - **Implementation:** [Where/how this is handled]
    **Error Scenarios:**
    1. **Error:** [Description]
    - **Handling:** [How error is caught and communicated]
    ## Performance Considerations
    [If applicable, discuss performance implications]
    - **Expected Load:** [Description]
    - **Performance Targets:** [Metrics]
    - **Optimization Strategy:** [Approach]
    ## Documentation Updates
    [List documentation that needs to be created or updated]
    - [ ] Update README.md with [new information]
    - [ ] Add docstrings to new functions
    - [ ] Update API documentation at [location]
    - [ ] [Additional documentation needs]
    ## Timeline Estimate
    [Optional: Provide a rough estimate if helpful, but avoid overly precise predictions]
    - Phase 1: [Rough estimate]
    - Phase 2: [Rough estimate]
    - Total: [Rough estimate]
    **Note:** These are estimates and may change based on discoveries during implementation.
    ## Open Questions
    [CRITICAL: This section should be EMPTY in the final plan. If there are open questions, they must be resolved before the plan is complete.]
    ---
    ## References
    **Research Documents:**
    - [Research: Topic Name](research-slug.md)
    **Experiment Reports:**
    - [Experiment: Approach Comparison](experiment-slug.md)
    **Files Analyzed:**
    - `path/to/file1.ext`
    - `path/to/file2.ext`
    **External Documentation:**
    - [Link to relevant docs]
    ---
    ## Review History
    [Track plan iterations and reviews]

  2. "Append to .agents/plan-<slug>.md" with "One section = one Write/Edit tool call" is mechanically impossible. Sections 2-10 all say "Append to ...", but Claude Code's Write tool overwrites the entire file (using it for section 2 would clobber section 1), and Edit requires a prior Read plus an exact old_string/new_string match — neither natively appends. Following the instruction literally either destroys prior sections or forces an unspecified re-read-and-edit pattern per section. The original Step 4 avoided this by using a single Write call.

    1. **Write each section directly to the file — do NOT output section content as response text first.** Generate the content only inside the Write/Edit tool call.
    2. **One section = one Write/Edit tool call = one save.** Never batch multiple sections into one tool call.
    3. **After each save, output only a one-line confirmation** (e.g., `✓ Section 1/10 saved`) before proceeding to the next section.
    4. If you feel the urge to "draft" a section in your response before saving it — stop. Write directly to the file instead.
    **Before writing:** resolve any remaining open questions by researching code or asking the user. Every decision must be made before you start writing.
    ---
    Write and save the plan in the following 10 sequential single-section writes:
    **Section 1 — Overview**
    - Write directly to `.agents/plan-<slug>.md` (creates the file)
    - Content: What, why, and high-level how (2-3 paragraphs max)
    - After saving: output `✓ Section 1/10 saved (Overview)`
    **Section 2 — Current State Analysis**
    - Append to `.agents/plan-<slug>.md`
    - Content: Existing code with file:line references, current behavior, current limitations
    - After saving: output `✓ Section 2/10 saved (Current State Analysis)`

Suggestion: the Assumptions additions are ready to merge; consider splitting them from the Step 4 rewrite. For the 504 timeout that motivated the rewrite (per your earlier comment), a single Write call of the full plan is closer to the original design and consistent with how /research, /experiment, and /handoff write their artifacts. If chunking is genuinely needed, the procedure should (a) cover all template sections or explicitly mark some optional, and (b) replace "Append" with a concrete Read + Edit pattern.

🤖 Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.

@cdcore09 cdcore09 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes based on the code review above (#99 (comment)).

The Assumptions additions are good and align with the PR's stated intent. The blockers are both in the Step 4 rewrite of plan.md.

Suggested changes

  1. Split the PR. Keep the Assumptions changes (orchestrator checklist line, plan-template ## Assumptions section, Assumptions Guidelines, and the plan-level checklist entry). Move the Step 4 procedural rewrite into a separate PR so each change can be reviewed on its own merits. The PR title and description currently only advertise the Assumptions work, which makes the Step 4 rewrite easy to miss.

  2. Address the template/script mismatch in Step 4. If the section-at-a-time approach stays, the script must either cover every ## section in plan-template.md (currently 18) or explicitly mark which template sections are optional. Right now an agent that follows Step 4 verbatim will produce a plan missing Migration Strategy, Risk Assessment, Edge Cases and Error Handling, Performance Considerations, Documentation Updates, Timeline Estimate, Open Questions, and Review History — while the Quality Checklist at L463 still requires Plan uses the official template from assets/.

  3. Replace "Append" with a mechanically valid instruction. "One section = one Write/Edit tool call" combined with "Append to .agents/plan-<slug>.md" is not executable — Write overwrites, Edit requires Read + exact-match old_string. Either (a) revert to a single Write call for the full plan (matches /research, /experiment, /handoff) or (b) specify the exact Read+Edit pattern to use for each appended section.

  4. For the 504 timeout that motivated the rewrite, consider documenting it as a known transport issue with a workaround (e.g., re-run with retry, switch network) rather than encoding the workaround into the plugin instructions for all users. If the chunked write pattern is genuinely needed, applying it uniformly across the other commands would be more consistent than making /plan the lone outlier.

Happy to re-review once these are addressed.

@gunisha30

Copy link
Copy Markdown
Author

@cdcore09 Thanks for the review!
I will create a new PR containing only the assumptions.
And, a PR might not be needed for the procedural rewrite, if point 4 from your suggested changes (‘For the 504 timeout..’) has to be followed. Thanks!

@gunisha30

Copy link
Copy Markdown
Author

Hi @cdcore09, as suggested in your comment (#99 (review)), I have reverted the commit which was a temporary fix for 504 timeout error. I agree that these changes have a few issues, and might not benefit the entire /plan as a whole. The PR now only contains 'Assumptions' section changes. Thanks.

@gunisha30

Copy link
Copy Markdown
Author

Hey @cdcore09 please review and suggest changes whenever possible. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants