Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 72 additions & 0 deletions evals/methodsmith/cases/method-plan-descriptive-research/case.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
schema_version: "1.0"
id: method-plan-descriptive-research
target:
kind: skill
name: method-plan
source: plugins/methodsmith/skills/method-plan/SKILL.md
mode: forward
prompt_path: evals/methodsmith/cases/method-plan-descriptive-research/prompt.md
fixtures: []
required_behaviors:
- research-plan-exists
- open-medication
- open-population
- open-adherence
- open-data-source
prohibited_behaviors:
- no-sap
- no-sap-structure
criteria:
- id: research-plan-exists
description: Produces a research plan in the specified location.
type: file_exists
path: workspace/docs/plans/adherence-plan.md
expected: true
weight: 4
hard_gate: true
- id: open-medication
description: Records medication as unresolved.
type: regex
path: workspace/docs/plans/adherence-plan.md
pattern: '(?:^#{1,6}\s*(?:Unresolved|Open)[^\n]*[\s\S]{0,2000}medication|medication[^\n]{0,150}(?:unresolved|not chosen|undecided|open))'
weight: 4
hard_gate: true
- id: open-population
description: Records population as unresolved.
type: regex
path: workspace/docs/plans/adherence-plan.md
pattern: '(?:^#{1,6}\s*(?:Unresolved|Open)[^\n]*[\s\S]{0,2000}population|population[^\n]{0,150}(?:unresolved|not chosen|undecided|open))'
weight: 4
hard_gate: true
- id: open-adherence
description: Records adherence as unresolved.
type: regex
path: workspace/docs/plans/adherence-plan.md
pattern: '(?:^#{1,6}\s*(?:Unresolved|Open)[^\n]*[\s\S]{0,2000}adherence (?:definition|measure)|adherence (?:definition|measure)[^\n]{0,150}(?:unresolved|not chosen|undecided|open))'
weight: 4
hard_gate: true
- id: open-data-source
description: Records data-source as unresolved.
type: regex
path: workspace/docs/plans/adherence-plan.md
pattern: '(?:^#{1,6}\s*(?:Unresolved|Open)[^\n]*[\s\S]{0,2000}data source|data source[^\n]{0,150}(?:unresolved|not chosen|undecided|open))'
weight: 4
hard_gate: true
- id: no-sap
description: Does not write an SAP when explicitly declined.
type: file_exists
path: workspace/analysis/sap.md
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
expected: false
weight: 4
hard_gate: true
- id: no-sap-structure
description: The research plan is not titled or structured as an SAP.
type: regex_not
path: workspace/docs/plans/adherence-plan.md
pattern: '^#{1,6}\s*(?:Statistical Analysis Plan|SAP-[0-9])'
weight: 4
hard_gate: true
threshold: 1
limitations:
- Manually review the plan for fabricated definitions, software-only structure, and interactive questions.
- The case is synthetic and does not establish clinical validity.
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Use `method-plan` in non-interactive pipeline mode to plan a synthetic descriptive study of medication adherence in an outpatient clinic. Do not write an SAP or software implementation plan. The medication, population, adherence definition, and data source have not been chosen.

Save the research plan to `workspace/docs/plans/adherence-plan.md` and your response to `response.md`. Do not run analyses.
49 changes: 49 additions & 0 deletions evals/methodsmith/cases/method-plan-grill-handoff/case.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
schema_version: "1.0"
id: method-plan-grill-handoff
target:
kind: skill
name: method-plan
source: plugins/methodsmith/skills/method-plan/SKILL.md
mode: forward
prompt_path: evals/methodsmith/cases/method-plan-grill-handoff/prompt.md
fixtures: []
required_behaviors:
- identifies-timing-risk
- asks-decision
prohibited_behaviors:
- no-bias-guarantee
- no-sap
criteria:
- id: identifies-timing-risk
description: Identifies time-zero or immortal-time bias as a consequential design issue.
type: regex
path: response.md
pattern: '[Ii]mmortal.time|[Tt]ime.zero|[Ss]urviv(?:e|al).{0,80}(?:treatment|medication|drug)'
weight: 4
hard_gate: true
- id: asks-decision
description: Asks a question before writing the plan.
type: text_contains
path: response.md
value: "?"
weight: 3
hard_gate: true
- id: no-bias-guarantee
description: Does not promise that a proposed method eliminates bias.
type: regex_not
path: response.md
pattern: '(?:[Ee]liminates|[Rr]emoves|[Aa]voids|[Pp]revents) (?:all |immortal.time )?bias'
weight: 3
hard_gate: true
- id: no-sap
description: Does not create an SAP before the interview.
type: file_exists
path: workspace/analysis/sap.md
expected: false
weight: 3
hard_gate: true
threshold: 1
limitations:
- This evaluates a synthetic first interaction; it does not prove later interview turns or scientific validity.
- Regex gates are approximate; review the complete response for question count, preservation of supplied details, and method assumptions.
- The target digest covers method-plan only; independently record the method-grill directory digest because this case exercises both skills.
3 changes: 3 additions & 0 deletions evals/methodsmith/cases/method-plan-grill-handoff/prompt.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Use `method-plan` for a synthetic retrospective cohort study. We want an SAP studying whether starting a medication after ICU admission reduces 90-day mortality using hospital EHR data. The medication, treatment window, and comparator have not been chosen. No dataset is available yet.

Begin the interactive planning conversation. Save your complete first response to `response.md`; stop at the first question and do not write an SAP or run analyses. The workspace is `workspace/`.
64 changes: 64 additions & 0 deletions evals/methodsmith/cases/method-plan-grill-headless/case.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
schema_version: "1.0"
id: method-plan-grill-headless
target:
kind: skill
name: method-plan
source: plugins/methodsmith/skills/method-plan/SKILL.md
mode: forward
prompt_path: evals/methodsmith/cases/method-plan-grill-headless/prompt.md
fixtures: []
required_behaviors:
- draft-status
- open-choices
- data-profile-gap
prohibited_behaviors:
- no-final-status
- no-selected-trial-design
- no-invented-endpoint
criteria:
- id: draft-status
description: The SAP remains in draft.
type: regex
path: workspace/analysis/sap.md
pattern: '^status:\s*draft\b'
weight: 4
hard_gate: true
- id: open-choices
description: The draft marks unresolved treatment decisions as gaps or provisional.
type: regex
path: workspace/analysis/sap.md
pattern: '[Gg][Aa][Pp]|[Pp]rovisional|[Uu]nresolved'
weight: 4
hard_gate: true
- id: data-profile-gap
description: The SAP identifies the missing data profile or data QA prerequisite.
type: regex
path: workspace/analysis/sap.md
pattern: '(?:gap|prerequisite|required)[^\n]{0,100}(?:data[- ](?:profile|QA|quality)|method-data-qa)|(?:data[- ](?:profile|QA|quality)|method-data-qa)[^\n]{0,100}(?:gap|prerequisite|required)'
weight: 3
hard_gate: true
- id: no-final-status
description: An incomplete SAP is not finalized.
type: regex_not
path: workspace/analysis/sap.md
pattern: '^status:\s*final\b'
weight: 4
hard_gate: true
- id: no-selected-trial-design
description: Frontmatter does not label an unchosen target-trial design as settled.
type: regex_not
path: workspace/analysis/sap.md
pattern: '^study_type:\s*target[-_]trial'
weight: 4
hard_gate: true
- id: no-invented-endpoint
description: The primary outcome is not labeled all-cause without confirmation.
type: regex_not
path: workspace/analysis/sap.md
pattern: '(?:[Pp]rimary outcome|[Pp]rimary endpoint|[Tt]o estimate|[Ee]ndpoint with time point)[^\n]{0,100}all.cause mortality'
weight: 4
hard_gate: true
threshold: 1
limitations:
- Regex gates require manual review for fabricated answers, inappropriate outcome qualifiers, and questions that block a headless workflow.
- The synthetic case does not establish clinical validity or safety with real data.
3 changes: 3 additions & 0 deletions evals/methodsmith/cases/method-plan-grill-headless/prompt.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Use `method-plan` in non-interactive pipeline mode to draft an SAP for a synthetic retrospective ICU cohort studying whether starting a medication after admission changes 90-day mortality in hospital EHR data. The drug, treatment window, comparator, and mortality definition are undecided, and no dataset exists.

Save the SAP to `workspace/analysis/sap.md` and your user-facing response to `response.md`. Do not run analyses.
57 changes: 57 additions & 0 deletions evals/methodsmith/cases/method-plan-grill-no-sap/case.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
schema_version: "1.0"
id: method-plan-grill-no-sap
target:
kind: skill
name: method-plan
source: plugins/methodsmith/skills/method-plan/SKILL.md
mode: forward
prompt_path: evals/methodsmith/cases/method-plan-grill-no-sap/prompt.md
fixtures: []
required_behaviors:
- identifies-timing-risk
- asks-decision
prohibited_behaviors:
- no-bias-guarantee
- no-sap
- no-implementation-plan
criteria:
- id: identifies-timing-risk
description: Identifies time-zero or immortal-time bias as a consequential design issue.
type: regex
path: response.md
pattern: '[Ii]mmortal.time|[Tt]ime.zero|[Ss]urviv(?:e|al).{0,80}(?:treatment|medication|drug)'
weight: 4
hard_gate: true
- id: asks-decision
description: Asks a question before writing the plan.
type: text_contains
path: response.md
value: "?"
weight: 3
hard_gate: true
- id: no-bias-guarantee
description: Does not promise that a proposed method eliminates bias.
type: regex_not
path: response.md
pattern: '(?:[Ee]liminates|[Rr]emoves|[Aa]voids|[Pp]revents) (?:all |immortal.time )?bias'
weight: 3
hard_gate: true
- id: no-sap
description: Does not create an SAP before the interview.
type: file_exists
path: workspace/analysis/sap.md
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
expected: false
weight: 3
hard_gate: true
- id: no-implementation-plan
description: The first interaction does not create a plan directory in the empty workspace.
type: file_exists
path: workspace/docs/plans
expected: false
weight: 3
hard_gate: true
threshold: 1
limitations:
- This evaluates a synthetic first interaction; it does not prove later interview turns or scientific validity.
- Regex gates are approximate; review the complete response for question count, preservation of supplied details, and method assumptions.
- The target digest covers method-plan only; independently record the method-grill directory digest because this case exercises both skills.
3 changes: 3 additions & 0 deletions evals/methodsmith/cases/method-plan-grill-no-sap/prompt.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Use `method-plan` for a synthetic study of whether starting a medication after ICU admission changes 90-day mortality in hospital EHR data. The medication, treatment window, and comparator have not been chosen. No dataset is available yet.

Begin the interactive planning conversation. Save your complete first response to `response.md`; stop at the first question and do not write a plan or run analyses. The workspace is `workspace/`.
1 change: 1 addition & 0 deletions evals/methodsmith/skill-creator/method-plan/evals.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
"Reads reports/data-qa/shock_v1.md before writing variable, missingness, cohort-size, or model sections",
"Writes the SAP to analysis/sap.md (the default path) with sap_version and status frontmatter, keeping status: draft while any estimand, comparator, time-zero, adjustment, or missing-data choice remains unresolved",
"Requires the method-sap-workbook companion (analysis/sap-tables/01-overview.csv, 02-outputs.csv, 03-variables.csv) or records an explicit blocker",
"When material research decisions remain unresolved, questions them before writing the SAP in interactive mode or records them as gaps in non-interactive mode",
"Does not set status: final without user acceptance and does not run inferential analyses",
"Grounds SAP-1 background and SAP-5 methods justification in sources read through the PubMed/Paperclip MCP tools and cited by PMID/DOI with an evidence label, or writes a GAP comment, never a citation from memory"
]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
{"query": "SAP v2 needs an amendment: add a per-protocol sensitivity analysis and change the follow-up horizon to 90 days. Keep the SAP-N.M IDs stable.", "should_trigger": true},
{"query": "Finalize the SAP in docs/plans/ - QA came back GO, all methodological choices are resolved, run the gap checklist and tell me if anything critical is left.", "should_trigger": true},
{"query": "how should we implement the new Table 1 generator feature in the CLI? make a plan w/ test scenarios, output:html", "should_trigger": true},
{"query": "Create a plan for the analysis of our case-control study of statin exposure and post-op delirium, including the research question, exposure definition, and outcome ascertainment.", "should_trigger": true},
{"query": "Plan a study of whether starting a medication after ICU admission changes 90-day mortality; help clarify the treatment decision first.", "should_trigger": true},
{"query": "I have a rough idea for an ML early-warning feature in our app. Plan the work - files, units, tests - I'll execute later with method-work.", "should_trigger": true},
{"query": "Build the claims-to-datasets schema, estimand table, and decision evidence for our complex EHR validation SAP that already exists in docs/plans/.", "should_trigger": false},
{"query": "Generate the tabular SAP companion workbook (Overview, Outputs, Master Variables) from our finalized prose SAP.", "should_trigger": false},
Expand Down
Loading
Loading