feat(development-pr-workflow): add /backtest-change command by dylanschmittle-uniswap · Pull Request #532 · Uniswap/ai-toolkit

dylanschmittle-uniswap · 2026-06-05T21:30:01Z

What

Adds a new agnostic slash command /backtest-change to the development-pr-workflow plugin: before opening a PR for a data-driven change (monitor threshold, alert routing / re-notify cadence, metric/log/trace query, sampling rate, perf tweak), pull the relevant live historical data, replay old-vs-new over the same window, and report whether the change actually achieves its goal — refusing to ship (or redirecting) when the data disproves the premise.

Registered in plugin.json + the README commands table; command body in commands/backtest-change.md.

Why

Codified from a real session where the obvious fix (raising a Datadog monitor's re-notify interval to cut incident.io AI-investigation cost) looked right but a backtest of live alert history showed the dominant driver was monitor flapping, not re-notification — so the PR was held instead of shipped. The reusable lesson: validate a measurable change against real data before the PR, and be willing to reverse. The plugin's existing commands cover review/merge mechanics but nothing gates a data-backed change on evidence — this fills that gap.

Key guardrails baked in:

Respects data-source sampling (metrics ~100%; indexed spans/logs keep errors but ~1% of successes — never compute rates from span/log counts).
Classifies EFFECTIVE / PARTIAL / INEFFECTIVE-PREMISE-DISPROVED and stops on the last.
Prefers dashboard/query links over stale typed numbers.

Test plan

Command frontmatter matches existing commands (description / argument-hint / allowed-tools)
Added to .claude-plugin/plugin.json commands array + README table
npm/bun plugin validation / CI green
Manual: /backtest-change is discoverable after install

AI-Generated Description

What

Adds a new agnostic slash command /backtest-change to the development-pr-workflow plugin. Before opening a PR for a data-driven change (monitor threshold, alert routing / re-notify cadence, metric/log/trace query, sampling rate, perf tweak), the command pulls the relevant live historical data, replays old-vs-new over the same window, and reports whether the change actually achieves its goal — refusing to ship (or redirecting) when the data disproves the premise.
Registered in plugin.json + the README commands table; command body in commands/backtest-change.md.

Why

Codified from a real session where the obvious fix (raising a Datadog monitor's re-notify interval to cut incident.io AI-investigation cost) looked right but a backtest of live alert history showed the dominant driver was monitor flapping, not re-notification — so the PR was held instead of shipped. The reusable lesson: validate a measurable change against real data before the PR, and be willing to reverse.
The plugin's existing commands cover review/merge mechanics but nothing gates a data-backed change on evidence — this fills that gap.
Key guardrails baked into the command body:

Respects data-source sampling (metrics ~100%; indexed spans/logs keep errors but ~1% of successes — never compute rates from span/log counts).
Classifies EFFECTIVE / PARTIAL / INEFFECTIVE-PREMISE-DISPROVED and stops on the last.
Prefers dashboard/query links over stale typed numbers.

Changes

File	Change
`packages/plugins/development-pr-workflow/commands/backtest-change.md`	New command (+54 lines) with frontmatter (`description`, `argument-hint`, `allowed-tools`) and a six-step workflow: hypothesis → data source → window → old-vs-new replay → classification → PR-or-stop
`packages/plugins/development-pr-workflow/.claude-plugin/plugin.json`	Register the new command in the `commands` array (alphabetized at the top); bump `version` `2.1.0` → `2.2.0` per the root `CLAUDE.md` policy ("After making any changes to files in `packages/plugins/`, Claude Code MUST bump the plugin version")
`packages/plugins/development-pr-workflow/README.md`	Add the `/backtest-change` row to the commands table
Total diff: +57 / -1 across 3 files.

Test plan

Command frontmatter matches the shape of the other commands in this plugin (description, argument-hint, allowed-tools)
Plugin commands array entry resolves to the new file path
README commands table renders cleanly with the new row
Plugin version bumped (minor: new command added — backward-compatible enhancement)
CI green (plugin validation / lint / typecheck)
Manual: /backtest-change is discoverable after install
Manual: invoke /backtest-change end-to-end on a real data-driven change and confirm the verdict classification gates the PR

A validate-before-PR command: for any data-driven change (monitor thresholds, alert routing/renotify, queries, sampling, perf tweaks), pull live historical data, replay old-vs-new, and report whether it actually achieves its goal — refusing to ship or redirecting when the data disproves the premise. Registered in plugin.json + README commands table.

vercel · 2026-06-05T21:30:07Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
ai-toolkit-slack-oauth-backend	Ready	Preview, Comment	Jun 9, 2026 5:59pm

github-actions · 2026-06-05T21:31:57Z

📚 Documentation Check ✅

Verdict: Passed

Version was correctly bumped (2.1.0 → 2.2.0, minor for new command), README was updated with the new command, and plugin.json references the new command file. All critical checks pass.

PR #532 Documentation Check — PASS

This PR adds the /backtest-change command to the development-pr-workflow plugin. All critical documentation requirements are satisfied:

Check	Status
Plugin version bump	✅ 2.1.0 → 2.2.0 (correct minor bump for new command)
README updated	✅ New command row added to commands table
plugin.json updated	✅ `./commands/backtest-change.md` added to `commands` array
CLAUDE.md updated	⚠️ Not in diff — may want to add the new command to the plugin's CLAUDE.md component list

The version bump is appropriate: a new user-invocable command is a backward-compatible addition, warranting a minor version increment.

Missing Updates

Type	File	Severity	Reason
📘 claude_md	`packages/plugins/development-pr-workflow/CLAUDE.md`	ℹ️ info	New command /backtest-change was added to the plugin but the CLAUDE.md component list was not updated in this PR. The CLAUDE.md may document plugin commands and should reflect the new addition.

Suggestions (1)

💡 Inline suggestions have been posted as review comments. Click "Commit suggestion" to apply each fix directly.

ℹ️ packages/plugins/development-pr-workflow/CLAUDE.md: The plugin's CLAUDE.md component inventory (if it lists commands) should be updated to include the new /backtest-change command so the documentation stays in sync with plugin.json and the README.

_{🤖 Generated by Claude Documentation Validator | Mode: suggest}

…r new command

github-actions · 2026-06-05T22:19:34Z

🤖 Claude Code Review

🔄 Review in progress... Claude is analyzing this pull request. If a review can be seen below, it will be replaced by the results of this one.

_{⏱️ Reviews typically complete within 5-15 minutes depending on PR size.}

Summary

This PR adds a new /backtest-change slash command to the development-pr-workflow plugin — a workflow guide for validating data-driven changes (monitor thresholds, alert routing, sampling rates, perf tweaks) against historical data before opening a PR. It includes a version bump (2.1.0 → 2.2.0) and README update.

The command file is documentation/instructions for Claude, not executable code, so there's no bug/security surface to evaluate. The change is well-scoped and follows the plugin conventions.

What works well

Version bump matches the CLAUDE.md policy (minor for a new command)
Filename backtest-change.md follows the verb-noun convention required for user-invocable commands
No name field in frontmatter — correctly relies on the filename, per plugin conventions
allowed-tools is scoped to read/inspection tools (Bash, Read, Grep, Glob, Task, AskUserQuestion) — appropriate for a pre-PR validation workflow
Plugin.json commands array and the README table are both updated in the same PR

Notes (non-blocking)

1. Root CLAUDE.md version table is stale. The project-level CLAUDE.md contains a "Current plugins" table with explicit versions, and a note: "Keep this table updated when versions change." The development-pr-workflow row still shows 2.1.0 and isn't updated in this PR. Not in the diff, but worth fixing here so the doc and the published version don't drift.

2. Notion Plugin Marketplace doc. Per .claude/rules/plugin-docs.md, new commands should be reflected in the Notion marketplace doc (skill/agent/command counts and the per-plugin section). That's an external action, so just flagging — easy to forget.

**3. Minor readability — Datadog \`in the data-source section.** The line reads: *"Metrics (Datadog, trace., CloudWatch) are ~100%"*. The bare `` `` reads like a placeholder/typo even though I think the intent is "any Datadog metric namespace." Consider something like ``.` `` or just "standard Datadog metrics" to avoid the dropped-token look. Cosmetic, not blocking.

4. README table width. The new row's description (Validate a data-driven change vs live history before PR) is longer than the visual column width of the existing rows. Markdown renders it fine, but if you're particular about source-readable alignment, the existing | --------- | separator widths are no longer representative. Trivially fixable, or ignorable.

Verdict

Approve. The change is small, additive, follows the plugin conventions, and has no code-execution surface. Worth quickly addressing the CLAUDE.md version table before merge so the docs stay in sync.

_{💡 Want a fresh review? Add a comment containing @request-claude-review to trigger a new review at any time.}

github-actions

📋 Review verdict: APPROVE

👆 The main review comment above is the source of truth for this PR review. It is automatically updated on each review cycle, so always refer to it for the most current feedback.

This formal review submission is for the verdict only. 2 inline comment(s) are attached below.

github-actions · 2026-06-05T22:20:58Z

+
+2. **Find the authoritative data source** and respect sampling:
+   - **Metrics** (Datadog `*`, `trace.*`, CloudWatch) are ~100% — use these to count rates/volumes/percentiles.
+   - **Indexed spans / logs** keep ~all errors but often only ~1% of successes — do NOT compute success-rate or totals from span/log *counts* (off by ~100×). Use them for inspection, not denominators.


Minor: Datadog \`reads like a dropped token. Intent appears to be "any Datadog metric" (contrasted with thetrace.subset), but the bare backtick-star is easy to misread as a typo. ConsiderDatadog metrics (.)` or just "standard Datadog metrics" for clarity.

Suggested change

- **Indexed spans / logs** keep ~all errors but often only ~1% of successes — do NOT compute success-rate or totals from span/log *counts* (off by ~100×). Use them for inspection, not denominators.

- **Metrics** (standard Datadog metrics, `trace.*`, CloudWatch) are ~100% — use these to count rates/volumes/percentiles.

github-actions · 2026-06-05T22:20:59Z

 {
  "name": "development-pr-workflow",
-  "version": "2.1.0",
+  "version": "2.2.0",


Version bump looks right (minor for new command per CLAUDE.md). Reminder: the root CLAUDE.md has a "Current plugins" version table that still lists development-pr-workflow at 2.1.0 — worth updating it in this PR so the table doesn't drift.

@claude address this suggestion

Claude finished @wkoutre's task in 55s —— View job

Addressed the suggestion ✅

Understand the request

Update root CLAUDE.md version table (development-pr-workflow: 2.1.0 → 2.2.0)

Commit and push

Updated the Current plugins version table in the root CLAUDE.md to reflect the 2.2.0 version bump introduced in this PR.

-| development-pr-workflow | 2.1.0 | +| development-pr-workflow | 2.2.0 |

Keep the Current plugins table in sync with the plugin.json bump (2.1.0 → 2.2.0) introduced in this PR for the new /backtest-change command. Co-authored-by: Nick Koutrelakos <wkoutre@users.noreply.github.com>

github-actions · 2026-06-09T17:59:05Z

🤖 Claude PR Metadata Generation

Status: ❌ Error
Job: View workflow run

⚠️ Generation Failed

The automated PR title and description generation encountered an error.

Next Steps:

Check the workflow logs for details
Verify the GitHub Actions configuration
Ensure the Anthropic API key is valid

You can manually set the PR title and description, or re-trigger by pushing a new commit.

github-actions · 2026-06-09T17:59:52Z

🤖 Claude Code Review

Status: ⚠️ Partial (script error)
Job: View workflow run

The review was completed but there was an error posting inline comments.

Raw Review Output

No review output available

Please check the workflow logs for details.

github-actions · 2026-06-09T17:59:53Z

🤖 Claude Code Review

Status: ❌ Error
Job: View workflow run

⚠️ Review Failed

The automated code review encountered an error and could not complete.

Next Steps:

Check the workflow logs for details
Verify the GitHub Actions configuration
Ensure the Anthropic API key is valid
Check for any permission issues

If the problem persists, please contact your repository administrator.

You can manually re-trigger the review by pushing a new commit or re-running the workflow.

vercel Bot deployed to Preview June 5, 2026 21:30 View deployment

chore(development-pr-workflow): bump plugin version 2.1.0 -> 2.2.0 fo…

15e5850

…r new command

vercel Bot deployed to Preview June 5, 2026 22:17 View deployment

dylanschmittle-uniswap marked this pull request as ready for review June 5, 2026 22:18

dylanschmittle-uniswap requested a review from a team as a code owner June 5, 2026 22:18

github-actions Bot approved these changes Jun 5, 2026

View reviewed changes

docs: update development-pr-workflow version to 2.2.0 in root CLAUDE.md

5fa3d8d

Keep the Current plugins table in sync with the plugin.json bump (2.1.0 → 2.2.0) introduced in this PR for the new /backtest-change command. Co-authored-by: Nick Koutrelakos <wkoutre@users.noreply.github.com>

vercel Bot deployed to Preview June 9, 2026 17:59 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(development-pr-workflow): add /backtest-change command#532

feat(development-pr-workflow): add /backtest-change command#532
dylanschmittle-uniswap wants to merge 3 commits into
nextfrom
add-backtest-change-command

dylanschmittle-uniswap commented Jun 5, 2026 •

edited by github-actions Bot

Loading

Uh oh!

vercel Bot commented Jun 5, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 5, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 5, 2026 •

edited

Loading

Uh oh!

github-actions Bot left a comment

Uh oh!

github-actions Bot Jun 5, 2026

Uh oh!

github-actions Bot Jun 5, 2026

Uh oh!

wkoutre Jun 9, 2026

Uh oh!

claude Bot Jun 9, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 9, 2026

Uh oh!

github-actions Bot commented Jun 9, 2026

Uh oh!

github-actions Bot commented Jun 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	- Indexed spans / logs keep ~all errors but often only ~1% of successes — do NOT compute success-rate or totals from span/log counts (off by ~100×). Use them for inspection, not denominators.
	- Metrics (standard Datadog metrics, `trace.*`, CloudWatch) are ~100% — use these to count rates/volumes/percentiles.

Conversation

dylanschmittle-uniswap commented Jun 5, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Test plan

What

Why

Changes

Test plan

Uh oh!

vercel Bot commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📚 Documentation Check ✅

PR #532 Documentation Check — PASS

Missing Updates

Suggestions (1)

Uh oh!

github-actions Bot commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤖 Claude Code Review

Summary

What works well

Notes (non-blocking)

Verdict

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

wkoutre Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Addressed the suggestion ✅

Uh oh!

github-actions Bot commented Jun 9, 2026

🤖 Claude PR Metadata Generation

⚠️ Generation Failed

Uh oh!

github-actions Bot commented Jun 9, 2026

🤖 Claude Code Review

Uh oh!

github-actions Bot commented Jun 9, 2026

🤖 Claude Code Review

⚠️ Review Failed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dylanschmittle-uniswap commented Jun 5, 2026 •

edited by github-actions Bot

Loading

vercel Bot commented Jun 5, 2026 •

edited

Loading

github-actions Bot commented Jun 5, 2026 •

edited

Loading

github-actions Bot commented Jun 5, 2026 •

edited

Loading

claude Bot Jun 9, 2026 •

edited

Loading