feat(cli): add stats subcommand — per-agent React Doctor leaderboard#932
Open
aidenybai wants to merge 17 commits into
Open
feat(cli): add stats subcommand — per-agent React Doctor leaderboard#932aidenybai wants to merge 17 commits into
stats subcommand — per-agent React Doctor leaderboard#932aidenybai wants to merge 17 commits into
Conversation
…d from agent history Adds `react-doctor stats`, which reads local AI agent history (Claude Code + Codex transcripts, the Cursor composer database), reconstructs the React code each model actually wrote, lints it with the existing engine, and ranks models and providers by a confidence-weighted React Doctor score. - Reconstructs faithful post-edit file content per provider (Claude snapshots, Cursor `afterContentId` blobs, Codex `apply_patch`), filtered to real React. - Confidence-weighted ranking: each group's raw score regresses toward the global mean by its evidence (files dominant, lightly discounted by sessions), so a tiny clean sample can't top the board. - Plain-language terminal leaderboard with color-coded tools (adds an `orange` to the shared highlighter for Claude); `--json` for the machine-readable report.
commit: |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Discovery loaded each candidate session from the Cursor SQLite DB synchronously, blocking the event loop so the ora spinner appeared frozen for a few seconds. Yield to the event loop periodically and report live "(N found)" progress during the history walk.
Cap the terminal table to the top 5 with a "+ N more" pointer to --json; the full ranking still ships in the JSON report and the best/worst callout.
Consolidate the asString/asRecord/asArray/parseJson coercers (copied across the Claude/Codex/Cursor adapters) into a shared coerce.ts, extract the "most common model" tally into most-common-key.ts, and reuse statMtimeMs in findJsonlFiles. Behavior unchanged.
The static `node:sqlite` import crashed the whole adapter test file on Node 20 (where the module doesn't exist), failing the 20.19 CI matrix job. Load it via a guarded require and skip the Cursor suite when unavailable, mirroring cursor-db.ts's runtime degradation.
- closeCursorDb now closes the underlying node:sqlite database instead of only dropping the cached reference, so the fixture file is unlocked and Windows can unlink the temp dir (was EBUSY in the adapter test teardown). - The reconstruct test compared emitted absolute paths against hardcoded POSIX strings; on Windows resolveAgainstCwd normalizes to backslashes, so expectations now mirror that normalization. Production code unchanged.
- A failed apply_patch update hunk left the prior in-session buffer in place and still emitted the file as faithfully reconstructed; drop it to unreconstructable so stale content is never linted as the model's output. - Sessions touching only non-lintable files (e.g. markdown) had zero reconstructed files and zero failures but were counted as "unreconstructable"; require an actual reconstruction failure for that bucket so the skip note stays accurate.
Replace the readFileSync + split("\n") transcript reader with a streaming
node:readline parser so memory stays flat on large Claude/Codex transcripts.
Makes session loading async (SessionCandidate.load + the parse adapters);
the Cursor composer load wraps its sync DB walk to match.
…--since) - Drop scans that error/skip/lint-fail instead of counting them as clean code, which was inflating the leaderboard. - Emit structured JSON on failure in --json mode (reuse enableJsonMode), which also silences the incidental score-API stderr warning. - Exclude unknown-timestamp candidates under --since so the filter is consistent. - Consolidate the path-inside predicate, move render magic numbers to constants, type-guard the provider flag, throw on invalid --limit, rename op -> operation.
… content A replace/Edit whose oldString isn't in the in-session buffer now marks the file unreconstructable (like a failed apply_patch hunk) rather than keeping the stale snapshot and scoring it as the model's final output.
Confidence weighting now counts only sessions that contributed scanned files, so non-React/failed/skipped sessions no longer raise session reliability or effective file weight. The reported per-group session count still reflects every analyzed session.
react-doctor has a runtime `deslop-js: workspace:*` dependency, but the Continuous Releases workflow didn't publish deslop-js, so pkg.pr.new couldn't rewrite the ref and `npx https://pkg.pr.new/react-doctor@<pr>` failed with EUNSUPPORTEDPROTOCOL ("workspace:"). Add deslop-js to the publish set.
…slop the engine Broaden which local agent history `stats` reads, and refine the engine. Coverage: - Cursor GUI: scan both the stable and Nightly builds' composer databases (was stable-only — a Nightly-only user got zero GUI sessions), and read a live, editor-locked database via SQLite's immutable mode instead of letting the lock crash the run. - Cursor CLI agent: new source for the per-session content-addressed stores under ~/.cursor and ~/.cursor-nightly — decode the hex meta row, parse the binary message manifest, and map Write/ApplyPatch/StrReplace/Delete tool calls to edits, capturing Read results as reconstruction bases. - Codex (~/.codex) was already covered; verified. Engine deslop (behavior-preserving): - consolidate the zip-slip path-inside guard into one audited core util (@react-doctor/core isPathInside), and share the node:sqlite read-only open and the empty-string-preserving string narrow (coerce asNullableString) instead of hand-rolling copies - drop dead code (write-only session timestamps, an unused export, an unreachable branch), replace forbidden nested ternaries with if/else and a lookup table, collapse a redundant variable and a pass-through wrapper - guard every SQLite close so a locked/unreadable store degrades to "skip" rather than sinking the whole stats run Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…board rows) Wrap `react-doctor stats` in a `cli.stats` root span with a discover/scan/ aggregate latency waterfall, and emit one queryable `stats.leaderboard_row` span per ranked model carrying its model, harness, confidence-weighted score, and files scored — the four leaderboard columns. Same gating as the scan path (no-op under --no-score, in tests, and for @react-doctor/api). - Extract the shared `modelLabel` helper (render + tracing) into one util. - Pure, exported `buildStatsRowAttributes` for testability, mirroring `buildRunEventAttributes`. - Fix `detectCommand`: `stats` runs were mis-tagged `command=inspect`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nity board
Send the same per-model rows the stats command puts on Sentry
({model, harness, score, files}) to our own /api/stats so we store them and get
back the community leaderboard, shown beneath the local board.
- stats/leaderboard-row.ts: one shared projection feeds BOTH the Sentry span
attributes and the /api/stats payload, so they can't drift and both stay
code-free (no source, paths, or identity ever leaves the machine).
- stats/report-stats-run.ts: best-effort gzip POST (null on any failure),
honoring an optional REACT_DOCTOR_STATS_API_URL override for local e2e.
- stats command: honors --no-score/--no-telemetry — skips the score API
(scores n/a, ranked by diagnostics-per-file) AND the /api/stats report, so a
--no-telemetry run is fully local and less rich.
- render-stats: appends a "Community leaderboard" table (with run counts) when
one is returned.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
| ["version", VERSION_FLAG_SPEC], | ||
| ["rules", RULES_FLAG_SPEC], | ||
| ["why", WHY_FLAG_SPEC], | ||
| ["stats", STATS_FLAG_SPEC], |
There was a problem hiding this comment.
Stats strips trailing no-score
Medium Severity
For react-doctor stats --no-score (or --no-telemetry), the pre-Commander flag stripper drops those globals when they appear after stats, so statsAction still treats telemetry as on and calls the score and /api/stats endpoints even though Sentry already opted out via raw process.argv.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit db52fc6. Configure here.
| sessionsRanked: results.filter((result) => result.filesScanned > 0).length, | ||
| sessionsNonReact: results.filter( | ||
| (result) => result.filesScanned === 0 && result.reconstructedFiles > 0, | ||
| ).length, |
There was a problem hiding this comment.
Lint failures labeled non-React
Low Severity
Sessions where React files were reconstructed but linting failed or was skipped are counted in sessionsNonReact, so the footer can say they “changed only non-React files” even when the pipeline dropped them for scan errors.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit db52fc6. Configure here.
| await new Promise<void>((resolve) => setImmediate(resolve)); | ||
| } | ||
| if (sessions.length >= scope.limit) break; | ||
| } |
There was a problem hiding this comment.
Repo scope scans all history
Medium Severity
With repo scope, if no candidate session passes the repo filter, discovery keeps loading every sorted candidate until the list ends, ignoring --limit, which can mean thousands of synchronous transcript/DB loads on a large machine.
Reviewed by Cursor Bugbot for commit db52fc6. Configure here.
The leaderboard payload is a handful of tiny rows ({model, harness, score,
files}), so gzip cost more than it saved — it was cargo-culted from the
diagnostics-heavy score API. Send plain JSON and drop the Content-Encoding
header.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 4 total unresolved issues (including 3 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 83a9210. Configure here.
| // phase failed yields zero diagnostics for reasons unrelated to code | ||
| // quality. Counting its files as clean would reward un-lintable code and | ||
| // inflate the leaderboard, so it joins the empty bucket instead. | ||
| if (!result.ok || result.skipped || result.didLintFail) return empty; |
There was a problem hiding this comment.
Failed lint labeled non-React
Medium Severity
When a session’s React files reconstruct successfully but runEditorScan errors, is skipped, or lint fails, the scan returns the same empty result shape as a non-React session (filesScanned === 0, reconstructedFiles > 0). The report then increments sessionsNonReact and shows “changed only non-React files” even though React code was replayed and lint never ran.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 83a9210. Configure here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
react-doctor stats: a per-model/per-tool code-quality leaderboard built from your local AI agent history. It answers one question — which agent writes the cleanest React code in my repo?It reads local agent history (Claude Code + Codex transcripts, and every place Cursor stores its chats), reconstructs the file content each model actually wrote, lints it with the existing engine, and ranks models and providers by a confidence-weighted React Doctor score.
apply_patchreplay. Only actual React files (JSX/TSX,use client/use server, or a React-ecosystem import) are scored, so backend/util/config files don't dilute the result.state.vscdb) for both the stable and Nightly builds, plus the cursor-agent CLI per-session stores under~/.cursor/~/.cursor-nightly(binary manifest +Write/ApplyPatch/StrReplace/Deletetool calls, withReadresults captured as reconstruction bases). A database a running editor holds locked is read via SQLite'simmutablemode rather than skipped.--json.orangeformatter to the sharedhighlighter(honors--no-color).--global(all repos),--since,--limit,--provider,--json. Default scope is the current repo.Coverage is honest about limits: Codex shell edits aren't reconstructable (surfaced as skipped), reading any Cursor database needs
node:sqlite(Node 22.13+), and the score requires network access.Test plan
pnpm --filter react-doctor test(32 new stats tests: adapters, reconstruct, apply-patch, aggregate/weighting, is-react-source, render)pnpm typecheck/pnpm lint/pnpm format:checkreact-doctor statsin a repo with local Cursor/Claude/Codex history renders a sane leaderboardreact-doctor stats --jsonemits{ schemaVersion, models, providers, best, worst, … }with bothscoreandweightedScorereact-doctor stats --globalranks across repos;--provider cursornarrows the sourceUpdate — broader Cursor/Codex coverage + engine deslop
Follow-up commit on this branch:
~/.cursor,~/.cursor-nightly), not just the stable GUIstate.vscdb. A live, editor-locked database is now read via SQLiteimmutablemode instead of crashing the run, and every SQLite close is guarded so an unreadable store degrades to "skip".@react-doctor/coreutil, shared thenode:sqliteread-only open and the empty-string-preserving string narrow, dropped dead code, and replaced forbidden nested ternaries. Verified by typecheck/lint/format, the full react-doctor suite, and a real-data smoke (451 GUI + 12 CLI sessions acrossclaude-opus-4-8/composer-2.5/gpt-5.5).Note
Medium Risk
Large new CLI surface that reads homedir agent/SQLite data and optionally sends anonymized scores to react.doctor and Sentry; mitigations include path guards, React-only linting, and telemetry gating, but coverage of transcript formats remains inherently fragile.
Overview
Adds
react-doctor stats, which mines local Claude Code, Codex, and Cursor history (GUIstate.vscdbfor stable/Nightly plus CLIstore.dbunder~/.cursor/~/.cursor-nightly), replays edits into file snapshots, lints only React sources, and ranks models/tools by a confidence-weighted score (raw score regressed toward the global mean so tiny samples can’t win). Repo scope is default;--global,--since,--limit,--provider, and--jsoncontrol scope and output.The pipeline drops failed/skipped scans and non-faithful reconstructions (e.g. Codex shell edits), uses shared
isPathInsidezip-slip checks for temp trees, and optionally POSTs code-free leaderboard rows to/api/statsand Sentrycli.statsspans when telemetry/score is on (--no-score/--no-telemetrykeeps runs local). Terminal UI adds provider-colored tables (includinghighlighter.orangefor Claude).Also wires
STATS_API_URLin core, registersstatsin CLI help/flag stripping/run context, and includesdeslop-jsin the pkg-pr-new publish workflow.Reviewed by Cursor Bugbot for commit 83a9210. Bugbot is set up for automated code reviews on this repo. Configure here.