feat: add evaluation system — datasets, evaluators, experiments, comparison#25
Merged
Conversation
…arison Enterprise-grade evaluation system for measuring agent output quality, not just structural regressions. Closes the #1 feature gap vs LangSmith. New crate: rewind-eval - DatasetManager: versioned test-case collections with JSONL import/export - 5 built-in deterministic scorers: exact_match, contains, regex, json_schema, tool_use_match - ExperimentRunner: subprocess protocol (stdin JSON → stdout JSON), timeout, error handling - ExperimentComparison: side-by-side delta computation with regression/improvement detection CLI (rewind eval): - dataset create/import/export/show/list/delete/add-from-session - evaluator create/list/delete - run: execute target command against dataset, score, aggregate - compare: diff two experiments with color-coded output - --json output for CI pipelines, --fail-below threshold with exit code 1 Storage: 6 new SQLite tables (datasets, dataset_examples, evaluators, experiments, experiment_results, experiment_scores) with versioning and content-addressed blob storage. Web API: 7 read-only GET routes under /api/eval/* MCP: 5 read-only tools (list/show datasets, list/show/compare experiments) Python SDK: Dataset, evaluate(), @evaluator decorator, compare(), 4 built-in scorers Web UI: EvalDashboard with tabs (Datasets/Experiments/Compare), DatasetBrowser, ExperimentList, ExperimentDetail with expandable scores, ExperimentComparison with color-coded deltas, ScoreBadge component Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove extraneous f-string prefix (F541) - Rename ambiguous variable `l` to `left_result` (E741) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Replace redundant closures with function references (clippy::redundant_closure) - Collapse nested if-let statements (clippy::collapsible_if) - Fix format! in println! args (clippy::format_in_format_args) - Allow too_many_arguments on update_experiment_aggregates Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add eval to tagline and capability table - New section: Evaluation — datasets, evaluators, experiments, comparison - Document CLI workflow: create dataset → import → evaluate → compare - Document CI integration with --fail-below and --json - Document Python SDK evaluate() with custom evaluators - Add eval commands to Commands table (8 new entries) - Add rewind-eval to Architecture crate list - Add 5 eval MCP tools to MCP tools table - Update roadmap: v0.4 shipped (eval system) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Enterprise-grade evaluation system for measuring agent output quality. Closes the #1 feature gap vs LangSmith.
rewind-evalcrate — DatasetManager (versioned test cases, JSONL import/export, session extraction), EvaluatorRegistry (5 built-in deterministic scorers), ExperimentRunner (subprocess protocol with timeout/error handling), ExperimentComparison (side-by-side deltas)rewind evalsubcommand tree: dataset CRUD, evaluator CRUD,runwith--json/--fail-belowfor CI,compare,experiments,show/api/eval/*Dataset,evaluate(),@evaluatordecorator,compare(), 4 built-in scorersTest plan
cargo test -p rewind-eval)cargo build --release)npx tsc --noEmit)--fail-belowexits with code 1 when avg_score < threshold (CI gate works)--jsonoutputs machine-parseable JSON withschema_version: 1Phase 2 (follow-up)
llm_judgeevaluator (experimental, withbase_urlfor internal LLM gateways)🤖 Generated with Claude Code