Skip to content

Add shared rule tests, calibration and agent guidance - #1

Merged
mrubens merged 3 commits into
mainfrom
codex/calibration-harness
Sep 29, 2026
Merged

mrubens merged 3 commits into
mainfrom
codex/calibration-harness

Conversation

@mrubens

@mrubens mrubens commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add shared rule-example testing and calibration to Judgement. judgement test evaluates configured thresholds and fails on wrong or incomplete results. judgement calibrate --rule <id> and --all compare candidates using repeated full checks in disposable Git repositories. Library APIs support existing application inference adapters.

Use .judgement/rules.json as the only policy location and .judgement/examples/<id>.json for fixtures. Base rules still govern policy edits. Suites validate and snapshot every selected fixture before inference, bound concurrency, disable verdict caching, and include input hashes in JSON reports. Generated reports are intended for local output or CI artifacts.

Add an agent guide shipped in the npm package, linked from the README, covering rule design, labels, commands, uncertainty, threshold selection, held-out validation, and credential handling. Repository AGENTS.md documents implementation ownership and validation requirements. Applications keep rules, examples, and thin inference adapters rather than copying the runner.

Includes the 0.2.0 version bump for the configuration change. Not published.

Validation

  • npm run check: 47 tests passed, including suite thresholds and exit semantics, preflight validation, input snapshots, cancellation, Git isolation, and CLI dry runs.
  • npm pack --dry-run --json: examples and agent guide included.
  • Earlier live custom-evaluator smoke test completed without operational failures and correctly declined to recommend an unsupported threshold.

@mrubens mrubens changed the title Add calibration harness and .judgement configuration layout Add shared rule tests, calibration and agent guidance Sep 29, 2026
@mrubens
mrubens merged commit e23c651 into main Sep 29, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant