From 631be3a5cfdad1aeaadb520fa2e095b3bc92912e Mon Sep 17 00:00:00 2001 From: Matt Rubens <2600+mrubens@users.noreply.github.com> Date: Wed, 30 Sep 2026 11:11:33 -0400 Subject: [PATCH] Add installable Judgement skill for recurring corrections --- README.md | 29 +++- docs/agents.md | 177 +-------------------- package.json | 3 +- skills/judgement/SKILL.md | 89 +++++++++++ skills/judgement/agents/openai.yaml | 3 + skills/judgement/references/agent-guide.md | 173 ++++++++++++++++++++ 6 files changed, 296 insertions(+), 178 deletions(-) create mode 100644 skills/judgement/SKILL.md create mode 100644 skills/judgement/agents/openai.yaml create mode 100644 skills/judgement/references/agent-guide.md diff --git a/README.md b/README.md index 214ff1e..b0b15df 100644 --- a/README.md +++ b/README.md @@ -23,11 +23,25 @@ Save this as `.judgement/rules.json` in your repository. Run Judgement before a or in CI. It uses [Jev](https://typesafe.ai) to make focused judgments and reads related source files from the exact Git snapshot being checked. -## Agent instructions +## Agent skill -Coding agents: read [the agent guide](docs/agents.md) before adding or tuning rules. -It is included in the npm package and covers fixtures, commands, uncertainty, and -validation. Projects should keep their rules and examples, not copy the runner. +Install the [Judgement skill](skills/judgement/SKILL.md) with the +[skills CLI](https://skills.sh/docs/cli): + +```sh +npx skills add RooCodeInc/judgement --skill judgement +``` + +The installer lets you choose the agents to use it with. The skill can be selected +for requests such as “never do that again,” “add a rule to prevent this mistake,” +or “calibrate this Judgement rule.” It guides the agent to scope the requirement, +reuse existing rules, add labeled examples, and test the chosen cutoff. Mechanical +checks and conversation-only preferences are routed to the appropriate tooling. + +You can also copy the complete `skills/judgement/` directory into your agent's +skill directory. Keep its `references/` directory alongside `SKILL.md`. The skill +is included in the npm package; installing it configures agent guidance. Projects +configure the Judgement CLI, rules, and inference separately. ## Install @@ -35,7 +49,7 @@ Node 22.20+ or 24+ and Git are required. Glob matching uses the small `picomatch Install the published package from npm. ```sh -npm install --save-dev --save-exact @roo-code/judgement@0.3.0 +npm install --save-dev --save-exact @roo-code/judgement@0.4.0 export TYPESAFE_API_KEY=... ./node_modules/.bin/judgement check --staged ``` @@ -57,8 +71,9 @@ or evidence is insufficient, it reports **incomplete** and allows the commit. It never presents an unfinished review as a pass. Process startup and operating system scheduling add overhead; use the benchmark to measure your environment. -A required CI check must finish that work before merging. Judgement does not -schedule a background check or configure branch protection for you. +Use strict checks in CI to finish that work. Projects choose whether the status +is required before merge. Judgement does not schedule a background check or +configure branch protection for you. For the [pre-commit framework](https://pre-commit.com), use this repository's `.pre-commit-hooks.yaml` with `rev: v0.1.5` and hook `id: judgement`. diff --git a/docs/agents.md b/docs/agents.md index a3ce46d..9c43760 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -1,173 +1,10 @@ -# Using Judgement as a coding agent +# Judgement for coding agents -Judgement checks repository diffs against natural-language rules. The project owns -its rules and labeled examples. Judgement owns running checks, testing examples, -comparing thresholds, and reporting results. +Use the [Judgement skill](../skills/judgement/SKILL.md) to turn recurring user +corrections into scoped repository rules and labeled examples. -## Find the project setup +The [agent guide](../skills/judgement/references/agent-guide.md) covers schemas, +commands, calibration, and interpreting results. These files are included in the +npm package and can be read directly without installing the skill. -1. Read the repository's agent instructions and `.judgement/rules.json`. -2. Inspect its package scripts and existing inference adapter. Use the project's - configured command and backend when available; a different model may score the - same examples differently. Do not add credentials to files or output. -3. Use the installed, pinned package. Do not install packages inside a commit hook. - The commands below require the release containing the calibration API; check - `judgement --help` rather than assuming an older installation supports them. - -## Write a rule - -Save rules in `.judgement/rules.json`: - -```json -{ - "criteria": [ - { - "id": "billing-audit", - "rule": "Every successful billing mutation must record an audit event for that mutation. Read-only queries do not require audit events.", - "files": ["src/billing/**"], - "context": ["src/audit/**"], - "threshold": 0.85 - } - ] -} -``` - -- Give each rule a stable ID. Use letters, digits and hyphens for easy filenames. -- Describe one observable requirement, its scope, and meaningful exceptions. - Split unrelated requirements when they need different evidence or thresholds. -- Scope `files` to relevant paths. Globs are relative to the repository root and - match hidden paths too. `context` selects unchanged supporting files from the - same Git snapshot. Keep it focused; it is not a request to inspect the whole repo. -- A threshold is a violation-probability cutoff, not proof of accuracy. The default 0.85 is a - starting point. Do not weaken the rule or lower its threshold just to get green. - -## Save labeled examples - -For each rule, write `.judgement/examples/.json`: - -```json -{ - "ruleId": "billing-audit", - "examples": [ - { - "name": "Mutation without audit event", - "path": "src/billing/update.ts", - "before": "export async function update(db, audit, id) { await db.update(id); await audit.record(id); }\n", - "after": "export async function update(db, audit, id) { await db.update(id); }\n", - "expected": "violation" - }, - { - "name": "Read-only query", - "path": "src/billing/read.ts", - "before": "export async function read(db) { return []; }\n", - "after": "export async function read(db) { return db.findMany(); }\n", - "expected": "pass" - } - ] -} -``` - -Label by the intended policy **before** looking at model scores. Include clear -violations, allowed exceptions, fixes, unrelated edits near old violations, and -realistic context. Keep failing examples; investigate them instead of changing the -expected label to match the model. - -`before` and `after` must differ. Use `null` for the absent side of an addition or -deletion. Optional `context` maps unchanged relative paths to file contents; the -rule's context globs still control which files are included. Fixtures are source -text, not programs to execute. Never put real secrets or private data in examples. -If a rule allows synthetic placeholders, those are valid cases, not positive -secret-detection tests. State that coverage limitation explicitly. - -## Test before tuning - -Run with the project's pinned CLI or equivalent adapter command: - -```sh -judgement test --dry-run -judgement test --rule billing-audit --repeats 3 -judgement test --format json > /tmp/judgement-tests.json -judgement calibrate --rule billing-audit --thresholds 0.8,0.85,0.9,0.95 -judgement calibrate --all --format json > /tmp/judgement-calibration.json -``` - -`test` uses each rule's configured threshold and requires every run to match its -label. A valid example must fully pass; an incomplete result fails the test. -`calibrate` compares candidates. A recommendation can still leave valid examples -incomplete, so read the full report. Live runs make paid model requests; dry runs -make none. A recommendation requires both valid and violating examples; a -single-label set can test behavior but cannot select a threshold. Testing and calibration are explicit operations, never part of the -normal commit hook. - -The harness uses disposable Git repositories, repeats without a verdict cache, -and preserves the real working tree and index. It reads working-tree rules so -edited candidate thresholds can be tested. Normal diff checks use the base policy, -so a rule edit cannot weaken the check on its own commit. - -Inspect false blocks, caught violations, incorrect passes, incomplete results, -operational failures, scores, and timing separately. When a case fails: - -1. Check its label, path, rule wording, and selected supporting context. -2. Distinguish an uncertain correct answer from a wrong answer. Lowering a cutoff can catch more violations and also introduce false blocks. - If valid and violating examples overlap, clarify the rule or its evidence. -3. Repeat the run. Never present one score as a guarantee. -4. Validate a candidate against separately labeled held-out examples before adopting - it. Use `--examples` for one rule, or `test --examples-dir` / `calibrate --all ---examples-dir` for a separate suite. Commit deliberate rule and fixture changes; - keep generated reports local or as CI artifacts. - -## Interpret exit codes and report honestly - -- `test`: 0 means all labels matched (or dry run); 1 means wrong, incomplete, or - operationally failed checks; 2 means invalid input/setup; 130 means interrupted. -- `calibrate`: 0 means a candidate was recommended (or dry run); 3 means no candidate - was recommended for at least one selected rule; 2 means invalid input/setup. -- `check --hook` allows incomplete results to keep commits moving. Strict `check` - rejects incomplete results. Neither behavior makes an incomplete result a pass. - -Report what was tested, repeated counts, backend differences, incomplete cases, -and untested categories. Passing small fixtures does not prove that large commits, -missing context, or every possible violation will be handled correctly. - -Applications with existing credentials should supply their evaluator to `testRules` -or `calibrateRules`. Keep that adapter thin: do not copy the runner, fixture loading, -isolated repository setup, threshold logic, or reporting into each application. - -## Inspect uncertainty in a model tester - -Use `prepareExamples` to obtain exact checker request packets from the project's -fixtures. Send each packet's `state` and `questions` to the configured backend; -keep `expected`, names, and thresholds as tester metadata. Inspect both initial -and expanded packets, with raw violation probabilities visible. -Preparation makes no inference calls. Generated inputs can be bundled as presets; -regenerate them when their policy, fixture, or package version changes. - -The model answers whether a change violates the rule. Its Noul value is the -probability of a violation. Scores at or above the cutoff flag a violation; -every lower score produces no finding, including an ambiguous score of 0.5. -Missing required evidence, partial screens, unsupported files, and failed requests -remain incomplete. The default binary evaluator does not request context expansion -based on its probability. Use explicit context and compare expanded packets when -investigating a missed violation. Recalibrate when changing question types; Choice -confidence and Noul probability use different meanings and scales. - -## Collect and compare feedback - -Use `judgement capture --rule --path --name --expected pass|violation` -to preview an index-based example. Review and redact it before saving a new file -with `--output`; capture never overwrites or stages files. Include unchanged -supporting evidence with `--context`. Reduce multi-file changes rather than -silently discarding context that changed. Captured content must satisfy the -repository's privacy requirements before it is committed. - -Save baseline and candidate `test --format json` reports locally, using identical -fixtures. `judgement compare --before --after ` exposes per-example -regressions and exits 1 when any are found. Do not change expected labels to hide -regressions. Compare both valid exceptions and violations; repeated observations -of the same example are not independent coverage. - -Finding snippets are changed-line previews, not model-localized accusations. -Read the supplied evidence before proposing a fix. Progress and cache-hit counts -are diagnostics; only a complete report establishes check completion. In CI, use -strict checks without `--hook` or `--advisory`, and require the resulting status -before merge. Missing inference credentials must not produce a successful check. +See the [README](../README.md#agent-skill) for skill installation. diff --git a/package.json b/package.json index 6671c21..147a348 100644 --- a/package.json +++ b/package.json @@ -22,7 +22,8 @@ "LICENSE", "examples", ".pre-commit-hooks.yaml", - "docs" + "docs", + "skills" ], "scripts": { "test": "node --test test/*.test.js", diff --git a/skills/judgement/SKILL.md b/skills/judgement/SKILL.md new file mode 100644 index 0000000..d6f8176 --- /dev/null +++ b/skills/judgement/SKILL.md @@ -0,0 +1,89 @@ +--- +name: judgement +description: Turn recurring repository mistakes and user corrections into calibrated Judgement rules. Use when a user asks an agent to "never do that again," prevent a mistake from recurring, enforce a contextual code or prose requirement, or add, test, or tune Judgement rules. Applies to requirements visible in repository changes; conversation-only preferences belong in agent instructions or memory. +--- + +# Judgement + +Make a recurring requirement concrete: a focused rule in `.judgement/rules.json`, +labeled examples, and evidence that the configured checker distinguishes the +mistake from allowed changes. Judgement checks Git diffs; it cannot guarantee that +an agent will never repeat a behavior. + +## Turn the correction into a requirement + +Use the user's correction and the surrounding task to identify the unwanted +behavior, where it matters, and the allowed exceptions. A clear request to prevent +recurrence is enough to make a scoped repository rule change. Ask a focused +question only if the intended requirement or scope is ambiguous. + +Choose the appropriate mechanism: + +- Use Judgement for contextual distinctions that need judgment: misleading setup + prose, exposing internal errors to users, or carrying credentials across a trust + boundary. +- Use an existing formatter, linter, type check, or deterministic test when it can + express the requirement reliably. Do not add a model check for a mechanical ban. +- Use the project's agent instructions or the user's preferred memory mechanism + for conversational or tool-use behavior that is not visible in committed files. + A rule about Git diffs cannot enforce how an agent talks in chat. + +Keep the scope the user intended. A correction to one situation does not establish +an unrelated repository-wide policy. Resolve equivalent existing rules before +adding another; preserve their stable IDs and useful examples. + +## Find the setup and write the rule + +Read repository instructions, `.judgement/rules.json`, existing fixtures, and the +package scripts. Use the project's pinned Judgement command and inference adapter. +Check its `--help` for available commands. If Judgement is not configured, explain +what setup is needed and use the repository's dependency and credential conventions +when setup is within the request. Installing this skill does not install the CLI +or configure inference. + +Read [the agent guide](references/agent-guide.md) for the rule and example schemas, +commands, calibration, and result semantics before editing rules or fixtures. + +Describe one observable requirement and its meaningful exceptions. Select relevant +`files` globs and only the supporting `context` needed to judge it. Favor localized +rules so evidence stays small and checks stay fast. Do not expand context to the +whole repository to compensate for an unclear rule. + +For example, “stop capitalizing session everywhere” may mean ordinary nouns in +user-facing prose, while sentence starts, headings, exact UI labels, and code +identifiers remain valid. Preserve the exceptions the user actually intends. + +## Demonstrate the behavior + +Add labeled examples under `.judgement/examples/.json`. Include the +mistake, a corrected version, allowed exceptions, and a nearby unrelated edit that +should pass. Label examples from the intended requirement before inspecting scores. +Use synthetic or redacted content; do not copy private data into fixtures. + +For staged or committed feedback, `judgement capture` can preview an example. +Review the preview before saving; do not stage unrelated user changes to capture +them. Preserve the index and working tree while testing. + +Use the configured command to dry-run the fixtures, then run a bounded repeated +test of the affected rule when its inference backend is available. Use calibration +to compare cutoffs and separately labeled examples to check the chosen cutoff. +A cutoff is a violation-probability threshold, not a measured accuracy guarantee. +The default is a starting point, not evidence that a new rule is calibrated. + +Keep intended labels even when the model gets them wrong. Inspect wording, scope, +and evidence when valid and violating scores overlap. Do not weaken a rule or tune +on one example merely to make a check pass. If inference is unavailable, validate +what you can and report that calibration remains untested. + +Normal checks read the base policy; a new rule is tested through the example +harness's working-tree policy before it becomes part of the base. Keep calibration +out of commit hooks. Store generated reports locally or as CI artifacts; check in +deliberate rules and examples. + +## Finish with an honest result + +Summarize the requirement, scope and exceptions, examples tested, chosen cutoff, +and any missed violations, false blocks, or incomplete runs. Do not promise +“never again” from passing examples. Preserve the project's choice of optional or +required CI checks and its existing hook behavior; adding a rule does not authorize +changing merge protection, publishing a package, or posting messages. diff --git a/skills/judgement/agents/openai.yaml b/skills/judgement/agents/openai.yaml new file mode 100644 index 0000000..29ee09b --- /dev/null +++ b/skills/judgement/agents/openai.yaml @@ -0,0 +1,3 @@ +interface: + display_name: "Judgement" + short_description: "Turn recurring mistakes into calibrated repository rules" diff --git a/skills/judgement/references/agent-guide.md b/skills/judgement/references/agent-guide.md new file mode 100644 index 0000000..0331474 --- /dev/null +++ b/skills/judgement/references/agent-guide.md @@ -0,0 +1,173 @@ +# Using Judgement as a coding agent + +Judgement checks repository diffs against natural-language rules. The project owns +its rules and labeled examples. Judgement owns running checks, testing examples, +comparing thresholds, and reporting results. + +## Find the project setup + +1. Read the repository's agent instructions and `.judgement/rules.json`. +2. Inspect its package scripts and existing inference adapter. Use the project's + configured command and backend when available; a different model may score the + same examples differently. Do not add credentials to files or output. +3. Use the installed, pinned package. Do not install packages inside a commit hook. + Check `judgement --help` for the commands supported by the installed version. + +## Write a rule + +Save rules in `.judgement/rules.json`: + +```json +{ + "criteria": [ + { + "id": "billing-audit", + "rule": "Every successful billing mutation must record an audit event for that mutation. Read-only queries do not require audit events.", + "files": ["src/billing/**"], + "context": ["src/audit/**"], + "threshold": 0.85 + } + ] +} +``` + +- Give each rule a stable ID. Use letters, digits and hyphens for easy filenames. +- Describe one observable requirement, its scope, and meaningful exceptions. + Split unrelated requirements when they need different evidence or thresholds. +- Scope `files` to relevant paths. Globs are relative to the repository root and + match hidden paths too. `context` selects unchanged supporting files from the + same Git snapshot. Keep it focused; it is not a request to inspect the whole repo. +- A threshold is a violation-probability cutoff, not proof of accuracy. The default 0.85 is a + starting point. Do not weaken the rule or lower its threshold just to get green. + +## Save labeled examples + +For each rule, write `.judgement/examples/.json`: + +```json +{ + "ruleId": "billing-audit", + "examples": [ + { + "name": "Mutation without audit event", + "path": "src/billing/update.ts", + "before": "export async function update(db, audit, id) { await db.update(id); await audit.record(id); }\n", + "after": "export async function update(db, audit, id) { await db.update(id); }\n", + "expected": "violation" + }, + { + "name": "Read-only query", + "path": "src/billing/read.ts", + "before": "export async function read(db) { return []; }\n", + "after": "export async function read(db) { return db.findMany(); }\n", + "expected": "pass" + } + ] +} +``` + +Label by the intended policy **before** looking at model scores. Include clear +violations, allowed exceptions, fixes, unrelated edits near old violations, and +realistic context. Keep failing examples; investigate them instead of changing the +expected label to match the model. + +`before` and `after` must differ. Use `null` for the absent side of an addition or +deletion. Optional `context` maps unchanged relative paths to file contents; the +rule's context globs still control which files are included. Fixtures are source +text, not programs to execute. Never put real secrets or private data in examples. +If a rule allows synthetic placeholders, those are valid cases, not positive +secret-detection tests. State that coverage limitation explicitly. + +## Test before tuning + +Run with the project's pinned CLI or equivalent adapter command: + +```sh +judgement test --dry-run +judgement test --rule billing-audit --repeats 3 +judgement test --format json > /tmp/judgement-tests.json +judgement calibrate --rule billing-audit --thresholds 0.8,0.85,0.9,0.95 +judgement calibrate --all --format json > /tmp/judgement-calibration.json +``` + +`test` uses each rule's configured threshold and requires every run to match its +label. A valid example must fully pass; an incomplete result fails the test. +`calibrate` compares candidates. A recommendation can still leave valid examples +incomplete, so read the full report. Live runs make paid model requests; dry runs +make none. A recommendation requires both valid and violating examples; a +single-label set can test behavior but cannot select a threshold. Testing and calibration are explicit operations, never part of the +normal commit hook. + +The harness uses disposable Git repositories, repeats without a verdict cache, +and preserves the real working tree and index. It reads working-tree rules so +edited candidate thresholds can be tested. Normal diff checks use the base policy, +so a rule edit cannot weaken the check on its own commit. + +Inspect false blocks, caught violations, incorrect passes, incomplete results, +operational failures, scores, and timing separately. When a case fails: + +1. Check its label, path, rule wording, and selected supporting context. +2. Distinguish an uncertain correct answer from a wrong answer. Lowering a cutoff can catch more violations and also introduce false blocks. + If valid and violating examples overlap, clarify the rule or its evidence. +3. Repeat the run. Never present one score as a guarantee. +4. Validate a candidate against separately labeled held-out examples before adopting + it. Use `--examples` for one rule, or `test --examples-dir` / `calibrate --all +--examples-dir` for a separate suite. Commit deliberate rule and fixture changes; + keep generated reports local or as CI artifacts. + +## Interpret exit codes and report honestly + +- `test`: 0 means all labels matched (or dry run); 1 means wrong, incomplete, or + operationally failed checks; 2 means invalid input/setup; 130 means interrupted. +- `calibrate`: 0 means a candidate was recommended (or dry run); 3 means no candidate + was recommended for at least one selected rule; 2 means invalid input/setup. +- `check --hook` allows incomplete results to keep commits moving. Strict `check` + rejects incomplete results. Neither behavior makes an incomplete result a pass. + +Report what was tested, repeated counts, backend differences, incomplete cases, +and untested categories. Passing small fixtures does not prove that large commits, +missing context, or every possible violation will be handled correctly. + +Applications with existing credentials should supply their evaluator to `testRules` +or `calibrateRules`. Keep that adapter thin: do not copy the runner, fixture loading, +isolated repository setup, threshold logic, or reporting into each application. + +## Inspect uncertainty in a model tester + +Use `prepareExamples` to obtain exact checker request packets from the project's +fixtures. Send each packet's `state` and `questions` to the configured backend; +keep `expected`, names, and thresholds as tester metadata. Inspect both initial +and expanded packets, with raw violation probabilities visible. +Preparation makes no inference calls. Generated inputs can be bundled as presets; +regenerate them when their policy, fixture, or package version changes. + +The model answers whether a change violates the rule. Its Noul value is the +probability of a violation. Scores at or above the cutoff flag a violation; +every lower score produces no finding, including an ambiguous score of 0.5. +Missing required evidence, partial screens, unsupported files, and failed requests +remain incomplete. The default binary evaluator does not request context expansion +based on its probability. Use explicit context and compare expanded packets when +investigating a missed violation. Recalibrate when changing question types; Choice +confidence and Noul probability use different meanings and scales. + +## Collect and compare feedback + +Use `judgement capture --rule --path --name --expected pass|violation` +to preview an index-based example. Review and redact it before saving a new file +with `--output`; capture never overwrites or stages files. Include unchanged +supporting evidence with `--context`. Reduce multi-file changes rather than +silently discarding context that changed. Captured content must satisfy the +repository's privacy requirements before it is committed. + +Save baseline and candidate `test --format json` reports locally, using identical +fixtures. `judgement compare --before --after ` exposes per-example +regressions and exits 1 when any are found. Do not change expected labels to hide +regressions. Compare both valid exceptions and violations; repeated observations +of the same example are not independent coverage. + +Finding snippets are changed-line previews, not model-localized accusations. +Read the supplied evidence before proposing a fix. Progress and cache-hit counts +are diagnostics; only a complete report establishes check completion. In CI, use +strict checks without `--hook` or `--advisory`. Preserve the project's choice of +optional or required status checks. Missing inference credentials must not produce +a successful check.