Add installable Judgement skill for recurring corrections

README.md

@@ -23,19 +23,33 @@ Save this as `.judgement/rules.json` in your repository. Run Judgement before a
 or in CI. It uses [Jev](https://typesafe.ai) to make focused judgments and reads
 related source files from the exact Git snapshot being checked.
 
-## Agent instructions
+## Agent skill
 
-Coding agents: read [the agent guide](docs/agents.md) before adding or tuning rules.
-It is included in the npm package and covers fixtures, commands, uncertainty, and
-validation. Projects should keep their rules and examples, not copy the runner.
+Install the [Judgement skill](skills/judgement/SKILL.md) with the
+[skills CLI](https://skills.sh/docs/cli):
+
+```sh
+npx skills add RooCodeInc/judgement --skill judgement
+```
+
+The installer lets you choose the agents to use it with. The skill can be selected
+for requests such as “never do that again,” “add a rule to prevent this mistake,”
+or “calibrate this Judgement rule.” It guides the agent to scope the requirement,
+reuse existing rules, add labeled examples, and test the chosen cutoff. Mechanical
+checks and conversation-only preferences are routed to the appropriate tooling.
+
+You can also copy the complete `skills/judgement/` directory into your agent's
+skill directory. Keep its `references/` directory alongside `SKILL.md`. The skill
+is included in the npm package; installing it configures agent guidance. Projects
+configure the Judgement CLI, rules, and inference separately.
 
 ## Install
 
 Node 22.20+ or 24+ and Git are required. Glob matching uses the small `picomatch` dependency.
 Install the published package from npm.
 
 ```sh
-npm install --save-dev --save-exact @roo-code/judgement@0.3.0
+npm install --save-dev --save-exact @roo-code/judgement@0.4.0
 export TYPESAFE_API_KEY=...
 ./node_modules/.bin/judgement check --staged
 ```
@@ -57,8 +71,9 @@ or evidence is insufficient, it reports **incomplete** and allows the commit.
 It never presents an unfinished review as a pass. Process startup and operating
 system scheduling add overhead; use the benchmark to measure your environment.
 
-A required CI check must finish that work before merging. Judgement does not
-schedule a background check or configure branch protection for you.
+Use strict checks in CI to finish that work. Projects choose whether the status
+is required before merge. Judgement does not schedule a background check or
+configure branch protection for you.
 
 For the [pre-commit framework](https://pre-commit.com), use this repository's
 `.pre-commit-hooks.yaml` with `rev: v0.1.5` and hook `id: judgement`.

docs/agents.md

@@ -1,173 +1,10 @@
-# Using Judgement as a coding agent
+# Judgement for coding agents
 
-Judgement checks repository diffs against natural-language rules. The project owns
-its rules and labeled examples. Judgement owns running checks, testing examples,
-comparing thresholds, and reporting results.
+Use the [Judgement skill](../skills/judgement/SKILL.md) to turn recurring user
+corrections into scoped repository rules and labeled examples.
 
-## Find the project setup
+The [agent guide](../skills/judgement/references/agent-guide.md) covers schemas,
+commands, calibration, and interpreting results. These files are included in the
+npm package and can be read directly without installing the skill.
 
-1. Read the repository's agent instructions and `.judgement/rules.json`.
-2. Inspect its package scripts and existing inference adapter. Use the project's
-   configured command and backend when available; a different model may score the
-   same examples differently. Do not add credentials to files or output.
-3. Use the installed, pinned package. Do not install packages inside a commit hook.
-   The commands below require the release containing the calibration API; check
-   `judgement --help` rather than assuming an older installation supports them.
-
-## Write a rule
-
-Save rules in `.judgement/rules.json`:
-
-```json
-{
-  "criteria": [
-    {
-      "id": "billing-audit",
-      "rule": "Every successful billing mutation must record an audit event for that mutation. Read-only queries do not require audit events.",
-      "files": ["src/billing/**"],
-      "context": ["src/audit/**"],
-      "threshold": 0.85
-    }
-  ]
-}
-```
-
-- Give each rule a stable ID. Use letters, digits and hyphens for easy filenames.
-- Describe one observable requirement, its scope, and meaningful exceptions.
-  Split unrelated requirements when they need different evidence or thresholds.
-- Scope `files` to relevant paths. Globs are relative to the repository root and
-  match hidden paths too. `context` selects unchanged supporting files from the
-  same Git snapshot. Keep it focused; it is not a request to inspect the whole repo.
-- A threshold is a violation-probability cutoff, not proof of accuracy. The default 0.85 is a
-  starting point. Do not weaken the rule or lower its threshold just to get green.
-
-## Save labeled examples
-
-For each rule, write `.judgement/examples/<id>.json`:
-
-```json
-{
-  "ruleId": "billing-audit",
-  "examples": [
-    {
-      "name": "Mutation without audit event",
-      "path": "src/billing/update.ts",
-      "before": "export async function update(db, audit, id) { await db.update(id); await audit.record(id); }\n",
-      "after": "export async function update(db, audit, id) { await db.update(id); }\n",
-      "expected": "violation"
-    },
-    {
-      "name": "Read-only query",
-      "path": "src/billing/read.ts",
-      "before": "export async function read(db) { return []; }\n",
-      "after": "export async function read(db) { return db.findMany(); }\n",
-      "expected": "pass"
-    }
-  ]
-}
-```
-
-Label by the intended policy **before** looking at model scores. Include clear
-violations, allowed exceptions, fixes, unrelated edits near old violations, and
-realistic context. Keep failing examples; investigate them instead of changing the
-expected label to match the model.
-
-`before` and `after` must differ. Use `null` for the absent side of an addition or
-deletion. Optional `context` maps unchanged relative paths to file contents; the
-rule's context globs still control which files are included. Fixtures are source
-text, not programs to execute. Never put real secrets or private data in examples.
-If a rule allows synthetic placeholders, those are valid cases, not positive
-secret-detection tests. State that coverage limitation explicitly.
-
-## Test before tuning
-
-Run with the project's pinned CLI or equivalent adapter command:
-
-```sh
-judgement test --dry-run
-judgement test --rule billing-audit --repeats 3
-judgement test --format json > /tmp/judgement-tests.json
-judgement calibrate --rule billing-audit --thresholds 0.8,0.85,0.9,0.95
-judgement calibrate --all --format json > /tmp/judgement-calibration.json
-```
-
-`test` uses each rule's configured threshold and requires every run to match its
-label. A valid example must fully pass; an incomplete result fails the test.
-`calibrate` compares candidates. A recommendation can still leave valid examples
-incomplete, so read the full report. Live runs make paid model requests; dry runs
-make none. A recommendation requires both valid and violating examples; a
-single-label set can test behavior but cannot select a threshold. Testing and calibration are explicit operations, never part of the
-normal commit hook.
-
-The harness uses disposable Git repositories, repeats without a verdict cache,
-and preserves the real working tree and index. It reads working-tree rules so
-edited candidate thresholds can be tested. Normal diff checks use the base policy,
-so a rule edit cannot weaken the check on its own commit.
-
-Inspect false blocks, caught violations, incorrect passes, incomplete results,
-operational failures, scores, and timing separately. When a case fails:
-
-1. Check its label, path, rule wording, and selected supporting context.
-2. Distinguish an uncertain correct answer from a wrong answer. Lowering a cutoff can catch more violations and also introduce false blocks.
-   If valid and violating examples overlap, clarify the rule or its evidence.
-3. Repeat the run. Never present one score as a guarantee.
-4. Validate a candidate against separately labeled held-out examples before adopting
-   it. Use `--examples` for one rule, or `test --examples-dir` / `calibrate --all
---examples-dir` for a separate suite. Commit deliberate rule and fixture changes;
-   keep generated reports local or as CI artifacts.
-
-## Interpret exit codes and report honestly
-
-- `test`: 0 means all labels matched (or dry run); 1 means wrong, incomplete, or
-  operationally failed checks; 2 means invalid input/setup; 130 means interrupted.
-- `calibrate`: 0 means a candidate was recommended (or dry run); 3 means no candidate
-  was recommended for at least one selected rule; 2 means invalid input/setup.
-- `check --hook` allows incomplete results to keep commits moving. Strict `check`
-  rejects incomplete results. Neither behavior makes an incomplete result a pass.
-
-Report what was tested, repeated counts, backend differences, incomplete cases,
-and untested categories. Passing small fixtures does not prove that large commits,
-missing context, or every possible violation will be handled correctly.
-
-Applications with existing credentials should supply their evaluator to `testRules`
-or `calibrateRules`. Keep that adapter thin: do not copy the runner, fixture loading,
-isolated repository setup, threshold logic, or reporting into each application.
-
-## Inspect uncertainty in a model tester
-
-Use `prepareExamples` to obtain exact checker request packets from the project's
-fixtures. Send each packet's `state` and `questions` to the configured backend;
-keep `expected`, names, and thresholds as tester metadata. Inspect both initial
-and expanded packets, with raw violation probabilities visible.
-Preparation makes no inference calls. Generated inputs can be bundled as presets;
-regenerate them when their policy, fixture, or package version changes.
-
-The model answers whether a change violates the rule. Its Noul value is the
-probability of a violation. Scores at or above the cutoff flag a violation;
-every lower score produces no finding, including an ambiguous score of 0.5.
-Missing required evidence, partial screens, unsupported files, and failed requests
-remain incomplete. The default binary evaluator does not request context expansion
-based on its probability. Use explicit context and compare expanded packets when
-investigating a missed violation. Recalibrate when changing question types; Choice
-confidence and Noul probability use different meanings and scales.
-
-## Collect and compare feedback
-
-Use `judgement capture --rule <id> --path <file> --name <name> --expected pass|violation`
-to preview an index-based example. Review and redact it before saving a new file
-with `--output`; capture never overwrites or stages files. Include unchanged
-supporting evidence with `--context`. Reduce multi-file changes rather than
-silently discarding context that changed. Captured content must satisfy the
-repository's privacy requirements before it is committed.
-
-Save baseline and candidate `test --format json` reports locally, using identical
-fixtures. `judgement compare --before <file> --after <file>` exposes per-example
-regressions and exits 1 when any are found. Do not change expected labels to hide
-regressions. Compare both valid exceptions and violations; repeated observations
-of the same example are not independent coverage.
-
-Finding snippets are changed-line previews, not model-localized accusations.
-Read the supplied evidence before proposing a fix. Progress and cache-hit counts
-are diagnostics; only a complete report establishes check completion. In CI, use
-strict checks without `--hook` or `--advisory`, and require the resulting status
-before merge. Missing inference credentials must not produce a successful check.
+See the [README](../README.md#agent-skill) for skill installation.

package.json

@@ -22,7 +22,8 @@
     "LICENSE",
     "examples",
     ".pre-commit-hooks.yaml",
-    "docs"
+    "docs",
+    "skills"
   ],
   "scripts": {
     "test": "node --test test/*.test.js",

skills/judgement/SKILL.md

@@ -0,0 +1,89 @@
+---
+name: judgement
+description: Turn recurring repository mistakes and user corrections into calibrated Judgement rules. Use when a user asks an agent to "never do that again," prevent a mistake from recurring, enforce a contextual code or prose requirement, or add, test, or tune Judgement rules. Applies to requirements visible in repository changes; conversation-only preferences belong in agent instructions or memory.
+---
+
+# Judgement
+
+Make a recurring requirement concrete: a focused rule in `.judgement/rules.json`,
+labeled examples, and evidence that the configured checker distinguishes the
+mistake from allowed changes. Judgement checks Git diffs; it cannot guarantee that
+an agent will never repeat a behavior.
+
+## Turn the correction into a requirement
+
+Use the user's correction and the surrounding task to identify the unwanted
+behavior, where it matters, and the allowed exceptions. A clear request to prevent
+recurrence is enough to make a scoped repository rule change. Ask a focused
+question only if the intended requirement or scope is ambiguous.
+
+Choose the appropriate mechanism:
+
+- Use Judgement for contextual distinctions that need judgment: misleading setup
+  prose, exposing internal errors to users, or carrying credentials across a trust
+  boundary.
+- Use an existing formatter, linter, type check, or deterministic test when it can
+  express the requirement reliably. Do not add a model check for a mechanical ban.
+- Use the project's agent instructions or the user's preferred memory mechanism
+  for conversational or tool-use behavior that is not visible in committed files.
+  A rule about Git diffs cannot enforce how an agent talks in chat.
+
+Keep the scope the user intended. A correction to one situation does not establish
+an unrelated repository-wide policy. Resolve equivalent existing rules before
+adding another; preserve their stable IDs and useful examples.
+
+## Find the setup and write the rule
+
+Read repository instructions, `.judgement/rules.json`, existing fixtures, and the
+package scripts. Use the project's pinned Judgement command and inference adapter.
+Check its `--help` for available commands. If Judgement is not configured, explain
+what setup is needed and use the repository's dependency and credential conventions
+when setup is within the request. Installing this skill does not install the CLI
+or configure inference.
+
+Read [the agent guide](references/agent-guide.md) for the rule and example schemas,
+commands, calibration, and result semantics before editing rules or fixtures.
+
+Describe one observable requirement and its meaningful exceptions. Select relevant
+`files` globs and only the supporting `context` needed to judge it. Favor localized
+rules so evidence stays small and checks stay fast. Do not expand context to the
+whole repository to compensate for an unclear rule.
+
+For example, “stop capitalizing session everywhere” may mean ordinary nouns in
+user-facing prose, while sentence starts, headings, exact UI labels, and code
+identifiers remain valid. Preserve the exceptions the user actually intends.
+
+## Demonstrate the behavior
+
+Add labeled examples under `.judgement/examples/<rule-id>.json`. Include the
+mistake, a corrected version, allowed exceptions, and a nearby unrelated edit that
+should pass. Label examples from the intended requirement before inspecting scores.
+Use synthetic or redacted content; do not copy private data into fixtures.
+
+For staged or committed feedback, `judgement capture` can preview an example.
+Review the preview before saving; do not stage unrelated user changes to capture
+them. Preserve the index and working tree while testing.
+
+Use the configured command to dry-run the fixtures, then run a bounded repeated
+test of the affected rule when its inference backend is available. Use calibration
+to compare cutoffs and separately labeled examples to check the chosen cutoff.
+A cutoff is a violation-probability threshold, not a measured accuracy guarantee.
+The default is a starting point, not evidence that a new rule is calibrated.
+
+Keep intended labels even when the model gets them wrong. Inspect wording, scope,
+and evidence when valid and violating scores overlap. Do not weaken a rule or tune
+on one example merely to make a check pass. If inference is unavailable, validate
+what you can and report that calibration remains untested.
+
+Normal checks read the base policy; a new rule is tested through the example
+harness's working-tree policy before it becomes part of the base. Keep calibration
+out of commit hooks. Store generated reports locally or as CI artifacts; check in
+deliberate rules and examples.
+
+## Finish with an honest result
+
+Summarize the requirement, scope and exceptions, examples tested, chosen cutoff,
+and any missed violations, false blocks, or incomplete runs. Do not promise
+“never again” from passing examples. Preserve the project's choice of optional or
+required CI checks and its existing hook behavior; adding a rule does not authorize
+changing merge protection, publishing a package, or posting messages.

skills/judgement/agents/openai.yaml

@@ -0,0 +1,3 @@
+interface:
+  display_name: "Judgement"
+  short_description: "Turn recurring mistakes into calibrated repository rules"