Skip to content

Upgrade Judgement to 0.2.0 with shared rule tests and prose rule - #3307

Merged
mrubens merged 3 commits into
developfrom
codex/calibrate-roomote-rules
Sep 29, 2026
Merged

mrubens merged 3 commits into
developfrom
codex/calibrate-roomote-rules

Conversation

@mrubens

@mrubens mrubens commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Upgrade the local and managed-worker integrations to published @roo-code/judgement@0.2.0. Store repository rules in .judgement/rules.json and use Judgement's shared example APIs through Roomote's existing model settings and credentials:

  • pnpm judgement test tests configured thresholds.
  • pnpm judgement calibrate --rule <id> compares threshold candidates; --all selects every rule.

Add 57 labeled examples across eight rules. The new prose rule asks for direct descriptions of current behavior and actions, with exemptions for release notes, migration guidance, security or compatibility requirements, and quoted examples. Existing rule thresholds remain unchanged.

Judgement owns fixture execution, isolated Git repositories, scoring and reports. Roomote supplies a thin inference adapter and argument handling. Generated results stay local. Update public and local documentation, worker test fixtures, lockfile integrity, and the exact first-party release-age exception for 0.2.0.

Validation

  • Published package installed from npm; both workspace consumers resolve 0.2.0. Frozen-lockfile installation passed.
  • Local and worker adapter tests: 12 passed.
  • pnpm check-types:fast: 27 tasks passed.
  • pnpm lint:fast and pnpm knip: passed.
  • CLI dry run validated all 57 examples without inference; changed-file formatting passed.
  • Live smoke test through the published package and Roomote CLI completed without backend errors. All three prose-rule violations were blocked, with no false blocks among seven valid examples. Four valid examples remained incomplete, so the example suite correctly exited nonzero. This is a known calibration limitation, not a passing calibration suite.

Privacy fixtures contain only allowed synthetic values and do not establish real-secret detection recall. The labeled examples preserve observed gaps rather than changing expectations to fit model output.

@roomote-community

roomote-community Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

No new code issues found. See task

Reviewed 71d21e7

@mrubens mrubens changed the title Add calibration examples for repository judgement rules Add labeled examples for repository judgement rules Sep 29, 2026
@mrubens mrubens changed the title Add labeled examples for repository judgement rules Upgrade Judgement to 0.2.0 with shared rule tests and prose rule Sep 29, 2026
@mrubens
mrubens merged commit bea3e6d into develop Sep 29, 2026
19 checks passed
@mrubens
mrubens deleted the codex/calibrate-roomote-rules branch September 29, 2026 20:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant