Skip to content

About

Define, execute, and validate behavioral contracts for MCP tools.

Topics

Resources

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

JOONS MCP Contract Test Runner

Preview Apache-2.0 Node >=20 CI

Define, execute, and validate behavioral contracts for MCP tools. JSON suites run against a JOONS MCP Mock Server started by the runner. Part of the JOONS ecosystem: Probe diagnoses a real server, Diff compares snapshots and makes snapshot files, Policy Gate checks snapshots against policies, and Mock Server serves a mock from a snapshot and fixtures.

Why

Check expected tool behavior consistently with readable assertions, capture chains, ordered reports, and CI-friendly exit codes.

Safety

Version 0.1.0 runs only against a JOONS MCP Mock Server. Before any tool call, the runner checks the server's name, title, and instructions. Real-server execution is not in 0.1.0; it is planned with explicit opt-in and an exact per-tool allowlist. Input is data, never executed. The child starts without a shell, in an empty temporary directory, with a minimal environment; it is cleaned up on all paths. See SECURITY.md for the library exception and limitations.

Quick start

Install Node.js 20 or newer, then install the runner and Mock Server separately:

npm install -g github:dbdefault/joons-mcp-contract-runner#v0.1.0
npm install -g github:dbdefault/joons-mcp-mock-server#v0.1.0

Alternatively pass the Mock Server's .js entry point with --mock-server PATH. Create a small suite such as suite.json:

{
  "format": "joons-mcp-contract-test/suite",
  "suiteVersion": 1,
  "tests": [
    {"id":"search", "tool":"search_products", "arguments":{"query":"boots"},
     "expect":{"outcome":"result", "isError":false, "text":{"contains":"boots"}}}
  ]
}

Using the included snapshot and fixtures from a checkout:

joons-mcp-contract-test validate suite.json --snapshot examples/snapshot.json
joons-mcp-contract-test run suite.json --snapshot examples/snapshot.json --fixtures examples/fixtures.json

If the globally installed Mock Server is not resolved from the working directory or runner installation, pass its .js path using --mock-server. More examples and the outage scenario are in examples/README.md.

Suite format overview

A suite has format, suiteVersion, and one to 1000 ordered tests; optional name, defaults, and tools configure it. Each test has a unique id, a tool, and an expect, with optional arguments, timeout, skip and captures. Tool inclusion checks produce a synthetic #tools result. Unknown schema keys are errors. See suite reference.

Assertions

Assertion Checks
outcome result or JSON-RPC error
isError, contentCount result error flag and number of content blocks
text equals, contains, or startsWith on joined text blocks
structuredContent exact equality or recursive subset match
fields pointer existence, equality, captured equality, type or length bounds
maxDurationMs elapsed time limit
errorCode, errorMessageContains JSON-RPC error code and message

Missing assertion inputs do not pass by default. See assertion examples.

Captures

Capture named values from a passing test's structured content using JSON pointers. Refer to an earlier capture in arguments with {"$capture":"name"}. Captures are copied, isolated between runs and omitted from reports. If a value is unavailable, dependent tests skip.

Concurrency and isolation

Concurrency defaults to 1 and ranges from 1 to 8. Independent tests may run in parallel; tests using or producing captures wait for earlier tests and run in order. Results remain in suite order. Per-test timeout defaults to 5000 ms. The total timeout defaults to 600000 ms and skips remaining tests when exceeded. A missing listed tool fails a result-expecting test without a call.

Reports and exit codes

Output formats are terminal, JSON, and Markdown; --json-out and --md-out write files atomically. --stable makes durations zero and omits timestamps. Arguments and captures are omitted, sensitive keys are masked, and untrusted terminal/Markdown text is sanitized. Skips alone do not fail a run.

Exit Meaning
0 PASS, or --fail-on none; validation valid
1 FAIL verdict
2 invalid input, mock unavailable, startup or identity failure
3 usage error
4 output write error

CI usage

For a checkout with development dependencies, a GitHub Actions step can run:

- uses: actions/checkout@v4
- uses: actions/setup-node@v4
  with:
    node-version: 20
    cache: npm
- run: npm ci
- run: node scripts/test.js
- run: node bin/joons-mcp-contract-test.js validate examples/shop.suite.json --snapshot examples/snapshot.json
- run: node bin/joons-mcp-contract-test.js run examples/shop.suite.json --snapshot examples/snapshot.json --fixtures examples/fixtures.json --mock-server node_modules/joons-mcp-mock-server/bin/joons-mcp-mock-server.js --stable

See CI guidance and the workflow.

Limits

Suites are limited to 8 MiB, snapshots to 64 MiB, fixtures to 16 MiB; JSON complexity and test counts are bounded. See suite reference for details. Mock behavior reflects supplied snapshot and fixtures, not a real server's current behavior.

Not in 0.1.0

The CLI cannot run against real servers. The planned real-server design requires explicit opt-in and an exact per-tool allowlist. The library has an explicit allowNonMock escape hatch; it is not exposed by the CLI.

Status

Preview, not on npm. Install from the GitHub tag shown above. Licensed under Apache-2.0.

About

Define, execute, and validate behavioral contracts for MCP tools.

Topics

Resources

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages