Shadow AI does not match the cMCP v0.2.0 catalog and audit formats it declares in tested_against

[devdiv07]

Summary

integrations/shadow-ai declares:

cmcp-runtime: "0.2.0"

I tested the current integration against the cMCP v0.2.0 git tag and found two independent producer/consumer contract mismatches.

Pinned revisions:

• agentrust-io/integrations@e63b2d039628e5dfd4e15290687fec356d3a46fd
• agentrust-io/cmcp@v0.2.0
• cMCP commit 572ba840dde70ab5bca24399cae51a72a479dcea
• Python 3.13.5

As a positive control, Shadow AI's own supported catalog shape and documented record shape returned no events. This confirms the scanner behaves as expected on the integration's own fixture shape.

Shadow AI is listed as Community tier. The repository documents Community integrations as structure-validated/listed and says their code is not run by maintainers; this report is therefore about the declared compatibility behavior, not an allegation that the tier promised end-to-end execution.

These are the exact catalog/audit inputs used by the integration's documented scanner path.

1. cMCP's shipped catalog cannot be loaded

cMCP v0.2.0 ships examples/minimal/catalog.json as a top-level JSON array of tool-centric entries:

[
  {
    "tool_name": "echo",
    "server": { "...": "..." },
    "approved_definition": { "...": "..." }
  }
]

Passing that exact file unchanged to ShadowAIScanner produces:

ValueError: Unrecognized catalog format in .../cmcp-v0.2.0/examples/minimal/catalog.json

ShadowAIScanner._load_catalog() currently accepts either an {"agents": [...]} mapping or a flat agent_id -> tools mapping. The cMCP v0.2.0 producer format is neither.

2. cMCP audit entries do not carry the agent identity the scanner requires

Independently of the catalog failure, I generated a real cMCP v0.2.0 AuditEntry using AuditChain.append() and serialized it with dataclasses.asdict(), matching the representation written by SqliteAuditStore.

The genuine record contains session_id, timestamp_utc, and tool_name, but no agent_id.

To isolate this audit boundary from the catalog incompatibility, I used a catalog shape already accepted by Shadow AI while leaving the generated cMCP audit record unchanged.

That genuine record produces:

{
    "agent_id": "unknown",
    "tool_name": "echo",
    "timestamp": "",
    "reason": "unregistered_agent",
    "suggested_manifest_id": "unknown",
}

The causal control isolates the missing identity field:

genuine cMCP record                 -> unregistered_agent
same record + agent_id              -> no event
same record + agent_id + timestamp  -> no event
same record + agent_id + bad tool   -> undeclared_tool

Adding agent_id alone removes the unregistered_agent classification for the declared tool. Adding timestamp does not change classification. With the identity present, changing the tool to one outside the declared set correctly produces undeclared_tool.

There is also a separate non-classification field mismatch: cMCP emits timestamp_utc, while Shadow AI reads timestamp, so genuine entries currently produce an empty event timestamp. This does not control classification.

Why I'm filing this as a contract issue rather than proposing a fix

This does not appear to be a simple field rename.

At cMCP v0.2.0:

• the catalog is tool-centric rather than an agent -> tools registry;
• the audit entry carries session/tool-call identity but no agent_id.

Shadow AI's advertised behavior requires determining whether an agent is registered and whether a tool is declared for that agent.

That seems to require a design decision about the authoritative source of agent identity and registration. Possible resolutions could include changing the identity source, narrowing the documented behavior or compatibility claim, or another approach maintainers prefer.

Scope

This is a declared-version integration functionality / producer-consumer contract mismatch.

No security vulnerability, authorization bypass, or cMCP defect is claimed.

[imran-siddique]

Narrow the compatibility claim to the inputs the scanner actually supports. Document the separate agent registry and enriched-record format, and withdraw the direct cMCP v0.2.0 compatibility claim until an adapter is tested.

Missing agent identity must be reported as unavailable, rather than classified as an unregistered agent named unknown. A future adapter needs an explicit, trustworthy session-to-agent mapping; the tool catalog cannot supply that identity.

Keep the timestamp mapping fix separate from the identity decision. Add genuine cMCP catalog/audit fixtures and positive controls before restoring the compatibility claim.

[devdiv07]

Thanks, Imran. I've reviewed the scanner's current result contract.

I can take the scoped compatibility correction: withdraw the untested direct cMCP v0.2.0 claim and update the README, manifest, and marketplace description to document the separate agent registry and enriched-record inputs.

One API detail I'd like to align on: DiscoveryEvent currently requires a string agent_id, derives suggested_manifest_id from it, and both scanner methods return list[DiscoveryEvent].

I propose reporting missing identity through a separate unavailable diagnostic, with an explicit scan-report interface. The existing scanner methods would fail clearly on missing identity rather than manufacture unknown or silently skip records.

Would that approach work, or would you prefer extending DiscoveryEvent to represent unavailable identity?

I'll keep timestamp normalization and the future cMCP identity adapter outside this PR.

[imran-siddique]

Use the separate diagnostic with an explicit scan-report interface. Keep DiscoveryEvent for findings where an agent identity is available.

The report should retain valid findings, identify unclassifiable records by line or index, and make incomplete classification explicit. Missing identity must produce neither an unregistered_agent finding nor a suggested manifest ID.

Keep the existing list-returning methods strict: raise a documented error for tool-call records without usable identity rather than returning an apparently complete list. Preserve their behavior for valid inputs, and document this compatibility change.

Include mixed-record tests and missing, null, blank and non-string identity cases. Timestamp normalization and the future cMCP adapter remain separate.