Skip to content

[Router] Interfaces + fixtures stub (foundation)#2408

Open
ramkrishna2910 wants to merge 6 commits into
mainfrom
router/2407-interfaces-fixtures
Open

[Router] Interfaces + fixtures stub (foundation)#2408
ramkrishna2910 wants to merge 6 commits into
mainfrom
router/2407-interfaces-fixtures

Conversation

@ramkrishna2910

@ramkrishna2910 ramkrishna2910 commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Closes #2407.

Foundation for the Lemonade Router milestone (#2389): the shared C++ contract surface, the policy/decision JSON schemas, the back-compat guardrails, and the per-tier example policies that every downstream engine/wiring issue codes against. Interfaces only — no engine behavior (route() is declared, not implemented; that's #2382).

Draft: gated on schema sign-off (#2375 / #2376). If a schema is revised, churn stays in the schema files + fixtures; the C++ surface barely moves.

Contract header — src/cpp/include/lemon/routing_policy.h

Std library + nlohmann/json only — no Router/backend include, per the engine invariant.

  • Data: RouteContext, Score (label→score), OnError, Rule, MatchExpr AST, TraceEntry, Decision, RoutePolicy.
  • Injection seam: ClassifierServices (embed / run_classifier / chatchat powers the L0a llm router), ClassifierContext, Classifier.
  • Evaluation seam: Condition, EvalContext (per-request state: classifier memo + trace), LeafFactory — the shared contract that lets the evaluator ([Router] Core types + match-expression evaluator (M6/M7) #2378) and registry ([Router] Classifier + Condition registry (M3) #2379) be built in parallel without colliding at the leaf boundary.
  • Engine: RoutingPolicyEngine ctor signature; route() declared, intentionally undefined.

Schemas — src/cpp/resources/schemas/

Back-compat guardrails

The README documents the contract — never redefine a shipped field; never delete a major's schema; migrate-on-load — enforced mechanically, not by convention:

  • schema-lock.json + test/test_schema_lock.py — canonical-hash snapshot guard (buf breaking-lite). A released major is immutable (any change fails CI → ship a new major); an unreleased major may change only with a refreshed lock in the same diff, so every schema edit is a visible, reviewed change. (v1 is released: false until sign-off.)
  • test/test_routing_fixtures.py — schemas are self-valid and every fixture conforms.

(Behavioral back-compat — golden policy → Decision — is the conformance corpus #2425, non-gating, tracked separately.)

Fixtures — test/cpp/fixtures/routing/

Lean local-form examples, verbatim from the locked level issues:

Fixture Level Source
l0a_llm_router.json L0(a) llm-as-router #2405
l1_keywords.json L1 deterministic (keywords / regex / chars) #2380
l1_metadata.json L1 deterministic (metadata match — equals/any/exists) #2380
l2_semantic.json L2 semantic_similarity #2381
l3_classifier.json L3 model-backed classifier #2379
decision_example.json Decision + trace #2376

The metadata leaf condition (lets rules branch on caller-supplied OpenAI metadata like task_class / consent) was contributed by @SlawomirNowaczyk.

Tests

  • test/cpp/test_routing_policy_contract.cpp → CTest RoutingPolicyContractTest: header compiles standalone, every type constructs, the Condition/EvalContext/LeafFactory seam is exercised, the fake ClassifierServices is callable, the engine is constructible, and fixtures satisfy the locked cross-field invariants (default_model ∈ candidates; every route_to ∈ candidates; classifier refs resolve).
  • test/cpp/fake_classifier_services.h: configurable fixed embeddings / scores / replies.
  • test/test_routing_fixtures.py: fixtures validate against the JSON Schemas + cross-field invariants (jsonschema, added to test/requirements.txt).
  • test/test_schema_lock.py: the frozen-major snapshot guard.

Acceptance (#2407) — all met

Verification

  • C++: compiles clean at /W4 (MSVC, VS2026); builds in-tree via CMake; ctest -R RoutingPolicyContract passes.
  • Python: test_routing_fixtures.py and test_schema_lock.py pass; Black-clean.
  • CI: these now run on every PR — RoutingPolicyContractTest is built + run in the C++ unit-test steps, and a fast server-less Routing Schema Tests lane runs the two Python guards (so the schema lock + fixture validation are enforced, not decorative).

🤖 Generated with Claude Code

@ramkrishna2910 ramkrishna2910 added this to the Lemonade Router milestone Jun 24, 2026
@github-actions github-actions Bot added enhancement New feature or request cpp labels Jun 24, 2026
@ramkrishna2910 ramkrishna2910 force-pushed the router/2407-interfaces-fixtures branch 3 times, most recently from f6030ca to 2b99fb6 Compare June 24, 2026 23:30
@ramkrishna2910 ramkrishna2910 force-pushed the router/2407-interfaces-fixtures branch from 2b99fb6 to 371aa92 Compare June 25, 2026 17:52
Land the shared C++ contract surface + schema fixtures the rest of the
routing engine codes against — the first development step after schema
sign-off, gating the parallel engine/wiring tracks.

- routing_policy.h: RouteContext, Score (label->score), ClassifierServices
  (embed/run_classifier/chat injection seam), Classifier, MatchExpr AST,
  Condition + EvalContext + LeafFactory (the runtime-evaluation seam shared by
  the evaluator and the registry), Rule, Decision/TraceEntry, RoutePolicy,
  RoutingPolicyEngine ctor. Std + nlohmann/json only; no Router/backend include.
  route() declared, not defined.
- src/cpp/resources/schemas: frozen route_policy + decision JSON Schemas + README.
  Both carry a required root `version` ("1") and pin engine-owned v1 semantics
  (keywords=substring, regex=ECMAScript, inclusive bands w/ default 0.5,
  min/max_chars=UTF-8 bytes, on_error default match_false, router desugaring).
  README documents the back-compat contract (never redefine a shipped field;
  never delete a major's schema; migrate-on-load).
- Mechanical enforcement: schema-lock.json + test/test_schema_lock.py
  (canonical-hash snapshot guard); test/test_routing_fixtures.py (schemas
  self-valid + fixtures conformant).
- test/cpp/fixtures/routing: lean L0a-L3 collection.router examples + decision.
- fake_classifier_services.h + test_routing_policy_contract.cpp (CTest
  RoutingPolicyContractTest): header compiles standalone, types construct,
  Condition/EvalContext/LeafFactory seam exercised, fake services callable,
  engine constructible, fixtures satisfy locked invariants.

Comments describe concepts, not tracker IDs; the issue linkage lives in this
commit and the PR. Depends on schema sign-off; landing as draft until then.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ramkrishna2910 ramkrishna2910 force-pushed the router/2407-interfaces-fixtures branch from 371aa92 to d5f9772 Compare June 25, 2026 19:11
@ramkrishna2910 ramkrishna2910 changed the title [Router] Interfaces + fixtures stub (foundation) (#2407) [Router] Interfaces + fixtures stub (foundation) Jun 25, 2026
@ramkrishna2910 ramkrishna2910 self-assigned this Jun 25, 2026
@ramkrishna2910 ramkrishna2910 marked this pull request as ready for review June 25, 2026 19:22
ramkrishna2910 and others added 2 commits June 25, 2026 12:43
The routing contract test and schema guards were added but nothing in CI ran
them — making the schema-lock and fixture validation decorative. Wire them in:

- cpp_server_build_test_release.yml: build test_routing_policy_contract and add
  RoutingPolicyContract to the ctest filter (both the Debian and macOS C++ unit
  test steps).
- routing_schema_tests.yml: new fast, server-less PR lane that runs
  test/test_routing_fixtures.py (fixtures validate against the schemas) and
  test/test_schema_lock.py (frozen-major snapshot guard).

Verified in-tree: cmake builds the target, ctest -R RoutingPolicyContract passes,
and both python tests pass as invoked.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`candidates` was overloaded: routing.candidates is the routing-target set (a
core concept), while the semantic_similarity classifier reused the same word for
its exemplar phrases — visibly ambiguous in the L2 fixture, which had `candidates`
meaning two different things. Rename the classifier field to `reference_phrases`
(per review on #2375); routing.candidates is unchanged. Lock refreshed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment thread src/cpp/resources/schemas/route_policy.schema.json Outdated
Comment thread src/cpp/resources/schemas/route_policy.schema.json
Comment thread src/cpp/resources/schemas/README.md
Comment thread src/cpp/resources/schemas/README.md

@eddierichter-amd eddierichter-amd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some small nits. Otherwise looks good.

Comment thread src/cpp/resources/schemas/decision.schema.json
Comment thread src/cpp/resources/schemas/route_policy.schema.json
Comment thread src/cpp/resources/schemas/route_policy.schema.json
SlawomirNowaczyk and others added 2 commits June 25, 2026 14:39
From @eddierichter-amd's review on #2408:

- Add request.schema.json + request_example.json fixture: covers the request-side
  extension surface from #2376 (OpenAI `metadata` with string values + optional
  `route_trace`) that was missing — completes the schema deliverable. Validates
  only the extension fields (additionalProperties: true); no version (rides on the
  stock OpenAI request).
- Constrain `min_score`/`max_score` to [0, 1] (the Score contract range).
- Classifier type-specific required fields via if/then: semantic_similarity needs
  reference_phrases, classifier needs model, llm needs model + prompt — moving
  type validation into the schema instead of leaving it to the parser.

Lock refreshed; README + both test harnesses updated (incl. a non-string-metadata
rejection test and a conditional-requirement test). Fixes a dangling-temporary
bug in the request check (nlohmann .items() on a .value() temporary).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ramkrishna2910

Copy link
Copy Markdown
Contributor Author

Thanks @eddierichter-amd — all three addressed in the latest commit:

  1. Request-extension schema — added request.schema.json + a request_example.json fixture covering the request side of [Schema][Router] Decision + trace object (pure selection) #2376: metadata (object, string values only — list values are comma-encoded) and the optional route_trace (bool). It validates only the extension fields and leaves the rest of the OpenAI body alone (additionalProperties: true), and intentionally has no version since it rides on the stock request rather than being a versioned document. Added a test that a non-string metadata value is rejected.
  2. [0,1] boundsmin_score/max_score now carry minimum: 0, maximum: 1 (the Score contract range).
  3. Type-specific required fields — classifier declarations now enforce, via if/then: semantic_similaritymodel + reference_phrases, classifiermodel, llmmodel + prompt. Type-specific validation now lives in the schema rather than deferring to the parser. (Reserved post-v1 preset types stay unconstrained for now.)

Lock refreshed, README + both test harnesses updated. Good catches — the request schema in particular was a real gap against #2376's deliverable.

Sign-off approves the design and unblocks development; it does not freeze the
schema. `released` stays false through implementation (schema still refinable
with a reviewed lock refresh) and flips to true at product release, when real
policies exist and immutability protects users.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ramkrishna2910 ramkrishna2910 enabled auto-merge June 25, 2026 22:24
@eddierichter-amd

Copy link
Copy Markdown
Contributor

@ramkrishna2910 one more question now that I am scoping out #2379. Should the classifier contract expose declared labels and default_label?

The registry/leaf factory needs this to:

  • reject condition label refs that are not in the classifier's declared labels
  • apply default_label when a classifier condition omits label

Something like:

const std::vectorstd::string& labels() const;
const std::optionalstd::string& default_label() const;

and corresponding protected constructor fields would make #2379 implementable without keeping a sidecar metadata table outside the Classifier.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cpp enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Router] Interfaces + fixtures stub (foundation)

3 participants