[Router] Milestone tracker — Lemonade Router (generic core)

[ramkrishna2910]

North star: ship a generic router for all customers; the Trust Router is the first customer. The engine does pure model selection (boolean rules + classifiers, first-match) and emits a decision trace. Everything trust-specific — verdicts, block, audit persistence, consent — is expressed on top via the per-rule outputs pass-through bag, refusal-routing, request metadata, and client-side trace logging. No trust code in the engine.

Policy lives in a collection.router collection JSON (sibling to collection.omni); pointing the model field at the collection name triggers the engine — no /v1/route, no "auto". Vocabulary mirrors vLLM Semantic Router for migration.

📋 Full Decision Log is maintained in the comments on this issue.

Approval gate — schemas ✅ SIGNED OFF

• [Schema][Router] route_policy — routing block in collection.router JSON #2375 — route_policy (candidates / classifiers / rules / route_to / default_model / routing.router) — Q1/Q2/Q3 resolved; signed off (PR [Router] Interfaces + fixtures stub (foundation) #2408 approved by @SlawomirNowaczyk + @eddierichter-amd; reference_phrases rename, metadata condition, [0,1] bounds, type-specific requirements incorporated)
• [Schema][Router] Decision + trace object (pure selection) #2376 — Decision + trace (pure selection; metadata inputs; route_trace opt-in) — resolved; signed off (request-side metadata/route_trace schema added per review)

🎯 Priority 0 — first demo slice (priority:p0)

The minimum spine to demo routing from main before the milestone is complete: a collection.router policy with deterministic rules (keywords / regex / metadata / length) routes a request to different candidate models. No model-backed classifiers, no backend wiring — pure CPU. 8 of the milestone's issues; critical path is 5 sequential levels deep.

• [Router] Interfaces + fixtures stub (foundation) #2407 — foundation (interfaces + fixtures) — in review (PR [Router] Interfaces + fixtures stub (foundation) #2408)
• [Router] Core types + match-expression evaluator (M6/M7) #2378 — match-expression evaluator
• [Router] Classifier + Condition registry (M3) #2379 — classifier + condition registry (leaf factory builds the deterministic rules)
• [Router] Deterministic conditions: keywords / regex / min_chars / has_* (M4) #2380 — deterministic conditions (keywords / regex / metadata / min_chars)
• [Router] RoutingPolicyEngine — assembly + evaluate() (M8) #2382 — RoutingPolicyEngine route()
• [Router] Policy parser + hot-reload + import allowlist (M9) #2383 — policy parser (load collection.router)
• [Router] collection.router recipe + dispatch #2385 — collection.router recipe + dispatch (intercept request → route → serve)
• [Router] Expose decision trace on response (additive) #2386 — expose the decision/trace on the response (so the demo can show why it routed)

Demo critical path: #2407 → [#2378 ‖ #2379] → #2380 → #2382 → [#2383 ‖ #2385] → #2386
Not on the demo path: #2381 (semantic), #2405 (llm router), #2384 (Router wiring), #2388 (docs/e2e), #2406 (UI), #2425 (corpus) — all additive after the first demo.

Foundation (first dev step after sign-off)

• [Router] Interfaces + fixtures stub (foundation) #2407 — interfaces + fixtures stub (shared headers + schema fixtures) — gates [Router] Core types + match-expression evaluator (M6/M7) #2378/[Router] Classifier + Condition registry (M3) #2379/[Router] ClassifierServices to Router wiring (M-svc) #2384 and trust [Router] AuditSink — CloudEvents JSONL (M10) #2387; unlocks the parallel tracks

Engine core (pure, fake services)

• [Router] Core types + match-expression evaluator (M6/M7) #2378 — types + match-expr evaluator (M6/M7; any/all/not)
• [Router] Classifier + Condition registry (M3) #2379 — Classifier + Condition registry (M3; score/label/band/on_error)
• [Router] Deterministic conditions: keywords / regex / min_chars / has_* (M4) #2380 — deterministic conditions (M4; keywords_*/regex/min_chars/has_*)
• [Router] semantic_similarity classifier (M5) #2381 — semantic_similarity classifier (M5)
• [Router] llm router classifier + routing.router desugaring (L0a) #2405 — llm router classifier + routing.router desugaring (L0a)
• [Router] RoutingPolicyEngine — assembly + evaluate() (M8) #2382 — RoutingPolicyEngine (M8; first-match, fail-open to default_model)

Policy loading

• [Router] Policy parser + hot-reload + import allowlist (M9) #2383 — parser + hot-reload + allowlist (M9)
• [Router] ClassifierServices to Router wiring (M-svc) #2384 — ClassifierServices ↔ Router (M·svc; embed + run_classifier + chat for L0a)

Dispatch

• [Router] collection.router recipe + dispatch #2385 — collection.router recipe + dispatch
• [Router] Expose decision trace on response (additive) #2386 — expose decision trace on response (additive)

Exit

• [Router] End-to-end tests + docs #2388 — end-to-end tests + docs (covers L0a + classifier + rule paths)

Optional / non-gating

• [Router][UI] Author collection.router in the custom-collection panel #2406 — author collection.router in the desktop custom-collection panel (UI-owned per AGENTS.md; not on the engine critical path — hand-authoring + CLI import work without it)
• [Router] Back-compat conformance corpus (golden policy → Decision tests) #2425 — back-compat conformance corpus (golden policy → Decision tests) — guard rail, not a build step: mechanically enforces "never break an older-major policy". Best filled once #2382 produces Decisions; corpus fixtures can be authored earlier.

Dependency graph

{#2375,#2376} → #2407 → #2378,#2379 → {#2380,#2381} → #2382 → {#2383,#2384} → {#2385, #2405} → #2386 → #2388

(#2407 interfaces gates #2378/#2379/#2384. #2405 depends on #2379/#2383/#2384 and feeds the #2388 e2e.)

Deferred → Trust Router (first customer) milestone

• [Schema][Router] Audit record — CloudEvents JSONL (route_audit.jsonl) #2377 — audit record (CloudEvents) — sink format
• [Router] AuditSink — CloudEvents JSONL (M10) #2387 — AuditSink — persists the trace
• Verdict / warn / block semantics (refusal-routing + outputs bag interpretation)
• Also out: L0(b) NL→policy compiler · L4 code hooks · vLLM-SR config.yaml→routing migration tool · token-based length.

Exit criterion: a collection.router collection, driven by a vanilla openai client, routes a PII-flagged query and a support query to different candidates and everything else to default_model, with the per-condition trace returned on the response.

[ramkrishna2910]

Decision Log — Lemonade Router (generic core) + Trust customer

Consolidated record of every design decision taken while scoping this milestone, so docs (#2388) draw from one source. Canonical issues in parentheses.

North star

• Generic router for all customers; the Trust Router is the first customer. The engine does pure model selection; everything trust-specific is expressed on top, not in the kernel.

Engine / evaluation

• Boolean rules, ordered, first-match-wins; operators any / all / not (+ nested AST; vLLM-SR's flat single-operator form is the subset). ([Router] Core types + match-expression evaluator (M6/M7) #2378)
• Short-circuit + per-request memoization keyed on classifier+input. ([Router] Core types + match-expression evaluator (M6/M7) #2378)
• Engine is const / thread-safe; fail-open to default_model; per-classifier on_error: match_true|match_false is the failure primitive (fail-closed = a trust authoring choice). ([Router] RoutingPolicyEngine — assembly + evaluate() (M8) #2382)

Classifiers & conditions (adopted from vLLM SR vocabulary)

• Classifier = runs a model → score 0–1 (+ optional label) + on_error. Classifier condition = references a classifier + a min_score/max_score band. ([Router] Classifier + Condition registry (M3) #2379)
• Deterministic conditions: keywords_any/keywords_all, regex, min_chars/max_chars (char-based — no tokenizer), has_tools, has_images. ([Router] Deterministic conditions: keywords / regex / min_chars / has_* (M4) #2380)
• semantic_similarity is the first model-backed classifier (cosine vs candidates → score), embed loops through Router::embeddings. ([Router] semantic_similarity classifier (M5) #2381, [Router] ClassifierServices to Router wiring (M-svc) #2384)
• Classifier infra is core (was deferred); model training stays out of scope. L3 classifiers (pii/jailbreak/domain/language) are pluggable types.

Config & invocation

• Policy lives embedded in a collection.router collection JSON (sibling recipe to collection.omni), distributed/pulled like an LMX model. The routing block = { default_model, classifiers, rules }. ([Schema][Router] route_policy — routing block in collection.router JSON #2375, [Router] Policy parser + hot-reload + import allowlist (M9) #2383)
• Components are the candidates AND the classifier models (Q-A option 1): candidates = components referenced by route_to/default_model; classifier models = components referenced by classifiers[].model. One self-contained, distributable unit.
• Invocation: point the model field at the collection name; recipe: collection.router flips the engine — no /v1/route, no "auto" (same mechanism as omni). ([Router] collection.router recipe + dispatch #2385)
• default_model (Q-B) names the fail-open component. ([Schema][Router] route_policy — routing block in collection.router JSON #2375)
• route_to names a candidate component; the local/private/cloud route category is derived from that component's recipe — no separate routes-map table.
• routing must be added to the import/export allowlists (kKnownKeys, EXPORT_KNOWN_KEYS) or it's stripped (same family as Move omni collection system prompt from app-global to per-model JSON definition #2367). ([Router] Policy parser + hot-reload + import allowlist (M9) #2383)

Decision output & API surface (#2376, locked)

• Output = route_to + matched_rule + default_used + pass-through outputs + trace. No verdict in core.
• Q1 transport: routing inputs ride the standard OpenAI metadata body field (max drop-in; lists comma-encoded). Custom field is a later escape hatch.
• Q2 trace: server-side audit always full; client-facing minimal by default, full only on route_trace opt-in (trace is policy-internal/sensitive).
• Q3 header/body: header x-lemonade-route = scalar matched_rule (survives streaming, configurable off); body x_lemonade_route = route_to/matched_rule/default_used always + trace[] on opt-in. Selected model already visible via standard model field.

Trust customer (Q-C) — no engine code

• verdict/warn = a verdict key in a rule's outputs bag, copied verbatim into the decision; the client interprets it.
• block = a policy routing route_to a refusal/guard component.
• Trust = an authored collection.router policy + client conventions + (trust-milestone) an audit sink. Three core seams: request metadata, the per-rule outputs bag, the decision trace.

Audit (deferred to Trust milestone)

• Core only emits the trace; persistence is the customer's. Sink format = CloudEvents (CNCF) JSONL, payload per OpenTelemetry GenAI semantic conventions, input hashed by default. ([Schema][Router] Audit record — CloudEvents JSONL (route_audit.jsonl) #2377, [Router] AuditSink — CloudEvents JSONL (M10) #2387)

Deferred (explicitly out of this milestone)

• Mode A /v1/route decision-only endpoint · model:"auto" · verdict/block semantics + audit persistence (→ Trust milestone) · L0(b) NL→policy compiler · L4 code hooks · vLLM-SR config.yaml→routing migration tool · token-based length · dashboard.

Process / facts

• Greenfield off main — no existing heuristic/rule router today (the v2 req-spec's "superset of today's heuristic routing" was aspirational). ([Router] Core types + match-expression evaluator (M6/M7) #2378/[Router] Deterministic conditions: keywords / regex / min_chars / has_* (M4) #2380/[Router] RoutingPolicyEngine — assembly + evaluate() (M8) #2382 build from scratch.)
• Recommended first dev step after schema sign-off: a small interfaces + fixtures PR (Classifier/Condition/ClassifierServices/Decision headers + schema fixtures) to unlock parallel tracks.
• Design doc (claude.ai/design — "Federated AI Trust Router design document", Trust Router Design Doc.dc.html) updated to this generic-core / trust-as-customer framing (§1–§8).

[ramkrishna2910]

Decision Log — addendum (schema gate #2375 resolved)

Classifier types in v1 (#2375 Q1)

• v1 implements three types: semantic_similarity (embeddings+cosine), classifier (generic text-classification → label/score), and llm (LLM-as-router, L0a — the on-ramp for most users, via routing.router sugar — [Router] llm router classifier + routing.router desugaring (L0a) #2405).
• vLLM-SR named types (pii_detection, prompt_safety, language_detection, domain_classification, complexity, sentiment) are reserved in the schema enum and ship as thin presets over classifier post-v1. Model training stays out of scope.

Component role marking (#2375 Q2) — mirrors vLLM SR

• vLLM SR separates routing targets (providers.models) from classifier models (a separate classifier config). Role = declaration site, not a label.
• Our adaptation: routing targets declared in routing.candidates, classifier models at classifiers[].model — both drawn from the bundled components/models[]. default_model ∈ candidates; every route_to ∈ candidates; named-by-neither → validation warning; in-both allowed.

Classifier output shape (#2375 Q3)

• classifier returns label → score ∈ [0,1]; declares optional labels + default_label; band default min_score: 0.5; scores engine-opaque. semantic_similarity is the fixed cosine-score exception (declares candidates, not labels).
• Dependency note: this constrains [Router] ClassifierServices to Router wiring (M-svc) #2384 run_classifier() — it must normalize the served model's response into {label: score} (a small per-type output adapter).

Decision + trace API surface (#2376) — locked earlier

• Q1 transport: routing inputs ride the OpenAI metadata body field (lists comma-encoded).
• Q2 trace: server-side audit always full; client-facing minimal by default, full on route_trace opt-in.
• Q3 header/body: header x-lemonade-route = scalar matched_rule; body x_lemonade_route = route_to/matched_rule/default_used always + trace[] on opt-in.

New issue

• [Router] llm router classifier + routing.router desugaring (L0a) #2405 — llm router classifier + routing.router desugaring (L0a). Stays inside the one-engine invariant by desugaring to one llm classifier + identity rules. Reuses the Router chat path; trace records the LLM's chosen candidate + rationale (so L0a is auditable). Tradeoff: least-deterministic tier — docs steer steady-state users to L0(b)/L1–L3.

Status: the schema gate (#2375 + #2376) is fully resolved and ready for stakeholder sign-off; development can start once approved (recommended first step: the interfaces + fixtures PR).

[ramkrishna2910]

Foundation issue filed: #2407 — interfaces + fixtures stub. It's the first development step after schema sign-off and gates #2378/#2379/#2384 (and the trust-milestone #2387, which codes against the trace contract). Dependency graph and the three dependents updated accordingly.

[ramkrishna2910]

Back-compat guard rail filed: #2425 — frozen conformance corpus (versioned collection.router policies + expected Decisions, replayed in CI). It mechanically enforces the milestone's hard rule — a future server must never break an older-major policy — that's currently held by discipline (the "never redefine a shipped field / never delete a major's schema" principles + the pinned v1 semantics in src/cpp/resources/schemas/README.md, both landed in #2408). Deterministic paths (keyword/regex/char rules, any/all/not, first-match, routing.router desugaring) assert the full Decision exactly; model-backed paths run against a pinned fake ClassifierServices so the test covers the engine's threshold/selection logic, not backend floats. Non-gating; best filled once #2382 implements route(). Live-backend tolerance and component-resolution (delisted model) compat are explicitly out of scope.

[ramkrishna2910]

Schema-correctness enforcement codified. "The schema stays correct" decomposes into four gates; assigning each so none is left to discipline:

1. Schema is self-valid — test/test_routing_fixtures.py meta-validates both schemas. ✅ in [Router] Interfaces + fixtures stub (foundation) #2408.
2. Fixtures conform — same test validates the L0a–L3 examples + decision example. ✅ in [Router] Interfaces + fixtures stub (foundation) #2408.
3. Schema ↔ parser parity — the keys the parser accepts must equal the keys the schema declares, driven from one shared key registry (mirrors the kKnownKeys/EXPORT_KNOWN_KEYS pattern) + a divergence test. → folded into [Router] Policy parser + hot-reload + import allowlist (M9) #2383 (which also now reads version and dispatches by major).
4. A change didn't break users (back-compat) — two parts:
   • Frozen-major snapshot guard — schema-lock.json + test/test_schema_lock.py (canonical-hash; a released major is immutable → changing it fails CI, an unreleased major may change only with a refreshed lock in the same diff). buf-breaking-lite. ✅ in [Router] Interfaces + fixtures stub (foundation) #2408. v1 is released: false until schema sign-off; flip to true at release.
   • Behavioral stability — the golden conformance corpus (old policy → expected Decision). → [Router] Back-compat conformance corpus (golden policy → Decision tests) #2425 (non-gating).

Net: gates 1, 2, and 4a are mechanical in PR #2408 today; 3 is owned by #2383; 4b by #2425. This is the standard layering used by Confluent Schema Registry (compat-on-register), Buf (buf breaking in CI), and Kubernetes (versioned conversion + round-trip tests) — adapted to our LMX-embedded, version-gated JSON format.

[ramkrishna2910]

Dev-wave alignment pass (issues re-synced to the as-built foundation). The engine-core issues were written before the #2407 header existed and before version/frozen-semantics were locked, so they had drifted. Reconciled:

• Foundation ([Router] Interfaces + fixtures stub (foundation) #2408) gained the runtime-evaluation seam — Condition + EvalContext + LeafFactory — so the evaluator ([Router] Core types + match-expression evaluator (M6/M7) #2378) and the registry ([Router] Classifier + Condition registry (M3) #2379) build against one shared contract and meet only at LeafFactory (no hidden serialization). #2407's scope had listed a Condition leaf; this delivers it.
• [Router] Core types + match-expression evaluator (M6/M7) #2378 — re-scoped from "declare the types" (they're in [Router] Interfaces + fixtures stub (foundation) #2407 now) to "implement the composite conditions + classifier-band + compile(MatchExpr, LeafFactory) + memoization." Fixed the stale Decision {…reasons…} → as-built {route_to, matched_rule, default_used, outputs, trace}; EvalState → EvalContext.
• [Router] Classifier + Condition registry (M3) #2379 — Score is a label→score map (score_of/primary), ClassifierServices now also exposes chat; owns make_classifier + the LeafFactory dispatch + ref resolution.
• [Router] Deterministic conditions: keywords / regex / min_chars / has_* (M4) #2380 — rewritten to the frozen v1 semantics: keywords = case-insensitive substring (not BM25), regex = ECMAScript, min/max_chars = UTF-8 bytes (not "character-based"), inclusive bounds.
• [Router] semantic_similarity classifier (M5) #2381 — writes the cosine into Score.labels[""] (single-score convention) + default min_score: 0.5. (Was already the closest to ready.)
• [Router] RoutingPolicyEngine — assembly + evaluate() (M8) #2382 — method is route(ctx, want_trace) (not evaluate), holds a RoutePolicy + services_, compiles rules at construction, trace only on opt-in.

Net: every issue from the foundation through the engine now references the real header types and the locked semantics, so they're pick-up-ready. #2383 already carries the version-dispatch + schema↔parser parity from the earlier pass.