perf(snowflake): composite_summary as one self-contained SQL statement

[vigneshnarayanaswamy]

What

SnowflakeLedgerBackend.composite_summary() now computes the full composite inventory in one self-contained SQL statement (CTE-based event replay over the backend's own MODELS + SNAPSHOTS tables).

Previously the method delegated to a V_COMPOSITES view that the backend never creates — an undocumented deployment dependency. Deployments without that view (i.e., any stock install) hit the SDK's sequential fallback: list_models() over the full inventory plus 2+ snapshot queries and per-edge model lookups for every composite.

Semantics — exact replication of the SDK fallback

The statement replicates Ledger.composite_summary()'s fallback behavior field by field:

• composites: MODELS rows with MODEL_TYPE IN (...) — pushed down to SQL, never filtered client-side
• member_count: replay of Ledger.members() — baseline = depends_on snapshots with payload.relationship = 'member_of', resolved against MODELS via payload.upstream_hash (name match first, then hash, mirroring Ledger.get); overlaid by member_added / member_removed events in timestamp order, latest op wins per (composite, member). member_added events whose member resolves to no registered model are no-ops in the replay and are dropped in SQL too.
• last_validated: MAX(timestamp) of validated snapshots on the composite
• open_observation_count: distinct observation_id values issued and never resolved — set semantics matching Ledger.open_observation_count (events without an observation_id payload key are ignored; an id issued then resolved is closed; a resolution without a matching issue is ignored)
• returned dict keys/types identical to the fallback: name, owner, tier, status, model_type, member_count, last_validated (datetime | None), open_observation_count, metadata

InMemoryLedgerBackend intentionally keeps no composite_summary — the SDK fallback stays the reference implementation, and the hasattr dispatch seam is now covered by an SDK test (test_dispatches_to_backend_composite_summary_when_available).

Parity proof (production-scale ledger: 28,848 models / 211,989 snapshots)

Run read-only against a live production deployment:

1. Old sequential fallback vs new SQL, 41 composites: forced the genuine fallback path (backend proxy hiding composite_summary) and diffed every field of every row against the new statement's output → 0 mismatching rows (41/41).
2. In-memory replay oracle vs new SQL, full type filter (2,334 rows): bulk-loaded all models + relevant snapshots into InMemoryLedgerBackend and ran the actual SDK fallback code locally, then diffed against the new SQL → 0 mismatching rows (2,334/2,334).

Performance (same production ledger)

path	time	round trips
old SDK fallback (41 composites)	92.6s	568
old view-based query (correlated subqueries per row)	2.5–23.5s depending on warehouse/cache	1
new single statement (2,334 rows)	0.3–0.6s warm; ~6s first-ever compile on a cold plan	1

Behavior note for downstream wrappers

For deployments that previously relied on the view's extra filtering: the view applied a metadata.source IS NOT NULL guard that the SDK fallback never had. This method now returns exactly what the SDK contract promises — every model whose model_type matches. Wrappers that want a narrower inventory should filter on metadata in their own layer (it's in every returned row).

Follow-up (diagnosed, not in this PR): investigate tool round trips

tools.investigate measured 10.0s / 39 round trips for a composite and 32.7s / 109 round trips for a model belonging to 3 groups. Breakdown: the dominant cost is per-edge Ledger.get() resolution inside dependencies() → members() / groups() (each edge costs a get_model_by_name + get_model pair), plus list_snapshots re-fetched for the same model up to 4 times (history(), dependencies(), members(), groups() each refetch). Fix plan:

1. resolve dependency edges with one batched IN (...) lookup (the batch_dependencies pattern that already exists) instead of per-edge get()
2. thread the already-fetched snapshot list from history() into members() / groups()

Estimated ~39 → ~6 round trips without semantic change. Left out of this PR because it touches the Ledger call graph rather than adding a backend method.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8fee072d93

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Count unresolved observations despite COUNT_IF NULLs

For an observation ID that has only observation_issued rows, Snowflake documents COUNT_IF(EVENT_TYPE = 'observation_resolved') as returning NULL when no rows satisfy the condition, so the = 0 predicate is not true. That drops every genuinely open observation from open_obs, causing open_observation_count to come back as 0 for the common case of issued-but-not-resolved observations; coalesce the count or use a SUM/IFF expression before comparing to zero.

Useful? React with 👍 / 👎.