feat(evals): add regression eval for edge function console output (mcp#375)

[Rjabov]

Adds a regression eval for supabase/mcp#375 (fix in supabase/mcp#376). @mattrossman asked for one there before the fix lands.

An order-sync edge function returns 200 on every call but logs a missing warehouse SKU with console.error, and the user asks for the exact error. The request envelopes in function_edge_logs show nothing wrong; the error text is only in function_logs.event_message, which the current query_logs hint doesn't mention. The scorer is deterministic: the answer has to quote the SKU and name one of the affected orders.

Checked with the published mcp-server-supabase 0.12.0 over stdio against platform-lite: get_logs is hidden, the hint names neither function_logs nor event_message, and the error text is only reachable through event_message. eval:dry, biome and typecheck pass. I haven't run an agent on it (no API credits), so could someone add run-evals-changed? Allow edits by maintainers is on.

[vercel]

@Rjabov is attempting to deploy a commit to the Supabase Team on Vercel.

A member of the Team first needs to authorize it.

[mattrossman]

Thanks for your patience here @Rjabov, taking a look at this now that we've tackled some other backlog items.

[mattrossman]

@Rjabov I pushed a couple commits to pull this branch up to speed with our latest shape on main, and because it's fork where our refresh workflow can't run, I pushed the new regression eval results from my 3-run tests directly.

In those runs, the agent (Codex with GPT 5.6 Luna) passed all runs already with & without the /supabase skill. In most cases it ran select distinct source from logs to figure out the list of sources before querying them, so it didn't stumble on the lack of explicit callout in the description.

If you're able to find a scenario where it stumbles, let me know! I'd prefer to start from a failing scenario before making server changes.

If we do end up tweaking the description, what I'd prefer is to have a single lookup object e.g. LOG_SOURCES that stores all the details for each log source in one place, including the MCP preset arg, human/agent-readable summary of the table's use case, ClickHouse columns, etc. to avoid making this a one-off fix for the edge function logs.

[Rjabov]

@mattrossman You were right about the discovery step. Sonnet-5 and Opus-5 pass 3/3 at every reasoning effort I tried against the pinned 0.12.0, and they name event_message in their first query without the description mentioning it, so the hint isn't what blocks them.

I think the eval was hiding a failure rather than showing there wasn't one. The prompt lets the agent ask a follow-up question instead of investigating, and Haiku-4.5 takes that exit almost every time: across 9 runs (low, medium, high effort, 3 each) exactly 1 issued a single query_logs call. That scores as a failure but says nothing about the description. I've pushed a one-sentence change saying the user is boarding a flight and can't answer follow-ups - no hint about sources or columns. With it, 5 of 9 runs investigate, and one fails the way #375 describes: 11 queries, runs select distinct source from logs, reaches source = 'function_logs', finds all three failing execution ids, then reads only log_attributes, trying ['message'], ['error'], ['text'], ['details'], and reports back "The error logs exist but contain no error message text. Error logged but no message captured."

Reproducing it needs a cheap Claude experiment, which the matrix doesn't have. I haven't added one since that's a row on your CI bill, but it's this:

import {
  claudeCodeAgent,
  defineExperiment,
  platformLiteRuntime,
  supabaseMcpServer,
} from '@supabase-evals/core';

export default defineExperiment({
  agent: claudeCodeAgent({ model: 'claude-haiku-4-5', reasoningEffort: 'low' }),
  runtime: platformLiteRuntime({ mcpServers: [supabaseMcpServer()] }),
  skills: [],
});

Run it with --runs 3 and expect roughly one failure in nine, not every run - Haiku's engagement is erratic and it still punts for unrelated reasons about half the time. Two things I can't claim yet: whether #376 actually fixes this, because it's branched from 0.11.0 two releases behind main, so an A/B against it measures the SDK v2 migration more than the one-line hint edit, and I'd rather rebase and remeasure than hand you a number I don't trust. Also the results you committed are now stale since the prompt changed.

One last thing worth knowing regardless of the fix: platform-lite's logs view projects message alongside event_message with identical text, and select * returns both, so an agent can pass this eval without ever naming the column it's testing - two of the passing runs did exactly that. Happy to tighten that and do the LOG_SOURCES consolidation you described.