feat: organize experiments by owner - #315
Rodriguespn wants to merge 1 commit into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
881432e to
a2edffd
Compare
| for (const entry of await readdir(ownerDir, { withFileTypes: true })) { | ||
| if (!entry.isFile() || !entry.name.endsWith(EXPERIMENT_SUFFIX)) continue; | ||
|
|
||
| files.push({ |
There was a problem hiding this comment.
In case a team ends up trying to kick off a refresh w/ experiment names that overlap across teams which would silently clash, maybe worth a small duplicate check + error surfaced here.
| skills, | ||
| } satisfies ExperimentPreset; | ||
|
|
||
| export const baselineExperiment = codexGpt56Luna; |
There was a problem hiding this comment.
What's the use case for this if codexGpt56Luna is already exported? Assuming this is scaffolding for future PR, maybe better to add once it's used for something.
There was a problem hiding this comment.
If the intention was to signal to other teams which base they should use—perhaps the other harness/model combos shouldn't be in presets.ts? Unless we know other teams want to use them. For now it seems like other teams will only be using 5.6 Luna.
| @@ -0,0 +1,8 @@ | |||
| import { defineExperiment } from '@supabase-evals/core'; | |||
| import { claudeCodeSonnet5 } from '../presets.ts'; | |||
There was a problem hiding this comment.
I think we'll want to fix the tsconfig btw, IIUC this .ts extension would normally fail the typecheck. It look like this was intended to be handled by the "experiments" includes entry in apps/framework/tsconfig.json but rn that resolves to apps/framework/experiments instead of root experiments/
| * @supabase/ai | ||
|
|
||
| /experiments/ai/ @supabase/ai | ||
| /experiments/presets.ts @supabase/ai |
There was a problem hiding this comment.
nit, these are already handled by the @supabase/ai catch-all
|
Sanity checking w/ a regression run to verify results look normal still: |
This branch has not been deployed
2 participants
This change organizes experiments under per-owner folders so CODEOWNERS can scope them by path. Experiment IDs and result names remain unchanged.
AI-owned base configs live in
experiments/presets.ts. The AI experiments extend those presets fromexperiments/ai/.How to review
Start with
experiment-files.ts, thenpresets.tsand the two discovery call sites inrun-eval.tsandexport-results.ts.Dry run one of the moved experiments using its unchanged ID:
Verification
pnpm --filter @supabase-evals/framework typecheckpnpm --filter @supabase-evals/framework test:vercel-runner(26 tests passed)pnpm eval -- listreturned the same 12 experiment IDsclaude-code-sonnet-5pnpm formatContext: Slack thread