Skip to content

feat(sandbox): pricing benchmark tracking provider rates over time - #453

Draft
HeyGarrison wants to merge 1 commit into
masterfrom
devin/1789758302-sandbox-pricing-bench
Draft

HeyGarrison wants to merge 1 commit into
masterfrom
devin/1789758302-sandbox-pricing-bench

Conversation

@HeyGarrison

Copy link
Copy Markdown
Collaborator

Summary

Adds a sandbox-pricing benchmark that turns provider pricing into a tracked time series, same shape as sandbox-dax: weekly run → results/sandbox-pricing/<YYYY-MM-DD>.json + latest.json, which dotcom reads off raw.githubusercontent.com (used to keep /compare honest over time).

  • benchmarks/sandbox/pricing.ts — PRICING_PROVIDERS: 26 sandbox providers, each with its public pricing page URL and extractors ({key, pattern, scale?}). Each pattern captures the literal price figure printed on the page (e.g. 0.000014 → ×3600 for per-second → per-hour). hourlyRate(rates) reduces extracted units to a comparable $/hr at the Dax spec (8 vCPU / 16 GiB); providers quoted per-core (Beam, Modal) use 4 * coreHour + 16 * gibHour.
  • benchmarks/sandbox/pricing.bench.ts — declarative bench, 1 iteration per provider, all fetched in parallel, no credentials needed. Task returns data: {hourlyRate, rates, fieldsMatched, fieldsTotal, httpStatus}; extraction misses and HTTP errors are recorded rather than thrown — a provider whose page stops matching is the signal this benchmark exists to catch.
  • pricing-legacy-results.ts — onComplete bridge writing the dated + latest.json results files.
  • .github/workflows/sandbox-pricing-benchmarks.yml — weekly Wednesday run + workflow_dispatch, commits results to master like the Dax workflow.

Verified locally with --no-ingest: 24/26 providers extract cleanly and match the verified rates on the compare page (e.g. e2b $0.6624, modal $0.9518, vercel $1.3632). Two known misses that stay recorded as misses: CodeSandbox (403 bot-block) and CreateOS (JS-rendered page).

Link to Devin session: https://app.devin.ai/sessions/6f63e57c31da447b95f7580d3324b40b
Open in Devin Desktop: https://app.devin.ai/desktop/session/6f63e57c31da447b95f7580d3324b40b?variant=devin
Requested by: @HeyGarrison

@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant