Conversation
Comment out both provider entries in benchmarks/sandbox/providers.ts so the benchmarks-daily matrix builder (which imports the providers array directly) stops generating jobs for them, and comment opencomputer out of the sandbox workflow matrices to match lightning's existing treatment. Co-Authored-By: Noah Kiser <noah@computesdk.com>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
Contributor License AgreementAll contributors are covered by a CLA. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Devin Review found 1 potential issue.
1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
| // { | ||
| // name: 'opencomputer', | ||
| // requiredEnvVars: ['OPENCOMPUTER_API_KEY', 'OPENCOMPUTER_API_URL'], | ||
| // createCompute: () => opencomputer({ | ||
| // apiKey: process.env.OPENCOMPUTER_API_KEY!, | ||
| // apiUrl: process.env.OPENCOMPUTER_API_URL!, | ||
| // }), | ||
| // sandboxOptions: { timeout: 600_000 }, | ||
| // }, |
There was a problem hiding this comment.
🟡 OpenComputer benchmark command always fails
Running pnpm bench:opencomputer selects the removed opencomputer participant. resolveParticipants finds no participant and aborts the command.
Learn more
The package command remains exposed at bench:opencomputer, but tti.bench.ts now receives a participant list without opencomputer. Explicit participant selection produces an empty list, and resolveParticipants throws NoAvailableParticipantsError before executing tasks.
Example: A contributor runs pnpm bench:opencomputer using the repository's named command. The runner receives --provider opencomputer, selects zero participants, and exits instead of benchmarking OpenComputer.
Recommended fix: Remove or disable the bench:opencomputer package script alongside this provider entry. Restore both together when OpenComputer is re-enabled.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Good catch, though this matches an existing pattern in the repo: bench:railway (and bench:render, bench:just-bash) still exist even though those providers are commented out of providers.ts, so a bench:* script pointing at a disabled provider already fails the same way today.
Keeping bench:opencomputer makes re-enabling a single-file uncomment; happy to remove it if we want the disable to be more permanent.
1 participant
Summary
PR #414 commented
lightningout of the sandbox workflow matrices, but benchmarks-daily builds its own matrix by importing theprovidersarray frombenchmarks/sandbox/providers.ts— so it kept generatingsandboxanddaxjobs for lightning.This PR comments out the
lightningandopencomputerentries (and their imports) inproviders.ts, which stops benchmarks-daily from emitting jobs for them, and comments- opencomputerout of the three sandbox workflow matrices (sandbox-tti-benchmarks.yml,sandbox-dax-benchmarks.yml,sandbox-capabilities.yml) to match lightning's existing treatment — otherwise those matrix jobs would dispatch and fail on an unknown provider.Left in place for easy re-enable: vault secret loading for both providers' env vars, the
opencomputerkey inDAX_RESOURCE_OPTIONS, the lightning sizing note indax.bench.ts, and theOpenComputerdisplay-name mapping ingenerate-svg.ts.benchmarks/scale/providers.ts(a separate list for the scale benchmark) is untouched.Verified with
pnpm typecheck.Link to Devin session: https://app.devin.ai/sessions/e6251273110540a099ae686989b05e20
Open in Devin Desktop: https://app.devin.ai/desktop/session/e6251273110540a099ae686989b05e20?variant=devin
Requested by: @kisernl