[Community] Provision GPU capacity for Community CI

[yifeif-nv]

Summary

Provision secure GPU capacity that can run qualification workloads for community pull requests without depending on manually available internal machines.

The solution must preserve a strict trust boundary: untrusted pull-request code must not gain access to repository secrets, persistent runner credentials, private infrastructure, or artifacts from other jobs.

Tasks

• Define the initial GPU, operating-system, architecture, TensorRT, CUDA, and driver matrix.
• Select the hosting model and establish an operating budget.
• Configure isolated runners with least-privilege access and reproducible environments.
• Define authorization rules for running community-submitted code on GPU infrastructure.
• Ensure jobs cannot access long-lived credentials, unrelated workloads, or residual data from previous jobs.
• Connect the runners to the Community CI workflow.
• Add at least one representative GPU qualification job.
• Pin each run to the exact pull-request revision being evaluated.
• Add public queue visibility, queue-time reporting, timeout handling, cancellation, and actionable failure reporting.
• Publish sanitized qualification results without exposing private infrastructure details.
• Monitor utilization, cost, queue time, reliability, and available capacity.
• Document capacity limits and the process for expanding the initial matrix.
• Document ownership for runner maintenance, security updates, credential rotation, and incident response.

Acceptance criteria

An authorized community pull request can trigger at least one representative GPU qualification job and receive public, actionable results for its exact revision without manual machine provisioning.

The qualification environment is isolated, least-privilege, reproducible, observable, and covered by documented maintenance and incident-response ownership.

[husizhtu]

Current capacity evidence: Community GPU Dev lanes are failing at Brev reservation with ResourceExhausted ... quota vpc.pool.count (retryable) and instance state FAILURE, before any validation runs. Seen today on PR #1402 (three attempts) and PR #1406; Stable CPU lanes pass. Raising �pc.pool.count or serializing Dev GPU reservations would unblock these PRs.

[chaofengw-nv]

Hi @husizhtu sorry for the Community GPU status. We're contacting staff at Brev to address this issue. But now the Community GPU isn't blocking PR merges. We can still merge via internal ci. Sorry for the confusion caused by the failure messages