Conversation
The agent links here when a save fails because the sandbox has the NVIDIA graphics or video driver capability enabled. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
Contributor
Author
|
Sibling PRs: agent https://github.com/CerebriumAI/cerebrium-checkpoint-agent/pull/100, gVisor https://github.com/CerebriumAI/gvisor/pull/68. Merge this first so the agent's new anchors are live. |
Vulkan/NVENC workloads do not run at all: runsc keeps the stock utility,compute allowed driver capabilities, so an app requiring the graphics or video capability fails to start or fails at driver init. State that rather than framing it as a checkpoint failure reason, and drop the two anchor-linked sections since no classifier links to them. Co-authored-by: Cursor <cursoragent@cursor.com>
Hkhan161
changed the title
Graphics and video are no longer the same story. ubuntu-ami#139 allowed compute,utility,graphics,video for NVIDIA NIM images, so video engines will run once that lands in a baked AMI, while Vulkan still fails at ICD load. Video therefore becomes a checkpoint failure cause with its own section and anchor, matching the agent's docsAnchorVideo. Co-authored-by: Cursor <cursoragent@cursor.com>
Hkhan161
changed the title
Co-authored-by: Cursor <cursoragent@cursor.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
This PR started out saying graphics and video were both unsupported. That is no longer right for video.
ubuntu-ami#139(merged Sep 1) allowedcompute,utility,graphics,videoon every GPU node so NVIDIA NIM images stop crash-looping — they shipNVIDIA_DRIVER_CAPABILITIES=compute,utility,video. So the two now need different answers:vulkaninfoon an A10G with the caps widened and all ICD libraries present; it fails at ICD load, so the capability being allowed isn't sufficient. Documented as unsupported, no checkpoint anchor.What
## Video encode and decodesection, anchoredvideo-encode-and-decodeto matchdocsAnchorVideoin checkpoint-agent PR Kats/increase-num-train-epochs #101.spawndoesn't help. That's the trap: the error people currently get tells them to stop forking, and I confirmed on hardware that a single-processffmpeg h264_nvencwith no children hits it.Merge order
Depends on gVisor #69 → agent #101 for the anchor to be reachable. Hold this until #139 is in a baked AMI — until then it would claim video runs when it doesn't.