osdc/hf-cache: self-heal rclone mount across a dead FUSE#876
Merged
Conversation
huydhn
force-pushed
the
hf-cache-selfheal
branch
from
July 17, 2026 17:26
9997305 to
fad219c
Compare
huydhn
marked this pull request as ready for review
July 17, 2026 17:57
jeanschmidt
reviewed
Jul 17, 2026
jeanschmidt
reviewed
Jul 17, 2026
jeanschmidt
approved these changes
Jul 17, 2026
jeanschmidt
approved these changes
Jul 17, 2026
huydhn
force-pushed
the
hf-cache-selfheal
branch
from
July 18, 2026 06:54
fad219c to
eeefb92
Compare
huydhn
added a commit
to huydhn/pytorch-ci-infra
that referenced
this pull request
Jul 18, 2026
…to 640Mi CPU-tier (r7i/m7i) rclone OOMs persist at buffer 0 AND at 512Mi (both clusters OOM'd again at ~14:00 and ~18:00 under load), because the OOM is rclone heap/vfs-item/ dir-cache overhead, not read-ahead. Raise CPU catch-all 256->640Mi and gpu1 512->640Mi (GOMEMLIMIT ~576MiB). 640Mi is the safe ceiling: 1Gi strands the tight nodes (c7i.metal-24xl/r7a.12xlarge for CPU, a10g/l4/t4 8xl for gpu1) per analyze_node_utilization; at 640Mi all fit (c7i.metal 328Mi / r7a.12xl 331Mi / g4dn.8xl 168Mi free). Residual peak-load OOMs still possible -> pair with pytorch#876 self-heal.
When rclone dies (e.g. OOM), the FUSE at /mnt/hf_cache goes stale
("transport endpoint is not connected") and containerd can't stat the
volumeMount source, so the rclone container can't restart and every job pod
on the node wedges in CreateContainerError until a manual unmount.
rclone now Bidirectional-mounts the stable parent /mnt instead of
/mnt/hf_cache, so containerd always stats a live dir and the mount self-heals
on restart; the init container also clears any dead FUSE on pod recreation.
Use fusermount3 (not fusermount) everywhere: the rclone image and the AL2023
host both ship only the FUSE3 binary, so the old 'fusermount -uz' cleanups
silently no-op'd (command not found) and never cleared a stale mount.
huydhn
added a commit
to huydhn/pytorch-ci-infra
that referenced
this pull request
Jul 18, 2026
…h#917) Makes rclone `--buffer-size` a **per-tier** value in `deploy.sh` instead of one global flag. Read-ahead RAM scales as (concurrent open shards) × buffer-size, and peak concurrency scales with a node's **pod density**, so the tight small tiers need less read-ahead: - **CPU catch-all (256Mi)** — densely packed (many runner pods per node share one mount) → `--buffer-size 0` (read-ahead off; served from the `--vfs-cache-mode full` on-disk cache). - **1-GPU tier (512Mi)** → `--buffer-size 2M` (a single HF model opening ~100+ shards at 4M overruns the 512Mi reserve; observed OOMs on g5.16xlarge gpu1 nodes). - **multi-GPU tiers (1–4Gi)** → `--buffer-size 4M` (roomier reserve, keep more prefetch). Addresses the OOM driver on both small tiers via buffer sizing. Note this is buffer-size only; the deeper fix (mount reserve sized to pod density) can follow, and pytorch#876 adds self-heal so any residual OOM recovers instead of wedging.
huydhn
force-pushed
the
hf-cache-selfheal
branch
2 times, most recently
from
July 18, 2026 23:12
481a92c to
2214fc0
Compare
huydhn
enabled auto-merge
July 22, 2026 06:36
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When rclone dies (e.g. OOM), the FUSE at
/mnt/hf_cachegoes stale ("transport endpoint is not connected") and containerd can't stat the volumeMount source, so the rclone container can't restart and every job pod on the node wedges inCreateContainerErroruntil a manual unmount.rclone now Bidirectional-mounts the stable parent
/mntinstead of/mnt/hf_cache, so containerd always stats a live dir and the mount self-heals on restart. The init container also clears any dead FUSE on pod recreation.Updated 2026-07-17:
fusermounttofusermount3: the rclone image (rclone/rclone:1.69.1) and the AL2023 host both ship only the FUSE3 binary, so the previousfusermount -uzcleanups silently no-op'd (command not found) and never cleared a stale mount — which is exactly why mounts wedged. Verified in-cluster:command -v fusermountis empty,/usr/bin/fusermount3is present./mnt) — needs a staging soak before prod.