[BUG-FIX]: download progress not reported to Python during file transfers

[tobocop2]

This blocks the companion PR huggingface/huggingface_hub#4059

Related issue: huggingface/huggingface_hub#4058

Problem

Download progress bars in huggingface_hub update in large jumps (e.g. 0% -> 26% -> 63% -> 100% for a 4GB file) instead of smoothly. The root cause is that ItemBridgeState::compute_diff hardcodes all total_transfer_bytes fields to 0, so the fine-grained per-HTTP-chunk progress that xet-core already tracks internally never reaches Python callbacks.

Fix

• Add transfer_bytes and transfer_bytes_completed to ItemProgressReport
• Update ItemBridgeState::compute_diff to diff transfer fields, matching the existing GroupBridgeState behavior
• Update GroupBridgeState::compute_diff item inclusion to also trigger on transfer progress

Reproduction

Single file

"""Verify smooth single-file download progress."""
import tempfile
from huggingface_hub import hf_hub_download

with tempfile.TemporaryDirectory() as tmp:
    hf_hub_download(
        "Qwen/Qwen3-4B",
        filename="model-00002-of-00003.safetensors",
        cache_dir=tmp,
        force_download=True,
    )

Multi-file

"""Verify smooth multi-file download progress."""
import tempfile
from huggingface_hub import snapshot_download

with tempfile.TemporaryDirectory() as tmp:
    snapshot_download(
        "Qwen/Qwen3-4B",
        allow_patterns=["model-00003-of-00003.safetensors", "config.json", "*.txt"],
        cache_dir=tmp,
        force_download=True,
    )

Before: tqdm bar jumps 0% -> 26% -> 63% -> 100% with ~4 callbacks for a 4GB file.
After: tqdm bar updates smoothly every 250ms.

See callback_bridge.rs tests for coverage of the new transfer byte diffing.

[hoytak]

LGTM -- makes sense.

[tobocop2]

Friendly bump on this, happy to address any feedback if needed. Let me know if there's anything blocking review.

[seanses]

@tobocop2 Hi, with the release of hf-xet 1.5.0 the legacy API are deprecated and replaced by the new APIs (see #792). Also the huggingface-hub side that your companion PR huggingface/huggingface_hub#4059 intends to fix will no longer exists as huggingface-hub switches to use hf-xet 1.5.0 in the next release. Can you retarget your fix to the new APIs if the problem persists?

[tobocop2]

no problem. I'm currently using this workaround huggingface/huggingface_hub#4129

I will try to step up to the latest hugging face hub and report back if the problem persists.

[tobocop2]

@seanses stepped up to huggingface_hub 1.16.1 / hf-xet 1.5.0. Problem persists on the download path.

Repro:

import os, shutil, time
os.environ["HF_HOME"] = "/tmp/xet_repro_cache"  # set before importing huggingface_hub
shutil.rmtree("/tmp/xet_repro_cache", ignore_errors=True)  # cold download each run

from huggingface_hub import hf_hub_download
from tqdm.auto import tqdm

REPO, FILE = "unsloth/Qwen3-4B-GGUF", "Qwen3-4B-UD-IQ1_S.gguf"  # ~1 GB, xet-backed

class Recorder(tqdm):
    def __init__(self, *a, **k):
        k["disable"] = True
        super().__init__(*a, **k)
        self._t = time.monotonic()
    def update(self, n=1):
        now = time.monotonic()
        print(f"+{int(n)/1e6:6.1f} MB  (+{now - self._t:5.1f}s gap)", flush=True)
        self._t = now
        return super().update(n)

hf_hub_download(REPO, FILE, tqdm_class=Recorder)

HF_HUB_DISABLE_XET=0 python repro.py   # xet
HF_HUB_DISABLE_XET=1 python repro.py   # http

Each line is one progress_updater callback (MB delivered, gap since previous). xet:

+   0.6 MB  (+  2.3s gap)
+   5.3 MB  (+  4.2s gap)
+ 134.1 MB  (+117.0s gap)
+ 133.3 MB  (+  4.3s gap)
+ 129.3 MB  (+ 61.2s gap)
+  14.8 MB  (+ 13.5s gap)
+  33.5 MB  (+  9.7s gap)
+  43.4 MB  (+ 22.2s gap)
+  71.9 MB  (+ 36.7s gap)
+ 104.6 MB  (+ 64.5s gap)
+  67.0 MB  (+ 46.7s gap)
...

xet delivers 100+ MB per callback with gaps up to ~2 min. HTTP (HF_HUB_DISABLE_XET=1) reports every 10 MB (DOWNLOAD_CHUNK_SIZE) at a steady cadence.

[seanses]

Apologies, the PR that updates huggingface_hub to use the new API has not landed yet. I'm aware that even using the new API the update gap can still be large, I'll discussed with the huggingface_hub maintainers to see if we can add a separate progress bar so the update gap there can be smaller, see relevant discussion here.