Efficiently download AI models from Hugging Face Hub with controlled download speeds, ideal for low-bandwidth networks.
Script for downloading large AI models in environments with limited bandwidth. It allows you to:
- Control download speeds to avoid network congestion.
- Resume interrupted downloads automatically.
- Download specific files selectively from model repositories.
Unlike the Hugging Face CLI download method, this script does not leverage blob storage for deduplication or hash-based integrity checks. However, this simpler approach provides greater ease of use, explicit control over download speeds, and straightforward file management for individual downloads.
- Speed Control: Prevent network congestion by limiting download speeds.
- Automatic Resume: Resume interrupted downloads seamlessly.
- File Selection: Choose specific files rather than downloading the entire repository.
Install the required packages:
pip install requests tqdmpython model_downloader.py --model MODEL_NAME --mode {slow|medium|fast} --token HF_TOKENDownload at medium speed (5MB/s):
python model_downloader.py --model deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --mode medium --token HF_TOKENDownload at custom speed (10MB/s):
python model_downloader.py --model deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --mode fast --speed 10 --token HF_TOKENDownload specific files only:
python model_downloader.py --model deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --mode slow --token HF_TOKEN --files model.safetensors config.jsonYour script downloads files directly, whereas Hugging Face Hub utilizes a blobs-based storage method.
Note: This comparison was created with the help of ChatGPT and may not be 100% accurate. Please verify details if needed.
| Aspect | This Script | Hugging Face's Native Method |
|---|---|---|
| Speed Control | ✅ Easy to set custom speeds | ❌ No speed throttling built-in |
| Resume capability | ✅ Supported | ✅ Built-in |
| Storage Efficiency | ❌ Downloads duplicates if files overlap | ✅ De-duplicates blobs storage |
| Integrity Checks | ❌ No built-in hashing/integrity checks | ✅ Hash-based file integrity |
| Ease of Use (Single Models) | ✅ Simple and minimal setup | |
| Version Management | ❌ Harder for multiple versions | ✅ Easy management of multiple versions |
| Low Bandwidth Performance | ✅ Optimal for bandwidth control | ❌ No direct control of download speed |
- Limited bandwidth environments.
- Occasional or one-off downloads.
- Selective downloading of specific files.
For extensive model management or frequent version updates, the Hugging Face native method may be more efficient.
This script, inspired by common patterns in the Python community, was developed for personal use during periods of limited bandwidth availability. It is shared as-is in the hope that it may be useful to others. However, it comes with no warranties or guarantees of any kind. I accept no responsibility for any issues, damages, or consequences arising from its use. Users are advised to review the code thoroughly before running it and to use it at their own risk.