Hi there to all of you,
since the hf download (as well as its cli variants) looks to being not able to resume any failed download, is there any solution to avoid spending days (and money for the internet connection) unsuccessfully downloading big models?
This is a known pain point with the newer Xet-backed transfer path specifically, there’s an open issue where hf download on Xet-backed repos can exit with status 0 even when it didn’t actually finish, leaving .incomplete blobs behind and no real resume happening. Two things worth trying:
HF_HUB_DISABLE_XET=1 hf download <repo> forces the older plain HTTP range-request path instead of Xet, that one does properly resume from existing bytes on retry, just slower per-connection.
If that’s still flaky, the community hfd script (curl+aria2c wrapper, see the padeoe gist on GitHub) is more robust for big downloads, it does integrity-aware resume, re-fetching only files that are missing or the wrong size rather than restarting the whole transfer, and lets you cap thread count if your connection can’t handle many parallel streams.
all of these appear to be not a real solution.
I mean if I’ve to dowload a model like the antirez’s q2 gguf of deepseek v4.1 flash that is a 366GB file is impossible to launch the download from the files card of the model because of a “400 bad request” on the huggingface site; and if you try the ds4 download script if the connection fails I’ve to start it again from beginning.
I’m now trying after I changed the _ETAG_RETRY_TIMEOUT parameter in the file_download.py ov the python venv of hf-cli, and I hope it will work because I’m trying 3 days in a row to download a single model.
About the hfd scipt how do this work when the direct access to the download link through the files section of a model give a “400 Bad Request”?
Found it, that’s a known bug specific to very large single files (this exact issue is tracked upstream, xet-core #592): the “CAS Bridge” that serves browser/website downloads and blob/main links breaks with a 400 on files past a certain size (people are hitting it on 300-600GB GGUF quants specifically), it only affects the website/CAS-bridge path, not the Xet client itself. So that 400 isn’t fixable from your end, it’s a server-side limit on that particular delivery path.
For a file that size the reliable path is the Xet-native client, hf download <repo> --include "*.gguf" (or the specific filename) uses the Xet protocol directly rather than going through the broken CAS bridge, that’s what’s confirmed working around this bug in the linked issue. On the hfd question, it won’t help here either since it also just wraps HTTP/curl requests to the same CAS-bridge links, so it’ll hit the same 400. Stick with hf download (Xet-enabled, which is default now) rather than any browser/curl/aria2c approach for this specific file, and if it still drops mid-transfer, that’s the separate resume bug from before, worth watching xet-core#592 for a fix on the 400 itself since it’s still open.
I see but the “server side problem + poor connection that makes hf download fail on timeout for CAS check” ends up as a fatal combo right on my back…
At least if the server side error was solved I could even use and rely on JDownloader to easily resume the downlod fails because of the connection fails.
I mean I think it looks link there is something wrong in the site and/or hub architecture if just 2 miserable problems alone turn into a fatal error by the customer side when they appear togheter.
Yeah, fair, and that’s a real gap, not you doing anything wrong. Right now there isn’t a single working path for a file that size on a bad connection: Xet client is the only thing that gets past the 400, but it doesn’t reliably resume on your connection; the website link would resume fine with JDownloader, but it’s blocked by the CAS bridge bug at this size. Two separate bugs that happen to cancel out each other’s workaround, which is exactly the “architecture problem” you’re describing.
Only thing left worth trying on your end: split the download into smaller chunks via hf_hub_download with explicit byte ranges if the model repo exposes it, or check if antirez/the quantizer split the GGUF into multiple smaller part files elsewhere (some large quants get split specifically to dodge this bridge limit), that’d let you use JDownloader on each part instead of one 366GB stream. Otherwise this really is on HF to fix xet-core#592, worth adding your case to that issue thread since more reports with concrete repro (repo name, file, connection type) helps it get prioritized.
I solve it by running this command trying to escape timeout and retry limits:
export HF_HUB_ETAG_TIMEOUT=300
export HF_HUB_DOWNLOAD_TIMEOUT=600
export HF_XET_CLIENT_CONNECT_TIMEOUT=90
export HF_XET_CLIENT_READ_TIMEOUT=600
export HF_XET_CLIENT_RETRY_BASE_DELAY=5
export HF_XET_CLIENT_RETRY_MAX_ATTEMPTS=100000
export HF_XET_CLIENT_RETRY_MAX_DURATION=36000
hf download antirez/deepseek-v4.1-flash-gguf DeepSeek-V4.1-Flash-Q2.gguf
Hope it can help someone, but much more I hope someone in hf could place himself and point to really make the xet work properly.