Qwen3-Embedding-8B produces NaN embeddings for token 474 ("import")

[cbwchuck]

System Info

Docker image: ghcr.io/huggingface/text-embeddings-inference:cuda-1.9
Start command: --tokenization-workers=16 --dtype float16 --auto-truncate --max-client-batch-size 128
Host OS: Ubuntu

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

I've been running Qwen3-Embedding-8B with float16 dtype and noticed that any input starts with the token "import" (token ID 474), such as "importance", "import", and "important" will cause all-NaN vectors returned.

# Reproduction

curl <TEI_URL>/embed \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"inputs": "importance"}'
# Returns: [[NaN, NaN, NaN, ...]]

Expected behavior

When investigating, I found people having the exactly same issue -> https://huggingface.co/Qwen/Qwen3-Embedding-8B/discussions/21, and padding the word with a leading space does seem to mitigate it.

I tried checking out tag v1.9.2 and traced through the model layer by layer and found that the NaN originates from
an FP16 overflow in the MLP layers. Here's the chain of events:

1. RMSNorm normalizes hidden states to ~1.0 — perfectly safe in F16
2. Attention runs fine on these normalized values
3. The MLP's down_proj output, however, reaches values around ~2.95 million for this token
4. F16 can only represent values up to ~65504, so this overflows to Inf
5. The residual add (Inf + finite) stays Inf
6. The next layer's RMSNorm receives Inf and produces NaN
7. NaN propagates through every remaining layer

The overflow first appears at layer 2 and corrupts the entire output from that point on.

Not sure whether it's reasonable, my hypothesis is that Qwen3-Embedding-8B was trained in BF16 -> The MLP weights learned during BF16 training produce activations exceed what F16 range so precision mismatch error occurred.

[Kyogre-Primal]

I have come across similar discussions regarding this issue on the Hugging Face model hub:

• Discussion #21
• Discussion #27

While it is a somewhat rough workaround, I have found a temporary solution that might be effective. If your input string begins with the word import, prepending a single space to the string (e.g., changing it to import) seems to successfully bypass the tokenizer error.

Here is a quick example:

if query.startswith('import'):
    query = ' ' + query

I hope this serves as a helpful temporary fix until the underlying issue is officially resolved.

[alvarobartt]

Hey @cbwchuck and @Kyogre-Primal thanks for sharing! This indeed seems to be related to the hidden states exceeding the maximum value that FP16 can represent, due to the model being trained with BF16. We saw similar issues with both Gemma3 where FP16 is disabled as the same issue will arise, same for Perplexity AI latest models. Given that this is an issue with the weights due to those being trained on BF16, the only "fix" we can do on Text Embeddings Inference is to add native support for BF16, which we're already doing as per bumping to the latest candle version, so expect that to land on v1.10.0, meaning that the models with torch_dtype or dtype set to bfloat16 will run inference normally, BF16 on both Metal and CUDA, and FP32 on CPU (as BF16 not supported there). I'll keep you posted, but might be worth having a look at earlier efforts at #809 🤗