Qwen3-0.6B requires .float() fix but leads to very low naive recall (0.35)

[lawen5]

Issue: Low Recall Score in Vector-Graph RAG Reproduction with Qwen3-0.6B After Fixing Embedding Type Error

Description

When reproducing the vector-graph RAG pipeline with the Qwen3-0.6B model, I encountered a dtype-related error in the embedding output. The pipeline would crash at the line:

return embeddings.cpu().numpy()


---
```markdown
## Issue：使用 Qwen3-0.6B 复现 Vector-Graph RAG 时，修改 Embedding 类型后召回分数极低

### 问题描述
在使用 `Qwen3-0.6B` 模型复现 `vector-graph` RAG 流程时，嵌入向量转换环节会报错，原代码为：
```python
return embeddings.cpu().numpy()

<img width="418" height="438" alt="Image" src="https://github.com/user-attachments/assets/c340dcec-465c-4102-9639-4a90d37d8194" />

Graph RAG recall@10：0.5788   ； Naive RAG recall@10：0.3502;

[zc277584121]

Thanks for reporting this.

The .float() change before converting embeddings to NumPy makes sense for models that output bfloat16, and we should make the HuggingFace embedding backend handle this more robustly.

However, the low score does not seem to be caused by the Graph RAG strategy itself. In your result, Naive RAG recall@10 is also quite low (0.3502), while Graph RAG improves it to 0.5788. That suggests the main issue may be in the embedding setup or evaluation configuration rather than the graph retrieval logic.

Could you share the exact reproduction command and model name you used? In particular:

• whether the model is Qwen/Qwen3-Embedding-0.6B rather than the base Qwen3-0.6B model
• which dataset and sample size were used
• whether the index was rebuilt with --force-reindex
• whether any embedding instruction / instruction template was passed

Qwen-style embedding models may require model-specific pooling and query/document formatting. If those are not aligned, both Naive RAG and Graph RAG recall can be lower than expected.

[lawen5]

Thank you for your reply: The model used is Qwen3-Embedding-0.6B, and the dataset is musique with a sample size of 1000. The --force-reindex parameter was used. I noticed the code uses an embedding instruction template by default. 一天574 ***@***.***

[lawen5]

Another question: what embedding model did you use when running the evaluation? I'm using BAAI/bge-large-en-v1.5 on the MuSique dataset, and only got a Recall@10 score of 73. However, in your results, the Recall@5 already reached 73. 一天574 ***@***.*** 原始邮件 发件人：Cheney Zhang ***@***.***> 发件时间：2026年4月29日 08:43 收件人：zilliztech/vector-graph-rag ***@***.***> 抄送：lawen5 ***@***.***>, Author ***@***.***> 主题：Re: [zilliztech/vector-graph-rag] Qwen3-0.6B requires .float() fix but leads to very low naive recall (0.35) (Issue #27) zc277584121 left a comment (zilliztech/vector-graph-rag#27) Thanks for reporting this. The .float() change before converting embeddings to NumPy makes sense for models that output bfloat16, and we should make the HuggingFace embedding backend handle this more robustly. However, the low score does not seem to be caused by the Graph RAG strategy itself. In your result, Naive RAG ***@***.*** is also quite low (0.3502), while Graph RAG improves it to 0.5788. That suggests the main issue may be in the embedding setup or evaluation configuration rather than the graph retrieval logic. Could you share the exact reproduction command and model name you used? In particular: whether the model is Qwen/Qwen3-Embedding-0.6B rather than the base Qwen3-0.6B model which dataset and sample size were used whether the index was rebuilt with --force-reindex whether any embedding instruction / instruction template was passed Qwen-style embedding models may require model-specific pooling and query/document formatting. If those are not aligned, both Naive RAG and Graph RAG recall can be lower than expected. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you authored the thread.

[zc277584121]

For the reported MuSiQue number, the evaluation setup is documented here:

• README result table: https://github.com/zilliztech/vector-graph-rag/blob/main/README.md#evaluation-results
• Detailed evaluation page: https://github.com/zilliztech/vector-graph-rag/blob/main/docs/evaluation.md#results
• Exact MuSiQue Graph RAG reproduction command: https://github.com/zilliztech/vector-graph-rag/blob/main/evaluation/README.md#graph-rag-evaluation

The MuSiQue result in the README is Recall@5 = 73.0 for Vector Graph RAG. The reproduction command uses:

• dataset: musique
• method: graph
• max samples: 300
• entity top-k: 20
• relation top-k: 20
• reranker: gpt-5-mini
• embedding model: BAAI/bge-large-en-v1.5

So if your run used 1000 samples and you are comparing Recall@10, it is not exactly the same setting as the README number. Could you share the exact command and the final output JSON/log? That would make it easier to compare the results under the same setup.

[lawen5]

Sorry, it was my mistake. I originally used the default `gpt-4o-mini` configuration in the code. Yesterday I switched to `gpt-5-mini` for retesting, and the scores are now normal. I have one more question: does the official evaluation only use 300 samples? I notice the MuSiQue dataset has a total of 1000 samples, and I achieved around 74% score when running the full 1000-sample evaluation. Below is my running command: ```bash python evaluation/evaluate.py --dataset musique --force-reindex --embedding-model bge-large-en-v1.5 --milvus-uri ./eval_bge_musique_graph.db --entity-top-k 20 --relation-top-k 20 --no-llm-cache --llm-model gpt-5-mini 一天574 ***@***.***

[zc277584121]

Thanks for confirming.

Yes, the public README number was produced with the documented reproduction command, which uses --max-samples 300. Running the full 1000 MuSiQue samples and getting around 74% is consistent with that result.

Also, for reproducibility, I would recommend using the full HuggingFace model name in the command: BAAI/bge-large-en-v1.5. The earlier low score was most likely due to the reranker configuration difference rather than the graph retrieval method itself.