Repository navigation
feat: semantic and hybrid retrieval, with a retrieval evaluation - #4
Merged
Merged
Conversation
This was referenced Oct 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3
What changed
Retrieval
VectorStoreprotocol (add_documents/search) inrag/vector_store.py, implemented byTfidfVectorStore(unchanged behavior), the newQdrantVectorStoreand the newHybridVectorStore.RagPipelinenow depends on the protocol.Embedderprotocol with a fastembed (ONNX) implementation usingsentence-transformers/paraphrase-multilingual-MiniLM-L12-v2. No PyTorch or TensorFlow.HybridVectorStorefuses the lexical and semantic rankings with Reciprocal Rank Fusion (rag/fusion.py).RETRIEVAL_MODE=tfidf|semantic|hybrid, defaulthybrid, documented in.env.example.tfidfnever builds an embedder, so it needs no model download.docker-compose.ymlandQDRANT_URLgive a real server.App
app.mainnever downloads a model or touches a database.create_app(pipeline=...)still accepts an injected pipeline./documentsand/querykeep their routes and response shapes. Version bumped to 1.1.0.Evaluation
docs_sample/grows from 4 to 18 original notes.evaluation/golden_set.json: 40 labeled questions in three kinds (lexical overlap, paraphrase, exact term).python -m evaluation.runreports hit rate@3 and MRR for every mode, overall and per question kind.--enforce-gate).Results
Printed by the
retrieval-evaluationjob (golden set v1,sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2):Semantic retrieval alone is worse than TF-IDF on exact terms, and hybrid misses one paraphrase that semantic alone answers. The README says so and explains the sample-size caveat.
Existing tests that changed
Only
tests/test_pipeline.pyandtests/test_api.py, and only in how the pipeline is built:RagPipelineno longer defaults to a TF-IDF store, so both now passstore=TfidfVectorStore()explicitly. Every assertion is unchanged.How it was checked
Locally, on Python 3.12.3 in a clean virtualenv built from
requirements-dev.txt:ruff check .,ruff format --check .andpytest(150 tests, offline). On GitHub Actions: thetestjob (lint and tests) and theretrieval-evaluationjob, which is the only place the real embedding model runs.Unrelated fix included in the README: the run command now passes
--env-file .envto uvicorn, since nothing in the app loads.envon its own.