Skip to content

feat: semantic and hybrid retrieval, with a retrieval evaluation - #4

Merged
LacerdaTraderCode merged 53 commits into
mainfrom
feat/3-semantic-retrieval
Oct 20, 2026
Merged

LacerdaTraderCode merged 53 commits into
mainfrom
feat/3-semantic-retrieval

Conversation

@LacerdaTraderCode

@LacerdaTraderCode LacerdaTraderCode commented Oct 20, 2026 •

Copy link
Copy Markdown
Owner

Closes #3

What changed

Retrieval

  • VectorStore protocol (add_documents / search) in rag/vector_store.py, implemented by TfidfVectorStore (unchanged behavior), the new QdrantVectorStore and the new HybridVectorStore. RagPipeline now depends on the protocol.
  • Embedder protocol with a fastembed (ONNX) implementation using sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2. No PyTorch or TensorFlow.
  • HybridVectorStore fuses the lexical and semantic rankings with Reciprocal Rank Fusion (rag/fusion.py).
  • RETRIEVAL_MODE=tfidf|semantic|hybrid, default hybrid, documented in .env.example. tfidf never builds an embedder, so it needs no model download.
  • Qdrant runs in memory by default; docker-compose.yml and QDRANT_URL give a real server.

App

  • The default pipeline is built in the application's startup hook instead of at import time, so importing app.main never downloads a model or touches a database. create_app(pipeline=...) still accepts an injected pipeline.
  • /documents and /query keep their routes and response shapes. Version bumped to 1.1.0.

Evaluation

  • docs_sample/ grows from 4 to 18 original notes.
  • evaluation/golden_set.json: 40 labeled questions in three kinds (lexical overlap, paraphrase, exact term).
  • python -m evaluation.run reports hit rate@3 and MRR for every mode, overall and per question kind.
  • CI runs it in a separate job with the embedding model cached, and fails when hybrid hit rate@3 is below the TF-IDF baseline (--enforce-gate).

Results

Printed by the retrieval-evaluation job (golden set v1, sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2):

Mode Hit rate@3 MRR
tfidf 0.875 (35/40) 0.852
semantic 0.900 (36/40) 0.877
hybrid 0.975 (39/40) 0.941
Hit rate@3 Lexical overlap (14) Paraphrase (14) Exact term (12)
tfidf 1.000 (14/14) 0.643 (9/14) 1.000 (12/12)
semantic 1.000 (14/14) 0.857 (12/14) 0.833 (10/12)
hybrid 1.000 (14/14) 0.929 (13/14) 1.000 (12/12)

Semantic retrieval alone is worse than TF-IDF on exact terms, and hybrid misses one paraphrase that semantic alone answers. The README says so and explains the sample-size caveat.

Existing tests that changed

Only tests/test_pipeline.py and tests/test_api.py, and only in how the pipeline is built: RagPipeline no longer defaults to a TF-IDF store, so both now pass store=TfidfVectorStore() explicitly. Every assertion is unchanged.

How it was checked

Locally, on Python 3.12.3 in a clean virtualenv built from requirements-dev.txt: ruff check ., ruff format --check . and pytest (150 tests, offline). On GitHub Actions: the test job (lint and tests) and the retrieval-evaluation job, which is the only place the real embedding model runs.

Unrelated fix included in the README: the run command now passes --env-file .env to uvicorn, since nothing in the app loads .env on its own.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add semantic and hybrid retrieval, and evaluate every retrieval mode

1 participant