Skip to content

README.md

Latest commit

 

History

History
239 lines (183 loc) · 12 KB

README.md

File metadata and controls

239 lines (183 loc) · 12 KB
Raw

📓 Simple NotebookLM

A document-grounded RAG (Retrieval-Augmented Generation) learning assistant inspired by Google's NotebookLM — upload PDFs and get grounded Q&A, summaries, quizzes, and flashcards, all with source citations back to the exact page they came from.

Built as an end-to-end system: local vector store, swappable LLM backends (Gemini / OpenAI / Claude) switchable at runtime without a restart, a FastAPI service, a Typer CLI, a Streamlit UI, and a full RAGAS-based evaluation pipeline used to empirically pick the best chunking and reranking strategy.

Python 3.10+ FastAPI Streamlit LangChain Qdrant License: MIT

📖 Read PIPELINE.md for the complete architecture guide, provider setup, configuration reference, and step-by-step reproduction of the benchmark results below.

✨ Features
💬 Grounded Q&A Ask questions in natural language and get answers with inline [Sn] citation markers linked to the exact source document and page
📝 Summarization Summarize a single document, a topic across documents, or the entire corpus — automatically map-reduces when content exceeds one context window
🧠 Quiz Generation Auto-generated multiple-choice quizzes with explanations, scoped to any document or topic, with instant scoring in the UI
🃏 Flashcards Study flashcards with click-to-reveal answers and optional hints
🌐 Multi-provider LLM Swap between Gemini, OpenAI, and Claude live — via the Streamlit sidebar or a REST call — no server restart required
🌏 Language-aware generation Output language automatically follows the source document (tested with English and Vietnamese corpora)
📊 Empirically tuned retrieval Chunking size/overlap and cross-encoder reranking were selected via a real RAGAS evaluation, not guesswork — see Results
🗂 Multi-document scoping Upload, list, and delete documents from the vector store; scope any feature to a subset of documents

🏗️ Architecture

┌─────────────────────────────────────────────────────────────┐
│                     User Interfaces                         │
│  Streamlit UI  ←→  FastAPI REST  ←→  Typer CLI              │
└────────────────────────┬────────────────────────────────────┘
                          │
            ┌─────────────▼─────────────────┐
            │      Application Layer        │
            │   src/rag.py  src/learning    │
            │  Q&A · Summary · Quiz · Cards │
            └──────────┬────────────────────┘
                        │
         ┌──────────────▼───────────────────────┐
         │            Storage Layer             │
         │   Qdrant (local, file-based) + PDFs  │
         └──────────┬───────────────────────────┘
                     │
         ┌───────────▼──────────────────────────────────────┐
         │      LLM Backend (pick one, switch anytime)      │
         │       Gemini   ·   OpenAI   ·   Claude           │
         └──────────────────────────────────────────────────┘

Indexing (once): PDF → PyPDFLoader → RecursiveCharacterTextSplitter → HuggingFace embeddings → Qdrant, tagged with filename, page, document_id, chunk_id.

Query (per request): question → embed → Qdrant similarity search (top-k) → Jinja2 prompt template → LLM → structured answer with citations.

All four learning features share this retrieve → prompt → generate pattern, differing only in their Jinja2 template and expected JSON output shape.

🧰 Tech Stack

Layer Technology
Orchestration LangChain
Vector store Qdrant (local, embedded — no server process)
Embeddings HuggingFace Sentence-Transformers (Vietnamese-optimized by default)
LLM providers Google Gemini · OpenAI · Anthropic Claude (hot-swappable)
Backend API FastAPI
CLI Typer
Frontend Streamlit
Prompting Jinja2 templates
Evaluation RAGAS (faithfulness, answer relevancy, context precision, context recall)
Reranking Cross-encoder (BAAI/bge-reranker-v2-m3)
Validation Pydantic v2

🚀 Quickstart

# 1. Clone and set up a virtual environment
git clone <your-repo-url>
cd simple_notebooklm
python -m venv .venv
.venv\Scripts\activate        # Windows
source .venv/bin/activate      # macOS / Linux

# 2. Install dependencies
pip install -r requirements.txt

# 3. Configure your LLM provider
cp .env.example .env
# then edit .env — at minimum set RAG_LLM_PROVIDER and its matching API key
# .env — minimal Gemini setup (free tier available)
RAG_LLM_PROVIDER=gemini
GOOGLE_API_KEY=your_key_here
# 4. Drop your PDFs into data/, then index them
python -m src.interfaces.cli ingest

# 5. Run it
uvicorn src.interfaces.api:app --reload          # terminal 1 — API on :8000
streamlit run src/interfaces/ui.py                # terminal 2 — UI on :8501

Or skip the UI entirely and use the CLI:

python -m src.interfaces.cli ask "What is LoRA?"
python -m src.interfaces.cli quiz --query "fine-tuning" --count 5

Full provider setup (Gemini / OpenAI / Claude), every configuration variable, and the complete REST API reference live in PIPELINE.md.

📊 Benchmark Results

Retrieval quality was evaluated on a 50-question benchmark using RAGAS across four metrics: Faithfulness, Answer Relevancy, Context Precision, and Context Recall. The full pipeline for reproducing these numbers is documented in PIPELINE.md § 7.

Chunking strategy comparison

Four recursive chunking configurations were evaluated to find the best size/overlap trade-off:

Figure 7 — Ragas metrics across recursive chunking configurations

Chunk Size / Overlap Faithfulness Answer Relevancy Context Precision Context Recall
500 / 50 0.787 0.751 0.815 0.920
800 / 100 0.811 0.753 0.857 0.920
1000 / 150 0.863 0.729 0.862 0.920
1500 / 200 0.881 0.759 0.815 0.920

1000/150 was selected as the default — the best balance of faithfulness and context precision without the diminishing returns of larger chunks.

Adding a cross-encoder reranker

Layering a BAAI/bge-reranker-v2-m3 reranker on top of the best chunking config (retrieve 15 → rerank → keep top 5) was compared against recursive and semantic chunking alone:

Figure 9 — Ragas metrics, chunking vs reranking

Method Faithfulness Answer Relevancy Context Precision Context Recall
Recursive (1000/150) 0.863 0.729 0.862 0.920
Semantic (Interquartile) 0.887 0.796 0.812 0.920
Recursive + Reranker 0.876 0.804 0.925 0.940

Adding a reranker was the single highest-leverage change — it improved context precision and recall beyond either chunking strategy alone, confirming that retrieval quality benefits more from re-scoring candidates than from further tuning chunk boundaries.

Reproduce these figures yourself:

python reproduce_figures.py --use-reference-data   # uses the numbers above
python reproduce_figures.py                        # or regenerate from your own eval run

📁 Project Structure

simple_notebooklm/
├── data/                      # PDF knowledge base
├── storage/qdrant/            # Local vector store (created on first ingest)
├── evaluation_results/        # Benchmark data + experiment outputs
│
├── src/
│   ├── config.py              # Pydantic settings (all env vars, 3 LLM providers)
│   ├── schemas.py              # Chunk / Citation / RagAnswer / Quiz / Flashcard models
│   ├── store.py                # Qdrant client + cached HuggingFace embeddings
│   ├── indexing.py             # PDF loading, chunking, ingestion
│   ├── filters.py               # Metadata filter → Qdrant filter (multi-doc support)
│   ├── rag.py                  # retrieve() / answer() / citation formatting
│   ├── llm.py                  # LLM factory + runtime provider switching
│   ├── learning.py              # summarize() / generate_quiz() / generate_flashcards()
│   ├── export.py                # Result → Markdown / JSON export
│   ├── prompts/                # Jinja2 templates (Q&A, summary, quiz, flashcards)
│   ├── interfaces/
│   │   ├── api.py               # FastAPI app (10 endpoints)
│   │   ├── cli.py               # Typer CLI (6 commands)
│   │   └── ui.py                # Streamlit frontend
│   └── evaluation/
│       ├── ragas_evaluator.py         # RAGAS metric setup
│       ├── chunking_strategies.py     # Recursive + semantic chunkers
│       ├── run_chunking.py            # Chunking experiment → Figure 7
│       └── run_reranking.py           # Reranking experiment → Figure 9
│
├── reproduce_figures.py       # Plots Figures 7 & 9 from experiment output
├── requirements.txt
├── .env.example
└── PIPELINE.md                 # Full architecture & operations guide

🔌 REST API at a glance

curl -X POST http://localhost:8000/ask \
     -H "Content-Type: application/json" \
     -d '{"question": "What is LoRA?"}'
Method Path Description
GET /health Liveness check
GET/POST /config/llm Read / switch active LLM provider & model at runtime
GET /documents List indexed documents
POST /upload Upload and index a PDF
DELETE /documents/{id} Remove all chunks for a document
POST /ask Grounded Q&A with citations
POST /summarize Document / topic / corpus summarization
POST /quiz Generate a multiple-choice quiz
POST /flashcards Generate study flashcards

Interactive Swagger docs are served at http://localhost:8000/docs once the API is running. Full request/response examples for every endpoint are in PIPELINE.md § 6.

🙋 Why this project

This started as a side project to build a scaled-down NotebookLM, but was extended well past the original scope: a three-provider LLM abstraction with hot-swapping, a proper metadata filtering layer for multi-document scoping, and a rigorous RAGAS evaluation harness to justify retrieval design choices with data instead of intuition. It's meant to demonstrate practical, production-shaped RAG engineering, not just a prompt wrapped around an LLM call.

📄 License

Licensed under the MIT License.