Noor Knowledge Engine

⚠️ Work in Progress — Disclaimer

This project is under active development and is not production-ready.

Search results, citations, and generated answers may not be 100% accurate or complete. Retrieval, ranking, and answer generation can produce incorrect, incomplete, or misleading information. Do not rely on this system as a sole source of religious or factual guidance. Always verify important questions with qualified scholars and authoritative primary sources. Current sources are Quran (Eng and Arab) + Muslim and Bukhari Hadith books.

Citation-based Islamic knowledge search with retrieval and grounded RAG answer generation. This phase provides project structure, environment configuration, logging, a modular document ingestion pipeline, pluggable embeddings, Qdrant vector storage, LLM-free retrieval, and configurable LLM answer generation with citations.

Architecture

The system has two main paths: an offline indexing pipeline that builds the vector index, and an online query pipeline that retrieves relevant passages and (optionally) generates a cited answer. Clients can use the CLI, FastAPI (/search, /ask), or the Next.js frontend.

flowchart TB
    subgraph offline["Offline — build the index"]
        RAW["raw/ corpus<br/>Quran XML · Hadith JSON"]
        ING["Ingestion pipeline<br/>load → clean → metadata → chunk"]
        EMB["Embedder<br/>Ollama · OpenAI · sentence-transformers"]
        QD[(Qdrant<br/>vector store)]
        RAW --> ING --> EMB --> QD
    end

    subgraph online["Online — answer a question"]
        CLIENT["CLI · REST API · Web UI"]
        RET["Retriever<br/>embed query → search → rerank"]
        GEN["Answer generator<br/>prompt + LLM → cited answer"]
        CLIENT -->|"question"| RET
        QD -->|"top-k passages + citation metadata"| RET
        RET --> GEN -->|"answer with [n] citations"| CLIENT
    end

    offline ~~~ online

Loading

Path	Entry points	What happens
Index	index CLI	Corpus files are chunked with citation metadata, embedded, and upserted into Qdrant.
Search	search CLI, POST /search	Question is embedded, Qdrant returns ranked passages — no LLM call.
Ask (RAG)	ask / repl CLI, POST /ask, frontend	Retrieve top passages, then generate an answer constrained to that context with inline citations.
Evaluate	eval CLI	Run the golden dataset through retrieval and/or full RAG; compare reports over time.

Pluggable providers are selected via .env: embedding model for indexing and retrieval, LLM for generation (only on the ask path).

Prerequisites

• Python 3.12+
• For local vectors: Qdrant (e.g. docker run -p 6333:6333 qdrant/qdrant)
• For Ollama embeddings or chat: a running Ollama server with models pulled (e.g. ollama pull bge-m3, ollama pull qwen3:8b)

Setup

python3 -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -e ".[dev]"
cp .env.example .env

Download the indexing corpus (third-party Quran + Hadith files; not committed to git):

chmod +x scripts/download_corpus.sh
./scripts/download_corpus.sh

Source URLs and license notes: DATA_LICENSES.md.

For local sentence-transformers models:

pip install -e ".[local]"

Edit .env as needed. Never commit .env.

Run

python -m noor_knowledge_engine
# or
noor-knowledge-engine

Index the raw/ corpus into Qdrant (requires Qdrant + a configured embedding provider):

# Example local setup
# docker run -p 6333:6333 qdrant/qdrant
# ollama pull nomic-embed-text
# EMBEDDING_PROVIDER=ollama

python -m noor_knowledge_engine index --raw-dir raw --source all
# or: --source quran | hadith

Search the indexed collection (requires a running Qdrant and indexed data):

python -m noor_knowledge_engine search "what is patience" --top-k 5 --source quran

Ask a question and generate a grounded answer with citations (requires retrieval plus a configured LLM):

python -m noor_knowledge_engine ask "what is patience" --top-k 5 --source quran

ask prints the generated answer followed by retrieved sources with similarity scores and passage snippets:

Patience is a virtue emphasized throughout the Quran [1].

Citations:
  [1] score=0.9123  quran | Mushaf | 2:153
      O you who have believed, seek help through patience and prayer...

For interactive RAG testing, use the REPL (same retrieval flags as ask):

python -m noor_knowledge_engine repl --top-k 5 --source quran

Typical manual test flow: index → repl or ask. Type quit to exit the REPL.

Optional flags: --top-k, --source, --score-threshold. Defaults come from RETRIEVAL_TOP_K and RETRIEVAL_SCORE_THRESHOLD.

Ingestion

Load plain text or structured JSON, clean text, normalize citation metadata, and produce chunks:

from noor_knowledge_engine.ingestion import IngestionPipeline

chunks = IngestionPipeline().run(["data/quran/sample.json"], source="quran")
for chunk in chunks:
    print(chunk.metadata.reference, chunk.text[:80])

Supported sources (adapters): quran, hadith, tafsir, and generic (default for unknown sources). Structured JSON uses an entries[] list so each ayah/hadith/commentary can become its own chunk with source, book, author, chapter, language, and reference.

Raw corpus layout (raw/)

Download files into raw/ with ./scripts/download_corpus.sh before indexing. See DATA_LICENSES.md for attribution.

Files	Loader
quran-simple.xml, en.sahih.xml, quran-metadata.xml	Tanzil Quran (one chunk per ayah; Arabic + Saheeh International)
bukhari.json, muslim.json	Sunnah-style Hadith (one chunk per hadith; Arabic + English)

en.transliteration.xml is not indexed. Programmatic load:

from noor_knowledge_engine.ingestion import load_raw_corpus

chunks = load_raw_corpus("raw", source="all")

Embeddings and vector storage

Embedding generation and Qdrant storage are separate modules. Configure the provider with EMBEDDING_PROVIDER:

Provider	Notes
openai	Needs OPENAI_API_KEY. Default model: text-embedding-3-small
sentence_transformers	Install .[local]. Default: multilingual MiniLM
ollama	Local server at OLLAMA_BASE_URL. Default model: nomic-embed-text

from noor_knowledge_engine.embeddings import get_embedder
from noor_knowledge_engine.storage import QdrantVectorStore
from noor_knowledge_engine.indexing import index_chunks
from noor_knowledge_engine.ingestion import IngestionPipeline

chunks = IngestionPipeline().run(["data/quran/sample.json"], source="quran")
embedder = get_embedder()
store = QdrantVectorStore()

index_chunks(chunks, embedder, store)

Qdrant payloads keep chunk text plus citation metadata (source, book, author, chapter, language, reference) for filtered retrieval.

Retrieval

The retrieval layer embeds a user question, searches Qdrant, and returns passages with text, citation metadata, and a similarity score. It does not call an LLM.

from noor_knowledge_engine.retrieval import get_retriever

result = get_retriever().retrieve(
    "what is patience",
    top_k=5,
    filters={"source": "quran"},
)
for hit in result.hits:
    print(hit.score, hit.metadata.reference, hit.text[:80])

Configure defaults with RETRIEVAL_TOP_K (default 5) and optional RETRIEVAL_SCORE_THRESHOLD (drops hits below the threshold when set).

Answer generation (RAG)

Generation is separate from retrieval. Prompt templates force answers to use only retrieved context, request inline [n] citations, and refuse when sources are insufficient. Configure the chat model with LLM_PROVIDER:

Provider	Notes
openai	Needs OPENAI_API_KEY. Default model: gpt-4o-mini
ollama	Local server at OLLAMA_BASE_URL. Default model: qwen3:8b

Also: LLM_MODEL, LLM_TEMPERATURE (default 0.0), LLM_MAX_TOKENS (default 1024).

from noor_knowledge_engine.generation import get_rag_pipeline

answer = get_rag_pipeline().answer(
    "what is patience",
    top_k=5,
    filters={"source": "quran"},
)
print(answer.answer)
for citation in answer.citations:
    print(citation.index, citation.label)

Or generate from an existing SearchResult without calling the retriever again:

from noor_knowledge_engine.generation import AnswerGenerator, get_llm
from noor_knowledge_engine.retrieval import get_retriever

result = get_retriever().retrieve("what is patience")
answer = AnswerGenerator(get_llm()).generate("what is patience", result)

Evaluation

Measure RAG quality with a golden question dataset, retrieval metrics, citation checks, and latency stats. Index the corpus into Qdrant before running live evaluation.

Dataset: eval/datasets/golden_v1.json — questions with expected (source, reference) pairs (e.g. quran:2:255, hadith:Bukhari 1).

Quality snapshot

Latest internal benchmark (full-rerank-v1, 2026-07-24): 20 hand-written questions on Quran + Bukhari/Muslim hadith, evaluated end-to-end with bge-m3 embeddings, qwen3:8b generation, top_k=5, and reranking (candidate_k=20). A case passes only when retrieval finds the expected source, inline citations are valid, and the answer cites the expected reference.

End-to-end pass rate — 11 / 20 cases (55%)

Outcome	Cases	Share
Passed	11	55%	███████████░░░░░░░░░
Failed	9	45%	░░░░░░░░░░░█████████

Retrieval and citation scores

Metric	Score		In plain terms
Hit@5	65%	█████████████░░░░░░░	Expected source appears in the top 5 retrieved chunks
Recall@5	65%	█████████████░░░░░░░	Share of expected references found in top 5
MRR	0.42	—	How high the first correct hit ranks (1.0 = always first)
Citation validity	100%	████████████████████	Inline [n] markers map to retrieved sources
Expected cite recall	55%	███████████░░░░░░░░░	Answer actually cites the reference we expected
Unsupported citations	0%	░░░░░░░░░░░░░░░░░░░░	Citations pointing outside retrieved context
Median latency	~10s	—	Retrieval + generation per question (local Ollama)

Read this carefully: This is a small dev benchmark, not a measure of real-world Islamic knowledge accuracy. About 45% of golden cases still fail end-to-end. Results change with models, corpus coverage, chunking, and indexing. Treat these numbers as a rough baseline — re-run eval locally after setup (see below).

Run evaluation (retrieval + citation checks by default):

python -m noor_knowledge_engine eval run \
  --dataset eval/datasets/golden_v1.json \
  --label before-chunk-fix \
  --mode full \
  --top-k 5

Retrieval-only (no LLM calls):

python -m noor_knowledge_engine eval run \
  --dataset eval/datasets/golden_v1.json \
  --label retrieval-baseline \
  --mode retrieval

Compare two saved reports (use eval labels or full paths):

python -m noor_knowledge_engine eval list

python -m noor_knowledge_engine eval compare before-chunk-fix after-chunk-fix

Labels resolve to the newest matching report in eval/reports/. You can also pass full paths such as eval/reports/before-chunk-fix_20260723T084924Z.json.

Reports are written to eval/reports/{label}_{timestamp}.json and include settings snapshot, aggregate metrics, and per-case results.

Metric	Meaning
Hit@k	Any expected source appears in top-k hits
Recall@k	Fraction of expected sources found in top-k
MRR	Reciprocal rank of the first expected hit
Citation validity	Fraction of inline [n] markers that map to retrieved citations
Expected citation recall	Fraction of expected sources cited in the answer
Unsupported citation rate	Inline citations outside the retrieved set

Programmatic use:

from noor_knowledge_engine.config import Settings
from noor_knowledge_engine.evaluation import run_evaluation, save_report, compare_reports, load_report

settings = Settings()
report = run_evaluation(
    dataset_path="eval/datasets/golden_v1.json",
    label="experiment-a",
    mode="retrieval",
    top_k=5,
    score_threshold=None,
    settings=settings,
)
path = save_report(report, "eval/reports")

HTTP API

Start the FastAPI server (same engine as the CLI search / ask commands):

python -m noor_knowledge_engine serve
# or with options:
python -m noor_knowledge_engine serve --host 0.0.0.0 --port 8000 --reload

Alternative:

uvicorn noor_knowledge_engine.api.app:app --reload

Interactive docs: http://127.0.0.1:8000/docs

Endpoints

Method	Path	Description
GET	/health	Qdrant connectivity and collection status
POST	/search	Retrieve relevant passages (no LLM)
POST	/ask	Retrieve passages and generate a grounded answer

Request body for /search and /ask:

{
  "query": "what is patience",
  "top_k": 5,
  "source": "quran",
  "score_threshold": 0.5
}

source is optional (quran or hadith). Other fields use settings defaults when omitted.

Examples:

curl -s http://127.0.0.1:8000/health | jq

curl -s -X POST http://127.0.0.1:8000/search \
  -H "Content-Type: application/json" \
  -d '{"query": "what is patience", "top_k": 5, "source": "quran"}' | jq

curl -s -X POST http://127.0.0.1:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"query": "what is patience", "top_k": 5, "source": "quran"}' | jq

Tests

Install dev dependencies (included in the setup above):

pip install -e ".[dev]"

Run the full Python test suite from the repo root (uses mocked Qdrant/Ollama/LLM — no live services required):

source .venv/bin/activate  # Windows: .venv\Scripts\activate
pytest

Useful variants:

# Quiet summary
pytest -q

# One file or directory
pytest tests/retrieval/
pytest tests/api/test_routes.py

# Stop on first failure
pytest -x

# Show print/log output
pytest -s

Frontend tests (if you work on frontend/):

cd frontend
npm install
npm test

Live RAG evaluation (requires indexed Qdrant + configured embedder) is separate from unit tests — see Evaluation.

Project layout

src/noor_knowledge_engine/
  config.py                  # Settings from .env (pydantic-settings)
  logging_setup.py           # Logging configuration
  indexing.py                # Thin glue: embed then upsert / embed then search
  __main__.py                # CLI entry point (index / search / ask / repl / eval / serve)
  api/                       # FastAPI HTTP layer (health / search / ask)
  evaluation/                # RAG eval dataset, metrics, runner, report compare
  ingestion/                 # Document load → extract → clean → metadata → chunk
    models.py                # Document, Chunk, DocumentMetadata
    pipeline.py              # IngestionPipeline
    corpus.py                # raw/ Tanzil + sunnah corpus loader
    loaders/                 # Text, JSON, Tanzil Quran, sunnah Hadith loaders
    chunking/                # Fixed-size and structure-aware chunkers
    sources/                 # Quran / Hadith / Tafsir / generic adapters
  embeddings/                # OpenAI / sentence-transformers / Ollama providers
  storage/                   # Qdrant vector store
  retrieval/                 # LLM-free query → embed → search
  generation/                # Prompts, LLM providers, grounded RAG answers
tests/
  fixtures/                  # Sample Quran, Hadith, Tafsir, and text files
  ingestion/                 # Ingestion unit and pipeline tests
  embeddings/                # Embedder unit tests (mocked backends)
  storage/                   # Qdrant store unit tests (mocked client)
  retrieval/                 # Retriever unit tests (mocked backends)
  generation/                # Generation unit tests (mocked LLMs)
  evaluation/                # RAG eval unit tests (mocked backends)
  api/                       # HTTP API unit tests (mocked engine)
eval/
  datasets/                  # Golden evaluation question sets
  reports/                   # Saved evaluation report JSON files

Out of scope (this phase)

Frontend and auth/rate-limiting are not included yet.