RAG Robustness Evaluation

A reproducible evaluation scaffold for studying retrieval-augmented generation under clean and adversarial conditions. The public repository currently establishes the clean Natural Questions baseline and the measurement pipeline used for later attack and detector comparisons.

Current public scope

• Curates a 5,000-document Wikipedia knowledge base
• Builds a persistent Chroma vector store
• Supports local generation through Ollama or Hugging Face
• Evaluates Exact Match and token-level F1 on Natural Questions
• Saves predictions and append-only experiment metadata

Architecture

flowchart LR
    A[Natural Questions] --> B[Wikipedia corpus builder]
    B --> C[Chunking and embeddings]
    C --> D[(Chroma vector store)]
    D --> E[Top-k retrieval]
    E --> F[Ollama or Hugging Face generation]
    F --> G[EM and token-F1 evaluation]
    G --> H[CSV metrics and JSONL predictions]

Loading

Repository map

scripts/
  build_kb.py                 Curate and chunk the clean knowledge base
  build_vectorstore.py        Create the persistent Chroma collection
  build_rag_pipeline.py       Retrieve context and generate short answers
  evaluate_rag.py             Run Natural Questions evaluation
  export_clean_embeddings.py  Export embeddings for downstream analysis

results/
  metrics.csv                 Logged baseline runs
  metrics_prompt_tuning.csv   Prompt comparison results
  predictions_*.jsonl         Inspectable model outputs

Verified baseline

The largest checked-in clean run evaluated 1,000 questions with no pipeline failures. The repository preserves the full settings for each run, including backend, model, collection, top-k, answer count, EM, and token F1.

These scores are baseline measurements, not final model-quality claims. Their value is that they establish a repeatable control condition for later poisoning and detection experiments.

Reproducibility path

1. Run scripts/build_kb.py to create the clean corpus.
2. Run scripts/build_vectorstore.py to create the Chroma collection.
3. Start Ollama locally or configure a supported Hugging Face model.
4. Use scripts/build_rag_pipeline.py for interactive clean retrieval.
5. Run scripts/evaluate_rag.py to append a comparable baseline result.

See README_Module3_Handoff.md for artifact paths, validated settings, and caveats.

Evaluation discipline

• Predictions are saved for inspection rather than reporting only aggregate scores.
• Dataset size, model backend, retrieval depth, and resolved collection are recorded with each run.
• Prompt changes are compared in a separate metrics file.
• Current results are explicitly labeled provisional until the full attack and detector matrix is complete.