DeAI — Decision AI

Turkish Legal Question Answering with an Optimized RAG Pipeline

An optimized Retrieval-Augmented Generation system for Turkish legal question answering, developed as part of CENG493 Term Project.

Current Status (April 12, 2026)

• Project scaffold and configuration
• Dataset download and exploration
• Text cleaning pipeline
• Document chunking with legal boundary detection
• FAISS indexing and dense retrieval
• Baseline RAG pipeline (retrieval only)
• Gold test set (30 QA pairs + 3 adversarial)
• Retrieval evaluation framework
• Embedding fine-tuning (planned: April-May)
• Reranker fine-tuning (planned: May)
• LLM QLoRA fine-tuning (planned: May-June)
• Full ablation evaluation (planned: June)

Problem

Given a Turkish legal question, produce a grounded, source-supported, context-aware answer with minimal hallucination and consistent citations.

Architecture

Question
  │
  ▼
┌─────────────────┐
│  Embedding Layer │  ← BAAI/bge-m3 (fine-tuned)
│  (Hybrid Search) │     Dense + Sparse + ColBERT
└────────┬────────┘
         │  top-k passages
         ▼
┌─────────────────┐
│    Reranker      │  ← Jina Reranker v2 (fine-tuned)
│  (Cross-Encoder) │     Token-level query-doc interaction
└────────┬────────┘
         │  top-n reranked
         ▼
┌─────────────────┐
│   LLM Generator  │  ← Qwen3-8B (QLoRA fine-tuned)
│  (RAG Prompting) │     Citation-aware answer generation
└────────┬────────┘
         │
         ▼
   Grounded Answer
   with [1][2] citations

Ablation Configurations

Config	Embedding	Reranker	LLM
Baseline	bge-m3 frozen	None	Qwen3-8B frozen
+Embedding	bge-m3 FT	None	Qwen3-8B frozen
+Reranker	bge-m3 FT	Jina v2 FT	Qwen3-8B frozen
+LLM	bge-m3 FT	Jina v2 FT	Qwen3-8B QLoRA
Full	bge-m3 FT	Jina v2 FT	Qwen3-8B QLoRA + RAG prompting

Datasets

• Fine-tune Corpus: turkishlaw-dataset + turkish-lawchatbot
• Gold Test Set: 150-300 manually verified Turkish legal QA pairs

Evaluation Metrics

Retrieval: Recall@5, Recall@10, MRR, nDCG
QA: Exact Match, F1, ROUGE-L, BERTScore
Hallucination: Faithfulness score, Citation accuracy (RAGAS)

Setup

git clone https://github.com/<username>/deai.git
cd deai
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Quick Start

# 1. Download datasets
python src/data/download.py

# 2. Clean and chunk
python src/data/clean.py
python src/data/chunk.py

# 3. Build FAISS index
python src/retrieval/indexer.py

# 4. Run baseline pipeline
python scripts/run_baseline.py --question "İş sözleşmesi feshi için yasal süre nedir?"

Project Timeline

Phase	Period	Focus
Phase 1	Mar 1 - Apr 12	Scaffolding, data pipeline, baseline retrieval
Phase 2	Apr - May	Embedding & reranker fine-tuning
Phase 3	May - Jun	LLM QLoRA, full evaluation, final report

License

Academic use only.