Local RAG Application 📚

A locally-running, multilingual-enabled, advanced Retrieval-Augmented Generation (RAG) system. This application provides semantic question-answering based on your documents using an AI-powered system.

🎯 Project Overview

AlbaraKa Document Q&A System is a comprehensive RAG solution designed for corporate document management and querying. Going beyond traditional search engines, with semantic search and contextual response generation capabilities, it offers:

• Multiple document format support (PDF, DOCX, TXT, CSV, XLSX)
• Multiple embedding model support (JinaAI, intfloat, mBERT, RoBERTa)
• Multiple LLM support (Llama 3.1/3.2/3.3, Gemma 3)
• Advanced query techniques (Multi-Query, Query Augmentation)
• Visualization (UMAP for query-document relationship)

Core Objectives

• Digitize corporate knowledge management
• Access information quickly and accurately from documents
• Provide localized RAG solutions with multilingual support (especially Turkish)
• Minimize LLM hallucinations
• Increase accuracy with advanced retrieval techniques

✨ Key Features

🧠 Advanced Retrieval Techniques

1. Multi-Query Generation

Generates multiple queries for complex questions to improve retrieval performance:

# Example: "What are the rules for opening a corporate bank account?"
# Generated queries:
- "Corporate bank account opening process and KYC documentation requirements."
- "Regulatory guidelines for corporate account eligibility and compliance."
- "Business banking onboarding policies and required financial records."

2. Query Augmentation

• Query Mode: Expands query with industry terminology
• Answer Mode: Generates a sample answer to optimize the embedding model

3. Multi-Model Embedding Support

Embedding Model	Dimensions	Language Support	Features
jinaai/jina-embeddings-v3	1024	Multilingual	Latest JinaAI model
Omerhan/intfloat-fine-tuned-14376-v4	768	Turkish	Fine-tuned for Turkish
intfloat/multilingual-e5-large-instruct	1024	Multilingual	Instruction-tuned E5
emrecan/bert-base-turkish-cased-mean-nli-stsb-tr	768	Turkish	BERT-based Turkish
atasoglu/roberta-small-turkish-clean-uncased-nli-stsb-tr	768	Turkish	Small RoBERTa-based model
atasoglu/distilbert-base-turkish-cased-nli-stsb-tr	768	Turkish	Fast DistilBERT-based

🤖 LLM Integration (Ollama)

LLM Model	Parameters	Use Case
llama3.2:3b	3 billion	Fast responses, simple questions
llama3.2:1b	1 billion	Very fast, resource-constrained systems
llama3.1:8b	8 billion	Balanced performance
llama3.3	70 billion	Highest quality (as needed)
llama3.2-vision	11 billion	Vision support (future release)
gemma3	4 billion	Google's open-source model

📊 Visualization

• UMAP (Uniform Manifold Approximation and Projection) for dimensionality reduction
• 2D space visualization of query and document chunks
• Visual analysis of retrieval quality

🖥️ User Interface (Gradio)

• Three tabs: Query, Document Management, About
• Advanced settings: Model selection, augmentation, multi-query
• Visual outputs: Response, sources, retrieved chunks, visualization
• Real-time status: Processing time, source indication

🛠️ Technologies Used

Technology	Version	Purpose
Python	3.8+	Main programming language
LangChain	0.3.15	RAG pipeline management
ChromaDB	Latest	Vector database
Sentence-Transformers	≥2.2.2	Embedding generation
HuggingFace	4.47.1	Model repository and tokenization
Ollama	Latest	Local LLM execution
Gradio	5.20.1	Web interface
PyTorch	2.6.0+cu126	Deep learning infrastructure
Flash Attention	2.7.4	Accelerated attention mechanism
UMAP-learn	Latest	Dimensionality reduction
PyNVML	12.0.0	GPU memory management
PyPDF	Latest	PDF processing
python-docx	Latest	Word document processing

📁 Project Structure

Local_RAG_Application/
├── 🐍 app.py                       # Gradio web interface
├── 🐍 populate_database.py         # Vector database creation
├── 🐍 query_data.py                # Query processing and RAG logic
├── 🐍 get_embedding_function.py    # Embedding model loading
├── 🐍 run_utils.py                # Helper functions
├── 🐍 ntest_rag.py                # RAG test script
├── 🐍 test_args.py                # Argument tests
├── 📂 data/                        # Documents to upload (auto-created)
├── 📂 chroma/                      # ChromaDB vector database (auto-created)
├── 📂 results/                     # Visualization results
│   ├── query_chunks_visualization_Omerhan_intfloat-fine-tuned-14376-v4.png
│   ├── query_chunks_visualization_atasoglu_mbert-base-cased-nli-stsb-tr.png
│   ├── query_chunks_visualization_atasoglu_xlm-roberta-base-nli-stsb-tr.png
│   └── query_chunks_visualization_jinaai_jina-embeddings-v3.png
├── 📄 requirements.txt             # Python requirements
├── 📄 LICENSE                     # Apache 2.0 License
├── 📄 README.md                   # This file
├── 📄 .gitignore                  # Git ignore rules
└── 📄 .gitattributes              # Git attributes

🔬 RAG Architecture

1. Document Loading and Processing

# Supported formats
PDF (.pdf)    → Text extraction with PyPDF
DOCX (.docx)  → Text extraction with python-docx
TXT (.txt)    → Direct reading
CSV (.csv)    → Processing with Pandas
XLSX (.xlsx)  → Processing with Pandas

# Text splitting (chunking)
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,        # Each chunk 800 characters
    chunk_overlap=80,      # 80 character overlap between chunks
    length_function=len,
    is_separator_regex=False
)

2. Embedding and Vectorization

# Embedding function example
embeddings = HuggingFaceEmbeddings(
    model_name="Omerhan/intfloat-fine-tuned-14376-v4",
    encode_kwargs={'normalize_embeddings': True},
    model_kwargs={
        'trust_remote_code': True,
        'device': 'cuda' if torch.cuda.is_available() else 'cpu'
    }
)

# Adding to ChromaDB
db = Chroma(persist_directory="chroma", embedding_function=embeddings)
db.add_documents(chunks, ids=chunk_ids)

3. Query and Retrieval

# Similarity search
results = db.similarity_search_with_relevance_scores(query, k=5)
# results: List of (Document, score) tuples

# Advanced search with Multi-query
queries = QueryData.generate_multi_query(original_query, model="llama3.2:3b")
for q in queries:
    results.extend(db.similarity_search_with_relevance_scores(q, k=4))

4. Response Generation with LLM

# Prompt template
PROMPT_TEMPLATE = """
Answer the question based only on the following context:

{context}

---

Answer the question based on the above context: {question}
"""

# LLM call
model = Ollama(model="llama3.2:3b")
prompt = prompt_template.format(context=context_text, question=query_text)
response_text = model.invoke(prompt)

🚀 Installation

Requirements

• Python: 3.8 or higher
• pip: Python package manager
• Ollama: For running local LLM (https://ollama.com)
• CUDA (optional): For GPU acceleration

Steps

1. Clone the repository:

   git clone https://github.com/TugayTalha/Local_RAG_Application.git
   cd Local_RAG_Application

2. Create a virtual environment (recommended):

   python -m venv venv
   source venv/bin/activate  # Linux/Mac
   # or
   venv\Scripts\activate     # Windows

3. Install requirements:

   pip install -r requirements.txt

   Note: For CUDA support with PyTorch:

   pip install torch==2.6.0+cu126 -f https://download.pytorch.org/whl/torch

4. Install Ollama and download models:

   # Install Ollama: https://ollama.com/download
   
   # Download LLM models
   ollama pull llama3.2:3b
   ollama pull llama3.1:8b   # optional
   
   # Embedding models are downloaded from HuggingFace automatically
   # on first run

💡 Usage

A. Using Web Interface (Recommended)

1. Start the application:

   python app.py

   Browser will open: http://localhost:7860

2. Upload documents:

   • Go to "Document Management" tab
   • Upload your PDF, DOCX, TXT, CSV, or XLSX files
   • Click "Upload Files" button

3. Create the database:

   • Select embedding model in "Document Management" tab
   • Check "Reset Database" if needed
   • Click "Populate Database" button
   • Wait for "✅ Database populated successfully!" message

4. Ask questions:

   • Go to "Query Documents" tab
   • Type your question (e.g., "Which ATMs can my customer withdraw money from?")
   • Select LLM and Embedding models
   • Adjust advanced settings (Multi-Query, Augmentation)
   • Click "Submit Query" button
   • Review response, sources, and visualization

B. Command Line Usage

1. Create Database

python populate_database.py --model-name "Omerhan/intfloat-fine-tuned-14376-v4" --model-type "sentence_transformer" --reset

2. Query

python query_data.py "Which ATMs can my customer withdraw money from?" --model-name "jinaai/jina-embeddings-v3" --model-type "sentence_transformer"

C. Python Script Usage

from run_utils import populate_database, QueryData
from get_embedding_function import get_embedding_function

# Prepare embedding function
embedding_func = get_embedding_function(
    model_name_or_path="atasoglu/roberta-small-turkish-clean-uncased-nli-stsb-tr",
    model_type="sentence_transformer"
)

# Create database
populate_database(reset=True, model_name="atasoglu/roberta-small-turkish-clean-uncased-nli-stsb-tr")

# Make query
response, chunks = QueryData.query_rag(
    query_text="What are the corporate account opening procedures?",
    embedding_function=embedding_func,
    model="llama3.2:3b",
    augmentation="query",
    multi_query=True
)

print(f"Response: {response}")
print(f"Sources: {[chunk['source'] for chunk in chunks]}")

📊 Visualization Example

The system visualizes your query and document chunks in 2D space using UMAP dimensionality reduction:

Visualization Components:

• 🔴 Red dot: Query vector
• 🟡 Yellow dots: Retrieved document chunks
• ⚪ Gray dots: All document chunks in database

This visualization helps you analyze how well your query matches the documents.

🧪 Testing and Evaluation

RAG Evaluation Metrics

The system automatically evaluates responses using LLM:

from run_utils import evaluate_response

evaluation = evaluate_response(
    actual_response="Corporate account requires KYC documents.",
    expected_response="Corporate account requires KYC documentation."
)
# evaluation: "true" or "false"

Performance Monitoring

• Query time: Processing time for each query
• Relevance score: Similarity score for each document chunk (0-1)
• Source count: Number of unique sources used

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

Contribution Areas

• Integration of new embedding models (OpenAI, Cohere, etc.)
• Different vector databases (FAISS, Pinecone, Weaviate)
• More document format support (HTML, Markdown, LaTeX)
• Query-answer validation metrics improvement
• UI/UX enhancements (Streamlit, Next.js, etc.)
• Multilingual support expansion
• Hybrid search (keyword + vector) addition
• Response streaming (token-based response flow)

📄 License

This project is licensed under the Apache License 2.0. You can find the license text in the LICENSE file.

📧 Contact

Tugay Talha İçen

• GitHub: @Tugaytalha
• Twitter: @TugayTalhaIcen
• LinkedIn: Tugay Talha İçen

Project Link: https://github.com/TugayTalha/Local_RAG_Application

🙏 Acknowledgments

• To LangChain team for excellent RAG tools
• To ChromaDB team for open-source vector database
• To HuggingFace for model repository and transformers library
• To Ollama for easy local LLM execution
• To JinaAI, intfloat, atasoglu, emrecan for Turkish embedding models
• To Gradio for quick web interface development

---

📚 Local, secure, multilingual RAG experience!

⭐ Don't forget to star this project if you found it useful!

🐛 Use the Issues tab for bug reports or suggestions.

🎯 Discover the power of corporate knowledge with AlbaraKa!