Multimodal RAG AI Assistant

Portfolio-grade, production-style AI assistant that combines multimodal ingestion, retrieval-augmented generation (RAG), intent-aware query planning, answer post-processing, and dual interfaces (FastAPI + Gradio).

Live Demo Video

• Full walkthrough: Watch the demo
• Includes: PDF + URL + YouTube ingestion, cross-source Q&A, summarization and humanization comparison

Why This Project Stands Out

• Multimodal knowledge ingestion: PDF, DOCX, TXT/MD, URL, YouTube transcripts, images
• Intent-aware retrieval pipeline: factual, summary, comparison, open-ended
• Pluggable architecture: FAISS/Chroma, Gemini/OpenAI, local/offline fallback paths
• Post-processing layer: optional summarization and answer humanization
• Engineering maturity: modular codebase, lazy initialization, typed exceptions, smoke tests

System Architecture

Ingestion Sources

Source	Handler
PDF	load_pdf()
DOCX	load_docx()
TXT/MD	load_text()
URL	load_url()
YouTube	load_youtube()
Image	load_image()

Repository Layout

multimodal-rag-assistant/
|-- core/                  # config, logger, exceptions
|-- ingestion/             # loaders, preprocessor, chunker
|-- retrieval/             # embeddings, vector store, retriever
|-- rag_pipeline/          # query processing and orchestration
|-- processing/
|   |-- summarizer/        # BART summarization
|   `-- humanizer/         # rule + LLM humanization
|-- api/                   # FastAPI server and schemas
|-- frontend/              # Gradio app
|-- assets/                # persisted vectors and history
|-- tests/                 # smoke tests
|-- setup.ps1              # Windows setup
|-- setup.sh               # Linux/macOS setup
`-- README.md

Quick Start

1. Environment Setup

Windows (PowerShell):

cd multimodal-rag-assistant
.\setup.ps1

Linux/macOS:

cd multimodal-rag-assistant
bash setup.sh

2. Configure Keys

Edit .env:

GEMINI_API_KEY=your_key_here
OPENAI_API_KEY=

3. Run API and UI

Terminal 1 (API):

.\venv\Scripts\python.exe -m uvicorn api.server:app --reload --port 8000

Terminal 2 (UI):

.\venv\Scripts\python.exe frontend\app.py

Endpoints:

• API docs: http://localhost:8000/docs
• Gradio UI: http://localhost:7860

API Endpoints

Method	Route	Purpose
GET	/health	Service health + KB status
POST	/ingest/file	Ingest uploaded files
POST	/ingest/url	Ingest web content
POST	/ingest/youtube	Ingest YouTube transcript
POST	/query	Ask grounded questions
GET	/history	Read conversation history
DELETE	/knowledge-base	Reset vector store

Example:

curl -X POST "http://localhost:8000/query" \
    -H "Content-Type: application/json" \
    -d '{
        "query": "Summarize the key implementation decisions",
        "top_k": 5,
        "apply_summarization": true,
        "apply_humanization": true
    }'

Configurable Components

Core options live in core/config.py.

• LLM model/provider
• embedding provider/model
• vector backend (faiss/chroma)
• chunk size/overlap
• summarization and humanization toggles

Test Status

Smoke tests:

.\venv\Scripts\python.exe tests\test_smoke.py

Current baseline: 19/19 passing.

Roadmap

• Add reranker layer for retrieval quality
• Add citation spans with page/time offsets
• Add async ingestion pipeline and queueing
• Add benchmark suite (faithfulness, context precision, latency)

Acknowledgements

This project is a merged and upgraded evolution of:

• RAG_multi_media_QA_system
• RAG-based_Chatbot
• Summarization_project
• AiHumanizer
• project-samarth

License

MIT