Vector Volume Discovery Pipeline

This repository is a workflow for discoverability of multi-modal content as embedded vectors in a QDrant database. It uses the ColPali model, a Vision Language Model to produce multi-vector embeddings from images of document pages. ColPali uses the PaliGemma-3B model, which combines SigLIP patch embeddings with a Gemma-2B language model, to generate ColBERT-style multi-vector representations of text and images. The vectors are available through a QDrant vector store.

Prerequisites

Before proceeding, ensure the following are installed on your server:

• Python 3.8+
• Pip (Python package manager)

Steps to Host the Models

Follow these steps to get the models running on your server.

1. Clone the Repository

First, clone the repository to your server using the following command:

git clone <repository-url>

2. Install Required Packages

Install all necessary dependencies by executing the command below:

pip install -r requirements.txt

3. Navigate to the Model Directory

In order to host colpali model and functionalities script, move into the below mentioned directory:

cd colpali

In order to host LLM Vision Model for inferencing(User response generation) move into the below mentioned directory:

cd llm_vision_models

4. Run the Model Hosting Scripts

To start the API for each model, use the following commands:

i. Colpali Image Retrieval Model and functionalities

Start the server for Colpali Image Retrieval Model using this command:

python -m uvicorn colpali_host_script.py:app --host 0.0.0.0 --port 8000 --reload

ii. LLaMA 3.2 Vision

Start the server for LLaMA 3.2 Vision using this command:

python -m uvicorn llama_3_2_vision_host_script.py:app --host 0.0.0.0 --port 8000 --reload

iii. Paligemma

Start the server for Paligemma using this command:

python -m uvicorn paligemma_host_script.py:app --host 0.0.0.0 --port 8000 --reload

5. Accessing the Models

Once the server is running, the models will be accessible via the provided API endpoints:

• Colpali: http://<server-ip>:8000
• LLaMA 3.2 Vision: http://<server-ip>:8000
• Paligemma: http://<server-ip>:8000

Use the relevant endpoint for inference tasks and interacting with the models.