Skip to content
akashnathaiPublic

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

Β 

History

6 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 

Repository files navigation

πŸ“Έ Sanskrit Image Captioning β€” CNN vs Vision-Language Models (VLMs)

πŸš€ Overview

This repository presents a comparative evaluation framework for Sanskrit image captioning:

  • 🧠 Custom Model (CNN-RNN based)

  • πŸ€– Vision-Language Models (VLMs)

    • PaliGemma
    • Qwen2-VL
    • LLaVA
    • BLIP variants

The goal is to evaluate:

βš–οΈ Can large pre-trained VLMs match domain-specific models in low-resource languages like Sanskrit?


πŸ“‚ Repository Structure

.
β”œβ”€β”€ data/
β”‚   β”œβ”€β”€ images/                  # Test images (600 samples)
β”‚   └── ground_truth.txt         # 5 Sanskrit captions per image
β”‚
β”œβ”€β”€ VLM_Results/
β”‚   β”œβ”€β”€ PaliGemma/
β”‚   β”œβ”€β”€ Qwen2VL/
β”‚   β”œβ”€β”€ Nalanda/
β”‚   β”œβ”€β”€ Chitralekha/
β”‚   β”œβ”€β”€ evaluation_all_models.json
β”‚   └── plots/
β”‚
β”œβ”€β”€ CNN_Model_Results/
β”‚   └── evaluation_cnn.json
β”‚
β”œβ”€β”€ notebooks/
β”‚   β”œβ”€β”€ vlm_evaluation.ipynb
β”‚   └── cnn_model_inference.ipynb
β”‚
└── README.md

πŸ”„ Pipeline

🧾 Input

  • Image dataset (Flickr8k-based Sanskrit dataset)

  • Each image has:

    • πŸ–ΌοΈ 1 image
    • πŸ“ 5 ground-truth Sanskrit captions

Dataset size:

  • Train: 6,691 images
  • Validation: 800 images
  • Test: 600 images

βš™οΈ Processing Flow

Image β†’ Caption Generation β†’ Save Predictions β†’ Evaluation β†’ Metrics β†’ Plots

Two parallel pipelines:

1️⃣ Custom Model Pipeline

  • Input image
  • Generate Sanskrit caption
  • Save predictions

2️⃣ VLM Pipeline (Few-shot)

  • Select n examples (n = 5, 10, 15, 20)

  • Provide:

    • Image + Sanskrit captions (few-shot context)
  • Generate caption for unseen images


πŸ“€ Output

Each model produces:

model_name_n5.txt
model_name_n10.txt
model_name_n15.txt
model_name_n20.txt

Format:

image_name.jpg | generated_caption

Additional outputs:

  • evaluation_*.json β†’ all metric scores
  • plots/ β†’ visual analysis
  • split_info.json β†’ few-shot sampling details

πŸ“Š Evaluation Metrics

All models are evaluated against 5 reference captions per image.

Metrics Used:

  • BLEU-1 / BLEU-2 / BLEU-3 / BLEU-4
  • ROUGE-L
  • METEOR
  • CIDEr

These are standard metrics used in image captioning research to measure:

  • lexical similarity
  • fluency
  • semantic overlap

⚠️ Note: These metrics may not fully capture human-level quality but remain standard benchmarks.


πŸ“ˆ Experimental Setup

Few-shot Evaluation

  • n ∈ {5, 10, 15, 20}
  • Random sampling with fixed seed
  • Remaining images used for evaluation

Evaluation Strategy

  • Multi-reference scoring (5 GT captions per image)

  • Invalid outputs filtered:

    • empty captions
    • errors
    • non-Sanskrit outputs

πŸ“Š Key Results

🧠 Custom CNN Model

  • Achieved:

    • BLEU-1 β‰ˆ 77.8%
  • Outperformed all tested VLMs on Sanskrit captioning

βœ” Strong:

  • grammatical correctness
  • semantic alignment
  • consistent Sanskrit output

πŸ€– VLM Performance Summary

Model Output Quality Issues Observed
LLaVA Moderate Works in few-shot
PaliGemma Partial Needs high n (n=20)
Qwen2-VL Unstable Detection mode, Hindi output
BLIP / GIT Failed English output only

🚨 Key Findings

  • ❌ VLMs fail in low-resource Sanskrit

  • ❌ Strong English models β‰  multilingual capability

  • ❌ Few-shot prompting is not enough

  • βœ… Task-specific training significantly improves performance

  • βœ… Attention-based CNN models perform best

  • βœ… Domain-specific datasets are critical


πŸ“‰ Visualization Outputs

Generated plots include:

  • πŸ“Š Metric-wise bar charts
  • πŸ“ˆ BLEU score comparison
  • πŸ“‰ ROUGE / METEOR / CIDEr trends
  • πŸ”₯ Heatmap of all metrics

πŸ§ͺ Reproducibility

  • Fixed random seed = 42
  • Consistent dataset split
  • Same evaluation metrics across models
  • Standardized output format

πŸ’‘ Key Takeaway

⚑ Pre-trained VLMs do NOT replace domain-specific models in low-resource languages.

Your custom Sanskrit captioning model:

  • Learns linguistic structure
  • Adapts to morphology
  • Produces meaningful captions

While VLMs:

  • Struggle with script consistency
  • Fail to generalize to Sanskrit
  • Often revert to English/Hindi

πŸ“Œ Future Work

  • Add CLIPScore / SPICE metrics
  • Improve VLM prompting strategies
  • Train multilingual VLMs for Sanskrit
  • Expand dataset beyond Flickr8k

πŸ“Ž Reference

This repository is based on the research paper: πŸ“„ Beyond Pre-trained VLMs: A CNN-RNN Framework for Low-Resource Sanskrit Image Captioning

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors