This repository presents a comparative evaluation framework for Sanskrit image captioning:
-
π§ Custom Model (CNN-RNN based)
-
π€ Vision-Language Models (VLMs)
- PaliGemma
- Qwen2-VL
- LLaVA
- BLIP variants
The goal is to evaluate:
βοΈ Can large pre-trained VLMs match domain-specific models in low-resource languages like Sanskrit?
.
βββ data/
β βββ images/ # Test images (600 samples)
β βββ ground_truth.txt # 5 Sanskrit captions per image
β
βββ VLM_Results/
β βββ PaliGemma/
β βββ Qwen2VL/
β βββ Nalanda/
β βββ Chitralekha/
β βββ evaluation_all_models.json
β βββ plots/
β
βββ CNN_Model_Results/
β βββ evaluation_cnn.json
β
βββ notebooks/
β βββ vlm_evaluation.ipynb
β βββ cnn_model_inference.ipynb
β
βββ README.md
-
Image dataset (Flickr8k-based Sanskrit dataset)
-
Each image has:
- πΌοΈ 1 image
- π 5 ground-truth Sanskrit captions
Dataset size:
- Train: 6,691 images
- Validation: 800 images
- Test: 600 images
Image β Caption Generation β Save Predictions β Evaluation β Metrics β Plots
Two parallel pipelines:
- Input image
- Generate Sanskrit caption
- Save predictions
-
Select n examples (n = 5, 10, 15, 20)
-
Provide:
- Image + Sanskrit captions (few-shot context)
-
Generate caption for unseen images
Each model produces:
model_name_n5.txt
model_name_n10.txt
model_name_n15.txt
model_name_n20.txt
Format:
image_name.jpg | generated_caption
Additional outputs:
evaluation_*.jsonβ all metric scoresplots/β visual analysissplit_info.jsonβ few-shot sampling details
All models are evaluated against 5 reference captions per image.
- BLEU-1 / BLEU-2 / BLEU-3 / BLEU-4
- ROUGE-L
- METEOR
- CIDEr
These are standard metrics used in image captioning research to measure:
- lexical similarity
- fluency
- semantic overlap
- n β {5, 10, 15, 20}
- Random sampling with fixed seed
- Remaining images used for evaluation
-
Multi-reference scoring (5 GT captions per image)
-
Invalid outputs filtered:
- empty captions
- errors
- non-Sanskrit outputs
-
Achieved:
- BLEU-1 β 77.8%
-
Outperformed all tested VLMs on Sanskrit captioning
β Strong:
- grammatical correctness
- semantic alignment
- consistent Sanskrit output
| Model | Output Quality | Issues Observed |
|---|---|---|
| LLaVA | Moderate | Works in few-shot |
| PaliGemma | Partial | Needs high n (n=20) |
| Qwen2-VL | Unstable | Detection mode, Hindi output |
| BLIP / GIT | Failed | English output only |
-
β VLMs fail in low-resource Sanskrit
-
β Strong English models β multilingual capability
-
β Few-shot prompting is not enough
-
β Task-specific training significantly improves performance
-
β Attention-based CNN models perform best
-
β Domain-specific datasets are critical
Generated plots include:
- π Metric-wise bar charts
- π BLEU score comparison
- π ROUGE / METEOR / CIDEr trends
- π₯ Heatmap of all metrics
- Fixed random seed = 42
- Consistent dataset split
- Same evaluation metrics across models
- Standardized output format
β‘ Pre-trained VLMs do NOT replace domain-specific models in low-resource languages.
Your custom Sanskrit captioning model:
- Learns linguistic structure
- Adapts to morphology
- Produces meaningful captions
While VLMs:
- Struggle with script consistency
- Fail to generalize to Sanskrit
- Often revert to English/Hindi
- Add CLIPScore / SPICE metrics
- Improve VLM prompting strategies
- Train multilingual VLMs for Sanskrit
- Expand dataset beyond Flickr8k
This repository is based on the research paper: π Beyond Pre-trained VLMs: A CNN-RNN Framework for Low-Resource Sanskrit Image Captioning