An independent research lab building small, specialist, open AI models
for Indian languages and domains.

9 open models on Hugging Face

11 public repos code, harnesses, leaderboards

Sthānika AI builds small, specialist open models and the benchmarks used to judge them, and releases everything with weights, datasets, evaluation code, and papers. Our work focuses on what general-purpose models get wrong about India: its languages, its units and number systems, its scripts, and its public-service domains such as agriculture.

sthanika.ai · Hugging Face · hello@sthanika.ai

Models, datasets, and the code behind them. Weights and datasets live on the Hugging Face Hub; the code that produced them is here on GitHub.

Decision models that return calibrated probabilities for typed questions — choice, yes/no, or score — about text or JSON, without generating text. Training code and evaluation are open.

Reproduce the full benchmark suite locally, or as a single Hugging Face Job, with decision-index.

A 12B crop-advisory model for Indian farmers, fine-tuned on real Kisan Call Centre advisory conversations across 11 Indian languages.

Judged on our own Indic-KCC-Agri-Advisory-Benchmark (1–5 scale, four axes), it more than closes the gap to models twice its size:

+1.14 correctness over the base model it was trained from, and ahead of the 27B baseline on the axis that matters most for advisory quality — at less than half the parameters.

Open-ended agricultural-advisory question answering in 11 Indian languages, built from real farmer questions and the answers given by human agents at India's Kisan Call Centre.

500 questions were sampled once in English, then translated into the other 10 languages — so every language scores the same 500 underlying questions, and cross-language comparisons are apples-to-apples rather than confounded by a different question mix per language.

5,500 rows · 22 baseline models evaluated · 2-stage LLM-judged scoring across 4 axes

⚠️ Benchmark only — not agronomic advice. KCC references are noisy call-centre transcripts; do not act on any answer as farming guidance.

Does your model know where it is?

A benchmark of things every Indian knows and frontier LLMs routinely fumble — lakh/crore arithmetic, Indian digit grouping, state-specific land units (bigha, katha, guntha), traditional mass units, the Indian fiscal year, agricultural crop seasons, government schemes, and structural identifiers (PAN, GSTIN, IFSC, PIN codes).

Every quantitative item has a matched control twin — arithmetically identical but framed internationally — so the score separates missing India-specific knowledge from weak arithmetic. That distinction is the whole point.

552 core items, each reviewed by two people · 19 models evaluated · best model scores just 53.2%

Open evaluation studies on how models handle Indian languages, scripts, and tokenization. Each repo ships the code and configs to reproduce it from a clean clone.

The same question in English, Hinglish, Hindi, and their romanized forms, across 16 open-weight models on MMLU and GSM8K. Scored with a floor-adjusted retention metric so weak models aren't flattered by guessing.

Code-mixing is not the problem. Romanization is. Hinglish is the easiest non-English form for every model on both tasks. Romanized Hindi, which removes both the English words and the Devanagari script, is where models collapse: on MMLU the average model keeps 86.5% of its English headroom on Hinglish but only 41.4% on romanized Hindi. English strength doesn't transfer: the best English model in the study is mid-table on robustness.

Can vision-language models read what India actually looks like: hand-painted shop signage, fare boards, and multi-script hoardings, photographed in the wild rather than scanned from clean pages? Eight open-weight VLMs transcribed all 1,319 test images (24,987 gold text items, 14 scripts) from the BSTD dataset, with identical prompts and programmatic, non-model-judged scoring.

The Qwen2.5-VL models lead (32B: 0.471 fuzzy F1; 7B: 0.466), and the weakest model scores below 0.10. Precision is reported as an upper bound on hallucination, since BSTD's annotations aren't guaranteed exhaustive.

A reproducible evaluation of 18 newer LLMs on MILU (AI4Bharat and IBM's Multi-task Indic Language Understanding benchmark): about 85,000 multiple-choice questions across 8 domains and 11 Indic languages. We run it as adopters, adding models the original paper predates and measured GPU-hours next to every accuracy number.

Along the way, it found a scoring-protocol bug that silently breaks four families of hybrid "thinking" models and would have inverted two models' ranks, and shows a 16.84-point swing for one model (Qwen3.6-27B) from the thinking setting alone.

How many tokens does each model's tokenizer need for the same sentence in English, Hindi, Telugu, Kannada, and Marathi? Measured on the full FLORES+ devtest split (1,012 aligned sentences) and translated into fertility (tokens per word), parity versus English, and serving cost. Everything runs by tokenizing locally or through free token-counting endpoints, so reproducing it costs no generation spend.