rag-lab

A benchmark rig that answers one question with measurements instead of folklore: for a Ukrainian-language corpus, does lexical handling and fusion actually change retrieval quality, and by how much?

It crosses exactly two axes over a public Ukrainian Wikipedia sample:

• Retrieval (4) — dense (pgvector, cosine, HNSW), lexical (Postgres tsvector), hybrid via Reciprocal Rank Fusion, and hybrid with tuned weights.
• Ukrainian text handling (3) — raw simple config, Unicode folding with lowercasing, and lemmatised.

That is 12 configurations, and it reports recall@5, nDCG@10, p50/p95 latency, index size and ingest cost for every one of them. Everything else is held fixed: markdown-header-aware chunking throughout, and no reranking. Why those two axes and not four is set out under Not measured here.

The axis that motivated the repository is text handling. Postgres ships no Ukrainian dictionary. The default fallback, to_tsvector('simple', ...), does nothing but lowercase and split on non-letters, which on a heavily inflected language means "Києва", "Києву" and "Києві" are three unrelated lexemes. Teams ship that, see the index build without error, and lose recall silently.

That is the mechanism the benchmark set out to measure. It is not what the run found. On this corpus lemmatising the lexical index recovered nothing — it scored below the naive simple baseline — and dense retrieval on its own saturates the golden set, which leaves the text-handling axis very little room to show an effect either way. The negative result stays in, and what it does and does not license is set out under Known limitations.

Every number below is written by make bench into HTML comment markers. Nothing in a marked block is typed by hand. Configurations that lost are reported as having lost.

Generated by make bench && make report in 52 seconds on Python 3.12.3, Linux-6.8.0-134-generic-x86_64-with-glibc2.39, with RAGLAB_THREADS=2. Corpus and chunk tables occupy 71.1 MB.

---

Results at a glance

• Configurations measured: 12 (24 test questions, 24 held out for tuning).
• Best: dense / simple at nDCG@10 0.969, recall@5 1.000, p50 54 ms.
• Worst: lexical / lemma at nDCG@10 0.726, recall@5 0.875.
• Dense-only baseline: nDCG@10 0.969, recall@5 1.000 -- the best configuration is level on nDCG@10.
• The configuration this repository exists to warn about (to_tsvector('simple', ...), lexical only): nDCG@10 0.782, recall@5 0.958.
• Embedding model intfloat/multilingual-e5-small, markdown_header chunking throughout, no reranking, 2 threads, candidate depth 30, scored at depth 10.

What actually mattered

Ranked by measured effect on nDCG@10, largest first. This list is generated from the run, including the entries that report an idea failing to pay for itself.

1. Retrieval is worth 0.212 nDCG@10. Averaged over everything else, dense scores 0.969 and lexical scores 0.757.
2. Lemmatising Ukrainian loses 0.020 nDCG@10 against to_tsvector('simple', ...). Across every configuration where the lexical index is read, lemma averages 0.886 and simple averages 0.907.
3. Unicode folding is worth -0.006 nDCG@10 (0.901 against 0.907). Postgres's own unaccent() would have been worth exactly zero: its rules cover Latin diacritics and leave Cyrillic untouched.
4. Hybrid via tuned weights is worth -0.002 nDCG@10 over dense alone (0.967 against 0.969).
5. Hybrid via RRF is worth +0.000 nDCG@10 over dense alone (0.969 against 0.969).
6. Lemmatising costs -0.1s of ingest against the naive variant (19.2s against 19.2s), and nothing at query time beyond normalising the query itself.

Best configurations

All twelve rows use markdown_header chunking and no reranking, so neither is a column. * marks a row where the text-handling axis is inert: dense retrieval never reads a tsvector, so its three text-handling variants are one measurement reported three times rather than three independent results.

#	Retrieval	Text handling	recall@5	nDCG@10	p50 ms	p95 ms	Index MB	Ingest s
1	dense	simple*	1.000	0.969	54	95	8.2	1286
2	dense	unaccent*	1.000	0.969	54	95	8.2	1286
3	dense	lemma*	1.000	0.969	54	95	8.2	1286
4	rrf	lemma	1.000	0.969	144	307	13.7	1288
5	weighted	unaccent	1.000	0.969	156	275	16.6	1288

Worst configurations

Included deliberately. A table that only shows winners is not evidence.

	Retrieval	Text handling	recall@5	nDCG@10	p50 ms	p95 ms	Index MB	Ingest s
	lexical	lemma	0.875	0.726	107	261	5.5	19
	lexical	unaccent	0.917	0.763	95	345	8.5	19
	lexical	simple	0.958	0.782	110	282	8.5	19

---

Axis by axis

Each table averages over the other axis. Averages hide interactions, so the per-configuration tables above and the full table below are the authority; these are for seeing which of the two axes is worth attention at all.

Retrieval

Retrieval	Configs	mean recall@5	mean nDCG@10	best nDCG@10	mean p50 ms	mean index MB
dense	3	1.000	0.969	0.969	54	8.2
rrf	3	1.000	0.969	0.969	155	15.7
weighted	3	1.000	0.967	0.969	159	15.7
lexical	3	0.917	0.757	0.782	104	7.5

Ukrainian text handling

Computed over the configurations where the axis does anything — that is, excluding dense-only retrieval, which never touches a tsvector. Including those rows would average three copies of one measurement into every variant and pull them all towards the same number.

Ukrainian text handling	Configs	mean recall@5	mean nDCG@10	best nDCG@10	mean p50 ms	mean index MB
simple	3	0.986	0.907	0.969	145	13.9
unaccent	3	0.972	0.901	0.969	137	13.9
lemma	3	0.958	0.886	0.969	136	10.9

---

Not measured here

Two axes were left out on purpose, and naming them is the point of this section: an omission that is not stated reads as an oversight.

Axis	Options that exist in the code	Why it is not in the table
Chunking strategy	fixed-token, sentence-window, paragraph, markdown-header, embedding-boundary semantic	Budget. Held fixed at markdown_header.
Reranking	none, BAAI/bge-reranker-base cross-encoder, score-threshold cutoff	Budget. Held fixed at none.

The reason in both cases is the run cost. A full cross of chunking x retrieval x text handling x reranking is 5 x 4 x 3 x 3 = 180 configurations, and the cross-encoder alone is roughly two orders of magnitude slower per query than every other stage put together. On the hardware this was built for — 2 vCPU, 3 GB RAM, no GPU — that is a multi-day run to answer a question that was never the one being asked. Twelve configurations over two axes answer the actual question, which is whether Ukrainian lexical handling and fusion move retrieval quality.

Both axes are registries rather than branches, so neither is a rewrite to add:

.venv/bin/python -m raglab.cli ingest --chunkers markdown_header fixed semantic
.venv/bin/python -m raglab.cli bench  --chunkers markdown_header fixed semantic \
                                      --rerankers none cross_encoder

The runner, the schema and the report do not count the axes; they read the registries. What widening the cross costs is time, not code.

Two things this repository therefore does not claim to know: whether header-aware chunking is the best chunker for this corpus, and whether a cross-encoder would pay for its latency here. Both are the obvious next measurements.

---

Ingest cost

Measured during make ingest on the machine described under Provenance. The normalisation column covers all three text variants together; the lemmatiser is essentially all of it.

Chunking	Chunks	mean tok	p95 tok	chunk s	embed s	normalise s	normalise ms/doc	total s	HNSW MB	GIN MB (3 variants)
markdown_header	4179	262	350	17.0	1266.6	4.3	39	1308.2	8.2	22.4

---

Tuning

rrf_k and the weighted-hybrid alpha are fitted on the dev half of the golden set and then frozen. Every number elsewhere in this README comes from the test half, which those parameters never saw.

Fitted on the 24 dev questions, then frozen for every number above.

Parameter	Grid searched	Chosen
rrf_k	10, 20, 40, 60, 100	10
alpha (weight on dense)	0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0	0.6

Dev-set mean nDCG@10 across the grids:

Parameter	Value	dev nDCG@10
rrf_k	10	0.8762
rrf_k	20	0.8762
rrf_k	40	0.8762
rrf_k	60	0.8762
rrf_k	100	0.8762
alpha	0.0	0.6994
alpha	0.1	0.7381
alpha	0.2	0.7547
alpha	0.3	0.7749
alpha	0.4	0.8008
alpha	0.5	0.8644
alpha	0.6	0.8991
alpha	0.7	0.8825
alpha	0.8	0.8695
alpha	0.9	0.8575
alpha	1.0	0.8504

---

Golden set and its bias

48 questions over the public Wikipedia sample, committed as JSONL at golden/ukwiki.golden.jsonl in rag-eval golden format, split 50/50 into dev and test by a hash of the question id. rag-eval's schema is closed, so the split, the question style and the source article travel as tags (split:test, kind:lead, source:ukwiki:123) rather than as fields of their own.

Style	dev	test	total
factoid	5	7	12
lead	4	8	12
lemma	6	6	12
section	9	3	12
total	24	24	48

How they were built

There is no LLM in this project, so the questions were not written by one. They are generated programmatically from the corpus in four styles, filtered, and then read and curated. make goldset regenerates the candidates.

Style	Construction	What it is for
lead	The definition half of an article's first sentence, with the article's own name stripped out	Nothing lexical to match on; dense retrieval should earn its keep here
section	Article title plus a section heading, as a short keyword query	Closest to how people actually search; should favour lexical matching
factoid	A mid-article sentence containing a date or figure, with the figure and the article name removed	Targets content outside the lead
lemma	Content words lifted from an inflected body sentence and written back in dictionary form	Built to exercise the inflection mismatch that to_tsvector('simple', ...) cannot handle

Stated bias

Three biases, none of them incidental:

1. The lemma style favours the lemmatised variant by construction. It writes queries in dictionary form against documents that contain inflected forms. That is a real failure mode and it is the one the repository is about, but a quarter of the golden set being built to exhibit it inflates the lemma variant's average. The per-style breakdown above is what makes the effect visible instead of buried.
2. Questions derive from the articles' own wording, which favours lexical retrieval generally. The lead style, which strips the subject entirely, is the counterweight.
3. Relevance is binary and single-document. Each question is marked relevant to exactly the article it was generated from. A retriever that surfaces a genuinely useful different article is scored as wrong.

Curation rule

Candidates were dropped for intrinsic reasons only:

• shorter than four tokens (two for the keyword-style section queries) or longer than twenty-four;
• fewer than three content words after stop-word removal;
• leaking the article title, except for section queries which name it on purpose;
• duplicating another candidate's set of content words;
• containing no term rarer than a fifth of the corpus. "Тривала з червня до листопада" identifies nothing — every word in it appears nearly everywhere, so no retriever could rank the intended article first and scoring against it measures noise. This filter reads corpus statistics, not retrieval results.

No candidate was ever dropped because retrieval failed on it. Filtering a benchmark by what the system already answers correctly is the most direct way to produce numbers that mean nothing.

Generation is also capped at one question per article, so a handful of very long articles cannot supply a third of the set.

---

Reproducing it

Requires Docker, Docker Compose and Python 3.12. No GPU, no API keys, and no torch — embeddings run on CPU through fastembed's ONNX runtime.

make up       # start Postgres (pgvector) on port 5460 and apply the schema
make ingest   # fetch the public corpus, chunk it, embed and index
make bench    # run the 12-configuration cross, write results/
make report   # rewrite the tables in this README from results/
make test     # unit tests; no database needed

make up brings up this repository's own Postgres on port 5460. It does not touch anything already running on 5432.

The corpus is pinned by revision id in data/ukwiki_manifest.json, so a run today fetches the same article revisions the committed numbers were measured against. corpus/ itself is gitignored; nothing fetched or private is ever committed.

Running it against your own data

export KB_DSN=postgresql://user:pass@host:5432/knowledge
export KB_TABLE=articles KB_BODY_COLUMN=body KB_TITLE_COLUMN=title
make ingest SOURCE=pg

The published golden set and every number in this README come from the public Wikipedia source, not from a private knowledge base.

---

How it is put together

raglab/
  chunking/     five strategies behind one interface; one is measured
  retrieval/    dense, lexical, and two hybrids -- the first measured axis
  rerank/       none, cross-encoder, score threshold; one is measured
  textproc.py   the three Ukrainian text handlings -- the second measured axis
  fusion.py     RRF and weighted fusion, pure functions
  metrics.py    a thin adapter over the rag-eval package
  bench.py      the cross; writes results/
  report.py     results/ -> the tables above

Each axis is a registry keyed by name. Adding an option is one class plus one line in the registry, and the runner picks it up without further changes. That holds for the text-handling axis too: the schema and the insert statement both generate their per-variant tsvector columns from the registry, so a fourth variant needs no SQL edit — only make up to apply the new column and make ingest to populate it.

Design notes

Decisions where more than one answer was defensible. Where the brief was ambiguous, the simpler option was taken.

• Two axes, not four. The reasoning and the cost are in Not measured here. The chunkers and rerankers that are not measured stay in the tree, registered and unit-tested, because the claim that widening the cross is a command-line argument is only worth making if it is true.
• markdown_header is the fixed chunker. The corpus is Wikipedia, which is already sectioned, so header boundaries are real semantic boundaries rather than a guess at where one ought to be. A corpus of chat logs would deserve a different fixed choice, and that choice would need measuring.
• Relevance is judged at document level, and the chunk ranking is collapsed to a document ranking before scoring. Chunk ids are not comparable across chunking strategies, so anything finer would make recall@5 mean a different thing for 250-token fixed chunks than for whole-section markdown ones. The collapse stays even with chunking held fixed, so widening the cross does not change what the metric means.
• Lexical queries use OR semantics, not plainto_tsquery's AND. A natural question is eight to fifteen words; requiring every one to appear in a single chunk returns nothing for most questions, which would make the text handling axis unmeasurable rather than merely poor. ts_rank_cd does the ordering. The same choice applies to all three variants, so the comparison stays fair.
• The middle text-handling variant does not stop at Postgres's unaccent(). Stock unaccent is a verified no-op on Cyrillic — its rules file covers Latin diacritics — and to_tsvector('simple', ...) already lowercases, so "unaccent + lowercase" as literally specified would have been a bit-identical copy of the baseline column. The variant instead performs the folding unaccent would perform if its rules covered Ukrainian: Unicode combining marks stripped (merging й/и and ї/і), ґ folded to г, apostrophes removed. The no-op itself is asserted in tests/test_integration_db.py.
• simplemma over pymorphy3. Pure Python, installs cleanly on 3.12, no separate dictionary package, and a single lemmatize(token, lang) call. pymorphy3 returns full morphological parses this benchmark would discard. Its weakness is documented and tested: simplemma lemmatises common nouns reliably but is inconsistent on proper nouns — it maps "Києва" and "Києву" to "Київ" but "Києві" and even the nominative "Київ" to "кий". Place and person names are exactly what questions contain most, which caps what the lemma variant can recover.
• One global alpha and rrf_k, not one per text handling. With roughly two dozen dev questions, fitting three separate alphas would be fitting noise.
• Indexes are partial, one per chunking strategy. That is what lets the index-size column be a real per-configuration measurement rather than a shared total divided several ways, and it is what makes a widened cross chargeable.
• Nearest-rank percentiles, not interpolated. With a couple of dozen samples per configuration, interpolation invents precision the sample size cannot carry. rag-eval ships an interpolated percentile, so this one is rag-lab's own and is the single numeric thing raglab/metrics.py still computes.
• intfloat/multilingual-e5-small registered as a custom fastembed model. fastembed's built-in catalogue has no multilingual model under 1 GB — the smallest is e5-large at 2.24 GB. intfloat publishes an ONNX export of the small variant (384 dimensions, ~0.45 GB), so it is registered directly rather than settling for a weaker encoder. The passage: and query: prefixes e5 was trained with are applied; dropping them costs real recall.
• rag-eval is pinned to a commit, not a tag or a branch. The numbers here are only reproducible if the code that computed them is. The pin lives in requirements.txt.

Known limitations

• Roughly two dozen test questions is a small sample. Differences below about 0.03 nDCG@10 between neighbouring configurations should not be treated as real. The measured spread across the three text handlings is smaller than that, so this run does not establish an ordering between them — it only rules out the large gain the lemmatised variant was built to demonstrate.
• The golden set is too easy for dense retrieval, and that is the main thing limiting this run. Nine of the twelve configurations score recall@5 of 1.000. Once the dense retriever returns the intended article for every question, no fusion scheme and no text handling can improve on it, and the benchmark measures a ceiling rather than a difference. The questions were generated from the articles' own sentences, which is what makes them recoverable by a multilingual embedding model. A harder golden set — paraphrased questions, or several plausible articles per question — is the change that would make the two axes separable at all, and it is a larger piece of work than rerunning the cross.
• The corpus is Wikipedia featured articles: long, well structured and cleanly punctuated. That flatters header-aware chunking specifically, and since chunking is held fixed rather than measured, the benchmark cannot say by how much.
• The golden set has one relevant document per question, so recall@5 is a hit-rate and nDCG@10 has little room to distinguish orderings past the first hit.
• Three of the twelve rows — dense retrieval across the three text handlings — are one measurement reported three times. They are flagged * and excluded from the text-handling averages, but they still occupy a quarter of the table.

---

Full results

Every configuration measured. Also written to results/results.md and results/results.csv.

All 12 configurations

	Retrieval	Text handling	recall@5	nDCG@10	p50 ms	p95 ms	Index MB	Ingest s
	dense	lemma*	1.000	0.969	54	95	8.2	1286
	dense	simple*	1.000	0.969	54	95	8.2	1286
	dense	unaccent*	1.000	0.969	54	95	8.2	1286
	lexical	lemma	0.875	0.726	107	261	5.5	19
	lexical	simple	0.958	0.782	110	282	8.5	19
	lexical	unaccent	0.917	0.763	95	345	8.5	19
	rrf	lemma	1.000	0.969	144	307	13.7	1288
	rrf	simple	1.000	0.969	159	342	16.6	1288
	rrf	unaccent	1.000	0.969	160	331	16.6	1288
	weighted	lemma	1.000	0.964	158	306	13.7	1288
	weighted	simple	1.000	0.969	165	322	16.6	1288
	weighted	unaccent	1.000	0.969	156	275	16.6	1288

---

Licence

MIT.