Smarter repetition handling for small models: DRY, XTC and typical sampling

[kfaracik]

Summary

repetitionPenalty is currently the only tool the library offers against repetition, and it is the wrong shape for the job: it is a blunt, context-blind logit divisor that cannot tell a degeneration loop from a word the answer legitimately needs twice. On small quantized models — the ones this library exists to run — that leaves us choosing between loops and damaged answers.

We would like to see modern, repetition-aware sampling in react-native-executorch: DRY, XTC and locally typical sampling. All three are implemented in llama.cpp and text-generation-webui, with settled defaults and a settled ordering, so there is a reference to follow rather than a design to invent.

There is also a concrete defect in the existing implementation, described at the end. It is the reason we cannot use the current knob at all, but it is separable — happy to split it into its own issue if you prefer.

Why the current penalty cannot solve this

repetition_penalty divides the logit of every token seen so far by a constant. It has no notion of sequence, so it cannot distinguish:

• Zmiesz wszystkie składniki, po czym zastosuj ich na zimnym kremie. repeated as steps 3, 5, 7, 9, 11, 13, 15, 17 of a recipe — a loop;
• a RAG answer that must say "30 dni rocznie" twice because the question asked about two departments — not a loop.

Raising the penalty enough to stop the first reliably damages the second. That is not a tuning failure; the mechanism has no information to separate them. A sequence-aware penalty does.

What we are asking for

1. DRY — sequence-aware repetition penalty

oobabooga/textgen#5677 (merged 2024-05-20), also in llama.cpp as llama_sampler_init_dry.

DRY penalizes a token in proportion to the length of the n-gram it would extend. If the tail of the context matches an earlier sequence of length n, the candidate that would continue that match is penalized by

multiplier * base ^ (n - allowed_length)

with no penalty below allowed_length. Defaults: multiplier 2.0, base 1.75, allowed_length 2. Sequence breakers (\n, :, ", * by default) reset matching so chat templates and list markers do not poison it.

This is the one we want most. It targets verbatim looping directly and leaves single legitimate repeats alone, which is exactly the boundary the current penalty cannot see.

2. XTC — non-verbatim repetition

oobabooga/textgen#6335 (merged 2024-09-28), also in llama.cpp as llama_sampler_init_xtc.

With probability xtc_probability (default 0.5), when at least two tokens exceed xtc_threshold (default 0.1), XTC removes all of them except the least likely. It inverts the usual truncation logic and breaks structural repetition — the model restating the same idea in the same shape — which no repetition penalty detects because no tokens actually repeat.

We see this shape often: a model answering the same question three times in slightly different words, each phrasing individually fine.

3. Locally typical sampling

Meister et al., arXiv:2202.00666, exposed as typical_p in transformers and typ_p in llama.cpp.

Truncates to tokens whose information content is close to the conditional entropy of the distribution, rather than to the most probable ones. The paper reports competitive quality on summarization and story generation while consistently reducing degenerate repetition against nucleus and top-k. It is a small addition next to the existing mask_topp / mask_topk / apply_min_p and complements the other two rather than competing with them.

Ordering

llama.cpp's default chain is penalties → dry → top_n_sigma → top_k → typ_p → top_p → min_p → xtc → temperature. Following it would save everyone from rediscovering which order works.

Why this matters more on-device than on a server

The models this library targets are the ones that degenerate most. From 259 stored answers on a physical device (Qwen 3 1.7B, production build):

• 12 (4.6%) were verbatim loops. Three of the four longest replies in the whole set were loops, not long answers.
• A loop runs to the token limit. On a phone that is the user's battery and 30+ seconds of their time, spent producing nothing. Where a server would waste some compute, here it visibly burns the device.

Two examples, both from that set:

3. Zmiesz wszystkie składniki, po czym zastosuj ich na zimnym kremie.
4. Przygotuj wypierdzone zimne cebulki, a następnie wyporządkuj wszystkie składniki...
5. Zmiesz wszystkie składniki, po czym zastosuj ich na zimnym kremie.
6. Przygotuj wypierdzone zimne cebulki, a następnie wyporządkuj wszystkie składniki...
        ... alternating to step 18

Lecie w przestrzeni, pośród gwiazd i planet. Lecie w przestrzeni, pośród gwiazd
i planet. Lecie w przestrzeni, pośród gwiazd i planet. ... (4.8 kB, to the cap)

Both are exactly what DRY's n-gram matching is built to stop, and both are invisible to a constant logit divisor at any value that does not also wreck grounded answers.

We currently mitigate this after generation, by detecting the repeated unit in the finished text and cutting it. That works, but it is a bandage: the loop has already consumed the full token budget by the time we can see it. Prevention belongs in the sampler.

The defect in the current implementation

Worth fixing regardless of whether the above lands, because it is why we run with repetitionPenalty disabled everywhere.

Sampler::apply_repetition_penalty (packages/react-native-executorch/legacy/cpp/runner/sampler.h) iterates recent_tokens and divides once per occurrence:

for (uint64_t id : recent_tokens) {
  ...
  val = static_cast<T>(static_cast<float>(val) / repetition_penalty_);

and recent_tokens is not recent — TextTokenGenerator::generate seeds it with the entire prompt:

std::vector<uint64_t> generated_tokens(tokens.begin(), tokens.end());

So a token appearing N times in the prompt has its logit divided by penalty^N. At penalty = 1.1, 20 occurrences means ~6.7x suppression; 50 means ~117x. The high-count tokens in a RAG prompt are precisely the key terms of the retrieved passage.

Two differences from the reference implementations:

• transformers' RepetitionPenaltyLogitsProcessor gathers at the input ids and scatters back, so each distinct token is penalized once, not once per occurrence;
• llama.cpp penalizes a window (repeat_last_n) of trailing tokens, not the whole context.

Measured, iPhone 17 Pro simulator, Qwen 3 1.7B, same document, same question, retrieval verified identical between runs:

repetitionPenalty	runs	result
1.1	2/2	wrong figures, an invented institution name, Cyrillic characters leaking into Polish text, one run also looping a phrase 5x
1 (off)	2/2	correct, both facts quoted from the passage

A short prompt with no retrieved context stayed coherent at 1.1 — consistent with the mechanism, since with nothing repeated in the prompt there is nothing to compound.

Introduced in #1099. Still present on main as of 2026-09-22.

SamplerTest.cpp already covers the penalty (RepetitionPenaltyReducesPositiveLogit, RepetitionPenaltyMultipliesNegativeLogit, RepetitionPenaltyNoRecentTokensHasNoEffect), but every case passes a short recent_tokens, so neither behaviour is visible. Two cases would pin a fix: a token repeated N times penalized the same as one appearing once, and a token outside the window not penalized at all.

One thing to decide

The sampler lives only in legacy/cpp/runner/; the newer cpp/ tree has none. If the legacy runner is on its way out, this issue is really a request for the new one's sampling design, and the defect above is the argument for not carrying the current semantics over.

Happy to contribute

We already run the window + once-per-distinct-token fix as a local patch-package patch and can open that as a PR with the two unit tests. If DRY is something you would take, we are willing to work on it — tell us the shape you want (config surface, whether sequence breakers are token ids or strings) and we will follow it.

Environment

• react-native-executorch 0.9.2, behaviour verified unchanged on main
• iOS 26 simulator, Android physical device (Pixel 10), XNNPACK
• Qwen 3 1.7B / 0.6B, Qwen 2.5 0.5B, Bielik v3.0 1.5B, Gemma 4 2B
• Reported from Private Mind

[msluszniak]

@barhanc, I guess we can add these mechanisms to new implementation of llm runner.

[kfaracik]

Cross-link for anyone finding this from the other side: we track the downstream half at software-mansion-labs/private-mind#255 (the user-visible symptom, with on-device measurements) and software-mansion-labs/private-mind#375 (whether we carry a local patch-package sampler fix in the meantime). Both public.

Our current position is that we run with repetitionPenalty disabled everywhere and clean loops up after generation, which we would rather stop doing.

[msluszniak]

Thank you :))