A zero-dependency library and test harness that measures how badly a retrieval/QA prompt degrades as you pad it with irrelevant filler text ("the lost in the middle" effect), using a deterministic local scorer so no API key or model call is ever required.
The foundation is a seeded needle/haystack fixture generator and an exact-match scorer, both fully deterministic. On top of that sits a sweep runner that scores a predictor across a range of context (haystack) lengths and records the resulting accuracy curve as plain JSON, so it can be snapshot-tested and later gated on in CI. The scorer sits behind a small pluggable interface, so a real model-backed caller can swap in its own correctness judge without touching the sweep runner or breaking test mode. On top of that, a sparkline renderer turns a curve into a compact ASCII or SVG visualization for logs, docs, or CI output, and a gate helper (library function plus CLI) turns the curve into a pass/fail CI check against configured degradation bounds. Everything is covered by golden-output unit tests.
npm installNo runtime dependencies are required — the project only uses Node.js
built-ins (node:test, node:assert).
import { generateFixture, exactMatch, scoreBatch } from './src/index.js';
// Deterministically build a prompt: a haystack of filler sentences with one
// "needle" sentence inserted at a given depth, plus the question and the
// expected answer.
const fixture = generateFixture({
seed: 'example-seed',
haystackLength: 20, // number of filler sentences (controls prompt length)
needleDepth: 0.5, // 0 = needle at the start, 1 = needle at the end
});
console.log(fixture.prompt); // full prompt text to send to a model
console.log(fixture.question); // question about the needle
console.log(fixture.answer); // expected exact-match answer
// Score a candidate answer against the expected answer.
exactMatch(' Amber-1234. ', 'amber-1234'); // true (case/whitespace insensitive)
// Score a batch of predictions and get an accuracy fraction.
scoreBatch([
{ predicted: 'amber-1234', expected: 'amber-1234' },
{ predicted: 'wrong', expected: 'amber-1234' },
]); // { accuracy: 0.5, results: [true, false] }Sweep a range of context lengths and record an accuracy curve:
import { runSweep, sweepToJSON, lostInMiddleAnswerer } from './src/index.js';
// `answer` is the predictor under test: given a fixture, it returns a
// predicted answer string. Swap in a real model call here; for now,
// `lostInMiddleAnswerer` is a deterministic stand-in that simulates
// accuracy dropping off as filler grows and the needle sits mid-haystack,
// so the sweep can be exercised and tested without any model or API key.
const result = runSweep({
answer: lostInMiddleAnswerer,
contextLengths: [0, 10, 20, 50, 100], // haystack lengths to sweep
trialsPerLength: 8, // fixtures averaged per length
needleDepth: 0.5,
seed: 'my-sweep',
});
console.log(result.points);
// [{ contextLength: 0, accuracy: 1, trials: 8 }, ...]
console.log(sweepToJSON(result)); // same data as a JSON stringrunSweep is deterministic: the same options always produce the same
curve, which is what makes it safe to compare against a golden JSON
snapshot in a test or CI gate. Use writeSweepReport(filePath, result) to
write that JSON straight to a file.
Turn a sweep result into a compact visualization of the degradation curve, either as a one-line Unicode sparkline for logs/terminals or as a small SVG for docs/dashboards. Both are pure functions of the accuracy values, so the same curve always renders to the exact same string - safe to snapshot-test and gate on in CI.
import { renderAsciiSparkline, renderSvgSparkline } from './src/index.js';
renderAsciiSparkline(result); // '██▄▃▃' - one block per context length
renderSvgSparkline(result, { width: 160, height: 32 });
// '<svg xmlns="http://www.w3.org/2000/svg" ...><rect .../>...</svg>'Each accuracy is quantized into 8 levels for the ASCII renderer (▁ lowest ..
█ highest) and into a proportional bar height for the SVG renderer. Either
function also accepts a plain array of accuracy numbers or SweepPoint
objects directly, so they don't require a full SweepResult.
Correctness judging is a small interface - (predicted, expected) => boolean
- not a hardcoded check, so a real model-backed caller can swap in its own
scorer without changing
scoreBatchorrunSweep:
import { runSweep, exactMatch, containsMatch } from './src/index.js';
// Default: strict exact match (after trim/case/whitespace normalization).
// This is what keeps test mode deterministic and API-key-free.
runSweep({ answer: myAnswerer, scorer: exactMatch });
// A real LLM often answers in a full sentence instead of a bare value
// ("The code is AMBER-1234."). containsMatch tolerates that by checking
// whether the expected answer appears anywhere in the prediction.
runSweep({ answer: myLLMAnswerer, scorer: containsMatch });
// Or bring your own: any function with this signature works.
const fuzzyScorer = (predicted, expected) => /* your logic */ true;
runSweep({ answer: myAnswerer, scorer: fuzzyScorer });If scorer is omitted, runSweep and scoreBatch both default to
exactMatch, so existing deterministic test-mode behavior is unchanged.
Check a sweep result against configured degradation bounds and fail the
build if the curve shape falls outside them. checkGate is a pure function
of the curve and the bounds, so it has no network or file access of its own
and is safe to unit test directly:
import { runSweep, checkGate, formatGateReport } from './src/index.js';
const result = runSweep({ answer: myAnswerer });
const gate = checkGate(result, {
minAccuracy: 0.5, // every point must stay at or above this floor
maxDrop: 0.4, // no point may fall more than this far below the
// baseline (the first point in the curve)
maxRelativeDrop: 0.5, // no point may fall below this fraction of baseline
});
console.log(formatGateReport(gate)); // 'PASS: ...' or 'FAIL: N bound(s) ...'
if (!gate.pass) process.exitCode = 1;At least one of minAccuracy, maxDrop, or maxRelativeDrop must be
given. gate.violations lists every bound that was crossed, with the
contextLength and accuracy of the offending point, so a CI log can point
straight at where the curve degraded.
A small CLI wraps the same check around a JSON report written by
writeSweepReport, for use directly in a CI step:
npx contextrot-gate sweep-report.json --min-accuracy 0.5 --max-relative-drop 0.5It prints the curve's ASCII sparkline and a PASS/FAIL report, and exits 0
on pass, 1 when a bound is violated, or 2 on a usage/input error - so a
CI pipeline can wire it in as a regression gate with no extra scripting.
Run the test suite:
npm testThis project is being built autonomously, one milestone at a time, and is gated on its automated test suite passing before any change is kept. Expect incomplete functionality outside of what the current milestone describes.