Skip to content

About

A zero-dependency library and test harness that measures how badly a retrieval/QA prompt template degrades as you pad it with irrelevant filler, using a deterministic local scorer so no API key is…

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

5 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

contextrot

A zero-dependency library and test harness that measures how badly a retrieval/QA prompt degrades as you pad it with irrelevant filler text ("the lost in the middle" effect), using a deterministic local scorer so no API key or model call is ever required.

The foundation is a seeded needle/haystack fixture generator and an exact-match scorer, both fully deterministic. On top of that sits a sweep runner that scores a predictor across a range of context (haystack) lengths and records the resulting accuracy curve as plain JSON, so it can be snapshot-tested and later gated on in CI. The scorer sits behind a small pluggable interface, so a real model-backed caller can swap in its own correctness judge without touching the sweep runner or breaking test mode. On top of that, a sparkline renderer turns a curve into a compact ASCII or SVG visualization for logs, docs, or CI output, and a gate helper (library function plus CLI) turns the curve into a pass/fail CI check against configured degradation bounds. Everything is covered by golden-output unit tests.

Install

npm install

No runtime dependencies are required — the project only uses Node.js built-ins (node:test, node:assert).

Usage

import { generateFixture, exactMatch, scoreBatch } from './src/index.js';

// Deterministically build a prompt: a haystack of filler sentences with one
// "needle" sentence inserted at a given depth, plus the question and the
// expected answer.
const fixture = generateFixture({
  seed: 'example-seed',
  haystackLength: 20,   // number of filler sentences (controls prompt length)
  needleDepth: 0.5,     // 0 = needle at the start, 1 = needle at the end
});

console.log(fixture.prompt);    // full prompt text to send to a model
console.log(fixture.question);  // question about the needle
console.log(fixture.answer);    // expected exact-match answer

// Score a candidate answer against the expected answer.
exactMatch(' Amber-1234. ', 'amber-1234'); // true (case/whitespace insensitive)

// Score a batch of predictions and get an accuracy fraction.
scoreBatch([
  { predicted: 'amber-1234', expected: 'amber-1234' },
  { predicted: 'wrong', expected: 'amber-1234' },
]); // { accuracy: 0.5, results: [true, false] }

Sweep a range of context lengths and record an accuracy curve:

import { runSweep, sweepToJSON, lostInMiddleAnswerer } from './src/index.js';

// `answer` is the predictor under test: given a fixture, it returns a
// predicted answer string. Swap in a real model call here; for now,
// `lostInMiddleAnswerer` is a deterministic stand-in that simulates
// accuracy dropping off as filler grows and the needle sits mid-haystack,
// so the sweep can be exercised and tested without any model or API key.
const result = runSweep({
  answer: lostInMiddleAnswerer,
  contextLengths: [0, 10, 20, 50, 100], // haystack lengths to sweep
  trialsPerLength: 8,                   // fixtures averaged per length
  needleDepth: 0.5,
  seed: 'my-sweep',
});

console.log(result.points);
// [{ contextLength: 0, accuracy: 1, trials: 8 }, ...]

console.log(sweepToJSON(result)); // same data as a JSON string

runSweep is deterministic: the same options always produce the same curve, which is what makes it safe to compare against a golden JSON snapshot in a test or CI gate. Use writeSweepReport(filePath, result) to write that JSON straight to a file.

Sparkline renderers

Turn a sweep result into a compact visualization of the degradation curve, either as a one-line Unicode sparkline for logs/terminals or as a small SVG for docs/dashboards. Both are pure functions of the accuracy values, so the same curve always renders to the exact same string - safe to snapshot-test and gate on in CI.

import { renderAsciiSparkline, renderSvgSparkline } from './src/index.js';

renderAsciiSparkline(result); // '██▄▃▃' - one block per context length

renderSvgSparkline(result, { width: 160, height: 32 });
// '<svg xmlns="http://www.w3.org/2000/svg" ...><rect .../>...</svg>'

Each accuracy is quantized into 8 levels for the ASCII renderer (▁ lowest .. █ highest) and into a proportional bar height for the SVG renderer. Either function also accepts a plain array of accuracy numbers or SweepPoint objects directly, so they don't require a full SweepResult.

Pluggable scorer

Correctness judging is a small interface - (predicted, expected) => boolean

  • not a hardcoded check, so a real model-backed caller can swap in its own scorer without changing scoreBatch or runSweep:
import { runSweep, exactMatch, containsMatch } from './src/index.js';

// Default: strict exact match (after trim/case/whitespace normalization).
// This is what keeps test mode deterministic and API-key-free.
runSweep({ answer: myAnswerer, scorer: exactMatch });

// A real LLM often answers in a full sentence instead of a bare value
// ("The code is AMBER-1234."). containsMatch tolerates that by checking
// whether the expected answer appears anywhere in the prediction.
runSweep({ answer: myLLMAnswerer, scorer: containsMatch });

// Or bring your own: any function with this signature works.
const fuzzyScorer = (predicted, expected) => /* your logic */ true;
runSweep({ answer: myAnswerer, scorer: fuzzyScorer });

If scorer is omitted, runSweep and scoreBatch both default to exactMatch, so existing deterministic test-mode behavior is unchanged.

CI gate

Check a sweep result against configured degradation bounds and fail the build if the curve shape falls outside them. checkGate is a pure function of the curve and the bounds, so it has no network or file access of its own and is safe to unit test directly:

import { runSweep, checkGate, formatGateReport } from './src/index.js';

const result = runSweep({ answer: myAnswerer });

const gate = checkGate(result, {
  minAccuracy: 0.5,      // every point must stay at or above this floor
  maxDrop: 0.4,           // no point may fall more than this far below the
                          // baseline (the first point in the curve)
  maxRelativeDrop: 0.5,  // no point may fall below this fraction of baseline
});

console.log(formatGateReport(gate)); // 'PASS: ...' or 'FAIL: N bound(s) ...'
if (!gate.pass) process.exitCode = 1;

At least one of minAccuracy, maxDrop, or maxRelativeDrop must be given. gate.violations lists every bound that was crossed, with the contextLength and accuracy of the offending point, so a CI log can point straight at where the curve degraded.

A small CLI wraps the same check around a JSON report written by writeSweepReport, for use directly in a CI step:

npx contextrot-gate sweep-report.json --min-accuracy 0.5 --max-relative-drop 0.5

It prints the curve's ASCII sparkline and a PASS/FAIL report, and exits 0 on pass, 1 when a bound is violated, or 2 on a usage/input error - so a CI pipeline can wire it in as a regression gate with no extra scripting.

Run the test suite:

npm test

Status

This project is being built autonomously, one milestone at a time, and is gated on its automated test suite passing before any change is kept. Expect incomplete functionality outside of what the current milestone describes.

About

A zero-dependency library and test harness that measures how badly a retrieval/QA prompt template degrades as you pad it with irrelevant filler, using a deterministic local scorer so no API key is…

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages