CacheQuake

An interactive explainer on Key–Value Caching, Limitations, and Alternate Approaches Built for DataForge 2026 — Pathway x Rime Track

🔗 Live demo: https://cachequake.vercel.app

🔗 Backend API: https://cachequake-backend.onrender.com

📄 One-page concept summary: docs/one_page_summary.md

📝 Blog writeup: docs/blog.md

---

The claim

A Transformer's KV cache grows linearly with every token it has ever seen because it stores an exact copy of the past; eviction and compression trade that exactness for a bounded budget, and architectures like BDH remove the growth altogether by replacing the cache with a fixed-size associative state that overwrites itself instead of appending.

CacheQuake lets you test this sentence directly: pick a cache policy, set a memory budget, and watch a real, small, working Transformer retrieve (or fail to retrieve) facts buried in a long sequence — live, not scripted.

Who this is for

Audience: anyone with a working understanding of attention and autoregressive generation (a data scientist, ML engineer, or advanced student) who has never had to reason about why KV caching exists or what eviction/compression actually trade away.

Prerequisites: you should already know what attention computes and what "decoding one token at a time" means. Nothing about BDH, cache eviction, or fixed-size recurrent state is assumed.

Learning objectives — after using CacheQuake, you should be able to:

1. Explain why autoregressive decoding needs a KV cache at all.
2. State why cache size scales with sequence length, not model size.
3. Predict what eviction and compression trade away, and where that trade-off breaks down.
4. Explain how BDH's fixed-shape recurrent state sidesteps the growth problem, and what it gives up in return.
5. Name at least one real limitation or common misconception about each approach.

What's live vs. precomputed vs. illustrative

We take this distinction seriously — nothing here is presented as "real" unless it genuinely is.

Component	Status
The toy Transformer's forward pass and KV cache	Live — real computation, every time you interact with it
full_cache, sliding_window, heavy_hitter, bdh_inspired_state policies	Live — genuinely computed per request, not precomputed lookups
data/precomputed/accuracy_vs_budget.json	Precomputed by us — our own toy simulator's results, generated ahead of time for speed, clearly labeled precomputed_by_us
data/precomputed/bdh_published_claims.json	Published claim, not reproduced by us — sourced directly from the BDH paper (arXiv:2509.26507), labeled published_claim_not_reproduced_by_us, never presented as something we measured

⚠️ Important: what "BDH" means in this project

BDH (Dragon Hatchling) is Pathway's own real, published architecture (arXiv:2509.26507). We do not train it, reproduce it, or run an official BDH checkpoint anywhere in this project — that's explicitly outside the scope of this track.

What we do build is our own small toy cache policy, bdh_inspired_state, that borrows BDH's core idea — a fixed-size state that gets overwritten via a Hebbian-style update rule instead of a growing list of past tokens. It is a simplified, independently-written layer inspired by that idea, not a reimplementation of BDH itself. Every place it appears in this codebase, the UI, and this README says so explicitly. If you find a place where that distinction isn't clear, please open an issue — that's a bug in our documentation, not a caveat we're trying to bury.

The KV-cache landscape

The topic brief asks us to compare eviction, compression, retrieval, linear attention, state-space, and fixed-size recurrent alternatives. CacheQuake simulates two of these live — eviction (via sliding-window and heavy-hitter policies) and fixed-size recurrent (via the BDH-inspired toy layer) — and discusses/cites the other four rather than building live demos for all six, in keeping with the brief's guidance to build one focused artifact rather than a dashboard covering an entire field.

• Compression — quantizing or low-rank-approximating stored K/V values instead of dropping them outright, trading precision for smaller memory footprint per token.
• Retrieval — treating the cache as a searchable index and pulling in only the tokens relevant to the current query, rather than keeping a fixed recency- or score-based set.
• Linear attention — replacing softmax attention with a kernel formulation that lets the model update a running summary incrementally, avoiding storing raw per-token K/V at all.
• State-space models — compressing the sequence into an evolving state vector via structured recurrence (distinct from BDH, which is explicitly not an SSM in the Mamba sense — see the BDH paper's own framing).

(Full citations for each of these are in research/citations.md.)

Try it yourself

1. Open the live demo — no sign-in required.
2. Start with the guided walkthrough — it opens with a preset already running.
3. Switch cache policies and watch the memory chart, cache heat-map, and retrieval accuracy change in real time.
4. Jump into sandbox mode once you've got the idea, and try to break the claim.

Running it locally

Full instructions are in backend/SETUP.md and frontend/README.md. Quick version:

# Backend
cd backend
python -m venv .venv
.venv\Scripts\activate          # Windows
# source .venv/bin/activate     # macOS/Linux
pip install -r requirements.txt
python -m pytest tests -v       # should show all tests passing, 0 failed
uvicorn app.main:app --reload

# Frontend, in a separate terminal
cd frontend
npm install
npm run dev

Architecture

See ARCHITECTURE.md for the full breakdown. Short version: a React frontend talks to a FastAPI backend running a real toy Transformer; precomputed data lives in its own top-level folder, physically separated from the live computation path, so what's live vs. shipped-as-data is structural, not just a claim in this README.

CacheQuake/
├── frontend/                # React UI
├── backend/                 # FastAPI + toy Transformer + cache policies
├── data/precomputed/        # Labeled non-live data only
├── research/                # Paper notes + citations
├── docs/                    # Submission deliverables (one-pager, blog, licenses, AI disclosure)
└── assets/                  # Diagrams, fonts

Primary sources

At least three recent (2022–2026) papers this project draws on directly, cited beside the specific claims they support throughout the codebase and docs:

• Zhang et al., H2O: Heavy-Hitter Oracle for Efficient Generative Inference of LLMs, arXiv:2306.14048 (NeurIPS 2023)
• Xiao et al., Efficient Streaming Language Models with Attention Sinks (StreamingLLM), arXiv:2309.17453 (ICLR 2024)
• Kosowski et al., The Dragon Hatchling: The Missing Link Between the Transformer and Models of the Brain, arXiv:2509.26507

Full list with claim-level citations in research/citations.md.

Known limitations

• full_cache exceeds our sub-1-second interaction target at longer sequence lengths (seq_len=128); documented in backend/PERFORMANCE_NOTES.md rather than hidden.
• The toy model is trained only on our synthetic needle-in-haystack task at small scale — its numbers demonstrate the mechanism, not production-scale KV-cache behavior.
• bdh_inspired_state's update rule is our own simplified design, not a validated reproduction of BDH's actual training dynamics.

AI assistance disclosure

This project was built with AI-assisted coding, research, and writing throughout. Full disclosure of what was AI-generated vs. human-written/reviewed is in docs/AI_DISCLOSURE.md.

License & credits

See docs/LICENSES.md for the full source and license record covering code, data, and any reused assets.

---

Built for DataForge 2026, Pathway x Rime Track — topic: Key–Value Caching, Limitations, and Alternate Approaches.

Team : Game of Codes

Barnali Tanti

Md Aftab Hossain

Sangramjeet Choudhury

Jayita Jana