Implementation of PagedAttention from vLLM paper - a breakthrough attention algorithm that treats KV cache like virtual memory. Eliminates memory fragmentation, increases batch sizes, and dramatically improves LLM serving throughput.
Research harness for evaluating query-time bounded elimination of reconstructable KV-cache witnesses in long-context transformer inference workloads. Related provisional filing: IN 202641062451.
Add a description, image, and links to the transformer-optimization topic page so that developers can more easily learn about it.
To associate your repository with the transformer-optimization topic, visit your repo's landing page and select "manage topics."