Skip to content

Commit 965d076

Browse files
hlin99bigPYJ1151
andauthored
[CPU] Specify required KV cache layout for CPU attention backend (vllm-project#42740)
Signed-off-by: Tony Lin <tony.lin@intel.com> Co-authored-by: Li, Jiang <jiang1.li@intel.com>
1 parent c38bed4 commit 965d076

1 file changed

Lines changed: 5 additions & 0 deletions

File tree

vllm/v1/attention/backends/cpu_attn.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@
2525
MultipleOf,
2626
)
2727
from vllm.v1.attention.backends.utils import (
28+
KVCacheLayoutType,
2829
split_decodes_and_prefills,
2930
)
3031
from vllm.v1.kv_cache_interface import AttentionSpec, CrossAttentionSpec
@@ -94,6 +95,10 @@ def get_kv_cache_shape(
9495
) -> tuple[int, ...]:
9596
return 2, num_blocks, num_kv_heads, block_size, head_size
9697

98+
@classmethod
99+
def get_required_kv_cache_layout(cls) -> "KVCacheLayoutType | None":
100+
return "HND"
101+
97102
@staticmethod
98103
def use_cascade_attention(*args, **kwargs) -> bool:
99104
return False

0 commit comments

Comments
 (0)