Skip to content
This repository was archived by the owner on May 27, 2026. It is now read-only.

Commit 573910e

Browse files
yma11yma11
authored
update readme for vLLM 0.17.0 release on Intel GPU (#971)
Signed-off-by: Yan Ma <yan.ma@intel.com>
1 parent 90e713c commit 573910e

1 file changed

Lines changed: 252 additions & 0 deletions

File tree

‎vllm/0.17.0-xpu.md‎

Lines changed: 252 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,252 @@
1+
# Optimize LLM Serving with vLLM on Intel® GPUs
2+
3+
vLLM is a fast and easy-to-use library for LLM inference and serving. It has grown into a community-driven project with contributions from both academia and industry. Intel, as an active community contributor, continues to improve vLLM performance and usability on Intel® platforms, including Intel® Xeon® Scalable Processors, Intel® discrete GPUs, and Intel® Gaudi® AI accelerators. This document focuses on Intel® discrete GPUs and provides the information needed to run these workloads effectively on Intel® graphics cards.
4+
5+
This release is the first to switch to the optimized kernel library [vllm-xpu-kernels](https://github.com/vllm-project/vllm-xpu-kernels) for Intel® GPUs. The vLLM build included in this container uses the same code base as [v0.17.0](https://github.com/vllm-project/vllm/tree/v0.17.0) and has been validated on [Intel® Arc™ Pro B-Series Graphics](https://www.intel.com/content/www/us/en/products/docs/discrete-gpus/arc/workstations/b-series/overview.html) cards. The following bill of materials was used for validation:
6+
7+
| Ingredients | Version |
8+
| --- | --- |
9+
| Host OS | Ubuntu 25.04 |
10+
| Python | 3.12 |
11+
| KMD Driver | 6.14.0 |
12+
| oneAPI | 2025.3.2.4 with hotfix |
13+
| PyTorch | 2.10 |
14+
| vllm-xpu-kernels | 0.1.4 |
15+
| oneCCL | 2021.15.7.8 |
16+
17+
## 1. What's Supported?
18+
19+
This release supports core vLLM serving capabilities on Intel® GPUs, including online FP8 quantization, multimodal models, pooling models, and multi-GPU scaling strategies. In addition to dense-model serving, it also includes experimental expert parallelism and validated support for MoE models.
20+
21+
| Feature | Description | Note |
22+
| --- | --- | --- |
23+
| FP8 Online Quantization | vLLM supports weight-only online dynamic quantization with FP8, enabling up to a 2x reduction in model memory requirements and up to a 1.6x throughput improvement with minimal accuracy impact. Models in BF16 or FP16 can be quantized dynamically to FP8 without calibration data. | See the [example](https://docs.vllm.ai/en/stable/features/quantization/fp8/?h=online+dynamic#online-dynamic-quantization). |
24+
| Multi-Modality Support | We support most of the popular multimodal models in upstream's [list](https://docs.vllm.ai/en/stable/models/supported_models/#list-of-multimodal-language-models), such as Qwen VL series, InternVL series, whisper-large-v3, DeepSeek-OCR, and PaddleOCR-VL. | For example, `Qwen/Qwen2.5-VL-32B-Instruct` can be launched on 4 Intel® Arc™ Pro B60 Graphics cards for multimodal processing. |
25+
| Pooling Models Support | vLLM supports pooling models such as embedding, classification, and reward models. All of these models are now supported on Intel® GPUs. | For detailed usage, refer to the [guide](https://docs.vllm.ai/en/latest/models/pooling_models.html). |
26+
| Pipeline Parallelism | Pipeline parallelism distributes model layers across multiple GPUs, with each GPU processing a different stage of the model in sequence. | On Intel® GPUs, this is supported on a single node with `mp` as the backend. |
27+
| Data Parallelism | vLLM supports [Data Parallelism](https://docs.vllm.ai/en/latest/serving/data_parallel_deployment.html), where model weights are replicated across separate instances or GPUs to process independent request batches. | Supports both dense and MoE models. |
28+
| Expert Parallelism | Experimental support for [Expert Parallelism](https://docs.vllm.ai/en/stable/serving/expert_parallel_deployment), which allows experts in Mixture-of-Experts (MoE) models to be deployed across separate GPUs. | In this release, `TP+DP+EP` is supported. |
29+
30+
In addition, features such as [reasoning_outputs](https://docs.vllm.ai/en/latest/features/reasoning_outputs.html), [structured_outputs](https://docs.vllm.ai/en/latest/features/structured_outputs.html), and [tool calling](https://docs.vllm.ai/en/latest/features/tool_calling.html) are supported. The following experimental features are also available:
31+
32+
* **torch.compile**: Can be enabled for the FP16/BF16 path.
33+
* **speculative decoding**: Supports methods `n-gram`, `EAGLE`, `EAGLE3`, `medusa` and `suffix`. For detailed usage, refer to [document](https://docs.vllm.ai/en/stable/features/speculative_decoding/).
34+
* **async scheduling**: Can be enabled by `--async-scheduling`. This may help reduce the CPU overheads, leading to better latency and throughput.
35+
36+
## 2. Supported Models
37+
38+
Please note that the following table contains only the models verified by Intel. Support on Intel® GPUs through vLLM extends to a wider array of models.
39+
40+
### Text Generation Models
41+
42+
These models primarily accept the LLM.generate API. Chat/Instruct models additionally support the LLM.chat API.
43+
44+
| Model (company/model name) | BF16/FP16 | Dynamic Online FP8 | MXFP4 |
45+
|-------------------------------------------| --- | --- | -- |
46+
| openai/gpt-oss-20b | | |✅︎|
47+
| openai/gpt-oss-120b | | |✅︎|
48+
| deepseek-ai/DeepSeek-R1-Distill-Llama-8B |✅︎|✅︎| |
49+
| deepseek-ai/DeepSeek-R1-Distill-Qwen-14B |✅︎|✅︎| |
50+
| deepseek-ai/DeepSeek-R1-Distill-Qwen-32B |✅︎|✅︎| |
51+
| deepseek-ai/DeepSeek-R1-Distill-Llama-70B |✅︎|✅︎| |
52+
| Qwen/Qwen2.5-72B-Instruct |✅︎|✅︎| |
53+
| Qwen/Qwen3-14B |✅︎|✅︎| |
54+
| Qwen/Qwen3-32B |✅︎|✅︎| |
55+
| Qwen/Qwen3-30B-A3B |✅︎|✅︎| |
56+
| Qwen/Qwen3-30B-A3B-GPTQ-Int4 |✅︎|✅︎| |
57+
| Qwen/Qwen3-coder-30B-A3B-Instruct |✅︎|✅︎| |
58+
| Qwen/QwQ-32B |✅︎|✅︎| |
59+
| openbmb/MiniCPM-V-4 |✅︎|✅︎| |
60+
| deepseek-ai/DeepSeek-V2-Lite |✅︎|✅︎| |
61+
| meta-llama/Llama-3.1-8B-Instruct |✅︎|✅︎| |
62+
| THUDM/GLM-4-9B-chat |✅︎|✅︎| |
63+
| THUDM/GLM-4v-9B-chat |✅︎|✅︎| |
64+
| THUDM/CodeGeex4-All-9B |✅︎|✅︎| |
65+
| chuhac/TeleChat2-35B |✅︎|✅︎| |
66+
| 01-ai/Yi1.5-34B-Chat |✅︎|✅︎| |
67+
| THUDM/CodeGeex4-All-9B |✅︎|✅︎| |
68+
| deepseek-ai/DeepSeek-Coder-33B-base |✅︎|✅︎| |
69+
| meta-llama/Llama-2-13b-chat-hf |✅︎|✅︎| |
70+
| Qwen/Qwen1.5-14B-Chat |✅︎|✅︎| |
71+
| Qwen/Qwen1.5-32B-Chat |✅︎|✅︎| |
72+
73+
### Multimodal Models
74+
75+
The modalities(text, image, video, audio) are supported depending on the model:
76+
77+
| Model (company/model name) | BF16/FP16 | Dynamic Online FP8 | Text | Image | Video | Audio |
78+
|-------------------------------------------| --- | --- | -- | -- | -- | -- |
79+
| openai/whisper-large-v3 |✅︎| | | | |✅︎|
80+
| deepseek-ai/DeepSeek-OCR |✅︎|✅︎|✅︎|✅︎| | |
81+
| PaddlePaddle/PaddleOCR-VL |✅︎|✅︎|✅︎|✅︎| | |
82+
| Qwen/Qwen2-VL-7B-Instruct |✅︎|✅︎|✅︎|✅︎|✅︎| |
83+
| Qwen/Qwen2.5-VL-72B-Instruct |✅︎|✅︎|✅︎|✅︎|✅︎| |
84+
| Qwen/Qwen2.5-VL-32B-Instruct |✅︎|✅︎|✅︎|✅︎|✅︎| |
85+
| OpenGVLab/InternVL3_5-8B |✅︎|✅︎|✅︎|✅︎|✅︎| |
86+
| OpenGVLab/InternVL3_5-14B |✅︎|✅︎|✅︎|✅︎|✅︎| |
87+
| OpenGVLab/InternVL3_5-38B |✅︎|✅︎|✅︎|✅︎|✅︎| |
88+
| OpenGVLab/InternVL3_5-30B-A3B |✅︎|✅︎|✅︎|✅︎|✅︎| |
89+
| openbmb/MiniCPM-V-4 |✅︎|✅︎|✅︎|✅︎|✅︎| |
90+
91+
### Pooling Models
92+
93+
These models primarily support the LLM.embed API. The following table lists those that are tested on XPU.
94+
95+
| Model Type | Model (company/model name) | BF16 | Dynamic Online FP8 |
96+
|-----------------|-------------------------------------------| --- | --- |
97+
| Embedding Model | Qwen/Qwen3-Embedding-8B |✅︎|✅︎|
98+
| Reranker Model | Qwen/Qwen3-Reranker-8B |✅︎|✅︎|
99+
100+
## 3. Limitations
101+
102+
Some vLLM features still require additional enablement or refinement and are not included in current release, like LoRA (Low-Rank Adaptation), pipeline parallelism on Ray, and MLA (Multi-head Latent Attention). CPU KV-cache offloading also needs further refinement due to kernel migration.
103+
104+
The following items are also known issues:
105+
106+
* Certain workloads may show lower performance than the 0.14.1 release, as this release focuses on establishing a solid functional baseline with vLLM XPU kernels and removing IPEX dependencies. Performance optimizations will continue in future releases.
107+
* Set the `SYCL_UR_USE_LEVEL_ZERO_V2=0` environment variable to avoid unexpected OOM errors during inference.
108+
* Set block size to `64` for better accuracy.
109+
* For `Qwen/Qwen3-30B-A3B` in FP16/BF16, set `PYTORCH_ALLOC_CONF=expandable_segments:True` or `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` to enable expandable blocks in the cache allocator.
110+
* W8A8 quantized models generated with `llm_compressor` are not supported yet, such as `RedHatAI/DeepSeek-R1-Distill-Qwen-32B-FP8-dynamic`.
111+
112+
## 4. How to Get Started
113+
114+
### 4.1. Prerequisite
115+
116+
| OS | Hardware |
117+
| ---------- | ---------- |
118+
| Ubuntu 25.04 | Intel® Arc™ B-Series |
119+
120+
### 4.2. Prepare a Serving Environment
121+
122+
1. Pull the released Docker image:
123+
124+
```bash
125+
docker pull intel/vllm:0.17.0-xpu
126+
```
127+
128+
2. Start a container:
129+
130+
```bash
131+
docker run -t -d --shm-size 10g --net=host --ipc=host --privileged \
132+
-v /dev/dri/by-path:/dev/dri/by-path --name=vllm-test \
133+
--device /dev/dri:/dev/dri --entrypoint= intel/vllm:0.17.0-xpu /bin/bash
134+
```
135+
136+
3. Open two terminals and run `docker exec -it vllm-test bash` in both of them. Use one terminal for the server and the other for the client.
137+
138+
From this point on, all commands are expected to be run inside the Docker container unless noted otherwise.
139+
140+
In both environments, you may want to set the `HUGGING_FACE_HUB_TOKEN` environment variable to ensure that required files can be downloaded from Hugging Face.
141+
142+
```bash
143+
export HUGGING_FACE_HUB_TOKEN=xxxxxx
144+
```
145+
146+
### 4.3. Launch Workloads
147+
148+
#### 4.3.1. Launch Server in the Server Environment
149+
150+
Command:
151+
152+
```bash
153+
VLLM_WORKER_MULTIPROC_METHOD=spawn vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
154+
--dtype=float16 \
155+
--enforce-eager \
156+
--port 8000 \
157+
--block-size 64 \
158+
--gpu-memory-util 0.9 \
159+
--no-enable-prefix-caching \
160+
--trust-remote-code \
161+
--disable-sliding-window \
162+
--max-num-batched-tokens=8192 \
163+
--max-model-len 4096 \
164+
-tp=4 \
165+
--quantization fp8
166+
```
167+
168+
Expected output:
169+
170+
```bash
171+
INFO 03-20 03:20:29 api_server.py:937] Starting vLLM API server on http://0.0.0.0:8000
172+
INFO 03-20 03:20:29 launcher.py:23] Available routes are:
173+
INFO 03-20 03:20:29 launcher.py:31] Route: /openapi.json, Methods: HEAD, GET
174+
INFO 03-20 03:20:29 launcher.py:31] Route: /docs, Methods: HEAD, GET
175+
INFO 03-20 03:20:29 launcher.py:31] Route: /docs/oauth2-redirect, Methods: HEAD, GET
176+
INFO 03-20 03:20:29 launcher.py:31] Route: /redoc, Methods: HEAD, GET
177+
INFO 03-20 03:20:29 launcher.py:31] Route: /health, Methods: GET
178+
INFO 03-20 03:20:29 launcher.py:31] Route: /ping, Methods: POST, GET
179+
INFO 03-20 03:20:29 launcher.py:31] Route: /tokenize, Methods: POST
180+
INFO 03-20 03:20:29 launcher.py:31] Route: /detokenize, Methods: POST
181+
INFO 03-20 03:20:29 launcher.py:31] Route: /v1/models, Methods: GET
182+
INFO 03-20 03:20:29 launcher.py:31] Route: /version, Methods: GET
183+
INFO 03-20 03:20:29 launcher.py:31] Route: /v1/chat/completions, Methods: POST
184+
INFO 03-20 03:20:29 launcher.py:31] Route: /v1/completions, Methods: POST
185+
INFO 03-20 03:20:29 launcher.py:31] Route: /v1/embeddings, Methods: POST
186+
INFO 03-20 03:20:29 launcher.py:31] Route: /pooling, Methods: POST
187+
INFO 03-20 03:20:29 launcher.py:31] Route: /score, Methods: POST
188+
INFO 03-20 03:20:29 launcher.py:31] Route: /v1/score, Methods: POST
189+
INFO 03-20 03:20:29 launcher.py:31] Route: /v1/audio/transcriptions, Methods: POST
190+
INFO 03-20 03:20:29 launcher.py:31] Route: /rerank, Methods: POST
191+
INFO 03-20 03:20:29 launcher.py:31] Route: /v1/rerank, Methods: POST
192+
INFO 03-20 03:20:29 launcher.py:31] Route: /v2/rerank, Methods: POST
193+
INFO 03-20 03:20:29 launcher.py:31] Route: /invocations, Methods: POST
194+
INFO: Started server process [1636943]
195+
INFO: Waiting for application startup.
196+
INFO: Application startup complete.
197+
```
198+
199+
Startup may take some time. When `INFO: Application startup complete.` appears, the server is ready.
200+
201+
#### 4.3.2. Raise Requests for Benchmarking in the Client Environment
202+
203+
Use the following command to send benchmark requests:
204+
205+
```bash
206+
vllm bench serve \
207+
--model deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
208+
--dataset-name random \
209+
--random-input-len=1024 \
210+
--random-output-len=1024 \
211+
--ignore-eos \
212+
--num-prompt 16 \
213+
--max-concurrency 16 \
214+
--request-rate inf \
215+
--backend vllm \
216+
--port=8000 \
217+
--host 0.0.0.0 \
218+
--ready-check-timeout-sec 1
219+
```
220+
221+
This command uses the `deepseek-ai/DeepSeek-R1-Distill-Qwen-32B` model. Both the input and output token lengths are set to `1024`, and up to `16` requests are processed concurrently by the server.
222+
223+
Expected output:
224+
225+
```bash
226+
Maximum request concurrency: 16
227+
============ Serving Benchmark Result ============
228+
Successful requests: 1
229+
Benchmark duration (s): xxx
230+
Total input tokens: 1024
231+
Total generated tokens: 1024
232+
Request throughput (req/s): xxx
233+
Output token throughput (tok/s): xxx
234+
Total Token throughput (tok/s): xxx
235+
---------------Time to First Token----------------
236+
Mean TTFT (ms): xxx
237+
Median TTFT (ms): xxx
238+
P99 TTFT (ms): xxx
239+
-----Time per Output Token (excl. 1st token)------
240+
Mean TPOT (ms): xxx
241+
Median TPOT (ms): xxx
242+
P99 TPOT (ms): xxx
243+
---------------Inter-token Latency----------------
244+
Mean ITL (ms): xxx
245+
Median ITL (ms): xxx
246+
P99 ITL (ms): xxx
247+
==================================================
248+
```
249+
250+
## 5. Need Assistance?
251+
252+
Should you encounter any issues or have any questions, please submit an issue ticket at [vLLM Github Issues](https://github.com/vllm-project/vllm/issues). Include the text `[Intel GPU]` in the issue title to ensure it gets noticed.

0 commit comments

Comments
 (0)