vllm serve pytorch/gemma-3-12b-it-FP8 failed with fbgemm-gpu-genai 1.4.2 on B200

[huydhn]

When running vLLM benchmark of pytorch/gemma-3-12b-it-FP8 on B200, I notice that the benchmark starts failing with fbgemm-gpu-genai=1.4.2 https://github.com/pytorch/pytorch-integration-testing/actions/runs/20177479851/job/57929181213#step:19:2183. The error when running vllm serve pytorch/gemma-3-12b-it-FP8 is as follows:

TMA benchmarks will be running without grid constant TMA descriptor.
(APIServer pid=7998) INFO 12-13 00:56:05 [api_server.py:1351] vLLM API server version 0.13.0rc2.dev105+gfdc135d76
(APIServer pid=7998) INFO 12-13 00:56:05 [utils.py:253] non-default args: {'model_tag': 'pytorch/gemma-3-12b-it-FP8', 'model': 'pytorch/gemma-3-12b-it-FP8'}
(APIServer pid=7998) INFO 12-13 00:56:05 [model.py:514] Resolved architecture: Gemma3ForConditionalGeneration
(APIServer pid=7998) INFO 12-13 00:56:05 [model.py:1636] Using max model len 131072
(APIServer pid=7998) INFO 12-13 00:56:05 [scheduler.py:228] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=7998) WARNING 12-13 00:56:05 [cuda.py:244] Forcing --disable_chunked_mm_input for models with multimodal-bidirectional attention.
TMA benchmarks will be running without grid constant TMA descriptor.
(EngineCore_DP0 pid=8153) INFO 12-13 00:56:13 [core.py:93] Initializing a V1 LLM engine (v0.13.0rc2.dev105+gfdc135d76) with config: model='pytorch/gemma-3-12b-it-FP8', speculative_config=None, tokenizer='pytorch/gemma-3-12b-it-FP8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=torchao, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=pytorch/gemma-3-12b-it-FP8, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False}, 'local_cache_dir': None}
(EngineCore_DP0 pid=8153) INFO 12-13 00:56:15 [parallel_state.py:1203] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.0.31.154:56647 backend=nccl
(EngineCore_DP0 pid=8153) INFO 12-13 00:56:15 [parallel_state.py:1411] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
(EngineCore_DP0 pid=8153) Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
(EngineCore_DP0 pid=8153) INFO 12-13 00:56:22 [gpu_model_runner.py:3562] Starting to load model pytorch/gemma-3-12b-it-FP8...
(EngineCore_DP0 pid=8153) /usr/local/lib/python3.12/dist-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
(EngineCore_DP0 pid=8153)   _C._set_float32_matmul_precision(precision)
(EngineCore_DP0 pid=8153) INFO 12-13 00:56:24 [layer.py:537] Using AttentionBackendEnum.FLASH_ATTN for MultiHeadAttention in multimodal encoder.
(EngineCore_DP0 pid=8153) INFO 12-13 00:56:26 [cuda.py:412] Using FLEX_ATTENTION attention backend out of potential backends: ['FLEX_ATTENTION']
Loading pt checkpoint shards:   0% Completed | 0/3 [00:00<?, ?it/s]
Loading pt checkpoint shards:  33% Completed | 1/3 [00:02<00:05,  2.80s/it]
Loading pt checkpoint shards:  67% Completed | 2/3 [00:05<00:02,  2.73s/it]
Loading pt checkpoint shards: 100% Completed | 3/3 [00:07<00:00,  2.25s/it]
Loading pt checkpoint shards: 100% Completed | 3/3 [00:07<00:00,  2.39s/it]
(EngineCore_DP0 pid=8153)
(EngineCore_DP0 pid=8153) INFO 12-13 00:56:34 [default_loader.py:308] Loading weights took 7.16 seconds
(EngineCore_DP0 pid=8153) INFO 12-13 00:56:34 [gpu_model_runner.py:3659] Model loading took 12.8828 GiB memory and 11.215734 seconds
(EngineCore_DP0 pid=8153) INFO 12-13 00:56:34 [gpu_model_runner.py:4446] Encoder cache will be initialized with a budget of 8192 tokens, and profiled with 31 image items of the maximum feature size.
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866] EngineCore failed to start.
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866] Traceback (most recent call last):
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 857, in run_engine_core
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 637, in __init__
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     super().__init__(
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 109, in __init__
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     num_gpu_blocks, num_cpu_blocks, kv_cache_config = self._initialize_kv_caches(
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 240, in _initialize_kv_caches
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 126, in determine_available_memory
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     return self.collective_rpc("determine_available_memory")
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/uniproc_executor.py", line 75, in collective_rpc
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/serial_utils.py", line 461, in run_method
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     return func(*args, **kwargs)
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     return func(*args, **kwargs)
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 340, in determine_available_memory
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     self.model_runner.profile_run()
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 4462, in profile_run
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     dummy_encoder_outputs = self.model.embed_multimodal(
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/gemma3_mm.py", line 603, in embed_multimodal
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     return self._process_image_input(image_input)
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/gemma3_mm.py", line 587, in _process_image_input
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     image_features = self._image_pixels_to_features(
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/gemma3_mm.py", line 576, in _image_pixels_to_features
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     return vision_tower(pixel_values)
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]            ^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/siglip.py", line 856, in forward
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     return self.vision_model(
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]            ^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/siglip.py", line 754, in forward
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     encoder_outputs = self.encoder(
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]                       ^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/siglip.py", line 562, in forward
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     hidden_states, _ = encoder_layer(hidden_states)
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/siglip.py", line 511, in forward
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     hidden_states, _ = self.self_attn(hidden_states=hidden_states)
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/siglip.py", line 429, in forward
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     qkv_states, _ = self.qkv_proj(hidden_states)
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/linear.py", line 565, in forward
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     output_parallel = self.quant_method.apply(self, input_, bias)
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/torchao.py", line 348, in apply
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     return F.linear(x, layer.weight, bias)
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/torchao/utils.py", line 655, in _dispatch__torch_function__
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     return cls._ATEN_OP_OR_TORCH_FN_TABLE[cls][func](func, types, args, kwargs)
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/torchao/utils.py", line 491, in wrapper
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     return func(f, types, args, kwargs)
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/torchao/quantization/quantize_/workflows/float8/float8_tensor.py", line 300, in _
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     res = torch.ops.fbgemm.f8f8bf16_rowwise(
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]   File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1255, in __call__
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]     return self._op(*args, **kwargs)
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8153) ERROR 12-13 00:56:35 [core.py:866] RuntimeError: cutlass cannot initialize

The same command vllm serve pytorch/gemma-3-12b-it-FP8 works fine with the previous version fbgemm-gpu-genai=1.4.1. The same error also happens with pytorch/gemma-3-27b-it-fp8

Here is the full list of pip packages:

sing Python 3.12.12 environment at: /usr
Package                            Version                     Editable project location
---------------------------------- --------------------------- -------------------------------------
absl-py                            2.1.0
accelerate                         1.0.1
aenum                              3.1.16
affine                             2.4.0
aiohappyeyeballs                   2.6.1
aiohttp                            3.13.0
aiohttp-cors                       0.8.1
aiosignal                          1.4.0
albucore                           0.0.16
albumentations                     1.4.6
alembic                            1.16.4
annotated-doc                      0.0.4
annotated-types                    0.7.0
anthropic                          0.71.0
antlr4-python3-runtime             4.9.3
anyio                              4.6.2.post1
apache-tvm-ffi                     0.1.5
arctic-inference                   0.1.1
argcomplete                        3.5.1
arrow                              1.3.0
astor                              0.8.1
attrs                              24.2.0
audioread                          3.0.1
backoff                            2.2.1
bitsandbytes                       0.46.1
black                              24.10.0
blake3                             1.0.8
blinker                            1.9.0
blobfile                           3.0.0
bm25s                              0.2.13
boto3                              1.35.57
botocore                           1.35.57
bounded-pool-executor              0.0.3
buildkite-test-collector           0.1.9
cachetools                         5.5.2
cbor2                              5.7.1
certifi                            2024.8.30
cffi                               1.17.1
cfgv                               3.5.0
chardet                            5.2.0
charset-normalizer                 3.4.0
chz                                0.3.0
click                              8.1.7
click-plugins                      1.1.1.2
cligj                              0.7.2
cloudpickle                        3.1.1
colorama                           0.4.6
colorful                           0.5.6
compressed-tensors                 0.12.2
contourpy                          1.3.0
coverage                           7.10.6
cramjam                            2.9.0
cryptography                       46.0.3
cuda-bindings                      13.1.1
cuda-pathfinder                    1.3.3
cuda-python                        13.1.1
cupy-cuda12x                       13.6.0
cycler                             0.12.1
databricks-sdk                     0.59.0
datamodel-code-generator           0.26.3
dataproperty                       1.0.1
datasets                           3.0.2
decorator                          5.1.1
decord                             0.6.0
deep-ep                            1.2.1+73b6ea4
deep-gemm                          2.1.0+594953a
depyf                              0.20.0
dill                               0.3.8
diskcache                          5.6.3
distlib                            0.3.9
distro                             1.9.0
dnspython                          2.7.0
docker                             7.1.0
docopt                             0.6.2
docstring-parser                   0.17.0
efficientnet-pytorch               0.7.1
einops                             0.8.1
einx                               0.3.0
email-validator                    2.2.0
encodec                            0.1.1
evaluate                           0.4.3
fastapi                            0.116.1
fastapi-cli                        0.0.16
fastapi-cloud-cli                  0.6.0
fastar                             0.8.0
fastparquet                        2024.11.0
fastrlock                          0.8.2
fastsafetensors                    0.1.10
fbgemm-gpu-genai                   1.4.2
filelock                           3.16.1
fiona                              1.10.1
flashinfer-cubin                   0.5.3
flashinfer-jit-cache               0.5.3+cu129
flashinfer-python                  0.5.3
flask                              3.1.1
fonttools                          4.55.0
fqdn                               1.5.1
frozendict                         2.4.6
frozenlist                         1.5.0
fsspec                             2024.9.0
ftfy                               6.3.1
genai-perf                         0.0.8
genson                             1.3.0
geopandas                          1.0.1
gguf                               0.17.1
gitdb                              4.0.12
gitpython                          3.1.44
google-api-core                    2.24.2
google-auth                        2.40.2
google-cloud-core                  2.4.3
google-cloud-storage               3.4.0
google-crc32c                      1.7.1
google-resumable-media             2.7.2
googleapis-common-protos           1.70.0
gpt-oss                            0.0.8
graphene                           3.4.3
graphql-core                       3.2.6
graphql-relay                      3.2.0
greenlet                           3.2.3
grpcio                             1.71.0
gunicorn                           23.0.0
h11                                0.14.0
h5py                               3.13.0
harfile                            0.3.0
hf-transfer                        0.1.9
hf-xet                             1.1.7
hiredis                            3.0.0
html2text                          2025.4.15
httpcore                           1.0.6
httptools                          0.7.1
httpx                              0.27.2
httpx-sse                          0.4.3
huggingface-hub                    0.34.3
humanize                           4.11.0
hydra-core                         1.3.2
hypothesis                         6.131.0
hypothesis-graphql                 0.11.1
hypothesis-jsonschema              0.23.1
identify                           2.6.15
idna                               3.10
ijson                              3.4.0.post0
imageio                            2.37.0
importlib-metadata                 8.7.0
importlib-resources                6.5.2
inflect                            5.6.2
iniconfig                          2.0.0
interegular                        0.3.3
isoduration                        20.11.0
isort                              5.13.2
itsdangerous                       2.2.0
jinja2                             3.1.6
jiter                              0.12.0
jiwer                              3.0.5
jmespath                           1.0.1
joblib                             1.4.2
jsonargparse                       4.35.0
jsonlines                          4.0.0
jsonpointer                        3.0.0
jsonschema                         4.23.0
jsonschema-specifications          2024.10.1
junit-xml                          1.9
kaleido                            0.2.1
kiwisolver                         1.4.7
kornia                             0.8.1
kornia-rs                          0.1.9
lark                               1.2.2
lazy-loader                        0.4
libnacl                            2.1.0
librosa                            0.10.2.post1
lightly                            1.5.20
lightly-utils                      0.0.2
lightning                          2.5.1.post0
lightning-utilities                0.14.3
llguidance                         1.3.0
llvmlite                           0.44.0
lm-eval                            0.4.9.1
lm-format-enforcer                 0.11.3
loguru                             0.7.3
lxml                               5.3.0
mako                               1.3.10
markdown                           3.8.2
markdown-it-py                     3.0.0
markupsafe                         3.0.1
matplotlib                         3.9.2
mbstrdecoder                       1.1.3
mcp                                1.24.0
mdurl                              0.1.2
mistral-common                     1.8.5
mlflow                             2.22.0
mlflow-skinny                      2.22.0
model-hosting-container-standards  0.1.11
more-itertools                     10.5.0
mpmath                             1.3.0
msgpack                            1.1.0
msgspec                            0.20.0
mteb                               2.1.2
multidict                          6.1.0
multiprocess                       0.70.16
munch                              4.0.0
mypy-extensions                    1.0.0
networkx                           3.2.1
ninja                              1.13.0
nltk                               3.9.1
nodeenv                            1.9.1
num2words                          0.5.14
numba                              0.61.2
numexpr                            2.10.1
numpy                              1.26.4
nvidia-cublas-cu12                 12.9.1.4
nvidia-cuda-cupti-cu12             12.9.79
nvidia-cuda-nvrtc-cu12             12.9.86
nvidia-cuda-runtime-cu12           12.9.79
nvidia-cudnn-cu12                  9.10.2.21
nvidia-cudnn-frontend              1.16.0
nvidia-cufft-cu12                  11.4.1.4
nvidia-cufile-cu12                 1.14.1.1
nvidia-curand-cu12                 10.3.10.19
nvidia-cusolver-cu12               11.7.5.82
nvidia-cusparse-cu12               12.5.10.65
nvidia-cusparselt-cu12             0.7.1
nvidia-cutlass-dsl                 4.3.3
nvidia-ml-py                       13.590.44
nvidia-nccl-cu12                   2.27.5
nvidia-nvjitlink-cu12              12.9.86
nvidia-nvshmem-cu12                3.3.20
nvidia-nvtx-cu12                   12.9.79
omegaconf                          2.3.0
open-clip-torch                    2.32.0
openai                             2.11.0
openai-harmony                     0.0.4
opencensus                         0.11.4
opencensus-context                 0.1.3
opencv-python-headless             4.11.0.86
opentelemetry-api                  1.35.0
opentelemetry-exporter-prometheus  0.56b0
opentelemetry-proto                1.36.0
opentelemetry-sdk                  1.35.0
opentelemetry-semantic-conventions 0.56b0
outlines-core                      0.2.11
packaging                          24.2
pandas                             2.2.3
partial-json-parser                0.2.1.1.post7
pathspec                           0.12.1
pathvalidate                       3.2.1
patsy                              1.0.1
peft                               0.16.0
pillow                             10.4.0
pip                                25.3
platformdirs                       4.3.6
plotly                             5.24.1
pluggy                             1.5.0
polars                             1.29.0
pooch                              1.8.2
portalocker                        2.10.1
pplx-kernels                       0.0.1
pqdm                               0.2.0
pre-commit                         4.0.1
pretrainedmodels                   0.7.4
prometheus-client                  0.22.0
prometheus-fastapi-instrumentator  7.1.0
propcache                          0.2.0
proto-plus                         1.26.1
protobuf                           5.28.3
psutil                             6.1.0
py                                 1.11.0
py-cpuinfo                         9.0.0
py-spy                             0.4.0
pyarrow                            18.0.0
pyasn1                             0.6.1
pyasn1-modules                     0.4.2
pybase64                           1.4.3
pybind11                           2.13.6
pycocotools                        2.0.8
pycountry                          24.6.1
pycparser                          2.22
pycryptodomex                      3.22.0
pydantic                           2.12.0
pydantic-core                      2.41.1
pydantic-extra-types               2.10.5
pydantic-settings                  2.12.0
pygments                           2.18.0
pyjwt                              2.10.1
pyogrio                            0.11.0
pyparsing                          3.2.0
pyproj                             3.7.1
pyrate-limiter                     3.7.0
pystemmer                          3.0.0
pytablewriter                      1.2.0
pytest                             8.3.5
pytest-asyncio                     0.24.0
pytest-cov                         6.3.0
pytest-forked                      1.6.0
pytest-mock                        3.14.0
pytest-rerunfailures               14.0
pytest-shard                       0.1.2
pytest-subtests                    0.14.1
pytest-timeout                     2.3.1
python-box                         7.3.2
python-dateutil                    2.9.0.post0
python-dotenv                      1.2.1
python-json-logger                 4.0.0
python-multipart                   0.0.20
python-rapidjson                   1.20
pytorch-lightning                  2.5.2
pytrec-eval-terrier                0.5.7
pytz                               2024.2
pyyaml                             6.0.2
pyzmq                              27.1.0
rapidfuzz                          3.12.1
rasterio                           1.4.3
ray                                2.48.0
redis                              5.2.0
referencing                        0.35.1
regex                              2024.9.11
requests                           2.32.3
responses                          0.25.3
rfc3339-validator                  0.1.4
rfc3987                            1.3.8
rich                               13.9.4
rich-toolkit                       0.17.0
rignore                            0.7.6
rioxarray                          0.19.0
rouge-score                        0.1.2
rpds-py                            0.20.1
rsa                                4.9.1
rtree                              1.4.0
runai-model-streamer               0.15.3
runai-model-streamer-gcs           0.15.3
runai-model-streamer-s3            0.15.3
s3transfer                         0.10.3
sacrebleu                          2.4.3
safetensors                        0.4.5
schemathesis                       3.39.15
scikit-image                       0.25.2
scikit-learn                       1.5.2
scipy                              1.13.1
segmentation-models-pytorch        0.4.0
sentence-transformers              3.2.1
sentencepiece                      0.2.1
sentry-sdk                         2.47.0
setproctitle                       1.3.7
setuptools                         77.0.3
shapely                            2.1.1
shellingham                        1.5.4
six                                1.16.0
smart-open                         7.1.0
smmap                              5.0.2
sniffio                            1.3.1
sortedcontainers                   2.4.0
soundfile                          0.12.1
soxr                               0.5.0.post1
sqlalchemy                         2.0.41
sqlitedict                         2.1.0
sqlparse                           0.5.3
sse-starlette                      3.0.3
starlette                          0.46.2
starlette-testclient               0.4.1
statsmodels                        0.14.4
structlog                          25.4.0
supervisor                         4.3.0
sympy                              1.13.3
tabledata                          1.3.3
tabulate                           0.9.0
tblib                              3.1.0
tcolorpy                           0.1.6
tenacity                           9.1.2
tensorboardx                       2.6.4
tensorizer                         2.10.1
termcolor                          3.1.0
terratorch                         1.0.2
threadpoolctl                      3.5.0
tifffile                           2025.3.30
tiktoken                           0.12.0
timm                               1.0.17
tokenizers                         0.22.0
tomli                              2.2.1
tomli-w                            1.2.0
torch                              2.9.0+cu129
torchao                            0.14.1
torchaudio                         2.9.0+cu129
torchgeo                           0.7.0
torchmetrics                       1.7.4
torchvision                        0.24.0+cu129
tqdm                               4.66.6
tqdm-multiprocess                  0.0.11
transformers                       4.57.3
transformers-stream-generator      0.0.5
triton                             3.5.0
tritonclient                       2.51.0
typepy                             1.3.2
typer                              0.15.2
types-python-dateutil              2.9.0.20241206
typeshed-client                    2.8.2
typing-extensions                  4.15.0
typing-inspection                  0.4.2
tzdata                             2024.2
uri-template                       1.3.0
urllib3                            2.2.3
uv                                 0.9.17
uvicorn                            0.35.0
uvloop                             0.22.1
vector-quantize-pytorch            1.21.2
virtualenv                         20.31.2
vllm                               0.13.0rc2.dev105+gfdc135d76
vllm-test-utils                    0.1                         /vllm-workspace/tests/vllm_test_utils
vocos                              0.1.0
watchfiles                         1.1.1
wcwidth                            0.2.13
webcolors                          24.11.1
websockets                         15.0.1
werkzeug                           3.1.3
word2number                        1.1
wrapt                              1.17.2
xarray                             2025.7.1
xgrammar                           0.1.27
xxhash                             3.5.0
yarl                               1.17.1
zipp                               3.23.0
zstandard                          0.23.0

cc @jainapurva

[Celestina-Annalise]

I'm encountering the same issue with pytorch/gemma-3-12b-it-QAT-INT4 using the latest vLLM Docker image (CUDA 12.9) on an RTX A5000. I tried manually installing torchao==0.14.1 and fbgemm-gpu-genai==1.4.1, but the model still fails to load. Has anyone found a solution for this configuration?