[Bug] nvcc related crash on first forward pass - resolved with FLA_DISABLE_BACKEND_DISPATCH=1

[mramendi]

Checklist

• I have checked FAQs and existing issues for similar problems
• Please report this bug in English to ensure wider understanding and support

Describe the Bug

When I run my training script after installing flash-linear-attention[cuda] , I get a complicated crash involving nvcc and incompatible headers. There is no nvcc on my system. I captured the full output of the script, including the crash, in the attached file.

error.txt

Thanks to Claude Code, I found a workaround. I set FLA_DISABLE_BACKEND_DISPATCH=1 and the system works.

If this cannot be easily fixed, could this workaround be added to the readme or FAQ?

Steps to Reproduce the Bug

Use standard Trainer to train a Qwen 3.5 model after installing flash-linear-attention[cuda] and causal-conv1d . The crash apparently happens at the first forward pass.

Expected Behavior

training goes on

Environment Information

1. Torch: 2.11.0
2. Triton: 3.6.0
3. CUDA 13.0 ; container image based on the NVidia CUDA 13.0 runtime image with Ubuntu 24.04

[zhiyuan1i]

Hi, thanks for the detailed report!

This is expected, not really a bug in your setup. Since v0.5.0 FLA dispatches some GDN / KDA / attention kernels to the TileLang backend by default, and TileLang's default CUDA compile path shells out to the system nvcc, which needs a full CUDA toolkit (compiler + dev headers) on the host. You're on the CUDA 13.0 runtime image, which only ships the runtime/driver libraries and no nvcc or headers, so the JIT compile blows up with the nvcc / "incompatible headers" errors you saw.

If you just want to train, the easiest fix is to disable only the TileLang backend — it'll fall back to the default Triton kernels for just those ops:

export FLA_TILELANG=0

That's a bit more targeted than FLA_DISABLE_BACKEND_DISPATCH=1, which turns off every backend (including things like FlashKDA).

If you'd rather keep TileLang for the speedup, you just need to give it a working compiler: either switch to a CUDA devel image / install cuda-toolkit, or use the pip-provided toolchain so it works without a host toolkit:

pip install "nvidia-cuda-nvcc>=13" "nvidia-cuda-cccl>=13"

We'll add FLA_TILELANG=0 to the FAQ and look into falling back to Triton gracefully when nvcc isn't available. Thanks for flagging!