RTX 3090 device loss (Xid 79) at --max-prefill-length 8192 in offload mode — known limitation, or host-specific?

[kngharv]

Before you start

• I have read the FAQ and my problem is not answered there.
• I have read the Roadmap and this is not already planned there.
• I have searched existing issues and found no duplicate.
• I am on the latest release, or on a freshly rebuilt main when building from source.

What happened

Running FreeToken in MoE offload mode on a single RTX 3090, the GPU falls off the PCIe bus 16-61 seconds after the first large prefill. The engine dies on a CUDA error and the device does not return on a warm reboot — it needs a cold power cycle.

What I expected:
in offload mode, large prefills to be slow. I accept that expert
fetches over PCIe are bandwidth-bound and that TTFT suffers.

What happened instead:
the device is lost at the hardware level, taking down the
whole serving process, and the host needs physical intervention to recover.

The single variable that controls it

--max-prefill-length	two-concurrent runs	crashed
8192	6	5 (16-61 s after the first large prefill)
2048	9	0

Strict two-concurrent Fisher exact p = 0.0476. One sequential single-worker run also crashed at 8192, so concurrency multiplies the pressure but does not cause it.

Not the build, not the model

Reproduced on both cac247a8 and cc1f5c2c, and with both
Qwen/Qwen3.6-35B-A3B-FP8 and ornith-ai/Ornith-1.5-35B-A3B-NVFP4. The only thing
that changed the outcome was the chunk size.

Signature

Instant, complete link loss at 150-213 W (43-61% of the card's 350 W limit), at full PCIe Gen3 (no down-shift), with no throttling ramp and no precursor event. That
reads as a hard fault, not gradual overload.

Engine side, the CUDA failure surfaces from a different leaf frame each time consistent with the device disappearing rather than with one buggy code path:

• fla/chunk.py:108 in prepare_chunk_indices
• moe/offload_cache.py:639 in _invalidate_prefill_buffer
• scheduler.py:317 _process_last_data -> torch/cuda/streams.py:254 synchronize

Fatal prefill trace immediately before loss:

#new-token: 8192  #cached-token: 0  #running-req: 0
#new-token: 8192  #cached-token: 0  #running-req: 0
#running-req: 1
<device lost>

Anything else

• Hardware: Dell Precision T7810, BIOS A34, 1350 W PSU with a custom dual-GPU power delivery board, no BMC/IPMI. Each occurrence therefore costs a hands-on cold power cycle, which is why the sample sizes above are small.

• What I ruled out: OOM (VRAM, cgroup, host RAM); sustained PSU overcurrent (peaks 242-248 W); PCIe ASPM/link oscillation (every death was at full Gen3, and surviving runs churned more link-state transitions than the failing ones); connector seating and cable sharing (separate 8-pin run to each GPU, verified); driver/GSP version (identical for the 14 days before the first occurrence, and no newer GSP exists for this driver); concurrency as a standalone cause.

• What I could not determine: the mechanism.
  It is entirely possible this host's PCIe path is marginal and FreeToken is simply the only workload I run that drives it hard enough. Offload fetches experts from host RAM over PCIe every step, and a commenter on ft bench bw picks hybrid on Ampere, but offload is 8.3x faster (DeepSeek-V4-Flash, 2x RTX 3090) #151 measured roughly 78% of practical Gen3 x16 bandwidth on a similar 2x3090 setup. I am not claiming the engine is broken.

• Why I think it still belongs here:
  the trigger is a parameter the engine controls, the correlation is strong and reproducible, and there is not a single other bus-drop report in the tracker — so either this is host-specific, or it is under-reported.

• A1000 note: the 154 line also fires for 0000:03:00 (the A1000) although it never logs a 79. I read that as the node-level recovery action being reported against every GPU the driver manages, not as a second device failing.

How did you install FreeToken

Built from source

FreeToken version

0.13 cac247a also reproduced on cc1f5c2

OS

Ubuntu

OS details

Ubuntu 26.04.1 LTS, kernel 7.0.0-31-generic

GPU and driver

RTX 3090 24 GB, driver 595.91.07, PCIe Gen3 x16, 0000:04:00.0. An RTX A1000 is also installed (0000:03:00.0) but is not used by FreeToken and is idle during these runs.

CPU and system RAM

2x Intel Xeon E5-2620 v3 (12 cores / 24 threads), 92 GiB

Checkpoint

Qwen/Qwen3.6-35B-A3B-FP8(snapshot 95a723d08a9490559dae23d0cff1d9466213d989) AND Ornith-ai/Ornith-1.5-35B-A3B-NVFP4

Command

ft \
  --model /root/.cache/huggingface/hub/models--Qwen--Qwen3.6-35B-A3B-FP8/snapshots/95a723d08a9490559dae23d0cff1d9466213d989 \
  --served-model-name Qwen3.6-35B-A3B-FP8 \
  --host 0.0.0.0 --port 1919 \
  --gpu 0 \
  --moe-strategy offload \
  --max-prefill-length 8192 \
  --max-running-requests 4 \
  --memory-ratio 0.90 \
  --kv-reserve-tokens 300000 \
  --max-seq-len-override 164000 \
  --enable-cache-report

Full log

Sep 22 00:21:09 dell7810 dockerd[2065]: time="2026-09-22T00:21:09.439140272-07:00" level=info msg="sbJoin: gwep4 ''->'7c79d612a10e', gwep6 ''->''" eid=7c79d612a10e ep=freetoken-qwen36moe net=t7810dockerNet nid=4e2b3338e547 spanID=a2d1e9d8a4b94f6a traceID=826b5edf34d4c71e491f89376c716a48
ServerArgs(model_path='/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-35B-A3B-FP8/snapshots/95a723d08a9490559dae23d0cff1d9466213d989', tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, max_running_req=4, attention_backend='auto', moe_strategy='offload', quant_backend=None, ple_backend='disk', expert_load='auto', moe_cache_size=0, moe_cache_rate=None, moe_cache_auto=True, kv_reserve_tokens=300000, moe_cache_policy='lru', moe_prefill_overlap=True, moe_prefill_hit_d2d=False, moe_collect_stats=False, moe_cpu_threads=0, moe_cpu_layers=None, moe_hybrid_max_fetch=-1, cuda_graph_bs=None, cuda_graph_max_bs=None, page_size=1, memory_ratio=0.9, linear_state_cache_ratio=2.0, swa_full_tokens_ratio=0.2, swa_num_pages_override=None, distributed_timeout=60.0, use_dummy_weight=False, use_pynccl=True, max_seq_len_override=164000, num_page_override=None, num_token_override=None, mm=MultimodalConfig(disabled_encoders=frozenset(), embed_cache_device='cpu', encoder_weights='host', image_min_tokens=None, image_max_tokens=None, processor_kwargs={}), max_extend_tokens=8192, cache_type='radix', offline_mode=False, decode_log_interval=40, special_token_ckpt=False, _unique_suffix='.pid=1', server_host='0.0.0.0', server_port=1919, num_tokenizer=0, silent_output=False, shell_mode=False, served_model_name='Qwen3.6-35B-A3B-FP8', tool_call_parser='qwen3_coder', reasoning_parser='qwen3', sampling_defaults='model', max_output_tokens=None, enable_cache_report=True, allowed_media_domains='', allowed_local_media_path='', cors_origins='tauri://localhost,http://tauri.localhost,http://localhost:1420', gpu=('0',), gpu_assigned=None)
/opt/freetoken/python/freetoken/engine/engine.py:1135: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead.
  torch.cuda.memory._set_allocator_settings("expandable_segments:True")
[974B blob data]
[3.8K blob data]
                 from /tmp/tmpc95vgwb3/cuda_utils.c:7:
/root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/pyconfig.h:1877:9: warning: ‘_POSIX_C_SOURCE’ redefined
 1877 | #define _POSIX_C_SOURCE 200809L
      |         ^~~~~~~~~~~~~~~
In file included from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:26,
                 from /opt/venv/lib/python3.12/site-packages/triton/backends/nvidia/include/cuda.h:56,
                 from /tmp/tmpc95vgwb3/cuda_utils.c:1:
/usr/include/features.h:319:10: note: this is the location of the previous definition
  319 | # define _POSIX_C_SOURCE        202405L
      |          ^~~~~~~~~~~~~~~
In file included from /root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/Python.h:12,
                 from /tmp/tmpuybi5zac/__triton_launcher.c:7:
/root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/pyconfig.h:1877:9: warning: ‘_POSIX_C_SOURCE’ redefined
 1877 | #define _POSIX_C_SOURCE 200809L
      |         ^~~~~~~~~~~~~~~
In file included from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:26,
                 from /opt/venv/lib/python3.12/site-packages/triton/backends/nvidia/include/cuda.h:56,
                 from /tmp/tmpuybi5zac/__triton_launcher.c:2:
/usr/include/features.h:319:10: note: this is the location of the previous definition
  319 | # define _POSIX_C_SOURCE        202405L
      |          ^~~~~~~~~~~~~~~
In file included from /root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/Python.h:12,
                 from /tmp/tmptkob053c/__triton_launcher.c:7:
/root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/pyconfig.h:1877:9: warning: ‘_POSIX_C_SOURCE’ redefined
 1877 | #define _POSIX_C_SOURCE 200809L
      |         ^~~~~~~~~~~~~~~
In file included from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:26,
                 from /opt/venv/lib/python3.12/site-packages/triton/backends/nvidia/include/cuda.h:56,
                 from /tmp/tmptkob053c/__triton_launcher.c:2:
/usr/include/features.h:319:10: note: this is the location of the previous definition
  319 | # define _POSIX_C_SOURCE        202405L
      |          ^~~~~~~~~~~~~~~
[288B blob data]
                 from /tmp/tmpzme54d_e/__triton_launcher.c:7:
/root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/pyconfig.h:1877:9: warning: ‘_POSIX_C_SOURCE’ redefined
 1877 | #define _POSIX_C_SOURCE 200809L
      |         ^~~~~~~~~~~~~~~
In file included from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:26,
                 from /opt/venv/lib/python3.12/site-packages/triton/backends/nvidia/include/cuda.h:56,
                 from /tmp/tmpzme54d_e/__triton_launcher.c:2:
/usr/include/features.h:319:10: note: this is the location of the previous definition
  319 | # define _POSIX_C_SOURCE        202405L
      |          ^~~~~~~~~~~~~~~
In file included from /root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/Python.h:12,
                 from /tmp/tmp2707wujq/__triton_launcher.c:7:
/root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/pyconfig.h:1877:9: warning: ‘_POSIX_C_SOURCE’ redefined
 1877 | #define _POSIX_C_SOURCE 200809L
      |         ^~~~~~~~~~~~~~~
In file included from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:26,
                 from /opt/venv/lib/python3.12/site-packages/triton/backends/nvidia/include/cuda.h:56,
                 from /tmp/tmp2707wujq/__triton_launcher.c:2:
/usr/include/features.h:319:10: note: this is the location of the previous definition
  319 | # define _POSIX_C_SOURCE        202405L
      |          ^~~~~~~~~~~~~~~
In file included from /root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/Python.h:12,
                 from /tmp/tmp_005lasw/__triton_launcher.c:7:
/root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/pyconfig.h:1877:9: warning: ‘_POSIX_C_SOURCE’ redefined
 1877 | #define _POSIX_C_SOURCE 200809L
      |         ^~~~~~~~~~~~~~~
In file included from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:26,
                 from /opt/venv/lib/python3.12/site-packages/triton/backends/nvidia/include/cuda.h:56,
                 from /tmp/tmp_005lasw/__triton_launcher.c:2:
/usr/include/features.h:319:10: note: this is the location of the previous definition
  319 | # define _POSIX_C_SOURCE        202405L
      |          ^~~~~~~~~~~~~~~
In file included from /root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/Python.h:12,
                 from /tmp/tmpbigqsw5e/__triton_launcher.c:7:
/root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/pyconfig.h:1877:9: warning: ‘_POSIX_C_SOURCE’ redefined
 1877 | #define _POSIX_C_SOURCE 200809L
      |         ^~~~~~~~~~~~~~~
In file included from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:26,
                 from /opt/venv/lib/python3.12/site-packages/triton/backends/nvidia/include/cuda.h:56,
                 from /tmp/tmpbigqsw5e/__triton_launcher.c:2:
/usr/include/features.h:319:10: note: this is the location of the previous definition
  319 | # define _POSIX_C_SOURCE        202405L
      |          ^~~~~~~~~~~~~~~
In file included from /root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/Python.h:12,
                 from /tmp/tmph3hihe7i/__triton_launcher.c:7:
/root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/pyconfig.h:1877:9: warning: ‘_POSIX_C_SOURCE’ redefined
 1877 | #define _POSIX_C_SOURCE 200809L
      |         ^~~~~~~~~~~~~~~
In file included from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:26,
                 from /opt/venv/lib/python3.12/site-packages/triton/backends/nvidia/include/cuda.h:56,
                 from /tmp/tmph3hihe7i/__triton_launcher.c:2:
/usr/include/features.h:319:10: note: this is the location of the previous definition
  319 | # define _POSIX_C_SOURCE        202405L
      |          ^~~~~~~~~~~~~~~
In file included from /root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/Python.h:12,
                 from /tmp/tmpsxjq3nf0/__triton_launcher.c:7:
/root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/pyconfig.h:1877:9: warning: ‘_POSIX_C_SOURCE’ redefined
 1877 | #define _POSIX_C_SOURCE 200809L
      |         ^~~~~~~~~~~~~~~
In file included from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:26,
                 from /opt/venv/lib/python3.12/site-packages/triton/backends/nvidia/include/cuda.h:56,
                 from /tmp/tmpsxjq3nf0/__triton_launcher.c:2:
/usr/include/features.h:319:10: note: this is the location of the previous definition
  319 | # define _POSIX_C_SOURCE        202405L
      |          ^~~~~~~~~~~~~~~
In file included from /root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/Python.h:12,
                 from /tmp/tmp0b605miw/__triton_launcher.c:7:
/root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/pyconfig.h:1877:9: warning: ‘_POSIX_C_SOURCE’ redefined
 1877 | #define _POSIX_C_SOURCE 200809L
      |         ^~~~~~~~~~~~~~~
In file included from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:26,
                 from /opt/venv/lib/python3.12/site-packages/triton/backends/nvidia/include/cuda.h:56,
                 from /tmp/tmp0b605miw/__triton_launcher.c:2:
/usr/include/features.h:319:10: note: this is the location of the previous definition
  319 | # define _POSIX_C_SOURCE        202405L
      |          ^~~~~~~~~~~~~~~
In file included from /root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/Python.h:12,
                 from /tmp/tmpjv9629lw/__triton_launcher.c:7:
/root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/pyconfig.h:1877:9: warning: ‘_POSIX_C_SOURCE’ redefined
 1877 | #define _POSIX_C_SOURCE 200809L
      |         ^~~~~~~~~~~~~~~
In file included from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:26,
                 from /opt/venv/lib/python3.12/site-packages/triton/backends/nvidia/include/cuda.h:56,
                 from /tmp/tmpjv9629lw/__triton_launcher.c:2:
/usr/include/features.h:319:10: note: this is the location of the previous definition
  319 | # define _POSIX_C_SOURCE        202405L
      |          ^~~~~~~~~~~~~~~
In file included from /root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/Python.h:12,
                 from /tmp/tmpkh0bqh96/__triton_launcher.c:7:
/root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/pyconfig.h:1877:9: warning: ‘_POSIX_C_SOURCE’ redefined
 1877 | #define _POSIX_C_SOURCE 200809L
      |         ^~~~~~~~~~~~~~~
In file included from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:26,
                 from /opt/venv/lib/python3.12/site-packages/triton/backends/nvidia/include/cuda.h:56,
                 from /tmp/tmpkh0bqh96/__triton_launcher.c:2:
/usr/include/features.h:319:10: note: this is the location of the previous definition
  319 | # define _POSIX_C_SOURCE        202405L
      |          ^~~~~~~~~~~~~~~
[559B blob data]
                 from /tmp/tmpullyyy6z/__triton_launcher.c:7:
/root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/pyconfig.h:1877:9: warning: ‘_POSIX_C_SOURCE’ redefined
 1877 | #define _POSIX_C_SOURCE 200809L
      |         ^~~~~~~~~~~~~~~
In file included from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:26,
                 from /opt/venv/lib/python3.12/site-packages/triton/backends/nvidia/include/cuda.h:56,
                 from /tmp/tmpullyyy6z/__triton_launcher.c:2:
/usr/include/features.h:319:10: note: this is the location of the previous definition
  319 | # define _POSIX_C_SOURCE        202405L
      |          ^~~~~~~~~~~~~~~
In file included from /root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/Python.h:12,
                 from /tmp/tmpamaum31i/__triton_launcher.c:7:
/root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/pyconfig.h:1877:9: warning: ‘_POSIX_C_SOURCE’ redefined
 1877 | #define _POSIX_C_SOURCE 200809L
      |         ^~~~~~~~~~~~~~~
In file included from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:26,
                 from /opt/venv/lib/python3.12/site-packages/triton/backends/nvidia/include/cuda.h:56,
                 from /tmp/tmpamaum31i/__triton_launcher.c:2:
/usr/include/features.h:319:10: note: this is the location of the previous definition
  319 | # define _POSIX_C_SOURCE        202405L
      |          ^~~~~~~~~~~~~~~
In file included from /root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/Python.h:12,
                 from /tmp/tmp95ki_yzd/__triton_launcher.c:7:
/root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/pyconfig.h:1877:9: warning: ‘_POSIX_C_SOURCE’ redefined
 1877 | #define _POSIX_C_SOURCE 200809L
      |         ^~~~~~~~~~~~~~~
In file included from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:26,
                 from /opt/venv/lib/python3.12/site-packages/triton/backends/nvidia/include/cuda.h:56,
                 from /tmp/tmp95ki_yzd/__triton_launcher.c:2:
/usr/include/features.h:319:10: note: this is the location of the previous definition
  319 | # define _POSIX_C_SOURCE        202405L
      |          ^~~~~~~~~~~~~~~
In file included from /root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/Python.h:12,
                 from /tmp/tmpi3zxzbl5/__triton_launcher.c:7:
/root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/pyconfig.h:1877:9: warning: ‘_POSIX_C_SOURCE’ redefined
 1877 | #define _POSIX_C_SOURCE 200809L
      |         ^~~~~~~~~~~~~~~
In file included from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:26,
                 from /opt/venv/lib/python3.12/site-packages/triton/backends/nvidia/include/cuda.h:56,
                 from /tmp/tmpi3zxzbl5/__triton_launcher.c:2:
/usr/include/features.h:319:10: note: this is the location of the previous definition
  319 | # define _POSIX_C_SOURCE        202405L
      |          ^~~~~~~~~~~~~~~
[240B blob data]
In file included from /root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/Python.h:12,
                 from /tmp/tmpoxfs5ueo/__triton_launcher.c:7:
/root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/pyconfig.h:1877:9: warning: ‘_POSIX_C_SOURCE’ redefined
 1877 | #define _POSIX_C_SOURCE 200809L
      |         ^~~~~~~~~~~~~~~
In file included from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:26,
                 from /opt/venv/lib/python3.12/site-packages/triton/backends/nvidia/include/cuda.h:56,
                 from /tmp/tmpoxfs5ueo/__triton_launcher.c:2:
/usr/include/features.h:319:10: note: this is the location of the previous definition
  319 | # define _POSIX_C_SOURCE        202405L
      |          ^~~~~~~~~~~~~~~
In file included from /root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/Python.h:12,
                 from /tmp/tmptptdjtg4/__triton_launcher.c:7:
/root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/pyconfig.h:1877:9: warning: ‘_POSIX_C_SOURCE’ redefined
 1877 | #define _POSIX_C_SOURCE 200809L
      |         ^~~~~~~~~~~~~~~
In file included from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:26,
                 from /opt/venv/lib/python3.12/site-packages/triton/backends/nvidia/include/cuda.h:56,
                 from /tmp/tmptptdjtg4/__triton_launcher.c:2:
/usr/include/features.h:319:10: note: this is the location of the previous definition
  319 | # define _POSIX_C_SOURCE        202405L
      |          ^~~~~~~~~~~~~~~
In file included from /root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/Python.h:12,
                 from /tmp/tmpl31pxulc/__triton_launcher.c:7:
/root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/pyconfig.h:1877:9: warning: ‘_POSIX_C_SOURCE’ redefined
 1877 | #define _POSIX_C_SOURCE 200809L
      |         ^~~~~~~~~~~~~~~
In file included from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:26,
                 from /opt/venv/lib/python3.12/site-packages/triton/backends/nvidia/include/cuda.h:56,
                 from /tmp/tmpl31pxulc/__triton_launcher.c:2:
/usr/include/features.h:319:10: note: this is the location of the previous definition
  319 | # define _POSIX_C_SOURCE        202405L
      |          ^~~~~~~~~~~~~~~
In file included from /root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/Python.h:12,
                 from /tmp/tmp5fq6i5jz/__triton_launcher.c:7:
/root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/pyconfig.h:1877:9: warning: ‘_POSIX_C_SOURCE’ redefined
 1877 | #define _POSIX_C_SOURCE 200809L
      |         ^~~~~~~~~~~~~~~
In file included from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:26,
                 from /opt/venv/lib/python3.12/site-packages/triton/backends/nvidia/include/cuda.h:56,
                 from /tmp/tmp5fq6i5jz/__triton_launcher.c:2:
/usr/include/features.h:319:10: note: this is the location of the previous definition
  319 | # define _POSIX_C_SOURCE        202405L
      |          ^~~~~~~~~~~~~~~
In file included from /root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/Python.h:12,
                 from /tmp/tmprrts8jrq/__triton_launcher.c:7:
/root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/pyconfig.h:1877:9: warning: ‘_POSIX_C_SOURCE’ redefined
 1877 | #define _POSIX_C_SOURCE 200809L
      |         ^~~~~~~~~~~~~~~
In file included from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:26,
                 from /opt/venv/lib/python3.12/site-packages/triton/backends/nvidia/include/cuda.h:56,
                 from /tmp/tmprrts8jrq/__triton_launcher.c:2:
/usr/include/features.h:319:10: note: this is the location of the previous definition
  319 | # define _POSIX_C_SOURCE        202405L
      |          ^~~~~~~~~~~~~~~
In file included from /root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/Python.h:12,
                 from /tmp/tmpj5n8y3ez/__triton_launcher.c:7:
/root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/pyconfig.h:1877:9: warning: ‘_POSIX_C_SOURCE’ redefined
 1877 | #define _POSIX_C_SOURCE 200809L
      |         ^~~~~~~~~~~~~~~
In file included from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:26,
                 from /opt/venv/lib/python3.12/site-packages/triton/backends/nvidia/include/cuda.h:56,
                 from /tmp/tmpj5n8y3ez/__triton_launcher.c:2:
/usr/include/features.h:319:10: note: this is the location of the previous definition
  319 | # define _POSIX_C_SOURCE        202405L
      |          ^~~~~~~~~~~~~~~
In file included from /root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/Python.h:12,
                 from /tmp/tmptwz1sa93/__triton_launcher.c:7:
/root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/pyconfig.h:1877:9: warning: ‘_POSIX_C_SOURCE’ redefined
 1877 | #define _POSIX_C_SOURCE 200809L
      |         ^~~~~~~~~~~~~~~
In file included from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:26,
                 from /opt/venv/lib/python3.12/site-packages/triton/backends/nvidia/include/cuda.h:56,
                 from /tmp/tmptwz1sa93/__triton_launcher.c:2:
/usr/include/features.h:319:10: note: this is the location of the previous definition
  319 | # define _POSIX_C_SOURCE        202405L
      |          ^~~~~~~~~~~~~~~
[2026-09-22|07:31:14|FrontendAPI] WARNING  Aborting request for user 22
Process freetoken-TP0-scheduler:
Traceback (most recent call last):
  File "/root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/root/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/freetoken/python/freetoken/server/launch.py", line 116, in _run_scheduler
    scheduler.run_forever()
  File "/opt/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/freetoken/python/freetoken/scheduler/scheduler.py", line 305, in run_forever
    data = self.overlap_loop(data)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/freetoken/python/freetoken/scheduler/scheduler.py", line 251, in overlap_loop
    ongoing_data = (forward_input, self._forward(forward_input))
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/freetoken/python/freetoken/scheduler/scheduler.py", line 909, in _forward
    forward_output = self.engine.forward_batch(batch, sample_args)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/freetoken/python/freetoken/engine/engine.py", line 993, in forward_batch
    logits = self.graph_runner.replay(batch) if use_graph else self.model.forward()
                                                               ^^^^^^^^^^^^^^^^^^^^
  File "/opt/freetoken/python/freetoken/models/qwen3_5_moe/model.py", line 114, in forward
    output = self.model.forward(get_global_ctx().batch.input_ids)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/freetoken/python/freetoken/models/qwen3_5_moe/model.py", line 95, in forward
    x, residual = layer.forward(x, residual)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/freetoken/python/freetoken/utils/torch_utils.py", line 33, in wrapper
    return fn(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/freetoken/python/freetoken/models/qwen3_5_moe/model.py", line 73, in forward
    hidden = self.mlp.forward(hidden)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/freetoken/python/freetoken/models/qwen3_5_moe/moe.py", line 81, in forward
    routed = self.experts.forward(hidden_states=hidden_states, router_logits=router_logits)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/freetoken/python/freetoken/layers/moe.py", line 199, in forward
    final_hidden_states = self.prefill_forward(hidden_states, router_logits)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/freetoken/python/freetoken/layers/moe.py", line 251, in prefill_forward
    return self._prefill_routed(hidden_states, topk_weights, topk_ids)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/freetoken/python/freetoken/layers/moe.py", line 360, in _prefill_routed
    views = self._wait_prefill_overlap(cache)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/freetoken/python/freetoken/layers/moe.py", line 395, in _wait_prefill_overlap
    cache.prefetch_prefill_layer(self.layer_id + 1)
  File "/opt/freetoken/python/freetoken/moe/offload_cache.py", line 697, in prefetch_prefill_layer
    copy()
  File "/opt/freetoken/python/freetoken/moe/offload_cache.py", line 685, in copy
    self._invalidate_prefill_buffer(buffer_id)
  File "/opt/freetoken/python/freetoken/moe/offload_cache.py", line 639, in _invalidate_prefill_buffer
    self.slot_for_id.view(-1)[old_ids[old_ids >= 0].long()] = -1
                              ~~~~~~~^^^^^^^^^^^^^^
torch.AcceleratorError: CUDA error: unspecified launch failure
Search for `cudaErrorLaunchFailure' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

terminate called after throwing an instance of 'c10::AcceleratorError'
  what():  CUDA error: unspecified launch failure
Search for `cudaErrorLaunchFailure' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from currentStreamCaptureStatusMayInitCtx at /pytorch/c10/cuda/CUDAGraphsC10Utils.h:71 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x728f0537305d in /opt/venv/lib/python3.12/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xc114 (0x728f05700114 in /opt/venv/lib/python3.12/site-packages/torch/lib/libc10_cuda.so)
frame #2: <unknown function> + 0xcc775e (0x728e5066475e in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
frame #3: <unknown function> + 0x7f8e4 (0x728f053548e4 in /opt/venv/lib/python3.12/site-packages/torch/lib/libc10.so)
frame #4: c10::TensorImpl::~TensorImpl() + 0x9 (0x728f0534e279 in /opt/venv/lib/python3.12/site-packages/torch/lib/libc10.so)
frame #5: <unknown function> + 0x869d55 (0x728e7d51dd55 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0x869df1 (0x728e7d51ddf1 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #7: /opt/venv/bin/python() [0x18290a3]
frame #8: /opt/venv/bin/python() [0x18291d0]
frame #9: /opt/venv/bin/python() [0x1829263]
frame #10: /opt/venv/bin/python() [0x1801408]
frame #11: /opt/venv/bin/python() [0x18290a3]
frame #12: /opt/venv/bin/python() [0x182d2e3]
frame #13: /opt/venv/bin/python() [0x182cffd]
frame #14: /opt/venv/bin/python() [0x182d020]
frame #15: /opt/venv/bin/python() [0x182d020]
frame #16: /opt/venv/bin/python() [0x182d020]
frame #17: /opt/venv/bin/python() [0x182d020]
frame #18: /opt/venv/bin/python() [0x182d020]
frame #19: /opt/venv/bin/python() [0x182cdcc]
frame #20: /opt/venv/bin/python() [0x18290a3]
frame #21: _PyEval_EvalFrameDefault + 0x30a4 (0x18145a4 in /opt/venv/bin/python)
frame #22: PyEval_EvalCode + 0xde (0x18a54ba in /opt/venv/bin/python)
frame #23: /opt/venv/bin/python() [0x18a5c9d]
frame #24: PyRun_SimpleStringFlags + 0xe0 (0x196783e in /opt/venv/bin/python)
frame #25: /opt/venv/bin/python() [0x1ad9815]
frame #26: /opt/venv/bin/python() [0x197f61f]
frame #27: /opt/venv/bin/python() [0x197f42c]
frame #28: <unknown function> + 0x2a601 (0x728f06238601 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #29: __libc_start_main + 0x88 (0x728f06238718 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #30: _start + 0x29 (0x19a78a9 in /opt/venv/bin/python)

[2026-09-22|07:45:01|FrontendAPI] ERROR    Backend supervisor: backend worker freetoken-TP0-scheduler exited
[2026-09-22|07:45:11|FrontendAPI] ERROR    Backend worker is gone and cannot be restarted; stopping the API server

Anything else

Proxies / virtualization: No proxy, no WSL, no VM — bare metal, container only.

Unusual setup

• Dell Precision T7810, BIOS A34, 1350 W PSU with a custom dual-GPU power-delivery board, no BMC/IPMI. Every occurrence costs a hands-on cold power cycle, which is why the sample sizes above are small and why I have not rebuilt main.
• Two GPUs installed (3090 on 04:00.0, A1000 on 03:00.0); only the 3090 is used, with a single GPU exposed to the container.
• --moe-strategy offload is an explicit choice, not the default: per ft bench bw picks hybrid on Ampere, but offload is 8.3x faster (DeepSeek-V4-Flash, 2x RTX 3090) #151, ft bench bw selects hybrid on Ampere while offload measured 8.3x faster on a 3090, so I override the benchmark's pick.
• The engine runs in a Docker image built from source at a pinned commit rather than through the documented pip / Desktop path, and is driven over the OpenAI-compatible API by an agent harness.
• Expert weights live on host RAM (92 GiB total) and PLE is offloaded to disk, so PCIe and local disk are both in the hot path.

What I ruled out (so you do not have to ask)

• OOM — VRAM, cgroup and host RAM all checked.
• Sustained PSU overcurrent — peaks 242-248 W against a 350 W card limit.
• PCIe ASPM / link oscillation — every death occurred at full Gen3, and surviving runs logged more link-state transitions than the failing ones.
• Connector seating and cable sharing — separate 8-pin run to each GPU, reseated and verified.
• Driver and GSP version — identical for the 14 days before the first occurrence, and no newer GSP ships for this driver.
• Concurrency as a standalone cause — a single sequential worker crashed too.

What I could not determine: the mechanism. This host's PCIe path may simply be marginal, with FreeToken being the only workload I run that drives it hard enough. Offload fetches experts over PCIe every step, and a commenter on #151 measured roughly 78% of practical Gen3 x16 bandwidth on a similar 2x3090 setup. I am not claiming the engine is broken.

The 154 line for 0000:03:00 is not a second failure. The A1000 never logs a 79; I read it as the node-level recovery action being reported against every GPU the driver manages.

Questions

1. Is device loss under large chunk prefill in offload mode a known limitation on Ampere / PCIe-Gen3 hosts? Is there a recommended --max-prefill-length ceiling for offload mode?
2. Could the engine detect device loss and report it clearly? The failure currently surfaces as a CUDA error from three unrelated leaf frames — and the engine itself prints that the stack trace may be incorrect because CUDA errors are asynchronous — with no hint that the GPU has left the bus, which sends you looking for an engine bug first.
3. Would a guardrail help, for example a startup warning or an automatic cap when offload runs on a device whose link is narrower than the model's per-step fetch?
4. I will test a specific commit or flag combination if that is useful, but please note the cold-power-cycle cost above, so I would want each attempt to count. If you need a current-main data point, say so and I will run one.

[jason-fxz]

A larger chunk increases the GPU compute per chunk, not the PCIe traffic — per-chunk PCIe transfer in offload mode is the same regardless of chunk size. So this looks like heavy GPU compute combined with a thermal issue on the card. I'd suggest checking the GPU's thermals, e.g. temperatures under sustained load.

As for detecting device loss: handling GPU/driver-level failures is outside what the engine should do.