[GSD-13481] [BMG B65 / xe / GuC 70.72.1] Dual-GPU TP2 workload: GuC consumes SCHED_CONTEXT but leaves enabled LRC undispatched; ring tail advances/head stalls until 'not started' timeout

[weddle]

Proposed primary destination: https://github.com/intel/compute-runtime/issues
(justification in filing-recommendation section of the report.)
Secondary cross-reference: freedesktop GitLab DRM Xe tracker.

---

Summary

On 2× Intel Arc Pro B65 (Battlemage G31, 8086:e222) under the xe driver,
a deterministic two-GPU tensor-parallel LLM serving startup (vLLM TP=2,
80 MiB FP16 all-reduce during warmup) fails every time on the default
collective path while two alternative collective algorithms pass every
time on identical hardware and inputs.

Host-side submission is proven complete by kernel trace: job create →
exec → run → submit → H2G CTB SCHED_CONTEXT (0x1000) write → MMIO
GuC doorbell, all within ~1 ms, with the queue REGISTERED|ENABLED.
Live GuC CTB sampling shows the firmware consuming every message
(memory-head tracks tail through the failure, broken=0). Yet the LRC
ring head never advances, the engine stays idle, no page fault occurs,
and ~108 s later the job times out as "not started" (error -62).

So: a received, enabled GuC context is sometimes never dispatched by GuC
firmware under this reproducible dual-GPU workload. TMP_BUF-style
avoidance (a different submission pattern via CCL_SYCL_ALLREDUCE_TMP_BUF=1)
never triggers it (5/5 clean), which localizes the trigger to the default
submission sequence, not to data movement or memory capacity.

Hardware

• 2× Intel Arc Pro B65 32 GB, Battlemage G31, PCI 8086:e222, subsys 1849:6027
• 06:00.0 chipset-attached (rank 0), 0c:00.0 CPU-attached (rank 1); both xe
• Board: AMD B550, ASRock; host Fedora CoreOS; boot includes pci=noats
• P2P verified working both directions (~2.7–3.1 GB/s bulk)

Software

• Fedora kernel 7.1.8-200.fc44.x86_64, xe driver (in-tree)
• GuC firmware xe/bmg_guc_70.bin 70.72.1, RUNNING both cards
  (from intel-gpu-firmware-20260810-1.fc44.noarch; only GuC blob on disk)
• Level Zero userspace 26.27.39122.11-0 (libze-intel-gpu1), IGC 2.38.2
• torch 2.13.0+xpu, oneCCL Gold-2022.0.0, SYCL RT 2026.0.0
• vLLM 0.28.1rc1.dev388+g8a728663c.xpu, vllm-xpu-kernels 0.1.14.1

GuC firmware

bmg_guc_70.bin, version 70.72.1 (wanted 70.54.0), RUNNING on both cards,
IP ver 20.2.0. Only bmg_guc_70 blob present in /lib/firmware/xe/.

Reproduction

Model: Qwen3.8-27B OrcaRouter AutoRound W4A16/G128 (public Hugging Face
weights), served with vLLM TP=2, FP16 activations, FP8 KV,
--max-model-len 262144 --max-num-seqs 32 --max-num-batched-tokens 8192 --gpu-memory-utilization 0.95, PIECEWISE XPU graphs. Exact launch command
in reproduction.txt.

Expected: engine passes determine_available_memory/profile warmup.
Actual: during warmup, right after an 80 MiB [8192,5120] FP16 all-reduce
is dispatched on both ranks, both workers fail at the next synchronize
with UR_RESULT_ERROR_DEVICE_LOST / UR_RESULT_ERROR_OUT_OF_RESOURCES;
dmesg shows GuC "not started" timeouts + engine resets on both GPUs.

Note: a standalone model-free 80 MiB all-reduce passes — full
model-resident/profile state (weights resident, compiled profiling) is
required for deterministic reproduction.

Reproduction rate: default 5/5 fail; CCL_SYCL_ALLREDUCE_TMP_BUF=1 0/5
fail (profile completes); CCL_ALLREDUCE=ring 0/5 fail. Single-GPU and
dual-independent-replica serving of the same model work.

Failure signature

• Timed-out job "not started" (e.g. 0c:00.0, guc22, seqno 360), no page
  faults required, predecessor completes normally, DRM dependencies drain,
  no TLB timeouts, no earlier stuck occupant in 15K-line kernel trace.
• Preserved devcoredumps (separate equivalent failures): single pending
  context, LRC tail advanced (~1120 B and ~2352 B cases), last completed
  seqno = timedout − 1, engine registers idle.

Evidence chain

1. xe schedules/submits job (create/exec/run/submit trace records, 1:1).
2. Ring tail advances (coredump LRC state).
3. H2G SCHED_CONTEXT emitted (trace record, action 0x1000).
4. GuC notify/MMIO doorbell invoked (kprobe record, same ms).
5. GuC CTB head consumes the submission (1 Hz sampled head==tail,
   advancing, broken=0).
6. LRC head never advances; engine idle.
7. Context remains undispatched until "not started" timeout + reset.

Control

CCL_SYCL_ALLREDUCE_TMP_BUF=1 avoids the issue (different submission/
memory pattern, much faster than forced ring). This is a workaround
observation, not a claim that oneCCL is defective — the failure as
localized is below the L0 API, in GuC consumption-to-scheduling.

Expected / actual

Expected: GuC schedules the enabled context after consuming its
SCHED_CONTEXT. Actual: GuC consumes control traffic but the context sits
pending/idle until TDR.

Attachments

summary.json (machine-readable findings), dmesg-key-window.txt
(timeout + reset block), xe-trace-key-window.txt (submit→silence→
timeout for the stuck job), guc-ctb-key-series.csv (head==tail through
failure), devcoredump (+metadata showing pending context/idle
engine), version-inventory.txt, reproduction.txt (exact launch),
SHA256SUMS. Full raw archives (trace, server logs, 1 Hz CTB series)
available on request.

Questions for Intel

1. Does this match a known GuC scheduling issue on BMG / GuC 70.72.1?
2. Are there newer or older GuC firmware versions you recommend A/B
   testing (only bmg_guc_70 is on disk here)?
3. Is there a GuC firmware log decoder/tool available for the preserved
   level-1 guc_log dumps?
4. Would higher GuC log verbosity help, and what exact module/debugfs
   configuration should be used?
5. Is there additional firmware/scheduler state in the devcoredump you
   want captured from the next failure?
6. Is CCL_SYCL_ALLREDUCE_TMP_BUF=1 an acceptable temporary workaround?

Related issues (reviewed; filing separately rather than attaching)

• intel/compute-runtime [GSD-13010] Sporadic permanent GPU wedge on dual Arc Pro B70 (BMG G31): ccs/bcs engine reset + "Fault response: Unsuccessful -ENOENT/-EINVAL" under sustained Level-Zero inference load #948 (dual-B70 sporadic page-fault wedge, GuC 70.58.0): same die class and TP=2 shape, but sporadic multi-hour fault-driven wedge vs this deterministic zero-fault startup hang on newer firmware. Probably distinct mechanism; shared GuC-scheduling neighborhood.
• uxlfoundation/oneCCL unable to build for centos7 #212 (stale IPC handle): mechanism-adjacent page-fault path, but this failure reproduces with IPC caches disabled and zero faults, and GET reuse is structurally bypassed on the failing path.
• uxlfoundation/oneCCL OpenCL clBuildProgram intermittent segfault on Intel HD 520 on Ubuntu Linux 19.04 #215 and [Bug]: [XPU] Multi-GPU TP serving hangs on Intel Arc Pro B60 with torch 2.13 wheels (oneCCL 2022.x): warmup allreduce never returns, GuC timeouts, DEVICE_LOST` vllm-project/vllm#50545 (B60 large-AR warmup hang): same symptom family, closed with workarounds on older stacks; this report adds scheduling-level localization (submit→doorbell→CTB-consumed, never dispatched) those threads lack.
• intel/compute-runtime [GSD-13036] 4x Arc Pro B60: multi-GPU DDP training fails with UR_RESULT_ERROR_OUT_OF_RESOURCES at first kernel (large models only; single-GPU and small-model DDP both work) #952/[GSD-13038] Regression in 26.09+: dual-GPU workloads eagerly pin a host-RAM mirror of every large device USM allocation on the peer device, exhausting system memory #953 (host-RAM pin OOM): considered and excluded (different mechanism).

Key evidence excerpts

dmesg: GuC timeout + reset block (both GPUs)

[Thu Sep 17 19:21:41 2026] xe 0000:06:00.0: [drm] Tile0: GT0: Check job timeout: seqno=2428257, lrc_seqno=2428257, guc_id=0, not started
[Thu Sep 17 19:21:41 2026] xe 0000:06:00.0: [drm] Tile0: GT0: Timedout job: seqno=2428257, lrc_seqno=2428257, guc_id=0, flags=0x73 in no process [-1]
[Thu Sep 17 19:21:41 2026] xe 0000:06:00.0: [drm] Tile0: GT0: trying reset from guc_exec_queue_timedout_job [xe]
[Thu Sep 17 19:21:41 2026] xe 0000:06:00.0: [drm] Tile0: GT0: reset queued
[Thu Sep 17 19:21:41 2026] xe 0000:06:00.0: [drm] Tile0: GT0: reset started
[Thu Sep 17 19:21:41 2026] xe 0000:06:00.0: [drm] Tile0: GT0: reset done
[Thu Sep 17 19:21:41 2026] xe 0000:06:00.0: [drm] Tile0: GT0: Timedout job: seqno=330, lrc_seqno=330, guc_id=22, flags=0x20 in python [924129]

xe trace: stuck job submit → silence → timeout (0c/guc22, seqno 360)

VLLM::Worker_TP-901818  [008] ..... 690903.176768: xe_sched_job_exec: dev=0000:0c:00.0, fence=00000000e8abfb21, seqno=359, lrc_seqno=359, gt=0, guc_id=22, batch_addr=0xffffd556a77f1ac0, guc_state=0x3, flags=0x20, error=0
   kworker/u48:2-902171  [011] ..... 690903.176774: xe_sched_job_run: dev=0000:0c:00.0, fence=00000000e8abfb21, seqno=359, lrc_seqno=359, gt=0, guc_id=22, batch_addr=0xffffd556a77f1ac0, guc_state=0x3, flags=0x20, error=0
 VLLM::Worker_TP-901818  [008] ..... 690903.259596: xe_sched_job_exec: dev=0000:0c:00.0, fence=00000000aa8e88b5, seqno=360, lrc_seqno=360, gt=0, guc_id=22, batch_addr=0xffffd556a77f1b80, guc_state=0x3, flags=0x20, error=0
   kworker/u48:4-888717  [005] ..... 690903.259605: xe_sched_job_run: dev=0000:0c:00.0, fence=00000000aa8e88b5, seqno=360, lrc_seqno=360, gt=0, guc_id=22, batch_addr=0xffffd556a77f1b80, guc_state=0x3, flags=0x20, error=0
   kworker/u48:2-902171  [009] ..... 691011.959433: xe_sched_job_timedout: dev=0000:0c:00.0, fence=00000000aa8e88b5, seqno=360, lrc_seqno=360, gt=0, guc_id=22, batch_addr=0xffffd556a77f1b80, guc_state=0x200, flags=0x20, error=0
   kworker/u48:2-902171  [009] ..... 691011.959434: xe_sched_job_set_error: dev=0000:0c:00.0, fence=00000000aa8e88b5, seqno=360, lrc_seqno=360, gt=0, guc_id=22, batch_addr=0xffffd556a77f1b80, guc_state=0x200, flags=0x20, error=-62

H2G CTB head==tail through failure (DW units, 1 Hz samples)

wall,gpu,h2g_mem_head_dw,h2g_mem_tail_dw
2026-09-17T19:21:20,06:00.0,984,984
2026-09-17T19:21:23,06:00.0,990,990
2026-09-17T19:21:26,06:00.0,993,993
2026-09-17T19:21:29,06:00.0,999,999
2026-09-17T19:21:33,06:00.0,1002,1002
2026-09-17T19:21:36,06:00.0,982,982
2026-09-17T19:21:39,06:00.0,982,982
2026-09-17T19:21:42,06:00.0,982,982
2026-09-17T19:21:45,06:00.0,985,985
2026-09-17T19:21:49,06:00.0,484,908

devcoredump ring state (independent equivalent failure)

Single pending bcs context per GPU (1120 B and 2352 B cases): LRC Head behind Tail, last completed seqno = timedout seqno − 1, Schedule State 0x201, engine registers idle. Full dumps available on request.

[kgibala]

Hi @weddle,

Thank you for your contribution.

We kindly ask that issues be reported using the template described in the Issue Submission Guide. Without following this template, resolution time tends to be longer, as we need to ask follow-up questions to gather information that isn't included otherwise.

Could you please edit this issue to match the required format?

Thank you for your understanding.