feat(server): opt-in logprobs on chat completions (AR path, qwen35 + ds4)

[Graffioh]

Summary

Adds logprobs / top_logprobs to /v1/chat/completions as a QA-only, opt-in feature so the fidelity tooling in luce_box/lucebox_qa (Luce-Org/luce_box#168) can run a margin bench against dflash_server. Nothing changes for requests that do not set logprobs: true.

• Request: logprobs: bool (default false), top_logprobs: int 0–20 (default 0, requires logprobs: true).
• Response: choices[0].logprobs = {content: [...], reasoning_content: [...]} with OpenAI-shaped entries (token, token_id, logprob, bytes, top_logprobs). Invariant: join(content[].token) == message.content, same for reasoning_content; stop-sequence truncation keeps it.
• Semantics: log-softmax over the raw target logits at the committed position, before penalties and temperature, double precision, ties to lower token id. This is deliberately not OpenAI's post-sampling semantics; it is what a fidelity measurement needs. Documented in server/docs/API.md → "Logprobs".
• Routing: a logprobs request sets force_ar_decode (existing knob) and bypasses the GPU-argmax / GPU-sampler shortcuts so the full logits row is read to the CPU each step; it also skips prefix-cache restore/save (an exact-hit restore yields the first token with no logits row).
• Plumbing: new common/logprobs.{h,cpp} (compute_token_logprob), GenerateRequest::logprobs_top_k, DaemonIO::emit_with_logits + on_token_logprob, ModelBackend::supports_logprobs() (true for qwen35 and deepseek4). The response builder fails with 500 if the record count does not match committed tokens, so a missed commit site can never serve shifted probabilities.

Scope / limits (400 with a clear message)

• non-streaming only (stream: true + logprobs → 400)
• single-slot only (--max-concurrency > 1 / scheduler path → 400 logprobs requires --max-concurrency 1)
• qwen35 and deepseek4 backends only (others → 400)

Follow-ups (not in this PR)

• logprobs on the speculative-decode path (DFlash chain verify, DSpark) — needs the verify-batch logit rows read per accepted token in every backend; perf-sensitive, more work
• streaming logprobs, scheduler/concurrency support
• teacher-forced per-token KLD dump (the other half of what lucebox_qa/logit_tools needs)

Cost

qwen35 (Qwen3.8-27B IQ4_XS, R9700): AR 30.9–31.0 tok/s → 29.7–30.0 tok/s with logprobs (~3–4%, the forced logits D2H per step). deepseek4 hybrid: ~0.4% (its AR path already keeps logits CPU-side). Both vs. the spec-decode default the request opts out of.

Test plan

• test_server_unit: 592/592 (new test_logprobs.cpp: log-softmax normalisation, argmax max, top-K order + tie-break, K=0/K>vocab, ±1e4 finite; new parse_request_logprobs validation cases)
• server/tests/test_server_logprobs.py against a real qwen35 server (Qwen3.8-27B + DFlash2 draft, block 16, gfx1201): 27/27 — shape, key absent when unrequested, bounds, top-K ordering and greedy chosen == top[0], concatenation invariants incl. thinking on, byte-identical message.content with and without logprobs (spec vs forced AR), six 400 cases, stop-sequence truncation
• qwen35 with --max-concurrency 4: logprobs → 400, plain → 200
• deepseek4 hybrid (docs/DS4.md R9700 recipe) AR-only: 27/27; with DSpark spec enabled: 26/26 + 1 warning — spec-verify vs cold-AR output differs on the hybrid expert path (17 + 25 = 42 vs 57, first token at logprob ≈ −0.98). Pre-existing numerics divergence unrelated to this change; the test has --allow-spec-divergence to record it as a warning when usage.spec_decode_ran is true
• Re-run on Strix Halo (gfx1151): 27/27 with DFlash2 draft (block 16), 27/27 single-slot --paged-attention; sampled-mode spot check OK
• Wire lucebox_qa/logit_tools/vendor/probe_router.py against this and record the first margin panel

Generated with Devin