feat(model): switch DEFAULT_MODEL to Qwen3-4B-Instruct-2507-GGUF

Decision driven by a head-to-head bench across 4 candidate LLMs on the
iran-showcase corpus (6 multi-event scenarios, same hardware, same
prompt v6, full results in /tmp/bench_1777810874/summary.md):

  model                            pass    avg    batch    p50
  Qwen3-4B-Instruct-2507-GGUF      5/6    6.7    58s      ~5s    ← winner
  Gemma-4-E4B-it-GGUF              6/6    7.0    170s     14s
  Qwen3.5-4B-GGUF                  4/6    6.4    80s      9s
  Gemma-4-E2B-it-GGUF              0/6    5.2    64s      10s

Why Qwen3-4B-Instruct-2507 wins:

  1. Highest pass-rate-per-second (5 PASSes in 58s = 5.2× Gemma's
     throughput for one fewer pass).
  2. Sub-5s p50 on 4 of 6 scenarios on M4 Pro — closest of the
     candidates to spec § 7.2 target (5s p50 / 8s p95 on Toughbook).
  3. **Non-thinking-only model** per HF card. Eliminates the entire
     `enable_thinking` patch tower we built for Qwen3.5 (no chain-
     of-thought leakage in reply text, no schema-template parroting
     under thinking-mode reload, no reasoning-content-burns-budget
     hangs). The chat_template_kwargs override and reply-text scrub
     stay in as defense-in-depth but become inert.
  4. Same Hermes tool-calling + same Qwen sampling distribution as
     Qwen3.5, so all our existing patches (parallel-call interceptor,
     server-side caps on free-text fields, sources coercion, insight
     prefix-strip, presence + frequency penalty, tighter max_tokens)
     apply unchanged.

Why NOT the alternates:
  - Gemma-4-E4B: perfect 6/6 PASS but 3× wall-time. Pick this only
    when the multi-axis-summary stress test has to pass.
  - Qwen3.5-4B: thinking-leakage class of failures we patched but
    can never fully eliminate. Strictly worse than 2507.
  - Gemma-4-E2B: 0/6 PASS — 2B params is below the schema-discipline
    threshold for Beacon's 14-tool extraction.

Across-repo updates (every Qwen3.5 reference now correctly listed as
a tested alternate, not the default):

  - src/beacon/agent.py             DEFAULT_MODEL + comment block + bench table
  - src/beacon/telemetry.py         docstring example (model_name + checkpoint)
  - dashboard/src/LLMStatusChip.tsx header pill example comment
  - CLAUDE.md                       Smoke baseline (with bench results) +
                                    Local-first inference paragraph
  - docs/manual-testing.md          backend matrix
  - docs/architecture-diagram.html  Model fact card + footer

Test updates:

  - test_qwen_model_injects_..._and_sampling: assertions on max_tokens
    aligned with the new 2048 cap and the split-knob anti-repetition
    (presence_penalty 0.3 + frequency_penalty 0.3, instead of
    presence_penalty 0.5 alone).
  - test_non_qwen_model_does_not_inject_chat_template_kwargs: asserts
    Qwen sampling overrides + frequency_penalty don't leak onto
    non-Qwen models, and the generic max_tokens=2048 floor still
    applies.

367 tests pass; 1 e2e skipped.

Author: kovtcharov

CLAUDE.md

@@ -75,18 +75,27 @@ of the 40 the spec calls for.
 **Phase 0 hardware probes were not run** because the dev box is a Mac M4 Pro,
 not Ryzen AI silicon. Lemonade still works (CPU/GPU path) — see Smoke baseline below.
 
-### Smoke baseline (2026-05-03, Mac M4 Pro, Qwen3.5-4B-GGUF on Lemonade)
+### Smoke baseline (2026-05-03, Mac M4 Pro, Qwen3-4B-Instruct-2507-GGUF on Lemonade)
 
-- iran-showcase batch (6 multi-event scenarios): **6/6 PASS** at avg 6.9/7.0,
-  total wall-clock ~22s on Apple Silicon GPU.
-- Single MEDEVAC extraction: **~3s**, all 9-line fields correct.
-- Canonical multi-event utterance: **~8s**, 4/4 tools called, all 12 params
-  correct (Class V correctly assigned to 5.56mm ammo).
+Head-to-head bench across 4 candidate models on the iran-showcase corpus
+(6 multi-event scenarios), same hardware, same prompt v6:
 
-Earlier baseline runs with Gemma-4-E4B-it-GGUF on the same hardware took
-5-30× longer (e.g. 198s for the multi-axis sitrep vs ~6s on Qwen3.5-4B) —
-Qwen3.5-4B is the new default for that reason; Gemma remains a tested
-alternate via `BEACON_MODEL=Gemma-4-E4B-it-GGUF`.
+```
+  model                            pass   avg    batch    p50/scenario
+  Qwen3-4B-Instruct-2507-GGUF      5/6    6.7    58s      ~5s    ← default
+  Gemma-4-E4B-it-GGUF              6/6    7.0    170s     14s
+  Qwen3.5-4B-GGUF                  4/6    6.4    80s      9s
+  Gemma-4-E2B-it-GGUF              0/6    5.2    64s      10s
+```
+
+Qwen3-4B-Instruct-2507 wins the tradeoff: highest pass-rate-per-second,
+sub-5s on 4 of 6 scenarios on M4 Pro, AND it's a non-thinking-only model
+per the HF card — so no `enable_thinking` patch tower is needed (eliminates
+the chain-of-thought-leakage class of failures we saw on Qwen3.5).
+
+Gemma-4-E4B-it-GGUF remains the highest-accuracy alternate at the cost of
+3× wall-time; pick it via `BEACON_MODEL=Gemma-4-E4B-it-GGUF` when the
+multi-axis-summary stress test must pass.
 
 Latency on this dev box is not representative of fielding hardware (Ryzen AI
 NPU acceleration is Windows/Linux only). On a Toughbook-class laptop the
@@ -201,7 +210,7 @@ Key seams that span multiple files:
 
 - **InsightsEngine + ReasoningLog**: `src/beacon/insights.py` is a **trigger-lane subscriber with a 1s trailing debounce** that calls `agent.analyze(snapshot_json)` — a tool-less LLM pass that returns a JSON array of `Insight`. Heuristics were removed (commit `95e64d6` + the agent-only rewrite per `docs/superpowers/specs/2026-05-03-remove-heuristic-insights-design.md`). Per AGENTS.md § 2.3, parse failures and connection errors return `[]` and log; we **do not fabricate insights to fill the gap**. `src/beacon/reasoning.py` is a 500-step in-memory ring buffer; `_execute_tool` writes TOOL steps, the InsightsEngine writes QUERY/SURFACE. The dashboard's reasoning column streams from `/agent/reasoning/stream`.
 
-- **Local-first inference**: Lemonade Server is the default. The agent's `DEFAULT_MODEL` is `Qwen3.5-4B-GGUF` (Unsloth's Qwen 3.5 4B, GGUF-quantised) — picked because it's 5-30× faster than Gemma-4-E4B on Apple Silicon for the same extraction tasks (iran-showcase batch: ~22s on Qwen vs ~6 minutes on Gemma), Lemonade-tagged `tool-calling` + `hot`, and vision-capable for the future image path. Override with `BEACON_MODEL=…` (other tested options: `Gemma-4-E4B-it-GGUF`, `Qwen3-4B-Instruct-2507-GGUF`). The 30B-class model the spec originally called for is still too large for current dev hardware; swap when Phase 0.1 is re-run on Ryzen AI silicon. OpenAI fallback is opt-in via `BEACON_USE_OPENAI=1`; Claude fallback is `BEACON_USE_CLAUDE=1`. If Lemonade is unreachable and no opt-in is set, the server returns 502 with an actionable detail — never silently fall back. The backend lifespan POSTs Lemonade `/api/v1/load` at startup with `ctx_size=BEACON_CTX_SIZE` (default 32768) so the active model serves with enough context for multi-turn extraction — Lemonade's server-wide `ctx_size` config (set once via `lemonade config set ctx_size=32768`) overrides per-request values, so changing `BEACON_CTX_SIZE` alone won't take effect unless the server config matches.
+- **Local-first inference**: Lemonade Server is the default. The agent's `DEFAULT_MODEL` is `Qwen3-4B-Instruct-2507-GGUF` (Qwen 3 4B Instruct, July 2025 release, GGUF-quantised) — picked from the head-to-head bench above because it ties the throughput sweet spot (5/6 PASS in 58s) with being a non-thinking-only model (no `enable_thinking` patches needed, no chain-of-thought leakage). Same Hermes tool-calling + Qwen sampling distribution as Qwen3.5, so the parallel-call interceptor and Qwen-specific sampling overrides apply unchanged. Override with `BEACON_MODEL=…` (other tested options: `Gemma-4-E4B-it-GGUF` for highest accuracy at 3× latency cost, `Qwen3.5-4B-GGUF` for vision support). The 30B-class model the spec originally called for is still too large for current dev hardware; swap when Phase 0.1 is re-run on Ryzen AI silicon. OpenAI fallback is opt-in via `BEACON_USE_OPENAI=1`; Claude fallback is `BEACON_USE_CLAUDE=1`. If Lemonade is unreachable and no opt-in is set, the server returns 502 with an actionable detail — never silently fall back. The backend lifespan POSTs Lemonade `/api/v1/load` at startup with `ctx_size=BEACON_CTX_SIZE` (default 32768) so the active model serves with enough context for multi-turn extraction — Lemonade's server-wide `ctx_size` config (set once via `lemonade config set ctx_size=32768`) overrides per-request values, so changing `BEACON_CTX_SIZE` alone won't take effect unless the server config matches.
 
   **Qwen-specific tuning** (lives in `BeaconAgent._intercept_chat_send`, fires whenever `"qwen" in self.model_id.lower()`):
   - **Thinking mode is force-disabled.** Qwen3-family GGUFs ship with reasoning ON by default — the model emits a multi-page `Thinking Process:` chain-of-thought before any answer. With thinking ON, `agent.analyze()`'s JSON-array-output contract burns 4+ minutes generating reasoning and never reaches the final array (content stays empty, 0 insights). We pass `chat_template_kwargs={"enable_thinking": False}` on every send and bump `max_tokens` to 16384 so non-thinking outputs have room. If/when token-level streaming to the dashboard reasoning panel lands (Phase 2), re-enable thinking and use `_capture_reasoning_content` to surface `reasoning_content` live as `REASONING` reasoning steps.

dashboard/src/LLMStatusChip.tsx

@@ -77,8 +77,9 @@ function formatTokenCount(n: number): string {
 }
 
 /** Trim noisy suffixes from the model name so it fits in the header pill.
- *  "Qwen3.5-4B-GGUF" → "Qwen3.5-4B". The full name is still in the
- *  hover tooltip, so curious operators can see the quantization level. */
+ *  "Qwen3-4B-Instruct-2507-GGUF" → "Qwen3-4B-Instruct-2507". The full
+ *  name is still in the hover tooltip, so curious operators can see the
+ *  quantization level. */
 function shortenModelName(name: string | null | undefined): string {
   if (!name) return "LLM";
   return name

docs/architecture-diagram.html

@@ -486,7 +486,7 @@ <h1>Beacon</h1>
     </div>
     <div class="brand-meta">
       <div>architecture diagram · <strong>real wiring only</strong></div>
-      <div>Phases 1–8 + 10 shipped · 346 tests · Qwen3.5-4B-GGUF + v6 prompt</div>
+      <div>Phases 1–8 + 10 shipped · 346 tests · Qwen3-4B-Instruct-2507-GGUF + v6 prompt</div>
     </div>
   </header>
 
@@ -552,7 +552,7 @@ <h2><span class="accent">▲ BeaconAgent</span> &nbsp;<span style="color:var(--t
       </div>
       <div class="fact">
         <span class="k">Model</span>
-        <span class="v">Qwen3.5-4B-GGUF<br/><span class="alt">32k ctx · POST /api/v1/load at boot · BEACON_MODEL / BEACON_CTX_SIZE overrides</span></span>
+        <span class="v">Qwen3-4B-Instruct-2507-GGUF<br/><span class="alt">32k ctx · non-thinking-only · POST /api/v1/load at boot · BEACON_MODEL / BEACON_CTX_SIZE overrides</span></span>
       </div>
       <div class="fact">
         <span class="k">Confirm mode</span>

docs/manual-testing.md

@@ -24,7 +24,7 @@ LLM backend — pick one:
 
 | Backend | How to enable | Notes |
 |---|---|---|
-| Local Lemonade (default) | `lemonade-server serve` in a separate shell | Default model is `Qwen3.5-4B-GGUF` (override via `BEACON_MODEL`; tested alternates: `Gemma-4-E4B-it-GGUF`, `Qwen3-4B-Instruct-2507-GGUF`). Beacon's lifespan startup POSTs `/api/v1/load` with `ctx_size=BEACON_CTX_SIZE` (default 32768) so multi-turn extraction has room — but Lemonade's server-wide config also caps it, so run `lemonade config set ctx_size=32768` once if requests overflow with the per-request override. |
+| Local Lemonade (default) | `lemonade-server serve` in a separate shell | Default model is `Qwen3-4B-Instruct-2507-GGUF` (override via `BEACON_MODEL`; tested alternates: `Gemma-4-E4B-it-GGUF` for max accuracy at 3× latency, `Qwen3.5-4B-GGUF` for vision support). Beacon's lifespan startup POSTs `/api/v1/load` with `ctx_size=BEACON_CTX_SIZE` (default 32768) so multi-turn extraction has room — but Lemonade's server-wide config also caps it, so run `lemonade config set ctx_size=32768` once if requests overflow with the per-request override. |
 | OpenAI | `export BEACON_USE_OPENAI=1 OPENAI_API_KEY=sk-…` | Use when no GPU / no Lemonade. |
 | Claude | `export BEACON_USE_CLAUDE=1 ANTHROPIC_API_KEY=sk-…` | Same. |
 | None (HTTP / dashboard smoke test only) | `export BEACON_SKIP_LEMONADE=1 BEACON_DEFER_AGENT=1` | `/agent/run` will 502, but `/health`, `/state`, dashboard polling all work. |

src/beacon/agent.py

@@ -104,27 +104,41 @@ class BeaconAgent(RAGToolsMixin, Agent):
     GAIA's actual `Agent.__init__` signature.
     """
 
-    # ``Qwen3.5-4B-GGUF`` is the canonical model. Unsloth's Qwen3.5 4B
-    # (GGUF-quantised for Lemonade's llama.cpp backend) — picked because:
-    #   - 5-30× faster than Gemma-4-E4B on Apple Silicon for the same
-    #     extraction tasks (iran-showcase: ~22s for 6 scenarios vs.
-    #     ~6 minutes on Gemma)
-    #   - Lemonade-tagged ``tool-calling`` + ``hot``: reliable structured
-    #     `tool_calls` emission, GAIA-compatible without our parallel-call
-    #     workaround having to fire on most inputs
-    #   - Honors Beacon's structured-output contract (schema validation
-    #     errors below 5% on the eval scenario set)
-    #   - Vision-capable (mmproj bundled) so the same instance can take
-    #     image inputs once we wire the vision path
+    # ``Qwen3-4B-Instruct-2507-GGUF`` is the canonical model. Picked from a
+    # head-to-head iran-showcase × 4-model bench (2026-05-03):
+    #
+    #   model                        pass    avg    batch    p50/scenario
+    #   Qwen3-4B-Instruct-2507       5/6    6.7    58s      ~5s    ← winner
+    #   Gemma-4-E4B-it-GGUF          6/6    7.0    170s     14s
+    #   Qwen3.5-4B-GGUF              4/6    6.4    80s      9s
+    #   Gemma-4-E2B-it-GGUF          0/6    5.2    64s      10s
+    #
+    # Why this one:
+    #   - Highest pass-rate-per-second (5 PASSes in 58s = 5.2× Gemma's
+    #     throughput for one fewer pass).
+    #   - **Non-thinking-only** model per HF card. No `enable_thinking`
+    #     patch tower needed; the chain-of-thought leakage we patched on
+    #     Qwen3.5 doesn't exist here at the source.
+    #   - Closest to spec target (5s p50 / 8s p95 on Toughbook-class
+    #     hardware) — sub-5s on 4 of 6 scenarios on M4 Pro.
+    #   - Same Hermes tool-calling + same Qwen sampling distribution as
+    #     Qwen3.5, so the parallel-call interceptor and Qwen sampling
+    #     overrides apply unchanged.
     #
     # Override via ``BEACON_MODEL`` env var or by passing ``model="…"``
-    # to the constructor. Tested alternates: ``Gemma-4-E4B-it-GGUF`` (10×
-    # slower on M-series Macs but available on every Lemonade installation),
-    # ``Qwen3-4B-Instruct-2507-GGUF``. The agent-spec originally called for
-    # the 30B-class ``Qwen3-Coder-30B-A3B-Instruct-GGUF`` for fielding-grade
-    # accuracy; revisit when Phase 0.1 (tool-calling reliability gate) is
-    # re-run on Ryzen AI hardware.
-    DEFAULT_MODEL = os.getenv("BEACON_MODEL", "Qwen3.5-4B-GGUF")
+    # to the constructor. Tested alternates and their tradeoffs:
+    #   * ``Gemma-4-E4B-it-GGUF`` — perfect 6/6 PASS but 3× slower; pick
+    #     this if accuracy on the multi-axis-summary stress test matters
+    #     more than dashboard responsiveness.
+    #   * ``Qwen3.5-4B-GGUF`` — slightly slower than 2507; suffers from
+    #     reasoning-mode leakage that we have to scrub server-side.
+    #   * ``Gemma-4-E2B-it-GGUF`` — fast, but 0/6 PASS. 2B params is
+    #     not enough for Beacon's schema discipline.
+    # The agent-spec originally called for the 30B-class
+    # ``Qwen3-Coder-30B-A3B-Instruct-GGUF`` for fielding-grade accuracy;
+    # revisit when Phase 0.1 (tool-calling reliability gate) is re-run
+    # on Ryzen AI hardware.
+    DEFAULT_MODEL = os.getenv("BEACON_MODEL", "Qwen3-4B-Instruct-2507-GGUF")
 
     def __init__(
         self,

src/beacon/telemetry.py

@@ -58,8 +58,8 @@
 
 Verified on
 -----------
-Apple M4 Pro (Mac16,8), macOS 26.2, Lemonade running Qwen3.5-4B-GGUF on
-port 8002 with ``-ngl 99``. Idle GPU baseline ~10%, inference peak 97%,
+Apple M4 Pro (Mac16,8), macOS 26.2, Lemonade running Qwen3-4B-Instruct-2507-GGUF
+on port 8002 with ``-ngl 99``. Idle GPU baseline ~10%, inference peak 97%,
 post-inference settle <5% within 2s.
 """
 
@@ -362,9 +362,9 @@ def fetch_lemonade_active_llm(force_refresh: bool = False) -> Optional[dict]:
     Returns shape (single dict from ``all_models_loaded``):
       {
         "backend_url": "http://127.0.0.1:8001/v1",
-        "checkpoint": "unsloth/Qwen3.5-4B-GGUF:Qwen3.5-4B-UD-Q4_K_XL.gguf",
+        "checkpoint": "unsloth/Qwen3-4B-Instruct-2507-GGUF:Qwen3-4B-Instruct-2507-Q4_K_M.gguf",
         "device": "gpu",
-        "model_name": "Qwen3.5-4B-GGUF",
+        "model_name": "Qwen3-4B-Instruct-2507-GGUF",
         "recipe_options": {"ctx_size": 32768},
         "type": "llm",
         ...
@@ -598,9 +598,10 @@ def read_llm_stats(force_refresh: bool = False) -> LLMStats:
         if m:
             max_ctx = int(m.group(1))
 
-    # Prefer the orchestrator's `model_name` for display ("Qwen3.5-4B-GGUF")
-    # over the gguf filename ("Qwen3.5-4B-UD-Q4_K_XL.gguf") — it's the user-
-    # facing alias they configured + matches what shows up in Lemonade's UI.
+    # Prefer the orchestrator's `model_name` for display
+    # ("Qwen3-4B-Instruct-2507-GGUF") over the gguf filename
+    # ("Qwen3-4B-Instruct-2507-Q4_K_M.gguf") — it's the user-facing alias
+    # they configured + matches what shows up in Lemonade's UI.
     orchestrator_model_name: Optional[str] = None
     if orchestrator_entry is not None:
         n = orchestrator_entry.get("model_name")

tests/test_agent.py

@@ -179,9 +179,10 @@ def test_qwen_model_injects_enable_thinking_false_max_tokens_and_sampling(tmp_db
 
     - ``chat_template_kwargs={"enable_thinking": False}`` (kills reasoning
       mode that otherwise burns the whole token budget on chain-of-thought)
-    - ``max_tokens=4096`` (enough for non-thinking JSON / tool_calls,
-      previously 16384 — that was thinking-on-era headroom and only
-      gave the model rope to drift into placeholder-parroting)
+    - ``max_tokens=2048`` (enough for non-thinking JSON / tool_calls;
+      tighter than 4096/16384 so when Qwen's sampling collapses into
+      a runaway-repetition loop, we hit `finish_reason=length` in ~50s
+      instead of 100s, halving the worst-case wasted-tokens cost)
     - Qwen-recommended sampling (``temperature=0.7``, ``top_p=0.8``,
       ``top_k=20``) per Unsloth + Qwen Quickstart docs. "DO NOT use
       greedy decoding (temp=0)" — Beacon's instinct toward determinism
@@ -212,9 +213,9 @@ def test_qwen_model_injects_enable_thinking_false_max_tokens_and_sampling(tmp_db
 
         assert len(captured) == 1
         kwargs = captured[0]
-        assert kwargs.get("max_tokens") == 4096, (
-            "Qwen non-thinking needs ~4K, not 16K — large budgets just "
-            f"give the model rope to drift; got {kwargs.get('max_tokens')!r}"
+        assert kwargs.get("max_tokens") == 2048, (
+            "Qwen non-thinking needs ~2K — tighter cap halves the worst-case "
+            f"runaway-loop cost; got {kwargs.get('max_tokens')!r}"
         )
         assert kwargs.get("chat_template_kwargs") == {"enable_thinking": False}, (
             "Qwen3.5-* default reasoning ON breaks JSON-array contracts; "
@@ -225,13 +226,19 @@ def test_qwen_model_injects_enable_thinking_false_max_tokens_and_sampling(tmp_db
         )
         assert kwargs.get("top_p") == 0.8, f"got top_p={kwargs.get('top_p')!r}"
         assert kwargs.get("top_k") == 20, f"got top_k={kwargs.get('top_k')!r}"
-        # presence_penalty breaks the "endless repetitions" failure mode
-        # we hit in production (4000-char word-salad in `details` fields).
-        # Small positive value per Qwen docs; ≤ 1.0 to avoid CJK leakage.
-        assert kwargs.get("presence_penalty") == 0.5, (
+        # presence_penalty + frequency_penalty break the "endless
+        # repetitions" failure mode (4000-char word-salad in `details`
+        # fields). Split across the two knobs because presence alone at
+        # higher values caused JSON-syntax drift; combining them at 0.3
+        # each gives anti-repetition without disrupting structure.
+        assert kwargs.get("presence_penalty") == 0.3, (
             f"presence_penalty must be set to break repetition loops; "
             f"got {kwargs.get('presence_penalty')!r}"
         )
+        assert kwargs.get("frequency_penalty") == 0.3, (
+            f"frequency_penalty targets recently-used tokens; "
+            f"got {kwargs.get('frequency_penalty')!r}"
+        )
     finally:
         a.close()
 
@@ -269,8 +276,8 @@ def test_non_qwen_model_does_not_inject_chat_template_kwargs(tmp_db):
                 f"Non-Qwen models must not get Qwen sampling override "
                 f"{forbidden}={kwargs.get(forbidden)!r}"
             )
-        # max_tokens is generic — should still be bumped.
-        assert kwargs.get("max_tokens") == 4096
+        # max_tokens is generic — should still be bumped (matches Qwen path).
+        assert kwargs.get("max_tokens") == 2048
     finally:
         a.close()