Skip to content

Phi-4-mini on CPU and GPU, GPTQ Vulkan through a gate proxy, and a stricter publish gate - #28

Merged
alpharomercoma merged 10 commits into
mainfrom
phi4-mini
Nov 9, 2026
Merged

alpharomercoma merged 10 commits into
mainfrom
phi4-mini

Conversation

@alpharomercoma

@alpharomercoma alpharomercoma commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

Adds Phi-4-mini-instruct (3.8B) on XNNPACK and Vulkan. Lets a Vulkan export take GPTQ codes, gated through the XNNPACK file built from the same codes. Closes the ways a file could be published without passing the checks its recipe calls for. Five rounds of adversarial review with codex found real defects in every round; all are fixed here.

Both repos below were built from this branch and are live: Phi-4-mini-instruct-3.8B-ExecuTorch and Qwen3-4B-Thinking-2507-ExecuTorch. Each has CPU and GPU files at 2k to 32k.

Phi-4-mini

  • Recipe. export_llm's phi_4_mini with our params and a converter that splits the fused qkv_proj and gate_up_proj where HF does.
    • Before quantization, ExecuTorch's model matches transformers' fp32 Phi3ForCausalLM within 0.003 in the logits on all 16 gate prompts (0.0003 at 8k against HF's long factors).
    • Outputs are named Phi-4-mini-instruct-3.8B (naming.SIZED_NAMES), because the app reads the size from the name.
  • ffn_dim_multiplier trap. Upstream's phi_4_mini config sets it to 1, which makes Rope build its table at twice the window. A 4k file would then run LongRoPE's long factors on every prompt. It's dropped for Phi, so 2k and 4k use HF's short factors exactly. 8k and larger windows use the long factors from position 0, as llama.cpp does by context size; the short-prompt cost is measured in PLAN.
  • Tools in the system message. Phi's chat template reads its tools from the system message, not tools=, so calibration and gate prompts carried no tool schema (found by codex). render_app now puts them there, byte-identical to the app's Phi4Prompt.
  • Int8 output head. XnnpackPlan.output_bits = 8: the tied 200,064 × 3072 head gets int8 per-channel weights.
    • Measured eager against fp32: an int4 head kept 7/8 tool calls, an int8 head 8/8.
    • On the real .pte it still keeps 7/8 (0.51 against fp32's 0.66), because fp32 itself sits near a tie on several rows.
  • Operator override. verify --accept-gate REASON publishes past a failed gate only with a recorded reason. The measured verdict stays passed: false, the reason is stored beside it, and the README states each window's shortfall. The published Phi files used it, with the operator's approval.

Vulkan with GPTQ codes

  • export-vulkan --codes --gate-proxy <xnnpack report>. The proxy must match the build: source repo and revision, window, codes sha256, ExecuTorch version, and every graph- or weight-shaping recipe field. Its gate must have passed or been accepted.
  • Round to nearest kept 0 of Phi's 8 tool calls; the first two Phi Vulkan files were withdrawn from the Hub for it.
  • The export-vulkan workflow does not pass --codes yet, so CI's Vulkan files remain round to nearest.

Gate and publish

  • Thinking-only templates. The gate closes a think block that a reasoning-only template leaves open (Qwen3-4B-Thinking-2507). Every published family renders as before. That model still reasons before deciding, so measures_tool_calling is false for it (PLAN, Known limits).
  • Gated reports must pass. verify re-gates a report whose gate is missing or failed, or whose .pte was rebuilt. publish-hf and publish-release share require_publishable: a GPTQ file needs its own passing gate (or, for Vulkan, a passing proxy).
  • Byte binding. Publish refuses a file whose sha256 differs from its report, and any model file its report doesn't list.
  • Codes integrity. An export reads its codes from a private, hashed copy. A codes file that leaves an int4 linear unsolved, or holds no codes at all, is refused.

Testing

  • Unit tests. pytest -q: 361 passed, 1 skipped; ruff check and ruff format --check clean. New tests cover:
    • Phi params against ExecuTorch's own config, the fused-weight split, and the RoPE table's factor choice per window;
    • the sized name and eligibility;
    • Phi's tool rendering with its real template, and the think-block fix;
    • the int8 head and refusing partial codes;
    • the gate proxy;
    • gate, override and byte binding in verify and publish.
  • Real exports. Ten Phi and ten Qwen3-4B-Thinking-2507 files on x86 EC2, each through export, verify (smoke test plus gate on XNNPACK, structural check plus proxy on Vulkan) and publish-hf.

🤖 Generated with Claude Code


Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need. Autofix is enabled.

alpharomercoma and others added 10 commits November 9, 2026 08:46
…g-only template

Phi-4-mini-instruct (Phi3ForCausalLM) gets a recipe for export_llm's phi_4_mini class:
- The fused qkv_proj and gate_up_proj are split where HF splits them.
- LongRoPE's factors pass through as ExecuTorch 1.5.1 reads them.
- ffn_dim_multiplier is dropped. Upstream's config sets it to 1, which makes Rope build its
  table at twice the window, so a 4k export would run every prompt on the long factors
  that HF only uses past 4096 tokens.
- Outputs are named Phi-4-mini-instruct-3.8B, because the app reads the size from the name
  and the release has none.
- Its chat template reads tools from the system message, not from tools=, so calibration and
  the gate now put the schema there, as the app does. Found by codex review: before this, the
  Phi rows carried no tools at all.

The gate closes a think block that a template leaves open. Qwen3-4B-Thinking-2507 opens one
whatever enable_thinking says, which put the first word of the reasoning in the graded
position. Every published family renders exactly as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
verify skipped any report that already had a smoke result, even when it was asked to gate.
So a first run that passed the smoke test and then failed, or never reached, the decision
gate left a report that a second run returned as verified with nothing gated. publish-hf
checked only the smoke result, so that report could be published. Now verify gates it again,
and publish refuses a report whose gate failed. A report with no gate at all (Vulkan, or a
run without an fp32 reference) publishes on its smoke test, as before. Found by codex review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phi-4-mini's GPTQ CPU export missed the decision gate by one row: it kept 7 of 8 tool calls,
at a mean of 0.45 against fp32's 0.66. Measured eager on the gate's prompts against fp32,
holding everything else fixed, the int4 output head was the cost:

- int4 everywhere kept 7/8 at 0.457 (the .pte measured 0.452);
- an fp32 head kept 8/8 at 0.521;
- an int8 per-channel head kept 8/8 at 0.526.

The head is the tied 200,064 x 3072 embedding, a sixth of the model, and fp32 is itself only
0.40 to 0.78 sure on these rows. XnnpackPlan.output_bits sets the head's bits; Phi-4-mini
uses 8, which adds about 0.27 GB to the file. The head gets int8 weights per channel with
the same dynamic int8 activations, and its int4 codes are skipped.

export-vulkan takes --codes. The Vulkan delegate runs the same torchao linears, and
round-to-nearest int4 kept none of Phi's 8 tool calls (0.05 against 0.66). There is no
Vulkan runtime on the host, so a Vulkan file is gated through the XNNPACK file solved from
the same codes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hat stands for it

Codex review, round three:

- publish-hf and publish-release share one check now. A GPTQ report must carry a passing
  gate of its own, because the solve always writes an fp32 reference, so a GPTQ report
  without a gate is one whose gate crashed or never ran. A Vulkan GPTQ report, which the
  host cannot run, must instead carry a passing gate proxy. publish-release checked
  nothing before.
- export-vulkan --codes needs --gate-proxy: the XNNPACK report from the same source
  revision, window, codes (sha256, now recorded in every GPTQ report) and output head, whose
  own gate passed. It is checked before the export and recorded in the Vulkan report.
- A codes file that leaves an int4 linear unsolved is refused, not exported as GPTQ with
  that linear rounded.
- The size and phone-fit estimates price an int8 output head at a byte a weight.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r's recorded reason

For Phi-4-mini the operator accepted the 4-bit files with an int8 head, which keep 7 of 8
tool calls. fp32 itself sits near a tie on several of those rows. The measured verdict stays
in the report as it is (passed: false), and the reason goes beside it as gate.override.
publish accepts a failed gate only when that reason is there, the Vulkan gate proxy carries
the reason through, and the README states the shortfall and the reason in the recipe line.
No family gets a lower bar: the override is per run, and has to be asked for by name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hole build

Codex review, round four:

- publish-hf and publish-release refuse a file whose sha256 is not the one its report
  records. verify re-checks a report whose file was rebuilt rather than skipping it on its
  old verdict.
- An export with codes reads them from its own copy, the one it hashes. A shared codes.pt
  replaced between the hash and the read can no longer leave the report naming other codes.
- An empty codes file is refused instead of exporting every linear rounded under a GPTQ
  label.
- A gate proxy has to match the source repo, the ExecuTorch version and every recipe field
  that shapes the graph or the weights: model class, converter, params, qmode, group size,
  embedding quantization, prefill chunk and output head. It used to match only the
  revision, window, codes and head. A proxy report that cannot be read is an ExportError.
- The window table prices the int8 head too, not only the chosen window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… report vouches for

Codex review, round five:

- The README's recipe line names each window published past its gate, with its own
  agreement count and reason, instead of taking the first report's verdict for all of them.
- publish-hf refuses a model file in the folder that the report does not list. An extra or
  renamed .pte had no verdict and no digest behind it.
- The export reads its private copy of the codes by absolute path, since its subprocess runs
  inside the work directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…te proxy, the thinking-only gate gap

What the 2026-11-09 run decided and measured, so the next export does not rediscover it:
- Phi-4-mini matches fp32 before quantization. Its windows of 8k and up run LongRoPE's long
  factors from position 0. It keeps 7 of 8 tool calls at 4 bits and was published through
  an operator override.
- Vulkan takes GPTQ codes only with an XNNPACK twin as its gate.
- The one-token gate cannot see Qwen3-4B-Thinking-2507's decision.
- 32k at 4B needs about 143 GB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o; tie a verdict to its gate reference

Codex review of the whole PR:

- publish-release had no check for model files its report does not list, and publish-hf did
  not check the output root. Both now share require_vouched: a .pte or .bin about to be
  uploaded must be in the report.
- A verdict records the sha256 of the gate.json it was measured against. verify measures
  again when the reference it is given is a different one, instead of answering with the
  old verdict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alpharomercoma
alpharomercoma merged commit c01c60e into main Nov 9, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants