Repository navigation
Phi-4-mini on CPU and GPU, GPTQ Vulkan through a gate proxy, and a stricter publish gate - #28
Merged
Merged
Conversation
…g-only template Phi-4-mini-instruct (Phi3ForCausalLM) gets a recipe for export_llm's phi_4_mini class: - The fused qkv_proj and gate_up_proj are split where HF splits them. - LongRoPE's factors pass through as ExecuTorch 1.5.1 reads them. - ffn_dim_multiplier is dropped. Upstream's config sets it to 1, which makes Rope build its table at twice the window, so a 4k export would run every prompt on the long factors that HF only uses past 4096 tokens. - Outputs are named Phi-4-mini-instruct-3.8B, because the app reads the size from the name and the release has none. - Its chat template reads tools from the system message, not from tools=, so calibration and the gate now put the schema there, as the app does. Found by codex review: before this, the Phi rows carried no tools at all. The gate closes a think block that a template leaves open. Qwen3-4B-Thinking-2507 opens one whatever enable_thinking says, which put the first word of the reasoning in the graded position. Every published family renders exactly as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
verify skipped any report that already had a smoke result, even when it was asked to gate. So a first run that passed the smoke test and then failed, or never reached, the decision gate left a report that a second run returned as verified with nothing gated. publish-hf checked only the smoke result, so that report could be published. Now verify gates it again, and publish refuses a report whose gate failed. A report with no gate at all (Vulkan, or a run without an fp32 reference) publishes on its smoke test, as before. Found by codex review. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phi-4-mini's GPTQ CPU export missed the decision gate by one row: it kept 7 of 8 tool calls, at a mean of 0.45 against fp32's 0.66. Measured eager on the gate's prompts against fp32, holding everything else fixed, the int4 output head was the cost: - int4 everywhere kept 7/8 at 0.457 (the .pte measured 0.452); - an fp32 head kept 8/8 at 0.521; - an int8 per-channel head kept 8/8 at 0.526. The head is the tied 200,064 x 3072 embedding, a sixth of the model, and fp32 is itself only 0.40 to 0.78 sure on these rows. XnnpackPlan.output_bits sets the head's bits; Phi-4-mini uses 8, which adds about 0.27 GB to the file. The head gets int8 weights per channel with the same dynamic int8 activations, and its int4 codes are skipped. export-vulkan takes --codes. The Vulkan delegate runs the same torchao linears, and round-to-nearest int4 kept none of Phi's 8 tool calls (0.05 against 0.66). There is no Vulkan runtime on the host, so a Vulkan file is gated through the XNNPACK file solved from the same codes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hat stands for it Codex review, round three: - publish-hf and publish-release share one check now. A GPTQ report must carry a passing gate of its own, because the solve always writes an fp32 reference, so a GPTQ report without a gate is one whose gate crashed or never ran. A Vulkan GPTQ report, which the host cannot run, must instead carry a passing gate proxy. publish-release checked nothing before. - export-vulkan --codes needs --gate-proxy: the XNNPACK report from the same source revision, window, codes (sha256, now recorded in every GPTQ report) and output head, whose own gate passed. It is checked before the export and recorded in the Vulkan report. - A codes file that leaves an int4 linear unsolved is refused, not exported as GPTQ with that linear rounded. - The size and phone-fit estimates price an int8 output head at a byte a weight. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r's recorded reason For Phi-4-mini the operator accepted the 4-bit files with an int8 head, which keep 7 of 8 tool calls. fp32 itself sits near a tie on several of those rows. The measured verdict stays in the report as it is (passed: false), and the reason goes beside it as gate.override. publish accepts a failed gate only when that reason is there, the Vulkan gate proxy carries the reason through, and the README states the shortfall and the reason in the recipe line. No family gets a lower bar: the override is per run, and has to be asked for by name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hole build Codex review, round four: - publish-hf and publish-release refuse a file whose sha256 is not the one its report records. verify re-checks a report whose file was rebuilt rather than skipping it on its old verdict. - An export with codes reads them from its own copy, the one it hashes. A shared codes.pt replaced between the hash and the read can no longer leave the report naming other codes. - An empty codes file is refused instead of exporting every linear rounded under a GPTQ label. - A gate proxy has to match the source repo, the ExecuTorch version and every recipe field that shapes the graph or the weights: model class, converter, params, qmode, group size, embedding quantization, prefill chunk and output head. It used to match only the revision, window, codes and head. A proxy report that cannot be read is an ExportError. - The window table prices the int8 head too, not only the chosen window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… report vouches for Codex review, round five: - The README's recipe line names each window published past its gate, with its own agreement count and reason, instead of taking the first report's verdict for all of them. - publish-hf refuses a model file in the folder that the report does not list. An extra or renamed .pte had no verdict and no digest behind it. - The export reads its private copy of the codes by absolute path, since its subprocess runs inside the work directory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…te proxy, the thinking-only gate gap What the 2026-11-09 run decided and measured, so the next export does not rediscover it: - Phi-4-mini matches fp32 before quantization. Its windows of 8k and up run LongRoPE's long factors from position 0. It keeps 7 of 8 tool calls at 4 bits and was published through an operator override. - Vulkan takes GPTQ codes only with an XNNPACK twin as its gate. - The one-token gate cannot see Qwen3-4B-Thinking-2507's decision. - 32k at 4B needs about 143 GB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o; tie a verdict to its gate reference Codex review of the whole PR: - publish-release had no check for model files its report does not list, and publish-hf did not check the output root. Both now share require_vouched: a .pte or .bin about to be uploaded must be in the report. - A verdict records the sha256 of the gate.json it was measured against. verify measures again when the reference it is given is a different one, instead of answering with the old verdict. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Phi-4-mini-instruct (3.8B) on XNNPACK and Vulkan. Lets a Vulkan export take GPTQ codes, gated through the XNNPACK file built from the same codes. Closes the ways a file could be published without passing the checks its recipe calls for. Five rounds of adversarial review with codex found real defects in every round; all are fixed here.
Both repos below were built from this branch and are live: Phi-4-mini-instruct-3.8B-ExecuTorch and Qwen3-4B-Thinking-2507-ExecuTorch. Each has CPU and GPU files at 2k to 32k.
Phi-4-mini
phi_4_miniwith our params and a converter that splits the fusedqkv_projandgate_up_projwhere HF does.Phi3ForCausalLMwithin 0.003 in the logits on all 16 gate prompts (0.0003 at 8k against HF's long factors).Phi-4-mini-instruct-3.8B(naming.SIZED_NAMES), because the app reads the size from the name.ffn_dim_multipliertrap. Upstream'sphi_4_miniconfig sets it to 1, which makesRopebuild its table at twice the window. A 4k file would then run LongRoPE's long factors on every prompt. It's dropped for Phi, so 2k and 4k use HF's short factors exactly. 8k and larger windows use the long factors from position 0, as llama.cpp does by context size; the short-prompt cost is measured in PLAN.tools=, so calibration and gate prompts carried no tool schema (found by codex).render_appnow puts them there, byte-identical to the app'sPhi4Prompt.XnnpackPlan.output_bits = 8: the tied 200,064 × 3072 head gets int8 per-channel weights..pteit still keeps 7/8 (0.51 against fp32's 0.66), because fp32 itself sits near a tie on several rows.verify --accept-gate REASONpublishes past a failed gate only with a recorded reason. The measured verdict stayspassed: false, the reason is stored beside it, and the README states each window's shortfall. The published Phi files used it, with the operator's approval.Vulkan with GPTQ codes
export-vulkan --codes --gate-proxy <xnnpack report>. The proxy must match the build: source repo and revision, window, codes sha256, ExecuTorch version, and every graph- or weight-shaping recipe field. Its gate must have passed or been accepted.export-vulkanworkflow does not pass--codesyet, so CI's Vulkan files remain round to nearest.Gate and publish
measures_tool_callingis false for it (PLAN, Known limits).verifyre-gates a report whose gate is missing or failed, or whose.ptewas rebuilt.publish-hfandpublish-releasesharerequire_publishable: a GPTQ file needs its own passing gate (or, for Vulkan, a passing proxy).Testing
pytest -q: 361 passed, 1 skipped;ruff checkandruff format --checkclean. New tests cover:🤖 Generated with Claude Code
Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is enabled.