Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions docs/PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,17 @@ so the app's `CompiledBackend.of` reads "vulkan". The Linux wheel's runner has n
kernels and the runner no GPU, so the check is structural (`smoke.structural_check`:
`forward` delegates to `VulkanBackend`); quality and speed are measured on the devices.

- **GPTQ codes, gated through the XNNPACK twin** (2026-11-09): `export-vulkan --codes
<codes.pt> --gate-proxy <xnnpack export-report-<w>.json>` puts the solve's codes into the
same torchao linears, as for XNNPACK. With no Vulkan runtime on the host, the XNNPACK file
built from the same codes is the gate. `export_xnnpack.check_gate_proxy` accepts it only
when it was built the same way: source repo and revision, window, codes sha256,
ExecuTorch version, and every recipe field that shapes the graph or the weights. Its own
gate must also have passed or been accepted. The proxy is recorded in the Vulkan report,
and `publish` refuses a GPTQ Vulkan file without one. Why: round to nearest kept none of
Phi-4-mini's 8 graded tool calls (mean 0.05 against fp32's 0.66, measured on an XNNPACK
file with the same quantization). The `export-vulkan` workflow does not pass `--codes`
yet, so CI's Vulkan files are still round to nearest.
- **Runners, per window** (2026-10-04): the export peak grows with the window squared (the
same per-layer causal masks as XNNPACK), so one fixed 8vcpu runner could not carry any 32k
window of Qwen3, Qwen2.5 or Llama-3.2-3B. The `targets` job runs `export-matrix`, which
Expand Down Expand Up @@ -406,6 +417,32 @@ G5, Dimensity 9300+, Exynos 2500; GSM8K, RetrievalQA, IFEval, PopQA, BFCL, Fresh

## Known limits

- **Phi-4-mini** (2026-11-09):
- **Recipe.** It is export_llm's `phi_4_mini` with execupack's params and converter
(fused qkv/gate_up split), and matches transformers' fp32 Phi3ForCausalLM to 0.003 in
the logits on the gate's prompts.
- **LongRoPE.** It is static per file: ExecuTorch picks the long factors when the RoPE
table is longer than 4096. `ffn_dim_multiplier` is dropped from Phi's params, because
upstream's 1 doubles the table and would put a 4k file on the long factors. So 8k and
larger windows run the long factors from position 0, as llama.cpp does by context
size. HF uses them only past 4096 tokens, so short prompts there differ from HF (KL
2.1, top token 14 of 16, more tool-eager).
- **Quantization.** Its fp32 tool-call decisions are near ties (0.40 to 0.78 on the
graded rows). GPTQ int4 kept 7 of 8. The int8 output head (`XnnpackPlan.output_bits`,
+0.27 GB) raised the mean but not the count on the .pte.
- **Override.** The published files went out through `verify --accept-gate REASON`,
which keeps the measured verdict (`passed: false`) beside the operator's reason. The
README states the shortfall per window.
- **The decision gate cannot see a thinking-only model's decision.**
Qwen3-4B-Thinking-2507's template opens `<think>` whatever `enable_thinking` says. The
gate closes it empty, as the app does with thinking off, but this model reasons anyway:
fp32's first token is "Okay" on every row, so `measures_tool_calling` is false. A
generation through the reasoning on the published CPU 4k file reached a tool call on 6 of
12 search rows within 600 tokens (the rest were still reasoning) and on none of the 4
quiet ones.
- **32k exports of 4B-class models need about 143 GB with headroom**: more than an
r7i.4xlarge (123 GiB), within an r7i.8xlarge.

- Export memory grows with the square of the window: every attention layer of ExecuTorch
1.4.0's transformer builds its own window × window causal mask (not stored in the
`.pte`). Qwen3-0.6B at 32k needs 30,064,771,072 bytes of masks alone and killed a
Expand Down
25 changes: 24 additions & 1 deletion pipeline/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,9 @@ def _verify(args) -> int:

try:
gate = Path(args.gate) if args.gate else None
summary = verify_module.run(Path(args.out), args.backend, Path(args.work), gate_reference=gate)
summary = verify_module.run(
Path(args.out), args.backend, Path(args.work), gate_reference=gate, accept_gate=args.accept_gate
)
except ExportError as error:
print(f"verify failed: {error}", file=sys.stderr)
return 2
Expand Down Expand Up @@ -126,6 +128,8 @@ def _export_vulkan(args) -> int:
Path(args.work),
context=args.context,
keep_work=args.keep_work,
codes=Path(args.codes) if args.codes else None,
gate_proxy=Path(args.gate_proxy) if args.gate_proxy else None,
backend="vulkan",
)
)
Expand Down Expand Up @@ -446,6 +450,12 @@ def main(argv: list[str] | None = None) -> int:
default=None,
help="gate.json from `solve`: the file must decide like the fp32 model before it publishes",
)
verify.add_argument(
"--accept-gate",
default=None,
metavar="REASON",
help="publish even if the gate fails, recording this reason beside the measured verdict",
)
verify.set_defaults(func=_verify)

audit = commands.add_parser("audit", help="gate a file that is already published, without rebuilding it")
Expand Down Expand Up @@ -480,6 +490,19 @@ def main(argv: list[str] | None = None) -> int:
vulkan.add_argument(
"--keep-work", action="store_true", help="keep the work dir after success (a failure always leaves it)"
)
# The Vulkan delegate runs the same torchao Int8DynamicActivationIntxWeight linears as
# XNNPACK, so the solve's codes go in the same way. There is no Vulkan runtime on the
# host to gate the file, so its gate is the XNNPACK file solved from the same codes.
vulkan.add_argument(
"--codes",
default=None,
help="codes.pt from `solve`; without it the int4 weights are rounded to nearest",
)
vulkan.add_argument(
"--gate-proxy",
default=None,
help="with --codes: the XNNPACK export-report-<window>.json from the same codes that passed its gate",
)
vulkan.set_defaults(func=_export_vulkan)

qnn = commands.add_parser("export-qnn", help="compile a Qualcomm HTP .pte for one chip")
Expand Down
23 changes: 21 additions & 2 deletions pipeline/calibration.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@

from __future__ import annotations

import json
from pathlib import Path

# Kept short on purpose: every prompt costs an fp32 generation pass on the runner, and the
Expand Down Expand Up @@ -273,9 +274,27 @@ def render_app(tokenizer, user_turn: str, thinking: bool = True) -> str:
ignore), so the next token is the answer or the tool call itself. Calibration leaves it
at the template's default and renders exactly as before.
"""
messages = [{"role": "system", "content": APP_SYSTEM}, *APP_PRIMER, {"role": "user", "content": user_turn}]
system = {"role": "system", "content": APP_SYSTEM}
extra = {} if thinking else {"enable_thinking": False}
return tokenizer.apply_chat_template(messages, tools=APP_TOOLS, tokenize=False, add_generation_prompt=True, **extra)
if system_carries_tools(tokenizer):
system["tools"] = json.dumps(APP_TOOLS)
else:
extra["tools"] = APP_TOOLS
messages = [system, *APP_PRIMER, {"role": "user", "content": user_turn}]
return tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, **extra)


def system_carries_tools(tokenizer) -> bool:
"""Whether the template reads its tools from the system message rather than ``tools=``.

Phi-4-mini's does: it prints a system message's ``tools`` field, a JSON string, between
``<|tool|>`` markers, and never reads the ``tools`` argument, so passing the schema the
usual way renders a prompt with no tools in it at all (codex review, 2026-11-09). The
app sends exactly that field: ``json.dumps`` of the list, ``", "`` between tools
(Phi4Prompt.render, ToolDefinition.asToolJson).
"""
template = getattr(tokenizer, "chat_template", None)
return isinstance(template, str) and "'tools' in message" in template


def app_head(tokenizer) -> str:
Expand Down
49 changes: 47 additions & 2 deletions pipeline/convert.py
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,24 @@
# as B, C and x stacked on dim 0, and ExecuTorch's ShortConv reads them apart.
_IN_PROJ = ("conv.in_proj.weight", ("conv.B_proj.weight", "conv.C_proj.weight", "conv.x_proj.weight"))
SPLIT_THREE = {"lfm2": _IN_PROJ, "lfm3": _IN_PROJ}
# Phi-4-mini (Phi3ForCausalLM) fuses q, k and v into one projection and the gate and up
# projections into another; everything else has Llama's names. HF splits them in forward:
# q, k, v at n_heads * head_dim and n_kv_heads * head_dim rows (Phi3Attention), gate before
# up in halves (Phi3MLP, gate_up_states.chunk(2, dim=-1)), as ExecuTorch's own
# examples/models/phi_4_mini/convert_weights.py does. The placeholders below never reach the
# checkpoint: convert() replaces each with its parts.
_PHI4_LAYER_KEYS = {
"self_attn.qkv_proj.weight": "attention.wqkv.weight",
"self_attn.o_proj.weight": "attention.wo.weight",
"input_layernorm.weight": "attention_norm.weight",
"post_attention_layernorm.weight": "ffn_norm.weight",
"mlp.gate_up_proj.weight": "feed_forward.w13.weight",
"mlp.down_proj.weight": "feed_forward.w2.weight",
}
_PHI4_FUSED = {
"attention.wqkv.weight": ("attention.wq.weight", "attention.wk.weight", "attention.wv.weight"),
"feed_forward.w13.weight": ("feed_forward.w1.weight", "feed_forward.w3.weight"),
}
# Buffers some checkpoints carry that the model recomputes.
_IGNORED = re.compile(r"(rotary_emb\.inv_freq|\.masked_bias|\.attn\.bias)$")

Expand All @@ -107,9 +125,11 @@
# common one; every layer is a short convolution or an attention block, never both.
"lfm2": (_LFM2_LAYER_KEYS, False),
"lfm3": (_LFM3_LAYER_KEYS, False),
# HF RoPE (use_hf_rope), so q and k stay in HF's layout.
"phi4": (_PHI4_LAYER_KEYS, False),
}
# Converters whose per-layer map stands alone instead of extending _LAYER_KEYS.
_STANDALONE = {"lfm2", "lfm3"}
_STANDALONE = {"lfm2", "lfm3", "phi4"}
# A conv layer and an attention layer carry different tensors, so the per-layer count that
# the other families check cannot apply; LFM2 is checked by total instead.
_MIXED_LAYERS = {"lfm2", "lfm3"}
Expand All @@ -128,6 +148,23 @@ def unpermute(weight, n_heads: int):
)


def _phi4_parts(fused: str, tensor, config: dict) -> tuple:
"""A fused Phi-4-mini projection cut where HF cuts it, refusing any other shape."""
rows, cols = tensor.shape
if fused == "feed_forward.w13.weight":
hidden = int(config["intermediate_size"])
if rows != 2 * hidden:
raise ValueError(f"gate_up_proj has {rows} rows, expected 2 x intermediate_size {hidden}")
return tensor[:hidden], tensor[hidden:]
n_heads = int(config["num_attention_heads"])
head_dim = int(config.get("head_dim") or int(config["hidden_size"]) // n_heads)
q = n_heads * head_dim
kv = int(config.get("num_key_value_heads") or n_heads) * head_dim
if rows != q + 2 * kv:
raise ValueError(f"qkv_proj has {rows} rows, expected {q} + 2 x {kv}")
return tensor[:q], tensor[q : q + kv], tensor[q + kv :]


def _shards(model_dir: Path) -> list[Path]:
index = model_dir / "model.safetensors.index.json"
if index.exists():
Expand Down Expand Up @@ -183,6 +220,12 @@ def convert(model_dir: Path, output: Path, converter: str, config: dict) -> dict
for name, part in zip(split[1], torch.chunk(tensor, 3, dim=0), strict=True):
state[head + name] = part
continue
fused = next((f for f in _PHI4_FUSED if target.endswith(f)), None) if converter == "phi4" else None
if fused:
head = target[: -len(fused)]
for name, part in zip(_PHI4_FUSED[fused], _phi4_parts(fused, tensor, config), strict=True):
state[head + name] = part
continue
if permute and target.endswith("attention.wq.weight"):
tensor = unpermute(tensor, n_heads)
elif permute and target.endswith("attention.wk.weight"):
Expand Down Expand Up @@ -210,7 +253,9 @@ def convert(model_dir: Path, output: Path, converter: str, config: dict) -> dict
if not widths or min(widths) < 6:
raise ValueError(f"a layer is missing tensors: widths {widths}")
else:
expected_per_layer = len(_LAYER_KEYS) + len(CONVERTERS[converter][0])
# Phi-4-mini's six HF tensors per layer become Llama's nine.
extra = 0 if converter == "phi4" else len(CONVERTERS[converter][0])
expected_per_layer = len(_LAYER_KEYS) + extra
if per_layer != n_layers * expected_per_layer:
raise ValueError(f"expected {n_layers} layers x {expected_per_layer} tensors, found {per_layer}")

Expand Down
4 changes: 4 additions & 0 deletions pipeline/eligibility.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
"lfm2": {"lfm25"},
# Ahead of the app (naming.AHEAD_OF_APP): it has no LFM3 template yet.
"lfm3": {"lfm3"},
"phi4_mini": {"phi4mini"},
}
# What a release of each family is called. The architecture class alone is not enough: a
# later generation can keep the class (a text-only "Qwen3.8" on Qwen3ForCausalLM would pass
Expand All @@ -34,6 +35,9 @@
"llama": re.compile(r"llama-?3\.2(?![.\d])|smollm2(?!\d)", re.IGNORECASE),
"gemma3": re.compile(r"gemma-?3(?![.\dn])", re.IGNORECASE),
"smollm3": re.compile(r"smollm3(?!\d)", re.IGNORECASE),
# Phi-4-mini only: Phi-3 and Phi-3.5 share the architecture class, and the app's template
# is Phi-4-mini's (Phi4Prompt).
"phi4_mini": re.compile(r"phi-?4-mini(?![-.]?\d)", re.IGNORECASE),
}
_INSTRUCT_TOKENS = {"instruct", "it", "chat"}
# Families whose chat models ship with no suffix at all and whose base models carry "-Base".
Expand Down
85 changes: 64 additions & 21 deletions pipeline/export_with_codes.py
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,59 @@ def inject(model, codes: dict, log=print) -> tuple[int, list[str]]:
return injected, left


def quantizer(upstream, codes: dict, output_int8: bool):
"""``upstream`` (source_transformation.quantize.quantize) with the solved codes put in,
and for ``output_int8`` the output head at int8 per channel rather than int4."""
import torch

codes_given = bool(codes) # main() refuses an empty codes file before this
if output_int8:
# The head is not int4 here, so its int4 codes have nowhere to go.
codes = {k: v for k, v in codes.items() if k != "output"}

def quantize(model, qmode, *rest, **kwargs):
if qmode != "8da4w":
return upstream(model, qmode, *rest, **kwargs)
head = None
if output_int8:
# Hidden from the int4 pass, which takes every nn.Linear, then given int8 weights
# per channel with the same dynamic int8 activations
# (Int8DynamicActivationIntxWeightConfig, as upstream's own 8da8w mode builds them).
head, model.output = model.output, torch.nn.Identity()
quantised = upstream(model, qmode, *rest, **kwargs)
if head is not None:
from torchao.quantization import Int8DynamicActivationIntxWeightConfig, quantize_
from torchao.quantization.granularity import PerAxis

with torch.no_grad():
quantize_(
head,
Int8DynamicActivationIntxWeightConfig(weight_dtype=torch.int8, weight_granularity=PerAxis(0)),
)
quantised.output = head
print("output head quantized to int8 per channel")
if not codes_given:
return quantised
injected, left = inject(quantised, codes)
# And every int4 tensor must have its code: a partial codes file would otherwise
# export, be labelled GPTQ, and carry the rest rounded (codex review, 2026-11-09). The
# int8 head has no int4 code by design.
rounded = [n for n in left if not (output_int8 and n == "output")]
if rounded:
raise ValueError(f"no solved codes for {len(rounded)} int4 linears: {rounded[:5]}")
# Every code must find its tensor. If the solve was run against a different
# checkpoint, or a name moved, the export would otherwise finish, be labelled GPTQ,
# and carry weights that were quietly rounded instead.
if injected != len(codes):
missing = sorted(set(codes) - {n for n, _ in quantised.named_modules()})
raise ValueError(
f"{len(codes)} solved tensors but {injected} were injected; these codes match no module: {missing[:5]}"
)
return quantised

return quantize


def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--config", required=True, help="the export_llm YAML the exporter wrote")
Expand All @@ -72,6 +125,11 @@ def main(argv: list[str] | None = None) -> int:
action="store_true",
help="clear the short convolution's state at position zero (pipeline/lfm2_state.py)",
)
parser.add_argument(
"--output-int8",
action="store_true",
help="quantize the output head to int8 per channel instead of int4 (XnnpackPlan.output_bits)",
)
parser.add_argument(
"--lfm3",
action="store_true",
Expand All @@ -93,31 +151,16 @@ def main(argv: list[str] | None = None) -> int:
lfm3_model.apply()
print("applied the LFM3 model definition (pipeline/lfm3_model.py)")

if args.codes:
if args.codes or args.output_int8:
from executorch.examples.models.llama.source_transformation import quantize as quantize_module

# weights_only: the codes file is an artifact passed between CI jobs, and it holds
# nothing but tensors, so there is no reason to let it unpickle arbitrary objects.
codes = torch.load(args.codes, map_location="cpu", weights_only=True)
upstream = quantize_module.quantize

def quantize(model, qmode, *rest, **kwargs):
quantised = upstream(model, qmode, *rest, **kwargs)
if qmode != "8da4w":
return quantised
injected, _ = inject(quantised, codes)
# Every code must find its tensor. If the solve was run against a different
# checkpoint, or a name moved, the export would otherwise finish, be labelled
# GPTQ, and carry weights that were quietly rounded instead.
if injected != len(codes):
missing = sorted(set(codes) - {n for n, _ in quantised.named_modules()})
raise ValueError(
f"{len(codes)} solved tensors but {injected} were injected; "
f"these codes match no module: {missing[:5]}"
)
return quantised

quantize_module.quantize = quantize
codes = torch.load(args.codes, map_location="cpu", weights_only=True) if args.codes else {}
if args.codes and not codes:
# An empty codes file would export every linear rounded under a GPTQ label.
raise ValueError(f"{args.codes} holds no codes")
quantize_module.quantize = quantizer(quantize_module.quantize, codes, args.output_int8)

from executorch.extension.llm.export.export_llm import main as export_llm_main

Expand Down
Loading
Loading