diff --git a/docs/PLAN.md b/docs/PLAN.md index 789b92c..58df6e4 100644 --- a/docs/PLAN.md +++ b/docs/PLAN.md @@ -248,6 +248,18 @@ calling ExecuTorch's own script in compile-only mode: - The window is fixed at compile time: `qnn.max_context_len` (2048 to start), or the workflow's `context` input (`--context`) for one run. - QAIRT's license: `THIRD_PARTY_NOTICES.md`. +- **Gemma 3 1B** (2026-11-09): gemma-3-1b-it (`gemma3-1b`, dcc83ea8) is published for SM8750 at + 2k to 32k, the first 32k QNN build here, one 1.15-1.21 GB `.pte` per window, run on EC2 rather + than Modal: 2k 1,087 s and 4k 1,070 s (peak RSS 49.5 and 46.3 GiB), 8k 2,202 s (47.9 GiB), + 16k 3,263 s (77.9 GiB) on r7i.4xlarge (16 vCPU), 32k 4,598 s on r7i.8xlarge (32 vCPU) with a + 146.9 GiB peak, past the 83 GiB `qnn-matrix` asks Modal for and past a 128 GiB host. A fresh + Ubuntu 24.04 image lacks `libatomic1`, which the QAIRT SDK's bundled libc++ needs; without it + `libQnnHtp.so` fails to load and the wheel reports only "Failed to set up QNN SDK". +- Qwen2.5-0.5B/1.5B/3B-Instruct and Qwen2.5-Math-1.5B-Instruct are not in the registry (it has + the base 0.5B and 1.5B only, and no 3B). The script takes `--checkpoint`, so their weights could + ride the base entries' graph and recipe, but those recipes are post-training int4 with no host + check, like the round-to-nearest int4 that failed the gate for Qwen2.5-1.5B on XNNPACK; held + back on 2026-11-09 by the operator's choice. ### MediaTek (phase 4) @@ -269,7 +281,10 @@ calling ExecuTorch's own script in compile-only mode: (`export_mtk.tool_env`): the wheel's `libextract_shared.so` needs the `libc++.so.1` bundled beside it but has no RPATH (docs/research, finding 21). - **Scripts:** the LLM export scripts are in ExecuTorch's source, not the wheel; the - workflow sparse-checks-out `examples/mediatek` at `EXECUTORCH_COMMIT`. + workflow sparse-checks-out `examples/mediatek` at `EXECUTORCH_COMMIT`. Each run executes + them in its own copy (31 MB) in its work folder (`export_mtk.private_examples`): they write + chunks to `pte/___chunks`, a name without the window, and two windows + of Gemma 3 on one EC2 box (2026-11-09) shared it until the first to finish deleted it. - **Recipe** (MediaTek's own, `shell_scripts/export_qwen.sh`): A16W4, the model cut into up to 4 chunks of equal layer counts, a 128-token prompt graph and a one-token generation graph over a 512-token cache, calibrated on MediaTek's `alpaca.txt` prompts (9, up to 9 @@ -285,7 +300,9 @@ calling ExecuTorch's own script in compile-only mode: - **Families:** the scripts build the model from `config.json`'s `model_type`, so any size works, not a fixed list. Qwen3 (`qwen.py`), Llama 3.2 and SmolLM2 (`llama.py`, with `"tokenizer": "pretrained_fast"` written into the config, SmolLM2 calibrating in ChatML) - are exported; Gemma 3 (`gemma3_text` vs `gemma3`) waits for validation. + are exported, and Gemma 3 (`gemma.py`, with `families.mtk_model_config` writing MediaTek's + spelling of the config: model_type `gemma3`, `sliding_window_attention_size`, + `global_local_attention_pattern`, `use_res_clamp` false). - **Stock RoPE was wrong** (2026-10-04, finding 37): `get_master_rot_emb` used base 10000 because neither `QwenConfig` nor `LlamaConfig` kept `rope_theta`, and it had no Llama 3 frequency scaling. In the fp32 graph, before quantization, Qwen3-0.6B was at KL 0.082 and @@ -302,9 +319,26 @@ calling ExecuTorch's own script in compile-only mode: full fp32 cache at the window, so Qwen3 at 32k (about 285 GB for the 0.6B), Qwen3-4B from 16k and Llama-3.2-3B at 32k fit no hosted runner and are left out of the matrix; the streaming calibration `lfm2.py` uses would lift that. +- **Gemma** (2026-11-09, findings 46-47). Gemma 3's sliding layers rotate at base 10000 and + its global ones at 1e6, and stock `Gemma3ModelChunk` takes one table for both: in fp32, + 10000 everywhere fails the decision gate (6/11, KL 0.18) and 1e6 everywhere, the runner + sample's setting, worse (5/11, KL 1.42). `mediatek-gemma.patch` adds a second table input + (`pos_emb_local`) and the per-layer choice, which is exact against Hugging Face (KL 0, + 11/11); the runner config names `rot_emb_local_base` and `rot_emb_inputs`, and the card + says MediaTek's 1.5.1 runner, which feeds one table, cannot run these files correctly. + gemma-3-1b-it is published for MT6991 (A16W8, 2 chunks) at 512 to 32k; the 32k export took + 6,944 s on an r7i.4xlarge (123 GiB), the box at 92.8 GB in use with the 16k export beside it. + Gemma 2 2B is exact + as MediaTek ships it up to its 4,096-token sliding window (the card says what differs past + it), but its 2.36 GB fp32 output head is over `mtk_converter`'s 2 GB protobuf at any chunk + count, plain, fp16 or with int8 weights folded, so `mtk_plan` refuses it and Gemma 2 has + no backend. - **Output** per chip: the chunk `.pte` files, the fp32 token embedding table the runner - reads from disk, and `config.json` with the flags MediaTek's LLM runner - (`examples/mediatek/executor_runner`) needs. They do not run on `TextLLMRunner`. + reads from disk (one per folder, shared by every window: `publish` refuses a run whose table + differs from the one the folder's other reports record, as a newer source revision's would, + unless `--replace-shared` withdraws those windows in the same commit for republishing), + and `config.json` with the flags MediaTek's LLM runner (`examples/mediatek/executor_runner`) + needs. They do not run on `TextLLMRunner`. - **Check:** structural; every chunk loads, has its two methods, and delegates to `NeuropilotBackend`. - **16k built with one attention layer per chunk** (2026-10-04, finding 39). LFM2.5-1.2B-Instruct @@ -389,8 +423,8 @@ pipeline exports it now; **todo** = ExecuTorch supports it and the pipeline does | Llama 3.2 1B / 3B (+Instruct) | done | todo | done: 1B/3B-Instruct (registry) | todo (validate `llama.py`: rope_type, tokenizer) | upstream | | SmolLM2 135M / 360M / 1.7B (+Instruct) | done | todo | done: 135M-Instruct (registry) | todo (validate, with Llama 3.2) | upstream | | SmolLM3 3B | upstream | upstream | done | upstream | upstream | -| Gemma 3 1B | upstream | upstream | done: gemma-3-1b-it | todo (validate `gemma.py`: model_type gemma3_text) | upstream | -| Gemma 2 2B | upstream | upstream | todo | todo | upstream | +| Gemma 3 1B | upstream | upstream | done: gemma-3-1b-it, SM8750 2k-32k | done: gemma-3-1b-it, MT6991 (`mediatek-gemma.patch`, two RoPE tables) | upstream | +| Gemma 2 2B | upstream | upstream | todo | blocked: the 2.36 GB head is over `mtk_converter`'s 2 GB protobuf (finding 47) | upstream | | Phi-4-mini 3.8B | todo (watch `microsoft`) | todo | todo | todo | upstream | | LFM2.5 350M / 1.2B, LFM2 350M / 700M / 1.2B | todo (watch `LiquidAI`) | todo | upstream | upstream | upstream | | Qwen3.5 0.8B / 2B / 4B | blocked: the app refuses `qwen35` names | — | upstream | upstream | upstream | diff --git a/docs/research/README.md b/docs/research/README.md index 30fe41d..f80203e 100644 --- a/docs/research/README.md +++ b/docs/research/README.md @@ -71,3 +71,5 @@ Findings from building and running this pipeline, each kept with its evidence. C | 43 | 2026-11-09 | XNNPACK | Qwen3-0.6B does not keep its tool decision at int4 under the gate that asks with reasoning switched off. A GPTQ solve (ExecuTorch 1.5.1) keeps 0 of 4 calling rows at every window, mean 0.016 against fp32's 0.619, and calls on 2 of 4 quiet rows; the October GPTQ file reads 0.086 and 0 of 4 on the same gate. The same model with fp32 linears (int8 embedding) reads 0.618 and 4 of 4. Qwen3-1.7B (12 of 12, 0.995 against 0.999) and Qwen3-4B (12 of 12, 0.9999) keep it at int4. | measured | verify logs of the rollout's twins; `pipeline audit Qwen/Qwen3-0.6B --context 2048` | Qwen3-0.6B ships fp32 linears on XNNPACK and no Vulkan folder. | | 44 | 2026-11-09 | XNNPACK | `pipeline solve` on macOS arm64 (torch 2.14.0, Apple M5) turns a calibration row's hidden state into NaN after a few quantised layers: SmolLM2-135M at layer 4 (row 39), Qwen2.5-0.5B at layer 9, then Cholesky fails on the NaN Hessian. The NaN sits in the prompt positions the Hessian does not count, and no submodule output is non-finite. The same commit solves both on Linux x86 without it. The Mac's fp32 gate reference is off too: for SmolLM2-135M it grades 9 calling rows at 0.713 where Linux grades 4 at 0.583, and the published, gated GPTQ file scores 0.0002 against the Mac's reference and 0.585 against Linux's on the same Mac runner. | measured | local runs on the Mac against the rollout's EC2 solves | Solve and compute gate references on Linux only; a Mac can export and run a gate against a Linux reference. | | 45 | 2026-11-09 | Both | XNNPACK and Vulkan `.pte` files hold their int4 weights in a form that can be read back and compared exactly: XNNPACK as `qbint4` values (unsigned nibbles around 8, low nibble first, bf16 scales) behind each fully-connected node's filter id; Vulkan as `linear_dq8ca_q4gsw` arguments (uint8 [out, in/2], fp32 scales [groups, out], int32 per-group code sums). Both serialise linears in execution order. Matched against fp32 sources, a published round-to-nearest file agrees with plain rounding on 99.0 to 99.2 percent of codes with every scale at amax/7.5; a GPTQ file on 67 to 87 percent, with 9 to 36 percent of scales at amax/7.5. | measured | SmolLM2-135M's published files; the rollout's byte checks | Byte checks before publish and after. | +| 46 | 2026-11-09 | MediaTek | Gemma 3's sliding layers rotate at base 10000 (`rope_local_base_freq`) and its global ones at 1e6, but MediaTek's `Gemma3ModelChunk` takes one RoPE table for every layer: its master table is built at 10000 and its runner sample sets `rot_emb_base` 1e6. In fp32 against Hugging Face (google/gemma-3-1b-it, 900-token passage and the decision gate's prompts), 10000 everywhere is at KL 0.177, 88% top-1, perplexity 10.77 against 9.58, gate 6/11; 1e6 everywhere at KL 1.42, 52% top-1, 22.7, gate 5/11; a table per layer type is exact (KL 0, 100%, 9.575, 11/11). Gemma 2 (one base, 10000) is exact as MediaTek ships it, with softcapping, up to its 4,096-token sliding window. | measured | [`mtk-gemma-vs-hf.py`](evidence/mtk-gemma-vs-hf.py), [`mtk-gemma-fp32-vs-hf.log`](evidence/mtk-gemma-fp32-vs-hf.log) | `mediatek-gemma.patch` gives the chunk a second input, `pos_emb_local`, built at the local base, and `export_mtk` writes `rot_emb_local_base` and `rot_emb_inputs` for the runner, which has to feed both tables (31 inputs per method). Published for MT6991 with that caveat on the card: MediaTek's 1.5.1 runner feeds one table. | +| 47 | 2026-11-09 | MediaTek | `mtk_converter` 8.13.0 holds a delegated graph's weights as fp32 in one protobuf `ModelProto`, which protobuf caps at 2 GB, and the output head sits whole in the last chunk. Gemma 2 2B's tied 256000 x 2304 head is 2,359,296,000 B: the export fails in the converter at 2 chunks and at 13 (two layers each; 2361344595 B), and the head alone (final norm + head, A16W8, MT6991) fails plain, with `convert_float32_weights_to_float16` and with `convert_pt2e(fold_quantize=True)`: "exceeds maximum protobuf size of 2GB: 2361344528" (2361344648 folded). Gemma 3 1B's 262144 x 1152 head is 1.21 GB and builds. | measured | [`mtk-gemma2-head-probe.py`](evidence/mtk-gemma2-head-probe.py), [`mtk-gemma2-head-probe.log`](evidence/mtk-gemma2-head-probe.log) | `families.mtk_plan` refuses a head of 2 GB or more in fp32 (`mtk_head_bytes`), so Gemma 2 has no backend. A way through would split the head across separately delegated graphs and join the halves on the CPU, which the runtime would have to provide; not attempted. | diff --git a/docs/research/evidence/mtk-gemma-fp32-vs-hf.log b/docs/research/evidence/mtk-gemma-fp32-vs-hf.log new file mode 100644 index 0000000..bf1a694 --- /dev/null +++ b/docs/research/evidence/mtk-gemma-fp32-vs-hf.log @@ -0,0 +1,40 @@ +# mtk-gemma-vs-hf.py, 2026-11-09, r7i.8xlarge, Ubuntu 24.04 x86_64, fp32, CPU. +# MediaTek's chunk classes at ExecuTorch 3b60683 (examples/mediatek/models/llm_models/modeling_gemma.py), +# one chunk of every layer, against transformers' fp32 eager model. Console output as printed by the +# harness's wrapper, one line per variant. Passage: 900 tokens of the model card, every position. +# Gate: execupack's 16 decision-gate prompts, last position (pipeline/gate.py). + +== gemma3 google/gemma-3-1b-it @ dcc83ea841ab6100d6b47a070329e1ba4cf78752 +# 1) Stock Gemma3ModelChunk: one RoPE table for every layer. Sliding layers use base 10000 and +# global ones 1e6 in Hugging Face's model; MediaTek's master table is built at 10000, its runner +# sample (run_gemma3_sample.sh) sets rot_emb_base 1e6. "per-layer" here emulated both tables. +MediaTek config: GemmaConfig Gemma3ModelChunk +passage 900 tokens; gate prompts 392 - 428 ; window 908 +MediaTek master table vs base 10000: max |diff| 2.00e+00 +MediaTek master table vs base 1e+06: max |diff| 6.10e-05 + rope=10000 everywhere mask=-100 passage KL=0.1769 (max 8.460) top1=0.884 ppl hf/mtk=9.575/10.772 | gate top1=0.438 KL=5.6725 verdict=False 6/11 quiet 1/3 + rope=10000 everywhere mask=-1e+09 passage KL=0.1769 (max 8.460) top1=0.884 ppl hf/mtk=9.575/10.772 | gate top1=0.438 KL=5.6725 verdict=False 6/11 quiet 1/3 + rope=1e6 everywhere mask=-100 passage KL=1.4200 (max 12.288) top1=0.521 ppl hf/mtk=9.575/22.726 | gate top1=0.312 KL=3.6574 verdict=False 5/11 quiet 0/3 + rope=1e6 everywhere mask=-1e+09 passage KL=1.4200 (max 12.288) top1=0.521 ppl hf/mtk=9.575/22.726 | gate top1=0.312 KL=3.6574 verdict=False 5/11 quiet 0/3 + rope=per-layer mask=-100 passage KL=0.0000 (max 0.000) top1=1.000 ppl hf/mtk=9.575/9.575 | gate top1=1.000 KL=0.0000 verdict=True 11/11 quiet 3/3 + rope=per-layer mask=-1e+09 passage KL=0.0000 (max 0.000) top1=1.000 ppl hf/mtk=9.575/9.575 | gate top1=1.000 KL=0.0000 verdict=True 11/11 quiet 3/3 + +# 2) Patched Gemma3ModelChunk (third_party/executorch/patches/mediatek-gemma.patch): the chunk +# takes pos_emb (global, 1e6) and pos_emb_local (10000) and gives each layer its own. +MediaTek config: GemmaConfig Gemma3ModelChunk +MediaTek master table vs base 10000: max |diff| 2.00e+00 +MediaTek master table vs base 1e+06: max |diff| 6.10e-05 + rope=10000 everywhere passage KL=0.17687 (max 8.4599) top1=0.884 ppl hf/mtk=9.575/10.772 | gate top1=0.438 verdict=False 6/11 quiet 1/3 + rope=1e6 everywhere passage KL=1.41998 (max 12.2876) top1=0.521 ppl hf/mtk=9.575/22.726 | gate top1=0.312 verdict=False 5/11 quiet 0/3 + rope=per-layer passage KL=0.00000 (max 0.0000) top1=1.000 ppl hf/mtk=9.575/9.575 | gate top1=1.000 verdict=True 11/11 quiet 3/3 + +== gemma2 google/gemma-2-2b-it @ 299a8560bedf22ed1c72a8a11e7dce4a7f9f51f8 +# Stock Gemma2ModelChunk, base 10000 on every layer as in Hugging Face's model. The template +# refuses a system turn, so the gate's system text is folded into the first user turn. +# Window 908 < sliding_window 4096, so the sliding layers see what they see in Hugging Face's. +MediaTek config: GemmaConfig Gemma2ModelChunk +passage 900 tokens; gate prompts 391 - 425 ; window 908 +MediaTek master table vs base 10000: max |diff| 1.22e-04 +MediaTek master table vs base 1e+06: max |diff| 2.00e+00 + rope=10000 everywhere mask=-100 passage KL=0.00000 (max 0.0000) top1=1.000 ppl hf/mtk=6.054/6.054 | gate top1=1.000 KL=0.00000 verdict=True 8/8 quiet 4/4 + rope=10000 everywhere mask=-1e+09 passage KL=0.00000 (max 0.0000) top1=1.000 ppl hf/mtk=6.054/6.054 | gate top1=1.000 KL=0.00000 verdict=True 8/8 quiet 4/4 diff --git a/docs/research/evidence/mtk-gemma-vs-hf.py b/docs/research/evidence/mtk-gemma-vs-hf.py new file mode 100644 index 0000000..83f0b99 --- /dev/null +++ b/docs/research/evidence/mtk-gemma-vs-hf.py @@ -0,0 +1,184 @@ +"""fp32 MediaTek Gemma graph against Hugging Face, before any quantization (docs/research, finding 46). + + python mtk-gemma-vs-hf.py + +MediaTek's chunk classes (examples/mediatek/models/llm_models/modeling_gemma.py) run on the CPU +in fp32 with mtk_converter stubbed (as in docs/research/evidence/mtk-conv-state-and-padding.py), +from the config execupack's families.mtk_model_config writes. They see one prompt in one feed +with an empty cache of WINDOW slots, masked the way MediaTek's MaskBuilder masks: 0 where +visible, MASK where not, and for Gemma 3's sliding layers a second mask that also hides keys +more than sliding_window tokens back. + +Scored against transformers' fp32 model (eager attention, which applies Gemma 2's softcapping) +on a ~900-token passage (every position: KL, top-1, perplexity) and on the decision gate's +16 prompts (the last position). Variants: + gemma3: RoPE at 10000 on every layer (MediaTek's master table), at 1e6 on every layer (its + runner sample's rot_emb_base), and per layer (10000 sliding, 1e6 global: what a + two-table graph would compute); each with mask -100 and with -1e9 + gemma2: RoPE at 10000 (every layer, as HF), mask -100 and -1e9 +""" + +import json +import os +import sys +import types +from pathlib import Path + +for name in ("mtk_converter", "mtk_converter.python", "mtk_converter.python.converters", "mtk_neuron"): + sys.modules[name] = types.ModuleType(name) +pytorch_mod = types.ModuleType("mtk_converter.python.converters.pytorch") +pytorch_mod.importer_v2 = types.ModuleType("importer_v2") +sys.modules["mtk_converter.python.converters.pytorch"] = pytorch_mod + +which, checkout, examples, out_path = sys.argv[1], Path(sys.argv[2]).resolve(), os.path.abspath(sys.argv[3]), Path(sys.argv[4]).resolve() +sys.path.insert(0, str(checkout)) +import torch # noqa: E402 +from huggingface_hub import HfApi, hf_hub_download, snapshot_download # noqa: E402 +from transformers import AutoModelForCausalLM, AutoTokenizer # noqa: E402 + +from pipeline import families, gate # noqa: E402 + +torch.set_num_threads(int(os.environ.get("THREADS", "8"))) +REPO = {"gemma3": "google/gemma-3-1b-it", "gemma2": "google/gemma-2-2b-it"}[which] +SHA = HfApi().model_info(REPO).sha +snap = Path(snapshot_download(REPO, revision=SHA, allow_patterns=["*.json", "*.safetensors", "*.model", "README.md"])) +stage = Path(os.environ.get("STAGE", f"/tmp/gemma-stage-{which}")) +stage.mkdir(parents=True, exist_ok=True) +for f in snap.iterdir(): + if f.name != "config.json" and not (stage / f.name).exists(): + (stage / f.name).symlink_to(f.resolve()) +hf_config = json.loads((snap / "config.json").read_text()) +(stage / "config.json").write_text(json.dumps(families.mtk_model_config(hf_config), indent=2)) + +os.chdir(examples) +sys.path.insert(0, examples) +from aot_utils.llm_utils.utils import get_embedding_layer, get_master_rot_emb, load_checkpoints, resolve_model_classes # noqa: E402 + +config, weight_dir, _, chunk_class = resolve_model_classes(str(stage / "config.json")) +print("MediaTek config:", type(config).__name__, chunk_class.__name__, flush=True) +state_dict = load_checkpoints(weight_dir) +embedding = get_embedding_layer(config, weight_dir, state_dict) +n_layers = config.num_hidden_layers +chunk = chunk_class(config, n_layers, chunk_idx=0, dtype=torch.float32, include_tail=True, jit_trace=True) +chunk = chunk.load_weights(state_dict, 0).eval() +del state_dict +head_dim, kv_heads = int(config.head_dim), config.num_key_value_heads + +tok = AutoTokenizer.from_pretrained(snap) +readme = (snap / "README.md").read_text() if (snap / "README.md").exists() else "" +body = readme.split("---", 2)[-1] +passage = tok(body, add_special_tokens=True)["input_ids"][:900] +# Gemma 2's template refuses a system turn; fold it into the first user turn, as Gemma 3's +# template does itself (the app has no Gemma 2 template to copy). +_apply = tok.apply_chat_template + + +def _folded(messages, *args, **kwargs): + if messages and messages[0]["role"] == "system": + first, rest = messages[1], messages[2:] + messages = [{"role": "user", "content": messages[0]["content"] + "\n\n" + first["content"]}, *rest] + return _apply(messages, *args, **kwargs) + + +if which == "gemma2": + tok.apply_chat_template = _folded +prompts = gate.prompts(tok) +gate_ids = {q: tok(t, add_special_tokens=False)["input_ids"] for q, t in prompts.items()} +WINDOW = max(len(passage), *map(len, gate_ids.values())) + 8 +print("passage", len(passage), "tokens; gate prompts", min(map(len, gate_ids.values())), "-", + max(map(len, gate_ids.values())), "; window", WINDOW, flush=True) + + +def rot_table(base): + inv = 1.0 / (base ** (torch.arange(0, head_dim, 2, dtype=torch.float32) / head_dim)) + freqs = torch.outer(torch.arange(WINDOW + 8, dtype=torch.float32), inv) + emb = torch.cat((freqs, freqs), dim=-1) + return torch.stack((emb.cos(), emb.sin()))[None] # (1, 2, len, head_dim) + + +master = get_master_rot_emb(config, dtype=torch.float32) +for base in (10000.0, 1e6): + ref = rot_table(base) + n = min(ref.shape[2], master.shape[2]) + print(f"MediaTek master table vs base {base:g}: max |diff| {float((master[:, :, :n] - ref[:, :, :n]).abs().max()):.2e}", + flush=True) + + +def masks(n, mask_value, sliding): + full = torch.full((1, 1, n, WINDOW + n), mask_value) + local = full.clone() + for r in range(n): + full[0, 0, r, WINDOW : WINDOW + r + 1] = 0.0 + first = max(0, r - sliding + 1) if sliding else 0 + local[0, 0, r, WINDOW + first : WINDOW + r + 1] = 0.0 + return full, local + + +def run(ids, mask_value, rope): + n = len(ids) + hidden = embedding(torch.tensor([ids], dtype=torch.int32)) + cache = [torch.zeros(1, kv_heads, WINDOW, head_dim) for _ in range(2 * n_layers)] + if which == "gemma2": + full, _ = masks(n, mask_value, 0) + pos = rope["global"][:, :, :n] + with torch.no_grad(): + return chunk(hidden, full, pos, *cache)[0][0] + full, local = masks(n, mask_value, config.sliding_window_attention_size) + # The patched Gemma3ModelChunk (mediatek-gemma.patch) takes the global table, then the + # sliding one; a single-table variant hands it the same table twice. + pos_g = rope["global"][:, :, :n] + pos_l = rope.get("local", rope["global"])[:, :, :n] + with torch.no_grad(): + return chunk(hidden, full, local, pos_g, pos_l, *cache)[0][0] + + +hf = AutoModelForCausalLM.from_pretrained(snap, dtype=torch.float32, attn_implementation="eager").eval() +with torch.no_grad(): + hf_passage = hf(torch.tensor([passage])).logits[0].double() + hf_gate = {q: hf(torch.tensor([ids])).logits[0, -1].double() for q, ids in gate_ids.items()} +del hf + + +def compare(a, b, ids=None): + la, lb = torch.log_softmax(a, -1), torch.log_softmax(b, -1) + kl = (la.exp() * (la - lb)).sum(-1) + out = {"kl_mean": float(kl.mean()), "kl_max": float(kl.max()), "top1": float((a.argmax(-1) == b.argmax(-1)).double().mean())} + if ids is not None: + target = torch.tensor(ids[1:]) + out["ppl_hf"] = float(torch.exp(-la[:-1].gather(1, target[:, None]).mean())) + out["ppl_mtk"] = float(torch.exp(-lb[:-1].gather(1, target[:, None]).mean())) + return out + + +def graded(mtk_gate): + ref, measured = {"rows": {}}, {} + for q in gate_ids: + p, m = torch.softmax(hf_gate[q], -1), torch.softmax(mtk_gate[q], -1) + t = int(p.argmax()) + ref["rows"][q] = {"token": t, "p": float(p[t])} + measured[q] = {"top": int(m.argmax()), "p": float(m[t])} + v = gate.check(ref, measured) + v.pop("rows") + return {k: v[k] for k in ("passed", "agreed", "graded", "fp32_mean", "export_mean", "quiet_rows_answered_like_fp32", + "quiet_rows", "problems")} + + +if which == "gemma2": + ropes = [{"name": "10000 everywhere", "global": rot_table(10000.0)}] +else: + ropes = [ + {"name": "10000 everywhere", "global": rot_table(10000.0)}, + {"name": "1e6 everywhere", "global": rot_table(1e6)}, + {"name": "per-layer", "local": rot_table(float(config.rope_local_base_freq)), "global": rot_table(1e6)}, + ] +results = {"repo": REPO, "sha": SHA, "passage_tokens": len(passage), "variants": []} +for rope in ropes: + for mask_value in (-100.0,): + mtk_passage = run(passage, mask_value, rope).double() + mtk_gate = {q: run(ids, mask_value, rope)[-1].double() for q, ids in gate_ids.items()} + row = {"rope": rope["name"], "mask": mask_value, "passage": compare(hf_passage, mtk_passage, passage), + "gate_last_token": compare(torch.stack(list(hf_gate.values())), torch.stack(list(mtk_gate.values()))), + "gate": graded(mtk_gate)} + results["variants"].append(row) + print(json.dumps(row), flush=True) +out_path.write_text(json.dumps(results, indent=1)) diff --git a/docs/research/evidence/mtk-gemma2-head-probe.log b/docs/research/evidence/mtk-gemma2-head-probe.log new file mode 100644 index 0000000..c0f29e6 --- /dev/null +++ b/docs/research/evidence/mtk-gemma2-head-probe.log @@ -0,0 +1,277 @@ +===== python mtk-gemma2-head-probe.py plain (2026-11-09, r7i.4xlarge, Ubuntu 24.04 x86_64) +Failed to load /opt/mtk-venv/lib/python3.10/site-packages/torchao/_C_cutlass_90a.abi3.so: Could not load this library: /opt/mtk-venv/lib/python3.10/site-packages/torchao/_C_cutlass_90a.abi3.so +Failed to load /opt/mtk-venv/lib/python3.10/site-packages/torchao/_C_mxfp8.cpython-310-x86_64-linux-gnu.so: Could not load this library: /opt/mtk-venv/lib/python3.10/site-packages/torchao/_C_mxfp8.cpython-310-x86_64-linux-gnu.so +W1109 06:55:19.472000 6640 torch/utils/_pytree.py:630] is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release. + +RuntimeWarning: An installation of PyTorch 2.14.0+cpu has been detected, which falls outside the supported version range: 2.6.0 > torch >= 2.1.0. As a result, the tool behavior is not fully-verified and may fail unexpectedly. It is recommended to install a supported PyTorch version. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_quantize_per_tensor_default/input_1' from DT_FLOAT64 to DT_FLOAT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'norm/aten_native_layer_norm_default/input_1' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_dequantize_per_tensor_default_1/input_1' from DT_FLOAT64 to DT_FLOAT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_quantize_per_tensor_default_2/input_1' from DT_FLOAT64 to DT_FLOAT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'lm_head/aten_permute_copy_default/input_1' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_quantize_per_tensor_default_2/input_2' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_dequantize_per_tensor_default_2/input_1' from DT_FLOAT64 to DT_FLOAT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor '_zero_point_0' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_dequantize_per_tensor_default_2/input_2' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'lm_head/aten_view_copy_default/input_1' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_quantize_per_tensor_default/input_2' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_quantize_per_tensor_default_1/input_2' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_dequantize_per_tensor_default/input_1' from DT_FLOAT64 to DT_FLOAT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_quantize_per_tensor_default_1/input_1' from DT_FLOAT64 to DT_FLOAT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'lm_head/aten_view_copy_default_1/input_1' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_dequantize_per_tensor_default/input_2' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_dequantize_per_tensor_default_1/input_2' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: The following operator types are not supported by PyTorch delegation: FakeQuantize. Expect longer execution time on PC. + +Traceback (most recent call last): + File "/home/ubuntu/probe/head_probe.py", line 51, in + delegated = to_backend(MethodProgramsPartitionerSpec({"head": edge}, {"head": NeuropilotPartitioner(spec)})) + File "/home/ubuntu/.local/share/uv/python/cpython-3.10-linux-x86_64-gnu/lib/python3.10/functools.py", line 889, in wrapper + return dispatch(args[0].__class__)(*args, **kw) + File "/opt/mtk-venv/lib/python3.10/site-packages/executorch/exir/backend/backend_api.py", line 768, in _ + lower_all_submodules_to_backend( + File "/opt/mtk-venv/lib/python3.10/site-packages/executorch/exir/backend/backend_api.py", line 597, in lower_all_submodules_to_backend + backend_cls.preprocess_multimethod( + File "/opt/mtk-venv/lib/python3.10/site-packages/executorch/backends/mediatek/preprocess.py", line 165, in preprocess_multimethod + preprocess_result = cls.preprocess(program, compile_spec_for_program) + File "/opt/mtk-venv/lib/python3.10/site-packages/executorch/backends/mediatek/preprocess.py", line 136, in preprocess + mlir_str = converter.convert_to_mlir() + File "mtk_converter/python/converters/base_converter.py", line 1418, in mtk_converter.python.converters.base_converter.BaseConverter.convert_to_mlir + File "mtk_converter/python/converters/base_converter.py", line 1350, in mtk_converter.python.converters.base_converter.BaseConverter._convert_to_nrpmodel + File "mtk_converter/python/converters/base_converter.py", line 1354, in mtk_converter.python.converters.base_converter.BaseConverter._convert_to_nrpmodel + File "mtk_converter/python/converters/transformer.py", line 988, in mtk_converter.python.converters.transformer.Transformer.run + File "mtk_converter/python/converters/transformer.py", line 1243, in mtk_converter.python.converters.transformer.Transformer.run_pre_quantization_transformations + File "mtk_converter/python/converters/transformer.py", line 449, in mtk_converter.python.converters.transformer.apply_transformations + File "mtk_converter/python/converters/transformer.py", line 478, in mtk_converter.python.converters.transformer.apply_transformations + File "mtk_converter/python/converters/transformer.py", line 463, in mtk_converter.python.converters.transformer.apply_transformations + File "mtk_converter/python/converters/transformations/resolve_fake_quantize.py", line 136, in mtk_converter.python.converters.transformations.resolve_fake_quantize.apply + File "mtk_converter/python/tools/executor.py", line 534, in mtk_converter.python.tools.executor.Executor.__init__ + File "mtk_converter/python/tools/executor.py", line 167, in mtk_converter.python.tools.executor._ModelExecutorImpl.__init__ + File "mtk_converter/python/tools/executor.py", line 387, in mtk_converter.python.tools.executor._ModelExecutorImpl._create_submodel_impls + File "mtk_converter/python/tools/executor.py", line 87, in mtk_converter.python.tools.executor._SubmodelExecutorImpl.__init__ + File "mtk_converter/python/py_interpreter.py", line 57, in mtk_converter.python.py_interpreter.PyInterpreter.__init__ +ValueError: Message mtk_converter.ModelProto exceeds maximum protobuf size of 2GB: 2361344528 +exit 1 + +===== python mtk-gemma2-head-probe.py fp16 (2026-11-09, r7i.4xlarge, Ubuntu 24.04 x86_64) +Failed to load /opt/mtk-venv/lib/python3.10/site-packages/torchao/_C_cutlass_90a.abi3.so: Could not load this library: /opt/mtk-venv/lib/python3.10/site-packages/torchao/_C_cutlass_90a.abi3.so +Failed to load /opt/mtk-venv/lib/python3.10/site-packages/torchao/_C_mxfp8.cpython-310-x86_64-linux-gnu.so: Could not load this library: /opt/mtk-venv/lib/python3.10/site-packages/torchao/_C_mxfp8.cpython-310-x86_64-linux-gnu.so +W1109 06:55:32.311000 6686 torch/utils/_pytree.py:630] is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release. + +RuntimeWarning: An installation of PyTorch 2.14.0+cpu has been detected, which falls outside the supported version range: 2.6.0 > torch >= 2.1.0. As a result, the tool behavior is not fully-verified and may fail unexpectedly. It is recommended to install a supported PyTorch version. + +converter: convert_float32_weights_to_float16 = True + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_dequantize_per_tensor_default_2/input_2' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_dequantize_per_tensor_default_1/input_2' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_dequantize_per_tensor_default/input_2' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_dequantize_per_tensor_default/input_1' from DT_FLOAT64 to DT_FLOAT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_quantize_per_tensor_default_1/input_2' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_quantize_per_tensor_default_2/input_1' from DT_FLOAT64 to DT_FLOAT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_dequantize_per_tensor_default_1/input_1' from DT_FLOAT64 to DT_FLOAT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'lm_head/aten_view_copy_default_1/input_1' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_quantize_per_tensor_default_2/input_2' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'norm/aten_native_layer_norm_default/input_1' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'lm_head/aten_permute_copy_default/input_1' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor '_zero_point_0' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_quantize_per_tensor_default/input_1' from DT_FLOAT64 to DT_FLOAT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_dequantize_per_tensor_default_2/input_1' from DT_FLOAT64 to DT_FLOAT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_quantize_per_tensor_default/input_2' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'lm_head/aten_view_copy_default/input_1' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_quantize_per_tensor_default_1/input_1' from DT_FLOAT64 to DT_FLOAT32. Expect quality drop in some cases. + + +RuntimeWarning: The following operator types are not supported by PyTorch delegation: FakeQuantize. Expect longer execution time on PC. + +Traceback (most recent call last): + File "/home/ubuntu/probe/head_probe.py", line 51, in + delegated = to_backend(MethodProgramsPartitionerSpec({"head": edge}, {"head": NeuropilotPartitioner(spec)})) + File "/home/ubuntu/.local/share/uv/python/cpython-3.10-linux-x86_64-gnu/lib/python3.10/functools.py", line 889, in wrapper + return dispatch(args[0].__class__)(*args, **kw) + File "/opt/mtk-venv/lib/python3.10/site-packages/executorch/exir/backend/backend_api.py", line 768, in _ + lower_all_submodules_to_backend( + File "/opt/mtk-venv/lib/python3.10/site-packages/executorch/exir/backend/backend_api.py", line 597, in lower_all_submodules_to_backend + backend_cls.preprocess_multimethod( + File "/opt/mtk-venv/lib/python3.10/site-packages/executorch/backends/mediatek/preprocess.py", line 165, in preprocess_multimethod + preprocess_result = cls.preprocess(program, compile_spec_for_program) + File "/opt/mtk-venv/lib/python3.10/site-packages/executorch/backends/mediatek/preprocess.py", line 136, in preprocess + mlir_str = converter.convert_to_mlir() + File "mtk_converter/python/converters/base_converter.py", line 1418, in mtk_converter.python.converters.base_converter.BaseConverter.convert_to_mlir + File "mtk_converter/python/converters/base_converter.py", line 1350, in mtk_converter.python.converters.base_converter.BaseConverter._convert_to_nrpmodel + File "mtk_converter/python/converters/base_converter.py", line 1354, in mtk_converter.python.converters.base_converter.BaseConverter._convert_to_nrpmodel + File "mtk_converter/python/converters/transformer.py", line 988, in mtk_converter.python.converters.transformer.Transformer.run + File "mtk_converter/python/converters/transformer.py", line 1243, in mtk_converter.python.converters.transformer.Transformer.run_pre_quantization_transformations + File "mtk_converter/python/converters/transformer.py", line 449, in mtk_converter.python.converters.transformer.apply_transformations + File "mtk_converter/python/converters/transformer.py", line 478, in mtk_converter.python.converters.transformer.apply_transformations + File "mtk_converter/python/converters/transformer.py", line 463, in mtk_converter.python.converters.transformer.apply_transformations + File "mtk_converter/python/converters/transformations/resolve_fake_quantize.py", line 136, in mtk_converter.python.converters.transformations.resolve_fake_quantize.apply + File "mtk_converter/python/tools/executor.py", line 534, in mtk_converter.python.tools.executor.Executor.__init__ + File "mtk_converter/python/tools/executor.py", line 167, in mtk_converter.python.tools.executor._ModelExecutorImpl.__init__ + File "mtk_converter/python/tools/executor.py", line 387, in mtk_converter.python.tools.executor._ModelExecutorImpl._create_submodel_impls + File "mtk_converter/python/tools/executor.py", line 87, in mtk_converter.python.tools.executor._SubmodelExecutorImpl.__init__ + File "mtk_converter/python/py_interpreter.py", line 57, in mtk_converter.python.py_interpreter.PyInterpreter.__init__ +ValueError: Message mtk_converter.ModelProto exceeds maximum protobuf size of 2GB: 2361344528 +exit 1 + +===== python mtk-gemma2-head-probe.py fold (2026-11-09, r7i.4xlarge, Ubuntu 24.04 x86_64) +Failed to load /opt/mtk-venv/lib/python3.10/site-packages/torchao/_C_cutlass_90a.abi3.so: Could not load this library: /opt/mtk-venv/lib/python3.10/site-packages/torchao/_C_cutlass_90a.abi3.so +Failed to load /opt/mtk-venv/lib/python3.10/site-packages/torchao/_C_mxfp8.cpython-310-x86_64-linux-gnu.so: Could not load this library: /opt/mtk-venv/lib/python3.10/site-packages/torchao/_C_mxfp8.cpython-310-x86_64-linux-gnu.so +W1109 06:55:59.392000 6767 torch/utils/_pytree.py:630] is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release. + +RuntimeWarning: An installation of PyTorch 2.14.0+cpu has been detected, which falls outside the supported version range: 2.6.0 > torch >= 2.1.0. As a result, the tool behavior is not fully-verified and may fail unexpectedly. It is recommended to install a supported PyTorch version. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'norm/aten_native_layer_norm_default/input_1' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_dequantize_per_tensor_default/input_1' from DT_FLOAT64 to DT_FLOAT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_quantize_per_tensor_default_2/input_2' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_dequantize_per_tensor_default_1/input_1' from DT_FLOAT64 to DT_FLOAT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor '_zero_point_0' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'lm_head/aten_view_copy_default_1/input_1' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_dequantize_per_tensor_default_2/input_2' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'lm_head/aten_permute_copy_default/input_1' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_quantize_per_tensor_default_1/input_1' from DT_FLOAT64 to DT_FLOAT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_quantize_per_tensor_default_2/input_1' from DT_FLOAT64 to DT_FLOAT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_quantize_per_tensor_default_1/input_2' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_dequantize_per_tensor_default_2/input_1' from DT_FLOAT64 to DT_FLOAT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_dequantize_per_tensor_default_1/input_2' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_dequantize_per_tensor_default/input_2' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_quantize_per_tensor_default/input_2' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'quantized_decomposed_quantize_per_tensor_default/input_1' from DT_FLOAT64 to DT_FLOAT32. Expect quality drop in some cases. + + +RuntimeWarning: Forcibly convert the unsupported type of tensor 'lm_head/aten_view_copy_default/input_1' from DT_INT64 to DT_INT32. Expect quality drop in some cases. + + +RuntimeWarning: The following operator types are not supported by PyTorch delegation: Torch_AtenFakeQuantizeAffine. Expect longer execution time on PC. + + +RuntimeWarning: The following operator types are not supported by PyTorch delegation: FakeQuantize. Expect longer execution time on PC. + +Traceback (most recent call last): + File "/home/ubuntu/probe/head_probe.py", line 51, in + delegated = to_backend(MethodProgramsPartitionerSpec({"head": edge}, {"head": NeuropilotPartitioner(spec)})) + File "/home/ubuntu/.local/share/uv/python/cpython-3.10-linux-x86_64-gnu/lib/python3.10/functools.py", line 889, in wrapper + return dispatch(args[0].__class__)(*args, **kw) + File "/opt/mtk-venv/lib/python3.10/site-packages/executorch/exir/backend/backend_api.py", line 768, in _ + lower_all_submodules_to_backend( + File "/opt/mtk-venv/lib/python3.10/site-packages/executorch/exir/backend/backend_api.py", line 597, in lower_all_submodules_to_backend + backend_cls.preprocess_multimethod( + File "/opt/mtk-venv/lib/python3.10/site-packages/executorch/backends/mediatek/preprocess.py", line 165, in preprocess_multimethod + preprocess_result = cls.preprocess(program, compile_spec_for_program) + File "/opt/mtk-venv/lib/python3.10/site-packages/executorch/backends/mediatek/preprocess.py", line 136, in preprocess + mlir_str = converter.convert_to_mlir() + File "mtk_converter/python/converters/base_converter.py", line 1418, in mtk_converter.python.converters.base_converter.BaseConverter.convert_to_mlir + File "mtk_converter/python/converters/base_converter.py", line 1350, in mtk_converter.python.converters.base_converter.BaseConverter._convert_to_nrpmodel + File "mtk_converter/python/converters/base_converter.py", line 1354, in mtk_converter.python.converters.base_converter.BaseConverter._convert_to_nrpmodel + File "mtk_converter/python/converters/transformer.py", line 988, in mtk_converter.python.converters.transformer.Transformer.run + File "mtk_converter/python/converters/transformer.py", line 1243, in mtk_converter.python.converters.transformer.Transformer.run_pre_quantization_transformations + File "mtk_converter/python/converters/transformer.py", line 449, in mtk_converter.python.converters.transformer.apply_transformations + File "mtk_converter/python/converters/transformer.py", line 478, in mtk_converter.python.converters.transformer.apply_transformations + File "mtk_converter/python/converters/transformer.py", line 463, in mtk_converter.python.converters.transformer.apply_transformations + File "mtk_converter/python/converters/transformations/resolve_fake_quantize.py", line 136, in mtk_converter.python.converters.transformations.resolve_fake_quantize.apply + File "mtk_converter/python/tools/executor.py", line 534, in mtk_converter.python.tools.executor.Executor.__init__ + File "mtk_converter/python/tools/executor.py", line 167, in mtk_converter.python.tools.executor._ModelExecutorImpl.__init__ + File "mtk_converter/python/tools/executor.py", line 387, in mtk_converter.python.tools.executor._ModelExecutorImpl._create_submodel_impls + File "mtk_converter/python/tools/executor.py", line 87, in mtk_converter.python.tools.executor._SubmodelExecutorImpl.__init__ + File "mtk_converter/python/py_interpreter.py", line 57, in mtk_converter.python.py_interpreter.PyInterpreter.__init__ +ValueError: Message mtk_converter.ModelProto exceeds maximum protobuf size of 2GB: 2361344648 +exit 1 + diff --git a/docs/research/evidence/mtk-gemma2-head-probe.py b/docs/research/evidence/mtk-gemma2-head-probe.py new file mode 100644 index 0000000..48099ae --- /dev/null +++ b/docs/research/evidence/mtk-gemma2-head-probe.py @@ -0,0 +1,61 @@ +"""Does MediaTek's converter take Gemma 2's 256000 x 2304 head? (docs/research, finding 47) + + /opt/mtk-venv/bin/python mtk-gemma2-head-probe.py + +Gemma 2 2B's final norm and tied output head alone, quantized A16W8 with NeuropilotQuantizer and +lowered for MT6991 as gemma.py lowers a chunk (ExecuTorch 3b60683, mtk_converter 8.13.0, +mtk_neuron 8.2.23). plain: as gemma.py does it; fp16: the converter's +convert_float32_weights_to_float16; fold: convert_pt2e(fold_quantize=True), int8 weight +constants. All three: "ValueError: Message mtk_converter.ModelProto exceeds maximum protobuf +size of 2GB" (mtk-gemma2-head-probe.log). +""" +import struct +import sys + +import mtk_converter +import torch +from executorch import exir +from executorch.backends.mediatek import NeuropilotPartitioner, NeuropilotQuantizer, Precision +from executorch.exir.backend.backend_api import MethodProgramsPartitionerSpec, to_backend +from executorch.exir.backend.backend_details import CompileSpec +from torchao.quantization.pt2e.quantize_pt2e import convert_pt2e, prepare_pt2e + +mode = sys.argv[1] +if mode == "fp16": + original = mtk_converter.PyTorchV2Converter.from_exported_program + + def from_exported_program(*args, **kwargs): + converter = original(*args, **kwargs) + converter.convert_float32_weights_to_float16 = True + print("converter: convert_float32_weights_to_float16 = True", flush=True) + return converter + + mtk_converter.PyTorchV2Converter.from_exported_program = staticmethod(from_exported_program) + + +class Head(torch.nn.Module): + def __init__(self): + super().__init__() + self.norm = torch.nn.LayerNorm(2304) + self.lm_head = torch.nn.Linear(2304, 256000, bias=False) + + def forward(self, x): + return self.lm_head(self.norm(x)) + + +torch.manual_seed(0) +model = Head().eval() +x = torch.randn(1, 1, 2304) +graph = torch.export.export(model, (x,), strict=True).module() +quantizer = NeuropilotQuantizer() +quantizer.setup_precision(Precision.A16W8) +prepared = prepare_pt2e(graph, quantizer) +prepared(x) +converted = convert_pt2e(prepared, fold_quantize=(mode == "fold")) +edge = exir.to_edge(torch.export.export(converted, (x,), strict=True)).exported_program() +spec = [CompileSpec("gno", b"LTS"), CompileSpec("gno-exp", b""), CompileSpec("gno-non-4d-tiling", b""), + CompileSpec("ImportForever", struct.pack("?", True)), CompileSpec("platform-config", b"mt6991")] +delegated = to_backend(MethodProgramsPartitionerSpec({"head": edge}, {"head": NeuropilotPartitioner(spec)})) +program = exir.EdgeProgramManager(delegated, compile_config=exir.EdgeCompileConfig(_check_ir_validity=False)) +buffer = program.to_executorch().buffer +print(f"OK {mode}: lowered, {len(buffer):,} bytes", flush=True) diff --git a/pipeline/__main__.py b/pipeline/__main__.py index 86d7d7d..9427242 100644 --- a/pipeline/__main__.py +++ b/pipeline/__main__.py @@ -392,6 +392,7 @@ def _publish_hf(args) -> int: args.target, args.context, allow_unpublished_twin=getattr(args, "allow_unpublished_twin", False), + replace_shared=getattr(args, "replace_shared", False), ) ) return 0 @@ -672,6 +673,13 @@ def main(argv: list[str] | None = None) -> int: action="store_true", help="publish a GPTQ Vulkan file whose gate-proxy XNNPACK twin is deliberately not on the Hub", ) + if name == "publish-hf": + command.add_argument( + "--replace-shared", + action="store_true", + help="replace a shared file (MediaTek's embedding) the folder's other windows were built on, " + "withdrawing those windows in the same commit", + ) command.set_defaults(func=func) args = parser.parse_args(argv) diff --git a/pipeline/eligibility.py b/pipeline/eligibility.py index e4704a8..ff74c3e 100644 --- a/pipeline/eligibility.py +++ b/pipeline/eligibility.py @@ -23,6 +23,8 @@ # Ahead of the app (naming.AHEAD_OF_APP): it has no LFM3 template yet. "lfm3": {"lfm3"}, "phi4_mini": {"phi4mini"}, + # Ahead of both apps (naming.AHEAD_OF_APP). + "gemma2": {"gemma2"}, } # What a release of each family is called. The architecture class alone is not enough: a # later generation can keep the class (a text-only "Qwen3.8" on Qwen3ForCausalLM would pass @@ -35,6 +37,7 @@ "llama": re.compile(r"llama-?3\.2(?![.\d])|smollm2(?!\d)", re.IGNORECASE), "gemma3": re.compile(r"gemma-?3(?![.\dn])", re.IGNORECASE), "smollm3": re.compile(r"smollm3(?!\d)", re.IGNORECASE), + "gemma2": re.compile(r"gemma-?2(?![.\dn])", re.IGNORECASE), # Phi-4-mini only: Phi-3 and Phi-3.5 share the architecture class, and the app's template # is Phi-4-mini's (Phi4Prompt). "phi4_mini": re.compile(r"phi-?4-mini(?![-.]?\d)", re.IGNORECASE), diff --git a/pipeline/export_mtk.py b/pipeline/export_mtk.py index a801eb9..21ebfe5 100644 --- a/pipeline/export_mtk.py +++ b/pipeline/export_mtk.py @@ -98,6 +98,16 @@ # which get_master_rot_emb otherwise replaces with 10000 (docs/research, finding 37). ROPE_MARKER = 'self.rope_theta = float(kwargs.pop("rope_theta", 10000.0) or 10000.0)' ROPE_CONFIGS = {"qwen.py": "configuration_qwen.py", "llama.py": "configuration_llama.py"} +# mediatek-gemma.patch: Gemma 3's chunk takes the sliding layers' RoPE table as an input of its +# own, and gemma.py builds and feeds it. Unpatched, the graph gives every layer one table +# (fp32 gate 6/11 or 5/11, docs/research finding 46) while runner_settings names two. +GEMMA_PATCH = "patches/mediatek-gemma.patch" +GEMMA3_MARKERS = { + "models/llm_models/modeling_gemma.py": ( + "def forward(self, inputs_embeds, global_mask, local_mask, pos_emb, pos_emb_local, *cache):" + ), + "model_export_scripts/gemma.py": "def local_rot_emb(config):", +} # third_party/executorch/patches/mediatek-lfm2.patch: lfm2.py calibrates every chunk in one # streaming pass over the long-context corpus (pipeline/mtk_corpus.py) and keeps no step, so # the Arrow round trip and the arrays patch do not apply to it. @@ -314,6 +324,50 @@ def modal_shape(window: int) -> tuple[int, int] | None: return None +def absolute_tool(tool_python: str) -> str: + """The tool python as a path that holds from any working directory; a bare command name is + left to PATH.""" + return str(Path(tool_python).resolve()) if os.sep in tool_python else tool_python + + +def absolute_inputs( + out_dir: Path, work_dir: Path, examples_dir: Path, tool_python: str +) -> tuple[Path, Path, Path, str]: + """run()'s paths, made absolute before anything else: every subprocess it starts runs from + another directory (pipeline.mtk_corpus from the repo, the export script from the run's own + copy of the scripts), where a caller's relative path means something else (codex review of + #30).""" + return out_dir.resolve(), work_dir.resolve(), examples_dir.resolve(), absolute_tool(tool_python) + + +def script_paths(tool_python: str, weight_dir: Path) -> tuple[str, Path]: + """The tool python and the weights' config.json as the script needs them: absolute, because + it runs in the run's own copy of the scripts (private_examples), where a relative --work or + tool path would point into that copy.""" + return absolute_tool(tool_python), (weight_dir / "config.json").resolve() + + +def private_examples(examples_dir: Path, work_dir: Path) -> Path: + """A copy of examples/mediatek (31 MB) in this run's work folder, to run the script in. + + The scripts write their chunks to pte/ under the working directory, and + (utils.get_exp_name) names the weights, precision and chunk count but not the window, so + two windows of one model shared the folder, and each run deletes it when done. On one host + (an EC2 box, 2026-11-09; CI gives every job its own runner) Gemma 3's 8k run finished + first and took the 16k run's chunk 0 with it. Its own copy keeps a run's chunks its own, + and goes with the work folder. + """ + run_dir = (work_dir / "mediatek").resolve() + source = examples_dir.resolve() + # The copy is deleted and rebuilt below, so it may neither be the scripts nor hold them, nor + # sit inside them (codex review of #30, round five: --examples /mediatek deleted them). + if run_dir == source or source.is_relative_to(run_dir) or run_dir.is_relative_to(source): + raise ExportError(f"--examples {source} overlaps the run's own copy at {run_dir}; keep it outside --work") + shutil.rmtree(run_dir, ignore_errors=True) + shutil.copytree(examples_dir, run_dir, symlinks=True, ignore=shutil.ignore_patterns("pte", "__pycache__")) + return run_dir + + def exp_name(weight_dir: Path, precision: str, chunks: int) -> str: """The name the scripts give the output folder and files (utils.get_exp_name).""" return f"{weight_dir.name}_{precision}_{chunks}_chunks" @@ -346,6 +400,27 @@ def runner_settings(config: dict, recipe: settings.MtkRecipe, bos: int | None, e "eos_token": eos_token if eos_token is not None else (eos[0] if eos else None), "eos_tokens": eos, "tokenizer_type": "hf", + **gemma3_runner_settings(config), + } + + +def gemma3_runner_settings(config: dict) -> dict: + """What a runner needs beyond the flags above to drive a Gemma 3 graph. + + ``window_size``: the sliding layers' window, for the second mask the graph takes after the + global one (mtk_llama_executor_runner --window_size, MaskBuilder::enableSlidingWindow). + ``rot_emb_local_base``: the base of the second RoPE table, the input that follows the + global one in a graph built with mediatek-gemma.patch. ``rot_emb_base`` stays the global + layers' (rope_theta). Neither MediaTek's stock runner nor any shipped app feeds that second + table yet; one that does not cannot run these files correctly. + """ + mtk = families.mtk_model_config(config) + if mtk.get("model_type") != "gemma3": + return {} + return { + "window_size": int(mtk["sliding_window_attention_size"]), + "rot_emb_local_base": float(mtk["rope_local_base_freq"]), + "rot_emb_inputs": ["global", "local"], } @@ -405,6 +480,7 @@ def run( keep_work: bool = False, context: int | None = None, ) -> dict: + out_dir, work_dir, examples_dir, tool_python = absolute_inputs(out_dir, work_dir, examples_dir, tool_python) cfg = settings.load() recipe = cfg.mtk if soc not in SOC_PLATFORMS: @@ -447,6 +523,10 @@ def run( script_text = script.read_text(encoding="utf-8") if not streaming and (PATCH_MARKER not in script_text or WRITER_MARKER not in script_text): raise ExportError(f"{script} lacks third_party/executorch/patches/mediatek-calibration-as-arrays.patch") + if families.text_config(families.mtk_model_config(source.config)).get("model_type") == "gemma3": + for relative, marker in GEMMA3_MARKERS.items(): + if marker not in (examples_dir / relative).read_text(encoding="utf-8"): + raise ExportError(f"{examples_dir / relative} lacks the patch third_party/executorch/{GEMMA_PATCH}") if plan.script in ROPE_CONFIGS: config_file = examples_dir / "models" / "llm_models" / ROPE_CONFIGS[plan.script] if ROPE_MARKER not in config_file.read_text(encoding="utf-8"): @@ -520,7 +600,10 @@ def run( tokenizer, licenses = copy_side_files(source, weight_dir, out_dir) # The MediaTek script reads the weights' own config.json: give it the keys it expects # (hub.normalize_config: the first LFM2 checkpoints spell them the older way). - model_config = hub.normalize_config(json.loads((weight_dir / "config.json").read_text(encoding="utf-8"))) + # Gemma 3's keys in MediaTek's spelling (families.mtk_model_config). + model_config = families.mtk_model_config( + hub.normalize_config(json.loads((weight_dir / "config.json").read_text(encoding="utf-8"))) + ) if plan.tokenizer: model_config["tokenizer"] = plan.tokenizer (weight_dir / "config.json").write_text(json.dumps(model_config, indent=2), encoding="utf-8") @@ -547,19 +630,23 @@ def run( ) # fmt: skip corpus = json.loads(summary_path.read_text(encoding="utf-8")) - command = export_command(tool_python, plan, recipe, soc, weight_dir / "config.json", dataset) - print("==> " + " ".join(command)) + # The script runs in the run's own copy of the scripts, so every path handed to it is absolute: + # a relative --work (or tool python) would be read from that copy (codex review of #30). + tool, config_path = script_paths(tool_python, weight_dir) + command = export_command(tool, plan, recipe, soc, config_path, dataset) + run_dir = private_examples(examples_dir, work_dir) + print("==> " + " ".join(command) + f" (in {run_dir})") export_started = time.time() env = tool_env(tool_python, dict(os.environ)) with MemorySampler() as memory: - subprocess.run(command, check=True, cwd=examples_dir, env=env) + subprocess.run(command, check=True, cwd=run_dir, env=env) export_seconds = time.time() - export_started exp = exp_name(weight_dir, recipe.precision, plan.num_chunks) - built = [examples_dir / "pte" / exp / f"{exp}_{i}.pte" for i in range(plan.num_chunks)] + built = [run_dir / "pte" / exp / f"{exp}_{i}.pte" for i in range(plan.num_chunks)] missing = [p.name for p in built if not p.is_file()] if missing: - raise ExportError(f"the export wrote no {missing} in {examples_dir / 'pte' / exp}") + raise ExportError(f"the export wrote no {missing} in {run_dir / 'pte' / exp}") chunks = [] for path, name in zip(built, chunk_names, strict=True): shutil.move(str(path), backend_dir / name) @@ -653,6 +740,9 @@ def run( }, "metadata": {}, "smoke": check, + # What this window computes differently from Hugging Face (families.mtk_limits); the card + # states each. + **({"limits": limits} if (limits := families.mtk_limits(source.config, window)) else {}), "measuring_gate": gate.measuring_gate(check.get("stats")), "run": manifest.run_info(), } @@ -661,8 +751,7 @@ def run( ) (backend_dir / report_file(window)).write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") if not keep_work: - shutil.rmtree(work_dir, ignore_errors=True) - shutil.rmtree(examples_dir / "pte" / exp, ignore_errors=True) + shutil.rmtree(work_dir, ignore_errors=True) # with the run's own copy of the scripts if not check["passed"]: raise ExportError("structural check failed: " + "; ".join(check["problems"])) return report diff --git a/pipeline/exporting.py b/pipeline/exporting.py index 9afd024..69788bd 100644 --- a/pipeline/exporting.py +++ b/pipeline/exporting.py @@ -220,7 +220,9 @@ def copy_side_files(source: hub.SourceModel, src_dir: Path, out_dir: Path) -> tu if (src_dir / name).exists(): shutil.copy2(src_dir / name, out_dir / name) licenses.append(name) - token = naming.app_family(naming.source_name(source.id)) + # The pipeline's family, not the app's: a family exported ahead of the app (Gemma 2) is + # still under its license's notice requirement. + token = naming.pipeline_family(naming.source_name(source.id)) if token in manifest.NOTICES: (out_dir / "NOTICE").write_text(manifest.NOTICES[token], encoding="utf-8") return tokenizer, licenses diff --git a/pipeline/families.py b/pipeline/families.py index cd9b666..9eac725 100644 --- a/pipeline/families.py +++ b/pipeline/families.py @@ -58,8 +58,8 @@ def supports(self, backend: str) -> bool: "through: the fp32 MediaTek graph is at KL 0.013 and 98% top-1 against Hugging Face before " "any quantization (docs/research, finding 37)" ) -_MTK_GEMMA3 = "not validated on the MediaTek scripts (they expect model_type gemma3, HF has gemma3_text)" _MTK_NO_MODEL = "no model definition for this architecture in ExecuTorch 1.5.1's examples/mediatek" +_GEMMA2_NO_QNN = "ExecuTorch 1.5.1's Qualcomm registry has gemma2-2b, but execupack has not wired or validated it" _LFM2_NO_VULKAN = ( "the Vulkan delegate has no kernel for the short convolution, and an LFM2.5 file that " "lowers to it segfaults in the 1.4.0 runtime at the first prefill" @@ -98,10 +98,19 @@ def supports(self, backend: str) -> bool: ), Family("qwen2_5", ("Qwen2ForCausalLM",), {"mtk": _MTK_QWEN2_MASK}), Family("llama", ("LlamaForCausalLM",)), + # Gemma 3 on MediaTek: examples/mediatek/model_export_scripts/gemma.py with the config + # mtk_model_config writes (HF's gemma3_text keys in MediaTek's spelling). Family( "gemma3", ("Gemma3ForCausalLM",), - {"xnnpack": _NOT_IN_EXPORT_LLM, "vulkan": _NOT_IN_EXPORT_LLM, "mtk": _MTK_GEMMA3}, + {"xnnpack": _NOT_IN_EXPORT_LLM, "vulkan": _NOT_IN_EXPORT_LLM}, + ), + # Gemma 2: MediaTek only (gemma.py's Gemma2ModelChunk), and ahead of both apps, neither of + # which has a Gemma 2 chat template yet (naming.AHEAD_OF_APP). + Family( + "gemma2", + ("Gemma2ForCausalLM",), + {"xnnpack": _NOT_IN_EXPORT_LLM, "vulkan": _NOT_IN_EXPORT_LLM, "qnn": _GEMMA2_NO_QNN}, ), # Phi-4-mini: Llama's block with fused q/k/v and gate/up projections (pipeline/convert.py # splits them), partial RoPE on 96 of 128 head dims, and LongRoPE. ExecuTorch 1.5.1 builds @@ -168,9 +177,96 @@ class MtkPlan: "qwen2_5": ("qwen.py", "qwen.json", "qwen2"), "lfm2": ("lfm2.py", "qwen3.json", "lfm2"), "llama": ("llama.py", "llama3.json", "llama"), + # shell_scripts/export_gemma.sh: gemma.py and gemma.json for both generations. + "gemma3": ("gemma.py", "gemma.json", "gemma3"), + "gemma2": ("gemma.py", "gemma.json", "gemma2"), } +# Gemma 3's sliding layers as MediaTek's Gemma3ModelChunk names them (modeling_gemma.py), keyed +# by Hugging Face's layer_types spelling. +_GEMMA3_MTK_LAYER = {"sliding_attention": "SLIDING_LOCAL", "full_attention": "GLOBAL"} + + +def mtk_model_config(config: dict) -> dict: + """The config.json MediaTek's scripts read, from a Hugging Face one. + + Gemma 3 (HF model_type gemma3_text) takes MediaTek's own keys for what HF spells another way + (examples/mediatek/models/llm_models/configuration_gemma.py and weights/gemma3_1b_it): + the sliding window as ``sliding_window_attention_size`` (MediaTek's default is 1024, the + checkpoint's 512), the layer pattern as ``global_local_attention_pattern`` over one period, + and ``use_res_clamp`` false, since HF's model clamps no residual and MediaTek's default does. + Every other family's config is read as it is. + """ + c = text_config(config) + if c.get("model_type") not in ("gemma3", "gemma3_text"): + return config + out = dict(c) + out["model_type"] = "gemma3" + # Every shape this cannot translate is an UnsupportedModel, which eligibility turns into a + # verdict; anything else (StopIteration, KeyError) would end the watcher's whole pass + # (codex review of #30). + _require(bool(c.get("sliding_window")), "Gemma 3 config names no sliding_window") + period = int(c.get("sliding_window_pattern") or 0) + layer_types = c.get("layer_types") + if layer_types: + unknown = sorted({t for t in layer_types if t not in _GEMMA3_MTK_LAYER}) + _require(not unknown, f"Gemma 3 layer_types {unknown} have no MediaTek layer") + if not period: + first_global = next((i + 1 for i, t in enumerate(layer_types) if t == "full_attention"), None) + _require(first_global is not None, "Gemma 3 layer_types name no full_attention layer") + period = first_global + pattern = [_GEMMA3_MTK_LAYER[t] for t in layer_types[:period]] + if any(_GEMMA3_MTK_LAYER[t] != pattern[i % period] for i, t in enumerate(layer_types)): + raise UnsupportedModel("Gemma 3 layer_types do not repeat with one period") + else: + _require(period > 0, "Gemma 3 config names neither layer_types nor sliding_window_pattern") + # transformers' Gemma3TextConfig: every period-th layer is global, the rest slide. + pattern = ["SLIDING_LOCAL"] * (period - 1) + ["GLOBAL"] + out.update( + sliding_window_attention_size=int(c["sliding_window"]), + sliding_window_pattern=period, + global_local_attention_pattern=pattern, + use_res_clamp=False, + ) + return out + + +def mtk_limits(config: dict, window: int) -> list[str]: + """What a MediaTek export at ``window`` computes differently from Hugging Face, for the card. + + Gemma 2 alternates sliding layers (``sliding_window``, 4096) with global ones, and MediaTek's + Gemma2ModelChunk takes one mask for every layer: up to the sliding window the graph is HF's + (exact in fp32, measured 2026-11-09), past it the sliding layers see the whole window. + """ + c = text_config(config) + sliding = c.get("sliding_window") + if c.get("model_type") == "gemma2" and sliding and window > int(sliding): + return [ + f"Gemma 2's sliding layers attend over {int(sliding):,} tokens in Hugging Face's model and " + f"over the whole {window:,}-token window in MediaTek's graph, which takes one mask for every " + f"layer: prompts up to {int(sliding):,} tokens are computed as Hugging Face computes them, " + "longer ones are not." + ] + return [] + + +# mtk_converter 8.13.0 (PyTorchV2Converter) builds each delegated chunk as one protobuf ModelProto +# holding its weights in fp32, and protobuf refuses a message over 2 GB. The output head sits +# whole in the last chunk, so a head over the cap fails at every chunk count. Gemma 2 2B's +# tied 256000 x 2304 head is 2,359,296,000 bytes. A head-only probe (final norm + head, A16W8, +# 2026-11-09) was refused plain, with convert_float32_weights_to_float16, and with +# convert_pt2e(fold_quantize=True): "exceeds maximum protobuf size of 2GB" at 2,361,344,528 bytes +# (2,361,344,648 folded). docs/research, finding 47. +MTK_CONVERTER_PROTO_BYTES = 2**31 + + +def mtk_head_bytes(config: dict) -> int: + """The output head's size as mtk_converter holds it: vocab x hidden in fp32 (0 if unstated).""" + c = text_config(config) + return int(c.get("vocab_size") or 0) * int(c.get("hidden_size") or 0) * 4 + + def mtk_chunks(n_layers: int, max_chunks: int) -> int: """Most chunks up to ``max_chunks`` that split the layers evenly (the scripts require it).""" return next(n for n in range(min(max_chunks, n_layers), 0, -1) if n_layers % n == 0) @@ -203,7 +299,7 @@ def mtk_plan(family: Family, config: dict, max_chunks: int) -> MtkPlan: if family.key not in _MTK_SCRIPTS: raise UnsupportedModel(f"no MediaTek export script mapped for {family.key}") script, preformatter, model_type = _MTK_SCRIPTS[family.key] - c = text_config(config) + c = text_config(mtk_model_config(config)) if c.get("model_type") != model_type: raise UnsupportedModel(f"model_type {c.get('model_type')!r}, the MediaTek {script} path expects {model_type!r}") scaling = rope_scaling(c) @@ -218,6 +314,12 @@ def mtk_plan(family: Family, config: dict, max_chunks: int) -> MtkPlan: scaling = None if scaling is not None: raise UnsupportedModel("RoPE scaling is not validated on the MediaTek scripts") + head = mtk_head_bytes(c) + if head >= MTK_CONVERTER_PROTO_BYTES: + raise UnsupportedModel( + f"the output head is {head:,} bytes in fp32, and mtk_converter puts each chunk in one protobuf " + "capped at 2 GB, with the whole head in the last chunk at any chunk count" + ) return MtkPlan(script, preformatter, mtk_chunks(int(c["num_hidden_layers"]), max_chunks), tokenizer) diff --git a/pipeline/manifest.py b/pipeline/manifest.py index 5935517..47e0f95 100644 --- a/pipeline/manifest.py +++ b/pipeline/manifest.py @@ -25,7 +25,7 @@ ) # Attribution the upstream licenses require of redistributed derivatives, keyed by the -# app family token of the source name. +# pipeline family token of the source name (naming.pipeline_family). NOTICES = { # Llama 3.2 Community License, section 1.b.iii. "llama32": ( @@ -34,6 +34,7 @@ ), # Gemma Terms of Use, section 3.1. "gemma3": ("Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms\n"), + "gemma2": ("Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms\n"), } BUILT_WITH = {"llama32": "Built with Llama"} @@ -143,7 +144,7 @@ def readme(repo_id: str, reports: list[dict], hub_tags: list[str], license_files key=lambda r: (list(BACKEND_TITLES).index(r["backend"]), r.get("target") or "", r["window"]["context"]), ) source = reports[0]["source"] - token = naming.app_family(naming.source_name(source["id"])) + token = naming.pipeline_family(naming.source_name(source["id"])) tags = sorted(set(hub_tags) | {r["backend"] for r in reports}) upstream = f"https://huggingface.co/{source['id']}" revisions = sorted({r["source"]["sha"] for r in reports}) @@ -155,13 +156,28 @@ def readme(repo_id: str, reports: list[dict], hub_tags: list[str], license_files if len(revisions) == 1 else "revisions " + ", ".join(f"`{sha[:12]}`" for sha in revisions) + " (see each variant's `source_revision`)" ) + # Only an XNNPACK or Vulkan file runs on any ExecuTorch runtime; an NPU file needs the runner + # its section names (codex review: a MediaTek-only Gemma 3 repo was promised to any runtime, + # and a repo with both kinds promised it for the NPU files too). + portable = any(r["backend"] in ("xnnpack", "vulkan") for r in reports) + npu = any(r["backend"] not in ("xnnpack", "vulkan") for r in reports) + runtime = f"ExecuTorch {reports[0]['toolchain']['executorch']} runtime" + app = "[openweights](https://github.com/alpharomercoma/openweights) Android app" + if portable and npu: + use = ( + f"for on-device inference with the {app}. The CPU (XNNPACK) and GPU (Vulkan) files also run on " + f"any {runtime}; the NPU files need the runner their section below names." + ) + elif portable: + use = f"for on-device inference with the {app} or any {runtime}." + else: + use = "for phone NPUs only: each file needs the runner its section below names, not a generic " + ( + "ExecuTorch runtime." + ) out += [ f"# {naming.source_name(source['id'])} for ExecuTorch", "", - f"ExecuTorch exports of [{source['id']}]({upstream}) ({revision_text}) " - "for on-device inference with the " - "[openweights](https://github.com/alpharomercoma/openweights) Android app or any " - f"ExecuTorch {reports[0]['toolchain']['executorch']} runtime.", + f"ExecuTorch exports of [{source['id']}]({upstream}) ({revision_text}) " + use, "", ] if source.get("variant") == "base": @@ -177,8 +193,12 @@ def readme(repo_id: str, reports: list[dict], hub_tags: list[str], license_files # they run there today. out += [ "The openweights app has no chat template for this model family yet, so it does not " - "load these files until it gains one. Any ExecuTorch runtime with the LLM runner's " - "custom ops runs them today" + "load these files until it gains one. " + + ( + "Any ExecuTorch runtime with the LLM runner's custom ops runs them today" + if portable + else "The runner each backend needs is named below" + ) + ( "." if source.get("variant") == "base" @@ -320,9 +340,28 @@ def readme(repo_id: str, reports: list[dict], hub_tags: list[str], license_files f"{sdk['build']} (mtk_converter {sdk['mtk_converter']}, mtk_neuron {sdk['mtk_neuron']}) " "from MediaTek Inc., used under MediaTek's license terms for that SDK. No MediaTek SDK or " "runtime library is included. They run on MediaTek's LLM runner from ExecuTorch " - "(`examples/mediatek/executor_runner`) with the device's NeuroPilot runtime; the settings it " - "needs are in each folder's `config.json`, under each variant's `runner`.", + "(`examples/mediatek/executor_runner`) with the device's NeuroPilot runtime. That runner takes " + "its settings as command-line flags and does not read `config.json`; the values to pass it are " + "in each folder's `config.json`, under each variant's `runner`.", ] + if any(r["runner"].get("rot_emb_local_base") for r in mtk): + # mediatek-gemma.patch: Gemma 3's graph takes the sliding layers' rotary table as an + # input of its own, which no runner ExecuTorch or the apps ship feeds yet. + local = next(r["runner"] for r in mtk if r["runner"].get("rot_emb_local_base")) + out += [ + "", + "Variants whose `runner` lists `rot_emb_inputs: [global, local]` take two rotary-embedding " + "inputs, in that order after the masks: the global layers' table at `rot_emb_base` " + f"({local['rot_emb_base']:g}) and the sliding layers' at `rot_emb_local_base` " + f"({local['rot_emb_local_base']:g}), with a sliding-window mask of `window_size` " + f"({local['window_size']}) beside the global one. MediaTek's runner as ExecuTorch 1.5.1 ships " + "it builds one table and cannot run them correctly, and neither app feeds the second table " + "yet; a runner has to build both. With one table for every layer this model was measured " + "far from Hugging Face's (perplexity 22.7 against 9.6, fp32); with both it matches exactly.", + ] + limits = sorted({line for r in mtk for line in r.get("limits", [])}) + if limits: + out += ["", *limits] if any(r["runner"].get("state_layout") == "per-layer" for r in mtk): # MediaTek's stock runner counts 2 * num_layer / num_chunk caches per chunk and # refuses these files at load, so the card has to say which runner reads them. diff --git a/pipeline/naming.py b/pipeline/naming.py index 9357533..1bcecb7 100644 --- a/pipeline/naming.py +++ b/pipeline/naming.py @@ -37,8 +37,9 @@ # the Kotlin port: the app reads these names as no family at all and refuses the file # (PromptTemplates.forModel returns null, loudly), so a published file waits for the # template without breaking anything. Each needs its own exporter support first; LFM3 has -# pipeline/lfm3_model.py. -AHEAD_OF_APP = ("lfm3",) +# pipeline/lfm3_model.py, Gemma 2 MediaTek's gemma.py (neither app has a Gemma 2 template), whose +# converter cannot hold Gemma 2 2B's head yet (docs/research, finding 47). +AHEAD_OF_APP = ("lfm3", "gemma2") _KEPT_PUNCTUATION = "-_." _SIZE_HINT = re.compile(r"(? None: raise ValueError(f"{unvouched} would be uploaded but are not in the report; nothing vouches for them") +def shared_file_clashes(report: dict, retained: list[dict]) -> list[tuple[dict, str]]: + """The other windows whose reports list a file this run uploads (MediaTek's embedding + table, one per folder) with different bytes, each with what differs. + + Replacing that file from another source revision would leave those windows' chunks running + on weights their reports never measured, under digests that no longer match (codex review + of #30). A rerun from the same revision writes the same bytes and clashes with nothing. + """ + ours = {entry["path"]: entry.get("sha256") for entry in report["files"]} + clashes = [] + for other in retained: + differ = [ + f"{entry['path']} ({str(entry.get('sha256'))[:12]} in the " + f"{naming.window_label(other['window']['context'])} report, {str(ours[entry['path']])[:12]} in this run's)" + for entry in other.get("files", []) + if entry["path"] in ours and entry.get("sha256") != ours[entry["path"]] + ] + if differ: + clashes.append((other, "; ".join(differ))) + return clashes + + def require_publishable(report: dict) -> None: """Refuse a report that has not passed every check its recipe calls for. @@ -188,11 +210,17 @@ def publish_hf( context: int | None = None, attempts: int = 6, allow_unpublished_twin: bool = False, + replace_shared: bool = False, ) -> str: """Commit this run's window into its backend folder beside the other windows, plus the shared root files; regenerate the folder's config.json and the repo README.md from every report in the repo. + A run whose shared file (MediaTek's embedding) differs from the one the folder's other + windows were built on is refused; with ``replace_shared`` those windows are withdrawn in + the same commit, so the folder never mixes tables, and are republished from this run's + revision afterwards, one run each. + Runs publish on their own, possibly at the same moment as another (other windows, other backends), so the commit is pinned to the revision the README was rendered from and retried on conflict. @@ -259,12 +287,56 @@ def publish_hf( if orphaned: dropped = {id(by_path[p]) for p in orphaned if p in by_path} reports = [r for r in reports if id(r) not in dropped] + others = [r for r in reports if r is not report] + # Only this folder's windows can be withdrawn (its config.json is the one rebuilt here); a + # clash with another backend's report cannot be settled by this run and is refused outright + # (codex review of #30, round four). + foreign = shared_file_clashes(report, [r for r in others if _folder(r["backend"], r.get("target")) != folder]) + if foreign: + raise ValueError( + "this run's files differ from what another backend's reports record: " + + "; ".join(why for _, why in foreign) + + ". Withdrawing windows in this folder cannot settle that." + ) + clashes = shared_file_clashes(report, [r for r in others if _folder(r["backend"], r.get("target")) == folder]) + withdrawn: list[str] = [] + if clashes and not replace_shared: + raise ValueError( + "this run's shared files differ from those other published windows were built on: " + + "; ".join(why for _, why in clashes) + + ". Pass --replace-shared to withdraw those windows in the same commit, then republish " + "each from this run's revision." + ) + if clashes: + gone = {id(other) for other, _ in clashes} + reports = [r for r in reports if id(r) not in gone] + # Only each withdrawn window's own files go: those in this folder that carry its window + # in their name, and its report. A file any kept report (this run's included) still + # lists, an unlabelled shared file, or a path in another folder is never touched, so a + # withdrawal cannot take a file a window that stays depends on (codex review of #30, + # round three). + kept = {entry["path"] for r in reports for entry in r.get("files", [])} + withdrawn = [path for path, other in by_path.items() if id(other) in gone] + for other, _ in clashes: + own = naming.window_label(other["window"]["context"]) + withdrawn += [ + entry["path"] + for entry in other["files"] + if entry["path"].startswith(f"{folder}/") + and "/" not in entry["path"][len(folder) + 1 :] + and window_of(entry["path"].rsplit("/", 1)[-1]) == own + ] + migrated.pop(f"{folder}/{exporting.report_file(other['window']['context'])}", None) + withdrawn = sorted( + {p for p in withdrawn if p in existing and p not in stale and p not in new_paths and p not in kept} + ) folder_reports = [r for r in reports if _folder(r["backend"], r.get("target")) == folder] config = json.dumps(manifest.backend_config(folder_reports), indent=2) + "\n" license_files = sorted({f for r in reports for f in r["source"].get("license_files", [])}) readme = manifest.readme(repo_id, reports, list(cfg.hub_tags), license_files) operations = [CommitOperationDelete(path_in_repo=path) for path in stale if path not in new_paths] + operations += [CommitOperationDelete(path_in_repo=path) for path in withdrawn] operations += [ CommitOperationAdd(path_in_repo=k, path_or_fileobj=str(v)) for k, v in new_paths.items() @@ -275,6 +347,9 @@ def publish_hf( operations += [CommitOperationAdd(path_in_repo=k, path_or_fileobj=str(v)) for k, v in root_files.items()] operations.append(CommitOperationAdd(path_in_repo="README.md", path_or_fileobj=readme.encode("utf-8"))) message = f"{folder} {label}: {report['source']['id']}@{report['source']['sha'][:12]}" + if clashes: + labels = sorted({naming.window_label(other["window"]["context"]) for other, _ in clashes}) + message += f"; withdraws {', '.join(labels)}, built on another shared file" if orphaned: operations += [CommitOperationDelete(path_in_repo=p) for p in orphaned] gpu_reports = [r for r in reports if _folder(r["backend"], r.get("target")) == "vulkan"] diff --git a/tests/conftest.py b/tests/conftest.py index 66691a4..7ebe449 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -24,6 +24,7 @@ "Qwen/Qwen3-30B-A3B": 30_532_122_624, "google/gemma-3-1b-it": 999_885_952, "microsoft/Phi-4-mini-instruct": 3_836_021_760, + "google/gemma-2-2b-it": 2_614_341_888, } CONFIG_FILES = { "Qwen/Qwen3-0.6B": "qwen3-0.6b", @@ -35,6 +36,7 @@ "Qwen/Qwen3-30B-A3B": "qwen3-30b-a3b", "google/gemma-3-1b-it": "gemma-3-1b-it", "microsoft/Phi-4-mini-instruct": "phi-4-mini-instruct", + "google/gemma-2-2b-it": "gemma-2-2b-it", } diff --git a/tests/fixtures/gemma-2-2b-it.config.json b/tests/fixtures/gemma-2-2b-it.config.json new file mode 100644 index 0000000..05131f6 --- /dev/null +++ b/tests/fixtures/gemma-2-2b-it.config.json @@ -0,0 +1,35 @@ +{ + "architectures": [ + "Gemma2ForCausalLM" + ], + "attention_bias": false, + "attention_dropout": 0.0, + "attn_logit_softcapping": 50.0, + "bos_token_id": 2, + "cache_implementation": "hybrid", + "eos_token_id": [ + 1, + 107 + ], + "final_logit_softcapping": 30.0, + "head_dim": 256, + "hidden_act": "gelu_pytorch_tanh", + "hidden_activation": "gelu_pytorch_tanh", + "hidden_size": 2304, + "initializer_range": 0.02, + "intermediate_size": 9216, + "max_position_embeddings": 8192, + "model_type": "gemma2", + "num_attention_heads": 8, + "num_hidden_layers": 26, + "num_key_value_heads": 4, + "pad_token_id": 0, + "query_pre_attn_scalar": 256, + "rms_norm_eps": 1e-06, + "rope_theta": 10000.0, + "sliding_window": 4096, + "torch_dtype": "bfloat16", + "transformers_version": "4.42.4", + "use_cache": true, + "vocab_size": 256000 +} diff --git a/tests/test_eligibility.py b/tests/test_eligibility.py index b5e31cb..0e3f6b3 100644 --- a/tests/test_eligibility.py +++ b/tests/test_eligibility.py @@ -84,13 +84,19 @@ def test_llama_family_covers_llama32_and_smollm2(): assert evaluate("HuggingFaceTB/SmolLM2-360M-Instruct").eligible -def test_gemma3_goes_to_qnn_only(): +def test_gemma3_goes_to_qnn_and_mediatek(): verdict = evaluate("google/gemma-3-1b-it") assert verdict.reasons == [] - assert verdict.export_backends == ["qnn"] + assert verdict.export_backends == ["qnn", "mtk"] assert "export_llm" in verdict.backends["xnnpack"] +def test_gemma2_goes_nowhere_its_head_is_over_mediateks_2gb_converter_cap(source): + verdict = eligibility.evaluate(source("google/gemma-2-2b-it"), settings.load()) + assert (verdict.family, verdict.variant, verdict.export_backends) == ("gemma2", "instruct", []) + assert "2,359,296,000 bytes" in verdict.backends["mtk"] and "2 GB" in verdict.backends["mtk"] + + def test_gated_repo_without_access_says_what_to_do(): verdict = evaluate("google/gemma-3-1b-it", config={}, access_error="gated repo and the HF_TOKEN account ...") assert verdict.reasons[0].startswith("gated repo") diff --git a/tests/test_mtk.py b/tests/test_mtk.py index a8fc36a..b0ad094 100644 --- a/tests/test_mtk.py +++ b/tests/test_mtk.py @@ -1,11 +1,12 @@ import dataclasses import json import subprocess +from pathlib import Path import pytest from conftest import MTK_FLEET, TOTAL_PARAMS, hf_config, load_json -from pipeline import export_mtk, families, manifest, naming, publish, settings +from pipeline import eligibility, export_mtk, families, manifest, naming, publish, settings CFG = settings.load() # The runner choice is tested on a three-size fleet (conftest); the configured tier is one. @@ -485,3 +486,200 @@ def test_the_patches_carry_what_the_export_checks_for(): assert sum(export_mtk.ROPE_MARKER in line for line in rope.splitlines() if line.startswith("+")) == 2 for config in export_mtk.ROPE_CONFIGS.values(): assert f"+++ b/examples/mediatek/models/llm_models/{config}" in rope + gemma = (patches / "mediatek-gemma.patch").read_text(encoding="utf-8") + for relative, marker in export_mtk.GEMMA3_MARKERS.items(): + assert f"+++ b/examples/mediatek/{relative}" in gemma + assert sum(line[1:].strip() == marker for line in gemma.splitlines() if line.startswith("+")) == 1 + + +def test_gemma3_config_is_written_in_mediateks_spelling(): + mtk = families.mtk_model_config(hf_config("google/gemma-3-1b-it")) + assert mtk["model_type"] == "gemma3" # HF says gemma3_text; configuration_gemma reads gemma3 + assert mtk["sliding_window_attention_size"] == 512 # MediaTek's default is 1024 + assert mtk["global_local_attention_pattern"] == ["SLIDING_LOCAL"] * 5 + ["GLOBAL"] + assert mtk["use_res_clamp"] is False # HF clamps no residual; MediaTek's default does + assert families.mtk_model_config(hf_config("Qwen/Qwen3-0.6B")) == hf_config("Qwen/Qwen3-0.6B") + + +def test_a_gemma3_layer_pattern_that_does_not_repeat_is_refused(): + config = {**hf_config("google/gemma-3-1b-it"), "sliding_window_pattern": None} + config["layer_types"] = ["sliding_attention"] * 5 + ["full_attention"] + ["full_attention"] * 20 + with pytest.raises(families.UnsupportedModel, match="one period"): + families.mtk_model_config(config) + + +def test_gemma3_runs_gemma_py_and_its_runner_config_names_both_rope_tables(): + config = hf_config("google/gemma-3-1b-it") + assert families.family_for(config).key == "gemma3" + plan = families.mtk_plan(families.family_for(config), config, CFG.mtk.max_chunks) + assert (plan.script, plan.preformatter, plan.num_chunks) == ("gemma.py", "gemma.json", 2) # 26 layers + three = export_mtk.runner_settings(hf_config("google/gemma-3-1b-it"), CFG.mtk, 2, [1, 106]) + assert (three["rot_emb_base"], three["rot_emb_local_base"], three["window_size"]) == (1e6, 10000.0, 512) + assert three["rot_emb_inputs"] == ["global", "local"] + two = export_mtk.runner_settings(hf_config("google/gemma-2-2b-it"), CFG.mtk, 2, [1, 107]) + assert two["rot_emb_base"] == 10000.0 and "rot_emb_local_base" not in two and "window_size" not in two + + +def test_a_head_over_mtk_converters_2gb_protobuf_is_refused_at_any_chunk_count(): + gemma2 = hf_config("google/gemma-2-2b-it") + assert families.family_for(gemma2).key == "gemma2" + assert families.mtk_head_bytes(gemma2) == 256000 * 2304 * 4 # measured refused, 2026-11-09 + for chunks in (2, 13, 26): + with pytest.raises(families.UnsupportedModel, match="2 GB"): + families.mtk_plan(families.family_for(gemma2), gemma2, chunks) + assert families.mtk_head_bytes(hf_config("google/gemma-3-1b-it")) == 262144 * 1152 * 4 # 1.2 GB, built + + +def gemma_report(model, window, **runner): + report = mtk_report() + name = model.rsplit("/", 1)[-1] + report["source"] = {"id": model, "sha": "0" * 12, "license": {"license": "gemma"}} + report["output_repo"] = f"experimentalmachines/{name}-ExecuTorch" + report["window"] = {"context": window, "kv_cache_bytes_per_token": None} + report["runner"] = {**report["runner"], **runner} + limits = families.mtk_limits(hf_config(model), window) + if limits: + report["limits"] = limits + return report + + +def test_a_mediatek_only_family_ahead_of_the_app_is_not_promised_to_any_runtime(): + text = manifest.readme( + "experimentalmachines/gemma-2-2b-it-ExecuTorch", [gemma_report("google/gemma-2-2b-it", 2048)], [], [] + ) + assert "no chat template for this model family yet" in text + assert "Any ExecuTorch runtime" not in text # the MediaTek files need MediaTek's runner + assert "or any ExecuTorch" not in text # nor in the opening sentence + + +def test_an_npu_only_card_promises_no_generic_runtime_and_a_portable_one_does(): + gemma3 = [gemma_report("google/gemma-3-1b-it", 512)] + opening = manifest.readme("experimentalmachines/gemma-3-1b-it-ExecuTorch", gemma3, [], []) + assert "for phone NPUs only" in opening and "or any ExecuTorch" not in opening + assert "does not read `config.json`" in opening # MediaTek's runner takes flags + # With CPU or GPU files beside the NPU ones, only those are promised to any runtime. + mixed = [*gemma3, {**gemma3[0], "backend": "xnnpack"}] + text = manifest.readme("experimentalmachines/gemma-3-1b-it-ExecuTorch", mixed, [], []) + assert "The CPU (XNNPACK) and GPU (Vulkan) files also run on any ExecuTorch" in text + assert "the NPU files need the runner their section below names" in text + assert "Android app or any ExecuTorch" not in text + + +def test_gemma2_past_its_sliding_window_says_what_it_computes_differently(): + assert families.mtk_limits(hf_config("google/gemma-2-2b-it"), 4096) == [] + reports = [gemma_report("google/gemma-2-2b-it", 4096), gemma_report("google/gemma-2-2b-it", 8192)] + text = manifest.readme("experimentalmachines/gemma-2-2b-it-ExecuTorch", reports, [], []) + assert text.count("over the whole 8,192-token window in MediaTek's graph") == 1 + assert "prompts up to 4,096 tokens are computed as Hugging Face computes them" in text + + +def test_the_gemma3_card_says_its_runner_must_feed_both_rope_tables(): + runner = export_mtk.runner_settings(hf_config("google/gemma-3-1b-it"), CFG.mtk, 2, [1, 106]) + report = gemma_report( + "google/gemma-3-1b-it", + 2048, + **{k: runner[k] for k in ("rot_emb_base", "rot_emb_local_base", "window_size", "rot_emb_inputs")}, + ) + text = manifest.readme("experimentalmachines/gemma-3-1b-it-ExecuTorch", [report], [], []) + assert "`rot_emb_inputs: [global, local]`" in text + assert "rot_emb_local_base` (10000)" in text and "`window_size` (512)" in text + assert "cannot run them correctly" in text + plain = manifest.readme("experimentalmachines/Qwen3-0.6B-ExecuTorch", [mtk_report()], [], []) + assert "rot_emb_inputs" not in plain + + +def test_each_run_builds_in_its_own_copy_of_the_scripts(tmp_path): + # Two windows of one model on one host shared pte/, which has no window in its name, + # and the first run to finish deleted the other's chunk (2026-11-09). + examples = tmp_path / "examples" / "mediatek" + (examples / "model_export_scripts").mkdir(parents=True) + (examples / "model_export_scripts" / "gemma.py").write_text("# script\n") + (examples / "pte" / "gemma-3-1b-it_A16W8_2_chunks").mkdir(parents=True) + (examples / "pte" / "gemma-3-1b-it_A16W8_2_chunks" / "stale_0.pte").write_bytes(b"x") + (examples / "model_export_scripts" / "__pycache__").mkdir() + runs = [export_mtk.private_examples(examples, tmp_path / f"work-{w}") for w in (8192, 16384)] + assert runs[0] != runs[1] and all(r.is_absolute() for r in runs) + for run in runs: + assert (run / "model_export_scripts" / "gemma.py").read_text() == "# script\n" + assert not (run / "pte").exists() and not (run / "model_export_scripts" / "__pycache__").exists() + (runs[0] / "leftover").write_text("from a crashed run") + again = export_mtk.private_examples(examples, tmp_path / "work-8192") + assert again == runs[0] and not (again / "leftover").exists() + assert (examples / "pte" / "gemma-3-1b-it_A16W8_2_chunks" / "stale_0.pte").exists() # never touched + + +def test_gemma3_is_refused_on_scripts_without_the_two_table_patch(tmp_path, monkeypatch, source): + examples = tmp_path / "mediatek" + (examples / "model_export_scripts").mkdir(parents=True) + (examples / "models" / "llm_models").mkdir(parents=True) + calibrated = f"{export_mtk.PATCH_MARKER}\n{export_mtk.WRITER_MARKER}\n" + (examples / "model_export_scripts" / "gemma.py").write_text(calibrated) + (examples / "models" / "llm_models" / "modeling_gemma.py").write_text("def forward(self, *args): ...\n") + monkeypatch.setattr(export_mtk.hub, "fetch", lambda model_id, revision: source(model_id)) + with pytest.raises(export_mtk.ExportError, match="modeling_gemma.py lacks the patch.*mediatek-gemma.patch"): + export_mtk.run("google/gemma-3-1b-it", "main", "MT6991", tmp_path, tmp_path, "p", examples) + + +@pytest.mark.parametrize( + ("change", "reason"), + [ + ({"layer_types": ["sliding_attention"] * 26, "sliding_window_pattern": None}, "no full_attention"), + ({"layer_types": ["chunked_attention"] * 26, "sliding_window_pattern": None}, "no MediaTek layer"), + ({"sliding_window": None}, "no sliding_window"), + ], +) +def test_a_gemma3_config_mediatek_cannot_translate_is_a_verdict_not_a_crash(change, reason, source): + # Anything but UnsupportedModel would end the watcher's whole pass (codex review of #30). + config = {**hf_config("google/gemma-3-1b-it"), **change} + with pytest.raises(families.UnsupportedModel, match=reason): + families.mtk_model_config(config) + verdict = eligibility.evaluate(source("google/gemma-3-1b-it", config), CFG) + assert reason in verdict.backends["mtk"] + + +def test_the_script_gets_absolute_paths_though_it_runs_in_the_runs_own_copy(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + tool, config = export_mtk.script_paths("venv/bin/python", Path("work/weights/m")) + assert tool == str(tmp_path / "venv/bin/python") and config == tmp_path / "work/weights/m/config.json" + assert export_mtk.script_paths("python3", Path("w"))[0] == "python3" # found on PATH, left alone + assert export_mtk.absolute_tool("venv/bin/python") == str(tmp_path / "venv/bin/python") + + +def test_run_makes_its_paths_absolute_before_anything_else(tmp_path, monkeypatch): + # pipeline.mtk_corpus runs from the repo before the script's paths were resolved, so a + # relative --work broke LFM2's streaming calibration (codex review of #30, round two). + monkeypatch.chdir(tmp_path) + seen = [] + + class Stop(Exception): + pass + + def spy(*args): + seen.append(args) + raise Stop + + original = export_mtk.absolute_inputs + monkeypatch.setattr(export_mtk, "absolute_inputs", spy) + with pytest.raises(Stop): + export_mtk.run("Qwen/Qwen3-0.6B", "main", "MT6991", Path("out"), Path("work"), "venv/bin/python", Path("ex")) + assert seen == [(Path("out"), Path("work"), Path("ex"), "venv/bin/python")] + assert original(Path("out"), Path("work"), Path("ex"), "venv/bin/python") == ( + tmp_path / "out", + tmp_path / "work", + tmp_path / "ex", + str(tmp_path / "venv/bin/python"), + ) + + +def test_the_runs_copy_never_overlaps_the_scripts_it_copies(tmp_path): + # --examples /mediatek made the copy the source itself: rmtree took the scripts + # (codex review of #30, round five). + work = tmp_path / "work" + scripts = work / "mediatek" + (scripts / "model_export_scripts").mkdir(parents=True) + (scripts / "model_export_scripts" / "gemma.py").write_text("# script\n") + with pytest.raises(export_mtk.ExportError, match="overlaps the run's own copy"): + export_mtk.private_examples(scripts, work) + assert (scripts / "model_export_scripts" / "gemma.py").exists() # untouched + with pytest.raises(export_mtk.ExportError, match="overlaps"): + export_mtk.private_examples(tmp_path, work) # the copy would land inside the scripts diff --git a/tests/test_naming.py b/tests/test_naming.py index 176f0e2..a87d4fd 100644 --- a/tests/test_naming.py +++ b/tests/test_naming.py @@ -115,3 +115,12 @@ def test_phi4_mini_is_named_with_its_size_and_reads_as_phi4mini_in_the_app(): assert naming.check_app_rules(repo, path, backend) == [] assert naming.app_family(naming.app_model_name(repo, path)) == "phi4mini" assert naming.app_size_hint(repo.split("/")[1]) == "3.8B" + + +def test_gemma2_is_exported_ahead_of_the_apps_with_its_notice(): + from pipeline import manifest + + assert naming.app_family("gemma-2-2b-it") is None # neither app has a Gemma 2 template + assert naming.pipeline_family("gemma-2-2b-it") == "gemma2" + assert naming.ahead_of_app("gemma-2-2b-it") + assert "Gemma Terms of Use" in manifest.NOTICES["gemma2"] diff --git a/tests/test_publish.py b/tests/test_publish.py index 594f0c3..dd1b4cc 100644 --- a/tests/test_publish.py +++ b/tests/test_publish.py @@ -547,3 +547,179 @@ def test_an_unreadable_gpu_release_is_an_error_not_an_absent_one(monkeypatch): monkeypatch.setattr(publish.subprocess, "run", FakeGh("api-error").run) with pytest.raises(RuntimeError, match="gh release view"): publish.withdraw_orphaned_gpu_release(report(4096)) + + +def mtk_window(context, embedding_sha): + r = report(context, backend="mtk", target="mt6991") + label = f"{context // 1024}k" + r["files"] = [ + { + "path": f"mtk/mt6991/Qwen3-1.7B-neuropilot-a16w4-{label}-chunk1of1.pte", + "bytes": 3, + "sha256": r["files"][0]["sha256"], + }, + {"path": "mtk/mt6991/Qwen3-1.7B-neuropilot-embedding-fp32.bin", "bytes": 3, "sha256": embedding_sha}, + ] + # What a MediaTek report carries for config.json and the card (export_mtk.run). + r["runner"] = {"token_embedding_path": "Qwen3-1.7B-neuropilot-embedding-fp32.bin"} + r["neuropilot"] = {"build": "8.0.8", "mtk_converter": "8.13.0", "mtk_neuron": "8.2.23"} + return r + + +def test_a_shared_file_must_match_what_the_other_windows_reports_describe(): + same, other = hashlib.sha256(b"emb").hexdigest(), hashlib.sha256(b"newer weights").hexdigest() + assert publish.shared_file_clashes(mtk_window(16384, same), [mtk_window(8192, same)]) == [] # one revision + [(clash, why)] = publish.shared_file_clashes(mtk_window(16384, other), [mtk_window(8192, same)]) + assert clash["window"]["context"] == 8192 and "embedding-fp32.bin" in why and "8k report" in why + + +def mtk_repo(tmp_path, monkeypatch): + """A MediaTek folder holding an 8k window on embedding A, and a 16k export on embedding B.""" + out = tmp_path / "out" + (out / "mtk" / "mt6991").mkdir(parents=True) + (out / "tokenizer.json").write_text("{}") + new = mtk_window(16384, hashlib.sha256(b"newer weights").hexdigest()) + (out / "mtk/mt6991/Qwen3-1.7B-neuropilot-a16w4-16k-chunk1of1.pte").write_bytes(b"pte") + (out / "mtk/mt6991/Qwen3-1.7B-neuropilot-embedding-fp32.bin").write_bytes(b"newer weights") + (out / "mtk/mt6991/export-report-16k.json").write_text(json.dumps(new)) + (out / "mtk/mt6991/config.json").write_text(json.dumps(manifest.backend_config(new))) # as export_mtk writes it + remote = { + "README.md": None, + "mtk/mt6991/Qwen3-1.7B-neuropilot-a16w4-8k-chunk1of1.pte": None, + "mtk/mt6991/Qwen3-1.7B-neuropilot-embedding-fp32.bin": None, + "mtk/mt6991/export-report-8k.json": mtk_window(8192, hashlib.sha256(b"emb").hexdigest()), + "mtk/mt6991/config.json": None, + } + hub = FakeHub(remote) + monkeypatch.setattr(publish.hub, "api", lambda: hub) + + def fake_download(repo_id, path, revision, token): + local = tmp_path / "dl" / path + local.parent.mkdir(parents=True, exist_ok=True) + local.write_text(json.dumps(remote[path])) + return str(local) + + import huggingface_hub + + monkeypatch.setattr(huggingface_hub, "hf_hub_download", fake_download) + return out, hub + + +def test_publish_hf_refuses_to_swap_the_embedding_other_mediatek_windows_run_on(tmp_path, monkeypatch): + # Publishing a 16k export built from a newer revision would have replaced the folder's one + # embedding table under the published 8k chunks (codex review of #30). + out, hub = mtk_repo(tmp_path, monkeypatch) + with pytest.raises(ValueError, match="Pass --replace-shared"): + publish.publish_hf(out, "mtk", "mt6991") + assert hub.commits == [] + + +def test_replace_shared_withdraws_the_windows_built_on_the_old_table_in_the_same_commit(tmp_path, monkeypatch): + # Without a way through, a revision change could never be published: every window would be + # refused for the others still on the old table (codex review of #30, round two). + out, hub = mtk_repo(tmp_path, monkeypatch) + publish.publish_hf(out, "mtk", "mt6991", replace_shared=True) + [(operations, message, _)] = hub.commits + deleted = sorted(op.path_in_repo for op in operations if type(op).__name__ == "CommitOperationDelete") + added = {op.path_in_repo: op for op in operations if type(op).__name__ == "CommitOperationAdd"} + assert deleted == ["mtk/mt6991/Qwen3-1.7B-neuropilot-a16w4-8k-chunk1of1.pte", "mtk/mt6991/export-report-8k.json"] + assert "mtk/mt6991/Qwen3-1.7B-neuropilot-embedding-fp32.bin" in added # the new table, not deleted + config = json.loads(added["mtk/mt6991/config.json"].path_or_fileobj) + assert [v["context"] for v in config["variants"]] == [16384] + assert "withdraws 8k" in message + + +def test_replace_shared_takes_only_the_withdrawn_windows_own_files(tmp_path, monkeypatch): + # The withdrawal must not take a file a kept window still lists, an unlabelled shared file, or + # a path in another folder; an old pre-matrix report on the old table goes with its window + # (codex review of #30, round three). + new_emb, old_emb = hashlib.sha256(b"newer weights").hexdigest(), hashlib.sha256(b"emb").hexdigest() + aux = {"path": "mtk/mt6991/aux-shared.bin", "bytes": 3, "sha256": "a" * 64} + out = tmp_path / "out" + (out / "mtk" / "mt6991").mkdir(parents=True) + (out / "tokenizer.json").write_text("{}") + new = mtk_window(16384, new_emb) + (out / "mtk/mt6991/Qwen3-1.7B-neuropilot-a16w4-16k-chunk1of1.pte").write_bytes(b"pte") + (out / "mtk/mt6991/Qwen3-1.7B-neuropilot-embedding-fp32.bin").write_bytes(b"newer weights") + (out / "mtk/mt6991/export-report-16k.json").write_text(json.dumps(new)) + (out / "mtk/mt6991/config.json").write_text(json.dumps(manifest.backend_config(new))) + clashing = mtk_window(8192, old_emb) + clashing["files"] += [aux, {"path": "xnnpack/Qwen3-1.7B-8da4w-8k.pte", "bytes": 3, "sha256": "b" * 64}] + kept = mtk_window(4096, new_emb) # already republished from the new revision + # The kept window also lists one of the withdrawn 2k window's labelled files: it must stay. + kept["files"] += [ + aux, + {"path": "mtk/mt6991/Qwen3-1.7B-neuropilot-a16w4-2k-chunk1of1.pte", "bytes": 3, "sha256": "c" * 64}, + ] + pre_matrix = mtk_window(2048, old_emb) + remote = { + "README.md": None, + "mtk/mt6991/Qwen3-1.7B-neuropilot-embedding-fp32.bin": None, + "mtk/mt6991/aux-shared.bin": None, + "mtk/mt6991/config.json": None, + "mtk/mt6991/Qwen3-1.7B-neuropilot-a16w4-8k-chunk1of1.pte": None, + "mtk/mt6991/export-report-8k.json": clashing, + "mtk/mt6991/Qwen3-1.7B-neuropilot-a16w4-4k-chunk1of1.pte": None, + "mtk/mt6991/export-report-4k.json": kept, + "mtk/mt6991/Qwen3-1.7B-neuropilot-a16w4-2k-chunk1of1.pte": None, + "mtk/mt6991/export-report.json": pre_matrix, + "xnnpack/Qwen3-1.7B-8da4w-8k.pte": None, + } + hub = FakeHub(remote) + monkeypatch.setattr(publish.hub, "api", lambda: hub) + + def fake_download(repo_id, path, revision, token): + local = tmp_path / "dl" / path + local.parent.mkdir(parents=True, exist_ok=True) + local.write_text(json.dumps(remote[path])) + return str(local) + + import huggingface_hub + + monkeypatch.setattr(huggingface_hub, "hf_hub_download", fake_download) + with pytest.raises(ValueError, match="Pass --replace-shared"): + publish.publish_hf(out, "mtk", "mt6991") + publish.publish_hf(out, "mtk", "mt6991", replace_shared=True) + [(operations, message, _)] = hub.commits + deleted = sorted(op.path_in_repo for op in operations if type(op).__name__ == "CommitOperationDelete") + added = {op.path_in_repo: op for op in operations if type(op).__name__ == "CommitOperationAdd"} + assert deleted == [ + "mtk/mt6991/Qwen3-1.7B-neuropilot-a16w4-8k-chunk1of1.pte", + "mtk/mt6991/export-report-8k.json", + "mtk/mt6991/export-report.json", # the pre-matrix report, not migrated to a 2k name + ] + assert "mtk/mt6991/export-report-2k.json" not in added + config = json.loads(added["mtk/mt6991/config.json"].path_or_fileobj) + assert [v["context"] for v in config["variants"]] == [4096, 16384] + readme = added["README.md"].path_or_fileobj.decode() + # The withdrawn 8k window is gone from the card; the 2k chunk stays listed, by the 4k report. + assert "at 4k to 16k" in readme and "-8k-chunk" not in readme + assert "withdraws 2k, 8k" in message + + +def test_a_clash_with_another_backends_report_is_refused_even_with_replace_shared(tmp_path, monkeypatch): + # Withdrawing windows here cannot settle a file another backend's report vouches for, and its + # folder's config.json would not be rebuilt (codex review of #30, round four). + out, hub = mtk_repo(tmp_path, monkeypatch) + import huggingface_hub + + remote = { + "README.md": None, + "xnnpack/Qwen3-1.7B-8da4w-8k.pte": None, + "xnnpack/export-report-8k.json": { + **report(8192), + "files": [{"path": "mtk/mt6991/Qwen3-1.7B-neuropilot-embedding-fp32.bin", "bytes": 3, "sha256": "d" * 64}], + }, + } + hub.files = remote + + def fake_download(repo_id, path, revision, token): + local = tmp_path / "dl2" / path + local.parent.mkdir(parents=True, exist_ok=True) + local.write_text(json.dumps(remote[path])) + return str(local) + + monkeypatch.setattr(huggingface_hub, "hf_hub_download", fake_download) + with pytest.raises(ValueError, match="another backend's reports"): + publish.publish_hf(out, "mtk", "mt6991", replace_shared=True) + assert hub.commits == [] diff --git a/third_party/executorch/patches/mediatek-gemma.patch b/third_party/executorch/patches/mediatek-gemma.patch new file mode 100644 index 0000000..cd54455 --- /dev/null +++ b/third_party/executorch/patches/mediatek-gemma.patch @@ -0,0 +1,232 @@ +diff --git a/examples/mediatek/model_export_scripts/gemma.py b/examples/mediatek/model_export_scripts/gemma.py +index ee3a4df..95548dd 100644 +--- a/examples/mediatek/model_export_scripts/gemma.py ++++ b/examples/mediatek/model_export_scripts/gemma.py +@@ -59,6 +59,49 @@ from tqdm import tqdm + warnings.filterwarnings("ignore") + + ++def _retry_compile_service_start(): ++ """execupack: MediaTek's compiler client starts a local service on a random port per ++ compile and polls it over HTTP; starting it fails now and then, after hours of work (the ++ same retry llama.py and qwen.py carry, mediatek-calibration-as-arrays.patch). The compile ++ is a pure function of its input, so it is asked again; any error that is not about the ++ service, or a third failure, still stops the export.""" ++ import time as _time ++ ++ try: ++ import mtk_neuron ++ except ImportError: ++ return ++ compile_once = getattr(mtk_neuron, "compile", None) ++ if compile_once is None or getattr(compile_once, "retries_service_start", False): ++ return ++ ++ def compile_with_retry(*args, **kwargs): ++ for attempt in range(3): ++ try: ++ return compile_once(*args, **kwargs) ++ except RuntimeError as error: ++ if "service" not in str(error) or attempt == 2: ++ raise ++ print(f"mtk_neuron: {error}; asking again ({attempt + 2} of 3)", flush=True) ++ _time.sleep(10) ++ ++ compile_with_retry.retries_service_start = True ++ mtk_neuron.compile = compile_with_retry ++ ++ ++_retry_compile_service_start() ++ ++ ++def local_rot_emb(config): ++ """execupack: Gemma 3's sliding layers' RoPE table, at rope_local_base_freq. get_master_rot_emb ++ builds at config.rope_theta (mediatek-rope-theta.patch), Gemma 3's global base.""" ++ import copy ++ ++ local = copy.copy(config) ++ local.rope_theta = config.rope_local_base_freq ++ return get_master_rot_emb(local, dtype=torch.float32) ++ ++ + def get_argument_parser(): + parser = argparse.ArgumentParser( + description="Run Export to ET for suppoorted LLM models.", allow_abbrev=False +@@ -207,28 +250,22 @@ def forward_and_save( + model_input_dict, + num_blocks_per_chunk, + batch_name, ++ pos_emb_local=None, + ): + for chunk_idx in range(len(models)): + cache_in = cache[chunk_idx] ++ saved = {"hidden_state": hidden_state, "mask": mask, "pos_emb": pos_emb, "cache": cache_in} ++ if pos_emb_local is not None: ++ saved["pos_emb_local"] = pos_emb_local # execupack: Gemma 3's sliding table + + try: + model_input_dict[str(chunk_idx)] = { + **model_input_dict[str(chunk_idx)], +- batch_name: { +- "hidden_state": hidden_state, +- "mask": mask, +- "pos_emb": pos_emb, +- "cache": cache_in, +- }, ++ batch_name: saved, + } + except: + model_input_dict[str(chunk_idx)] = { +- batch_name: { +- "hidden_state": hidden_state, +- "mask": mask, +- "pos_emb": pos_emb, +- "cache": cache_in, +- } ++ batch_name: saved, + } + + if isinstance(mask, dict): +@@ -240,6 +277,7 @@ def forward_and_save( + global_mask, + local_mask, + pos_emb, ++ pos_emb_local, + *torch.split(cache_in, 1, dim=0), + ) + else: +@@ -267,6 +305,7 @@ def prepare_model_inputs( + eos_token_id_tensor, + response_cap, + window_size, ++ master_rot_emb_local=None, + ): + model_input_dict = {str(i): None for i in range(len(models))} + input_ids = inp.pop("input_ids") +@@ -287,6 +326,7 @@ def prepare_model_inputs( + else: + mask = global_mask + pos_emb = master_rot_emb[:, :, :input_length, :] ++ pos_emb_local = None if master_rot_emb_local is None else master_rot_emb_local[:, :, :input_length, :] + # cache shape: num chunks of 2*num_block, num kv heads, c, head dim + cache = reset_cache( + len(models), num_key_value_heads, num_blocks_per_chunk, head_dim, max_cache_size +@@ -300,6 +340,7 @@ def prepare_model_inputs( + model_input_dict, + num_blocks_per_chunk, + "prompt", ++ pos_emb_local=pos_emb_local, + ) + next_token_logits = logits[:, -1, :] # last layer logits + next_token = torch.argmax(next_token_logits, dim=-1) +@@ -325,6 +366,11 @@ def prepare_model_inputs( + else: + mask = global_mask + pos_emb = master_rot_emb[:, :, seq_length : seq_length + input_length, :] ++ pos_emb_local = ( ++ None ++ if master_rot_emb_local is None ++ else master_rot_emb_local[:, :, seq_length : seq_length + input_length, :] ++ ) + logits, cache = forward_and_save( + models, + hidden_state, +@@ -334,6 +380,7 @@ def prepare_model_inputs( + model_input_dict, + num_blocks_per_chunk, + f"response{response_count}", ++ pos_emb_local=pos_emb_local, + ) + next_token_logits = logits[:, -1, :] + next_token = torch.argmax(next_token_logits, dim=-1) +@@ -362,11 +409,13 @@ def calibrate_model(model, cal_dataset, chunk_idx: str): + if isinstance(mask, dict): + global_mask = torch.tensor(mask["GLOBAL"]) + local_mask = torch.tensor(mask["SLIDING_LOCAL"]) ++ pos_emb_local = torch.tensor(inp[chunk_idx][batch]["pos_emb_local"]) + model( + inputs_embeds, + global_mask, + local_mask, + pos_emb, ++ pos_emb_local, + *torch.split(cache, 1, dim=0), + ) + else: +@@ -545,6 +594,7 @@ def main(): + tokenize_dataset, fn_kwargs={"tokenizer": tokenizer} + ) + window_size = getattr(config, "sliding_window_attention_size", None) ++ master_rot_emb_local = local_rot_emb(config) if config.model_type == "gemma3" else None + print("Preparing Model Calibration Inputs...") + cal_dataset = cal_dataset.map( + prepare_model_inputs, +@@ -559,8 +609,15 @@ def main(): + "eos_token_id_tensor": torch.tensor(tokenizer.eos_token_id), + "response_cap": args.response_cap, + "window_size": window_size, ++ "master_rot_emb_local": master_rot_emb_local, + }, ++ # execupack: each row is one prompt's every step, each a full fp32 KV cache at the ++ # window; the default writer_batch_size of 1000 kept all of them resident before ++ # any reached Arrow. One row at a time is what export_mtk.calibration_bytes sizes. ++ writer_batch_size=1, + ) ++ # execupack: read the prepared tensors back as arrays rather than nested Python lists. ++ cal_dataset = cal_dataset.with_format("numpy") + + for chunk_idx, chunk in enumerate(models): + export_to_et_ir( +diff --git a/examples/mediatek/models/llm_models/modeling_gemma.py b/examples/mediatek/models/llm_models/modeling_gemma.py +index e1ee4d7..7b55617 100644 +--- a/examples/mediatek/models/llm_models/modeling_gemma.py ++++ b/examples/mediatek/models/llm_models/modeling_gemma.py +@@ -1022,7 +1022,11 @@ class Gemma3ModelChunk(ModelChunk): + self.sliding_window_pattern = self.config.sliding_window_pattern + self.global_local_attention_pattern = self.config.global_local_attention_pattern + +- def forward(self, inputs_embeds, global_mask, local_mask, pos_emb, *cache): ++ def forward(self, inputs_embeds, global_mask, local_mask, pos_emb, pos_emb_local, *cache): ++ # execupack: Gemma 3 rotates its sliding layers at rope_local_base_freq (10000) and its ++ # global layers at rope_theta (1e6). One table for every layer was measured against ++ # Hugging Face in fp32: perplexity 22.7 against 9.6 at 1e6, top-1 52%; per-layer tables ++ # match exactly. pos_emb is the global table, pos_emb_local the sliding one. + if not self.jit_trace: + assert ( + len(cache) == 2 * self.num_blocks +@@ -1058,15 +1062,17 @@ class Gemma3ModelChunk(ModelChunk): + mask_type = self.global_local_attention_pattern[mask_pos] + if mask_type == "SLIDING_LOCAL": + mask = local_mask ++ layer_pos_emb = pos_emb_local + elif mask_type == "GLOBAL": + mask = global_mask ++ layer_pos_emb = pos_emb + else: + raise TypeError(f"Unrecognized mask type: {mask_type}") + + decoder_outputs = decoder_layer( + hidden_states.to(self.device_list[idx]), + mask=mask.to(self.device_list[idx]), +- pos_emb=pos_emb.to(self.device_list[idx]), ++ pos_emb=layer_pos_emb.to(self.device_list[idx]), + past_key=cache[idx].to(self.device_list[idx]), + past_value=cache[self.num_blocks + idx].to(self.device_list[idx]), + ) +@@ -1540,6 +1546,8 @@ class Gemma3ModelChunk(ModelChunk): + dtype=torch.float32, + ), + torch.randn(1, 2, num_token, head_dim, device="cpu", dtype=torch.float32), ++ # execupack: the sliding layers' table (forward's pos_emb_local). ++ torch.randn(1, 2, num_token, head_dim, device="cpu", dtype=torch.float32), + *[ + torch.randn( + 1, +@@ -1563,6 +1571,7 @@ class Gemma3ModelChunk(ModelChunk): + {0: Dim.STATIC, 1: Dim.STATIC, 2: nt, 3: nt + cache_size}, + {0: Dim.STATIC, 1: Dim.STATIC, 2: nt, 3: nt + cache_size}, + {0: Dim.STATIC, 1: Dim.STATIC, 2: nt, 3: Dim.STATIC}, ++ {0: Dim.STATIC, 1: Dim.STATIC, 2: nt, 3: Dim.STATIC}, + cache_dims, + ) + return example_inputs, dynamic_shapes