Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
0163034
Gemma 3 and Gemma 2 on MediaTek: MediaTek's config spelling, a second…
alpharomercoma Nov 9, 2026
123ce37
MediaTek cards: Gemma 3's runner must feed both RoPE tables, Gemma 2 …
alpharomercoma Nov 9, 2026
fa7caf0
MediaTek: refuse a head over mtk_converter's 2 GB protobuf, because G…
alpharomercoma Nov 9, 2026
bdf1f44
MediaTek cards: an NPU-only repo promises no generic runtime, because…
alpharomercoma Nov 9, 2026
1800b41
MediaTek export: each run builds in its own copy of the scripts, beca…
alpharomercoma Nov 9, 2026
0f165cc
PLAN: MediaTek runs build in their own copy of the scripts, because t…
alpharomercoma Nov 9, 2026
242f7d8
MediaTek export: refuse Gemma 3 on scripts without the two-table patc…
alpharomercoma Nov 9, 2026
8c40d93
PLAN: Gemma 3 1B is published for MT6991 at 512 to 32k, with the 32k …
alpharomercoma Nov 9, 2026
09efe33
PLAN: Gemma 3 1B is published for QNN SM8750 at 2k to 32k, with the r…
alpharomercoma Nov 9, 2026
4e90c1e
Codex review of #30: a malformed Gemma 3 config is a verdict, a windo…
alpharomercoma Nov 9, 2026
0eee9a5
Codex review of #30, round two: --replace-shared withdraws the window…
alpharomercoma Nov 9, 2026
38efcd2
Codex review of #30, round three: a withdrawal takes only the withdra…
alpharomercoma Nov 9, 2026
de80840
Codex review of #30, round four: only this folder's windows can be wi…
alpharomercoma Nov 9, 2026
e083471
Codex review of #30, round five: the run's copy of the scripts may no…
alpharomercoma Nov 9, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 40 additions & 6 deletions docs/PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -248,6 +248,18 @@ calling ExecuTorch's own script in compile-only mode:
- The window is fixed at compile time: `qnn.max_context_len` (2048 to start), or the
workflow's `context` input (`--context`) for one run.
- QAIRT's license: `THIRD_PARTY_NOTICES.md`.
- **Gemma 3 1B** (2026-11-09): gemma-3-1b-it (`gemma3-1b`, dcc83ea8) is published for SM8750 at
2k to 32k, the first 32k QNN build here, one 1.15-1.21 GB `.pte` per window, run on EC2 rather
than Modal: 2k 1,087 s and 4k 1,070 s (peak RSS 49.5 and 46.3 GiB), 8k 2,202 s (47.9 GiB),
16k 3,263 s (77.9 GiB) on r7i.4xlarge (16 vCPU), 32k 4,598 s on r7i.8xlarge (32 vCPU) with a
146.9 GiB peak, past the 83 GiB `qnn-matrix` asks Modal for and past a 128 GiB host. A fresh
Ubuntu 24.04 image lacks `libatomic1`, which the QAIRT SDK's bundled libc++ needs; without it
`libQnnHtp.so` fails to load and the wheel reports only "Failed to set up QNN SDK".
- Qwen2.5-0.5B/1.5B/3B-Instruct and Qwen2.5-Math-1.5B-Instruct are not in the registry (it has
the base 0.5B and 1.5B only, and no 3B). The script takes `--checkpoint`, so their weights could
ride the base entries' graph and recipe, but those recipes are post-training int4 with no host
check, like the round-to-nearest int4 that failed the gate for Qwen2.5-1.5B on XNNPACK; held
back on 2026-11-09 by the operator's choice.

### MediaTek (phase 4)

Expand All @@ -269,7 +281,10 @@ calling ExecuTorch's own script in compile-only mode:
(`export_mtk.tool_env`): the wheel's `libextract_shared.so` needs the `libc++.so.1`
bundled beside it but has no RPATH (docs/research, finding 21).
- **Scripts:** the LLM export scripts are in ExecuTorch's source, not the wheel; the
workflow sparse-checks-out `examples/mediatek` at `EXECUTORCH_COMMIT`.
workflow sparse-checks-out `examples/mediatek` at `EXECUTORCH_COMMIT`. Each run executes
them in its own copy (31 MB) in its work folder (`export_mtk.private_examples`): they write
chunks to `pte/<weights>_<precision>_<n>_chunks`, a name without the window, and two windows
of Gemma 3 on one EC2 box (2026-11-09) shared it until the first to finish deleted it.
- **Recipe** (MediaTek's own, `shell_scripts/export_qwen.sh`): A16W4, the model cut into up
to 4 chunks of equal layer counts, a 128-token prompt graph and a one-token generation
graph over a 512-token cache, calibrated on MediaTek's `alpaca.txt` prompts (9, up to 9
Expand All @@ -285,7 +300,9 @@ calling ExecuTorch's own script in compile-only mode:
- **Families:** the scripts build the model from `config.json`'s `model_type`, so any size
works, not a fixed list. Qwen3 (`qwen.py`), Llama 3.2 and SmolLM2 (`llama.py`, with
`"tokenizer": "pretrained_fast"` written into the config, SmolLM2 calibrating in ChatML)
are exported; Gemma 3 (`gemma3_text` vs `gemma3`) waits for validation.
are exported, and Gemma 3 (`gemma.py`, with `families.mtk_model_config` writing MediaTek's
spelling of the config: model_type `gemma3`, `sliding_window_attention_size`,
`global_local_attention_pattern`, `use_res_clamp` false).
- **Stock RoPE was wrong** (2026-10-04, finding 37): `get_master_rot_emb` used base 10000
because neither `QwenConfig` nor `LlamaConfig` kept `rope_theta`, and it had no Llama 3
frequency scaling. In the fp32 graph, before quantization, Qwen3-0.6B was at KL 0.082 and
Expand All @@ -302,9 +319,26 @@ calling ExecuTorch's own script in compile-only mode:
full fp32 cache at the window, so Qwen3 at 32k (about 285 GB for the 0.6B), Qwen3-4B from
16k and Llama-3.2-3B at 32k fit no hosted runner and are left out of the matrix; the
streaming calibration `lfm2.py` uses would lift that.
- **Gemma** (2026-11-09, findings 46-47). Gemma 3's sliding layers rotate at base 10000 and
its global ones at 1e6, and stock `Gemma3ModelChunk` takes one table for both: in fp32,
10000 everywhere fails the decision gate (6/11, KL 0.18) and 1e6 everywhere, the runner
sample's setting, worse (5/11, KL 1.42). `mediatek-gemma.patch` adds a second table input
(`pos_emb_local`) and the per-layer choice, which is exact against Hugging Face (KL 0,
11/11); the runner config names `rot_emb_local_base` and `rot_emb_inputs`, and the card
says MediaTek's 1.5.1 runner, which feeds one table, cannot run these files correctly.
gemma-3-1b-it is published for MT6991 (A16W8, 2 chunks) at 512 to 32k; the 32k export took
6,944 s on an r7i.4xlarge (123 GiB), the box at 92.8 GB in use with the 16k export beside it.
Gemma 2 2B is exact
as MediaTek ships it up to its 4,096-token sliding window (the card says what differs past
it), but its 2.36 GB fp32 output head is over `mtk_converter`'s 2 GB protobuf at any chunk
count, plain, fp16 or with int8 weights folded, so `mtk_plan` refuses it and Gemma 2 has
no backend.
- **Output** per chip: the chunk `.pte` files, the fp32 token embedding table the runner
reads from disk, and `config.json` with the flags MediaTek's LLM runner
(`examples/mediatek/executor_runner`) needs. They do not run on `TextLLMRunner`.
reads from disk (one per folder, shared by every window: `publish` refuses a run whose table
differs from the one the folder's other reports record, as a newer source revision's would,
unless `--replace-shared` withdraws those windows in the same commit for republishing),
and `config.json` with the flags MediaTek's LLM runner (`examples/mediatek/executor_runner`)
needs. They do not run on `TextLLMRunner`.
- **Check:** structural; every chunk loads, has its two methods, and delegates to
`NeuropilotBackend`.
- **16k built with one attention layer per chunk** (2026-10-04, finding 39). LFM2.5-1.2B-Instruct
Expand Down Expand Up @@ -389,8 +423,8 @@ pipeline exports it now; **todo** = ExecuTorch supports it and the pipeline does
| Llama 3.2 1B / 3B (+Instruct) | done | todo | done: 1B/3B-Instruct (registry) | todo (validate `llama.py`: rope_type, tokenizer) | upstream |
| SmolLM2 135M / 360M / 1.7B (+Instruct) | done | todo | done: 135M-Instruct (registry) | todo (validate, with Llama 3.2) | upstream |
| SmolLM3 3B | upstream | upstream | done | upstream | upstream |
| Gemma 3 1B | upstream | upstream | done: gemma-3-1b-it | todo (validate `gemma.py`: model_type gemma3_text) | upstream |
| Gemma 2 2B | upstream | upstream | todo | todo | upstream |
| Gemma 3 1B | upstream | upstream | done: gemma-3-1b-it, SM8750 2k-32k | done: gemma-3-1b-it, MT6991 (`mediatek-gemma.patch`, two RoPE tables) | upstream |
| Gemma 2 2B | upstream | upstream | todo | blocked: the 2.36 GB head is over `mtk_converter`'s 2 GB protobuf (finding 47) | upstream |
| Phi-4-mini 3.8B | todo (watch `microsoft`) | todo | todo | todo | upstream |
| LFM2.5 350M / 1.2B, LFM2 350M / 700M / 1.2B | todo (watch `LiquidAI`) | todo | upstream | upstream | upstream |
| Qwen3.5 0.8B / 2B / 4B | blocked: the app refuses `qwen35` names | — | upstream | upstream | upstream |
Expand Down
2 changes: 2 additions & 0 deletions docs/research/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,3 +71,5 @@ Findings from building and running this pipeline, each kept with its evidence. C
| 43 | 2026-11-09 | XNNPACK | Qwen3-0.6B does not keep its tool decision at int4 under the gate that asks with reasoning switched off. A GPTQ solve (ExecuTorch 1.5.1) keeps 0 of 4 calling rows at every window, mean 0.016 against fp32's 0.619, and calls on 2 of 4 quiet rows; the October GPTQ file reads 0.086 and 0 of 4 on the same gate. The same model with fp32 linears (int8 embedding) reads 0.618 and 4 of 4. Qwen3-1.7B (12 of 12, 0.995 against 0.999) and Qwen3-4B (12 of 12, 0.9999) keep it at int4. | measured | verify logs of the rollout's twins; `pipeline audit Qwen/Qwen3-0.6B --context 2048` | Qwen3-0.6B ships fp32 linears on XNNPACK and no Vulkan folder. |
| 44 | 2026-11-09 | XNNPACK | `pipeline solve` on macOS arm64 (torch 2.14.0, Apple M5) turns a calibration row's hidden state into NaN after a few quantised layers: SmolLM2-135M at layer 4 (row 39), Qwen2.5-0.5B at layer 9, then Cholesky fails on the NaN Hessian. The NaN sits in the prompt positions the Hessian does not count, and no submodule output is non-finite. The same commit solves both on Linux x86 without it. The Mac's fp32 gate reference is off too: for SmolLM2-135M it grades 9 calling rows at 0.713 where Linux grades 4 at 0.583, and the published, gated GPTQ file scores 0.0002 against the Mac's reference and 0.585 against Linux's on the same Mac runner. | measured | local runs on the Mac against the rollout's EC2 solves | Solve and compute gate references on Linux only; a Mac can export and run a gate against a Linux reference. |
| 45 | 2026-11-09 | Both | XNNPACK and Vulkan `.pte` files hold their int4 weights in a form that can be read back and compared exactly: XNNPACK as `qbint4` values (unsigned nibbles around 8, low nibble first, bf16 scales) behind each fully-connected node's filter id; Vulkan as `linear_dq8ca_q4gsw` arguments (uint8 [out, in/2], fp32 scales [groups, out], int32 per-group code sums). Both serialise linears in execution order. Matched against fp32 sources, a published round-to-nearest file agrees with plain rounding on 99.0 to 99.2 percent of codes with every scale at amax/7.5; a GPTQ file on 67 to 87 percent, with 9 to 36 percent of scales at amax/7.5. | measured | SmolLM2-135M's published files; the rollout's byte checks | Byte checks before publish and after. |
| 46 | 2026-11-09 | MediaTek | Gemma 3's sliding layers rotate at base 10000 (`rope_local_base_freq`) and its global ones at 1e6, but MediaTek's `Gemma3ModelChunk` takes one RoPE table for every layer: its master table is built at 10000 and its runner sample sets `rot_emb_base` 1e6. In fp32 against Hugging Face (google/gemma-3-1b-it, 900-token passage and the decision gate's prompts), 10000 everywhere is at KL 0.177, 88% top-1, perplexity 10.77 against 9.58, gate 6/11; 1e6 everywhere at KL 1.42, 52% top-1, 22.7, gate 5/11; a table per layer type is exact (KL 0, 100%, 9.575, 11/11). Gemma 2 (one base, 10000) is exact as MediaTek ships it, with softcapping, up to its 4,096-token sliding window. | measured | [`mtk-gemma-vs-hf.py`](evidence/mtk-gemma-vs-hf.py), [`mtk-gemma-fp32-vs-hf.log`](evidence/mtk-gemma-fp32-vs-hf.log) | `mediatek-gemma.patch` gives the chunk a second input, `pos_emb_local`, built at the local base, and `export_mtk` writes `rot_emb_local_base` and `rot_emb_inputs` for the runner, which has to feed both tables (31 inputs per method). Published for MT6991 with that caveat on the card: MediaTek's 1.5.1 runner feeds one table. |
| 47 | 2026-11-09 | MediaTek | `mtk_converter` 8.13.0 holds a delegated graph's weights as fp32 in one protobuf `ModelProto`, which protobuf caps at 2 GB, and the output head sits whole in the last chunk. Gemma 2 2B's tied 256000 x 2304 head is 2,359,296,000 B: the export fails in the converter at 2 chunks and at 13 (two layers each; 2361344595 B), and the head alone (final norm + head, A16W8, MT6991) fails plain, with `convert_float32_weights_to_float16` and with `convert_pt2e(fold_quantize=True)`: "exceeds maximum protobuf size of 2GB: 2361344528" (2361344648 folded). Gemma 3 1B's 262144 x 1152 head is 1.21 GB and builds. | measured | [`mtk-gemma2-head-probe.py`](evidence/mtk-gemma2-head-probe.py), [`mtk-gemma2-head-probe.log`](evidence/mtk-gemma2-head-probe.log) | `families.mtk_plan` refuses a head of 2 GB or more in fp32 (`mtk_head_bytes`), so Gemma 2 has no backend. A way through would split the head across separately delegated graphs and join the halves on the CPU, which the runtime would have to provide; not attempted. |
40 changes: 40 additions & 0 deletions docs/research/evidence/mtk-gemma-fp32-vs-hf.log
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# mtk-gemma-vs-hf.py, 2026-11-09, r7i.8xlarge, Ubuntu 24.04 x86_64, fp32, CPU.
# MediaTek's chunk classes at ExecuTorch 3b60683 (examples/mediatek/models/llm_models/modeling_gemma.py),
# one chunk of every layer, against transformers' fp32 eager model. Console output as printed by the
# harness's wrapper, one line per variant. Passage: 900 tokens of the model card, every position.
# Gate: execupack's 16 decision-gate prompts, last position (pipeline/gate.py).

== gemma3 google/gemma-3-1b-it @ dcc83ea841ab6100d6b47a070329e1ba4cf78752
# 1) Stock Gemma3ModelChunk: one RoPE table for every layer. Sliding layers use base 10000 and
# global ones 1e6 in Hugging Face's model; MediaTek's master table is built at 10000, its runner
# sample (run_gemma3_sample.sh) sets rot_emb_base 1e6. "per-layer" here emulated both tables.
MediaTek config: GemmaConfig Gemma3ModelChunk
passage 900 tokens; gate prompts 392 - 428 ; window 908
MediaTek master table vs base 10000: max |diff| 2.00e+00
MediaTek master table vs base 1e+06: max |diff| 6.10e-05
rope=10000 everywhere mask=-100 passage KL=0.1769 (max 8.460) top1=0.884 ppl hf/mtk=9.575/10.772 | gate top1=0.438 KL=5.6725 verdict=False 6/11 quiet 1/3
rope=10000 everywhere mask=-1e+09 passage KL=0.1769 (max 8.460) top1=0.884 ppl hf/mtk=9.575/10.772 | gate top1=0.438 KL=5.6725 verdict=False 6/11 quiet 1/3
rope=1e6 everywhere mask=-100 passage KL=1.4200 (max 12.288) top1=0.521 ppl hf/mtk=9.575/22.726 | gate top1=0.312 KL=3.6574 verdict=False 5/11 quiet 0/3
rope=1e6 everywhere mask=-1e+09 passage KL=1.4200 (max 12.288) top1=0.521 ppl hf/mtk=9.575/22.726 | gate top1=0.312 KL=3.6574 verdict=False 5/11 quiet 0/3
rope=per-layer mask=-100 passage KL=0.0000 (max 0.000) top1=1.000 ppl hf/mtk=9.575/9.575 | gate top1=1.000 KL=0.0000 verdict=True 11/11 quiet 3/3
rope=per-layer mask=-1e+09 passage KL=0.0000 (max 0.000) top1=1.000 ppl hf/mtk=9.575/9.575 | gate top1=1.000 KL=0.0000 verdict=True 11/11 quiet 3/3

# 2) Patched Gemma3ModelChunk (third_party/executorch/patches/mediatek-gemma.patch): the chunk
# takes pos_emb (global, 1e6) and pos_emb_local (10000) and gives each layer its own.
MediaTek config: GemmaConfig Gemma3ModelChunk
MediaTek master table vs base 10000: max |diff| 2.00e+00
MediaTek master table vs base 1e+06: max |diff| 6.10e-05
rope=10000 everywhere passage KL=0.17687 (max 8.4599) top1=0.884 ppl hf/mtk=9.575/10.772 | gate top1=0.438 verdict=False 6/11 quiet 1/3
rope=1e6 everywhere passage KL=1.41998 (max 12.2876) top1=0.521 ppl hf/mtk=9.575/22.726 | gate top1=0.312 verdict=False 5/11 quiet 0/3
rope=per-layer passage KL=0.00000 (max 0.0000) top1=1.000 ppl hf/mtk=9.575/9.575 | gate top1=1.000 verdict=True 11/11 quiet 3/3

== gemma2 google/gemma-2-2b-it @ 299a8560bedf22ed1c72a8a11e7dce4a7f9f51f8
# Stock Gemma2ModelChunk, base 10000 on every layer as in Hugging Face's model. The template
# refuses a system turn, so the gate's system text is folded into the first user turn.
# Window 908 < sliding_window 4096, so the sliding layers see what they see in Hugging Face's.
MediaTek config: GemmaConfig Gemma2ModelChunk
passage 900 tokens; gate prompts 391 - 425 ; window 908
MediaTek master table vs base 10000: max |diff| 1.22e-04
MediaTek master table vs base 1e+06: max |diff| 2.00e+00
rope=10000 everywhere mask=-100 passage KL=0.00000 (max 0.0000) top1=1.000 ppl hf/mtk=6.054/6.054 | gate top1=1.000 KL=0.00000 verdict=True 8/8 quiet 4/4
rope=10000 everywhere mask=-1e+09 passage KL=0.00000 (max 0.0000) top1=1.000 ppl hf/mtk=6.054/6.054 | gate top1=1.000 KL=0.00000 verdict=True 8/8 quiet 4/4
Loading
Loading