Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/export-mtk.yml
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,10 @@ jobs:
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: ./.github/actions/setup-light
# As in export-xnnpack.yml: nothing newer than the apps' runtime is built to be published.
- name: Refuse to build what could not be published
if: ${{ inputs.publish_hf || inputs.publish_release }}
run: python3 -m pipeline publish-hold --backend mtk
- id: socs
env:
SOCS: ${{ inputs.socs }}
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/export-qnn.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,10 @@ jobs:
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: ./.github/actions/setup-light
# As in export-xnnpack.yml: nothing newer than the apps' runtime is built to be published.
- name: Refuse to build what could not be published
if: ${{ inputs.publish_hf || inputs.publish_release }}
run: python3 -m pipeline publish-hold --backend qnn
- id: socs
env:
SOCS: ${{ inputs.socs }}
Expand Down
5 changes: 5 additions & 0 deletions .github/workflows/export-xnnpack.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,11 @@ jobs:
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: ./.github/actions/setup-light
# Before any solve or export: a run that is to publish files newer than the apps' runtime
# would only be refused at its publish step, after hours of work (APP_RUNTIME_VERSION).
- name: Refuse to build what could not be published
if: ${{ inputs.publish_hf || inputs.publish_release }}
run: python3 -m pipeline publish-hold --backend xnnpack
- id: contexts
env:
CONTEXTS: ${{ inputs.contexts }}
Expand Down
13 changes: 9 additions & 4 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ limits. Read it before changing sizing, backends or naming, and update it when a
```sh
pip install -r requirements/dev.txt # enough for lint + all tests except tests/test_convert.py
# test_convert.py is skipped unless torch + safetensors are present (CI installs them from the CPU index):
pip install torch==2.14.0 --index-url https://download.pytorch.org/whl/cpu && pip install safetensors==0.8.0 numpy
pip install torch==2.15.0 --index-url https://download.pytorch.org/whl/cpu && pip install safetensors==0.8.0 numpy

ruff check . && ruff format --check . # CI runs both; line length 120
pytest -q # all tests
Expand Down Expand Up @@ -92,15 +92,20 @@ the app's `config.json` from a folder's reports. **`export-mtk.yml`** → `expor

## Constraints that aren't obvious from the code

- **Version lock:** `EXECUTORCH_VERSION` in `config/versions.env` must equal the executorch-android AAR the app ships.
- **Version lock:** nothing built with a newer ExecuTorch than the apps' runtime may be published. `APP_RUNTIME_VERSION`
in `config/versions.env` is the oldest executorch-android AAR the apps ship, and `publish.require_publishable` refuses
any report built with a newer release; move it only once both apps ship that runtime. `EXECUTORCH_VERSION` (the
CPU/GPU exporter) may run ahead of it to prepare an upgrade. The NPU toolchains stay one minor behind on 1.5.1:
Qualcomm (`QNN_EXECUTORCH_VERSION`), because 1.6.0 compiles with QAIRT 2.51, newer than the apps' QNN runtime, and
MediaTek (`MTK_EXECUTORCH_VERSION`), because `mtk_converter` is cp310-only and ExecuTorch 1.6.0 has no cp310 wheel.
A newer exporter can emit methods the older runtime lacks. `tests/test_versions.py` checks the pip pins against it;
bump pins, `versions.env` and `QAIRT_VERSION` together.
- **App-imposed naming rules** (from the app source, see PLAN.md): the output repo name must never contain `xnnpack`,
must keep the size (`1.7B`), and repo name + file stem must contain an app family token and none of `vl`, `vision`,
`coder`, `guard`, `qwen35`.
- **Tokenizer only at the repo root.** A tokenizer inside any backend folder stops the app lending the root one to the
other folders; `publish_hf` refuses it.
- `third_party/executorch/` holds params files copied verbatim from ExecuTorch v1.4.0, unchanged in v1.5.1 (the wheel doesn't ship them) for
- `third_party/executorch/` holds params files copied verbatim from ExecuTorch v1.4.0, unchanged in v1.5.1 and v1.6.0 (the wheel doesn't ship them) for
the QNN registry. Refresh them when the ExecuTorch pin moves.
- QNN `.pte` metadata always has BOS 1 / EOS 2 (hard-coded upstream); stop tokens must come from the tokenizer.
- Upstream licenses are copied verbatim into published repos; vendor SDKs (QAIRT, NeuroPilot) are downloaded at run
Expand All @@ -117,7 +122,7 @@ is pinned (`tests/test_versions.py` enforces it, and that the backend files agre

`tests/fixtures/*.config.json` are real HF `config.json` files, and `et-*.params.json` are ExecuTorch's own params for
the same models. `tests/test_convert.py` and `tests/test_qnn.py` skip without torch/executorch; CI installs both (CPU
index), and locally a scratch venv with `torch==2.14.0 executorch==1.5.1` runs them on macOS too. Tests check generated params against them and against the vendored `third_party` copies. Build a
index), and locally a scratch venv with `torch==2.15.0 executorch==1.6.0` (Python 3.11+) runs them on macOS too. Tests check generated params against them and against the vendored `third_party` copies. Build a
`SourceModel` with the `source` fixture / `conftest.make_source`; `TOTAL_PARAMS` holds real Hub parameter counts. The
watcher takes injectable `lister`/`fetch`/`dispatcher` callables, so tests never touch the network.

Expand Down
2 changes: 1 addition & 1 deletion THIRD_PARTY_NOTICES.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,5 +78,5 @@ from MediaTek Inc. and that no MediaTek SDK or runtime library is included.
## ExecuTorch examples/mediatek

`export-mtk.yml` also fetches `examples/mediatek` from the ExecuTorch source at
`EXECUTORCH_COMMIT` (v1.4.0) at run time; those scripts are © MediaTek Inc., under the
`EXECUTORCH_COMMIT` (v1.5.1) at run time; those scripts are © MediaTek Inc., under the
BSD-style license in ExecuTorch's repository root.
36 changes: 29 additions & 7 deletions config/versions.env
Original file line number Diff line number Diff line change
@@ -1,11 +1,25 @@
# Toolchain versions that are not pip packages, plus the ExecuTorch release every backend
# must match. EXECUTORCH_VERSION has to equal the executorch-android AAR the openweights app
# ships (gradle/libs.versions.toml there): a newer exporter can emit method names the older
# runtime does not know. tests/test_versions.py checks the pip pins agree with it.
EXECUTORCH_VERSION=1.5.1
# Toolchain versions that are not pip packages, plus the ExecuTorch releases the backends build
# with. EXECUTORCH_VERSION is the CPU and GPU exporter's; tests/test_versions.py checks the pip
# pins agree with it. It may run ahead of the apps' runtime (APP_RUNTIME_VERSION below), whose
# executorch-android AAR (gradle/libs.versions.toml in each app) decides what may be published:
# a newer exporter can emit method names the older runtime does not know. The NPU backends
# build one release behind: see QNN_EXECUTORCH_VERSION and MTK_EXECUTORCH_VERSION.
EXECUTORCH_VERSION=1.6.0
# The oldest ExecuTorch runtime the apps ship (openweights 1.5.1-openweights.4, ExecuServe
# 1.5.1-execuserve.4). Publishing refuses a file built with a newer release than this
# (publish.require_publishable), because ExecuTorch promises an older program loads on a newer
# runtime and nothing the other way. Move it only once both apps ship the newer runtime.
APP_RUNTIME_VERSION=1.5.1
PYTHON_VERSION=3.11
# Qualcomm exports stay on ExecuTorch 1.5.1 and its QAIRT 2.37. ExecuTorch 1.6.0's wheel pins
# and downloads QAIRT 2.51.0.260929 (backends/qualcomm/scripts/qnn_config.sh), and a context
# binary from a newer SDK is not promised to load on an older QNN runtime: both apps ship
# qnn-runtime 2.39.0, which loads 2.37's binaries. A 1.6.0 runtime loads 1.5.1 programs (one
# minor back), so these files stay loadable as the apps move to 1.6.0. Move this pin, and
# QAIRT_VERSION with it, once the apps' QNN runtime is at least the SDK it would compile with.
QNN_EXECUTORCH_VERSION=1.5.1
# QAIRT (Qualcomm AI Runtime SDK) the executorch 1.5.1 wheel is built against and downloads
# on first import (backends/qualcomm/scripts/qnn_config.sh). executorch-android-qnn:1.5.1
# when asked (backends/qualcomm/scripts/qnn_config.sh). executorch-android-qnn:1.5.1
# depends on qnn-runtime 2.37.0. Used as the Actions cache key for the SDK.
QAIRT_VERSION=2.37.0.250724
# MediaTek NeuroPilot Express SDK 8.0.8 (neuropilot-express-sdk-8.0.8-build20250925.tar.gz),
Expand All @@ -22,5 +36,13 @@ NEUROPILOT_SDK_BUILD=8.0.8-build20250925
MTK_CONVERTER_VERSION=8.13.0
MTK_NEURON_VERSION=8.2.23
MTK_PYTHON_VERSION=3.10
# ExecuTorch source for examples/mediatek (the LLM export scripts are not in the wheel).
# MediaTek exports stay on ExecuTorch 1.5.1 too: mtk_converter is cp310-only and 1.6.0 publishes
# no cp310 wheel (it requires Python >= 3.11). Nothing the export runs moved in 1.6.0:
# backends/mediatek's Python and examples/mediatek's scripts are unchanged from 1.5.1 (only
# build files differ: two scripts' -j and the runtime's BUCK), so the patches apply to either.
# A 1.6.0 runtime loads 1.5.1 programs (one minor back is ExecuTorch's promise), so files from
# this toolchain stay inside it.
MTK_EXECUTORCH_VERSION=1.5.1
# ExecuTorch source for examples/mediatek (the LLM export scripts are not in the wheel), at
# MTK_EXECUTORCH_VERSION's tag (v1.5.1) so the scripts match the wheel that runs them.
EXECUTORCH_COMMIT=3b60683923245cf472b7323426920e15623ba361
27 changes: 26 additions & 1 deletion docs/PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ for it, Samsung Exynos (ENN). iOS is deferred.
| Outputs | Hugging Face Hub, GitHub Releases (files ≤ 2 GiB), Actions artifacts |
| HF layout | One repo per model, backend folders; NPU exports published even though the app can't load them yet |
| Context window | Every window of 2k/4k/8k/16k/32k, per model and backend, one workflow job each; the app and the benchmarker decide what fits a device. A window the runner cannot build is skipped and reported, not failed |
| Runtime pin | ExecuTorch **1.5.1** everywhere since 2026-10-04 (1.4.0 before; PR #6, after the apps moved to the 1.5.1 runtime); QAIRT **2.37.0**, unchanged by 1.5.1 |
| Runtime pin | XNNPACK and Vulkan export with ExecuTorch **1.6.0** (torch 2.15.0) since 2026-11-13; the NPU toolchains stay on **1.5.1**, one minor behind, which a 1.6.0 runtime is promised to load: Qualcomm because 1.6.0 compiles with QAIRT 2.51, newer than the apps' `qnn-runtime` 2.39 (QAIRT stays **2.37.0**), MediaTek because `mtk_converter` is cp310-only and 1.6.0 has no cp310 wheel. The apps still ship 1.5.1, so `APP_RUNTIME_VERSION` (1.5.1) makes `publish` refuse any 1.6.0 file until both apps move; regression measured in finding 48. Before: 1.5.1 everywhere from 2026-10-04 (PR #6), 1.4.0 before that |
| Vendor SDKs | Downloaded from the vendor at run time, cached only in this repo's Actions cache, never re-hosted |

## Hugging Face repo layout
Expand Down Expand Up @@ -473,6 +473,31 @@ G5, Dimensity 9300+, Exynos 2500; GSM8K, RetrievalQA, IFEval, PopQA, BFCL, Fresh

## Known limits

- **ExecuTorch 1.6.0** (2026-11-14, finding 48):
- **Held.** The CPU and GPU exporter is on 1.6.0 but the apps still ship 1.5.1, so `publish`
refuses every 1.6.0 file (`APP_RUNTIME_VERSION`), and the export workflows stop before
building unless both publish inputs are off. Move `APP_RUNTIME_VERSION` once both apps ship
1.6.0.
- **NPU toolchains.** QNN stays on 1.5.1 and QAIRT 2.37 until the apps' QNN runtime is 2.51 or
later; MediaTek stays on 1.5.1 until MediaTek publishes a `mtk_converter` for Python 3.11.
- **Two releases back.** A 1.6.0 runtime is promised 1.5.1 files, not the 1.4.0 ones still
published (the CPU files of LFM2.5-1.2B-Instruct, LFM2.5-2.6B, their heretic finetunes and
SmolLM2-360M-Instruct, and 145 of the 178 MediaTek chunks). On the Python runtime the CPU ones
load, and their first token on today's gate prompts is the 1.4.0 runtime's on 79 of 80 rows
(78 on 1.5.1). Rebuild them with 1.6.0 if the promise has to hold.
- **Smoke test.** Qwen2.5-Math-1.5B's published file fails it on 1.6.0 (no "paris" for the
capital of France) while passing its gate; its reply turns on an eighth-token split, and a
1.6.0 rebuild passed only because its reply reached "Paris" later. The smoke test asks every
model of 500 million parameters or more for "paris" (`smoke.ANSWER_REQUIRED_FROM_PARAMS`).
- **Mixed folders.** Windows are republished one at a time, so a folder can hold files from two
releases. Each variant in `config.json` names its own `runtime_version`, and the folder names
the newest, so an installed app on an older runtime is warned about a newer file. An app that
reads only the folder's version is not warned about an older file two releases back (a 1.4.0
window beside a 1.6.0 one): ExecuServe reads only the folder's today, and openweights neither.
- **Not measured.** The Linux side (Modal image install, the solve on torch 2.15, the x86 C++
runner of the verify job), Phi-4-mini's export, a GPU or phone, and the apps' own runtime
builds. The first CI run on this pin, built with publishing off, is that test.

- **Phi-4-mini** (2026-11-09):
- **Recipe.** It is export_llm's `phi_4_mini` with execupack's params and converter
(fused qkv/gate_up split), and matches transformers' fp32 Phi3ForCausalLM to 0.003 in
Expand Down
1 change: 1 addition & 0 deletions docs/research/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,3 +73,4 @@ Findings from building and running this pipeline, each kept with its evidence. C
| 45 | 2026-11-09 | Both | XNNPACK and Vulkan `.pte` files hold their int4 weights in a form that can be read back and compared exactly: XNNPACK as `qbint4` values (unsigned nibbles around 8, low nibble first, bf16 scales) behind each fully-connected node's filter id; Vulkan as `linear_dq8ca_q4gsw` arguments (uint8 [out, in/2], fp32 scales [groups, out], int32 per-group code sums). Both serialise linears in execution order. Matched against fp32 sources, a published round-to-nearest file agrees with plain rounding on 99.0 to 99.2 percent of codes with every scale at amax/7.5; a GPTQ file on 67 to 87 percent, with 9 to 36 percent of scales at amax/7.5. | measured | SmolLM2-135M's published files; the rollout's byte checks | Byte checks before publish and after. |
| 46 | 2026-11-09 | MediaTek | Gemma 3's sliding layers rotate at base 10000 (`rope_local_base_freq`) and its global ones at 1e6, but MediaTek's `Gemma3ModelChunk` takes one RoPE table for every layer: its master table is built at 10000 and its runner sample sets `rot_emb_base` 1e6. In fp32 against Hugging Face (google/gemma-3-1b-it, 900-token passage and the decision gate's prompts), 10000 everywhere is at KL 0.177, 88% top-1, perplexity 10.77 against 9.58, gate 6/11; 1e6 everywhere at KL 1.42, 52% top-1, 22.7, gate 5/11; a table per layer type is exact (KL 0, 100%, 9.575, 11/11). Gemma 2 (one base, 10000) is exact as MediaTek ships it, with softcapping, up to its 4,096-token sliding window. | measured | [`mtk-gemma-vs-hf.py`](evidence/mtk-gemma-vs-hf.py), [`mtk-gemma-fp32-vs-hf.log`](evidence/mtk-gemma-fp32-vs-hf.log) | `mediatek-gemma.patch` gives the chunk a second input, `pos_emb_local`, built at the local base, and `export_mtk` writes `rot_emb_local_base` and `rot_emb_inputs` for the runner, which has to feed both tables (31 inputs per method). Published for MT6991 with that caveat on the card: MediaTek's 1.5.1 runner feeds one table. |
| 47 | 2026-11-09 | MediaTek | `mtk_converter` 8.13.0 holds a delegated graph's weights as fp32 in one protobuf `ModelProto`, which protobuf caps at 2 GB, and the output head sits whole in the last chunk. Gemma 2 2B's tied 256000 x 2304 head is 2,359,296,000 B: the export fails in the converter at 2 chunks and at 13 (two layers each; 2361344595 B), and the head alone (final norm + head, A16W8, MT6991) fails plain, with `convert_float32_weights_to_float16` and with `convert_pt2e(fold_quantize=True)`: "exceeds maximum protobuf size of 2GB: 2361344528" (2361344648 folded). Gemma 3 1B's 262144 x 1152 head is 1.21 GB and builds. | measured | [`mtk-gemma2-head-probe.py`](evidence/mtk-gemma2-head-probe.py), [`mtk-gemma2-head-probe.log`](evidence/mtk-gemma2-head-probe.log) | `families.mtk_plan` refuses a head of 2 GB or more in fp32 (`mtk_head_bytes`), so Gemma 2 has no backend. A way through would split the head across separately delegated graphs and join the halves on the CPU, which the runtime would have to provide; not attempted. |
| 48 | 2026-11-14 | Runtime | ExecuTorch 1.6.0 (torch 2.15.0) against 1.5.1, on the Python runtime (macOS arm64). **Published files:** all 368 `.pte` files parse under 1.6.0's program schema, among them 145 MediaTek chunks built with 1.4.0, which the host cannot run. All 20 published CPU files (one 2k window per repo, five of them 1.4.0 builds) load and run on 1.6.0 with the same methods and metadata. The int4 files' outputs move (KL 0.0025 to 0.064 on a probe; gate means by up to 0.090, Phi-4-mini, and 0.054 for the rest), the fp32 ones' do not. Of the 16 with an archived gate reference, 15 keep 1.5.1's verdict and agreed count on 1.6.0, and Phi-4-mini keeps one more row (8/8 against 7/8). Qwen2.5-Math-1.5B fails the smoke test on 1.6.0 ("parallelogram … lyle" for the capital of France) while its gate passes 3/3: at the answer its top tokens have probabilities of 0.03 to 0.05 on both runtimes, and an eighth-token split (0.256 against 0.240) goes the other way. **1.4.0 builds:** their first token on today's gate prompts matches the 1.4.0 runtime's on 79 of 80 rows on 1.6.0 (78 on 1.5.1); on the other row the top two first tokens are 0.18 apart on 1.4.0 and 0.07 on 1.6.0, neither a tool call. **New 1.6.0 exports** from the same codes: SmolLM2-135M, Qwen2.5-0.5B, Qwen2.5-Math-1.5B, Qwen3-0.6B (fp32) and Llama-3.2-1B pass their gates, LFM3-1.7B its gate with the conv state cleared at position 0; the SmolLM2 Vulkan file passed only its structural check and gate proxy, the rounded LFM2.5-1.2B only its smoke test; the int4 codes are byte-exact in every GPTQ one. Against a 1.5.1 build on the same host, only the two RoPE tables (by 1.19e-7), the program's value table and the XNNPACK blobs (27 KB smaller in all) differ. A 1.6.0 file loads and passes its gate on the 1.5.1 runtime (SmolLM2-135M). **Cannot:** build for MediaTek (no cp310 wheel) or for the apps' QNN runtime (1.6.0's Qualcomm scripts compile with QAIRT 2.51; the apps ship `qnn-runtime` 2.39). | measured + source | [`executorch-1.6.0-regression.md`](evidence/executorch-1.6.0-regression.md) and the `et160-*.py` probes beside it; executorch 1.6.0's PyPI metadata (Requires-Python >= 3.11, torch==2.15.0); ExecuTorch v1.6.0 `backends/qualcomm/scripts/qnn_config.sh` | XNNPACK and Vulkan export with 1.6.0; QNN and MediaTek stay on 1.5.1 (`QNN_EXECUTORCH_VERSION`, `MTK_EXECUTORCH_VERSION`); `APP_RUNTIME_VERSION` 1.5.1 holds every 1.6.0 file back from publishing until both apps ship 1.6.0. Not measured: Linux (Modal image, solve on torch 2.15, the x86 C++ runner), a GPU or phone, the apps' own runtime builds, Phi-4-mini's export. The smoke test requires "paris" of the Math model, whose reply on these runs turned on an eighth-token split. |
Loading
Loading