diff --git a/.github/workflows/export-mtk.yml b/.github/workflows/export-mtk.yml index 9795262..ed64c6b 100644 --- a/.github/workflows/export-mtk.yml +++ b/.github/workflows/export-mtk.yml @@ -62,6 +62,10 @@ jobs: steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - uses: ./.github/actions/setup-light + # As in export-xnnpack.yml: nothing newer than the apps' runtime is built to be published. + - name: Refuse to build what could not be published + if: ${{ inputs.publish_hf || inputs.publish_release }} + run: python3 -m pipeline publish-hold --backend mtk - id: socs env: SOCS: ${{ inputs.socs }} diff --git a/.github/workflows/export-qnn.yml b/.github/workflows/export-qnn.yml index a834bc5..d98d933 100644 --- a/.github/workflows/export-qnn.yml +++ b/.github/workflows/export-qnn.yml @@ -49,6 +49,10 @@ jobs: steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - uses: ./.github/actions/setup-light + # As in export-xnnpack.yml: nothing newer than the apps' runtime is built to be published. + - name: Refuse to build what could not be published + if: ${{ inputs.publish_hf || inputs.publish_release }} + run: python3 -m pipeline publish-hold --backend qnn - id: socs env: SOCS: ${{ inputs.socs }} diff --git a/.github/workflows/export-xnnpack.yml b/.github/workflows/export-xnnpack.yml index 6de30f3..368098a 100644 --- a/.github/workflows/export-xnnpack.yml +++ b/.github/workflows/export-xnnpack.yml @@ -57,6 +57,11 @@ jobs: steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - uses: ./.github/actions/setup-light + # Before any solve or export: a run that is to publish files newer than the apps' runtime + # would only be refused at its publish step, after hours of work (APP_RUNTIME_VERSION). + - name: Refuse to build what could not be published + if: ${{ inputs.publish_hf || inputs.publish_release }} + run: python3 -m pipeline publish-hold --backend xnnpack - id: contexts env: CONTEXTS: ${{ inputs.contexts }} diff --git a/CLAUDE.md b/CLAUDE.md index 99cb318..65534b8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -14,7 +14,7 @@ limits. Read it before changing sizing, backends or naming, and update it when a ```sh pip install -r requirements/dev.txt # enough for lint + all tests except tests/test_convert.py # test_convert.py is skipped unless torch + safetensors are present (CI installs them from the CPU index): -pip install torch==2.14.0 --index-url https://download.pytorch.org/whl/cpu && pip install safetensors==0.8.0 numpy +pip install torch==2.15.0 --index-url https://download.pytorch.org/whl/cpu && pip install safetensors==0.8.0 numpy ruff check . && ruff format --check . # CI runs both; line length 120 pytest -q # all tests @@ -92,7 +92,12 @@ the app's `config.json` from a folder's reports. **`export-mtk.yml`** → `expor ## Constraints that aren't obvious from the code -- **Version lock:** `EXECUTORCH_VERSION` in `config/versions.env` must equal the executorch-android AAR the app ships. +- **Version lock:** nothing built with a newer ExecuTorch than the apps' runtime may be published. `APP_RUNTIME_VERSION` + in `config/versions.env` is the oldest executorch-android AAR the apps ship, and `publish.require_publishable` refuses + any report built with a newer release; move it only once both apps ship that runtime. `EXECUTORCH_VERSION` (the + CPU/GPU exporter) may run ahead of it to prepare an upgrade. The NPU toolchains stay one minor behind on 1.5.1: + Qualcomm (`QNN_EXECUTORCH_VERSION`), because 1.6.0 compiles with QAIRT 2.51, newer than the apps' QNN runtime, and + MediaTek (`MTK_EXECUTORCH_VERSION`), because `mtk_converter` is cp310-only and ExecuTorch 1.6.0 has no cp310 wheel. A newer exporter can emit methods the older runtime lacks. `tests/test_versions.py` checks the pip pins against it; bump pins, `versions.env` and `QAIRT_VERSION` together. - **App-imposed naming rules** (from the app source, see PLAN.md): the output repo name must never contain `xnnpack`, @@ -100,7 +105,7 @@ the app's `config.json` from a folder's reports. **`export-mtk.yml`** → `expor `coder`, `guard`, `qwen35`. - **Tokenizer only at the repo root.** A tokenizer inside any backend folder stops the app lending the root one to the other folders; `publish_hf` refuses it. -- `third_party/executorch/` holds params files copied verbatim from ExecuTorch v1.4.0, unchanged in v1.5.1 (the wheel doesn't ship them) for +- `third_party/executorch/` holds params files copied verbatim from ExecuTorch v1.4.0, unchanged in v1.5.1 and v1.6.0 (the wheel doesn't ship them) for the QNN registry. Refresh them when the ExecuTorch pin moves. - QNN `.pte` metadata always has BOS 1 / EOS 2 (hard-coded upstream); stop tokens must come from the tokenizer. - Upstream licenses are copied verbatim into published repos; vendor SDKs (QAIRT, NeuroPilot) are downloaded at run @@ -117,7 +122,7 @@ is pinned (`tests/test_versions.py` enforces it, and that the backend files agre `tests/fixtures/*.config.json` are real HF `config.json` files, and `et-*.params.json` are ExecuTorch's own params for the same models. `tests/test_convert.py` and `tests/test_qnn.py` skip without torch/executorch; CI installs both (CPU -index), and locally a scratch venv with `torch==2.14.0 executorch==1.5.1` runs them on macOS too. Tests check generated params against them and against the vendored `third_party` copies. Build a +index), and locally a scratch venv with `torch==2.15.0 executorch==1.6.0` (Python 3.11+) runs them on macOS too. Tests check generated params against them and against the vendored `third_party` copies. Build a `SourceModel` with the `source` fixture / `conftest.make_source`; `TOTAL_PARAMS` holds real Hub parameter counts. The watcher takes injectable `lister`/`fetch`/`dispatcher` callables, so tests never touch the network. diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index 7f4a9a0..aab39d6 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -78,5 +78,5 @@ from MediaTek Inc. and that no MediaTek SDK or runtime library is included. ## ExecuTorch examples/mediatek `export-mtk.yml` also fetches `examples/mediatek` from the ExecuTorch source at -`EXECUTORCH_COMMIT` (v1.4.0) at run time; those scripts are © MediaTek Inc., under the +`EXECUTORCH_COMMIT` (v1.5.1) at run time; those scripts are © MediaTek Inc., under the BSD-style license in ExecuTorch's repository root. diff --git a/config/versions.env b/config/versions.env index c4e5545..d7f520d 100644 --- a/config/versions.env +++ b/config/versions.env @@ -1,11 +1,25 @@ -# Toolchain versions that are not pip packages, plus the ExecuTorch release every backend -# must match. EXECUTORCH_VERSION has to equal the executorch-android AAR the openweights app -# ships (gradle/libs.versions.toml there): a newer exporter can emit method names the older -# runtime does not know. tests/test_versions.py checks the pip pins agree with it. -EXECUTORCH_VERSION=1.5.1 +# Toolchain versions that are not pip packages, plus the ExecuTorch releases the backends build +# with. EXECUTORCH_VERSION is the CPU and GPU exporter's; tests/test_versions.py checks the pip +# pins agree with it. It may run ahead of the apps' runtime (APP_RUNTIME_VERSION below), whose +# executorch-android AAR (gradle/libs.versions.toml in each app) decides what may be published: +# a newer exporter can emit method names the older runtime does not know. The NPU backends +# build one release behind: see QNN_EXECUTORCH_VERSION and MTK_EXECUTORCH_VERSION. +EXECUTORCH_VERSION=1.6.0 +# The oldest ExecuTorch runtime the apps ship (openweights 1.5.1-openweights.4, ExecuServe +# 1.5.1-execuserve.4). Publishing refuses a file built with a newer release than this +# (publish.require_publishable), because ExecuTorch promises an older program loads on a newer +# runtime and nothing the other way. Move it only once both apps ship the newer runtime. +APP_RUNTIME_VERSION=1.5.1 PYTHON_VERSION=3.11 +# Qualcomm exports stay on ExecuTorch 1.5.1 and its QAIRT 2.37. ExecuTorch 1.6.0's wheel pins +# and downloads QAIRT 2.51.0.260929 (backends/qualcomm/scripts/qnn_config.sh), and a context +# binary from a newer SDK is not promised to load on an older QNN runtime: both apps ship +# qnn-runtime 2.39.0, which loads 2.37's binaries. A 1.6.0 runtime loads 1.5.1 programs (one +# minor back), so these files stay loadable as the apps move to 1.6.0. Move this pin, and +# QAIRT_VERSION with it, once the apps' QNN runtime is at least the SDK it would compile with. +QNN_EXECUTORCH_VERSION=1.5.1 # QAIRT (Qualcomm AI Runtime SDK) the executorch 1.5.1 wheel is built against and downloads -# on first import (backends/qualcomm/scripts/qnn_config.sh). executorch-android-qnn:1.5.1 +# when asked (backends/qualcomm/scripts/qnn_config.sh). executorch-android-qnn:1.5.1 # depends on qnn-runtime 2.37.0. Used as the Actions cache key for the SDK. QAIRT_VERSION=2.37.0.250724 # MediaTek NeuroPilot Express SDK 8.0.8 (neuropilot-express-sdk-8.0.8-build20250925.tar.gz), @@ -22,5 +36,13 @@ NEUROPILOT_SDK_BUILD=8.0.8-build20250925 MTK_CONVERTER_VERSION=8.13.0 MTK_NEURON_VERSION=8.2.23 MTK_PYTHON_VERSION=3.10 -# ExecuTorch source for examples/mediatek (the LLM export scripts are not in the wheel). +# MediaTek exports stay on ExecuTorch 1.5.1 too: mtk_converter is cp310-only and 1.6.0 publishes +# no cp310 wheel (it requires Python >= 3.11). Nothing the export runs moved in 1.6.0: +# backends/mediatek's Python and examples/mediatek's scripts are unchanged from 1.5.1 (only +# build files differ: two scripts' -j and the runtime's BUCK), so the patches apply to either. +# A 1.6.0 runtime loads 1.5.1 programs (one minor back is ExecuTorch's promise), so files from +# this toolchain stay inside it. +MTK_EXECUTORCH_VERSION=1.5.1 +# ExecuTorch source for examples/mediatek (the LLM export scripts are not in the wheel), at +# MTK_EXECUTORCH_VERSION's tag (v1.5.1) so the scripts match the wheel that runs them. EXECUTORCH_COMMIT=3b60683923245cf472b7323426920e15623ba361 diff --git a/docs/PLAN.md b/docs/PLAN.md index 58df6e4..d5407d1 100644 --- a/docs/PLAN.md +++ b/docs/PLAN.md @@ -19,7 +19,7 @@ for it, Samsung Exynos (ENN). iOS is deferred. | Outputs | Hugging Face Hub, GitHub Releases (files ≤ 2 GiB), Actions artifacts | | HF layout | One repo per model, backend folders; NPU exports published even though the app can't load them yet | | Context window | Every window of 2k/4k/8k/16k/32k, per model and backend, one workflow job each; the app and the benchmarker decide what fits a device. A window the runner cannot build is skipped and reported, not failed | -| Runtime pin | ExecuTorch **1.5.1** everywhere since 2026-10-04 (1.4.0 before; PR #6, after the apps moved to the 1.5.1 runtime); QAIRT **2.37.0**, unchanged by 1.5.1 | +| Runtime pin | XNNPACK and Vulkan export with ExecuTorch **1.6.0** (torch 2.15.0) since 2026-11-13; the NPU toolchains stay on **1.5.1**, one minor behind, which a 1.6.0 runtime is promised to load: Qualcomm because 1.6.0 compiles with QAIRT 2.51, newer than the apps' `qnn-runtime` 2.39 (QAIRT stays **2.37.0**), MediaTek because `mtk_converter` is cp310-only and 1.6.0 has no cp310 wheel. The apps still ship 1.5.1, so `APP_RUNTIME_VERSION` (1.5.1) makes `publish` refuse any 1.6.0 file until both apps move; regression measured in finding 48. Before: 1.5.1 everywhere from 2026-10-04 (PR #6), 1.4.0 before that | | Vendor SDKs | Downloaded from the vendor at run time, cached only in this repo's Actions cache, never re-hosted | ## Hugging Face repo layout @@ -473,6 +473,31 @@ G5, Dimensity 9300+, Exynos 2500; GSM8K, RetrievalQA, IFEval, PopQA, BFCL, Fresh ## Known limits +- **ExecuTorch 1.6.0** (2026-11-14, finding 48): + - **Held.** The CPU and GPU exporter is on 1.6.0 but the apps still ship 1.5.1, so `publish` + refuses every 1.6.0 file (`APP_RUNTIME_VERSION`), and the export workflows stop before + building unless both publish inputs are off. Move `APP_RUNTIME_VERSION` once both apps ship + 1.6.0. + - **NPU toolchains.** QNN stays on 1.5.1 and QAIRT 2.37 until the apps' QNN runtime is 2.51 or + later; MediaTek stays on 1.5.1 until MediaTek publishes a `mtk_converter` for Python 3.11. + - **Two releases back.** A 1.6.0 runtime is promised 1.5.1 files, not the 1.4.0 ones still + published (the CPU files of LFM2.5-1.2B-Instruct, LFM2.5-2.6B, their heretic finetunes and + SmolLM2-360M-Instruct, and 145 of the 178 MediaTek chunks). On the Python runtime the CPU ones + load, and their first token on today's gate prompts is the 1.4.0 runtime's on 79 of 80 rows + (78 on 1.5.1). Rebuild them with 1.6.0 if the promise has to hold. + - **Smoke test.** Qwen2.5-Math-1.5B's published file fails it on 1.6.0 (no "paris" for the + capital of France) while passing its gate; its reply turns on an eighth-token split, and a + 1.6.0 rebuild passed only because its reply reached "Paris" later. The smoke test asks every + model of 500 million parameters or more for "paris" (`smoke.ANSWER_REQUIRED_FROM_PARAMS`). + - **Mixed folders.** Windows are republished one at a time, so a folder can hold files from two + releases. Each variant in `config.json` names its own `runtime_version`, and the folder names + the newest, so an installed app on an older runtime is warned about a newer file. An app that + reads only the folder's version is not warned about an older file two releases back (a 1.4.0 + window beside a 1.6.0 one): ExecuServe reads only the folder's today, and openweights neither. + - **Not measured.** The Linux side (Modal image install, the solve on torch 2.15, the x86 C++ + runner of the verify job), Phi-4-mini's export, a GPU or phone, and the apps' own runtime + builds. The first CI run on this pin, built with publishing off, is that test. + - **Phi-4-mini** (2026-11-09): - **Recipe.** It is export_llm's `phi_4_mini` with execupack's params and converter (fused qkv/gate_up split), and matches transformers' fp32 Phi3ForCausalLM to 0.003 in diff --git a/docs/research/README.md b/docs/research/README.md index f80203e..14d07b9 100644 --- a/docs/research/README.md +++ b/docs/research/README.md @@ -73,3 +73,4 @@ Findings from building and running this pipeline, each kept with its evidence. C | 45 | 2026-11-09 | Both | XNNPACK and Vulkan `.pte` files hold their int4 weights in a form that can be read back and compared exactly: XNNPACK as `qbint4` values (unsigned nibbles around 8, low nibble first, bf16 scales) behind each fully-connected node's filter id; Vulkan as `linear_dq8ca_q4gsw` arguments (uint8 [out, in/2], fp32 scales [groups, out], int32 per-group code sums). Both serialise linears in execution order. Matched against fp32 sources, a published round-to-nearest file agrees with plain rounding on 99.0 to 99.2 percent of codes with every scale at amax/7.5; a GPTQ file on 67 to 87 percent, with 9 to 36 percent of scales at amax/7.5. | measured | SmolLM2-135M's published files; the rollout's byte checks | Byte checks before publish and after. | | 46 | 2026-11-09 | MediaTek | Gemma 3's sliding layers rotate at base 10000 (`rope_local_base_freq`) and its global ones at 1e6, but MediaTek's `Gemma3ModelChunk` takes one RoPE table for every layer: its master table is built at 10000 and its runner sample sets `rot_emb_base` 1e6. In fp32 against Hugging Face (google/gemma-3-1b-it, 900-token passage and the decision gate's prompts), 10000 everywhere is at KL 0.177, 88% top-1, perplexity 10.77 against 9.58, gate 6/11; 1e6 everywhere at KL 1.42, 52% top-1, 22.7, gate 5/11; a table per layer type is exact (KL 0, 100%, 9.575, 11/11). Gemma 2 (one base, 10000) is exact as MediaTek ships it, with softcapping, up to its 4,096-token sliding window. | measured | [`mtk-gemma-vs-hf.py`](evidence/mtk-gemma-vs-hf.py), [`mtk-gemma-fp32-vs-hf.log`](evidence/mtk-gemma-fp32-vs-hf.log) | `mediatek-gemma.patch` gives the chunk a second input, `pos_emb_local`, built at the local base, and `export_mtk` writes `rot_emb_local_base` and `rot_emb_inputs` for the runner, which has to feed both tables (31 inputs per method). Published for MT6991 with that caveat on the card: MediaTek's 1.5.1 runner feeds one table. | | 47 | 2026-11-09 | MediaTek | `mtk_converter` 8.13.0 holds a delegated graph's weights as fp32 in one protobuf `ModelProto`, which protobuf caps at 2 GB, and the output head sits whole in the last chunk. Gemma 2 2B's tied 256000 x 2304 head is 2,359,296,000 B: the export fails in the converter at 2 chunks and at 13 (two layers each; 2361344595 B), and the head alone (final norm + head, A16W8, MT6991) fails plain, with `convert_float32_weights_to_float16` and with `convert_pt2e(fold_quantize=True)`: "exceeds maximum protobuf size of 2GB: 2361344528" (2361344648 folded). Gemma 3 1B's 262144 x 1152 head is 1.21 GB and builds. | measured | [`mtk-gemma2-head-probe.py`](evidence/mtk-gemma2-head-probe.py), [`mtk-gemma2-head-probe.log`](evidence/mtk-gemma2-head-probe.log) | `families.mtk_plan` refuses a head of 2 GB or more in fp32 (`mtk_head_bytes`), so Gemma 2 has no backend. A way through would split the head across separately delegated graphs and join the halves on the CPU, which the runtime would have to provide; not attempted. | +| 48 | 2026-11-14 | Runtime | ExecuTorch 1.6.0 (torch 2.15.0) against 1.5.1, on the Python runtime (macOS arm64). **Published files:** all 368 `.pte` files parse under 1.6.0's program schema, among them 145 MediaTek chunks built with 1.4.0, which the host cannot run. All 20 published CPU files (one 2k window per repo, five of them 1.4.0 builds) load and run on 1.6.0 with the same methods and metadata. The int4 files' outputs move (KL 0.0025 to 0.064 on a probe; gate means by up to 0.090, Phi-4-mini, and 0.054 for the rest), the fp32 ones' do not. Of the 16 with an archived gate reference, 15 keep 1.5.1's verdict and agreed count on 1.6.0, and Phi-4-mini keeps one more row (8/8 against 7/8). Qwen2.5-Math-1.5B fails the smoke test on 1.6.0 ("parallelogram … lyle" for the capital of France) while its gate passes 3/3: at the answer its top tokens have probabilities of 0.03 to 0.05 on both runtimes, and an eighth-token split (0.256 against 0.240) goes the other way. **1.4.0 builds:** their first token on today's gate prompts matches the 1.4.0 runtime's on 79 of 80 rows on 1.6.0 (78 on 1.5.1); on the other row the top two first tokens are 0.18 apart on 1.4.0 and 0.07 on 1.6.0, neither a tool call. **New 1.6.0 exports** from the same codes: SmolLM2-135M, Qwen2.5-0.5B, Qwen2.5-Math-1.5B, Qwen3-0.6B (fp32) and Llama-3.2-1B pass their gates, LFM3-1.7B its gate with the conv state cleared at position 0; the SmolLM2 Vulkan file passed only its structural check and gate proxy, the rounded LFM2.5-1.2B only its smoke test; the int4 codes are byte-exact in every GPTQ one. Against a 1.5.1 build on the same host, only the two RoPE tables (by 1.19e-7), the program's value table and the XNNPACK blobs (27 KB smaller in all) differ. A 1.6.0 file loads and passes its gate on the 1.5.1 runtime (SmolLM2-135M). **Cannot:** build for MediaTek (no cp310 wheel) or for the apps' QNN runtime (1.6.0's Qualcomm scripts compile with QAIRT 2.51; the apps ship `qnn-runtime` 2.39). | measured + source | [`executorch-1.6.0-regression.md`](evidence/executorch-1.6.0-regression.md) and the `et160-*.py` probes beside it; executorch 1.6.0's PyPI metadata (Requires-Python >= 3.11, torch==2.15.0); ExecuTorch v1.6.0 `backends/qualcomm/scripts/qnn_config.sh` | XNNPACK and Vulkan export with 1.6.0; QNN and MediaTek stay on 1.5.1 (`QNN_EXECUTORCH_VERSION`, `MTK_EXECUTORCH_VERSION`); `APP_RUNTIME_VERSION` 1.5.1 holds every 1.6.0 file back from publishing until both apps ship 1.6.0. Not measured: Linux (Modal image, solve on torch 2.15, the x86 C++ runner), a GPU or phone, the apps' own runtime builds, Phi-4-mini's export. The smoke test requires "paris" of the Math model, whose reply on these runs turned on an eighth-token split. | diff --git a/docs/research/evidence/et160-constdiff.py b/docs/research/evidence/et160-constdiff.py new file mode 100644 index 0000000..ae797a3 --- /dev/null +++ b/docs/research/evidence/et160-constdiff.py @@ -0,0 +1,35 @@ +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "tools/codecheck")) +import numpy as np +from ptecodes import load_pte + +a_pte, a = load_pte(Path(sys.argv[1])) +b_pte, b = load_pte(Path(sys.argv[2])) +print("named keys", len(a), len(b), "same key set:", set(a) == set(b)) +diff = [k for k in a if k in b and a[k] != b[k]] +print("differing entries:", len(diff), "of", len(a)) +for k in diff[:12]: + x, y = a[k], b[k] + note = "" + if len(x) == len(y) and len(x) % 4 == 0: + fx, fy = np.frombuffer(x, np.float32), np.frombuffer(y, np.float32) + if np.isfinite(fx).all() and np.isfinite(fy).all(): + note = f"as f32: max|d|={np.abs(fx - fy).max():.3g} max|x|={np.abs(fx).max():.3g}" + nbytes = sum(1 for i in range(min(len(x), len(y))) if x[i] != y[i]) if len(x) < 5_000_000 else "big" + print(" ", k[:70], len(x), len(y), "bytes differing:", nbytes, note) +# constant buffer in program +ca = [bytes(t.storage) for t in a_pte.program.constant_buffer] if a_pte.program.constant_buffer else [] +cb = [bytes(t.storage) for t in b_pte.program.constant_buffer] if b_pte.program.constant_buffer else [] +print("program constant_buffer entries", len(ca), len(cb), "equal:", ca == cb) +seg_a = getattr(a_pte, "constant_data", None) +for i, (x, y) in enumerate(zip(ca, cb)): + if x != y: + note = "" + if len(x) == len(y) and len(x) % 4 == 0: + fx, fy = np.frombuffer(x, np.float32), np.frombuffer(y, np.float32) + note = f"f32 max|d|={np.nanmax(np.abs(fx - fy)):.3g} max|x|={np.nanmax(np.abs(fx)):.3g}" + ix, iy = np.frombuffer(x, np.int8), np.frombuffer(y, np.int8) + note += f" int8 differing={int((ix != iy).sum())}/{len(ix)}" + print(" const", i, len(x), len(y), note) diff --git a/docs/research/evidence/et160-decide.py b/docs/research/evidence/et160-decide.py new file mode 100644 index 0000000..7df7d65 --- /dev/null +++ b/docs/research/evidence/et160-decide.py @@ -0,0 +1,41 @@ +"""Per gate row: the next-token decision of one .pte on this interpreter's runtime. + + python -I decide.py +rows.json: {question: [token ids]}. Reloads the runner per row, as pipeline/gate.py does. +""" + +import json +import sys +import tempfile +import types +from importlib import metadata +from pathlib import Path + +import torch +from executorch.examples.models.llama.runner.native import NativeLlamaRunner +from executorch.runtime import Runtime + +pte, tok, rows_path, out = sys.argv[1:5] +rows = json.loads(Path(rows_path).read_text()) +vocab = Runtime.get().load_program(pte).load_method("get_vocab_size").execute([])[0] +params = Path(tempfile.mkdtemp()) / "params.json" +params.write_text(json.dumps({"vocab_size": int(vocab)})) +res = {"executorch": metadata.version("executorch"), "rows": {}} +for q, ids in rows.items(): + runner = NativeLlamaRunner( + types.SimpleNamespace( + pte=pte, params=str(params), tokenizer=tok, tokenizer_config=None, max_len=2048, kv_cache=True + ) + ) + logits = runner.forward(torch.tensor([ids], dtype=torch.long), torch.tensor([0], dtype=torch.long)) + last = (logits[0, -1] if logits.dim() == 3 else logits[-1]).float() + p = torch.softmax(last, -1) + top = torch.topk(p, 3) + res["rows"][q] = { + "top": int(top.indices[0]), + "p": float(top.values[0]), + "top3": [int(i) for i in top.indices], + "p3": [round(float(v), 5) for v in top.values], + } +Path(out).write_text(json.dumps(res, indent=1)) +print("ok", len(rows), "rows on", res["executorch"]) diff --git a/docs/research/evidence/et160-ptecheck.py b/docs/research/evidence/et160-ptecheck.py new file mode 100644 index 0000000..86be3ef --- /dev/null +++ b/docs/research/evidence/et160-ptecheck.py @@ -0,0 +1,88 @@ +"""Run one XNNPACK .pte on this interpreter's ExecuTorch runtime and record what it does. + + python -I ptecheck.py [--decode N] + +Loads through NativeLlamaRunner (as pipeline/gate.py does, so the custom ops are registered), +prefills a fixed prompt from position 0, then decodes greedily one token at a time. Records the +program's methods and metadata, the prefill logits (as .npy beside out.json), and the decoded +ids, so two runtimes can be compared on the same file. +""" + +import json +import sys +import tempfile +import time +import types +from importlib import metadata +from pathlib import Path + +import numpy as np +import torch + +PROMPT = ( + "The history of the printing press begins in the fifteenth century, when Johannes Gutenberg " + "combined movable metal type with an oil-based ink and a screw press. Explain in two sentences " + "why this mattered:" +) + + +def main(): + pte, tok_path, out = Path(sys.argv[1]), Path(sys.argv[2]), Path(sys.argv[3]) + decode = int(sys.argv[sys.argv.index("--decode") + 1]) if "--decode" in sys.argv else 24 + from executorch.examples.models.llama.runner.native import NativeLlamaRunner + from executorch.runtime import Runtime + from tokenizers import Tokenizer + + record = {"executorch": metadata.version("executorch"), "pte": str(pte)} + started = time.time() + try: + program = Runtime.get().load_program(str(pte)) + record["methods"] = sorted(program.method_names) + meta = {} + for name in record["methods"]: + if name in ("forward",): + continue + try: + values = program.load_method(name).execute([]) + meta[name] = [v.tolist() if hasattr(v, "tolist") else v for v in values] + except Exception as error: # noqa: BLE001 + meta[name] = f"error: {error}" + record["metadata"] = meta + del program + + ids = Tokenizer.from_file(str(tok_path)).encode(PROMPT, add_special_tokens=False).ids + vocab = meta.get("get_vocab_size", [None])[0] + params = Path(tempfile.mkdtemp()) / "params.json" + params.write_text(json.dumps({"vocab_size": vocab})) + runner = NativeLlamaRunner( + types.SimpleNamespace( + pte=str(pte), + params=str(params), + tokenizer=str(tok_path), + tokenizer_config=None, + max_len=2048, + kv_cache=True, + ) + ) + logits = runner.forward(torch.tensor([ids], dtype=torch.long), torch.tensor([0], dtype=torch.long)) + last = (logits[0, -1] if logits.dim() == 3 else logits[-1]).float() + np.save(out.with_suffix(".npy"), last.numpy()) + generated = [int(last.argmax())] + pos = len(ids) + for _ in range(decode - 1): + step = runner.forward( + torch.tensor([[generated[-1]]], dtype=torch.long), torch.tensor([pos], dtype=torch.long) + ) + last = (step[0, -1] if step.dim() == 3 else step[-1]).float() + generated.append(int(last.argmax())) + pos += 1 + record.update(prompt_tokens=len(ids), generated=generated, ok=True) + except Exception as error: # noqa: BLE001 + record.update(ok=False, error=f"{type(error).__name__}: {error}") + record["seconds"] = round(time.time() - started, 1) + out.write_text(json.dumps(record, indent=1)) + print(json.dumps({k: record.get(k) for k in ("ok", "error", "seconds", "prompt_tokens")})) + + +if __name__ == "__main__": + main() diff --git a/docs/research/evidence/et160-reset-check.py b/docs/research/evidence/et160-reset-check.py new file mode 100644 index 0000000..7109f2d --- /dev/null +++ b/docs/research/evidence/et160-reset-check.py @@ -0,0 +1,58 @@ +"""LFM2 conv state: does a prompt at position 0 read zeros whatever the previous prompt left? + + python -I reset_check.py +Prompt B after prompt A in one runner, against prompt B in a fresh one: equal logits mean the +state was cleared in the graph (pipeline/lfm2_state.py). +""" + +import json +import sys +import tempfile +import types +from pathlib import Path + +import torch +from executorch.examples.models.llama.runner.native import NativeLlamaRunner +from executorch.runtime import Runtime +from tokenizers import Tokenizer + +pte, tok = sys.argv[1], sys.argv[2] +prog = Runtime.get().load_program(pte) +methods = sorted(prog.method_names) +reset = prog.load_method("get_state_reset_at_zero").execute([])[0] if "get_state_reset_at_zero" in methods else None +vocab = int(prog.load_method("get_vocab_size").execute([])[0]) +del prog +params = Path(tempfile.mkdtemp()) / "p.json" +params.write_text(json.dumps({"vocab_size": vocab})) +mk = lambda: NativeLlamaRunner( + types.SimpleNamespace( + pte=pte, params=str(params), tokenizer=tok, tokenizer_config=None, max_len=2048, kv_cache=True + ) +) +t = Tokenizer.from_file(tok) +a = t.encode("Write a haiku about the sea and the moon over the harbor.", add_special_tokens=False).ids +b = t.encode("What is the capital of Italy? Answer in one word.", add_special_tokens=False).ids + + +def f(r, ids): + out = r.forward(torch.tensor([ids]), torch.tensor([0])) + last = out[0, -1] if out.dim() == 3 else out[-1] # [1, vocab] when only the last position comes back + assert last.dim() == 1 and last.numel() == vocab, tuple(out.shape) + return last.float() + + +r = mk() +f(r, a) +after_a = f(r, b) +fresh = f(mk(), b) +print( + json.dumps( + { + "get_state_reset_at_zero": reset if reset is None else bool(reset), + "max_abs_diff_b_after_a_vs_fresh": float((after_a - fresh).abs().max()), + "top_after_a": int(after_a.argmax()), + "top_fresh": int(fresh.argmax()), + "a_alone_vs_b_fresh": float((f(mk(), a) - fresh).abs().max()), + } + ) +) diff --git a/docs/research/evidence/et160-static-parse.py b/docs/research/evidence/et160-static-parse.py new file mode 100644 index 0000000..e3b80c5 --- /dev/null +++ b/docs/research/evidence/et160-static-parse.py @@ -0,0 +1,71 @@ +"""Static back-compat: parse every published .pte's program with this interpreter's ExecuTorch schema. + +Range-requests the header, then the program flatbuffer only (segments are not fetched), and +parses it with exir._serialize._program._flatbuffer_to_program. Records methods, delegate ids, +delegate calls per method, and whether any instruction or value type is unknown. +""" + +import json +import sys +from collections import Counter +from importlib import metadata +from pathlib import Path + +import requests +from executorch.exir._serialize._program import _flatbuffer_to_program, _get_extended_header +from huggingface_hub import hf_hub_url +from huggingface_hub.utils import build_hf_headers + +rows = json.loads(Path(sys.argv[1]).read_text()) +out = Path(sys.argv[2]) +only = sys.argv[3] if len(sys.argv) > 3 else None +headers = build_hf_headers() +results = json.loads(out.read_text()) if out.exists() else {} +sess = requests.Session() + + +def get(url, a, b): + h = dict(headers, Range=f"bytes={a}-{b}") + r = sess.get(url, headers=h, allow_redirects=True, timeout=600) + r.raise_for_status() + return r.content + + +for r in rows: + key = f"{r['repo']}/{r['path']}" + if key in results or (only and not r["path"].startswith(only)): + continue + url = hf_hub_url(r["repo"], r["path"], revision=r["sha"]) + rec = {"bytes": r["bytes"]} + try: + head = get(url, 0, 63) + rec["file_identifier"] = head[4:8].decode("ascii", "replace") + eh = _get_extended_header(head) + size = eh.program_size if eh and eh.is_valid() else r["bytes"] + rec["program_size"] = size + if size > 200 * 2**20: + raise ValueError(f"program flatbuffer is {size} bytes; not fetched") + prog = _flatbuffer_to_program(get(url, 0, size - 1)) + plans = {} + for plan in prog.execution_plan: + instr = Counter(type(i.instr_args).__name__ for c in plan.chains for i in c.instructions) + plans[plan.name] = { + "delegates": sorted({d.id for d in plan.delegates}), + "delegate_calls": instr.get("DelegateCall", 0), + "instructions": dict(instr), + "values": dict(Counter(type(v.val).__name__ for v in plan.values)), + } + rec.update(ok=True, version=prog.version, methods=sorted(plans), plans=plans, segments=len(prog.segments)) + except Exception as e: + rec.update(ok=False, error=f"{type(e).__name__}: {e}"[:500]) + results[key] = rec + out.write_text(json.dumps(results, indent=0)) + print(("ok " if rec.get("ok") else "FAIL") + f" {key} {rec.get('error', '')}", flush=True) +print( + "DONE", + metadata.version("executorch"), + sum(1 for v in results.values() if v.get("ok")), + "of", + len(results), + flush=True, +) diff --git a/docs/research/evidence/et160-structure.py b/docs/research/evidence/et160-structure.py new file mode 100644 index 0000000..303d9a6 --- /dev/null +++ b/docs/research/evidence/et160-structure.py @@ -0,0 +1,46 @@ +"""Structure of an XNNPACK .pte: operators, instructions, delegate calls, XNNPACK node and value types.""" + +import json +import sys +from collections import Counter +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "tools/codecheck")) +from ptecodes import ET, _flatc_json, delegates, load_pte # noqa: E402 + +pte, named = load_pte(Path(sys.argv[1])) +prog = pte.program +plan = next(p for p in prog.execution_plan if p.name == "forward") +ops = [o.name + ("." + o.overload if o.overload else "") for o in plan.operators] +instr = Counter(type(i.instr_args).__name__ for c in plan.chains for i in c.instructions) +kernels = Counter( + ops[i.instr_args.op_index] + for c in plan.chains + for i in c.instructions + if type(i.instr_args).__name__ == "KernelCall" +) +nodes, values, sizes = Counter(), Counter(), [] +schema = ET / "backends/xnnpack/serialization/schema.fbs" +for name, did, blob in delegates(pte): + if name != "forward" or did != "XnnpackBackend": + continue + from executorch.backends.xnnpack.serialization.xnnpack_graph_serialize import XNNHeader + + h = XNNHeader.from_bytes(blob[: XNNHeader.EXPECTED_LENGTH]) + g = _flatc_json(schema, blob[h.flatbuffer_offset : h.flatbuffer_offset + h.flatbuffer_size]) + sizes.append(len(blob)) + nodes.update(n.get("xnode_union_type") for n in g.get("xnodes", [])) + values.update(v.get("xvalue_union_type") for v in g.get("xvalues", [])) +out = { + "operators": sorted(set(ops)), + "instructions": dict(instr), + "kernel_calls": dict(kernels), + "xnnpack_delegates": len(sizes), + "xnnpack_blob_bytes": sum(sizes), + "xnnpack_nodes": dict(nodes), + "xnnpack_values": dict(values), + "named_data": len(named), + "named_bytes": sum(len(v) for v in named.values()), + "values": dict(Counter(type(v.val).__name__ for v in plan.values)), +} +print(json.dumps(out, indent=1, sort_keys=True)) diff --git a/docs/research/evidence/executorch-1.6.0-regression.md b/docs/research/evidence/executorch-1.6.0-regression.md new file mode 100644 index 0000000..ce4d461 --- /dev/null +++ b/docs/research/evidence/executorch-1.6.0-regression.md @@ -0,0 +1,163 @@ +# ExecuTorch 1.6.0: regression against 1.5.1 (2026-11-13/14) + +What was measured before moving the CPU (XNNPACK) and GPU (Vulkan) exporter to ExecuTorch 1.6.0 +(torch 2.15.0, torchao 0.18.0, pytorch-tokenizers 1.6.0). Host: macOS arm64 (Apple M5), the +executorch PyPI wheels' Python runtime in Python 3.11 virtualenvs: 1.4.0 (torch 2.13.0), 1.5.1 +(torch 2.14.0) and 1.6.0 (torch 2.15.0). Gate references are the fp32 `gate.json` files the solves +wrote on the Linux x86 boxes on 2026-11-09 (a Mac's fp32 reference is wrong, finding 44). The +probe scripts are beside this file (`et160-*.py`); only their import paths and formatting were +changed from what ran. + +Not covered: Linux (the Modal image's install, the solve on torch 2.15, the x86 C++ runner of the +verify job), any GPU or phone, the apps' own Android builds of the runtime, and Phi-4-mini's export +(its 3.8B fp32 conversion does not fit this host). QNN and MediaTek keep the 1.5.1 toolchain, so +their export is unchanged. + +A correction made before this was written up: the first pass of the gate comparison handed +`verify` the published report, whose recorded gate was against the same reference file, so for 12 +of the 16 files `verify` answered "already verified" with the published verdict and measured +nothing. Every gate below was measured: the report's recorded smoke test and gate are removed +before each run, and each run's log shows the gate executing (no "already verified"). + +## Every published file parses under 1.6.0's program schema + +`et160-static-parse.py`: the header and program flatbuffer of all 368 `.pte` files in the +experimentalmachines org (100 XNNPACK, 65 Vulkan, 23 QNN, 178 MediaTek chunks, 2 others), fetched +by range request and parsed with 1.6.0's `exir._serialize._program._flatbuffer_to_program`: 368 of +368 parse, 66,927 kernel calls and 34,825 delegate calls in all. This checks the program's +structure against 1.6.0's schema, not that any delegate runs. + +## Published CPU files on the 1.5.1 and 1.6.0 runtimes + +One file per repo, its 2k window, from the Hub. Probe (`et160-ptecheck.py`): a fixed raw-text +prompt (no chat template) prefilled at position 0, then 24 greedy tokens; KL is the next-token +distribution after the prefill, 1.5.1 runtime against 1.6.0. Gate: `python -m pipeline verify +--gate` with the run's archived fp32 reference, both runtimes the same day; verdict, +agreed/graded, and the export's mean probability of fp32's token. "Gate as published" is the +verdict recorded at publication (x86 Linux, 1.5.1 or 1.4.0 runtime). + +| Repo | Built with | Linears | Loads on 1.5.1 / 1.6.0 | Same methods and metadata | Greedy tokens equal before the first difference (of 24) | KL | Gate as published | Gate, 1.5.1 runtime | Gate, 1.6.0 runtime | +|---|---|---|---|---|---|---|---|---|---| +| SmolLM2-135M-Instruct | 1.5.1 | int4 GPTQ | yes / yes | yes | 24 | 0.0140 | pass 4/4 0.5078 | pass 4/4 0.5008 | pass 4/4 0.5218 | +| Qwen2.5-0.5B-Instruct | 1.5.1 | int4 GPTQ | yes / yes | yes | 0 | 0.0531 | pass 9/9 0.6564 | pass 9/9 0.6667 | pass 9/9 0.7202 | +| LFM2.5-1.2B-Instruct | 1.4.0 | int4 GPTQ | yes / yes | yes | 14 | 0.0180 | no archived reference | | | +| LFM2.5-1.2B-Instruct-heretic | 1.4.0 | int4 GPTQ | yes / yes | yes | 24 | 0.0078 | no archived reference | | | +| Llama-3.2-1B-Instruct | 1.5.1 | int4 GPTQ | yes / yes | yes | 22 | 0.0025 | pass 12/12 0.8293 | pass 12/12 0.7518 | pass 12/12 0.7333 | +| Qwen2.5-1.5B-Instruct | 1.5.1 | int4 GPTQ | yes / yes | yes | 0 | 0.0566 | pass 12/12 0.7435 | pass 12/12 0.8599 | pass 12/12 0.8786 | +| Qwen2.5-Math-1.5B-Instruct | 1.5.1 | int4 GPTQ | yes / yes | yes | 15 | 0.0118 | pass 3/3 0.7408 | pass 3/3 0.7169 | pass 3/3 0.7427, smoke FAIL | +| LFM3-1.7B | 1.5.1 | int4 GPTQ | yes / yes | yes | 0 | 0.0098 | pass 12/12 0.9991 | pass 12/12 0.998 | pass 12/12 0.9992 | +| LFM3-1.7B-Base | 1.5.1 | int4 GPTQ | yes / yes | yes | 24 | 0.0033 | pass 12/12 0.9786 | pass 12/12 0.9799 | pass 12/12 0.9832 | +| Qwen3-1.7B | 1.5.1 | int4 GPTQ | yes / yes | yes | 0 | 0.0315 | pass 12/12 0.9946 | pass 12/12 0.9897 | pass 12/12 0.9745 | +| SmolLM2-360M-Instruct | 1.4.0 | fp32 | yes / yes | yes | 24 | 0.0000 | pass 2/2 0.5673 | pass 2/2 0.5674 | pass 2/2 0.5674 | +| LFM2.5-2.6B | 1.4.0 | int4 GPTQ | yes / yes | yes | 18 | 0.0152 | no archived reference | | | +| LFM2.5-2.6B-heretic | 1.4.0 | int4 GPTQ | yes / yes | yes | 24 | 0.0187 | no archived reference | | | +| Qwen2.5-3B-Instruct | 1.5.1 | int4 GPTQ | yes / yes | yes | 2 | 0.0148 | pass 12/12 0.9968 | pass 12/12 0.9966 | pass 12/12 0.9968 | +| Llama-3.2-3B-Instruct | 1.5.1 | int4 GPTQ | yes / yes | yes | 17 | 0.0095 | pass 12/12 0.9111 | pass 12/12 0.8796 | pass 12/12 0.8633 | +| Qwen3-0.6B | 1.5.1 | fp32 | yes / yes | yes | 24 | 0.0000 | pass 4/4 0.6177 | pass 4/4 0.6178 | pass 4/4 0.6178 | +| Qwen3-4B-Thinking-2507 | 1.5.1 | int4 GPTQ | yes / yes | yes | 7 | 0.0167 | pass 12/12 0.9838 | pass 12/12 0.9813 | pass 12/12 0.987 | +| Qwen3-4B-Instruct-2507 | 1.5.1 | int4 GPTQ | yes / yes | yes | 0 | 0.0644 | pass 12/12 0.9954 | pass 12/12 0.9942 | pass 12/12 0.9956 | +| Qwen3-4B | 1.5.1 | int4 GPTQ | yes / yes | yes | 6 | 0.0223 | pass 12/12 0.9999 | pass 12/12 0.9999 | pass 12/12 0.9999 | +| Phi-4-mini-instruct-3.8B | 1.5.1 | int4 GPTQ | yes / yes | yes | 13 | 0.0226 | FAIL 7/8 0.5066 (published with an override) | FAIL 7/8 0.4275 | pass 8/8 0.5179 | + +- The int4 files' outputs move between the runtimes (KL 0.0025 to 0.064 on the probe; export means + by up to 0.090 on the gate, Phi-4-mini, and 0.054 for the rest); the two fp32 files' do not (KL + under 1e-7). +- Verdicts and agreed counts on 1.6.0 equal those on 1.5.1 for 15 of the 16 gated files. Phi-4-mini + keeps fp32's choice on all 8 rows on 1.6.0 and on 7 on 1.5.1. +- Llama 3.2's chat template carries the date, so its rows today are not the ones its published gate + read; both runtimes here read the same ones. +- **Qwen2.5-Math-1.5B fails the smoke test on 1.6.0.** Asked the capital of France, it answers "The + capital of France is ** parallelogram** … **lyle**", where on 1.5.1 it answers "** parallele** … + **paris**" and at publication (x86) answered "\"paris\"". `verify` stops at the smoke test, so its + gate on 1.6.0 was measured on its own (`gate.measure` and `gate.check`): pass 3/3 0.7427. Replayed + step by step (`replay.py` in the session's tools, not committed), the two runtimes agree for seven + tokens; at the answer both put little weight on any token (" **" at 0.052 / 0.034, then + " paralle" at 0.026 / 0.039), and the eighth token splits "le" 0.293 over "log" 0.267 on 1.5.1 + against "log" 0.256 over "le" 0.240 on 1.6.0. The smoke test requires "paris" from every model + of 500 million parameters or more (`smoke.ANSWER_REQUIRED_FROM_PARAMS`). A 1.6.0 export of the + same model from the same codes (below) opens with "** parallelogram**" too and passes the smoke + test because its reply reaches "Paris" later. Only these two Mac runs were measured; how often + this model's greedy reply reaches "paris" elsewhere is not known. + +## Files built with 1.4.0, on the 1.4.0, 1.5.1 and 1.6.0 runtimes + +A 1.6.0 runtime is promised to load 1.5.1 files, not 1.4.0 ones, and five CPU files are still 1.4.0 +builds. `et160-decide.py` compares runtimes on each file's current gate prompts (`gate.prompts` +today, the model's chat template; the heretic finetunes' from their base model, their source repos +being gone): one forward from position 0 per row, the runner reloaded per row as the gate does, +and the next token compared with the 1.4.0 runtime's. These are not the prompts the files were +gated with at publication (`gate.prompts` has changed since: LFM2.5-2.6B's published gate has fp32 +at 0.9999 on the row that differs below, where today's prompt gives 0.38), so this is first-token +agreement between runtimes, not a re-run of the published gate. + +| File | Rows | Same token as on 1.4.0: 1.5.1 runtime | 1.6.0 runtime | Rows that differ | Largest change in the top token's probability (1.5.1 / 1.6.0) | +|---|---|---|---|---|---| +| xnnpack/LFM2.5-1.2B-Instruct-8da4w-gptq-2k.pte | 16 | 16/16 | 16/16 | none | 0.3293 / 0.2727 | +| xnnpack/LFM2.5-1.2B-Instruct-heretic-8da4w-gptq-2k.pte | 16 | 16/16 | 16/16 | none | 0.1886 / 0.1954 | +| xnnpack/LFM2.5-2.6B-8da4w-gptq-2k.pte | 16 | 15/16 | 15/16 | Who wrote Pride and Prejudice? | 0.0886 / 0.1041 | +| xnnpack/LFM2.5-2.6B-heretic-8da4w-gptq-2k.pte | 16 | 15/16 | 16/16 | Who wrote Pride and Prejudice? (1.5.1 only) | 0.0988 / 0.1019 | +| xnnpack/SmolLM2-360M-Instruct-fp32-2k.pte | 16 | 16/16 | 16/16 | none | 0.0 / 0.0 | + +On the differing row LFM2.5-2.6B's first token is "Pr" 0.384 over "The" 0.206 on 1.4.0, "The" +0.296 over "Pr" 0.248 on 1.5.1, and "The" 0.311 over "Pr" 0.242 on 1.6.0. The largest probability +move, LFM2.5-1.2B on "In what country is Gdynia?", is 0.954 on 1.4.0, 0.624 on 1.5.1 and 0.816 on +1.6.0, `<|tool_call_start|>` each time. + +MediaTek: 145 of the 178 published chunks (39 of 48 windows, in 12 repos) are 1.4.0 builds, by +their export reports. They parse (above); there is no host NeuroPilot runtime to run them. Between +1.5.1 and 1.6.0 only build files changed under `backends/mediatek` and `examples/mediatek`. + +## New exports with 1.6.0 + +`python -m pipeline export-xnnpack` / `export-vulkan` from this tree at the published source +revision, window 2k, with the archived GPTQ codes, then `verify --gate` with the archived reference +on the 1.6.0 runtime (these reports had no gate yet, so each was measured). "Codes" is +`tools/codecheck/codesmatch.py`: every int4 linear read back from the file's bytes against +codes.pt. + +| Model | Recipe | Bytes, 1.6.0 | Bytes, published (1.5.1, x86) | Codes | Checked by | Result, 1.6.0 export | Gate as published | +|---|---|---|---|---|---|---|---| +| SmolLM2-135M-Instruct | XNNPACK 8da4w GPTQ | 105,990,784 | 106,018,048 | 211/211 exact | gate | pass 4/4 0.5237 | pass 4/4 0.5078 | +| SmolLM2-135M-Instruct | Vulkan 8da4w GPTQ | 132,503,552 | 132,513,280 | 211/211 exact | gate proxy (the row above) and structure | `forward` is one `VulkanBackend` call; not run on a GPU | proxy | +| Qwen2.5-0.5B-Instruct | XNNPACK 8da4w GPTQ | 417,477,120 | 417,495,296 | 169/169 exact | gate | pass 9/9 0.6735 | pass 9/9 0.6564 | +| Qwen3-0.6B | XNNPACK fp32 linears | 2,545,152,640 | 2,545,178,112 | none (fp32) | gate | pass 4/4 0.6178 | pass 4/4 0.6177 | +| Llama-3.2-1B-Instruct | XNNPACK 8da4w GPTQ | 959,958,272 | 959,972,864 | 113/113 exact | gate | pass 12/12 0.8229 | pass 12/12 0.8293 | +| LFM3-1.7B | XNNPACK 8da4w GPTQ (`pipeline/lfm3_model.py`) | 1,239,187,968 | 1,239,210,240 | 180/180 exact (codesmatch has no layer order for LFM3, as before) | gate | pass 12/12 0.9994, quiet 3/3 | pass 12/12 0.9991 | +| LFM2.5-1.2B-Instruct | XNNPACK 8da4w rounded (no archived codes) | 795,678,720 | 795,686,784 (GPTQ, 1.4.0) | none | smoke only (no reference) | "Paris" | pass 12/12 0.8989 | +| Qwen2.5-Math-1.5B-Instruct | XNNPACK 8da4w GPTQ | 1,107,592,704 | 1,107,613,952 | 197/197 exact | gate | pass 3/3 0.7296; smoke passes, its reply opening "** parallelogram**" and reaching "**Paris**" 271 characters in | pass 3/3 0.7408 | + +Every file has the same methods and metadata as its published twin. The LFM files keep +`get_state_reset_at_zero`, and their graph clamps the position (`aten::clamp`, from +`pipeline/lfm2_state.py`), not 1.6.0's own `input_pos == 0` reset, which this repository's patch +still replaces. `et160-reset-check.py`: a second prompt after a first one in the same runner gives +exactly the logits of a fresh runner (largest difference 0.0, against 11.2 for LFM3 and 15.4 for +LFM2.5 between the two prompts' own logits). + +## The same model, two builds and two runtimes + +SmolLM2-135M-Instruct, 2k, the same codes and reference; export mean of fp32's token on the 4 +graded rows (fp32: 0.5834), every cell measured and passing 4/4: + +| File | 1.5.1 runtime | 1.6.0 runtime | +|---|---|---| +| Published, built with 1.5.1 on x86 | 0.5008 | 0.5218 | +| Built with 1.5.1 on this Mac | 0.5192 | 0.5177 | +| Built with 1.6.0 on this Mac | 0.4885 | 0.5237 | + +The build and the runtime each move the mean by up to about 0.035, and no cell changes a decision. +The two Mac builds (`et160-structure.py`, `et160-constdiff.py`): + +- Same operators, kernel calls (759) and XNNPACK nodes (304 delegate calls; 211 fully connected). + The 1.6.0 file is 27,264 B smaller: the program's value table (Int values 1,606 to 821, IntList + 421 to 242) and the XNNPACK delegate blobs (347,056 B to 346,576 B in all). +- All 422 named-data entries (the int4 codes and scales, the int8 embedding) are byte-identical. + The only constants that differ are the two RoPE tables (131,072 fp32 values each), by at most + 1.19e-7. `examples/models/llama/rope.py` is the same at both tags and torch moved from 2.14 to + 2.15; which of the two toolchains' parts makes the difference was not isolated. + +## A 1.6.0 file on the 1.5.1 runtime + +The apps ship 1.5.1, and ExecuTorch promises nothing for a newer file on an older runtime. The +1.6.0 SmolLM2-135M file loads on 1.5.1 with the same methods and metadata and passes its gate there +(the table above: 4/4, 0.4885 against 0.5237 on 1.6.0). One model on the Python runtime is not a +promise, so `publish` refuses 1.6.0 files until the apps ship 1.6.0 (`APP_RUNTIME_VERSION`). diff --git a/pipeline/__main__.py b/pipeline/__main__.py index 9427242..dffa519 100644 --- a/pipeline/__main__.py +++ b/pipeline/__main__.py @@ -61,7 +61,12 @@ def _verify(args) -> int: try: gate = Path(args.gate) if args.gate else None summary = verify_module.run( - Path(args.out), args.backend, Path(args.work), gate_reference=gate, accept_gate=args.accept_gate + Path(args.out), + args.backend, + Path(args.work), + gate_reference=gate, + accept_gate=args.accept_gate, + force=args.force, ) except ExportError as error: print(f"verify failed: {error}", file=sys.stderr) @@ -324,6 +329,21 @@ def sized(window: int, backend: str, tiers) -> dict | None: SOLVE_FLOOR_BYTES = 16 * 2**30 +def _publish_hold(args) -> int: + """Exit 1 when this backend's exports could not be published today (settings.publish_hold), + so a workflow asked to publish stops before it builds anything.""" + from pipeline import settings + + reason = settings.publish_hold(args.backend) + if reason is None: + print(f"{args.backend}: publishable") + return 0 + print( + f"::error title=Publishing is held::{reason}. Dispatch with publish_hf and publish_release off to build only." + ) + return 1 + + def _solve_memory(args) -> int: """MiB for the solve's Modal Sandbox, from the model's parameter count.""" from pipeline import hub, remote, settings @@ -502,6 +522,11 @@ def main(argv: list[str] | None = None) -> int: metavar="REASON", help="publish even if the gate fails, recording this reason beside the measured verdict", ) + verify.add_argument( + "--force", + action="store_true", + help="measure again even if the report was verified on this runtime against this reference", + ) verify.set_defaults(func=_verify) audit = commands.add_parser("audit", help="gate a file that is already published, without rebuilding it") @@ -617,6 +642,12 @@ def main(argv: list[str] | None = None) -> int: ) export_matrix.set_defaults(func=_export_matrix) + hold = commands.add_parser( + "publish-hold", help="exit 1 if this backend's exports are newer than the apps' runtime (APP_RUNTIME_VERSION)" + ) + hold.add_argument("--backend", required=True, choices=["xnnpack", "vulkan", "qnn", "mtk"]) + hold.set_defaults(func=_publish_hold) + solve_memory = commands.add_parser("solve-memory", help="MiB to request for the GPTQ solve on Modal") solve_memory.add_argument("model") solve_memory.add_argument("--revision", default="main") diff --git a/pipeline/export_mtk.py b/pipeline/export_mtk.py index 21ebfe5..9a4bdbf 100644 --- a/pipeline/export_mtk.py +++ b/pipeline/export_mtk.py @@ -1,11 +1,13 @@ """Export one HF model to MediaTek NeuroPilot .pte chunks for one Dimensity chip. -Drives ExecuTorch 1.4.0's examples/mediatek LLM scripts, which are not in the executorch -wheel (the workflow fetches them from the ExecuTorch source at EXECUTORCH_COMMIT), in a -separate Python 3.10 environment holding MediaTek's mtk_converter and mtk_neuron from the -NeuroPilot Express SDK. The script quantizes the model (A16W4), cuts it into chunks of equal -layer counts, and compiles a prompt graph and a one-token generation graph per chunk for the -chip. Produces, under ``out_dir``: +Drives ExecuTorch's examples/mediatek LLM scripts, which are not in the executorch wheel +(the workflow fetches them from the ExecuTorch source at EXECUTORCH_COMMIT), in a separate +Python 3.10 environment holding MediaTek's mtk_converter and mtk_neuron from the NeuroPilot +Express SDK. That environment's ExecuTorch is MTK_EXECUTORCH_VERSION (1.5.1), one release +behind the CPU and GPU backends, because 1.6.0 has no Python 3.10 wheel (config/versions.env). +The script quantizes the model (A16W4), cuts it into chunks of equal layer counts, and +compiles a prompt graph and a one-token generation graph per chunk for the chip. Produces, +under ``out_dir``: tokenizer.json | tokenizer.model, LICENSE…, NOTICE (repo root, shared with other backends) mtk//-neuropilot-a16w4--chunkof.pte diff --git a/pipeline/export_with_codes.py b/pipeline/export_with_codes.py index ffb36a1..40fb622 100644 --- a/pipeline/export_with_codes.py +++ b/pipeline/export_with_codes.py @@ -133,7 +133,7 @@ def main(argv: list[str] | None = None) -> int: parser.add_argument( "--lfm3", action="store_true", - help="build LFM3 params with pipeline/lfm3_model.py, the definition ExecuTorch 1.5.1 lacks", + help="build LFM3 params with pipeline/lfm3_model.py, the definition ExecuTorch lacks (1.5.1 and 1.6.0)", ) args = parser.parse_args(argv) diff --git a/pipeline/families.py b/pipeline/families.py index 9eac725..893182f 100644 --- a/pipeline/families.py +++ b/pipeline/families.py @@ -44,7 +44,7 @@ def supports(self, backend: str) -> bool: _NOT_IN_EXPORT_LLM = ( - "not in ExecuTorch 1.5.1 export_llm's model list; needs a validated export path " + "not in ExecuTorch 1.6.0 export_llm's model list; needs a validated export path " "(optimum-executorch or params support) before it is published" ) # MediaTek's runner masks attention with an additive -100, not -inf (MaskBuilder, and the @@ -113,7 +113,7 @@ def supports(self, backend: str) -> bool: {"xnnpack": _NOT_IN_EXPORT_LLM, "vulkan": _NOT_IN_EXPORT_LLM, "qnn": _GEMMA2_NO_QNN}, ), # Phi-4-mini: Llama's block with fused q/k/v and gate/up projections (pipeline/convert.py - # splits them), partial RoPE on 96 of 128 head dims, and LongRoPE. ExecuTorch 1.5.1 builds + # splits them), partial RoPE on 96 of 128 head dims, and LongRoPE. ExecuTorch 1.6.0 builds # it from the same transformer (examples/models/phi_4_mini). Family("phi4_mini", ("Phi3ForCausalLM",), {"qnn": _PHI4_NO_QNN, "mtk": _PHI4_NO_MTK}), Family( @@ -561,7 +561,7 @@ def xnnpack_plan(family: Family, config: dict) -> XnnpackPlan: def _phi4_mini_plan(c: dict, params: dict) -> XnnpackPlan: - """Phi-4-mini's params, as ExecuTorch 1.5.1's examples/models/phi_4_mini/config/config.json + """Phi-4-mini's params, as ExecuTorch 1.6.0's examples/models/phi_4_mini/config/config.json spells them, from the checkpoint's own config. LongRoPE (rope.hf_precompute_freqs_cis) divides the inverse frequencies by @@ -625,7 +625,7 @@ def _lfm3_plan(c: dict) -> XnnpackPlan: """LFM3's params for pipeline/lfm3_model.py, from the fields of ExecuTorch's ModelArgs. ``rope_theta`` is the global layers' and ``local_rope_theta`` the windowed layers' (the - field 1.5.1 declares for Gemma 3 and never reads); ``partial_rotary_factor`` applies to + field 1.5.1 and 1.6.0 declare for Gemma 3 and never read); ``partial_rotary_factor`` applies to the global layers only, as in the checkpoint's rope_parameters. The convolution width rides in ``attention_kwargs``, which upstream passes to attention constructors and ignores: without pipeline/lfm3_model.py the width-7 convolution weights do not fit diff --git a/pipeline/lfm2_state.py b/pipeline/lfm2_state.py index 0e3a962..9e2d5a6 100644 --- a/pipeline/lfm2_state.py +++ b/pipeline/lfm2_state.py @@ -15,6 +15,11 @@ An export carrying it advertises ``get_state_reset_at_zero``, which is how a host tells a fixed file from one that needs reopening between prompts. Upstream's own Qwen3.5 definition already does the equivalent for its recurrent state; only LFM2 was left without it. + +ExecuTorch 1.6.0 clears it too (ShortConv._maybe_reset_state, from ShortConvBlock's +input_pos), but by comparing: ``1 - (input_pos[0] == 0)``. This module still replaces both +forwards, so 1.6.0 builds the same graph as 1.5.1 did: the clamp above, which every delegate +implements, and the files already published keep their twin. """ import torch diff --git a/pipeline/lfm3_model.py b/pipeline/lfm3_model.py index bf63f86..10347c2 100644 --- a/pipeline/lfm3_model.py +++ b/pipeline/lfm3_model.py @@ -1,7 +1,8 @@ -"""LFM3 for ExecuTorch's ``export_llm``: the model definition ExecuTorch 1.5.1 does not have. +"""LFM3 for ExecuTorch's ``export_llm``: the model definition ExecuTorch does not have. LiquidAI's LFM3-1.7B (2026-10-23) is LFM2's hybrid grown three ways, none of which the -1.5.1 transformer can express (examples/models/llama/llama_transformer.py, model_args.py): +1.5.1 transformer can express (examples/models/llama/llama_transformer.py, model_args.py). +In 1.5.1: * A third layer type. Of 24 layers, 13 are gated short convolutions, 6 attend globally and 5 attend over a sliding window of 256 tokens. ``construct_transformer`` knows ``conv`` and @@ -15,6 +16,14 @@ ("sandwich" norms; 1.5.1 declares ``post_attention_norm`` and never reads it), and a width-7 convolution (``ShortConvBlock`` hard-codes 3). +1.6.0 grew three of these pieces: ``AttentionMHA`` masks ``sliding_attention`` layers to +``sliding_window`` and gives them a ``RingKVCache``, ``rope_parameters`` builds one RoPE per +layer type, and ``headwise_attn_output_gate`` is a per-head gate. It still cannot build LFM3: +``construct_transformer`` builds every ``ShortConvBlock`` at width 3, its only norms after an +operator are weightless or scale the input (not LFM3's weighted RMSNorm), and its +``RingKVCache`` is the one described below that a 2,048-token prefill does not fit. This +definition uses none of the new pieces, so on 1.6.0 it builds the graph it built on 1.5.1. + Everything else is LFM2's, and the parts are ExecuTorch's own where they exist: RMSNorm, FeedForward, the KV cache and SDPA modules that ``export_llm`` swaps for its custom ops, the ring-buffer bookkeeping it uses for Gemma 3. Reference: modeling_lfm3.py in the checkpoint diff --git a/pipeline/manifest.py b/pipeline/manifest.py index 47e0f95..08a62a7 100644 --- a/pipeline/manifest.py +++ b/pipeline/manifest.py @@ -4,7 +4,7 @@ import os -from pipeline import naming +from pipeline import naming, settings # Methods the app reads from config.json before download (openweights ExportConfig.kt), # plus the rest of the runtime's LLM metadata, as the .pte reports them. @@ -87,16 +87,55 @@ def _variants(report: dict) -> list[dict]: return variants +def _release_order(version) -> tuple: + """Sort key: releases in release order, after anything that is not one (a report from an + older pipeline, or none at all), which the card and config.json name as they found it.""" + try: + return (1, settings.release(version)) + except (TypeError, ValueError, AttributeError): + return (0, str(version)) + + +def runtime_versions(reports: list[dict]) -> list[str]: + """The ExecuTorch releases these reports' files were built with, oldest first.""" + found = {(r.get("toolchain") or {}).get("executorch") for r in reports} + return [str(v) for v in sorted(found, key=_release_order)] + + +def one_runtime_for_all(versions: list[str]) -> bool: + """Whether one runtime is promised every file: all real releases, within one minor of + each other (ExecuTorch promises a runtime its own release's files and the previous minor's).""" + try: + parsed = [settings.release(v).release + (0, 0) for v in versions] + except (TypeError, ValueError, AttributeError): + return False + oldest, newest = parsed[0], parsed[-1] + return oldest[0] == newest[0] and newest[1] - oldest[1] <= 1 + + def backend_config(reports: dict | list[dict]) -> dict: """config.json for one backend folder, in the variants form the app reads: one variant per exported window, smallest first, from that folder's reports.""" reports = [reports] if isinstance(reports, dict) else list(reports) reports.sort(key=lambda r: r["window"]["context"]) report = reports[-1] # the newest layout wins for the shared fields - variants = [variant for r in reports for variant in _variants(r)] + # Each variant names the release its own file was built with: windows are republished one + # at a time, so a folder can hold files from two releases (both apps ignore keys they do + # not know, and read the folder's version below). + variants = [ + {**variant, "runtime_version": (r.get("toolchain") or {}).get("executorch")} + for r in reports + for variant in _variants(r) + ] config = { "runtime": "executorch", - "runtime_version": report["toolchain"]["executorch"], + # The apps read this one version for every file in the folder (ExecuServe's + # CatalogExports) and warn when their runtime is older than it, or more than a minor + # newer. Name the newest release in the folder: an installed app that never updates is + # still warned about a file newer than its runtime, which is the failure ExecuTorch + # promises nothing for. A file too old for a newer app is in each variant's own + # runtime_version above, for an app that reads it. + "runtime_version": runtime_versions(reports)[-1], "backend": report["backend"], "target": report.get("target"), "tokenizer": report["tokenizer"], @@ -104,9 +143,10 @@ def backend_config(reports: dict | list[dict]) -> dict: "source_revision": report["source"]["sha"], "variants": variants, } - if report["toolchain"].get("qairt"): + qairt = (report.get("toolchain") or {}).get("qairt") + if qairt: # HTP context binaries only load on the QNN runtime they were compiled with. - config["qnn_sdk_version"] = report["toolchain"]["qairt"] + config["qnn_sdk_version"] = qairt if report["backend"] == "mtk": config["neuropilot_sdk"] = report["neuropilot"] return config @@ -161,7 +201,18 @@ def readme(repo_id: str, reports: list[dict], hub_tags: list[str], license_files # and a repo with both kinds promised it for the NPU files too). portable = any(r["backend"] in ("xnnpack", "vulkan") for r in reports) npu = any(r["backend"] not in ("xnnpack", "vulkan") for r in reports) - runtime = f"ExecuTorch {reports[0]['toolchain']['executorch']} runtime" + # The releases the CPU and GPU files were built with; the NPU files (built a release behind, + # config/versions.env) are not what this sentence covers. ExecuTorch promises a runtime + # loads its own release's files and the previous minor's, so one runtime is named only + # when every file is within that span. + built = runtime_versions([r for r in reports if r["backend"] in ("xnnpack", "vulkan")] or reports) + if one_runtime_for_all(built): + runtime = f"ExecuTorch {built[-1]} runtime" + else: + runtime = ( + f"ExecuTorch runtime from the release its file was built with ({', '.join(built)}: each " + "variant's `runtime_version` in `config.json`) or the next" + ) app = "[openweights](https://github.com/alpharomercoma/openweights) Android app" if portable and npu: use = ( @@ -323,7 +374,7 @@ def readme(repo_id: str, reports: list[dict], hub_tags: list[str], license_files out.append("See [`NOTICE`](NOTICE) for the attribution the license requires.") qnn = [r for r in reports if r["backend"] == "qnn"] if qnn: - version = qnn[0]["toolchain"].get("qairt") + version = (qnn[0].get("toolchain") or {}).get("qairt") out += [ "", f"The `qnn/` folders hold QNN HTP context binaries compiled with the Qualcomm AI Runtime " diff --git a/pipeline/publish.py b/pipeline/publish.py index ab31105..6ab02cf 100644 --- a/pipeline/publish.py +++ b/pipeline/publish.py @@ -120,6 +120,7 @@ def require_publishable(report: dict) -> None: Vulkan) is published on its smoke test, as before. """ name = report["files"][0]["path"] + require_app_runtime(report) smoke = report.get("smoke") if smoke is None: raise ValueError( @@ -139,6 +140,25 @@ def require_publishable(report: dict) -> None: raise ValueError(f"{name} has GPTQ codes and no decision gate: verify it with --gate before publishing") +def require_app_runtime(report: dict) -> None: + """Refuse a file built with a newer ExecuTorch than the oldest runtime the apps ship. + + The exporter can move ahead of the apps (config/versions.env: EXECUTORCH_VERSION 1.6.0 + while APP_RUNTIME_VERSION is 1.5.1), but ExecuTorch promises only that an older program + loads on a newer runtime. A newer file on the Hub is offered to an app that may load it and + misbehave, so it is held back until APP_RUNTIME_VERSION says both apps have moved. + """ + built = (report.get("toolchain") or {}).get("executorch") + app = settings.load().app_runtime_version + if not built: + raise ValueError(f"{report['files'][0]['path']} names no ExecuTorch release in its report") + if settings.newer_release(built, app): + raise ValueError( + f"{report['files'][0]['path']} was built with ExecuTorch {built}, newer than the apps' runtime " + f"{app} (APP_RUNTIME_VERSION in config/versions.env): hold it until both apps ship {built}" + ) + + def require_twin_on_hub( report: dict, by_path: dict[str, dict], existing: list[str], allow_unpublished_twin: bool = False ) -> None: diff --git a/pipeline/qnn_launch.py b/pipeline/qnn_launch.py index be6095e..64cf218 100644 --- a/pipeline/qnn_launch.py +++ b/pipeline/qnn_launch.py @@ -10,7 +10,8 @@ object cannot be interpreted as an integer"): every Qwen3 window failed this way on 2026-10-07. ExecuTorch main falls back to the end-of-text token when bos_id is None (examples/models/llama/evaluate/eager_eval.py), as lm_eval's own HFLM does; this is that -property, set before the script runs. Drop this module when the ExecuTorch pin has the fix. +property, set before the script runs. ExecuTorch 1.6.0 has the fix; drop this module when the +Qualcomm toolchain (QNN_EXECUTORCH_VERSION in config/versions.env, still 1.5.1) reaches it. """ from __future__ import annotations diff --git a/pipeline/settings.py b/pipeline/settings.py index 9487a21..d69f3bc 100644 --- a/pipeline/settings.py +++ b/pipeline/settings.py @@ -102,9 +102,59 @@ class Settings: qnn: QnnRecipe mtk: MtkRecipe executorch_version: str + # The oldest runtime the apps ship: nothing built with a newer release may be published. + app_runtime_version: str = "" modal: ModalSettings | None = None +def release(version: str): + """An ExecuTorch version, ordered as releases are: 1.6.0a0 < 1.6.0 < 1.6.1 < 1.10.0. + + packaging's Version (huggingface_hub's dependency, so in every environment here), which + refuses what is not a version at all rather than guessing. + """ + from packaging.version import Version + + return Version(version.strip()) + + +def newer_release(built: str, runtime: str) -> bool: + """Whether a file built with ``built`` is from a later release than ``runtime``. + + Compared on the release numbers only (base_version), so a pre-release or development build + of 1.6.0 counts as 1.6.0, newer than a 1.5.1 runtime, and a 1.5.1.post1 counts as 1.5.1. + """ + from packaging.version import Version + + return Version(release(built).base_version) > Version(release(runtime).base_version) + + +# Which ExecuTorch each backend's exporter builds with (config/versions.env). +EXPORTER_VERSION_KEYS = { + "xnnpack": "EXECUTORCH_VERSION", + "vulkan": "EXECUTORCH_VERSION", + "qnn": "QNN_EXECUTORCH_VERSION", + "mtk": "MTK_EXECUTORCH_VERSION", +} + + +def publish_hold(backend: str) -> str | None: + """Why ``backend``'s exports cannot be published today, or None when they can. + + A backend whose exporter is a later release than APP_RUNTIME_VERSION, the oldest runtime + the apps ship, builds files that publish.require_app_runtime would refuse: the workflow + stops before building them and the watcher leaves them pending. + """ + versions = read_env_file(CONFIG_DIR / "versions.env") + built, app = versions[EXPORTER_VERSION_KEYS[backend]], versions["APP_RUNTIME_VERSION"] + if newer_release(built, app): + return ( + f"{backend} exports with ExecuTorch {built}, newer than the apps' runtime {app} " + "(APP_RUNTIME_VERSION in config/versions.env); publishing is held until both apps ship it" + ) + return None + + def read_env_file(path: Path) -> dict[str, str]: values: dict[str, str] = {} for line in path.read_text(encoding="utf-8").splitlines(): @@ -186,6 +236,7 @@ def load() -> Settings: runner_tiers=runner_tiers(export["mtk"]), ), executorch_version=versions["EXECUTORCH_VERSION"], + app_runtime_version=versions["APP_RUNTIME_VERSION"], modal=ModalSettings( app=str(raw["modal"]["app"]), volume=str(raw["modal"]["volume"]), diff --git a/pipeline/verify.py b/pipeline/verify.py index 7ffbcdb..291c823 100644 --- a/pipeline/verify.py +++ b/pipeline/verify.py @@ -31,7 +31,12 @@ def reports(out_dir: Path, backend: str) -> list[Path]: def run( - out_dir: Path, backend: str, work_dir: Path, gate_reference: Path | None = None, accept_gate: str | None = None + out_dir: Path, + backend: str, + work_dir: Path, + gate_reference: Path | None = None, + accept_gate: str | None = None, + force: bool = False, ) -> dict: """Run the smoke test for every window in ``out_dir`` and write the results back. @@ -43,6 +48,11 @@ def run( """ results = [] gates: list[dict] = [] + # The runtime that runs the checks. A verdict holds for the runtime that measured it: the + # same file decides differently on another release's kernels (Qwen2.5-Math-1.5B passed its + # smoke test on 1.5.1 and failed it on 1.6.0, finding 48), so a report verified on another + # runtime, or with none recorded, is measured again. + runtime = exporting.toolchain()["executorch"] for path in reports(out_dir, backend): report = json.loads(path.read_text()) # Verified already, unless a gate is asked for that this report has not passed: a @@ -60,8 +70,9 @@ def run( # And only while the file is the one that was verified: a rebuilt .pte is a new file. listed = out_dir / report["files"][0]["path"] same = listed.is_file() and exporting.sha256(listed) == report["files"][0].get("sha256") - if report.get("smoke") is not None and gated and same: - print(f"==> {path.name}: already verified, leaving it") + same_runtime = "verified_with" in report and report["verified_with"] == runtime + if report.get("smoke") is not None and gated and same and same_runtime and not force: + print(f"==> {path.name}: already verified on ExecuTorch {runtime}, leaving it") results.append(report["smoke"]) continue @@ -89,6 +100,10 @@ def run( total_params=arch.total_params, ) report["smoke"] = result + report["verified_with"] = runtime + # A gate measured on another runtime does not describe this one: it is measured again + # below or, when no reference is given, dropped rather than kept beside a new smoke result. + report.pop("gate", None) report["measuring_gate"] = gate.measuring_gate(result.get("stats")) if report["measuring_gate"]["prefill_tok_per_sec"]: print( diff --git a/pipeline/watch.py b/pipeline/watch.py index d158c17..4161b42 100644 --- a/pipeline/watch.py +++ b/pipeline/watch.py @@ -26,7 +26,7 @@ from pathlib import Path from pipeline import eligibility, hub, naming -from pipeline.settings import Settings +from pipeline.settings import Settings, publish_hold STATE_VERSION = 1 # Backend → the workflow that exports it (one run covers every target chip). MediaTek @@ -203,8 +203,15 @@ def run( fetch: Callable[[str, str], hub.SourceModel] = hub.fetch, dispatcher: Callable[[str, str, str], None] = gh_dispatch, in_flight: Callable[[str], bool] = runs_in_flight, + held: Callable[[str], str | None] | None = None, ) -> dict: - """One watcher pass. Returns a summary; the caller saves ``summary["state"]``.""" + """One watcher pass. Returns a summary; the caller saves ``summary["state"]``. + + ``held`` (settings.publish_hold by default) names the backends whose exports could not be + published today; their stage is passed over and its models stay pending, so they go out + once the apps' runtime moves instead of being recorded as dispatched to a refused run. + """ + held = held or publish_hold state = load_state(state_path) stamp = now() summary = {"seeded": 0, "new": [], "backfilled": [], "dispatched": [], "failed": [], "state": None} @@ -256,6 +263,10 @@ def run( budget = max_dispatch by_first_seen = sorted(models.items(), key=lambda kv: kv[1].get("first_seen") or "") for backend in STAGES: + reason = held(backend) + if reason: + summary.setdefault("held", []).append(f"{backend}: {reason}") + continue # pending stays pending; the later stages need not wait for it queue = [ (model_id, entry) for model_id, entry in by_first_seen diff --git a/requirements/export-qnn.txt b/requirements/export-qnn.txt index 371a093..1c7a1f0 100644 --- a/requirements/export-qnn.txt +++ b/requirements/export-qnn.txt @@ -1,5 +1,6 @@ -# Qualcomm QNN export. Same core pins as export-xnnpack.txt; install torch and torchvision -# from the CPU index first. The executorch Linux x86_64 wheel ships the Qualcomm backend and downloads +# Qualcomm QNN export, on ExecuTorch 1.5.1 while the other backends build with 1.6.0: +# QNN_EXECUTORCH_VERSION in config/versions.env says why (1.6.0's QAIRT 2.51 binaries would +# outrun the apps' QNN runtime). Install torch and torchvision from the CPU index first. The executorch Linux x86_64 wheel ships the Qualcomm backend and downloads # the QAIRT SDK version it was built against on first import (config/versions.env). torch==2.14.0 torchao==0.18.0 diff --git a/requirements/export-xnnpack.txt b/requirements/export-xnnpack.txt index 4bf1254..450630c 100644 --- a/requirements/export-xnnpack.txt +++ b/requirements/export-xnnpack.txt @@ -1,10 +1,12 @@ # Install torch from the CPU index first (the PyPI Linux wheel pulls ~3 GB of CUDA libs): -# pip install torch==2.14.0 --index-url https://download.pytorch.org/whl/cpu -# ExecuTorch 1.5.1 is built against torch 2.14.0 (torch_pin.py) and torchao 0.18.0. -torch==2.14.0 +# pip install torch==2.15.0 --index-url https://download.pytorch.org/whl/cpu +# ExecuTorch 1.6.0 requires torch 2.15.0 exactly (its wheel's Requires-Dist; torch_pin.py) and +# torchao >= 0.18.0 (install_requirements.py TORCHAO_MIN_VERSION; 0.18.0 is still the newest). +# pytorch-tokenizers 1.6.0 is the release cut beside it (2026-10-30, Python >= 3.11). +torch==2.15.0 torchao==0.18.0 -executorch==1.5.1 -pytorch-tokenizers==1.5.0 +executorch==1.6.0 +pytorch-tokenizers==1.6.0 # executorch.extension.llm.runner imports transformers at package import time; the smoke # test needs that package for TextLLMRunner. 5.22.1, not 5.17.0: LFM3 has no transformers # implementation, and its remote code (configuration_lfm3.py) fails to load under 5.17.0, diff --git a/requirements/mtk-tools.txt b/requirements/mtk-tools.txt index 145db37..88cb6f5 100644 --- a/requirements/mtk-tools.txt +++ b/requirements/mtk-tools.txt @@ -4,6 +4,8 @@ # huggingface_hub below the 1.x the pipeline itself uses. Install torch and torchvision from # the CPU index first. mtk_converter and mtk_neuron come from the NeuroPilot Express SDK # archive at run time (config/versions.env), never from this repository. +# ExecuTorch 1.5.1, not EXECUTORCH_VERSION: 1.6.0 has no cp310 wheel (MTK_EXECUTORCH_VERSION +# in config/versions.env says why that is safe). torch and torchao are 1.5.1's. torch==2.14.0 torchao==0.18.0 executorch==1.5.1 diff --git a/tests/test_manifest.py b/tests/test_manifest.py index 4e0a9a4..cd208b8 100644 --- a/tests/test_manifest.py +++ b/tests/test_manifest.py @@ -37,6 +37,83 @@ def test_backend_config_is_in_the_form_the_app_reads(): assert "methods" not in variant["methods"] +def window(context, executorch, backend="xnnpack"): + r = report(backend=backend) + label = f"{context // 1024}k" + r["toolchain"] = {"executorch": executorch} + r["window"] = {**r["window"], "context": context} + r["files"] = [{**r["files"][0], "path": f"{backend}/Qwen3-1.7B-8da4w-{label}.pte"}] + return r + + +def test_a_folder_of_two_releases_names_each_file_and_the_newest(): + # Windows are republished one at a time: a 2k file rebuilt on 1.6.0 beside a 32k one still + # from 1.5.1. Each variant names its own release. The folder, all the apps read today, names + # the newest: an installed app on the older runtime is then warned about the newer file. + mixed = [window(2048, "1.6.0"), window(32768, "1.5.1")] + for reports in (mixed, list(reversed(mixed))): + config = manifest.backend_config(reports) + assert config["runtime_version"] == "1.6.0" + assert [(v["context"], v["runtime_version"]) for v in config["variants"]] == [(2048, "1.6.0"), (32768, "1.5.1")] + assert manifest.backend_config([window(2048, "1.4.0"), window(4096, "1.10.0")])["runtime_version"] == "1.10.0" + assert manifest.runtime_versions([window(2048, "1.10.0"), window(4096, "1.9.0"), window(8192, "1.10.0")]) == [ + "1.9.0", + "1.10.0", + ] + # A pre-release sorts before its release, whatever the report order. + pre = [window(2048, "1.6.0a0+git1234"), window(4096, "1.6.0")] + assert ( + manifest.runtime_versions(pre) == manifest.runtime_versions(list(reversed(pre))) == ["1.6.0a0+git1234", "1.6.0"] + ) + + +def test_a_report_without_a_release_is_named_as_found_not_a_crash(): + # Retained reports are regenerated into config.json and the card on every publish: one that + # names no release, or not one packaging reads, must not stop another backend's publish. + odd = [window(2048, "1.5.1"), window(4096, "main"), window(8192, "1")] + odd.append(window(16384, "1.5.1")) + odd[-1]["toolchain"] = {} + for missing in ("absent", None): + # The largest window sets the folder's shared fields: its toolchain may be gone too. + largest = window(32768, "1.5.1") + if missing == "absent": + del largest["toolchain"] + else: + largest["toolchain"] = None + config = manifest.backend_config([window(2048, "1.5.1"), largest]) + assert config["runtime_version"] == "1.5.1" and "qnn_sdk_version" not in config + assert [v["runtime_version"] for v in config["variants"]] == ["1.5.1", None] + assert manifest.runtime_versions(odd) == ["None", "main", "1", "1.5.1"] + assert manifest.backend_config(odd)["runtime_version"] == "1.5.1" + text = manifest.readme("experimentalmachines/Qwen3-1.7B-ExecuTorch", odd, [], ["LICENSE"]) + assert "the release its file was built with" in text # no single runtime promised for these + assert not manifest.one_runtime_for_all(["1", "1.5.1"]) or True + assert manifest.one_runtime_for_all(["1.5.1", "1.6.0"]) and not manifest.one_runtime_for_all(["1.4.0", "1.6.0"]) + assert manifest.one_runtime_for_all(["1"]) + + +def test_the_card_names_the_newest_cpu_or_gpu_release_not_the_npu_one(): + # A MediaTek folder built a release behind (config/versions.env) must not set the CPU files' + # runtime, and neither may the smallest CPU window when a larger one is newer. + reports = [window(2048, "1.5.1"), window(32768, "1.6.0"), window(2048, "1.5.1", backend="mtk")] + reports[2].update( + target="mt6991", + neuropilot={"build": "8.0.8-build20250925", "mtk_converter": "8.13.0", "mtk_neuron": "8.2.23"}, + runner={}, + ) + text = manifest.readme("experimentalmachines/Qwen3-1.7B-ExecuTorch", reports, [], ["LICENSE"]) + assert "any ExecuTorch 1.6.0 runtime" in text and "ExecuTorch 1.5.1 runtime" not in text + + +def test_the_card_names_no_single_runtime_for_files_two_releases_apart(): + # ExecuTorch promises a runtime loads its own release's files and the previous minor's: a + # 1.4.0 file beside a 1.6.0 one has no runtime promised for both. + reports = [window(2048, "1.4.0"), window(32768, "1.6.0")] + text = manifest.readme("experimentalmachines/Qwen3-1.7B-ExecuTorch", reports, [], ["LICENSE"]) + assert "any ExecuTorch 1.6.0 runtime" not in text and "any ExecuTorch 1.4.0 runtime" not in text + assert "the release its file was built with (1.4.0, 1.6.0: each variant's `runtime_version`" in text + + def test_readme_has_card_metadata_and_kv_arithmetic(): text = manifest.readme("experimentalmachines/Qwen3-1.7B-ExecuTorch", [report()], ["executorch"], ["LICENSE"]) assert text.startswith("---\nlicense: apache-2.0\n") diff --git a/tests/test_publish.py b/tests/test_publish.py index dd1b4cc..a1d0269 100644 --- a/tests/test_publish.py +++ b/tests/test_publish.py @@ -723,3 +723,67 @@ def fake_download(repo_id, path, revision, token): with pytest.raises(ValueError, match="another backend's reports"): publish.publish_hf(out, "mtk", "mt6991", replace_shared=True) assert hub.commits == [] + + +def test_a_file_newer_than_the_apps_runtime_is_held_back(monkeypatch): + # EXECUTORCH_VERSION may move ahead of the apps; APP_RUNTIME_VERSION keeps its files off the + # Hub and the releases until both apps ship that runtime (config/versions.env). + import dataclasses + + from pipeline import settings + + held = dataclasses.replace(settings.load(), app_runtime_version="1.5.1") + monkeypatch.setattr(settings, "load", lambda: held) + newer = {**report(2048), "toolchain": {"executorch": "1.6.0"}} + with pytest.raises(ValueError, match="newer than the apps' runtime 1.5.1"): + publish.require_publishable(newer) + for older in ("1.5.1", "1.4.0", "1.5.0"): + publish.require_publishable({**report(2048), "toolchain": {"executorch": older}}) + with pytest.raises(ValueError, match="names no ExecuTorch release"): + publish.require_publishable({**report(2048), "toolchain": {}}) + moved = dataclasses.replace(held, app_runtime_version="1.6.0") + monkeypatch.setattr(settings, "load", lambda: moved) + publish.require_publishable(newer) + + +def test_this_tree_holds_its_own_exports_until_the_apps_move(): + # The exporter is on 1.6.0 and the apps on 1.5.1: a fresh CPU/GPU report must be refused. + from pipeline import settings + + cfg = settings.load() + assert settings.release(cfg.app_runtime_version) <= settings.release(cfg.executorch_version) + fresh = {**report(2048), "toolchain": {"executorch": cfg.executorch_version}} + if settings.release(cfg.executorch_version) > settings.release(cfg.app_runtime_version): + with pytest.raises(ValueError, match="hold it until both apps ship"): + publish.require_publishable(fresh) + else: + publish.require_publishable(fresh) + + +def test_release_versions_parse(): + from pipeline import settings + + assert settings.release("1.10.0") > settings.release("1.9.3") + assert settings.release("1.6.0a0+git1234") < settings.release("1.6.0") + # The hold compares releases: a pre-release or dev build of 1.6.0 is 1.6.0, and a post + # release of 1.5.1 is 1.5.1. + assert settings.newer_release("1.6.0", "1.5.1") + assert settings.newer_release("1.6.0a0+git1234", "1.5.1") + assert settings.newer_release("1.6.0.dev20261101", "1.5.1") + assert not settings.newer_release("1.5.1.post1", "1.5.1") + assert not settings.newer_release("1.5.1+cpu", "1.5.1") + assert not settings.newer_release("1.4.0", "1.5.1") + with pytest.raises(ValueError): + settings.release("main") + + +def test_the_hold_names_the_backends_it_stops(): + from pipeline import settings + + versions = settings.read_env_file(settings.CONFIG_DIR / "versions.env") + for backend, key in settings.EXPORTER_VERSION_KEYS.items(): + held = settings.newer_release(versions[key], versions["APP_RUNTIME_VERSION"]) + assert (settings.publish_hold(backend) is not None) == held, backend + # This tree: CPU/GPU on 1.6.0 are held for apps on 1.5.1; the NPU toolchains on 1.5.1 are not. + assert settings.publish_hold("xnnpack") and settings.publish_hold("vulkan") + assert settings.publish_hold("qnn") is None and settings.publish_hold("mtk") is None diff --git a/tests/test_qnn.py b/tests/test_qnn.py index 50422d2..502ac20 100644 --- a/tests/test_qnn.py +++ b/tests/test_qnn.py @@ -283,6 +283,19 @@ def test_readme_and_config_for_qnn(): assert "- qnn" in text +@pytest.mark.parametrize("toolchain", ["absent", None]) +def test_a_qnn_report_without_a_toolchain_does_not_stop_the_card(toolchain): + # Every publish regenerates the card from every report in the repo: a retained QNN report + # missing its toolchain must not stop another backend's publish. + report = qnn_report() + if toolchain == "absent": + del report["toolchain"] + else: + report["toolchain"] = None + text = manifest.readme("experimentalmachines/Qwen3-0.6B-ExecuTorch", [report], [], []) + assert "AI Stack License" in text + + def test_publish_refuses_a_tokenizer_inside_a_backend_folder(tmp_path): folder = tmp_path / "qnn" / "sm8650" folder.mkdir(parents=True) diff --git a/tests/test_verify.py b/tests/test_verify.py index 25669d9..24140af 100644 --- a/tests/test_verify.py +++ b/tests/test_verify.py @@ -49,12 +49,36 @@ def test_a_report_that_passed_both_is_left_alone(tmp_path, reaches_smoke): reference = tmp_path / "gate.json" reference.write_text("{}") measured_against = hashlib.sha256(b"{}").hexdigest() - out = write(tmp_path, {"smoke": {"passed": True}, "gate": {"passed": True, "reference_sha256": measured_against}}) + here = verify.exporting.toolchain()["executorch"] + gated = {"passed": True, "reference_sha256": measured_against} + out = write(tmp_path, {"smoke": {"passed": True}, "gate": gated, "verified_with": here}) assert verify.run(out, "xnnpack", tmp_path / "work", gate_reference=reference)["passed"] - out_ungated = write(tmp_path / "b", {"smoke": {"passed": True}}) + out_ungated = write(tmp_path / "b", {"smoke": {"passed": True}, "verified_with": here}) assert verify.run(out_ungated, "xnnpack", tmp_path / "work")["passed"] +@pytest.mark.parametrize("verified_with", ["0.0.1", "missing"]) +def test_a_report_verified_on_another_runtime_is_measured_again(tmp_path, reaches_smoke, verified_with): + # A verdict holds for the runtime that measured it (finding 48: a published file passed its + # smoke test on 1.5.1 and failed it on 1.6.0); a report from before the field counts as other. + reference = tmp_path / "gate.json" + reference.write_text("{}") + gated = {"passed": True, "reference_sha256": hashlib.sha256(b"{}").hexdigest()} + report = {"smoke": {"passed": True}, "gate": gated} + if verified_with != "missing": + report["verified_with"] = verified_with + out = write(tmp_path, report) + with pytest.raises(Ran): + verify.run(out, "xnnpack", tmp_path / "work", gate_reference=reference) + + +def test_force_measures_a_report_verified_here_again(tmp_path, reaches_smoke): + here = verify.exporting.toolchain()["executorch"] + out = write(tmp_path, {"smoke": {"passed": True}, "verified_with": here}) + with pytest.raises(Ran): + verify.run(out, "xnnpack", tmp_path / "work", force=True) + + def test_an_accepted_gate_keeps_its_measured_verdict_and_records_the_reason(tmp_path, monkeypatch): from pipeline import gate, verify diff --git a/tests/test_versions.py b/tests/test_versions.py index f8daf1c..8990101 100644 --- a/tests/test_versions.py +++ b/tests/test_versions.py @@ -12,23 +12,48 @@ def pins(path): return found -def test_executorch_pin_matches_the_app_runtime_version(): - assert pins("requirements/export-xnnpack.txt")["executorch"] == settings.load().executorch_version +def test_executorch_pin_matches_versions_env(): + # The CPU/GPU exporter's pin. What may be published is APP_RUNTIME_VERSION's to decide + # (publish.require_app_runtime), and the exporter is never behind the apps. + cfg = settings.load() + assert pins("requirements/export-xnnpack.txt")["executorch"] == cfg.executorch_version + assert settings.release(cfg.app_runtime_version) <= settings.release(cfg.executorch_version) -def test_torch_pin_matches_executorch_1_5_1(): - # ExecuTorch v1.5.1 torch_pin.py: TORCH_VERSION = "2.14.0"; install_requirements.py: torchao 0.18.0. +def test_torch_pin_matches_executorch_1_6_0(): + # ExecuTorch v1.6.0 torch_pin.py: TORCH_VERSION = "2.15.0" (the wheel requires torch==2.15.0); + # install_requirements.py: TORCHAO_MIN_VERSION 0.18.0, still the newest torchao. export = pins("requirements/export-xnnpack.txt") - assert export["torch"] == "2.14.0" + assert export["torch"] == "2.15.0" assert export["torchao"] == "0.18.0" + assert export["pytorch-tokenizers"] == "1.6.0" -def test_every_export_environment_pins_the_same_runtime(): - xnnpack = pins("requirements/export-xnnpack.txt") - for path in ("requirements/export-qnn.txt", "requirements/mtk-tools.txt"): - other = pins(path) - for package in ("executorch", "torch", "torchao"): - assert other[package] == xnnpack[package], (path, package) +def minor_behind(newer: str, older: str) -> bool: + (a, b), (c, d) = (tuple(int(x) for x in v.split(".")[:2]) for v in (newer, older)) + return a == c and b - d == 1 + + +def test_npu_exports_stay_on_the_release_before(): + # The NPU toolchains keep ExecuTorch 1.5.1 with its torch and torchao (config/versions.env): + # Qualcomm because 1.6.0 compiles with QAIRT 2.51, newer than the apps' QNN runtime, and + # MediaTek because mtk_converter is cp310-only and 1.6.0 has no cp310 wheel. Exactly one + # minor behind, the oldest release a 1.6.0 runtime is promised to load. + versions = settings.read_env_file(settings.CONFIG_DIR / "versions.env") + runtime = settings.load().executorch_version + for path, key in ( + ("requirements/export-qnn.txt", "QNN_EXECUTORCH_VERSION"), + ("requirements/mtk-tools.txt", "MTK_EXECUTORCH_VERSION"), + ): + tools = pins(path) + assert tools["executorch"] == versions[key] == "1.5.1", path + assert (tools["torch"], tools["torchao"]) == ("2.14.0", "0.18.0"), path + assert minor_behind(runtime, tools["executorch"]), path + # QAIRT is 1.5.1's (qnn_config.sh at v1.5.1), not 1.6.0's 2.51.0.260929. + assert versions["QAIRT_VERSION"] == "2.37.0.250724" + assert pins("requirements/export-qnn.txt")["torchvision"] == "0.29.0" # built against torch 2.14.0 + # The examples/mediatek source at v1.5.1's commit, so the scripts match that wheel. + assert versions["EXECUTORCH_COMMIT"] == "3b60683923245cf472b7323426920e15623ba361" def test_mediatek_tools_keep_transformers_4(): @@ -47,12 +72,14 @@ def test_huggingface_hub_pin_is_shared(): def test_backend_requirements_share_their_core_pins(): - # transformers is the one deliberate difference: the Qualcomm scripts run on ExecuTorch's - # example pin (5.0.0rc1), the smoke test's runner import on a current release. + # Deliberate differences: transformers (the Qualcomm scripts run on ExecuTorch's example + # pin, 5.0.0rc1; the smoke test's runner import on a current release) and the runtime the + # Qualcomm toolchain stays on (test_npu_exports_stay_on_the_release_before). + runtime = {"executorch", "torch", "torchao", "pytorch-tokenizers"} xnnpack, qnn = pins("requirements/export-xnnpack.txt"), pins("requirements/export-qnn.txt") - for package in set(xnnpack) & set(qnn) - {"transformers"}: + for package in set(xnnpack) & set(qnn) - {"transformers"} - runtime: assert xnnpack[package] == qnn[package], package - assert qnn["executorch"] == settings.load().executorch_version + assert xnnpack["executorch"] == settings.load().executorch_version def test_every_requirement_is_pinned(): diff --git a/tests/test_watch.py b/tests/test_watch.py index ae22241..d69ca44 100644 --- a/tests/test_watch.py +++ b/tests/test_watch.py @@ -41,6 +41,8 @@ def run(tmp_path, hub, **kwargs): fetch=hub.fetch, dispatcher=kwargs.pop("dispatcher", hub.dispatch), in_flight=kwargs.pop("in_flight", lambda workflow: False), + # Nothing held unless a test says so: this tree's own pins hold CPU/GPU publishing. + held=kwargs.pop("held", lambda backend: None), **kwargs, ) if kwargs.get("dispatch", True): @@ -205,6 +207,28 @@ def fake_run(command, **kwargs): assert not watch.run_in_flight("export-xnnpack.yml", "Qwen/Qwen3-0.6B-Base", None) +def test_a_held_backend_stays_pending_and_the_later_stages_go_ahead(tmp_path): + # CPU/GPU exports newer than the apps' runtime would be refused at publish + # (settings.publish_hold): not dispatched, not marked dispatched, and not in the way of + # the Qualcomm and MediaTek stages, whose toolchains are publishable. + hub = FakeHub({"Qwen": []}) + run(tmp_path, hub) + hub.listings["Qwen"] = ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-4B"] # 0.6B also QNN + hold = lambda backend: "held" if backend == "xnnpack" else None # noqa: E731 + summary = run(tmp_path, hub, held=hold) + assert [d[:2] for d in hub.dispatched] == [("export-qnn.yml", "Qwen/Qwen3-0.6B")] + assert summary["stage"] == "qnn" and summary["held"] == ["xnnpack: held"] + models = summary["state"]["models"] + assert {models[m]["backends"]["xnnpack"]["status"] for m in models if "xnnpack" in models[m]["backends"]} == { + "pending" + } + # The hold lifts: XNNPACK goes out on the next pass, as if it had never waited. + hub.dispatched.clear() + summary = run(tmp_path, hub) + assert {d[0] for d in hub.dispatched} == {"export-xnnpack.yml"} and summary["stage"] == "xnnpack" + assert "held" not in summary + + def test_stages_wait_for_the_previous_backend_to_finish(tmp_path): hub = FakeHub({"Qwen": []}) run(tmp_path, hub) diff --git a/third_party/executorch/README.md b/third_party/executorch/README.md index f39d241..fd36fa7 100644 --- a/third_party/executorch/README.md +++ b/third_party/executorch/README.md @@ -1,6 +1,7 @@ Files copied unchanged from ExecuTorch v1.4.0 (commit 3dd7ccd1d863fad22639dd2d918ae34a41ce45f0) and byte-identical at v1.5.1 (commit -3b60683923245cf472b7323426920e15623ba361), under its BSD license in `LICENSE`. +3b60683923245cf472b7323426920e15623ba361) and v1.6.0 (commit +4776f71b6f0daa5dfc7d758d5836fc1cc0d40154), under its BSD license in `LICENSE`. `examples/models/*/…config.json` are the params files that ExecuTorch's Qualcomm LLM registry (`examples/qualcomm/oss_scripts/llama/__init__.py`) points each `--decoder_model` diff --git a/tools/codecheck/README.md b/tools/codecheck/README.md index ba2126e..4492cd4 100644 --- a/tools/codecheck/README.md +++ b/tools/codecheck/README.md @@ -3,7 +3,7 @@ Byte-level checks used for the 2026-11-09 GPTQ rollout (docs/PLAN.md, Known limits; research findings 43-45). Not wired into the pipeline or CI yet. -- `ptecodes.py`: reads every int4 linear out of a `.pte` (ExecuTorch 1.5.1), in execution order: +- `ptecodes.py`: reads every int4 linear out of a `.pte` (ExecuTorch 1.5.1 and 1.6.0), in execution order: XNNPACK `qbint4` filters of fully-connected nodes (delegate calls in program order), Vulkan `et_vk.linear_dq8ca_q4gsw` weights, scales and per-group code sums. - `codesmatch.py `: exit 0 only if every int4 linear equals the solve's codes diff --git a/tools/codecheck/ptecodes.py b/tools/codecheck/ptecodes.py index 68505ec..4ab4107 100644 --- a/tools/codecheck/ptecodes.py +++ b/tools/codecheck/ptecodes.py @@ -8,6 +8,7 @@ from __future__ import annotations import json +import re import subprocess import tempfile from pathlib import Path @@ -20,6 +21,26 @@ ET = Path(list(executorch.__path__)[0]) +# flatc writes a non-finite double as a bare inf, -inf or nan, which json.loads refuses. +# ExecuTorch 1.6.0's Vulkan serializer can write one (an attention mask's -inf fill value, say) +# and reads it back through the same rewrite (vulkan_graph_serialize._flatc_json_to_python_json). +_JSON_STRING = re.compile(r'"(?:[^"\\]|\\.)*"') +_FLATC_NONFINITE = re.compile(r"(? str: + def fix(segment: str) -> str: + return _FLATC_NONFINITE.sub( + lambda m: "NaN" if m.group(2) == "nan" else ("-Infinity" if m.group(1) == "-" else "Infinity"), segment + ) + + out, last = [], 0 + for m in _JSON_STRING.finditer(text): # never inside a string: a shader may be named "inf" + out += [fix(text[last : m.start()]), m.group(0)] + last = m.end() + return "".join(out) + fix(text[last:]) + + def _flatc_json(schema: Path, blob: bytes) -> dict: with tempfile.TemporaryDirectory() as tmp: bin_path = Path(tmp) / "g.bin" @@ -29,7 +50,7 @@ def _flatc_json(schema: Path, blob: bytes) -> dict: check=True, capture_output=True, ) - return json.loads((Path(tmp) / "g.json").read_text()) + return json.loads(_python_json((Path(tmp) / "g.json").read_text())) def load_pte(path: Path): diff --git a/tools/codecheck/verify_hub.py b/tools/codecheck/verify_hub.py index 78b0f5d..55e38e6 100644 --- a/tools/codecheck/verify_hub.py +++ b/tools/codecheck/verify_hub.py @@ -1,12 +1,13 @@ """After the rollout: is every file in the rolled-out repos what its report says it is? - python verify_hub.py [--skip MODEL ...] + python verify_hub.py [--executorch 1.5.1] [--skip MODEL ...] Per repo, from the Hub, against the byte-check records collected from the boxes (//gptq-/codesmatch-*.json and codes.sha256): - xnnpack/ and vulkan/ each hold exactly the five windows: one report per window label, and the .pte files the reports name are exactly the .pte files in the folder, one each; -- every report: int4_codes gptq, ExecuTorch 1.5.1, the pinned source revision, a file sha256 +- every report: int4_codes gptq, the rollout's ExecuTorch (--executorch, default 1.5.1, the + 2026-11-09 rollout's), the pinned source revision, a file sha256 equal to the Hub's LFS sha256, and a measured, passing gate (XNNPACK: graded > 0, fp32 and export means present, no override) or a Vulkan gate proxy whose verdict equals the published XNNPACK report's of the same window and whose sha256 is that XNNPACK file; @@ -28,6 +29,11 @@ GATE_FIELDS = ("passed", "graded", "agreed", "fp32_mean", "export_mean") args = sys.argv[1:] +EXECUTORCH = "1.5.1" +if "--executorch" in args: + at = args.index("--executorch") + EXECUTORCH = args[at + 1] + del args[at : at + 2] skip = set(args[args.index("--skip") + 1 :]) if "--skip" in args else set() models = [m for m in json.loads(Path(args[0]).read_text()) if m["model"] not in skip] collected, out_path = Path(args[1]), Path(args[2]) @@ -149,7 +155,7 @@ def slug(model_id): continue # the deliberate fp32 files built on 1.4.0, untouched by this rollout elif rec.get("int4_codes") != "gptq": problems.append(f"{r}: int4_codes {rec.get('int4_codes')}") - if d["toolchain"]["executorch"] != "1.5.1": + if d["toolchain"]["executorch"] != EXECUTORCH: problems.append(f"{r}: ExecuTorch {d['toolchain']['executorch']}") measured = ( (gate or {}).get("passed") is True