Skip to content

ExecuTorch 1.6.0 for CPU and GPU exports, held from publishing until both apps ship its runtime - #31

Open
alpharomercoma wants to merge 2 commits into
mainfrom
executorch-1.6
Open

alpharomercoma wants to merge 2 commits into
mainfrom
executorch-1.6

Conversation

@alpharomercoma

@alpharomercoma alpharomercoma commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

What changes

  • CPU (XNNPACK) and GPU (Vulkan) export with ExecuTorch 1.6.0, torch 2.15.0, torchao 0.18.0 and pytorch-tokenizers 1.6.0.

  • Qualcomm (QNN) stays on 1.5.1 + QAIRT 2.37 (QNN_EXECUTORCH_VERSION): 1.6.0's wheel compiles with QAIRT 2.51, and both apps ship qnn-runtime 2.39.

  • MediaTek stays on 1.5.1 (MTK_EXECUTORCH_VERSION): mtk_converter is cp310 only, and 1.6.0 ships no cp310 wheel.

  • Publishing is held by APP_RUNTIME_VERSION=1.5.1. An older program loads on the next runtime, but nothing is promised the other way, so no file built with a newer release than the apps run gets published:

    • publish refuses such a report.
    • The export workflows stop at their first step when asked to publish one (python -m pipeline publish-hold). A build-only run still goes through.
    • The watcher leaves a held backend's models pending and dispatches the later stages.

    Merging this as it stands pauses CPU/GPU publishing. Setting APP_RUNTIME_VERSION=1.6.0 lifts the hold once ExecuServe and openweights are live on 1.6.0. Both are in Play review now.

  • config.json gives each variant its own runtime_version, and the folder's value is the newest release in it.

  • verify used to return the stored verdict for any report already gated. It now re-measures when the installed ExecuTorch differs from the one recorded in verified_with, and --force re-measures regardless.

Regression results (finding 48)

  • Published files:
    • All 368 published .pte files parse under 1.6.0's schema.
    • All 20 CPU files load and run on 1.6.0. 15 of the 16 with a gate keep their 1.5.1 verdict, and Phi-4-mini improves to 8/8.
    • Qwen2.5-Math-1.5B's published file fails the smoke prompt on 1.6.0 while still passing its gate (3/3). Its choice flips at the eighth token, and a 1.6.0 rebuild passes both.
  • Older 1.4.0 files: five CPU files and 145 of 178 MediaTek chunks are two releases behind 1.6.0.
    • The CPU ones agree with the 1.4.0 runtime on the first token for 79 of 80 rows.
    • The MediaTek ones were checked only structurally.
  • Fresh 1.6.0 exports: they pass their gates with byte-identical GPTQ codes. A 2×2 of SmolLM2 builds and runtimes (1.5.1 and 1.6.0) passes in every cell.

Evidence and scripts: docs/research/evidence/executorch-1.6.0-regression.md. Reviewed adversarially by Codex over five rounds, and every finding was fixed or documented.

Test plan

  • pytest -q: 422 passed, 1 skipped (Python 3.11, torch 2.15.0, executorch 1.6.0); ruff check and ruff format --check clean.
  • A build-only export-xnnpack run on this branch (publishing off): run 43719116925, SmolLM2-360M-Instruct at 2k.
    • The GPTQ solve on torch 2.15 finished 225 linears in 140 s.
    • The int4 file passed the smoke test ("Paris") on the x86 runner, and passed the gate on its 2 confident rows: 0.5759 against fp32's 0.5698, same choice on both.
    • The Vulkan file built and passed the structural check. It is 323,019,008 B, 10,368 B smaller than the published 1.5.1 file. It ran on no GPU.
  • python -m pipeline publish-hold in a requirements/dev.txt-only Python 3.11 env (what setup-light installs): it fails xnnpack and vulkan and passes qnn and mtk. The build-only run skips this step, so it was checked separately.
  • Before lifting the hold: decide on rebuilding Qwen2.5-Math-1.5B and the 1.4.0 files.

🤖 Generated with Claude Code


Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need. Autofix is enabled.

alpharomercoma and others added 2 commits November 15, 2026 21:27
…n 1.5.1, and nothing newer than the apps' runtime publishes

1.6.0 needs torch 2.15.0 and Python 3.11 or newer. XNNPACK and Vulkan move to it.
QNN stays on 1.5.1 with QAIRT 2.37 (QNN_EXECUTORCH_VERSION), because 1.6.0's wheel
compiles with QAIRT 2.51 and both apps ship qnn-runtime 2.39. MediaTek stays on 1.5.1
(MTK_EXECUTORCH_VERSION), because mtk_converter is cp310 only and 1.6.0 has no cp310 wheel.

An older program loads on the next runtime, never the reverse, so APP_RUNTIME_VERSION
holds publishing until both apps ship the exporter's release:
- publish refuses a report built newer than APP_RUNTIME_VERSION
- the export workflows stop at their first step when asked to publish such a build
  (python -m pipeline publish-hold); a build-only run still goes through
- the watcher leaves a held backend's models pending and dispatches the later stages

config.json gives every variant its own runtime_version, and the folder's value is the
newest release in it. verify re-measures when the installed ExecuTorch differs from the
one a report was verified with (verified_with), and --force re-measures regardless:
before this, a re-run returned the stored verdict for any report already gated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
….0 runtime and fresh 1.6.0 builds against 1.5.1's

All 368 published .pte files parse under 1.6.0's schema. All 20 CPU files load and run on
it, and 15 of 16 gated ones keep their 1.5.1 verdict (Phi-4-mini improves to 8/8).
Qwen2.5-Math-1.5B's published file fails the smoke prompt on 1.6.0 while passing its
gate; a 1.6.0 rebuild passes both. Fresh 1.6.0 exports pass their gates with the same
GPTQ codes. Linux, GPU, phone and the apps' own runtime builds were not measured here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant