Skip to content

Gemma 3 1B on MediaTek MT6991 and Qualcomm SM8750; Gemma 2 2B refused at mtk_converter's 2 GB cap - #30

Merged
alpharomercoma merged 14 commits into
mainfrom
gemma-mtk
Nov 9, 2026
Merged

alpharomercoma merged 14 commits into
mainfrom
gemma-mtk

Conversation

@alpharomercoma

@alpharomercoma alpharomercoma commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

Summary

Gemma 3 1B on the two NPU paths, and what stopped Gemma 2 2B. The files are already on the Hub, built from this branch: experimentalmachines/gemma-3-1b-it-ExecuTorch.

Backend Chip Windows Files
MediaTek NeuroPilot, A16W8 MT6991 (Dimensity 9400) 512, 2k, 4k, 8k, 16k, 32k 2 chunks + fp32 embedding table per window
Qualcomm QNN (HTP), hybrid SM8750 (Snapdragon 8 Elite) 2k, 4k, 8k, 16k, 32k one .pte per window (1.15–1.21 GB)

Changes

Gemma 3 on MediaTek (third_party/executorch/patches/mediatek-gemma.patch, families.mtk_model_config, export_mtk)

  • Gemma 3's sliding layers rotate at base 10000 and its global layers at 1e6. MediaTek's Gemma3ModelChunk gives every layer one table: its master table is built at 10000, and its runner sample sets rot_emb_base 1e6. In fp32 against Hugging Face, either base alone fails the decision gate (6/11 and 5/11). The patch adds a second table input, pos_emb_local, and chooses per layer, which is exact (KL 0, 11/11). See finding 46 and its evidence.
  • The pipeline writes the config in MediaTek's spelling: model_type gemma3, sliding_window_attention_size, global_local_attention_pattern, and use_res_clamp false. The runner config names window_size, rot_emb_local_base and rot_emb_inputs.
  • The export refuses Gemma 3 on scripts without the patch (GEMMA3_MARKERS), as the other MediaTek patches already do.
  • Known limit, stated on the card: MediaTek's runner as ExecuTorch 1.5.1 ships it feeds one table, and neither app feeds the second yet, so these MediaTek files need a runner that builds both tables.

Gemma 2 2B: refused, nothing published (families.mtk_plan, mtk_head_bytes, finding 47)

  • mtk_converter 8.13.0 holds a chunk's weights in fp32 in one protobuf capped at 2 GB, and the output head sits whole in the last chunk. Gemma 2's tied 256000 × 2304 head is 2.36 GB. The export failed at 2 and 13 chunks. The head alone also failed plain, with convert_float32_weights_to_float16, and with the int8 quantization folded. mtk_plan now says so before any download.
  • The graph itself is exact against Hugging Face up to Gemma 2's 4,096-token sliding window. mtk_limits and the card say what differs past it, in case the head problem is solved later.

MediaTek export robustness

  • Each run executes MediaTek's scripts in its own 31 MB copy, in its work folder (export_mtk.private_examples). The chunk folder pte/<weights>_<precision>_<n>_chunks has no window in its name. Two windows of Gemma 3 on one EC2 box shared that folder, and the first to finish deleted the other's chunk. CI gives each job its own runner, so it never hit this. The published files were checked: each window's chunks carry their own window's method names.

Model cards (manifest)

  • A repo with only NPU files no longer promises "any ExecuTorch runtime" in its first sentence.
  • The MediaTek section says the runner takes flags and does not read config.json.
  • The card has a paragraph for the two-table variants, prints each report's limits, and uses naming.pipeline_family for the NOTICE (Gemma Terms of Use for gemma2).

Docs

  • PLAN covers the Gemma results, the per-run scripts copy, and the Gemma 3 QNN runs: times, memory peaks, and the libatomic1 that a fresh Ubuntu 24.04 image lacks and the QAIRT SDK needs.
  • Findings 46 and 47 come with the fp32 harness, its output, and the head probe as evidence.
  • PLAN records that the Qwen2.5 Instruct, 3B and Math models were held back from QNN. They are not in Qualcomm's registry, and the --checkpoint route would ship post-training int4 with no host check.

Review and testing

  • Codex reviewed the PR adversarially over six rounds; the last one reported nothing blocking. Fixes along the way:
    • NPU-only card wording: the opening sentence no longer promises "any ExecuTorch runtime" for NPU files. Repos with both kinds now say only the CPU/GPU files run on any runtime.
    • Watcher robustness: a Gemma 3 config MediaTek can't translate is now an UnsupportedModel verdict. Before, it raised StopIteration/KeyError, which would have ended the watcher's whole pass.
    • Shared MediaTek embedding: every window in a folder shares one embedding table. publish now refuses a run whose table differs from the one the folder's other reports record, as a newer source revision's would.
      • publish-hf --replace-shared withdraws those windows in the same commit (only their own files and reports in this folder), so they can be republished from the new revision.
      • A clash with another backend's report is always refused.
      • The published Gemma 3 windows all record ee65de48, built from dcc83ea8.
    • Paths: export_mtk.run makes its paths absolute before any subprocess (relative --work broke LFM2's streaming calibration and the script). The run's copy of the scripts may not overlap --examples.
  • ruff check, ruff format --check: clean. pytest: 350 passed, 9 skipped locally (the torch-dependent modules run in CI).
  • Real exports: Gemma 3 MediaTek at 512 to 32k and QNN at 2k to 32k, on EC2. Every published file passed its structural check, and each file's sha256 matches its report.
  • The branch was rebased on CI builds Vulkan files from the XNNPACK run's GPTQ codes; the 2026-11-09 RTN-to-GPTQ rollout and the checks that proved it #29. That PR took findings 43–45, so the Gemma findings were renumbered 46 and 47 everywhere, commit messages included.

🤖 Generated with Claude Code

alpharomercoma and others added 14 commits November 9, 2026 19:50
… RoPE table for Gemma 3, and Gemma 2 ahead of the apps

MediaTek's examples/mediatek has gemma.py and Gemma2/Gemma3 chunk models. Measured in fp32
against Hugging Face, before any quantization (MediaTek's chunk classes on the CPU):
- Gemma 2 2B: exact (KL 0, top-1 100%, perplexity 6.054 both, gate 8/8).
- Gemma 3 1B: exact only when each layer gets its own RoPE table. The stock graph takes one
  table for all layers, and Gemma 3 rotates its sliding layers at 10000 and its global ones
  at 1e6. Our calibration's single table at 1e6 gave perplexity 22.7 against 9.6 and top-1
  52%; at 10000 it gave 10.8 and 88%. The gate failed either way.

The fixes:
- families.mtk_model_config writes HF's gemma3_text keys the way configuration_gemma reads
  them: model_type gemma3, sliding_window_attention_size 512 (MediaTek's default is 1024),
  global_local_attention_pattern and use_res_clamp false.
- mediatek-gemma.patch gives Gemma3ModelChunk a pos_emb_local input for its sliding layers,
  built at rope_local_base_freq. It also gives gemma.py the one-row calibration and the
  compiler-service retry that the other scripts carry.
- The MediaTek runner block records window_size and rot_emb_local_base for Gemma 3. No runner
  shipped today feeds the second table, so these files need one that does.
- Gemma 2 is a family exported ahead of both apps, which have no Gemma 2 template. The
  Gemma Terms of Use notice is keyed by the pipeline's family, so it covers Gemma 2 too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…past 4,096 tokens, no runtime promised for NPU-only files

Codex review of the Gemma change:

- The MediaTek section of the model card named ExecuTorch's MediaTek runner for every variant.
  A Gemma 3 variant's runner block lists rot_emb_inputs [global, local], and the card now
  says what that runner has to feed: both tables, and the sliding-window mask beside the
  global one. MediaTek's runner as 1.5.1 ships it builds one table and cannot run these files.
- A Gemma 2 window past the model's 4,096-token sliding window records a limit
  (families.mtk_limits), and the card prints it: MediaTek's Gemma 2 graph takes one mask, so
  its sliding layers see the whole window. Up to 4,096 tokens it matches Hugging Face
  exactly in fp32.
- The ahead-of-the-app line promised that any ExecuTorch runtime runs the files. It now says
  so only when the repo has XNNPACK or Vulkan files; Gemma 2 here is MediaTek-only.

Codex's other finding, that GemmaConfig.print_config breaks Gemma 2, does not hold:
print_config reads use_qk_norm only inside its gemma3 branch, and the Gemma 2 harness and
exports load that config.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…emma 2 2B's cannot be built at any chunk count

mtk_converter 8.13.0 holds a delegated graph's weights as fp32 in one protobuf ModelProto, and
the output head sits whole in the last chunk. Gemma 2 2B's tied 256000 x 2304 head is 2.36 GB:
its export failed at 2 and 13 chunks, and the head alone failed plain, with fp16 weights and
with the int8 quantization folded. mtk_plan now says so before any download (finding 47).

PLAN and findings 46-47 record the Gemma 3 two-table RoPE result and the Gemma 2 block, with
the fp32 harness, its output, and the head probe as evidence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… its files need the runner its section names

The opening sentence offered every repo to "any ExecuTorch runtime", which a MediaTek-only
repo such as gemma-3-1b-it cannot keep: its files run only on MediaTek's runner, and Gemma 3's
only on one that feeds both RoPE tables. The MediaTek section also said the runner's settings
are in config.json, which read as if the runner loads it; it takes flags. Found by codex review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…use two windows on one host deleted each other's chunks

MediaTek's scripts write chunks to pte/<exp> under their working directory, and <exp> has no
window in it, so Gemma 3's 8k and 16k runs on one EC2 box shared the folder. The 8k run
finished first and its cleanup took the 16k run's chunk 0; the 16k export failed on the
missing file. The 8k files were its own: the structural check matches method names that carry
the window, and the published reports show 1t8192c/128t8192c on both chunks.

The script now runs in a 31 MB copy of examples/mediatek inside the run's work folder, which
goes with it. CI gives every job its own runner and never hit this.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…wo windows on one host shared the chunk folder

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h, because the runner config would name a table the graph lacks

Every other MediaTek patch has a marker the export checks before any download; the Gemma one
had none. On an unpatched tree Gemma 3 would build with one RoPE table for every layer (fp32
gate 6/11, finding 46) under a runner config naming two. The markers are checked against the
patch file in the tests and were found in the patched tree on the EC2 box.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…run's time and the box's memory

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uns' times and peaks and the libatomic1 a fresh image lacks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…w cannot swap the shared embedding, absolute script paths, scoped runtime promise

- families.mtk_model_config raised StopIteration for layer_types with no full_attention layer
  (KeyError for an unknown layer type or no sliding_window). Eligibility catches only
  UnsupportedModel, so such a model would have ended the watcher's whole pass; each is now an
  UnsupportedModel with its reason.
- Every MediaTek window in a folder shares one embedding table, and each report records its
  sha256. Publishing a window exported from another source revision replaced it under the
  other windows' chunks. publish.require_shared_files_agree refuses that (the published Gemma 3
  windows all record ee65de48, from dcc83ea8).
- export_mtk.script_paths hands the script an absolute tool python and config.json: it runs in
  the run's own copy of the scripts, where a relative --work pointed into that copy.
- A repo with CPU or GPU files beside NPU ones promised every file to any ExecuTorch runtime;
  the card now says only the CPU and GPU files run there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s on the old embedding, and run() resolves its paths before any subprocess

- The shared-file refusal had no way through: publish_hf commits one window per run, so after
  a revision change every rebuilt window was refused for the others still on the old table.
  publish-hf --replace-shared now withdraws those windows (chunks and reports) in the same
  commit, so the folder never mixes tables; each is then republished from the new revision.
- export_mtk.run makes out, work, examples and the tool python absolute before anything else:
  pipeline.mtk_corpus runs from the repo, before script_paths was reached, and LFM2's streaming
  calibration read relative paths from there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…wn windows' own files, and run()'s paths go through one tested helper

- --replace-shared deleted every file a clashing report listed unless this run listed it too,
  which could take a file a kept window still lists or a path in another folder. It now takes
  only each withdrawn window's report and its window-labelled files in this folder, never a
  file any kept report lists.
- The test now covers a kept window, an unlabelled shared file, a path in another folder, a
  pre-matrix report on the old table, and the rebuilt README; a spy shows run() makes its
  paths absolute (export_mtk.absolute_inputs) before anything else.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…thdrawn, and a clash with another backend's report is refused

The clash check compared against every folder's reports, so --replace-shared could delete
another backend's report without rebuilding its config.json. Withdrawal now considers only
this folder's windows; a clash with another backend's report is refused even with the flag.
The withdrawal test now has a kept window that lists a withdrawn window's labelled file, which
must survive.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t overlap them

--examples <work>/mediatek made the run's copy the scripts themselves, and private_examples
deleted them before copying. It now refuses any overlap, either way, before deleting anything.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alpharomercoma
alpharomercoma merged commit 34c5511 into main Nov 9, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant