Repository navigation
Gemma 3 1B on MediaTek MT6991 and Qualcomm SM8750; Gemma 2 2B refused at mtk_converter's 2 GB cap - #30
Merged
Conversation
… RoPE table for Gemma 3, and Gemma 2 ahead of the apps MediaTek's examples/mediatek has gemma.py and Gemma2/Gemma3 chunk models. Measured in fp32 against Hugging Face, before any quantization (MediaTek's chunk classes on the CPU): - Gemma 2 2B: exact (KL 0, top-1 100%, perplexity 6.054 both, gate 8/8). - Gemma 3 1B: exact only when each layer gets its own RoPE table. The stock graph takes one table for all layers, and Gemma 3 rotates its sliding layers at 10000 and its global ones at 1e6. Our calibration's single table at 1e6 gave perplexity 22.7 against 9.6 and top-1 52%; at 10000 it gave 10.8 and 88%. The gate failed either way. The fixes: - families.mtk_model_config writes HF's gemma3_text keys the way configuration_gemma reads them: model_type gemma3, sliding_window_attention_size 512 (MediaTek's default is 1024), global_local_attention_pattern and use_res_clamp false. - mediatek-gemma.patch gives Gemma3ModelChunk a pos_emb_local input for its sliding layers, built at rope_local_base_freq. It also gives gemma.py the one-row calibration and the compiler-service retry that the other scripts carry. - The MediaTek runner block records window_size and rot_emb_local_base for Gemma 3. No runner shipped today feeds the second table, so these files need one that does. - Gemma 2 is a family exported ahead of both apps, which have no Gemma 2 template. The Gemma Terms of Use notice is keyed by the pipeline's family, so it covers Gemma 2 too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…past 4,096 tokens, no runtime promised for NPU-only files Codex review of the Gemma change: - The MediaTek section of the model card named ExecuTorch's MediaTek runner for every variant. A Gemma 3 variant's runner block lists rot_emb_inputs [global, local], and the card now says what that runner has to feed: both tables, and the sliding-window mask beside the global one. MediaTek's runner as 1.5.1 ships it builds one table and cannot run these files. - A Gemma 2 window past the model's 4,096-token sliding window records a limit (families.mtk_limits), and the card prints it: MediaTek's Gemma 2 graph takes one mask, so its sliding layers see the whole window. Up to 4,096 tokens it matches Hugging Face exactly in fp32. - The ahead-of-the-app line promised that any ExecuTorch runtime runs the files. It now says so only when the repo has XNNPACK or Vulkan files; Gemma 2 here is MediaTek-only. Codex's other finding, that GemmaConfig.print_config breaks Gemma 2, does not hold: print_config reads use_qk_norm only inside its gemma3 branch, and the Gemma 2 harness and exports load that config. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…emma 2 2B's cannot be built at any chunk count mtk_converter 8.13.0 holds a delegated graph's weights as fp32 in one protobuf ModelProto, and the output head sits whole in the last chunk. Gemma 2 2B's tied 256000 x 2304 head is 2.36 GB: its export failed at 2 and 13 chunks, and the head alone failed plain, with fp16 weights and with the int8 quantization folded. mtk_plan now says so before any download (finding 47). PLAN and findings 46-47 record the Gemma 3 two-table RoPE result and the Gemma 2 block, with the fp32 harness, its output, and the head probe as evidence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… its files need the runner its section names The opening sentence offered every repo to "any ExecuTorch runtime", which a MediaTek-only repo such as gemma-3-1b-it cannot keep: its files run only on MediaTek's runner, and Gemma 3's only on one that feeds both RoPE tables. The MediaTek section also said the runner's settings are in config.json, which read as if the runner loads it; it takes flags. Found by codex review. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…use two windows on one host deleted each other's chunks MediaTek's scripts write chunks to pte/<exp> under their working directory, and <exp> has no window in it, so Gemma 3's 8k and 16k runs on one EC2 box shared the folder. The 8k run finished first and its cleanup took the 16k run's chunk 0; the 16k export failed on the missing file. The 8k files were its own: the structural check matches method names that carry the window, and the published reports show 1t8192c/128t8192c on both chunks. The script now runs in a 31 MB copy of examples/mediatek inside the run's work folder, which goes with it. CI gives every job its own runner and never hit this. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…wo windows on one host shared the chunk folder Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h, because the runner config would name a table the graph lacks Every other MediaTek patch has a marker the export checks before any download; the Gemma one had none. On an unpatched tree Gemma 3 would build with one RoPE table for every layer (fp32 gate 6/11, finding 46) under a runner config naming two. The markers are checked against the patch file in the tests and were found in the patched tree on the EC2 box. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…run's time and the box's memory Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uns' times and peaks and the libatomic1 a fresh image lacks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…w cannot swap the shared embedding, absolute script paths, scoped runtime promise - families.mtk_model_config raised StopIteration for layer_types with no full_attention layer (KeyError for an unknown layer type or no sliding_window). Eligibility catches only UnsupportedModel, so such a model would have ended the watcher's whole pass; each is now an UnsupportedModel with its reason. - Every MediaTek window in a folder shares one embedding table, and each report records its sha256. Publishing a window exported from another source revision replaced it under the other windows' chunks. publish.require_shared_files_agree refuses that (the published Gemma 3 windows all record ee65de48, from dcc83ea8). - export_mtk.script_paths hands the script an absolute tool python and config.json: it runs in the run's own copy of the scripts, where a relative --work pointed into that copy. - A repo with CPU or GPU files beside NPU ones promised every file to any ExecuTorch runtime; the card now says only the CPU and GPU files run there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s on the old embedding, and run() resolves its paths before any subprocess - The shared-file refusal had no way through: publish_hf commits one window per run, so after a revision change every rebuilt window was refused for the others still on the old table. publish-hf --replace-shared now withdraws those windows (chunks and reports) in the same commit, so the folder never mixes tables; each is then republished from the new revision. - export_mtk.run makes out, work, examples and the tool python absolute before anything else: pipeline.mtk_corpus runs from the repo, before script_paths was reached, and LFM2's streaming calibration read relative paths from there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…wn windows' own files, and run()'s paths go through one tested helper - --replace-shared deleted every file a clashing report listed unless this run listed it too, which could take a file a kept window still lists or a path in another folder. It now takes only each withdrawn window's report and its window-labelled files in this folder, never a file any kept report lists. - The test now covers a kept window, an unlabelled shared file, a path in another folder, a pre-matrix report on the old table, and the rebuilt README; a spy shows run() makes its paths absolute (export_mtk.absolute_inputs) before anything else. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…thdrawn, and a clash with another backend's report is refused The clash check compared against every folder's reports, so --replace-shared could delete another backend's report without rebuilding its config.json. Withdrawal now considers only this folder's windows; a clash with another backend's report is refused even with the flag. The withdrawal test now has a kept window that lists a withdrawn window's labelled file, which must survive. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t overlap them --examples <work>/mediatek made the run's copy the scripts themselves, and private_examples deleted them before copying. It now refuses any overlap, either way, before deleting anything. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Gemma 3 1B on the two NPU paths, and what stopped Gemma 2 2B. The files are already on the Hub, built from this branch: experimentalmachines/gemma-3-1b-it-ExecuTorch.
.pteper window (1.15–1.21 GB)Changes
Gemma 3 on MediaTek (
third_party/executorch/patches/mediatek-gemma.patch,families.mtk_model_config,export_mtk)Gemma3ModelChunkgives every layer one table: its master table is built at 10000, and its runner sample setsrot_emb_base1e6. In fp32 against Hugging Face, either base alone fails the decision gate (6/11 and 5/11). The patch adds a second table input,pos_emb_local, and chooses per layer, which is exact (KL 0, 11/11). See finding 46 and its evidence.gemma3,sliding_window_attention_size,global_local_attention_pattern, anduse_res_clampfalse. The runner config nameswindow_size,rot_emb_local_baseandrot_emb_inputs.GEMMA3_MARKERS), as the other MediaTek patches already do.Gemma 2 2B: refused, nothing published (
families.mtk_plan,mtk_head_bytes, finding 47)mtk_converter8.13.0 holds a chunk's weights in fp32 in one protobuf capped at 2 GB, and the output head sits whole in the last chunk. Gemma 2's tied 256000 × 2304 head is 2.36 GB. The export failed at 2 and 13 chunks. The head alone also failed plain, withconvert_float32_weights_to_float16, and with the int8 quantization folded.mtk_plannow says so before any download.mtk_limitsand the card say what differs past it, in case the head problem is solved later.MediaTek export robustness
export_mtk.private_examples). The chunk folderpte/<weights>_<precision>_<n>_chunkshas no window in its name. Two windows of Gemma 3 on one EC2 box shared that folder, and the first to finish deleted the other's chunk. CI gives each job its own runner, so it never hit this. The published files were checked: each window's chunks carry their own window's method names.Model cards (
manifest)config.json.limits, and usesnaming.pipeline_familyfor the NOTICE (Gemma Terms of Use for gemma2).Docs
libatomic1that a fresh Ubuntu 24.04 image lacks and the QAIRT SDK needs.--checkpointroute would ship post-training int4 with no host check.Review and testing
UnsupportedModelverdict. Before, it raisedStopIteration/KeyError, which would have ended the watcher's whole pass.publishnow refuses a run whose table differs from the one the folder's other reports record, as a newer source revision's would.publish-hf --replace-sharedwithdraws those windows in the same commit (only their own files and reports in this folder), so they can be republished from the new revision.ee65de48, built fromdcc83ea8.export_mtk.runmakes its paths absolute before any subprocess (relative--workbroke LFM2's streaming calibration and the script). The run's copy of the scripts may not overlap--examples.ruff check,ruff format --check: clean.pytest: 350 passed, 9 skipped locally (the torch-dependent modules run in CI).🤖 Generated with Claude Code