docs: add a benchmarks section

[msluszniak]

Description

Adds a Benchmarks section to the docs with the v0.10.0 measurements from #1363.

• 124 Android variants on a Galaxy S26 Ultra, a Pixel 10 and a Galaxy S20+
• 124 iOS variants on an iPhone 17 and an iPhone SE (3rd gen)
• 23 LLM variants on the S26 Ultra

Four pages: an overview (devices, the input each task was given, what the data says), per-task tables for Android and iOS, and LLM decode throughput. New sidebar category between Core & Advanced and API Reference.

Results and inputs only. How the measurements are produced is deliberately left out until the harness in #1363 lands, so nothing here documents a command that does not exist on main.

Tables are generated from the measurement data rather than retyped, so the numbers match the source comments exactly.

Three things the tables surface that the raw comments did not:

• Core ML's 14.7x median lead on iOS is partly an artifact of what XNNPACK had to work with. Restricted to the 14 families where XNNPACK has an int8 build it drops to 5.33x, against 17.99x for the 44 fp32-only ones.
• Vulkan inverts between GPUs: it wins 14/16 families on the S26 Ultra's Adreno and only 5/15 on the Pixel 10. paraphrase-multilingual-minilm-l12-v2 is 1.21x faster on Adreno and 3.7x slower on Tensor G5.
• Core ML's lead grows on older hardware. XNNPACK is 2.49x slower on the A15 than the A19, Core ML only 1.53x.

Introduces a breaking change?

• Yes
• No

Type of change

• Bug fix (change which fixes an issue)
• New feature (change which adds functionality)
• Documentation update (improves or adds clarity to existing documentation)
• Other (chores, tests, code style improvements etc.)

Tested on

• iOS
• Android

Testing instructions

cd docs && yarn install && yarn build

Passes with no broken links. yarn start to read the pages.

Screenshots

Related issues

Data from #1363. Feeds the deprecation tracker in #1456.

Checklist

• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have updated the documentation accordingly
• My changes generate no new warnings

Additional notes

Docs-only, so no device testing applies.

Open question for review: the data is v0.10.0 and lives in docs/docs (next), not in the frozen versioned_docs/version-0.10.0. Say the word if it belongs in both.

[msluszniak]

Measured the three hooks that had no accelerator numbers on iOS. Device: iPhone 16 (iPhone17,3), v0.10.0, release build, 10 iterations after 2 warmups, medians below.

variant	median ms	execution ms	peak MB	load ms	IQR
image-embeddings/clip-vit-base-patch32-xnnpack-fp32	67.154	66.687	432.13	68.6	3.304
image-embeddings/clip-vit-base-patch32-coreml-fp16	3.481	3.342	242.47	53.0	0.102
image-embeddings/clip-vit-base-patch32-mlx-int8	12.118	11.848	281.92	23.2	0.045
ocr/paddle-ppocrv6-small-xnnpack	128.037	125.882	753.05	24.1	4.568
ocr/paddle-ppocrv6-small-coreml	6.646	5.130	524.58	96.5	0.175
text-to-speech/supertonic-xnnpack-fp32	2996.950	2986.967	1061.16	114.7	4.523
text-to-speech/supertonic-mlx-fp32	793.520	761.097	1124.52	73.4	9.340

Speedups over the XNNPACK baseline on the same device:

• image embeddings: Core ML 19.3x, MLX 5.5x
• OCR: Core ML 19.3x
• text to speech: MLX 3.8x

Both Core ML rows also cut peak memory (432 to 242 MB for CLIP, 753 to 525 MB for OCR). Supertonic MLX is the exception: it is 3.8x faster but peaks slightly higher than XNNPACK.

These three hooks previously had only Android Vulkan numbers, so the docs tables understated them: OCR was listed at 4.8x and text to speech at 1.4x.

[msluszniak]

@barhanc I guess we can add benchmarks now, and update with llms data later on.

[barhanc]

Some general remarks:

• I feel there is too much text in overview that nobody is going to read. It should be more concentrated and punchy (e.g. one table for devices' info, one table for tasks info with benchmark notes column, etc.). Important info marked with :::info and so on.
• We shouldn't separate Android and iOS benchmarks into separate pages, makes it difficult to compare the two (see next point).
• We discussed in the past that raw measurement tables are not the best way to showcase the benchmarks and here this problem is even more visible because of the number of model/device/backend/precision configurations. Benchmarks are walls of numbers that are difficult to read and aren't very pleasant to the eye. This definitely needs to be changed. What I'd rather see are:
  • interactive column charts where you can select model/metric and columns get rendered for different device/backend/precision configurations grouped by device (for comparing performance of given model on different devices)
  • a way to also compare different models for a given task either by taking a mean or best results per devices or something like that.

I attach an example of what that interactive chart could look like

[msluszniak]

Some general remarks:

• I feel there is too much text in overview that nobody is going to read. It should be more concentrated and punchy (e.g. one table for devices' info, one table for tasks info with benchmark notes column, etc.). Important info marked with :::info and so on.

• We shouldn't separate Android and iOS benchmarks into separate pages, makes it difficult to compare the two (see next point).

• We discussed in the past that raw measurement tables are not the best way to showcase the benchmarks and here this problem is even more visible because of the number of model/device/backend/precision configurations. Benchmarks are walls of numbers that are difficult to read and aren't very pleasant to the eye. This definitely needs to be changed. What I'd rather see are:

  • interactive column charts where you can select model/metric and columns get rendered for different device/backend/precision configurations grouped by device (for comparing performance of given model on different devices)
  • a way to also compare different models for a given task either by taking a mean or best results per devices or something like that.

I attach an example of what that interactive chart could look like

@barhanc Totally agreed. If you want, feel free to updating this PR by yourself so we don't postpone releasing the benchmark results too much :))

[barhanc]

I've added commits that implement the interactive charts in question. Right now users can choose the model in a dropdown menu and values for different devices are rendered -- this works nicely because the number of devices is relatively small so the diagrams are clear and not cluttered.

[msluszniak]

A few things:

1. Devices table: S26 Ultra has 123 XNNPACK rows / 147 variants in benchmarks.json (LLMs not counted), not 100 / 124.
2. S26 Ultra sizeMb is ~1.048x every other device (efficientnet-v2-s 86 vs 82), looks like MiB vs MB. Both selfie-segmentation models are 0 there.
3. The even-count median in BenchmarkChart.tsx uses Math.round, so sub-1 MB models show 0 or 1. Once 2 is fixed, sizes are device-independent and the median can go.
4. "Model Time Share" says "Higher is better", but it isn't a quality metric. I'd drop the direction for it.
5. Worth one line with RNE/ExecuTorch version, date, and that these are release builds.