Add benchmarking infrastructure #727

Activity

1. chmjkb

   added this to the v1.0.0 milestone on Jan 14, 2026
2. chmjkb

   added the

   Task

   issue type on Jan 14, 2026
3. chmjkb

   added

   documentationImprovements or additions to documentationImprovements or additions to documentation

   featurePRs that implement a new featurePRs that implement a new feature

   on Jan 14, 2026
4. msluszniak

   added

   performanceRelated to all issues and tasks focused on improving performanceRelated to all issues and tasks focused on improving performance

   and removed

   featurePRs that implement a new featurePRs that implement a new feature

   on Jan 16, 2026
5. mkopcins

   added

   core

   on Jan 16, 2026
6. barhanc

   self-assigned this

   on May 18, 2026
7. barhanc

   removed their assignment

   on Sep 2, 2026
8. msluszniak commented on Sep 9, 2026

   

   msluszniak

   on Sep 9, 2026

   Last edited by msluszniak

   Member

   More actions

   v0.10.0 on SM-S948B (Android 16, SM8850, 8 cores, 11 GB)

   The published Android estate: all 124 variants ok. Release build, one repeat, 20 timed iterations after 3 warmups, memory sampled over 5, load time the median of 3 cold loads. Every case enters at or below 37 C with the device not throttling; the host holds the run until dumpsys battery says so.

   median and mean are the whole pipeline (pre-processing, execute, post-processing), which is what an app actually waits for. ET is the time ExecuTorch itself spent inside those same iterations, tallied natively per execute call; share is ET / mean. The tally resets after the warmups, so it covers exactly the timed iterations and belongs against mean, not against median. It is below the pipeline mean on all 124 variants, worst case 99.9%.

   classification

   variant	backend	precision	median ms	mean ms	ET ms	share	peak MB	load ms	size MB
   efficientnet-v2-s	xnnpack	fp32	73.94	82.51	80.56	98%	316	55	86
   efficientnet-v2-s	xnnpack	int8	80.83	72.91	70.49	97%	237	29	23

   style-transfer

   variant	backend	precision	median ms	mean ms	ET ms	share	peak MB	load ms	size MB
   mosaic	xnnpack	int8	357.66	358.08	354.74	99%	1116	53	2
   candy	xnnpack	int8	364.77	365.81	362.33	99%	1105	32	2
   udnie	xnnpack	int8	436.14	421.38	417.62	99%	1139	42	2
   rain-princess	xnnpack	int8	557.45	545.96	541.26	99%	1103	53	2
   mosaic	xnnpack	fp32	700.36	686.82	682.54	99%	1256	46	7
   udnie	xnnpack	fp32	799.17	815.20	809.82	99%	1259	57	7
   candy	xnnpack	fp32	864.79	840.18	834.96	99%	1249	47	7
   rain-princess	xnnpack	fp32	897.18	900.52	894.54	99%	1256	79	7

   semantic-segmentation

   variant	backend	precision	median ms	mean ms	ET ms	share	peak MB	load ms	size MB
   selfie-segmentation-landscape	xnnpack	fp32	5.54	5.33	4.28	80%	215	1	0
   selfie-segmentation	xnnpack	fp32	5.59	5.42	3.96	73%	224	2	0
   lraspp-mobilenet-v3-large	xnnpack	int8	29.09	28.68	21.22	74%	379	46	4
   lraspp-mobilenet-v3-large	xnnpack	fp32	33.50	33.52	26.42	79%	415	35	13
   deeplab-v3-mobilenet-v3-large	xnnpack	int8	41.09	40.58	28.44	70%	368	62	11
   deeplab-v3-mobilenet-v3-large	xnnpack	fp32	92.39	90.53	75.66	84%	393	91	44
   fcn-resnet50	xnnpack	int8	117.92	130.20	118.93	91%	421	138	36
   deeplab-v3-resnet50	xnnpack	int8	161.89	160.99	148.53	92%	473	106	42
   fcn-resnet101	xnnpack	int8	203.31	203.27	191.86	94%	521	179	55
   deeplab-v3-resnet101	xnnpack	int8	242.39	248.02	238.04	96%	603	110	62
   fcn-resnet50	xnnpack	fp32	500.19	494.81	483.33	98%	612	227	132
   deeplab-v3-resnet50	xnnpack	fp32	522.28	520.08	512.06	98%	637	240	159
   fcn-resnet101	xnnpack	fp32	669.64	659.59	651.75	99%	687	186	208
   deeplab-v3-resnet101	xnnpack	fp32	791.02	766.57	755.11	99%	761	294	234

   keypoint-detection

   variant	backend	precision	median ms	mean ms	ET ms	share	peak MB	load ms	size MB
   blazeface	xnnpack	fp32	9.29	9.37	8.87	95%	238	2	1
   yolo26-pose-size-384	xnnpack	fp32	18.98	27.26	26.42	97%	255	21	12
   yolo26-pose-size-640	xnnpack	fp32	52.54	63.04	61.86	98%	293	20	12
   yolo26-pose-size-512	xnnpack	fp32	53.50	55.27	54.48	99%	271	24	12
   rfdetr-keypoint	xnnpack	fp32	788.95	792.66	791.71	100%	1017	134	146

   object-detection

   variant	backend	precision	median ms	mean ms	ET ms	share	peak MB	load ms	size MB
   ssdlite320-mobilenet-v3-large	xnnpack	fp32	22.26	23.99	22.91	96%	230	10	14
   yolo26-nano-size-384	xnnpack	fp32	39.26	39.11	38.27	98%	224	7	10
   yolo26-nano-size-512	xnnpack	fp32	39.95	51.02	50.03	98%	247	13	10
   yolo26-small-size-384	xnnpack	fp32	42.15	42.00	41.34	98%	283	31	38
   yolo26-small-size-512	xnnpack	fp32	63.08	59.86	58.89	98%	309	42	38
   yolo26-nano-size-640	xnnpack	fp32	75.77	77.39	76.32	99%	275	18	10
   yolo26-medium-size-384	xnnpack	fp32	86.73	87.73	86.98	99%	358	72	82
   yolo26-small-size-640	xnnpack	fp32	92.02	98.48	97.55	99%	349	34	38
   yolo26-large-size-384	xnnpack	fp32	115.62	113.10	112.29	99%	388	87	100
   yolo26-medium-size-512	xnnpack	fp32	151.40	159.31	158.43	99%	411	67	82
   rfdetr-nano	xnnpack	fp32	153.42	160.06	159.25	99%	335	72	112
   yolo26-large-size-512	xnnpack	fp32	188.97	190.69	189.75	100%	427	90	100
   yolo26-xlarge-size-384	xnnpack	fp32	202.27	203.77	202.86	100%	557	179	223
   yolo26-medium-size-640	xnnpack	fp32	208.26	208.06	207.17	100%	478	77	82
   yolo26-large-size-640	xnnpack	fp32	253.26	254.76	253.78	100%	500	85	100
   yolo26-xlarge-size-512	xnnpack	fp32	293.75	295.64	294.67	100%	629	174	223
   yolo26-xlarge-size-640	xnnpack	fp32	397.96	400.44	399.43	100%	699	175	223

   instance-segmentation

   Post-processing is mask-bound; read Execute %.

   variant	backend	precision	median ms	mean ms	ET ms	share	peak MB	load ms	size MB
   yolo26-nano-size-384	xnnpack	fp32	25.74	26.46	25.40	96%	372	18	11
   yolo26-nano-size-512	xnnpack	fp32	44.22	44.09	42.32	96%	397	20	11
   yolo26-small-size-384	xnnpack	fp32	56.44	54.95	53.63	98%	394	44	42
   yolo26-nano-size-640	xnnpack	fp32	73.49	78.23	75.62	97%	458	28	11
   yolo26-small-size-512	xnnpack	fp32	96.03	98.23	96.71	98%	436	49	42
   fastsam-s	xnnpack	fp32	131.96	133.25	130.38	98%	468	62	47
   yolo26-medium-size-384	xnnpack	fp32	143.04	155.93	154.63	99%	460	89	95
   yolo26-small-size-640	xnnpack	fp32	143.39	155.36	153.05	99%	499	47	42
   yolo26-large-size-384	xnnpack	fp32	158.58	169.21	167.87	99%	403	85	112
   rfdetr-nano	xnnpack	fp32	161.67	158.73	157.85	99%	494	78	124
   yolo26-medium-size-512	xnnpack	fp32	231.94	234.96	233.07	99%	523	89	95
   yolo26-large-size-512	xnnpack	fp32	235.35	244.08	242.42	99%	464	78	112
   yolo26-xlarge-size-384	xnnpack	fp32	251.27	255.41	254.14	100%	574	157	252
   yolo26-medium-size-640	xnnpack	fp32	282.58	285.26	283.49	99%	531	75	95
   yolo26-large-size-640	xnnpack	fp32	320.86	325.93	324.14	99%	548	85	112
   yolo26-xlarge-size-512	xnnpack	fp32	384.07	384.14	382.65	100%	660	158	252
   fastsam-x	xnnpack	fp32	548.29	550.43	547.66	99%	824	271	289
   yolo26-xlarge-size-640	xnnpack	fp32	557.96	551.60	549.25	100%	752	189	252

   privacy-filter

   variant	backend	precision	median ms	mean ms	ET ms	share	peak MB	load ms	size MB
   openai-xnnpack-8-da4-w	xnnpack	8da4w	255.30	252.78	251.25	99%	2151	2395	1270
   nemotron-xnnpack-8-da4-w	xnnpack	8da4w	258.73	251.88	247.41	98%	2047	2182	1270

   text-embeddings

   variant	backend	precision	median ms	mean ms	ET ms	share	peak MB	load ms	size MB
   all-minilm-l6-v2	vulkan	fp16	3.82	3.82	3.73	98%	337	79	46
   multi-qa-minilm-l6-cos-v1	vulkan	fp16	3.94	3.99	3.85	97%	368	92	46
   multi-qa-minilm-l6-cos-v1	xnnpack	fp32	5.54	5.60	5.41	97%	408	75	91
   all-minilm-l6-v2	xnnpack	fp32	6.57	6.72	6.52	97%	349	101	91
   paraphrase-multilingual-minilm-l12-v2	vulkan	fp16	7.42	7.44	7.33	99%	686	918	252
   distiluse-base-multilingual-cased-v2-xnnpack-8-da4-w	xnnpack	8da4w	7.91	7.93	7.81	98%	691	351	396
   paraphrase-multilingual-minilm-l12-v2-xnnpack-8-da4-w	xnnpack	8da4w	8.95	9.10	8.94	98%	827	949	414
   distiluse-base-multilingual-cased-v2	vulkan	fp16	10.13	10.14	10.06	99%	586	382	273
   clip-vit-base-patch32-text	vulkan	fp16	12.35	12.38	11.90	96%	456	239	129
   paraphrase-multilingual-minilm-l12-v2	xnnpack	fp32	13.24	13.36	12.94	97%	893	1052	487
   distiluse-base-multilingual-cased-v2	xnnpack	fp32	13.78	13.74	13.53	98%	820	487	544
   clip-vit-base-patch32-text	xnnpack	fp32	16.84	16.83	16.14	96%	560	252	256
   multi-qa-mpnet-base-dot-v1	xnnpack	fp32	17.28	17.34	17.12	99%	758	320	437
   all-mpnet-base-v2	vulkan	fp16	17.41	17.42	17.31	99%	553	246	219
   all-mpnet-base-v2	xnnpack	fp32	17.42	17.47	17.25	99%	725	394	437
   multi-qa-mpnet-base-dot-v1	vulkan	fp16	17.59	17.72	17.54	99%	587	226	219
   all-mpnet-base-v2	vulkan	int8	22.34	22.35	22.25	100%	460	180	134
   multi-qa-mpnet-base-dot-v1	vulkan	int8	22.55	22.58	22.48	100%	491	151	134
   lfm2-5-embedding-350-m-xnnpack-8-da4-w	xnnpack	8da4w	37.72	38.00	37.55	99%	943	871	580

   image-embeddings

   variant	backend	precision	median ms	mean ms	ET ms	share	peak MB	load ms	size MB
   clip-vit-base-patch32	vulkan	fp16	17.39	17.29	17.12	99%	516	108	176
   clip-vit-base-patch32	xnnpack	fp32	24.64	25.64	25.17	98%	673	248	352

   ocr

   Recognizer cost scales with the regions the detector finds.

   variant	backend	precision	median ms	mean ms	ET ms	share	peak MB	load ms	size MB
   paddle-ppocrv6-small-vulkan	vulkan	default	38.78	38.85	34.29	88%	583	45	26
   paddle-ppocrv6-small-xnnpack	xnnpack	default	187.74	184.88	179.63	97%	569	74	24
   paddle-ppocrv6-small	xnnpack	fp32	239.68	237.42	231.64	98%	690	65	31

   voice-activity-detection

   variant	backend	precision	median ms	mean ms	ET ms	share	peak MB	load ms	size MB
   fsmn-vad	xnnpack	fp32	8.96	8.91	7.70	86%	244	3	2

   speech-to-text

   Decode length is input-dependent; read Execute %.

   variant	backend	precision	median ms	mean ms	ET ms	share	peak MB	load ms	size MB
   whisper-en-tiny	vulkan	int8	74.02	74.00	72.61	98%	639	142	134
   whisper-tiny	vulkan	int8	79.43	79.93	78.36	98%	621	137	134
   whisper-en-tiny	vulkan	fp16	81.07	81.30	80.02	98%	716	146	218
   whisper-tiny	vulkan	fp16	91.34	91.35	89.78	98%	698	158	218
   whisper-en-tiny	xnnpack	int8	118.51	118.66	117.59	99%	536	159	180
   whisper-en-tiny	xnnpack	fp32	143.66	146.23	145.06	99%	700	143	237
   whisper-tiny	xnnpack	fp32	170.11	168.75	167.33	99%	659	176	237
   whisper-base	vulkan	int8	178.60	177.79	175.19	99%	821	206	210
   whisper-en-base	vulkan	fp16	188.89	190.40	188.02	99%	987	204	355
   whisper-en-base	vulkan	int8	202.90	202.49	199.25	98%	858	216	210
   whisper-base	vulkan	fp16	207.76	206.18	203.63	99%	963	249	355
   whisper-en-base	xnnpack	int8	256.91	255.04	253.60	99%	676	192	252
   whisper-en-base	xnnpack	fp32	397.55	402.24	400.24	100%	930	223	403
   whisper-base	xnnpack	fp32	400.47	400.63	398.19	99%	917	234	403
   whisper-en-small	vulkan	int8	619.10	703.88	697.56	99%	1589	470	512
   whisper-small	vulkan	int8	647.84	642.87	636.21	99%	1577	415	512
   whisper-small	vulkan	fp16	751.62	811.09	804.47	99%	1971	682	928
   whisper-en-small	vulkan	fp16	873.17	866.83	857.78	99%	1944	846	928
   whisper-en-small	xnnpack	int8	1034.09	1028.65	1026.59	100%	971	349	452
   whisper-en-small	xnnpack	fp32	1732.60	1684.70	1682.12	100%	1761	619	1133
   whisper-small	xnnpack	fp32	2121.47	2010.64	2007.44	100%	1744	868	1133

   text-to-speech

   Timed on the RN thread; includes per-chunk thread hops.

   The nine Kokoro entries are three networks, not nine: en-us, en-gb, es, fr, it, pt and hi all resolve to the same standard bundle, pl to polish and de to german. What differs between rows of the same bundle is the phonemizer and the length of that language's test sentence, so the spread here is input cost, not model cost.

   Read the median for kokoro-pt: one of its 20 iterations stalled for 217 s, which is what drags the mean to 13.5 s. Its next-slowest iteration is 5.0 s, in line with the rest.

   variant	backend	precision	median ms	mean ms	ET ms	share	peak MB	load ms	size MB
   supertonic	vulkan	fp16	660.49	852.50	838.13	98%	1586	317	202
   supertonic	xnnpack	fp32	904.83	905.55	895.71	99%	1154	344	401
   kokoro-en-us	xnnpack	fp32	1339.09	1422.17	1409.82	99%	1234	798	350
   kokoro-fr	xnnpack	fp32	1841.53	1830.07	1806.07	99%	1181	453	339
   kokoro-de	xnnpack	fp32	1859.56	1845.00	1821.68	99%	1180	439	339
   kokoro-en-gb	xnnpack	fp32	1873.29	1861.76	1844.43	99%	1257	667	349
   kokoro-it	xnnpack	fp32	1895.42	1926.05	1903.15	99%	1133	488	340
   kokoro-es	xnnpack	fp32	1935.62	2711.84	2684.47	99%	1118	493	340
   kokoro-pl	xnnpack	fp32	1963.21	2600.01	2568.60	99%	1111	453	339
   kokoro-hi	xnnpack	fp32	2173.63	2229.65	2198.41	99%	1178	806	343
   kokoro-pt	xnnpack	fp32	2657.99	13464.04	13434.33	100%	1079	476	340

   text-to-image

   variant	backend	precision	median ms	mean ms	ET ms	share	peak MB	load ms	size MB
   sdxs-512-dreamshaper	xnnpack	fp32	1216.52	1154.80	1145.35	99%	2559	1509	1764
9. msluszniak

   modified the milestones: v1.0.0, v0.11.0 on Sep 22, 2026
10. msluszniak

    self-assigned this

    on Oct 7, 2026
11. msluszniak

    linked a pull request that will close this issue[RNE Rewrite] feat: benchmark harness for every registry variant #1363on Oct 7, 2026
12. msluszniak

    added a commit that references this issue on Oct 7, 2026

    docs(benchmarks): version the benchmark data with the docs (#1518)

    ...

    2655c91