Edge AI Bench

Same ONNX models, different devices and execution providers. Generated 2026-11-17 17:10 UTC from 1 result file(s).

Devices

Apple M5 Max (Darwin 27.2)

CPU
Apple M5 Max
Model
Mac17,6
OS
Darwin 27.2
RAM
48.0 GB
ONNX Runtime
1.32.0
EPs available
CoreML, Azure, CPU
Measured
2026-11-17 17:10 UTC

Skipped EPs:

  • qnn: QNNExecutionProvider not in this onnxruntime build. Install `onnxruntime-qnn` on a Snapdragon device (Windows on Snapdragon / supported Linux). Android/Termux CPU builds do not include QNN.
  • cuda: CUDAExecutionProvider not in this onnxruntime build. Install `onnxruntime-gpu` on a machine with an NVIDIA GPU and CUDA runtime.

digits-cnn

MobileNet-style depthwise-separable CNN, 1x32x32 digits, 10 classes

Apple M5 Max (Darwin 27.2) · INT8 · cpup50 0.078 ms · p90 0.081 ms0.078 msApple M5 Max (Darwin 27.2) · INT8 · coremlp50 0.084 ms · p90 0.087 ms0.084 msApple M5 Max (Darwin 27.2) · FP32 · coremlp50 0.101 ms · p90 0.105 ms0.101 msApple M5 Max (Darwin 27.2) · FP32 · cpup50 0.110 ms · p90 0.122 ms0.110 msApple M5 Max (Darwin 27.2) · W8A16 · cpup50 0.158 ms · p90 0.175 ms0.158 msApple M5 Max (Darwin 27.2) · W8A16 · coremlp50 0.172 ms · p90 0.192 ms0.172 msbar = p50 latency (batch 1), thin line = p90

Quantized (QDQ) vs FP32

DeviceEPVariantFP32 accQuant accΔ accp50 speed-upSize reduction
Apple M5 Max (Darwin 27.2)cpuINT897.49%97.77%0.28 pp1.41×2.99×
Apple M5 Max (Darwin 27.2)cpuW8A1697.49%97.49%0.00 pp0.70×3.00×
Apple M5 Max (Darwin 27.2)coremlINT897.49%97.77%0.28 pp1.20×2.99×
Apple M5 Max (Darwin 27.2)coremlW8A1697.49%97.49%0.00 pp0.58×3.00×

All measurements

DeviceVariantEPp50p90p99ThroughputEP coverageLoadPeak RSSSizeAccuracy
Apple M5 Max (Darwin 27.2)FP32coreml0.101 ms0.105 ms0.142 ms42,622/s (b=16)24/24 nodes, 1 part.78 ms94 MB287 KB97.49%
Apple M5 Max (Darwin 27.2)FP32cpu0.110 ms0.122 ms0.134 ms39,781/s (b=16)—15 ms75 MB287 KB97.49%
Apple M5 Max (Darwin 27.2)INT8coreml0.084 ms0.087 ms0.117 ms18,671/s (b=16)1/65 nodes, 1 part.68 ms83 MB96 KB97.77%
Apple M5 Max (Darwin 27.2)INT8cpu0.078 ms0.081 ms0.087 ms19,842/s (b=16)—16 ms69 MB96 KB97.77%
Apple M5 Max (Darwin 27.2)W8A16coreml0.172 ms0.192 ms0.199 ms22,858/s (b=16)1/65 nodes, 1 part.59 ms89 MB96 KB97.49%
Apple M5 Max (Darwin 27.2)W8A16cpu0.158 ms0.175 ms0.220 ms23,812/s (b=16)—18 ms72 MB96 KB97.49%

sms-transformer

2-layer byte-level transformer encoder (d=96), SMS spam vs ham

Apple M5 Max (Darwin 27.2) · FP32 · coremlp50 0.227 ms · p90 0.238 ms0.227 msApple M5 Max (Darwin 27.2) · FP32 · cpup50 0.291 ms · p90 0.304 ms0.291 msApple M5 Max (Darwin 27.2) · INT8 · cpup50 0.436 ms · p90 0.510 ms0.436 msApple M5 Max (Darwin 27.2) · W8A16 · cpup50 0.697 ms · p90 0.781 ms0.697 msApple M5 Max (Darwin 27.2) · W8A16 · coremlp50 1.856 ms · p90 2.145 ms1.856 msApple M5 Max (Darwin 27.2) · INT8 · coremlp50 1.857 ms · p90 2.236 ms1.857 msbar = p50 latency (batch 1), thin line = p90

Quantized (QDQ) vs FP32

DeviceEPVariantFP32 accQuant accΔ accp50 speed-upSize reduction
Apple M5 Max (Darwin 27.2)cpuINT897.40%94.34%-3.05 pp0.67×2.85×
Apple M5 Max (Darwin 27.2)cpuW8A1697.40%97.40%0.00 pp0.42×2.50×
Apple M5 Max (Darwin 27.2)coremlINT897.40%94.34%-3.05 pp0.12×2.85×
Apple M5 Max (Darwin 27.2)coremlW8A1697.40%97.40%0.00 pp0.12×2.50×

All measurements

DeviceVariantEPp50p90p99ThroughputEP coverageLoadPeak RSSSizeAccuracy
Apple M5 Max (Darwin 27.2)FP32coreml0.227 ms0.238 ms0.270 ms31,528/s (b=16)87/88 nodes, 1 part.248 ms112 MB751 KB97.40%
Apple M5 Max (Darwin 27.2)FP32cpu0.291 ms0.304 ms0.348 ms8,335/s (b=16)—20 ms104 MB751 KB97.40%
Apple M5 Max (Darwin 27.2)INT8coreml1.857 ms2.236 ms2.588 ms1,621/s (b=16)59/308 nodes, 45 part.1,207 ms212 MB263 KB94.34%
Apple M5 Max (Darwin 27.2)INT8cpu0.436 ms0.510 ms0.532 ms4,514/s (b=16)—23 ms107 MB263 KB94.34%
Apple M5 Max (Darwin 27.2)W8A16coreml1.856 ms2.145 ms2.387 ms1,603/s (b=16)59/308 nodes, 45 part.855 ms215 MB300 KB97.40%
Apple M5 Max (Darwin 27.2)W8A16cpu0.697 ms0.781 ms0.819 ms3,461/s (b=16)—24 ms115 MB300 KB97.40%

mobilenetv3-small

torchvision MobileNetV3-Small (ImageNet), 3x224x224 - latency/size only

Apple M5 Max (Darwin 27.2) · INT8 · cpup50 0.747 ms · p90 0.894 ms0.747 msApple M5 Max (Darwin 27.2) · FP32 · cpup50 1.059 ms · p90 1.123 ms1.059 msApple M5 Max (Darwin 27.2) · W8A16 · cpup50 2.075 ms · p90 2.211 ms2.075 msApple M5 Max (Darwin 27.2) · FP32 · coremlp50 2.993 ms · p90 3.030 ms2.993 msApple M5 Max (Darwin 27.2) · INT8 · coremlp50 4.906 ms · p90 5.015 ms4.906 msApple M5 Max (Darwin 27.2) · W8A16 · coremlp50 5.751 ms · p90 6.213 ms5.751 msbar = p50 latency (batch 1), thin line = p90

Quantized (QDQ) vs FP32

DeviceEPVariantFP32 accQuant accΔ accp50 speed-upSize reduction
Apple M5 Max (Darwin 27.2)cpuINT8———1.42×3.68×
Apple M5 Max (Darwin 27.2)cpuW8A16———0.51×3.68×
Apple M5 Max (Darwin 27.2)coremlINT8———0.61×3.68×
Apple M5 Max (Darwin 27.2)coremlW8A16———0.52×3.68×

All measurements

DeviceVariantEPp50p90p99ThroughputEP coverageLoadPeak RSSSizeAccuracy
Apple M5 Max (Darwin 27.2)FP32coreml2.993 ms3.030 ms3.061 ms628/s (b=16)38/60 nodes, 19 part.1,002 ms440 MB9,942 KB—
Apple M5 Max (Darwin 27.2)FP32cpu1.059 ms1.123 ms1.152 ms1,958/s (b=16)—21 ms189 MB9,942 KB—
Apple M5 Max (Darwin 27.2)INT8coreml4.906 ms5.015 ms5.432 ms352/s (b=16)72/525 nodes, 72 part.1,326 ms554 MB2,699 KB—
Apple M5 Max (Darwin 27.2)INT8cpu0.747 ms0.894 ms0.920 ms2,053/s (b=16)—28 ms137 MB2,699 KB—
Apple M5 Max (Darwin 27.2)W8A16coreml5.751 ms6.213 ms6.740 ms388/s (b=16)72/525 nodes, 72 part.1,350 ms564 MB2,704 KB—
Apple M5 Max (Darwin 27.2)W8A16cpu2.075 ms2.211 ms2.289 ms1,050/s (b=16)—29 ms201 MB2,704 KB—