On-Device AI Performance Benchmarks: Apple Silicon vs Cloud APIs

Benchmark methodology

Bad benchmark methodology produces misleading numbers that lead to bad decisions. The following methodology produces results that generalize to real app behavior:

• —Physical hardware only. The iOS Simulator doesn't have a Neural Engine. All latency measurements must be taken on a physical device. Simulator benchmarks are meaningless for on-device AI.
• —Warm-up runs excluded. The first 1–3 predictions after model load include model compilation time (when using .mlpackage ). Measure steady-state latency after at least 5 warm-up predictions.
• —Median, not average. Core ML latency has a non-normal distribution — occasional outliers from thermal throttling or system activity skew averages. Use median of at least 50 measurements.
• —Thermal state controlled. Sustained inference generates heat. Benchmark at both ambient temperature and after 30 seconds of sustained load to measure throttled performance — what users experience during extended use.
• —Compute units declared explicitly. Use MLModelConfiguration to pin compute units. Benchmark both .all and .cpuAndNeuralEngine — sometimes excluding the GPU produces faster results for latency-sensitive tasks.

Apple's official benchmarking tool is the Core ML Performance Report in Xcode, available by clicking the Performance tab in the model inspector. It runs on-device benchmarks automatically and reports per-layer compute unit assignment.

Core ML inference latency across Apple Silicon

The following table shows median inference latency for standard model architectures across Apple Silicon generations. All measurements use FP16 precision, mlprogram format, compute units .cpuAndNeuralEngine, batch size 1.

Median of 100 measurements at ambient temperature, physical devices, iOS 17 / macOS 14. Input size: 224×224 for vision models, 128 tokens for NLP. Values rounded to nearest 0.5ms.

On-device vs cloud inference latency

For practical decision-making, the comparison is total latency including network round-trip. Cloud inference adds 100–400ms of latency under typical conditions — not just API processing time.

* Foundation Models first-token latency for short prompts. Cloud API latency measured from real apps to GPT-4o (US East region) under normal load conditions.

Quantization impact on performance

Quantization affects both model size and inference latency via two mechanisms: reduced memory bandwidth (fewer bytes to load) and native precision matching (ANE runs FP16 natively). The combined effect depends on whether your model is compute-bound or memory-bandwidth-bound.

Measurements on iPhone 13 Pro (A15), iOS 17, 100-run median. Accuracy measured on 10K ImageNet validation subset. 2-bit palettization accuracy loss (6.1 points) is significant and should only be used when model size is the hard constraint.

The practical recommendation: use FP16 as the default. Add 8-bit palettization if the model is over 10MB. Only go to 4-bit if size constraints demand it — validate accuracy carefully first. See the full optimization workflow in Core ML Optimization Techniques.

Compute unit impact

The choice of compute units in MLModelConfiguration has a significant impact on latency. The ANE is fastest for latency-optimized use cases; the GPU is fastest for throughput-optimized (multiple concurrent inferences). The CPU is usually slowest but most compatible.

For interactive single-inference use cases (user taps, camera classification), use .cpuAndNeuralEngine. It achieves the same latency as .all while consuming significantly less power — important for sustained use on battery.

Using benchmark data for shipping decisions

Benchmarks answer one question: is the model fast enough on the target hardware? Define your performance budget before looking at benchmarks — what latency is acceptable for your use case?

• —Real-time camera inference: budget of ~16ms (60fps) or ~33ms (30fps) for the inference step. Most mobilenet-class models on A13+ are well within this.
• —Interactive single prediction: under 200ms total from user action to displayed result. Inference latency is rarely the bottleneck — image preprocessing and UI updates usually take longer.
• —Background batch processing: no hard latency constraint — optimize for throughput (images/second) and battery impact while a background task runs.

Benchmark on the oldest device in your supported hardware range. If you support iOS 15, test on an A12 device — not on the latest hardware. Performance gaps between generations are significant.

Common questions