Shipping on-device LLMs on iPhone: 5 lessons from benchmarking 9 models.

We ship apps where the AI runs on the phone itself, with no server behind it. To choose models for Ragondin Assistant, we ran nine open LLMs through 220 benchmark questions on an iPhone 17e with 8 GB of memory. The full results are on the Ragondin Assistant blog. This post is about what the exercise taught us about building on-device AI features, the part that matters if you are planning one.

1. Fitting in memory is not the same as surviving load

Bonsai 27B, compressed to about one bit per weight, is a 3.8 GB file. It loads on an 8 GB iPhone and chats normally. Under a benchmark, which keeps the GPU busy with no pause, the GPU hit a hardware error early in the run and iOS restarted it (IOGPUScheduler::hardware_error_interrupt, then GPURestartBegin). From then on, every generation returned empty in about 40 milliseconds, with no exception raised by the inference library.

What to do: test your model under sustained load, not only in a demo conversation, and detect empty or impossibly fast generations as failures. A silent empty answer is worse than an error message.

2. Read the raw outputs before you trust a score

Our first full run put Gemma 4 E2B far below where it belonged: 120 correct answers out of 220. The totals looked plausible. The answers did not: fragments of one answer appeared in the next. Gemma uses sliding-window attention, and the prompt cache that speeds up consecutive requests in llama.cpp was being reused across unrelated prompts, which is not safe for that kind of model. Clearing the KV cache between unrelated prompts for sliding-window models brought it to 147.

What to do: if you reuse the KV cache for speed, check the model's attention type first, and spot-check generated text whenever you change models or library versions.

3. Decide what "too long" means, and enforce it

We capped answers at 2,048 tokens and counted any answer still being written at the cap as wrong. On a phone, an answer the user never sees is no answer. That one rule moved a model from first to third on the phone: LFM2.5 2.6B would have scored 75.5 without it and scored 71.9 with it, because 33 of its 220 answers hit the cap.

What to do: treat output length as a product constraint. A small model that reasons at length can be both accurate and unusable.

4. Latency per answer beats tokens per second

LFM2.5 2.6B generates faster than Qwen 3.5 4B (14.8 against 10.5 tokens per second), yet it takes about a minute per question against about 18 seconds, because it writes a hidden reasoning draft before almost every answer, even with its thinking mode off. MiniCPM5 2B, about four points behind the leader, answers in about 8 seconds.

What to do: measure time to a complete answer on your real prompts, on the real device. Benchmarked tokens per second will mislead you.

5. Choose a model per device, and be honest about the margin

With 220 questions, scores are good to about ±5 points. The four best phone models, Qwen 3.5 4B (74.5), LFM2.5 2.6B (71.9), MiniCPM5 2B (70.3) and Gemma 4 E2B (69.1), sit inside that margin. Below 2 billion parameters, scores drop to between 34 and 56. The practical decision is not "which model is smartest" but which one answers fast enough, fits the device's memory, and keeps working through a long session.

What to do: pick the model from the device's memory at first launch, keep a smaller fallback, and plan for model files of 1.5 to 3 GB that users download once.

How we ran it

iPhone 17e (8 GB, iOS 27), llama.rn 0.12.9 on llama.cpp b10256, GPU inference inside a shipping React Native app. 220 questions with a fixed seed from MMLU-Pro, GSM8K, MGSM (French and Chinese) and IFEval, temperature 0, phone cooled to iOS's "nominal" thermal state before each model. Full method and results.

Planning an on-device AI feature?

Model choice per device, shipping gigabytes of weights, iOS memory limits, and the failure modes above are most of the work. We build these features for our own apps and for clients, in weeks. Talk to Ragondin Dev.