offgrid:AI: Shipping Fully Offline LLM Inference on iOS

Context

In 2024, every AI assistant app on iOS required an active internet connection and transmitted user prompts to cloud infrastructure. The market assumption was that language model inference was too compute-intensive to run on a mobile device. The use cases for a genuinely offline AI assistant were real and unserved: field workers without reliable connectivity, travelers in areas with high data costs, privacy-conscious users who would not send prompts to a cloud API, and emergency preparedness scenarios where connectivity cannot be assumed.

Problem

The technical barriers to production-viable on-device LLM inference on iOS in 2024 were not theoretical — they were real constraints that had to be solved simultaneously:

• —Model size: a usable language model is 3–16 GB. That's a significant portion of a device's storage.
• —Memory: LLM inference requires holding model weights and the KV cache in memory simultaneously. The iPhone's unified memory architecture helps, but context window size is directly limited by available RAM.
• —Battery: sustained inference draws significant CPU and Neural Engine power. An app that drains 50% battery per hour is not useful.
• —Apple Foundation Models framework: not available until iOS 26. A cross-version strategy was required.
• —App Store: Apple's guidelines restrict some model hosting patterns. Approval required deliberate preparation.

Architecture

Implementation: Token Streaming to SwiftUI

llama.cpp produces tokens via a C callback. Bridging that to SwiftUI's reactive update model requires an AsyncStream that emits each token as it's generated:

// Bridge llama.cpp token callback to Swift AsyncStream
actor InferenceEngine {
    private var model: OpaquePointer?
    private var context: OpaquePointer?

    func generate(prompt: String) -> AsyncStream<String> {
        AsyncStream { continuation in
            Task.detached(priority: .userInitiated) { [weak self] in
                guard let self else { return }

                let tokens = await self.tokenize(prompt)
                var response = ""

                for token in await self.generateTokens(from: tokens) {
                    // Check thermal state before each token
                    let thermal = ProcessInfo.processInfo.thermalState
                    if thermal == .critical {
                        await self.throttleInference()
                    }

                    let piece = await self.tokenToPiece(token)
                    response += piece
                    continuation.yield(piece)
                }

                continuation.finish()
            }
        }
    }
}

// In SwiftUI
struct ChatView: View {
    @State private var response = ""

    var body: some View {
        Text(response)
            .task {
                for await token in engine.generate(prompt: userMessage) {
                    response += token
                }
            }
    }
}

Key Technical Learnings

Technical FAQ