@mmerterden/multi-agent-pipeline 20.7.0 → 20.8.0

package/pipeline/skills/shared/external/apple-on-device-ai/references/mlx-swift.md

@@ -1,289 +1,275 @@
-# MLX Swift & llama.cpp Reference
-
-Complete reference for running open-source LLMs on Apple platforms using
-MLX Swift and llama.cpp.
+# MLX Swift, llama.cpp and routing between backends
 
 ## Contents
 
-- [MLX Swift](#mlx-swift)
+- [How MLX works](#how-mlx-works)
+- [Loading and generating](#loading-and-generating)
+- [Model sizes by device](#model-sizes-by-device)
+- [Memory rules](#memory-rules)
+- [Owning the model lifecycle](#owning-the-model-lifecycle)
+- [Going to the background](#going-to-the-background)
 - [llama.cpp](#llamacpp)
-- [Multi-Backend Architecture](#multi-backend-architecture)
-- [Built-in Apple Frameworks](#built-in-apple-frameworks)
-- [Performance Best Practices](#performance-best-practices)
-- [Review Checklist](#review-checklist)
-
-## MLX Swift
-
-Apple's ML framework for Swift. Highest sustained generation throughput on
-Apple Silicon via unified memory architecture.
+- [GGUF quant types](#gguf-quant-types)
+- [llama.cpp or MLX Swift](#llamacpp-or-mlx-swift)
+- [Routing across backends](#routing-across-backends)
+- [Built-in frameworks first](#built-in-frameworks-first)
+- [Performance habits](#performance-habits)
+- [Checklist](#checklist)
 
-### Key Characteristics
+## How MLX works
 
-- Unified memory: operations run on CPU or GPU without data transfer
-- Lazy computation: operations computed only when needed
-- Automatic differentiation for training
-- Metal GPU acceleration
-- Research-oriented but increasingly used in production
+- CPU and GPU share unified memory, so an array can move between them without
+  a copy.
+- Evaluation is lazy; values are computed only when something needs them.
+- It has automatic differentiation for training and runs on the GPU through
+  Metal.
+- It started as a research tool and is used more and more in shipping apps.
 
-### Loading and Running LLMs
+## Loading and generating
 
 ```swift
-import MLX
-import MLXLLM
-import MLXLMCommon
-import MLXLMHFAPI
 import MLXLMTokenizers
+import MLXLMHFAPI
+import MLXLMCommon
+import MLXLLM
+import MLX
 
+let llama = ModelConfiguration(id: "mlx-community/Llama-3.2-3B-Instruct-4bit")
 let container = try await LLMModelFactory.shared.loadContainer(
     from: HubClient.default,
     using: TokenizersLoader(),
-    configuration: .init(id: "mlx-community/Qwen3-4B-4bit")
+    configuration: llama
 )
+```
 
-let session = ChatSession(container)
-print(try await session.respond(to: "Hello"))
+`ChatSession(container)` is the simple path. For control over streaming and
+sampling, drive the container directly:
 
-// Use ModelContainer directly when you need streaming control.
-let input = try await container.prepare(input: UserInput(prompt: "Hello"))
-let stream = try await container.generate(
-    input: input,
-    parameters: GenerateParameters(temperature: 0.0)
-)
-for await event in stream {
-    if case .chunk(let text) = event {
-        print(text, terminator: "")
-    }
+```swift
+let question = UserInput(prompt: "Name three knots for climbing.")
+let prepared = try await container.prepare(input: question)
+let greedy = GenerateParameters(temperature: 0)
+let events = try await container.generate(input: prepared, parameters: greedy)
+for await item in events {
+    guard case .chunk(let piece) = item else { continue }
+    transcript += piece
 }
 ```
 
-### Recommended Models by Device
+## Model sizes by device
+
+| Device | Download size | Resident memory | Suggested model |
+|---|---|---|---|
+| iPhone 12 to 14 | about 278 MB | about 0.3 GB | SmolLM2-135M, or Qwen 2.5 at 0.5B |
+| iPhone 15 Pro or newer | about 2.7 GB | about 3.5 GB | Gemma 3n E4B at 4 bits |
+| Mac, 8 GB | about 1.8 GB | about 3 GB | Llama 3.2 3B at 4 bits |
+| Mac, 16 GB or more | about 4 GB | about 6 GB | Mistral 7B at 4 bits |
 
-| Device | RAM | Recommended Model | Disk Size | RAM Usage |
-|---|---|---|---|---|
-| iPhone 12-14 | 4-6 GB | SmolLM2-135M or Qwen 2.5 0.5B | ~278 MB | ~0.3 GB |
-| iPhone 15 Pro+ | 8 GB | Gemma 3n E4B 4-bit | ~2.7 GB | ~3.5 GB |
-| Mac 8 GB | 8 GB | Llama 3.2 3B 4-bit | ~1.8 GB | ~3 GB |
-| Mac 16 GB+ | 16 GB+ | Mistral 7B 4-bit | ~4 GB | ~6 GB |
+## Memory rules
 
-### Memory Management Rules
+- On iOS keep total use under 60% of device RAM.
+- Cap the buffer cache: `Memory.cacheLimit = 512 * 1024 * 1024`.
+- Read `Memory.snapshot()` to watch usage, and lower the cache limit when the
+  system reports pressure.
+- Before loading a model, check that the free memory can hold it.
+- For larger models on iOS, add the Increased Memory Limit entitlement.
 
-1. Never exceed 60% of total RAM on iOS
-2. Set MLX cache limits:
-   ```swift
-   Memory.cacheLimit = 512 * 1024 * 1024 // 512 MB
-   ```
-3. Monitor memory pressure with `Memory.snapshot()` and reduce cache under pressure
-4. Unload MLX and llama.cpp models on backgrounding or memory pressure; for MLX,
-   also call `Memory.clearCache()` after generation-heavy phases
-5. Use "Increased Memory Limit" entitlement for larger models on iOS
-6. Pre-flight memory checks before loading models
-7. Validate MLX Swift and llama.cpp on physical Apple Silicon; Simulator cannot
-   exercise Metal-dependent inference, memory, or performance
+## Owning the model lifecycle
 
-### Model Lifecycle Management
+Keep the loaded model in one object that knows whether it is loaded, whether it
+is generating, and how to let go of it.
 
 ```swift
+import Observation
+
+@MainActor
 @Observable
-class ModelManager {
-    private var model: ModelContainer?
-    private var generationCount = 0
-
-    func loadModel() async throws {
-        model = try await LLMModelFactory.shared.loadContainer(
-            from: HubClient.default,
-            using: TokenizersLoader(),
-            configuration: .init(id: "mlx-community/Qwen3-4B-4bit")
-        )
+final class LocalModelHost {
+    private(set) var container: ModelContainer?
+    private(set) var runningGenerations = 0
+    private var generationTask: Task<Void, Never>?
+
+    var isBusy: Bool { runningGenerations > 0 }
+
+    func load(_ configuration: ModelConfiguration) async throws {
+        guard container == nil else { return }
+        let factory = LLMModelFactory.shared
+        container = try await factory.loadContainer(
+            from: HubClient.default, using: TokenizersLoader(), configuration: configuration)
     }
 
-    func unloadModel() {
-        model = nil
+    func cancelGeneration() { generationTask?.cancel() }
+
+    func unload() {
+        generationTask?.cancel()
+        container = nil
         Memory.clearCache()
     }
+
+    func track(_ task: Task<Void, Never>) {
+        generationTask = task
+        runningGenerations += 1
+        Task { await task.value; runningGenerations -= 1 }
+    }
 }
 ```
 
-Key lifecycle patterns:
-- Track active generation count to distinguish "loaded but idle" from
-  "generating"
-- Unconditional cancellation on app backgrounding
-- 5-second delayed force-unload after backgrounding
-- Platform-specific memory monitoring (UIKit on iOS, DispatchSource on macOS)
+- Count active generations so "loaded but idle" is distinguishable from
+  "working".
+- When the app goes to the background, cancel generation every time, without
+  exceptions.
+- After a short grace period in the background (five seconds works well),
+  unload the model.
+- Memory warnings arrive differently per platform: through UIKit notifications
+  on iOS and through a `DispatchSource` memory-pressure source on macOS.
 
-### Background Handling
+## Going to the background
 
 ```swift
-// iOS: Observe app lifecycle
-NotificationCenter.default.addObserver(
-    forName: UIApplication.didEnterBackgroundNotification,
-    object: nil, queue: .main
-) { _ in
-    modelManager.cancelGeneration()
-    Task {
-        try await Task.sleep(for: .seconds(5))
-        modelManager.unloadModel()
+import UIKit
+
+@MainActor
+func releaseModelWhenBackgrounded(_ host: LocalModelHost) async {
+    let grace = Duration.seconds(5)
+    let moves = NotificationCenter.default.notifications(
+        named: UIApplication.didEnterBackgroundNotification)
+    for await _ in moves {
+        host.cancelGeneration()
+        try? await Task.sleep(for: grace)
+        host.unload()
     }
 }
 ```
 
 ## llama.cpp
 
-C/C++ LLM inference engine. Best cross-platform support. Uses GGUF model format.
-
-### Swift Integration (swift-llama-cpp)
+llama.cpp is a C and C++ inference engine with the widest platform support. It
+loads GGUF files. From Swift, the `swift-llama-cpp` package wraps it:
 
 ```swift
+import Foundation
 import SwiftLlamaCpp
 
-let service = LlamaService(
-    modelUrl: modelURL,
-    config: .init(
-        batchSize: 256,
-        maxTokenCount: 4096,
-        useGPU: true
-    )
+let engine = LlamaService(
+    modelUrl: ggufURL,
+    config: .init(batchSize: 512, maxTokenCount: 2048, useGPU: true)
 )
-
-let messages = [
-    LlamaChatMessage(role: .system, content: "You are helpful."),
-    LlamaChatMessage(role: .user, content: "Hello")
+let conversation = [
+    LlamaChatMessage(role: .system, content: "Answer as a hiking guide."),
+    LlamaChatMessage(role: .user, content: "What should a day pack contain?")
 ]
-
-let stream = try await service.streamCompletion(
-    of: messages,
+for try await token in try await engine.streamCompletion(
+    of: conversation,
     samplingConfig: .init(temperature: 0.8)
-)
-for try await token in stream {
-    print(token, terminator: "")
+) {
+    reply += token
 }
 ```
 
-### GGUF Quantization Levels
+## GGUF quant types
 
-| Level | Quality | Size | Use Case |
+| Type | Pick it when | Output quality | File size |
 |---|---|---|---|
-| Q2_K | Lowest | Smallest | Extreme memory constraints |
-| Q4_K_M | Good | Balanced | Mobile devices (recommended) |
-| Q5_K_M | Higher | Larger | When quality matters more |
-| Q8_0 | Near-original | Largest | Desktop with ample RAM |
+| Q2_K | memory is extremely tight | lowest | smallest |
+| Q4_K_M | shipping on phones; the usual default | good | balanced |
+| Q5_K_M | a bit more quality is worth the space | higher | larger |
+| Q8_0 | desktops with plenty of RAM | close to the original | largest |
 
-### llama.cpp vs MLX Swift
+## llama.cpp or MLX Swift
 
-| Aspect | llama.cpp | MLX Swift |
+| Question | MLX Swift | llama.cpp |
 |---|---|---|
-| Model format | GGUF | Hugging Face / MLX format |
-| Platform support | Cross-platform | Apple only |
-| Throughput (Apple Silicon) | Good | Best |
-| Model ecosystem | Broadest | mlx-community models |
-| Maturity | Very mature | Evolving |
-| Memory efficiency | Excellent | Good |
+| Where does it run? | Apple platforms only | iOS, Android, desktop, servers |
+| Weights it loads | MLX / Hugging Face format | GGUF files |
+| Speed on Apple Silicon | the fastest | good |
+| How efficient with memory? | good | excellent |
+| How settled is it? | still changing quickly | long established |
+| Where do models come from? | the `mlx-community` organisation | the widest range of sources |
 
-## Multi-Backend Architecture
+## Routing across backends
 
-When an app needs multiple AI backends:
+Order of preference:
 
-### Fallback Chain Pattern
+1. Foundation Models, integrated with the system.
+2. MLX Swift, for throughput on Apple Silicon.
+3. llama.cpp, for compatibility.
+4. Nothing available: throw `AIError.noBackendAvailable`.
 
 ```swift
-func respond(to prompt: String) async throws -> String {
-    // Try Foundation Models first when available (system-integrated backend)
-    if SystemLanguageModel.default.isAvailable {
-        return try await foundationModelsRespond(prompt)
+enum AIError: Error { case noBackendAvailable }
+
+actor ModelRouter {
+    enum Backend: Sendable { case foundationModels, mlx, llamaCpp }
+
+    private(set) var activeBackend: Backend?
+
+    func resolve() async throws -> Backend {
+        if let activeBackend { return activeBackend }
+        let choice: Backend
+        if SystemLanguageModel.default.isAvailable {
+            choice = .foundationModels
+        } else if await MLXSupport.canLoad(for: DeviceTier.current) {
+            choice = .mlx
+        } else if LlamaSupport.hasBundledModel {
+            choice = .llamaCpp
+        } else {
+            throw AIError.noBackendAvailable
+        }
+        activeBackend = choice
+        return choice
     }
-
-    // Fall back to MLX Swift (best throughput)
-    if canLoadMLXModel() {
-        return try await mlxRespond(prompt)
-    }
-
-    // Fall back to llama.cpp (broadest compatibility)
-    if llamaModelAvailable() {
-        return try await llamaRespond(prompt)
-    }
-
-    throw AIError.noBackendAvailable
 }
 ```
 
-### Architecture Guidelines
-
-1. Create a router that checks Foundation Models availability first
-2. Fall back to MLX or llama.cpp when Foundation Models is unavailable
-3. Define model tiers based on device capabilities
-4. Serialize all model access through a coordinator actor to prevent contention
-5. Ensure tool systems work across backends (schema translation may be needed)
-
-### Coordinator Actor
-
-```swift
-actor ModelCoordinator {
-    private var activeBackend: Backend?
-
-    func withExclusiveAccess<T>(
-        _ work: () async throws -> T
-    ) async rethrows -> T {
-        try await work()
-    }
-
-    enum Backend {
-        case foundationModels
-        case mlx
-        case llamaCpp
-    }
-}
-```
-
-## Built-in Apple Frameworks
-
-Before reaching for custom models, consider built-in frameworks:
-
-### Natural Language Framework
+Guidelines:
 
-No model downloads required:
+- Ask the system model first; only when it is missing try MLX, then llama.cpp.
+- Define model tiers per device class so each device gets a model it can hold.
+- Serialise every model call through one coordinator actor (see SKILL.md for a
+  gate that actually queues requests).
+- Tools may need translating: a Foundation Models `Tool` schema is not the
+  format an open model expects, so map tool definitions per backend.
 
-- `NLLanguageRecognizer` -- Language detection
-- `NLTokenizer` -- Word, sentence, paragraph tokenization
-- `NLTagger` -- Parts of speech, named entity recognition, sentiment
-- `NLEmbedding` -- Word and sentence vectors, similarity search
+## Built-in frameworks first
 
-### Vision Framework
+Before shipping a custom model, check whether the system already does the job.
 
-Built-in computer vision (legacy `VN*` API; for iOS 18+ prefer modern Swift equivalents like `RecognizeTextRequest`):
+Natural Language, with nothing to download:
 
-- `VNRecognizeTextRequest` -- OCR
-- `VNClassifyImageRequest` -- Image classification
-- `VNDetectFaceRectanglesRequest` -- Face detection
-- `VNDetectHumanBodyPoseRequest` -- Body pose estimation
+- Which language is this? `NLLanguageRecognizer`.
+- Where are the word, sentence or paragraph boundaries? `NLTokenizer`.
+- Tagging for sentiment, named entities or parts of speech: `NLTagger`.
+- Vectors for words and sentences, and how close two are: `NLEmbedding`.
 
-### Create ML
+Vision (legacy `VN` names shown; from iOS 18 prefer the Swift request types
+such as `RecognizeTextRequest`):
 
-Training custom classifiers directly on device or Mac:
+- Reading text: `VNRecognizeTextRequest`.
+- Labelling a whole image: `VNClassifyImageRequest`.
+- Finding faces: `VNDetectFaceRectanglesRequest`.
+- Human body pose: `VNDetectHumanBodyPoseRequest`.
 
-- Image classification
-- Text classification
-- Tabular data models
-- Sound classification
+Create ML trains image, text, tabular and sound classifiers on a Mac or on
+device.
 
-## Performance Best Practices
+## Performance habits
 
-1. Run outside debugger for accurate benchmarks (Xcode: Cmd-Opt-R, uncheck
-   "Debug Executable")
-2. Use `session.prewarm()` for Foundation Models before user interaction
-3. Batch Vision framework requests in a single `perform()` call
-4. Use `.fast` recognition level for real-time camera processing
-5. Neural Engine (Core ML) is most energy-efficient for compatible operations
-6. For MLX Swift, monitor token generation speed and adjust model size if
-   below acceptable thresholds
+- Send several Vision requests in one `perform()` call.
+- Use the `.fast` recognition level for live camera text.
+- Core ML on the Neural Engine is the most energy-efficient option for the ops
+  it supports.
+- With MLX, track tokens per second; if it falls below what the feature needs,
+  move to a smaller model.
 
-## Review Checklist
+## Checklist
 
-- [ ] Model size appropriate for target device RAM
-- [ ] Memory pressure monitoring implemented
-- [ ] Models unloaded on app backgrounding
-- [ ] MLX cache limits set appropriately
-- [ ] Pre-flight memory check before loading large models
-- [ ] Fallback strategy when model unavailable
-- [ ] All model access serialized through coordinator
-- [ ] Quantization level appropriate for quality/size tradeoff
-- [ ] Energy efficiency considered (Neural Engine vs GPU)
-- [ ] Physical device testing (not simulator) for Metal-dependent code
+- [ ] The model fits the device's RAM.
+- [ ] Memory pressure is monitored.
+- [ ] Models unload when the app goes to the background.
+- [ ] MLX cache limit is set.
+- [ ] A memory check runs before loading a large model.
+- [ ] There is a fallback when no model is available.
+- [ ] All model access goes through one coordinator.
+- [ ] The quantization level matches the quality and size trade-off.
+- [ ] Energy use was considered (Neural Engine versus GPU).
+- [ ] Metal inference was tested on physical devices.