@mmerterden/multi-agent-pipeline 20.7.0 → 20.8.0

package/pipeline/skills/shared/external/apple-on-device-ai/references/coreml-optimization.md

@@ -1,347 +1,252 @@
-# Core ML Optimization Reference
+# Core ML compression and profiling
 
-Complete reference for optimizing Core ML models: quantization, palettization,
-pruning, performance tuning, and profiling.
-Keep conversion/optimization handoffs focused on Python-side tooling and
-profiling decisions. Defer Swift prediction/runtime wiring to the sibling
-`coreml` skill unless the user explicitly asks for app integration code.
+Keep the work here on the Python tooling and on profiling. Swift prediction
+code and runtime setup belong to the `coreml` skill unless the user asked for
+app code; the Swift section below is only a quick orientation.
 
 ## Contents
 
-- [Optimization Technique Selection](#optimization-technique-selection)
-- [Post-Training Weight Quantization (Data-Free)](#post-training-weight-quantization-data-free)
-- [Palettization (Weight Clustering)](#palettization-weight-clustering)
-- [Pruning (Weight Sparsification)](#pruning-weight-sparsification)
-- [Joint Compression (Stacking Techniques)](#joint-compression-stacking-techniques)
-- [Per-Op Configuration](#per-op-configuration)
-- [Quantization-Aware Training (QAT)](#quantization-aware-training-qat)
-- [Swift Integration](#swift-integration)
-- [MLTensor (iOS 18+)](#mltensor-ios-18)
-- [Neural Engine Best Practices](#neural-engine-best-practices)
-- [Model Loading Optimization](#model-loading-optimization)
+- [Picking a technique](#picking-a-technique)
+- [Data-free weight quantization](#data-free-weight-quantization)
+- [Palettization](#palettization)
+- [Pruning](#pruning)
+- [Combining techniques](#combining-techniques)
+- [Per-op settings](#per-op-settings)
+- [Quantization-aware training](#quantization-aware-training)
+- [Swift at a glance](#swift-at-a-glance)
+- [MLTensor](#mltensor)
+- [Neural Engine habits](#neural-engine-habits)
+- [Faster loading](#faster-loading)
 - [Profiling](#profiling)
-- [Common Optimization Mistakes](#common-optimization-mistakes)
+- [Compression mistakes](#compression-mistakes)
 
-## Optimization Technique Selection
+## Picking a technique
 
-| Technique | Size Reduction | Accuracy Impact | Best Compute Unit | Min OS |
+| Technique | Minimum OS | Size cut | Accuracy risk | Best on |
 |---|---|---|---|---|
-| INT8 per-channel | ~4x | Low | CPU/GPU | iOS 16 |
-| INT4 per-block | ~8x | Medium | GPU | iOS 18 |
-| Palettization 4-bit | ~8x | Low-Medium | Neural Engine | iOS 16 |
-| Palettization 2-bit | ~16x | Medium-High | Neural Engine | iOS 16 |
-| W8A8 (weights+activations) | ~4x | Low | ANE (A17 Pro/M4+) | iOS 17 |
-| Pruning 50% | ~2x | Low | CPU/ANE | iOS 16 |
-| Pruning 75% | ~4x | Medium | CPU/ANE | iOS 16 |
+| 50% pruning | iOS 16 | about 2x | low | CPU, Neural Engine |
+| 75% pruning | iOS 16 | about 4x | medium | CPU, Neural Engine |
+| 4-bit palettization | iOS 16 | about 8x | low to medium | Neural Engine |
+| 2-bit palettization | iOS 16 | about 16x | medium to high | Neural Engine |
+| INT8, per channel | iOS 16 | about 4x | low | CPU and GPU |
+| W8A8 | iOS 17 | about 4x | low | Neural Engine (A17 Pro, M4 and later) |
+| INT4, per block | iOS 18 | about 8x | medium | GPU |
 
-## Post-Training Weight Quantization (Data-Free)
+## Data-free weight quantization
 
-### INT8 Per-Channel Symmetric
+INT8 on a converted model:
 
 ```python
-import coremltools as ct
+# Data-free INT8 on an already converted model.
 import coremltools.optimize as cto
 
-model = ct.models.MLModel("model.mlpackage")
-
-op_config = cto.coreml.OpLinearQuantizerConfig(
-    mode="linear_symmetric",  # or "linear" (asymmetric with zero-point)
-    weight_threshold=512,     # only quantize tensors with > N elements
+int8_op = cto.coreml.OpLinearQuantizerConfig(
+    weight_threshold=512,          # skip tensors with this many elements or fewer
+    mode="linear_symmetric",       # "linear" is the asymmetric variant with a zero point
 )
-config = cto.coreml.OptimizationConfig(global_config=op_config)
-compressed = cto.coreml.linear_quantize_weights(model, config=config)
-compressed.save("model_int8.mlpackage")
+int8 = cto.coreml.OptimizationConfig(global_config=int8_op)
+small = cto.coreml.linear_quantize_weights(mlmodel, config=int8)
 ```
 
-### INT4 Per-Block (PyTorch, Data-Free)
+INT4 per-block, applied to the PyTorch model before conversion:
 
 ```python
-import coremltools.optimize as cto
+import coremltools.optimize.torch.quantization as ptq
 
-config = cto.torch.quantization.PostTrainingQuantizerConfig.from_dict({
-    "global_config": {
-        "weight_dtype": "int4",
-        "granularity": "per_block",
-        "block_size": 128,
-    }
-})
-quantizer = cto.torch.quantization.PostTrainingQuantizer(model, config)
-quantized_model = quantizer.compress()
+int4_settings = {"block_size": 128, "granularity": "per_block", "weight_dtype": "int4"}
+cfg = ptq.PostTrainingQuantizerConfig.from_dict({"global_config": int4_settings})
+compressed = ptq.PostTrainingQuantizer(torch_model, cfg).compress()
 ```
 
-### GPTQ Calibration-Based Quantization
+With calibration data, GPTQ usually holds accuracy better:
 
 ```python
-config = cto.torch.layerwise_compression.LayerwiseCompressorConfig.from_dict({
-    "global_config": {
-        "algorithm": "gptq",
-        "weight_dtype": 4,
-        "granularity": "per_block",
-        "block_size": 128,
-    },
+import coremltools.optimize.torch.layerwise_compression as lwc
+
+gptq = lwc.LayerwiseCompressorConfig.from_dict({
     "calibration_nsamples": 16,
+    "global_config": {"block_size": 128, "granularity": "per_block",
+                      "weight_dtype": "uint4", "algorithm": "gptq"},
 })
-compressor = cto.torch.layerwise_compression.LayerwiseCompressor(model, config)
-compressed_model = compressor.compress(calibration_dataloader)
+compressed = lwc.LayerwiseCompressor(torch_model, gptq).compress(calibration_loader)
 ```
 
-## Palettization (Weight Clustering)
-
-Especially effective on the Neural Engine. 4-bit palettization typically
-preserves accuracy better than 4-bit linear quantization.
+## Palettization
 
-### Post-Conversion Palettization
+Palettization replaces weights with indices into a small table of values. The
+Neural Engine handles it well, and a 4-bit palette tends to lose less accuracy
+than 4-bit linear quantizing does.
 
 ```python
-op_config = cto.coreml.OpPalettizerConfig(
-    mode="kmeans",                     # "kmeans" or "uniform"
-    nbits=4,                           # {1, 2, 3, 4, 6, 8}
-    granularity="per_grouped_channel", # iOS 18+ for grouped
+palette_op = cto.coreml.OpPalettizerConfig(
+    nbits=4,                              # allowed: 8, 6, 4, 3, 2, 1
+    mode="kmeans",                        # the other mode is "uniform"
     group_size=16,
+    granularity="per_grouped_channel",    # iOS 18+
 )
-config = cto.coreml.OptimizationConfig(global_config=op_config)
-palettized = cto.coreml.palettize_weights(model, config=config)
+palette = cto.coreml.OptimizationConfig(global_config=palette_op)
+paletted = cto.coreml.palettize_weights(mlmodel, config=palette)
 ```
 
-### Available Bit Widths
-
-| Bits | Unique Values | Size Reduction | Typical Quality |
+| Bits | Size cut | Typical quality | Distinct values |
 |---|---|---|---|
-| 8 | 256 | ~2x | Excellent |
-| 6 | 64 | ~2.7x | Very good |
-| 4 | 16 | ~8x | Good |
-| 3 | 8 | ~10.7x | Moderate |
-| 2 | 4 | ~16x | Fair |
-| 1 | 2 | ~32x | Poor (binary) |
+| 8 | about 2x | excellent | 256 |
+| 6 | about 2.7x | very good | 64 |
+| 4 | about 8x | good | 16 |
+| 3 | about 10.7x | moderate | 8 |
+| 2 | about 16x | fair | 4 |
+| 1 | about 32x | poor | 2 |
 
-## Pruning (Weight Sparsification)
+## Pruning
 
-### Magnitude Pruning
+Zero out small weights by magnitude:
 
 ```python
-config = cto.coreml.OptimizationConfig(
-    global_config=cto.coreml.OpMagnitudePrunerConfig(
-        target_sparsity=0.75,
-        weight_threshold=2048,
-    )
-)
-pruned = cto.coreml.prune_weights(model, config=config)
+by_size = cto.coreml.OpMagnitudePrunerConfig(weight_threshold=2048, target_sparsity=0.75)
+prune_cfg = cto.coreml.OptimizationConfig(global_config=by_size)
+sparse = cto.coreml.prune_weights(mlmodel, config=prune_cfg)
 ```
 
-### Threshold Pruning
+Or by an absolute threshold:
 
 ```python
-config = cto.coreml.OptimizationConfig(
-    global_config=cto.coreml.OpThresholdPrunerConfig(
-        threshold=1e-12,
-        minimum_sparsity_percentile=0.5,
-    )
-)
-pruned = cto.coreml.prune_weights(model, config=config)
+tiny = cto.coreml.OpThresholdPrunerConfig(minimum_sparsity_percentile=0.5, threshold=1e-12)
+sparse = cto.coreml.prune_weights(
+    mlmodel, config=cto.coreml.OptimizationConfig(global_config=tiny))
 ```
 
-## Joint Compression (Stacking Techniques)
+## Combining techniques
 
-Apply multiple compression techniques in sequence:
+Palettize first, then prune with `joint_compression=True` so the sparsity is
+applied on top of the palette:
 
 ```python
-# Palettize first, then prune on top
-palettized = cto.coreml.palettize_weights(model, pal_config)
-final = cto.coreml.prune_weights(
-    palettized, prune_config, joint_compression=True
-)
+paletted = cto.coreml.palettize_weights(mlmodel, config=palette)
+both = cto.coreml.prune_weights(paletted, config=prune_cfg, joint_compression=True)
 ```
 
-## Per-Op Configuration
+## Per-op settings
 
-Fine-grained control over which operations get compressed:
+`OptimizationConfig` can vary the treatment by op type or by op name. `None`
+leaves that op uncompressed, which is useful for embeddings and other sensitive
+layers.
 
 ```python
-config = cto.coreml.OptimizationConfig(
-    global_config=global_op_config,
+mixed = cto.coreml.OptimizationConfig(
+    global_config=cto.coreml.OpPalettizerConfig(nbits=6),
     op_type_configs={
-        "linear": linear_config,
-        "conv": conv_config,
-    },
-    op_name_configs={
-        "embedding_layer": None,  # None = skip compression
+        "linear": cto.coreml.OpPalettizerConfig(nbits=4),
+        "conv": cto.coreml.OpPalettizerConfig(nbits=6),
     },
+    op_name_configs={"token_embedding": None},
 )
 ```
 
-## Quantization-Aware Training (QAT)
+## Quantization-aware training
 
-Train with quantization in the loop for best accuracy:
+QAT gives the best accuracy at low bit widths because the model learns around
+the quantization during fine-tuning:
 
 ```python
-from coremltools.optimize.torch.quantization import (
-    LinearQuantizer, LinearQuantizerConfig, ModuleLinearQuantizerConfig
-)
+import coremltools.optimize.torch.quantization as ptq
 
-config = LinearQuantizerConfig(
-    global_config=ModuleLinearQuantizerConfig(
-        quantization_scheme="symmetric",
-        milestones=[0, 1000, 1000, 0],
-    )
+module_cfg = ptq.ModuleLinearQuantizerConfig(
+    milestones=[0, 200, 400, 400],
+    quantization_scheme="symmetric",
 )
-quantizer = LinearQuantizer(model, config)
-quantizer.prepare(example_inputs=[1, 3, 224, 224], inplace=True)
-
-# Training loop
-for inputs, labels in data:
-    output = model(inputs)
-    loss = loss_fn(output, labels)
-    loss.backward()
-    optimizer.step()
-    quantizer.step()
+qat_cfg = ptq.LinearQuantizerConfig(global_config=module_cfg)
+quantizer = ptq.LinearQuantizer(torch_model, qat_cfg)
+torch_model = quantizer.prepare(example_inputs=(sample,), inplace=True)
 
-model = quantizer.finalize(inplace=True)
-```
-
-## Swift Integration
-
-### Loading Models
-
-```swift
-// From Xcode-compiled model (auto-generated class)
-let model = try MyImageClassifier(configuration: MLModelConfiguration())
-
-// From URL at runtime
-let config = MLModelConfiguration()
-config.computeUnits = .all
-let model = try MLModel(contentsOf: modelURL, configuration: config)
-
-// From pre-compiled model (.mlmodelc) for faster loading
-let compiledURL = try MLModel.compileModel(at: sourceModelURL)
-let model = try MLModel(contentsOf: compiledURL)
-```
-
-### MLModelConfiguration
-
-```swift
-let config = MLModelConfiguration()
-config.computeUnits = .all
-config.allowLowPrecisionAccumulationOnGPU = true
-// config.functionName = "adapter_1"  // For multifunction models (iOS 18+)
-```
-
-### Synchronous Prediction
-
-```swift
-let input = MyModelInput(image: pixelBuffer)
-let output = try model.prediction(input: input)
-let label = output.classLabel
-```
-
-### Async Prediction (iOS 17+)
-
-```swift
-let output = try await model.prediction(input: input)
-```
-
-Thread-safe, supports Task cancellation, integrates with Swift concurrency.
-~60% faster than synchronous for batch workloads.
-
-### Batch Prediction
+for batch in train_loader:
+    train_step(torch_model, batch)
+    quantizer.step()
 
-```swift
-let batchInputs: [MyModelInput] = images.map { MyModelInput(image: $0) }
-let batchOutputs = try model.predictions(inputs: batchInputs)
+torch_model = quantizer.finalize(inplace=True)
 ```
 
-### MLFeatureProvider
+## Swift at a glance
 
-```swift
-let features = try MLDictionaryFeatureProvider(dictionary: [
-    "input": MLFeatureValue(pixelBuffer: pixelBuffer),
-    "threshold": MLFeatureValue(double: 0.5),
-])
-let output = try model.prediction(from: features)
-```
-
-### Vision Framework Integration
+For orientation only; loading, configuration and prediction lifecycle in an app
+belong to the `coreml` skill.
 
 ```swift
-import Vision
 import CoreML
 
-let vnModel = try VNCoreMLModel(for: MyDetector().model)
-let request = VNCoreMLRequest(model: vnModel) { request, error in
-    guard let results = request.results as? [VNClassificationObservation] else { return }
-    let topResult = results.first
-    print("\(topResult?.identifier ?? ""): \(topResult?.confidence ?? 0)")
-}
-let handler = VNImageRequestHandler(cgImage: image)
-try handler.perform([request])
-```
-
-### Natural Language Integration
-
-```swift
-import NaturalLanguage
+let runSettings = MLModelConfiguration()
+runSettings.computeUnits = .all
+runSettings.allowLowPrecisionAccumulationOnGPU = true
+// runSettings.functionName = "legal"   // multifunction models, iOS 18+
 
-let nlModel = try NLModel(mlModel: SentimentClassifier().model)
-let sentiment = nlModel.predictedLabel(for: "Great product!")
+let generated = try LeafClassifier(configuration: runSettings)   // Xcode-generated class
+let compiledLocation = try await MLModel.compileModel(at: downloadedPackageURL)
+let runtime = try MLModel(contentsOf: compiledLocation, configuration: runSettings)
 ```
 
-## MLTensor (iOS 18+)
+- `model.prediction(input:)` is synchronous. From iOS 17, the async
+  `try await model.prediction(input:)` is thread-safe and cancellable, and
+  running several predictions concurrently this way can raise throughput for
+  batch workloads; measure the gain on your model.
+- `model.predictions(inputs:)` runs a batch.
+- Without a generated class, build inputs with
+  `MLDictionaryFeatureProvider(dictionary:)` using
+  `MLFeatureValue(pixelBuffer:)` or `MLFeatureValue(double:)` and call
+  `prediction(from:)`.
+- Vision: wrap with `VNCoreMLModel(for:)`, run a `VNCoreMLRequest` through
+  `VNImageRequestHandler(cgImage:).perform([request])`, and read
+  `[VNClassificationObservation]` in the completion.
+- Natural Language: `NLModel(mlModel:)` then `predictedLabel(for:)`.
 
-Swift type for multidimensional array operations:
+## MLTensor
 
-```swift
-import CoreML
+From iOS 18, `MLTensor` handles multidimensional math next to Core ML models:
 
-let tensor = MLTensor([1.0, 2.0, 3.0, 4.0])
-let reshaped = tensor.reshaped(to: [2, 2])
-let result = tensor.softmax()
-let matmulResult = tensorA.matmul(tensorB)
+```swift
+let logits = MLTensor([2.0, 0.5, -1.0, 3.0] as [Float]).reshaped(to: [2, 2])
+let probabilities = logits.softmax()
+let identity = MLTensor(shape: [2, 2], scalars: [1, 0, 0, 1] as [Float])
+let projected = logits.matmul(identity)
 ```
 
-## Neural Engine Best Practices
+## Neural Engine habits
 
-1. Use EnumeratedShapes instead of RangeDim for ANE optimization
-2. Avoid unsupported ANE ops -- they cause fallback to CPU/GPU with transfer
-   overhead
-3. Use palettization (4-bit or 6-bit) for best ANE memory/latency gains
-4. W8A8 quantization on A17 Pro / M4+ enables optimized INT8 compute on ANE
+- Ops the Neural Engine cannot run push the graph to the CPU or GPU and add
+  transfer cost. Find and replace them.
+- Use 4-bit or 6-bit palettization for the best Neural Engine memory use and
+  latency.
+- On A17 Pro and M4 or later, W8A8 unlocks fast INT8 compute on the Neural
+  Engine.
 
-## Model Loading Optimization
+## Faster loading
 
-1. Pre-compile models -- use `.mlmodelc` for instant loading after first
-   compilation
-2. Cache compiled models to a fixed location after `MLModel.compileModel(at:)`
-3. Use `bisect_model()` for very large models that are slow to load
-4. Use `MLComputePlan` (iOS 17.4+) for programmatic profiling
+- Ship or cache `.mlmodelc`. The first load of an uncompiled model runs a slow,
+  device-specific compile; after `MLModel.compileModel(at:)`, move the result
+  to a permanent location and reuse it.
+- Split very large, slow-to-load models with `bisect_model()`.
+- `MLComputePlan` (iOS 17.4+) reports per-op placement and cost from code.
 
 ## Profiling
 
-1. **Xcode Performance tab** -- open .mlpackage in Xcode to see load time,
-   prediction time, per-op compute unit assignment
-2. **Core ML Instrument** in Instruments app -- runtime profiling
-3. **MLComputePlan API** -- programmatic access to profiling data
-4. **coremltools debugging** -- MLModelValidator, MLModelComparator,
-   MLModelInspector, MLModelBenchmarker
-
-### Reshape Frequency Hint
+- Xcode: open the `.mlpackage` and use the Performance tab for load time,
+  prediction time and the compute unit each op runs on.
+- Instruments: the Core ML instrument.
+- Code: `MLComputePlan`.
+- coremltools debugging helpers: `MLModelValidator`, `MLModelComparator`,
+  `MLModelInspector`, `MLModelBenchmarker`.
+- Tell Core ML that input shapes rarely change:
 
 ```python
-model = ct.models.MLModel("model.mlpackage",
-    optimization_hints={
-        "reshapeFrequency": ct.ReshapeFrequency.Infrequent
-    })
+hints = {"reshapeFrequency": ct.ReshapeFrequency.Infrequent}
+tuned = ct.models.MLModel("Segmenter.mlpackage", optimization_hints=hints)
 ```
 
-## Common Optimization Mistakes
-
-1. **Applying quantization without checking accuracy.** Always validate after
-   compression. Use MLModelComparator to compare outputs.
-2. **Ignoring weight_threshold.** Small tensors (< 512 elements) should not be
-   quantized -- overhead outweighs the benefit.
-3. **Using synchronous predictions in async contexts.** Use async prediction
-   (iOS 17+) in Swift concurrency code.
-4. **Not pre-compiling models.** First load triggers device-specific
-   compilation, which can be slow.
-5. **Ignoring compute_units configuration.** Default `.all` is correct for
-   production. `.cpuOnly` is for debugging only.
-6. **Not testing on physical devices.** Simulator does not support Metal GPU or
-   Neural Engine.
+## Compression mistakes
+
+| Mistake | Fix |
+|---|---|
+| Compressing without checking accuracy | Compare outputs with the original, for example with `MLModelComparator` |
+| Ignoring `weight_threshold` | Leave tensors under about 512 elements unquantized; the overhead outweighs the saving |
+| Calling sync prediction from async code | Use async prediction (iOS 17+) |
+| Skipping precompilation | Compile once and cache `.mlmodelc` |
+| Wrong compute units | `.all` in production, `.cpuOnly` only while debugging |
+| Testing only in the Simulator | It has no Metal GPU or Neural Engine; use devices |