@mmerterden/multi-agent-pipeline 20.6.0 → 20.8.0

package/pipeline/skills/shared/external/coreml/evals/evals.json

@@ -3,41 +3,44 @@
   "evals": [
     {
       "id": 1,
-      "prompt": "Review this Core ML prediction service for iOS 26: it loads a compiled model with MLModel(contentsOf:), calls `try await model.prediction(from:)`, uses `model.predictions(from: batchProvider)`, and says async prediction is available on iOS 17. Provide corrected Swift snippets and availability notes without turning this into model conversion guidance.",
-      "expected_output": "A concise review that keeps model loading asynchronous, corrects prediction and batch prediction API usage, and preserves the Swift integration boundary.",
+      "prompt": "Please audit a model wrapper from our iOS 26 app. An actor exposes an async method; on first call it builds the model via MLModel(contentsOf:), then calls `model.prediction(from: input)` with no await. Batches go through `model.predictions(from: batchProvider)`. A comment in the file claims Core ML has had no async prediction API until iOS 18. List the problems and the fixes.",
+      "expected_output": "Points out that inside an async function on an iOS 17+ target the async prediction overload is selected, so the call must be `try await model.prediction(from:)`; that async prediction exists since iOS 17; recommends MLModel.load(contentsOf:configuration:) (iOS 15+) for loading; corrects the batch call; stays on Swift integration.",
       "files": [],
       "assertions": [
-        "States that stateless MLModel prediction APIs are synchronous and does not put `await` on stateless `model.prediction(...)` calls.",
-        "Uses `MLModel.load(contentsOf:configuration:)` for asynchronous model loading and notes it is available from iOS 15+.",
-        "Uses `predictions(fromBatch:)` for batch prediction when no explicit `MLPredictionOptions` are supplied.",
-        "Notes that `MLModel.compileModel(at:)` async runtime compilation is iOS 16+ when discussing runtime compilation.",
-        "Keeps the answer focused on Swift Core ML integration rather than Python-side conversion, quantization, or framework selection."
+        "States that MLModel has an async prediction(from:options:) from iOS 17, so the comment claiming nothing async before iOS 18 is wrong",
+        "States that inside an async function with an iOS 17+ deployment target the call must be written `try await model.prediction(from:)`, and that the synchronous form is for synchronous contexts or older targets",
+        "Suggests `MLModel.load(contentsOf:configuration:)` as the loader and mentions its iOS 15 floor",
+        "Replaces `predictions(from: batchProvider)` with `predictions(fromBatch:)` when no options are passed, or `predictions(from:options:)` when MLPredictionOptions are needed",
+        "When compiling on device is discussed, gives iOS 16 as the floor for the async `MLModel.compileModel(at:)` and says the compiled result has to be kept",
+        "Keeps the answer on the Swift integration; no detour into Python model conversion, quantization or choosing a framework"
       ]
     },
     {
       "id": 2,
-      "prompt": "Implement snippets for a stateful audio Core ML model that uses MLState and MLTensor preprocessing on iOS 18+. Include the concurrency rules and materialization step. Avoid unsupported tensor APIs.",
-      "expected_output": "Swift snippets that use MLState correctly, serialize stateful predictions, and use source-grounded MLTensor materialization and operations.",
+      "prompt": "I have a stateful audio model on iOS 18+. I want to keep an MLState across 1-second windows, post-process the logits with MLTensor (softmax plus a standard deviation for a confidence score), and possibly process two microphones at once. Show me the Swift.",
+      "expected_output": "Creates state with model.makeState() once per stream, passes it to every prediction, serializes predictions per state, uses one state per microphone, materializes tensors with await shapedArray(of:), and builds the spread from documented MLTensor operations.",
       "files": [],
       "assertions": [
-        "Creates state with `model.makeState()` and passes it to `model.prediction(from:using:)` without `await` on the prediction call.",
-        "States that predictions sharing the same `MLState` must be serialized and that state buffers should not be read or written while a prediction is in flight.",
-        "Does not claim `MLState` is non-Sendable.",
-        "Uses `await tensor.shapedArray(of:)` when materializing an `MLTensor`.",
-        "Avoids unsupported examples such as `MLTensor(multiArray)`, `tensor.std()`, and `tensor.standardDeviation()`."
+        "Makes one state with `model.makeState()` and hands that same state to `prediction(from:using:)` for every window",
+        "Calls synchronous `prediction(from:using:)` only from synchronous code and uses `try await` with `prediction(from:using:options:)` inside async code",
+        "Runs calls on any one state strictly one after another and leaves its buffers alone during a call",
+        "Gives each concurrent microphone stream its own state from `makeState()`",
+        "Treats MLState as Sendable, while making clear that Sendable is no license for overlapping calls on a single state",
+        "Reads tensor values with `await tensor.shapedArray(of:)`",
+        "Does not use `MLTensor(multiArray)`, `std()` or `standardDeviation()`; builds the spread from documented operations such as mean, arithmetic and squareRoot, or computes it outside the tensor pipeline"
       ]
     },
     {
       "id": 3,
-      "prompt": "Write a deployment note for a 600 MB Core ML model in an iOS app. Compare bundling, Background Assets, On-Demand Resources, and server delivery; include memory/background compute guidance and App Store size caveats.",
-      "expected_output": "A deployment note that prefers current asset-delivery guidance, avoids stale download-limit claims, and keeps background compute guidance conservative.",
+      "prompt": "Draft a brief note on how to ship a Core ML model of roughly 600 MB with our iOS app: weigh putting it in the app bundle, Background Assets, on-demand resources, and hosting it on our servers.",
+      "expected_output": "Recommends Background Assets for a large, updatable model, treats on-demand resources as legacy, avoids invented size rules, points at current App Store Connect limits, recommends precompiled .mlmodelc and device validation, and keeps compute-unit guidance policy-based.",
       "files": [],
       "assertions": [
-        "Prefers Background Assets for new large or updateable model assets and frames On-Demand Resources as legacy or existing-project guidance.",
-        "Does not repeat a blanket 200 MB cellular download limit or a rule that models over 50 MB must use ODR.",
-        "Mentions checking current App Store Connect size limits or gives the iOS/iPadOS 18+ thinned app bundle and ODR asset-pack limits with appropriate caveats.",
-        "Recommends precompiled `.mlmodelc` assets and physical-device memory/performance validation.",
-        "Softens `.cpuOnly` background guidance to policy, contention, thermal, or energy constraints instead of claiming GPU or Neural Engine are categorically unavailable."
+        "Picks Background Assets for a big model that will be updated, and presents on-demand resources as the older option kept for apps already on it",
+        "Does not invent a blanket 200 MB cellular limit or a rule that anything over 50 MB must use on-demand resources",
+        "Tells the reader to check current App Store Connect limits, or cites the iOS/iPadOS 18+ figures (4 GB thinned app bundle, 8 GB thinned on-demand-resource asset pack) with a caveat that they change",
+        "Recommends shipping or caching a precompiled `.mlmodelc` and validating load and prediction on physical devices, including the lowest-memory one supported",
+        "Frames `.cpuOnly` for background-adjacent work as a tradeoff driven by policy, contention, thermal state and energy, not as a rule"
       ]
     }
   ]