@mmerterden/multi-agent-pipeline 20.7.0 → 20.8.1

package/pipeline/skills/shared/external/apple-on-device-ai/evals/evals.json

@@ -3,44 +3,41 @@
   "evals": [
     {
       "id": 0,
-      "name": "foundation-models-tool-calling-plan",
-      "prompt": "I'm adding an iOS 26 meal-planning feature that uses Apple's on-device Foundation Models to return a typed MealPlan, call a pantry lookup tool, stream partial results into SwiftUI, and fall back when Apple Intelligence is unavailable. Sketch the implementation plan and review checklist for another engineer.",
-      "expected_output": "A Foundation Models implementation outline that is availability-safe, locale-aware, schema-conscious, tool-calling-aware, and ready for SwiftUI streaming without overclaiming model readiness.",
+      "prompt": "Plan an iOS 26 feature for a gardening app: the user describes their balcony and the app returns a typed PlantingPlan (plants, sowing dates, watering notes) generated on device. It should look up the user's saved seed inventory through a tool, stream the plan into a SwiftUI view as it forms, and still work in some form when Apple Intelligence is unavailable. Give the plan and a review checklist.",
+      "expected_output": "A Foundation Models plan covering availability and locale gating, typed streaming output, a small tool set, context budget and error handling, with a fallback path.",
       "files": [],
-      "expectations": [
-        "Checks `SystemLanguageModel.default.availability` before generation and handles `appleIntelligenceNotEnabled`, `modelNotReady`, and `deviceNotEligible` with fallback UI.",
-        "Uses `LanguageModelSession`, `@Generable`, concise `@Guide` constraints, and structured streaming with partial generated content.",
-        "Prefers `supportsLocale(_:)` for locale gating and accounts for the shared input/output context window with `contextSize` or token counting.",
-        "Defines a small set of `Sendable` tools at session creation, notes that tool schemas consume context, and does not promise forced tool invocation.",
-        "Covers generation error handling for guardrails, context overflow, concurrent requests, unsupported guide or locale, assets unavailable, refusals, and rate limiting."
+      "assertions": [
+        "Checks SystemLanguageModel.default.availability and gives fallback UI for appleIntelligenceNotEnabled, modelNotReady and deviceNotEligible.",
+        "Uses LanguageModelSession with a @Generable PlantingPlan, short @Guide descriptions, and streaming of partially generated values.",
+        "Gates generation with supportsLocale(_:) and keeps prompt plus reply within one budget, checked with contextSize or tokenCount(for:).",
+        "Passes only a few Sendable tools when the session is created, says each tool schema takes context space, and never promises the model will call the tool.",
+        "Covers these GenerationError cases: guardrail violation, exceeded context window, concurrent requests, unsupported guide, unsupported language or locale, assets unavailable, refusal, and rate limiting."
       ]
     },
     {
       "id": 1,
-      "name": "coreml-conversion-optimization-review",
-      "prompt": "Review this conversion plan before I hand it to an ML engineer: export a PyTorch vision transformer straight from training mode, convert it to a `.mlmodel` neuralnetwork with dynamic RangeDim inputs, quantize everything to INT4, and profile with MLComputePlan on iOS 17.0. Target devices are iPhone 15 Pro and M4 iPad.",
-      "expected_output": "A correction-focused Core ML conversion review that replaces stale or risky guidance with modern mlprogram, explicit deployment target, shape, compression, and profiling advice.",
+      "prompt": "Here is our Core ML plan, please critique it: we trace our PyTorch image segmentation model straight out of the training loop, save the result as a neural-network-format .mlmodel using RangeDim on height and width, apply INT4 to every weight, and use MLComputePlan on iOS 17.0 to check op placement. We ship to an M4 iPad and to iPhone 15 Pro.",
+      "expected_output": "Feedback that fixes the export, format, shape, compression and profiling steps and keeps Swift integration out of scope.",
       "files": [],
-      "expectations": [
-        "Requires `model.eval()` before tracing/export and recommends validating outputs against the source model after conversion.",
-        "Uses mlprogram `.mlpackage` for new models with an explicit `minimum_deployment_target` instead of legacy neuralnetwork `.mlmodel` output.",
-        "Prefers `EnumeratedShapes` over broad `RangeDim` when valid shapes are known, especially for Neural Engine performance.",
-        "Warns that compression must be accuracy-tested and should avoid blindly quantizing every tensor.",
-        "Corrects `MLComputePlan` availability to iOS 17.4+ and distinguishes profiling from Swift app integration details."
+      "assertions": [
+        "Requires model.eval() before tracing and validating converted outputs against the source model.",
+        "Recommends an ML program .mlpackage with an explicit minimum_deployment_target.",
+        "Recommends EnumeratedShapes over a broad RangeDim when the Neural Engine is the target.",
+        "Says compression needs accuracy testing and should not be applied blindly to every tensor, for example respecting weight_threshold or per-op configs.",
+        "States that MLComputePlan needs iOS 17.4 or later and keeps profiling separate from Swift app integration."
       ]
     },
     {
       "id": 2,
-      "name": "backend-selection-boundary",
-      "prompt": "We have a private journaling app that needs summarization on iOS 26 when possible, an open-source LLM fallback for some Apple Silicon devices, and a custom sentiment classifier we already trained. Decide which Apple on-device AI stack belongs to each piece, and call out what should be delegated to the sibling Core ML integration skill rather than explained here.",
-      "expected_output": "A boundary-aware backend selection answer that routes text generation to Foundation Models, custom classifier deployment to Core ML conversion/optimization, open-source fallback to MLX Swift or llama.cpp, and detailed Swift Core ML integration to the sibling Core ML skill.",
+      "prompt": "We are building a private voice-memo app. On iOS 26 we want to summarise memos, on some Apple Silicon devices we want an open-source LLM as a fallback, and we already trained a topic classifier in PyTorch. Which stack handles each piece, and what should go to the Core ML integration skill instead?",
+      "expected_output": "A per-piece stack decision with a clear hand-off of Swift Core ML work to the coreml skill.",
       "files": [],
-      "expectations": [
-        "Selects Foundation Models for on-device summarization when Apple Intelligence is available and includes fallback behavior for unsupported devices.",
-        "Routes the trained sentiment classifier through Core ML conversion and optimization guidance rather than treating it as a Foundation Models prompt.",
-        "Chooses MLX Swift or llama.cpp for open-source LLM fallback based on Apple Silicon throughput, GGUF compatibility, device memory, and cross-platform needs.",
-        "Mentions MLX memory limits, background unloading, and physical-device validation for Metal-dependent behavior.",
-        "Keeps detailed Swift Core ML app integration, Vision request wiring, and model loading code as a handoff to the sibling Core ML skill."
+      "assertions": [
+        "Uses Foundation Models for summaries with an availability check and a fallback.",
+        "Converts and optimises the trained classifier for Core ML instead of prompting an LLM to classify.",
+        "Picks MLX Swift or llama.cpp by weighing speed, whether GGUF weights exist, RAM, and whether other platforms matter.",
+        "Raises the MLX RAM ceiling, releasing models when backgrounded, and testing on real hardware.",
+        "Hands Swift Core ML integration, Vision wiring and model loading code to the coreml skill."
       ]
     }
   ]