@mmerterden/multi-agent-pipeline 20.7.0 → 20.8.0

package/pipeline/skills/shared/external/mapkit-location/evals/evals.json

@@ -3,44 +3,48 @@
   "evals": [
     {
       "id": 0,
-      "prompt": "I'm building a SwiftUI store locator for iOS 18. Sketch the MapKit implementation for a searchable map with autocomplete, markers for results, marker selection that shows details, a route overlay to the selected store, and a user-location button. Include the privacy and cancellation details that matter.",
-      "expected_output": "A SwiftUI MapKit implementation plan that uses Map, Marker or Annotation, MapCameraPosition, MKLocalSearchCompleter plus MKLocalSearch, Map selection tags, MKDirections and MapPolyline, UserAnnotation or MapUserLocationButton, and CoreLocation authorization/task cancellation guidance.",
+      "name": "searchable-store-locator-ios18",
+      "prompt": "Build an iOS 18 SwiftUI screen for a pharmacy chain: a map the user can search with suggestions as they type, markers for the results, tap a marker to see its details, a route line from the user to the chosen pharmacy, and a button to jump back to the user's position. Include what we need for location privacy and make sure nothing keeps running after the screen closes.",
+      "expected_output": "Builds the screen with SwiftUI MapKit (Map, Marker or Annotation, a MapCameraPosition binding, mapControls with MapUserLocationButton, UserAnnotation) instead of wrapping MKMapView. Suggestions come from MKLocalSearchCompleter with debounced input, and the chosen suggestion runs MKLocalSearch.Request(completion:) or a naturalLanguageQuery. Selection uses a binding whose type matches the marker tags. The route comes from MKDirections.Request and is drawn with MapPolyline(route.polyline). Adds NSLocationWhenInUseUsageDescription, a denied-location path with a Settings link, and cancels the location stream and pending searches when the screen goes away.",
       "files": [],
       "expectations": [
-        "Uses SwiftUI MapKit APIs such as Map, Marker or Annotation, MapCameraPosition, map controls, and MapPolyline rather than defaulting to MKMapView.",
-        "Describes MKLocalSearchCompleter for autocomplete and MKLocalSearch.Request(completion:) or naturalLanguageQuery for full search results, with debounced input.",
-        "Handles map selection with a binding and tags whose type matches the selected value.",
-        "Builds directions with MKDirections.Request and displays the selected route with MapPolyline(route.polyline).",
-        "Includes NSLocationWhenInUseUsageDescription and a denied-location UI path such as a Settings deep link.",
-        "Cancels or scopes live location/search tasks to the view lifecycle and avoids running location updates indefinitely."
+        "Draws the screen with the SwiftUI Map API (Map, Marker or Annotation, MapCameraPosition, mapControls, MapPolyline); MKMapView is not the default choice",
+        "Suggestions come from MKLocalSearchCompleter with a debounce, and the full lookup uses MKLocalSearch.Request(completion:) or naturalLanguageQuery",
+        "The marker tag type equals the selection binding type",
+        "Computes directions with MKDirections.Request and draws MapPolyline(route.polyline)",
+        "Adds NSLocationWhenInUseUsageDescription and shows a Settings link when location is denied",
+        "Location and search work stops when the screen disappears (cancelled or tied to the view's task lifetime)",
+        "Shows the user with UserAnnotation or offers MapUserLocationButton"
       ]
     },
     {
       "id": 1,
-      "prompt": "Review this location feature plan: create a CLMonitor every time the view appears, monitor 45 circular regions for coupons, request Always location on launch, start liveUpdates forever, and use background location without a Live Activity or CLBackgroundActivitySession. What should I change for iOS 18+?",
-      "expected_output": "A correction-focused CoreLocation review that uses CLServiceSession only for active user-facing features, limits CLMonitor conditions, keeps monitor/event subscriptions stable, uses CLLocationUpdate.liveUpdates with filtering and cancellation, and explains the background-location requirements.",
+      "name": "review-geofence-plan",
+      "prompt": "A delivery-driver app plans the following. Every time the dashboard appears it makes a fresh CLMonitor; it watches 45 drop-off zones at once; it asks for Always permission on first launch; it starts CLLocationUpdate.liveUpdates() at launch and never stops it; and it tracks drivers in the background with neither a Live Activity nor a CLBackgroundActivitySession. Critique the plan and propose the fix.",
+      "expected_output": "Says asking for Always on first launch is wrong: start with when-in-use, and on iOS 18 hold a CLServiceSession only while a driver-facing feature is running. Names the limit of 20 conditions per app and suggests ranking or rotating the zones. Points out that making a CLMonitor again under a name still in use crashes, so keep one monitor for the app's life, update it by diffing adds and removes, and subscribe to its events once. Treats liveUpdates() as an async sequence the feature owns, discards inaccurate or invalid fixes and cancels it when idle. For background tracking, requires the Location updates background mode together with a CLBackgroundActivitySession or a Live Activity, plus a running update or monitor subscription. Reacts to denied, globally denied and unavailable diagnostics with a reduced mode rather than silence.",
       "files": [],
       "expectations": [
-        "Rejects requesting Always authorization on launch and recommends when-in-use first, with CLServiceSession scoped to the feature lifetime on iOS 18+.",
-        "States that CLMonitor condition monitoring is limited to 20 conditions per app and the app must prioritize or rotate regions.",
-        "Warns against rapidly recreating CLMonitor instances with the same name and recommends a long-lived monitor plus add/remove diffing.",
-        "Uses CLLocationUpdate.liveUpdates() as an async sequence and filters invalid or inaccurate locations while cancelling the task when not needed.",
-        "Explains that background updates require the Location updates background mode and either a CLBackgroundActivitySession or Live Activity, with active update or monitor subscriptions.",
-        "Mentions handling denied/global-denied/location-unavailable diagnostics or degraded behavior instead of silently failing."
+        "Says Always at first launch is wrong; recommends when-in-use and, on iOS 18+, a CLServiceSession that lives only as long as the feature",
+        "Names the per-app cap of 20 conditions and suggests ranking or rotating zones",
+        "Flags the crash from recreating a CLMonitor under a live name; keeps one monitor, diffs adds and removes, subscribes to events once",
+        "Iterates liveUpdates() as an async sequence, drops bad fixes, and cancels it when idle",
+        "Background tracking needs the Location updates mode, a CLBackgroundActivitySession or Live Activity, and a running update or monitor subscription",
+        "Denied, globally denied and unavailable diagnostics lead to a visible reduced mode, not silence"
       ]
     },
     {
       "id": 2,
-      "prompt": "I need a navigation experience that shows a map inside CarPlay, opens a small App Clip from a place card, and pairs a BLE beacon when the user enters a geofence. Which parts belong in this MapKit skill, and where should the other pieces go?",
-      "expected_output": "A boundary-aware answer that keeps MapKit/CoreLocation responsibilities in scope while routing CarPlay templates, App Clip invocation, and BLE accessory setup to the appropriate sibling skills or frameworks.",
+      "name": "scope-boundaries",
+      "prompt": "Our festival app needs three things: a CarPlay screen with a map to the venue, an App Clip that opens when someone taps our venue's place card in Maps, and automatic pairing with a Bluetooth wristband beacon when the user enters the festival geofence. Which of this is MapKit and CoreLocation work, and where does the rest go?",
+      "expected_output": "Limits this skill to maps, annotations, routes, search and geocoding, the user's position, CLMonitor geofences and the background-location rules. The CarPlay screen (templates, navigation alerts, CarPlay UI) goes to CarPlay framework guidance; opening an App Clip from the place card, its size limits and the handoff go to app-clips; pairing the wristband goes to core-bluetooth or AccessorySetupKit. Never says MapKit itself renders CarPlay templates, launches App Clips or pairs Bluetooth hardware. Describes where the pieces meet: the geofence entry event starts the pairing flow, a computed route or search result feeds the CarPlay screen, and the place card carries the App Clip link.",
       "files": [],
       "expectations": [
-        "Keeps MapKit/CoreLocation scope to maps, annotations, routes, geocoding/search, user location, CLMonitor geofencing, and background location constraints.",
-        "Routes CarPlay map templates, navigation alerts, and CarPlay-specific UI to the CarPlay skill or CarPlay framework guidance.",
-        "Routes App Clip invocation, size limits, and handoff behavior to the App Clips skill.",
-        "Routes BLE beacon pairing or accessory onboarding to Core Bluetooth or AccessorySetupKit rather than expanding MapKit scope.",
-        "Does not claim MapKit handles CarPlay templates, App Clip launch infrastructure, or BLE pairing directly.",
-        "Still explains the handoff points between geofence events, routes/search results, and the adjacent frameworks."
+        "This skill owns maps, annotations, routes, search and geocoding, the user's position, CLMonitor geofences and background location",
+        "The CarPlay map screen, its templates and navigation alerts are sent to CarPlay framework guidance",
+        "App Clip launch from the place card, size limits and handoff are sent to app-clips",
+        "Wristband or beacon pairing is sent to core-bluetooth or AccessorySetupKit",
+        "MapKit is never credited with CarPlay templates, App Clip launch or Bluetooth pairing",
+        "Describes where the frameworks meet: geofence entry, a route or search result, the place card link"
       ]
     }
   ]