@mmerterden/multi-agent-pipeline 20.7.0 → 20.8.0

package/pipeline/skills/shared/external/push-notifications/evals/evals.json

@@ -2,44 +2,45 @@
   "skill_name": "push-notifications",
   "evals": [
     {
-      "id": 1,
+      "id": 0,
       "name": "apns-registration-permission-review",
-      "prompt": "Review this SwiftUI push setup: it only calls registerForRemoteNotifications after the user grants alert permission, stores the APNs token in UserDefaults so it can skip uploads on future launches, and treats a simulator failure as a production bug. What should change?",
-      "expected_output": "A correction-focused review that separates APNs token registration from user-visible notification authorization, uploads APNs tokens on every callback without local token caching, and handles simulator push testing correctly.",
+      "prompt": "Our SwiftUI fitness app only calls registerForRemoteNotifications() after requestAuthorization returns .authorized. We store the device token in UserDefaults and skip the upload when the stored value matches, and we reject tokens that are not 64 hex characters. QA also says registration fails on the Simulator. Review this setup.",
+      "expected_output": "Explains that alert permission and APNs registration are separate, so registration must not wait for .authorized; the token is hex-encoded and uploaded from every didRegister callback, the local copy is only a cache and its length is never assumed; Simulator registration failure is normal and pushes can be simulated with a .apns file or xcrun simctl push.",
       "files": [],
-      "assertions": [
-        "Explains that requestAuthorization is for alerts, sounds, and badges, while registerForRemoteNotifications is the APNs token registration path.",
-        "Says APNs registration should not be gated on .authorized when the app needs silent pushes or server token binding.",
-        "Requires sending the device token to the provider server on every didRegister callback and not persisting it locally as a source of truth.",
-        "Notes that Simulator can simulate pushes with .apns or simctl but does not provide real APNs device-token registration."
+      "expectations": [
+        "Separates the two calls: requestAuthorization governs what the user sees and hears, registerForRemoteNotifications fetches the token, and neither depends on the other",
+        "Says registration must not be gated on .authorized because silent pushes and server token binding still need it",
+        "Requires an upload from each didRegister call and treats the UserDefaults value as a disposable cache",
+        "Rejects the fixed 64-character check because token length is not guaranteed",
+        "Notes the Simulator gets no real APNs token but can show simulated pushes from a .apns file or simctl push"
       ]
     },
     {
-      "id": 2,
+      "id": 1,
       "name": "background-push-payload-review",
-      "prompt": "A backend wants to keep chat fresh with silent pushes every few minutes and apns-priority 10. The app already has visible notification UI elsewhere. Review only the APNs payload, headers, and app-side notification handling.",
-      "expected_output": "A source-grounded review that requires a background payload and headers, rejects high-priority/frequent silent pushes, names the Remote notifications background mode, and keeps long-running background work in the background-processing boundary.",
+      "prompt": "Backend wants to keep a stock-price widget fresh by sending a push every two minutes with {\"aps\": {\"content-available\": 1, \"sound\": \"default\"}} at priority 10, and the app would show a banner from the handler after downloading prices for up to two minutes. Review the payload and the plan.",
+      "expected_output": "Strips the payload down to content-available 1 (nothing that alerts), sent at apns-priority 5 under apns-push-type background; explains that APNs throttles these, may drop them and will not honour a two-minute cadence; asks for Remote notifications in Background Modes and a quick handler returning the matching UIBackgroundFetchResult; routes visible UI and long work elsewhere, pointing long work at BGTaskScheduler in background-processing.",
       "files": [],
-      "assertions": [
-        "Requires aps content-available: 1 with no alert, sound, or badge for a background notification.",
-        "Requires apns-push-type: background and apns-priority: 5.",
-        "Warns that background notifications are low priority, throttled, not guaranteed, and should not be sent every few minutes.",
-        "Requires Background Modes > Remote notifications and prompt UIBackgroundFetchResult completion.",
-        "Keeps visible-notification UI and long-running BGTaskScheduler work out of the push-notifications scope except as handoffs."
+      "expectations": [
+        "Drops the sound key so the aps dictionary holds content-available: 1 alone",
+        "Requires the apns-push-type: background header and apns-priority: 5",
+        "Warns that delivery is throttled and not guaranteed, and that a push every two minutes will not work",
+        "Asks for Remote notifications under Background Modes plus a short handler whose UIBackgroundFetchResult reflects the outcome",
+        "Treats user-visible behaviour and long downloads as out of scope for the silent push and hands long work to BGTaskScheduler via background-processing"
       ]
     },
     {
-      "id": 3,
+      "id": 2,
       "name": "rich-notification-extension-review",
-      "prompt": "Review a rich-notification plan: use mutable-content on a silent push to decrypt text, store the decryption key in an App Group UserDefaults suite, attach a downloaded file from any URL, use INSendMessageIntent without extra capability setup, and skip explicit fallback handling when work fails or time expires.",
-      "expected_output": "A review that corrects service-extension triggering, secure shared storage, attachment handling, communication notification setup, and fallback behavior for failed or timed-out service-extension work.",
+      "prompt": "We plan a Notification Service Extension for our encrypted chat app. It should run for our silent sync pushes too, store the decryption key in the App Group UserDefaults, attach sender photos by passing the remote image URL straight to UNNotificationAttachment, show the sender avatar, and call the content handler when the download finishes. Review the design.",
+      "expected_output": "Explains the extension runs only for alerting pushes with mutable-content 1, never for silent pushes; moves the key to a shared Keychain access group and keeps App Groups for files and defaults; requires downloading the image into a supported local file, which the system then checks and takes over; lists the communication-notification setup (the capability, an NSUserActivityTypes entry, a donated INInteraction, then content.updating(from:)); and requires the content handler to be called exactly once on success, failure and serviceExtensionTimeWillExpire with updated, best-attempt or original content.",
       "files": [],
-      "assertions": [
-        "Says a Notification Service Extension requires mutable-content: 1 and an alerting remote notification; silent pushes do not trigger it.",
-        "Uses Keychain Sharing for decryption secrets and App Groups only for shared files or UserDefaults.",
-        "Requires attachments to be supported files on disk and notes the system validates and stores them.",
-        "Requires Communication Notifications capability, NSUserActivityTypes intent entries, intent donation, and content.updating(from:).",
-        "Requires calling the content handler on success, failure, and serviceExtensionTimeWillExpire timeout paths with original, best-attempt, or updated content."
+      "expectations": [
+        "Explains that only an alerting push flagged mutable-content: 1 launches the extension, so silent sync pushes never will",
+        "Puts the decryption key in a shared Keychain group; the App Group stays for files and suite defaults",
+        "Requires writing the attachment to a supported on-disk file instead of passing a remote URL, noting the system validates and stores it",
+        "Names each communication-notification step: enabling the capability, declaring NSUserActivityTypes, donating an INInteraction, and calling content.updating(from:)",
+        "Requires exactly one content-handler call on every path including download failure and serviceExtensionTimeWillExpire"
       ]
     }
   ]