@mmerterden/multi-agent-pipeline 20.7.0 → 20.8.0

package/pipeline/skills/shared/external/ios-accessibility/evals/evals.json

@@ -4,46 +4,47 @@
     {
       "id": 0,
       "name": "assistive-technology-review",
-      "prompt": "Review this SwiftUI shopping cart screen for accessibility before implementation: each row has an icon, product name, price, stepper-like +/- icons built with tap gestures, a swipe-to-delete action, duplicate \"Edit\" buttons, a favorite heart-only button, a checkout sheet, and product photos. Product wants VoiceOver, Voice Control, Switch Control, Dynamic Type, and Reduce Motion support. What should we fix?",
-      "expected_output": "A practical accessibility review that covers labels, values, traits, grouping, gesture alternatives, Voice Control input labels, Switch Control custom actions, focus restoration, Dynamic Type, decorative/content images, and Reduce Motion.",
+      "prompt": "Before we build it, look over the plan for our SwiftUI pharmacy basket. A row has a medicine icon, the product name, the price, and a plus/minus quantity control made from tap gestures. Swiping a row deletes it. Every row carries a visible 'Edit' button, a heart-only button favourites the item, 'Place order' opens a sheet, and product photos sit at the top. What has to change so it works with the screen reader, spoken commands, switch scanning, larger text sizes and the reduced-motion setting?",
+      "expected_output": "Goes through each assistive path in turn. The heart button and the quantity control get a useful label, a value and the right trait, and none of the labels repeat the control's type; the quantity control becomes adjustable. Every basket row collapses into one element with accessibilityElement(children: .combine), which cuts the stops for both the screen reader and switch scanning. Deleting, favouriting and changing quantity are each available as a named accessibility action. The repeated Edit buttons get distinct spoken names, long product names get accessibilityInputLabels, and testing uses the Show Names and Show Numbers overlays. When the order sheet closes, focus goes back to Place order through @AccessibilityFocusState or a UIKit notification. Text uses system styles, the layout adapts, sizes use @ScaledMetric, and motion checks accessibilityReduceMotion. Product photos get descriptions while purely decorative art is hidden.",
       "files": [],
-      "assertions": [
-        "Requires meaningful accessibility labels, values, and traits for icon-only buttons and custom stepper controls without putting control types like \"button\" in the label.",
-        "Recommends grouping each cart row with `accessibilityElement(children: .combine)` or an equivalent pattern to reduce VoiceOver and Switch Control scan stops.",
-        "Provides `.accessibilityAction(named:)` custom actions for swipe-to-delete, favorite, quantity changes, or other gesture-only behavior so Switch Control and VoiceOver can operate them.",
-        "Mentions Voice Control testing with \"Show Names\" and/or \"Show Numbers\" and recommends unique, speakable labels plus `accessibilityInputLabels` for long or awkward names.",
-        "Requires returning accessibility focus to the checkout trigger after dismissing the sheet with `@AccessibilityFocusState` or an equivalent UIKit notification.",
-        "Covers Dynamic Type and Reduce Motion using system text styles, adaptive layout, `@ScaledMetric`, and `accessibilityReduceMotion` rather than fixed fonts or motion-only feedback."
+      "expectations": [
+        "The heart button and the custom quantity control receive a label, value and trait, and no label states the kind of control",
+        "Rows are merged with accessibilityElement(children: .combine) or an equivalent so the screen reader and switch scanning stop once per row",
+        "Delete, favourite, quantity changes and any other gesture-only action are exposed with accessibilityAction(named:)",
+        "Unique, pronounceable names for the Edit buttons, accessibilityInputLabels for long product names, and testing with the Show Names and/or Show Numbers overlay",
+        "Focus is returned to the Place order button once the sheet closes, using @AccessibilityFocusState or a UIKit accessibility notification",
+        "Larger text and reduced motion are handled with text styles, adaptive layout, @ScaledMetric and accessibilityReduceMotion instead of fixed font sizes or feedback that exists only as motion",
+        "Separates product photos, which need a description, from decorative images, which are hidden"
       ]
     },
     {
       "id": 1,
       "name": "nutrition-labels-current-categories",
-      "prompt": "We're preparing App Store Accessibility Nutrition Labels for a media-heavy iOS 26 app. The team wants to claim VoiceOver, Switch Control, Full Keyboard Access, Closed Captions / SDH, Audio Descriptions, Larger Text, Reduce Motion, and Increase Contrast. Audit this list against Apple's current label categories and tell us what evidence to collect before answering in App Store Connect.",
-      "expected_output": "A source-grounded label audit that corrects stale category names, routes non-label assistive technology work appropriately, and describes evidence needed for common-task claims.",
+      "prompt": "Our language-learning app for iOS 26 is mostly video. In App Store Connect we plan to tick Larger Text, Increase Contrast, VoiceOver, Closed Captions / SDH, Switch Control, Reduce Motion, Audio Descriptions and Full Keyboard Access. Which of these are real labels we can declare, and what evidence do we gather first?",
+      "expected_output": "Gives the current set of nine labels and no others: Captions, Audio Descriptions, VoiceOver, Voice Control, Larger Text, Sufficient Contrast, Dark Interface, Reduced Motion and Differentiate Without Color Alone. Points out that keyboard access and switch scanning are real accessibility work yet have no label of their own. Maps the old names to the current ones (Closed Captions / SDH becomes Captions, Increase Contrast becomes Sufficient Contrast, and the label is Reduced Motion). A label is claimed only when every common task can be completed with the feature on that device type. The evidence to collect: a matrix of common tasks, passes with the screen reader and with spoken commands, text sizes up to at least 200% where supported, contrast measurements, checks with the reduced-motion setting on, and a review of caption tracks, description tracks or transcripts. Ends by warning that these answers have to stay true and are not a marketing surface.",
       "files": [],
-      "assertions": [
-        "Lists the current App Store Accessibility Nutrition Label categories as VoiceOver, Voice Control, Larger Text, Dark Interface, Differentiate Without Color Alone, Sufficient Contrast, Reduced Motion, Captions, and Audio Descriptions.",
-        "States that Switch Control and Full Keyboard Access are important accessibility work but are not current App Store Accessibility Nutrition Label categories.",
-        "Renames Closed Captions / SDH to Captions and Increase Contrast to Sufficient Contrast, while also covering Audio Descriptions and Reduced Motion.",
-        "Explains that labels should be claimed only when users can complete all common tasks with that feature on the relevant device type.",
-        "Calls for evidence such as a common-task test matrix, VoiceOver and Voice Control testing, text-size testing to at least 200% where supported, contrast checks, Reduce Motion checks, and caption/audio-description track or transcript review.",
-        "Warns that App Store accessibility label answers must remain accurate and should not be treated as marketing claims."
+      "expectations": [
+        "Gives exactly the nine current label categories",
+        "Says keyboard access and switch scanning matter but are not label categories",
+        "Replaces Closed Captions / SDH with Captions and Increase Contrast with Sufficient Contrast, and also covers Audio Descriptions and Reduced Motion",
+        "A label is only claimed when every common task works with the feature on the relevant device type",
+        "Evidence covers a common-task matrix, screen reader and spoken-command testing, text sizes of 200% or more where supported, contrast checks, reduced-motion checks, and review of caption or description tracks or transcripts",
+        "Warns that the App Store Connect accessibility answers must remain accurate and are not marketing"
       ]
     },
     {
       "id": 2,
       "name": "focus-boundary-routing",
-      "prompt": "A team asks the iOS accessibility skill to fix all focus bugs in a SwiftUI iPad app: VoiceOver focus is lost after closing a custom modal, Tab order skips a custom card, arrow-key movement between panels feels wrong, a tvOS build needs UIFocusGuide routing, and Voice Control names are duplicated. Give a concise scope review and the first fixes.",
-      "expected_output": "A boundary-aware accessibility focus answer that fixes assistive-technology focus and Voice Control issues while routing keyboard, directional, tvOS, and UIFocusGuide work to the focus-engine skill.",
+      "prompt": "Our iPad SwiftUI notes app has several focus bugs. When a custom formatting overlay closes, the screen reader cursor lands somewhere random. Pressing Tab skips our custom tag cards. Arrow keys hop strangely between the sidebar and the editor. The tvOS companion app needs UIFocusGuide routing. Voice Control shows two buttons that are both called 'More'. Which of these are accessibility work, which are keyboard focus work, and how is each fixed?",
+      "expected_output": "Treats the lost screen reader cursor after the overlay as accessibility work: restore it with @AccessibilityFocusState or an accessibility notification, and give the overlay the isModal trait plus an escape action. Treats the two 'More' buttons as accessibility work too: give each a distinct spoken name and add accessibilityInputLabels. Sends the keyboard mechanics to hig-inputs: UIFocusGuide, tvOS directional focus, focus sections, arrow-key movement, @FocusState, .focusable(), the skipped cards and Tab order. Explains that the assistive-technology cursor and keyboard or directional scene focus are separate systems, and still records how element order and grouping shape traversal for the screen reader, switch scanning, spoken-command overlays and keyboard access.",
       "files": [],
-      "assertions": [
-        "Keeps VoiceOver/accessibility focus restoration after modal dismissal in ios-accessibility scope and recommends `@AccessibilityFocusState` or accessibility notifications.",
-        "Keeps duplicate Voice Control names in ios-accessibility scope and recommends unique, speakable labels and `accessibilityInputLabels` where appropriate.",
-        "Routes Tab order, `.focusable()`, `@FocusState`, arrow-key movement, focus sections, tvOS directional focus, and `UIFocusGuide` details to the focus-engine skill rather than expanding the accessibility skill into that domain.",
-        "Explains the distinction between accessibility focus for assistive technologies and keyboard/directional focus in the scene.",
-        "Still notes that accessibility element order can affect VoiceOver, Switch Control, Voice Control, and Full Keyboard Access traversal when reviewing user-facing UI."
+      "expectations": [
+        "Restoring the screen reader cursor after the overlay closes stays in this skill and uses @AccessibilityFocusState or accessibility notifications",
+        "The duplicate 'More' names stay in this skill and are fixed with distinct spoken labels and accessibilityInputLabels",
+        "hig-inputs receives the keyboard mechanics: UIFocusGuide, tvOS directional focus, focus sections, arrow keys, @FocusState, .focusable() and Tab order",
+        "Explains how assistive-technology focus differs from keyboard or directional scene focus",
+        "Notes that element order still affects traversal for the screen reader, switch scanning, Voice Control and keyboard access"
       ]
     }
   ]
-}
+}