@mmerterden/multi-agent-pipeline 20.6.0 → 20.8.0

package/pipeline/skills/shared/external/metrickit-diagnostics/evals/evals.json

@@ -3,43 +3,46 @@
   "evals": [
     {
       "id": 0,
-      "prompt": "Review this MetricKit telemetry plan for an iOS app: the subscriber is created in a dashboard view, only didReceive([MXMetricPayload]) is implemented, payloads are parsed and uploaded before saving, and missed reports are ignored. Give corrected guidance with concise Swift where useful.",
-      "expected_output": "A source-grounded MetricKit setup review that registers early, handles both metric and diagnostic payloads, persists raw JSON before processing, offloads heavier work, and retrieves past reports precisely.",
+      "name": "production-ingestion-pipeline",
+      "prompt": "We want MetricKit data from our shipping iOS app in our own backend: daily performance numbers plus crashes and hangs with symbolicated stacks. A teammate registered the subscriber in the settings screen and parses everything inside the callback. Design the pipeline properly.",
+      "expected_output": "Registers one long-lived subscriber at startup (didFinishLaunchingWithOptions or App.init), implements both didReceive callbacks, writes each payload's jsonRepresentation() to disk before any parsing, symbolication or upload, moves heavy work off the callback, backfills from pastPayloads and pastDiagnosticPayloads, and does not promise that daily metric payloads show up immediately while developing.",
       "files": [],
       "expectations": [
-        "Recommends registering a long-lived MXMetricManagerSubscriber during app startup, such as UIApplication launch or App.init.",
-        "Implements or explicitly requires both didReceive(_: [MXMetricPayload]) and didReceive(_: [MXDiagnosticPayload]).",
-        "Persists jsonRepresentation() before parsing, symbolication, or upload work.",
-        "Moves expensive parsing or uploads out of the subscriber callback after persistence.",
-        "Uses pastPayloads and pastDiagnosticPayloads, or otherwise addresses missed reports during startup backfill.",
-        "Does not claim that MetricKit callbacks are a normal immediate development-time signal for daily metric payloads."
+        "Subscribes a long-lived object at app startup rather than in a screen",
+        "Implements both the metric and the diagnostic callback",
+        "Persists raw JSON before parsing, symbolicating or uploading",
+        "Moves heavy processing out of the callback",
+        "Backfills with pastPayloads and pastDiagnosticPayloads",
+        "Does not claim daily metric payloads arrive immediately during development"
       ]
     },
     {
       "id": 1,
-      "prompt": "Fix this MetricKit code review comment: the team uses MXSignpostIntervalData.makeSignpostID(log:), passes a custom signpostID into mxSignpost, and calls MXMetricManager.shared.extendLaunchMeasurement without try from a background task. What should the corrected guidance say?",
-      "expected_output": "A correction-focused answer that removes the fake signpost helper, preserves MetricKit mxSignpost defaults, and uses the current extended launch type methods with availability and main-thread constraints.",
+      "name": "signpost-and-launch-fix",
+      "prompt": "Fix this code: it creates an ID with MXSignpostIntervalData.makeSignpostID(log:), passes it to every mxSignpost call, and extends launch with MXMetricManager.shared.extendLaunchMeasurement(forTaskID:) from a background queue after the first screen appears. Where do the signpost numbers end up?",
+      "expected_output": "Says makeSignpostID(log:) is not documented MetricKit API, uses MXMetricManager.makeLogHandle(category:) with default mxSignpost arguments, and points to MXMetricPayload.signpostMetrics in the daily payload. Rewrites extended launch with the throwing type methods extendLaunchMeasurement(forTaskID:) and finishExtendedLaunchMeasurement(forTaskID:), called with try on the main thread before the first scene is active, with at most 16 overlapping tasks that must all be finished.",
       "files": [],
       "expectations": [
-        "States that MXSignpostIntervalData.makeSignpostID(log:) is not the documented MetricKit API.",
-        "Uses MXMetricManager.makeLogHandle(category:) and mxSignpost(.begin/.end, log:name:) without custom dso, signpostID, or format parameters for the basic MetricKit pattern.",
-        "States that MetricKit custom signpost metrics appear under MXMetricPayload.signpostMetrics.",
-        "Uses try MXMetricManager.extendLaunchMeasurement(forTaskID:) and try or try? MXMetricManager.finishExtendedLaunchMeasurement(forTaskID:) as type methods, not shared instance methods.",
-        "Mentions Apple's operational constraints for extended launch measurement: main thread, early scene activation or state restoration timing, max 16 tasks, overlapping task windows, or finishing every started task."
+        "Rejects makeSignpostID(log:) as undocumented",
+        "Uses makeLogHandle(category:) and default mxSignpost parameters",
+        "Names MXMetricPayload.signpostMetrics as where the data appears",
+        "Uses try with the static extendLaunchMeasurement and finishExtendedLaunchMeasurement methods instead of the shared instance",
+        "States the main-thread, early-start, 16-task, overlapping-window and finish-every-task rules"
       ]
     },
     {
       "id": 2,
-      "prompt": "My SwiftUI feed stutters in development, Instruments shows a hot body update, and product wants production hang/crash telemetry in our analytics backend. What should the MetricKit skill handle directly, and what should be routed to adjacent skills?",
-      "expected_output": "A boundary-aware answer that keeps production MetricKit ingestion in scope while routing SwiftUI code remediation and local Instruments profiling details to sibling skills.",
+      "name": "scope-routing",
+      "prompt": "Organizer shows rising hang rates and a crash spike after our SwiftUI redesign. Tell us what MetricKit can do here, and what we should take to other tools.",
+      "expected_output": "Keeps MetricKit to manager setup, payload handling, reading MXHangDiagnostic and MXCrashDiagnostic, and uploading call-stack JSON with dSYMs to a backend. Sends SwiftUI body cost and invalidation fixes to swiftui-performance and local Instruments, LLDB, Memory Graph or xctrace work to debugging-instruments. Explains that MetricKit is production telemetry aggregated daily, and does not turn into a refactoring plan or an Instruments tutorial.",
       "files": [],
       "expectations": [
-        "Keeps MXMetricManager setup, MXMetricPayload/MXDiagnosticPayload handling, MXHangDiagnostic or MXCrashDiagnostic processing, and backend upload guidance in the MetricKit scope.",
-        "Routes SwiftUI view invalidation, body-cost, identity, and remediation patterns to the swiftui-performance skill.",
-        "Routes local Instruments capture, LLDB, Memory Graph, and xctrace workflow details to the debugging-instruments skill.",
-        "Mentions using MXCallStackTree JSON plus dSYMs or an analytics symbolication pipeline for crash and hang triage.",
-        "Mentions MetricKit as production telemetry rather than immediate local debugging, including daily aggregation or non-realtime behavior.",
-        "Does not expand into a full SwiftUI refactor or full Instruments tutorial."
+        "Keeps manager setup, payload handling, hang and crash diagnostics and backend upload in MetricKit",
+        "Mentions call-stack JSON uploaded with dSYMs",
+        "Sends SwiftUI fixes to swiftui-performance",
+        "Sends Instruments, LLDB, Memory Graph and xctrace work to debugging-instruments",
+        "Describes MetricKit as production data aggregated daily",
+        "Does not expand into a refactor or an Instruments walkthrough"
       ]
     }
   ]

package/pipeline/skills/shared/external/metrickit-diagnostics/references/metrickit-patterns.md

@@ -1,180 +1,173 @@
-# MetricKit Extended Patterns
+# MetricKit patterns
 
-Overflow reference for the `metrickit` skill. Contains deeper payload analysis, export patterns, custom signpost metrics, and extended launch measurement.
+Worked examples for the `metrickit-diagnostics` skill ([../SKILL.md](../SKILL.md)).
 
-## Contents
+## Reading call-stack trees
 
-- [Call Stack Trees](#call-stack-trees)
-- [Custom Signpost Metrics](#custom-signpost-metrics)
-- [Exporting and Uploading Payloads](#exporting-and-uploading-payloads)
-- [Past Payloads](#past-payloads)
-- [Extended Launch Measurement](#extended-launch-measurement)
-- [Xcode Organizer Integration](#xcode-organizer-integration)
-
-## Call Stack Trees
-
-`MXCallStackTree` is attached to each diagnostic (crash, hang, CPU exception,
-disk write, app launch). Use `jsonRepresentation()` to extract and symbolicate.
+Every diagnostic class (crashes, hangs, CPU and disk-write exceptions, slow
+launches) exposes a `callStackTree`.
 
 ```swift
-func handleCrash(_ crash: MXCrashDiagnostic) {
-    let tree = crash.callStackTree
-    let treeJSON = tree.jsonRepresentation()
-
-    let exceptionType = crash.exceptionType
-    let signal = crash.signal
-    let reason = crash.terminationReason
-
-    uploadDiagnostic(
-        type: "crash",
-        exceptionType: exceptionType,
-        signal: signal,
-        reason: reason,
-        callStack: treeJSON
+import MetricKit
+
+func describe(_ crash: MXCrashDiagnostic) -> CrashRecord {
+    CrashRecord(
+        stack: crash.callStackTree.jsonRepresentation(),
+        exception: crash.exceptionType?.intValue,
+        signal: crash.signal?.intValue,
+        reason: crash.terminationReason
     )
 }
 
-func handleHang(_ hang: MXHangDiagnostic) {
-    let tree = hang.callStackTree
-    let duration = hang.hangDuration  // Measurement<UnitDuration>
-    uploadDiagnostic(type: "hang", duration: duration, callStack: tree.jsonRepresentation())
+func describe(_ hang: MXHangDiagnostic) -> HangRecord {
+    let seconds = hang.hangDuration.converted(to: .seconds).value
+    return HangRecord(stack: hang.callStackTree.jsonRepresentation(), seconds: seconds)
 }
-```
-
-The JSON structure contains an array of call stack frames with binary name,
-offset, and address. Symbolicate using `atos` or upload dSYMs to your
-analytics service.
 
-**Availability**: `MXCallStackTree`  -  iOS 14.0+, iPadOS 14.0+,
-Mac Catalyst 14.0+, macOS 12.0+, visionOS 1.0+
+struct CrashRecord { let stack: Data; let exception: Int?; let signal: Int?; let reason: String? }
+struct HangRecord { let stack: Data; let seconds: Double }
+```
 
-## Custom Signpost Metrics
+The stack JSON lists each frame with its binary name, offset into the binary
+and address. That is what `atos` or a crash service needs, together with the
+matching dSYMs.
 
-Use `mxSignpost` with a MetricKit log handle to capture custom performance
-intervals. These appear in the daily `MXMetricPayload` under `signpostMetrics`.
+## Custom signposts end to end
 
-### Creating a Log Handle
-
-```swift
-let metricLog = MXMetricManager.makeLogHandle(category: "Networking")
-```
-
-### Emitting Signposts
+### Emitting
 
 ```swift
+import MetricKit
 import os
 
-func fetchData() async throws -> Data {
-    mxSignpost(.begin, log: metricLog, name: "DataFetch")
-    let data = try await URLSession.shared.data(from: url).0
-    mxSignpost(.end, log: metricLog, name: "DataFetch")
+let syncLog = MXMetricManager.makeLogHandle(category: "Sync")
 
-    return data
+func pullLatestRoutes(from endpoint: URL) async throws -> Data {
+    mxSignpost(.begin, log: syncLog, name: "PullRoutes")
+    defer { mxSignpost(.end, log: syncLog, name: "PullRoutes") }
+    let session = URLSession(configuration: .ephemeral)
+    let (payload, _) = try await session.data(from: endpoint)
+    return payload
 }
 ```
 
-For MetricKit custom metrics, create the log with
-`MXMetricManager.makeLogHandle(category:)` and leave the `mxSignpost` overload's
-advanced `dso`, `signpostID`, and `format` parameters at their documented
-defaults.
-
-### Reading Custom Metrics from Payload
+### Reading them back
 
 ```swift
-if let signposts = payload.signpostMetrics {
-    for metric in signposts {
-        let name = metric.signpostName       // "DataFetch"
-        let category = metric.signpostCategory // "Networking"
-        let count = metric.totalCount
-        if let intervalData = metric.signpostIntervalData {
-            let avgMemory = intervalData.averageMemory
-            let cumulativeCPUTime = intervalData.cumulativeCPUTime
-        }
+func logSignposts(in payload: MXMetricPayload) {
+    for metric in payload.signpostMetrics ?? [] {
+        let intervals = metric.signpostIntervalData
+        telemetryLog.info("""
+            \(metric.signpostCategory)/\(metric.signpostName): \(metric.totalCount) runs, \
+            avg memory \(String(describing: intervals?.averageMemory)), \
+            cpu \(String(describing: intervals?.cumulativeCPUTime))
+            """)
     }
 }
+
+let telemetryLog = Logger(subsystem: "com.example.trailhead", category: "metrics")
 ```
 
-> The system limits the number of custom signpost metrics per log to reduce
-> on-device overhead. Reserve custom metrics for critical code paths.
+The system caps how many custom signpost metrics each log may record, to keep
+overhead low. Spend them on the few paths that matter.
 
-## Exporting and Uploading Payloads
+## Exporting and uploading
 
-Both payload types conform to `NSSecureCoding` and provide
-`jsonRepresentation()` for easy serialization.
+Both payload classes also conform to `NSSecureCoding`, but JSON is the
+format backends want.
 
 ```swift
-func persistPayload(_ jsonData: Data, from: Date? = nil, to: Date? = nil) {
-    let fileName = "metrics_\(ISO8601DateFormatter().string(from: Date())).json"
-    let url = FileManager.default.temporaryDirectory.appending(path: fileName)
-    try? jsonData.write(to: url)
-}
+enum PayloadArchive {
+    static func save(_ json: Data, kind: String) throws -> URL {
+        let stamp = ISO8601DateFormatter().string(from: .now)
+        let file = FileManager.default.temporaryDirectory
+            .appendingPathComponent("metrickit-\(kind)-\(stamp).json")
+        try json.write(to: file, options: .atomic)
+        return file
+    }
 
-func uploadPayloads(_ jsonData: Data) {
-    Task.detached(priority: .utility) {
-        var request = URLRequest(url: URL(string: "https://api.example.com/metrics")!)
-        request.httpMethod = "POST"
-        request.setValue("application/json", forHTTPHeaderField: "Content-Type")
-        request.httpBody = jsonData
-        _ = try? await URLSession.shared.data(for: request)
+    static func upload(_ file: URL, to collector: URL) {
+        Task.detached(priority: .utility) {
+            var request = URLRequest(url: collector)
+            request.httpMethod = "POST"
+            request.addValue("application/json", forHTTPHeaderField: "Content-Type")
+            let json = try Data(contentsOf: file)
+            let (_, reply) = try await URLSession.shared.upload(for: request, from: json)
+            guard (reply as? HTTPURLResponse)?.statusCode == 200 else { return }
+            try FileManager.default.removeItem(at: file)
+        }
     }
 }
 ```
 
-## Past Payloads
+In production prefer Application Support or Caches over the temporary
+directory if files must survive until the next launch.
 
-If the subscriber was not registered when payloads arrived, retrieve them
-using `pastPayloads` and `pastDiagnosticPayloads`. These return reports
-generated since the last allocation of the shared manager instance.
+## Catching up on missed payloads
+
+Deliveries that happened while no subscriber was registered are not lost;
+they stay reachable through `MXMetricManager.shared.pastPayloads` and
+`pastDiagnosticPayloads`, which cover everything produced since the shared
+instance was last allocated.
 
 ```swift
-let pastMetrics = MXMetricManager.shared.pastPayloads
-let pastDiags = MXMetricManager.shared.pastDiagnosticPayloads
+func backfill() throws {
+    let manager = MXMetricManager.shared
+    for payload in manager.pastPayloads {
+        _ = try PayloadArchive.save(payload.jsonRepresentation(), kind: "metric")
+    }
+    for payload in manager.pastDiagnosticPayloads {
+        _ = try PayloadArchive.save(payload.jsonRepresentation(), kind: "diagnostic")
+    }
+}
 ```
 
-## Extended Launch Measurement
+## Extended launch rules
 
-Track post-first-draw setup work (loading databases, restoring state) as part
-of the launch metric using extended launch measurement on iOS 16+, iPadOS 16+,
-Mac Catalyst 16+, macOS 13+, and visionOS 1+.
+Typical tasks worth including: opening the database, restoring saved state,
+fetching the data the first screen shows.
 
-```swift
-let taskID = MXLaunchTaskID("com.example.app.loadDatabase")
+- Begin the earliest task no later than state restoration, while the first
+  scene is still inactive.
+- Up to 16 tasks can be registered.
+- Task windows may overlap.
+- Measurement ends once every running task has finished, so finish each one,
+  including on error paths.
 
-try MXMetricManager.extendLaunchMeasurement(forTaskID: taskID)
-defer { try? MXMetricManager.finishExtendedLaunchMeasurement(forTaskID: taskID) }
-
-restoreCachedState()
-connectInitialSceneData()
+```swift
+@MainActor
+func restoreSession() async {
+    let restore = MXLaunchTaskID("com.example.trailhead.restore")
+    let routes = MXLaunchTaskID("com.example.trailhead.routes")
+    try? MXMetricManager.extendLaunchMeasurement(forTaskID: restore)
+    try? MXMetricManager.extendLaunchMeasurement(forTaskID: routes)
+
+    await sessionStore.restore()
+    try? MXMetricManager.finishExtendedLaunchMeasurement(forTaskID: restore)
+
+    await routeCache.warm()
+    try? MXMetricManager.finishExtendedLaunchMeasurement(forTaskID: routes)
+}
 ```
 
-Extended launch times appear under `histogrammedExtendedLaunch` in
-`MXAppLaunchMetric`.
-
-Use these throwing type methods on the main thread. Start the first task before
-or during state restoration, or before the first scene becomes active. The
-system supports up to 16 tasks; task windows need to overlap, and extended
-launch measurement ends when all running tasks finish.
-
-## Xcode Organizer Integration
-
-Xcode Organizer shows the same MetricKit data aggregated across all users
-who have opted in to share diagnostics. Use Organizer for trend analysis:
+## Xcode Organizer
 
-- **Metrics tab**: Battery, performance, and disk-write metrics over time
-- **Regressions tab**: Automatic detection of metric regressions per version
-- **Crashes tab**: Crash logs with symbolicated stack traces
+| Tab | Shows |
+|---|---|
+| Metrics | Battery, performance and disk writes over time |
+| Regressions | Regressions detected automatically between versions |
+| Crashes | Symbolicated crash logs |
 
-MetricKit on-device collection complements Organizer by letting you route
-raw data to your own backend for custom dashboards, alerting, and filtering
-by user cohort.
+Organizer covers the aggregate view. Your own on-device collection adds what
+it cannot: your own dashboards and alerts, plus slicing by user segment.
 
-## Apple Documentation Links
+## Apple documentation
 
-- [MetricKit framework](https://sosumi.ai/documentation/metrickit)
-- [MXMetricManager](https://sosumi.ai/documentation/metrickit/mxmetricmanager)
-- [MXMetricManagerSubscriber](https://sosumi.ai/documentation/metrickit/mxmetricmanagersubscriber)
-- [MXCallStackTree](https://sosumi.ai/documentation/metrickit/mxcallstacktree)
-- [MXSignpostMetric](https://sosumi.ai/documentation/metrickit/mxsignpostmetric)
-- [MXAppLaunchMetric](https://sosumi.ai/documentation/metrickit/mxapplaunchmetric)
-- [MXCrashDiagnostic](https://sosumi.ai/documentation/metrickit/mxcrashdiagnostic)
-- [Analyzing the performance of your shipping app](https://sosumi.ai/documentation/xcode/analyzing-the-performance-of-your-shipping-app)
+- https://developer.apple.com/documentation/metrickit
+- https://developer.apple.com/documentation/metrickit/mxmetricmanager
+- https://developer.apple.com/documentation/metrickit/mxmetricmanagersubscriber
+- https://developer.apple.com/documentation/metrickit/mxcallstacktree
+- https://developer.apple.com/documentation/metrickit/mxsignpostmetric
+- https://developer.apple.com/documentation/metrickit/mxapplaunchmetric
+- https://developer.apple.com/documentation/metrickit/mxcrashdiagnostic
+- Apple article on analyzing a shipping app's performance, in the Xcode documentation