@mmerterden/multi-agent-pipeline 20.7.0 → 20.8.1

package/pipeline/skills/shared/external/device-integrity/evals/evals.json

@@ -4,41 +4,45 @@
     {
       "id": 0,
       "name": "app-attest-attestation-verifier",
-      "prompt": "We're adding server-side App Attest verification for an iOS app. The client sends keyId, attestationObject, and the original one-time challenge. What exact checks should the verifier perform before storing anything?",
-      "expected_output": "A source-grounded App Attest attestation verification checklist that covers CBOR/certificate validation, nonce composition, App ID and environment checks, credential ID/keyId checks, and storing verified public key plus receipt.",
+      "prompt": "Our backend team is writing the endpoint that receives App Attest attestation objects from the iOS client. List what the verifier has to check and what it should persist afterwards.",
+      "expected_output": "Walks through CBOR decoding, certificate chain validation against the App Attest root CA, nonce = SHA256(authData || SHA256(challenge)) matched against the certificate value under OID 1.2.840.113635.100.8.2, public key hash equal to keyId, RP ID hash equal to SHA256(teamID.bundleID), counter 0, aaguid for the expected environment, credentialId equal to keyId, then storing the public key and receipt and consuming the challenge only after success.",
       "files": [],
       "expectations": [
-        "Computes `clientDataHash = SHA256(challenge)`, then verifies the certificate nonce against `SHA256(authData || clientDataHash)` rather than `SHA256(challenge)` alone.",
-        "Names the credential certificate extension OID `1.2.840.113635.100.8.2` as the source of the nonce value to compare.",
-        "Verifies the App ID/RP ID hash, environment `aaguid`, initial counter, `credentialId`, and keyId/public-key hash relationship.",
-        "Stores the verified public key and receipt for future assertion and fraud-risk use instead of treating the raw attestation object as the trusted durable record."
+        "Composes the nonce from authData and the hashed challenge, SHA256(authData || SHA256(challenge)), rather than hashing the challenge alone",
+        "Compares the nonce with the certificate extension at 1.2.840.113635.100.8.2",
+        "Checks the RP ID or App ID hash against team ID plus bundle ID",
+        "Checks aaguid for the expected environment and an initial counter of 0",
+        "Requires credentialId and the public key hash to match keyId",
+        "Keeps the public key and receipt after verification, not the raw attestation object"
       ]
     },
     {
       "id": 1,
-      "name": "app-attest-sensitive-request-assertion",
-      "prompt": "Design the App Attest assertion contract for a premium-content download endpoint. The app already has an attested key. What should the client sign and what should the server verify on every sensitive request?",
-      "expected_output": "A replay-safe assertion design that signs client data containing a server challenge and request context, then verifies assertion CBOR, signature, RP ID, counter, challenge, and request binding server-side.",
+      "name": "gift-card-assertion-flow",
+      "prompt": "We want to protect our 'redeem gift card' POST endpoint with App Attest assertions. Show the client side and explain what the server verifies.",
+      "expected_output": "Fetches a fresh one-time challenge per request, builds client data with the challenge plus method, path and a body digest, hashes it, calls generateAssertion, sends assertion and client data in headers; server checks that the stored key signed SHA256(authenticatorData || SHA256(clientData)), RP ID, a strictly increasing counter, and that the challenge and request context match.",
       "files": [],
       "expectations": [
-        "Requires a fresh one-time server challenge for the assertion flow and embeds it in the client data.",
-        "Includes request context or a request-body hash in the signed client data so the assertion is bound to the sensitive request.",
-        "Verifies the assertion signature over `SHA256(authenticatorData || SHA256(clientData))` using the public key stored from attestation.",
-        "Checks the RP ID hash, monotonically increasing counter, and embedded challenge before accepting the request."
+        "Uses a fresh server-issued one-time challenge inside the client data",
+        "Binds request context or a body hash into the signed client data",
+        "Describes the signature as covering SHA256(authenticatorData || SHA256(clientData))",
+        "Server checks the RP ID hash and that the counter increases",
+        "Server confirms the challenge was issued and not yet consumed"
       ]
     },
     {
       "id": 2,
-      "name": "device-integrity-boundary-and-errors",
-      "prompt": "Review this plan: cache a DCDevice token for the whole install, use one App Attest key for all users, retry any attestKey failure with a new key immediately, treat invalidKey as an OS update problem, and add OAuth/session-token/certificate-pinning advice to the same device-integrity checklist. What should be corrected?",
-      "expected_output": "A boundary-aware correction list that fixes DeviceCheck token reuse, App Attest key scope, retry/error handling, environment behavior, and routes unrelated authentication/network security work to sibling guidance.",
+      "name": "integrity-plan-review",
+      "prompt": "Review our plan: cache the DeviceCheck token for a day, generate a new App Attest key each launch, retry every DCError with a new key, and use App Attest instead of OAuth sessions and certificate pinning. What would you change?",
+      "expected_output": "Makes DCDevice tokens single-use, keeps one key per account per device and warns that extra keys affect risk metrics, retries serverUnavailable reusing its keyId and clientDataHash while discarding the key for other attestation errors, explains invalidKey causes (already attested, not attested, rejected) without inventing causes such as system updates or enclave resets, and routes OAuth, sessions and pinning to the authentication and swift-security skills.",
       "files": [],
       "expectations": [
-        "States that `DCDevice.generateToken()` tokens are single-use and should not be cached for the whole install.",
-        "Requires App Attest keys to be unique per user account on each device and warns against unnecessary key generation because it affects risk metrics.",
-        "For `serverUnavailable`, retries attestation later with the same key and same `clientDataHash`; for other attestation errors, discards the key identifier before a new-key retry.",
-        "Describes `invalidKey` using Apple-documented causes: already-attested key, unattested assertion key, or service rejection, without attributing it to OS updates or Secure Enclave resets.",
-        "Keeps OAuth/session-token design, certificate pinning, and broad networking security out of the device-integrity checklist except as sibling-skill handoffs."
+        "Says DCDevice tokens are single-use and must not be cached",
+        "Keeps one key per account per device and mentions the key-count risk metric",
+        "On serverUnavailable, retries later reusing the identical keyId and clientDataHash",
+        "Discards the key and generates a new one for other attestation errors",
+        "Gives the three real invalidKey causes and does not invent others such as system updates or enclave resets",
+        "Routes OAuth, sessions and pinning to sibling skills instead of App Attest"
       ]
     }
   ]

package/pipeline/skills/shared/external/device-integrity/references/device-integrity-patterns.md

@@ -1,223 +1,187 @@
-# Device Integrity Extended Patterns
-
-Overflow reference for the `device-integrity` skill. Contains server verification details, advanced error handling, and integration patterns.
-
-## Contents
-
-- [Server-Side Attestation Verification](#server-side-attestation-verification)
-- [Server-Side Assertion Verification](#server-side-assertion-verification)
-- [Server Architecture](#server-architecture)
-- [Error Handling](#error-handling)
-- [Retry Strategy](#retry-strategy)
-- [Handling Rejected Keys](#handling-rejected-keys)
-- [Full Integration Manager](#full-integration-manager)
-- [Gradual Rollout](#gradual-rollout)
-- [Environment Entitlement](#environment-entitlement)
-
-## Server-Side Attestation Verification
-
-Your server must:
-1. Verify the attestation object is a valid CBOR-encoded structure.
-2. Extract the certificate chain and validate it against Apple's App Attest root CA.
-3. Compute `clientDataHash = SHA256(challenge)`, append it to the decoded
-   `authData`, then compute `nonce = SHA256(authData || clientDataHash)`.
-4. Extract the credential certificate extension with OID
-   `1.2.840.113635.100.8.2` and verify its octet string equals `nonce`.
-5. Verify the public-key hash matches the app-provided `keyId`.
-6. Verify the `RP ID` hash matches `SHA256(teamID + "." + bundleID)`.
-7. Verify the initial counter is `0`, the `aaguid` matches the expected
-   development or production environment, and `credentialId` equals `keyId`.
-8. Store the verified public key and receipt for future assertion verification.
-9. Mark the challenge consumed only after every verification step succeeds,
-   ideally in the same transaction that stores the key state.
-
-See [Validating apps that connect to your server](https://sosumi.ai/documentation/devicecheck/validating-apps-that-connect-to-your-server) for the full server verification algorithm.
-
-## Server-Side Assertion Verification
-
-Your server must:
-1. Decode the assertion (CBOR).
-2. Recompute `clientDataHash = SHA256(clientData)`, where `clientData`
-   includes a one-time server challenge and request context.
-3. Verify the signature using the stored public key over
+# Device Integrity Patterns
+
+Server verification, backend layout, error recovery and rollout for DeviceCheck
+and App Attest. The client basics are in the [skill overview](../SKILL.md).
+
+## Verifying an attestation on the server
+
+Run every step; reject on the first failure.
+
+1. Decode the attestation object and confirm the CBOR structure is well formed.
+2. Extract the certificates and confirm they chain up to Apple's App Attest root.
+3. Hash the challenge you issued (`clientDataHash = SHA256(challenge)`), concatenate
+   `authData` and that hash, then hash again: `nonce = SHA256(authData || clientDataHash)`.
+4. The leaf certificate carries an extension, OID `1.2.840.113635.100.8.2`,
+   whose octet string has to be byte-for-byte equal to `nonce`.
+5. Hash the credential certificate's public key. It must match the `keyId` the app sent.
+6. The RP ID hash in `authData` must equal `SHA256(teamID + "." + bundleID)`.
+7. The counter must be `0`, `aaguid` must name the environment you expect
+   (development or production), and `credentialId` must equal `keyId`.
+8. Persist the public key you just verified plus the receipt. Keep those, not the raw attestation object.
+9. Burn the challenge only once steps 1 to 8 all pass, ideally in the same
+   database transaction that writes the key record.
+
+## Verifying an assertion on the server
+
+1. Decode the CBOR assertion into `authenticatorData` and `signature`.
+2. Compute `clientDataHash = SHA256(clientData)`. The client data carries the
+   one-time challenge and the request context.
+3. Verify `signature` with the stored public key over
    `SHA256(authenticatorData || clientDataHash)`.
-4. Verify the `RP ID` hash and the counter (greater than the stored counter, or
-   greater than `0` for the first assertion).
-5. Confirm the embedded challenge matches the issued challenge and the request
-   context binds the assertion to the received request.
-6. Mark the challenge consumed and update the stored counter only after every
-   verification step succeeds, ideally atomically.
-
-## Server Architecture
-
-### Attestation vs. Assertion
-
-| Phase | When | What It Proves | Frequency |
-|-------|------|---------------|-----------|
-| **Attestation** | After key generation | The key lives on a genuine Apple device running a legitimate instance of your app | Once per key |
-| **Assertion** | With each sensitive request | The request came from the attested app instance | Per request |
-
-### Recommended Server Architecture
-
-1. **Challenge endpoint** -- generate a random nonce with at least 16 bytes of entropy, store it server-side with a short TTL (e.g., 5 minutes), purpose, and expected request/key context.
-2. **Attestation verification endpoint** -- validate the attestation object, store the public key and receipt keyed by `keyId`.
-3. **Assertion verification middleware** -- verify assertions on sensitive endpoints (purchases, account changes).
-
-Reject expired, missing, mismatched, or already-consumed challenges. Consume a
-challenge only after the corresponding attestation or assertion is fully
-verified; consuming on receipt can block safe retries after transient failures.
-
-### Risk Assessment
-
-Combine App Attest with [fraud risk assessment](https://sosumi.ai/documentation/devicecheck/assessing-fraud-risk) for defense in depth. App Attest alone does not guarantee the user is not abusing the app -- it confirms the app is genuine.
-
-App Attest is not a user authentication, session, entitlement, TLS, certificate
-pinning, or subscription validation system. Keep those controls in the
-appropriate authentication, networking, or broader security layer, and require
-them in addition to App Attest on protected endpoints.
-
-## Error Handling
-
-### DCError Codes
+4. Check the RP ID hash, and require the counter to be greater than the stored
+   counter; on a key's first assertion anything above zero passes.
+5. Confirm the embedded challenge is the one you issued and that the request
+   context (method, path, body digest) matches the request that arrived.
+6. Consume the challenge and store the new counter only after everything passes,
+   preferably atomically.
+
+## Server architecture
+
+### Attestation versus assertion
+
+| | Attestation | Assertion |
+|---|---|---|
+| When | Right after key generation | On every sensitive request |
+| Proves | Real hardware, real app, holds this key | This request came from that attested instance |
+| Frequency | Once per key | Once per request |
+
+### Suggested services
+
+- **Challenge endpoint** - hands out random nonces (16 or more bytes of
+  entropy). Store it with a short time to live (about five minutes), its purpose,
+  and the key or request it is meant for.
+- **Attestation endpoint** - runs the attestation steps and stores the public
+  key and receipt indexed by `keyId`.
+- **Assertion middleware** - sits in front of purchases, account changes and
+  other sensitive routes and runs the assertion steps.
+
+Reject a challenge that is missing, expired, already used, or issued for a
+different context. Consume it only after full verification; burning it the
+moment it arrives turns a transient failure into a lost challenge and blocks a
+safe retry.
+
+### Risk assessment
+
+Pair App Attest with Apple's fraud risk assessment for defense in depth. App
+Attest confirms the binary is genuine; it cannot tell you whether the person
+using it is abusing your service. It is also not user authentication, session
+management, entitlement or subscription validation, TLS or pinning. Protected
+endpoints still require all of those. Route sign-in work to `authentication`,
+transport security to `ios-networking`, and Keychain or pinning policy to
+`swift-security`.
+
+## Handling `DCError`
 
 ```swift
 import DeviceCheck
 
-func handleAttestError(_ error: Error) {
-    if let dcError = error as? DCError {
-        switch dcError.code {
-        case .unknownSystemFailure:
-            // Transient system error -- retry with exponential backoff
-            break
-        case .featureUnsupported:
-            // Device or OS does not support this feature
-            // Fall back to alternative verification
-            break
-        case .invalidKey:
-            // Already-attested key, unattested assertion key, or service rejection
-            // Inspect local/server state; discard and regenerate only when bad
-            break
-        case .invalidInput:
-            // The clientDataHash or keyId was malformed
-            break
-        case .serverUnavailable:
-            // Retry attestation later with the same keyId and clientDataHash
-            break
-        @unknown default:
-            break
-        }
+enum AttestRecovery {
+    case retryLater, useDeviceCheck, inspectKeyState, fixInput, giveUp
+}
+
+func recovery(for error: DCError) -> AttestRecovery {
+    switch error.code {
+    case .unknownSystemFailure:
+        return .retryLater          // transient system problem, back off
+    case .featureUnsupported:
+        return .useDeviceCheck      // no App Attest on this device or OS
+    case .invalidKey:
+        return .inspectKeyState     // regenerate only if the key is confirmed bad
+    case .invalidInput:
+        return .fixInput            // malformed clientDataHash or keyId
+    case .serverUnavailable:
+        return .retryLater          // same keyId and same hash on retry
+    @unknown default:
+        return .giveUp
     }
 }
 ```
 
-## Retry Strategy
+## Retry strategy
 
 ```swift
-import CryptoKit
-
-extension AppAttestManager {
-    func attestKeyWithRetry(challenge: Data, maxAttempts: Int = 3) async throws -> Data {
-        guard let keyId else {
-            throw DeviceIntegrityError.keyNotGenerated
-        }
-
-        let clientDataHash = Data(SHA256.hash(data: challenge))
-        var lastError: Error?
-
-        for attempt in 0..<maxAttempts {
-            do {
-                return try await service.attestKey(keyId, clientDataHash: clientDataHash)
-            } catch let error as DCError where error.code == .serverUnavailable {
-                lastError = error
-                if attempt < maxAttempts - 1 {
-                    try await Task.sleep(for: .seconds(pow(2.0, Double(attempt + 1))))
-                }
-            } catch {
-                throw error // Non-retryable errors propagate immediately
-            }
+func withAttestRetry<T>(
+    maxAttempts: Int = 3,
+    _ operation: () async throws -> T
+) async throws -> T {
+    var lastError: Error?
+    for attempt in 0..<maxAttempts {
+        do {
+            return try await operation()
+        } catch let failure as DCError {
+            guard failure.code == .serverUnavailable else { throw failure }
+            lastError = failure
+            let backoff = Duration.seconds(1 << (attempt + 1))
+            try await Task.sleep(for: backoff)
         }
-
-        throw lastError ?? DeviceIntegrityError.attestationFailed
     }
+    throw lastError ?? IntegrityError.attestationFailed
 }
 ```
 
-Use the same `challenge`, `keyId`, and `clientDataHash` for each retry after
-`.serverUnavailable`. Do not fetch a fresh challenge for that retry loop unless
-you are also starting over with a new attestation attempt.
+Only `.serverUnavailable` loops; every other error leaves immediately. Across
+those retries nothing changes: one challenge, one `keyId`, one `clientDataHash`. Ask for a
+new challenge only when you begin a brand-new attestation attempt.
 
-## Handling Rejected Keys
+## Rejected keys
 
-`DCError.invalidKey` means the app called `attestKey` for an already-attested
-key, called `generateAssertion` with an unattested key, or the App Attest service
-rejected the key. If local/server state confirms the key cannot be used, delete
-the stored `keyId` and generate a new key:
+`invalidKey` has three causes: `attestKey` on a key that is already attested,
+`generateAssertion` with a key that was never attested, or the service
+rejecting the key. Check local and server state first. When it confirms the key
+is unusable, remove it from the Keychain with the **same account-scoped query
+used to save it**, clear the cached value, and enroll again:
 
 ```swift
-extension AppAttestManager {
-    func handleRejectedKey() async throws -> String {
-        deleteKeyIdFromKeychain()
-        keyId = nil
+extension AttestKeyStore {
+    func discardKeyAndReenroll() async throws -> String {
+        let status = SecItemDelete(baseQuery as CFDictionary)
+        if status != errSecSuccess && status != errSecItemNotFound {
+            throw IntegrityError.keyNotGenerated
+        }
+        keyID = nil
         return try await generateKeyIfNeeded()
     }
-
-    private func deleteKeyIdFromKeychain() {
-        let query: [String: Any] = [
-            kSecClass as String: kSecClassGenericPassword,
-            kSecAttrAccount as String: "app-attest-key-id",
-            kSecAttrService as String: Bundle.main.bundleIdentifier ?? ""
-        ]
-        SecItemDelete(query as CFDictionary)
-    }
 }
 ```
 
-## Full Integration Manager
-
-Combine the patterns above into a single `actor` that manages the full lifecycle:
-1. Check `isSupported` and fall back to `DCDevice` tokens on unsupported devices.
-2. Call `generateKeyIfNeeded()` for each user account on each device, reuse the
-   account/device-scoped `keyId`, and limit new key generation to new
-   account/device/install enrollment or confirmed bad-key recovery.
-3. Attest once per key; if `.serverUnavailable` occurs, retry with the same
-   challenge, key, and `clientDataHash`.
-4. For each sensitive request, obtain a one-time assertion challenge and sign
-   client data that includes the challenge plus request context.
-5. Handle `DCError.invalidKey` by checking whether the key was already attested,
-   not yet attested, or rejected before regenerating.
+`baseQuery` is the actor property from the skill overview and already
+includes `"attest-key.\(accountID)"` as `kSecAttrAccount`. Building a separate
+delete query with a bare account string would miss the stored item, leave the
+stale `keyId` behind, and the next launch would load it again.
 
-## Gradual Rollout
+## Integration outline
 
-Apple recommends a gradual rollout. Gate App Attest behind a remote feature
-flag and fall back to `DCDevice` tokens on unsupported devices. For large apps,
-ramp production adoption gradually and be prepared to reduce attestation traffic
-if `.serverUnavailable` or rate-limit behavior increases during rollout.
+1. Check `DCAppAttestService.shared.isSupported`; when false, fall back to `DCDevice`.
+2. Call `generateKeyIfNeeded()` for the signed-in account. A new key is justified
+   only for a new account, device or install enrollment, or for confirmed
+   bad-key recovery.
+3. Attest once per key. On `.serverUnavailable`, retry with the same challenge,
+   key and hash.
+4. For each sensitive request, fetch a one-time challenge and sign it together
+   with the request context.
+5. On `invalidKey`, establish whether the key is already attested, not attested,
+   or rejected before replacing it.
 
-## Environment Entitlement
+## Gradual rollout
 
-Set the App Attest environment in your entitlements file. Use `development`
-during testing and `production` for App Store builds:
+Apple recommends enabling App Attest gradually. Put it behind a remote feature
+flag, keep the `DCDevice` fallback for unsupported devices, and ramp large
+audiences in steps. Be ready to switch attestation traffic off if
+`.serverUnavailable` errors or rate limiting climb.
 
-```xml
-<key>com.apple.developer.devicecheck.appattest-environment</key>
-<string>production</string>
-```
-
-When the entitlement is omitted during development, the app uses the App Attest
-sandbox by default. After distribution through TestFlight, the App Store, or the
-Apple Developer Enterprise Program, the app ignores the entitlement value and
-uses production. Sandbox keys and receipts do not work in production, and
-production keys and receipts do not work in sandbox.
+## Environment entitlement
 
-If an App Clip or extension uses App Attest, configure the capability for that
-target too. App Attest is supported only in Action, extensible SSO, and watchOS
-extensions; other extension types are unsupported even if `isSupported` returns
-`true`.
+The environment entitlement (`...devicecheck.appattest-environment`) is
+`development` or `production`. Development builds without it use the sandbox; TestFlight, App
+Store and Enterprise builds always use production. Keys and receipts from one
+environment fail in the other, so route server verification to match. App
+Clips and extensions that call App Attest need the capability on their own
+target as well.
 
-### Error Type
+### Error type used in the examples
 
 ```swift
-enum DeviceIntegrityError: Error {
+enum IntegrityError: Error {
     case deviceCheckUnsupported
     case keyNotGenerated
     case attestationFailed
@@ -227,14 +191,12 @@ enum DeviceIntegrityError: Error {
 }
 ```
 
-## Apple Documentation Links
-
-- [DeviceCheck framework](https://sosumi.ai/documentation/devicecheck)
-- [DCDevice](https://sosumi.ai/documentation/devicecheck/dcdevice)
-- [DCAppAttestService](https://sosumi.ai/documentation/devicecheck/dcappattestservice)
-- [Establishing your app's integrity](https://sosumi.ai/documentation/devicecheck/establishing-your-app-s-integrity)
-- [Validating apps that connect to your server](https://sosumi.ai/documentation/devicecheck/validating-apps-that-connect-to-your-server)
-- [Assessing fraud risk](https://sosumi.ai/documentation/devicecheck/assessing-fraud-risk)
-- [Preparing to use the app attest service](https://sosumi.ai/documentation/devicecheck/preparing-to-use-the-app-attest-service)
-- [Attestation Object Validation Guide](https://sosumi.ai/documentation/devicecheck/attestation-object-validation-guide)
-- [App Attest Environment](https://sosumi.ai/documentation/bundleresources/entitlements/com.apple.developer.devicecheck.appattest-environment)
+## Apple documentation
+
+- [DeviceCheck framework](https://developer.apple.com/documentation/devicecheck), whose article list includes
+  the app integrity overview, validating apps that talk to your server, the
+  attestation object guide, and preparation steps for the App Attest service
+- [Fraud risk assessment](https://developer.apple.com/documentation/devicecheck/assessing-fraud-risk)
+- [DCAppAttestService](https://developer.apple.com/documentation/devicecheck/dcappattestservice)
+- [DCDevice](https://developer.apple.com/documentation/devicecheck/dcdevice)
+- The App Attest environment page under Bundle Resources > Entitlements