@mmerterden/multi-agent-pipeline 20.7.0 → 20.8.0

package/pipeline/skills/shared/external/speech-recognition/evals/evals.json

@@ -4,45 +4,42 @@
     {
       "id": 0,
       "name": "speechanalyzer-live-transcription",
-      "prompt": "I'm building an iOS 26 meeting recorder with live transcript text and word highlighting during playback. Sketch the SpeechAnalyzer implementation shape, including model setup, audio input, result handling, and cleanup.",
-      "expected_output": "An iOS 26 SpeechAnalyzer plan that uses current SpeechTranscriber APIs, installs assets, converts live audio buffers, handles volatile/final results, and explicitly finishes the analyzer session.",
-      "files": [],
+      "prompt": "Build live captions for an iOS 26 lecture-recording app with SpeechAnalyzer. Captions should appear while the lecturer speaks and later highlight words during playback. A draft we found uses SpeechTranscriber(locale:preset: .offlineTranscription) and simply finishes the input stream when recording stops. Correct it and outline the full flow.",
+      "expected_output": "Replaces the nonexistent .offlineTranscription with a documented preset such as .timeIndexedProgressiveTranscription, checks SpeechTranscriber.isAvailable and supportedLocale(equivalentTo:), installs assets through AssetInventory.assetInstallationRequest(supporting:), converts microphone buffers to SpeechAnalyzer.bestAvailableAudioFormat(compatibleWith:), keeps volatile and final text apart, and ends the session explicitly with a finalize or cancel call rather than relying on the input stream closing.",
       "assertions": [
-        "Uses SpeechAnalyzer with SpeechTranscriber and a documented preset such as .timeIndexedProgressiveTranscription, not .offlineTranscription.",
-        "Checks SpeechTranscriber.isAvailable and supportedLocale(equivalentTo:) before starting transcription.",
-        "Installs or verifies model assets with AssetInventory.assetInstallationRequest(supporting:) before analysis.",
-        "Converts AVAudioEngine microphone buffers to SpeechAnalyzer.bestAvailableAudioFormat(compatibleWith:) before yielding AnalyzerInput.",
-        "Handles volatile results separately from finalized results so live text is replaced rather than duplicated.",
-        "Explicitly finalizes or cancels the analyzer session after input ends."
+        "Uses a documented preset such as .timeIndexedProgressiveTranscription and rejects .offlineTranscription",
+        "Checks SpeechTranscriber.isAvailable and supportedLocale(equivalentTo:)",
+        "Installs assets with AssetInventory.assetInstallationRequest(supporting:)",
+        "Converts live buffers to bestAvailableAudioFormat(compatibleWith:)",
+        "Replaces volatile results with final ones instead of duplicating text",
+        "Ends the analyzer explicitly with finalizeAndFinish, finalizeAndFinishThroughEndOfInput or cancelAndFinishNow"
       ]
     },
     {
       "id": 1,
       "name": "sfspeechrecognizer-live-review",
-      "prompt": "Review this iOS live dictation plan: request only NSSpeechRecognitionUsageDescription, start AVAudioEngine immediately, use SFSpeechRecognizer forever in one recognition task, force requiresOnDeviceRecognition for every locale, and ignore availability changes. Give corrected guidance and focused Swift snippets.",
-      "expected_output": "A correction-focused SFSpeechRecognizer review that covers speech and microphone authorization, live audio setup, availability changes, on-device checks, task cleanup, and recognition duration limits.",
-      "files": [],
+      "prompt": "Review our SFSpeechRecognizer voice-notes feature for an app that still supports iOS 16. We only add NSSpeechRecognitionUsageDescription, request speech permission after starting the audio engine, set requiresOnDeviceRecognition = true everywhere, and just call task.cancel() when the user taps Stop. Notes can run for several minutes.",
+      "expected_output": "Adds NSMicrophoneUsageDescription next to the speech key, requests both speech and microphone permission before activating the session and starting the engine, adopts SFSpeechRecognizerDelegate for availability changes, checks supportsOnDeviceRecognition before forcing on-device with a fallback or message, plans for the per-task duration limit by rolling over or chunking, and cleans up completely: stop the engine, remove the tap, call endAudio() and cancel the task.",
       "assertions": [
-        "Requires both NSSpeechRecognitionUsageDescription and NSMicrophoneUsageDescription for live microphone transcription.",
-        "Requests speech recognition authorization and microphone record permission before activating the audio session or starting AVAudioEngine.",
-        "Uses SFSpeechRecognizerDelegate or availability handling for recognition service changes.",
-        "Checks supportsOnDeviceRecognition before setting requiresOnDeviceRecognition and gives a fallback when unsupported.",
-        "Accounts for SFSpeechRecognizer recognition duration/service limits instead of running one unbounded task.",
-        "Stops AVAudioEngine, removes the input tap, ends audio, and cancels or clears the recognition task during cleanup."
+        "Requires both NSSpeechRecognitionUsageDescription and NSMicrophoneUsageDescription",
+        "Requests speech and microphone permission before the session and engine start",
+        "Handles availability changes through the recognizer delegate",
+        "Checks supportsOnDeviceRecognition before setting requiresOnDeviceRecognition, with a fallback",
+        "Accounts for the task duration limit on multi-minute notes",
+        "Stops the engine, removes the tap, calls endAudio() and cancels the task"
       ]
     },
     {
       "id": 2,
       "name": "speech-boundary-routing",
-      "prompt": "A notes feature records a spoken note, transcribes it, detects the text language and sentiment, translates a summary to Spanish, plays back the recording with captions, and optionally uses Apple Intelligence to summarize it. Which parts belong in the speech-recognition skill and which should be handed to sibling skills?",
-      "expected_output": "A boundary-aware routing answer that keeps speech-to-text and microphone capture in Speech, and routes text analysis, translation, playback UI, and generative summarization to the correct sibling skills.",
-      "files": [],
+      "prompt": "Our meeting app records audio, shows a live transcript, detects which language each speaker used, rates the mood of the meeting, translates the transcript, plays recordings back with captions, and writes an AI summary. Which pieces does the speech skill own?",
+      "expected_output": "Keeps audio capture, speech and microphone permission, transcription and result handling in the speech skill. Sends language detection, mood or sentiment, embeddings and translation of the transcript to natural-language, playback with captions and AVPlayer to avkit, and Apple Intelligence or Foundation Models summaries to apple-on-device-ai. Does not blur NaturalLanguage and Speech together.",
       "assertions": [
-        "Keeps microphone capture, speech authorization, SpeechAnalyzer or SFSpeechRecognizer transcription, and transcript result handling in the speech-recognition scope.",
-        "Routes language detection, sentiment, text embeddings, and translation to the natural-language skill after transcription exists.",
-        "Routes playback UI, captions during media playback, or AVPlayer/AVKit concerns to the avkit skill.",
-        "Routes Apple Intelligence or Foundation Models summarization to the apple-on-device-ai skill.",
-        "Does not present NaturalLanguage as a speech-to-text framework or Speech as a general text-analysis framework."
+        "Keeps capture, authorization, transcription and result handling in this skill",
+        "Sends language detection, sentiment, embeddings and translation to natural-language",
+        "Sends playback, captions and AVPlayer work to avkit",
+        "Sends generated summaries to apple-on-device-ai",
+        "Keeps the NaturalLanguage and Speech frameworks distinct"
       ]
     }
   ]

package/pipeline/skills/shared/external/speech-recognition/references/speechanalyzer-patterns.md

@@ -1,171 +1,179 @@
-# SpeechAnalyzer Patterns
+# SpeechAnalyzer patterns (iOS 26+)
 
-Use this reference when implementing iOS 26+ speech-to-text with
-`SpeechAnalyzer`, `SpeechTranscriber`, `DictationTranscriber`, `SpeechDetector`,
-or `AssetInventory`.
+Companion to the `speech-recognition` skill ([../SKILL.md](../SKILL.md)).
+Every API here needs iOS 26 or later.
 
-## Contents
-
-- [Choosing Modules](#choosing-modules)
-- [Preparing Assets](#preparing-assets)
-- [Transcribing Files](#transcribing-files)
-- [Live Audio](#live-audio)
-- [Handling Results](#handling-results)
-- [Finishing Sessions](#finishing-sessions)
-- [References](#references)
-
-## Choosing Modules
-
-Prefer `SpeechTranscriber` for the newer general-purpose on-device model.
-Before building UI around it, check both device and locale support:
+## Picking modules
 
 ```swift
-guard SpeechTranscriber.isAvailable,
-      let locale = SpeechTranscriber.supportedLocale(equivalentTo: Locale.current)
-else {
-    // Disable the feature or try DictationTranscriber for compatible devices/locales.
-    return
+import Speech
+
+func makeModule(for locale: Locale) async -> (any SpeechModule)? {
+    if SpeechTranscriber.isAvailable,
+       let matched = await SpeechTranscriber.supportedLocale(equivalentTo: locale) {
+        return SpeechTranscriber(locale: matched, preset: .progressiveTranscription)
+    }
+    if let matched = await DictationTranscriber.supportedLocale(equivalentTo: locale) {
+        return DictationTranscriber(locale: matched, preset: .progressiveLongDictation)
+    }
+    return nil        // disable the feature for this device and locale
 }
 ```
 
-Use documented presets only:
+### SpeechTranscriber presets
 
-- `.transcription` for basic accurate transcription.
-- `.transcriptionWithAlternatives` for editing suggestions.
-- `.timeIndexedTranscriptionWithAlternatives` for audio-time metadata plus alternatives.
-- `.progressiveTranscription` for low-latency live UI updates.
-- `.timeIndexedProgressiveTranscription` for live UI updates with time ranges.
+| Preset | Use it for |
+|---|---|
+| `.transcription` | Plain, accurate text |
+| `.transcriptionWithAlternatives` | Editing UI that offers other candidate words |
+| `.timeIndexedTranscriptionWithAlternatives` | Alternatives plus timing for each span |
+| `.progressiveTranscription` | Low-latency live captions |
+| `.timeIndexedProgressiveTranscription` | Live captions that also carry time ranges |
 
-Use `DictationTranscriber` when `SpeechTranscriber` is unavailable and the app
-can accept dictation-model behavior. Add `SpeechDetector` only with a
-transcriber module, and only when voice activity detection is worth the risk of
-dropping speech-like audio.
+`SpeechDetector` never works alone; attach it next to a transcriber, and only
+when skipping silence saves enough to justify occasionally dropping audio
+that sounded like noise.
 
-## Preparing Assets
+## Model assets
 
-`SpeechAnalyzer` modules require model assets. The system installs and shares
-them outside the app bundle, but the app must request installation for the
-module configuration it plans to use.
+Speech models are not in your bundle. The system downloads and shares them
+between apps, and your app asks for the ones its configuration needs:
 
 ```swift
-let transcriber = SpeechTranscriber(locale: locale, preset: .transcription)
-
-if let request = try await AssetInventory.assetInstallationRequest(
-    supporting: [transcriber]
-) {
-    try await request.downloadAndInstall()
+func ensureAssets(for transcriber: SpeechTranscriber) async throws {
+    if let install = try await AssetInventory.assetInstallationRequest(supporting: [transcriber]) {
+        try await install.downloadAndInstall()
+    }
 }
 ```
 
-For language pickers, use `installedLocales`, `supportedLocales`, and
-`AssetInventory.status(forModules:)` to distinguish installed, downloadable,
-and unsupported choices. The app has a limited number of locale reservations;
-release unused reservations with `AssetInventory.release(reservedLocale:)`.
+For a language picker, combine `await SpeechTranscriber.installedLocales`,
+`await SpeechTranscriber.supportedLocales` and
+`AssetInventory.status(forModules:)`, which reports `.installed`,
+`.downloading`, `.supported` (can be downloaded) or `.unsupported`.
 
-## Transcribing Files
+An app can reserve only a limited number of locales. Give one back with
+`AssetInventory.release(reservedLocale:)` when the user drops a language.
 
-For files, let the analyzer convert the file to a compatible format and finish
-the session after the file is consumed.
+## Transcribing a file
 
-```swift
-func transcribeFile(at url: URL, locale: Locale) async throws -> AttributedString {
-    guard let supportedLocale = SpeechTranscriber.supportedLocale(equivalentTo: locale) else {
-        throw SpeechError.unsupportedLocale
-    }
+The analyzer converts the file's audio format on its own.
 
-    let transcriber = SpeechTranscriber(
-        locale: supportedLocale,
-        preset: .transcription
-    )
+```swift
+import AVFoundation
+import Speech
 
-    if let request = try await AssetInventory.assetInstallationRequest(
-        supporting: [transcriber]
-    ) {
-        try await request.downloadAndInstall()
+func transcript(of recording: URL, locale: Locale) async throws -> AttributedString {
+    guard let matched = await SpeechTranscriber.supportedLocale(equivalentTo: locale) else {
+        throw CocoaError(.featureUnsupported)
     }
+    let transcriber = SpeechTranscriber(locale: matched, preset: .transcription)
+    try await ensureAssets(for: transcriber)
 
     let analyzer = SpeechAnalyzer(modules: [transcriber])
-    async let transcript = transcriber.results.reduce(into: AttributedString()) {
-        text, result in
-        text.append(result.text)
+    async let collected = transcriber.results.reduce(into: AttributedString()) { text, piece in
+        text.append(piece.text)
     }
 
-    let file = try AVAudioFile(forReading: url)
-    let lastSampleTime = try await analyzer.analyzeSequence(from: file)
-    if let lastSampleTime {
-        try await analyzer.finalizeAndFinish(through: lastSampleTime)
+    let audio = try AVAudioFile(forReading: recording)
+    if let lastSample = try await analyzer.analyzeSequence(from: audio) {
+        try await analyzer.finalizeAndFinish(through: lastSample)
     } else {
-        try analyzer.cancelAndFinishNow()
+        await analyzer.cancelAndFinishNow()
     }
-
-    return try await transcript
+    return try await collected
 }
 ```
 
-## Live Audio
-
-For live audio, create an `AsyncStream<AnalyzerInput>`, convert microphone
-buffers to the analyzer-compatible format, yield them, and consume results in a
-separate task.
+## Live audio
 
 ```swift
-let transcriber = SpeechTranscriber(
-    locale: locale,
-    preset: .timeIndexedProgressiveTranscription
-)
-let analyzer = SpeechAnalyzer(modules: [transcriber])
-let audioFormat = await SpeechAnalyzer.bestAvailableAudioFormat(
-    compatibleWith: [transcriber]
-)
-let (inputSequence, inputBuilder) = AsyncStream.makeStream(of: AnalyzerInput.self)
-
-// In the audio-engine tap, convert each AVAudioPCMBuffer to audioFormat first.
-inputBuilder.yield(AnalyzerInput(buffer: convertedBuffer))
-```
-
-Use `AVAudioConverter` or an existing project audio pipeline for the conversion.
-Do not feed arbitrary input-node formats directly unless they already match a
-compatible analyzer format.
-
-## Handling Results
+import AVFoundation
+import Speech
+
+@MainActor
+final class LiveTranscript {
+    private(set) var settled = AttributedString()
+    private(set) var pending = AttributedString()
+    private var analyzer: SpeechAnalyzer?
+    private var feed: AsyncStream<AnalyzerInput>.Continuation?
+    private var reader: Task<Void, Never>?
+
+    func start(locale: Locale) async throws -> AVAudioFormat? {
+        let transcriber = SpeechTranscriber(locale: locale, preset: .timeIndexedProgressiveTranscription)
+        try await ensureAssets(for: transcriber)
+        let analyzer = SpeechAnalyzer(modules: [transcriber])
+        let format = await SpeechAnalyzer.bestAvailableAudioFormat(compatibleWith: [transcriber])
+
+        let (inputs, feed) = AsyncStream.makeStream(of: AnalyzerInput.self)
+        self.feed = feed
+        self.analyzer = analyzer
+
+        reader = Task { [weak self] in
+            do {
+                for try await piece in transcriber.results {
+                    self?.absorb(piece)
+                }
+            } catch {
+                // surface the error in the UI
+            }
+        }
+        try await analyzer.start(inputSequence: inputs)
+        return format        // convert microphone buffers to this before yielding
+    }
 
-`SpeechTranscriber.Result.text` is an `AttributedString`. Time-indexed presets
-include audio time range attributes that can drive playback highlighting.
+    func push(_ converted: AVAudioPCMBuffer) {
+        feed?.yield(AnalyzerInput(buffer: converted))
+    }
 
-When using progressive presets, volatile results may be replaced by later final
-results. Keep volatile display state separate so the UI does not duplicate text.
+    func stop() async throws {
+        feed?.finish()
+        try await analyzer?.finalizeAndFinishThroughEndOfInput()
+        await reader?.value
+    }
 
-```swift
-for try await result in transcriber.results {
-    if result.isFinal {
-        volatileTranscript = AttributedString()
-        finalizedTranscript.append(result.text)
-    } else {
-        volatileTranscript = result.text
+    private func absorb(_ piece: SpeechTranscriber.Result) {
+        if piece.isFinal {
+            settled.append(piece.text)
+            pending = AttributedString()
+        } else {
+            pending = piece.text
+        }
     }
 }
 ```
 
-## Finishing Sessions
+Convert tap buffers with `AVAudioConverter` (or an audio pipeline you
+already have) into the format returned by `bestAvailableAudioFormat`. Do not
+feed raw input-node buffers unless their format already matches.
+
+## Reading results
 
-The analyzer can only analyze one input sequence at a time. Ending your stream
-does not finish the analyzer session; call a finish or cancel method.
+`SpeechTranscriber.Result.text` is an `AttributedString`. With a
+time-indexed preset its runs carry audio time-range attributes, which is how
+playback can highlight the word being spoken. Progressive presets deliver
+volatile results that a later final result replaces, so keep the volatile
+text in its own state, as `pending` does above.
 
-Use:
+## Ending a session
 
-- `finalizeAndFinish(through:)` after `analyzeSequence(_:)` returns a final sample time.
-- `finalizeAndFinishThroughEndOfInput()` after autonomous `start(inputSequence:)`.
-- `cancelAndFinishNow()` for immediate cancellation.
+- An analyzer processes one input sequence at a time, and finishing your
+  input stream does not end the session.
+- After `analyzeSequence(_:)` returns the last sample time, call
+  `finalizeAndFinish(through:)`.
+- After starting with `start(inputSequence:)`, call
+  `finalizeAndFinishThroughEndOfInput()` once input is complete.
+- `cancelAndFinishNow()` stops immediately and discards pending work.
 
-After the session finishes, result streams terminate and most analyzer methods
-no longer accept new work. Create a new analyzer for a new finished session.
+Once finished, the module result streams end and the analyzer refuses most
+further calls. Create a new analyzer for the next session.
 
-## References
+## Further reading
 
-- [SpeechAnalyzer](https://sosumi.ai/documentation/speech/speechanalyzer)
-- [SpeechTranscriber](https://sosumi.ai/documentation/speech/speechtranscriber)
-- [SpeechTranscriber.Preset](https://sosumi.ai/documentation/speech/speechtranscriber/preset)
-- [DictationTranscriber](https://sosumi.ai/documentation/speech/dictationtranscriber)
-- [SpeechDetector](https://sosumi.ai/documentation/speech/speechdetector)
-- [AssetInventory](https://sosumi.ai/documentation/speech/assetinventory)
-- [WWDC25: Bring advanced speech-to-text to your app with SpeechAnalyzer](https://sosumi.ai/videos/play/wwdc2025/277)
+- https://developer.apple.com/documentation/speech/speechanalyzer
+- https://developer.apple.com/documentation/speech/speechtranscriber
+- https://developer.apple.com/documentation/speech/speechtranscriber/preset
+- https://developer.apple.com/documentation/speech/dictationtranscriber
+- https://developer.apple.com/documentation/speech/speechdetector
+- https://developer.apple.com/documentation/speech/assetinventory
+- WWDC25 session 277 on SpeechAnalyzer