@mmerterden/multi-agent-pipeline 20.7.0 → 20.8.0

package/pipeline/skills/shared/external/natural-language/SKILL.md

@@ -1,444 +1,292 @@
 ---
 name: natural-language
-description: "Tokenize, tag, and analyze natural language text using Apple's NaturalLanguage framework and translate between languages with the Translation framework. Use when adding language identification, sentiment analysis, named entity recognition, part-of-speech tagging, text embeddings, or in-app translation to iOS/macOS/visionOS apps."
+description: "NaturalLanguage and Translation frameworks: tokenizing, language identification, part-of-speech tagging, named entities, sentiment, word and sentence embeddings, custom NLModel classifiers, translation sheet, TranslationSession, batch translation. Use when analyzing or translating text in-app on iOS, macOS or visionOS."
 metadata:
-  source: "dpearson2699/swift-ios-skills (PolyForm Perimeter 1.0.0)"
+  source: multi-agent-pipeline
 ---
 
-# NaturalLanguage + Translation
-
-Analyze natural language text for tokenization, part-of-speech tagging, named
-entity recognition, sentiment analysis, language identification, and word/sentence
-embeddings. Translate text between languages with the Translation framework.
-Targets Swift 6.3 / iOS 26+.
-
-> This skill covers two related frameworks: **NaturalLanguage** (`NLTokenizer`, `NLTagger`, `NLEmbedding`) for on-device text analysis, and **Translation** (`TranslationSession`, `LanguageAvailability`) for language translation.
-
-**Scope boundary:** Use this skill after you already have text. It owns
-tokenization, language identification, POS/NER tagging, sentiment, embeddings,
-custom `NLModel` classifiers/taggers, and in-app translation. Hand off OCR to
-`vision-framework`, speech-to-text to `speech-recognition`, UI strings and
-locale formatting to `ios-localization`, and generative summarization or Apple
-Intelligence workflows to `apple-on-device-ai`.
-
-## Contents
-
-- [Setup](#setup)
-- [Tokenization](#tokenization)
-- [Language Identification](#language-identification)
-- [Part-of-Speech Tagging](#part-of-speech-tagging)
-- [Named Entity Recognition](#named-entity-recognition)
-- [Sentiment Analysis](#sentiment-analysis)
-- [Text Embeddings](#text-embeddings)
-- [Translation](#translation)
-- [Common Mistakes](#common-mistakes)
-- [Review Checklist](#review-checklist)
-- [References](#references)
-
-## Setup
-
-Import `NaturalLanguage` for text analysis and `Translation` for language
-translation. No special entitlements or capabilities are required for
-NaturalLanguage. Translation has split availability: system translation
-presentation is iOS 17.4+ / macOS 14.4+, while `TranslationSession`,
-`.translationTask()`, `LanguageAvailability`, and batch translation require
-iOS 18+ / macOS 15+.
-Direct `TranslationSession(installedSource:target:)` is the non-UI option, but
-only when the source and target languages are already installed on device.
+# NaturalLanguage and Translation
 
-```swift
-import NaturalLanguage
-import Translation
-```
+Two frameworks, one job each. NaturalLanguage analyses text on the device:
+tokens, languages, word classes, entities, sentiment, embeddings and custom
+Create ML models. Translation turns text from one language into another,
+either through a system sheet or through a session you drive. Code assumes
+Swift 6.3 and an iOS 26 deployment target.
+
+## Scope
 
-NaturalLanguage classes (`NLTokenizer`, `NLTagger`) are **not thread-safe**.
-Use each instance from one thread or dispatch queue at a time.
+This skill starts once you already have text. Hand off elsewhere for:
 
-## Tokenization
+| Need | Skill |
+|---|---|
+| Reading text out of images (OCR) | `vision-framework` |
+| Turning speech into text | `speech-recognition` |
+| UI strings, String Catalogs, plurals, right-to-left layout, locale formatting | `ios-localization` |
+| Generative summaries and other Apple Intelligence features | `apple-on-device-ai` |
 
-Segment text into words, sentences, or paragraphs with `NLTokenizer`.
+## Setup and availability
 
 ```swift
 import NaturalLanguage
-
-func tokenizeWords(in text: String) -> [String] {
-    let tokenizer = NLTokenizer(unit: .word)
-    tokenizer.string = text
-
-    let range = text.startIndex..<text.endIndex
-    return tokenizer.tokens(for: range).map { String(text[$0]) }
-}
+import Translation
 ```
 
-### Token Units
+NaturalLanguage needs no capability or entitlement. For Translation:
 
-| Unit | Description |
+| API | Minimum |
 |---|---|
-| `.word` | Individual words |
-| `.sentence` | Sentences |
-| `.paragraph` | Paragraphs |
-| `.document` | Entire document |
+| `.translationPresentation(isPresented:text:)` system sheet | iOS 17.4, macOS 14.4 |
+| `TranslationSession`, `.translationTask`, `LanguageAvailability`, batch requests | iOS 18, macOS 15 |
+| `TranslationSession(installedSource:target:)` without any UI | iOS 26, macOS 26 |
+| `TranslationSession.Strategy` selection | iOS 26.4, macOS 26.4 |
 
-### Enumerating with Attributes
+The Translation APIs are marked unavailable on visionOS, tvOS and watchOS
+in the 26 SDKs; NaturalLanguage itself works on every Apple platform.
+Use the UI-free `installedSource` initializer only when both languages are
+already downloaded.
 
-Use `enumerateTokens(in:using:)` to detect numeric or emoji tokens.
+`NLTokenizer` and `NLTagger` are not thread-safe. Keep each instance on a
+single thread or queue at a time.
 
-```swift
-let tokenizer = NLTokenizer(unit: .word)
-tokenizer.string = text
+## Tokenizing
 
-tokenizer.enumerateTokens(in: text.startIndex..<text.endIndex) { range, attributes in
-    if attributes.contains(.numeric) {
-        print("Number: \(text[range])")
-    }
-    return true // continue enumeration
-}
+```swift
+let splitter = NLTokenizer(unit: .sentence)
+splitter.string = review
+let sentences = splitter.tokens(for: review.startIndex..<review.endIndex).map { review[$0] }
 ```
 
-## Language Identification
-
-Detect the dominant language of a string with `NLLanguageRecognizer`.
+Units are `.word`, `.sentence`, `.paragraph` and `.document`. Enumeration
+also reports token attributes such as numbers and emoji:
 
 ```swift
-func detectLanguage(for text: String) -> NLLanguage? {
-    NLLanguageRecognizer.dominantLanguage(for: text)
-}
-
-// Multiple hypotheses with confidence scores
-func languageHypotheses(for text: String, max: Int = 5) -> [NLLanguage: Double] {
-    let recognizer = NLLanguageRecognizer()
-    recognizer.processString(text)
-    return recognizer.languageHypotheses(withMaximum: max)
+let words = NLTokenizer(unit: .word)
+words.string = receipt
+words.enumerateTokens(in: receipt.startIndex..<receipt.endIndex) { span, traits in
+    if traits.contains(.numeric) { amounts.append(String(receipt[span])) }
+    return true                    // keep going
 }
 ```
 
-Constrain the recognizer to expected languages for better accuracy on short text.
+## Identifying the language
 
 ```swift
+let guess = NLLanguageRecognizer.dominantLanguage(for: "Hvor er togstationen?")
+
 let recognizer = NLLanguageRecognizer()
-recognizer.languageConstraints = [.english, .french, .spanish]
-recognizer.processString(text)
-let detected = recognizer.dominantLanguage
+recognizer.processString(snippet)
+let ranked = recognizer.languageHypotheses(withMaximum: 3)   // [NLLanguage: Double]
+
+recognizer.reset()
+recognizer.languageConstraints = [.danish, .norwegian, .swedish]
+recognizer.processString(snippet)
+let bestNordic = recognizer.dominantLanguage
 ```
 
-## Part-of-Speech Tagging
+Constraining the candidate set helps a lot with short input.
 
-Identify nouns, verbs, adjectives, and other lexical classes with `NLTagger`.
+## Part-of-speech tagging
 
 ```swift
-func tagPartsOfSpeech(in text: String) -> [(String, NLTag)] {
-    let tagger = NLTagger(tagSchemes: [.lexicalClass])
-    tagger.string = text
-
-    var results: [(String, NLTag)] = []
-    let range = text.startIndex..<text.endIndex
-    let options: NLTagger.Options = [.omitPunctuation, .omitWhitespace]
-
-    tagger.enumerateTags(in: range, unit: .word, scheme: .lexicalClass, options: options) { tag, tokenRange in
-        if let tag {
-            results.append((String(text[tokenRange]), tag))
-        }
-        return true
-    }
-    return results
+let tagger = NLTagger(tagSchemes: [.lexicalClass])
+tagger.string = sentence
+tagger.enumerateTags(in: sentence.startIndex..<sentence.endIndex,
+                     unit: .word,
+                     scheme: .lexicalClass,
+                     options: [.omitPunctuation, .omitWhitespace]) { tag, span in
+    if tag == .verb { verbs.append(String(sentence[span])) }
+    return true
 }
 ```
 
-### Common Tag Schemes
-
-| Scheme | Output |
+| Scheme | Produces |
 |---|---|
-| `.lexicalClass` | Part of speech (noun, verb, adjective) |
-| `.nameType` | Named entity type (person, place, organization) |
-| `.nameTypeOrLexicalClass` | Combined NER + POS |
-| `.lemma` | Base form of a word |
-| `.language` | Per-token language |
-| `.sentimentScore` | Sentiment polarity score |
-
-## Named Entity Recognition
+| `.lexicalClass` | Word class: noun, verb, adjective and so on |
+| `.nameType` | Entity type |
+| `.nameTypeOrLexicalClass` | Entity type where there is one, word class otherwise |
+| `.lemma` | Dictionary form of the word |
+| `.language` | Language of each token |
+| `.sentimentScore` | Polarity score |
 
-Extract people, places, and organizations.
+## Named entities
 
 ```swift
-func extractEntities(from text: String) -> [(String, NLTag)] {
-    let tagger = NLTagger(tagSchemes: [.nameType])
-    tagger.string = text
-
-    var entities: [(String, NLTag)] = []
-    let options: NLTagger.Options = [.omitPunctuation, .omitWhitespace, .joinNames]
-
-    tagger.enumerateTags(
-        in: text.startIndex..<text.endIndex,
-        unit: .word,
-        scheme: .nameType,
-        options: options
-    ) { tag, tokenRange in
-        if let tag, tag != .other {
-            entities.append((String(text[tokenRange]), tag))
-        }
-        return true
+let finder = NLTagger(tagSchemes: [.nameType])
+finder.string = article
+finder.enumerateTags(in: article.startIndex..<article.endIndex,
+                     unit: .word,
+                     scheme: .nameType,
+                     options: [.omitPunctuation, .omitWhitespace, .joinNames]) { tag, span in
+    guard let tag, tag != .other else { return true }
+    switch tag {
+    case .personalName: people.insert(String(article[span]))
+    case .placeName: places.insert(String(article[span]))
+    case .organizationName: companies.insert(String(article[span]))
+    default: break
     }
-    return entities
+    return true
 }
-// NLTag values: .personalName, .placeName, .organizationName
 ```
 
-## Sentiment Analysis
+`.joinNames` keeps "Ada Lovelace" together as one entity.
+
+## Sentiment
 
-Score text sentiment from -1.0 (negative) to +1.0 (positive).
+Scores run from -1.0 (negative) to +1.0 (positive). Some languages have no
+model, so the result is optional.
 
 ```swift
-func sentimentScore(for text: String) -> Double? {
-    let tagger = NLTagger(tagSchemes: [.sentimentScore])
-    tagger.string = text
-
-    let (tag, _) = tagger.tag(
-        at: text.startIndex,
-        unit: .paragraph,
-        scheme: .sentimentScore
-    )
+func mood(of paragraph: String) -> Double? {
+    let scorer = NLTagger(tagSchemes: [.sentimentScore])
+    scorer.string = paragraph
+    let (tag, _) = scorer.tag(at: paragraph.startIndex, unit: .paragraph, scheme: .sentimentScore)
     return tag.flatMap { Double($0.rawValue) }
 }
 ```
 
-## Text Embeddings
-
-Measure semantic similarity between words or sentences with `NLEmbedding`.
+## Embeddings
 
 ```swift
-func wordSimilarity(_ word1: String, _ word2: String) -> Double? {
-    guard let embedding = NLEmbedding.wordEmbedding(for: .english) else { return nil }
-    return embedding.distance(between: word1, and: word2, distanceType: .cosine)
+func wordNeighbors() -> (gap: NLDistance, similar: [(String, NLDistance)])? {
+    guard let vectors = NLEmbedding.wordEmbedding(for: .english) else { return nil }
+    let gap = vectors.distance(between: "harbor", and: "port", distanceType: .cosine)
+    let similar = vectors.neighbors(for: "harbor", maximumCount: 5, distanceType: .cosine)
+    return (gap, similar)
 }
 
-func findSimilarWords(to word: String, count: Int = 5) -> [(String, Double)] {
-    guard let embedding = NLEmbedding.wordEmbedding(for: .english) else { return [] }
-    return embedding.neighbors(for: word, maximumCount: count, distanceType: .cosine)
+func sentenceGap() -> NLDistance? {
+    NLEmbedding.sentenceEmbedding(for: .english)?.distance(between: "The ferry is late.",
+                                                           and: "Our boat has been delayed.",
+                                                           distanceType: .cosine)
 }
 ```
 
-Sentence embeddings compare entire sentences.
-
-```swift
-func sentenceSimilarity(_ s1: String, _ s2: String) -> Double? {
-    guard let embedding = NLEmbedding.sentenceEmbedding(for: .english) else { return nil }
-    return embedding.distance(between: s1, and: s2, distanceType: .cosine)
-}
-```
+Not every language ships embeddings; both factories can return `nil`.
+`NLDistance` is a `Double`; smaller means closer.
 
 ## Translation
 
-### System Translation Overlay
-
-Show the built-in translation UI with `.translationPresentation()`.
+### The system sheet
 
 ```swift
-import SwiftUI
-import Translation
-
-struct TranslatableView: View {
-    @State private var showTranslation = false
-    let text = "Hello, how are you?"
+struct PhraseCard: View {
+    let phrase: String
+    @State private var showsTranslation = false
 
     var body: some View {
-        Button { showTranslation = true } label: {
-            Text(text)
-        }
-        .buttonStyle(.plain)
-        .translationPresentation(
-            isPresented: $showTranslation,
-            text: text
-        )
+        Button(phrase) { showsTranslation = true }
+            .buttonStyle(.plain)
+            .translationPresentation(isPresented: $showsTranslation, text: phrase)
     }
 }
 ```
 
-### Programmatic Translation
+### Translating in code
 
-Use `.translationTask()` for programmatic translations within a view context.
+`.translationTask` hands you a session whenever its configuration becomes
+non-nil or changes, so assigning a configuration is what starts the work.
+Written inside `body`, the action closure is main-actor isolated, and
+`TranslationSession` is not `Sendable`: Swift 6 rejects calling
+`translate(_:)` from there. Mark the closure `@concurrent` and hop back to the
+main actor to store the result.
 
 ```swift
-struct TranslatingView: View {
-    @State private var translatedText = ""
-    @State private var translationErrorMessage: String?
-    @State private var configuration: TranslationSession.Configuration?
+struct MenuTranslator: View {
+    let dish: String
+    @State private var request: TranslationSession.Configuration?
+    @State private var output = ""
 
     var body: some View {
         VStack {
-            Text(translatedText)
-            Button("Translate") {
-                configuration = .init(source: Locale.Language(identifier: "en"),
-                                      target: Locale.Language(identifier: "es"))
+            Text(output)
+            Button("Translate to Japanese") {
+                request = .init(source: Locale.Language(identifier: "it"),
+                                target: Locale.Language(identifier: "ja"))
             }
         }
-        .translationTask(configuration) { session in
+        .translationTask(request) { @concurrent session in
             do {
-                let response = try await session.translate("Hello, world!")
-                await MainActor.run {
-                    translatedText = response.targetText
-                    translationErrorMessage = nil
-                }
+                let reply = try await session.translate(dish)
+                await MainActor.run { output = reply.targetText }
             } catch {
-                let message = error.localizedDescription
-                await MainActor.run {
-                    translationErrorMessage = message
-                }
+                await MainActor.run { output = "Translation failed: \(error.localizedDescription)" }
             }
         }
     }
 }
 ```
 
-### Batch Translation
+### Several strings at once
 
-Translate multiple strings in a single session.
+Give each request a `clientIdentifier` so you can match answers to inputs.
 
 ```swift
-.translationTask(configuration) { session in
-    do {
-        let requests = texts.enumerated().map { index, text in
-            TranslationSession.Request(sourceText: text,
-                                       clientIdentifier: "\(index)")
-        }
-        let responses = try await session.translations(from: requests)
-        for response in responses {
-            print("\(response.sourceText) -> \(response.targetText)")
-        }
-    } catch {
-        // Handle cancellation, unsupported languages, or download refusal.
-    }
+let requests = menuItems.map {
+    TranslationSession.Request(sourceText: $0.name, clientIdentifier: $0.id)
 }
-```
-
-### Checking Language Availability
-
-```swift
-let availability = LanguageAvailability()
-let status = await availability.status(
-    from: Locale.Language(identifier: "en"),
-    to: Locale.Language(identifier: "ja")
-)
-switch status {
-case .installed: break    // Ready to translate offline
-case .supported: break    // Needs download
-case .unsupported: break  // Language pair not available
-}
-```
-
-## Common Mistakes
-
-### DON'T: Share NLTagger/NLTokenizer across threads
-
-These classes are not thread-safe and will produce incorrect results or crash.
-
-```swift
-// WRONG
-let sharedTagger = NLTagger(tagSchemes: [.lexicalClass])
-DispatchQueue.concurrentPerform(iterations: 10) { _ in
-    sharedTagger.string = someText  // Data race
-}
-
-// CORRECT
-await withTaskGroup(of: Void.self) { group in
-    for _ in 0..<10 {
-        group.addTask {
-            let tagger = NLTagger(tagSchemes: [.lexicalClass])
-            tagger.string = someText
-            // process...
-        }
+do {
+    let answers = try await session.translations(from: requests)
+    for answer in answers {
+        translated[answer.clientIdentifier ?? ""] = answer.targetText
     }
+} catch {
+    // cancelled, unsupported pair, or the user declined the download
 }
 ```
 
-### DON'T: Confuse NaturalLanguage with Core ML
-
-NaturalLanguage provides built-in linguistic analysis. Use Core ML for custom
-trained models. They complement each other via `NLModel`.
-
-```swift
-// WRONG: Trying to do NER with raw Core ML
-let coreMLModel = try MLModel(contentsOf: modelURL)
-
-// CORRECT: Use NLTagger for built-in NER
-let tagger = NLTagger(tagSchemes: [.nameType])
-
-// Or load a custom Core ML model via NLModel
-let nlModel = try NLModel(mlModel: coreMLModel)
-tagger.setModels([nlModel], forTagScheme: .nameType)
-```
-
-### DON'T: Assume embeddings exist for all languages
-
-Not all languages have word or sentence embeddings available on device.
-
-```swift
-// WRONG: Force unwrap
-let embedding = NLEmbedding.wordEmbedding(for: .japanese)!
-
-// CORRECT: Handle nil
-guard let embedding = NLEmbedding.wordEmbedding(for: .japanese) else {
-    // Embedding not available for this language
-    return
-}
-```
-
-### DON'T: Create a new tagger per token
-
-Creating and configuring a tagger is expensive. Reuse it for the same text.
+### Checking a language pair first
 
 ```swift
-// WRONG: New tagger per word
-for word in words {
-    let tagger = NLTagger(tagSchemes: [.lexicalClass])
-    tagger.string = word
-}
-
-// CORRECT: Set string once, enumerate
-let tagger = NLTagger(tagSchemes: [.lexicalClass])
-tagger.string = fullText
-tagger.enumerateTags(in: fullText.startIndex..<fullText.endIndex,
-                     unit: .word, scheme: .lexicalClass, options: []) { tag, range in
-    return true
+let pairState = await LanguageAvailability().status(
+    from: Locale.Language(identifier: "pt"),
+    to: Locale.Language(identifier: "de")
+)
+switch pairState {
+case .installed: break        // ready offline
+case .supported: break        // works after a download
+case .unsupported: break      // cannot translate this pair
+@unknown default: break
 }
 ```
 
-### DON'T: Ignore language hints for short text
-
-Language detection on short strings (under ~20 characters) is unreliable.
-Set constraints or hints to improve accuracy.
-
-```swift
-// WRONG: Detect language of a single word
-let lang = NLLanguageRecognizer.dominantLanguage(for: "chat")  // French or English?
-
-// CORRECT: Provide context
-let recognizer = NLLanguageRecognizer()
-recognizer.languageHints = [.english: 0.8, .french: 0.2]
-recognizer.processString("chat")
-```
-
-## Review Checklist
-
-- [ ] `NLTokenizer` and `NLTagger` instances used from a single thread
+## Common mistakes
+
+- **Sharing a tagger between threads.** Driving one `NLTagger` from
+  `concurrentPerform` is a data race. Give each task its own instance, for
+  example one per child task in a task group.
+- **Confusing NaturalLanguage with Core ML.** NaturalLanguage covers the
+  built-in analyses. A custom model trained in Create ML comes in through
+  `NLModel(mlModel:)` and `tagger.setModels(_:forTagScheme:)`, not as a raw
+  `MLModel` wired up by hand.
+- **Force-unwrapping `wordEmbedding(for:)`.** Many languages have none.
+- **A tagger per token.** Create the tagger once, set the whole string, then
+  enumerate.
+- **Trusting detection on tiny strings.** Under about 20 characters it is
+  guesswork; add `languageHints` such as `[.english: 0.8, .french: 0.2]` or
+  `languageConstraints`.
+
+## Review checklist
+
+- [ ] Each tokenizer and tagger used from one thread at a time
 - [ ] Tagger created once per text, not per token
-- [ ] Language detection uses constraints/hints for short text
-- [ ] `NLEmbedding` availability checked before use (returns nil if unavailable)
-- [ ] Translation `LanguageAvailability` checked before attempting translation
-- [ ] `.translationTask()` used within a SwiftUI view hierarchy
-- [ ] Batch translation uses `clientIdentifier` to match responses to requests
-- [ ] Sentiment scores handled as optional (may return nil for unsupported languages)
-- [ ] `.joinNames` option used with NER to keep multi-word names together
-- [ ] Custom ML models loaded via `NLModel`, not raw Core ML
+- [ ] Short text gets constraints or hints
+- [ ] `NLEmbedding` results checked for `nil`
+- [ ] `LanguageAvailability` consulted before translating
+- [ ] `.translationTask` attached inside the SwiftUI view hierarchy
+- [ ] Batch requests carry a `clientIdentifier`
+- [ ] Sentiment handled as optional (unsupported languages give `nil`)
+- [ ] Entity extraction uses `.joinNames`
+- [ ] Custom models loaded through `NLModel`
 
 ## References
 
-- Extended patterns (custom models, contextual embeddings, gazetteers): [references/translation-patterns.md](references/translation-patterns.md)
-- [Natural Language framework](https://sosumi.ai/documentation/naturallanguage)
-- [NLTokenizer](https://sosumi.ai/documentation/naturallanguage/nltokenizer)
-- [NLTagger](https://sosumi.ai/documentation/naturallanguage/nltagger)
-- [NLEmbedding](https://sosumi.ai/documentation/naturallanguage/nlembedding)
-- [NLLanguageRecognizer](https://sosumi.ai/documentation/naturallanguage/nllanguagerecognizer)
-- [Translation framework](https://sosumi.ai/documentation/translation)
-- [TranslationSession](https://sosumi.ai/documentation/translation/translationsession)
-- [TranslationSession.Strategy](https://sosumi.ai/documentation/translation/translationsession/strategy)
-- [LanguageAvailability](https://sosumi.ai/documentation/translation/languageavailability)
+- [translation and analysis patterns](references/translation-patterns.md): custom models, contextual embeddings, gazetteers, multi-scheme tagging, in-place translation, strategies, session preparation, SwiftUI analyzer, tagger assets, lemmas
+- https://developer.apple.com/documentation/naturallanguage
+- https://developer.apple.com/documentation/naturallanguage/nltokenizer
+- https://developer.apple.com/documentation/naturallanguage/nltagger
+- https://developer.apple.com/documentation/naturallanguage/nlembedding
+- https://developer.apple.com/documentation/naturallanguage/nllanguagerecognizer
+- https://developer.apple.com/documentation/translation
+- https://developer.apple.com/documentation/translation/translationsession
+- https://developer.apple.com/documentation/translation/translationsession/strategy
+- https://developer.apple.com/documentation/translation/languageavailability