@mmerterden/multi-agent-pipeline 20.7.0 → 20.8.0

package/pipeline/skills/shared/external/app-store-optimization/evals/evals.json

@@ -3,43 +3,47 @@
   "evals": [
     {
       "id": 1,
-      "prompt": "I have a new habit tracker launching on the App Store. Draft an ASO plan covering app name/subtitle keyword placement, keyword field priorities, the description hook, first three screenshots, promotional text, custom product pages for ads, in-app events, product page optimization tests, localization, and review prompt timing.",
-      "expected_output": "A complete App Store Optimization plan that separates search visibility from conversion, uses Apple-backed metadata surfaces correctly, and gives practical copy/testing guidance without drifting into App Review compliance or StoreKit implementation depth.",
+      "name": "language-app-full-plan",
+      "prompt": "We are launching a language-learning flashcard app in English, German and Japanese. Build us an ASO plan: what goes in the name, subtitle and keyword field, how the product page should convert, which Custom Product Pages and events make sense, how we should test the page, and when the app should ask for a rating.",
+      "expected_output": "Organizes the plan into visibility and conversion. Picks relevant, high-intent keywords over popular unrelated ones and lists name, subtitle, keyword field, primary category, ratings, Custom Product Page keywords and event cards as visibility levers without stuffing the description. For conversion covers the opening description lines, benefit-led first screenshots, current promotional text and research per locale. Maps Custom Product Pages to audiences or channels and mentions the 70-page capacity or assigned keywords. Proposes a test with one hypothesis, at most three treatments and chosen localizations. Places review prompts after positive moments and notes StoreKit may not show them.",
       "files": [],
       "expectations": [
-        "Separates search visibility work from conversion work and prioritizes relevance over high-volume but weakly related keywords.",
-        "Uses app name, subtitle, keyword field, primary category, ratings/reviews, Custom Product Page assigned keywords, and in-app event cards as the main search/visibility surfaces rather than keyword-stuffing the description.",
-        "Recommends a conversion-focused description hook, benefit-oriented first screenshots, current promotional text, and per-locale metadata research.",
-        "Plans Custom Product Pages around audience or acquisition-channel differences and mentions the current 70-page capacity or assigned keyword search visibility.",
-        "Plans Product Page Optimization around one hypothesis at a time while acknowledging up to three treatments and selected localizations.",
-        "Places in-app review prompts only after positive user moments and notes that StoreKit may not show every requested prompt."
+        "Splits the plan into a visibility part and a conversion part",
+        "Puts relevance and intent ahead of raw search volume",
+        "Names name, subtitle, keyword field, category, ratings, Custom Product Page keywords and event cards as visibility levers, and keeps keywords out of the description",
+        "Covers the description opening, benefit-first screenshots, up-to-date promotional text and keyword research per locale",
+        "Ties Custom Product Pages to audiences or channels and mentions the 70-page limit or assigned search keywords",
+        "Designs a test with one hypothesis, up to three treatments and selected localizations",
+        "Asks for reviews after positive moments and says StoreKit decides whether the prompt appears"
       ]
     },
     {
       "id": 2,
-      "prompt": "Review this ASO note for accuracy: 'Custom Product Pages are capped at 35 and only work through direct links. Product Page Optimization compares one treatment in the default locale only. Put repeated keywords at the start of the keyword field because position is a ranking factor. Add more keywords to the description. In-app event metadata is indexed like the keyword field. Japan and Korea also index English keyword fields. Ratings below 4.0 always tank conversion.' What should change?",
-      "expected_output": "A correction-focused review that replaces stale or unsupported App Store Optimization claims with current Apple-source-grounded guidance.",
+      "name": "correct-aso-notes",
+      "prompt": "Fix this internal ASO note: 'We can make 35 Custom Product Pages and they only work through links. Product page tests allow one treatment for all locales and give no confidence data. Words at the start of the keyword field rank higher. Add keywords to the description too. Event names are a hidden keyword field. Japan and Korea index the English keyword field, and a rating under 4.0 always kills conversion.'",
+      "expected_output": "Raises the Custom Product Page limit to 70 and adds that assigned keywords let pages appear in search as well as via links. Says a test can have up to three treatments, localizations are selectable and App Analytics reports confidence. Rejects the idea that keyword position inside the field changes ranking, removes keywords from the description, and describes event names and short descriptions as visible card copy. Drops the unsupported claims about cross-locale indexing and a hard 4.0 threshold, while keeping research per market and treating ratings as a real signal.",
       "files": [],
       "expectations": [
-        "Corrects Custom Product Pages to the current 70-page limit and mentions assigned-keyword search visibility in addition to direct URLs.",
-        "Corrects Product Page Optimization to up to three treatments, selectable localizations, and App Analytics confidence/reporting behavior instead of one default-locale treatment.",
-        "Rejects unsupported keyword-position ranking claims and recommends spending keyword-field characters on relevant, non-duplicated terms.",
-        "Keeps the description conversion-focused and avoids adding unnecessary keywords for search ranking.",
-        "Frames in-app event name and short description as visible event-card/search-result copy rather than a hidden keyword field.",
-        "Removes unsupported cross-locale English-indexing and fixed rating-threshold claims while preserving per-market research and ratings/reviews as search and conversion signals."
+        "Corrects the limit to 70 Custom Product Pages and adds assigned-keyword search visibility",
+        "States up to three treatments, selectable localizations and confidence reporting in App Analytics",
+        "Rejects the claim that position within the keyword field affects ranking",
+        "Advises against adding keywords to the description",
+        "Describes event metadata as visible card copy rather than a keyword field",
+        "Removes the unsupported cross-locale indexing and fixed-threshold rating claims while keeping per-market research and ratings as signals"
       ]
     },
     {
       "id": 3,
-      "prompt": "I need help before submission: my App Store listing has ASO keyword ideas, but also PrivacyInfo.xcprivacy entries, ATT wording, subscription paywall code, and a possible metadata rejection because screenshots show mockups. Which pieces should app-store-optimization own, and where should the rest go?",
-      "expected_output": "A boundary-aware answer that keeps app-store-optimization focused on discoverability and conversion strategy while routing compliance, privacy manifest, ATT, rejection, and StoreKit implementation details to sibling skills.",
+      "name": "ownership-boundary",
+      "prompt": "Our growth, compliance and payments teams keep arguing over who owns what on the App Store listing and the paywall. Split the work between ASO, App Store review and StoreKit, and say what ASO should still double-check.",
+      "expected_output": "ASO keeps keywords, name and subtitle, description copy, screenshots and captions, promotional text, Custom Product Pages, events, page tests, listing localization and ratings. Rejection risk, screenshot compliance, PrivacyInfo.xcprivacy, metadata policy and ATT go to app-store-review. Paywall code, purchase handling, entitlements and product setup go to storekit. ASO still reviews listing copy and screenshots for accuracy and conversion, and never recommends competitor names, unrelated keywords or misleading screenshots.",
       "files": [],
       "expectations": [
-        "Keeps app-store-optimization focused on keyword strategy, title/subtitle, description conversion copy, screenshots/captions, promotional text, Custom Product Pages, In-App Events, PPO tests, localization, and ratings strategy.",
-        "Routes App Store rejection risks, screenshot compliance, PrivacyInfo.xcprivacy, metadata policy, and ATT wording to app-store-review rather than solving them in depth.",
-        "Routes subscription paywall code, purchase handling, entitlement checks, and StoreKit product setup to storekit rather than expanding ASO into IAP implementation.",
-        "Still gives the ASO-safe next step: cross-check listing copy and screenshots for both conversion quality and compliance before submission.",
-        "Avoids recommending competitor names, unrelated keywords, or misleading screenshots as ASO tactics."
+        "Assigns keywords, name, subtitle, description, screenshots, captions, promotional text, Custom Product Pages, events, tests, listing localization and ratings to ASO",
+        "Sends rejection risk, screenshot compliance, PrivacyInfo.xcprivacy, metadata policy and ATT to app-store-review",
+        "Sends paywall code, purchase handling, entitlements and product setup to storekit",
+        "Keeps an ASO cross-check of listing copy and screenshots for accuracy and conversion",
+        "Never suggests competitor names, unrelated keywords or misleading screenshots"
       ]
     }
   ]

package/pipeline/skills/shared/external/app-store-optimization/references/keyword-research-methodology.md

@@ -1,174 +1,157 @@
 # Keyword Research Methodology
 
-Step-by-step process for identifying, scoring, and maintaining the keyword field for App Store search optimization.
+## What App Store search reads
 
-## Contents
+Apple documents two kinds of search signal: text relevance from the name,
+subtitle, keyword field and primary category, and user behavior such as
+downloads, ratings and reviews. The table below is a planning aid, not a list
+of ranking weights.
 
-- [How Apple Indexes Metadata](#how-apple-indexes-metadata)
-- [Research Process](#research-process)
-- [Keyword Scoring Framework](#keyword-scoring-framework)
-- [Slot Allocation Strategy](#slot-allocation-strategy)
-- [Seasonal Keyword Rotation](#seasonal-keyword-rotation)
-- [Before and After Examples](#before-and-after-examples)
+| Field | Role in search | Limit | Changing it | Guidance |
+|---|---|---|---|---|
+| App name | Indexed | 30 characters | With a new version, when status allows | Distinctive, and hints at what the app does |
+| Subtitle | Indexed | 30 characters | With a new version | Short statement of the value or main use |
+| Keyword field | Indexed | 100 bytes | With a new version | Commas between terms, no spaces after them |
+| Primary category | Counts toward relevance | One choice | With a new version | Never repeat its name in keywords |
+| Custom Product Page keywords | Indexed for approved, visible pages | Set per page | In App Store Connect | Sends matching searches to that page |
+| In-App Event name and short description | Shown on cards in Search | Set per event | Per event | Visible copy, not a hidden keyword field |
+| Description | Conversion | 4,000 characters | With a new version | Apple's product page guidance says not to add keywords for search here |
+| Promotional text | No effect on ranking, per Apple | 170 characters | Any time | Timely conversion message |
+| Ratings and reviews | Behavior signal | Ongoing | Earned | Mine for vocabulary; never a place to stuff terms |
 
-## How Apple Indexes Metadata
+Additional rules:
 
-Apple documents the main App Store search signals as text relevance, including matches for the app title, subtitle, keywords, and primary category, plus user behavior such as downloads, ratings, and reviews. Use this table as a practical planning map, not as a precise ranking-weight model:
+- Singular and plural of a word already present count as the same word.
+- Use a space only inside a real multi-word phrase.
+- Two different words that are both searched, such as "photo" and
+  "photography", may both earn a place.
+- Latin-script characters take one byte each; many other scripts take two or
+  three, so the field holds fewer of them.
 
-| Field | Indexed for search? | Controllable per version? | Notes |
-|-------|---------------------|--------------------------|-------|
-| App name | Yes | Yes (with new version/status permitting) | 30 characters. Choose a distinctive name that hints at what the app does. |
-| Subtitle | Yes | Yes (with new version/status permitting) | 30 characters. Summarize value or typical use in a concise phrase. |
-| Keyword field | Yes | Yes (with new version/status permitting) | Up to 100 bytes. Use comma-separated terms with no spaces after commas. |
-| Primary category | Yes | Yes (with new version/status permitting) | Apple includes the primary category in search relevance. Do not repeat the category name in keywords. |
-| Custom Product Page assigned keywords | Yes, for approved visible pages | Yes (in App Store Connect) | Can route matching searches to a custom page instead of the default product page. |
-| In-app event name and short description | Visible in Search/event cards | Yes (per event) | Use query-relevant language for the event card, but avoid treating event copy as a hidden keyword field. |
-| Description | Conversion-focused | Yes (with new version/status permitting) | Apple's product page guidance says not to add unnecessary keywords to improve search results. |
-| Promotional text | No ranking effect | Yes (anytime) | Apple says promotional text does not affect search ranking. Use it for timely conversion messaging. |
-| Ratings and reviews | Behavior/search signal | Ongoing | Ratings and reviews influence search and conversion, but review text should be used as customer research rather than keyword stuffing. |
+## Research process
 
-### Indexing behavior details
+### 1. Competitors
 
-- Avoid plurals of words already included in singular form; Apple treats them as duplicates.
-- Do not repeat words from the app name, subtitle, or category in the keyword field.
-- Use spaces only inside real keyword phrases; do not add spaces after commas.
-- Include separate related terms when both are genuine search terms, such as "photo" and "photography".
+Pick the top 5 to 10 apps that rank for your category's main terms. For each,
+record:
 
-## Research Process
+- the keywords in its name and subtitle,
+- positioning words that recur across names, subtitles and screenshots,
+- terms nobody covers.
 
-### Step 1: Competitor audit
+Type partial queries into App Store search and note the autocomplete
+suggestions; they come from what people actually search.
 
-Identify the top 5-10 apps ranking for the primary category terms. For each competitor, note:
+### 2. Brainstorm by type
 
-- Title and subtitle keywords
-- Visible positioning patterns (terms that appear frequently across top-ranking titles, subtitles, and screenshots)
-- Gaps -- relevant terms that competitors are not targeting
+| Type | Example for a hiking app |
+|---|---|
+| What it does | trail, map, route |
+| What it is used for | hike, backpack, summit |
+| Who uses it | beginner, family, solo |
+| Nearby activities | camping, climbing, birding |
 
-Use App Store search autocomplete to discover what users actually type. Start typing a category term and note the suggested completions.
+Terms that appear in many competitor names usually mean high volume and hard
+competition. Terms that autocomplete offers but competitors ignore are the
+openings.
 
-### Step 2: Category analysis
+### 3. Apple Ads discovery
 
-List every term a user might associate with the app's functionality:
+Run a discovery campaign with broad match for two to four weeks. Export the
+search terms report, sort by tap-through rate to find intent and by conversion
+rate to find terms that lead to downloads, and move the winners into the
+keyword field.
 
-- Primary function terms (what the app does)
-- Use case terms (why a user would want it)
-- Audience terms (who the user is)
-- Adjacent terms (related activities or workflows)
+### 4. Every update
 
-Filter this list against competitor findings. Terms that appear in many competitor titles are high-volume but high-competition. Terms that appear in autocomplete but not in competitor metadata represent opportunities.
+- Check impressions, tap-through rate and conversion in App Analytics.
+- Swap out terms with low impressions or low tap-through.
+- Try newly discovered terms.
+- Note what competitors changed in their metadata.
 
-### Step 3: Search Ads discovery
+## Scoring
 
-Run Apple Search Ads discovery campaigns with broad-match keywords matching the primary function. After 2-4 weeks of data:
+Rate each candidate from 1 to 5 on three scales:
 
-- Export the search term report to see actual queries that triggered impressions.
-- Sort by tap-through rate (TTR) to identify high-intent terms.
-- Sort by conversion rate to identify terms where users who find the app actually download it.
-- Add high-performing discovered terms to the keyword field.
+- **Relevance**: 5 means it describes the core function, 1 means it is
+  tangential.
+- **Volume**: estimated from Apple Ads impressions.
+- **Difficulty**: 5 means little competition, 1 means large apps dominate it.
 
-### Step 4: Iteration cycle
+Only terms with Relevance of 3 or more are considered. Priority is
+`(Relevance x 2) + Volume + Difficulty`.
 
-With each app update:
+Rule a term out when:
 
-1. Review App Analytics for keyword impressions, tap-through rates, and conversion rates.
-2. Identify underperforming keywords (low impressions or low TTR) and replace them.
-3. Test new terms discovered through Search Ads or category changes.
-4. Check if competitor metadata has shifted -- new competitors or renamed apps can change the keyword landscape.
+- Relevance is below 3,
+- it is another company's brand name, which breaks the guidelines,
+- it already appears in the name or subtitle,
+- it is the primary category's name.
 
-## Keyword Scoring Framework
+## Filling the slots
 
-Score each candidate keyword on three dimensions:
+Two worked examples. Lengths are counted exactly.
 
-| Dimension | Score range | How to assess |
-|-----------|-------------|---------------|
-| **Relevance** | 1-5 | Does the keyword describe what the app actually does? 5 = core function, 1 = tangentially related |
-| **Volume** | 1-5 | Is this term frequently searched? Use Search Ads impression data as a proxy. 5 = high impressions, 1 = minimal |
-| **Difficulty** | 1-5 | How many established apps target this term? 5 = low competition (good), 1 = dominated by major apps |
+**Meditation app**
 
-**Decision rule:** Include a keyword only if Relevance >= 3. Among qualifying keywords, prioritize by (Relevance x 2) + Volume + Difficulty. This weights relevance heavily while still favoring discoverable, winnable terms.
-
-**Disqualify keywords that:**
-- Do not describe the app's actual functionality (Relevance < 3)
-- Are competitor brand names (violates App Store guidelines)
-- Duplicate a word already in the title or subtitle
-- Are the app's primary category name (Apple adds it automatically)
-
-## Slot Allocation Strategy
-
-The 100-character keyword field requires careful allocation. Here are allocation patterns for different app types:
-
-### Example: Fitness tracking app
-
-Title: `FitTrack -- Workout Log` (23 chars used)
-Subtitle: `Exercise & Activity Stats` (25 chars used)
-
-Keyword field strategy -- do not repeat: fittrack, workout, log, exercise, activity, stats
+- Name: `Stillpoint - Meditation Timer`
+- Subtitle: `Sleep Sounds & Calm Breathing`
+- Keywords (15 terms, 98 bytes):
 
 ```
-gym,training,health,run,step,calorie,weight,strength,cardio,hiit,yoga,plank,squat,routine,progress
+mindfulness,anxiety,stress,relax,focus,yoga,insomnia,nap,journal,mantra,zen,rain,noise,guided,mood
 ```
 
-(96 characters -- 4 remaining, all unique against title/subtitle)
-
-### Example: Photo editing app
+**Recipe app**
 
-Title: `PixelPop -- Photo Editor` (24 chars used)
-Subtitle: `Filters & Retouch Tools` (23 chars used)
-
-Keyword field -- do not repeat: pixelpop, photo, editor, filter, retouch, tool
+- Name: `Pantry Chef - Recipe Planner`
+- Subtitle: `Meal Prep & Grocery Lists`
+- Keywords (14 terms, 100 bytes):
 
 ```
-selfie,portrait,collage,crop,blur,preset,enhance,adjust,brightness,contrast,saturation,beauty,skin
+cooking,dinner,vegan,keto,healthy,shopping,budget,family,easy,quick,bake,cookbook,leftover,nutrition
 ```
 
-(99 characters -- all unique against title/subtitle)
-
-### Allocation principles
+Notes:
 
-- Each comma costs 1 character. Fewer, longer keywords are not inherently better than more, shorter ones -- Apple matches individual words, not phrases.
-- If two terms are searched as a phrase (e.g., "meal prep"), use the phrase when it matches real query behavior; use separate terms when each word is independently valuable.
-- Reserve 5-10 characters for experimental keywords that rotate each release based on performance data.
+- Every comma costs one byte. Search matches single words, so long keywords
+  are not better in themselves.
+- Keep a phrase such as "meal prep" together when people search it that way;
+  split words that are valuable on their own.
+- Hold back 5 to 10 bytes for experimental terms you rotate each release.
 
-## Seasonal Keyword Rotation
+## Seasonal rotation
 
-Some keywords have seasonal search volume spikes. Plan rotation around predictable events:
+| Season | Terms | Ship in |
+|---|---|---|
+| New year | resolution, goal, habit, new year | December release |
+| Back to school | student, school, study, planner | July or August |
+| Holiday shopping | gift, holiday, deal, wish list | October or November |
+| Summer fitness | outdoor, running, summer, beach | April or May |
+| Tax season | tax, receipt, expense, deduction | January or February |
 
-| Season/Event | Example keywords to add | When to deploy |
-|-------------|------------------------|----------------|
-| New Year | resolution,goal,habit,new year | December release |
-| Back to school | student,school,study,planner | July/August release |
-| Holiday shopping | gift,holiday,deal,wish list | October/November release |
-| Summer fitness | outdoor,running,summer,beach | April/May release |
-| Tax season | tax,receipt,expense,deduction | January/February release |
+Swap two or three terms per season. Keep the core 70 to 80 bytes stable and use
+the remaining 20 to 30 for seasonal experiments.
 
-Rotate 2-3 keyword slots per seasonal cycle. Keep the core 70-80 characters stable and use the remaining 20-30 characters for seasonal experimentation.
+## Before and after
 
-## Before and After Examples
+**Wasted bytes.** For a surf forecast app:
 
-### Problem: Wasted characters
-
-**Before:**
 ```
-fitness tracker, workout app, best exercise, health and wellness, gym
+Before (66): best surf app, surf forecast, wave report, and tide times, surfing
+After  (48): wave,swell,tide,wind,beach,buoy,report,spot,reef
 ```
-Issues: spaces after commas (-5 chars wasted), "fitness" and "tracker" may duplicate title, "best" and "app" have no search value, "and" is a stop word.
 
-**After:**
-```
-gym,health,run,step,calorie,weight,strength,cardio,hiit,yoga,plank,squat,routine,progress,streak
-```
-Fixes: no spaces, no duplicates of title/subtitle, no filler words, all terms are genuine search queries.
+The first version pays for spaces after commas, filler ("best", "app"), a
+stop word ("and") and words that belong in the name. The second holds nine real
+terms in fewer bytes and leaves 52 bytes for more.
 
-### Problem: Duplicate keywords
+**Duplicates.** With the name `SwellWatch - Surf Forecast`:
 
-**Before** (title is "RunMate -- Running Tracker"):
 ```
-running,tracker,run,jog,marathon,5k,pace,distance,route,gps,heart,rate,training,fitness,health
+Before (65): surf,forecast,forecasts,swell,wave,tide,wind,buoy,beach,reef,spot
+After  (59): swell,wave,tide,wind,buoy,beach,reef,spot,kite,marine,chart
 ```
-Issues: "running", "tracker", and "run" duplicate the title -- 17 characters wasted.
 
-**After:**
-```
-jog,marathon,5k,pace,distance,route,gps,heart,rate,training,fitness,health,stride,tempo,interval
-```
-Fixes: removed title duplicates, added 3 new high-value terms with the recovered space.
+Removing "surf", "forecast" and "forecasts" frees 24 bytes; "kite", "marine"
+and "chart" use 18 of them, and 41 bytes remain for further terms.