@mmerterden/multi-agent-pipeline 20.6.0 → 20.7.0

package/pipeline/skills/shared/external/localization-reuse-map/reference/sources-and-recipes.md

@@ -1,176 +1,139 @@
-# Sources & retrieval recipes (the rigorous method)
+# Sources and recipes
 
-> Bundled detail for [../SKILL.md](../SKILL.md). The data model and the exact, copy-paste recipe per
-> source. **Retrieval is a cross-reference, not a skim**  -  that is the whole point of this skill. Every
-> recipe reads files or uses `gh`; no other skill required.
+Where every column of the map comes from, how to trace it, and the traps that produce a confidently wrong row. The whole point of the skill is a cross-reference: each column is checked against at least one other source, never skimmed from one.
 
-## The data model  -  two sides, sources per side
+## Where each input lives
 
-| | UI element → key | key → translation value |
+| Input | Source | Access |
 |---|---|---|
-| **New (redesign)** | `design-export` components-used **×** component registry `localizationKeys` **×** the screen's code (what it wires). Cross-reference all three. | `<resources-repo>` → `Sources/Suggested/<Key>.json` (8 langs)  -  `scripts/resolve-new-values.py` |
-| **Legacy** | the screen **traced** through its code: iOS VC→VM→DataSource→**cell presentation models**; Android layout `@string/` + fragment `R.string.`; web component → `t('key')`/`$t('key')` call site | iOS shipped label map `<lang>.lproj/language.plist` (`Mobile-<Key>`, 8 langs)  -  `scripts/resolve-legacy-values.py`; gaps → labels-API snapshot; web → its `locales/<lang>.json` (or equivalent i18n catalog), same snapshot script |
-
-Read locally: the redesign app (code) + `<resources-repo>` (new values + registry).
-Over `gh` (no checkout): `<specs-repo>`  -  holds `design-export/`, the legacy iOS/Android source, the
-shipped `language.plist`, and `specs/`. New translation **values are shared across platforms** (one
-Suggested value per key); only key *usage* is per-platform.
-
-**A third value source  -  the CMS (content team) copy.** Beyond `new` (design/resources) and `legacy`, the
-content team writes the **actual final copy** as Figma **Dev Mode annotations** ("Final UX Writing", `TR:/EN:`).
-That copy is fetched live (`fetch-annotations.py`) and shown additively as the **CMS** columns  -  see *CMS
-values* below. And because the static element→key cross-reference misses runtime strings, a **scan of the
-implemented screen** (`scan-screen-keys.py`) recovers error / dynamic / registry-missed keys  -  see *Catching
-the keys the cross-reference misses* below.
-
-## New side  -  element → key (CROSS-REFERENCE, not code-only)
-
-A component's registry `localizationKeys` are what it renders **internally**; the screen's code shows
-what it **wires**; neither alone is complete. Do all three:
-
-1. **Which components + nodes the screen comprises**  -  `design-export` (single-repo, the snapshot model):
-   ```bash
-   gh api "repos/<org>/<specs-repo>/contents/design-export/<project>/screens/<screen-slug>/components-used.md" -H "Accept: application/vnd.github.raw"
-   ```
-   Read **every state frame** of the screen (empty / 1-item / filled)  -  components only present in the
-   filled state (e.g. an item list) still belong to the screen. The folder also has `screenshot.png`,
-   `tree.json` (node ids), `tokens.json`. *(Live alternative if a frame isn't exported: Figma REST via a
-   host Figma tool  -  REST works with a token; MCP/live needs Figma Desktop in Dev Mode. The export is
-   preferred  -  single-repo.)*
-2. **Each component's owned keys**  -  registry, local & instant:
-   ```bash
-   python3 -c "import json;d=json.load(open('Resources/Figma/Components/<NODE-DASH>.json'));print([k['key'] for k in d.get('localizationKeys',[])])"
-   ```
-   These include keys the component renders that **never appear in the screen code** (e.g.
-   `ItemListGroup.Title`, `ItemListRow.Delete`). Beware the inverse: a shared
-   component (e.g. `HeaderMain`, `BottomActions`) owns many keys the screen does **not** use because the
-   screen passes its own title in  -  confirm against the code.
-3. **What the screen wires**  -  read the scene/screen file: each `LocalizationStringKey.<Ns>.<leaf>` call
-   site is one element; the call site is the label. Screen-specific keys (`SignUpAccountDetails.*`) and
-   reused keys (`ModalsCustoms.Ok`) live here.
-
-Union the three; tag each key **screen-specific / reusable-component / shared-primitive**. Component-only
-keys: map once where that component is mapped  -  flag, don't duplicate per screen.
-
-**New values:** `python3 scripts/resolve-new-values.py --resources-root <resources-repo> --keys "..." --langs all`.
-
-## Legacy side  -  element → key (TRACE the screen, don't skim the map)
-
-1. Start from `specs/<area>/<screen>-spec.md`  -  its `> Legacy reference (iOS):` blockquote names the entry
-   class (VC/VM).
-2. **iOS  -  follow the chain.** Fetch the VM raw; it builds sections/cells. Form-field labels are **not** in
-   the VM  -  they default inside the **cell presentation models** it instantiates. Follow them:
-   ```bash
-   gh api ".../contents/<path>/<Screen>ViewModel.swift" -H "Accept: application/vnd.github.raw" > /tmp/vm.swift
-   grep -nE 'PresentationModel|\.localized' /tmp/vm.swift          # find the cell classes it creates
-   # then fetch each cell class and grep its keys:
-   gh api ".../Forms/TextInputCell/PresentationModels/<Field>TextInputCellPresentationModel.swift" -H "Accept: application/vnd.github.raw" | grep -oE '"[^"]+"\.localized'
-   ```
-   This is how `NewSurnameReq` / `OrderNumber` / `ShippingAddressPlaceHolder` surface  -  a VM-only grep
-   misses them. Keys are the string literal before `.localized`.
-3. **Android.** Layout XML carries labels as `@string/Key`; fragment carries runtime strings as `R.string.`:
-   ```bash
-   gh api ".../res/layout/<screen>.xml" -H "Accept: application/vnd.github.raw" | grep -oE '@string/[A-Za-z0-9_]+' | sort -u
-   ```
-4. **Web (when the legacy app is a web frontend).** The component wires the key as an i18n call, not a
-   string literal  -  grep the component tree for the call site, not the copy:
-   ```bash
-   gh api ".../src/screens/<Screen>/index.tsx" -H "Accept: application/vnd.github.raw" \
-     | grep -oE "(\\\$?t\(['\"]|useTranslation\(['\"])[A-Za-z0-9_.]+" | sort -u
-   ```
-   Covers `t('key')` / `i18n.t('key')` (react-i18next, next-intl) and `$t('key')` (vue-i18n). A template
-   literal or variable argument (`` t(`errors.${field}`) ``) is a **dynamic** key exactly like iOS
-   `String(format:)`  -  record the pattern, not a resolvable literal, same rule `scan-screen-keys.py`
-   applies. Namespace prefixes (`t('signup.email_label')` inside `useTranslation('signup')`) resolve to
-   `signup.email_label`  -  read the hook/provider call, don't assume the literal alone is the full key.
-
-iOS, Android and web usually share the flat key for shared elements; when any platform differs it is
-drift worth a **review** (e.g. `AgentaUserAlreadyAdded` iOS typo vs `AgencyUserAlreadyAdded` Android).
-Record every column present; `  -  ` where a platform has none.
-
-## Legacy translation values  -  the in-repo legacy-label snapshot (default), refreshed from the live service
-
-Values come from your legacy backend’s label service (the same map the legacy apps render)  -  but **snapshotted
-in-repo** so the mapper reads them offline, no network round-trip per run (single-repo, like design-export):
+| Figma structure (components, nodes, screenshot) | In-repo design export | Local files |
+| CMS annotations and overlay geometry | Figma REST API (keychain `FIGMA_ACCESS_TOKEN`); Figma MCP in Dev Mode as the rate-limit-free alternative; the export as offline fallback | Network |
+| Screenshot | `screens/<frame>/screenshot.png` in the design export | Local |
+| Specs repo files | `gh api repos/<owner>/<repo>/contents/<path>` with `Accept: application/vnd.github.raw`; no checkout needed | Network |
+| Redesign app and resources repo | Local checkouts | Local |
+| Legacy values | `resources/Localization/Legacy/<lang>.json`, a flat snapshot that is refreshed from the label service | Local, network on refresh |
+
+The Android app's committed `strings.xml` is only a key catalog: its values default to the key itself. Real legacy values live in the iOS bundled `language.plist` and, authoritatively, in the label service.
+
+## New keys: the three-way union
+
+A screen's keys come from three places, and each one misses something the others see:
+
+1. **Design export**: read `screens/<frame>/components-used.md` of each state frame the screen has (default, error, loading, filled...).
+2. **Component registry**: each used component's `<node>.json -> localizationKeys`. These are keys the component renders internally, and they are often absent from the screen code.
+3. **Screen code**: on iOS, each call site of `LocalizationStringKey.<Namespace>.<leaf>` counts as one element.
+
+Take the union and tag every key: screen-specific, component-owned, or shared (a reusable component or a shared primitive).
+
+Registry caveat: a shared component such as `HeaderMain` or `BottomActions` may list keys that this screen never shows, since the screen supplies its own title. Check the code before a registry key goes into the map.
+
+### Keys the union cannot see
+
+Error, validation and alert strings, and keys assembled at runtime, rarely appear in the design export or the registry. Run the scanner on the implemented screen:
+
 ```bash
-python3 scripts/resolve-legacy-values.py --langs all \
-  --snapshot-root <specs-repo>/resources/Localization/Legacy \
-  --keys "Continue,EmailAddress,PasswordRule,EmailAlreadyExists"
+python3 scripts/scan-screen-keys.py --out scan.json --screen-path <screen-dir>
 ```
-The snapshot is `resources/Localization/Legacy/<lang>.json`  -  a flat `{key: value}` map per language.
-If your backend namespaces keys under a prefix the app strips (a stored `Mobile-Continue` for the app's
-`Continue`), pass `--prefix Mobile-` and keep the mapping's keys bare; set the mapping's `legacyKeyPrefix`
-so the rendered table still shows the content team the full stored key.
 
-**Refresh** the snapshot (the only network step) with `fetch-legacy-labels.py`. Legacy translations are
-served by *your* backend, so no endpoint is built in  -  you describe it once:
+Every `error` and `dynamic` hit becomes a row. Compare the `static` hits with the union; whatever remains is a key the code wires but the map lacks. Skipping this step drops those strings silently.
+
+## New values
+
+Values are authored per key in `Suggested/<Key>.json`, in eight languages, and they are shared by all platforms: one value per key, only the key usage differs per platform.
+
 ```bash
-python3 scripts/fetch-legacy-labels.py --out <specs-repo>/resources/Localization/Legacy --langs all \
-  --endpoint 'https://labels.example.internal/{env}/labels/{lang}' --env dev \
-  --headers-file .secrets/label-headers.json \
-  --fail-status '9999=gateway blocked the request' --fail-status '50=client version not accepted'
+python3 scripts/resolve-new-values.py --keys "<k1>,<k2>" --resources-root <res> --langs all \
+  --catalog <path/to/shipped/Localizable.xcstrings>
 ```
-- Keep tokens in `--headers-file` (a JSON object), never in `--header`  -  argv lands in shell history and CI logs.
-- The response shape is discovered, not assumed: the largest flat `{string: string}` map anywhere in the JSON
-  wins, so an enveloped payload works without configuration.
-- `--fail-status CODE=message` turns your backend's in-band error codes into a clear abort instead of an
-  empty snapshot. Find these once and record them in your project's own notes.
 
-`resolve-legacy-values.py --live` (same `--endpoint` / `--headers-file`) bypasses the snapshot to read the
-service directly, to verify or refresh. `--plist-root <dir>` is a last-resort offline fallback for a legacy
-iOS app's bundled `<lang>.lproj/language.plist`  -  typically a stale subset, so a hit there deserves a note.
-Never invent a legacy value  -  blank + a note beats a guess.
+Always pass `--catalog`. A missing Suggested file means "not in this snapshot", not "unauthored": the snapshot can lag the shipped catalog. With the catalog the value still resolves and the script names the stale snapshot so it can be refreshed with `snapshot-resources.sh`. Only a key absent from both the snapshot and the catalog is truly unauthored. Never open an authoring request for a key the catalog resolved.
 
-## CMS values  -  the content team's actual copy (Figma Dev Mode annotations)
+## CMS copy (Figma annotations)
 
-The content team annotates each text node with the **final** bilingual copy as a Dev Mode annotation
-(category *Final UX Writing*), `TR: ...` / `EN: ...`. The design text is usually a placeholder ("Giriniz",
-"Lorem ipsum")  -  the **annotation**, not the design text, is their real value. Figma is edited continuously,
-so fetch it **live**, with fallbacks (priority order):
+The content team writes final copy as Dev Mode annotations in the category "Final UX Writing", usually as `TR: ...` and `EN: ...` lines. The text drawn in the design is often placeholder copy; the annotation is the real value.
 
 ```bash
-# 1) REST (default)  -  needs a Figma token (keychain FIGMA_ACCESS_TOKEN, env, or --token):
-python3 scripts/fetch-annotations.py --mapping <map>.json --out _annotations.json
-#    (or --file <fileKey> --nodes "1024:4096,1024:4112")
-# 2) MCP (no rate limit; the host's Figma plugin dumped the nodes' annotations):
-python3 scripts/fetch-annotations.py --from-mcp _annotations.raw.json --out _annotations.json
-# 3) local fallback (offline; only if design-export tree.json carries annotations):
-python3 scripts/fetch-annotations.py --local design-export/<project>/screens/<slug> --out _annotations.json
+python3 scripts/fetch-annotations.py --out _annotations.json --mapping <map>.json
+python3 scripts/fetch-annotations.py --out _annotations.json --from-mcp <raw-mcp-dump>.json
+python3 scripts/fetch-annotations.py --out _annotations.json --local <design-export>/screens/<frame>
 ```
-It calls Figma `GET /v1/files/<key>/nodes?ids=...` (header `X-Figma-Token`), walks `node.annotations[].label`,
-parses `TR:/EN:` (the same `split_lang` rules the web skill uses), and emits
-`{nodeId, designText, tr, en, raw, mode}` per annotated node. Map each annotation onto its row by
-**`nodeId`** (store it as the row's `cmsNodeId`) → fill the row's `cms:{tr,en}`. A `429` aborts with a "use
-`--from-mcp`" hint. **Never invent a CMS value**  -  leave `cms` absent if there's no annotation. A CMS value
-that differs from `new` is fine (the renderer marks it ⚠); a CMS-only `TR` with no `EN` is flagged back to
-the content team.
-
-## Catching the keys the cross-reference misses  -  scan the implemented screen
-
-The static element→key cross-reference (registry × design-export × code) is biased toward **statically-wired
-literal** keys. It is blind to **error/validation/alert** copy (lives in validators, error mappers, alert
-builders, VM error branches  -  not the VC→VM→cell chain) and to **dynamic/interpolated** keys
-(`String(format:)`, variable `.localized` receivers, `LocalizationStringKey(<expr>)`, Android
-`getString(<var>)`). Recover them by scanning the implemented screen directly:
+
+Pair every annotation with its row through `nodeId`: the id is saved in the row as `cmsNodeId`, and the copy goes into `cms: {tr, en}`. An element without an annotation keeps a blank CMS cell. The script lists nodes that have TR but no EN; send those back to the content team.
+
+## Legacy keys: trace, do not skim
+
+### iOS
+
+The spec names the entry point: `specs/<area>/<screen>-spec.md` carries a blockquote `> Legacy reference (iOS):` naming the class. Follow the chain from there:
+
+view controller, then view model, then data source, then the cell presentation models, where the `"Key".localized` calls usually are.
+
+Form-field labels are normally defaulted inside the cell presentation models (often shared input-cell models), not in the view model. Do not stop at the view model.
+
+### Android
+
+Labels come from the layout XML as `@string/Key`; strings set at runtime come from the fragment as `R.string.<name>`. For form-field labels the layout is the most reliable place to read them.
+
+### Web
+
+Call sites are `t('key')` or `i18n.t('key')` (react-i18next, next-intl) and `$t('key')` (vue-i18n), resolved against `locales/<lang>.json`. A template literal or a variable argument is a dynamic key. A namespace from `useTranslation('<ns>')` is prepended to the key, so `t('email_label')` under `useTranslation('signup')` is `signup.email_label`.
+
+### Cross-platform drift
+
+The same element can carry different keys per platform (`AgentaUserAlreadyAdded` on one, `AgencyUserAlreadyAdded` on the other). That is a `review`. Keep every platform column that has a key and render a dash where a platform has none.
+
+## Legacy values
 
 ```bash
-python3 scripts/scan-screen-keys.py --screen-path <impl-screen-dir>     # [--category error|dynamic|static]
+python3 scripts/resolve-legacy-values.py --keys "Continue,EmailAddress" \
+  --snapshot-root resources/Localization/Legacy --langs all --prefix Mobile-
+```
+
+- Reading the snapshot needs no network. `fetch-legacy-labels.py` refreshes it and is the only step that goes online.
+- `--live` checks against the service directly; if the live call fails, the script falls back to the snapshot, then to the plists.
+- `--plist-root` reads the bundled `<lang>.lproj/language.plist` as a last resort. It is usually a stale subset, so a hit from it deserves a note.
+- **Backend prefix.** When the backend stores `Mobile-Continue` but the app asks for `Continue`, pass `--prefix Mobile-`, keep the mapping keys bare, and set the mapping's `legacyKeyPrefix` so the table shows the full stored key.
+
+## Overlay geometry
+
+The overlay does not depend on annotations. With zero annotations every keyed element still gets a gray "awaiting copy" card. The number on a card and on its dot is the index of the matched row in `mapping.rows`, counted from 1, so it agrees with the table rather than with the order on screen.
+
+**REST (preferred).** Needs `figmaFileKey` and the real frame node. Take the frame node-id from the design export's `tree.json` root `nodeId`: folder-name suffixes and ids copied from older maps are often wrong. Check that the node text belongs to this screen; background nodes that match no row are dropped automatically.
+
+**`--spec` (screenshot-anchored).** Use it when there is no token or when REST geometry is noisy. Measure the box of every visible element on the clean screenshot, give the rows synthetic `cmsNodeId`s (`n1`, `n2`, ...), and write:
+
+```json
+{"fileKey": "...", "pages": [{"nodeId": "1024:4096", "image": "screenshot.png",
+  "frame": {"w": 375, "h": 812},
+  "nodes": [{"id": "n1", "x": 16, "y": 64, "w": 220, "h": 28, "side": "left"}]}]}
 ```
-It greps `.swift`/`.kt`/`.xml`/`.ts`/`.tsx`/`.js`/`.jsx`/`.vue` and classifies each hit `static | error |
-dynamic` with file:line. **Fold the
-`error` and `dynamic` hits into the mapping** (each still gets the normal new/legacy/CMS resolution); diff the
-`static` hits against the registry union to catch keys wired in code but absent from the registry. Dynamic
-keys that can't be statically resolved are mapped with a note (`verdict: review` / "dynamic  -  enumerate at
-runtime"), never silently dropped.
+
+`image` is either a data URL or a PNG path, resolved from the folder of the spec file. `side` is `left` (default) or `right`: it puts the card on the side where its target sits, which two-column layouts need. Each side is numbered and stacked independently, and right-side connectors are mirrored.
+
+The export's `tree.json` stops at component instances and carries no geometry per label, so take only the frame node-id from it.
+
+Component-owned keys (list rows, promotion grids, summary cards) are marked on the overlay too, flagged `(owned by <Component>)`, with their keys taken from the registry `localizationKeys`.
+
+**Multi-state screens.** Each state gets an overlay of its own, built from its own `--spec`, screenshot and `cmsNodeId`s; a row may keep the same `cmsNodeId` in several states. Stack the results vertically in a single PNG, padded to the widest one with a `#1e1e1e` gutter between them, point the mapping's `overlay` at that PNG, and keep the per-state images next to it.
+
+**Isolating a component variant.** Once the library is re-versioned, node-ids exported from it stop resolving: `/components` may return nothing and `/images` returns `null`. Walk the live screen frame instead (`GET /v1/files/<key>/nodes?ids=<frame>&depth=6`), find the `INSTANCE` whose `name` matches, render that id, and take label boxes from its text nodes' `absoluteBoundingBox` multiplied by the render `scale`. Without a token, or when the render fails, cut the variant from a screenshot of one of the states.
+
+## Keyshots
+
+For each matched row, `render-key-shots.py` saves a crop with the element outlined in red as `keyshot__<NN>__<key>.png` inside `keyshots/`, where NN is the row number in the table, and it writes `<slug>.keyshots.manifest.json`. A row with no match gets a dash, never a guessed crop. Set the mapping's `keyshots` to the manifest file name.
+
+Geometry reuses the overlay inputs: REST (visible text only, filtered for ancestor visibility and cumulative opacity) or the same `--spec` files, one per state, repeatable; `image` may be a path such as the committed `screenshot.png`.
 
 ## Pitfalls
-- **Code-only / map-skim misses keys.** New side needs registry × design-export × code; legacy side needs
-  the cell-chain trace. This is the difference between right and wrong keys.
-- **Static cross-reference misses runtime strings.** Error/validation/alert copy and dynamic/interpolated
-  keys are invisible to the literal greps  -  run `scan-screen-keys.py` over the implemented screen and fold
-  in the `error`/`dynamic` hits, or they silently vanish from the map.
-- The recursive Git tree of `<specs-repo>` truncates (~43k paths)  -  use `gh search code` / direct
-  `contents/` calls, not `git/trees/HEAD?recursive=1`.
-- New namespace ≠ legacy flat key  -  "reuse" means reuse the *value* (or alias the key), not string equality.
-- Legacy values default to the key in the committed Android `strings.xml`  -  that file is a key *catalog*,
-  not values; values live in the iOS `language.plist` (and the service).
+
+- **Truncated trees.** On a large specs repo the recursive Git tree is cut off at roughly 43k paths. Search with `gh search code` or fetch through `contents/` calls rather than `git/trees/HEAD?recursive=1`.
+- **Namespaced vs flat keys.** `SignUpAccountDetails.ContinueButton` against legacy `Continue`: reuse is about the value, not the key string.
+- **Hidden form labels.** They sit in shared input-cell models; read them from the Android layout.
+- **Blank is fine.** Empty Old and CMS columns are expected states, not errors.
+- **Scanner skipped.** Error, validation, alert and dynamic keys never show up in the static cross-reference.
+- **Frame origin drift.** A rendered frame PNG may not share the frame's `absoluteBoundingBox` origin. Subtracting the frame x/y from text-node x/y has shifted boxes by tens of pixels, a full row off. Check one crop by eye, and prefer boxes read off the actual screenshot to coordinates mixed from two sources.
+- **Stale snapshot.** Missing from Suggested is not unauthored; pass `--catalog` and refresh the snapshot instead of filing requests.
+- **Value anchoring.** Never anchor a keyshot by value across a whole file: short labels recur on unrelated screens and give a confidently wrong crop, which is worse than a dash. Recover a row only through its own `cmsNodeId`, and leave `characters` out of nodes handed to `render-key-shots.py --spec`, because text present turns value matching back on and can bind an unrelated row.
+- **Confluence auth.** Server / Data Center takes a Bearer PAT (keychain `CONFLUENCE_API_TOKEN`), not the Cloud `user:token` form. Pages are looked up with the `content?title=` query, which reads the database; CQL is avoided because its index lags and leads to duplicate pages.
+- **Secrets.** Never commit a label-service token, Confluence PAT or Figma token into a mapping, a taxonomy file or the docs.
+- **TLS proxies.** Publishing runs through `curl` so a corporate proxy's trust store is honoured (Python `urllib` fails verification there). The token goes in a `chmod 600` config passed with `-K`, never on argv. For the Figma scripts behind such a proxy, export the system roots with `security find-certificate -a -p > roots.pem` and pass `--ca-file roots.pem`.