toga-ai 1.0.454 → 1.0.455

package/knowledge/2.0/apps/_underscore/features/model-magic-field-access.md

@@ -6,7 +6,7 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-06-25
+updated: 2026-07-28
 owners: ["jcardinal"]
 files:
   - _underscore/Model/Core/Model.php
@@ -43,7 +43,15 @@ gate on `isset($model->field)` or `$model->field ?? $default` — both silently
 
 ## Gotchas / known issues
 
-- `isset($model->magicField)` / `$model->magicField ?? null` are **always** false/null for DB-backed
+- **Treat `empty()` / `isset()` / `??` on a magic field as simply UNRELIABLE — the exact behavior
+  is not uniform.** Fields live in `$_model_fields[...]['value']` behind `__get()`. In the 2026-06-25
+  case `??` short-circuited to null; in the 2026-07-28 cross-client case the opposite mix appeared —
+  `empty($model->id)` reported a **populated** field (value `49`) as absent, while
+  `$model->id ?? $default` read it fine. Do not reason about which way it will fail. **Read the
+  value into a plain local variable first and test that local.** This cost three debugging rounds
+  because it failed in the "looks unauthenticated" direction — a populated identity read as empty.
+  A codebase-wide sweep for `empty(`/`isset(` on model fields is warranted.
+- `isset($model->magicField)` / `$model->magicField ?? null` cannot be trusted for DB-backed
   magic fields. Use `getFieldConfig()` + `array_key_exists` to test presence.
 - A bare `__get` on an unconfigured field **throws** — never read a maybe-absent field without the
   `getFieldConfig()` guard.
@@ -52,6 +60,12 @@ gate on `isset($model->field)` or `$model->field ?? $default` — both silently
 
 ## Change history
 
+- 2026-07-28 — Broadened the rule after the cross-client work: `empty()` can report a **populated**
+  field as absent while `??` reads it correctly — the inverse of the 2026-06-25 symptom. The safe
+  rule is now "copy to a plain local, then test the local," and a codebase-wide sweep is warranted
+  because it fails in the "looks unauthenticated" direction. Also noted the sibling trap: `_Model`
+  writes every **declared** field explicitly, so a DB column not declared on the model both makes
+  `__get` throw and means the column's SQL default never applies on insert. (jcardinal)
 - 2026-06-25 — Documented the `__get`-without-`__isset` gap and the `getFieldConfig()` +
   `array_key_exists` safe-read pattern, discovered while debugging a missing `uuid` in the API2
   translation fallback warning. (jcardinal)

package/knowledge/2.0/apps/_underscore/features/per-client-database-connections.md

@@ -6,7 +6,7 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-07-27
+updated: 2026-07-28
 owners: ["dfranks", "jcardinal", "mhammontree", "apeterson", "kyalamarthi"]
 files:
   - _underscore/Database.php
@@ -50,8 +50,11 @@ The three per-client aliases and their concrete local schemas:
 | `ClientLogs` (`DB_CLIENT_LOGS`) | `Logs_<Id>` | **API transaction logging** |
 | `Archive` | `Archive_<Id>` | historical snapshots |
 
-`<Id>` is the client's `clientIdentifier` from `Core.Clients` (e.g. `Growrk` → `Client_Growrk`,
-`Logs_Growrk`, `Archive_Growrk`). A `Logs_<Id>` schema has 5 tables:
+`<Id>` is *usually* the client's `clientIdentifier` from `Core.Clients` (e.g. `Growrk` →
+`Client_Growrk`, `Logs_Growrk`, `Archive_Growrk`) — but that is a naming convention, **not** a rule
+you may rely on in code. **Always resolve the real schema name through
+`Core.Clients.clientDatabaseId` → `Core.Databases.name`** (see the gotcha below). A `Logs_<Id>`
+schema has 5 tables:
 `Api, CustomRecordField, Error, Record, RecordField`.
 
 ## Data model
@@ -63,18 +66,28 @@ Engine InnoDB, `utf8mb4_unicode_ci`.
 
 ## Client variations
 
-None — uniform across clients; only the `<Id>` suffix differs.
+Structurally uniform, but two per-client values genuinely differ and must never be assumed:
+
+- **The schema NAME** — resolve `Core.Clients.clientDatabaseId` → `Core.Databases.name`. Observed
+  divergence: `clientIdentifier` = `Compass_Usa` but the database is `Client_Compass`.
+- **The CHARSET/COLLATION** — client databases do **not** share one. `utf8mb4_unicode_ci`,
+  `utf8mb4_0900_ai_ci` and `latin1_swedish_ci` were all observed on the same logical column across
+  tenants.
 
 ## Related shared clusters (not per-client)
 
 Alongside the three per-client aliases and the shared **Core** cluster, a dedicated shared
-**Cache** cluster was added (2026-07-07): `Core.Databases` id **145**, `_underscore` alias
-`DB_CACHE`, `CACHE_DATABASE_ID = 145`. It is modeled like Core (a single shared cluster, **not**
+**Cache** cluster was added (2026-07-07), alias `DB_CACHE`. It is resolved **by name**
+(`Databases.name = 'Cache'`) in both api2 and worker2 — there is deliberately **no
+`CACHE_DATABASE_ID` constant**, because `Databases.id` is auto-assigned per Core instance and
+differs per environment, so a hardcoded id works only where it happens to match and fails silently
+elsewhere. It is modeled like Core (a single shared cluster, **not**
 per-client), region-aware (1=us-east-1, 2=us-west-2, 3=eu-west-1) with per-region `DatabaseHosts`
 (`hostCluster` DevOps-TBD). It exists to keep cache churn off Core and backs the api2
 [multi-client data retrieval](../../api2/features/cross-client-data-retrieval.md) engine. Boot
-registration is region-aware in `api2/Controller/Index.php`; the alias const is in `api2/_.php`.
-`Records.ttlCache` (TINYINT UNSIGNED, default 15) governs cache TTL.
+registration is name-based in `api2/Controller/Index.php` and `worker2/Controller/Index.php`.
+`Records.ttlCache` (TINYINT UNSIGNED, default 15) governs cache TTL; `0` means "effectively
+uncached" but is **floored at 1 minute** so an entry always outlives the request that built it.
 
 ### The `Team` schema rides the Core cluster (`DB_TEAM`)
 
@@ -94,6 +107,15 @@ here — they live in `Config/*.ini`.)
 
 ## Gotchas / known issues
 
+- **NEVER build a client's database name as `'Client_' . $clientIdentifier`.** They diverge in
+  production (`Compass_Usa` → `Client_Compass`). Resolve
+  `Core.Clients.clientDatabaseId` → `Core.Databases.name`. This mirrors the log-database resolution
+  already done in `_underscore`'s `Email.php`. A string-built name fails only for the clients whose
+  names happen to diverge, so it passes local testing and breaks in production.
+- **Client DBs do not share a charset, so any hardcoded `COLLATE` is invalid somewhere.** Emitting
+  e.g. `COLLATE utf8mb4_unicode_ci` against a `latin1` column errors with `COLLATION ... is not
+  valid for CHARACTER SET 'latin1'`. For cross-tenant string comparison use `CAST(expr AS BINARY)`
+  (byte ordering) rather than naming a collation.
 - **The "logs DB write trap" (local dev).** A laptop usually imports only `Client_<Id>`, not
   `Logs_<Id>`/`Archive_<Id>`. Because logging is **on by default**, the first request 500s on
   `Unknown database 'logs_<id>'` from `ApiRequest.php`'s logging write — *after* the real work
@@ -130,6 +152,14 @@ here — they live in `Config/*.ini`.)
 
 ## Change history
 
+- 2026-07-28 — **Corrected a wrong rule:** a client's schema name is *not* reliably
+  `'Client_' . clientIdentifier` (`Compass_Usa` → `Client_Compass`); resolve
+  `Clients.clientDatabaseId` → `Databases.name`. Recorded that client DBs do **not** share a
+  charset (utf8mb4_unicode_ci / utf8mb4_0900_ai_ci / latin1_swedish_ci observed on one column), so
+  hardcoded `COLLATE` is invalid somewhere — use `CAST(... AS BINARY)`. Replaced the Cache
+  cluster's hardcoded id 145 / `CACHE_DATABASE_ID` with **name-based resolution**
+  (`Databases.name = 'Cache'`), since ids are per-Core-instance and a hardcoded id fails silently
+  per environment. (jcardinal)
 - 2026-07-27 — Recorded that the three per-client aliases key **all** live connection state, so
   re-pointing an alias mid-request does not reconnect on its own; split the detail into
   [Re-pointing a DB alias mid-request](./database-alias-repointing.md). (jcardinal)

package/knowledge/2.0/apps/api2/INDEX.md

@@ -3,7 +3,7 @@
 | Doc | Summary | Files |
 |-----|---------|-------|
 | [API (api2 / TOGa API v2) Architecture](architecture.md) | `api2` is the backend powering the public **TOGa 2.0 API**. | api2/Controller/Index.php, api2/Component/Api/V2/V2.php, api2/Component/Api/Cxml/Cxml.php, api2/Component/Api/V2/Response/Response.php, api2/Config/ |
-| [Multi-Client (Cross-Client) Data Retrieval](features/cross-client-data-retrieval.md) | A single authenticated V2 GET listing can return records across **many** clients (designed for 1000+) that the caller is entitled to, honoring **each target cli | api2/Component/Api/CrossClient/CrossClient.php, api2/Component/Api/V2/V2.php, api2/Controller/Index.php, api2/_.php, _underscore/Model/Cache/Table.php, _underscore/Model/Cache/Tables/Client.php, dbchanges2/Core/2026-06-30b - CacheClusterRegistrationAndRecordTtl.sql, dbchanges2/Cache/2026-06-30a - MultiClientCacheTables.sql |
+| [Multi-Client (Cross-Client) Data Retrieval](features/cross-client-data-retrieval.md) | A single authenticated V2 GET listing can return records across **many** clients (designed for 1000+) that the caller is entitled to, honoring **each target cli | api2/Component/Api/CrossClient/CrossClient.php, api2/Component/Api/V2/V2.php, api2/Controller/Index.php, _underscore/Model/Cache/Table.php, _underscore/Model/Cache/Tables/Client.php, _underscore/Model/Core/Record.php, worker2/Worker/Platform/Cache.php, worker2/Controller/Index.php, worker2/_.php, dbchanges2/Cache/2026-06-30a - MultiClientCacheTables.sql, dbchanges2/Core/2026-06-30b - CacheClusterRegistrationAndRecordTtl.sql, dbchanges2/Core/2026-07-27a - PlatformCacheCleanCron.sql |
 | [Encrypted-User-UUID Auth Handoff (/auth/encrypted-user-uuid)](features/encrypted-user-uuid-auth-handoff.md) | `POST /auth/encrypted-user-uuid` is the intended **cross-client / SSO-handoff identity mechanism**: given an encrypted `{client, user}` UUID pair, it mints a fr | api2/Component/Api/CrossClient/CrossClient.php |
 | [Health-check endpoint (/health liveness short-circuit)](features/health-check-endpoint.md) | `_Controller_Index::api()` short-circuits **liveness/health-probe** requests to an HTTP 200 **before** any routing, DB bootstrap, or V2 engine work runs. | api2/Controller/Index.php |
 | [Language Translation Layer (audience.language + sidecar tables)](features/language-translation-layer.md) | Serves the same TOGa data (Item title/description/longDescription, plus item **feature** text — `Features.name`, `ItemCategoryFeatureGroups.name`, `ItemFeatures | api2/Component/Api/V2/V2.php, api2/Component/Api/V2/Response/Response.php, _underscore/Model/Core/Setting.php, _underscore/Model/Core/RecordField.php, _underscore/Model/Core/DefaultGlobalSetting.php, _underscore/Model/Client/ItemTranslation.php, _underscore/Model/Client/FeatureTranslation.php, _underscore/Model/Client/ItemCategoryFeatureGroupTranslation.php, _underscore/Model/Client/ItemFeatureTranslation.php, dbchanges2/Client/2026-06-23a - ItemTranslations.sql, dbchanges2/Client/2026-06-23b - ItemTranslationsAcl.sql, dbchanges2/Client/2026-07-13a - FeatureTranslations.sql, dbchanges2/Client/2026-07-13b - FeatureTranslationsAcl.sql, dbchanges2/Core/2026-06-23a - RecordFieldsTranslationColumn.sql, dbchanges2/Core/2026-06-23b - ItemTranslationsRecord.sql, dbchanges2/Core/2026-07-13 - FeatureTranslationsRecord.sql |

package/knowledge/2.0/apps/api2/features/cross-client-data-retrieval.md

@@ -5,23 +5,29 @@ repo: api2
 project: API
 client: shared
 type: feature
-status: draft
-updated: 2026-07-27
+status: active
+updated: 2026-07-28
 owners: [jcardinal]
 files:
   - api2/Component/Api/CrossClient/CrossClient.php
   - api2/Component/Api/V2/V2.php
   - api2/Controller/Index.php
-  - api2/_.php
   - _underscore/Model/Cache/Table.php
   - _underscore/Model/Cache/Tables/Client.php
-  - dbchanges2/Core/2026-06-30b - CacheClusterRegistrationAndRecordTtl.sql
+  - _underscore/Model/Core/Record.php
+  - worker2/Worker/Platform/Cache.php
+  - worker2/Controller/Index.php
+  - worker2/_.php
   - dbchanges2/Cache/2026-06-30a - MultiClientCacheTables.sql
+  - dbchanges2/Core/2026-06-30b - CacheClusterRegistrationAndRecordTtl.sql
+  - dbchanges2/Core/2026-07-27a - PlatformCacheCleanCron.sql
 related:
   - ./encrypted-user-uuid-auth-handoff.md
+  - ../../worker2/features/platform-cache-cleanup.md
   - ../../_underscore/features/per-client-database-connections.md
   - ../../_underscore/features/acl-permission-chain.md
   - ../../_underscore/features/database-alias-repointing.md
+  - ../../_underscore/features/model-magic-field-access.md
 ---
 
 ## What it is
@@ -33,13 +39,13 @@ single-client path is byte-for-byte unchanged.
 
 Because client DBs live on **different clusters**, a cross-DB JOIN/UNION is impossible. The feature
 is a **scatter-gather** engine: it fans out to each entitled client's own V2 API, merges the
-streams, and caches the result per-user on a dedicated Cache cluster.
+streams with a watermark, and caches the merged result per-user/per-query on a dedicated Cache
+cluster.
 
-> **Status: code-complete + reviewer-hardened, NOT yet runtime-tested.** Result rows are stored as
-> a JSON `rowData` column (deliberate v1 deviation from real typed columns). Phases 4 (single-record
-> cross-client), 5 (worker2 page-ahead prefetch), and 7 (live load test / verify `_Query`/`_Database`
-> method names against a running instance) are open. Per-client token hardening (§0) is an
-> owner-accepted deferred risk.
+> **Status: working end to end on local dev** (multi-client fan-out, ACL-filtered `client` object,
+> typed multi-column sort, keyset paging, narrowed `fields`, per-query cache isolation).
+> **Not yet deployed** — next stop is external testing. See
+> [api2 architecture](../architecture.md) for the accepted risks carried into that test.
 
 ## How it works
 
@@ -47,84 +53,168 @@ Entry point is `_Component_Api_CrossClient` (`api2/Component/Api/CrossClient/Cro
 delegated to from a guard at the **top** of the `ACTION__LIST` case in
 `_Component_Api_V2::processRoutePairs()`: when `isCrossClientRequest($httpOptions)` is true (a
 non-empty `client` option), it calls `handleListing()`, merges result rows into `$outData` and meta
-into `$this->response->meta`, sets status 200, and breaks. The guard is **inert** on ordinary
-single-client GETs.
+into `$this->response->meta`, and breaks. The guard is **inert** on ordinary single-client GETs.
 
 1. **resolveScope** — the caller's entitled clients resolve via the `Users_Clients` bridge
    (`Users.homeClientUserId` maps one home user to per-client user records across tenant DBs); each
-   target yields a minted per-client identity.
+   target yields a minted per-client identity. This is **serial** — one DB context switch + query
+   per permitted client, on every request including cache hits.
 2. **queryHash** — SHA-256 of the normalized query; the cache key.
-3. **Build lock** — a named `GET_LOCK` (`tablecache:<clientId>:<userId>:<hash>`) prevents duplicate
-   concurrent builds.
+3. **Build lock** — a named `GET_LOCK` over a **fixed-width digest** of client/user/hash.
 4. **Fan out** — a two-phase concurrent HTTP handshake per client (see
    [encrypted-user-uuid-auth-handoff](./encrypted-user-uuid-auth-handoff.md)): Phase A mints an
    access token, Phase B issues the authenticated GET.
-5. **Watermark k-way merge** — client streams are merged into a globally-correct sort order using a
-   watermark (`safeRowCount`/`keyBeyond`); a `deepen` loop pulls more from lagging streams, capped
-   by a 50-iteration backstop.
-6. **Cache build/serve** — rows are cached on the Cache cluster; `servePage` emits a keyset page.
-
-### Total order + keyset pagination (V2 query builder, Phase 0a/0b)
-
-Cross-client global sort correctness requires each client's stream to be a **total order**:
-- **0a** — the LIST query builder always appends a **primary-key ASC tiebreaker** to `ORDER BY`
-  (also improves single-client determinism).
-- **0b** — a new keyset/seek pagination mode runs alongside offset mode; it engages **only** when
-  the caller passes `seekValue` + `seekId` options. It injects a seek `WHERE` predicate (V2.php
-  ~line 3782) and switches `LIMIT` from offset to `recordsPerPage` (~line 3861). The composite
-  unique sort key is `(sortfield, clientId, primaryKeyId)`.
-
-### Query-string forwarding (load-bearing gotcha)
-
-V2 parses list options (`fields`/`where`/`sort`/`join`/`group`…) from the **raw** query-string
-values into structured `$httpOptions` — re-serializing the parsed array does **not** round-trip.
-The fan-out therefore forwards the caller's **original** `$_SERVER['QUERY_STRING']` through an
-explicit **allowlist** of forwardable keys (`fields, where, sort, group, distinct, join, ojoin,
-depth, calcDepth, _`), re-stamping `recordsPerPage`, `seekValue`/`seekId`, and a fresh
-`transactionId` per client. The allowlist (not a denylist) prevents internal flags like
-`surface`/`debug` from leaking into sub-requests.
-
-### Cache cluster
-
-A new dedicated **Cache** cluster (Core-style shared cluster, `Databases` id **145**, alias
-`DB_CACHE`, `CACHE_DATABASE_ID = 145`) keeps cache churn off Core. It is region-aware
-(1=us-east-1, 2=us-west-2, 3=eu-west-1) with per-region `DatabaseHosts`; `hostCluster` values are
-DevOps TBD. Boot registration lives in `api2/Controller/Index.php`; the alias const in `api2/_.php`.
-See [per-client-database-connections](../../_underscore/features/per-client-database-connections.md)
-for how it sits among the shared/per-client clusters.
+5. **Watermark k-way merge** — client streams merge into a globally-correct sort order behind a
+   watermark; a `deepen` loop pulls more from the lagging stream, capped by a 50-iteration backstop
+   plus a **stall guard** (a pass that returns no new rows with nobody exhausted logs and breaks).
+6. **Cache build/serve** — merged rows are written to a per-query result table on the Cache
+   cluster; `servePage` emits a keyset page.
+
+### Row identity is the record `uuid` — never the numeric id
+
+A V2 LIST response **never exposes `id`**. The original design keyed cached rows on
+`primaryKeyId` + `seekId`, which could not work: every row arrived with `id = 0`, collided on the
+unique key, and the cache retained exactly one row per page. Result rows are now keyed on
+**`rowUuid CHAR(36)`** and the keyset cursor tiebreaker is **`seekUuid`**. The default sort is
+`uuid` (it used to be `id`, a field the API never returns → null sort values and a cursor that
+could not advance).
+
+### Per-query result tables with TYPED sort columns
+
+Each cached query gets its own `TableResults_<Tables.id>` table carrying **one typed column per
+requested sort field** — type taken from the model field, or `FIELDOPT_SQL_TYPE` for calculated
+`FIELD_SQL` fields, defaulting to `CHAR`. Tables are per-**query**, not per record type, precisely
+so the sort columns can be typed for that query. The index declares per-column **ASC/DESC
+directions matching the ORDER BY**; without that, any DESC sort filesorts the entire table.
+
+### Multi-column keyset seek (V2)
+
+The seek generalised from single-column `seekValue` + `seekId` to **`seekValues`** (a JSON array)
++ **`seekUuid`**: a lexicographic predicate with NULL-safe `<=>` links and typed literals, tie-broken
+on `CAST(uuid AS BINARY)`. The fan-out must send `sort` **explicitly** (rebuilt from the resolved
+sort specs) — V2 only engages its keyset when the sub-request itself carries sort terms, and the
+term count must match the `seekValues` tuple.
+
+### Deepen-on-demand paging + watermark-only deepening
+
+`Tables.safeRecordCount` records how deep the cache is **provably complete**. The raw stored row
+count must **never** be used for that decision: a fast client can leave rows cached that a lagging
+client would displace. Duplicate rows are de-duplicated by `clientId|rowUuid` before counting —
+double-counting inflated `safeRecordCount` (25 real rows reported as 50), so later pages skipped
+deepening and returned empty pages.
+
+**Only the watermark-holding client is deepened per pass.** A client whose cursor is already past
+the watermark cannot make any row safe, so querying it is a wasted round trip. At 1000 clients that
+is ~1 sub-request per pass instead of 1000 — the single biggest efficiency property of the design.
+
+### Per-row `client` object (ACL-filtered)
+
+Every returned row carries a `client` object filtered by the caller's **CORE** roles
+(`id.core.roles`) against the Core `Clients` record. `getAclFieldPermissions()` interprets role ids
+"in the perspective of `$record->aclDatabase`", so a CORE record requires **core** role ids — client
+role ids silently yield nothing.
+
+### Narrowed `fields`
+
+The fan-out injects `uuid` + the sort fields into each sub-request (the merge cannot identify or
+order rows without them) and strips them back out of the returned rows. This resolves **per
+request**, not per fan-out, so cache hits strip too.
+
+### Failure surfacing
+
+Sub-request failures reach the caller: `meta.crossClient.failures[]` plus an ERROR per failed client
+in `messages[]`, and the response no longer reports 200/success when **every** in-scope client
+failed. Previously a 500 inside a sub-request returned 200 with an empty page.
+
+Cross-client `meta` now mirrors the single-client LIST meta exactly — `depth`, `calcDepth`, `page`,
+`prevPage`, `nextPage`, `totalPageCount`, `recordsPerPage`, `pageRecordCount`, `totalRecordCount`,
+`prohibitedRecordCount` — plus the `crossClient` block.
+
+### Cache cluster & cleanup
+
+The dedicated **Cache** cluster is registered for **all** environments, and both api2 and worker2
+resolve it **by name** (`Databases.name = 'Cache'`). There is deliberately **no
+`CACHE_DATABASE_ID` constant** — `Databases.id` is auto-assigned per Core instance and differs per
+environment, so a hardcoded id works only where it happens to match and fails silently everywhere
+else. This was caught as a deploy blocker.
+
+Expired `Tables` rows and orphaned `TableResults_*` tables are reaped by
+[`_Worker_Platform_Cache::Clean()`](../../worker2/features/platform-cache-cleanup.md) on a `*/5`
+cron.
 
 ## Key rules
 
 - **Triggered only by a non-empty `client` option** — single-client GETs are unaffected.
-- **Each target client's ACL is honored** via a minted per-client identity, not a global aggregate
-  view. This preserves ACL fidelity across tenants.
-- **Every outbound sub-request needs its own globally-unique `transactionId`** — the API enforces
-  uniqueness; it is the load-bearing idempotency/audit key across Core + Client logs.
-- **Forward the original query string, not the parsed options** — parsed `$httpOptions` do not
-  round-trip through re-serialization.
-- `MAX_CONCURRENT_CLIENT_FETCHES = 25` bounds fan-out concurrency (PHP-FPM has no in-process
-  concurrency, so transport is an HTTP pool via `curl_multi`).
+- **Row identity is the `uuid`**, never the numeric primary key. Chosen over exposing internal ids
+  over HTTP, consistent with the platform never emitting `id`.
+- **Cross-client is USER-LEVEL ONLY.** API credentials (`Client.Apis`) have no cross-client
+  entitlement bridge — `Users_Clients` is user-keyed and there is no `Apis_Clients` equivalent.
+  `homeClientApiId` was dropped from the schema entirely.
+- **`Records.ttlCache = 0` means "effectively uncached" but is FLOORED at 1 minute**, so a cache
+  entry always outlives the request that built it and the cleanup worker cannot delete rows
+  mid-read.
+- **String/uuid comparison uses BYTE ordering** — `utf8mb4_bin` in the cache, `CAST(... AS BINARY)`
+  in client queries — because the PHP-side watermark merge compares bytes. Any linguistic collation
+  would order differently and the merge could certify rows complete that MySQL then re-orders.
+- **Never use the raw stored row count** to decide paging depth; only `safeRecordCount`.
+- **Every outbound sub-request needs its own globally-unique `transactionId`.**
+- **Forward the caller's original query string verbatim** — parsed `$httpOptions` do not round-trip
+  (see the url-decode gotcha below). `MAX_CONCURRENT_CLIENT_FETCHES = 25` bounds fan-out.
 
 ## Gotchas
 
-- **Cross-client output row shape must match single-client.** Served rows are cast to `(object)` in
-  `servePage` so payload interceptors reading `$row->prop` don't throw — the single-client path
-  emits `(object)$outRow`.
-- **The X-Cross-Client / X-Cross-User custom-header transport was a dead stub** — the V2 engine
-  never reads such headers. The real transport is the two-phase encrypted-UUID auth handshake.
-- **Switching the `Client` DB alias mid-request used to silently not switch.** `CrossClient.php`
-  (~lines 193/201/207) uses `_Database::registerClientDatabases()` + `_underscore::DB_CLIENT`, and
-  `_Database` keys its live connection state by **alias**, not physical schema — so cross-client
-  reads were executing against the *home* client's DB and returning 0 rows. Fixed in
-  `_underscore/Database.php` (no api2 edit needed); see
+- **V2 NEVER url-decodes its query string.** It splits `QUERY_STRING` on `&` and `=` and consumes
+  the raw values. Anything containing `,`, `[`, `"`, `%`, `+` or a space arrives mangled — this bit
+  the feature twice (the JSON `seekValues` array, and a comma-separated `fields` list corrupted into
+  a single field literally named `number%2Cuuid`). Corollary: a fan-out must forward options
+  **verbatim**, never `parse_str` → `http_build_query`.
+- **V2 builds `$httpOptions` from an explicit allowlist.** `client`, `seekValues` and `seekUuid`
+  were never parsed into it, so the cross-client branch was unreachable and the keyset could never
+  engage. Adding an option means adding it to that allowlist.
+- **Client databases do NOT share a charset** — `utf8mb4_unicode_ci`, `utf8mb4_0900_ai_ci` and
+  `latin1_swedish_ci` were all observed on the same column across tenants. Any hardcoded `COLLATE`
+  is invalid somewhere (`COLLATION ... is not valid for CHARACTER SET 'latin1'`). Use
+  `CAST(... AS BINARY)`.
+- **A client's database NAME must come from `Core.Clients.clientDatabaseId` → `Core.Databases.name`**
+  — never built as `'Client_' . clientIdentifier`. They diverge in practice (identifier
+  `Compass_Usa`, database `Client_Compass`).
+- **MySQL `GET_LOCK` names cap at 64 chars and ERROR rather than truncate.** The original lock name
+  was 80 chars.
+- **`_Query::fetchRows()` returns OBJECTS**, not arrays. CrossClient read them with array syntax
+  throughout.
+- **A V2 LIST nests records under the route key** (`data: {"salesOrders": [...]}`);
+  `parseClientResponse` required `data` to be a flat array and discarded every successful
+  sub-response as unusable.
+- **`_Model` writes every declared field explicitly, so column defaults never apply.**
+  `Tables.safeRecordCount` is NOT NULL and `buildCache` never assigned it. Likewise a Core column
+  (`Records.ttlCache`) that exists in SQL but is not declared on the model makes `__get` throw.
+- **Dangling foreign keys used to fail an entire listing.** `getFullModelData()` threw out of
+  `_Model::initialize()` when a referenced row did not exist; one orphaned pointer killed the whole
+  page. It now logs and expands the relationship to `null`. **This affects every REST listing/GET in
+  api2, not just cross-client.**
+- **`empty()`/`isset()` on a `_Model` field is unreliable** and fails in the "looks unauthenticated"
+  direction — see [_Model magic-field access](../../_underscore/features/model-magic-field-access.md).
+- **Switching the `Client` DB alias mid-request** needs the park/restore path — `_Database` keys
+  live connection state by **alias**, not physical schema. See
   [Re-pointing a DB alias mid-request](../../_underscore/features/database-alias-repointing.md).
 - **`curl_multi` busy-spin guard** — both multi loops `usleep(100)` when
   `curl_multi_select() === -1`.
 
 ## Change history
+- 2026-07-28 — Took the feature from never-executed to working end to end. Row identity moved from
+  the numeric id to `rowUuid`/`seekUuid` (a V2 LIST never returns `id`); per-query
+  `TableResults_<id>` tables with typed, direction-indexed sort columns; multi-column `seekValues`
+  keyset; `safeRecordCount` watermark with de-dup by `clientId|rowUuid` and watermark-only
+  deepening; ACL-filtered per-row `client` object (CORE roles); narrowed `fields`; sub-request
+  failures surfaced in `meta.crossClient.failures[]` + `messages[]`; Cache cluster registered for
+  all environments and resolved **by name** (no `CACHE_DATABASE_ID`); worker2 cache reaper on a
+  `*/5` cron. Cross-client declared **user-level only** (`homeClientApiId` dropped). Fixed a dozen
+  latent runtime bugs (namespace, option allowlist, 80-char `GET_LOCK`, undeclared `ttlCache` field,
+  unassigned `safeRecordCount`, route-key-nested `data`, `id` default sort, missing explicit `sort`,
+  duplicate counting, object-vs-array `fetchRows`, dangling-FK listing failure). (jcardinal)
 - 2026-07-27 — Root-caused cross-client reads hitting the home client's DB: `_Database` keys live
   connection state by alias, so re-registering `DB_CLIENT` never swapped the open link. Fixed in
   `_underscore/Database.php`; CrossClient needed no edit. (jcardinal)
 - 2026-07-07 — Initial capture: scatter-gather cross-client retrieval engine (orchestrator, watermark
-  k-way merge, keyset pagination Phase 0a/0b, `client`-option delegation, Cache cluster id 145).
+  k-way merge, keyset pagination, `client`-option delegation, Cache cluster).
   Code-complete + reviewer-hardened, not yet runtime-tested. (jcardinal)

package/knowledge/2.0/apps/api2/features/encrypted-user-uuid-auth-handoff.md

@@ -6,7 +6,7 @@ project: API
 client: shared
 type: feature
 status: active
-updated: 2026-07-07
+updated: 2026-07-28
 owners: [jcardinal]
 files:
   - api2/Component/Api/CrossClient/CrossClient.php
@@ -22,6 +22,12 @@ without the caller holding the target client's credentials. It is what the scatt
 [cross-client data retrieval](./cross-client-data-retrieval.md) engine uses to authenticate each
 per-client sub-request.
 
+> **Correction (2026-07-28): the route is ORIGIN-GATED and browser-shaped.** It resolves the
+> `Origin` header against `Core.Domains` to establish `appId`/`clientId` *before* minting, and
+> rejects an unknown origin with **EN-6**. It therefore **cannot** be used server-to-server without
+> presenting a domain registered for the target client. Earlier revisions of this doc described it
+> as the cross-client engine's auth mechanism without noting this constraint — plan for it.
+
 ## How it works
 
 - The encrypted `{client, user}` pairs are produced during **normal** auth: `/auth/api` and
@@ -41,6 +47,12 @@ per-client sub-request.
 
 ## Key rules
 
+- **Origin gating is mandatory.** The `Origin` header must resolve to a `Core.Domains` row for the
+  target client; that lookup — not the caller's own session — establishes the `appId`/`clientId` the
+  token is minted under. Unknown origin → **EN-6**.
+- **The minted token's `appId` is the resolved DOMAIN's app, not the caller's app.** Observed in
+  cross-client fan-out: sub-request tokens carried `appId 1` (TOGa Supply) while the caller was
+  Talos. Harmless for reads today; it must be revisited before anything ACLs on `appId`.
 - **Identity source of truth is the `Users_Clients` bridge** — `Users.homeClientUserId` maps one
   home user to its per-client user records across tenant DBs; that is where the per-client UUIDs
   come from.
@@ -50,5 +62,10 @@ per-client sub-request.
   tolerates the rotation set.
 
 ## Change history
+- 2026-07-28 — Corrected: the route is **origin-gated** (resolves `Origin` against `Core.Domains`
+  to establish `appId`/`clientId`, EN-6 on unknown origin), so it is browser-shaped and cannot be
+  used server-to-server without a registered domain for the target client. Also recorded that the
+  minted token's `appId` follows the resolved domain's app, not the caller's. Found while getting
+  the cross-client fan-out running end to end. (jcardinal)
 - 2026-07-07 — Documented `/auth/encrypted-user-uuid` as the cross-client/SSO identity-handoff route
   and its two-phase use by the cross-client fan-out engine. (jcardinal)

package/knowledge/2.0/apps/worker2/INDEX.md

@@ -24,10 +24,11 @@
 | [Background Email-Template Worker (_Worker_Notification_EmailTemplate)](features/notification-email-template.md) | `_Worker_Notification_EmailTemplate::Send(...)` dispatches a **stored, client-defined `EmailTemplates` row off-thread** as a background WorkerJob. | worker2/Worker/Notification/EmailTemplate.php, worker2/Worker/Client/True.php, _underscore/Model/Client/EmailTemplate.php |
 | [DB-Driven Notification (Internal) Email](features/notification-email.md) | Internal/notification emails (merge-conflict alerts, ops notices — anything system-generated, not client-facing transactional mail) are sent through one worker | worker2/Worker/Notification/Email.php, _underscore/Model/Client/EmailTemplate.php, dbchanges2/Client/2026-06-23a - EmailTemplateWrapper.sql, dbchanges2/Client_True/2026-06-23a - EmailTemplateWrapper.sql |
 | [OneUptime push-metric monitors for 2.0 workers](features/oneuptime-worker2-monitoring.md) | A second, **OneUptime-reporting** monitoring pattern for the 2.0 worker2 tier, ported from the 1.0 `App_SystemMonitor_Compass` monitors. | worker2/Worker/Monitor/Compass.php, worker2/Worker/Client/Compass.php, worker2/composer.json, _underscore/Cloud.php |
+| [Platform Cache Cleanup (_Worker_Platform_Cache::Clean)](features/platform-cache-cleanup.md) | `_Worker_Platform_Cache::Clean()` is the reaper for the shared **Cache** cluster that backs api2's [multi-client data retrieval](../../api2/features/cross-clien | worker2/Worker/Platform/Cache.php, worker2/Controller/Index.php, worker2/_.php, dbchanges2/Core/2026-07-27a - PlatformCacheCleanCron.sql |
 | [Startech Webhook Handler (worker2)](features/startech-webhook-handler.md) | Receives inbound webhook events from Startech (Easeedesk) and creates or updates the corresponding ticket in TOGA 2.0. | worker2/Worker/Startech.php |
 | [Talos (TOGa IQ) Meeting-Notes Integration & Token Auto-Refresh (consumer)](features/talos-meeting-notes-integration.md) | How a **dev tool / agent consumes Talos (TOGa IQ)** to query the team meeting-notes corpus programmatically. | .claude/skills/plan-ticket/scripts/talos.js |
 | [Talos Pricing Automation (worker2 Cron — AWS Actuals, Calibration, Monthly Report)](features/talos-pricing-automation.md) | The worker2 half of the **Talos Pricing Platform** (see the talos `pricing-cogs-model` and tools `talos-pricing-ui` docs for the other halves). | worker2/Worker/Talos/Pricing.php, worker2/Database/TalosPricingCrons.sql |
-| [Talos Transcript Ingestion Pipeline (worker2 → AWS Bedrock KBs)](features/talos-transcript-ingestion.md) | `_Worker_Team_Transcripts` runs a fully automated, cron-driven pipeline that ingests raw Teams transcripts into the **Talos / TOGa IQ** AWS Bedrock knowledge ba | worker2/Worker/Team/Transcripts.php, worker2/bin/sync-knowledge-bases.php, worker2/Config/production.ini, worker2/Database/TeamsTranscriptExports.sql, dbchanges2/Team/2026-06-30a, dbchanges2/Team/2026-06-30b, dbchanges2/Team/2026-06-30c, dbchanges2/Team/2026-06-30d, dbchanges2/Team/2026-06-30e, dbchanges2/Core/2026-06-30a, dbchanges2/Core/2026-07-02a, dbchanges2/Team/2026-07-02a, dbchanges2/Team/2026-07-08a, dbchanges2/Team/2026-07-09a, dbchanges2/Team/2026-07-10a |
+| [Talos Transcript Ingestion Pipeline (worker2 → AWS Bedrock KBs)](features/talos-transcript-ingestion.md) | `_Worker_Team_Transcripts` runs a fully automated, cron-driven pipeline that ingests raw Teams transcripts into the **Talos / TOGa IQ** AWS Bedrock knowledge ba | worker2/Worker/Team/Transcripts.php, worker2/bin/sync-knowledge-bases.php, worker2/Config/production.ini, worker2/Database/TeamsTranscriptExports.sql, dbchanges2/Team/2026-06-30a, dbchanges2/Team/2026-06-30b, dbchanges2/Team/2026-06-30c, dbchanges2/Team/2026-06-30d, dbchanges2/Team/2026-06-30e, dbchanges2/Core/2026-06-30a, dbchanges2/Core/2026-07-02a, dbchanges2/Team/2026-07-02a, dbchanges2/Team/2026-07-08a, dbchanges2/Team/2026-07-09a, dbchanges2/Team/2026-07-10a, dbchanges2/Team/2026-07-28a - TranscriptProcessingRetryAttempts.sql, dbchanges2/Team/2026-07-28b - TranscriptPromptTemplateConverseModel.sql, dbchanges2/Core/2026-07-28a - TeamsTranscriptRetryCron.sql |
 | [Team Sprint Management & Reporting](features/team-sprint-management.md) | `_Worker_Team_Sprint` (file `Worker/Team/Sprint.php`) is the engine behind TOGA's internal **development-sprint process and reporting**. | worker2/Worker/Team/Sprint.php, _underscore/Model/Team/Sprint.php, dbchanges2/Core/CronJobs (SprintLockScheduled seed) |
 | [Teams Meeting Transcript Export](features/teams-transcript-export.md) | > **SUPERSEDED (2026-07-09) — the S3-staging model below is history.** `Export` is now a thin > **GRAPH-DIRECT** cron poller: it no longer archives raw VTT to ` | worker2/Worker/Team/Transcripts.php, worker2/Config/production.ini, worker2/Database/TeamsTranscriptExports.sql, dbchanges2/Core/2026-06-18a - Teams Transcript Export schedule.sql |
 | [VAPI Webhook Handler (worker2 — AI-BDR end-of-call processing)](features/vapi-webhook-handler.md) | `_Worker_Vapi` ([worker2/Worker/Vapi.php](worker2/Worker/Vapi.php)) is the **PHP side of the AI-BDR call loop** — the webhook that receives VAPI's end-of-call r | worker2/Worker/Vapi.php, worker2/Worker/Ai/Bdr/Vapi.php |

package/knowledge/2.0/apps/worker2/features/platform-cache-cleanup.md

@@ -0,0 +1,69 @@
+---
+title: Platform Cache Cleanup (_Worker_Platform_Cache::Clean)
+framework: "2.0"
+repo: worker2
+project: Worker
+client: shared
+type: feature
+status: active
+updated: 2026-07-28
+owners: [jcardinal]
+files:
+  - worker2/Worker/Platform/Cache.php
+  - worker2/Controller/Index.php
+  - worker2/_.php
+  - dbchanges2/Core/2026-07-27a - PlatformCacheCleanCron.sql
+related:
+  - ../../api2/features/cross-client-data-retrieval.md
+  - ./creating-worker-actions.md
+  - ../../_underscore/features/per-client-database-connections.md
+---
+
+## Summary
+
+`_Worker_Platform_Cache::Clean()` is the reaper for the shared **Cache** cluster that backs api2's
+[multi-client data retrieval](../../api2/features/cross-client-data-retrieval.md). It deletes
+expired `Cache.Tables` rows and drops the per-query `TableResults_<Tables.id>` tables that no
+longer have a parent. Without it the Cache schema grows one table per distinct cross-client query,
+forever.
+
+Action path `Platform/Cache/Clean`, scheduled on a **`*/5`** cron registered in
+`dbchanges2/Core/2026-07-27a - PlatformCacheCleanCron.sql`.
+
+## How it works
+
+1. **Reap expired parents.** Delete `Cache.Tables` rows whose TTL has elapsed, with a **2-minute
+   grace period** so a request that is mid-read cannot have its rows deleted out from under it.
+   This pairs with the api2 side flooring `Records.ttlCache = 0` at 1 minute — together they
+   guarantee a cache entry outlives the request that built it.
+2. **Drop orphans.** Enumerate `TableResults_*` tables and drop the ones with no surviving parent
+   row. Ordering is load-bearing: the parent row is **deleted and committed first**, and orphan
+   status is **re-verified on the write host immediately before each `DROP`** — a `DROP TABLE` is
+   irreversible and non-transactional, so a stale read replica must never be the basis for one.
+
+## Key rules
+
+- **Delete-parent-then-drop-child, and re-check on the writer.** Never drop a result table based on
+  a list read earlier in the run or from a reader host.
+- **Never shorten the grace period below the longest plausible in-flight request.** The whole
+  correctness argument for reaping while requests are live rests on it.
+- The Cache DB is registered as `DB_CACHE` in `worker2/_.php` and resolved **by name**
+  (`Databases.name = 'Cache'`) in `worker2/Controller/Index.php` — the same rule as api2. There is
+  deliberately no `CACHE_DATABASE_ID`: `Databases.id` is auto-assigned per Core instance and a
+  hardcoded id fails silently in every environment where it does not happen to match.
+
+## Gotchas
+
+- **Worker classes are `abstract class _Worker_<Path>` with `public static` entry methods**,
+  dispatched as `$className::$functionName()`. They do **not** `extends _Worker` — that is the
+  dispatcher. See [Creating Worker Actions](./creating-worker-actions.md). (The 2.0
+  `standards/framework-rules.md` worker contract still describes the older `extends _Worker` +
+  `run()` shape and needs reconciling.)
+- **Adding the cron is a dbchanges2 change, not a code change** — the schedule lives in Core, so a
+  new worker action is not actually scheduled until that SQL is applied per environment.
+
+## Change history
+- 2026-07-28 — Initial capture: `_Worker_Platform_Cache::Clean()` reaps expired `Cache.Tables` rows
+  (2-minute grace) and drops orphaned `TableResults_*` tables (parent deleted+committed first,
+  orphan status re-verified on the write host immediately before each DROP); `DB_CACHE` registered
+  by name in worker2 boot; `*/5` cron added in dbchanges2 `Core/2026-07-27a`. (jcardinal)

package/knowledge/2.0/apps/worker2/features/talos-transcript-ingestion.md

@@ -24,6 +24,9 @@ files:
   - dbchanges2/Team/2026-07-08a
   - dbchanges2/Team/2026-07-09a
   - dbchanges2/Team/2026-07-10a
+  - dbchanges2/Team/2026-07-28a - TranscriptProcessingRetryAttempts.sql
+  - dbchanges2/Team/2026-07-28b - TranscriptPromptTemplateConverseModel.sql
+  - dbchanges2/Core/2026-07-28a - TeamsTranscriptRetryCron.sql
 related:
   - ./teams-transcript-export.md
   - ./notification-email-template.md
@@ -56,7 +59,7 @@ JSON that PHP renders to an inline-styled HTML recap email, enqueued to the orga
 
 ## Key files / entry points
 
-- `Worker/Team/Transcripts.php` — actions `Export`, `Process`, `SyncKb`,
+- `Worker/Team/Transcripts.php` — actions `Export`, `Process`, `Retry`, `SyncKb`,
   `SyncKnowledgeBases`. **`Scan` was removed.**
 - `bin/sync-knowledge-bases.php` — CLI entry for the Bedrock→`Team.KnowledgeBases` registry
   sync (`--dry-run` supported).
@@ -111,6 +114,53 @@ the Graph date filter and gets stuck "observing" forever.
 `alreadyExported()` and `recordExport()` were **removed** (superseded by the observation
 helpers).
 
+### `Retry(?int $limit = null, array $cc = [], array $bcc = [])` (cron) — bounded re-drive of failures
+Added 2026-07-28. Before it existed, a failed `Process()` was **permanently terminal** (see
+Gotchas). `Retry` sweeps `Team.TranscriptProcessing` for `status='failed'` and re-enqueues
+`Team/Transcripts/Process` **directly from the `Team.TranscriptExports` ledger row**, which
+already holds every Graph id `Process` needs.
+
+Two deliberate non-behaviors:
+- It does **not** clear `dtEnqueued`, and it does **not** route back through `Export`. Going
+  through `Export` would re-download every VTT purely to re-measure its size and would perturb
+  the observe-until-stable state; the ledger is sufficient.
+- Nothing "resets" a failure — the row stays `failed` until a `Process` run succeeds.
+
+**Bounded on three axes** so a permanently unprocessable transcript *parks itself* instead of
+retrying forever (budget sized for a multi-day AI-endpoint outage, not a blip):
+- `RETRY_MAX_ATTEMPTS = 20`
+- doubling backoff: `RETRY_BACKOFF_MINUTES = 15`, doubling up to `RETRY_BACKOFF_CAP_MINUTES = 360`
+- `RETRY_MAX_AGE_DAYS = 30`
+
+**Atomic claim (do not "simplify" this).** `claimRetryAttempt()` increments the attempt counter
+with a status-guarded UPDATE — `... WHERE transcriptIdentifier = ? AND status = 'failed'` — reads
+`_Query::getAffectedRows()` **before** the commit (affected-rows reflects the last statement on
+the connection), and **skips the enqueue when zero rows were claimed**. `loadRetryCandidates()`
+takes no locks and `Process()` only short-circuits when status is already `'synced'`, so an
+unconditional increment let two overlapping sweeps both run the full pipeline for one meeting:
+two AI passes, two Bedrock syncs, and **two recap emails to the organizer**. Claim-then-enqueue is
+the pattern for making any sweep-and-requeue worker safe without locking.
+
+**Unprocessable rows are excluded in the candidate SQL, not skipped in the loop.** A ledger row
+with no `meetingIdentifier` can never succeed (`Process()` throws without one); leaving such rows
+selectable let a handful fill the `LIMIT` on every sweep and starve rows that could succeed. They
+are surfaced separately by `countUnprocessableFailures()` in the sweep summary.
+
+Private helpers: `loadRetryCandidates()`, `claimRetryAttempt()`, `countUnprocessableFailures()`,
+`toIsoUtc()`.
+
+**Schema (`Team/2026-07-28a`):** `Team.TranscriptProcessing.retryAttempts`
+(`TINYINT UNSIGNED NOT NULL DEFAULT 0`), `dtLastRetried` (`DATETIME NULL`), plus a
+`(status, retryAttempts)` index. The defaults make every already-failed row eligible on the first
+sweep, so **no backfill or one-off UPDATE is needed** (an explicit requirement). The retry join
+needs no new index — `TranscriptExports.transcriptIdentifier` already has the `uq_transcript`
+UNIQUE index.
+
+**Cron (`Core/2026-07-28a`): `10,40 9-18 * * 1-5`** — deliberately offset from the `Export` cron
+(`15,45 9-18 * * 1-5`, `Core.CronJobs` id 18) so the two never contend for the Graph token or the
+AI endpoint in the same minute, and kept inside Export's business-hours window because a re-drive
+emails a recap to the meeting organizer.
+
 ### `Process(int $transcriptId, string $graphUserId, string $meetingId, string $organizerEmail, string $subject, string $meetingStartIso, array $cc = [], array $bcc = [])`
 1. Download the VTT **straight from Graph** (no S3 round trip).
 2. `stripAndGroupVtt()` — strips WebVTT timestamps and **merges consecutive same-speaker cues
@@ -225,6 +275,48 @@ part of the ingestion loop** (raw reads removed). Credential values live only in
 
 ## Gotchas / known issues
 
+- **THE AI MODEL FOR THIS PIPELINE COMES FROM THE DATABASE, NOT `production.ini`.** `Process()`
+  resolves it as `($template->model ?? _Config::talos('model'))` against the **active
+  `Team.TranscriptPromptTemplate` row**, and because that table's `model` column is
+  `VARCHAR(255) NOT NULL`, the ini value is **unreachable — dead fallback on this path**. Editing
+  `worker2/Config/production.ini` does nothing here. Both AI passes (`callAICleaningAPI`
+  clean/classify and `buildRecapEmailBody` recap) read the **same row**, so one row update fixes
+  both. There is **no admin screen** for the prompt template in `tools/mvc/talos/`, so a model
+  change belongs in a dbchanges2 migration like its original seed (`Team/2026-06-30c`). Write such
+  a migration to match on the **OLD model value** (not id/uuid) so it no-ops in an environment
+  already hand-corrected and cannot clobber one deliberately pinned to a different model.
+  This was the root cause of a five-day outage: Talos `/api/ai/generate` returned
+  `500 {"error":"internal_error"}` for `bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0`
+  because the endpoint now requires the **`bedrock_converse/` prefix** (verified against
+  `Logs_True.Api` id 5581286, `dtStamp 2026-07-28 09:45:54`, route `/api/ai/generate`).
+  Fixed by `Team/2026-07-28b`.
+- **A failed `Process()` used to be PERMANENTLY TERMINAL — "just re-run it" never recovered it.**
+  Three mechanisms combined: (a) workers return **HTTP 200 even on failure** by framework
+  contract, so SQS never redelivers; (b) `Export()` stamps `TranscriptExports.dtEnqueued`
+  **before** the SQS handoff and **nothing ever clears it**, while Export's first check skips any
+  transcript whose `dtEnqueued` is set — so `dtEnqueued` is a **one-way latch**; (c) nothing else
+  re-read `status='failed'` rows. Observed impact: **25 meetings stranded 2026-07-22 → 2026-07-27**
+  (last successful sync 2026-07-22 16:15) while `Export` reported
+  `found: 106, enqueued: 0, skipped: 106` — i.e. fixing the underlying error alone would **not**
+  have recovered them. **Diagnostic tell:** Export/Process pairs *do* appear in
+  `Core.WorkerJobs`; an `Export` with `enqueued: 0` simply has no `Process` after it, which is
+  easy to misread as a missing dispatch. The `Retry` action is the fix.
+- **Losing the `'synced'` status write is uniquely dangerous now that `Retry` exists.**
+  `setStatus()` catches `Throwable` and only `error_log`s. If the failure hits the **`'synced'`**
+  write specifically, the pipeline **completed** and the recap was **already sent**, but the row
+  still reads `failed` — so the `Retry` sweep re-runs everything and re-sends the recap,
+  repeatedly. It now logs a distinct **CRITICAL** line naming the uuid and telling the operator to
+  set the row to `synced` manually. Residual risk **accepted** rather than adding a
+  `dtRecapSent` column.
+- **DEPLOY ORDER (2026-07-28 change): Team migration → worker2 code → Core cron row.** Any other
+  order is safe but inert. Also note: draining the 25-meeting backlog **sends 25 recap emails** to
+  the original organizers for meetings up to six days old.
+- **Two of the 25 stranded meetings are not the outage and will not self-heal** — they will
+  exhaust the 20-attempt budget and park, which is the intended behavior of the bound.
+  "Agilant & The Tow Foundation Sync" (2026-07-22, 179,779 bytes, the largest in the set) failed
+  **HTTP 422**, not 500, against an active-template `maxTokens` of 40000 — it likely exceeds the
+  model's input budget and needs **chunking**. "Tow Receipt Project Update" is **86 bytes**, an
+  effectively empty transcript.
 - **DEPLOY ORDER: apply migration `2026-07-10a` BEFORE deploying the new worker2 code.** The
   code reads/writes `dtEnqueued`/`dtLastChanged`; deploying code first errors on every poll.
   The migration also backfills every existing row as handed-off (`dtEnqueued=dtExported`) so
@@ -294,6 +386,23 @@ part of the ingestion loop** (raw reads removed). Credential values live only in
 
 ## Change history
 
+- 2026-07-28 — **Added `Team/Transcripts/Retry`, a bounded automatic re-drive of failed
+  transcripts, and fixed the five-day 500 outage.** Discovered a failed `Process()` was
+  *permanently* terminal (HTTP-200 workers + the `dtEnqueued` one-way latch + nothing re-reading
+  `status='failed'`), stranding 25 meetings from 2026-07-22. `Retry` re-enqueues `Process` straight
+  from the `TranscriptExports` ledger, bounded by `RETRY_MAX_ATTEMPTS=20`, a 15→360-minute doubling
+  backoff, and `RETRY_MAX_AGE_DAYS=30`. Review findings addressed: made the attempt increment an
+  **atomic status-guarded claim** (read `getAffectedRows()` pre-commit, skip enqueue on 0 rows) —
+  the unconditional version let concurrent sweeps double-run the pipeline and send **two recap
+  emails**; excluded `meetingIdentifier`-less unprocessable rows in the candidate SQL so they can't
+  starve the `LIMIT`; added a CRITICAL log when the `'synced'` status write is lost. Outage root
+  cause was a stale model id **in `Team.TranscriptPromptTemplate`, not `production.ini`** (the ini
+  is dead fallback) — Talos now requires the `bedrock_converse/` prefix. dbchanges2:
+  `Team/2026-07-28a` (retryAttempts/dtLastRetried, defaults make failures eligible with **no
+  backfill**), `Team/2026-07-28b` (model, matched on the old value so it no-ops if hand-fixed),
+  `Core/2026-07-28a` (cron `10,40 9-18 * * 1-5`, offset from Export's `15,45`). `php -l` clean;
+  php-reviewer 1 critical + 1 high + 1 medium all addressed; sql-reviewer SAFE TO MERGE.
+  **Nothing deployed or applied yet.** (jcardinal)
 - 2026-07-28 — **`callTalosEndpoint()` migrated from raw curl to `_ApiRequest`** so both AI
   passes are recorded in the api log; `initialize()` now also registers `Logs_True` under
   `_underscore::DB_CLIENT_LOGS` (without it the logging is a silent no-op). Deliberately kept

package/knowledge/INDEX.md

@@ -18,7 +18,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 ## 2.0 framework
 
 - **_underscore** (_Underscore) _(framework core)_ — 39 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
-- **worker2** (Worker) — 31 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
+- **worker2** (Worker) — 32 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
 - **api2** (API) — 18 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
 - **dbchanges2** (Database Changes) _(framework core)_ — 3 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
 - **toga2-supply** (TOGa Supply) — 3 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.454",
+  "version": "1.0.455",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",