toga-ai 1.0.694 → 1.0.695

package/knowledge/2.0/apps/api2/INDEX.md

@@ -7,8 +7,8 @@
 | [Multi-Client (Cross-Client) Data Retrieval](features/cross-client-data-retrieval.md) | A single authenticated V2 GET listing can return records across **many** clients (designed for 1000+) that the caller is entitled to, honoring **each target cli | api2/Component/Api/CrossClient/CrossClient.php, api2/Component/Api/V2/V2.php, api2/Controller/Index.php, _underscore/Model/Cache/Table.php, _underscore/Model/Cache/Tables/Client.php, _underscore/Model/Core/Record.php, worker2/Worker/Platform/Cache.php, worker2/Controller/Index.php, worker2/_.php, dbchanges2/Cache/2026-06-30a - MultiClientCacheTables.sql, dbchanges2/Core/2026-06-30b - CacheClusterRegistrationAndRecordTtl.sql, dbchanges2/Core/2026-07-27a - PlatformCacheCleanCron.sql |
 | [cXML ShipNotice Gateway (ASN ingestion, carrier resolution, per-client provisioning)](features/cxml-shipnotice-gateway.md) | `_Component_Api_Cxml` accepts a supplier `ShipNoticeRequest` and translates it into a `POST /v2/advance-shipping-notices` on the V2 JSON engine. | api2/Component/Api/Cxml/Cxml.php, api2/Component/Api/V2/V2.php, api2/Controller/Index.php |
 | [Encrypted-User-UUID Auth Handoff (/auth/encrypted-user-uuid)](features/encrypted-user-uuid-auth-handoff.md) | `POST /auth/encrypted-user-uuid` is the intended **cross-client / SSO-handoff identity mechanism**: given an encrypted `{client, user}` UUID pair, it mints a fr | api2/Component/Api/CrossClient/CrossClient.php |
-| [ENVIRONMENT (not the EB environment name) decides the _underscore branch and Config file](features/environment-variable-drives-underscore-branch.md) | An api2 Elastic Beanstalk instance decides **which `_underscore` branch it clones** and **which `Config/<env>.ini` it loads** from the EB environment property * | api2/.ebextensions/git.php, api2/.ebextensions/php_include_underscore.config, api2/.ebextensions/git.sandbox-dev.json, api2/.ebextensions/git.sandbox-client.json, api2/Config/beta.ini, api2/Config/sandbox-dev.ini, api2/Component/Api/V2/V2.php |
-| [Health-check endpoint (/health liveness short-circuit)](features/health-check-endpoint.md) | `_Controller_Index::api()` short-circuits **liveness/health-probe** requests to an HTTP 200 **before** any routing, DB bootstrap, or V2 engine work runs. | api2/Controller/Index.php |
+| [ENVIRONMENT (not the EB environment name) decides the _underscore branch and Config file](features/environment-variable-drives-underscore-branch.md) | An api2 Elastic Beanstalk instance decides **which `_underscore` branch it clones** and **which `Config/<env>.ini` it loads** from the EB environment property * | api2/.ebextensions/git.php, api2/.platform/hooks/prebuild/git.sh, api2/.ebextensions/php_include_underscore.config, api2/.ebextensions/git.sandbox-dev.json, api2/.ebextensions/git.sandbox-client.json, api2/Config/beta.ini, api2/Config/sandbox-dev.ini, api2/Component/Api/V2/V2.php |
+| [Health-check endpoint (/health liveness — static file + PHP short-circuit)](features/health-check-endpoint.md) | The EB/ALB liveness probe (`/health`) is served two ways, and the **load-bearing** one is now a **static Apache-served file**, not PHP: - **`/health` → a static | api2/health, api2/.platform/httpd/conf.d/health_probe.conf, api2/Controller/Index.php |
 | [Language Translation Layer (audience.language + sidecar tables)](features/language-translation-layer.md) | Serves the same TOGa data (Item title/description/longDescription, plus item **feature** text — `Features.name`, `ItemCategoryFeatureGroups.name`, `ItemFeatures | api2/Component/Api/V2/V2.php, api2/Component/Api/V2/Response/Response.php, _underscore/Model/Core/Setting.php, _underscore/Model/Core/RecordField.php, _underscore/Model/Core/DefaultGlobalSetting.php, _underscore/Model/Client/ItemTranslation.php, _underscore/Model/Client/FeatureTranslation.php, _underscore/Model/Client/ItemCategoryFeatureGroupTranslation.php, _underscore/Model/Client/ItemFeatureTranslation.php, dbchanges2/Client/2026-06-23a - ItemTranslations.sql, dbchanges2/Client/2026-06-23b - ItemTranslationsAcl.sql, dbchanges2/Client/2026-07-13a - FeatureTranslations.sql, dbchanges2/Client/2026-07-13b - FeatureTranslationsAcl.sql, dbchanges2/Core/2026-06-23a - RecordFieldsTranslationColumn.sql, dbchanges2/Core/2026-06-23b - ItemTranslationsRecord.sql, dbchanges2/Core/2026-07-13 - FeatureTranslationsRecord.sql |
 | [/auth/login resolves the client from the email domain, not the Bearer token (cross-client user path)](features/login-cross-client-user-resolution.md) | `POST /v2/auth/login` (email/password user login) can silently swap the target client mid-request. | api2/Component/Api/V2/V2.php, _underscore/String.php |
 | [Nested FK object embedding is gated by the CHILD record's own ACL](features/nested-fk-acl-embedding.md) | When the V2 JSON engine serializes a foreign-key field into a **nested object** (in `getFullModelData()`, ~V2.php L6016-6060), it re-checks the **child** record | api2/Component/Api/V2/V2.php, dbchanges2/Client_Compass/2026-07-23b - PurchaseOrdersRecordReadAcl.sql |
@@ -22,7 +22,7 @@
 | [TableView field/column metadata (TableViewFields, hidden projected columns)](features/tableview-field-metadata.md) | The columns of a 2.0 table view are defined by DB metadata, not code. | _underscore/Model/Client/TableView.php, api2/Component/Api/V2/V2.php, dbchanges2/Client/2026-07-20 - ItemsUuidForPurchaseOrderItemsTableView.sql, dbchanges2/Core/2026-08-17a - ItemFulfillmentQuantityFieldTypeNumber.sql, dbchanges2/Client/2026-08-17a - ItemFulfillmentColumnsCopyable.sql, dbchanges2/Client_Quad/2026-08-18c - SalesOrderListingSortByDateOrderDesc.sql, dbchanges2/Client_Quad/2026-08-24 - Quad Multi Currency Item Pricing.sql, dbchanges2/Client_Elite/2026-08-18 - InventoryUnitsItemColumns.sql, dbchanges2/Client_Nychh/2026-08-18 - InventoryUnitsItemColumns.sql, dbchanges2/Client_CompassCanada/2026-08-27a - ItemFulfillmentTableViewsRebuild.sql |
 | [Tickets API (/v2/tickets)](features/tickets-api.md) | The generic ticket endpoint of the 2.0 REST API. | Component/Api/V2/V2.php |
 | [V2 API error/message codes (EV/EZ troubleshooting map)](features/v2-api-error-codes.md) | The V2 JSON engine (`Component/Api/V2/V2.php`) returns short **message codes** in the response `error` field, grouped by family: `EN-*` authentication, `EZ-*` a | api2/Component/Api/V2/V2.php, api2/Component/Api/V2/Response/Response.php, api2/Component/Api/V2/Response/Oauth/Oauth.php, api2/Controller/Index.php, toga2-supply/src/globalTypes.ts, toga2-supply/src/pages/Orders/api/OrdersApi.ts, toga2-supply/src/api/toga.ts, _underscore/Model/Client/TrackingNumber.php |
-| [V2 request deadlock-retry — route-scoped in-process replay](features/v2-deadlock-retry.md) | api2's front controller can **detect a MySQL deadlock (1213) / lock-wait timeout (1205) and replay the whole request in-process**, so a transient lock collision | api2/Controller/Index.php, _underscore/Database.php, _underscore/Query.php |
+| [V2 request retry (Prong A) — deadlock AND retriable unique-key race, route-scoped in-process replay](features/v2-deadlock-retry.md) | api2's front controller ("**Prong A**") can **detect a retriable concurrency failure and replay the whole request in-process**, so a transient collision self-re | api2/Controller/Index.php, api2/Component/Api/V2/V2.php, _underscore/Database.php, _underscore/Query.php, _underscore/Model/Compass/AdvanceShippingNotice.php |
 | [V2 REST query contract (params, where grammar, encoding, ACL behavior)](features/v2-rest-query-contract.md) | What an **HTTP client** has to get right to query the Toga v2 REST API: which query params are recognized, the exact `where` grammar, how the query string is (n | api2/Component/Api/V2/V2.php |
 | [V2 reverse hasMany collections must be named in the fetch fields whitelist](features/v2-reverse-hasmany-fields-whitelist.md) | In the V2 JSON engine, a **reverse hasMany** relationship — the collection of child records that foreign-key back to a parent (e.g. | api2/Component/Api/V2/V2.php |
 | [AWS CodePipeline Deployment via CodeConnections (GitHub → Elastic Beanstalk)](workflows/codepipeline-codeconnections-deploy.md) | 2.0 apps (`api2`, `_underscore`) are deployed through **AWS CodePipeline**. | api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, api2/ebs/register_instance_to_shared_application_load_balancer.php, api2/.platform/hooks/prebuild/git.sh, api2/.ebextensions/git.php, api2/.ebextensions/git.sandbox-dev.json |

package/knowledge/2.0/apps/api2/architecture.md

@@ -6,7 +6,7 @@ project: API
 client: shared
 type: architecture
 status: active
-updated: 2026-08-26
+updated: 2026-08-28
 owners: [jcardinal, bala, mhammontree, dfranks]
 files:
   - api2/Controller/Index.php
@@ -254,6 +254,27 @@ interceptor's plain-`Exception` rejections — all now returning 4xx/fixed. Dist
 reading the ALB 5xx metric and `Logs.Issue`: zero 5xx + a traffic spike = transient EB snapshot;
 a sustained 5xx rate = a code bug to fix.
 
+**A third cause — a `Degraded`/`Severe` flip with NO stated cause is the PHP-served `/health`
+probe being STARVED, not client traffic failing.** Confirmed on `api-production-1` (acct
+`654654170868`): off-peak **bursts of slow, failing `POST /v2/advance-shipping-notices`**
+(Compass ASN; 17–63 s each, ~100% HTTP 500) saturate the **PHP-FPM worker pool** on an
+instance. The ALB `/health` check — historically **PHP-served, 5 s timeout** — then can't get a
+worker and fails a few checks, so EB flags the target group "reduced health" **with no cause
+cited**. Two reasons this looks invisible: (1) **health-check traffic does NOT appear in ALB
+`RequestCount`/`5xx` metrics**, so the ALB view reads clean; (2) **`HealthyHostCount` never
+drops**, because the probe failure is shorter than the **5-check / 75 s deregistration
+threshold** — it flaps EB enhanced-health without ever deregistering a target. A separate, benign
+contributor is genuine **sub-minute `/health` transients** on a single instance. Correlation
+proof: a **06:00–06:08 CST ASN burst == the 11:05 UTC EB `Severe` flip**. Diagnosed via AWS CLI
+(`describe-events`, `describe-configuration-settings`, CloudWatch `get-metric-data`) + prod
+`Logs.Api`/`Logs.Issue`.
+
+**Rule:** a "reduced health / no cause" flip on this 2-instance PHP env means the ALB `/health`
+probe was starved of a PHP-FPM worker — do **not** chase client-traffic 5xx. **Fix shipped
+(working tree):** `/health` is now served as a **static Apache file** (`api2/health`), decoupling
+liveness from PHP-FPM worker availability so a slow-request burst can no longer flap it — see
+[health-check endpoint](features/health-check-endpoint.md).
+
 ## Known issues / accepted risks
 
 Open items a maintainer should know before changing this tier. None are "bugs to fix right now" —
@@ -314,7 +335,11 @@ they are the known sharp edges. Do not re-discover these from scratch.
   class lives in `_underscore` (`Model/Client/...`).
 - **Client-specific behavior** belongs in `_Model_<ClientSlug>_X` overrides and
   `Client_ApiPayloadInterceptor` pre/post hooks — not in `V2.php`.
-- **Don't break `/health`** or weaken `enforce_https.conf`.
+- **Don't break `/health`.** It is now a **static Apache-served file** (`api2/health`, body
+  `OK`) — **not** a PHP short-circuit — so the EB/ALB liveness probe is decoupled from PHP-FPM
+  worker availability. Do **not** route `/health` back through PHP or remove the file, and don't
+  weaken `enforce_https.conf`. (`/v2/health` remains PHP-handled.) See
+  [health-check endpoint](features/health-check-endpoint.md).
 - **504s:** verify `long_gateway_timeout.conf` on both this tier and the proxy
   (LB 3600s → proxy Apache 1800s → proxy cURL 900s → api Apache 1800s).
 - **`transactionId` uniqueness is load-bearing** — the idempotency/audit key; don't bypass
@@ -327,6 +352,7 @@ they are the known sharp edges. Do not re-discover these from scratch.
   single-caller branches in `V2.php` as unverified until exercised directly.
 
 ## Change history
+- 2026-08-28 — Recorded the third EB-`Degraded` cause: a no-stated-cause "reduced health" flip on `api-production-1` is the **PHP-served `/health` probe starved of a PHP-FPM worker** by off-peak bursts of slow failing Compass ASN POSTs (17–63 s, ~100% 500) — invisible in ALB `RequestCount`/`5xx` and never dropping `HealthyHostCount` (flap < 75 s deregistration). Correlated a 06:00–06:08 CST ASN burst to the 11:05 UTC Severe flip. Noted the fix (static-file `/health` decouples liveness from PHP-FPM) and updated the "Don't break `/health`" rule to the static-file form. (jcardinal)
 - 2026-08-26 — Documented and fixed a `/v2/auth/public` OOM: a present-but-null client `transactionId` (`{"transactionId": null}`) flowed into the request-logger's `_Model::load()` dup-check; because `Logs.Api.transactionId` is `FIELD_CHAR`, a null value collapses the WHERE and full-table-scans prod `Logs.Api` → memory exhaustion (the `_underscore` collapsed-WHERE mechanism). V2 only auto-uuids an *absent* transactionId (~L543), so a present null fell through (~L553 read → null). Fix (commit `36ef31f`): coerce null/empty transactionId → uuid in the auth block, and skip `load()` in the final logger dup-check (~L2277) when null/empty. Added the "don't blame the latest deploy — check first-occurrence" triage note. (jcardinal)
 - 2026-08-25 — **Resolved the auto-generated `Api.transactionId` 1062 collision** (was the 2026-07-28
   gotcha): the two DB-insert log sites now uuid the log-row `transactionId` (`_String::generateUuid()`)

package/knowledge/2.0/apps/api2/features/environment-variable-drives-underscore-branch.md

@@ -6,10 +6,11 @@ project: API
 client: shared
 type: feature
 status: active
-updated: 2026-08-25
-owners: ["bala", "mhammontree", "apeterson"]
+updated: 2026-08-28
+owners: ["bala", "mhammontree", "apeterson", "jcardinal"]
 files:
   - api2/.ebextensions/git.php
+  - api2/.platform/hooks/prebuild/git.sh
   - api2/.ebextensions/php_include_underscore.config
   - api2/.ebextensions/git.sandbox-dev.json
   - api2/.ebextensions/git.sandbox-client.json
@@ -177,6 +178,21 @@ the tier trades a stale-code bug for a total outage. Full procedure:
 
 ## Gotchas / known issues
 
+- **⚠ Cross-repo change ordering: land `_underscore` BEFORE deploying api2.** Because api2's EB
+  deploy **clones `_underscore` (branch `_<ENVIRONMENT>`, e.g. `_production`) at BUILD time**
+  (`.platform/hooks/prebuild/git.sh`), a change that spans both repos must have its `_underscore`
+  side merged to `_<ENVIRONMENT>` **first** — otherwise the freshly-built api2 will call a framework
+  method/class that isn't on the box and fatal. Landing `_underscore` first is **safe on its own**:
+  the old api2 controller simply doesn't use the new framework code yet. Order: merge `_underscore`
+  → (verify it's on the branch) → deploy api2.
+- **⚠ A "Could not find required file for `_Model_...`" fatal on prod can be a STALE-BUILD
+  artifact, not a missing file.** `Logs.Issue #476`
+  ("Could not find required file for '_Model_Client_ItemFulfillments_InventoryAdjustment'") fired
+  because that model file **was** committed to `_production` but `api-production-1` was still running
+  an **older build**; a **redeploy** (which re-pulls `_production`) resolved it with no recurrence.
+  Before assuming a model file is absent, check the **running build's `_underscore` against
+  `_production`** (file presence on the box; `git merge-base --is-ancestor`) — the build-time clone
+  means a merged file may simply not be on that instance yet.
 - **⚠ Merging to the branch named after the EB environment can be a no-op.** Merge to
   `_` + `ENVIRONMENT`. If your change "deployed successfully" but the behavior is absent and no
   error appears, check this before debugging the code. A missing class also throws `Error`
@@ -241,6 +257,14 @@ the tier trades a stale-code bug for a total outage. Full procedure:
 
 ## Change history
 
+- 2026-08-28 — Recorded two build-time-clone consequences: (1) **cross-repo deploy ordering** — api2
+  clones `_underscore` (`_<ENVIRONMENT>`) at build via `.platform/hooks/prebuild/git.sh`, so for a
+  change spanning both repos, `_underscore` must land on the deploy branch **before** api2 is
+  deployed (landing `_underscore` first is safe alone); and (2) a **"Could not find required file
+  for `_Model_...`" fatal can be a stale build**, not a missing file — `Logs.Issue #476` fired for
+  a model committed to `_production` while `api-production-1` ran an older build, and a redeploy
+  resolved it. Check the running build's `_underscore` vs `_production` before assuming the file is
+  absent. (jcardinal)
 - 2026-08-27 — Recorded the **inverse hazard**: moving a tier **forward** onto its correct
   `_underscore` branch is a **schema event**. The 2026-08-26 deploy moved `sandbox-client` off the
   `_production` framework (which the 2026-08-25 entry below explains it had been running for weeks)

package/knowledge/2.0/apps/api2/features/health-check-endpoint.md

@@ -1,14 +1,16 @@
 ---
-title: "Health-check endpoint (/health liveness short-circuit)"
+title: "Health-check endpoint (/health liveness — static file + PHP short-circuit)"
 framework: "2.0"
 repo: api2
 project: API
 client: shared
 type: feature
 status: active
-updated: 2026-07-24
+updated: 2026-08-28
 owners: [jcardinal]
 files:
+  - api2/health
+  - api2/.platform/httpd/conf.d/health_probe.conf
   - api2/Controller/Index.php
 related:
   - ../architecture.md
@@ -17,13 +19,37 @@ related:
 
 ## Summary
 
-`_Controller_Index::api()` short-circuits **liveness/health-probe** requests to an HTTP 200
-**before** any routing, DB bootstrap, or V2 engine work runs. EB/LB (Elastic Beanstalk /
-load balancer) probes depend on this always returning 200 cheaply — the architecture doc's
-critical rule is literally "Don't break `/health`". This doc records the **matching contract**
-so a teammate doesn't reintroduce a brittle exact-string check.
+The EB/ALB liveness probe (`/health`) is served two ways, and the **load-bearing** one is now
+a **static Apache-served file**, not PHP:
 
-## How it works
+- **`/health` → a static file `api2/health`** (literal body `OK`), served by Apache **without
+  ever reaching PHP-FPM**. This is what the EB target group health-check hits. Decoupling the
+  probe from PHP means a PHP-FPM worker-pool exhaustion (e.g. a burst of slow ASN requests
+  saturating all workers) or a PHP transient can **no longer flap EB health** — the probe
+  answers from Apache regardless of app load.
+- **`/v2/health` (and `/health` as a fallback) → PHP short-circuit** in
+  `_Controller_Index::api()`, an HTTP 200 returned **before** any routing, DB bootstrap, or V2
+  engine work. This still exists for callers that hit the PHP entrypoint directly.
+
+This doc records the **matching contract** for both so a teammate doesn't reintroduce a brittle
+exact-string check or accidentally route the liveness probe back through PHP.
+
+## The static probe (the one EB actually uses) — 2026-08-28
+
+- **`api2/health`** — a one-line static file (`OK`). Apache serves it directly.
+- **`api2/.platform/httpd/conf.d/health_probe.conf`** — comment-only documentation of why the
+  probe is static (no directives; it exists to explain the decision at the Apache layer).
+- **Why it works without a rewrite fight:** `api2/.htaccess` only rewrites to `index.php` when
+  the request target is **not a real file** (`RewriteCond %{REQUEST_FILENAME} !-f`). Because
+  `health` is a real file on disk, the rewrite is skipped and Apache serves it straight.
+- **Why `enforce_https` doesn't redirect it:** EB health checks arrive on **HTTP:80 with no
+  `X-Forwarded-Proto`**, and `enforce_https.conf` only redirects requests that carry the
+  forwarded-proto marker — so the probe is not bounced to HTTPS.
+- **Do not** move `/health` back into a PHP-served route or rename/remove `api2/health` — that
+  re-couples EB liveness to PHP-FPM worker availability, which is the exact failure this fixed
+  (see [api2 architecture — EB "Degraded" health](../architecture.md)).
+
+## PHP short-circuit (for `/v2/health` and direct hits)
 
 The short-circuit lives at the top of `Index.php::api()`, before host dispatch and before the
 Core/Logs DB bootstrap. It matches against a class constant:
@@ -55,6 +81,15 @@ normalized form: lowercase, no trailing slash, no query) rather than re-adding a
 branch.
 
 ## Change history
+- 2026-08-28 — Moved the **EB/ALB liveness probe off PHP**: `/health` is now a **static file**
+  (`api2/health`, body `OK`) served by Apache — it never reaches PHP-FPM, so a slow-request burst
+  or a PHP transient can no longer starve the probe and flap EB health. Works because
+  `.htaccess` only rewrites to `index.php` for non-files (`!-f`), and `enforce_https` skips the
+  probe (HTTP:80, no `X-Forwarded-Proto`). Added a comment-only
+  `.platform/httpd/conf.d/health_probe.conf` documenting the decision. `/v2/health` stays
+  PHP-handled and the `HEALTH_CHECK_ROUTES` PHP short-circuit remains as a fallback. Working tree
+  only (uncommitted). See the [EB "Degraded" health](../architecture.md) diagnosis this fixes.
+  (jcardinal)
 - 2026-07-24 — Replaced the brittle exact-string `REQUEST_URI == '/health'` check with a
   `HEALTH_CHECK_ROUTES` constant + normalized (lowercase / strip query / strip trailing slash)
   strict `in_array` match, so `/v2/health`, `/health/`, and `/health?x` all short-circuit to 200

package/knowledge/2.0/apps/api2/features/v2-deadlock-retry.md

@@ -1,38 +1,101 @@
 ---
-title: V2 request deadlock-retry — route-scoped in-process replay
+title: V2 request retry (Prong A) — deadlock AND retriable unique-key race, route-scoped in-process replay
 framework: "2.0"
 repo: api2
 project: API
 client: shared
 type: feature
 status: draft
-updated: 2026-08-13
+updated: 2026-08-28
 owners: [jcardinal]
 files:
   - api2/Controller/Index.php
+  - api2/Component/Api/V2/V2.php
   - _underscore/Database.php
   - _underscore/Query.php
+  - _underscore/Model/Compass/AdvanceShippingNotice.php
 related:
   - ../../_underscore/features/model-save-parent-cascade-stored-field-deadlock.md
+  - ../../../../clients/compass-usa/features/asn-to-item-fulfillment.md
   - ../architecture.md
   - ./v2-api-error-codes.md
 ---
 
 ## Summary
 
-api2's front controller can **detect a MySQL deadlock (1213) / lock-wait timeout (1205) and replay
-the whole request in-process**, so a transient lock collision self-recovers instead of returning an
-HTTP 400 to the caller. The replay is **route-scoped by an allowlist**
-(`DEADLOCK_RETRIABLE_ROUTES`) — currently only `/v2/advance-shipping-notices` — because whole-
-request in-process replay is **only safe for routes with no non-rollbackable external side effect**.
-This is the safety-net half of the ASN deadlock fix; the source-level fix (stop the collateral
-parent re-save) is in
-[_underscore's save-cascade doc](../../_underscore/features/model-save-parent-cascade-stored-field-deadlock.md).
+api2's front controller ("**Prong A**") can **detect a retriable concurrency failure and replay
+the whole request in-process**, so a transient collision self-recovers instead of returning an
+error to the caller. Two failure classes now qualify:
 
-> Status (2026-08-13): **in the working tree only** — not committed or deployed. Pending sandbox
+1. **MySQL deadlock (1213) / lock-wait timeout (1205)** — the original case (a same-PO ASN write
+   deadlock; see the
+   [save-cascade doc](../../_underscore/features/model-save-parent-cascade-stored-field-deadlock.md)).
+2. **Retriable unique-key get-or-create race (1062)** — a concurrent same-key INSERT lost the
+   race; a whole-request replay on a fresh snapshot lets the retry's GET reuse the winner's
+   committed row (added 2026-08-28 for the Compass ASN data-loss race).
+
+The replay is **route-scoped by an allowlist** (`DEADLOCK_RETRIABLE_ROUTES`) — currently only
+`/v2/advance-shipping-notices` — because whole-request in-process replay is **only safe for routes
+with no non-rollbackable external side effect**. The retry gate is now
+**`wasDeadlock() || wasRetriableUniqueRace()`**.
+
+> Status (2026-08-28): **in the working tree only** — not committed or deployed. Pending sandbox
 > validation + regression tests.
 
-## How it works
+## The retriable unique-key race (1062) — Compass ASN data-loss (2026-08-28)
+
+`_Model_Compass_AdvanceShippingNotice::postPost` **get-or-creates** an SO-numbered Item
+Fulfillment (GET by number, else POST). `ItemFulfillments.number` is UNIQUE and vendors
+re-transmit the same shipment file, so two concurrent same-number ASNs race: the loser's INSERT
+trips the unique key. Pre-fix, `internalApiRequest` threw (default `throwExceptionsOnError=true`),
+the 500 propagated out of `postPost`, and **the api2 controller rolled back the whole request —
+losing the ASN row and its tracking numbers** (`Logs.Issue #239`, EV-10). This is a bug 1062
+must NOT be treated like a normal error, because a replay recovers it cleanly.
+
+**Fix (Approach B — a CTO review rejected a simpler in-place `FOR UPDATE` recovery, because an
+interceptor cannot recover a poisoned response in place; see *V2 concurrency mechanics* below).**
+A generalized retriable-unique-race path:
+
+- **`_Database`** gains `const ERRNO_DUPLICATE_ENTRY = 1062`, a sticky
+  `public static bool $retriableUniqueRace` (reset per attempt in `clearLastError()`), and
+  `wasRetriableUniqueRace()` — true **only** when the caller-set flag is set **AND**
+  `lastErrorNumber === 1062`. The flag is caller-opt-in on purpose: a bare 1062 is a real
+  constraint violation and must **not** silently retry; only a caller that knows the 1062 is a
+  benign get-or-create race sets it.
+- **The ASN model** calls the IF-create `internalApiRequest` with **`throwExceptionsOnError:false`**;
+  on an empty response with `lastErrorNumber === 1062` it sets `$retriableUniqueRace = true` and
+  returns null. `postPost` then **aborts immediately** (issues no further query).
+- **The controller's Prong A gate** becomes `wasDeadlock() || wasRetriableUniqueRace()`. Replaying
+  the whole request on a fresh REPEATABLE-READ snapshot lets the retry's GET see the winner's
+  committed IF and reuse it.
+
+Bounded by `MAX_DEADLOCK_ATTEMPTS = 3`, gated to `DEADLOCK_RETRIABLE_ROUTES =
+['/v2/advance-shipping-notices']`.
+
+## V2 concurrency mechanics (durable framework knowledge)
+
+These are the V2-engine facts that decide **how any interceptor-level write race must be handled**
+— they are why the fix is a whole-request replay and not an in-place recovery:
+
+- **Response poisoning (load-bearing).** A failed nested save via `internalApiRequest` calls
+  `addDefinedMessage(EV-10)` on the **shared** `$api->response`, setting a non-2xx status **before**
+  control returns. So even if an interceptor catches the throw and "recovers," `execute()`
+  recomputes `isSuccess = false` (`V2.php` ~2154) and **self-rolls-back DB_CORE + DB_CLIENT**
+  (~2157-2159). **An interceptor therefore CANNOT recover a failed internal save in place** — the
+  only correct recovery is a whole-request replay. This same poisoning is what drives the Prong A
+  retry gate and, on exhaustion, produces the coherent EV-10 response.
+- **`internalApiRequest($method, $route, $payload, $options, $throwExceptionsOnError = true)`** — a
+  **5th argument** controls throw-vs-return-on-error. Pass `false` to inspect `lastErrorNumber`
+  instead of unwinding.
+- **`_Database::$lastErrorNumber` is sticky.** Set only on query failure, never cleared by a later
+  success; reset per attempt via `clearLastError()`. A recorded **deadlock (1213/1205) is protected
+  from overwrite** by a later error; a **1062 is NOT** protected — so read/act on a 1062 promptly.
+- **The prod client cluster runs REPEATABLE-READ.** A same-transaction re-SELECT after a losing
+  concurrent INSERT is **blind to the winner's commit** — only a locking read or a **fresh-snapshot
+  whole-request replay** sees it. This is why in-place retry inside the transaction cannot work and
+  the replay must start a new transaction.
+
+## How it works — deadlock path
 
 **Detection primitive (`_underscore`, reusable).** `_Database` gains
 `public static ?int $lastErrorNumber`, the consts `ERRNO_DEADLOCK` (1213) and
@@ -42,7 +105,8 @@ driver errno **at its throw site**, and the recording is **sticky**: a later unr
 inside a model-save `try/catch` still be visible to the outer controller and trigger a retry.
 
 **Retry loop (`api2/Controller/Index.php`).** The v2 dispatch is wrapped in a bounded loop
-(`MAX_DEADLOCK_ATTEMPTS = 3`). On an unsuccessful response where `_Database::wasDeadlock()` is true
+(`MAX_DEADLOCK_ATTEMPTS = 3`). On an unsuccessful response where
+`_Database::wasDeadlock() || _Database::wasRetriableUniqueRace()` is true
 and the route is in `DEADLOCK_RETRIABLE_ROUTES`, the controller:
 
 1. rolls back all open transactions;
@@ -82,6 +146,22 @@ superglobal. Default is to leave a route off the list.
   not a substitute for removing the deadlock at its source.
 
 ## Change history
+- 2026-08-28 — **Generalized Prong A to a second retriable class: the unique-key (1062)
+  get-or-create race**, fixing the Compass ASN data-loss bug (`Logs.Issue #239`, EV-10) where a
+  concurrent same-`ItemFulfillments.number` INSERT lost the race, `internalApiRequest` threw, and
+  the whole request rolled back — losing the ASN row and its tracking. Added
+  `_Database::ERRNO_DUPLICATE_ENTRY = 1062`, a sticky caller-opt-in `$retriableUniqueRace`
+  (reset in `clearLastError()`), and `wasRetriableUniqueRace()` (flag AND `lastErrorNumber===1062`);
+  the ASN model calls the IF-create with `throwExceptionsOnError:false`, sets the flag on an empty
+  1062 response, and aborts; the controller gate became
+  `wasDeadlock() || wasRetriableUniqueRace()`. Chose the whole-request replay (Approach B) over an
+  in-place `FOR UPDATE` recovery after a CTO review, because V2 **response poisoning** (a failed
+  nested save flips the shared `$api->response` to non-2xx and `execute()` self-rolls-back
+  DB_CORE+DB_CLIENT, `V2.php` ~2154-2159) makes in-place interceptor recovery impossible, and the
+  prod cluster's REPEATABLE-READ isolation makes a same-transaction re-SELECT blind to the winner's
+  commit. Recorded the durable V2 concurrency mechanics (response poisoning, `internalApiRequest`
+  5th arg, sticky-`lastErrorNumber` with 1213 protected / 1062 not, REPEATABLE-READ). Working tree
+  only, pending sandbox validation. (jcardinal)
 - 2026-08-13 — Added deadlock/lock-wait detection to `_Database`/`_Query` (`$lastErrorNumber`,
   `wasDeadlock()`, sticky errno recording) and a bounded, route-scoped in-process replay loop in
   `Controller/Index.php`, gated to `/v2/advance-shipping-notices` only. Documented why whole-request

package/knowledge/clients/compass-usa/features/asn-to-item-fulfillment.md

@@ -5,7 +5,7 @@ project: _Underscore
 client: compass-usa
 type: client-feature
 status: active
-updated: 2026-08-27
+updated: 2026-08-28
 owners: [jcardinal, bala]
 files:
   - _underscore/Model/Compass/AdvanceShippingNotice.php
@@ -200,6 +200,20 @@ Compass Canada (`Model/Compass/Canada/`) is a separate sub-client. The Compass c
   (`$_model_skipParentCascadeOnSave`), and api2 retries the ASN route on a detected deadlock. Full
   mechanic:
   [save-cascade stored-field deadlock](../../../2.0/apps/_underscore/features/model-save-parent-cascade-stored-field-deadlock.md).
+- **Concurrent same-SO-number ASNs lost the ASN + its tracking to a unique-key race (fixed
+  2026-08-28, working tree).** `postPost` get-or-creates the SO-numbered IF (GET by number, else
+  POST). `ItemFulfillments.number` is UNIQUE and vendors re-transmit the same shipment file, so two
+  concurrent same-number ASNs race: the loser's IF INSERT hits the unique key (MySQL **1062**),
+  `internalApiRequest` threw (default `throwExceptionsOnError=true`), the 500 propagated out of
+  `postPost`, and the api2 controller **rolled back the whole request — losing the ASN row AND its
+  tracking numbers** (`Logs.Issue #239`, EV-10). Fix: the IF-create call now passes
+  `throwExceptionsOnError:false`; on an empty response with `lastErrorNumber===1062` the model sets
+  `_Database::$retriableUniqueRace = true` and returns null, `postPost` aborts, and api2's Prong A
+  replays the whole request on a fresh REPEATABLE-READ snapshot so the retry's GET reuses the
+  winner's committed IF. Distinct from the 1213 deadlock above (a different concurrency failure on
+  the same re-transmission traffic). Full mechanic + why in-place recovery is impossible (V2
+  response poisoning):
+  [V2 request retry — Prong A](../../../2.0/apps/api2/features/v2-deadlock-retry.md).
 - **Cross-line tracking contamination from bad SOI↔POI bridge rows (root cause, fixed
   forward 2026-06-11):** `postPost` resolves which IFI(s) a tracking number attaches to by
   joining `AdvanceShippingNoticeItems → SalesOrderItems_PurchaseOrderItems → SalesOrderItems`.
@@ -313,6 +327,13 @@ feed cannot currently support a line-level completeness test.
 
 ## Change history
 Dated one-liners, newest first.
+- 2026-08-28 — Fixed a **data-loss unique-key race** on concurrent same-SO-number ASNs: the loser's
+  SO-numbered IF INSERT trips `ItemFulfillments.number` UNIQUE (1062), `internalApiRequest` threw,
+  and api2 rolled back the whole request, losing the ASN row + tracking (`Logs.Issue #239`, EV-10).
+  The model now creates the IF with `throwExceptionsOnError:false`, flags a retriable 1062
+  (`_Database::$retriableUniqueRace`) and aborts, and api2's Prong A replays the request on a fresh
+  snapshot so the retry reuses the winner's IF. Distinct from the 2026-08-13 1213 deadlock. Working
+  tree only; also swept 6 banned `throw new _Exception(...)` → `\Exception` in the file. (jcardinal)
 - 2026-08-27 — Quantified the **Office Depot duplicate-ASN** problem on prod and settled what it is:
   of ODP PO lines with more than one shipping notice, **10,012 exceed the PO line quantity and only 8
   are genuine split shipments**, so a second notice on an ODP line is a **duplicate**, not a partial.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.694",
+  "version": "1.0.695",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",