toga-ai 1.0.694 → 1.0.696

package/knowledge/2.0/apps/api2/INDEX.md

@@ -7,8 +7,8 @@
 | [Multi-Client (Cross-Client) Data Retrieval](features/cross-client-data-retrieval.md) | A single authenticated V2 GET listing can return records across **many** clients (designed for 1000+) that the caller is entitled to, honoring **each target cli | api2/Component/Api/CrossClient/CrossClient.php, api2/Component/Api/V2/V2.php, api2/Controller/Index.php, _underscore/Model/Cache/Table.php, _underscore/Model/Cache/Tables/Client.php, _underscore/Model/Core/Record.php, worker2/Worker/Platform/Cache.php, worker2/Controller/Index.php, worker2/_.php, dbchanges2/Cache/2026-06-30a - MultiClientCacheTables.sql, dbchanges2/Core/2026-06-30b - CacheClusterRegistrationAndRecordTtl.sql, dbchanges2/Core/2026-07-27a - PlatformCacheCleanCron.sql |
 | [cXML ShipNotice Gateway (ASN ingestion, carrier resolution, per-client provisioning)](features/cxml-shipnotice-gateway.md) | `_Component_Api_Cxml` accepts a supplier `ShipNoticeRequest` and translates it into a `POST /v2/advance-shipping-notices` on the V2 JSON engine. | api2/Component/Api/Cxml/Cxml.php, api2/Component/Api/V2/V2.php, api2/Controller/Index.php |
 | [Encrypted-User-UUID Auth Handoff (/auth/encrypted-user-uuid)](features/encrypted-user-uuid-auth-handoff.md) | `POST /auth/encrypted-user-uuid` is the intended **cross-client / SSO-handoff identity mechanism**: given an encrypted `{client, user}` UUID pair, it mints a fr | api2/Component/Api/CrossClient/CrossClient.php |
-| [ENVIRONMENT (not the EB environment name) decides the _underscore branch and Config file](features/environment-variable-drives-underscore-branch.md) | An api2 Elastic Beanstalk instance decides **which `_underscore` branch it clones** and **which `Config/<env>.ini` it loads** from the EB environment property * | api2/.ebextensions/git.php, api2/.ebextensions/php_include_underscore.config, api2/.ebextensions/git.sandbox-dev.json, api2/.ebextensions/git.sandbox-client.json, api2/Config/beta.ini, api2/Config/sandbox-dev.ini, api2/Component/Api/V2/V2.php |
-| [Health-check endpoint (/health liveness short-circuit)](features/health-check-endpoint.md) | `_Controller_Index::api()` short-circuits **liveness/health-probe** requests to an HTTP 200 **before** any routing, DB bootstrap, or V2 engine work runs. | api2/Controller/Index.php |
+| [ENVIRONMENT (not the EB environment name) decides the _underscore branch and Config file](features/environment-variable-drives-underscore-branch.md) | An api2 Elastic Beanstalk instance decides **which `_underscore` branch it clones** and **which `Config/<env>.ini` it loads** from the EB environment property * | api2/.ebextensions/git.php, api2/.platform/hooks/prebuild/git.sh, api2/.ebextensions/php_include_underscore.config, api2/.ebextensions/git.sandbox-dev.json, api2/.ebextensions/git.sandbox-client.json, api2/Config/beta.ini, api2/Config/sandbox-dev.ini, api2/Component/Api/V2/V2.php |
+| [Health-check endpoint (/health liveness — static file + PHP short-circuit)](features/health-check-endpoint.md) | The EB/ALB liveness probe (`/health`) is served two ways, and the **load-bearing** one is now a **static Apache-served file**, not PHP: - **`/health` → a static | api2/health, api2/.platform/httpd/conf.d/health_probe.conf, api2/Controller/Index.php |
 | [Language Translation Layer (audience.language + sidecar tables)](features/language-translation-layer.md) | Serves the same TOGa data (Item title/description/longDescription, plus item **feature** text — `Features.name`, `ItemCategoryFeatureGroups.name`, `ItemFeatures | api2/Component/Api/V2/V2.php, api2/Component/Api/V2/Response/Response.php, _underscore/Model/Core/Setting.php, _underscore/Model/Core/RecordField.php, _underscore/Model/Core/DefaultGlobalSetting.php, _underscore/Model/Client/ItemTranslation.php, _underscore/Model/Client/FeatureTranslation.php, _underscore/Model/Client/ItemCategoryFeatureGroupTranslation.php, _underscore/Model/Client/ItemFeatureTranslation.php, dbchanges2/Client/2026-06-23a - ItemTranslations.sql, dbchanges2/Client/2026-06-23b - ItemTranslationsAcl.sql, dbchanges2/Client/2026-07-13a - FeatureTranslations.sql, dbchanges2/Client/2026-07-13b - FeatureTranslationsAcl.sql, dbchanges2/Core/2026-06-23a - RecordFieldsTranslationColumn.sql, dbchanges2/Core/2026-06-23b - ItemTranslationsRecord.sql, dbchanges2/Core/2026-07-13 - FeatureTranslationsRecord.sql |
 | [/auth/login resolves the client from the email domain, not the Bearer token (cross-client user path)](features/login-cross-client-user-resolution.md) | `POST /v2/auth/login` (email/password user login) can silently swap the target client mid-request. | api2/Component/Api/V2/V2.php, _underscore/String.php |
 | [Nested FK object embedding is gated by the CHILD record's own ACL](features/nested-fk-acl-embedding.md) | When the V2 JSON engine serializes a foreign-key field into a **nested object** (in `getFullModelData()`, ~V2.php L6016-6060), it re-checks the **child** record | api2/Component/Api/V2/V2.php, dbchanges2/Client_Compass/2026-07-23b - PurchaseOrdersRecordReadAcl.sql |
@@ -22,8 +22,8 @@
 | [TableView field/column metadata (TableViewFields, hidden projected columns)](features/tableview-field-metadata.md) | The columns of a 2.0 table view are defined by DB metadata, not code. | _underscore/Model/Client/TableView.php, api2/Component/Api/V2/V2.php, dbchanges2/Client/2026-07-20 - ItemsUuidForPurchaseOrderItemsTableView.sql, dbchanges2/Core/2026-08-17a - ItemFulfillmentQuantityFieldTypeNumber.sql, dbchanges2/Client/2026-08-17a - ItemFulfillmentColumnsCopyable.sql, dbchanges2/Client_Quad/2026-08-18c - SalesOrderListingSortByDateOrderDesc.sql, dbchanges2/Client_Quad/2026-08-24 - Quad Multi Currency Item Pricing.sql, dbchanges2/Client_Elite/2026-08-18 - InventoryUnitsItemColumns.sql, dbchanges2/Client_Nychh/2026-08-18 - InventoryUnitsItemColumns.sql, dbchanges2/Client_CompassCanada/2026-08-27a - ItemFulfillmentTableViewsRebuild.sql |
 | [Tickets API (/v2/tickets)](features/tickets-api.md) | The generic ticket endpoint of the 2.0 REST API. | Component/Api/V2/V2.php |
 | [V2 API error/message codes (EV/EZ troubleshooting map)](features/v2-api-error-codes.md) | The V2 JSON engine (`Component/Api/V2/V2.php`) returns short **message codes** in the response `error` field, grouped by family: `EN-*` authentication, `EZ-*` a | api2/Component/Api/V2/V2.php, api2/Component/Api/V2/Response/Response.php, api2/Component/Api/V2/Response/Oauth/Oauth.php, api2/Controller/Index.php, toga2-supply/src/globalTypes.ts, toga2-supply/src/pages/Orders/api/OrdersApi.ts, toga2-supply/src/api/toga.ts, _underscore/Model/Client/TrackingNumber.php |
-| [V2 request deadlock-retry — route-scoped in-process replay](features/v2-deadlock-retry.md) | api2's front controller can **detect a MySQL deadlock (1213) / lock-wait timeout (1205) and replay the whole request in-process**, so a transient lock collision | api2/Controller/Index.php, _underscore/Database.php, _underscore/Query.php |
+| [V2 request retry (Prong A) — deadlock AND retriable unique-key race, route-scoped in-process replay](features/v2-deadlock-retry.md) | api2's front controller ("**Prong A**") can **detect a retriable concurrency failure and replay the whole request in-process**, so a transient collision self-re | api2/Controller/Index.php, api2/Component/Api/V2/V2.php, _underscore/Database.php, _underscore/Query.php, _underscore/Model/Compass/AdvanceShippingNotice.php |
 | [V2 REST query contract (params, where grammar, encoding, ACL behavior)](features/v2-rest-query-contract.md) | What an **HTTP client** has to get right to query the Toga v2 REST API: which query params are recognized, the exact `where` grammar, how the query string is (n | api2/Component/Api/V2/V2.php |
 | [V2 reverse hasMany collections must be named in the fetch fields whitelist](features/v2-reverse-hasmany-fields-whitelist.md) | In the V2 JSON engine, a **reverse hasMany** relationship — the collection of child records that foreign-key back to a parent (e.g. | api2/Component/Api/V2/V2.php |
-| [AWS CodePipeline Deployment via CodeConnections (GitHub → Elastic Beanstalk)](workflows/codepipeline-codeconnections-deploy.md) | 2.0 apps (`api2`, `_underscore`) are deployed through **AWS CodePipeline**. | api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, api2/ebs/register_instance_to_shared_application_load_balancer.php, api2/.platform/hooks/prebuild/git.sh, api2/.ebextensions/git.php, api2/.ebextensions/git.sandbox-dev.json |
+| [AWS CodePipeline Deployment via CodeConnections (GitHub → Elastic Beanstalk)](workflows/codepipeline-codeconnections-deploy.md) | 2.0 apps (`api2`, `_underscore`) are deployed through **AWS CodePipeline**. | api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, api2/ebs/register_instance_to_shared_application_load_balancer.php, api2/.platform/hooks/prebuild/git.sh, api2/.platform/hooks/postdeploy/015_install_composer.sh, api2/.ebextensions/git.php, api2/.ebextensions/git.sandbox-dev.json |
 | [New Environment Configuration & Provisioning (api2)](workflows/environment-configuration-and-provisioning.md) | What it takes for a 2.0 API environment (e.g. | api2/Config/<environment>.ini, api2/composer.json, api2/composer.lock, api2/Controller/Index.php, dbchanges2/Core/2026-06-16a - DatabaseHosts for new QA QC stage demo environments.sql, dbchanges2/Logs/, _underscore/Route.php |

package/knowledge/2.0/apps/api2/architecture.md

@@ -6,7 +6,7 @@ project: API
 client: shared
 type: architecture
 status: active
-updated: 2026-08-26
+updated: 2026-08-28
 owners: [jcardinal, bala, mhammontree, dfranks]
 files:
   - api2/Controller/Index.php
@@ -197,6 +197,12 @@ always exits 0). See
 [the worker2 reference implementation](../worker2/features/alb-target-group-auto-registration.md) —
 api2's copy is the **unhardened original** and must be brought up to it.
 
+**Deploy policy (2026-08-28):** the three production envs (`api-production-1/2/3`, 2 instances each)
+run **Rolling / `BatchSize` 1** (was `AllAtOnce`/100%), target-group `HealthyThresholdCount` 2 — so
+a deploy keeps ≥1 healthy backend and no longer emits the ELB-502 blip. Each CodePipeline
+auto-deploys on a push to `_production`. See
+[the CodePipeline deploy workflow](workflows/codepipeline-codeconnections-deploy.md).
+
 ## CI — commit message policy
 
 `.github/workflows/true-devteam-requirements.yml` enforces **`TRUE-{ticket}: {Subject}`** —
@@ -254,6 +260,48 @@ interceptor's plain-`Exception` rejections — all now returning 4xx/fixed. Dist
 reading the ALB 5xx metric and `Logs.Issue`: zero 5xx + a traffic spike = transient EB snapshot;
 a sustained 5xx rate = a code bug to fix.
 
+**A third cause — a `Degraded`/`Severe` flip with NO stated cause is the PHP-served `/health`
+probe being STARVED, not client traffic failing.** Confirmed on `api-production-1` (acct
+`654654170868`): off-peak **bursts of slow, failing `POST /v2/advance-shipping-notices`**
+(Compass ASN; 17–63 s each, ~100% HTTP 500) saturate the **PHP-FPM worker pool** on an
+instance. The ALB `/health` check — historically **PHP-served, 5 s timeout** — then can't get a
+worker and fails a few checks, so EB flags the target group "reduced health" **with no cause
+cited**. Two reasons this looks invisible: (1) **health-check traffic does NOT appear in ALB
+`RequestCount`/`5xx` metrics**, so the ALB view reads clean; (2) **`HealthyHostCount` never
+drops**, because the probe failure is shorter than the **5-check / 75 s deregistration
+threshold** — it flaps EB enhanced-health without ever deregistering a target. A separate, benign
+contributor is genuine **sub-minute `/health` transients** on a single instance. Correlation
+proof: a **06:00–06:08 CST ASN burst == the 11:05 UTC EB `Severe` flip**. Diagnosed via AWS CLI
+(`describe-events`, `describe-configuration-settings`, CloudWatch `get-metric-data`) + prod
+`Logs.Api`/`Logs.Issue`.
+
+**Rule:** a "reduced health / no cause" flip on this 2-instance PHP env means the ALB `/health`
+probe was starved of a PHP-FPM worker — do **not** chase client-traffic 5xx. **Fix shipped
+(working tree):** `/health` is now served as a **static Apache file** (`api2/health`), decoupling
+liveness from PHP-FPM worker availability so a slow-request burst can no longer flap it — see
+[health-check endpoint](features/health-check-endpoint.md).
+
+**A fourth cause (deploy-triggered) — a burst of ELB `502`s when both instances restart at once.**
+Confirmed on `api-production-1` (2026-08-28 23:51 UTC deploy: **43 × 502 in one minute**, the only
+ELB 5xx in a 7-hour window). Under the old `AllAtOnce`/100% deploy policy, every deploy to a
+2-instance env restarts Apache/PHP-FPM on **both** instances simultaneously, so in-flight requests
+have their connections dropped mid-response → a burst of **ELB-generated HTTP 502**. EB flags "X%
+of requests to the ELB are failing with HTTP 5xx" `Degraded` ~3–4 min later and recovers in ~5 min.
+Distinguishing fingerprint vs. a real app failure: it is **502** (a dropped backend connection),
+**not 503** (no healthy backend) or **504** (backend timeout); **`HTTPCode_Target_5XX_Count` (app
+5xx) = 0**; **`HealthyHostCount` never drops below 2**; and there is **NOTHING in `Logs.Api`** —
+api2 logs a request only *after* the response, and a 502 never completes one, so the request leaves
+no log row. Diagnosis path: CloudWatch `HTTPCode_ELB_502/503/504_Count` vs
+`HTTPCode_Target_5XX_Count`, plus EB `describe-events` (a version change) and CodePipeline
+`list-pipeline-executions` — because a push to `_production` auto-deploys all three prod tiers, "we
+didn't deploy" can be false (see
+[the CodePipeline deploy workflow](workflows/codepipeline-codeconnections-deploy.md)).
+**Rule:** an EB "% 5xx" `Degraded` with an **empty `Logs.Api` and 0 target-5xx** right after a
+deploy is a deploy-restart 502 artifact, not an application bug — check the ELB-5xx breakdown and
+the deploy/version timeline before hunting app bugs. **Fixed 2026-08-28:** all three prod envs moved
+to **Rolling / batch-size 1** (≥1 healthy backend throughout a deploy), which eliminates the blip —
+config + tradeoffs in the [deploy workflow](workflows/codepipeline-codeconnections-deploy.md).
+
 ## Known issues / accepted risks
 
 Open items a maintainer should know before changing this tier. None are "bugs to fix right now" —
@@ -314,7 +362,11 @@ they are the known sharp edges. Do not re-discover these from scratch.
   class lives in `_underscore` (`Model/Client/...`).
 - **Client-specific behavior** belongs in `_Model_<ClientSlug>_X` overrides and
   `Client_ApiPayloadInterceptor` pre/post hooks — not in `V2.php`.
-- **Don't break `/health`** or weaken `enforce_https.conf`.
+- **Don't break `/health`.** It is now a **static Apache-served file** (`api2/health`, body
+  `OK`) — **not** a PHP short-circuit — so the EB/ALB liveness probe is decoupled from PHP-FPM
+  worker availability. Do **not** route `/health` back through PHP or remove the file, and don't
+  weaken `enforce_https.conf`. (`/v2/health` remains PHP-handled.) See
+  [health-check endpoint](features/health-check-endpoint.md).
 - **504s:** verify `long_gateway_timeout.conf` on both this tier and the proxy
   (LB 3600s → proxy Apache 1800s → proxy cURL 900s → api Apache 1800s).
 - **`transactionId` uniqueness is load-bearing** — the idempotency/audit key; don't bypass
@@ -327,6 +379,16 @@ they are the known sharp edges. Do not re-discover these from scratch.
   single-caller branches in `V2.php` as unverified until exercised directly.
 
 ## Change history
+- 2026-08-28 — Recorded a **fourth EB-`Degraded` cause — the deploy-restart ELB 502**: under the old
+  `AllAtOnce`/100% policy a deploy restarts both instances of a 2-instance env at once, dropping
+  in-flight connections → an ELB-502 burst (43 in one minute on `api-production-1`, 2026-08-28) that
+  EB flags as "X% 5xx" Degraded ~3–4 min later. Fingerprint vs. an app bug: **502** (not 503/504),
+  **`HTTPCode_Target_5XX_Count`=0**, `HealthyHostCount` never < 2, and **nothing in `Logs.Api`** (a
+  502 never completes a PHP response, which api2 logs only after). Diagnose via the ELB-5xx breakdown
+  + `describe-events`/`list-pipeline-executions` — a push to `_production` auto-deploys all three prod
+  tiers, so "we didn't deploy" can be false. Fixed by moving to Rolling/batch-1 (deploy config +
+  policy tradeoffs captured in the CodePipeline deploy workflow). (jcardinal)
+- 2026-08-28 — Recorded the third EB-`Degraded` cause: a no-stated-cause "reduced health" flip on `api-production-1` is the **PHP-served `/health` probe starved of a PHP-FPM worker** by off-peak bursts of slow failing Compass ASN POSTs (17–63 s, ~100% 500) — invisible in ALB `RequestCount`/`5xx` and never dropping `HealthyHostCount` (flap < 75 s deregistration). Correlated a 06:00–06:08 CST ASN burst to the 11:05 UTC Severe flip. Noted the fix (static-file `/health` decouples liveness from PHP-FPM) and updated the "Don't break `/health`" rule to the static-file form. (jcardinal)
 - 2026-08-26 — Documented and fixed a `/v2/auth/public` OOM: a present-but-null client `transactionId` (`{"transactionId": null}`) flowed into the request-logger's `_Model::load()` dup-check; because `Logs.Api.transactionId` is `FIELD_CHAR`, a null value collapses the WHERE and full-table-scans prod `Logs.Api` → memory exhaustion (the `_underscore` collapsed-WHERE mechanism). V2 only auto-uuids an *absent* transactionId (~L543), so a present null fell through (~L553 read → null). Fix (commit `36ef31f`): coerce null/empty transactionId → uuid in the auth block, and skip `load()` in the final logger dup-check (~L2277) when null/empty. Added the "don't blame the latest deploy — check first-occurrence" triage note. (jcardinal)
 - 2026-08-25 — **Resolved the auto-generated `Api.transactionId` 1062 collision** (was the 2026-07-28
   gotcha): the two DB-insert log sites now uuid the log-row `transactionId` (`_String::generateUuid()`)

package/knowledge/2.0/apps/api2/features/environment-variable-drives-underscore-branch.md

@@ -6,10 +6,11 @@ project: API
 client: shared
 type: feature
 status: active
-updated: 2026-08-25
-owners: ["bala", "mhammontree", "apeterson"]
+updated: 2026-08-28
+owners: ["bala", "mhammontree", "apeterson", "jcardinal"]
 files:
   - api2/.ebextensions/git.php
+  - api2/.platform/hooks/prebuild/git.sh
   - api2/.ebextensions/php_include_underscore.config
   - api2/.ebextensions/git.sandbox-dev.json
   - api2/.ebextensions/git.sandbox-client.json
@@ -177,6 +178,21 @@ the tier trades a stale-code bug for a total outage. Full procedure:
 
 ## Gotchas / known issues
 
+- **⚠ Cross-repo change ordering: land `_underscore` BEFORE deploying api2.** Because api2's EB
+  deploy **clones `_underscore` (branch `_<ENVIRONMENT>`, e.g. `_production`) at BUILD time**
+  (`.platform/hooks/prebuild/git.sh`), a change that spans both repos must have its `_underscore`
+  side merged to `_<ENVIRONMENT>` **first** — otherwise the freshly-built api2 will call a framework
+  method/class that isn't on the box and fatal. Landing `_underscore` first is **safe on its own**:
+  the old api2 controller simply doesn't use the new framework code yet. Order: merge `_underscore`
+  → (verify it's on the branch) → deploy api2.
+- **⚠ A "Could not find required file for `_Model_...`" fatal on prod can be a STALE-BUILD
+  artifact, not a missing file.** `Logs.Issue #476`
+  ("Could not find required file for '_Model_Client_ItemFulfillments_InventoryAdjustment'") fired
+  because that model file **was** committed to `_production` but `api-production-1` was still running
+  an **older build**; a **redeploy** (which re-pulls `_production`) resolved it with no recurrence.
+  Before assuming a model file is absent, check the **running build's `_underscore` against
+  `_production`** (file presence on the box; `git merge-base --is-ancestor`) — the build-time clone
+  means a merged file may simply not be on that instance yet.
 - **⚠ Merging to the branch named after the EB environment can be a no-op.** Merge to
   `_` + `ENVIRONMENT`. If your change "deployed successfully" but the behavior is absent and no
   error appears, check this before debugging the code. A missing class also throws `Error`
@@ -241,6 +257,14 @@ the tier trades a stale-code bug for a total outage. Full procedure:
 
 ## Change history
 
+- 2026-08-28 — Recorded two build-time-clone consequences: (1) **cross-repo deploy ordering** — api2
+  clones `_underscore` (`_<ENVIRONMENT>`) at build via `.platform/hooks/prebuild/git.sh`, so for a
+  change spanning both repos, `_underscore` must land on the deploy branch **before** api2 is
+  deployed (landing `_underscore` first is safe alone); and (2) a **"Could not find required file
+  for `_Model_...`" fatal can be a stale build**, not a missing file — `Logs.Issue #476` fired for
+  a model committed to `_production` while `api-production-1` ran an older build, and a redeploy
+  resolved it. Check the running build's `_underscore` vs `_production` before assuming the file is
+  absent. (jcardinal)
 - 2026-08-27 — Recorded the **inverse hazard**: moving a tier **forward** onto its correct
   `_underscore` branch is a **schema event**. The 2026-08-26 deploy moved `sandbox-client` off the
   `_production` framework (which the 2026-08-25 entry below explains it had been running for weeks)

package/knowledge/2.0/apps/api2/features/health-check-endpoint.md

@@ -1,14 +1,16 @@
 ---
-title: "Health-check endpoint (/health liveness short-circuit)"
+title: "Health-check endpoint (/health liveness — static file + PHP short-circuit)"
 framework: "2.0"
 repo: api2
 project: API
 client: shared
 type: feature
 status: active
-updated: 2026-07-24
+updated: 2026-08-28
 owners: [jcardinal]
 files:
+  - api2/health
+  - api2/.platform/httpd/conf.d/health_probe.conf
   - api2/Controller/Index.php
 related:
   - ../architecture.md
@@ -17,13 +19,37 @@ related:
 
 ## Summary
 
-`_Controller_Index::api()` short-circuits **liveness/health-probe** requests to an HTTP 200
-**before** any routing, DB bootstrap, or V2 engine work runs. EB/LB (Elastic Beanstalk /
-load balancer) probes depend on this always returning 200 cheaply — the architecture doc's
-critical rule is literally "Don't break `/health`". This doc records the **matching contract**
-so a teammate doesn't reintroduce a brittle exact-string check.
+The EB/ALB liveness probe (`/health`) is served two ways, and the **load-bearing** one is now
+a **static Apache-served file**, not PHP:
 
-## How it works
+- **`/health` → a static file `api2/health`** (literal body `OK`), served by Apache **without
+  ever reaching PHP-FPM**. This is what the EB target group health-check hits. Decoupling the
+  probe from PHP means a PHP-FPM worker-pool exhaustion (e.g. a burst of slow ASN requests
+  saturating all workers) or a PHP transient can **no longer flap EB health** — the probe
+  answers from Apache regardless of app load.
+- **`/v2/health` (and `/health` as a fallback) → PHP short-circuit** in
+  `_Controller_Index::api()`, an HTTP 200 returned **before** any routing, DB bootstrap, or V2
+  engine work. This still exists for callers that hit the PHP entrypoint directly.
+
+This doc records the **matching contract** for both so a teammate doesn't reintroduce a brittle
+exact-string check or accidentally route the liveness probe back through PHP.
+
+## The static probe (the one EB actually uses) — 2026-08-28
+
+- **`api2/health`** — a one-line static file (`OK`). Apache serves it directly.
+- **`api2/.platform/httpd/conf.d/health_probe.conf`** — comment-only documentation of why the
+  probe is static (no directives; it exists to explain the decision at the Apache layer).
+- **Why it works without a rewrite fight:** `api2/.htaccess` only rewrites to `index.php` when
+  the request target is **not a real file** (`RewriteCond %{REQUEST_FILENAME} !-f`). Because
+  `health` is a real file on disk, the rewrite is skipped and Apache serves it straight.
+- **Why `enforce_https` doesn't redirect it:** EB health checks arrive on **HTTP:80 with no
+  `X-Forwarded-Proto`**, and `enforce_https.conf` only redirects requests that carry the
+  forwarded-proto marker — so the probe is not bounced to HTTPS.
+- **Do not** move `/health` back into a PHP-served route or rename/remove `api2/health` — that
+  re-couples EB liveness to PHP-FPM worker availability, which is the exact failure this fixed
+  (see [api2 architecture — EB "Degraded" health](../architecture.md)).
+
+## PHP short-circuit (for `/v2/health` and direct hits)
 
 The short-circuit lives at the top of `Index.php::api()`, before host dispatch and before the
 Core/Logs DB bootstrap. It matches against a class constant:
@@ -55,6 +81,15 @@ normalized form: lowercase, no trailing slash, no query) rather than re-adding a
 branch.
 
 ## Change history
+- 2026-08-28 — Moved the **EB/ALB liveness probe off PHP**: `/health` is now a **static file**
+  (`api2/health`, body `OK`) served by Apache — it never reaches PHP-FPM, so a slow-request burst
+  or a PHP transient can no longer starve the probe and flap EB health. Works because
+  `.htaccess` only rewrites to `index.php` for non-files (`!-f`), and `enforce_https` skips the
+  probe (HTTP:80, no `X-Forwarded-Proto`). Added a comment-only
+  `.platform/httpd/conf.d/health_probe.conf` documenting the decision. `/v2/health` stays
+  PHP-handled and the `HEALTH_CHECK_ROUTES` PHP short-circuit remains as a fallback. Working tree
+  only (uncommitted). See the [EB "Degraded" health](../architecture.md) diagnosis this fixes.
+  (jcardinal)
 - 2026-07-24 — Replaced the brittle exact-string `REQUEST_URI == '/health'` check with a
   `HEALTH_CHECK_ROUTES` constant + normalized (lowercase / strip query / strip trailing slash)
   strict `in_array` match, so `/v2/health`, `/health/`, and `/health?x` all short-circuit to 200

package/knowledge/2.0/apps/api2/features/v2-deadlock-retry.md

@@ -1,38 +1,101 @@
 ---
-title: V2 request deadlock-retry — route-scoped in-process replay
+title: V2 request retry (Prong A) — deadlock AND retriable unique-key race, route-scoped in-process replay
 framework: "2.0"
 repo: api2
 project: API
 client: shared
 type: feature
 status: draft
-updated: 2026-08-13
+updated: 2026-08-28
 owners: [jcardinal]
 files:
   - api2/Controller/Index.php
+  - api2/Component/Api/V2/V2.php
   - _underscore/Database.php
   - _underscore/Query.php
+  - _underscore/Model/Compass/AdvanceShippingNotice.php
 related:
   - ../../_underscore/features/model-save-parent-cascade-stored-field-deadlock.md
+  - ../../../../clients/compass-usa/features/asn-to-item-fulfillment.md
   - ../architecture.md
   - ./v2-api-error-codes.md
 ---
 
 ## Summary
 
-api2's front controller can **detect a MySQL deadlock (1213) / lock-wait timeout (1205) and replay
-the whole request in-process**, so a transient lock collision self-recovers instead of returning an
-HTTP 400 to the caller. The replay is **route-scoped by an allowlist**
-(`DEADLOCK_RETRIABLE_ROUTES`) — currently only `/v2/advance-shipping-notices` — because whole-
-request in-process replay is **only safe for routes with no non-rollbackable external side effect**.
-This is the safety-net half of the ASN deadlock fix; the source-level fix (stop the collateral
-parent re-save) is in
-[_underscore's save-cascade doc](../../_underscore/features/model-save-parent-cascade-stored-field-deadlock.md).
+api2's front controller ("**Prong A**") can **detect a retriable concurrency failure and replay
+the whole request in-process**, so a transient collision self-recovers instead of returning an
+error to the caller. Two failure classes now qualify:
 
-> Status (2026-08-13): **in the working tree only** — not committed or deployed. Pending sandbox
+1. **MySQL deadlock (1213) / lock-wait timeout (1205)** — the original case (a same-PO ASN write
+   deadlock; see the
+   [save-cascade doc](../../_underscore/features/model-save-parent-cascade-stored-field-deadlock.md)).
+2. **Retriable unique-key get-or-create race (1062)** — a concurrent same-key INSERT lost the
+   race; a whole-request replay on a fresh snapshot lets the retry's GET reuse the winner's
+   committed row (added 2026-08-28 for the Compass ASN data-loss race).
+
+The replay is **route-scoped by an allowlist** (`DEADLOCK_RETRIABLE_ROUTES`) — currently only
+`/v2/advance-shipping-notices` — because whole-request in-process replay is **only safe for routes
+with no non-rollbackable external side effect**. The retry gate is now
+**`wasDeadlock() || wasRetriableUniqueRace()`**.
+
+> Status (2026-08-28): **in the working tree only** — not committed or deployed. Pending sandbox
 > validation + regression tests.
 
-## How it works
+## The retriable unique-key race (1062) — Compass ASN data-loss (2026-08-28)
+
+`_Model_Compass_AdvanceShippingNotice::postPost` **get-or-creates** an SO-numbered Item
+Fulfillment (GET by number, else POST). `ItemFulfillments.number` is UNIQUE and vendors
+re-transmit the same shipment file, so two concurrent same-number ASNs race: the loser's INSERT
+trips the unique key. Pre-fix, `internalApiRequest` threw (default `throwExceptionsOnError=true`),
+the 500 propagated out of `postPost`, and **the api2 controller rolled back the whole request —
+losing the ASN row and its tracking numbers** (`Logs.Issue #239`, EV-10). This is a bug 1062
+must NOT be treated like a normal error, because a replay recovers it cleanly.
+
+**Fix (Approach B — a CTO review rejected a simpler in-place `FOR UPDATE` recovery, because an
+interceptor cannot recover a poisoned response in place; see *V2 concurrency mechanics* below).**
+A generalized retriable-unique-race path:
+
+- **`_Database`** gains `const ERRNO_DUPLICATE_ENTRY = 1062`, a sticky
+  `public static bool $retriableUniqueRace` (reset per attempt in `clearLastError()`), and
+  `wasRetriableUniqueRace()` — true **only** when the caller-set flag is set **AND**
+  `lastErrorNumber === 1062`. The flag is caller-opt-in on purpose: a bare 1062 is a real
+  constraint violation and must **not** silently retry; only a caller that knows the 1062 is a
+  benign get-or-create race sets it.
+- **The ASN model** calls the IF-create `internalApiRequest` with **`throwExceptionsOnError:false`**;
+  on an empty response with `lastErrorNumber === 1062` it sets `$retriableUniqueRace = true` and
+  returns null. `postPost` then **aborts immediately** (issues no further query).
+- **The controller's Prong A gate** becomes `wasDeadlock() || wasRetriableUniqueRace()`. Replaying
+  the whole request on a fresh REPEATABLE-READ snapshot lets the retry's GET see the winner's
+  committed IF and reuse it.
+
+Bounded by `MAX_DEADLOCK_ATTEMPTS = 3`, gated to `DEADLOCK_RETRIABLE_ROUTES =
+['/v2/advance-shipping-notices']`.
+
+## V2 concurrency mechanics (durable framework knowledge)
+
+These are the V2-engine facts that decide **how any interceptor-level write race must be handled**
+— they are why the fix is a whole-request replay and not an in-place recovery:
+
+- **Response poisoning (load-bearing).** A failed nested save via `internalApiRequest` calls
+  `addDefinedMessage(EV-10)` on the **shared** `$api->response`, setting a non-2xx status **before**
+  control returns. So even if an interceptor catches the throw and "recovers," `execute()`
+  recomputes `isSuccess = false` (`V2.php` ~2154) and **self-rolls-back DB_CORE + DB_CLIENT**
+  (~2157-2159). **An interceptor therefore CANNOT recover a failed internal save in place** — the
+  only correct recovery is a whole-request replay. This same poisoning is what drives the Prong A
+  retry gate and, on exhaustion, produces the coherent EV-10 response.
+- **`internalApiRequest($method, $route, $payload, $options, $throwExceptionsOnError = true)`** — a
+  **5th argument** controls throw-vs-return-on-error. Pass `false` to inspect `lastErrorNumber`
+  instead of unwinding.
+- **`_Database::$lastErrorNumber` is sticky.** Set only on query failure, never cleared by a later
+  success; reset per attempt via `clearLastError()`. A recorded **deadlock (1213/1205) is protected
+  from overwrite** by a later error; a **1062 is NOT** protected — so read/act on a 1062 promptly.
+- **The prod client cluster runs REPEATABLE-READ.** A same-transaction re-SELECT after a losing
+  concurrent INSERT is **blind to the winner's commit** — only a locking read or a **fresh-snapshot
+  whole-request replay** sees it. This is why in-place retry inside the transaction cannot work and
+  the replay must start a new transaction.
+
+## How it works — deadlock path
 
 **Detection primitive (`_underscore`, reusable).** `_Database` gains
 `public static ?int $lastErrorNumber`, the consts `ERRNO_DEADLOCK` (1213) and
@@ -42,7 +105,8 @@ driver errno **at its throw site**, and the recording is **sticky**: a later unr
 inside a model-save `try/catch` still be visible to the outer controller and trigger a retry.
 
 **Retry loop (`api2/Controller/Index.php`).** The v2 dispatch is wrapped in a bounded loop
-(`MAX_DEADLOCK_ATTEMPTS = 3`). On an unsuccessful response where `_Database::wasDeadlock()` is true
+(`MAX_DEADLOCK_ATTEMPTS = 3`). On an unsuccessful response where
+`_Database::wasDeadlock() || _Database::wasRetriableUniqueRace()` is true
 and the route is in `DEADLOCK_RETRIABLE_ROUTES`, the controller:
 
 1. rolls back all open transactions;
@@ -82,6 +146,22 @@ superglobal. Default is to leave a route off the list.
   not a substitute for removing the deadlock at its source.
 
 ## Change history
+- 2026-08-28 — **Generalized Prong A to a second retriable class: the unique-key (1062)
+  get-or-create race**, fixing the Compass ASN data-loss bug (`Logs.Issue #239`, EV-10) where a
+  concurrent same-`ItemFulfillments.number` INSERT lost the race, `internalApiRequest` threw, and
+  the whole request rolled back — losing the ASN row and its tracking. Added
+  `_Database::ERRNO_DUPLICATE_ENTRY = 1062`, a sticky caller-opt-in `$retriableUniqueRace`
+  (reset in `clearLastError()`), and `wasRetriableUniqueRace()` (flag AND `lastErrorNumber===1062`);
+  the ASN model calls the IF-create with `throwExceptionsOnError:false`, sets the flag on an empty
+  1062 response, and aborts; the controller gate became
+  `wasDeadlock() || wasRetriableUniqueRace()`. Chose the whole-request replay (Approach B) over an
+  in-place `FOR UPDATE` recovery after a CTO review, because V2 **response poisoning** (a failed
+  nested save flips the shared `$api->response` to non-2xx and `execute()` self-rolls-back
+  DB_CORE+DB_CLIENT, `V2.php` ~2154-2159) makes in-place interceptor recovery impossible, and the
+  prod cluster's REPEATABLE-READ isolation makes a same-transaction re-SELECT blind to the winner's
+  commit. Recorded the durable V2 concurrency mechanics (response poisoning, `internalApiRequest`
+  5th arg, sticky-`lastErrorNumber` with 1213 protected / 1062 not, REPEATABLE-READ). Working tree
+  only, pending sandbox validation. (jcardinal)
 - 2026-08-13 — Added deadlock/lock-wait detection to `_Database`/`_Query` (`$lastErrorNumber`,
   `wasDeadlock()`, sticky errno recording) and a bounded, route-scoped in-process replay loop in
   `Controller/Index.php`, gated to `/v2/advance-shipping-notices` only. Documented why whole-request

package/knowledge/2.0/apps/api2/workflows/codepipeline-codeconnections-deploy.md

@@ -6,12 +6,13 @@ project: API
 client: shared
 type: workflow
 status: active
-updated: 2026-08-25
+updated: 2026-08-28
 owners: ["jcardinal", "mhammontree", "bala", "apeterson"]
 files:
   - api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh
   - api2/ebs/register_instance_to_shared_application_load_balancer.php
   - api2/.platform/hooks/prebuild/git.sh
+  - api2/.platform/hooks/postdeploy/015_install_composer.sh
   - api2/.ebextensions/git.php
   - api2/.ebextensions/git.sandbox-dev.json
 related:
@@ -62,8 +63,11 @@ are actually changing before you merge anything:
   `_<environment>`). Consequence: **a framework-only change requires REDEPLOYING THE CONSUMING
   TIER even when `api2` itself has zero changes** — an instance **restart will not pick it up**,
   because the clone happens in prebuild.
-- **`api2` has no deploy workflow.** Its only GitHub Action is the `TRUE-#####:` commit-message
-  linter, so **EB deploys are manual**. Do not wait for CI to ship a framework change.
+- **`api2` has no GitHub *Actions* deploy workflow** — its only GitHub Action is the
+  `TRUE-#####:` commit-message linter. **The production tiers, however, auto-deploy via AWS
+  CodePipeline on every push to `_production`** (a different system from GitHub Actions — see
+  *Production deploy: auto-trigger + rolling policy* below). Non-pipeline-triggered tiers are
+  still deployed by hand; don't wait for a GitHub Action to ship a framework change.
 
 ### ⚠ The `_underscore` SOURCE ACTION has its own branch trigger — and a green pipeline hides a wrong one
 
@@ -105,6 +109,64 @@ beta env actually received and returned when Sentry isn't wired up. (The Rate AI
 scans this same `Logs_Rate.Api` for failed outbound calls — see
 `clients/rate/features/aig-contract-creation.md`.)
 
+## Production deploy: auto-trigger + rolling policy (2026-08-28)
+
+### Production auto-deploys on every push to `_production` — a merge IS a deployment
+The `api-production-1` (us-east-1), `api-production-2` (us-west-2), and `api-production-3`
+(eu-west-1) CodePipelines each **auto-deploy on every push to branch `_production`** — a
+**WebhookV2** trigger on the CodeConnections GitHub connection, watching source repos
+`agilantsolutions/_underscore` and `agilantsolutions/api2`. There is **no separate manual deploy
+step for production**: pushing or merging to `_production` **is** a production deployment to all
+three regions. (This is why deploying a fix can itself page a moment later — see the
+deploy-restart 502 below.)
+
+**Triage consequence:** "we didn't deploy" can be false — any push to `_production` is a deploy.
+Correlate an incident against the actual pipeline runs, not just remembered manual actions:
+```bash
+aws codepipeline list-pipeline-executions --pipeline-name api-production-1 --region us-east-1
+aws elasticbeanstalk describe-events --environment-name api-production-1 --region us-east-1  # version-change events
+```
+
+### ⚠ An EB `Degraded` right after a deploy = the deploy-restart 502
+Under the old `AllAtOnce`/100% policy every deploy restarted Apache/PHP-FPM on **both**
+instances of a 2-instance env simultaneously, dropping in-flight connections → a burst of
+ELB-generated **HTTP 502** that EB flags as an "X% 5xx" `Degraded` ~3–4 min later (recovers in
+~5 min). This is a **deploy artifact, not an application failure** — the full diagnostic
+fingerprint (502 not 503/504, app `HTTPCode_Target_5XX_Count`=0, `HealthyHostCount` never < 2,
+**nothing in `Logs.Api`** because a 502 never completes a PHP response) lives in
+[api2 architecture — EB "Degraded" health](../architecture.md). The rolling policy below
+eliminates it.
+
+### Deploy policy: Rolling, batch size 1 (changed 2026-08-28)
+All three production envs (app **"API"**, 2 instances each) were switched from
+`AllAtOnce`/`Percentage`/`100` to **`DeploymentPolicy=Rolling`, `BatchSizeType=Fixed`,
+`BatchSize=1`**, and the target group **`HealthyThresholdCount` dropped 3 → 2** (health path
+`/health`, `HealthCheckInterval` 15 s, `UnhealthyThresholdCount` 5 unchanged). Rolling batch-1 on
+a 2-instance env deploys in **2 batches with ≥1 healthy backend throughout**, so a deploy no
+longer drops in-flight requests → **no more deploy-restart 502 blip**; the 3→2 threshold shaves
+one 15 s interval (~45 s → 30 s) of per-batch health settle.
+
+- Applied via a **config-only `update-environment`** (AWS CLI, per env) — no instance
+  replacement, no downtime; **takes effect on the NEXT deploy**.
+- Namespaces: `aws:elasticbeanstalk:command` (`DeploymentPolicy` / `BatchSizeType` /
+  `BatchSize`); `aws:elasticbeanstalk:environment:process:default` (`HealthyThresholdCount`).
+- Trade-off: Rolling is slower end-to-end than the old ~31 s `AllAtOnce`, and capacity briefly
+  drops to 1/2 instances per batch (fine at current traffic).
+
+### Deploy-policy guidance (don't reach for immutable expecting speed)
+- **Immutable does NOT reduce deploy time.** It is the **slowest** policy — it boots a fresh
+  instance batch, health-checks it, then swaps, and temporarily **doubles** the instance count.
+  It does fully eliminate deploy 5xx and gives instant rollback, but it is not "faster."
+- **`AllAtOnce` is fast (~31 s) precisely BECAUSE both instances restart at once** — which is
+  what causes the 502s. You cannot have "as fast as AllAtOnce" **and** "zero 502s."
+- **Rolling batch-1 is the sweet spot** for a 2-instance env.
+- **`RollingWithAdditionalBatch`** is the option only if zero capacity loss during a deploy is
+  ever required (it adds a temporary instance; slower and costlier).
+- The real deploy-**time** levers are independent of the policy: the health-check settle
+  (`HealthCheckInterval` × `HealthyThresholdCount`) and per-instance postdeploy work — api2 runs
+  `composer install` on **every** deploy via
+  `.platform/hooks/postdeploy/015_install_composer.sh` (a caching / pre-bake candidate).
+
 ## Steps
 1. A push to the watched branch (e.g. `_stage`) triggers the pipeline.
 2. The Source stage uses `CodeStarSourceConnection` actions, each referencing:
@@ -233,6 +295,20 @@ aws codeconnections get-connection --connection-arn "<CONN_ARN>" --region "$REGI
 ```
 
 ## Change history
+- 2026-08-28 — Recorded the **production deploy trigger + policy change**: the `api-production-1/2/3`
+  CodePipelines **auto-deploy on every push to `_production`** (WebhookV2 on the CodeConnections
+  connection, repos `_underscore` + `api2`), so a merge to `_production` **is** a three-region prod
+  deploy — correlate incidents against `list-pipeline-executions`, not remembered manual actions.
+  Switched all three 2-instance envs from `AllAtOnce`/100% to **Rolling / Fixed / BatchSize 1** and
+  dropped the target-group **`HealthyThresholdCount` 3 → 2** (config-only `update-environment`,
+  effective next deploy) to eliminate the **deploy-restart 502 blip** (both instances restarting at
+  once dropped in-flight requests → ELB 502 → EB "X% 5xx" Degraded). Added deploy-policy guidance:
+  immutable is the **slowest** policy (not faster) and doubles instances; `AllAtOnce` is fast only
+  because it restarts both at once (the 502 cause); Rolling batch-1 is the sweet spot;
+  `RollingWithAdditionalBatch` only if zero capacity loss is needed; the real deploy-time levers are
+  the health-check settle and the per-deploy `015_install_composer.sh` `composer install`. Corrected
+  the "EB deploys are manual" note — that is only true of the absent GitHub *Actions* workflow; prod
+  deploys via CodePipeline are automatic. (jcardinal)
 - 2026-08-25 — Documented that the Source stage's **`_underscore` action has its own AWS-side branch
   trigger**, independent of `git.<env>.json` and invisible in the repo — a second way to silently run
   the wrong framework branch. `ApiSandboxClient` ran `_production` from a mis-set source action while

package/knowledge/clients/compass-usa/features/asn-to-item-fulfillment.md

@@ -5,7 +5,7 @@ project: _Underscore
 client: compass-usa
 type: client-feature
 status: active
-updated: 2026-08-27
+updated: 2026-08-28
 owners: [jcardinal, bala]
 files:
   - _underscore/Model/Compass/AdvanceShippingNotice.php
@@ -200,6 +200,20 @@ Compass Canada (`Model/Compass/Canada/`) is a separate sub-client. The Compass c
   (`$_model_skipParentCascadeOnSave`), and api2 retries the ASN route on a detected deadlock. Full
   mechanic:
   [save-cascade stored-field deadlock](../../../2.0/apps/_underscore/features/model-save-parent-cascade-stored-field-deadlock.md).
+- **Concurrent same-SO-number ASNs lost the ASN + its tracking to a unique-key race (fixed
+  2026-08-28, working tree).** `postPost` get-or-creates the SO-numbered IF (GET by number, else
+  POST). `ItemFulfillments.number` is UNIQUE and vendors re-transmit the same shipment file, so two
+  concurrent same-number ASNs race: the loser's IF INSERT hits the unique key (MySQL **1062**),
+  `internalApiRequest` threw (default `throwExceptionsOnError=true`), the 500 propagated out of
+  `postPost`, and the api2 controller **rolled back the whole request — losing the ASN row AND its
+  tracking numbers** (`Logs.Issue #239`, EV-10). Fix: the IF-create call now passes
+  `throwExceptionsOnError:false`; on an empty response with `lastErrorNumber===1062` the model sets
+  `_Database::$retriableUniqueRace = true` and returns null, `postPost` aborts, and api2's Prong A
+  replays the whole request on a fresh REPEATABLE-READ snapshot so the retry's GET reuses the
+  winner's committed IF. Distinct from the 1213 deadlock above (a different concurrency failure on
+  the same re-transmission traffic). Full mechanic + why in-place recovery is impossible (V2
+  response poisoning):
+  [V2 request retry — Prong A](../../../2.0/apps/api2/features/v2-deadlock-retry.md).
 - **Cross-line tracking contamination from bad SOI↔POI bridge rows (root cause, fixed
   forward 2026-06-11):** `postPost` resolves which IFI(s) a tracking number attaches to by
   joining `AdvanceShippingNoticeItems → SalesOrderItems_PurchaseOrderItems → SalesOrderItems`.
@@ -313,6 +327,13 @@ feed cannot currently support a line-level completeness test.
 
 ## Change history
 Dated one-liners, newest first.
+- 2026-08-28 — Fixed a **data-loss unique-key race** on concurrent same-SO-number ASNs: the loser's
+  SO-numbered IF INSERT trips `ItemFulfillments.number` UNIQUE (1062), `internalApiRequest` threw,
+  and api2 rolled back the whole request, losing the ASN row + tracking (`Logs.Issue #239`, EV-10).
+  The model now creates the IF with `throwExceptionsOnError:false`, flags a retriable 1062
+  (`_Database::$retriableUniqueRace`) and aborts, and api2's Prong A replays the request on a fresh
+  snapshot so the retry reuses the winner's IF. Distinct from the 2026-08-13 1213 deadlock. Working
+  tree only; also swept 6 banned `throw new _Exception(...)` → `\Exception` in the file. (jcardinal)
 - 2026-08-27 — Quantified the **Office Depot duplicate-ASN** problem on prod and settled what it is:
   of ODP PO lines with more than one shipping notice, **10,012 exceed the PO line quantity and only 8
   are genuine split shipments**, so a second notice on an ODP line is a **duplicate**, not a partial.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.694",
+  "version": "1.0.696",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",