toga-ai 1.0.867 → 1.0.869

package/knowledge/2.0/apps/_underscore/features/autoincrement-record-numbering.md

@@ -6,14 +6,15 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
-owners: ["bala"]
+updated: 2026-09-22
+owners: ["bala", "rgirish"]
 files:
   - _underscore/Model.php
   - _underscore/Model/Client/SalesOrder.php
   - _underscore/Model/Client/TransferOrder.php
 related:
   - ../architecture.md
+  - ../../api2/features/v2-deadlock-retry.md
   - ../../worker2/features/netsuite-transferorder-outbound-push.md
   - ../../worker2/features/netsuite-salesorder-outbound-push.md
 ---
@@ -42,6 +43,10 @@ Verified in `_underscore/Model.php` ~L546:
 2. **The next value is `MAX(numeric part) + 1` scoped by `WHERE number LIKE '<prefix>%'`.** So pre-existing numbers in another format are ignored: NYCHH's 1,930 legacy transfer-order numbers (e.g. `286906`) do not participate, and the sequence starts clean at `TA100000`.
 3. **There is NO zero padding.** The output is prefix + integer, so `TA00000` is not achievable without changing `Model.php`.
 
+**⚠ The generator is NOT concurrency-safe — read and insert can split.** `SELECT MAX(...)+1` and the `INSERT` are two separate statements with **no lock between them**. Two concurrent POSTs read the same MAX, pick the same number, and the loser gets **MySQL 1062** — a 400 to the caller, not a retry. Seen live on Prudential `/v2/service-requests` (`Duplicate entry 'SR131023' for key 'ServiceRequests.number'`).
+
+**Today's mitigation is a retry, not a fix.** api2 replays the whole request for two routes only (`/v2/advance-shipping-notices`, `/v2/service-requests`) — see [V2 request retry](../../api2/features/v2-deadlock-retry.md). **Every other `FIELD_AUTOINCREMENT` model has the same exposure and no mitigation**: SalesOrder, PurchaseOrder, Ticket, ItemReceipt, InventoryAdjustment, TransferOrder. A duplicate-number clash on those routes still returns 400. The durable answer is making the read and insert one operation in `Model.php`; that is a bigger, separate change and is **not done**.
+
 ## Gotchas
 
 - **⚠ A prefix on a SHARED model is not unique across clients.** Because the option lives on `_Model_Client_TransferOrder`, every tenant starts at `TA100000` — the scan is per client database. Any cross-client identifier must therefore NOT be keyed on `number`: NetSuite account **1095849 is shared by every client** and enforces `externalId` uniqueness account-wide, which is why both pushes key their external id on the record **uuid** (`toga-so-<uuid>` / `toga-to-<uuid>`) — see [the transfer-order push](../../worker2/features/netsuite-transferorder-outbound-push.md).
@@ -49,6 +54,7 @@ Verified in `_underscore/Model.php` ~L546:
   - **Keeping a Toga-side number permanently takes TWO changes, not one** (written 2026-09-02, then reverted at the developer's request — recorded so nobody rediscovers it): (1) a guard in `writeBackNetSuiteId()` so the push does not overwrite `number`, and (2) dropping `number` from the PUT in both `library/app/api/toga2.php` (~L4295) and `_underscore/Trait/Netsuite/TransferOrder.php`. Miss (2) and the 1.0 importer compares `number` and PUTs NetSuite's over the top on its next run, so the guard alone achieves nothing. Both files are back to untouched (verified with an empty `git diff`).
 - **`FIELD_AUTOINCREMENT` is one of the field types special-cased by `buildSqlFieldValue()`**, so a null value on it does NOT collapse a `load()` WHERE clause the way a null `FIELD_CHAR` does — see the collapsed-WHERE OOM note in [_underscore architecture](../architecture.md).
 - **Adding the option to a model does not renumber anything** and does not touch existing rows; it only affects inserts that arrive with `number` NULL.
+- **Concurrent inserts can pick the same number** (unlocked `MAX+1` then `INSERT`) → MySQL 1062 → 400. Only two api2 routes retry it; every other model is exposed. See [V2 request retry](../../api2/features/v2-deadlock-retry.md).
 - **A front-end that sends `number` silently disables the feature.** There is no error — the posted value is simply accepted. Check the actual POST body before concluding the generator is broken.
 
 ## Related
@@ -56,4 +62,7 @@ Verified in `_underscore/Model.php` ~L546:
 - [_underscore architecture](../architecture.md)
 - [NetSuite TransferOrder outbound push](../../worker2/features/netsuite-transferorder-outbound-push.md)
 - [NetSuite SalesOrder outbound push](../../worker2/features/netsuite-salesorder-outbound-push.md)
-</content>
+
+## Change history
+- 2026-09-22 — Recorded that the `MAX+1` generator is not concurrency-safe (unlocked read-then-insert → 1062), which routes retry it today, and that every other FIELD_AUTOINCREMENT model is still exposed. (rgirish)
+- 2026-09-02 — `TA` / 100000 added to the shared `_Model_Client_TransferOrder`. (bala)

package/knowledge/2.0/apps/api2/features/request-logging.md

@@ -6,8 +6,8 @@ project: API
 client: shared
 type: feature
 status: active
-updated: 2026-09-21
-owners: ["mhammontree", "dfranks", "bala", "jcardinal"]
+updated: 2026-09-22
+owners: ["mhammontree", "dfranks", "bala", "jcardinal", "rgirish"]
 files:
   - api2/Component/Api/V2/V2.php
   - api2/Controller/Index.php
@@ -98,6 +98,10 @@ $log->save();
 - The prefix is truncated to `TRANSACTION_ID_MAX_LENGTH` (255) minus the suffix, because a silent MySQL column-width chop would re-collide.
 - The loop is **deliberately unbounded and cannot spin**: every UNIQUE column changes per attempt (`_Model` mints a fresh `uuid` inside each `save()`, `id` is auto-increment, the resend number only climbs), so each try eliminates one taken value and never re-asks for the same one.
 - **It also removes one `SELECT` per API request** against an ~11.5M-row table.
+- **⚠ The loop's own 1062s must NOT leak into the caller's error state (fixed 2026-09-22).** `saveApiLogUnderItsOwnTransactionId()` runs **between** a failed business INSERT and `Controller/Index.php`'s retry check, so its deliberate collisions were **overwriting `_Database::$lastErrorNumber`** and silently disarming the whole unique-race retry — `/v2/service-requests` returned 21 × `400 EV-10` duplicate-number errors that should have self-healed. The fix saves and restores both `_Database::$lastErrorNumber` and `$retriableUniqueRace` in a `try/finally` **inside this method** (not at the call site), so only this loop's own errors are undone. **Rule: logging is bookkeeping and must never change the request's retry decision.** Full chain in [V2 request retry](./v2-deadlock-retry.md).
+- **A null `transactionId` does NOT break the first save** — checked and rejected as a finding, recorded so it is not re-raised. `Logs.Api.transactionId` is UNIQUE with `Null: YES`, and **MySQL allows unlimited NULLs in a UNIQUE index**; 4 such rows already exist in `dev-sandbox` `Logs_Prudential.Api`. A null never raises 1062 and never enters the resend loop.
+- **The loop is unbounded** (`for ($resendNumber = 1; ; ...)`) — reviewed 2026-09-22 and deliberately left as-is, because every pass changes a UNIQUE column so it cannot re-ask for a taken value. Do not add a bound without re-checking that reasoning.
+- **⚠ This method only exists on `_sandbox-dev`.** Commit `db94a26` never landed on `_production`. A branch cut from `_production` can therefore carry the **call site** without the **method**, and every DB-logged request fatals with *call to undefined method*. This bit branch `TRUE-82095`. **When you branch off `_production`, verify the callee exists — not just the caller.**
 
 **The duplicate comes from BELOW the application — you cannot stop it, only tolerate it.** Do not chase this as a double-click. The frontend axios interceptor (`toga2-commerce/src/api/axiosInstance.ts`) mints a **fresh uuid per request**, so a genuine double-click produces **two different ids**. Two requests sharing one id means the **identical bytes were replayed by something under the app**:
 - **2026-08-18 (production)** — the two copies arrived through **two different Zscaler proxy nodes** (`136.226.49.71` and `136.226.49.80`) with **different ALB trace ids**.
@@ -202,6 +206,9 @@ Both a **fallback recovery path** and a **scope diagnostic**. In the TRUE-80179
 - **A throw from a PRE interceptor leaves NO Api-log row** — 400 with null `transactionId`/`authority`/`audience`, real message in `messages[0].identifiers.message`. The only durable record is a `Logs.Issue` from `_Exception_Business`; a plain `_Exception_Validation` leaves no trace anywhere.
 - **An uncaught PHP fatal can leave NO log row anywhere** (incl. GETs, which return an empty 500 body). Empty 500 body + missing Api row = a fatal thrown before response assembly. Diagnose from the data (orphan sweeps), not the log.
 - **apiproxy retries a 5xx across regions** — one action = up to six near-identical 500 rows sharing one `transactionId`; a succeeding retry logs a clean 200 to the client log. Collapse on `transactionId`; join the two logs by it.
+- **The log-save must save/restore `_Database::$lastErrorNumber` + `$retriableUniqueRace`** — its own deliberate 1062s silently disarmed the unique-race retry until 2026-09-22. Logging must never change a request's retry decision.
+- **`saveApiLogUnderItsOwnTransactionId()` (`db94a26`) is on `_sandbox-dev` only** — a branch off `_production` may have the call site but not the method, and fatals with *call to undefined method*.
+- **A null `transactionId` is fine** — UNIQUE allows unlimited NULLs in MySQL; it never raises 1062.
 - **A duplicate `transactionId` must not fail the request** — save first, rename on 1062 (`-1`, `-2`, from the original id), swallow non-1062. A check-then-insert cannot work (uncommitted rows are invisible). The duplicate originates below the app (proxies/LBs resending a GET's bytes); the server tolerates, the client cannot prevent. Accepted exposure: two concurrent duplicate POSTs would both commit.
 - **`Logs.Api.transactionId` is UNIQUE on prod/`sandbox-client`/`dev-sandbox` but only a NON-unique index locally** — a local concurrency test proves nothing; confirm the constraint in the target env.
 - **`Logs.Api.transactionId` stores a timestamp/random uuid, not the envelope's `transactionId`** — you cannot look a request up by its envelope `transactionId`; use `dtStamp` window + `route`. The inner sub-request row's `transactionId` is random by design.
@@ -210,6 +217,7 @@ Both a **fallback recovery path** and a **scope diagnostic**. In the TRUE-80179
 - **Never copy logged rows into committed artifacts** — they carry live Bearer tokens and PII.
 
 ## Change history
+- 2026-09-22 — Log-save now saves/restores `_Database` error state in `try/finally`; its own 1062s were disarming the unique-race retry on `/v2/service-requests`. Also recorded the `_production`-branch missing-method hazard and the rejected null-`transactionId` finding. (rgirish)
 
 - 2026-09-21 — Added `Logs_<Client>.Api.originalUserId` (delegation audit: NULL = none, never equal to `userId`) and the rule that outbound `direction = OUT` rows stamp neither user column. (bala)
 

package/knowledge/2.0/apps/api2/features/v2-deadlock-retry.md

@@ -5,15 +5,16 @@ repo: api2
 project: API
 client: shared
 type: feature
-status: draft
-updated: 2026-09-16
-owners: [jcardinal, bala, ajean]
+status: active
+updated: 2026-09-22
+owners: [jcardinal, bala, ajean, rgirish]
 files:
   - api2/Controller/Index.php
   - api2/Component/Api/V2/V2.php
   - _underscore/Database.php
   - _underscore/Query.php
   - _underscore/Model/Compass/AdvanceShippingNotice.php
+  - _underscore/Model.php
 related:
   - ../../_underscore/features/model-save-parent-cascade-stored-field-deadlock.md
   - ../../../../clients/compass-usa/features/asn-to-item-fulfillment.md
@@ -31,9 +32,11 @@ api2's front controller ("**Prong A**") can **detect a retriable concurrency fai
 1. **MySQL deadlock (1213) / lock-wait timeout (1205)** — the original case (a same-PO ASN write deadlock; see the [save-cascade doc](../../_underscore/features/model-save-parent-cascade-stored-field-deadlock.md)).
 2. **Retriable unique-key get-or-create race (1062)** — a concurrent same-key INSERT lost the race; a whole-request replay on a fresh snapshot lets the retry's GET reuse the winner's committed row (added 2026-08-28 for the Compass ASN data-loss race).
 
-The replay is **route-scoped by an allowlist** (`DEADLOCK_RETRIABLE_ROUTES`) — currently only `/v2/advance-shipping-notices` — because whole-request in-process replay is **only safe for routes with no non-rollbackable external side effect**. The retry gate is now **`wasDeadlock() || wasRetriableUniqueRace()`**.
+The replay is **route-scoped by an allowlist** (`DEADLOCK_RETRIABLE_ROUTES` — currently `['/v2/advance-shipping-notices', '/v2/service-requests']`) — because whole-request in-process replay is **only safe for routes with no non-rollbackable external side effect**. The retry gate is now **`wasDeadlock() || wasRetriableUniqueRace()`**.
 
-> Status (2026-08-28): **in the working tree only** — not committed or deployed. Pending sandbox validation + regression tests.
+> **Status (2026-09-22).** Working on `_sandbox-dev` and **proven under real concurrency** (10/10 concurrent SR POSTs, zero duplicates). **Production still has NO race handling at all** for this — prod needs this fix plus the original `TRUE-82095` work. Prod evidence: 12 duplicate-number failures since 2026-08-18, latest `RITM6811124` / `REQ4510546`, which only succeeded on a manual resubmit as `SR131026`. The race is now **confirmed from production traffic**, not just the load test — see *Confirmed in production* below.
+
+**Constants (verified 2026-09-22):** `MAX_DEADLOCK_ATTEMPTS = 5` (not 3), backoff 20ms-120ms widening per attempt, `DEADLOCK_RETRIABLE_ROUTES = ['/v2/advance-shipping-notices', '/v2/service-requests']`.
 
 ## How it works
 
@@ -42,7 +45,7 @@ The replay is **route-scoped by an allowlist** (`DEADLOCK_RETRIABLE_ROUTES`) —
 - **The ASN model** calls the IF-create `internalApiRequest` with **`throwExceptionsOnError:false`**; on an empty response with `lastErrorNumber === 1062` it sets `$retriableUniqueRace = true` and returns null. `postPost` then **aborts immediately** (issues no further query).
 - **The controller's Prong A gate** becomes `wasDeadlock() || wasRetriableUniqueRace()`. Replaying the whole request on a fresh REPEATABLE-READ snapshot lets the retry's GET see the winner's committed IF and reuse it.
 
-Bounded by `MAX_DEADLOCK_ATTEMPTS = 3`, gated to `DEADLOCK_RETRIABLE_ROUTES = ['/v2/advance-shipping-notices']`.
+Bounded by `MAX_DEADLOCK_ATTEMPTS = 5`, gated to `DEADLOCK_RETRIABLE_ROUTES = ['/v2/advance-shipping-notices', '/v2/service-requests']`.
 
 **V2 concurrency mechanics (durable framework knowledge)** — the V2-engine facts that decide **how any interceptor-level write race must be handled**, and why the fix is a whole-request replay and not an in-place recovery:
 - **Response poisoning (load-bearing).** A failed nested save via `internalApiRequest` calls `addDefinedMessage(EV-10)` on the **shared** `$api->response`, setting a non-2xx status **before** control returns. So even if an interceptor catches the throw and "recovers," `execute()` recomputes `isSuccess = false` (`V2.php` ~2154) and **self-rolls-back DB_CORE + DB_CLIENT** (~2157-2159). **An interceptor therefore CANNOT recover a failed internal save in place** — the only correct recovery is a whole-request replay. This same poisoning drives the Prong A retry gate and, on exhaustion, produces the coherent EV-10 response.
@@ -50,7 +53,7 @@ Bounded by `MAX_DEADLOCK_ATTEMPTS = 3`, gated to `DEADLOCK_RETRIABLE_ROUTES = ['
 - **`_Database::$lastErrorNumber` is sticky.** Set only on query failure, never cleared by a later success; reset per attempt via `clearLastError()`. A recorded **deadlock (1213/1205) is protected from overwrite** by a later error; a **1062 is NOT** protected — so read/act on a 1062 promptly.
 - **The prod client cluster runs REPEATABLE-READ.** A same-transaction re-SELECT after a losing concurrent INSERT is **blind to the winner's commit** — only a locking read or a **fresh-snapshot whole-request replay** sees it. This is why in-place retry inside the transaction cannot work and the replay must start a new transaction.
 
-**How it works — deadlock path.** Detection primitive (`_underscore`, reusable): `_Database` gains `public static ?int $lastErrorNumber`, the consts `ERRNO_DEADLOCK` (1213) and `ERRNO_LOCK_WAIT_TIMEOUT` (1205), plus `clearLastError()` and `wasDeadlock()`. `_Query` records the driver errno **at its throw site**, and the recording is **sticky**: a later unrelated error does **not** overwrite an already-recorded deadlock. That stickiness lets a deadlock caught inside a model-save `try/catch` still be visible to the outer controller and trigger a retry. Retry loop (`api2/Controller/Index.php`): the v2 dispatch is wrapped in a bounded loop (`MAX_DEADLOCK_ATTEMPTS = 3`). On an unsuccessful response where `_Database::wasDeadlock() || _Database::wasRetriableUniqueRace()` is true and the route is in `DEADLOCK_RETRIABLE_ROUTES`, the controller:
+**How it works — deadlock path.** Detection primitive (`_underscore`, reusable): `_Database` gains `public static ?int $lastErrorNumber`, the consts `ERRNO_DEADLOCK` (1213) and `ERRNO_LOCK_WAIT_TIMEOUT` (1205), plus `clearLastError()` and `wasDeadlock()`. `_Query` records the driver errno **at its throw site**, and the recording is **sticky**: a later unrelated error does **not** overwrite an already-recorded deadlock. That stickiness lets a deadlock caught inside a model-save `try/catch` still be visible to the outer controller and trigger a retry. Retry loop (`api2/Controller/Index.php`): the v2 dispatch is wrapped in a bounded loop (`MAX_DEADLOCK_ATTEMPTS = 5`). On an unsuccessful response where `_Database::wasDeadlock() || _Database::wasRetriableUniqueRace()` is true and the route is in `DEADLOCK_RETRIABLE_ROUTES`, the controller:
 1. rolls back all open transactions;
 2. wipes `_Database::$_transactionStarts` so every connection re-begins via the umbrella lazy-transaction path on replay;
 3. resets the query + model caches;
@@ -74,12 +77,77 @@ Two consequences of the typed exception worth knowing:
 - **`_Exception_Database` extends `Exception` deliberately — not `RuntimeException`.** A design choice, not an oversight: callers that catch `RuntimeException` to mean "bad data, skip this row" must **not** accidentally swallow a DB failure. It also means any *new* exception type added near this code must extend `Exception` directly, or existing catch blocks change meaning — a live case in [Forecast.Sales import](../../_underscore/features/forecast-sale-import.md).
 - **Every SQL failure in worker2 is now `_Exception_Database`.** So any code that compares `get_class($e)` or matches on DB error **text** changes behaviour under this protocol. Audit for those before assuming the change is transparent.
 
+### ⚠ The API logger used to SILENTLY DISARM this retry — `$lastErrorNumber` must be saved/restored (2026-09-22)
+
+**Symptom:** `/v2/service-requests` returned `400 EV-10` *"Duplicate entry 'SR131023' for key 'ServiceRequests.number'"* — 21 consecutive failures, all ~1.8s with no backoff visible. ServiceNow was **not** sending duplicates. The retry existed and was correctly wired; it just never fired.
+
+**The root-cause chain, in order:**
+1. **`ServiceRequests.number` is not a MySQL auto-increment.** `_Model` `FIELD_AUTOINCREMENT` does a `SELECT MAX(number)+1` and then an `INSERT`, **with no lock between them** (see [FIELD_AUTOINCREMENT record numbering](../../_underscore/features/autoincrement-record-numbering.md)). Two concurrent POSTs read the same MAX, pick the same number, and the loser gets MySQL 1062.
+2. **The retry was already built for exactly this.** `_underscore/Model.php` sets `_Database::$retriableUniqueRace` on that 1062, and `api2/Controller/Index.php` replays the whole request when `_Database::wasRetriableUniqueRace()` is true. That helper requires **both** the flag **and** `$lastErrorNumber === 1062`.
+3. **The logger destroyed the second half of that condition.** `_Component_Api_V2::saveApiLogUnderItsOwnTransactionId()` runs **between the failed INSERT and the retry check**, inside `execute()`. Its resend loop **deliberately causes its own 1062s** — it renames `transactionId` on collision and retries (see [V2 request logging](./request-logging.md)). Those bookkeeping 1062s **overwrote the caller's `$lastErrorNumber`**, so `wasRetriableUniqueRace()` returned false and the replay never ran.
+
+**Fix:** save and restore **both** `_Database::$lastErrorNumber` and `_Database::$retriableUniqueRace` in a `try/finally` **inside** `saveApiLogUnderItsOwnTransactionId()`. Scoped inside that method — *not* at the call site — so it only undoes that loop's own errors; deadlocks, the ASN route, and every other route behave exactly as before.
+
+**The durable rule: logging is bookkeeping and must never change the request's retry decision.** Any code that runs between a failure and the code that inspects it must leave `_Database`'s error state exactly as it found it. Recall that `$lastErrorNumber` is **sticky but a 1062 is NOT protected from overwrite** (unlike a recorded deadlock) — so a 1062 is the error class most easily lost this way.
+
+**Side benefit:** the same fix repairs **ASN** (`_Model_Compass_AdvanceShippingNotice`), which sets the same flag and had the identical exposure.
+
+### Confirmed in production — three same-second ServiceNow POSTs, one 1062 (2026-09-22)
+
+The race is **not** synthetic. Real Prudential/ServiceNow bulk traffic reproduced it in prod, and prod is **still exposed** (the fix is on `_sandbox-dev` only).
+
+Three POSTs to `/v2/service-requests` arrived from ServiceNow (`sourceIp 199.91.136.12`) inside the same second, **2026-09-21 23:27:47–48 CST**:
+
+| `Logs_Prudential.Api.id` | Result | Number |
+|---|---|---|
+| 8736482 | **201** | `SR131023` |
+| 8736484 | **201** | `SR131024` |
+| 8736483 | **400 `EV-10`** | — |
+
+The 400's `responsePayload` carried the exact failure: **`Duplicate entry 'SR131023' for key 'ServiceRequests.number'`**, MySQL errno **1062**, on `INSERT INTO ServiceRequests`. So all three reads of `MAX(number)` returned the same value; two won (`131023`, `131024`) and the third re-tried `131023` and hit the UNIQUE key. Exactly the unlocked `FIELD_AUTOINCREMENT` generator described above.
+
+**⚠ Note the number in the error is NOT the failed request's number.** `SR131023` belongs to the *winner*. A client report of "duplicate SR131023" is about a request that never got a number at all — do not go looking for two `SR131023` rows.
+
+### ⚠ Open risk before the prod deploy — ServiceNow ALSO resends, so a fixed TOGA could create TWO SRs
+
+The failed `REQ4510546` above was **re-sent from the ServiceNow side ~6 hours later** (2026-09-22 05:23:32 CST, `transactionId 179007261259565713857`) and succeeded as **`SR131026`**. So today the outside resend is the only recovery — and it works.
+
+Once the `TRUE-82095` retry ships to prod, **TOGA retries internally within the same request**. If ServiceNow still resends the earlier failure on its own schedule, the same REQ can end up with **two ServiceRequests rows**. Confirm ServiceNow's resend behaviour with the client (contact: **Ferdaus**) before or alongside the prod deploy. Nothing in this retry protocol de-duplicates by `c_reqNumber`.
+
+### Verifying a concurrency fix — the gate-file pattern
+
+A plain background loop spreads requests out over time and can **miss the collision window entirely**, so it produces a false pass. What works: N `curl` workers each **block on a shared gate file**, then a single `touch` releases all of them at the same instant.
+
+Two details that cost time:
+- The v2 auth token is at **`.data.tokens.access`** in the auth response — **not** `.data.token`.
+- Confirm the UNIQUE constraint exists in the environment you are testing (see [V2 request logging](./request-logging.md)) — a local pass proves nothing.
+
+**Result (2026-09-22, `sandbox-dev`):** 10 concurrent SR POSTs → **10/10 HTTP 201**, `SR130589..SR130598`, zero duplicates. Response times stair-stepped 1.32s → 4.78s in ~0.4s steps — that shape **is** the retry backoff actually running, and is the signal that the retry engaged. **Control:** the same route the previous day, before the fix, returned **21 × 400**.
+
+### Deferred hardening (reviewed 2026-09-22, deliberately NOT changed)
+
+Both were considered and left alone on purpose. Do not "fix" either without re-checking the reasoning.
+
+- **The `V2.php` resend loop is unbounded** (`for ($resendNumber = 1; ; ...)`). Safe **today only because every pass changes a UNIQUE column**, so it cannot re-ask for a taken value.
+- **`_underscore/Model.php` marks ANY 1062 as a retriable race** without checking *which* unique key failed. `ServiceRequests` has three unique keys (`PRIMARY`, `uuid`, `number`) and only `number` is the benign race. Harmless today: `uuid` is a random uuid so a collision is effectively impossible, and `PRIMARY` cannot duplicate on INSERT.
+
+### Rejected finding — a null `transactionId` does NOT break the first log save
+
+Checked and **wrong**; recorded so it is not raised again. `Logs.Api.transactionId` is a UNIQUE index with `Null: YES`. **MySQL permits unlimited NULLs in a UNIQUE index**, and 4 such rows already exist in `dev-sandbox` `Logs_Prudential.Api`. A null therefore never raises 1062 and never enters the retry loop.
+
+
 ## Gotchas
 
 - Detection is built on the **MySQL errno** (1213/1205), not on matching the response text — deliberately, because the raw error text is not a reliable signal (and, separately, api2's validation error body still leaks the failing SQL + DB name; see [V2 API error codes](./v2-api-error-codes.md)).
-- Retry is bounded (`MAX_DEADLOCK_ATTEMPTS = 3`) with jittered backoff — a collision damper, not a substitute for removing the deadlock at its source.
+- Retry is bounded (**`MAX_DEADLOCK_ATTEMPTS = 5`**, not 3) with a 20ms-120ms widening backoff — a collision damper, not a substitute for removing the collision at its source.
+- **Anything running between the failing query and the retry check must save/restore `_Database::$lastErrorNumber` + `$retriableUniqueRace`.** The API logger's own 1062s disarmed this retry completely for weeks; a 1062 is not protected from overwrite the way a deadlock is.
+- **Test concurrency with a gate file**, not a background loop — a loop spreads the requests out and can miss the race. Auth token is at `.data.tokens.access`.
+- **A branch cut from `_production` may have the CALL SITE but not the METHOD.** `saveApiLogUnderItsOwnTransactionId()` (commit `db94a26`) exists only on `_sandbox-dev`; branch `TRUE-82095` came off `_production` and would have fataled with *call to undefined method* on every DB-logged request. When branching off `_production`, verify the **callee** exists, not just the caller.
 - `ERRNO_DUPLICATE_ENTRY` is absent from `_beta`/`_sandbox-dev` — naming it directly fatals there; define a local `1062` constant and read `$lastErrorNumber` via `property_exists()`.
-- Only `/v2/advance-shipping-notices` is on `DEADLOCK_RETRIABLE_ROUTES`; do not add a route without proving it is replay-safe (no irreversible side effect, no mutated superglobal).
+- `DEADLOCK_RETRIABLE_ROUTES` is now `/v2/advance-shipping-notices` **and** `/v2/service-requests`; do not add a route without proving it is replay-safe (no irreversible side effect, no mutated superglobal).
+- **Every other `FIELD_AUTOINCREMENT` model has the same unlocked generator** — SalesOrder, PurchaseOrder, Ticket, ItemReceipt, InventoryAdjustment, TransferOrder — and **none is on `DEADLOCK_RETRIABLE_ROUTES`**, so a duplicate-number clash on those routes still returns 400. The durable answer is fixing the number generator so the read and the insert cannot split; that is a bigger, separate change.
+- **The `SR…` number inside a `Duplicate entry` error belongs to the WINNER, not the failed request.** The loser never got a number. A "duplicate SR131023" report means one request was lost, not that two rows share a number.
+- **ServiceNow resends a failed SR on its own** (proven: `REQ4510546` failed, came back ~6h later as `SR131026`). After the prod deploy adds an internal retry, an outside resend could produce a **second SR for the same REQ** — confirm with the client first.
 - Do not reach for this protocol for an HTTP 429 — use `_ApiRequest::setAutoRetry()`.
 
 ## Related
@@ -90,3 +158,10 @@ Two consequences of the typed exception worth knowing:
 - [V2 API error codes](./v2-api-error-codes.md)
 - [V2 request logging](./request-logging.md)
 - [ENVIRONMENT decides the `_underscore` branch](./environment-variable-drives-underscore-branch.md)
+
+## Change history
+- 2026-09-22 — Added production confirmation of the race (3 same-second ServiceNow POSTs, `Logs_Prudential.Api` 8736482/8736483/8736484, 1062 on `SR131023`) and the open double-SR risk from ServiceNow's own resend. Investigation only, no code change. (rgirish)
+- 2026-09-22 — Found and fixed the reason this retry never fired on `/v2/service-requests`: the API logger's own 1062s overwrote `_Database::$lastErrorNumber`; now saved/restored in `try/finally` inside `saveApiLogUnderItsOwnTransactionId()`. Proven 10/10 under real concurrency on sandbox-dev. Also fixes ASN. Prod still unpatched. (rgirish)
+- 2026-09-22 — Corrected constants: `MAX_DEADLOCK_ATTEMPTS = 5` (was documented as 3); `DEADLOCK_RETRIABLE_ROUTES` now includes `/v2/service-requests`. (rgirish)
+- 2026-09-16 — Scoped the protocol against `_ApiRequest::setAutoRetry()` (MySQL vs HTTP retry). (bala)
+- 2026-08-28 — Added the retriable unique-key race (1062) path for the Compass ASN data-loss race. (jcardinal)

package/knowledge/clients/prudential/features/service-request-auth-rejection-alert.md

@@ -6,8 +6,8 @@ project: Worker
 client: prudential
 type: client-feature
 status: active
-updated: 2026-09-16
-owners: ["bala"]
+updated: 2026-09-22
+owners: ["bala", "rgirish"]
 files:
   - worker/crons/notifications/reports/prudential/prudential_auth_rejection_alert.php
   - worker/schedules/cron.worker.notification.json
@@ -19,6 +19,7 @@ related:
   - ../profile.md
   - ../../../2.0/apps/api2/features/request-logging.md
   - ../../../2.0/apps/api2/features/v2-api-error-codes.md
+  - ../../../2.0/apps/api2/features/v2-deadlock-retry.md
 ---
 The 15-min cron that alerts on refused (401/403) inbound Prudential REQs, where each code is logged, and how to tell "we refused it" from "they never sent it"; open when a Prudential REQ is reported missing.
 
@@ -68,7 +69,25 @@ Two different failure shapes get reported the same way ("REQ is missing"):
 
 Always check **both** schemas before answering — a 401 will not be in `Logs_Prudential`.
 
+### Triaging a "duplicate SR number" report — the recipe (2026-09-22)
+
+A third shape, and the report is misleading on its face: ServiceNow says *"duplicate `SR131023`"*, but **that number belongs to a different request that succeeded**. The reported REQ got no number at all — it lost the `FIELD_AUTOINCREMENT` race and died on `1062` (see [V2 request retry](../../../2.0/apps/api2/features/v2-deadlock-retry.md)). Do not go hunting for two rows with the same number.
+
+Where the Prudential data actually lives:
+- ServiceNow REQ/RITM land in `Client_Prudential.ServiceRequests` as **`c_reqNumber`** and **`c_ritmNumber`** — *not* `c_snowRequestNumber` / `c_snowRitmNumber`. The other ServiceNow columns on that table are `c_sysId` and `c_sysCreatedOn`.
+- `ServiceRequests` has **no `dtModified` column** — the timestamps are `dtCreated` / `dtUpdated`.
+- `c_sysCreatedOn` is ServiceNow time (**EST**); `dtCreated` is TOGA DB time (**CST**). **A one-hour gap between them is expected**, not a bug.
+
+How to find the failure:
+1. Look the call up in `Logs_Prudential.Api` by **`transactionId`** — it is UNIQUE and indexed, so this is instant.
+2. Read the full MySQL error out of **`responsePayload.messages[].identifiers.error`** — that is where `Duplicate entry 'SR131023' for key 'ServiceRequests.number'` and errno `1062` live.
+3. Check whether the REQ came back later on its own: ServiceNow does resend. `REQ4510546` failed on 2026-09-21 and succeeded ~6h later as `SR131026`.
+
+**⚠ Never run an unbounded `requestPayload LIKE '%REQ…%'` against `Logs_Prudential.Api`** — `requestPayload` is not indexed, so it full-scans the table and times out. Always bound it with a **`dtStamp` window plus `route` / `method`** first (`dtStamp` is indexed; `route` and `responseCode` are not).
+
 ## Gotchas
+- **Prudential SR columns are `c_reqNumber` / `c_ritmNumber`**, there is no `dtModified` (it is `dtCreated`/`dtUpdated`), and `c_sysCreatedOn` (EST) sitting ~1h off `dtCreated` (CST) is correct.
+- **Never `LIKE` on `requestPayload` without a `dtStamp` window** — it scans `Logs_Prudential.Api` and times out.
 - **Filter `method = 'POST'` or you get browser noise.** Unauthenticated **GET**s on this route from human sourceIps (`99.179.145.25`, `107.139.5.190`) also log 401, with an **empty `requestPayload`** — they show up as REQ "Unknown". Only a POST is a real ServiceNow submission.
 - **`prudential_api_500_retry.php` is broken and has almost certainly never worked.** At lines 57 and 70 it references a bare constant **`PRUDENTIAL_CLIENT_UUID` that is defined nowhere** in any repo (grep returns only its own two uses); it should be `App_Api_Toga2::CLIENT_UUID_PRUDENTIAL`. On PHP 8 an undefined constant is a **fatal Error**, so the cron dies every run; on PHP 7 it evaluates to the literal string and the client check never matches, so it silently retries nothing. It also appears in **no** `schedules/*.json` file, so it may never have been registered. **Not fixed as of 2026-09-10 — reported only.**
 - **`Logs.Api.dtStamp` IS indexed** (`Api_dtStamp_IDX`) in both base `Logs` and `Logs_Prudential`, which is what makes the 15-minute window read cheap. But `route` and `responseCode` are **not** indexed, so a wide `dtStamp` range plus a route filter still times out (a 2.5-month grouped query aborted at 300 s). Keep the window tight. See [request-logging.md](../../../2.0/apps/api2/features/request-logging.md).
@@ -78,3 +97,6 @@ Always check **both** schemas before answering — a 401 will not be in `Logs_Pr
 - [Service request address validation](service-request-address-validation.md)
 - [Prudential profile](../profile.md)
 - [V2 request logging](../../../2.0/apps/api2/features/request-logging.md), [V2 API error codes](../../../2.0/apps/api2/features/v2-api-error-codes.md)
+
+## Change history
+- 2026-09-22 — Added the "duplicate SR number" triage recipe: real column names, where the MySQL error text sits in `responsePayload`, the `requestPayload LIKE` timeout trap, and the EST/CST one-hour offset. Prod investigation only, no code change. (rgirish)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.867",
+  "version": "1.0.869",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",