toga-ai 1.0.868 → 1.0.869

package/knowledge/2.0/apps/api2/features/v2-deadlock-retry.md

@@ -34,7 +34,7 @@ api2's front controller ("**Prong A**") can **detect a retriable concurrency fai
 
 The replay is **route-scoped by an allowlist** (`DEADLOCK_RETRIABLE_ROUTES` — currently `['/v2/advance-shipping-notices', '/v2/service-requests']`) — because whole-request in-process replay is **only safe for routes with no non-rollbackable external side effect**. The retry gate is now **`wasDeadlock() || wasRetriableUniqueRace()`**.
 
-> **Status (2026-09-22).** Working on `_sandbox-dev` and **proven under real concurrency** (10/10 concurrent SR POSTs, zero duplicates). **Production still has NO race handling at all** for this — prod needs this fix plus the original `TRUE-82095` work. Prod evidence: 12 duplicate-number failures since 2026-08-18, latest `RITM6811124` / `REQ4510546`, which only succeeded on a manual resubmit as `SR131026`.
+> **Status (2026-09-22).** Working on `_sandbox-dev` and **proven under real concurrency** (10/10 concurrent SR POSTs, zero duplicates). **Production still has NO race handling at all** for this — prod needs this fix plus the original `TRUE-82095` work. Prod evidence: 12 duplicate-number failures since 2026-08-18, latest `RITM6811124` / `REQ4510546`, which only succeeded on a manual resubmit as `SR131026`. The race is now **confirmed from production traffic**, not just the load test — see *Confirmed in production* below.
 
 **Constants (verified 2026-09-22):** `MAX_DEADLOCK_ATTEMPTS = 5` (not 3), backoff 20ms-120ms widening per attempt, `DEADLOCK_RETRIABLE_ROUTES = ['/v2/advance-shipping-notices', '/v2/service-requests']`.
 
@@ -92,6 +92,28 @@ Two consequences of the typed exception worth knowing:
 
 **Side benefit:** the same fix repairs **ASN** (`_Model_Compass_AdvanceShippingNotice`), which sets the same flag and had the identical exposure.
 
+### Confirmed in production — three same-second ServiceNow POSTs, one 1062 (2026-09-22)
+
+The race is **not** synthetic. Real Prudential/ServiceNow bulk traffic reproduced it in prod, and prod is **still exposed** (the fix is on `_sandbox-dev` only).
+
+Three POSTs to `/v2/service-requests` arrived from ServiceNow (`sourceIp 199.91.136.12`) inside the same second, **2026-09-21 23:27:47–48 CST**:
+
+| `Logs_Prudential.Api.id` | Result | Number |
+|---|---|---|
+| 8736482 | **201** | `SR131023` |
+| 8736484 | **201** | `SR131024` |
+| 8736483 | **400 `EV-10`** | — |
+
+The 400's `responsePayload` carried the exact failure: **`Duplicate entry 'SR131023' for key 'ServiceRequests.number'`**, MySQL errno **1062**, on `INSERT INTO ServiceRequests`. So all three reads of `MAX(number)` returned the same value; two won (`131023`, `131024`) and the third re-tried `131023` and hit the UNIQUE key. Exactly the unlocked `FIELD_AUTOINCREMENT` generator described above.
+
+**⚠ Note the number in the error is NOT the failed request's number.** `SR131023` belongs to the *winner*. A client report of "duplicate SR131023" is about a request that never got a number at all — do not go looking for two `SR131023` rows.
+
+### ⚠ Open risk before the prod deploy — ServiceNow ALSO resends, so a fixed TOGA could create TWO SRs
+
+The failed `REQ4510546` above was **re-sent from the ServiceNow side ~6 hours later** (2026-09-22 05:23:32 CST, `transactionId 179007261259565713857`) and succeeded as **`SR131026`**. So today the outside resend is the only recovery — and it works.
+
+Once the `TRUE-82095` retry ships to prod, **TOGA retries internally within the same request**. If ServiceNow still resends the earlier failure on its own schedule, the same REQ can end up with **two ServiceRequests rows**. Confirm ServiceNow's resend behaviour with the client (contact: **Ferdaus**) before or alongside the prod deploy. Nothing in this retry protocol de-duplicates by `c_reqNumber`.
+
 ### Verifying a concurrency fix — the gate-file pattern
 
 A plain background loop spreads requests out over time and can **miss the collision window entirely**, so it produces a false pass. What works: N `curl` workers each **block on a shared gate file**, then a single `touch` releases all of them at the same instant.
@@ -124,6 +146,8 @@ Checked and **wrong**; recorded so it is not raised again. `Logs.Api.transaction
 - `ERRNO_DUPLICATE_ENTRY` is absent from `_beta`/`_sandbox-dev` — naming it directly fatals there; define a local `1062` constant and read `$lastErrorNumber` via `property_exists()`.
 - `DEADLOCK_RETRIABLE_ROUTES` is now `/v2/advance-shipping-notices` **and** `/v2/service-requests`; do not add a route without proving it is replay-safe (no irreversible side effect, no mutated superglobal).
 - **Every other `FIELD_AUTOINCREMENT` model has the same unlocked generator** — SalesOrder, PurchaseOrder, Ticket, ItemReceipt, InventoryAdjustment, TransferOrder — and **none is on `DEADLOCK_RETRIABLE_ROUTES`**, so a duplicate-number clash on those routes still returns 400. The durable answer is fixing the number generator so the read and the insert cannot split; that is a bigger, separate change.
+- **The `SR…` number inside a `Duplicate entry` error belongs to the WINNER, not the failed request.** The loser never got a number. A "duplicate SR131023" report means one request was lost, not that two rows share a number.
+- **ServiceNow resends a failed SR on its own** (proven: `REQ4510546` failed, came back ~6h later as `SR131026`). After the prod deploy adds an internal retry, an outside resend could produce a **second SR for the same REQ** — confirm with the client first.
 - Do not reach for this protocol for an HTTP 429 — use `_ApiRequest::setAutoRetry()`.
 
 ## Related
@@ -136,6 +160,7 @@ Checked and **wrong**; recorded so it is not raised again. `Logs.Api.transaction
 - [ENVIRONMENT decides the `_underscore` branch](./environment-variable-drives-underscore-branch.md)
 
 ## Change history
+- 2026-09-22 — Added production confirmation of the race (3 same-second ServiceNow POSTs, `Logs_Prudential.Api` 8736482/8736483/8736484, 1062 on `SR131023`) and the open double-SR risk from ServiceNow's own resend. Investigation only, no code change. (rgirish)
 - 2026-09-22 — Found and fixed the reason this retry never fired on `/v2/service-requests`: the API logger's own 1062s overwrote `_Database::$lastErrorNumber`; now saved/restored in `try/finally` inside `saveApiLogUnderItsOwnTransactionId()`. Proven 10/10 under real concurrency on sandbox-dev. Also fixes ASN. Prod still unpatched. (rgirish)
 - 2026-09-22 — Corrected constants: `MAX_DEADLOCK_ATTEMPTS = 5` (was documented as 3); `DEADLOCK_RETRIABLE_ROUTES` now includes `/v2/service-requests`. (rgirish)
 - 2026-09-16 — Scoped the protocol against `_ApiRequest::setAutoRetry()` (MySQL vs HTTP retry). (bala)

package/knowledge/clients/prudential/features/service-request-auth-rejection-alert.md

@@ -6,8 +6,8 @@ project: Worker
 client: prudential
 type: client-feature
 status: active
-updated: 2026-09-16
-owners: ["bala"]
+updated: 2026-09-22
+owners: ["bala", "rgirish"]
 files:
   - worker/crons/notifications/reports/prudential/prudential_auth_rejection_alert.php
   - worker/schedules/cron.worker.notification.json
@@ -19,6 +19,7 @@ related:
   - ../profile.md
   - ../../../2.0/apps/api2/features/request-logging.md
   - ../../../2.0/apps/api2/features/v2-api-error-codes.md
+  - ../../../2.0/apps/api2/features/v2-deadlock-retry.md
 ---
 The 15-min cron that alerts on refused (401/403) inbound Prudential REQs, where each code is logged, and how to tell "we refused it" from "they never sent it"; open when a Prudential REQ is reported missing.
 
@@ -68,7 +69,25 @@ Two different failure shapes get reported the same way ("REQ is missing"):
 
 Always check **both** schemas before answering — a 401 will not be in `Logs_Prudential`.
 
+### Triaging a "duplicate SR number" report — the recipe (2026-09-22)
+
+A third shape, and the report is misleading on its face: ServiceNow says *"duplicate `SR131023`"*, but **that number belongs to a different request that succeeded**. The reported REQ got no number at all — it lost the `FIELD_AUTOINCREMENT` race and died on `1062` (see [V2 request retry](../../../2.0/apps/api2/features/v2-deadlock-retry.md)). Do not go hunting for two rows with the same number.
+
+Where the Prudential data actually lives:
+- ServiceNow REQ/RITM land in `Client_Prudential.ServiceRequests` as **`c_reqNumber`** and **`c_ritmNumber`** — *not* `c_snowRequestNumber` / `c_snowRitmNumber`. The other ServiceNow columns on that table are `c_sysId` and `c_sysCreatedOn`.
+- `ServiceRequests` has **no `dtModified` column** — the timestamps are `dtCreated` / `dtUpdated`.
+- `c_sysCreatedOn` is ServiceNow time (**EST**); `dtCreated` is TOGA DB time (**CST**). **A one-hour gap between them is expected**, not a bug.
+
+How to find the failure:
+1. Look the call up in `Logs_Prudential.Api` by **`transactionId`** — it is UNIQUE and indexed, so this is instant.
+2. Read the full MySQL error out of **`responsePayload.messages[].identifiers.error`** — that is where `Duplicate entry 'SR131023' for key 'ServiceRequests.number'` and errno `1062` live.
+3. Check whether the REQ came back later on its own: ServiceNow does resend. `REQ4510546` failed on 2026-09-21 and succeeded ~6h later as `SR131026`.
+
+**⚠ Never run an unbounded `requestPayload LIKE '%REQ…%'` against `Logs_Prudential.Api`** — `requestPayload` is not indexed, so it full-scans the table and times out. Always bound it with a **`dtStamp` window plus `route` / `method`** first (`dtStamp` is indexed; `route` and `responseCode` are not).
+
 ## Gotchas
+- **Prudential SR columns are `c_reqNumber` / `c_ritmNumber`**, there is no `dtModified` (it is `dtCreated`/`dtUpdated`), and `c_sysCreatedOn` (EST) sitting ~1h off `dtCreated` (CST) is correct.
+- **Never `LIKE` on `requestPayload` without a `dtStamp` window** — it scans `Logs_Prudential.Api` and times out.
 - **Filter `method = 'POST'` or you get browser noise.** Unauthenticated **GET**s on this route from human sourceIps (`99.179.145.25`, `107.139.5.190`) also log 401, with an **empty `requestPayload`** — they show up as REQ "Unknown". Only a POST is a real ServiceNow submission.
 - **`prudential_api_500_retry.php` is broken and has almost certainly never worked.** At lines 57 and 70 it references a bare constant **`PRUDENTIAL_CLIENT_UUID` that is defined nowhere** in any repo (grep returns only its own two uses); it should be `App_Api_Toga2::CLIENT_UUID_PRUDENTIAL`. On PHP 8 an undefined constant is a **fatal Error**, so the cron dies every run; on PHP 7 it evaluates to the literal string and the client check never matches, so it silently retries nothing. It also appears in **no** `schedules/*.json` file, so it may never have been registered. **Not fixed as of 2026-09-10 — reported only.**
 - **`Logs.Api.dtStamp` IS indexed** (`Api_dtStamp_IDX`) in both base `Logs` and `Logs_Prudential`, which is what makes the 15-minute window read cheap. But `route` and `responseCode` are **not** indexed, so a wide `dtStamp` range plus a route filter still times out (a 2.5-month grouped query aborted at 300 s). Keep the window tight. See [request-logging.md](../../../2.0/apps/api2/features/request-logging.md).
@@ -78,3 +97,6 @@ Always check **both** schemas before answering — a 401 will not be in `Logs_Pr
 - [Service request address validation](service-request-address-validation.md)
 - [Prudential profile](../profile.md)
 - [V2 request logging](../../../2.0/apps/api2/features/request-logging.md), [V2 API error codes](../../../2.0/apps/api2/features/v2-api-error-codes.md)
+
+## Change history
+- 2026-09-22 — Added the "duplicate SR number" triage recipe: real column names, where the MySQL error text sits in `responsePayload`, the `requestPayload LIKE` timeout trap, and the EST/CST one-hour offset. Prod investigation only, no code change. (rgirish)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.868",
+  "version": "1.0.869",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",