toga-ai 1.0.866 → 1.0.868

package/knowledge/2.0/apps/_underscore/features/autoincrement-record-numbering.md

@@ -6,14 +6,15 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
-owners: ["bala"]
+updated: 2026-09-22
+owners: ["bala", "rgirish"]
 files:
   - _underscore/Model.php
   - _underscore/Model/Client/SalesOrder.php
   - _underscore/Model/Client/TransferOrder.php
 related:
   - ../architecture.md
+  - ../../api2/features/v2-deadlock-retry.md
   - ../../worker2/features/netsuite-transferorder-outbound-push.md
   - ../../worker2/features/netsuite-salesorder-outbound-push.md
 ---
@@ -42,6 +43,10 @@ Verified in `_underscore/Model.php` ~L546:
 2. **The next value is `MAX(numeric part) + 1` scoped by `WHERE number LIKE '<prefix>%'`.** So pre-existing numbers in another format are ignored: NYCHH's 1,930 legacy transfer-order numbers (e.g. `286906`) do not participate, and the sequence starts clean at `TA100000`.
 3. **There is NO zero padding.** The output is prefix + integer, so `TA00000` is not achievable without changing `Model.php`.
 
+**⚠ The generator is NOT concurrency-safe — read and insert can split.** `SELECT MAX(...)+1` and the `INSERT` are two separate statements with **no lock between them**. Two concurrent POSTs read the same MAX, pick the same number, and the loser gets **MySQL 1062** — a 400 to the caller, not a retry. Seen live on Prudential `/v2/service-requests` (`Duplicate entry 'SR131023' for key 'ServiceRequests.number'`).
+
+**Today's mitigation is a retry, not a fix.** api2 replays the whole request for two routes only (`/v2/advance-shipping-notices`, `/v2/service-requests`) — see [V2 request retry](../../api2/features/v2-deadlock-retry.md). **Every other `FIELD_AUTOINCREMENT` model has the same exposure and no mitigation**: SalesOrder, PurchaseOrder, Ticket, ItemReceipt, InventoryAdjustment, TransferOrder. A duplicate-number clash on those routes still returns 400. The durable answer is making the read and insert one operation in `Model.php`; that is a bigger, separate change and is **not done**.
+
 ## Gotchas
 
 - **⚠ A prefix on a SHARED model is not unique across clients.** Because the option lives on `_Model_Client_TransferOrder`, every tenant starts at `TA100000` — the scan is per client database. Any cross-client identifier must therefore NOT be keyed on `number`: NetSuite account **1095849 is shared by every client** and enforces `externalId` uniqueness account-wide, which is why both pushes key their external id on the record **uuid** (`toga-so-<uuid>` / `toga-to-<uuid>`) — see [the transfer-order push](../../worker2/features/netsuite-transferorder-outbound-push.md).
@@ -49,6 +54,7 @@ Verified in `_underscore/Model.php` ~L546:
   - **Keeping a Toga-side number permanently takes TWO changes, not one** (written 2026-09-02, then reverted at the developer's request — recorded so nobody rediscovers it): (1) a guard in `writeBackNetSuiteId()` so the push does not overwrite `number`, and (2) dropping `number` from the PUT in both `library/app/api/toga2.php` (~L4295) and `_underscore/Trait/Netsuite/TransferOrder.php`. Miss (2) and the 1.0 importer compares `number` and PUTs NetSuite's over the top on its next run, so the guard alone achieves nothing. Both files are back to untouched (verified with an empty `git diff`).
 - **`FIELD_AUTOINCREMENT` is one of the field types special-cased by `buildSqlFieldValue()`**, so a null value on it does NOT collapse a `load()` WHERE clause the way a null `FIELD_CHAR` does — see the collapsed-WHERE OOM note in [_underscore architecture](../architecture.md).
 - **Adding the option to a model does not renumber anything** and does not touch existing rows; it only affects inserts that arrive with `number` NULL.
+- **Concurrent inserts can pick the same number** (unlocked `MAX+1` then `INSERT`) → MySQL 1062 → 400. Only two api2 routes retry it; every other model is exposed. See [V2 request retry](../../api2/features/v2-deadlock-retry.md).
 - **A front-end that sends `number` silently disables the feature.** There is no error — the posted value is simply accepted. Check the actual POST body before concluding the generator is broken.
 
 ## Related
@@ -56,4 +62,7 @@ Verified in `_underscore/Model.php` ~L546:
 - [_underscore architecture](../architecture.md)
 - [NetSuite TransferOrder outbound push](../../worker2/features/netsuite-transferorder-outbound-push.md)
 - [NetSuite SalesOrder outbound push](../../worker2/features/netsuite-salesorder-outbound-push.md)
-</content>
+
+## Change history
+- 2026-09-22 — Recorded that the `MAX+1` generator is not concurrency-safe (unlocked read-then-insert → 1062), which routes retry it today, and that every other FIELD_AUTOINCREMENT model is still exposed. (rgirish)
+- 2026-09-02 — `TA` / 100000 added to the shared `_Model_Client_TransferOrder`. (bala)

package/knowledge/2.0/apps/api2/features/request-logging.md

@@ -6,8 +6,8 @@ project: API
 client: shared
 type: feature
 status: active
-updated: 2026-09-21
-owners: ["mhammontree", "dfranks", "bala", "jcardinal"]
+updated: 2026-09-22
+owners: ["mhammontree", "dfranks", "bala", "jcardinal", "rgirish"]
 files:
   - api2/Component/Api/V2/V2.php
   - api2/Controller/Index.php
@@ -98,6 +98,10 @@ $log->save();
 - The prefix is truncated to `TRANSACTION_ID_MAX_LENGTH` (255) minus the suffix, because a silent MySQL column-width chop would re-collide.
 - The loop is **deliberately unbounded and cannot spin**: every UNIQUE column changes per attempt (`_Model` mints a fresh `uuid` inside each `save()`, `id` is auto-increment, the resend number only climbs), so each try eliminates one taken value and never re-asks for the same one.
 - **It also removes one `SELECT` per API request** against an ~11.5M-row table.
+- **⚠ The loop's own 1062s must NOT leak into the caller's error state (fixed 2026-09-22).** `saveApiLogUnderItsOwnTransactionId()` runs **between** a failed business INSERT and `Controller/Index.php`'s retry check, so its deliberate collisions were **overwriting `_Database::$lastErrorNumber`** and silently disarming the whole unique-race retry — `/v2/service-requests` returned 21 × `400 EV-10` duplicate-number errors that should have self-healed. The fix saves and restores both `_Database::$lastErrorNumber` and `$retriableUniqueRace` in a `try/finally` **inside this method** (not at the call site), so only this loop's own errors are undone. **Rule: logging is bookkeeping and must never change the request's retry decision.** Full chain in [V2 request retry](./v2-deadlock-retry.md).
+- **A null `transactionId` does NOT break the first save** — checked and rejected as a finding, recorded so it is not re-raised. `Logs.Api.transactionId` is UNIQUE with `Null: YES`, and **MySQL allows unlimited NULLs in a UNIQUE index**; 4 such rows already exist in `dev-sandbox` `Logs_Prudential.Api`. A null never raises 1062 and never enters the resend loop.
+- **The loop is unbounded** (`for ($resendNumber = 1; ; ...)`) — reviewed 2026-09-22 and deliberately left as-is, because every pass changes a UNIQUE column so it cannot re-ask for a taken value. Do not add a bound without re-checking that reasoning.
+- **⚠ This method only exists on `_sandbox-dev`.** Commit `db94a26` never landed on `_production`. A branch cut from `_production` can therefore carry the **call site** without the **method**, and every DB-logged request fatals with *call to undefined method*. This bit branch `TRUE-82095`. **When you branch off `_production`, verify the callee exists — not just the caller.**
 
 **The duplicate comes from BELOW the application — you cannot stop it, only tolerate it.** Do not chase this as a double-click. The frontend axios interceptor (`toga2-commerce/src/api/axiosInstance.ts`) mints a **fresh uuid per request**, so a genuine double-click produces **two different ids**. Two requests sharing one id means the **identical bytes were replayed by something under the app**:
 - **2026-08-18 (production)** — the two copies arrived through **two different Zscaler proxy nodes** (`136.226.49.71` and `136.226.49.80`) with **different ALB trace ids**.
@@ -202,6 +206,9 @@ Both a **fallback recovery path** and a **scope diagnostic**. In the TRUE-80179
 - **A throw from a PRE interceptor leaves NO Api-log row** — 400 with null `transactionId`/`authority`/`audience`, real message in `messages[0].identifiers.message`. The only durable record is a `Logs.Issue` from `_Exception_Business`; a plain `_Exception_Validation` leaves no trace anywhere.
 - **An uncaught PHP fatal can leave NO log row anywhere** (incl. GETs, which return an empty 500 body). Empty 500 body + missing Api row = a fatal thrown before response assembly. Diagnose from the data (orphan sweeps), not the log.
 - **apiproxy retries a 5xx across regions** — one action = up to six near-identical 500 rows sharing one `transactionId`; a succeeding retry logs a clean 200 to the client log. Collapse on `transactionId`; join the two logs by it.
+- **The log-save must save/restore `_Database::$lastErrorNumber` + `$retriableUniqueRace`** — its own deliberate 1062s silently disarmed the unique-race retry until 2026-09-22. Logging must never change a request's retry decision.
+- **`saveApiLogUnderItsOwnTransactionId()` (`db94a26`) is on `_sandbox-dev` only** — a branch off `_production` may have the call site but not the method, and fatals with *call to undefined method*.
+- **A null `transactionId` is fine** — UNIQUE allows unlimited NULLs in MySQL; it never raises 1062.
 - **A duplicate `transactionId` must not fail the request** — save first, rename on 1062 (`-1`, `-2`, from the original id), swallow non-1062. A check-then-insert cannot work (uncommitted rows are invisible). The duplicate originates below the app (proxies/LBs resending a GET's bytes); the server tolerates, the client cannot prevent. Accepted exposure: two concurrent duplicate POSTs would both commit.
 - **`Logs.Api.transactionId` is UNIQUE on prod/`sandbox-client`/`dev-sandbox` but only a NON-unique index locally** — a local concurrency test proves nothing; confirm the constraint in the target env.
 - **`Logs.Api.transactionId` stores a timestamp/random uuid, not the envelope's `transactionId`** — you cannot look a request up by its envelope `transactionId`; use `dtStamp` window + `route`. The inner sub-request row's `transactionId` is random by design.
@@ -210,6 +217,7 @@ Both a **fallback recovery path** and a **scope diagnostic**. In the TRUE-80179
 - **Never copy logged rows into committed artifacts** — they carry live Bearer tokens and PII.
 
 ## Change history
+- 2026-09-22 — Log-save now saves/restores `_Database` error state in `try/finally`; its own 1062s were disarming the unique-race retry on `/v2/service-requests`. Also recorded the `_production`-branch missing-method hazard and the rejected null-`transactionId` finding. (rgirish)
 
 - 2026-09-21 — Added `Logs_<Client>.Api.originalUserId` (delegation audit: NULL = none, never equal to `userId`) and the rule that outbound `direction = OUT` rows stamp neither user column. (bala)
 

package/knowledge/2.0/apps/api2/features/v2-deadlock-retry.md

@@ -5,15 +5,16 @@ repo: api2
 project: API
 client: shared
 type: feature
-status: draft
-updated: 2026-09-16
-owners: [jcardinal, bala, ajean]
+status: active
+updated: 2026-09-22
+owners: [jcardinal, bala, ajean, rgirish]
 files:
   - api2/Controller/Index.php
   - api2/Component/Api/V2/V2.php
   - _underscore/Database.php
   - _underscore/Query.php
   - _underscore/Model/Compass/AdvanceShippingNotice.php
+  - _underscore/Model.php
 related:
   - ../../_underscore/features/model-save-parent-cascade-stored-field-deadlock.md
   - ../../../../clients/compass-usa/features/asn-to-item-fulfillment.md
@@ -31,9 +32,11 @@ api2's front controller ("**Prong A**") can **detect a retriable concurrency fai
 1. **MySQL deadlock (1213) / lock-wait timeout (1205)** — the original case (a same-PO ASN write deadlock; see the [save-cascade doc](../../_underscore/features/model-save-parent-cascade-stored-field-deadlock.md)).
 2. **Retriable unique-key get-or-create race (1062)** — a concurrent same-key INSERT lost the race; a whole-request replay on a fresh snapshot lets the retry's GET reuse the winner's committed row (added 2026-08-28 for the Compass ASN data-loss race).
 
-The replay is **route-scoped by an allowlist** (`DEADLOCK_RETRIABLE_ROUTES`) — currently only `/v2/advance-shipping-notices` — because whole-request in-process replay is **only safe for routes with no non-rollbackable external side effect**. The retry gate is now **`wasDeadlock() || wasRetriableUniqueRace()`**.
+The replay is **route-scoped by an allowlist** (`DEADLOCK_RETRIABLE_ROUTES` — currently `['/v2/advance-shipping-notices', '/v2/service-requests']`) — because whole-request in-process replay is **only safe for routes with no non-rollbackable external side effect**. The retry gate is now **`wasDeadlock() || wasRetriableUniqueRace()`**.
 
-> Status (2026-08-28): **in the working tree only** — not committed or deployed. Pending sandbox validation + regression tests.
+> **Status (2026-09-22).** Working on `_sandbox-dev` and **proven under real concurrency** (10/10 concurrent SR POSTs, zero duplicates). **Production still has NO race handling at all** for this — prod needs this fix plus the original `TRUE-82095` work. Prod evidence: 12 duplicate-number failures since 2026-08-18, latest `RITM6811124` / `REQ4510546`, which only succeeded on a manual resubmit as `SR131026`.
+
+**Constants (verified 2026-09-22):** `MAX_DEADLOCK_ATTEMPTS = 5` (not 3), backoff 20ms-120ms widening per attempt, `DEADLOCK_RETRIABLE_ROUTES = ['/v2/advance-shipping-notices', '/v2/service-requests']`.
 
 ## How it works
 
@@ -42,7 +45,7 @@ The replay is **route-scoped by an allowlist** (`DEADLOCK_RETRIABLE_ROUTES`) —
 - **The ASN model** calls the IF-create `internalApiRequest` with **`throwExceptionsOnError:false`**; on an empty response with `lastErrorNumber === 1062` it sets `$retriableUniqueRace = true` and returns null. `postPost` then **aborts immediately** (issues no further query).
 - **The controller's Prong A gate** becomes `wasDeadlock() || wasRetriableUniqueRace()`. Replaying the whole request on a fresh REPEATABLE-READ snapshot lets the retry's GET see the winner's committed IF and reuse it.
 
-Bounded by `MAX_DEADLOCK_ATTEMPTS = 3`, gated to `DEADLOCK_RETRIABLE_ROUTES = ['/v2/advance-shipping-notices']`.
+Bounded by `MAX_DEADLOCK_ATTEMPTS = 5`, gated to `DEADLOCK_RETRIABLE_ROUTES = ['/v2/advance-shipping-notices', '/v2/service-requests']`.
 
 **V2 concurrency mechanics (durable framework knowledge)** — the V2-engine facts that decide **how any interceptor-level write race must be handled**, and why the fix is a whole-request replay and not an in-place recovery:
 - **Response poisoning (load-bearing).** A failed nested save via `internalApiRequest` calls `addDefinedMessage(EV-10)` on the **shared** `$api->response`, setting a non-2xx status **before** control returns. So even if an interceptor catches the throw and "recovers," `execute()` recomputes `isSuccess = false` (`V2.php` ~2154) and **self-rolls-back DB_CORE + DB_CLIENT** (~2157-2159). **An interceptor therefore CANNOT recover a failed internal save in place** — the only correct recovery is a whole-request replay. This same poisoning drives the Prong A retry gate and, on exhaustion, produces the coherent EV-10 response.
@@ -50,7 +53,7 @@ Bounded by `MAX_DEADLOCK_ATTEMPTS = 3`, gated to `DEADLOCK_RETRIABLE_ROUTES = ['
 - **`_Database::$lastErrorNumber` is sticky.** Set only on query failure, never cleared by a later success; reset per attempt via `clearLastError()`. A recorded **deadlock (1213/1205) is protected from overwrite** by a later error; a **1062 is NOT** protected — so read/act on a 1062 promptly.
 - **The prod client cluster runs REPEATABLE-READ.** A same-transaction re-SELECT after a losing concurrent INSERT is **blind to the winner's commit** — only a locking read or a **fresh-snapshot whole-request replay** sees it. This is why in-place retry inside the transaction cannot work and the replay must start a new transaction.
 
-**How it works — deadlock path.** Detection primitive (`_underscore`, reusable): `_Database` gains `public static ?int $lastErrorNumber`, the consts `ERRNO_DEADLOCK` (1213) and `ERRNO_LOCK_WAIT_TIMEOUT` (1205), plus `clearLastError()` and `wasDeadlock()`. `_Query` records the driver errno **at its throw site**, and the recording is **sticky**: a later unrelated error does **not** overwrite an already-recorded deadlock. That stickiness lets a deadlock caught inside a model-save `try/catch` still be visible to the outer controller and trigger a retry. Retry loop (`api2/Controller/Index.php`): the v2 dispatch is wrapped in a bounded loop (`MAX_DEADLOCK_ATTEMPTS = 3`). On an unsuccessful response where `_Database::wasDeadlock() || _Database::wasRetriableUniqueRace()` is true and the route is in `DEADLOCK_RETRIABLE_ROUTES`, the controller:
+**How it works — deadlock path.** Detection primitive (`_underscore`, reusable): `_Database` gains `public static ?int $lastErrorNumber`, the consts `ERRNO_DEADLOCK` (1213) and `ERRNO_LOCK_WAIT_TIMEOUT` (1205), plus `clearLastError()` and `wasDeadlock()`. `_Query` records the driver errno **at its throw site**, and the recording is **sticky**: a later unrelated error does **not** overwrite an already-recorded deadlock. That stickiness lets a deadlock caught inside a model-save `try/catch` still be visible to the outer controller and trigger a retry. Retry loop (`api2/Controller/Index.php`): the v2 dispatch is wrapped in a bounded loop (`MAX_DEADLOCK_ATTEMPTS = 5`). On an unsuccessful response where `_Database::wasDeadlock() || _Database::wasRetriableUniqueRace()` is true and the route is in `DEADLOCK_RETRIABLE_ROUTES`, the controller:
 1. rolls back all open transactions;
 2. wipes `_Database::$_transactionStarts` so every connection re-begins via the umbrella lazy-transaction path on replay;
 3. resets the query + model caches;
@@ -74,12 +77,53 @@ Two consequences of the typed exception worth knowing:
 - **`_Exception_Database` extends `Exception` deliberately — not `RuntimeException`.** A design choice, not an oversight: callers that catch `RuntimeException` to mean "bad data, skip this row" must **not** accidentally swallow a DB failure. It also means any *new* exception type added near this code must extend `Exception` directly, or existing catch blocks change meaning — a live case in [Forecast.Sales import](../../_underscore/features/forecast-sale-import.md).
 - **Every SQL failure in worker2 is now `_Exception_Database`.** So any code that compares `get_class($e)` or matches on DB error **text** changes behaviour under this protocol. Audit for those before assuming the change is transparent.
 
+### ⚠ The API logger used to SILENTLY DISARM this retry — `$lastErrorNumber` must be saved/restored (2026-09-22)
+
+**Symptom:** `/v2/service-requests` returned `400 EV-10` *"Duplicate entry 'SR131023' for key 'ServiceRequests.number'"* — 21 consecutive failures, all ~1.8s with no backoff visible. ServiceNow was **not** sending duplicates. The retry existed and was correctly wired; it just never fired.
+
+**The root-cause chain, in order:**
+1. **`ServiceRequests.number` is not a MySQL auto-increment.** `_Model` `FIELD_AUTOINCREMENT` does a `SELECT MAX(number)+1` and then an `INSERT`, **with no lock between them** (see [FIELD_AUTOINCREMENT record numbering](../../_underscore/features/autoincrement-record-numbering.md)). Two concurrent POSTs read the same MAX, pick the same number, and the loser gets MySQL 1062.
+2. **The retry was already built for exactly this.** `_underscore/Model.php` sets `_Database::$retriableUniqueRace` on that 1062, and `api2/Controller/Index.php` replays the whole request when `_Database::wasRetriableUniqueRace()` is true. That helper requires **both** the flag **and** `$lastErrorNumber === 1062`.
+3. **The logger destroyed the second half of that condition.** `_Component_Api_V2::saveApiLogUnderItsOwnTransactionId()` runs **between the failed INSERT and the retry check**, inside `execute()`. Its resend loop **deliberately causes its own 1062s** — it renames `transactionId` on collision and retries (see [V2 request logging](./request-logging.md)). Those bookkeeping 1062s **overwrote the caller's `$lastErrorNumber`**, so `wasRetriableUniqueRace()` returned false and the replay never ran.
+
+**Fix:** save and restore **both** `_Database::$lastErrorNumber` and `_Database::$retriableUniqueRace` in a `try/finally` **inside** `saveApiLogUnderItsOwnTransactionId()`. Scoped inside that method — *not* at the call site — so it only undoes that loop's own errors; deadlocks, the ASN route, and every other route behave exactly as before.
+
+**The durable rule: logging is bookkeeping and must never change the request's retry decision.** Any code that runs between a failure and the code that inspects it must leave `_Database`'s error state exactly as it found it. Recall that `$lastErrorNumber` is **sticky but a 1062 is NOT protected from overwrite** (unlike a recorded deadlock) — so a 1062 is the error class most easily lost this way.
+
+**Side benefit:** the same fix repairs **ASN** (`_Model_Compass_AdvanceShippingNotice`), which sets the same flag and had the identical exposure.
+
+### Verifying a concurrency fix — the gate-file pattern
+
+A plain background loop spreads requests out over time and can **miss the collision window entirely**, so it produces a false pass. What works: N `curl` workers each **block on a shared gate file**, then a single `touch` releases all of them at the same instant.
+
+Two details that cost time:
+- The v2 auth token is at **`.data.tokens.access`** in the auth response — **not** `.data.token`.
+- Confirm the UNIQUE constraint exists in the environment you are testing (see [V2 request logging](./request-logging.md)) — a local pass proves nothing.
+
+**Result (2026-09-22, `sandbox-dev`):** 10 concurrent SR POSTs → **10/10 HTTP 201**, `SR130589..SR130598`, zero duplicates. Response times stair-stepped 1.32s → 4.78s in ~0.4s steps — that shape **is** the retry backoff actually running, and is the signal that the retry engaged. **Control:** the same route the previous day, before the fix, returned **21 × 400**.
+
+### Deferred hardening (reviewed 2026-09-22, deliberately NOT changed)
+
+Both were considered and left alone on purpose. Do not "fix" either without re-checking the reasoning.
+
+- **The `V2.php` resend loop is unbounded** (`for ($resendNumber = 1; ; ...)`). Safe **today only because every pass changes a UNIQUE column**, so it cannot re-ask for a taken value.
+- **`_underscore/Model.php` marks ANY 1062 as a retriable race** without checking *which* unique key failed. `ServiceRequests` has three unique keys (`PRIMARY`, `uuid`, `number`) and only `number` is the benign race. Harmless today: `uuid` is a random uuid so a collision is effectively impossible, and `PRIMARY` cannot duplicate on INSERT.
+
+### Rejected finding — a null `transactionId` does NOT break the first log save
+
+Checked and **wrong**; recorded so it is not raised again. `Logs.Api.transactionId` is a UNIQUE index with `Null: YES`. **MySQL permits unlimited NULLs in a UNIQUE index**, and 4 such rows already exist in `dev-sandbox` `Logs_Prudential.Api`. A null therefore never raises 1062 and never enters the retry loop.
+
+
 ## Gotchas
 
 - Detection is built on the **MySQL errno** (1213/1205), not on matching the response text — deliberately, because the raw error text is not a reliable signal (and, separately, api2's validation error body still leaks the failing SQL + DB name; see [V2 API error codes](./v2-api-error-codes.md)).
-- Retry is bounded (`MAX_DEADLOCK_ATTEMPTS = 3`) with jittered backoff — a collision damper, not a substitute for removing the deadlock at its source.
+- Retry is bounded (**`MAX_DEADLOCK_ATTEMPTS = 5`**, not 3) with a 20ms-120ms widening backoff — a collision damper, not a substitute for removing the collision at its source.
+- **Anything running between the failing query and the retry check must save/restore `_Database::$lastErrorNumber` + `$retriableUniqueRace`.** The API logger's own 1062s disarmed this retry completely for weeks; a 1062 is not protected from overwrite the way a deadlock is.
+- **Test concurrency with a gate file**, not a background loop — a loop spreads the requests out and can miss the race. Auth token is at `.data.tokens.access`.
+- **A branch cut from `_production` may have the CALL SITE but not the METHOD.** `saveApiLogUnderItsOwnTransactionId()` (commit `db94a26`) exists only on `_sandbox-dev`; branch `TRUE-82095` came off `_production` and would have fataled with *call to undefined method* on every DB-logged request. When branching off `_production`, verify the **callee** exists, not just the caller.
 - `ERRNO_DUPLICATE_ENTRY` is absent from `_beta`/`_sandbox-dev` — naming it directly fatals there; define a local `1062` constant and read `$lastErrorNumber` via `property_exists()`.
-- Only `/v2/advance-shipping-notices` is on `DEADLOCK_RETRIABLE_ROUTES`; do not add a route without proving it is replay-safe (no irreversible side effect, no mutated superglobal).
+- `DEADLOCK_RETRIABLE_ROUTES` is now `/v2/advance-shipping-notices` **and** `/v2/service-requests`; do not add a route without proving it is replay-safe (no irreversible side effect, no mutated superglobal).
+- **Every other `FIELD_AUTOINCREMENT` model has the same unlocked generator** — SalesOrder, PurchaseOrder, Ticket, ItemReceipt, InventoryAdjustment, TransferOrder — and **none is on `DEADLOCK_RETRIABLE_ROUTES`**, so a duplicate-number clash on those routes still returns 400. The durable answer is fixing the number generator so the read and the insert cannot split; that is a bigger, separate change.
 - Do not reach for this protocol for an HTTP 429 — use `_ApiRequest::setAutoRetry()`.
 
 ## Related
@@ -90,3 +134,9 @@ Two consequences of the typed exception worth knowing:
 - [V2 API error codes](./v2-api-error-codes.md)
 - [V2 request logging](./request-logging.md)
 - [ENVIRONMENT decides the `_underscore` branch](./environment-variable-drives-underscore-branch.md)
+
+## Change history
+- 2026-09-22 — Found and fixed the reason this retry never fired on `/v2/service-requests`: the API logger's own 1062s overwrote `_Database::$lastErrorNumber`; now saved/restored in `try/finally` inside `saveApiLogUnderItsOwnTransactionId()`. Proven 10/10 under real concurrency on sandbox-dev. Also fixes ASN. Prod still unpatched. (rgirish)
+- 2026-09-22 — Corrected constants: `MAX_DEADLOCK_ATTEMPTS = 5` (was documented as 3); `DEADLOCK_RETRIABLE_ROUTES` now includes `/v2/service-requests`. (rgirish)
+- 2026-09-16 — Scoped the protocol against `_ApiRequest::setAutoRetry()` (MySQL vs HTTP retry). (bala)
+- 2026-08-28 — Added the retriable unique-key race (1062) path for the Compass ASN data-loss race. (jcardinal)

package/knowledge/2.0/apps/worker2/features/clickup-webhook-health-monitor.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-09-21
+updated: 2026-09-22
 owners: [jcardinal]
 files:
   - worker2/Worker/Monitor/Clickup.php
@@ -25,14 +25,19 @@ worker2 cron that checks the team's ClickUp webhook subscriptions and pages via
 
 Catches the **silent webhook outage** EB health checks cannot see: if ClickUp suspends or fails our inbound webhook (`https://webhook.togahub.com/clickup`), inbound tasks stop arriving and nothing else notices.
 
-Distinct from and **stricter than** the sibling email watchdog [`_Worker_Clickup_Health`](./clickup-connectivity-watchdog.md): this alarms on any `togahub.com` webhook not `health.status = "active"`; the watchdog only emails on `status = "failing"` in business hours.
+Distinct from the sibling email watchdog [`_Worker_Clickup_Health`](./clickup-connectivity-watchdog.md): the watchdog emails on `status = "failing"` in business hours; this monitor reports to OneUptime around the clock and only escalates on `"suspended"`, a missing subscription, or a blind read (see the alarm rule below). A plain `"failing"` status is **not** alarmed here — that softer signal stays with the email watchdog.
 
 ## How it works
 
 - Reads subscriptions: `_Component_Api_Clickup::send('GET','/team')` → for each team `_Component_Api_Clickup::send('GET','/team/{id}/webhook')`.
-- POSTs a metric body via `_ApiRequest` to a OneUptime incoming-request heartbeat URL: `alarm` (`OK`/`DEGRADED`), `probe` (`OK`/`DEGRADED`), plus `webhookCount`, `activeCount`, `failures`.
-- **Alarm rule:** `DEGRADED` when any webhook whose endpoint contains `togahub.com` is not `health.status = "active"`.
-- **Push URL is an in-file `$ONEUPTIME_URL` local, not `[oneuptime]` config** — the established team convention for every `Worker/Monitor/*` file (Operations, Staples, Compass, Aig, Prudential, Nycdoe, Fleet). A cso review suggested reading it from config; the developer rejected that on purpose — the heartbeat URL is treated as low-sensitivity and stays inline, matching the siblings. See the credential note in [OneUptime push-metric monitors](./oneuptime-worker2-monitoring.md). Ships blank with a soft-fail guard until the real URL is pasted in.
+- POSTs a metric body via `_ApiRequest` to a OneUptime incoming-request heartbeat URL: `alarm` (`OK`/`DEGRADED`/`HIGH`), `probe` (`OK`/`DEGRADED`), plus `webhookCount`, `okCount`, `suspendedCount`, `blindCount`, `failureCount`, `failures`, `checkedAtUtc`.
+- **Alarm rule (three levels, revised 2026-09-22 — quieter than the original "not active = bad"):**
+  - `HIGH` (full failure / Offline) — **blind read**: the ClickUp call threw, OR a monitored `togahub.com` webhook has no readable `health.status`. "We don't know the status."
+  - `DEGRADED` — a monitored `togahub.com` webhook has `health.status = "suspended"`, OR no `togahub.com` subscription exists at all. A known outage we can see.
+  - `OK` — every monitored webhook is readable and not suspended (so `active`, `failing`, any other readable status → OK).
+  - Precedence: `HIGH` > `DEGRADED` > `OK`. `probe` is `OK` on any successful read (even when `alarm=HIGH` from a blind webhook) and `DEGRADED` only when the ClickUp read itself failed.
+- **OneUptime criteria must match the tokens** (config lives outside the repo): Request Body Contains `"alarm":"HIGH"` → Offline + incident; Contains `"alarm":"DEGRADED"` → Degraded; received in 1 min AND Contains `"alarm":"OK"` → Online (OK-gated recovery).
+- **Push URL is an in-file `$ONEUPTIME_URL` local, not `[oneuptime]` config** — the established team convention for every `Worker/Monitor/*` file (Operations, Staples, Compass, Aig, Prudential, Nycdoe, Fleet). A cso review suggested reading it from config; the developer rejected that on purpose — the heartbeat URL is treated as low-sensitivity and stays inline, matching the siblings. See the credential note in [OneUptime push-metric monitors](./oneuptime-worker2-monitoring.md). The `$push` guard still soft-fails on a blank URL so the job can deploy inert.
 - Cron row: `dbchanges2/Core/2026-09-21c - Insert ClickUp Webhook Health Monitor CronJob.sql`. OneUptime import artifact `clickup-webhook-health.json` lives in a local staging dir, not the repo (import artifacts are not version-controlled — see the pattern doc's provisioning runbook).
 
 ## Gotchas
@@ -42,6 +47,7 @@ Distinct from and **stricter than** the sibling email watchdog [`_Worker_Clickup
 - Keep the metric POST non-fatal — a monitoring outage must never break the worker job.
 
 ## Change history
+- 2026-09-22 — Reworked the alarm rule from "any status ≠ active → DEGRADED" to three levels: `HIGH` on a blind read (ClickUp call failed, or a webhook has no readable `health.status`), `DEGRADED` on `"suspended"` or no `togahub.com` subscription, `OK` for everything else (`active`/`failing`/other). Original was too sensitive. Also hardened: blind-path OneUptime push wrapped in try/catch (non-fatal), `health` guarded against a non-object, stale "Left BLANK" URL comment removed (URL is live). Payload fields changed: `activeCount` → `okCount`, added `suspendedCount`/`blindCount`. OneUptime monitor criteria must be set to match `"alarm":"HIGH"` and `"alarm":"DEGRADED"`. (jcardinal)
 - 2026-09-21 — Created `_Worker_Monitor_Clickup::CheckWebhook` (OneUptime, `*/10`, stricter than the email watchdog). Recreated the inbound webhook under the service account (`pk_87469444`, new id `bd6168ab-…`, deleting the old OAuth-owned one) so both this monitor and `_Worker_Clickup_Health` can see it — the two were previously blind to it (token-ownership gotcha above). (jcardinal)
 
 ## Related

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.866",
+  "version": "1.0.868",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",