toga-ai 1.0.868 → 1.0.870

package/knowledge/2.0/apps/api2/architecture.md

@@ -6,7 +6,7 @@ project: API
 client: shared
 type: architecture
 status: active
-updated: 2026-09-16
+updated: 2026-09-22
 owners: [jcardinal, bala, mhammontree, dfranks]
 files:
   - api2/Controller/Index.php
@@ -134,7 +134,9 @@ api2's EB target group health-checks **`/health`** (interval 15 s, timeout 5 s,
 
 **A fourth cause (deploy-triggered) — a burst of ELB `502`s when both instances restart at once.** Confirmed on `api-production-1` (2026-08-28 23:51 UTC deploy: **43 × 502 in one minute**, the only ELB 5xx in a 7-hour window). Under the old `AllAtOnce`/100% policy, every deploy to a 2-instance env restarts Apache/PHP-FPM on **both** instances simultaneously, so in-flight requests have their connections dropped mid-response → a burst of **ELB-generated HTTP 502**. EB flags "X% of requests to the ELB are failing with HTTP 5xx" `Degraded` ~3–4 min later and recovers in ~5 min. Fingerprint vs. a real app failure: it is **502** (dropped backend connection), **not 503** (no healthy backend) or **504** (backend timeout); **`HTTPCode_Target_5XX_Count` (app 5xx) = 0**; **`HealthyHostCount` never drops below 2**; and there is **NOTHING in `Logs.Api`** — api2 logs a request only *after* the response, and a 502 never completes one. Diagnosis path: CloudWatch `HTTPCode_ELB_502/503/504_Count` vs `HTTPCode_Target_5XX_Count`, plus EB `describe-events` (a version change) and CodePipeline `list-pipeline-executions` — because a push to `_production` auto-deploys all three prod tiers, "we didn't deploy" can be false (see [CodePipeline deploy workflow](workflows/codepipeline-codeconnections-deploy.md)). **Rule:** an EB "% 5xx" `Degraded` with an **empty `Logs.Api` and 0 target-5xx** right after a deploy is a deploy-restart 502 artifact, not an application bug. **Fixed 2026-08-28:** all three prod envs moved to **Rolling / batch-size 1** — config + tradeoffs in the [deploy workflow](workflows/codepipeline-codeconnections-deploy.md).
 
-**A fifth cause — a brief, self-healing "no-cause TargetGroup reduced health" flip on a LOW-TRAFFIC, IDLE 2-instance env is EB Enhanced Health FLAPPINESS, not worker starvation.** Distinct from the third cause (the *busy* env under Compass ASN saturation). This is a single missed ALB `/health` check — a network/agent transient. Fingerprint (verified `api-production-3`, eu-west-1, 2026-08-29 13:31 UTC `Severe`, static-`/health` build deployed): self-heals in ~2–5 min; throughout, `HealthyHostCount` stays 2, **0** target/ELB 5xx, sub-second latency, idle CPU, near-zero traffic; ALB target health already reads "healthy" by the time you check. On an idle env there is no worker starvation, so **no server-side app fix applies** — do not chase it. Mitigation is alert de-sensitization: the worker2 EB health monitor applies a **persistence-delay gate** (a non-5xx `Degraded`/`Severe` must persist ≥ `nonFivexxPersistMinutes`, default 5, before it pages). See [worker2 EB health monitor](../worker2/features/elastic-beanstalk-health-monitor.md).
+**A fifth cause — a brief, self-healing "no-cause TargetGroup reduced health" flip on a LOW-TRAFFIC, IDLE 2-instance env is EB Enhanced Health FLAPPINESS, not worker starvation.** Distinct from the third cause (the *busy* env under Compass ASN saturation). This is a single missed ALB `/health` check — a network/agent transient. Fingerprint (verified `api-production-3`, eu-west-1, 2026-08-29 13:31 UTC `Severe`, static-`/health` build deployed; **recurred 2026-09-22** — three ~5-min Ok→Warning→Ok cycles, no cause cited): self-heals in ~2–5 min; throughout, `HealthyHostCount` stays 2, **0** target/ELB 5xx, sub-second latency (or NO DATA on a fully idle env), idle CPU, near-zero traffic, empty ALB `RequestCount`; ALB target health already reads "healthy" by the time you check. On an idle env there is no worker starvation, so **no server-side app fix applies** — do not chase it.
+
+Two paging facts confirmed 2026-09-22 on `api-production-3`: (1) **EB native notifications are OFF** (`aws:elasticbeanstalk:sns:topics` endpoint/topic all null) — EB itself never emails on a transition; the **only** paging path is the worker2 → OneUptime cron `Infrastructure/CloudWatch/ElasticBeanstalkHealth`. (2) That monitor pages only on `Degraded`/`Severe`, so a **`Warning`-only flip is below its alarm threshold and never pages** — the 2026-09-22 Warning flaps did not page. Mitigation is alert de-sensitization: the monitor's non-5xx persistence gate (`nonFivexxPersistMinutes`) was **raised from the default 5 to 8** on the API + API-Proxy monitors (dbchanges2 `2026-09-22d`), clearing the ~5-min idle flaps with margin; the 2026-09-01 4xx-rule disable (#6) and Rolling/batch-1 (#4) remain in place. See [worker2 EB health monitor](../worker2/features/elastic-beanstalk-health-monitor.md).
 
 **A sixth cause — a 4xx BURST flips the env to `Degraded`/`Severe` while the ALB stays fully healthy, because EB Enhanced Health counts application/ELB 4xx toward environment health.** Not a deploy (#4), not worker starvation (#3), not idle flappiness (#5) — EB reacting to *client* 4xx responses working as designed. EB Enhanced Health's `ConfigDocument` (namespace `aws:elasticbeanstalk:healthreporting:system`) ships with two rules **Enabled by default**: `Rules.Environment.Application.ApplicationRequests4xx` and `Rules.Environment.ELB.ELBRequests4xx`. So a short burst of client 4xx (e.g. a client polling with an expired bearer token) flips the env to `Degraded`/`Severe` even though the ALB never marked a target unhealthy. Verified live (AWS CLI + CloudWatch + prod `Logs.Api`, acct `654654170868`):
 - `api-production-1` (us-east-1) flip 2026-09-01 01:39–01:44 UTC, no cause cited, **not** during a deploy. Throughout: `HealthyHostCount=2`, `UnHealthyHostCount=0`, **0** target 5xx, **0** ELB 5xx, **0** connection errors.

package/knowledge/2.0/apps/api2/features/v2-deadlock-retry.md

@@ -34,7 +34,7 @@ api2's front controller ("**Prong A**") can **detect a retriable concurrency fai
 
 The replay is **route-scoped by an allowlist** (`DEADLOCK_RETRIABLE_ROUTES` — currently `['/v2/advance-shipping-notices', '/v2/service-requests']`) — because whole-request in-process replay is **only safe for routes with no non-rollbackable external side effect**. The retry gate is now **`wasDeadlock() || wasRetriableUniqueRace()`**.
 
-> **Status (2026-09-22).** Working on `_sandbox-dev` and **proven under real concurrency** (10/10 concurrent SR POSTs, zero duplicates). **Production still has NO race handling at all** for this — prod needs this fix plus the original `TRUE-82095` work. Prod evidence: 12 duplicate-number failures since 2026-08-18, latest `RITM6811124` / `REQ4510546`, which only succeeded on a manual resubmit as `SR131026`.
+> **Status (2026-09-22).** Working on `_sandbox-dev` and **proven under real concurrency** (10/10 concurrent SR POSTs, zero duplicates). **Production still has NO race handling at all** for this — prod needs this fix plus the original `TRUE-82095` work. Prod evidence: 12 duplicate-number failures since 2026-08-18, latest `RITM6811124` / `REQ4510546`, which only succeeded on a manual resubmit as `SR131026`. The race is now **confirmed from production traffic**, not just the load test — see *Confirmed in production* below.
 
 **Constants (verified 2026-09-22):** `MAX_DEADLOCK_ATTEMPTS = 5` (not 3), backoff 20ms-120ms widening per attempt, `DEADLOCK_RETRIABLE_ROUTES = ['/v2/advance-shipping-notices', '/v2/service-requests']`.
 
@@ -92,6 +92,28 @@ Two consequences of the typed exception worth knowing:
 
 **Side benefit:** the same fix repairs **ASN** (`_Model_Compass_AdvanceShippingNotice`), which sets the same flag and had the identical exposure.
 
+### Confirmed in production — three same-second ServiceNow POSTs, one 1062 (2026-09-22)
+
+The race is **not** synthetic. Real Prudential/ServiceNow bulk traffic reproduced it in prod, and prod is **still exposed** (the fix is on `_sandbox-dev` only).
+
+Three POSTs to `/v2/service-requests` arrived from ServiceNow (`sourceIp 199.91.136.12`) inside the same second, **2026-09-21 23:27:47–48 CST**:
+
+| `Logs_Prudential.Api.id` | Result | Number |
+|---|---|---|
+| 8736482 | **201** | `SR131023` |
+| 8736484 | **201** | `SR131024` |
+| 8736483 | **400 `EV-10`** | — |
+
+The 400's `responsePayload` carried the exact failure: **`Duplicate entry 'SR131023' for key 'ServiceRequests.number'`**, MySQL errno **1062**, on `INSERT INTO ServiceRequests`. So all three reads of `MAX(number)` returned the same value; two won (`131023`, `131024`) and the third re-tried `131023` and hit the UNIQUE key. Exactly the unlocked `FIELD_AUTOINCREMENT` generator described above.
+
+**⚠ Note the number in the error is NOT the failed request's number.** `SR131023` belongs to the *winner*. A client report of "duplicate SR131023" is about a request that never got a number at all — do not go looking for two `SR131023` rows.
+
+### ⚠ Open risk before the prod deploy — ServiceNow ALSO resends, so a fixed TOGA could create TWO SRs
+
+The failed `REQ4510546` above was **re-sent from the ServiceNow side ~6 hours later** (2026-09-22 05:23:32 CST, `transactionId 179007261259565713857`) and succeeded as **`SR131026`**. So today the outside resend is the only recovery — and it works.
+
+Once the `TRUE-82095` retry ships to prod, **TOGA retries internally within the same request**. If ServiceNow still resends the earlier failure on its own schedule, the same REQ can end up with **two ServiceRequests rows**. Confirm ServiceNow's resend behaviour with the client (contact: **Ferdaus**) before or alongside the prod deploy. Nothing in this retry protocol de-duplicates by `c_reqNumber`.
+
 ### Verifying a concurrency fix — the gate-file pattern
 
 A plain background loop spreads requests out over time and can **miss the collision window entirely**, so it produces a false pass. What works: N `curl` workers each **block on a shared gate file**, then a single `touch` releases all of them at the same instant.
@@ -124,6 +146,8 @@ Checked and **wrong**; recorded so it is not raised again. `Logs.Api.transaction
 - `ERRNO_DUPLICATE_ENTRY` is absent from `_beta`/`_sandbox-dev` — naming it directly fatals there; define a local `1062` constant and read `$lastErrorNumber` via `property_exists()`.
 - `DEADLOCK_RETRIABLE_ROUTES` is now `/v2/advance-shipping-notices` **and** `/v2/service-requests`; do not add a route without proving it is replay-safe (no irreversible side effect, no mutated superglobal).
 - **Every other `FIELD_AUTOINCREMENT` model has the same unlocked generator** — SalesOrder, PurchaseOrder, Ticket, ItemReceipt, InventoryAdjustment, TransferOrder — and **none is on `DEADLOCK_RETRIABLE_ROUTES`**, so a duplicate-number clash on those routes still returns 400. The durable answer is fixing the number generator so the read and the insert cannot split; that is a bigger, separate change.
+- **The `SR…` number inside a `Duplicate entry` error belongs to the WINNER, not the failed request.** The loser never got a number. A "duplicate SR131023" report means one request was lost, not that two rows share a number.
+- **ServiceNow resends a failed SR on its own** (proven: `REQ4510546` failed, came back ~6h later as `SR131026`). After the prod deploy adds an internal retry, an outside resend could produce a **second SR for the same REQ** — confirm with the client first.
 - Do not reach for this protocol for an HTTP 429 — use `_ApiRequest::setAutoRetry()`.
 
 ## Related
@@ -136,6 +160,7 @@ Checked and **wrong**; recorded so it is not raised again. `Logs.Api.transaction
 - [ENVIRONMENT decides the `_underscore` branch](./environment-variable-drives-underscore-branch.md)
 
 ## Change history
+- 2026-09-22 — Added production confirmation of the race (3 same-second ServiceNow POSTs, `Logs_Prudential.Api` 8736482/8736483/8736484, 1062 on `SR131023`) and the open double-SR risk from ServiceNow's own resend. Investigation only, no code change. (rgirish)
 - 2026-09-22 — Found and fixed the reason this retry never fired on `/v2/service-requests`: the API logger's own 1062s overwrote `_Database::$lastErrorNumber`; now saved/restored in `try/finally` inside `saveApiLogUnderItsOwnTransactionId()`. Proven 10/10 under real concurrency on sandbox-dev. Also fixes ASN. Prod still unpatched. (rgirish)
 - 2026-09-22 — Corrected constants: `MAX_DEADLOCK_ATTEMPTS = 5` (was documented as 3); `DEADLOCK_RETRIABLE_ROUTES` now includes `/v2/service-requests`. (rgirish)
 - 2026-09-16 — Scoped the protocol against `_ApiRequest::setAutoRetry()` (MySQL vs HTTP retry). (bala)

package/knowledge/2.0/apps/worker2/features/elastic-beanstalk-health-monitor.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
+updated: 2026-09-22
 owners: [jcardinal]
 files:
   - worker2/Worker/Infrastructure/CloudWatch.php
@@ -65,8 +65,8 @@ Removed this iteration: `shouldPageForHealth()` and the old `HTTP_5XX_ALARM_RATI
 
 A single `ElasticBeanstalkHealth` cron watches environments of **very different scale**, so thresholds are tuned per monitor. Measured 3h baselines: api-production-1 ~922–1737 req/5min, apiproxy ~400–1650, but eu-west-1 envs only ~40–90 req/5min; 5xx was 0 across all.
 
-- **Worker + worker 1.0 monitors** keep the **method defaults** (`100 / 20% / 20`) — the low absolute floor suits ~150 req/5min traffic.
-- **API and API-Proxy monitors** (each also including the low-traffic eu-west-1 env) are overridden to **`windowMinutes=5, minRequests=30, min5xxPercent=20, min5xxAbsolute=100`** — the lower request floor keeps the small env on the share gate, while the higher absolute floor stops the busy envs paging on minor blips (20 scattered 500s on ~1300 req = <2%). The override is applied by the dbchanges2 migration below.
+- **Worker + worker 1.0 monitors** keep the **method defaults** (`100 / 20% / 20`, and `nonFivexxPersistMinutes` default 5) — the low absolute floor suits ~150 req/5min traffic.
+- **API and API-Proxy monitors** (each also including the low-traffic eu-west-1 env) are overridden to **`windowMinutes=5, minRequests=30, min5xxPercent=20, min5xxAbsolute=100, nonFivexxPersistMinutes=8`** — the lower request floor keeps the small env on the share gate, the higher absolute floor stops the busy envs paging on minor blips (20 scattered 500s on ~1300 req = <2%), and the **8-minute** non-5xx persistence raises the gate above the observed ~5-min idle-flap length (the default 5 sat at the flap boundary). The overrides are applied by the two dbchanges2 migrations below (`2026-08-25a` = the four rate keys; `2026-09-22d` = `nonFivexxPersistMinutes=8`).
 
 #### Fail-safe: an unmeasurable solely-5xx event still pages
 
@@ -78,7 +78,7 @@ A CloudWatch read *failure* (as opposed to empty data) sets **`probe=DEGRADED`**
 
 A non-5xx `Degraded`/`Severe` pages only after the **current non-Ok episode has persisted ≥ `nonFivexxPersistMinutes`** (default 5). This exists because the brief "no-cause TargetGroup reduced health" flips on low-traffic 2-instance envs are **EB Enhanced Health flappiness** — a single missed ALB `/health` check (a network/agent transient), not a real app problem — and they self-heal in ~2–5 min. A persistence delay suppresses those while a genuinely sustained non-5xx degradation still pages, delayed only by the threshold. (The windowed 5xx-flood gate is **unchanged** — it has its own, independent decision.)
 
-**Fingerprint of the idle-env flap** (verified `api-production-3`, eu-west-1, 2026-08-29 13:31 UTC `Severe`, static-`/health` build deployed): self-heals in ~2–5 min; throughout, `HealthyHostCount` stays 2 (1-min minimum), **0** target/ELB 5xx, sub-second latency, idle CPU, near-zero traffic; ALB target health already reads "healthy" by the time you look. On an idle env there is **no worker starvation**, so nothing to fix server-side — the mitigation is this monitor persistence-delay gate, not more app code. **This is DISTINCT from the PHP-`/health`-starvation cause** (that one is the *busy* env, `api-production-1`, under ASN worker saturation, fixed by the static-`/health` file) — see [api2 architecture — EB "Degraded" health](../../api2/architecture.md).
+**Fingerprint of the idle-env flap** (verified `api-production-3`, eu-west-1, 2026-08-29 13:31 UTC `Severe`, static-`/health` build deployed; **recurred 2026-09-22** as three ~5-min Ok→Warning→Ok cycles on the same env, no cause cited): self-heals in ~2–5 min; throughout, `HealthyHostCount` stays 2 (1-min minimum), **0** target/ELB 5xx, latency/4xx/5xx = NO DATA (idle, zero traffic), ALB `RequestCount` empty; ALB target health already reads "healthy" by the time you look. The 2026-09-22 recurrence sat at the **~5-min boundary of the default `nonFivexxPersistMinutes=5`**, which is why the API + API-Proxy monitors were raised to **8** (migration `2026-09-22d`). Note the 2026-09-22 flips peaked at `Warning`, which is **below the monitor's alarm threshold** (the EB-health path pages only on `Degraded`/`Severe`), so they never paged; the gate raise is **preventive** for the `Severe`-flavored idle flap (e.g. 2026-08-29). On an idle env there is **no worker starvation**, so nothing to fix server-side — the mitigation is this monitor persistence-delay gate, not more app code. **This is DISTINCT from the PHP-`/health`-starvation cause** (that one is the *busy* env, `api-production-1`, under ASN worker saturation, fixed by the static-`/health` file) — see [api2 architecture — EB "Degraded" health](../../api2/architecture.md).
 
 **Measuring episode age — `fetchNonOkDepartureAgeSeconds()`** reads EB `describeEvents`:
 - The most recent **"transitioned from Ok to …"** event marks the current episode's start. Once non-Ok, later transitions stay between non-Ok states until a to-Ok event; because `describeEvents` returns **newest-first**, the first "from Ok to …" match wins.
@@ -129,6 +129,8 @@ WHERE action = 'Infrastructure/CloudWatch/ElasticBeanstalkHealth'
 
 `Core.CronJobs.parameters` is confirmed a native `JSON` column, so `JSON_SET` merges rather than overwrites.
 
+**`dbchanges2/Core/2026-09-22d - Beanstalk Health NonFivexx Persist Gate.sql`** (same pattern) adds only `$.nonFivexxPersistMinutes = 8` on the same two rows (matched by `action` + `name IN (API-Production, API Proxy-Production)`), raising the non-5xx persistence gate above the ~5-min idle-flap length seen in the 2026-09-22 `api-production-3` recurrence. Existing threshold keys are `JSON_TYPE=INTEGER`, so a bare `8` lands as an integer. Same **code-before-params deploy order** applies: worker2's `nonFivexxPersistMinutes` support (added 2026-08-29) must be live on `_production` first, or the monitor throws "Unknown named parameter" and goes dark. Ran on prod Core 2026-09-22.
+
 ### OneUptime monitor config (external, not code)
 
 Paging depends on the OneUptime Incoming-Request monitor's **string-match** criteria on the POSTed body — `Contains "alarm":"HIGH"` pages.
@@ -146,6 +148,12 @@ Paging depends on the OneUptime Incoming-Request monitor's **string-match** crit
 - **Deploy fails without degrading health** — still not caught (accepted residual).
 - **`ElasticBeanstalkHealth()` is long** and grew with the persistence gate — php-reviewer (0 critical) flagged it as a decomposition candidate; deferred pending the missing harness.
 
+## Change history
+
+- 2026-09-22 — API + API-Proxy monitors set `nonFivexxPersistMinutes=8` (was default 5) via dbchanges2 `2026-09-22d`, after the `api-production-3` idle-flap recurred at the ~5-min default boundary; default 5 kept on Worker/Worker-1.0. (jcardinal)
+- 2026-08-29 — added the non-5xx persistence gate (`nonFivexxPersistMinutes`, default 5) to suppress brief self-healing idle-env EB flaps. (jcardinal)
+- 2026-08-25 — 5xx page moved to a CloudWatch windowed request-rate gate; API/API-Proxy override applied via dbchanges2 `2026-08-25a`. (jcardinal)
+
 ## Related
 
 - [Cross-account AWS access](./cross-account-aws-access.md)

package/knowledge/clients/prudential/features/service-request-auth-rejection-alert.md

@@ -6,8 +6,8 @@ project: Worker
 client: prudential
 type: client-feature
 status: active
-updated: 2026-09-16
-owners: ["bala"]
+updated: 2026-09-22
+owners: ["bala", "rgirish"]
 files:
   - worker/crons/notifications/reports/prudential/prudential_auth_rejection_alert.php
   - worker/schedules/cron.worker.notification.json
@@ -19,6 +19,7 @@ related:
   - ../profile.md
   - ../../../2.0/apps/api2/features/request-logging.md
   - ../../../2.0/apps/api2/features/v2-api-error-codes.md
+  - ../../../2.0/apps/api2/features/v2-deadlock-retry.md
 ---
 The 15-min cron that alerts on refused (401/403) inbound Prudential REQs, where each code is logged, and how to tell "we refused it" from "they never sent it"; open when a Prudential REQ is reported missing.
 
@@ -68,7 +69,25 @@ Two different failure shapes get reported the same way ("REQ is missing"):
 
 Always check **both** schemas before answering — a 401 will not be in `Logs_Prudential`.
 
+### Triaging a "duplicate SR number" report — the recipe (2026-09-22)
+
+A third shape, and the report is misleading on its face: ServiceNow says *"duplicate `SR131023`"*, but **that number belongs to a different request that succeeded**. The reported REQ got no number at all — it lost the `FIELD_AUTOINCREMENT` race and died on `1062` (see [V2 request retry](../../../2.0/apps/api2/features/v2-deadlock-retry.md)). Do not go hunting for two rows with the same number.
+
+Where the Prudential data actually lives:
+- ServiceNow REQ/RITM land in `Client_Prudential.ServiceRequests` as **`c_reqNumber`** and **`c_ritmNumber`** — *not* `c_snowRequestNumber` / `c_snowRitmNumber`. The other ServiceNow columns on that table are `c_sysId` and `c_sysCreatedOn`.
+- `ServiceRequests` has **no `dtModified` column** — the timestamps are `dtCreated` / `dtUpdated`.
+- `c_sysCreatedOn` is ServiceNow time (**EST**); `dtCreated` is TOGA DB time (**CST**). **A one-hour gap between them is expected**, not a bug.
+
+How to find the failure:
+1. Look the call up in `Logs_Prudential.Api` by **`transactionId`** — it is UNIQUE and indexed, so this is instant.
+2. Read the full MySQL error out of **`responsePayload.messages[].identifiers.error`** — that is where `Duplicate entry 'SR131023' for key 'ServiceRequests.number'` and errno `1062` live.
+3. Check whether the REQ came back later on its own: ServiceNow does resend. `REQ4510546` failed on 2026-09-21 and succeeded ~6h later as `SR131026`.
+
+**⚠ Never run an unbounded `requestPayload LIKE '%REQ…%'` against `Logs_Prudential.Api`** — `requestPayload` is not indexed, so it full-scans the table and times out. Always bound it with a **`dtStamp` window plus `route` / `method`** first (`dtStamp` is indexed; `route` and `responseCode` are not).
+
 ## Gotchas
+- **Prudential SR columns are `c_reqNumber` / `c_ritmNumber`**, there is no `dtModified` (it is `dtCreated`/`dtUpdated`), and `c_sysCreatedOn` (EST) sitting ~1h off `dtCreated` (CST) is correct.
+- **Never `LIKE` on `requestPayload` without a `dtStamp` window** — it scans `Logs_Prudential.Api` and times out.
 - **Filter `method = 'POST'` or you get browser noise.** Unauthenticated **GET**s on this route from human sourceIps (`99.179.145.25`, `107.139.5.190`) also log 401, with an **empty `requestPayload`** — they show up as REQ "Unknown". Only a POST is a real ServiceNow submission.
 - **`prudential_api_500_retry.php` is broken and has almost certainly never worked.** At lines 57 and 70 it references a bare constant **`PRUDENTIAL_CLIENT_UUID` that is defined nowhere** in any repo (grep returns only its own two uses); it should be `App_Api_Toga2::CLIENT_UUID_PRUDENTIAL`. On PHP 8 an undefined constant is a **fatal Error**, so the cron dies every run; on PHP 7 it evaluates to the literal string and the client check never matches, so it silently retries nothing. It also appears in **no** `schedules/*.json` file, so it may never have been registered. **Not fixed as of 2026-09-10 — reported only.**
 - **`Logs.Api.dtStamp` IS indexed** (`Api_dtStamp_IDX`) in both base `Logs` and `Logs_Prudential`, which is what makes the 15-minute window read cheap. But `route` and `responseCode` are **not** indexed, so a wide `dtStamp` range plus a route filter still times out (a 2.5-month grouped query aborted at 300 s). Keep the window tight. See [request-logging.md](../../../2.0/apps/api2/features/request-logging.md).
@@ -78,3 +97,6 @@ Always check **both** schemas before answering — a 401 will not be in `Logs_Pr
 - [Service request address validation](service-request-address-validation.md)
 - [Prudential profile](../profile.md)
 - [V2 request logging](../../../2.0/apps/api2/features/request-logging.md), [V2 API error codes](../../../2.0/apps/api2/features/v2-api-error-codes.md)
+
+## Change history
+- 2026-09-22 — Added the "duplicate SR number" triage recipe: real column names, where the MySQL error text sits in `responsePayload`, the `requestPayload LIKE` timeout trap, and the EST/CST one-hour offset. Prod investigation only, no code change. (rgirish)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.868",
+  "version": "1.0.870",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",