toga-ai 1.0.794 → 1.0.795

package/knowledge/2.0/apps/api2/features/request-logging.md

@@ -6,7 +6,7 @@ project: API
 client: shared
 type: feature
 status: active
-updated: 2026-09-01
+updated: 2026-09-10
 owners: ["mhammontree", "dfranks", "bala", "jcardinal"]
 files:
   - api2/Component/Api/V2/V2.php
@@ -21,6 +21,7 @@ related:
   - ./environment-variable-drives-underscore-branch.md
   - ../../_underscore/features/error-reporting-issue-event.md
   - ../../../../clients/aig/features/entitlement-intake.md
+  - ../../../../clients/prudential/features/service-request-auth-rejection-alert.md
   - ../../../../clients/compass-usa/features/mits-po-to-so-item-linking.md
   - ../../../../clients/compass-usa/workflows/order-lifecycle-and-data-integrity.md
 ---
@@ -53,6 +54,24 @@ In `api2/Component/Api/V2/V2.php` (~L2168) every request is logged and routed:
 - There is also a **config-gated branch** (~L2166): when `_Config::api('log_filepath')` is set,
   the log entry is written to a **file** on the host instead of the DB.
 
+### 401 lands in base `Logs`; 403 lands in `Logs_<Client>`
+
+The pre-scope/post-scope split above has a sharp, practical edge on the two auth codes:
+
+- **401** (`EN-1`..`EN-6`, e.g. an expired token) is decided in the auth chain in `V2.php`
+  (~L2088, inside `execute()`) **before routing**, so the client is not resolved and the row can
+  only go to the **shared base `Logs`**.
+- **403** happens **after** the client IS known, so that row lands in **`Logs_<Client>`**.
+
+Verified on Prudential for 2026-08-31 – 09-01: `Logs_Prudential.Api` held **56** rows for
+`/v2/service-requests`, **all 201**, while every one of the **36 401s** sat in base `Logs`.
+
+**A 401 therefore can never reach a model interceptor** — no `prePost`/`postPost` hook can alert on
+it, because routing and the model are never reached. Anything that must react to refused inbound
+traffic has to **scan the base log**, not hook the model. The caller is still identifiable: decode
+the Bearer token in `requestHeaders` and read `id.client.uuid`. See
+[Prudential refused-REQ alert](../../../../clients/prudential/features/service-request-auth-rejection-alert.md).
+
 ### A rejected request (HTTP 400) is NOT unlogged
 
 `api2/Controller/Index.php` (L369-377) handles a non-success response by **committing** the
@@ -359,6 +378,13 @@ The working pattern, in order:
 > a field its response echo omitted (see the [V2 query contract](v2-rest-query-contract.md) PUT-echo
 > gotcha).
 
+> **What is actually indexed (checked with `SHOW INDEX`, 2026-09-10):** `dtStamp` **IS** indexed
+> — `Api_dtStamp_IDX` — in **both** the base `Logs` (~3.67M rows) and `Logs_Prudential` (~7.86M
+> rows). `route` and `responseCode` are **NOT** indexed. So a *tight* `dtStamp` window is cheap
+> (a 15-minute slice is trivial), but a **wide** `dtStamp` range combined with a `route` filter
+> still times out — a 2.5-month grouped query aborted at 300 s. Narrow the time window first;
+> the other predicates do no work for you.
+
 1. **Narrow with indexed / non-text columns only** — a `dtStamp` range plus `route`,
    `method`, `responseCode`, `hostname`. A
    `GROUP BY hostname, route, method, responseCode` over one day is cheap and is the fastest way
@@ -400,6 +426,16 @@ re-send.
 > replay operationally, not by pasting payloads into the KB.
 
 ## Change history
+- 2026-09-10 — Sharpened the client-vs-core routing on the auth codes: a **401 always lands in the
+  base `Logs`** (decided in the auth chain ~L2088 before routing, client unresolved) while a **403
+  lands in `Logs_<Client>`** (client already known) — verified on Prudential 31 Aug–1 Sep, where
+  `Logs_Prudential.Api` held 56 rows for `/v2/service-requests` (all 201) and all **36 401s** were
+  in base `Logs`. Consequence: **no model interceptor can ever fire for a 401**, so alerting on
+  refused inbound traffic must scan the base log; the caller is recoverable by decoding
+  `id.client.uuid` from the Bearer token. Also corrected the index picture with `SHOW INDEX`:
+  `Logs.Api.dtStamp` **is** indexed (`Api_dtStamp_IDX`) in both base `Logs` (~3.67M rows) and
+  `Logs_Prudential` (~7.86M rows), but `route`/`responseCode` are **not** — a tight window is cheap,
+  a wide window plus a route filter still aborts at 300 s. No code change. (bala)
 - 2026-09-01 — **Fixed the duplicate-`transactionId` 500** (`Logs.Issue CT`, `TRUE-81056` /
   `db94a26`): the log-save at the end of `execute()` did a **check-then-insert**, which cannot work
   under concurrency because the rival copy's row is still uncommitted and therefore invisible to the

package/knowledge/INDEX.md

@@ -5,7 +5,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 ## 1.0 framework
 
 - **library** (Library) _(framework core)_ — 25 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
-- **worker** (Worker) — 34 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
+- **worker** (Worker) — 35 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
 - **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
 - **worker1.5** (Worker 1.5) — 0 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
 - **togadesk** (TOGa Desk) — 13 doc(s) → [1.0/apps/togadesk/INDEX.md](1.0/apps/togadesk/INDEX.md)

package/knowledge/clients/prudential/INDEX.md

@@ -8,6 +8,7 @@
 | [Prudential: OneUptime order-pipeline backlog monitors (Monitor/Prudential/*)](features/oneuptime-order-pipeline-monitors.md) | 2.0 | Three worker2 OneUptime push monitors (TRUE-80587) that watch the Prudential/Dell order pipeline for **stalls**, which were previously invisible — a stuck pipel |
 | [Prudential: order email PDF attachments (which guide goes on which email)](features/order-email-pdf-attachments.md) | 1.0 | Every customer-facing Prudential order email carries S3-hosted PDF guides, and **which** guide is attached is decided by the service request type. |
 | [Prudential: Service Request Regional Address Validation](features/service-request-address-validation.md) | 2.0 | The `prePost` interceptor on `_Model_Prudential_ServiceRequest` validates `deliverToAddress` fields differently depending on which Prudential regional customer |
+| [Prudential: refused REQ alert (401/403 on POST /v2/service-requests)](features/service-request-auth-rejection-alert.md) | 1.0 | ServiceNow posts every Prudential REQ to `POST /v2/service-requests`. |
 | [Prudential: Service Request rejection alert email](features/service-request-rejection-alert-email.md) | 2.0 | When a Prudential ServiceNow→TOGa service-request submission (`POST /v2/service-requests`) is **rejected by validation**, TOGa now sends a real-time internal al |
 | [Prudential: Sales Orders CSV export (TOGa 2.5 Supply)](features/supply-orders-csv-export.md) | 2.0 | The **Export** button on the TOGa 2.5 Supply *Sales Orders* page is **Prudential-only**: it downloads a 22-column CSV (`prudential-orders-<YYYY-MM-DD>.csv`) cov |
 | [Prudential Order Shipped Email — transmit_ordershipped_updates_prudential.php](features/transmit-ordershipped-email.md) | 1.0 | Cron script that transmits "Order Shipped" updates to ServiceNow (RITM) and sends a shipped notification email to the end user. |

package/knowledge/clients/prudential/features/service-request-auth-rejection-alert.md

@@ -0,0 +1,156 @@
+---
+title: "Prudential: refused REQ alert (401/403 on POST /v2/service-requests)"
+framework: "1.0"
+repo: worker
+project: Worker
+client: prudential
+type: client-feature
+status: active
+updated: 2026-09-10
+owners: ["bala"]
+files:
+  - worker/crons/notifications/reports/prudential/prudential_auth_rejection_alert.php
+  - worker/schedules/cron.worker.notification.json
+  - worker/crons/notifications/reports/prudential_exception_report.php
+  - worker/crons/notifications/reports/prudential/prudential_api_500_retry.php
+related:
+  - service-request-rejection-alert-email.md
+  - service-request-address-validation.md
+  - ../profile.md
+  - ../../../2.0/apps/api2/features/request-logging.md
+  - ../../../2.0/apps/api2/features/v2-api-error-codes.md
+---
+
+## Summary
+
+ServiceNow posts every Prudential REQ to `POST /v2/service-requests`. When we **refuse** that
+request at the door — **401** (login token bad or expired) or **403** (permissions) — no
+ServiceRequest row is created, no interceptor runs, and **no existing report shows it**. The REQ
+simply does not exist on our side and nobody is told. The cron
+`worker/crons/notifications/reports/prudential/prudential_auth_rejection_alert.php` closes that
+gap: every 15 minutes it reads refused inbound REQs straight from the API logs and emails them
+with the full payload, so they can be resubmitted the same hour instead of being found weeks later.
+
+This is the **auth-layer** sibling of the 2.0 validation alert
+([service-request-rejection-alert-email.md](service-request-rejection-alert-email.md)), which
+covers 400s. The two never overlap: a 400 reaches the model, a 401 never does.
+
+## Why it exists (the 2026-08/09 incident)
+
+REQ4492868 was reported missing. It **did** reach us on 2026-08-31 09:45:18 (POST
+`/v2/service-requests`, BREAKFIX, RITM6763537) and came back **401 `EN-4` "The provided token has
+expired"**. ServiceNow (sourceIp `199.91.136.12`) collected a JWT at 07:40:47 that expired at
+08:40:47 (1-hour life) and then kept reusing that same dead token for hours — every refused row
+carries the **identical `exp`**. **36 REQs** were refused this way across 31 Aug and 1 Sep. 14 were
+later resent and created fine (the Reclaims, replayed 1 Sep ~15:00); **22, including REQ4492868,
+were never resent and are still missing.** The whole class of failure was silent for a week.
+
+## Why nothing else caught it
+
+- `worker/crons/notifications/reports/prudential_exception_report.php` (daily, 9:00 AM) only queries
+  response codes **201, 400 and 500**, and only against **`Logs_Prudential`** (`db_prod_logs_prudential`).
+- `prudential_api_500_retry.php` only queries **500**.
+
+So **401 and 403 were the two codes nobody was looking at** — and a 401 is not even in the schema
+those reports read. This is also why the new cron cannot double-report: no other job touches these
+codes.
+
+## Where a refused request is actually logged (the load-bearing fact)
+
+- **401 → the SHARED BASE `Logs` schema** (`db_prod_logs`), **not** `Logs_Prudential`. The
+  expired-token check runs in the auth chain in `api2/Component/Api/V2/V2.php` (~L2088, inside
+  `execute()`) **before** routing, so the client is not resolved yet and the row cannot be written
+  to the client log schema.
+- **403 → `Logs_Prudential`.** By then the client **is** known.
+
+Verified for 31 Aug – 1 Sep: `Logs_Prudential.Api` holds **56 rows** for `/v2/service-requests`,
+**all 201**; every one of the **36 401s** is in base `Logs`.
+
+**Consequence: a model `prePost` / interceptor can NEVER fire for a 401** — the request never
+reaches routing or the model. That is why this alert had to be a log scan and not a hook. The
+client is still identifiable: decode the expired Bearer token and read `id.client.uuid`.
+
+## How it works
+
+1. Floor the clock to the quarter hour and read exactly `[end - 15 min, end)`.
+2. Read **401s from base `Logs`** (`db_prod_logs`) and **403s from `Logs_Prudential`**
+   (`db_prod_logs_prudential`), both filtered to `direction = 'IN'`, `method = 'POST'`,
+   `route = '/v2/service-requests'`.
+3. Decode the Bearer token in `requestHeaders` to confirm the caller is Prudential
+   (`App_Api_Toga2::CLIENT_UUID_PRUDENTIAL`). A **missing** token yields no uuid — those rows are
+   **still reported**, deliberately, rather than dropped.
+4. Map the `EN-*` code from the response payload to a plain-English sentence
+   (`EN-1` no token sent · `EN-2` unreadable · `EN-3` not valid · `EN-4` expired · `EN-5` altered ·
+   `EN-6` address not allowed), and classify the row as **Login** (401) or **Permissions** (403).
+5. Email one grouped HTML table — REQ number, RITM number, type, time raised in ServiceNow, time it
+   reached us, reason — plus a **CSV attachment whose last column carries the full request
+   payload**, so the mail can be forwarded to Prudential as proof of what they sent.
+6. Send **nothing** on a quiet run. Cap of `MAX_ROWS_PER_RUN = 500` so one bad hour cannot produce a
+   giant email.
+
+Schedule entry in `worker/schedules/cron.worker.notification.json`: `"*/15 * * * *"` →
+`notifications/reports/prudential/prudential_auth_rejection_alert.php` (notification worker role).
+Recipients: `vburks`, `rgirish`, `ADiamond`, `kmaramreddy` @togatech.com.
+
+## Design decision — no state table, no state file (clock-aligned window instead)
+
+The first design kept a "last processed log id" marker. Both marker forms were rejected:
+
+- **A file marker is wrong here.** The worker cache folder is `/var/www/cache`, **local to each
+  Elastic Beanstalk instance** and recreated on every deploy (see
+  `worker/ebs/setup_export_cache_folders.php`). The same rejections would be emailed once *per
+  instance*, and the marker is lost on every deploy.
+- **A DB marker table** was drafted, then rejected as too much machinery for the value.
+
+**Final design:** each run floors the clock to the quarter hour and reads `[end - 15 min, end)`.
+Consecutive slices touch but never overlap, so no row can be read twice — **with no state at all.**
+
+**Known trade-off, accepted:** a skipped run (deploy, restart) loses that 15-minute slice. Do not
+"fix" this by re-adding a marker without re-reading the two reasons above.
+
+## Gotchas
+
+- **Filter `method = 'POST'` or you get browser noise.** Unauthenticated **GET**s on this route from
+  human sourceIps (`99.179.145.25`, `107.139.5.190`) also log 401, with an **empty `requestPayload`**
+  — they show up as REQ "Unknown". Only a POST is a real ServiceNow submission.
+- **`prudential_api_500_retry.php` is broken and has almost certainly never worked.** At lines 57
+  and 70 it references a bare constant **`PRUDENTIAL_CLIENT_UUID` that is defined nowhere** in any
+  repo (grep returns only its own two uses); it should be `App_Api_Toga2::CLIENT_UUID_PRUDENTIAL`.
+  On PHP 8 an undefined constant is a **fatal Error**, so the cron dies every run; on PHP 7 it
+  evaluates to the literal string and the client check never matches, so it silently retries
+  nothing. It also appears in **no** `schedules/*.json` file, so it may never have been registered.
+  **Not fixed as of 2026-09-10 — reported only.**
+- **`Logs.Api.dtStamp` IS indexed** (`Api_dtStamp_IDX`) in both base `Logs` and `Logs_Prudential`,
+  which is what makes the 15-minute window read cheap. But `route` and `responseCode` are **not**
+  indexed, so a wide `dtStamp` range plus a route filter still times out (a 2.5-month grouped query
+  aborted at 300 s). Keep the window tight. See
+  [request-logging.md](../../../2.0/apps/api2/features/request-logging.md).
+
+## Telling "we refused it" from "they never sent it"
+
+Two different failure shapes get reported the same way ("REQ is missing"):
+
+- **Refused by us** — there IS an inbound POST row on `/v2/service-requests` with a 401/403 and a
+  full `requestPayload`. That is REQ4492868. Prudential must resubmit.
+- **Never sent** — no inbound row at all. That is **REQ4398272**: absent from
+  `Client_Prudential.ServiceRequests` under both `c_reqNumber` and `c_ritmNumber`, and absent from
+  `Logs_Prudential.Api` (1 May – 15 Jul) and base `Logs.Api` (10–16 May) — no POST, no 401, no 400,
+  no 500. By the surrounding REQ numbers it would have been raised 2026-05-12 between 11:23 and
+  12:53 (REQ4398185 landed 11:23, REQ4398357 landed 12:53). Base `Logs` retention reaches back to
+  Aug 2025, so the window is genuinely covered. The problem is on the ServiceNow side.
+
+Always check **both** schemas before answering — a 401 will not be in `Logs_Prudential`.
+
+## Change history
+- 2026-09-10 — Created. New 15-minute cron
+  `notifications/reports/prudential/prudential_auth_rejection_alert.php` alerting on Prudential REQs
+  refused with **401 (base `Logs`) or 403 (`Logs_Prudential`)**, with an HTML table plus a
+  full-payload CSV for forwarding to Prudential. Found after REQ4492868 was reported missing: it was
+  refused 2026-08-31 09:45:18 with `EN-4` (expired token), one of **36** refused across 31 Aug–1 Sep
+  from a JWT ServiceNow collected at 07:40:47 and reused past its 08:40:47 expiry — **22 still
+  missing**. Recorded why nothing caught it (the daily exception report reads only 201/400/500 and
+  only `Logs_Prudential`), why a `prePost` hook cannot work for a 401, the **no-state clock-aligned
+  window** decision (the `/var/www/cache` marker is per-instance and dies on deploy), the
+  **POST-only** filter that excludes browser 401 noise, that `dtStamp` is indexed while
+  `route`/`responseCode` are not, and that `prudential_api_500_retry.php` is dead on an undefined
+  `PRUDENTIAL_CLIENT_UUID` constant. (bala)

package/knowledge/clients/prudential/features/service-request-rejection-alert-email.md

@@ -6,13 +6,14 @@ project: Worker
 client: prudential
 type: client-feature
 status: active
-updated: 2026-07-28
+updated: 2026-09-10
 owners: ["bala"]
 files:
   - worker2/Worker/Client/Prudential/reports/ReqRejectionEmail.php
   - _underscore/Model/Prudential/ServiceRequest.php
 related:
   - service-request-address-validation.md
+  - service-request-auth-rejection-alert.md
   - ../../../2.0/apps/_underscore/features/email-send-pipeline.md
   - ../profile.md
 ---
@@ -79,11 +80,21 @@ known residual risk.
   validation block completes (e.g. the unguarded `serviceRequestUnits[0]->unit` access — see
   `service-request-address-validation.md`) produces a **500 and no alert**. Fix that guard to
   keep the alert reliable.
+- **This alert only covers 400s — a 401/403 never reaches `prePost` at all.** An expired or bad
+  ServiceNow token is refused in the auth chain *before* routing, so the model never runs and no
+  interceptor can fire; the row is also logged to the **base `Logs`** schema, not `Logs_Prudential`.
+  Those refusals are alerted separately by the 1.0 worker cron in
+  [service-request-auth-rejection-alert.md](service-request-auth-rejection-alert.md). Do not extend
+  this interceptor to try to catch them.
 - **"Plain text" is transmitted HTML anyway.** `setIsHtml(false)` is honored by `_Email` when it
   queues the row, but the worker2 `Infrastructure/Email/Send` action calls `IsHTML(true)`
   unconditionally, so the message is sent HTML-mode regardless (see `email-send-pipeline.md`).
 
 ## Change history
+- 2026-09-10 — Scoped: this alert fires only for **validation 400s**. A **401/403** is decided in
+  the auth chain before routing, so `prePost` never runs and this email never sends for it — those
+  refused REQs are covered by the separate 15-minute worker cron
+  `prudential_auth_rejection_alert.php`. No code change. (bala)
 - 2026-07-28 — Created: real-time internal alert on `POST /v2/service-requests` validation
   rejection. `prePost()` now collects all validation errors and enqueues the
   `ReqRejectionEmail/sendRejectionEmail` worker task (REQ# + reason + full payload) before

package/knowledge/clients/prudential/profile.md

@@ -15,7 +15,7 @@ project: _Underscore
 client: prudential
 type: profile
 status: active
-updated: 2026-08-28
+updated: 2026-09-10
 owners: ["jcardinal", "rgirish", "bala", "mhammontree"]
 files: []
 related:
@@ -23,10 +23,13 @@ related:
   - features/dell-asn-units-interceptor.md
   - features/service-request-address-validation.md
   - features/service-request-rejection-alert-email.md
+  - features/service-request-auth-rejection-alert.md
   - features/dell-lch-iop-transmissions.md
   - features/device-information-import-and-contact-linking.md
   - workflows/http2-alb-workaround.md
-  - features/supply-orders-csv-export.md
+  - features/service-request-auth-rejection-alert.md — the 15-minute refused-REQ (401/403) alert cron,
+  where each code is logged, and how to tell "we refused it" from "they never sent it".
+- features/supply-orders-csv-export.md
   - features/order-email-pdf-attachments.md
 ---
 
@@ -89,6 +92,13 @@ The order-created cron branches on this id; the shipped/delivered crons branch o
   (E_ALL→ErrorException), so `!== false` guards do not work — use the `safeFetchPdf()` pattern. See
   the exception report and the transmit-ordershipped-email feature doc.
 
+- **Inbound REQs we refuse (401/403) are invisible in `Logs_Prudential`.** A 401 is logged to the
+  **shared base `Logs`** schema because the client is not resolved yet; only a 403 lands in
+  `Logs_Prudential`. The daily `prudential_exception_report.php` reads only 201/400/500 in the
+  client schema, so refused REQs went unseen for a week in Aug 2026 (36 refused, 22 never resent).
+  The 15-minute `prudential_auth_rejection_alert.php` cron now alerts on them — see
+  features/service-request-auth-rejection-alert.md.
+
 ## Monitoring
 - `Monitor/Prudential/{SalesOrderGenerationBacklog,DellPurchaseOrderTransmissionBacklog,OrderShippedUpdateBacklog}`
   (worker2 → OneUptime) watch the order pipeline for stalls. Thresholds are DB-tunable and the

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.794",
+  "version": "1.0.795",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",