toga-ai 1.0.763 → 1.0.764

package/knowledge/2.0/apps/worker2/features/oneuptime-worker2-monitoring.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-31
+updated: 2026-09-03
 owners: ["jcardinal", "mhammontree", "bala"]
 files:
   - worker2/Worker/Monitor/Compass.php
@@ -69,9 +69,17 @@ Each measures one Compass/Office Depot integration metric and pushes a decided a
   `pendingCount > 10` **OR** `oldestPendingAgeMinutes > 180` (3h), so a single order stuck too
   long pages even under the count threshold — this catches the 1.0 cron's
   [poison-order starvation](../../../clients/compass-usa/features/mits-sales-order-transmission-alerting.md).
-  **Age is measured from the FINAL (stage-2) approval decision** (`ApprovalDecisions` stage 2
-  `dtDecision` where `isApproved = 1` — not stage 1, not order date; that is the "eligible to
-  transmit" clock). The query **excludes canceled orders** (`salesOrderStageId IS NULL OR
+  **Age is measured from the LATER of the two approval decisions —
+  `GREATEST(stage-1, stage-2) dtDecision`** — because an order is eligible to transmit only once
+  **both** stages are approved. Measuring from **stage-2 alone is wrong** (it was the source of a
+  false HIGH — see the 2026-09-03 change-history): Compass **VIP auto-approve stamps the stage-2
+  (manager) decision at order creation** (`postPost`), so stage 1 (admin) can land days later; a
+  stage-2-only clock reads a huge age the instant stage-1 approves on a perfectly healthy order.
+  **NULL-guarded:** an `isApproved = 1` decision can still carry a **NULL `dtDecision`**, and a bare
+  `GREATEST` returns NULL — so `MAX()` would silently drop that order and hide a genuinely stuck one.
+  Each stage is therefore `COALESCE`d to the other stage's decision, then to `SalesOrders.dtCreated`,
+  so the age clock is never NULL and a stuck order always surfaces (fail loud). The query
+  **excludes canceled orders** (`salesOrderStageId IS NULL OR
   <> 8`), mirroring the transmit cron's WHERE, so an order canceled after approval (e.g.
   SA136567, now stage 8) neither counts nor false-alarms forever — this also fixed a
   pre-existing count mismatch vs. the cron. Config: `$ALARM_AGE_MINUTES = 180`; body echoes
@@ -430,6 +438,26 @@ check for another client.
   monitors must pass the region (see [Cloud S3 helpers](../../_underscore/features/cloud-s3-helpers.md)).
 
 ## Change history
+- 2026-09-03 — **Fixed a false HIGH age alarm** in `CompassSalesOrderTransmission()`.
+  `oldestPendingAgeMinutes` was measured from **stage-2 approval only**
+  (`TIMESTAMPDIFF(MINUTE, ApprovalDecisionsStage2.dtDecision, NOW())`), but stage 2 is **not**
+  always the final approval: Compass **VIP auto-approve stamps the stage-2 (manager) decision at
+  order creation** (`postPost`), so stage-1 (admin) can land days later. An order transmits only
+  once **both** stages approve, so the correct clock is the **later of the two** decisions.
+  **Incident 2026-09-03:** OneUptime HIGH, `pendingCount=1`, `oldestPendingAgeMinutes=2360`
+  (threshold 180) — order SA136844 (`Client_Compass` id 120481): stage-2 VIP auto-approved at
+  creation 09-01 19:55, stage-1 admin approved 09-03 11:15, transmitted 09-03 11:18 (3 min after
+  becoming eligible). The alarm fired the exact minute stage-1 approved — a **monitor** bug, the
+  `dtDecision` values were correct. Scale (prod, last 30 days both-approved = 1927): 439 had
+  stage-2 before stage-1, **133 with that gap > 180 min** (each a false HIGH if pending at run
+  time). **Fix:** age now `GREATEST(stage-1, stage-2) dtDecision`, and because an `isApproved=1`
+  row can carry a **NULL `dtDecision`** (1,415 such rows in prod) where a bare `GREATEST` returns
+  NULL and `MAX()` would silently drop a stuck order, each stage is `COALESCE`d to the other
+  stage's decision then to `SalesOrders.dtCreated` — the clock is never NULL, a stuck order always
+  surfaces (fail loud). Summary string changed from "since stage-2 approval" to "since eligible
+  (later of both approvals)". Timezone confirmed **not** an issue (DB `NOW()` and `dtDecision` both
+  Central). No OneUptime/SQL/config change — the age alarm still rides the existing `"alarm":"HIGH"`
+  token. sql-reviewer SAFE TO MERGE, php -l clean; worker2-only, not deployed. (jcardinal)
 - 2026-08-31 — Added **age-based detection** to `CompassSalesOrderTransmission()`: it now alarms
   on `pendingCount > 10` **OR** `oldestPendingAgeMinutes > 180` (new `$ALARM_AGE_MINUTES = 180`),
   so one order stuck too long pages even under the count threshold — catching the 1.0 transmit

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.763",
+  "version": "1.0.764",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",