toga-ai 1.0.716 → 1.0.717

package/knowledge/2.0/apps/worker2/features/oneuptime-worker2-monitoring.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-28
+updated: 2026-08-31
 owners: ["jcardinal", "mhammontree", "bala"]
 files:
   - worker2/Worker/Monitor/Compass.php
@@ -65,9 +65,24 @@ Each measures one Compass/Office Depot integration metric and pushes a decided a
 - **OfficeDepotEdiImportQueue** (S3) — counts files >10 min old in
   `s3://agilant-as2/OfficeDepot/`, excluding `OUTBOX/`, `SENT/` and directory placeholders.
 - **CompassSalesOrderTransmission** (DB) — approved SOs (both approval stages) with
-  `c_dtTransmittedToCompass` NULL (not yet transmitted to MITS).
+  `c_dtTransmittedToCompass` NULL (not yet transmitted to MITS). Alarms on **either**
+  `pendingCount > 10` **OR** `oldestPendingAgeMinutes > 180` (3h), so a single order stuck too
+  long pages even under the count threshold — this catches the 1.0 cron's
+  [poison-order starvation](../../../clients/compass-usa/features/mits-sales-order-transmission-alerting.md).
+  **Age is measured from the FINAL (stage-2) approval decision** (`ApprovalDecisions` stage 2
+  `dtDecision` where `isApproved = 1` — not stage 1, not order date; that is the "eligible to
+  transmit" clock). The query **excludes canceled orders** (`salesOrderStageId IS NULL OR
+  <> 8`), mirroring the transmit cron's WHERE, so an order canceled after approval (e.g.
+  SA136567, now stage 8) neither counts nor false-alarms forever — this also fixed a
+  pre-existing count mismatch vs. the cron. Config: `$ALARM_AGE_MINUTES = 180`; body echoes
+  `oldestPendingAgeMinutes` + `alarmAgeMinutesThreshold`.
 - **CompassIncomingPurchaseOrders** (DB) — `SA%` SOs transmitted to MITS >4h ago (on/after
-  the 2026-04-30 cutoff) with no PO returned.
+  the 2026-04-30 cutoff) with no PO returned. (This supersedes the 1.0
+  `App_SystemMonitor_Compass` "orders returned from MITS" sub-check, which is a **latent
+  no-op**: it selects `so`/`number`/etc but then reads `$row['pendingCount']`, a column the
+  query never returns, so `$pendingCount` is always null and that sub-check silently always
+  reports OK. Verified 2026-08-31 in `library/app/systemmonitor/compass.php` ~lines 111-140,
+  TRUE-78712.)
 - **CompassPurchaseOrderTransmissionsToVendors** (DB) — unsubmitted POs for monitored
   vendors (Office Depot, Strategic Systems, Compass, Presidio) due today or earlier.
 - **EmailedAsnImport** (mailbox) — counts inbox messages in the emailed-ASN mailbox via
@@ -415,6 +430,18 @@ check for another client.
   monitors must pass the region (see [Cloud S3 helpers](../../_underscore/features/cloud-s3-helpers.md)).
 
 ## Change history
+- 2026-08-31 — Added **age-based detection** to `CompassSalesOrderTransmission()`: it now alarms
+  on `pendingCount > 10` **OR** `oldestPendingAgeMinutes > 180` (new `$ALARM_AGE_MINUTES = 180`),
+  so one order stuck too long pages even under the count threshold — catching the 1.0 transmit
+  cron's poison-order starvation. Age is measured from the **stage-2** (final) approval decision,
+  not stage 1 or order date. The query now also **excludes canceled orders** (`salesOrderStageId
+  IS NULL OR <> 8`), mirroring the transmit cron's WHERE, fixing a pre-existing count mismatch and
+  a canceled order (SA136567) false-alarming forever. New body fields `oldestPendingAgeMinutes` +
+  `alarmAgeMinutesThreshold`; no OneUptime criteria change needed — the age alarm rides the
+  existing `"alarm":"HIGH"` token ("dumb reporter, smart monitor"). Also recorded that the 1.0
+  `App_SystemMonitor_Compass` "orders returned from MITS" sub-check is a **latent no-op** (reads
+  `$row['pendingCount']`, a column its query never selects — always reports OK; TRUE-78712),
+  superseded by `CompassIncomingPurchaseOrders`. worker2-only, php -l passes, not deployed. (jcardinal)
 - 2026-08-28 — Made `EmailedAsnImport()` resilient to a transient IMAP TLS handshake timeout
   that was flipping the monitor Offline and paging even though it self-recovered next run:
   wrapped token+connect+count in an **in-run retry loop** (`MAX_CONNECT_ATTEMPTS = 3`,

package/knowledge/clients/compass-usa/features/mits-sales-order-transmission-alerting.md

@@ -6,7 +6,7 @@ client: compass-usa
 type: client-feature
 status: active
 updated: 2026-08-31
-owners: ["dfranks", "bala"]
+owners: ["dfranks", "bala", "jcardinal"]
 files:
   - worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php
   - worker/crons/toga2/compasscanada/workflow/1_transmit_compassCanada_sales_orders_to_mits.php
@@ -100,6 +100,44 @@ Configuration is **never retroactive**: `shouldNotify()` fires on the first occu
 escalation, or when the backed-off reminder interval elapses, and returns false for a window with
 zero events. The first email lands on the **next live rejection**, not on existing history.
 
+## Why the backlog happens — poison-order starvation (root cause)
+
+The alert almost never fires on **volume**. It fires because one bad order **aborts the whole
+run** and every un-processed order behind it starves. Diagnosed 2026-08-31 against a live
+backlog (0 new, several >24h, one ~20 days old).
+
+- The cron loops approved-but-untransmitted orders and `throw`s on any MITS rejection
+  (`App_Exception_Business` on a non-200, plain `Exception` on "No Items Found"). A throw
+  **ends the entire run**, so every order not yet reached that run stays untransmitted.
+- The driver query uses **`ORDER BY RAND()`**, so *which* orders get processed before the bad
+  one is random each run. A stuck "poison" order therefore blocks a **different** set of good
+  orders every run — good orders can wait days behind one bad one.
+- Throughput ceiling: the cron sleeps **30s per order** (86400s time limit), scheduled every
+  3 min but overlap-guarded, so it is effectively one long-running instance — ~110 orders/hr.
+
+**Two rejection signatures seen in prod** (Logs.Issue id 24, issueKey
+`COMPASS_SALES_ORDER_MITS_TRANSMIT_REJECTED`, ~2329 occurrences since 2026-08-03):
+
+1. **HTTP 400 `The AssetType field is required`** — a line item's catalog Item has
+   `assetTypeId = NULL`, so the payload `assetType` is null and MITS rejects the **whole
+   order**. This was the active blocker (e.g. item `4U555A-S`; ~10 distinct NULL-assetType
+   items blocked 9 of 11 pending orders).
+2. **HTTP 403 `SalesOrder:Exists`** — MITS already created the order on a prior attempt, but
+   our side never recorded `c_dtTransmittedToCompass`, so the cron **retries it forever**.
+
+**Fix the data:** assign the correct `assetTypeId` to the affected `Client_Compass.Items`
+(business owns the mapping), and check whether the NetSuite item sync owns `assetType` — if it
+does, it will revert the fix unless the source is corrected too.
+
+**Recommended code hardening (NOT yet implemented):** wrap each order's transmit in
+`try/catch` so one MITS rejection raises the business issue but **continues to the next
+order** instead of aborting the batch. This removes the starvation entirely.
+
+The worker2 OneUptime monitor now catches this starvation directly: `CompassSalesOrderTransmission`
+alarms on `pendingCount > 10` **OR** `oldestAgeMinutes > 180`, so a single order stuck too long
+pages even when the count is under threshold. See
+[OneUptime push-metric monitors for 2.0 workers](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md).
+
 ## Gotchas / known issues
 
 - **Canada is the pre-2026-08-03 design, and it still carries the two problems that migration
@@ -131,6 +169,14 @@ zero events. The first email lands on the **next live rejection**, not on existi
   when this issue next fires. Watch that run rather than assuming it works.
 
 ## Change history
+- 2026-08-31 — Diagnosed the recurring backlog's **root cause**: not volume but **poison-order
+  starvation**. One MITS rejection `throw`s and aborts the whole run, and because the driver
+  query uses `ORDER BY RAND()`, a stuck order blocks a random set of good orders every run.
+  Recorded the two live rejection signatures (`assetTypeId = NULL` → HTTP 400 "The AssetType
+  field is required"; already-created order → HTTP 403 "SalesOrder:Exists" retried forever),
+  the data fix (assign `assetTypeId`; check NetSuite item sync ownership), and the recommended
+  per-order `try/catch` hardening (not yet implemented). Cross-linked the new worker2 age-based
+  monitor. (jcardinal)
 - 2026-08-31 — Added `vburks@togatech.com` alongside `devteam@togatech.com` on **every** in-cron
   `App_Email_Agilant` alert in both MITS transmit crons (USA has one, Canada has three), so the
   business stakeholder is alerted on transmission failures. Broadened this doc to cover **both**

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.716",
+  "version": "1.0.717",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",