toga-ai 1.0.716 → 1.0.718

package/knowledge/2.0/apps/api2/architecture.md

@@ -6,7 +6,7 @@ project: API
 client: shared
 type: architecture
 status: active
-updated: 2026-08-28
+updated: 2026-09-01
 owners: [jcardinal, bala, mhammontree, dfranks]
 files:
   - api2/Controller/Index.php
@@ -315,6 +315,52 @@ worker2 EB health monitor now applies a **persistence-delay gate** (a non-5xx `D
 must persist ≥ `nonFivexxPersistMinutes`, default 5, before it pages), which suppresses these flaps.
 See [worker2 EB health monitor](../worker2/features/elastic-beanstalk-health-monitor.md).
 
+**A sixth cause — a 4xx BURST flips the env to `Degraded`/`Severe` while the ALB stays fully
+healthy, because EB Enhanced Health counts application/ELB 4xx toward environment health.**
+Distinct from every cause above: it is not a deploy (cause #4), not worker starvation (cause #3),
+and not idle flappiness (cause #5) — it is EB reacting to *client* 4xx responses that are working
+as designed. EB Enhanced Health's `ConfigDocument` (namespace
+`aws:elasticbeanstalk:healthreporting:system`) ships with two rules **Enabled by default**:
+`Rules.Environment.Application.ApplicationRequests4xx` and `Rules.Environment.ELB.ELBRequests4xx`.
+So a short burst of client 4xx (e.g. a client polling with an expired bearer token) flips the env
+to `Degraded`/`Severe` even though the ALB never marked a target unhealthy. Verified live (AWS CLI
+`describe-events`/`describe-configuration-settings` + CloudWatch + prod `Logs.Api`, acct
+`654654170868`):
+- `api-production-1` (us-east-1) flip 2026-09-01 01:39–01:44 UTC, no cause cited ("TargetGroups
+  reduced health state"), **not** during a deploy. Throughout: `HealthyHostCount=2`,
+  `UnHealthyHostCount=0`, **0** target 5xx, **0** ELB 5xx, **0** connection errors — the ALB never
+  deregistered a target.
+- Trigger burst ~5 min earlier at 01:34 UTC: `RequestCount` 738/min, ~560 of them 4xx (~77%),
+  `TargetResponseTime` max 5.2 s. Prod `Logs.Api` for the window: **398 × HTTP 401 on
+  `GET /v2/users/me`** from `sourceIp 34.232.23.158` (the Compass/Veyer feed) on `api1.togahub.com`,
+  split across both instances (`i-0fa76fc1365cdf583`, `i-068e89c284eb26133`) — a client polling with
+  an expired/invalid bearer token.
+- `api-production-3` (eu-west-1) 2026-09-01 01:06 UTC stated it plainly: "Environment health
+  transitioned Ok→Severe: 80.7% of requests erroring with HTTP 4xx" — same root cause.
+- Other causes ruled out: `/health` confirmed served as a **static Apache file** (response carries
+  `Last-Modified`/`ETag`/`Accept-Ranges`, body `OK`, `Content-Length 3`, no PHP) → cause #3 not in
+  play; `DeploymentPolicy=Rolling`, `BatchSize=1` Fixed confirmed → cause #4 not in play. This is
+  why the prior four fixes (culminating in Rolling/batch-1, which only addressed cause #4) never
+  stopped these flips — they are unrelated to deploys.
+
+**Rule:** an EB "% 4xx" flag, or a no-cause "TargetGroup reduced health" flip, where
+`HealthyHostCount` never drops and target/ELB **5xx = 0**, is EB Enhanced Health reacting to a 4xx
+burst — **not** a backend failure. Check the ALB 4xx metric and `Logs.Api` for a client 4xx flood
+before chasing app bugs.
+
+**Fix shipped 2026-09-01 (config-only, all three prod envs).** Both 4xx rules were **disabled** on
+`api-production-1/2/3` by updating the `ConfigDocument` option (namespace
+`aws:elasticbeanstalk:healthreporting:system`) via `aws elasticbeanstalk update-environment
+--option-settings`, setting `ApplicationRequests4xx.Enabled=false` and
+`ELBRequests4xx.Enabled=false` while preserving every other rule and the `CloudWatchMetrics` block
+(**5xx and latency rules stay ON, so real backend failures still page**). It is a config-only update
+(~1–2 min "Updating", no instance replacement) and is **reversible** by flipping the flags back to
+`true`. Verified after apply: all three Ready/Green/Ok with both 4xx rules `Enabled=false`. This is
+the **server-side complement** to the worker2 monitor's non-5xx persistence-delay gate
+(`nonFivexxPersistMinutes`, cause #5) — one de-sensitizes the alert, this stops EB from generating
+the flip at all. (The dev ran the three `update-environment` commands; the harness auto-mode
+classifier correctly blocks the assistant from mutating prod infra or self-granting that permission.)
+
 **Triage aid — break an EB "% requests failing with HTTP 5xx" flag into ELB 502/503/504 to find
 the cause:** **503** = no healthy backend (a deploy with no capacity); **504** = target timeout
 (worker saturation); **502** = connection dropped mid-response (an `AllAtOnce` deploy killing
@@ -400,6 +446,20 @@ they are the known sharp edges. Do not re-discover these from scratch.
   single-caller branches in `V2.php` as unverified until exercised directly.
 
 ## Change history
+- 2026-09-01 — Recorded a **sixth EB-`Degraded` cause — a client 4xx BURST**: EB Enhanced Health
+  counts application/ELB 4xx toward env health via two default-Enabled `ConfigDocument` rules
+  (`ApplicationRequests4xx`, `ELBRequests4xx`, namespace
+  `aws:elasticbeanstalk:healthreporting:system`), so a short 4xx spike flips the env to
+  `Degraded`/`Severe` while the ALB stays fully healthy (`HealthyHostCount`=2, 0 target/ELB 5xx).
+  Verified on `api-production-1` (2026-09-01 01:39 UTC flip, not a deploy) from a 01:34 UTC burst of
+  398×HTTP 401 on `GET /v2/users/me` from the Compass/Veyer feed (`34.232.23.158`) polling with an
+  expired token; `api-production-3` stated it as "80.7% of requests erroring with HTTP 4xx". Ruled
+  out causes #3 (static-file `/health` confirmed) and #4 (Rolling/batch-1 confirmed) — which is why
+  the prior deploy-focused fixes never stopped it. Triage rule: no-cause/%-4xx flip + steady
+  `HealthyHostCount` + 0 5xx = a 4xx burst, check the ALB 4xx metric and `Logs.Api` first. Fixed
+  config-only on all three prod envs — disabled both 4xx rules via `update-environment
+  --option-settings` (5xx/latency rules kept ON), reversible; the server-side complement to the
+  worker2 monitor's non-5xx persistence-delay gate. (jcardinal)
 - 2026-08-29 — Recorded a **fifth EB-`Degraded` cause**: a brief self-healing "no-cause reduced
   health" flip on a low-traffic *idle* 2-instance env (e.g. `api-production-3`, eu-west-1) is EB
   Enhanced Health flappiness (a single missed ALB `/health` check), distinct from the busy-env

package/knowledge/2.0/apps/worker2/features/oneuptime-worker2-monitoring.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-28
+updated: 2026-08-31
 owners: ["jcardinal", "mhammontree", "bala"]
 files:
   - worker2/Worker/Monitor/Compass.php
@@ -65,9 +65,24 @@ Each measures one Compass/Office Depot integration metric and pushes a decided a
 - **OfficeDepotEdiImportQueue** (S3) — counts files >10 min old in
   `s3://agilant-as2/OfficeDepot/`, excluding `OUTBOX/`, `SENT/` and directory placeholders.
 - **CompassSalesOrderTransmission** (DB) — approved SOs (both approval stages) with
-  `c_dtTransmittedToCompass` NULL (not yet transmitted to MITS).
+  `c_dtTransmittedToCompass` NULL (not yet transmitted to MITS). Alarms on **either**
+  `pendingCount > 10` **OR** `oldestPendingAgeMinutes > 180` (3h), so a single order stuck too
+  long pages even under the count threshold — this catches the 1.0 cron's
+  [poison-order starvation](../../../clients/compass-usa/features/mits-sales-order-transmission-alerting.md).
+  **Age is measured from the FINAL (stage-2) approval decision** (`ApprovalDecisions` stage 2
+  `dtDecision` where `isApproved = 1` — not stage 1, not order date; that is the "eligible to
+  transmit" clock). The query **excludes canceled orders** (`salesOrderStageId IS NULL OR
+  <> 8`), mirroring the transmit cron's WHERE, so an order canceled after approval (e.g.
+  SA136567, now stage 8) neither counts nor false-alarms forever — this also fixed a
+  pre-existing count mismatch vs. the cron. Config: `$ALARM_AGE_MINUTES = 180`; body echoes
+  `oldestPendingAgeMinutes` + `alarmAgeMinutesThreshold`.
 - **CompassIncomingPurchaseOrders** (DB) — `SA%` SOs transmitted to MITS >4h ago (on/after
-  the 2026-04-30 cutoff) with no PO returned.
+  the 2026-04-30 cutoff) with no PO returned. (This supersedes the 1.0
+  `App_SystemMonitor_Compass` "orders returned from MITS" sub-check, which is a **latent
+  no-op**: it selects `so`/`number`/etc but then reads `$row['pendingCount']`, a column the
+  query never returns, so `$pendingCount` is always null and that sub-check silently always
+  reports OK. Verified 2026-08-31 in `library/app/systemmonitor/compass.php` ~lines 111-140,
+  TRUE-78712.)
 - **CompassPurchaseOrderTransmissionsToVendors** (DB) — unsubmitted POs for monitored
   vendors (Office Depot, Strategic Systems, Compass, Presidio) due today or earlier.
 - **EmailedAsnImport** (mailbox) — counts inbox messages in the emailed-ASN mailbox via
@@ -415,6 +430,18 @@ check for another client.
   monitors must pass the region (see [Cloud S3 helpers](../../_underscore/features/cloud-s3-helpers.md)).
 
 ## Change history
+- 2026-08-31 — Added **age-based detection** to `CompassSalesOrderTransmission()`: it now alarms
+  on `pendingCount > 10` **OR** `oldestPendingAgeMinutes > 180` (new `$ALARM_AGE_MINUTES = 180`),
+  so one order stuck too long pages even under the count threshold — catching the 1.0 transmit
+  cron's poison-order starvation. Age is measured from the **stage-2** (final) approval decision,
+  not stage 1 or order date. The query now also **excludes canceled orders** (`salesOrderStageId
+  IS NULL OR <> 8`), mirroring the transmit cron's WHERE, fixing a pre-existing count mismatch and
+  a canceled order (SA136567) false-alarming forever. New body fields `oldestPendingAgeMinutes` +
+  `alarmAgeMinutesThreshold`; no OneUptime criteria change needed — the age alarm rides the
+  existing `"alarm":"HIGH"` token ("dumb reporter, smart monitor"). Also recorded that the 1.0
+  `App_SystemMonitor_Compass` "orders returned from MITS" sub-check is a **latent no-op** (reads
+  `$row['pendingCount']`, a column its query never selects — always reports OK; TRUE-78712),
+  superseded by `CompassIncomingPurchaseOrders`. worker2-only, php -l passes, not deployed. (jcardinal)
 - 2026-08-28 — Made `EmailedAsnImport()` resilient to a transient IMAP TLS handshake timeout
   that was flipping the monitor Offline and paging even though it self-recovered next run:
   wrapped token+connect+count in an **in-run retry loop** (`MAX_CONNECT_ATTEMPTS = 3`,

package/knowledge/clients/compass-usa/features/mits-sales-order-transmission-alerting.md

@@ -6,7 +6,7 @@ client: compass-usa
 type: client-feature
 status: active
 updated: 2026-08-31
-owners: ["dfranks", "bala"]
+owners: ["dfranks", "bala", "jcardinal"]
 files:
   - worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php
   - worker/crons/toga2/compasscanada/workflow/1_transmit_compassCanada_sales_orders_to_mits.php
@@ -100,6 +100,44 @@ Configuration is **never retroactive**: `shouldNotify()` fires on the first occu
 escalation, or when the backed-off reminder interval elapses, and returns false for a window with
 zero events. The first email lands on the **next live rejection**, not on existing history.
 
+## Why the backlog happens — poison-order starvation (root cause)
+
+The alert almost never fires on **volume**. It fires because one bad order **aborts the whole
+run** and every un-processed order behind it starves. Diagnosed 2026-08-31 against a live
+backlog (0 new, several >24h, one ~20 days old).
+
+- The cron loops approved-but-untransmitted orders and `throw`s on any MITS rejection
+  (`App_Exception_Business` on a non-200, plain `Exception` on "No Items Found"). A throw
+  **ends the entire run**, so every order not yet reached that run stays untransmitted.
+- The driver query uses **`ORDER BY RAND()`**, so *which* orders get processed before the bad
+  one is random each run. A stuck "poison" order therefore blocks a **different** set of good
+  orders every run — good orders can wait days behind one bad one.
+- Throughput ceiling: the cron sleeps **30s per order** (86400s time limit), scheduled every
+  3 min but overlap-guarded, so it is effectively one long-running instance — ~110 orders/hr.
+
+**Two rejection signatures seen in prod** (Logs.Issue id 24, issueKey
+`COMPASS_SALES_ORDER_MITS_TRANSMIT_REJECTED`, ~2329 occurrences since 2026-08-03):
+
+1. **HTTP 400 `The AssetType field is required`** — a line item's catalog Item has
+   `assetTypeId = NULL`, so the payload `assetType` is null and MITS rejects the **whole
+   order**. This was the active blocker (e.g. item `4U555A-S`; ~10 distinct NULL-assetType
+   items blocked 9 of 11 pending orders).
+2. **HTTP 403 `SalesOrder:Exists`** — MITS already created the order on a prior attempt, but
+   our side never recorded `c_dtTransmittedToCompass`, so the cron **retries it forever**.
+
+**Fix the data:** assign the correct `assetTypeId` to the affected `Client_Compass.Items`
+(business owns the mapping), and check whether the NetSuite item sync owns `assetType` — if it
+does, it will revert the fix unless the source is corrected too.
+
+**Recommended code hardening (NOT yet implemented):** wrap each order's transmit in
+`try/catch` so one MITS rejection raises the business issue but **continues to the next
+order** instead of aborting the batch. This removes the starvation entirely.
+
+The worker2 OneUptime monitor now catches this starvation directly: `CompassSalesOrderTransmission`
+alarms on `pendingCount > 10` **OR** `oldestAgeMinutes > 180`, so a single order stuck too long
+pages even when the count is under threshold. See
+[OneUptime push-metric monitors for 2.0 workers](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md).
+
 ## Gotchas / known issues
 
 - **Canada is the pre-2026-08-03 design, and it still carries the two problems that migration
@@ -131,6 +169,14 @@ zero events. The first email lands on the **next live rejection**, not on existi
   when this issue next fires. Watch that run rather than assuming it works.
 
 ## Change history
+- 2026-08-31 — Diagnosed the recurring backlog's **root cause**: not volume but **poison-order
+  starvation**. One MITS rejection `throw`s and aborts the whole run, and because the driver
+  query uses `ORDER BY RAND()`, a stuck order blocks a random set of good orders every run.
+  Recorded the two live rejection signatures (`assetTypeId = NULL` → HTTP 400 "The AssetType
+  field is required"; already-created order → HTTP 403 "SalesOrder:Exists" retried forever),
+  the data fix (assign `assetTypeId`; check NetSuite item sync ownership), and the recommended
+  per-order `try/catch` hardening (not yet implemented). Cross-linked the new worker2 age-based
+  monitor. (jcardinal)
 - 2026-08-31 — Added `vburks@togatech.com` alongside `devteam@togatech.com` on **every** in-cron
   `App_Email_Agilant` alert in both MITS transmit crons (USA has one, Canada has three), so the
   business stakeholder is alerted on transmission failures. Broadened this doc to cover **both**

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.716",
+  "version": "1.0.718",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",