toga-ai 1.0.715 → 1.0.717

package/knowledge/1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-30
+updated: 2026-08-31
 owners: ["dfranks", "bala", "jcardinal", "mhammontree", "snaredla"]
 files:
   - worker/crons/toga2/netsuite/common_sync_togasupply.php
@@ -732,6 +732,35 @@ Parameters are stored **per client DB** but accessed **through the TOGa2 API**,
   line). **Fix (`library`):** skip lines whose `adjustQtyBy` is null in **BOTH** passes over
   `inventoryList->inventory` (the item-build pass and the serial/unit pass). Shared fix; unblocked NYCHH
   inventory adjustments.
+- **⚠ Item-receipt sync froze on a PARTIAL receipt's NULL-quantity PO lines — the item-receipt TWIN of the
+  inventory-adjustment NULL-quantity fix above (fixed 2026-08-31).** For a partial item receipt NetSuite
+  returns **all** the PO's linked lines but sets `quantity=null` on the lines **not received on this
+  receipt** (SuiteQL `transactionline.quantity` is NULL for an un-received PO line). The REST shim
+  `App_Api_Netsuite_Rest::fetchItemReceiptById` (`rest.php` ~L2196) faithfully carries that null, and
+  `syncItemReceiptFromNetsuite` wrote those null lines into the `POST /v2/item-receipts` payload →
+  `ItemReceiptItems.quantity` is `NOT NULL` → api2 `EV-10` / MySQL **1048** "Column 'quantity' cannot be
+  null", the section threw (no per-record try/catch — fail-loud by design), and froze at `1-RUNNING`
+  re-trying the same poison receipt every 5 min. Seen on **NYCHH** ITEM_RECEIPTS: stuck on receipt **#22302**
+  (NetSuite internalId 3425518, dated 2023-05-01; 5 PO lines, 2 received qty 260, 3 returned null), watermark
+  frozen at 2023-05-01, throwing every run (`Logs.Issue 578`, "Invalid API Response" from
+  `send()` `toga2.php:3517`). **Fix (`library`):** skip line items whose `quantity` **`is_null`** (NOT
+  `empty`, so a legitimate received qty of 0 still writes) in **BOTH** passes over
+  `$nsItemReceipt->itemList->item` — the payload-build loop (~L3376) and the unit/serial pass (~L3538).
+  Shared fix; no signature change, no SQL. Recovery is automatic: the watermark sits at the failing window's
+  start, so on deploy the section self-heals (RUNNING→IDLE, watermark advances past #22302, window regrows).
+  Diagnosis chain (reinforces the playbook above): `Client_Nychh.Parameters` to localize the stuck section
+  (`1-RUNNING` + frozen watermark) → `Logs.Event` filtered by `clientId=3` for the crash file:line +
+  `transactionId` (`Logs.Event` carries `clientId`, `Logs.Issue` does not) → `Logs_Nychh.Api` on that
+  `transactionId` for the untruncated `EV-10`/1048 rejection and the payload showing the `quantity:null`
+  lines.
+- **⚠ OPEN, not yet root-caused (found 2026-08-31): NYCHH ITEM_FULFILLMENTS watermark frozen at 2021-12-29.**
+  `NETSUITE_EXECUTION_MODE_ITEM_FULFILLMENTS='3-IDLE'` with the window collapsed to ~3s (it has been
+  throwing recently) but currently **IDLE** — so it crawls a few seconds per run rather than sitting in a
+  hard `1-RUNNING` freeze. Distinct from the fixed ITEM_RECEIPTS freeze above; needs its own investigation.
+  Context: the 2026-08-31 full 19-client togasupply prod sweep found all other clients healthy
+  (`864000-IDLE`) except Elite SALES_ORDERS `864000-RUNNING` (healthy mid-run, large window, not stuck) and
+  AIG SALES_ORDERS `8-RUNNING` (the known held account, already documented). Only NYCHH ITEM_RECEIPTS was in
+  a true `1-RUNNING` freeze.
 - **⚠ Item-fulfillment stale-item prune must delete tracking-number children before the unit AND the item
   (fixed 2026-08-30).** In `syncItemFulfillmentFromNetsuite`'s cleanup loop, deleting a stale
   item-fulfillment-item-**unit** failed api2 `EV-11` / MySQL **1451** (FK
@@ -792,6 +821,16 @@ Parameters are stored **per client DB** but accessed **through the TOGa2 API**,
 
 ## Change history
 
+- 2026-08-31 — **Item-receipt partial-receipt NULL-quantity fix (shared, `library`).**
+  `syncItemReceiptFromNetsuite` now **skips line items whose `quantity` is null** (unreceived PO lines on a
+  partial receipt) in both passes over `$nsItemReceipt->itemList->item` (payload-build ~L3376 + unit/serial
+  ~L3538), guarded on `is_null` so a received qty of 0 still writes. This is the item-receipt twin of the
+  2026-08-30 inventory-adjustment NULL-quantity skip — the receipt path never got that guard. It unfroze
+  NYCHH ITEM_RECEIPTS (stuck `1-RUNNING` on receipt #22302 / NetSuite 3425518 / 2023-05-01 → `EV-10`/1048
+  "quantity cannot be null", `Logs.Issue 578`); self-heals on deploy. Also flagged an OPEN item: NYCHH
+  ITEM_FULFILLMENTS watermark frozen at 2021-12-29 (`3-IDLE`, window collapsed to ~3s), not yet root-caused.
+  19-client sweep otherwise healthy. `library` only, no SQL, `php -l` clean; not deployed at capture time.
+  (jcardinal)
 - 2026-08-31 — **Invoice + serial-unit + duplicate-Item root-cause fixes (shared, `library`).**
   `syncInvoiceFromNetsuite` now assigns a **sequential 1..N invoice `lineNumber`** (NetSuite reuses `line`
   across multi-SO invoices → `InvoiceItems UNIQUE(invoiceId,lineNumber)` 1062 → atomic rollback → NYCHH

package/knowledge/2.0/apps/worker2/features/oneuptime-worker2-monitoring.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-28
+updated: 2026-08-31
 owners: ["jcardinal", "mhammontree", "bala"]
 files:
   - worker2/Worker/Monitor/Compass.php
@@ -65,9 +65,24 @@ Each measures one Compass/Office Depot integration metric and pushes a decided a
 - **OfficeDepotEdiImportQueue** (S3) — counts files >10 min old in
   `s3://agilant-as2/OfficeDepot/`, excluding `OUTBOX/`, `SENT/` and directory placeholders.
 - **CompassSalesOrderTransmission** (DB) — approved SOs (both approval stages) with
-  `c_dtTransmittedToCompass` NULL (not yet transmitted to MITS).
+  `c_dtTransmittedToCompass` NULL (not yet transmitted to MITS). Alarms on **either**
+  `pendingCount > 10` **OR** `oldestPendingAgeMinutes > 180` (3h), so a single order stuck too
+  long pages even under the count threshold — this catches the 1.0 cron's
+  [poison-order starvation](../../../clients/compass-usa/features/mits-sales-order-transmission-alerting.md).
+  **Age is measured from the FINAL (stage-2) approval decision** (`ApprovalDecisions` stage 2
+  `dtDecision` where `isApproved = 1` — not stage 1, not order date; that is the "eligible to
+  transmit" clock). The query **excludes canceled orders** (`salesOrderStageId IS NULL OR
+  <> 8`), mirroring the transmit cron's WHERE, so an order canceled after approval (e.g.
+  SA136567, now stage 8) neither counts nor false-alarms forever — this also fixed a
+  pre-existing count mismatch vs. the cron. Config: `$ALARM_AGE_MINUTES = 180`; body echoes
+  `oldestPendingAgeMinutes` + `alarmAgeMinutesThreshold`.
 - **CompassIncomingPurchaseOrders** (DB) — `SA%` SOs transmitted to MITS >4h ago (on/after
-  the 2026-04-30 cutoff) with no PO returned.
+  the 2026-04-30 cutoff) with no PO returned. (This supersedes the 1.0
+  `App_SystemMonitor_Compass` "orders returned from MITS" sub-check, which is a **latent
+  no-op**: it selects `so`/`number`/etc but then reads `$row['pendingCount']`, a column the
+  query never returns, so `$pendingCount` is always null and that sub-check silently always
+  reports OK. Verified 2026-08-31 in `library/app/systemmonitor/compass.php` ~lines 111-140,
+  TRUE-78712.)
 - **CompassPurchaseOrderTransmissionsToVendors** (DB) — unsubmitted POs for monitored
   vendors (Office Depot, Strategic Systems, Compass, Presidio) due today or earlier.
 - **EmailedAsnImport** (mailbox) — counts inbox messages in the emailed-ASN mailbox via
@@ -415,6 +430,18 @@ check for another client.
   monitors must pass the region (see [Cloud S3 helpers](../../_underscore/features/cloud-s3-helpers.md)).
 
 ## Change history
+- 2026-08-31 — Added **age-based detection** to `CompassSalesOrderTransmission()`: it now alarms
+  on `pendingCount > 10` **OR** `oldestPendingAgeMinutes > 180` (new `$ALARM_AGE_MINUTES = 180`),
+  so one order stuck too long pages even under the count threshold — catching the 1.0 transmit
+  cron's poison-order starvation. Age is measured from the **stage-2** (final) approval decision,
+  not stage 1 or order date. The query now also **excludes canceled orders** (`salesOrderStageId
+  IS NULL OR <> 8`), mirroring the transmit cron's WHERE, fixing a pre-existing count mismatch and
+  a canceled order (SA136567) false-alarming forever. New body fields `oldestPendingAgeMinutes` +
+  `alarmAgeMinutesThreshold`; no OneUptime criteria change needed — the age alarm rides the
+  existing `"alarm":"HIGH"` token ("dumb reporter, smart monitor"). Also recorded that the 1.0
+  `App_SystemMonitor_Compass` "orders returned from MITS" sub-check is a **latent no-op** (reads
+  `$row['pendingCount']`, a column its query never selects — always reports OK; TRUE-78712),
+  superseded by `CompassIncomingPurchaseOrders`. worker2-only, php -l passes, not deployed. (jcardinal)
 - 2026-08-28 — Made `EmailedAsnImport()` resilient to a transient IMAP TLS handshake timeout
   that was flipping the monitor Offline and paging even though it self-recovered next run:
   wrapped token+connect+count in an **in-run retry loop** (`MAX_CONNECT_ATTEMPTS = 3`,

package/knowledge/clients/compass-usa/features/mits-sales-order-transmission-alerting.md

@@ -6,7 +6,7 @@ client: compass-usa
 type: client-feature
 status: active
 updated: 2026-08-31
-owners: ["dfranks", "bala"]
+owners: ["dfranks", "bala", "jcardinal"]
 files:
   - worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php
   - worker/crons/toga2/compasscanada/workflow/1_transmit_compassCanada_sales_orders_to_mits.php
@@ -100,6 +100,44 @@ Configuration is **never retroactive**: `shouldNotify()` fires on the first occu
 escalation, or when the backed-off reminder interval elapses, and returns false for a window with
 zero events. The first email lands on the **next live rejection**, not on existing history.
 
+## Why the backlog happens — poison-order starvation (root cause)
+
+The alert almost never fires on **volume**. It fires because one bad order **aborts the whole
+run** and every un-processed order behind it starves. Diagnosed 2026-08-31 against a live
+backlog (0 new, several >24h, one ~20 days old).
+
+- The cron loops approved-but-untransmitted orders and `throw`s on any MITS rejection
+  (`App_Exception_Business` on a non-200, plain `Exception` on "No Items Found"). A throw
+  **ends the entire run**, so every order not yet reached that run stays untransmitted.
+- The driver query uses **`ORDER BY RAND()`**, so *which* orders get processed before the bad
+  one is random each run. A stuck "poison" order therefore blocks a **different** set of good
+  orders every run — good orders can wait days behind one bad one.
+- Throughput ceiling: the cron sleeps **30s per order** (86400s time limit), scheduled every
+  3 min but overlap-guarded, so it is effectively one long-running instance — ~110 orders/hr.
+
+**Two rejection signatures seen in prod** (Logs.Issue id 24, issueKey
+`COMPASS_SALES_ORDER_MITS_TRANSMIT_REJECTED`, ~2329 occurrences since 2026-08-03):
+
+1. **HTTP 400 `The AssetType field is required`** — a line item's catalog Item has
+   `assetTypeId = NULL`, so the payload `assetType` is null and MITS rejects the **whole
+   order**. This was the active blocker (e.g. item `4U555A-S`; ~10 distinct NULL-assetType
+   items blocked 9 of 11 pending orders).
+2. **HTTP 403 `SalesOrder:Exists`** — MITS already created the order on a prior attempt, but
+   our side never recorded `c_dtTransmittedToCompass`, so the cron **retries it forever**.
+
+**Fix the data:** assign the correct `assetTypeId` to the affected `Client_Compass.Items`
+(business owns the mapping), and check whether the NetSuite item sync owns `assetType` — if it
+does, it will revert the fix unless the source is corrected too.
+
+**Recommended code hardening (NOT yet implemented):** wrap each order's transmit in
+`try/catch` so one MITS rejection raises the business issue but **continues to the next
+order** instead of aborting the batch. This removes the starvation entirely.
+
+The worker2 OneUptime monitor now catches this starvation directly: `CompassSalesOrderTransmission`
+alarms on `pendingCount > 10` **OR** `oldestAgeMinutes > 180`, so a single order stuck too long
+pages even when the count is under threshold. See
+[OneUptime push-metric monitors for 2.0 workers](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md).
+
 ## Gotchas / known issues
 
 - **Canada is the pre-2026-08-03 design, and it still carries the two problems that migration
@@ -131,6 +169,14 @@ zero events. The first email lands on the **next live rejection**, not on existi
   when this issue next fires. Watch that run rather than assuming it works.
 
 ## Change history
+- 2026-08-31 — Diagnosed the recurring backlog's **root cause**: not volume but **poison-order
+  starvation**. One MITS rejection `throw`s and aborts the whole run, and because the driver
+  query uses `ORDER BY RAND()`, a stuck order blocks a random set of good orders every run.
+  Recorded the two live rejection signatures (`assetTypeId = NULL` → HTTP 400 "The AssetType
+  field is required"; already-created order → HTTP 403 "SalesOrder:Exists" retried forever),
+  the data fix (assign `assetTypeId`; check NetSuite item sync ownership), and the recommended
+  per-order `try/catch` hardening (not yet implemented). Cross-linked the new worker2 age-based
+  monitor. (jcardinal)
 - 2026-08-31 — Added `vburks@togatech.com` alongside `devteam@togatech.com` on **every** in-cron
   `App_Email_Agilant` alert in both MITS transmit crons (USA has one, Canada has three), so the
   business stakeholder is alerted on transmission failures. Broadened this doc to cover **both**

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.715",
+  "version": "1.0.717",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",