toga-ai 1.0.300 → 1.0.301

package/knowledge/1.0/apps/worker/INDEX.md

@@ -6,6 +6,6 @@
 | [Compass MA Sales Order Exception Report](features/compass-ma-sales-order-exception-report.md) | A worker cron that emails operations the "Compass Refresh Exception Report" — Compass `MA%` sales orders whose corresponding Office Depot (ODP) sales order has | worker/crons/toga2/compass/workflow/7_generate_ma_sales_order_exception_report.php |
 | [Compass Partial In-Transit & Delivered Emails (per package)](features/compass-partial-in-transit-delivered-emails.md) | Compass USA and Compass Canada send a **per-package** in-transit email (and a matching delivered email) instead of one email listing the whole order. | worker/crons/toga2/compass/update_salesorder_status_from_odp.php, worker/crons/toga2/compasscanada/workflow/3_update_salesorder_status_from_grand_and_toy.php |
 | [Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling](features/forecast2-netsuite-reconciliation.md) | CLI tools to **audit** and **repair** drift between the production `Forecast` DB (core2) and NetSuite. | test/@dave/checker.php, _underscore/Component/Forecast/SaleImport/SaleImport.php, test/@dave/looper.php, test/@dave/reconcile_netsuite_totals.php, test/@dave/fixer.php, test/@dave/analyze_netsuite_forecast_diff.php, test/@dave/trueup_sales.php, test/@dave/trueup_open_orders.php, test/@dave/loop_trueup_open_orders.php, test/@dave/trueup_opportunities.php, test/@dave/probe_sales_gap_direct.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_profit_invoices.php, test/@dave/probe_profit_gap.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_open_orders.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/schedules/cron.worker.infrastructure.json |
-| [NetSuite → TOGa Supply Per-Client Sync (thin wrappers)](features/netsuite-togasupply-per-client-sync.md) | Syncs NetSuite transactions (sales orders, purchase orders, invoices, item receipts, item fulfillments, inventory adjustments) into each TOGa Supply (2.0) clien | worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_canon.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql |
+| [NetSuite → TOGa Supply Per-Client Sync (thin wrappers)](features/netsuite-togasupply-per-client-sync.md) | Syncs NetSuite transactions (sales orders, purchase orders, invoices, item receipts, item fulfillments, inventory adjustments) into each TOGa Supply (2.0) clien | worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_canon.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql, library/app/api/netsuite/rest.php, library/app/systemmonitor/netsuiteintegration.php |
 | [Prudential: Send Shipments for the Day report (daily cron)](features/send-shipments-for-the-day.md) | Daily cron (9:00 PM) that emails Prudential and Dell stakeholders an Excel report of all devices shipped that day, including tracking number, serial number, emp | worker/crons/notifications/reports/send_shipments_for_the_day.php |
 | [Onboarding a Client to the NetSuite TOGa Supply Sync](workflows/onboarding-client-to-netsuite-togasupply-sync.md) | How to add a new TOGa 2 client to the per-client NetSuite → TOGa Supply importer (`worker/crons/toga2/netsuite/`). | worker/crons/toga2/netsuite/sync_togasupply.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql |

package/knowledge/1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md

@@ -6,13 +6,15 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-06-30
+updated: 2026-07-09
 owners: ["dfranks", "bala"]
 files:
   - worker/crons/toga2/netsuite/common_sync_togasupply.php
   - worker/crons/toga2/netsuite/sync_togasupply_canon.php
   - worker/schedules/cron.worker.sync.json
   - dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql
+  - library/app/api/netsuite/rest.php
+  - library/app/systemmonitor/netsuiteintegration.php
 related:
   - ../architecture.md
   - forecast2-netsuite-reconciliation.md
@@ -128,9 +130,34 @@ Parameters are stored **per client DB** but accessed **through the TOGa2 API**,
   older** integration (`syncWithToga`/`syncWithTogadesk`) with its own legacy parameter keys
   (`NETSUITE_LAST_*_INTEGRATION_DATETIME`) — NOT the supply sync. The supply wrapper for AIG
   lives in the `netsuite/` folder. Both key sets coexist harmlessly.
-- **Invoices lag ~24h by design:** the invoice section caps its upper bound at `now − 86400s`
-  (invoices may still be attaching), so `LAST_SYNC_DATETIME_INVOICES` always trails the others
-  by a day. Not a fault.
+- **Invoices lag ~24h by design + are keyed on `lastmodifieddate`:** the invoice section caps its
+  upper bound at `now − 86400s` (invoices may still be attaching), so `LAST_SYNC_DATETIME_INVOICES`
+  always trails the others by a day. Not a fault. Invoices are also pulled by
+  **`lastmodifieddate`, NOT `trandate`** — `App_Api_Netsuite_Rest::listSales()` defaults
+  `$dateField='lastmodifieddate'` and the engine calls it without the 4th arg. So a backdated
+  `trandate` is irrelevant to whether an invoice imports, and a just-modified invoice still won't
+  import until it is >24h old (the upper-bound cap).
+- **A parse error in the shared `library/app/api/netsuite/rest.php` takes down EVERY client at
+  once (tier-wide outage).** All wrappers autoload `App_Api_Netsuite_Rest` in the pre-section
+  LOOKUPS/setup phase of `common_sync_togasupply.php` (~line 433). Because PHP parses the whole
+  file before executing, any fatal parse error in `rest.php` prevents the class from loading and
+  crashes **all** clients' sync at the same instant — before any record section runs.
+  **Localizing the crash from `Parameters`:** `NETSUITE_EXECUTION_MODE_*` is `<seconds>-IDLE` vs
+  `<seconds>-RUNNING`; `startModeIteration()` writes RUNNING at the *start* of a section. If a
+  frozen client's mode is stuck **IDLE** (not RUNNING), the script is dying **before** any section
+  — i.e. in the LOOKUPS/setup phase (autoload/parse error, or a lookup 403/exception). Stuck
+  **RUNNING** means it crashed *inside* that section. And because `SALES_ORDERS` advances its
+  watermark to ~now on every completed run (no 24h cap), a SALES_ORDERS watermark frozen for days
+  means no run has *completed*, even if the cron is firing. To confirm tier-wide vs per-client:
+  compare several clients' watermarks — **all frozen at the same timestamp = shared/upstream cause**
+  (e.g. this parse error), not per-client data. When the fix deploys, MODE flips to RUNNING and the
+  watermark jumps to ~now on completion (a positive live signal); the sync self-recovers and
+  backfills the gap.
+- **Monitoring blind spot — not every client is in the NetSuite sync monitor.**
+  `App_SystemMonitor_NetSuiteIntegration` (`library/app/systemmonitor/netsuiteintegration.php`)
+  alerts ClickUp when any client's checkpoint is >48h stale, but its `CLIENT_CONFIGURATIONS` list
+  (~line 16) **omits Quad, AIG, and Growrk**. Their stalls raise no alert (Quad's 3-day July-2026
+  outage went unflagged for that reason). Follow-up: add Quad/AIG/Growrk to `CLIENT_CONFIGURATIONS`.
 - **Seeded-but-stale = stalled sync.** Verify health by reading `LAST_SYNC_DATETIME_*` in each
   `Client_<Id>.Parameters`: a healthy client advances to ~now every 5 min. As of 2026-06-17,
   Quad was seeded but frozen at 2025-04/05 (not advancing) while the other 12 deployed clients
@@ -151,6 +178,19 @@ Parameters are stored **per client DB** but accessed **through the TOGa2 API**,
 
 ## Change history
 
+- 2026-07-09 — **Tier-wide outage (2026-07-06 → 07-09): a dropped brace in shared `rest.php`
+  killed every client's sync for 3 days.** A bad merge (`c3caa11a`, "Merge branch '_production'
+  into TRUE-79078", 2026-06-24) silently dropped two tokens at the `listLocations()` /
+  `listPurchaseOrders()` boundary in `library/app/api/netsuite/rest.php` (~line 1119) during
+  conflict resolution — the closing `}` of `listLocations()` and the `/**` opening the
+  `listPurchaseOrders()` docblock — leaving `return $out;` immediately followed by a bare
+  ` * Returns purchase orders...`, a fatal `syntax error, unexpected '*'`. `App_Api_Netsuite_Rest`
+  then failed to load entirely, so every client's autoload in the LOOKUPS phase crashed. Shipped to
+  prod via automated merge `c85e39e5` at 2026-07-06 16:17 UTC — the exact minute all clients' last
+  successful sync ran. Fix: restored the missing `}` and `/**`; verified `php -l` clean. Sync
+  self-recovered and backfilled the July 6–9 gap (Quad invoice 313319 / order 281251 / $2,356.22
+  imported, all watermarks current). See the new tier-wide-parse-error and monitor-blind-spot
+  gotchas above. (bala)
 - 2026-06-30 — The IF section's `syncItemFulfillmentFromNetsuite` (`library/app/api/toga2.php`) now
   imports **all** fulfillment statuses (`['_picked','_packed','_shipped']`, not just shipped),
   stamping/advancing the IF stage from `c_netsuiteInternalItemFulfillmentStatus`. Full mechanics

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.300",
+  "version": "1.0.301",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",