toga-ai 1.0.750 → 1.0.751

package/knowledge/1.0/apps/library/features/netsuite-sync-alert-monitor.md

@@ -6,8 +6,8 @@ project: Library
 client: shared
 type: feature
 status: active
-updated: 2026-07-10
-owners: ["bala"]
+updated: 2026-09-02
+owners: ["bala", "jcardinal"]
 files:
   - library/app/systemmonitor/netsuiteintegration.php
   - worker/crons/infrastructure/system_monitors.php
@@ -104,8 +104,22 @@ that worker job does the syncing, this library class notices when it stops.
   checkpoints mean anything; and a client that is held / never transacts supply (AIG) must be
   left **off** this list or it produces permanent false-positive tickets.
 
+- **⚠ The pending cursor-rename in the sync will SILENTLY BLIND this monitor (BUILT 2026-09-02, not yet
+  deployed).** The per-record resume cursor work on the sync renames each section's position key from
+  `NETSUITE_LAST_SYNC_DATETIME_<SECTION>` to `NETSUITE_LAST_SYNC_CURSOR_<SECTION>` (value becomes
+  `"<lastmodifieddate>|<netsuite id>"`). This monitor's `NETSUITE_SYNC_PARAMETER_KEYS` still lists the old
+  `DATETIME` keys, so after the rename its `where..in` `/parameters` read matches **zero rows** and it
+  reports OK forever — no stale-checkpoint alerts. The sync developer **intentionally left the monitor
+  un-updated** this session (accepted gap). When the rename deploys, update `NETSUITE_SYNC_PARAMETER_KEYS`
+  to the `CURSOR` names **and** parse the `<date>|<id>` value (`strtotime` the part before `|`) before the
+  48h compare. See the [per-client sync doc](../../worker/features/netsuite-togasupply-per-client-sync.md).
+
 ## Change history
 
+- 2026-09-02 — Flagged that the sync's pending `NETSUITE_LAST_SYNC_DATETIME_*` →
+  `NETSUITE_LAST_SYNC_CURSOR_*` key rename (BUILT, not yet deployed) will make this monitor match zero rows
+  and stop alerting; it was intentionally left un-updated. Update the key list + parse the `<date>|<id>`
+  value when the rename ships. (jcardinal)
 - 2026-07-10 — **Fixed the monitor timing out and silently not running since 2026-06-29.**
   Rewrote `getClientStatus()` to read all six checkpoints per client in one TOGa2
   `where..in` `/parameters` call (was 84 serialized calls ≈ 230s, over the 300s

package/knowledge/1.0/apps/worker/INDEX.md

@@ -10,7 +10,7 @@
 | [Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling](features/forecast2-netsuite-reconciliation.md) | CLI tools to **audit** and **repair** drift between the production `Forecast` DB (core2) and NetSuite. | test/@dave/checker.php, worker2/Component/Forecast/SaleImport/SaleImport.php, test/@dave/looper.php, test/@dave/reconcile_netsuite_totals.php, test/@dave/fixer.php, tools/bin/forecast/fixer.php, tools/bin/forecast/checker.php, test/@dave/analyze_netsuite_forecast_diff.php, test/@dave/trueup_sales.php, test/@dave/reconcile_drift_2023plus.php, test/@dave/probe_invoice_gap_2026.php, test/@dave/probe_creditmemo_gap_detail.php, test/@dave/trueup_open_orders.php, test/@dave/loop_trueup_open_orders.php, test/@dave/trueup_opportunities.php, test/@dave/probe_sales_gap_direct.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_profit_invoices.php, test/@dave/probe_profit_gap.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_open_orders.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/schedules/cron.worker.infrastructure.json |
 | [Forecast2 Supporting-Records Nightly Import (1.0 cron) — Customers & ConsolidatedCustomers](features/forecast2-supporting-records-import.md) | The nightly **1.0** pull that keeps the Forecast2 lookup/dimension tables (`Forecast.Accounts`, `Classifications`, `Customers`, `ConsolidatedCustomers`, `Employ | worker/crons/toga2/forecast2/import_supporting_records.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker/schedules/cron.worker.infrastructure.json, worker/.ebextensions/009_setup_phpini.config, worker/crons/sync/netsuite/netsuite_customers.php, library/app/api/netsuite/rest.php, library/app/model/forecast2/customer.php, library/app/model/forecast2/consolidatedcustomer.php |
 | [NetSuite Sales Order Sales Rep Sourcing (Staples & ODP EDI orders)](features/netsuite-sales-order-sales-rep-sourcing.md) | How the **sales rep** on a NetSuite Sales Order is determined for the two 1.0 `worker` EDI order-creation integrations (Staples cXML and Compass/ODP EDI). | worker/crons/toga2/compass/workflow/5_create_netsuite_sales_orders_from_office_depot_purchase_orders.php, worker/crons/sync/staples/sync_staples_cxml.php, test/@Mark/NetSuite/TRUE_80451_customer_salesrep_diag.php |
-| [NetSuite → TOGa Supply Per-Client Sync (thin wrappers)](features/netsuite-togasupply-per-client-sync.md) | Syncs NetSuite transactions (sales orders, purchase orders, invoices, item receipts, item fulfillments, inventory adjustments) into each TOGa Supply (2.0) clien | worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_canon.php, worker/crons/toga2/netsuite/sync_togasupply_elite.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql, library/app/api/toga2.php, library/app/api/netsuite/rest.php, library/app/framework.php, library/app/systemmonitor/netsuiteintegration.php, test/@srija/Elite Testing/Service Requests/test_sync_togasupply_elite_section.php, test/@srija/Elite Testing/Service Requests/test_diagnose_togasupply_elite.php |
+| [NetSuite → TOGa Supply Per-Client Sync (thin wrappers)](features/netsuite-togasupply-per-client-sync.md) | Syncs NetSuite transactions (sales orders, purchase orders, invoices, item receipts, item fulfillments, inventory adjustments) into each TOGa Supply (2.0) clien | worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_canon.php, worker/crons/toga2/netsuite/sync_togasupply_elite.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql, library/app/api/toga2.php, library/app/api/netsuite/rest.php, library/app/query.php, library/app/database.php, library/app/framework.php, library/app/systemmonitor/netsuiteintegration.php, dbchanges2/Client/2026-09-02b - NetsuiteSyncCursorRename.sql, test/@srija/Elite Testing/Service Requests/test_sync_togasupply_elite_section.php, test/@srija/Elite Testing/Service Requests/test_diagnose_togasupply_elite.php |
 | [OneUptime Server monitor + disk/memory hygiene on the 1.0 worker EB host](features/oneuptime-server-monitor-host-hygiene.md) | The 1.0 `agilant-worker` EB environment runs on the **legacy Amazon Linux 1 PHP 7.2 platform** (Apache httpd/prefork, s3fs mounts, cron) and repeatedly went dow | worker/.ebextensions/040_disk_memory_hygiene.config, worker/.ebextensions/045_oneuptime_agent.config, worker/ebs/cron.worker.php, worker/ebs/mount-s3fs-folders.php, worker/ebs/apache_settings.php, worker/ebs/setup_phpini.php |
 | [OneUptime external uptime monitoring for 1.0 workers](features/oneuptime-worker-uptime-monitoring.md) | Every 1.0 worker box self-reports its liveness to an external OneUptime monitor once per minute by curl-POSTing to a per-worker "Incoming Request" heartbeat URL | library/app/worker.php, worker/crons/worker/worker_heartbeat.php, worker/ebs/cron.worker.php, worker/.ebextensions/045_oneuptime_agent.config |
 | [Prudential: Send Shipments for the Day report (daily cron)](features/send-shipments-for-the-day.md) | Daily cron (9:00 PM) that emails Prudential and Dell stakeholders an Excel report of all devices shipped that day, including tracking number, serial number, emp | worker/crons/notifications/reports/send_shipments_for_the_day.php |

package/knowledge/1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md

@@ -16,8 +16,11 @@ files:
   - dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql
   - library/app/api/toga2.php
   - library/app/api/netsuite/rest.php
+  - library/app/query.php
+  - library/app/database.php
   - library/app/framework.php
   - library/app/systemmonitor/netsuiteintegration.php
+  - dbchanges2/Client/2026-09-02b - NetsuiteSyncCursorRename.sql
   - test/@srija/Elite Testing/Service Requests/test_sync_togasupply_elite_section.php
   - test/@srija/Elite Testing/Service Requests/test_diagnose_togasupply_elite.php
 related:
@@ -89,6 +92,28 @@ its IDLE write. **There is no timeout, memory cap, or resource limit anywhere in
 never *tune the window size down*. (Corollary: a window that has grown back to the cap is positive
 evidence the section is completing.)
 
+### Per-record resume cursor — checkpoint after EACH record, not window-end (BUILT 2026-09-02, NOT yet deployed)
+
+The historical weakness of window-end-only checkpointing: a mid-window crash (poison record, or the
+log-DB doom loop above) re-processes the **whole** window every run — the giant-fulfillment doom loop is a
+direct consequence. This change converts the checkpoint to **per-record keyset (seek) resume** so a
+transient crash resumes at the next record instead of redoing the window.
+
+- **Key rename per section:** `NETSUITE_LAST_SYNC_DATETIME_<SECTION>` → `NETSUITE_LAST_SYNC_CURSOR_<SECTION>`,
+  value format **`"<lastmodifieddate>|<netsuite internal id>"`**. The id breaks same-second ties.
+- **Fetch change (`library/app/api/netsuite/rest.php`, all 6 header queries):** add
+  `ORDER BY lastmodifieddate, id` and change the lower bound to
+  `lastmodifieddate > :t OR (lastmodifieddate = :t AND id > :fromId)` (upper bound `<= :to` kept). A new
+  `$fromId` param was threaded into the 6 list funcs. Added `lastModifiedDate` to the fetched record object
+  for **PURCHASE_ORDERS, ITEM_RECEIPTS, ITEM_FULFILLMENTS, INVENTORY_ADJUSTMENTS** (SALES_ORDERS + INVOICES
+  already carried it).
+- **Save after EACH main record** (one PUT of `lastmod|id`); on a clean window with 0 records, PUT
+  `"<windowTop>|0"` so empty date ranges still advance. No new `try/catch` — stays fail-loud.
+- **Migration is MANDATORY with the code (see the reset trap in Gotchas):**
+  `dbchanges2/Client/2026-09-02b - NetsuiteSyncCursorRename.sql` (Client fan-out) renames the keys **and**
+  appends `|0` to carry the value across; safe to re-run. The new-client seed module
+  (`dbchanges2/_modules/netsuite/2026-08-05 - CLEAN NETSUITE CLINET.SQL`) was updated to CURSOR.
+
 ### ⚠ Per-record isolation is SECTION-SPECIFIC — and SALES_ORDERS has NONE (corrected 2026-08-17)
 
 Whether a bad record is "stepped over" or freezes the section depends entirely on whether **that
@@ -305,6 +330,10 @@ Per-client `Parameters` table (2.0 client DB, e.g. `Client_Canon.Parameters`; co
 
 Parameters are stored **per client DB** but accessed **through the TOGa2 API**, not direct SQL.
 
+**Pending rename (BUILT 2026-09-02, not yet deployed):** the position keys become
+`NETSUITE_LAST_SYNC_CURSOR_<SECTION>` with value `"<lastmodifieddate>|<netsuite internal id>"` — see the
+per-record resume cursor in *How it works*. The `NETSUITE_EXECUTION_MODE_*` keys are unchanged.
+
 ## Client variations
 
 - **Transfer-order routing is opt-in per wrapper (default OFF).** A client whose NetSuite
@@ -856,6 +885,15 @@ Parameters are stored **per client DB** but accessed **through the TOGa2 API**,
   line). **Fix (`library`):** skip lines whose `adjustQtyBy` is null in **BOTH** passes over
   `inventoryList->inventory` (the item-build pass and the serial/unit pass). Shared fix; unblocked NYCHH
   inventory adjustments.
+  - **Prod-confirmed 2026-09-02** (commit `8bae8a2a`, 2026-08-29; the null-skip lands at
+    `toga2.php:5728` and `:5879`). Before it, NYCHH `INVENTORY_ADJUSTMENTS` was frozen `3-IDLE` (window
+    collapsed to ~3s), watermark stuck at **2022-12-28**, advancing only seconds/run — `Logs.Issue`
+    **542** (`clientId 3`), 35 occurrences 2026-08-28..09-01. The fix reached prod ~2026-09-01 afternoon
+    and **Issue 542 stopped 09-01 14:08**. This `3-IDLE`-crawl-with-frozen-watermark signature is the same
+    poison-record shrink as a `1-RUNNING` freeze — a section that throws on a record too old to fall
+    outside the collapsed window keeps re-throwing; once the window shrinks past that record's timestamp it
+    reports IDLE but crawls seconds/run and can never advance past it. Read it as "still throwing," not
+    "small/slow section" (see the item-fulfillment doom-loop gotcha below for the same signature).
 - **⚠ Item-receipt sync froze on a PARTIAL receipt's NULL-quantity PO lines — the item-receipt TWIN of the
   inventory-adjustment NULL-quantity fix above (fixed 2026-08-31).** For a partial item receipt NetSuite
   returns **all** the PO's linked lines but sets `quantity=null` on the lines **not received on this
@@ -877,14 +915,33 @@ Parameters are stored **per client DB** but accessed **through the TOGa2 API**,
   `transactionId` (`Logs.Event` carries `clientId`, `Logs.Issue` does not) → `Logs_Nychh.Api` on that
   `transactionId` for the untruncated `EV-10`/1048 rejection and the payload showing the `quantity:null`
   lines.
-- **⚠ OPEN, not yet root-caused (found 2026-08-31): NYCHH ITEM_FULFILLMENTS watermark frozen at 2021-12-29.**
-  `NETSUITE_EXECUTION_MODE_ITEM_FULFILLMENTS='3-IDLE'` with the window collapsed to ~3s (it has been
-  throwing recently) but currently **IDLE** — so it crawls a few seconds per run rather than sitting in a
-  hard `1-RUNNING` freeze. Distinct from the fixed ITEM_RECEIPTS freeze above; needs its own investigation.
-  Context: the 2026-08-31 full 19-client togasupply prod sweep found all other clients healthy
-  (`864000-IDLE`) except Elite SALES_ORDERS `864000-RUNNING` (healthy mid-run, large window, not stuck) and
-  AIG SALES_ORDERS `8-RUNNING` (the known held account, already documented). Only NYCHH ITEM_RECEIPTS was in
-  a true `1-RUNNING` freeze.
+- **NYCHH ITEM_FULFILLMENTS watermark frozen at 2021-12-29 (found 2026-08-31; ROOT-CAUSED 2026-09-02 —
+  the log-DB doom loop below).** `NETSUITE_EXECUTION_MODE_ITEM_FULFILLMENTS='3-IDLE'` with the window
+  collapsed to ~3s (throwing recently) but currently **IDLE** — so it crawls a few seconds per run rather
+  than a hard `1-RUNNING` freeze. This is the exact signature of the doom loop in the next gotcha: a giant
+  fulfillment throws every run, the window shrinks ÷3 until it no longer spans that fulfillment, then goes
+  IDLE but can never advance past it. Context: the 2026-08-31 full 19-client togasupply prod sweep found
+  all other clients healthy (`864000-IDLE`) except Elite SALES_ORDERS `864000-RUNNING` (healthy mid-run,
+  large window, not stuck) and AIG SALES_ORDERS `8-RUNNING` (the known held account).
+- **⚠ Item-fulfillment DOOM LOOP from a dropped LOG-DB connection during a giant fulfillment (found
+  2026-09-02, NOT yet fixed — the per-record cursor below is the mitigation).** A single very large
+  fulfillment POSTs thousands of `/item-fulfillment-item-units` over api2 HTTP; that long unit-POST phase
+  lets the **worker's log-DB connection idle past MySQL `wait_timeout`**. The **next** fulfillment's refetch
+  — `fetchItemFulfillmentById` at `library/app/api/toga2.php:4440`, the **first line** of
+  `syncItemFulfillmentFromNetsuite` — tries to log that NetSuite REST call and dies at
+  `library/app/query.php:168` `mysqli_select_db` *"MySQL server has gone away / Connection timed out"*
+  (`Logs.Issue` **589**). The 1.0 framework escalates that warning to `exit`, so the run aborts mid-window;
+  the watermark only advances at window end (`finishModeIteration`), so the whole window — including the
+  giant fulfillment — **re-processes next run, times out at the same spot, and the window shrinks ÷3 each
+  run** (→ the `3-IDLE` crawl signature above once the window no longer spans the giant fulfillment).
+  - **Root of the crash:** the "gone away" recovery in `App_Query::execute` (`library/app/query.php`,
+    `database.php:431`) only guards the `mysqli_query` call (errno **2006** reconnect), **NOT** the
+    `mysqli_select_db` that runs before it — which is the first touch of the stale connection, so the
+    reconnect never gets a chance. A durable framework gap, not sync-specific.
+  - **Durable engine rule reconfirmed:** a shrinking window in this sync **ALWAYS** means a thrown
+    exception — there is **no** volume/time/memory backoff anywhere in the script. "Window not 864000" ⇒
+    something threw. The log-connection idle-out is a concrete cause to add to the poison-record and
+    shim-shape causes above.
 - **⚠ Item-fulfillment stale-item prune must delete tracking-number children before the unit AND the item
   (fixed 2026-08-30).** In `syncItemFulfillmentFromNetsuite`'s cleanup loop, deleting a stale
   item-fulfillment-item-**unit** failed api2 `EV-11` / MySQL **1451** (FK
@@ -967,8 +1024,51 @@ Parameters are stored **per client DB** but accessed **through the TOGa2 API**,
   A "silent freeze" here is a serial/data collision, not necessarily a catalog/item-matching problem —
   read the untruncated Api-log rejection before assuming item matching.
 
+- **⚠ The position key defaults to the EARLIEST date when MISSING — a code-only cursor rename WITHOUT
+  the data migration resets every client to re-sync from scratch (found 2026-09-02).** When the section's
+  position key is absent, the engine falls back to `MIN_DATETIME_TO_CHECK_FOR_NETSUITE_DATA`
+  (`common_sync_togasupply.php` ~L573). So shipping the `NETSUITE_LAST_SYNC_DATETIME_*` →
+  `NETSUITE_LAST_SYNC_CURSOR_*` rename in **code only** makes every client's old key invisible → each
+  section restarts from the earliest date and re-syncs its entire history. The migration
+  (`dbchanges2/Client/2026-09-02b - NetsuiteSyncCursorRename.sql`, Client fan-out) **must carry the value
+  across** — it renames the key and appends `|0` in one `UPDATE ... REPLACE(...)`, re-runnable. **Ship the
+  migration on every client DB before (or with) the code.**
+- **⚠ The NetSuite Sync Alert monitor will SILENTLY STOP after the cursor rename (known, accepted
+  2026-09-02).** `App_SystemMonitor_NetSuiteIntegration` (`library/app/systemmonitor/netsuiteintegration.php`)
+  hardcodes the old `NETSUITE_LAST_SYNC_DATETIME_*` keys, so once the rename ships its `where..in`
+  `/parameters` read matches **zero rows** and it stops alerting on stale checkpoints. The developer
+  **intentionally left the monitor un-updated** this session — a documented, accepted gap. Update the
+  monitor's key list to `NETSUITE_LAST_SYNC_CURSOR_*` (and parse the `<date>|<id>` value) when the rename
+  deploys, or NetSuite sync stalls go unmonitored. Cross-ref:
+  [NetSuite Sync Alert Monitor](../../library/features/netsuite-sync-alert-monitor.md).
+
 ## Change history
 
+- 2026-09-02 — **Per-record resume cursor BUILT (shared: `worker` + `library` + `dbchanges2`; pending
+  review + deploy, NOT shipped).** Converts the section checkpoint from window-end-only to per-record
+  keyset resume so a mid-window crash (poison record or the log-DB doom loop) resumes at the next record
+  instead of re-processing the whole window. Position key `NETSUITE_LAST_SYNC_DATETIME_<SECTION>` →
+  `NETSUITE_LAST_SYNC_CURSOR_<SECTION>`, value `"<lastmodifieddate>|<netsuite id>"` (id breaks same-second
+  ties); `rest.php`'s 6 header queries gain `ORDER BY lastmodifieddate, id` + a keyset lower bound + a
+  `$fromId` param; cursor PUT after each record (and `"<windowTop>|0"` on an empty window); `lastModifiedDate`
+  added to the PO/IR/IF/InvAdj fetched objects. Migration `Client/2026-09-02b - NetsuiteSyncCursorRename.sql`
+  (fan-out) renames the keys and appends `|0` — **mandatory**, because a missing key defaults to the earliest
+  date and would re-sync every client's full history. Monitor `netsuiteintegration.php` intentionally left on
+  the old DATETIME keys (will stop alerting after the rename — accepted). (jcardinal)
+- 2026-09-02 — **Item-fulfillment doom loop root-caused (found, NOT fixed — the cursor above mitigates).** A
+  giant fulfillment's long unit-POST phase idles the worker's log-DB connection past MySQL `wait_timeout`; the
+  next fulfillment's `fetchItemFulfillmentById` (`toga2.php:4440`) dies logging its NetSuite call at
+  `query.php:168` `mysqli_select_db` "server has gone away" (`Logs.Issue 589`) → 1.0 warning→`exit` → window
+  re-processes and shrinks ÷3 each run (the `3-IDLE` crawl). Root cause: `App_Query::execute`'s gone-away
+  recovery (`query.php`/`database.php:431`) guards only the `mysqli_query` (2006 reconnect), not the
+  `mysqli_select_db` before it. Explains the NYCHH ITEM_FULFILLMENTS freeze at 2021-12-29. Reconfirmed the
+  engine rule: a shrinking window ALWAYS means a throw (no volume/time/memory backoff exists). (jcardinal)
+- 2026-09-02 — **Prod-confirmed the NYCHH inventory-adjustment NULL-`adjustQtyBy` freeze + its fix (commit
+  `8bae8a2a`, `toga2.php:5728`/`:5879`).** `INVENTORY_ADJUSTMENTS` had been frozen `3-IDLE`, watermark stuck
+  2022-12-28, crawling seconds/run — `Logs.Issue 542` (`clientId 3`), 35 occ 2026-08-28..09-01, EV-10/MySQL
+  1048 "quantityAdjustBy cannot be null" on NetSuite serialized-detail sub-lines. The null-skip reached prod
+  ~2026-09-01 afternoon; Issue 542 stopped 09-01 14:08. Recorded that `3-IDLE`-crawl = the same poison-record
+  shrink as `1-RUNNING`, not a slow section. (jcardinal)
 - 2026-09-02 — **PURCHASE_ORDERS cleanup DELETE freeze fixed (shared, `library` only — WRITTEN +
   REVIEWED, NOT yet committed/deployed).** `syncPurchaseOrderFromNetsuite`'s "remove dropped items"
   cleanup deleted a NetSuite-dropped PO line with a bare throwing DELETE; a leftover item receipt

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.750",
+  "version": "1.0.751",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",