toga-ai 1.0.757 → 1.0.758

package/knowledge/1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-09-02
+updated: 2026-09-03
 owners: ["dfranks", "bala", "jcardinal", "mhammontree", "snaredla"]
 files:
   - worker/crons/toga2/netsuite/common_sync_togasupply.php
@@ -88,9 +88,27 @@ The single most expensive misreading of this engine. `startModeIteration()` writ
 `<seconds>-RUNNING` at the **start** of every section and `finishModeIteration()` writes `-IDLE` at
 the end; the ÷3 `REDUCE_INTERVAL_FACTOR` shrink fires only because the *previous* run never reached
 its IDLE write. **There is no timeout, memory cap, or resource limit anywhere in the script** — so
-"didn't finish" can only mean **it threw**. A collapsing window therefore means *go read Sentry*,
-never *tune the window size down*. (Corollary: a window that has grown back to the cap is positive
-evidence the section is completing.)
+"didn't finish" can only mean **it threw**. A collapsing window therefore means *go read the error
+in prod* (see below), never *tune the window size down*. (Corollary: a window that has grown back to
+the cap is positive evidence the section is completing.)
+
+**Where the error is: prod `Logs.Issue` / `Logs.Event`, NOT Sentry.** Read worker errors from the
+production log DB by default — you do not need a developer to point you there. The base `Logs`
+schema (prod-logs cluster) holds `Issue` (deduped, with `errorMessage` + `trace`) and `Event` (each
+occurrence, with `clientId`, `errorMessage`, `trace`, and a `context` JSON carrying `errorFile` +
+`errorLine`). Find a sync crash like this (NYCHH `clientId = 3`):
+
+```sql
+SELECT e.dtCreated, e.issueId, e.errorMessage, e.context
+FROM Event e
+WHERE e.clientId = 3 AND e.dtCreated > NOW() - INTERVAL 2 DAY
+ORDER BY e.dtCreated DESC;
+```
+
+Filter on the `context.REFERENCE.areaPath` = `crons/toga2/netsuite`. **The aggregated `Issue.trace`
+can show an OLD line; trust the newest `Event.context.errorLine` for the real spot.** `context` also
+gives `memoryPeakMb` — a small peak (e.g. 20 MB) rules out "giant record / OOM" and points at a
+plain thrown warning instead.
 
 ### Per-record resume cursor — checkpoint after EACH record, not window-end (LIVE 2026-09-02)
 
@@ -151,6 +169,39 @@ back in the account **display** format (here `M/D/YYYY`, date-only). Proven: `SE
   but the same trap if either ever becomes one.
 - Deployed 2026-09-02 (worker + library re-deploy; both files had changed).
 
+### ⚠ The by-reference refetch dropped `lastModifiedDate` → ITEM_FULFILLMENTS + INVENTORY_ADJUSTMENTS froze (found 2026-09-03, NYCHH prod)
+
+The per-record cursor introduced a trap specific to the two sections whose `sync*FromNetsuite()`
+**refetches the record by reference**. `syncItemFulfillmentFromNetsuite(&$nsItemFulfillment, …)`
+(`library/app/api/toga2.php:4535`) and `syncInventoryAdjustmentFromNetsuite(&$nsInventoryAdjustment,
+…)` (`:5655`) each **replace the caller's object** with a fetch-by-id shim
+(`fetchItemFulfillmentById` / `fetchInventoryAdjustmentById`). Those two fetch shims **did not set
+`->lastModifiedDate`**. So the per-record cursor PUT right after the call
+(`common_sync_togasupply.php:1235` fulfillments, `:1323` inventory adjustments) read
+`$record->lastModifiedDate` on the **replaced** object → *Undefined property: stdClass::$lastModifiedDate*
+→ 1.0 `App_Error::handleError` → `exit` → the section died **before** its cursor PUT and
+`finishModeIteration()`, freezing it at RUNNING with the watermark stuck years back. Confirmed on
+NYCHH prod (`Logs.Issue` 649, `clientId 3`): fulfillments frozen ~2023-01-23, inventory adjustments
+~2022-12-29, both windows shrinking. `memoryPeakMb` was 20 — a fast thrown warning, **not** a
+giant-record/OOM. **GroWrk was the reported alert** (ITEM_FULFILLMENTS re-throwing every 5 min
+since the 2026-09-02 per-record cursor deploy). Because the fix is in **shared library code**, all
+four active togasupply clients that hit these sections were affected: **GroWrk, NYCHH,
+Prudential-EDI, S&P Global**.
+
+- **ITEM_RECEIPTS was never affected**: `fetchItemReceiptById` already set `lastModifiedDate`, so its
+  cursor PUT succeeded (a real `…|<internalId>` value) while the other two carried the migration's
+  `…|0` and never moved.
+- **The `|0` is NOT the bug.** It is the intended empty-window marker (see the per-record cursor
+  section). The frozen sections showed `|0` only because they never completed a record since the
+  2026-09-02 rename.
+- **Fix (library only, `php -l` clean):** set `$shim->lastModifiedDate` in **both**
+  `fetchItemFulfillmentById` (from the REST header's `lastModifiedDate`) and
+  `fetchInventoryAdjustmentById` (add `lastmodifieddate` to its SuiteQL SELECT, then set the shim).
+  No worker change, no SQL, fail-loud preserved. After deploy, the two sections resume and backfill
+  the gap on their own (idempotent) — rewinding the cursor is optional to speed catch-up.
+- **Lesson:** any `sync*FromNetsuite()` that takes the record `&$byReference` and refetches must have
+  its fetch-by-id shim carry **every** field the caller reads afterward — `lastModifiedDate` first.
+
 ### ⚠ Per-record isolation is SECTION-SPECIFIC — and SALES_ORDERS has NONE (corrected 2026-08-17)
 
 Whether a bad record is "stepped over" or freezes the section depends entirely on whether **that
@@ -1124,6 +1175,22 @@ the code deploy).
 
 ## Change history
 
+- 2026-09-03 — **By-reference refetch dropped `lastModifiedDate` → ITEM_FULFILLMENTS + INVENTORY_ADJUSTMENTS
+  froze (fixed, `library` only).** GroWrk's ITEM_FULFILLMENTS was re-throwing every 5 min since the
+  2026-09-02 per-record cursor deploy (`Undefined property: stdClass::$lastModifiedDate` at
+  `common_sync_togasupply.php:1235`). Root cause: `syncItemFulfillmentFromNetsuite(&$rec)` and
+  `syncInventoryAdjustmentFromNetsuite(&$rec)` **replace** the by-ref record with a fetch-by-id shim
+  (`fetchItemFulfillmentById` / `fetchInventoryAdjustmentById` in `rest.php`), and neither shim set
+  `->lastModifiedDate` — so the per-record cursor PUT that reads it after the call threw → `App_Error`
+  `exit()` → section died before `finishModeIteration()` → frozen RUNNING, window ÷3. `fetchItemReceiptById`
+  already set it, so ITEM_RECEIPTS was never hit; INVENTORY_ADJUSTMENTS was latent (no client had processed
+  one since the rename). Shared code → all four active togasupply clients affected (GroWrk, NYCHH,
+  Prudential-EDI, S&P Global); no data lost (the record write commits each run), self-heals after deploy.
+  Fix: set `$shim->lastModifiedDate` in both shims (add `lastmodifieddate` to InventoryAdjustment's SuiteQL
+  SELECT). `php -l` clean; no worker/SQL change. Lesson: any `sync*FromNetsuite()` that refetches its
+  `&$byReference` record must have its fetch-by-id shim carry every field the caller reads afterward. Full
+  detail in *How it works* above; shim-side entry on the
+  [SuiteQL/REST shim doc](../../library/features/netsuite-suiteql-rest-shim.md). (jcardinal)
 - 2026-09-02 — **Per-record cursor was storing a date-ONLY, US-format value — FIXED + DEPLOYED (shared:
   `worker` + `library`).** The datetime half was `M/D/YYYY` with no time (e.g. `9/2/2026|6302323`), so the
   cursor sat at midnight and the same-second id tiebreak never matched — every run re-fetched the whole day.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.757",
+  "version": "1.0.758",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",