toga-ai 1.0.752 → 1.0.754

package/knowledge/1.0/apps/library/features/netsuite-suiteql-rest-shim.md

@@ -6,7 +6,7 @@ project: Library
 client: shared
 type: feature
 status: active
-updated: 2026-09-01
+updated: 2026-09-02
 owners: [dfranks, jcardinal, kyalamarthi]
 files:
   - library/app/api/netsuite/rest.php
@@ -98,6 +98,21 @@ Forecast2 tables already store.
 - **TO_DATE rejects impossible dates** (e.g. `2026-06-31`) with an opaque HTTP 400 — validate
   calendar dates before building SuiteQL.
 
+- **⚠ SuiteQL returns date/datetime columns in the account DISPLAY format (here `M/D/YYYY`,
+  date-only) unless you wrap them in `TO_CHAR` — losing the time is silent (2026-09-02).** A bare
+  `SELECT lastmodifieddate` came back `9/2/2026` (no time, locale-dependent order), while
+  `TO_CHAR(lastmodifieddate,'YYYY-MM-DD HH24:MI:SS')` gave `2026-09-02 17:39:21`. Any shim that
+  **stores or compares a NetSuite date STRING** must `TO_CHAR(col,'YYYY-MM-DD HH24:MI:SS')` it, or it
+  drops the time and inherits the account's locale format. This defeated the per-client sync's
+  per-record resume cursor: it stored `9/2/2026|<id>`, sat at midnight, and re-scanned the whole day
+  every run — fixed by `TO_CHAR`-ing `lastmodifieddate AS lastmodifieddate` in the 8 cursor-feeding
+  SELECTs (`listSalesOrders`, `fetchSalesOrderByIdFull`, `listSales`, `listPurchaseOrders`,
+  `fetchPurchaseOrderById`, `listItemReceipts`, `listItemFulfillments`, `listInventoryAdjustments`).
+  `WHERE`/`ORDER BY` are unaffected by the alias — bound via `TO_TIMESTAMP` — so they still use the base
+  column. **Still latent:** `listOpportunities` and `listItems` return raw date-only `lastmodifieddate`
+  into their shims; harmless only because neither feeds a resume cursor yet. Engine-side detail on the
+  [per-client-sync doc](../../worker/features/netsuite-togasupply-per-client-sync.md).
+
 - **The line SuiteQL must SELECT the line location or it is always NULL — the shim reads it but
   didn't select it.** `listSalesOrders()`'s line query now selects `tl.location` and
   `BUILTIN.DF(tl.location) AS location_name`. The line-shim assembly code already *read* those two
@@ -275,6 +290,12 @@ None — uniform across clients (NetSuite is a single shared account).
 
 ## Change history
 
+- 2026-09-02 — Recorded that **SuiteQL returns date columns in the account DISPLAY format (date-only
+  `M/D/YYYY` here) unless wrapped in `TO_CHAR`** — a bare `lastmodifieddate` silently drops the time and
+  is locale-dependent. This defeated the per-client sync's per-record resume cursor (stored `9/2/2026|id`,
+  sat at midnight, re-scanned the whole day); fixed by `TO_CHAR(...,'YYYY-MM-DD HH24:MI:SS') AS
+  lastmodifieddate` in the 8 cursor-feeding SELECTs. `listOpportunities`/`listItems` still return raw
+  date-only but are latent (no cursor). (jcardinal)
 - 2026-09-01 — Recorded that **`listPurchaseOrders()` scoped POs by the VENDOR header `entity`** (via
   the reused `buildEntityInFilter()`), silently dropping every drop-ship / special-order PO for all
   togasupply clients since the SOAP→REST migration. A PO header `entity` is the vendor, not the

package/knowledge/1.0/apps/library/features/toga2-api-client-and-bridge.md

@@ -418,9 +418,14 @@ enable flags** and an optional `$monitorTogadeskDepartmentIds[]`:
   still had a RESTRICT child (item receipt `ItemReceiptItems.purchaseOrderItemId`, or an SO/TO link).
   Fixed to **unlink first, then delete**: delete the four PO bridge routes, unhook item receipts with
   `PUT /item-receipt-items/{uuid}` `['purchaseOrderItem' => null]`, then DELETE the PO line kept
-  **throwing** as the fail-loud backstop — mirroring the SALES_ORDERS cleanup. Written + reviewed,
-  **not yet committed/deployed**; carries a schema deploy dependency (nullable
-  `ItemReceiptItems.purchaseOrderItemId`). Full detail, the mirror-NetSuite decision, and the api2
+  **throwing** as the fail-loud backstop — mirroring the SALES_ORDERS cleanup. **Now committed to
+  `_production` and confirmed live on Elite** (carried a schema deploy dependency: nullable
+  `ItemReceiptItems.purchaseOrderItemId`). **Follow-up fix same day:** the bridge-clear step's **GET**
+  was also a throwing `send()` — a client without an ACL grant on the transfer-order bridge routes
+  (Elite, Core records 329/330) got **403 EZ-2** and re-froze; the bridge/link GET loop now passes
+  `$throwExceptionOnApiError=false` and **skips** a link table on a no-access response (403 OR EZ-2),
+  while any other failure still throws. (Ops note: merging `library` to `_production` does not deploy
+  the worker boxes — a worker deploy does.) Full detail, the mirror-NetSuite decision, and the api2
   bridge-route-key mechanic (kebab→camel, no pluralization) live on
   [NetSuite → TOGa Supply per-client sync](../../worker/features/netsuite-togasupply-per-client-sync.md)
   (2026-09-02). (jcardinal)

package/knowledge/1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md

@@ -122,6 +122,35 @@ rename migration run on every client DB first (see the deploy-order gotcha below
   the same guarded `UPDATE ... WHERE key LIKE 'NETSUITE_LAST_SYNC_DATETIME_%'` so future new clients land on
   CURSOR keys. Still uncommitted in the working tree.
 
+### ⚠ The cursor was storing a date-ONLY, US-format value — the same-second tiebreak was dead (FIXED + DEPLOYED 2026-09-02)
+
+The per-record cursor "advanced" but its datetime half was `M/D/YYYY` with **no time** (e.g.
+`9/2/2026|6302323`), so it defeated its own same-second id tiebreak: the cursor sat at **midnight**,
+which no record's real timestamp equals, so every run re-fetched the **whole day** — the resume was
+defeated even while it looked healthy. Root cause: the SuiteQL SELECTs in
+`library/app/api/netsuite/rest.php` returned `lastmodifieddate` **raw**, and NetSuite hands that column
+back in the account **display** format (here `M/D/YYYY`, date-only). Proven: `SELECT lastmodifieddate`
+→ `9/2/2026`, but `TO_CHAR(lastmodifieddate,'YYYY-MM-DD HH24:MI:SS')` → `2026-09-02 17:39:21`. See the
+[SuiteQL/REST shim doc](../../library/features/netsuite-suiteql-rest-shim.md) for the general rule.
+
+- **Fix part 1 (`rest.php`):** wrapped the select as
+  `TO_CHAR(lastmodifieddate, 'YYYY-MM-DD HH24:MI:SS') AS lastmodifieddate` in the **8** SuiteQL SELECTs
+  that feed a cursor shim — `listSalesOrders`, `fetchSalesOrderByIdFull`, `listSales`,
+  `listPurchaseOrders`, `fetchPurchaseOrderById`, `listItemReceipts`, `listItemFulfillments`,
+  `listInventoryAdjustments`. `WHERE`/`ORDER BY` still use the **base** `lastmodifieddate` column (the
+  alias doesn't affect them) — the query bounds were already correct ISO via
+  `TO_TIMESTAMP('YYYY-MM-DD HH24:MI:SS')`.
+- **Fix part 2 (`common_sync_togasupply.php`):** normalized the datetime half at the **6** per-record
+  cursor PUTs to canonical `Y-m-d H:i:s` via `strtotime()` + `date()` (falls back to the raw string if
+  `strtotime` fails). The empty-window `"<windowTop>|0"` PUT was already canonical
+  (`App_Date::convertToSQLDate`).
+- **Old date-only cursors self-heal.** `strtotime('9/2/2026')` parses to midnight (one-time re-scan of
+  that day), then the next successful record overwrites with the full timestamp.
+- **LATENT (noted, not fixed):** `listOpportunities` and `listItems` in `rest.php` also return raw
+  date-only `lastmodifieddate` into their shims — harmless today because neither feeds a resume cursor,
+  but the same trap if either ever becomes one.
+- Deployed 2026-09-02 (worker + library re-deploy; both files had changed).
+
 ### ⚠ Per-record isolation is SECTION-SPECIFIC — and SALES_ORDERS has NONE (corrected 2026-08-17)
 
 Whether a bad record is "stepped over" or freezes the section depends entirely on whether **that
@@ -649,8 +678,8 @@ the code deploy).
   — for the throwing order.
 
 - **⚠ The PURCHASE_ORDERS cleanup DELETE froze the whole section when NetSuite drops a PO line that
-  still has an item receipt or SO/TO link (fixed 2026-09-02, `library` only — WRITTEN + REVIEWED,
-  NOT yet committed/deployed).** Same freeze class as the SALES_ORDERS renumber-DELETE above, on a
+  still has an item receipt or SO/TO link (fixed + DEPLOYED 2026-09-02, `library` only — committed to
+  `_production`, confirmed live on Elite).** Same freeze class as the SALES_ORDERS renumber-DELETE above, on a
   different path. `syncPurchaseOrderFromNetsuite`'s cleanup loop ("remove remaining items if
   updating", `library/app/api/toga2.php`) deleted a leftover Toga PO line with a **bare, throwing**
   `send()` DELETE `/purchase-order-items/{uuid}`. When the dropped line still had an item receipt
@@ -681,6 +710,26 @@ the code deploy).
     `dbchanges2/Client/2026-06-19a - TransferOrderSchemaFixes.sql` (section 7, guarded/re-runnable).
     **Confirm that migration has run on every client DB before deploying the code fix**, or the PUT
     `purchaseOrderItem = null` fails.
+  - **⚠ Follow-up (deployed same day): the bridge-CLEAR GET must TOLERATE a 403 for a client that
+    lacks the bridge (transfer-order routes).** After the first fix deployed, Elite hit a NEW freeze.
+    The cleanup **GETs** each of the four bridge routes to find rows to remove, and that GET was a
+    **throwing** `send()`. Elite has **no ACL grant** on the transfer-order bridge Core records (**329**
+    `purchase-order-items-transfer-order-items`, **330** `transfer-order-items-purchase-order-items`)
+    because Elite does not use transfer orders, so `GET /purchase-order-items-transfer-order-items`
+    returned **403 EZ-2** ("no permission to read the specified fields", roleIds [1,3]) → the whole
+    section aborted, watermark still frozen at 2025-10-13. **Fix:** the bridge/link GET loop now calls
+    `App_Api_Toga2::send(..., $throwExceptionOnApiError = false)` and, on a **no-access** response
+    (status **403** OR error code **EZ-2**), **skips that link table** — no access means the client
+    cannot have rows there, so there is nothing to clear. **Any other failure still throws** (fail
+    loud). Covers both transfer-order bridge routes. Confirmed: Elite's PO import unfroze after
+    redeploy.
+  - **Confirming the new cleanup code is actually live (from `Logs_<Client>.Api`).** A run on the
+    **NEW** code shows, before the PO-line delete: `GET /purchase-order-item-sales-order-items`,
+    `GET /sales-order-item-purchase-order-items`, `GET /purchase-order-items-transfer-order-items`
+    (may be **403 and skipped**), `GET /item-receipt-items`, then `PUT /item-receipt-items/{uuid}`
+    (the unhook). The **OLD** code goes straight from the purchase-order PUT to a bare
+    `DELETE /purchase-order-items/{uuid}` with none of those. This call pattern is the ground truth
+    for whether the worker box is running the fix.
   - **api2 route mechanics (verified against `api2/Component/Api/V2/V2.php`).** For these list/bridge
     routes the response collection key is a **plain kebab→camel conversion of the route, no
     pluralization** (`/purchase-order-item-sales-order-items` → `purchaseOrderItemSalesOrderItems`).
@@ -695,6 +744,14 @@ the code deploy).
     "move the receipt to the surviving line" is **unsafe** — double-count / `UNIQUE(itemReceiptId,
     purchaseOrderItemId)` collision. Only clean cases self-heal; tangled ones must throw for a human.
 
+- **⚠ Merging `library` to `_production` on GitHub does NOT update the running worker EC2 boxes — a
+  worker DEPLOY does.** After a `library` merge to `_production`, the worker instances keep running the
+  **OLD** `library` code across cron cycles until an actual worker deploy pushes it onto the boxes. On
+  2026-09-02 the PO-cleanup fix merged at 14:48 but the worker ran old code for multiple 5-min cron
+  cycles (the api2 log still showed the old bare DELETE, none of the new calls) until it was deployed.
+  **Before concluding a `library` fix "didn't work," confirm the worker box is actually running it** —
+  the `Logs_<Client>.Api` call pattern is the ground truth (see the PO-cleanup call-pattern note above).
+
 - **⚠ Inventory-adjustment sync: two shared-engine bugs + it is still SOAP (2026-08-28).** Both fixed
   in `syncInventoryAdjustmentFromNetsuite` (`library/app/api/toga2.php`) and reachable by any client
   running inventory adjustments: (a) the stale-item prune deleted an `InventoryAdjustmentItem` before
@@ -1067,6 +1124,14 @@ the code deploy).
 
 ## Change history
 
+- 2026-09-02 — **Per-record cursor was storing a date-ONLY, US-format value — FIXED + DEPLOYED (shared:
+  `worker` + `library`).** The datetime half was `M/D/YYYY` with no time (e.g. `9/2/2026|6302323`), so the
+  cursor sat at midnight and the same-second id tiebreak never matched — every run re-fetched the whole day.
+  Root cause: `rest.php` SELECTed `lastmodifieddate` raw and NetSuite returns it in the account display format
+  (date-only here). Fixed by `TO_CHAR(lastmodifieddate,'YYYY-MM-DD HH24:MI:SS') AS lastmodifieddate` in the 8
+  cursor-feeding SuiteQL SELECTs, plus normalizing the 6 per-record cursor PUTs to canonical `Y-m-d H:i:s`
+  (`strtotime`+`date`, raw fallback). Old cursors self-heal (midnight → one re-scan → full timestamp).
+  `listOpportunities`/`listItems` share the raw-date trap but are latent (no cursor). (jcardinal)
 - 2026-09-02 — **Per-record resume cursor DEPLOYED to prod (shared: `worker` + `library` + `dbchanges2`).**
   Now live: worker commit `06e8f9ef` + library commit `3a70b59c` (tips of `_production`); rename migration
   `Client/2026-09-02b - NetsuiteSyncCursorRename.sql` run on all client DBs **before** the code. Recorded the
@@ -1100,21 +1165,30 @@ the code deploy).
   1048 "quantityAdjustBy cannot be null" on NetSuite serialized-detail sub-lines. The null-skip reached prod
   ~2026-09-01 afternoon; Issue 542 stopped 09-01 14:08. Recorded that `3-IDLE`-crawl = the same poison-record
   shrink as `1-RUNNING`, not a slow section. (jcardinal)
-- 2026-09-02 — **PURCHASE_ORDERS cleanup DELETE freeze fixed (shared, `library` only — WRITTEN +
-  REVIEWED, NOT yet committed/deployed).** `syncPurchaseOrderFromNetsuite`'s "remove dropped items"
-  cleanup deleted a NetSuite-dropped PO line with a bare throwing DELETE; a leftover item receipt
-  (`ItemReceiptItems.purchaseOrderItemId`, RESTRICT) or SO/TO link → 1451 → EV-11 → threw → froze the
-  whole PURCHASE_ORDERS section at `1-RUNNING` (Elite watermark stuck 2025-10-13, window ÷3 to 1s,
-  poison PO re-tried every 5 min). Fix mirrors the SALES_ORDERS cleanup: unlink all four PO bridge
-  routes, unhook item receipts via `PUT /item-receipt-items/{uuid}` `purchaseOrderItem = null`, then
-  DELETE the PO line kept **throwing** as the fail-loud backstop. **Decision:** the sync mirrors
-  NetSuite — remove our line + its links but keep the real records on the other side; never
-  log-and-skip (a log-and-skip guard was tried this session and rejected as data damage). **Deploy
-  dependency:** `ItemReceiptItems.purchaseOrderItemId` must be nullable
-  (`dbchanges2/Client/2026-06-19a - TransferOrderSchemaFixes.sql` §7) on every client DB first.
-  Recorded the api2 bridge-route key mechanic (plain kebab→camel, no pluralization; PUT writes SQL
-  NULL) and Elite's 78 duplicate PO line-groups (4 with receipts, all tangled → must fail loud).
-  No prod data was modified (read-only SELECTs + NetSuite reads). (jcardinal)
+- 2026-09-02 — **PURCHASE_ORDERS cleanup DELETE freeze fixed + DEPLOYED, plus a follow-up 403 fix
+  (shared, `library` only — committed to `_production`, confirmed live on Elite).**
+  `syncPurchaseOrderFromNetsuite`'s "remove dropped items" cleanup deleted a NetSuite-dropped PO line
+  with a bare throwing DELETE; a leftover item receipt (`ItemReceiptItems.purchaseOrderItemId`,
+  RESTRICT) or SO/TO link → 1451 → EV-11 → threw → froze the whole PURCHASE_ORDERS section at
+  `1-RUNNING` (Elite watermark stuck 2025-10-13, window ÷3 to 1s, poison PO re-tried every 5 min). Fix
+  mirrors the SALES_ORDERS cleanup: unlink all four PO bridge routes, unhook item receipts via
+  `PUT /item-receipt-items/{uuid}` `purchaseOrderItem = null`, then DELETE the PO line kept **throwing**
+  as the fail-loud backstop. **Decision:** the sync mirrors NetSuite — remove our line + its links but
+  keep the real records on the other side; never log-and-skip (a log-and-skip guard was tried this
+  session and rejected as data damage). **Deploy dependency:** `ItemReceiptItems.purchaseOrderItemId`
+  must be nullable (`dbchanges2/Client/2026-06-19a - TransferOrderSchemaFixes.sql` §7) on every client
+  DB first. Recorded the api2 bridge-route key mechanic (plain kebab→camel, no pluralization; PUT writes
+  SQL NULL) and Elite's 78 duplicate PO line-groups (4 with receipts, all tangled → must fail loud).
+  **Follow-up after deploy:** the cleanup's **bridge-clear GET** was itself a throwing `send()`, and a
+  client without an ACL grant on the transfer-order bridge routes (Elite — no transfer orders — Core
+  records 329/330) got **403 EZ-2** on `GET /purchase-order-items-transfer-order-items` and re-froze;
+  the bridge/link GET loop now passes `$throwExceptionOnApiError=false` and **skips** a link table on a
+  no-access response (403 OR EZ-2), while any other failure still throws. Elite PO import unfroze after
+  redeploy. **Two ops lessons:** (1) merging `library` to `_production` does NOT deploy the worker EC2
+  boxes — only a worker deploy does (old code ran for cron cycles after the 14:48 merge); (2) the
+  `Logs_<Client>.Api` call pattern (bridge GETs + item-receipt unhook PUT vs. a bare PO-line DELETE) is
+  the ground truth for whether the fix is live. No prod data was modified (read-only SELECTs + NetSuite
+  reads). (jcardinal)
 - 2026-09-01 — **Customer-PO `dateOrder`/`cost` fixes + NYCHH inventory-adjustment serial-case unfreeze
   (shared, `library` only, working-tree — not yet committed/deployed).** `syncSalesOrderFromNetsuite`
   now sets the customer PO's `dateOrder` from the SO `tranDate` (was always NULL) and its PO lines'

package/knowledge/clients/elite/features/netsuite-togasupply-sync.md

@@ -126,8 +126,8 @@ the trait declares must exist as a column, a `CustomRecordFields` row **and** an
 ## Change history
 
 - 2026-09-02 — **Elite PURCHASE_ORDERS froze on a dropped PO line that still had a receipt** (shared
-  fix, `library` only — WRITTEN + REVIEWED, NOT yet committed/deployed). Elite was the confirmed prod
-  case: `NETSUITE_EXECUTION_MODE_PURCHASE_ORDERS='1-RUNNING'`, watermark frozen at **2025-10-13**,
+  fix, `library` only — now DEPLOYED to `_production`, PO import confirmed unfrozen on Elite). Elite was
+  the confirmed prod case: `NETSUITE_EXECUTION_MODE_PURCHASE_ORDERS='1-RUNNING'`, watermark frozen at **2025-10-13**,
   window collapsed to 1s, same poison PO re-tried every 5 min. Root cause is shared, not Elite —
   `syncPurchaseOrderFromNetsuite`'s cleanup DELETE threw on a leftover item receipt / SO link. Elite
   data behind it: **78 duplicate `(purchaseOrder, vendorItem)` PO line-groups**, only **4** carry a
@@ -138,7 +138,17 @@ the trait declares must exist as a column, a `CustomRecordFields` row **and** an
   (double-count / `UNIQUE(itemReceiptId, purchaseOrderItemId)` collision) — tangled cases fail loud
   for a human, only clean cases self-heal. **Before deploy:** confirm
   `ItemReceiptItems.purchaseOrderItemId` is nullable on `Client_Elite`
-  (`dbchanges2/Client/2026-06-19a - TransferOrderSchemaFixes.sql` §7). Full mechanics on the
+  (`dbchanges2/Client/2026-06-19a - TransferOrderSchemaFixes.sql` §7). **Follow-up after deploy:** a
+  second freeze appeared — the cleanup's **bridge-clear GET** was a throwing `send()`, and Elite has
+  **no ACL grant** on the transfer-order bridge Core records (**329**
+  `purchase-order-items-transfer-order-items`, **330** `transfer-order-items-purchase-order-items`)
+  because Elite doesn't use transfer orders, so `GET /purchase-order-items-transfer-order-items`
+  returned **403 EZ-2** (roleIds [1,3]) and aborted the section. Fixed by making the bridge GET loop
+  tolerate a no-access response (`$throwExceptionOnApiError=false`, skip that link table on 403/EZ-2,
+  else throw); Elite unfroze after redeploy. Note: merging `library` to `_production` did **not**
+  deploy the worker boxes — an actual worker deploy was required, and the `Logs_Elite.Api` call
+  pattern (bridge GETs + item-receipt unhook PUT vs. a bare PO-line DELETE) is how you confirm the fix
+  is live. Full mechanics on the
   [engine doc](../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md). (jcardinal)
 - 2026-09-01 — **Drop-ship / special-order POs were never importing for Elite** (e.g. PO tranid
   128238). Root cause was in shared code, not Elite: `listPurchaseOrders()` filtered the PO header by

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.752",
+  "version": "1.0.754",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",