toga-ai 1.0.710 → 1.0.711

package/knowledge/1.0/apps/worker/INDEX.md

@@ -17,5 +17,6 @@
 | [Staples cXML Order Import (SFTP → NetSuite)](features/staples-cxml-order-import.md) | `sync_staples_cxml.php` is an **hourly** cron (runs at **:45**) that imports Staples cXML purchase orders from SFTP into NetSuite as Sales Orders, then writes a | worker/crons/sync/staples/sync_staples_cxml.php |
 | [Diagnosing frozen 1.0 worker cron check-ins (Sentry "missed" flood)](workflows/diagnosing-frozen-cron-checkins.md) | When 1.0 worker cron timestamps freeze and Sentry project `worker1` fills with **`missed`** check-ins, the intuitive diagnosis — a wedged `App_Framework::isProc | worker/.ebextensions/cron.config, library/app/worker.php |
 | [isFulfillable Multi-Client Backfill (all togasupply clients)](workflows/isfulfillable-multi-client-backfill.md) | One-time backfill that catches up `Items.isFulfillable` on **existing** items across **all 17 togasupply clients** (AIG, Broward Sheriff, Canon, Endeavor Health | worker/crons/toga2/netsuite/backfill_isfulfillable_all_clients.php, library/app/api/toga2.php |
+| [Deduplicating NetSuite duplicate Items across client DBs (recursive MERGE)](workflows/netsuite-duplicate-items-dedup.md) | A one-time, per-client data cleanup that collapses **duplicate `Items` rows sharing a `partNumber`** in each NetSuite → TOGa Supply client DB down to a single c | library/app/api/toga2.php, worker/crons/toga2/netsuite/common_sync_togasupply.php |
 | [Onboarding a Client to the NetSuite TOGa Supply Sync](workflows/onboarding-client-to-netsuite-togasupply-sync.md) | How to add a new TOGa 2 client to the per-client NetSuite → TOGa Supply importer (`worker/crons/toga2/netsuite/`). | worker/crons/toga2/netsuite/sync_togasupply.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_elite.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql, dbchanges2/_modules/netsuite/2026-08-05 - CLEAN NETSUITE CLINET.SQL |
 | [Tracing a 1.0 worker cron run in production (no stdout, silent skips)](workflows/tracing-a-worker-cron-run-in-production.md) | How to answer *"did this cron actually run, and what did it do?"* on the 1.0 `worker` tier, where **there is no usable stdout** and **a skipped run leaves no tr | worker/ebs/cron.worker.php, worker/schedules/cron.worker.sync.json, worker/schedules/cron.worker.infrastructure.json, worker/.ebextensions/009_setup_phpini.config, library/app/framework.php |

package/knowledge/1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md

@@ -755,9 +755,55 @@ Parameters are stored **per client DB** but accessed **through the TOGa2 API**,
   2**) and scope the `Units` lookup by it (returns exactly 1 → reuse). Latent data problem behind it:
   **duplicate `Items` with the same `partNumber`** should be reconciled later (the fix is robust to it).
   GroWrk ITEM_FULFILLMENTS went from stuck `1-RUNNING` (since Aug 21) to green.
+  - **⚠ SUPERSEDED 2026-08-31 — item-scoping ALONE was WRONG for NYCHH.** Where the serial's single
+    existing unit sits under a **different** duplicate `Item` sharing the part number, scoping the `Units`
+    lookup by `$uuidItemForLine` found nothing and re-created the unit → the SAME 1062 collision. Correct
+    rule: **a serial identifies a physical device, so query `Units` by `serialNumber` ONLY**, then *prefer*
+    the unit whose item matches `$uuidItemForLine` when several share the serial (GroWrk's 2-unit case),
+    else reuse the first; create only when **no** unit carries the serial. Handles the 1-unit (NYCHH),
+    2-unit (GroWrk), and new-serial cases. (The duplicate `Items` themselves are the real defect — see the
+    Agilant-catalog root cause and the dedup workflow below.)
+
+- **⚠ Invoice line numbers are NOT unique on multi-SO invoices — assign a sequential `lineNumber`
+  (fixed 2026-08-31).** A NetSuite invoice that bills **multiple sales orders** restarts `line` per source
+  SO, so `$nsItem->line` is not unique within one invoice → `InvoiceItems UNIQUE(invoiceId, lineNumber)` →
+  `POST /v2/invoices` `EV-10` / MySQL **1062**, and because the invoice POST is **atomic** the whole
+  invoice rolls back, freezing `INVOICES` at `1-RUNNING` (NYCHH). **Fix (`syncInvoiceFromNetsuite`):**
+  assign a unique **1..N sequential** `lineNumber` per imported line instead of `$nsItem->line` (identical
+  for the common single-SO invoice; differs only on multi-SO). Safe because nothing downstream keys off
+  invoice `lineNumber` — the SO link uses `orderLine`/`salesOrderItem`.
+- **⚠ Transfer-order invoices are NOT imported (fixed 2026-08-31).** Under NYCHH stocking-flag routing a
+  $0 NetSuite Sales Order becomes a `TransferOrder`, not a `SalesOrder`; `Invoices` has only
+  `salesOrderId` (no `transferOrderId`), so such an invoice imported with a **null order link**. Team
+  decision: **do not import transfer-order invoices at all.** **Fix:** in the "billed order not found in
+  SalesOrders" branch, classify the order authoritatively (`App_Api_Netsuite_Rest::fetchSalesOrderByIdFull`
+  + `self::isTransferOrder`) and **return early** when it is a transfer order, gated on
+  `IS_ENABLED_INTEGRATION_TRANSFER_ORDERS`. Existing TO-billed invoices were deleted by the developer.
+- **⚠ ROOT CAUSE of the duplicate `Items`: the inventory-adjustment item-create OMITTED the catalog
+  (fixed 2026-08-31).** `getCreateItem` (`toga2.php:6233`) and the fulfillment create (~4951) stamp
+  `'catalog' => ['name' => 'Agilant']`; `syncInventoryAdjustmentFromNetsuite`'s item-create did **not** →
+  it produced **NULL-catalog `Items`**. The shared item lookup (`common_sync_togasupply.php` ~L313) is
+  populated **only from Agilant-catalog items**, so those NULL-catalog dupes are **invisible** to it → a
+  NEW duplicate item was created **every run** for the same part number — the source of every
+  `(itemId, serialNumber)` unit collision seen this session. **Fix:** stamp
+  `'catalog' => ['name' => 'Agilant']` on the inventory-adjustment item-create. Stops all new duplicates.
+  The one-time cleanup of the accumulated ~4,300 duplicate part numbers across all 17 clients is the
+  [NetSuite duplicate-Items dedup workflow](../workflows/netsuite-duplicate-items-dedup.md).
 
 ## Change history
 
+- 2026-08-31 — **Invoice + serial-unit + duplicate-Item root-cause fixes (shared, `library`).**
+  `syncInvoiceFromNetsuite` now assigns a **sequential 1..N invoice `lineNumber`** (NetSuite reuses `line`
+  across multi-SO invoices → `InvoiceItems UNIQUE(invoiceId,lineNumber)` 1062 → atomic rollback → NYCHH
+  INVOICES `1-RUNNING`) and **skips transfer-order invoices** (classify via `fetchSalesOrderByIdFull` +
+  `isTransferOrder`, return early, gated on `IS_ENABLED_INTEGRATION_TRANSFER_ORDERS` — `Invoices` has no
+  `transferOrderId`; team decision not to import them). **Corrected** the 2026-08-30 item-scoped serial
+  unit lookup (which broke NYCHH): a serial identifies a physical device, so query `Units` by
+  `serialNumber` ONLY, prefer the unit matching `$uuidItemForLine` when several share it, else reuse first,
+  create only when none exists. Found + fixed the **root cause of the duplicate `Items`**: the
+  inventory-adjustment item-create omitted `catalog => Agilant`, producing NULL-catalog items invisible to
+  the Agilant-only sync lookup → a new dup every run. Companion one-time cleanup across all 17 clients:
+  [NetSuite duplicate-Items dedup workflow](../workflows/netsuite-duplicate-items-dedup.md). (jcardinal)
 - 2026-08-30 — **Inventory-adjustment serialized-unit path now auto-creates a missing unit warehouse
   (`library`).** `syncInventoryAdjustmentFromNetsuite` took the unit's location from the inventory
   number's `custitemnumber_location` but never created a missing warehouse (unlike

package/knowledge/1.0/apps/worker/workflows/netsuite-duplicate-items-dedup.md

@@ -0,0 +1,102 @@
+---
+title: "Deduplicating NetSuite duplicate Items across client DBs (recursive MERGE)"
+framework: "1.0"
+repo: worker
+project: Worker
+client: shared
+type: workflow
+status: active
+updated: 2026-08-31
+owners: [jcardinal]
+files:
+  - library/app/api/toga2.php
+  - worker/crons/toga2/netsuite/common_sync_togasupply.php
+related:
+  - ../features/netsuite-togasupply-per-client-sync.md
+  - ../../library/features/netsuite-suiteql-rest-shim.md
+---
+
+## Summary
+
+A one-time, per-client data cleanup that collapses **duplicate `Items` rows sharing a `partNumber`**
+in each NetSuite → TOGa Supply client DB down to a single canonical item, re-pointing every reference.
+Duplicates accumulated because the NetSuite inventory-adjustment sync created **NULL-catalog `Items`**
+that the sync's own lookup could not see, so a new duplicate was minted every run (root cause below).
+Deleting a duplicate item is a **recursive MERGE**, not a simple `DELETE` — ~23 `itemId` tables plus
+two FK cascades (`VendorItems`, `Units`) reference it. Run **per client, in its own transaction**,
+after the code fix is deployed.
+
+**Verified in prod (2026-08-31):** all **17** NetSuite clients now show **0 duplicate part numbers**
+(e.g. `Client_Nychh` 3954→1818 items, `Client_Canon` 913→425; 0 orphans, 0 collisions).
+
+## Root cause — why the duplicates existed at all
+
+`syncInventoryAdjustmentFromNetsuite`'s item-create **omitted the catalog** that
+`getCreateItem`/`toga2.php:6233` and the fulfillment create both stamp
+(`'catalog' => ['name' => 'Agilant']`), so it produced **NULL-catalog `Items`**. The shared sync item
+lookup in `worker/crons/toga2/netsuite/common_sync_togasupply.php` (~L313) is populated **only from
+Agilant-catalog items**, so a NULL-catalog dupe is **invisible** to it → the sync re-created the item
+every run for the same part number. **The fix (stamp `catalog => Agilant` on that create) stops all NEW
+duplicates** and must be deployed first — see
+[per-client sync](../features/netsuite-togasupply-per-client-sync.md). This cleanup removes the
+**existing** backlog (~4,300 duplicate part numbers / ~15,700 `Items` rows accumulated over time).
+
+## When to run
+
+- After the Agilant-catalog item-create fix is deployed (otherwise the backlog regrows every sync run).
+- Per client, once. Re-runnable safely (idempotent: a client already at 0 duplicates is a no-op).
+
+## Procedure
+
+1. **Report first (read-only).** Run `test/team/2.0 deployment/duplicate_items_merge_analysis.sql`
+   against the client DB — a per-client duplicate + collision report. Confirm the duplicate count and the
+   `(itemId, serialNumber)` / `(vendorId, itemId, vendorPartNumber)` collisions before mutating anything.
+2. **Pick the canonical item per `partNumber`** in this priority order: **Agilant + manufacturer**
+   preferred → else **Agilant** → else **has-manufacturer** → else **lowest `id`**.
+3. **Re-point every `itemId` reference** across the ~23 `itemId` tables **plus `Entitlements.saleItemId`**
+   from each non-canonical item to the canonical one.
+4. **Handle the two FK cascades** (children must move before their parent duplicate is deleted):
+   - **`VendorItems`** is referenced by `PurchaseOrderItems`, `VendorItemCosts`, `Personas_VendorItems`.
+   - **`Units`** is referenced by **10** `*ItemUnits` child tables.
+5. **Resolve unique-key collisions with a SURVIVOR-per-key approach**, NOT a plain "`UPDATE IGNORE` +
+   delete leftover" (that leaves child FKs dangling on dupe-vs-dupe collisions). For each colliding key
+   (`Units.(itemId, serialNumber)`, `VendorItems.(vendorId, itemId, vendorPartNumber)`): choose one
+   survivor row, **re-point the non-survivor rows' children onto the survivor**, delete the non-survivors,
+   then re-point the rest.
+6. **Delete the duplicate `Items` rows LAST** (after all references and cascades are moved).
+7. **Run each client in its own transaction**, continue-on-error (see the missing-table gotcha).
+
+## Validation
+
+- Validate against **local copies of the client DBs before prod** (this session: `Client_Nychh`
+  3954→1818, `Client_Canon` 913→425, **0 orphans, 0 collisions**).
+- After each prod run, re-run the analysis SQL: the client must report **0 duplicate part numbers**.
+
+## Deliverables (live in the repo, not the knowledge base)
+
+- `test/team/2.0 deployment/duplicate_items_merge_analysis.sql` — read-only per-client duplicate +
+  collision report.
+- The per-client merge migration (the recursive MERGE implementing steps 2–6).
+
+## Gotchas / known issues
+
+- **The sync item lookup is Agilant-catalog-only.** Any non-Agilant item is **invisible** to
+  `common_sync_togasupply.php` and will be **re-created** — this is the whole mechanism behind the
+  duplicates. Deploy the catalog-stamp fix before (and independently of) this cleanup.
+- **`Items` has NO unique constraint on `partNumber`.** Duplicates persist silently; nothing at the DB
+  level prevents them, which is why this is a MERGE-and-clean rather than a rely-on-the-constraint fix.
+- **Client DBs are schema-similar but NOT identical.** Some lack optional tables (e.g.
+  `EntitlementServiceItems`). Run the merge **continue-on-error** so a missing-table MySQL **1146** is a
+  **benign skip**, not a failed migration.
+- **A plain `UPDATE IGNORE` + delete-leftover is insufficient.** Dupe-vs-dupe unique-key collisions leave
+  orphaned child FKs; only the survivor-per-key re-pointing (step 5) is correct.
+
+## Change history
+
+- 2026-08-31 — Documented the one-time duplicate-`Items` dedup across all 17 NetSuite clients (recursive
+  MERGE over ~23 `itemId` tables + `Entitlements.saleItemId`, `VendorItems`/`Units` FK cascades,
+  survivor-per-key collision resolution), its root cause (the inventory-adjustment item-create omitted
+  `catalog => Agilant` → NULL-catalog items invisible to the Agilant-only sync lookup → a new dup every
+  run), and the durable gotchas (Agilant-only lookup, no `partNumber` unique constraint, per-client schema
+  drift → continue-on-error). Validated on local copies then run per client in prod; all 17 clients at 0
+  duplicate part numbers. (jcardinal)

package/knowledge/INDEX.md

@@ -5,7 +5,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 ## 1.0 framework
 
 - **library** (Library) _(framework core)_ — 23 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
-- **worker** (Worker) — 31 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
+- **worker** (Worker) — 32 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
 - **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
 - **worker1.5** (Worker 1.5) — 0 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
 - **togadesk** (TOGa Desk) — 13 doc(s) → [1.0/apps/togadesk/INDEX.md](1.0/apps/togadesk/INDEX.md)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.710",
+  "version": "1.0.711",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",