toga-ai 1.0.720 → 1.0.722
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/1.0/apps/worker/workflows/netsuite-duplicate-items-dedup.md +33 -1
- package/knowledge/clients/compass-canada/profile.md +7 -1
- package/knowledge/clients/compass-usa/INDEX.md +2 -1
- package/knowledge/clients/compass-usa/features/item-catalogs-and-duplicate-items.md +51 -3
- package/knowledge/clients/compass-usa/features/mits-sales-order-transmission-alerting.md +33 -1
- package/knowledge/clients/compass-usa/profile.md +1 -0
- package/knowledge/clients/compass-usa/workflows/catalog-fold-reversal.md +120 -0
- package/package.json +1 -1
|
@@ -6,7 +6,7 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: workflow
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-
|
|
9
|
+
updated: 2026-09-01
|
|
10
10
|
owners: [jcardinal]
|
|
11
11
|
files:
|
|
12
12
|
- library/app/api/toga2.php
|
|
@@ -14,6 +14,8 @@ files:
|
|
|
14
14
|
related:
|
|
15
15
|
- ../features/netsuite-togasupply-per-client-sync.md
|
|
16
16
|
- ../../library/features/netsuite-suiteql-rest-shim.md
|
|
17
|
+
- ../../../clients/compass-usa/workflows/catalog-fold-reversal.md
|
|
18
|
+
- ../../../clients/compass-usa/features/item-catalogs-and-duplicate-items.md
|
|
17
19
|
---
|
|
18
20
|
|
|
19
21
|
## Summary
|
|
@@ -29,6 +31,16 @@ after the code fix is deployed.
|
|
|
29
31
|
**Verified in prod (2026-08-31):** all **17** NetSuite clients now show **0 duplicate part numbers**
|
|
30
32
|
(e.g. `Client_Nychh` 3954→1818 items, `Client_Canon` 913→425; 0 orphans, 0 collisions).
|
|
31
33
|
|
|
34
|
+
> **⚠ CORRECTION (2026-09-01): "0 duplicate part numbers" is NOT proof of a clean run for a
|
|
35
|
+
> multi-catalog client.** This merge is **catalog-blind** — it groups on `UPPER(partNumber)` only.
|
|
36
|
+
> For **Compass**, the same part number **legitimately exists once per catalog** (catalog 1 =
|
|
37
|
+
> store, catalog 2 = Agilant), so the merge folded **199 intentional catalog-1 store items into
|
|
38
|
+
> their Agilant canonical**, broke MITS transmission platform-wide, and had to be reversed. Before
|
|
39
|
+
> running this on any client, confirm the client is **single-catalog**; if not, the dedup must be
|
|
40
|
+
> **catalog-aware** (dedup within a catalog, never across). See the Compass
|
|
41
|
+
> [catalog-fold reversal](../../../clients/compass-usa/workflows/catalog-fold-reversal.md) and
|
|
42
|
+
> [Compass item catalogs & duplicate items](../../../clients/compass-usa/features/item-catalogs-and-duplicate-items.md).
|
|
43
|
+
|
|
32
44
|
## Root cause — why the duplicates existed at all
|
|
33
45
|
|
|
34
46
|
`syncInventoryAdjustmentFromNetsuite`'s item-create **omitted the catalog** that
|
|
@@ -90,9 +102,29 @@ duplicates** and must be deployed first — see
|
|
|
90
102
|
**benign skip**, not a failed migration.
|
|
91
103
|
- **A plain `UPDATE IGNORE` + delete-leftover is insufficient.** Dupe-vs-dupe unique-key collisions leave
|
|
92
104
|
orphaned child FKs; only the survivor-per-key re-pointing (step 5) is correct.
|
|
105
|
+
- **⚠ The merge is CATALOG-BLIND — it does NOT handle multi-catalog clients.** It groups on
|
|
106
|
+
`UPPER(partNumber)` only, with an Agilant-preferred canonical rule. For a client that keeps the same
|
|
107
|
+
part number in two intentional catalogs (Compass: store vs Agilant), it folds legitimate per-catalog
|
|
108
|
+
twins together and **destroys** the store items. Confirm a client is single-catalog before running;
|
|
109
|
+
otherwise make the dedup catalog-scoped. The Compass fold deleted 199 store items and broke MITS —
|
|
110
|
+
see the [catalog-fold reversal](../../../clients/compass-usa/workflows/catalog-fold-reversal.md).
|
|
111
|
+
- **`UPDATE IGNORE ...; DELETE ... WHERE fk=dupe` permanently DELETES collision rows** and drops its
|
|
112
|
+
helper maps, so prod cannot self-reverse. A reversal needs a **pre-merge snapshot** restored as a
|
|
113
|
+
scratch DB — plan the backup before running any variant of this.
|
|
114
|
+
- **Blast radius of the 2026-08-31 `duplicate_items_merge_ALL_CLIENTS.sql`:** it ran on 17 clients
|
|
115
|
+
(Masonite, Ncci, Erau, Shrss, Wmchealth, Trividiahealth, Prudential, Miamidade, Browardsheriff,
|
|
116
|
+
Nychh, Canon, Quad, Compass, Elite, Growrk, Spglobal, Endeavorhealth). At-risk = two-catalog +
|
|
117
|
+
TOGa Commerce clients like Compass; a **per-client assessment is outstanding**.
|
|
93
118
|
|
|
94
119
|
## Change history
|
|
95
120
|
|
|
121
|
+
- 2026-09-01 — **⚠ Corrected the success claim.** "0 duplicate part numbers" is NOT a clean-run proof
|
|
122
|
+
for a multi-catalog client. This merge is catalog-blind (groups on `UPPER(partNumber)` only), so for
|
|
123
|
+
Compass it folded 199 intentional catalog-1 store items into their Agilant canonical, broke MITS
|
|
124
|
+
transmission platform-wide, and had to be reversed from a pre-merge snapshot. Added the
|
|
125
|
+
confirm-single-catalog precondition, the permanent-delete / no-self-reverse warning, and the
|
|
126
|
+
17-client blast radius with the outstanding per-client assessment. See the Compass
|
|
127
|
+
[catalog-fold reversal](../../../clients/compass-usa/workflows/catalog-fold-reversal.md). (jcardinal)
|
|
96
128
|
- 2026-08-31 — Documented the one-time duplicate-`Items` dedup across all 17 NetSuite clients (recursive
|
|
97
129
|
MERGE over ~23 `itemId` tables + `Entitlements.saleItemId`, `VendorItems`/`Units` FK cascades,
|
|
98
130
|
survivor-per-key collision resolution), its root cause (the inventory-adjustment item-create omitted
|
|
@@ -16,7 +16,7 @@ project: _Underscore
|
|
|
16
16
|
client: compass-canada
|
|
17
17
|
type: profile
|
|
18
18
|
status: active
|
|
19
|
-
updated: 2026-
|
|
19
|
+
updated: 2026-09-01
|
|
20
20
|
owners: [jcardinal, bala, tcox, apeterson]
|
|
21
21
|
files: []
|
|
22
22
|
related:
|
|
@@ -190,4 +190,10 @@ to but distinct from Compass USA. Like Compass USA it spans the **2.0** commerce
|
|
|
190
190
|
8 Manager. USA grants the fulfillment records to roles 3 + 7; Canada grants most of them to role 1
|
|
191
191
|
only. Nearly every logged-in Canada user carries role 1, so this is usually invisible — but ~10
|
|
192
192
|
users hold Compass Base without Base and still hit the gap.
|
|
193
|
+
- **Canada was NOT damaged by the 2026-08-31 cross-client item merge, and is structurally
|
|
194
|
+
immune.** It was **not** in the `duplicate_items_merge_ALL_CLIENTS.sql` client list; it has a
|
|
195
|
+
**single catalog** ("compass"), 0 part numbers in multiple catalogs, and 0 duplicate part
|
|
196
|
+
numbers — because it does **not** use NetSuite, so it never hit the inventory-adjustment
|
|
197
|
+
dupe-item bug that fed the merge. The catalog-fold damage and reversal were Compass USA only.
|
|
198
|
+
See [Compass USA catalog-fold reversal](../compass-usa/workflows/catalog-fold-reversal.md).
|
|
193
199
|
- Related: [Compass USA](../compass-usa/profile.md).
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
| [Compass ASN → ItemFulfillment Auto-Creation](features/asn-to-item-fulfillment.md) | 2.0 | For Compass USA, posting an AdvanceShippingNotice (ASN) auto-creates the ItemFulfillment (IF) on the upstream SalesOrder. | _underscore/Model/Compass/AdvanceShippingNotice.php, _underscore/Model/Compass/PurchaseOrder.php, api2/Component/Api/Cxml/Cxml.php, dbchanges2/Client_Compass/2026-06-11 - AsnItemTrackingNumberAcl.sql, dbchanges2/Client_Compass/2026-06-15b - BackfillSA132781ItemFulfillmentTracking.sql, dbchanges2/Client_Compass/2026-06-16 - CleanupSA132763CrossLineTracking.sql, dbchanges2/Client_Compass/2026-06-16b - CleanupSA132743CrossLineTracking.sql, dbchanges2/Client_Compass/2026-06-16c - BackfillSA132763C40QYUCTracking.sql, dbchanges2/Client_Compass/2026-06-18a - CleanupSA132898DuplicateTracking.sql, dbchanges2/Client_Compass/2026-06-18b - CleanupSA132881DuplicateTracking.sql, dbchanges2/Client_Compass/2026-07-02a - FixSA133377TrackingSerialAndDuplicateIF.sql |
|
|
7
7
|
| [Cost Centers — Unit Locations, numeric-only policy](features/cost-centers.md) | 2.0 | A Compass "cost center" — the value a user picks in commerce and that lands on an order — is **not** a `CostCenters` row. | toga2-commerce/src/pages/Cart/api/CartApi.ts, worker1.5/crons/toga2/compass/import_locations.php, _underscore/Model/Compass/SalesOrder.php, api2/Component/Api/V2/V2.php, dbchanges2/Client_Compass/2026-07-06 - RemoveNonNumericCostCenters.sql |
|
|
8
8
|
| [Compass isFulfillable — Data Quality & the Type-Derived Rule](features/isfulfillable-data-quality-and-type-rule.md) | 2.0 | A **prod-data investigation (2026-08-12, read-only)** into why so many Compass storefront lines still have a dead **Qty Fulfilled** cell. | _underscore/Model/Client/Item.php, library/app/api/toga2.php, worker/crons/toga2/netsuite/backfill_isfulfillable_all_clients.php, dbchanges2/Core/2026-07-17 - RegisterItemIsFulfillableInterceptors.sql, toga2-supply/src/pages/Orders/view/OrderView/components/sections/OrderItemsTableSection.tsx |
|
|
9
|
-
| [Compass Item Catalogs (Compass vs Agilant) & the Duplicate Items the NetSuite Sync Creates](features/item-catalogs-and-duplicate-items.md) | 1.0 | `Client_Compass.Items` is **split across catalogs by which pipeline created the item**, and the two halves are routinely mistaken for one. | library/app/api/toga2.php, _underscore/Model/Compass/PurchaseOrder.php, _underscore/Model/Compass/Usa/PurchaseOrder.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/compass/workflow/3a_import_office_depot_purchase_orders.php |
|
|
9
|
+
| [Compass Item Catalogs (Compass vs Agilant) & the Duplicate Items the NetSuite Sync Creates](features/item-catalogs-and-duplicate-items.md) | 1.0 | `Client_Compass.Items` is **split across catalogs by which pipeline created the item**, and the two halves are routinely mistaken for one. | library/app/api/toga2.php, _underscore/Model/Compass/PurchaseOrder.php, _underscore/Model/Compass/Usa/PurchaseOrder.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/compass/workflow/3a_import_office_depot_purchase_orders.php, worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php |
|
|
10
10
|
| [Compass: Item-Fulfillment TableViews (for-sales-order-items & for-sales-orders, tracking via bridge)](features/item-fulfillment-tracking-tableview.md) | 2.0 | Two sibling Compass TableViews in `Client_Compass` display fulfilled items in toga2-supply, both driven by `TableViews` / `TableViewJoins` / `TableViewFields` c | dbchanges2/Client_Compass/2026-06-10 - ItemFulfillmentsForSalesOrderItemsTableView.sql, dbchanges2/Client_Compass/2026-06-11 - ItemFulfillmentsForSalesOrdersTableView.sql, dbchanges2/Client_Compass/2026-06-15a - FixItemFulfillmentTrackingNumberJoins.sql, dbchanges2/Client/2026-07-15a - ExcludeFeeItemsFromItemFulfillmentsForSalesOrdersView.sql |
|
|
11
11
|
| [Compass MITS PO → SO Item Linking](features/mits-po-to-so-item-linking.md) | 2.0 | MITS sends Compass inbound Purchase Orders (`POST /v2/purchase-orders`) against a Sales Order (`mitsSalesOrder`). | _underscore/Model/Compass/PurchaseOrder.php, _underscore/Model/Compass/Usa/PurchaseOrder.php, api2/Component/Api/V2/V2.php, worker/crons/toga2/compass/workflow/3a_import_office_depot_purchase_orders.php |
|
|
12
12
|
| [Compass MITS PO Transmission to Vendors](features/mits-po-transmission-to-vendors.md) | 2.0 | The 1.0 worker cron `2_transmit_mits_purchase_orders_to_vendors.php` transmits Compass PurchaseOrders to their vendors (Office Depot, Strategic Systems, Compass | worker/crons/toga2/compass/workflow/2_transmit_mits_purchase_orders_to_vendors.php, worker/crons/toga2/compasscanada/workflow/2_transmit_mits_purchase_orders_to_vendors.php, library/app/client/compass.php |
|
|
@@ -22,6 +22,7 @@
|
|
|
22
22
|
| [Compass Sales-Order Line-Number Renumbering (and why it hangs off the SO hooks)](features/sales-order-line-renumbering.md) | 2.0 | Compass sales-order lines must stay numbered **1..N with no gaps** after any add, edit, or delete — downstream MITS/PO linking reads `lineNumber` as an identity | _underscore/Model/Compass/SalesOrderItem.php, _underscore/Model/Compass/SalesOrder.php |
|
|
23
23
|
| [Stranded Approval Reassignment (repointing approvals off dead duplicate Compass Users rows)](features/stranded-approval-reassignment.md) | 2.0 | A Compass employee who leaves and comes back after more than `CONTACT_UNLINK_GRACE_DAYS` (5) is **inserted as a brand-new `Users` row** by the PEOPLE importer i | _underscore/Model/Compass/ApprovalDecision.php, worker2/Worker/Client/Compass/ApprovalReassignment.php, worker2/Worker/Client/Compass/PeopleFile.php |
|
|
24
24
|
| [Compass USA](profile.md) | 2.0 | Compass USA is a TOGA client running a multi-tier supply-chain commerce operation. | |
|
|
25
|
+
| [Compass USA — reversing the 199-item catalog fold (MITS outage recovery)](workflows/catalog-fold-reversal.md) | 2.0 | The **root cause** of the Compass MITS transmission outage (the `assetType`-NULL and duplicate-vendor symptoms) was an ad-hoc cross-client **duplicate-items mer | test/team/2.0 deployment/duplicate_items_merge_ALL_CLIENTS.sql, test/team/2.0 deployment/duplicate_items_merge_analysis.sql, worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php, library/app/api/toga2.php, worker/crons/toga2/netsuite/common_sync_togasupply.php |
|
|
25
26
|
| [Compass Cross-Kit Bundle Corruption — Detection & Repair](workflows/cross-kit-bundle-corruption.md) | 2.0 | A frontend regression in `toga2-commerce`'s edit-order bundle submission mis-attributed bundle (kit) line items and **fees/warranties** to the **wrong kit**, pe | src/api/syncSalesOrderItemsFromLocalStorageCartToApi.ts |
|
|
26
27
|
| [Granting a Compass user the same toga25-supply navigation as another user (role vs. SurfaceOverride)](workflows/granting-navigation-access.md) | 2.0 | "Give user X the same menu items user Y has" is the **navigation analog** of the recurring [bundle/persona grant](./granting-persona-bundle-access.md) request. | dbchanges2/Client_Compass/2026-08-12a - NavigationRoleGrants.sql, dbchanges2/Client_Compass/2026-02-25 - AgilantUsersViewAccess.sql, dbchanges2/Client_Compass/2026-08-11 - CompassODPViewVendorItemsAccess.sql, dbchanges2/Core/2026-08-12a - NavigationSurfaceSeed.sql |
|
|
27
28
|
| [Granting a Compass user access to a bundle (persona grant) and the SuperUser role](workflows/granting-persona-bundle-access.md) | 2.0 | "Give user X sight of kit N" is a **recurring** Compass request, usually paired with "and make them a super user". | dbchanges2/Client_Compass/2026-08-26a - CompassCreativeStudioPersona.sql |
|
|
@@ -7,19 +7,22 @@ client: compass-usa
|
|
|
7
7
|
type: client-feature
|
|
8
8
|
status: active
|
|
9
9
|
updated: 2026-09-01
|
|
10
|
-
owners: ["bala"]
|
|
10
|
+
owners: ["bala", "jcardinal"]
|
|
11
11
|
files:
|
|
12
12
|
- library/app/api/toga2.php
|
|
13
13
|
- _underscore/Model/Compass/PurchaseOrder.php
|
|
14
14
|
- _underscore/Model/Compass/Usa/PurchaseOrder.php
|
|
15
15
|
- worker/crons/toga2/netsuite/common_sync_togasupply.php
|
|
16
16
|
- worker/crons/toga2/compass/workflow/3a_import_office_depot_purchase_orders.php
|
|
17
|
+
- worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php
|
|
17
18
|
related:
|
|
18
19
|
- odp-edi-850-item-resolution.md
|
|
19
20
|
- mits-po-to-so-item-linking.md
|
|
21
|
+
- mits-sales-order-transmission-alerting.md
|
|
20
22
|
- isfulfillable-data-quality-and-type-rule.md
|
|
21
23
|
- ../workflows/odp-duplicate-po-line-cleanup.md
|
|
22
24
|
- ../workflows/odp-order-pipeline-to-netsuite.md
|
|
25
|
+
- ../workflows/catalog-fold-reversal.md
|
|
23
26
|
- ../profile.md
|
|
24
27
|
- ../../../1.0/apps/library/features/netsuite-item-isfulfillable-sync.md
|
|
25
28
|
- ../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md
|
|
@@ -62,6 +65,26 @@ the same list — they are **two different source systems' views** of overlappin
|
|
|
62
65
|
(`vendorId = 1`) point at **catalog-2** items. Filtering the ODP item lookup to catalog 1 therefore
|
|
63
66
|
breaks a large slice of the Office Depot pipeline outright.
|
|
64
67
|
|
|
68
|
+
## Customer → catalog rule (which catalog an order resolves to)
|
|
69
|
+
|
|
70
|
+
Confirmed by the developer and by pre-merge data (2026-08-31):
|
|
71
|
+
|
|
72
|
+
- **`customerId = 2` (Compass) sales orders → catalog 1 (Compass/store) items.**
|
|
73
|
+
- Everyone else — **Office Depot `customerId = 1`, Agilant `customerId = 3`** → catalog 2
|
|
74
|
+
(Agilant) items.
|
|
75
|
+
|
|
76
|
+
Pre-merge snapshot: `customerId = 2` order lines were ~100% on catalog 1 (**182,787 vs 1**).
|
|
77
|
+
The store (catalog-1) item is the one that carries the MITS classification
|
|
78
|
+
(`assetType`/`category`/`type`); a Compass line that resolves to a catalog-2 Agilant item has
|
|
79
|
+
NULL classification and **MITS rejects the whole order** (HTTP 400 "The AssetType field is
|
|
80
|
+
required"). So this mapping is load-bearing.
|
|
81
|
+
|
|
82
|
+
**⚠ Every catalog re-point MUST be scoped by `customerId`.** An ad-hoc
|
|
83
|
+
`UPDATE ... WHERE itemId = <agilantId>` with no customer filter wrongly moved **687
|
|
84
|
+
Office-Depot lines** onto a Compass item and had to be corrected. This is the durable lesson
|
|
85
|
+
from the 2026-08-31 catalog-fold reversal — see
|
|
86
|
+
[catalog-fold reversal](../workflows/catalog-fold-reversal.md).
|
|
87
|
+
|
|
65
88
|
## Where the "Agilant" catalog is hardcoded
|
|
66
89
|
|
|
67
90
|
The 1.0 side writes and reads catalog 2 **by name**, in four places — none of them configurable:
|
|
@@ -93,13 +116,21 @@ Plus `-1` / `-2` suffixed copies: items **2431, 2432, 2443, 2653**.
|
|
|
93
116
|
**Two distinct duplicate shapes, and only the first exists in prod today:**
|
|
94
117
|
- **Different part numbers** (the table above) — the two rows never match on part number, so each
|
|
95
118
|
pipeline keeps using "its own" item and quantities/fulfillments diverge. This is the live shape.
|
|
96
|
-
- **Same part number, two catalogs — NOT present in prod.** This is the shape behind
|
|
97
|
-
[dual-catalog duplicate PO-line bug](../workflows/odp-duplicate-po-line-cleanup.md) (4,886
|
|
119
|
+
- **Same part number, two catalogs — NOT present in prod (2026-09-01).** This is the shape behind
|
|
120
|
+
the [dual-catalog duplicate PO-line bug](../workflows/odp-duplicate-po-line-cleanup.md) (4,886
|
|
98
121
|
duplicate PO lines across 1,320 Office Depot POs when that was investigated).
|
|
99
122
|
**Verified on prod `Client_Compass` 2026-09-01: zero part numbers are duplicated at all, and zero
|
|
100
123
|
span more than one catalog.** It matters only as a thing to re-check, because a part-number lookup
|
|
101
124
|
with no catalog filter has **no ambiguity check on its exact-match branch**
|
|
102
125
|
([850 line-item resolution](odp-edi-850-item-resolution.md)).
|
|
126
|
+
**⚠ Unresolved tension — reconcile before relying on "partNumber is unique."** The
|
|
127
|
+
[2026-08-31 catalog-fold reversal](../workflows/catalog-fold-reversal.md) is recorded as having
|
|
128
|
+
folded **199 catalog-1 items into their catalog-2 Agilant canonical by `UPPER(partNumber)`**,
|
|
129
|
+
which requires those 199 to have shared a part number across the two catalogs pre-merge, and the
|
|
130
|
+
reversal restored them. That does not square with "zero part numbers span two catalogs" verified
|
|
131
|
+
the same day. Confirm on prod whether the restored catalog-1 twins carry **distinct** part numbers
|
|
132
|
+
(e.g. the `#B1H` / `-S` variant forms above) or genuinely-shared ones before treating table-wide
|
|
133
|
+
`partNumber` uniqueness as settled.
|
|
103
134
|
|
|
104
135
|
## "Who changed this part number?" — the audit trail says nobody
|
|
105
136
|
|
|
@@ -143,8 +174,25 @@ children" rule:
|
|
|
143
174
|
- **⚠ Do not "fix" a mismatch by creating an item** so a string matches. That is how the junk items
|
|
144
175
|
got there in the first place — fix the `VendorItems` mapping instead
|
|
145
176
|
([850 line-item resolution](odp-edi-850-item-resolution.md)).
|
|
177
|
+
- **The NetSuite sync UPDATE path never touches `assetType`/`category`/`type`.** `getCreateItem`
|
|
178
|
+
stamps them as NULL on create (so a **new** Agilant item is born unclassified → a Compass line
|
|
179
|
+
on it fails MITS), but the sync UPDATE only PUTs `inventoryType` / `isFulfillable`. Practical
|
|
180
|
+
consequence: once classification is set on an item it is **durable across sync runs** (the sync
|
|
181
|
+
will not revert it) — the exposure is only newly-created items. This is why a store-catalog data
|
|
182
|
+
repair survives, but the Compass NetSuite sync must still be **stopped before** any catalog
|
|
183
|
+
re-point (it re-creates Compass order lines on Agilant items). See
|
|
184
|
+
[catalog-fold reversal](../workflows/catalog-fold-reversal.md).
|
|
146
185
|
|
|
147
186
|
## Change history
|
|
187
|
+
- 2026-09-01 — Folded in the **customer → catalog rule** (`customerId = 2` → catalog 1, everyone
|
|
188
|
+
else → catalog 2; verified 182,787 vs 1 pre-merge) and the mandatory `customerId` scope on any
|
|
189
|
+
catalog re-point (an unscoped one moved 687 Office-Depot lines). Added that the NetSuite sync
|
|
190
|
+
UPDATE path never touches `assetType`/`category`/`type` (so classification is durable once set;
|
|
191
|
+
only new items are born unclassified) and that the Compass sync must be stopped before a re-point.
|
|
192
|
+
Cross-linked the [catalog-fold reversal](../workflows/catalog-fold-reversal.md) and flagged an
|
|
193
|
+
**unresolved tension**: that reversal folded/restored 199 catalog-1 items by `UPPER(partNumber)`,
|
|
194
|
+
which implies shared cross-catalog part numbers pre-merge and does not square with the same-day
|
|
195
|
+
"partNumber is unique table-wide" verification — needs a prod re-check. (jcardinal)
|
|
148
196
|
- 2026-09-01 — Recorded a **second, independent victim of the same catalog-scoping trap, on the 2.0
|
|
149
197
|
side**: `_Model_Compass_PurchaseOrder::handleMrOrder` references MR sales-order items with a
|
|
150
198
|
hardcoded `COMPASS_CATALOG_UUID` (`961dbcbc…` = catalog **1**), which is the mirror image of the
|
|
@@ -17,6 +17,8 @@ files:
|
|
|
17
17
|
- dbchanges2/Logs/2026-08-04a - MITS rejection business recipients.sql
|
|
18
18
|
related:
|
|
19
19
|
- mits-po-transmission-to-vendors.md
|
|
20
|
+
- item-catalogs-and-duplicate-items.md
|
|
21
|
+
- ../workflows/catalog-fold-reversal.md
|
|
20
22
|
- ../../compass-canada/profile.md
|
|
21
23
|
- ../../../2.0/apps/worker2/features/error-escalation-cron.md
|
|
22
24
|
- ../../../2.0/apps/_underscore/features/error-reporting-issue-event.md
|
|
@@ -125,9 +127,20 @@ backlog (0 new, several >24h, one ~20 days old).
|
|
|
125
127
|
2. **HTTP 403 `SalesOrder:Exists`** — MITS already created the order on a prior attempt, but
|
|
126
128
|
our side never recorded `c_dtTransmittedToCompass`, so the cron **retries it forever**.
|
|
127
129
|
|
|
130
|
+
**Deeper root cause of the 2026-08-31 outage — a wrong catalog merge, not just missing data.**
|
|
131
|
+
The NULL-assetType items were largely created by an ad-hoc cross-client dedup that folded
|
|
132
|
+
Compass's **199 catalog-1 (store) items into their Agilant (catalog-2) canonical**, so Compass
|
|
133
|
+
order lines resolved to Agilant items that have no store classification. The same merge fused
|
|
134
|
+
their VendorItems, which is what triggered the SQL 1242 abort (see gotchas below). Full root
|
|
135
|
+
cause + the reversal playbook: [catalog-fold reversal](../workflows/catalog-fold-reversal.md).
|
|
136
|
+
Compass keeps two intentional catalogs and the customer→catalog rule is load-bearing — see
|
|
137
|
+
[Compass item catalogs & duplicate items](./item-catalogs-and-duplicate-items.md).
|
|
138
|
+
|
|
128
139
|
**Fix the data:** assign the correct `assetTypeId` to the affected `Client_Compass.Items`
|
|
129
140
|
(business owns the mapping), and check whether the NetSuite item sync owns `assetType` — if it
|
|
130
|
-
does, it will revert the fix unless the source is corrected too.
|
|
141
|
+
does, it will revert the fix unless the source is corrected too. (Verified: the sync UPDATE path
|
|
142
|
+
never touches `assetType`/`category`/`type`, so a set value is durable — but new items are born
|
|
143
|
+
unclassified on the Agilant catalog. See the two-catalog doc.)
|
|
131
144
|
|
|
132
145
|
**Recommended code hardening (NOT yet implemented):** wrap each order's transmit in
|
|
133
146
|
`try/catch` so one MITS rejection raises the business issue but **continues to the next
|
|
@@ -158,6 +171,17 @@ pages even when the count is under threshold. See
|
|
|
158
171
|
name the order, not the person. (This is precisely what Canada still does wrong.)
|
|
159
172
|
- **The MITS crons hold their API credentials inline in the file.** Treat both files as
|
|
160
173
|
secret-bearing when copying snippets anywhere (tickets, PR bodies, knowledge docs).
|
|
174
|
+
- **The transmit item query's `GROUP BY SalesOrderItems.id` hides false "duplicates."** In
|
|
175
|
+
`1_transmit_compass_sales_orders_to_mits.php` (~L178-251, `GROUP BY` at L247) the
|
|
176
|
+
`GROUP BY SalesOrderItems.id` collapses the `LEFT JOIN VendorItems` / `LEFT JOIN
|
|
177
|
+
VendorItemCosts` fan-out to one row per line. Running that SELECT **without** the GROUP BY
|
|
178
|
+
shows FALSE duplicate fee lines (fees have 2 `VendorItemCosts` rows) — the real cron sends
|
|
179
|
+
one line per SalesOrderItem. Do not "diagnose" duplicates from the un-grouped query.
|
|
180
|
+
- **`c_preferredVendorId` scalar subquery has no `LIMIT` → SQL 1242 on multi-vendor items.**
|
|
181
|
+
Any item with >1 active `VendorItem` makes that scalar subquery return >1 row and **aborts
|
|
182
|
+
the whole run**. This was the latent bug the catalog merge exposed (it fused VendorItems).
|
|
183
|
+
A one-line `ORDER BY VendorItems.id LIMIT 1` guard fixes it, but the proper two-catalog data
|
|
184
|
+
fix makes Compass items single-vendor so the guard is not needed.
|
|
161
185
|
- **Adding a recipient does not turn ClickUp ticketing off.** Routing is both-and, not
|
|
162
186
|
either-or: `_Worker_Infrastructure_Errors` runs `if ($recipients !== null)` and
|
|
163
187
|
`if ((bool) $issue->createsClickupTask)` as **two independent** branches.
|
|
@@ -169,6 +193,14 @@ pages even when the count is under threshold. See
|
|
|
169
193
|
when this issue next fires. Watch that run rather than assuming it works.
|
|
170
194
|
|
|
171
195
|
## Change history
|
|
196
|
+
- 2026-09-01 — Traced the 2026-08-31 outage to its **deeper root cause**: an ad-hoc
|
|
197
|
+
cross-client dedup folded Compass's 199 catalog-1 store items into their Agilant canonical, so
|
|
198
|
+
order lines resolved to unclassified Agilant items (HTTP 400) and their fused VendorItems
|
|
199
|
+
tripped the transmit's un-`LIMIT`ed `c_preferredVendorId` scalar subquery (SQL 1242, whole run
|
|
200
|
+
aborts). Recorded the two transmit-query gotchas (`GROUP BY SalesOrderItems.id` hides false
|
|
201
|
+
duplicate fee lines from the VendorItemCosts fan-out; the `c_preferredVendorId` subquery needs
|
|
202
|
+
`ORDER BY VendorItems.id LIMIT 1` or single-vendor data). Cross-linked the new catalog-fold
|
|
203
|
+
reversal workflow and two-catalog item-model doc. (jcardinal)
|
|
172
204
|
- 2026-08-31 — Diagnosed the recurring backlog's **root cause**: not volume but **poison-order
|
|
173
205
|
starvation**. One MITS rejection `throw`s and aborts the whole run, and because the driver
|
|
174
206
|
query uses `ORDER BY RAND()`, a stuck order blocks a random set of good orders every run.
|
|
@@ -31,6 +31,7 @@ related:
|
|
|
31
31
|
- workflows/granting-persona-bundle-access.md
|
|
32
32
|
- workflows/granting-navigation-access.md
|
|
33
33
|
- features/mits-sales-order-transmission-alerting.md
|
|
34
|
+
- workflows/catalog-fold-reversal.md
|
|
34
35
|
- features/asn-to-item-fulfillment.md
|
|
35
36
|
- features/order-fulfillment-status-per-line.md
|
|
36
37
|
- features/cost-centers.md
|
|
@@ -0,0 +1,120 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Compass USA — reversing the 199-item catalog fold (MITS outage recovery)
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
project: _Underscore
|
|
5
|
+
client: compass-usa
|
|
6
|
+
type: workflow
|
|
7
|
+
status: active
|
|
8
|
+
updated: 2026-09-01
|
|
9
|
+
owners: ["jcardinal"]
|
|
10
|
+
files:
|
|
11
|
+
- test/team/2.0 deployment/duplicate_items_merge_ALL_CLIENTS.sql
|
|
12
|
+
- test/team/2.0 deployment/duplicate_items_merge_analysis.sql
|
|
13
|
+
- worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php
|
|
14
|
+
- library/app/api/toga2.php
|
|
15
|
+
- worker/crons/toga2/netsuite/common_sync_togasupply.php
|
|
16
|
+
related:
|
|
17
|
+
- ../features/item-catalogs-and-duplicate-items.md
|
|
18
|
+
- ../features/mits-sales-order-transmission-alerting.md
|
|
19
|
+
- ../../../1.0/apps/worker/workflows/netsuite-duplicate-items-dedup.md
|
|
20
|
+
---
|
|
21
|
+
|
|
22
|
+
## Summary
|
|
23
|
+
|
|
24
|
+
The **root cause** of the Compass MITS transmission outage (the `assetType`-NULL and
|
|
25
|
+
duplicate-vendor symptoms) was an ad-hoc cross-client **duplicate-items merge** that wrongly
|
|
26
|
+
folded Compass's **two intentional catalogs** together. This workflow records the damage and
|
|
27
|
+
the **durable playbook** used to reverse it. See the
|
|
28
|
+
[two-catalog item model](../features/item-catalogs-and-duplicate-items.md) for why the catalogs are
|
|
29
|
+
intentional.
|
|
30
|
+
|
|
31
|
+
## What broke (root cause)
|
|
32
|
+
|
|
33
|
+
An ad-hoc merge run **2026-08-31 ~12:43** — `test/team/2.0 deployment/
|
|
34
|
+
duplicate_items_merge_ALL_CLIENTS.sql`, Compass block lines **1201-1297**, committed — treated
|
|
35
|
+
Compass's two catalogs as duplicates and folded **199 catalog-1 (store) items into their
|
|
36
|
+
catalog-2 (Agilant) canonical**. Its canonical rule was **Agilant-preferred**:
|
|
37
|
+
|
|
38
|
+
```
|
|
39
|
+
ROW_NUMBER() OVER (PARTITION BY UPPER(partNumber)
|
|
40
|
+
ORDER BY (catalog='Agilant' AND manufacturerId IS NOT NULL) DESC,
|
|
41
|
+
(catalog='Agilant') DESC,
|
|
42
|
+
(manufacturerId IS NOT NULL) DESC, id)
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
For each of the ~35 reference tables it ran
|
|
46
|
+
`UPDATE IGNORE <t> SET fk=canon WHERE fk=dupe; DELETE <t> WHERE fk=dupe;` — so **collision
|
|
47
|
+
rows were permanently DELETED** — then `DELETE Items` on the Compass items. Its helper maps
|
|
48
|
+
(`_merge_map` / `_unit_map` / `_vi_map`) were dropped, so **prod cannot self-reverse**.
|
|
49
|
+
|
|
50
|
+
**Two failure signatures resulted:**
|
|
51
|
+
1. Compass sales orders resolved to Agilant items lacking store classification
|
|
52
|
+
(`assetType`/`category`/`type`) → **MITS HTTP 400** "The AssetType field is required".
|
|
53
|
+
2. The Agilant items now had **duplicate active VendorItems** (the merge fused them), so the
|
|
54
|
+
transmit's `c_preferredVendorId` scalar subquery returned **>1** → **SQL 1242**, aborting
|
|
55
|
+
the whole run.
|
|
56
|
+
|
|
57
|
+
**Not all of the merge was damage.** Of 550 total deleted dupes it correctly removed **150
|
|
58
|
+
NULL-catalog junk items** and **199 duplicate-Agilant items** (from the NetSuite
|
|
59
|
+
inventory-adjustment item-create bug). **Only the 199 catalog-1 → catalog-2 folds were the
|
|
60
|
+
damage.**
|
|
61
|
+
|
|
62
|
+
## The reversal playbook (durable)
|
|
63
|
+
|
|
64
|
+
1. **Restore a pre-merge snapshot as a scratch DB** — e.g. `Client_Compass_20260831_120000`
|
|
65
|
+
(taken ~12:00, before the ~12:43 merge). **Validate it is pre-merge**: catalog-1 twins
|
|
66
|
+
present, `customerId = 2` lines on catalog 1.
|
|
67
|
+
2. **Rebuild the merge's damage map from the snapshot** — the Agilant-preferred ranking above,
|
|
68
|
+
filtered to `dupe.catalog = 1 AND canon.catalog = 2` = the 199.
|
|
69
|
+
3. **Write a set-based, single-transaction, FK-checks-ON, NO `IGNORE`, COMMIT-left-commented
|
|
70
|
+
script** (`SNAP_DB.` / `TARGET_DB.` qualifiers) that:
|
|
71
|
+
- Restores the 199 Compass `Items` + their `VendorItems` / `VendorItemCosts` **by id**
|
|
72
|
+
(`NOT EXISTS` guard).
|
|
73
|
+
- **Re-points current `customerId = 2` `SalesOrderItems`** from the Agilant canonical back
|
|
74
|
+
to the Compass twin — **on CURRENT data, not a uuid-restore.** The sync re-created order
|
|
75
|
+
lines with **new uuids**, so restoring snapshot lines by uuid **collided 1062** on
|
|
76
|
+
`UNIQUE(salesOrderId, lineNumber)`.
|
|
77
|
+
4. **Rehearse on a same-day full prod clone** — this run: ~49,267 re-points, 199 items, 0
|
|
78
|
+
orphans, held items 0-missing-assetType / 0-multi-vendor.
|
|
79
|
+
5. **Apply to prod with a pre-run backup.** Result: queue drained to 0 pending; all
|
|
80
|
+
`customerId = 2` lines back on catalog 1.
|
|
81
|
+
|
|
82
|
+
**CTO review gated this** (rehearse-on-clone, fail-loud, customer-scoped, tenant-locked).
|
|
83
|
+
|
|
84
|
+
## Critical rules
|
|
85
|
+
|
|
86
|
+
- **Scope every catalog re-point by `customerId`.** An unscoped
|
|
87
|
+
`UPDATE ... WHERE itemId = <agilantId>` wrongly moved **687 Office-Depot lines** onto the
|
|
88
|
+
Compass item and had to be corrected.
|
|
89
|
+
- **Re-point on CURRENT data, never restore order lines by uuid** — the sync minted new uuids,
|
|
90
|
+
so a uuid-restore collides on `UNIQUE(salesOrderId, lineNumber)` (1062).
|
|
91
|
+
- **Stop the Compass NetSuite sync first.** The sync re-creates Compass order lines on Agilant
|
|
92
|
+
items, so the reversal is only durable if the sync is stopped; re-enabling risks re-pointing
|
|
93
|
+
new/re-synced orders back to Agilant (2 lines seen after a test re-enable). See the
|
|
94
|
+
[two-catalog item model](../features/item-catalogs-and-duplicate-items.md).
|
|
95
|
+
- **Never `UPDATE IGNORE` + delete-leftover for a catalog operation** — it permanently deletes
|
|
96
|
+
collision rows and cannot be reversed. Use FK-checks-ON, fail-loud, single transaction.
|
|
97
|
+
|
|
98
|
+
## Blast radius — other clients (OUTSTANDING)
|
|
99
|
+
|
|
100
|
+
The `duplicate_items_merge_ALL_CLIENTS.sql` ran `USE Client_<X>` on **17 clients**: Masonite,
|
|
101
|
+
Ncci, Erau, Shrss, Wmchealth, Trividiahealth, Prudential, Miamidade, Browardsheriff, Nychh,
|
|
102
|
+
Canon, Quad, **Compass**, Elite, Growrk, Spglobal, Endeavorhealth.
|
|
103
|
+
|
|
104
|
+
- **Not all are damaged.** The NYCHH run was a legitimate dedup.
|
|
105
|
+
- **At-risk = clients with the two-catalog + TOGa Commerce setup like Compass.** A per-client
|
|
106
|
+
assessment (customer-scoped, rehearsed) is **outstanding for a future session**.
|
|
107
|
+
- **Compass Canada was NOT damaged and is structurally immune** — it was not in the merge's
|
|
108
|
+
client list, has a single catalog, and does not use NetSuite (so no dupe-item bug).
|
|
109
|
+
|
|
110
|
+
## Change history
|
|
111
|
+
- 2026-09-01 — Recorded the root cause of the Compass MITS outage: the ad-hoc cross-client
|
|
112
|
+
`duplicate_items_merge_ALL_CLIENTS.sql` (2026-08-31 ~12:43) folded 199 Compass catalog-1
|
|
113
|
+
store items into their Agilant canonical, producing MITS HTTP 400 (NULL assetType) and SQL
|
|
114
|
+
1242 (fused duplicate VendorItems). Documented the durable reversal playbook (restore
|
|
115
|
+
pre-merge snapshot → rebuild damage map → set-based FK-checks-ON re-point of CURRENT
|
|
116
|
+
customerId=2 lines, NOT a uuid-restore → rehearse on a same-day clone → apply with backup),
|
|
117
|
+
the customer-scope rule (an unscoped re-point moved 687 Office-Depot lines), the
|
|
118
|
+
stop-the-sync-first requirement, and the 17-client blast radius with the outstanding
|
|
119
|
+
per-client assessment. CTO-gated. (jcardinal)
|
|
120
|
+
</content>
|
package/package.json
CHANGED