toga-ai 1.0.863 → 1.0.865
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/2.0/apps/_underscore/architecture.md +23 -7
- package/knowledge/2.0/apps/_underscore/features/recursive-item-fulfillments.md +19 -2
- package/knowledge/2.0/apps/_underscore/features/tracking-number-bridges.md +9 -2
- package/knowledge/2.0/apps/api2/architecture.md +2 -2
- package/knowledge/2.0/apps/apiproxy/INDEX.md +5 -0
- package/knowledge/2.0/apps/apiproxy/architecture.md +150 -0
- package/knowledge/2.0/apps/dbchanges2/features/rerunnable-additive-inserts.md +69 -1
- package/knowledge/2.0/apps/saml/architecture.md +12 -3
- package/knowledge/INDEX.md +2 -1
- package/knowledge/clients/compass-usa/INDEX.md +1 -0
- package/knowledge/clients/compass-usa/features/item-fulfillment-tracking-tableview.md +9 -2
- package/knowledge/clients/compass-usa/features/multi-tier-tracking-and-serial-mirroring.md +151 -0
- package/knowledge/clients/compass-usa/profile.md +4 -2
- package/knowledge/registry.json +7 -0
- package/package.json +1 -1
|
@@ -6,7 +6,7 @@ project: _Underscore
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: architecture
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
9
|
+
updated: 2026-09-22
|
|
10
10
|
owners: ["jcardinal", "rgirish", "mhammontree", "ajean"]
|
|
11
11
|
files:
|
|
12
12
|
- _underscore/_underscore.php
|
|
@@ -253,12 +253,28 @@ const DATABASE = _underscore::DB_CLIENT; // or DB_CORE
|
|
|
253
253
|
const TABLE = 'TableName';
|
|
254
254
|
```
|
|
255
255
|
|
|
256
|
-
Columns are declared as `public` properties set to field-type constants
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
`
|
|
260
|
-
|
|
261
|
-
|
|
256
|
+
Columns are declared as `public` properties set to field-type constants:
|
|
257
|
+
|
|
258
|
+
- **Numeric:** `FIELD_PRIMARYKEY`, `FIELD_FOREIGNKEY` (array form, see below), `FIELD_INTEGER`,
|
|
259
|
+
`FIELD_AUTOINCREMENT` (see [autoincrement-record-numbering](./features/autoincrement-record-numbering.md)),
|
|
260
|
+
`FIELD_BOOLEAN`, `FIELD_DECIMAL`.
|
|
261
|
+
- **String — several apply an on-write/on-read transform, so pick the specific type instead of a
|
|
262
|
+
plain `FIELD_CHAR` plus manual formatting:** `FIELD_CHAR` (plain), `FIELD_CHAR_CAPS` (forced
|
|
263
|
+
uppercase), `FIELD_CHAR_UCWORDS` (title-case), `FIELD_CHAR_HTMLENTITY` (HTML-escaped —
|
|
264
|
+
XSS-relevant), `FIELD_CHAR_PASSWORD` (hashed — never store a plaintext password in a plain
|
|
265
|
+
`FIELD_CHAR`), `FIELD_CHAR_PHONE` (phone-formatted), `FIELD_CHAR_SERIALIZED` (PHP-serialized),
|
|
266
|
+
`FIELD_CHAR_UUID`, `FIELD_CHAR_ZIPCODE` (ZIP-formatted), `FIELD_LIST` (enum/list value — see the
|
|
267
|
+
`FIELD_LIST` enum-validation caveat in `rules/common/security.md`).
|
|
268
|
+
- **Date/time:** `FIELD_DATE`, `FIELD_DATETIME`, `FIELD_DATETIME_CREATED` (auto-set on insert),
|
|
269
|
+
`FIELD_DATETIME_UPDATED` (auto-set on save), `FIELD_TIME`, `FIELD_YEAR`.
|
|
270
|
+
- **Binary/file:** `FIELD_BLOB`, `FIELD_STORAGE` (S3 or filesystem — see the storage-write gotcha
|
|
271
|
+
below and [field-storage-row-hydration](./features/field-storage-row-hydration.md)).
|
|
272
|
+
- **Calculated:** `FIELD_SQL` (see below).
|
|
273
|
+
|
|
274
|
+
**Field options** (the `FIELDOPT_*` array keys that tune a field): `FIELDOPT_DEFAULT` (default
|
|
275
|
+
value), `FIELDOPT_FOREIGNKEY_MODEL` (target model class), `FIELDOPT_AUTOINCREMENT_PREFIX` /
|
|
276
|
+
`FIELDOPT_AUTOINCREMENT_FIRSTNUMBER` / `FIELDOPT_AUTOINCREMENT_SUFFIX` (numbering scheme, see the
|
|
277
|
+
autoincrement feature), and `FIELDOPT_SQL_STORED` / `FIELDOPT_SQL_TYPE` (calculated fields, below).
|
|
262
278
|
|
|
263
279
|
**Foreign keys** are arrays with the target model:
|
|
264
280
|
|
|
@@ -6,13 +6,17 @@ project: _Underscore
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
9
|
+
updated: 2026-09-22
|
|
10
10
|
owners: [jcardinal, rgirish]
|
|
11
11
|
files:
|
|
12
12
|
- _underscore/Model/Client/ItemFulfillment.php
|
|
13
13
|
- _underscore/Model/Client/ItemFulfillmentItem.php
|
|
14
14
|
- _underscore/Model/Client/ItemFulfillmentItemUnit.php
|
|
15
15
|
- _underscore/Model/Client/ItemFulfillmentPackage.php
|
|
16
|
+
- _underscore/Model/Client/ItemFulfillments/TrackingNumber.php
|
|
17
|
+
- _underscore/Model/Client/ItemFulfillmentItems/TrackingNumber.php
|
|
18
|
+
- _underscore/Model/Client/ItemFulfillmentItemUnits/TrackingNumber.php
|
|
19
|
+
- dbchanges2/Core/2026-09-18a - ItemFulfillmentTrackingNumberInterceptors.sql
|
|
16
20
|
- _underscore/Model/Compass/AdvanceShippingNotice.php
|
|
17
21
|
- dbchanges2/Core/2026-02-13 - 75601 - RecursiveItemFulfillmentCreation.sql
|
|
18
22
|
- dbchanges2/Core/2026-06-04 - RecursiveItemFulfillmentPut.sql
|
|
@@ -22,6 +26,7 @@ related:
|
|
|
22
26
|
- ../architecture.md
|
|
23
27
|
- ../../api2/architecture.md
|
|
24
28
|
- ../../../../clients/compass-usa/features/asn-to-item-fulfillment.md
|
|
29
|
+
- ../../../../clients/compass-usa/features/multi-tier-tracking-and-serial-mirroring.md
|
|
25
30
|
---
|
|
26
31
|
|
|
27
32
|
Engine that mirrors a downstream ItemFulfillment up the SO→PO→SO chain onto every customer-facing SO; open when touching IF creation, bundle scaling, or upstream-duplicate/half-quantity bugs.
|
|
@@ -40,9 +45,14 @@ The whole engine is in `_Model_Client_ItemFulfillment`; the three child models a
|
|
|
40
45
|
| `Model/Client/ItemFulfillmentItem.php` | `postPost`, `postPut` | resolve owning IF uuid → driver |
|
|
41
46
|
| `Model/Client/ItemFulfillmentItemUnit.php` | `postPost`, `postPut` | resolve owning IF uuid → driver |
|
|
42
47
|
| `Model/Client/ItemFulfillmentPackage.php` | `postPost`, `postPut` | resolve owning IF uuid → driver |
|
|
48
|
+
| `Model/Client/ItemFulfillments/TrackingNumber.php` | `postPost`, `postPut` | record **317** header-tracking bridge → `triggerUpstreamSync()` → driver (added 2026-09-18) |
|
|
49
|
+
| `Model/Client/ItemFulfillmentItems/TrackingNumber.php` | `postPost`, `postPut` | record **318** item-tracking bridge → driver (added 2026-09-18) |
|
|
50
|
+
| `Model/Client/ItemFulfillmentItemUnits/TrackingNumber.php` | `postPost`, `postPut` | record **319** unit-tracking bridge → driver (added 2026-09-18) |
|
|
43
51
|
|
|
44
52
|
Triggers live on every child (not just the IF) so the sync is robust to all paths: the ASN flow (POSTs the IF then each child individually), nested IF POSTs, and direct child edits.
|
|
45
53
|
|
|
54
|
+
**⚠ The three bridge models live in SUBFOLDERS named after the parent table, not as top-level files.** The path is `Model/Client/ItemFulfillments/TrackingNumber.php` (class `_Model_Client_ItemFulfillments_TrackingNumber`) — **not** `Model/Client/ItemFulfillmentTrackingNumber.php`. Note the folder is the **plural table name** while the sibling top-level models are singular (`ItemFulfillmentItem.php`). Searching for the singular filename finds nothing and reads as "this model does not exist."
|
|
55
|
+
|
|
46
56
|
### Driver and level walk
|
|
47
57
|
**Driver — `ensureUpstreamItemFulfillment($api, $ifUuid)`:** a static `$isReconciling` re-entrancy guard makes nested re-entries (from our own `internalApiRequest` writes, which fire those records' interceptors) no-op. A `while` loop walks UP one level at a time via `reconcileUpstreamLevel`, with a `$visited` set to prevent cycles.
|
|
48
58
|
|
|
@@ -78,7 +88,9 @@ Engine is shared `_Model_Client_*` logic. Compass uses **empty pass-through subc
|
|
|
78
88
|
- **Transfer-order fulfillments are explicitly excluded from the upstream walk.** `reconcileUpstreamLevel()` returns `null` immediately when the downstream IF has `transferOrderId` set (no `salesOrderId`). Intentional for standalone TOs (GroWrk v1). When SO→TO→SO chain scenarios are needed, Phase 5 of the GroWrk transfer order plan extends this method to walk `TransferOrders_SalesOrders` bridge tables — see `clients/growrk/features/transfer-order-flow.md`.
|
|
79
89
|
- **Broken SOI↔POI bridge → empty header-only duplicate IFs (root cause, guard added 2026-07-02).** The item walk maps a downstream IFI up via `IFI.salesOrderItemId → PurchaseOrderItems_SalesOrderItems → SalesOrderItems_PurchaseOrderItems`. When that upstream `SalesOrderItems_PurchaseOrderItems` bridge is **missing/broken** (the same integrity class as the ~116 off-by-one Compass orders — see the Compass `asn-to-item-fulfillment` doc), `$desiredItems` resolves to **zero** even though the downstream IF has items. Pre-guard, step 4's eager resolve-or-create then POSTed an **empty header-only upstream IF** (e.g. Compass SA133377 got duplicate empty customer-facing IFs 71724/71725 alongside the legit direct-ASN IF 70635). **Guard:** `reconcileUpstreamLevel` now, after building `$desiredItems`, if it is empty **AND** the downstream IF has >0 ItemFulfillmentItems, `error_log()`s a diagnostic and returns `null` (skips the upstream mirror) — distinguished from the legitimate header-first POST case (downstream IF genuinely has 0 items), which is unchanged.
|
|
80
90
|
- **Duplicate customer-facing IF: the engine only adopts an upstream IF it created itself, never a pre-existing one (known gap, adoption DEFERRED).** `reconcileUpstreamLevel` finds an existing upstream IF only via `upstreamItemFulfillmentId`; it does **not** adopt a pre-existing IF that already sits on the upstream SO from another flow (e.g. a direct-ASN-created IF). So a broken bridge could yield two customer-facing IFs on the same SO (SA133377: direct-ASN IF 70635 + a mirror-created duplicate). **Naive adoption is dangerous and was intentionally not implemented:** step 5's delete-stale step deletes upstream IFIs with no matching downstream group, so adopting an IF while the item walk is empty/broken would **DELETE the adopted IF's legitimate line items**. Requires a CTO design review before implementing adoption semantics. The 2026-07-02 broken-bridge guard prevents the duplicate cascade for this failure mode without needing adoption.
|
|
81
|
-
-
|
|
91
|
+
- **⚠ NARROWED 2026-09-18 — the 317/318/319 interceptors ARE now required, for a different job than the one rejected in August.** The August decision below still stands for **populating** item/unit tracking from a header number; that work belongs in the NetSuite source writer. But the bridges also need an interceptor to **trigger the upstream walk** when tracking is attached after the chain already exists, and that one was missing entirely. Records **317/318/319 had ZERO `ApiPayloadInterceptors` rows** while their parents 28/29/30 had them: header tracking used to live in `ItemFulfillmentPackages` (record 41) with its own interceptor, and when that table was consolidated into the plain bridge `ItemFulfillments_TrackingNumbers`, **the registration did not follow the data**. So `reconcileUpstreamPackages()` — correct code — never fired on a tracking POST. Cost for Compass: **1,819 stranded orders since 2026-08-01, 95% of them tracking added after the chain was built.** Fix = `postPost`/`postPut` + `triggerUpstreamSync()` on the three bridge models **plus** `dbchanges2/Core/2026-09-18a - ItemFulfillmentTrackingNumberInterceptors.sql`. The two jobs do not fight: the writer decides *what* tracking a row carries, the interceptor only re-runs the mirror. See [Compass multi-tier tracking & serial mirroring](../../../clients/compass-usa/features/multi-tier-tracking-and-serial-mirroring.md). **Generalize the lesson: when a table is consolidated or replaced, audit `ApiPayloadInterceptors` for the old record id — a lost registration is silent, never an error.**
|
|
92
|
+
- **Units mirror on `unitId` ONLY — never on part number.** A unit is one physical device, so the shared `unitId` is the only safe key. **32,585** `upstreamItemFulfillmentItemId` links legitimately join **different** part numbers (kit/bundle roll-ups, vendor suffixes like `C40QYUC#ABA` → `C40QYUC`, service lines). Those are correct data; a "repair" that aligns part numbers across tiers corrupts them.
|
|
93
|
+
- **Source-side IF tracking population belongs in the NetSuite sync writer, not a 317 interceptor (design decision, CTO-reviewed 2026-08-26; narrowed 2026-09-18 — see above).** For NetSuite-integrated clients the fix that populates item/unit tracking (318/319) from a fulfillment's single header tracking number lives in the **source writer** `App_Api_Toga2::syncItemFulfillmentFromNetsuite` (see the [per-client sync doc](../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md)) — **not** an `_underscore` payload interceptor on `ItemFulfillments_TrackingNumbers` (317). An interceptor fires **per row** as tracking is attached, so it cannot know at insertion time whether the shipment is single- or multi-package, and it would **fight this engine**: `reconcileUpstreamUnits`/`reconcileUpstreamPackages` own upstream 319 with delete-stale logic. The source writer knows NetSuite's one tracking number and the full item/unit set up front, and this recursive engine then mirrors it upstream. A 317-interceptor approach was prototyped and **abandoned**.
|
|
82
94
|
- **Out of scope:** NetSuite sync of upstream IFs; transfer-order fulfillments (`transferOrderId`) are skipped.
|
|
83
95
|
- **Performance:** every child write re-runs a full upstream walk (read-heavy, but each upstream record is written at most once — idempotent). Fine for normal fulfillment sizes.
|
|
84
96
|
|
|
@@ -87,3 +99,8 @@ Engine is shared `_Model_Client_*` logic. Compass uses **empty pass-through subc
|
|
|
87
99
|
- GroWrk transfer order flow plan: `clients/growrk/features/transfer-order-flow.md`.
|
|
88
100
|
- `_underscore` architecture (interceptors, `internalApiRequest`, `_Model` layer).
|
|
89
101
|
- `api2` architecture (V2 metadata engine that fires these interceptors; the commit-logs / rollback-data-on-failure transaction invariant the throws rely on).
|
|
102
|
+
|
|
103
|
+
## Change history
|
|
104
|
+
- 2026-09-22 — Narrowed the "no 317 interceptor" decision: the three tracking bridges (317/318/319) had **zero** `ApiPayloadInterceptors` rows — lost when `ItemFulfillmentPackages`/record 41 was consolidated — so the upstream walk never ran when tracking was attached after the chain existed (1,819 stranded Compass orders). Added `postPost`/`postPut` to the three bridge models + a Core registration migration. Recorded that units mirror on `unitId` only (32,585 legitimate cross-part links) and that the bridge models live in plural-named subfolders. (rgirish)
|
|
105
|
+
- 2026-08-18 — Bundle scaling re-gated on distinct downstream **items** rather than downstream SOI lines, fixing same-item over-shipments mirroring as fractional quantities (Compass SA135471). (jcardinal)
|
|
106
|
+
- 2026-07-02 — Added the broken-bridge guard: an empty `$desiredItems` on a downstream IF that has items now logs and skips instead of POSTing an empty header-only upstream IF (Compass SA133377 duplicates). (jcardinal)
|
|
@@ -6,8 +6,8 @@ project: _Underscore
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
10
|
-
owners: ["jcardinal", "mhammontree", "dfranks", "apeterson", "bala", "tcox"]
|
|
9
|
+
updated: 2026-09-22
|
|
10
|
+
owners: ["jcardinal", "mhammontree", "dfranks", "apeterson", "bala", "tcox", "rgirish"]
|
|
11
11
|
files:
|
|
12
12
|
- api2/Component/Api/V2/V2.php
|
|
13
13
|
- _underscore/Model/Client/AdvanceShippingNoticeItemUnit.php
|
|
@@ -27,10 +27,12 @@ files:
|
|
|
27
27
|
- dbchanges2/Client_Quad/2026-08-26a - BackfillQuadUnitLevelTrackingSingleTn.sql
|
|
28
28
|
- dbchanges2/Client_Quad/2026-08-26b - BackfillQuadItemLevelTrackingSingleTn.sql
|
|
29
29
|
- dbchanges2/Client_CompassCanada/2026-08-27a - ItemFulfillmentTableViewsRebuild.sql
|
|
30
|
+
- dbchanges2/Core/2026-09-18a - ItemFulfillmentTrackingNumberInterceptors.sql
|
|
30
31
|
related:
|
|
31
32
|
- recursive-item-fulfillments.md
|
|
32
33
|
- ../../api2/features/cxml-shipnotice-gateway.md
|
|
33
34
|
- ../../../../clients/compass-canada/features/item-fulfillment-tableviews.md
|
|
35
|
+
- ../../../../clients/compass-usa/features/multi-tier-tracking-and-serial-mirroring.md
|
|
34
36
|
---
|
|
35
37
|
|
|
36
38
|
How tracking numbers moved from scalar FK columns to `*_TrackingNumbers` bridge tables across ASN/IF/IR; open before writing/reading tracking, choosing which bridge a client populates, or debugging 403/blank tracking columns.
|
|
@@ -87,6 +89,8 @@ All bridges are registered as **`InherentRecordChildren`** of their parent recor
|
|
|
87
89
|
## Gotchas
|
|
88
90
|
|
|
89
91
|
- **⚠ The three ASN bridge `trackingNumberId` fields do NOT share a `childPolicy`, and a new client must be provisioned per-API.** Core defaults: **1437** (unit bridge) `MATCH_CREATE`, **1441** (item bridge) `MATCH`, **1445** (header bridge) `MATCH`. Only the unit bridge may mint a `TrackingNumbers` row on its own. A producer that posts tracking at the item or header level (the cXML ShipNotice translator does, since 2026-06-11) therefore needs `MATCH_CREATE` **overrides** in that tenant's `Apis_RecordFields` for the writing API, or the write fails **EV-12 on field `trackingNumber`**. Compass USA has all three overridden (`apiId 4`); Compass Canada had none, which rejected 100% of its ship notices for 67 days (2026-06-11 → 2026-08-20). **Add these three override rows to the onboarding checklist for any client whose ASNs arrive by cXML** — and ship them as a `dbchanges2/Client_<Tenant>/` migration, since a manual patch dies on the next reseed. Do **not** flip the Core defaults: Core ids are shared by every tenant (the same reason the IF-side 2184 flip was rejected). Mechanics: [nested-relationship writes](../../api2/features/nested-relationship-writes.md) and the [cXML ShipNotice gateway](../../api2/features/cxml-shipnotice-gateway.md).
|
|
92
|
+
- **⚠ Consolidating record 41 into bridge 317 dropped its INTERCEPTOR registration, and nobody noticed for months (found 2026-09-18).** `ItemFulfillmentPackages` (record 41) had its own `Core.ApiPayloadInterceptors` rows. When it was folded into the plain bridge `ItemFulfillments_TrackingNumbers` (317), the rows did **not** follow — and 318/319 never had any either, while their parents 28/29/30 did. Interceptors are **DB-registered, never auto-discovered**, so the recursive upstream mirror simply stopped running whenever tracking was attached after the fulfillment chain existed. Compass cost: **1,819 stranded orders since 2026-08-01.** Fixed by `dbchanges2/Core/2026-09-18a - ItemFulfillmentTrackingNumberInterceptors.sql` + `postPost`/`postPut` on the three bridge models. **Rule for any future table consolidation: audit `ApiPayloadInterceptors` for the OLD record id before dropping it — a lost registration throws nothing and logs nothing.** See [recursive item fulfillments](recursive-item-fulfillments.md) and [Compass multi-tier tracking & serial mirroring](../../../clients/compass-usa/features/multi-tier-tracking-and-serial-mirroring.md).
|
|
93
|
+
- **The bridge models live in plural-named SUBFOLDERS.** `Model/Client/ItemFulfillments/TrackingNumber.php` → class `_Model_Client_ItemFulfillments_TrackingNumber`. There is no top-level `ItemFulfillmentTrackingNumber.php`; searching for one finds nothing and reads as "the model does not exist."
|
|
90
94
|
- **Tracking is now an inherent child, not a scalar.** Any producer that sent a flat `trackingNumber` on a unit must nest it under the unit's `*TrackingNumbers` child array, or it is silently dropped.
|
|
91
95
|
- **Pruning a unit/item means deleting its bridge rows FIRST — the bridge FKs are `RESTRICT`, not cascade.** The IF bridge FKs (`ItemFulfillments/Items/ItemUnits_TrackingNumbers` → parent) are `ON DELETE RESTRICT`, so deleting a parent unit/item that NetSuite dropped errors unless you delete its 317/318/319 bridge links first. The 1.0 `App_Api_Toga2::syncItemFulfillmentFromNetsuite` full reconcile does exactly this (delete bridges, then parent). Any code that prunes IF units/items must do the same ordering. See the [per-client sync doc](../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md).
|
|
92
96
|
- **Migration FK pitfalls (all handled in the SQL):** deleting `RecordFields` hits `DefaultRecordFieldSettings` FKs (repoint first); deleting `AclRecordPermissions` hits the `AclLogicGroups`→`AclLogicGroupExpressions` chain (record-41 teardown wraps it in `SET FOREIGN_KEY_CHECKS=0`); deleting `TableViewFields` hits `TableViews.sortPrimaryTableViewFieldId` (null it first, re-point after); self-referencing `INSERT … NOT EXISTS` on the target table errors 1093 (use a derived table); and orphaned/dangling `trackingNumberId` values fail the bridge FK 1452 (join `TrackingNumbers` so orphans become NULL / are skipped).
|
|
@@ -129,3 +133,6 @@ All bridges are registered as **`InherentRecordChildren`** of their parent recor
|
|
|
129
133
|
- compass-usa: ASN→Item Fulfillment, and the Item Fulfillment tracking TableView.
|
|
130
134
|
- prudential: Dell ASN units interceptor.
|
|
131
135
|
- [Compass Canada item-fulfillment TableViews](../../../../clients/compass-canada/features/item-fulfillment-tableviews.md).
|
|
136
|
+
|
|
137
|
+
## Change history
|
|
138
|
+
- 2026-09-22 — Recorded that folding record 41 (`ItemFulfillmentPackages`) into bridge 317 silently dropped its `ApiPayloadInterceptors` registration, and that 318/319 never had one — breaking the recursive upstream mirror for any tracking attached after the chain was built (1,819 stranded Compass orders). Added the subfolder/class-name note for the bridge models. (rgirish)
|
|
@@ -62,9 +62,9 @@ third party → apiproxy (region failover) → EB ALB → Apache → index.php
|
|
|
62
62
|
|
|
63
63
|
One ~2,000-line `execute()` then `processRoutePairs()`:
|
|
64
64
|
|
|
65
|
-
1. **Parse/normalize** — reads "now" from Core (`SELECT NOW()`) for token expiry; parses method/headers/route (strips `/v2`)/body; builds Core-metadata lookups (`Parameters`, `Records`, `InherentRecordChildren`, `RecordFields`).
|
|
65
|
+
1. **Parse/normalize** — reads "now" from Core (`SELECT NOW()`) for token expiry; parses method/headers/route (strips `/v2`)/body; builds Core-metadata lookups (`Parameters`, `Records`, `InherentRecordChildren`, `RecordFields`). A legacy **`apiKey` compatibility patch** rewrites a payload `apiKey` field into `api`/`secret` before auth, so old integrations that send one `apiKey` value still authenticate.
|
|
66
66
|
2. **Options** (`fields`, `where`, `join`/`ojoin`, `sort`, `group`, `distinct`, `page`, `recordsPerPage` default 25 / max 10000, `depth` default 3, `calcDepth` default 1) — query params or one base64(JSON) `_` blob.
|
|
67
|
-
3. **Authentication** — `transactionId` required and **globally unique** (checked vs Core + client logs; reuse → `EV-5`). Auth routes under `/v2/auth/*`: `api`, `refresh`, `oauth`, `oauth/refresh`, `login` (email/pw; AD via LDAP bind for AD-configured clients), `public`, `delegator`/`encrypted`. JWTs HS256, `sub` access/refresh (access 3600s, refresh 30d); signing secret **rotated** with current+previous accepted across the boundary. The JWT `id` claim carries the full identity used for ACL. **`/v2/auth/oauth` credential transport:** OAuth 2.0 `client_credentials`. `client_id` + `client_secret` may arrive as **HTTP Basic** *or* POST body fields — **Basic takes priority** when both present. `scope` is **required**; `grant_type` is **optional**, defaults to `client_credentials` (accepted: `client_credentials`, `password`, `refresh_token`). Mints a JWT; subsequent requests carry `Authorization: Bearer`. (`V2.php` L414-460.)
|
|
67
|
+
3. **Authentication** — `transactionId` required and **globally unique** (checked vs Core + client logs; reuse → `EV-5`). Auth routes under `/v2/auth/*`: `api`, `refresh`, `oauth`, `oauth/refresh`, `login` (email/pw; AD via LDAP bind for AD-configured clients), `public`, `delegator`/`encrypted`. JWTs HS256, `sub` access/refresh (access 3600s, refresh 30d); signing secret **rotated** with current+previous accepted across the boundary. The JWT `id` claim carries the full identity used for ACL — shape: `id.core.roles[]` (core role ids); `id.client.{uuid, slug, roles[], personas[]}`; `id.client.user.uuid` / `id.client.api.uuid` (whichever authenticated); `id.app.uuid` (from the `Domain`); `id.delegator` (present only when acting on behalf of another client/user). **`/v2/auth/oauth` credential transport:** OAuth 2.0 `client_credentials`. `client_id` + `client_secret` may arrive as **HTTP Basic** *or* POST body fields — **Basic takes priority** when both present. `scope` is **required**; `grant_type` is **optional**, defaults to `client_credentials` (accepted: `client_credentials`, `password`, `refresh_token`). Mints a JWT; subsequent requests carry `Authorization: Bearer`. (`V2.php` L414-460.)
|
|
68
68
|
4. **Response envelope** — `{transactionId, timestamp, authority, audience, isSuccess, status, error, messages[], meta{}, data{}}`; `isSuccess` = 2xx status.
|
|
69
69
|
5. **Transaction logging** — every request logged (to client/core Logs DB, or a JSONL line shipped by CloudWatch when `[api] log_filepath` is set).
|
|
70
70
|
|
|
@@ -0,0 +1,150 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: API Proxy (apiproxy) Architecture
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
repo: apiproxy
|
|
5
|
+
project: API Proxy
|
|
6
|
+
client: shared
|
|
7
|
+
type: architecture
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-09-22
|
|
10
|
+
owners: [jcardinal]
|
|
11
|
+
files:
|
|
12
|
+
- apiproxy/Controller/Index.php
|
|
13
|
+
- apiproxy/_.php
|
|
14
|
+
- apiproxy/index.php
|
|
15
|
+
- apiproxy/Config/production.ini
|
|
16
|
+
- apiproxy/.htaccess
|
|
17
|
+
- apiproxy/.platform/httpd/conf.d/long_gateway_timeout.conf
|
|
18
|
+
related: []
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
Thin, stateless reverse-proxy / failover router that fronts every public TOGa API tier. All external API traffic hits apiproxy first; it forwards the request verbatim to a configured backend and retries the next backend on a 5xx, so integrators never see a single-region/single-tier failure. Built on `_underscore` but DB-free in the request path. Open for the config-group resolution, the 5xx-only failover contract, and the two-tier timeout ceiling.
|
|
22
|
+
|
|
23
|
+
## Summary
|
|
24
|
+
|
|
25
|
+
`apiproxy` is the public edge for the TOGa 2.0 API. A request (e.g. `GET https://api.togahub.com/sales-orders/<uuid>`) lands here; Apache rewrites all to `index.php`, `_underscore` boots and — because no real controller matches the incoming URI — dispatch falls through to the default `['_Controller_Index','api']`. `api()` **is** the entire proxy: it picks a config group from hostname + region, walks an ordered list of `attempts_*` from the INI, and cURLs `endpoint + REQUEST_URI` forwarding the original method/headers/body. The first non-5xx response is relayed back unchanged; a 5xx moves to the next attempt. `/health` short-circuits to `200 OK` without proxying.
|
|
26
|
+
|
|
27
|
+
Runtime: PHP 8.1+, AWS Elastic Beanstalk (Amazon Linux, Apache/httpd → PHP-FPM). `_underscore` is **pulled at deploy, not vendored** (cloned by a prebuild hook). **No database is used in the request path** despite the framework supporting one — keep it that way.
|
|
28
|
+
|
|
29
|
+
**Critical rules:** (1) **Failover fires on HTTP 5xx only** (`floor(code/100) == 5`). Any 2xx/3xx/4xx is final and returned immediately, including the last attempt's 5xx if all fail — changing this trigger changes what counts as "try the next backend." (2) **Every fronted hostname/region needs a matching config group** (`<SERVER_NAME>:<region>` preferred, `<SERVER_NAME>` fallback); a missing group makes `api()` **throw**. (3) **`/health` must keep returning 200 without proxying** — it is the EB/LB probe. (4) **Keep the proxy stateless and DB-free in the request path.**
|
|
30
|
+
|
|
31
|
+
## Dependencies
|
|
32
|
+
|
|
33
|
+
- **`_underscore` (framework core)** — apiproxy is a minimal app over the framework boot/route/config layers (`_Config`, `_Environment`, `_Cloud`, `_Route`). Read its architecture first. Cloned at build from branch `_<ENVIRONMENT>`, not vendored.
|
|
34
|
+
- **`api2` / cXML tiers (downstream)** — apiproxy forwards to the real API backends (`api1/api2/api3.togahub.com`, `api-writer.togahub.com`, `cxml*.togahub.com`, `webhook.togahub.com`, etc.). api2's own architecture doc names apiproxy as the front edge; the timeout chain is shared across both tiers.
|
|
35
|
+
|
|
36
|
+
## Request lifecycle
|
|
37
|
+
|
|
38
|
+
```
|
|
39
|
+
client → LB (3600s) → apiproxy [Apache → PHP-FPM] → cURL → api2/cxml [Apache → PHP-FPM] → DB
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
1. **`.htaccess`** — `RewriteRule ^(.*) index.php` sends every non-file/non-dir request to the front controller; direct access to `.php`/`.ini` is denied.
|
|
43
|
+
2. **`index.php`** → `require '_underscore.php'` — version check, autoloader, `_Config`, `_Environment`, `_underscore::initialize()`, `_Route`.
|
|
44
|
+
3. **Routing (key fact)** — `_underscore::DEFAULT_CONTROLLER_METHOD = ['_Controller_Index','api']` (in `_.php`). The incoming URI (`/sales-orders/...`) matches no real controller/method, so `_Route` falls back to this default. The proxy has **no per-route controllers on purpose** — everything funnels into `Index::api()`, which is why the full original `REQUEST_URI` survives to forward downstream.
|
|
45
|
+
4. **`_Controller_Index::api()`** — the whole proxy lives here (`Controller/Index.php`), plus the `/health` short-circuit.
|
|
46
|
+
|
|
47
|
+
## Config-group resolution
|
|
48
|
+
|
|
49
|
+
`api()` selects which backend list to use by building a config-group name and looking it up in `Config/<environment>.ini` (`<environment>` = `_Environment::$name`). It tries two names, region-specific first:
|
|
50
|
+
|
|
51
|
+
```
|
|
52
|
+
1. "<SERVER_NAME>:<region>" e.g. api.togahub.com:us-east-1 (preferred)
|
|
53
|
+
2. "<SERVER_NAME>" e.g. api.togahub.com (fallback)
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
- `SERVER_NAME` = the Host the request arrived on.
|
|
57
|
+
- `region` = `_Cloud::getRegion()`: in **production** it reads `storage/region.txt` (written at deploy by `.platform` hook `041-write-region.sh` from IMDS); in **debug mode** (`debug_mode = 1`) it returns `_Environment::$name` itself.
|
|
58
|
+
- Locally the dev INI uses `[apiproxy:dev-<name>]` because `SERVER_NAME` is `apiproxy` and the debug "region" resolves to the env name. Production defines real groups: `[api.togahub.com:us-east-1]`, `[cxml.togahub.com:us-west-2]`, `[webhook.togahub.com]`, etc.
|
|
59
|
+
- **Neither group found → `api()` throws.** Adding a fronted host or region means adding a matching group to every relevant `Config/*.ini`.
|
|
60
|
+
|
|
61
|
+
## Retry / failover mechanism
|
|
62
|
+
|
|
63
|
+
Each config group is a flat, ordered list of attempts keyed with underscore-numbered keys:
|
|
64
|
+
|
|
65
|
+
```ini
|
|
66
|
+
attempts_0_read_endpoint = https://api1.togahub.com
|
|
67
|
+
attempts_0_write_endpoint = https://api-writer.togahub.com
|
|
68
|
+
attempts_0_delay = 0 ; microseconds to usleep BEFORE this attempt
|
|
69
|
+
attempts_1_read_endpoint = https://api1.togahub.com
|
|
70
|
+
attempts_1_write_endpoint = https://api-writer.togahub.com
|
|
71
|
+
attempts_1_delay = 2000000 ; 2 seconds
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
How `api()` consumes them:
|
|
75
|
+
|
|
76
|
+
- **Read vs write:** `GET` uses `*_read_endpoint`; every other method (`POST`/`PUT`/`DELETE`/...) uses `*_write_endpoint`. Reads spread across read replicas (`api1`/`api2`/`api3`); writes always go to the dedicated writer.
|
|
77
|
+
- The loop reads `attempts_0_*`, `attempts_1_*`, ... until a missing key stops it. **Empty list → throws.**
|
|
78
|
+
- Before each attempt, if `delay > 0` it `usleep()`s (back-off; the first attempt's delay is always `0`).
|
|
79
|
+
- Builds `destinationUrl = endpoint + REQUEST_URI` and cURLs it, forwarding the original **method, body (`php://input`), and headers**.
|
|
80
|
+
- **Failover trigger: a 5xx only.** Non-5xx (2xx/3xx/4xx) is final and returned at once. If the last attempt is still 5xx, that 5xx is returned.
|
|
81
|
+
- Region-ordered endpoint lists differ per region so each region prefers its local tier first (e.g. `us-east-1` prefers `api1`, `us-west-2` prefers `api2`).
|
|
82
|
+
|
|
83
|
+
### Header handling
|
|
84
|
+
|
|
85
|
+
- **Request → backend:** all incoming headers forwarded except `EXCLUDE_HEADERS_REQUEST` (`Host`).
|
|
86
|
+
- **Backend → client:** all returned headers relayed except `EXCLUDE_HEADERS_RESPONSE` (`Transfer-Encoding`, `Content-Length`, `Connection`, `Server`, `Date`) — dropped so the proxy/Apache set them correctly for the relayed body.
|
|
87
|
+
|
|
88
|
+
### Health check
|
|
89
|
+
|
|
90
|
+
`GET /health` (any path reducing to `health`) short-circuits to `200 OK` without proxying — the EB/ALB liveness probe. Keep it working.
|
|
91
|
+
|
|
92
|
+
## Timeouts (read before debugging 504s)
|
|
93
|
+
|
|
94
|
+
A request crosses **two** Apache→PHP-FPM tiers, each with its own timeout ceiling:
|
|
95
|
+
|
|
96
|
+
| Layer | Setting | Where |
|
|
97
|
+
|---|---|---|
|
|
98
|
+
| Load balancer idle timeout | 3600s | EB / ALB config |
|
|
99
|
+
| apiproxy Apache→FPM proxy | `ProxyTimeout 1800` / `Timeout 1800` | `.platform/httpd/conf.d/long_gateway_timeout.conf` |
|
|
100
|
+
| apiproxy PHP max execution | `time_limit = 900` | `Config/*.ini` `[_underscore]` |
|
|
101
|
+
| apiproxy → backend cURL | `CURLOPT_TIMEOUT = 900` (connect 10s) | `Controller/Index.php` |
|
|
102
|
+
| backend (api2/cxml) Apache→FPM | `ProxyTimeout 1800` / `Timeout 1800` | `api2/.platform/httpd/conf.d/long_gateway_timeout.conf` |
|
|
103
|
+
|
|
104
|
+
- **The "exactly 60 seconds" 504:** EB's Amazon Linux PHP platform proxies to PHP-FPM via `mod_proxy_fcgi`, whose default `ProxyTimeout` is **60s**. If `long_gateway_timeout.conf` is missing for a tier, that tier 504s at exactly 60s regardless of LB/PHP/cURL limits. This file must exist in **every** httpd-platform tier handling long requests (apiproxy, api2, saml, worker2). On any 60s 504, check for this file first.
|
|
105
|
+
- **Effective single-attempt ceiling is ~900s** — the cURL `CURLOPT_TIMEOUT`, not 1800s. Apache allows 1800 but cURL aborts the upstream call at 900. Raise `CURLOPT_TIMEOUT` in `Index.php` if a single backend call legitimately needs longer.
|
|
106
|
+
- **Retry amplification:** because a 5xx triggers failover, a slow request that *eventually* 5xxs can burn several multi-minute attempts back-to-back. A slow-but-successful request does not retry (non-5xx is final), so normal large-record reads are safe.
|
|
107
|
+
|
|
108
|
+
## Deployment (Elastic Beanstalk)
|
|
109
|
+
|
|
110
|
+
`.platform/` hooks and `.ebextensions/` configure the instance:
|
|
111
|
+
|
|
112
|
+
- **`.ebextensions/php_include_underscore.config`** — sets PHP `include_path` to `.:/var/www/html/_underscore` so the framework is importable.
|
|
113
|
+
- **`.ebextensions/git.php` + `.platform/hooks/prebuild/git.sh`** — at build, clones the shared `_underscore` repo (branch `_<ENVIRONMENT>`, e.g. `_production`) into place, strips its `.git`, swaps it in. **Framework source is NOT vendored — pulled at deploy.** Env-specific overrides come from `git.<ENVIRONMENT>.json` if present.
|
|
114
|
+
- **`.platform/hooks/postdeploy/015_install_composer.sh`** — installs Composer and runs `composer install` (currently no runtime deps; `composer.json` `require` is empty).
|
|
115
|
+
- **`.platform/hooks/_shared/040-write-instance-id.sh` & `041-write-region.sh`** — write `storage/instance-id.txt` and `storage/region.txt` from IMDSv2. `_Cloud::getRegion()`/`getInstanceId()` read these. Symlinked into `prestart` and `postdeploy`.
|
|
116
|
+
- **`.platform/httpd/conf.d/long_gateway_timeout.conf`** — the timeout fix above.
|
|
117
|
+
|
|
118
|
+
## Local development
|
|
119
|
+
|
|
120
|
+
- Config chosen by `_Environment::$name`. The dev file (`Config/dev-<name>.ini`) sets `debug_mode = 1` and points its attempts at local backends (`http://api2`, `http://cxml2`).
|
|
121
|
+
- `cache_folder` / `model_storage_folder` in the dev INI use Windows paths (`C:\CACHE`, `C:\STORAGE`).
|
|
122
|
+
- To front a new host/region locally, add a `[<SERVER_NAME>:<env-name>]` (or bare `[<SERVER_NAME>]`) group with `attempts_*` keys.
|
|
123
|
+
|
|
124
|
+
## File map
|
|
125
|
+
|
|
126
|
+
| Path | Purpose |
|
|
127
|
+
|---|---|
|
|
128
|
+
| `index.php` | Front controller — just requires the framework |
|
|
129
|
+
| `_.php` | Project class `_underscore` — namespace, defaults, `DEFAULT_CONTROLLER_METHOD` |
|
|
130
|
+
| `Controller/Index.php` | **The entire proxy logic** (`api()`) + `/health` |
|
|
131
|
+
| `Controller/Error.php` | Error route controller (sets HTTP code) |
|
|
132
|
+
| `Config/production.ini` | Production config groups (real hostnames/regions/endpoints) |
|
|
133
|
+
| `Config/dev-*.ini` | Per-developer local config |
|
|
134
|
+
| `.htaccess` | Rewrite-all-to-`index.php` + deny direct `.php`/`.ini` |
|
|
135
|
+
| `.platform/`, `.ebextensions/` | EB deploy configuration (see Deployment) |
|
|
136
|
+
| `Layout/`, `View/`, `Theme/`, `Component/`, `Model/`, `Assets/` | Framework scaffolding — largely unused; placeholder `.gitkeep`s / boilerplate |
|
|
137
|
+
|
|
138
|
+
## When making changes here
|
|
139
|
+
|
|
140
|
+
- **Adding a fronted hostname/region:** add a matching config group (`<host>:<region>` and/or `<host>`) with `attempts_*` keys to every relevant `Config/*.ini`. Missing group → `api()` throws.
|
|
141
|
+
- **Changing failover behavior:** the only retry trigger is a 5xx (`floor(code/100) == 5`). That check is the lever for what counts as "try the next backend."
|
|
142
|
+
- **Long-running requests / 504s:** verify `long_gateway_timeout.conf` exists for the tier, then check `CURLOPT_TIMEOUT` (900s) and PHP `time_limit` (900s). The LB is at 3600s and is rarely the cause.
|
|
143
|
+
- **Don't break `/health`** — the LB health probe.
|
|
144
|
+
- **Keep it stateless and DB-free in the request path** unless there is a strong reason not to.
|
|
145
|
+
|
|
146
|
+
## Security notes
|
|
147
|
+
|
|
148
|
+
- **No DB and no per-request secrets in the proxy path** — apiproxy forwards the client's own `Authorization` header downstream unchanged; it does not mint or hold tokens. Auth stays a backend (api2) concern.
|
|
149
|
+
- **Deploy-time credential exposure to check:** the `git.php` / `git.<ENVIRONMENT>.json` clone config supplies whatever credential `prebuild/git.sh` uses to pull `_underscore` at build. Sibling 2.0 tiers (api2) ship a **committed plaintext GitHub PAT** in their per-env `git.*.json`; if apiproxy's clone config carries a token the same way, treat it as compromised, rotate it, and move it to SSM / EB env properties. Verify before touching deploy config. (Location/remediation only — never record a token value.)
|
|
150
|
+
- **`_underscore` is cloned from a moving branch** (`_<ENVIRONMENT>`), not pinned to a commit, so two deploys of the same apiproxy commit can behave differently — check the framework branch state when triaging an "it worked yesterday" regression.
|
|
@@ -6,7 +6,7 @@ project: Database Changes
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
9
|
+
updated: 2026-09-22
|
|
10
10
|
owners: [bala, kyalamarthi, apeterson, ajean, jcardinal, rgirish]
|
|
11
11
|
files:
|
|
12
12
|
- dbchanges2/Client_Compass/2026-08-26a - CompassCreativeStudioPersona.sql
|
|
@@ -14,6 +14,8 @@ files:
|
|
|
14
14
|
- dbchanges2/Client/2026-09-01a - PurchaseOrderItemQtyFieldsApiRoleRead.sql
|
|
15
15
|
- dbchanges2/Core/2026-09-08e - Insert QcBatch CronJob.sql
|
|
16
16
|
- dbchanges2/_modules/netsuite/2026-09-11c - ItemClassRecordPermission.sql
|
|
17
|
+
- dbchanges2/Client_Compass/2026-09-18a - BackfillUpstreamItemFulfillmentTracking.sql
|
|
18
|
+
- dbchanges2/Client_Compass/2026-09-21a - BackfillUpstreamItemFulfillmentUnits.sql
|
|
17
19
|
related:
|
|
18
20
|
- ../architecture.md
|
|
19
21
|
- ../../worker2/features/netsuite-salesorder-open-orders-sync.md
|
|
@@ -22,6 +24,7 @@ related:
|
|
|
22
24
|
- ../../_underscore/features/tracking-number-bridges.md
|
|
23
25
|
- ../../../../clients/compass-usa/workflows/granting-persona-bundle-access.md
|
|
24
26
|
- ../../../../clients/compass-usa/workflows/granting-navigation-access.md
|
|
27
|
+
- ../../../../clients/compass-usa/features/multi-tier-tracking-and-serial-mirroring.md
|
|
25
28
|
---
|
|
26
29
|
|
|
27
30
|
## Summary
|
|
@@ -168,6 +171,20 @@ before you write, not after.
|
|
|
168
171
|
- **No unique key** — a re-run **silently duplicates every row**, and nothing tells you. Always guard
|
|
169
172
|
these.
|
|
170
173
|
|
|
174
|
+
**Check `information_schema.STATISTICS` for a COMPOSITE unique key before every bulk INSERT** — not
|
|
175
|
+
just for a single-column one, and not by reading the `CREATE TABLE` from memory. Measured 2026-09 on
|
|
176
|
+
the item-fulfillment bridges, which sit side by side and do **not** agree:
|
|
177
|
+
|
|
178
|
+
| Table | Composite UNIQUE? |
|
|
179
|
+
|---|---|
|
|
180
|
+
| `ItemFulfillments_TrackingNumbers` | yes — `(itemFulfillmentId, trackingNumberId)` |
|
|
181
|
+
| `ItemFulfillmentItems_TrackingNumbers` | yes — `(itemFulfillmentItemId, trackingNumberId)` |
|
|
182
|
+
| `ItemFulfillmentItemUnits_TrackingNumbers` | yes — `(itemFulfillmentItemUnitId, trackingNumberId)` |
|
|
183
|
+
| `ItemFulfillmentItemUnits` | **no** — and it already holds **254** duplicate `(itemFulfillmentItemId, unitId)` rows as a result |
|
|
184
|
+
|
|
185
|
+
The last row is the whole point: the table with no unique key had already accumulated duplicates
|
|
186
|
+
that nobody noticed, because nothing ever errored.
|
|
187
|
+
|
|
171
188
|
Both guard shapes are in use and equivalent:
|
|
172
189
|
|
|
173
190
|
```sql
|
|
@@ -320,6 +337,56 @@ phases leaves `@parentId` `NULL` and the child inserts either fail on the FK or
|
|
|
320
337
|
in the file header, and make the unguarded parent insert the one phase you skip on a re-run (set
|
|
321
338
|
`@parentId` to the existing row by hand instead).
|
|
322
339
|
|
|
340
|
+
### Backfilling a multi-tier chain: run the SAME file repeatedly until the verify count is 0
|
|
341
|
+
|
|
342
|
+
A backfill that walks a parent/child **chain** (Compass's Compass SO → Office Depot SO → Agilant SO
|
|
343
|
+
fulfillment tiers) moves data up **exactly one tier per pass**. Section N+1 can only act on rows
|
|
344
|
+
section N created — and it does not see them until the **next** run, because each section reads a
|
|
345
|
+
snapshot taken before the file started. So the file is not "run once"; it is **run until the verify
|
|
346
|
+
section reports 0**. Both 2026-09 Compass tracking/unit backfills needed **3 passes** each.
|
|
347
|
+
|
|
348
|
+
Write it accordingly:
|
|
349
|
+
|
|
350
|
+
1. Make every section a guarded no-op on re-run (see above) so extra passes are free.
|
|
351
|
+
2. End the file with a **verify section** that counts the rows still needing work.
|
|
352
|
+
3. Put "re-run until the verify count is 0" in the file header. Without that line, the executor runs
|
|
353
|
+
it once, sees rows inserted, and calls it done — leaving the top tier unfixed.
|
|
354
|
+
|
|
355
|
+
### ⚠ `SELECT DISTINCT` is not enough when many downstream rows roll up to ONE upstream row
|
|
356
|
+
|
|
357
|
+
Distinct from the uuid trap above: even with the uuid correctly generated in the outer select,
|
|
358
|
+
`DISTINCT` over the **source** rows still emits the same **target** pair several times when the
|
|
359
|
+
mapping is many-to-one. Compass bundles had **6** downstream items feeding **one** upstream item, so
|
|
360
|
+
the same `(itemFulfillmentItemId, trackingNumberId)` pair was emitted 6 times and tripped
|
|
361
|
+
`ItemFulfillmentItems_TrackingNumbers_UNIQUE_1`.
|
|
362
|
+
|
|
363
|
+
**Fix: `GROUP BY` the target key pair and `MIN()` the passthrough columns.** `DISTINCT` dedupes the
|
|
364
|
+
whole row; `GROUP BY` dedupes the thing the unique key actually covers.
|
|
365
|
+
|
|
366
|
+
```sql
|
|
367
|
+
SELECT
|
|
368
|
+
<uuid expression>,
|
|
369
|
+
src.targetId,
|
|
370
|
+
src.trackingNumberId,
|
|
371
|
+
MIN(src.somePassthrough)
|
|
372
|
+
FROM ( … ) src
|
|
373
|
+
GROUP BY src.targetId, src.trackingNumberId
|
|
374
|
+
```
|
|
375
|
+
|
|
376
|
+
### Measure the safe / ambiguous / no-match split BEFORE writing any bulk DML
|
|
377
|
+
|
|
378
|
+
The methodology that worked on the 2026-09 Compass link repair, and the one to reuse: before writing
|
|
379
|
+
a single `UPDATE`, run a counting query that buckets every candidate row into **safe** (exactly one
|
|
380
|
+
match), **ambiguous** (more than one), and **no-match**. Then **skip the ambiguous rows rather than
|
|
381
|
+
guessing** — a wrong link is far more expensive to unpick than a row left alone for a follow-up
|
|
382
|
+
ticket.
|
|
383
|
+
|
|
384
|
+
Worked example: 4,900 broken `upstreamItemFulfillmentItemId` links = **1,982 safe** (unique part
|
|
385
|
+
match) + **0 ambiguous** + **2,918 no-match** (missing bridge rows — raised as its own ticket, not
|
|
386
|
+
patched). A second pre-flight check confirmed **0** upstream rows were claimed by two downstream
|
|
387
|
+
rows, so the repair could not fan out. See
|
|
388
|
+
[Compass multi-tier tracking & serial mirroring](../../../../clients/compass-usa/features/multi-tier-tracking-and-serial-mirroring.md).
|
|
389
|
+
|
|
323
390
|
## Dry-running a destructive migration before it goes near an environment
|
|
324
391
|
|
|
325
392
|
A rebuild file (DELETE + re-INSERT, FK detaches, index renumbers) cannot be proved safe by reading
|
|
@@ -410,6 +477,7 @@ concluding a migration is production-safe.
|
|
|
410
477
|
[FIELD_SQL calculated fields](../../_underscore/features/calculated-sql-fields.md).
|
|
411
478
|
|
|
412
479
|
## Change history
|
|
480
|
+
- 2026-09-22 — Added multi-tier chain backfill rules from the Compass tracking/serial backfills: re-run the same file until the verify count is 0 (each pass moves data up one tier; both files needed 3 passes); `GROUP BY` the target key pair instead of `SELECT DISTINCT` when many downstream rows roll up to one upstream row (bundles tripped a bridge UNIQUE 6-to-1); measure the safe/ambiguous/no-match split before writing bulk DML and skip the ambiguous rows (4,900 broken links = 1,982 safe + 0 ambiguous + 2,918 no-match). Sharpened the unique-key rule with measured composite keys across the four item-fulfillment tables — `ItemFulfillmentItemUnits` has none and already carries 254 duplicates. (rgirish)
|
|
413
481
|
- 2026-09-21 — Added three lessons from the platform-wide `Users.contactId` NOT NULL backfill: a guarded
|
|
414
482
|
`NOT NULL` ALTER (`sql_mode ''` silently coerces leftover NULLs to `0`, so gate on a residual count —
|
|
415
483
|
single-tenant aborts loud, fan-out `DO 0` skips); latin1 schema drift breaks a forced
|
|
@@ -6,7 +6,7 @@ project: SAML SSO Gateway
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: architecture
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
9
|
+
updated: 2026-09-22
|
|
10
10
|
owners: ["rgirish", "jcardinal"]
|
|
11
11
|
files:
|
|
12
12
|
- saml/index.php
|
|
@@ -65,6 +65,13 @@ Legacy `/?meta`, `/?acs`, `/?sls` query-style paths are rewritten automatically.
|
|
|
65
65
|
|
|
66
66
|
Identity mapping lives in `_underscore/Model/<ClientIdentifier>/ClientAuthentication.php`. Each class extends `_Model_Core_ClientAuthentication` + uses `_Trait_ClientAuthentication`. To onboard a new SSO client, add the class in `_underscore` — not in this repo.
|
|
67
67
|
|
|
68
|
+
## Runtime & config
|
|
69
|
+
|
|
70
|
+
- **PHP 8.1+ in production**, on Amazon Linux with **Apache/httpd fronting PHP-FPM**. `composer.json` pins `platform.php` to **`7.2.33` for dependency resolution only** — that is not the runtime version; do not treat it as the target.
|
|
71
|
+
- **Config files are `Config/<environment>.ini`, selected by `_Environment::$name`.** Production sets `time_limit = 300`, `debug_mode = 0`. Many per-developer/per-stage configs exist (`alpha`, `beta`, `gamma`, `dev-*`, `rohan-mac`).
|
|
72
|
+
- **SP host per environment is a framework constant**, not local config: `_Model_Core_ClientAuthentication::SAML_ASSERTION_CONSUMER_SERVICE_HOST_URL_BY_ENVIRONMENT` — `dev → http://saml`, `beta`/`production → https://saml.togahub.com`. This is the ACS host IdPs POST to; it must match what `/meta` advertises.
|
|
73
|
+
- **Apache `.platform` configs (keep these):** `enforce_https.conf` redirects any request with `X-Forwarded-Proto != https` to HTTPS — **SAML must run over TLS only; do not weaken it**. `long_gateway_timeout.conf` sets `ProxyTimeout 1800` / `Timeout 1800` (same override as `apiproxy`). `php_include_underscore.config` adds `/var/www/html/_underscore` to PHP `include_path`.
|
|
74
|
+
|
|
68
75
|
## Deployment
|
|
69
76
|
|
|
70
77
|
- **EB env:** `saml-production` (us-east-1). DNS: `saml.togahub.com → saml-production.us-east-1.elasticbeanstalk.com`.
|
|
@@ -82,8 +89,10 @@ Identity mapping lives in `_underscore/Model/<ClientIdentifier>/ClientAuthentica
|
|
|
82
89
|
## Gotchas
|
|
83
90
|
|
|
84
91
|
- **`_Database::register()` auto-starts a lazy transaction (since Apr 2 2026).** Any code that registers DBs then writes must call `_Database::transactionCommit()` before exiting — otherwise MySQL silently rolls back all writes on connection close. `/acs` calls `transactionCommit()` after `getAuthenticatedSsoUser()` succeeds and before the redirect. See `_underscore/Database.php:48`.
|
|
85
|
-
- **SAML signature verification is commented out — repo's top security priority.** The IdP X509 cert verification block in `/acs` is disabled, so the gateway accepts any well-formed status-success `SAMLResponse` without proving it was signed by the expected IdP. The only remaining gate is the encrypted RelayState, which authenticates the *originating TOGa request* — not the *asserting IdP*. Threat: an attacker who can craft/replay a `SAMLResponse` (or a malicious/compromised IdP) can forge an identity assertion for any user. Re-enabling per-IdP X509 signature verification is the single highest-impact security fix.
|
|
86
|
-
- **Plaintext secrets in source — move to AWS SSM Parameter Store.** `saml/Config/production.ini`
|
|
92
|
+
- **SAML signature verification is commented out — repo's top security priority.** The IdP X509 cert verification block in `/acs` is disabled, so the gateway accepts any well-formed status-success `SAMLResponse` without proving it was signed by the expected IdP. The only remaining gate is the encrypted RelayState, which authenticates the *originating TOGa request* — not the *asserting IdP*. Threat: an attacker who can craft/replay a `SAMLResponse` (or a malicious/compromised IdP) can forge an identity assertion for any user. Re-enabling per-IdP X509 signature verification is the single highest-impact security fix. The per-client IdP verification certs already exist in source as base64 blobs (Compass, Office Depot, Prudential, RUMCSI, Agilant) — they are what a re-enabled verifier would load; verification is per-client and each IdP's signing cert must match.
|
|
93
|
+
- **Plaintext secrets in source — move to AWS SSM Parameter Store.** Checked-in credentials, all live and to be rotated: `saml/Config/production.ini` holds a **DB password, an AWS access key id + secret, and NetSuite credentials** in cleartext; `saml/Controller/Index.php` hardcodes the **Sentry DSN**; `.ebextensions/git.json` carries a **GitHub PAT** (same leak class as `apiproxy`). Migrate to AWS SSM Parameter Store (SecureString), load at runtime; source tree should reference parameter names only, never literal credential values.
|
|
94
|
+
- **Stale test scaffolding in `/acs` — remove it.** A large commented-out block in `saml/Controller/Index.php` hardcodes captured real `SAMLResponse` payloads and a synthetic `RelayState` for Office Depot / Compass testing. It is inert (commented out) but captured assertions living in source are a confidentiality/hygiene risk — delete it.
|
|
95
|
+
- **RelayState time check bounds the future, not the past.** `MAX_TIME_TO_LIVE` (300s) sanity-checks how far in the *future* a RelayState timestamp may be. It does not clearly bound replay of an *old* captured RelayState — review before relying on it as replay protection.
|
|
87
96
|
- **`/meta` cert lines must be `trim()`-ed (CRLF).** The `.crt` files in `_underscore/Assets/ssl` are CRLF-encoded; cert-format code does `explode("\n", ...)` and indents each line, so without a per-line `trim()` a stray `\r` ends up on every base64 line inside `<ds:X509Certificate>`. Trim each line in both signing and encryption blocks.
|
|
88
97
|
- **SP encryption certs load from Secrets Manager at request time.** `/acs` holds no encryption key material on disk (`_underscore/Assets/ssl/` no longer carries the SP encryption key). Rotation = edit the `ENCRYPTION_SECRET_NAMES` list + the secrets themselves, not a redeploy of on-disk `.crt`/`.key` files.
|
|
89
98
|
- **Committed AWS key must be rotated.** A hardcoded AWS key was removed from `_underscore/Cloud.php`; because it was committed to git history it is compromised — rotate it. Still open: plaintext creds in `production.ini` and the hardcoded Sentry DSN.
|
package/knowledge/INDEX.md
CHANGED
|
@@ -19,12 +19,13 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
19
19
|
|
|
20
20
|
## 2.0 framework
|
|
21
21
|
|
|
22
|
-
- **_underscore** (_Underscore) _(framework core)_ —
|
|
22
|
+
- **_underscore** (_Underscore) _(framework core)_ — 90 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
|
|
23
23
|
- **worker2** (Worker) — 74 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
|
|
24
24
|
- **api2** (API) — 26 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
|
|
25
25
|
- **dbchanges2** (Database Changes) _(framework core)_ — 19 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
|
|
26
26
|
- **toga2-supply** (TOGa Supply) — 9 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)
|
|
27
27
|
- **saml** (SAML SSO Gateway) — 6 doc(s) → [2.0/apps/saml/INDEX.md](2.0/apps/saml/INDEX.md)
|
|
28
|
+
- **apiproxy** (API Proxy) — 1 doc(s) → [2.0/apps/apiproxy/INDEX.md](2.0/apps/apiproxy/INDEX.md)
|
|
28
29
|
- **toga2-view** (TOGa View Frontend) — 12 doc(s) → [2.0/apps/toga2-view/INDEX.md](2.0/apps/toga2-view/INDEX.md)
|
|
29
30
|
- **toga2-hub** (TOGa Hub) — 2 doc(s) → [2.0/apps/toga2-hub/INDEX.md](2.0/apps/toga2-hub/INDEX.md)
|
|
30
31
|
- **talos** (TOGa IQ) — 8 doc(s) → [2.0/apps/talos/INDEX.md](2.0/apps/talos/INDEX.md)
|
|
@@ -14,6 +14,7 @@
|
|
|
14
14
|
| [Compass MITS PO rejection — zero line items (400 + Issue + PM alert email)](features/mits-po-zero-line-item-rejection.md) | 2.0 | Guard that rejects MITS Compass purchase orders carrying zero line items (HTTP 400 + Logs.Issue + PM alert email); open when working on the zero-item guard, its |
|
|
15
15
|
| [Compass MITS Sales-Order Transmit Alerting (US Issue/Event vs Canada in-cron email)](features/mits-sales-order-transmission-alerting.md) | 2.0 | How a `1_transmit_..._sales_orders_to_mits.php` failure reaches a human — USA rejections route through Logs Issue/Event, Canada still emails in-cron; open for " |
|
|
16
16
|
| [Compass MR/MA Order Auto-Approval & Status Gate](features/mr-ma-order-approval-and-status.md) | 2.0 | Compass MR/MA order auto-approval and how the `_status` approval gate strands orders; open when an order is stuck at "Pending Initial Approval" or you're tempte |
|
|
17
|
+
| [Compass: multi-tier tracking & serial mirroring (3-tier drop-ship chain) and the `_trackingNumbers` field](features/multi-tier-tracking-and-serial-mirroring.md) | 2.0 | Why a Compass customer order can show no tracking or serial while the goods shipped: the 3-tier drop-ship chain, the missing 317/318/319 interceptor rows, and t |
|
|
17
18
|
| [Compass ODP EDI 850 Line-Item Resolution (VA part number to IN SKU fallback)](features/odp-edi-850-item-resolution.md) | 1.0 | How each PO1 line on an inbound Office Depot EDI 850 resolves to a real Compass catalog item before cron 3a writes anything; open for ODP 850 import failures, u |
|
|
18
19
|
| [Compass Office Depot EDI 855 Acknowledgement + Over-Quantity PO Guard](features/odp-edi-855-acknowledgement-and-overquantity-guard.md) | 1.0 | How Compass acknowledges ODP EDI 850 POs with an X12 855 and rejects a duplicate/over-quantity PO before import; open for 855 accept/reject behaviour, over-quan |
|
|
19
20
|
| [Compass ODP EDI File Retention & Import Retry (cron 3a)](features/odp-edi-file-retention-and-retry.md) | 1.0 | When cron 3a keeps vs deletes an ODP EDI file in S3, and why an api2 outage used to silently lose orders; open for lost ODP imports, retry behaviour, or the sti |
|
|
@@ -5,8 +5,8 @@ project: _Underscore
|
|
|
5
5
|
client: compass-usa
|
|
6
6
|
type: client-feature
|
|
7
7
|
status: active
|
|
8
|
-
updated: 2026-09-
|
|
9
|
-
owners: ["jcardinal", "bala"]
|
|
8
|
+
updated: 2026-09-22
|
|
9
|
+
owners: ["jcardinal", "bala", "rgirish"]
|
|
10
10
|
files:
|
|
11
11
|
- dbchanges2/Client_Compass/2026-06-10 - ItemFulfillmentsForSalesOrderItemsTableView.sql
|
|
12
12
|
- dbchanges2/Client_Compass/2026-06-11 - ItemFulfillmentsForSalesOrdersTableView.sql
|
|
@@ -15,6 +15,7 @@ files:
|
|
|
15
15
|
related:
|
|
16
16
|
- ../../../2.0/apps/_underscore/features/tracking-number-bridges.md
|
|
17
17
|
- ../../../2.0/apps/api2/features/tableview-apiwhereclause-row-filtering.md
|
|
18
|
+
- ./multi-tier-tracking-and-serial-mirroring.md
|
|
18
19
|
---
|
|
19
20
|
Two config-only Compass TableViews (ids 12 & 13) showing fulfilled items in toga2-supply, re-rooted off Units and tracking sourced at item level; open when a fulfilled-items view shows 0 rows or blank tracking, or when editing these views.
|
|
20
21
|
|
|
@@ -48,6 +49,8 @@ Stable Core ids used: records 29 IFI, 30 IFIU, 31 Units, 15 SalesOrderItems, 14
|
|
|
48
49
|
Compass-only. The underlying bridge model is shared (see the _underscore feature doc).
|
|
49
50
|
|
|
50
51
|
## Gotchas
|
|
52
|
+
|
|
53
|
+
- **⚠ A blank Tracking # here while the Sales Orders GRID shows one is a DATA problem, not a view problem (2026-09-18).** View 12 reads only `ItemFulfillmentItems_TrackingNumbers` (318) scoped to **this** order, and the Shipments panel (`toga25-supply/.../hooks/useShipments.ts`) scopes to `salesOrderId = this order` too. The grid's `_trackingNumbers()` computed field used to walk a 4-hop PO bridge out to **another** order's fulfillment, so it found tracking that genuinely was not on this order. Compass's 3-tier chain had stopped mirroring tracking upward — check that before touching the view definition. See [multi-tier tracking & serial mirroring](multi-tier-tracking-and-serial-mirroring.md).
|
|
51
54
|
- **A Unit-rooted view returns 0 rows whenever the order has no unit-level breakdown.** Symptom on SA132740 (salesOrderId 105826): 13 SalesOrderItems, 7 ItemFulfillmentItems, but **0 ItemFulfillmentItemUnits / 0 Units** → view 12 showed "0 records" while re-rooted sibling view 13 showed the items fine. Fix is structural (re-root + OUTER chain), independent of the missing-unit data issue.
|
|
52
55
|
- **Tracking must be sourced at the ITEM level** via `ItemFulfillmentItems_TrackingNumbers` (record 318), joined directly onto `ItemFulfillmentItems` (rec 29). The original re-root (2026-06-10/11) left tracking on the **unit-level** bridge `ItemFulfillmentItemUnits_TrackingNumbers` (record 319), which is empty for non-serialized lines (no `ItemFulfillmentItemUnits` rows) → the OUTER join yielded NULL and the tracking column rendered blank even though the data was correct. Symptom: SO **MR243437** (id 105730) showed tracking at order level but not item level; item 266006 had 0 unit rows but a valid `ItemFulfillmentItems_TrackingNumbers` row (→ TN 67248). Fixed 2026-06-15.
|
|
53
56
|
- **Compass imports exactly one tracking number per item fulfillment** (NetSuite does not support multiple tracking numbers at the unit level; a different tracking number arrives as a separate item fulfillment). So the item-level bridge (318) is always the complete source and re-pointing away from the unit-level bridge (319) loses nothing.
|
|
@@ -61,3 +64,7 @@ Compass-only. The underlying bridge model is shared (see the _underscore feature
|
|
|
61
64
|
## Related
|
|
62
65
|
- [2.0 _underscore: Tracking-Number Bridge Migration](../../../2.0/apps/_underscore/features/tracking-number-bridges.md).
|
|
63
66
|
- [2.0 api2: TableView row-filtering via apiWhereClause](../../../2.0/apps/api2/features/tableview-apiwhereclause-row-filtering.md) — the mechanism used for the FEE exclusion.
|
|
67
|
+
|
|
68
|
+
## Change history
|
|
69
|
+
- 2026-09-22 — Added the "blank tracking here but populated in the grid" gotcha: the divergence was the upstream mirroring gap in Compass's 3-tier chain, not a view misconfiguration. (rgirish)
|
|
70
|
+
- 2026-06-11 — View 12 re-rooted at `ItemFulfillmentItems` (29) with an OUTER unit/tracking chain, fixing the 0-records result for orders with no serialized units. View 13 re-rooted 2026-06-10. (jcardinal)
|
|
@@ -0,0 +1,151 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Compass: multi-tier tracking & serial mirroring (3-tier drop-ship chain) and the `_trackingNumbers` field"
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
repo: _underscore
|
|
5
|
+
project: _Underscore
|
|
6
|
+
client: compass-usa
|
|
7
|
+
type: client-feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-09-22
|
|
10
|
+
owners: ["rgirish"]
|
|
11
|
+
files:
|
|
12
|
+
- _underscore/Model/Compass/SalesOrder.php
|
|
13
|
+
- _underscore/Model/Client/ItemFulfillments/TrackingNumber.php
|
|
14
|
+
- _underscore/Model/Client/ItemFulfillmentItems/TrackingNumber.php
|
|
15
|
+
- _underscore/Model/Client/ItemFulfillmentItemUnits/TrackingNumber.php
|
|
16
|
+
- dbchanges2/Core/2026-09-18a - ItemFulfillmentTrackingNumberInterceptors.sql
|
|
17
|
+
- dbchanges2/Client_Compass/2026-09-18a - BackfillUpstreamItemFulfillmentTracking.sql
|
|
18
|
+
- dbchanges2/Client_Compass/2026-09-21a - BackfillUpstreamItemFulfillmentUnits.sql
|
|
19
|
+
related:
|
|
20
|
+
- ../../../2.0/apps/_underscore/features/recursive-item-fulfillments.md
|
|
21
|
+
- ../../../2.0/apps/_underscore/features/tracking-number-bridges.md
|
|
22
|
+
- ../../../2.0/apps/api2/features/api-payload-interceptors.md
|
|
23
|
+
- ./item-fulfillment-tracking-tableview.md
|
|
24
|
+
- ./tracking-number-origin-and-asn-link.md
|
|
25
|
+
---
|
|
26
|
+
Why a Compass customer order can show no tracking or serial while the goods shipped: the 3-tier drop-ship chain, the missing 317/318/319 interceptor rows, and the two backfills. Open when tracking/serials are blank on a Compass order page but the grid looks fine.
|
|
27
|
+
|
|
28
|
+
## Summary
|
|
29
|
+
|
|
30
|
+
Compass runs a **3-tier drop-ship chain**: **Compass SO → Office Depot SO → Agilant SO**. Agilant
|
|
31
|
+
physically ships and records tracking numbers and unit serials against the *Agilant* fulfillment.
|
|
32
|
+
Those must mirror **UP** to the customer-facing Compass order. Max chain depth observed in prod is
|
|
33
|
+
**3** (zero rows at depth 4).
|
|
34
|
+
|
|
35
|
+
The tiers are linked by `ItemFulfillments.upstreamItemFulfillmentId` and
|
|
36
|
+
`ItemFulfillmentItems.upstreamItemFulfillmentItemId`. The shared
|
|
37
|
+
[recursive item fulfillment engine](../../../2.0/apps/_underscore/features/recursive-item-fulfillments.md)
|
|
38
|
+
does the mirroring — it is **not** Compass code.
|
|
39
|
+
|
|
40
|
+
**The bug (found 2026-09-18):** the mirroring only ever ran when the fulfillment *chain* was built.
|
|
41
|
+
Adding a tracking number afterwards triggered nothing, so the tracking and serials stayed on the
|
|
42
|
+
Agilant tier. **1,819 stranded cases since 2026-08-01; 1,723 of them (95%) had tracking created
|
|
43
|
+
AFTER the chain was built.** Compass-only — `Client_Elite` returns 0 stranded rows because it has
|
|
44
|
+
no multi-tier chain.
|
|
45
|
+
|
|
46
|
+
## How it works
|
|
47
|
+
|
|
48
|
+
### The three tiers
|
|
49
|
+
|
|
50
|
+
```
|
|
51
|
+
Compass SO (customer-facing, what the user opens)
|
|
52
|
+
↑ upstreamItemFulfillmentId
|
|
53
|
+
Office Depot SO (intermediate)
|
|
54
|
+
↑ upstreamItemFulfillmentId
|
|
55
|
+
Agilant SO (ships; owns the real tracking + serials)
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
Units are shared records: a unit mirrors up matched on **`unitId` only**. Tracking mirrors up via
|
|
59
|
+
the three bridges — 317 `ItemFulfillments_TrackingNumbers` (header), 318
|
|
60
|
+
`ItemFulfillmentItems_TrackingNumbers` (item), 319 `ItemFulfillmentItemUnits_TrackingNumbers`
|
|
61
|
+
(unit). See [tracking-number bridges](../../../2.0/apps/_underscore/features/tracking-number-bridges.md).
|
|
62
|
+
|
|
63
|
+
### Root cause — records 317/318/319 had ZERO interceptor rows
|
|
64
|
+
|
|
65
|
+
Parent records **28/29/30** (IF / IF item / IF item unit) had `Core.ApiPayloadInterceptors` rows;
|
|
66
|
+
the three tracking **bridge** records **317/318/319 had none**. Header tracking used to live in its
|
|
67
|
+
own table `ItemFulfillmentPackages` (record 41) with its own interceptor; when that was consolidated
|
|
68
|
+
into the plain bridge `ItemFulfillments_TrackingNumbers`, **the interceptor registration did not
|
|
69
|
+
follow**. So `_Model_Client_ItemFulfillment::reconcileUpstreamPackages()` (~L691) — which is
|
|
70
|
+
correct code — never fired when a tracking number was attached.
|
|
71
|
+
|
|
72
|
+
### The fix (two halves — both required)
|
|
73
|
+
|
|
74
|
+
1. **PHP** — `postPost` / `postPut` + a private `triggerUpstreamSync()` added to all three bridge
|
|
75
|
+
models, calling `_Model_Client_ItemFulfillment::ensureUpstreamItemFulfillment()`. Pattern copied
|
|
76
|
+
verbatim from the working sibling `Model/Client/ItemFulfillmentItem.php` (L18-46).
|
|
77
|
+
2. **SQL** — `dbchanges2/Core/2026-09-18a - ItemFulfillmentTrackingNumberInterceptors.sql` registers
|
|
78
|
+
POST/POST + POST/PUT for records 317, 318, 319 (6 INSERTs). **The PHP does nothing without it**
|
|
79
|
+
— interceptors are DB-registered, never auto-discovered
|
|
80
|
+
([api payload interceptors](../../../2.0/apps/api2/features/api-payload-interceptors.md)).
|
|
81
|
+
|
|
82
|
+
### `_Model_Compass_SalesOrder::_trackingNumbers()` — the field that papered over the gap
|
|
83
|
+
|
|
84
|
+
A `FIELD_SQL` computed field (`Client_Compass.CustomRecordFields` id **83**, recordId **14**). It is
|
|
85
|
+
a UNION of two branches:
|
|
86
|
+
|
|
87
|
+
- **ASN branch** — reads `AdvanceShippingNoticeItemUnits_TrackingNumbers`. **Load-bearing; do NOT
|
|
88
|
+
delete it.** 9,681 orders system-wide have ASN tracking that exists on **no** fulfillment.
|
|
89
|
+
Declining (658 in May 2026 → 7 in Sep 2026) but still live. Verified: SA137652 carries
|
|
90
|
+
`483164100002` only via the ASN.
|
|
91
|
+
- **Fulfillment branch** — used to walk a hardcoded **4-hop PO bridge**
|
|
92
|
+
(`SalesOrders_PurchaseOrders` / `PurchaseOrders_SalesOrders`, with aliases literally named
|
|
93
|
+
`CompassSO_PO` / `CompassPO_OfficeDepotSO` / `OfficeDepotSO_PO` / `OfficeDepotPO_AgilantSO`) to
|
|
94
|
+
read tracking off **another order's** fulfillment. Simplified 2026-09-18 to a direct 3-table join
|
|
95
|
+
on **this order's own** `ItemFulfillments`.
|
|
96
|
+
|
|
97
|
+
**This is why the Sales Orders grid looked right while the order page looked broken.** The grid used
|
|
98
|
+
`_trackingNumbers()`, which reached across to the Agilant order; the **Shipments panel**
|
|
99
|
+
(`toga25-supply/src/pages/SalesOrders/view/SalesOrderRecordModalLayout/hooks/useShipments.ts`) and
|
|
100
|
+
the **Item Fulfillments modal** (TableView 12, reads only `ItemFulfillmentItems_TrackingNumbers`)
|
|
101
|
+
both scope to `salesOrderId = this order`, so they showed nothing. Same order, two answers.
|
|
102
|
+
|
|
103
|
+
The simplification was proven a **strict superset** across 8,336 orders: old-finds-new-misses = **0**;
|
|
104
|
+
the new query finds tracking on **5,264** orders the old one missed; 1,046 differ, all supersets.
|
|
105
|
+
|
|
106
|
+
## Gotchas
|
|
107
|
+
|
|
108
|
+
- **⚠ The interceptor SQL is NOT run and `_underscore` is NOT deployed (as of 2026-09-22).** Until
|
|
109
|
+
both land, new shipments keep stranding at roughly **365 every 3 days**. The backfills only fix
|
|
110
|
+
history.
|
|
111
|
+
- **⚠ No end-to-end test exists.** No sandbox POST has proven the new hooks actually fire. The
|
|
112
|
+
registration and the method names were verified by reading, not by running.
|
|
113
|
+
- **Never match units on part number — only on `unitId`.** **32,585** `upstreamItemFulfillmentItemId`
|
|
114
|
+
links legitimately join **different** part numbers: real kitting/bundle roll-ups
|
|
115
|
+
(`PC-KIT-OPT2` → `C40QYUC`), vendor suffixes (`C40QYUC#ABA` → `C40QYUC`), and service lines
|
|
116
|
+
(`ODP-COMPASS-ASSETTAG1`). These are correct and **must not be "fixed"**. A unit is one physical
|
|
117
|
+
device, so `unitId` is the only safe key.
|
|
118
|
+
- **Asset Tag is a capture gap, not a sync gap.** `Units.assetTag` is NULL on **every** tier
|
|
119
|
+
including the origin. There is nothing to mirror — chasing it through the chain is wasted work.
|
|
120
|
+
Needs its own ticket at the capture end.
|
|
121
|
+
- **Open bug — broken PO↔SO item bridge (needs its own ticket).** **2,918** `ItemFulfillmentItems`
|
|
122
|
+
have no matching upstream part and cannot be linked, because their
|
|
123
|
+
`PurchaseOrderItems_SalesOrderItems` rows are **missing entirely**. Example: Office Depot SO lines
|
|
124
|
+
619922 / 619923 / 619924 have **0** bridge rows, while line 619925 wrongly maps to all 4 Compass
|
|
125
|
+
lines. Same integrity class as the off-by-one bridge breakage in
|
|
126
|
+
[asn-to-item-fulfillment](asn-to-item-fulfillment.md).
|
|
127
|
+
- **Latent — `triggerUpstreamSync()` skips silently.** It returns quietly when no parent fulfillment
|
|
128
|
+
is found (`if ($itemFulfillmentUuid)`). This matches the existing sibling's behaviour but violates
|
|
129
|
+
the team's throw-don't-skip rule. Harden all four files together, not one at a time.
|
|
130
|
+
|
|
131
|
+
## Backfill results (both run and complete)
|
|
132
|
+
|
|
133
|
+
| Backfill | Compass customer orders fixed | Rows added |
|
|
134
|
+
|---|---|---|
|
|
135
|
+
| Tracking (`2026-09-18a`) | **1,651** (+4,293 Office Depot intermediate) | 7,436 header + 11,856 item |
|
|
136
|
+
| Units/serials (`2026-09-21a`) | **1,096** (+1,530 Office Depot, 658 Agilant) | unit rows + their tracking bridges |
|
|
137
|
+
|
|
138
|
+
Verify counts all **0**. Integrity clean: 0 duplicates, 0 orphans, 0 self-links, 0 serials leaking
|
|
139
|
+
across unrelated orders. The multi-pass technique these backfills needed is written up in
|
|
140
|
+
[re-runnable additive migrations](../../../2.0/apps/dbchanges2/features/rerunnable-additive-inserts.md).
|
|
141
|
+
|
|
142
|
+
## Change history
|
|
143
|
+
- 2026-09-22 — First write-up. Root-caused blank tracking/serials on Compass customer orders to
|
|
144
|
+
`Core.ApiPayloadInterceptors` having **zero** rows for bridge records 317/318/319 (lost when
|
|
145
|
+
`ItemFulfillmentPackages`/record 41 was consolidated into `ItemFulfillments_TrackingNumbers`), so
|
|
146
|
+
`reconcileUpstreamPackages()` never fired on tracking attach — 1,819 stranded cases since
|
|
147
|
+
2026-08-01, 95% with tracking added after the chain was built. Added `postPost`/`postPut` to the
|
|
148
|
+
three bridge models, simplified `_trackingNumbers()`'s fulfillment branch from a 4-hop PO-bridge
|
|
149
|
+
walk to a direct join on the order's own fulfillments (proven strict superset over 8,336 orders),
|
|
150
|
+
and kept the load-bearing ASN branch. Backfilled 1,651 orders' tracking and 1,096 orders' serials.
|
|
151
|
+
Core interceptor SQL still unrun and `_underscore` undeployed. (rgirish)
|
|
@@ -20,8 +20,8 @@ project: _Underscore
|
|
|
20
20
|
client: compass-usa
|
|
21
21
|
type: profile
|
|
22
22
|
status: active
|
|
23
|
-
updated: 2026-09-
|
|
24
|
-
owners: [jcardinal, bala, tcox, apeterson, dfranks, akhokhani, ajean]
|
|
23
|
+
updated: 2026-09-22
|
|
24
|
+
owners: [jcardinal, bala, tcox, apeterson, dfranks, akhokhani, ajean, rgirish]
|
|
25
25
|
files: []
|
|
26
26
|
related:
|
|
27
27
|
- ../../2.0/apps/_underscore/features/sales-order-po-number-sourcing.md
|
|
@@ -37,6 +37,7 @@ related:
|
|
|
37
37
|
- features/oneuptime-ma-refresh-order-monitor.md
|
|
38
38
|
- workflows/catalog-fold-reversal.md
|
|
39
39
|
- features/asn-to-item-fulfillment.md
|
|
40
|
+
- features/multi-tier-tracking-and-serial-mirroring.md
|
|
40
41
|
- features/order-fulfillment-status-per-line.md
|
|
41
42
|
- features/cost-centers.md
|
|
42
43
|
- features/isfulfillable-data-quality-and-type-rule.md
|
|
@@ -272,6 +273,7 @@ separate, related client (see its own profile).
|
|
|
272
273
|
clean; a **live Agilant Drop Ship spike** started 2026-08-11.
|
|
273
274
|
[MR/MA Order Approval & Status Gate](features/mr-ma-order-approval-and-status.md).
|
|
274
275
|
- Built on the shared 2.0 Recursive Item Fulfillments engine (upstream mirroring).
|
|
276
|
+
- **⚠ Compass is a THREE-tier drop-ship chain (Compass SO → Office Depot SO → Agilant SO), and tracking/serials were not mirroring up (2026-09-18, fix NOT deployed).** Agilant ships and owns the real tracking numbers and unit serials; they must mirror up to the customer-facing Compass order. They did not, because `Core.ApiPayloadInterceptors` had **zero** rows for the three tracking-bridge records 317/318/319 — lost when `ItemFulfillmentPackages` was consolidated. **1,819 stranded orders since 2026-08-01**; Elite is unaffected (no multi-tier chain). Backfills fixed 1,651 orders' tracking and 1,096 orders' serials, but the **Core interceptor SQL is still unrun and `_underscore` undeployed**, so new shipments keep stranding at ~365 every 3 days. Also explains why the Sales Orders **grid** showed tracking while the order page's Shipments panel and Item Fulfillments modal showed none. [Multi-tier tracking & serial mirroring](features/multi-tier-tracking-and-serial-mirroring.md).
|
|
275
277
|
- **`Items.isFulfillable` gates the storefront Qty Fulfilled cell (2026-07).** Sourced from
|
|
276
278
|
NetSuite onto the Agilant source item during the 1.0 item sync
|
|
277
279
|
([Phase 1](../../1.0/apps/library/features/netsuite-item-isfulfillable-sync.md)) and propagated
|
package/knowledge/registry.json
CHANGED
package/package.json
CHANGED