toga-ai 1.0.590 → 1.0.592
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/1.0/apps/library/features/error-capture-1-0.md +16 -1
- package/knowledge/1.0/apps/library/features/toga2-api-client-and-bridge.md +18 -1
- package/knowledge/1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md +60 -8
- package/knowledge/2.0/apps/_underscore/features/per-client-database-connections.md +34 -28
- package/knowledge/2.0/apps/api2/features/api-payload-interceptors.md +9 -2
- package/knowledge/2.0/apps/api2/features/nested-relationship-writes.md +67 -0
- package/knowledge/2.0/apps/dbchanges2/INDEX.md +1 -1
- package/knowledge/2.0/apps/dbchanges2/workflows/nonprod-metadata-drift-repair.md +38 -0
- package/knowledge/clients/aig/features/entitlement-intake.md +59 -24
- package/package.json +1 -1
|
@@ -6,7 +6,7 @@ project: Library
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-08-
|
|
9
|
+
updated: 2026-08-17
|
|
10
10
|
owners: ["jcardinal", "mhammontree"]
|
|
11
11
|
files:
|
|
12
12
|
- library/app/error/capture.php
|
|
@@ -177,6 +177,15 @@ exists in `api2`/`worker2`. So 1.0 reads the AL1 container config at
|
|
|
177
177
|
|
|
178
178
|
## Gotchas / known issues
|
|
179
179
|
|
|
180
|
+
- **⚠ A grouped `Logs.Issue` row can show a STALE error id — read the live api2 transaction log for
|
|
181
|
+
the current error.** Two facts combine to mislead: (1) `Event.errorMessage` is **`varchar(255)`**,
|
|
182
|
+
so a long API response is **truncated** — often before the trailing error id; and (2)
|
|
183
|
+
`Logs.Issue.subject` / `Issue.errorMessage` hold the **first-occurrence** text, so a long-lived
|
|
184
|
+
issue keeps showing the error id from whenever it was first seen even after the per-call error has
|
|
185
|
+
changed. Observed 2026-08-17: issue `3H` still displayed `EO-1` / `3G-6` (the original
|
|
186
|
+
undefined-method fatal) while the current per-call failure was actually `EV-11` (an FK 1451 on a
|
|
187
|
+
DELETE). **To see what is failing *now*, read the live api2 transaction log
|
|
188
|
+
(`Logs_<client>.Api.responsePayload`) for the specific call — not the grouped `Issue` row.**
|
|
180
189
|
- **Any PHP warning inside capture code would kill the request.** 1.0's
|
|
181
190
|
`App_Error::handleError()` routes **every** PHP warning into `handleException()`, which renders
|
|
182
191
|
the red box and calls `exit()` — so a single warning raised while capturing (an unreachable IMDS
|
|
@@ -220,6 +229,12 @@ exists in `api2`/`worker2`. So 1.0 reads the AL1 container config at
|
|
|
220
229
|
|
|
221
230
|
## Change history
|
|
222
231
|
|
|
232
|
+
- 2026-08-17 — Recorded a diagnostic trap: a grouped `Logs.Issue` row can show a **stale error id**
|
|
233
|
+
because `Event.errorMessage` is `varchar(255)` (truncates a long API response, often before the
|
|
234
|
+
error id) and `Issue.subject`/`errorMessage` keep the **first-occurrence** text — observed issue
|
|
235
|
+
`3H` still showing `EO-1`/`3G-6` while the live per-call error was `EV-11` (FK 1451). Read the live
|
|
236
|
+
`Logs_<client>.Api.responsePayload` for the actual current error. Surfaced diagnosing the Compass
|
|
237
|
+
USA NetSuite SO sync freeze. (jcardinal)
|
|
223
238
|
- 2026-08-05 — Recorded the legacy `DELETE_DOA_PRE-SENTRY_error.php` handler leaking the full
|
|
224
239
|
config array (prod DB / SES / NetSuite / Payeezy / SSL Store credentials + ticket PII) on a
|
|
225
240
|
live towfoundation.togaview.com error page, meaning that host evaluates as dev mode in
|
|
@@ -6,7 +6,7 @@ project: Library
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-08-
|
|
9
|
+
updated: 2026-08-17
|
|
10
10
|
owners: [jcardinal, mhammontree, bala]
|
|
11
11
|
files:
|
|
12
12
|
- library/app/api/toga2.php
|
|
@@ -328,6 +328,16 @@ enable flags** and an optional `$monitorTogadeskDepartmentIds[]`:
|
|
|
328
328
|
**and** the consuming code. (The general 2.0-side metadata behavior — reverse relations resolved via
|
|
329
329
|
the child model's `FOREIGNKEY_MODEL` and named as the camelCase plural of the child model — is a
|
|
330
330
|
framework-level fact captured separately.)
|
|
331
|
+
- **⚠ Depth alone does NOT return every reverse-hasMany collection — `GET /sales-orders` at depth 4
|
|
332
|
+
omits `itemFulfillmentItems` and `invoiceItems`.** Verified 2026-08-17 against prod
|
|
333
|
+
`Logs_Compass.Api` responses (no fields whitelist): a depth-4 sales-order GET returns
|
|
334
|
+
`salesOrderItems` and the **PO-link bridge** collections
|
|
335
|
+
(`salesOrderItemPurchaseOrderItems`, `purchaseOrderItemSalesOrderItems`) but **not**
|
|
336
|
+
`itemFulfillmentItems` or `invoiceItems`. So any importer code that guards on
|
|
337
|
+
`$salesOrderItem->itemFulfillmentItems` / `->invoiceItems` from that GET is a **silent no-op** —
|
|
338
|
+
same reverse-hasMany trap as above, now with a concrete consequence: the NetSuite SO renumber fix
|
|
339
|
+
keys off a recorded "could-not-delete" set instead of trusting those collections (see
|
|
340
|
+
[NetSuite → TOGa Supply per-client sync](../../worker/features/netsuite-togasupply-per-client-sync.md)).
|
|
331
341
|
- **`TOGA_AIG.Customers.emailAddress` is `varchar(255)`** (V1 legacy, latin1_swedish_ci) — wide
|
|
332
342
|
enough to hold several `'; '`-joined emails, so TRUE-79401 needed **no** schema/dbchanges migration
|
|
333
343
|
(the earlier fear that it was ~VARCHAR(55) was wrong) — but **it is a real ceiling: ~6–7
|
|
@@ -386,6 +396,13 @@ enable flags** and an optional `$monitorTogadeskDepartmentIds[]`:
|
|
|
386
396
|
|
|
387
397
|
## Change history
|
|
388
398
|
|
|
399
|
+
- 2026-08-17 — Recorded that **depth-4 `GET /sales-orders` omits the `itemFulfillmentItems` and
|
|
400
|
+
`invoiceItems` reverse-hasMany collections** (it does return `salesOrderItems` and the PO-link
|
|
401
|
+
bridges), verified against prod `Logs_Compass.Api` — so code guarding on those collections from
|
|
402
|
+
that GET is a silent no-op, reinforcing the reverse-hasMany-fields trap. Surfaced fixing the
|
|
403
|
+
NetSuite SO renumber freeze (Compass USA), which now keys off a recorded "could-not-delete" set;
|
|
404
|
+
see [NetSuite → TOGa Supply per-client sync](../../worker/features/netsuite-togasupply-per-client-sync.md).
|
|
405
|
+
(jcardinal)
|
|
389
406
|
- 2026-08-12 — Recorded the **`send()` return contract**: it always returns a response carrying
|
|
390
407
|
`isSuccess` (early-returns on true, falls through to `return $response` on false, throws when the
|
|
391
408
|
property is absent), so `empty($response->isSuccess)` is a safe failure check on anything it
|
|
@@ -6,7 +6,7 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-08-
|
|
9
|
+
updated: 2026-08-17
|
|
10
10
|
owners: ["dfranks", "bala", "jcardinal", "mhammontree", "snaredla"]
|
|
11
11
|
files:
|
|
12
12
|
- worker/crons/toga2/netsuite/common_sync_togasupply.php
|
|
@@ -87,13 +87,28 @@ its IDLE write. **There is no timeout, memory cap, or resource limit anywhere in
|
|
|
87
87
|
never *tune the window size down*. (Corollary: a window that has grown back to the cap is positive
|
|
88
88
|
evidence the section is completing.)
|
|
89
89
|
|
|
90
|
-
### ⚠
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
90
|
+
### ⚠ Per-record isolation is SECTION-SPECIFIC — and SALES_ORDERS has NONE (corrected 2026-08-17)
|
|
91
|
+
|
|
92
|
+
Whether a bad record is "stepped over" or freezes the section depends entirely on whether **that
|
|
93
|
+
section's** record loop wraps the `sync*FromNetsuite()` call in a per-record `try/catch`. It is not
|
|
94
|
+
uniform across the six sections, and the earlier blanket claim here ("a per-record failure does not
|
|
95
|
+
stop the cursor — the record is skipped forever") is **false for the SALES_ORDERS section**.
|
|
96
|
+
|
|
97
|
+
- **Where a section DOES isolate per record:** the cursor write (`LAST_SYNC_DATETIME_*`) and
|
|
98
|
+
`finishModeIteration()` run after the loop, so a caught record is skipped and the section still
|
|
99
|
+
reports IDLE and advances. There, **"section IDLE + cursor advancing" is not proof of a complete
|
|
100
|
+
import** — it only proves the run didn't die (recovery = rewind the cursor; re-import is
|
|
101
|
+
idempotent).
|
|
102
|
+
- **⚠ SALES_ORDERS has NO per-record `try/catch`.** The `SALES_ORDERS` foreach in
|
|
103
|
+
`common_sync_togasupply.php` (~L637-673) calls `App_Api_Toga2::syncSalesOrderFromNetsuite()`
|
|
104
|
+
**bare**. `send()` throws on any non-success by default, so **one poison order throws, propagates,
|
|
105
|
+
and aborts the whole section BEFORE** the cursor PUT + `finishModeIteration()`. The mode stays
|
|
106
|
+
`1-RUNNING`, the watermark freezes, the window shrinks ÷3 each run, and the **same** order is
|
|
107
|
+
re-tried and re-throws every 5 minutes — a permanent freeze, not a skip. So a stuck
|
|
108
|
+
`SALES_ORDERS` at `1-RUNNING` with a days-frozen watermark means **it is throwing on a specific
|
|
109
|
+
order** (go read the live api2 transaction log for that client — see the poison-order gotcha
|
|
110
|
+
below). The "shrinking window is a bug signal" rule still holds; the "record is stepped over"
|
|
111
|
+
corollary does **not** for this section.
|
|
97
112
|
|
|
98
113
|
Proven concretely on Elite: sales order **169441** (NetSuite internal id 4624062, trandate
|
|
99
114
|
2024-04-15, customer PO `BEGGS/04152024/…`) was simply absent from `Client_Elite.SalesOrders`
|
|
@@ -361,9 +376,46 @@ Parameters are stored **per client DB** but accessed **through the TOGa2 API**,
|
|
|
361
376
|
a new line is actually created. Full root cause, the prod cleanup migration, and the
|
|
362
377
|
deploy-fix-before-cleanup sequencing live in the
|
|
363
378
|
[Compass ODP Duplicate PO-Line Cleanup](../../../clients/compass-usa/workflows/odp-duplicate-po-line-cleanup.md).
|
|
379
|
+
- **⚠ A single un-deletable renumbered line could freeze the entire SALES_ORDERS section (fixed
|
|
380
|
+
2026-08-17).** When NetSuite renumbers a line, `syncSalesOrderFromNetsuite` renumbers by
|
|
381
|
+
**DELETE `/sales-order-items/{uuid}` then re-create in the per-item loop**. Its "pinned row" guard
|
|
382
|
+
only skipped rows carrying `itemFulfillmentItems` / `salesOrderItemPurchaseOrderItems` /
|
|
383
|
+
`purchaseOrderItemSalesOrderItems`; an **INVOICED** line
|
|
384
|
+
(`InvoiceItems.salesOrderItemId`, `ON DELETE RESTRICT`) was **not** guarded, so the DELETE failed
|
|
385
|
+
with MySQL **1451** → api2 **EV-11** 400, and — because the SALES_ORDERS loop has no per-record
|
|
386
|
+
`try/catch` (see the corrected gotcha above) — the throw aborted the section, freezing Compass USA
|
|
387
|
+
for ~5 days (`NETSUITE_EXECUTION_MODE_SALES_ORDERS='1-RUNNING'`, watermark stuck at
|
|
388
|
+
`2026-08-12 06:13:26`) while it re-tried the same poison order every run (SO **7447** /
|
|
389
|
+
`SalesOrderItem` 47828, invoiced by `InvoiceItem` 25515). **Fix (library-only, PHP 7.2-safe, no
|
|
390
|
+
signature changes)** in `syncSalesOrderFromNetsuite`: (1) the renumber-DELETE now passes
|
|
391
|
+
`$throwExceptionOnApiError=false`, checks `empty($resp->isSuccess)`, and records any row it cannot
|
|
392
|
+
delete in a new `$pinnedRenumberLineUniqueKeys` set (keyed by `c_netsuiteLineUniqueKey`); (2) the
|
|
393
|
+
per-item PUT loop treats a row in that set as **pinned** — it keeps its current `lineNumber` (no
|
|
394
|
+
`UNIQUE(salesOrderId,lineNumber)` collision) and its PUT is made non-throwing + `error_log` **for
|
|
395
|
+
pinned rows only** (normal rows keep throw-on-error so real failures still surface); (3) the
|
|
396
|
+
trailing "remove dropped items" DELETE now skips `invoiceItems` and checks `isSuccess`. Net: **no
|
|
397
|
+
single order can freeze the section again regardless of which `RESTRICT` child blocks a delete**;
|
|
398
|
+
the invoiced line simply stays at its existing `lineNumber` (a documented unresolvable
|
|
399
|
+
NetSuite-vs-invoiced case). Deploy `library` only — no SQL, no other repos — then the section
|
|
400
|
+
completes, mode flips IDLE, and the backlog backfills idempotently. Note the collection guard added
|
|
401
|
+
for `invoiceItems` is **best-effort only**: depth-4 `GET /sales-orders` does **not** return the
|
|
402
|
+
`itemFulfillmentItems` or `invoiceItems` reverse-hasMany collections (only the PO-link bridges), so
|
|
403
|
+
the fix keys off the recorded "could-not-delete" set rather than trusting those collections — see
|
|
404
|
+
[App_Api_Toga2 client & bridge](../../library/features/toga2-api-client-and-bridge.md).
|
|
364
405
|
|
|
365
406
|
## Change history
|
|
366
407
|
|
|
408
|
+
- 2026-08-17 — Fixed a Compass USA ~5-day SALES_ORDERS freeze: an **invoiced** line that NetSuite
|
|
409
|
+
renumbered could not be DELETEd (FK `InvoiceItems.salesOrderItemId` `RESTRICT` → MySQL 1451 →
|
|
410
|
+
api2 EV-11), and because the SALES_ORDERS loop has **no per-record `try/catch`** the throw aborted
|
|
411
|
+
the whole section (mode stuck `1-RUNNING`, watermark frozen, poison order SO 7447 re-tried every
|
|
412
|
+
run). `syncSalesOrderFromNetsuite` now makes the renumber-DELETE non-throwing, records
|
|
413
|
+
un-deletable rows in a `$pinnedRenumberLineUniqueKeys` set, and keeps a pinned row at its existing
|
|
414
|
+
`lineNumber` with a non-throwing PUT (normal rows still throw on error); the trailing dropped-item
|
|
415
|
+
DELETE now skips `invoiceItems` and checks `isSuccess`. **Corrected** the prior gotcha that claimed
|
|
416
|
+
a per-record failure never stops the cursor — that is **section-specific and false for
|
|
417
|
+
SALES_ORDERS**, which has no per-record isolation. Library-only deploy; no SQL/other repos.
|
|
418
|
+
Diagnosis touched worker/api2/_underscore (read-only). (jcardinal)
|
|
367
419
|
- 2026-08-12 — TRUE-80499 (Elite onboarding, shared-engine work). **Fixed** the sparse-client
|
|
368
420
|
`TypeError`: the two per-client item lookups are now initialised at
|
|
369
421
|
`common_sync_togasupply.php:74-75` instead of only inside the items loop, so a client whose items
|
|
@@ -178,41 +178,47 @@ here — they live in `Config/*.ini`.)
|
|
|
178
178
|
(`db_logs`) — the laptop trap there is documented separately in the worker NetSuite bootstrap
|
|
179
179
|
notes.
|
|
180
180
|
|
|
181
|
-
##
|
|
181
|
+
## WITHDRAWN — the "alias-keyed `$_modelCache` cross-tenant leak" hypothesis (2026-08-17)
|
|
182
182
|
|
|
183
|
-
**
|
|
184
|
-
|
|
183
|
+
**There is no known cross-tenant leak. Do not cite this as a security concern.** An earlier entry in
|
|
184
|
+
this doc (same day, TRUE-80562) raised an *unproven* hypothesis that `_Database::$_modelCache`,
|
|
185
|
+
being keyed by the shared alias `'Client'`, was emitting one tenant's `c_` columns on another
|
|
186
|
+
tenant's `INSERT`. **That is not what happened.** The trigger was an AIG `Units` write failing MySQL
|
|
187
|
+
1054 on six `c_` columns; the real cause was a **missing migration on beta**, not a cache collision:
|
|
185
188
|
|
|
186
|
-
|
|
187
|
-
`
|
|
188
|
-
`
|
|
189
|
-
`
|
|
189
|
+
- `dbchanges2/_modules/netsuite/2026-07-10a - UnitInventoryFields.sql` adds all six columns
|
|
190
|
+
(`c_itemDescription`, `c_lastModified`, `c_chargingBrick`, `c_deviceStatus`, `c_chargingCable`,
|
|
191
|
+
`c_grade`) **and** their `CustomRecordFields` rows to every client whose `_modules.txt` lists
|
|
192
|
+
`netsuite`. **`Client_Aig/_modules.txt` contains exactly `netsuite`** — AIG is opted in, so those
|
|
193
|
+
are **AIG's own fields**, not another tenant's.
|
|
194
|
+
- **Prod `Client_Aig.Units` has all 7 custom columns.** Beta simply never ran that module file — the
|
|
195
|
+
same execution gap as `Entitlements.serviceAddressId`. After the module migration was run manually
|
|
196
|
+
on beta `Client_Aig`, `Units` gained all 7 columns and the nested unit write succeeded.
|
|
190
197
|
|
|
191
|
-
**
|
|
198
|
+
**Methodology lesson — why the wrong conclusion was reached.** The model-class comparison that
|
|
199
|
+
produced the leak hypothesis was made against the **local `_underscore` checkout, which was on branch
|
|
200
|
+
`TRUE-80282`, while beta deploys `_sandbox-dev`.** Comparing beta *runtime* behavior against a local
|
|
201
|
+
checkout on a different branch is **not evidence**. api2's architecture doc already warns that
|
|
202
|
+
`_underscore` is cloned at build time from a moving branch
|
|
203
|
+
([environment-variable-drives-underscore-branch](../../api2/features/environment-variable-drives-underscore-branch.md));
|
|
204
|
+
this is that hazard biting in practice. Before attributing a runtime field list to a model class,
|
|
205
|
+
confirm which branch the environment actually built from.
|
|
192
206
|
|
|
193
|
-
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
- Those six columns exist **only** in `Client_Adyen`, `Client_Elite`, `Client_Growrk`,
|
|
197
|
-
`Client_Prudential`. Five are declared **only** on `_Model_Growrk_Unit`; `_Model_Aig_Unit` extends
|
|
198
|
-
`_Model_Client_Unit` and declares none. `c_lastModified` is declared in **no** model anywhere in
|
|
199
|
-
`_underscore`.
|
|
200
|
-
- So the field list is **neither** AIG's schema, **nor** AIG's metadata, **nor** any model class AIG
|
|
201
|
-
uses — it is exactly **four other tenants'** column set.
|
|
202
|
-
|
|
203
|
-
**Hypothesis (not proven):** `_underscore/Model.php` (~676–687) caches query results as
|
|
204
|
-
`_Database::$_modelCache[$this::DATABASE][$sql]`, and `DATABASE` is the shared **alias** `'Client'`
|
|
205
|
-
— *not* the tenant schema name — so two tenants collide on the same cache key (the same alias-keying
|
|
206
|
-
already documented in the gotcha above). **Unknown: whether that static survives across requests
|
|
207
|
-
under PHP-FPM.** That single unknown is the difference between a harmless within-request artifact and
|
|
208
|
-
a **cross-tenant data-leak vector**, which is why it must be answered rather than assumed.
|
|
209
|
-
|
|
210
|
-
**Where to look:** `_underscore/Model.php` ~676–687, `api2/Component/Api/V2/V2.php` ~8918–8931,
|
|
211
|
-
`_underscore/Model/Growrk/Unit.php`, `_underscore/Model/Aig/Unit.php`. Context:
|
|
212
|
-
[AIG entitlement intake](../../../clients/aig/features/entitlement-intake.md).
|
|
207
|
+
**The real generalizable fact** — the beta migration fan-out gap is **not limited to `Client/`; it
|
|
208
|
+
also hits `_modules/<module>/`** — is recorded in
|
|
209
|
+
[non-prod metadata drift repair](../../dbchanges2/workflows/nonprod-metadata-drift-repair.md).
|
|
213
210
|
|
|
214
211
|
## Change history
|
|
215
212
|
|
|
213
|
+
- 2026-08-17 (later pass) — **WITHDRAWN, supersedes the entry below.** The alias-keyed
|
|
214
|
+
`$_modelCache` cross-tenant-leak hypothesis is **not supported** and is no longer a live security
|
|
215
|
+
concern. The six `c_` columns are **AIG's own**: `_modules/netsuite/2026-07-10a -
|
|
216
|
+
UnitInventoryFields.sql` adds them to every client whose `_modules.txt` lists `netsuite`, and
|
|
217
|
+
`Client_Aig/_modules.txt` contains exactly that; prod `Client_Aig.Units` has all 7 custom columns
|
|
218
|
+
and beta had simply never run the module file. Running it on beta fixed the write. The wrong
|
|
219
|
+
conclusion came from comparing beta runtime behavior against a **local `_underscore` checkout on
|
|
220
|
+
branch `TRUE-80282`** while beta deploys `_sandbox-dev` — the moving-branch hazard the api2
|
|
221
|
+
architecture doc already warns about. (mhammontree)
|
|
216
222
|
- 2026-08-17 — Recorded an **unproven open question** (TRUE-80562): an AIG nested `Units` write
|
|
217
223
|
emitted four *other* tenants' `c_` columns (1054 → EV-12) although AIG's schema, its
|
|
218
224
|
`CustomRecordFields` metadata, and its model class all declare exactly one custom field. Suspected
|
|
@@ -6,8 +6,8 @@ project: API
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-08-
|
|
10
|
-
owners: ["mhammontree", "dfranks", "bala", "snaredla"]
|
|
9
|
+
updated: 2026-08-17
|
|
10
|
+
owners: ["mhammontree", "dfranks", "bala", "snaredla", "jcardinal"]
|
|
11
11
|
files:
|
|
12
12
|
- api2/Component/Api/V2/V2.php
|
|
13
13
|
- api2/Controller/Index.php
|
|
@@ -448,6 +448,13 @@ and the failing environment**. It is a small table, and the drift is usually exa
|
|
|
448
448
|
|
|
449
449
|
## Change history
|
|
450
450
|
|
|
451
|
+
- 2026-08-17 — Corroborated that the unguarded-dispatch fatal is **systemic, not client-specific**:
|
|
452
|
+
the prod "undefined method" issue bucket (`Logs.Issue` ref 3G / id 59) grouped **both**
|
|
453
|
+
`_Model_Compass_Usa_SalesOrderItem::postPost()` **and** `_Model_Elite_SalesOrder::postPut()` — two
|
|
454
|
+
different tenants, one shared fragility. (This was the original Compass USA failure on 2026-08-12,
|
|
455
|
+
before it was mitigated on the 2.0 side and the client's NetSuite sync progressed to an unrelated
|
|
456
|
+
FK failure downstream.) Reinforces the standing fix direction: guard all four `V2.php` call sites
|
|
457
|
+
with `method_exists()`. Investigation only, no code change. (jcardinal)
|
|
451
458
|
- 2026-08-14 — TRUE-80489, from building the Compass inactive-standalone-item order guard.
|
|
452
459
|
(1) **A PRE interceptor on a PARENT record fires on the nested child POST, with the CHILD as the
|
|
453
460
|
payload.** `POST /sales-orders/{uuid}/sales-order-items` resolves `SalesOrders` as the
|
|
@@ -94,6 +94,60 @@ fixing. Until then the caller-side workaround is a follow-up identifier-only `PU
|
|
|
94
94
|
self-heal that detects the null/mismatched echo (shipped on the BDR funnel side — see
|
|
95
95
|
web-funnel-app.md).
|
|
96
96
|
|
|
97
|
+
## ⚠ CREATE path: a nested singular FK is matched TENANT-WIDE by VALUE and RE-PARENTED
|
|
98
|
+
|
|
99
|
+
The forward-singular-FK back-reference machinery has a second, worse manifestation — on the
|
|
100
|
+
**CREATE** path, and the damage is a **stolen child row** rather than a null pointer.
|
|
101
|
+
|
|
102
|
+
A nested `primaryContactEmailAddress` / `primaryContactPhoneNumber` object is matched **across the
|
|
103
|
+
whole tenant by its VALUE** (`emailAddress` / `phoneNumber`) — **not** by `uuid` — and the matched
|
|
104
|
+
child's back-reference `contactId` is then **flipped to the newest writer**, silently detaching it
|
|
105
|
+
from its previous owner.
|
|
106
|
+
|
|
107
|
+
**Measured on beta `Client_Aig` 2026-08-17 (TRUE-80562).** Two sequential POSTs each created a new
|
|
108
|
+
Contact (**854** at 10:19, **856** at 10:41). Afterwards:
|
|
109
|
+
|
|
110
|
+
- Contacts **854 AND 856** both have `primaryContactEmailAddressId = 250` and
|
|
111
|
+
`primaryContactPhoneNumberId = 18`.
|
|
112
|
+
- `ContactEmailAddresses` row **250** has `contactId = 856`.
|
|
113
|
+
- So contact **854 still points at a child row that now belongs to contact 856** — 854's primary
|
|
114
|
+
email is effectively **detached**.
|
|
115
|
+
|
|
116
|
+
**Singular and collection paths DIVERGE — only the forward singular FK re-parents.** The collection
|
|
117
|
+
members were **not** stolen: the nested `contactEmailAddresses` entries produced rows **251/252**
|
|
118
|
+
owned by 854 and **separate new rows 255/256** owned by 856. Same payload, same request — different
|
|
119
|
+
behavior per path.
|
|
120
|
+
|
|
121
|
+
> **Production implication.** Any two contacts sharing an email address or phone number — spouses, a
|
|
122
|
+
> shared household line, a corporate address — **collapse onto one child row whose owner flips to
|
|
123
|
+
> whoever wrote last**. For AIG this compounds with the `c_aigCustomerId` integer overflow (314 of
|
|
124
|
+
> 353 prod contacts clamped to a single identifier value), so **AIG contact identity on prod is not
|
|
125
|
+
> reliably distinct**. See
|
|
126
|
+
> [AIG entitlement intake](../../../clients/aig/features/entitlement-intake.md).
|
|
127
|
+
|
|
128
|
+
Same back-reference machinery as the
|
|
129
|
+
[UPDATE-path back-reference injection](#update-path-back-reference-injection-defeats-the-uuid-forced-match-primary-pointer-bug)
|
|
130
|
+
above; treat them as one defect with two symptoms when a platform fix is scoped.
|
|
131
|
+
|
|
132
|
+
## Matching is "link-AND-UPDATE", not "link" — a matched row is mutated from the payload
|
|
133
|
+
|
|
134
|
+
A nested related object that **matches** an existing record does not merely link to it — the engine
|
|
135
|
+
also **writes the payload's fields onto that existing row**. So partner-supplied data can **mutate
|
|
136
|
+
shared catalog rows**.
|
|
137
|
+
|
|
138
|
+
**Measured (beta `Client_Aig`, 2026-08-17):** the nested `unit.item` matched `Client_Aig.Items`
|
|
139
|
+
id **1** by `partNumber`, and the payload's `modelNumber` was then written onto it — `Items` id 1
|
|
140
|
+
`modelNumber` went from `NULL` to a test value, `dtUpdated 2026-08-17 10:41:17`.
|
|
141
|
+
|
|
142
|
+
> **"Link vs. create" is really "link-and-update vs. create". Matching is not read-only.**
|
|
143
|
+
|
|
144
|
+
**Caveat on this example — do not over-read it.** The test payload put the **sale item** part number
|
|
145
|
+
in `unit.item`, whereas AIG's real payload sends the **device** part number there. Per the AIG data
|
|
146
|
+
model those are two different row kinds in the dual-purpose `Items` table (sale item =
|
|
147
|
+
`itemCategoryId` NULL; unit/device item = `itemCategoryId` set), so **real traffic would not mutate
|
|
148
|
+
this particular row** — but it would do exactly the same thing to the **device item's** row. The
|
|
149
|
+
behavior is general; the specific row in the measurement is an artifact of the test payload.
|
|
150
|
+
|
|
97
151
|
## Per-API overrides (`Apis_RecordFields`)
|
|
98
152
|
|
|
99
153
|
The base identifier flags above (`Core.RecordFields.isIdentifier`) can be **overridden per API**
|
|
@@ -206,6 +260,19 @@ anywhere — no `messages[]`, nothing in the client `Logs` or the core/writer `L
|
|
|
206
260
|
|
|
207
261
|
## Change history
|
|
208
262
|
|
|
263
|
+
- 2026-08-17 (later pass) — Two new measured behaviors from the TRUE-80562 verification run on beta
|
|
264
|
+
`Client_Aig`. (1) **CREATE-path re-parenting:** a nested forward singular FK
|
|
265
|
+
(`primaryContactEmailAddress` / `primaryContactPhoneNumber`) is matched **tenant-wide by VALUE**,
|
|
266
|
+
not by uuid, and the matched child's back-reference `contactId` is **flipped to the newest writer**
|
|
267
|
+
— Contacts 854 and 856 both point at `ContactEmailAddresses` 250, which now belongs to 856, so
|
|
268
|
+
854's primary email is detached. **Collection members are NOT stolen** (rows 251/252 stayed with
|
|
269
|
+
854; 856 got new rows 255/256), so the singular and collection paths diverge. Prod implication: any
|
|
270
|
+
two contacts sharing an email/phone collapse onto one child row, compounding the AIG
|
|
271
|
+
`c_aigCustomerId` overflow. Same back-reference machinery as the 2026-07-20 UPDATE-path bug, but on
|
|
272
|
+
CREATE and with a stolen child instead of a null pointer. (2) **Matching is link-AND-UPDATE:** a
|
|
273
|
+
matched nested object is **mutated from the payload** — `unit.item` matched `Items` id 1 by
|
|
274
|
+
`partNumber` and wrote the payload's `modelNumber` onto it, so partner data can mutate shared
|
|
275
|
+
catalog rows. (mhammontree)
|
|
209
276
|
- 2026-08-17 — TRUE-80562: the TRUE-79978 **manual one-row beta patch did not survive** — the same
|
|
210
277
|
EV-12 recurred on beta 2026-07-21 (`Logs_Aig.Api` ids 93/95/98) after a tenant reseed (log rows
|
|
211
278
|
call `apiId 2` "AIG Integrations"; beta `Client_Aig.Apis` now names it "Agilant"). Identifier
|
|
@@ -7,4 +7,4 @@
|
|
|
7
7
|
| [2.0 New-Client Onboarding (manual process)](workflows/client-onboarding.md) | > **A local browser wizard now automates this.** Steps 2–9 below (create DBs, generate Core/API > inserts, append to `Clients_Db.txt`) — plus the dbchanges2 bla | Client/, Client_<Tenant>/, Core/, Logs_Client/ |
|
|
8
8
|
| [Auditing a client DB that drifted from its models (partially applied module migration)](workflows/client-schema-drift-audit.md) | A recurring 2.0 failure mode: **one client's database drifts from what the PHP models declare**, usually because a `_modules/<module>/` migration was applied to | dbchanges2/Client_Growrk/2026-05-28.sql, dbchanges2/Client_Growrk/2026-08-10c - GrowrkServiceRequestCustomFieldsCatchUp.sql, dbchanges2/Client_Growrk/2026-08-10d - GrowrkServiceRequestTypeAndDispositionSeeds.sql, dbchanges2/_modules/netsuite/2026-07-10a - UnitInventoryFields.sql, dbchanges2/Client_Growrk/2026-08-10 - GrowrkUnitInventoryFieldsCatchUp.sql, dbchanges2/Client_Growrk/2026-08-10b - GrowrkUnitItemDescriptionAcl.sql, dbchanges2/Client_Growrk/_modules.txt |
|
|
9
9
|
| [Local vs prod MySQL config parity — why “it passed locally” is not evidence](workflows/local-vs-prod-mysql-config-parity.md) | Several migration failures that look like "prod-only bugs" are actually **per-machine MySQL server-configuration differences**. | |
|
|
10
|
-
| [Repairing non-prod metadata drift (works in prod, broken in beta/dev-sandbox)](workflows/nonprod-metadata-drift-repair.md) | Almost all 2.0 platform behavior is **metadata** — `Core.Records`/`RecordFields`, `Core.RecordScripts`, `Core.ApiPayloadInterceptors`, and per-client `Acl*` row | dbchanges2/Core/2026-06-30a - ItemFulfillmentStageDefaultInterceptor.sql, dbchanges2/Client_Compass/2026-08-06 - RemoveBrokenSalesOrderItemPostPostInterceptor.sql, dbchanges2/Core/2026-07-16a - TrackingNumberSignatureTypeRecordField.sql, dbchanges2/Client/2026-07-22c - TrackingNumberMeasureIdsFieldPermission.sql, api2/Config/beta.ini, api2/Config/sandbox-dev.ini, dbchanges2/Client/2026-07-22a - EntitlementServiceAddressId.sql, dbchanges2/Client/2026-07-23a - EntitlementServiceAddressIdFieldPermission.sql |
|
|
10
|
+
| [Repairing non-prod metadata drift (works in prod, broken in beta/dev-sandbox)](workflows/nonprod-metadata-drift-repair.md) | Almost all 2.0 platform behavior is **metadata** — `Core.Records`/`RecordFields`, `Core.RecordScripts`, `Core.ApiPayloadInterceptors`, and per-client `Acl*` row | dbchanges2/Core/2026-06-30a - ItemFulfillmentStageDefaultInterceptor.sql, dbchanges2/Client_Compass/2026-08-06 - RemoveBrokenSalesOrderItemPostPostInterceptor.sql, dbchanges2/Core/2026-07-16a - TrackingNumberSignatureTypeRecordField.sql, dbchanges2/_modules/netsuite/2026-07-10a - UnitInventoryFields.sql, dbchanges2/Client_Aig/_modules.txt, dbchanges2/Client/2026-07-22c - TrackingNumberMeasureIdsFieldPermission.sql, api2/Config/beta.ini, api2/Config/sandbox-dev.ini, dbchanges2/Client/2026-07-22a - EntitlementServiceAddressId.sql, dbchanges2/Client/2026-07-23a - EntitlementServiceAddressIdFieldPermission.sql |
|
|
@@ -12,6 +12,8 @@ files:
|
|
|
12
12
|
- dbchanges2/Core/2026-06-30a - ItemFulfillmentStageDefaultInterceptor.sql
|
|
13
13
|
- dbchanges2/Client_Compass/2026-08-06 - RemoveBrokenSalesOrderItemPostPostInterceptor.sql
|
|
14
14
|
- dbchanges2/Core/2026-07-16a - TrackingNumberSignatureTypeRecordField.sql
|
|
15
|
+
- dbchanges2/_modules/netsuite/2026-07-10a - UnitInventoryFields.sql
|
|
16
|
+
- dbchanges2/Client_Aig/_modules.txt
|
|
15
17
|
- dbchanges2/Client/2026-07-22c - TrackingNumberMeasureIdsFieldPermission.sql
|
|
16
18
|
- api2/Config/beta.ini
|
|
17
19
|
- api2/Config/sandbox-dev.ini
|
|
@@ -110,6 +112,34 @@ debugging. That migration's own header states the column must exist for all tena
|
|
|
110
112
|
> A skipped `07-22a` sits **beside a `07-22b` that ran**. Check file-by-file; a date cutoff will
|
|
111
113
|
> mislead you.
|
|
112
114
|
|
|
115
|
+
### The gap is NOT limited to `Client/` — `_modules/<module>/` is skipped too
|
|
116
|
+
|
|
117
|
+
Verified 2026-08-17 on beta `Client_Aig`: **`_modules/netsuite/2026-07-10a - UnitInventoryFields.sql`
|
|
118
|
+
had never run** either, so `Client_Aig.Units` was missing six `c_` columns that **production has**.
|
|
119
|
+
`Client_Aig/_modules.txt` contains `netsuite`, so the client *is* opted in — the file was simply not
|
|
120
|
+
applied. Symptom is identical: **MySQL 1054 on a nested write**, surfaced as EV-12 with a
|
|
121
|
+
`modelSaveError`.
|
|
122
|
+
|
|
123
|
+
> **Audit the module folders as well as `Client/`.** For each tenant, read its `_modules.txt` and
|
|
124
|
+
> diff every `_modules/<module>/` file against the tenant's actual schema. A `Client/`-only audit
|
|
125
|
+
> reports a clean bill of health on a tenant that is still broken.
|
|
126
|
+
|
|
127
|
+
**This strengthens the "specific dates were skipped" reading.** The missing module file is dated
|
|
128
|
+
**`2026-07-10a`** — the **same date** as the already-recorded missing
|
|
129
|
+
`Client/2026-07-10a` (`ItemFulfillments.returnAddressId`). Two different fan-out folders both missing
|
|
130
|
+
the same date is far better explained by *that date's run failing* than by "beta is behind as of
|
|
131
|
+
date X."
|
|
132
|
+
|
|
133
|
+
**⚠ Diagnostic trap: do not attribute an unexpected runtime field list to a model class you read
|
|
134
|
+
locally.** During this session the missing-module 1054 was initially mis-diagnosed as a *cross-tenant
|
|
135
|
+
`$_modelCache` leak*, because the model-class comparison was made against a **local `_underscore`
|
|
136
|
+
checkout on branch `TRUE-80282`** while beta deploys **`_sandbox-dev`**. Beta runtime behavior
|
|
137
|
+
compared to a local checkout on a different branch is **not evidence** —
|
|
138
|
+
`_underscore` is cloned at build time from a moving branch
|
|
139
|
+
([environment-variable-drives-underscore-branch](../../api2/features/environment-variable-drives-underscore-branch.md)).
|
|
140
|
+
Check the environment's built branch **before** concluding anything about model classes; the boring
|
|
141
|
+
"a migration never ran" explanation is the one that keeps being true.
|
|
142
|
+
|
|
113
143
|
**Fix direction: run the existing pending `Client/` files — do NOT author a new migration**, which
|
|
114
144
|
would duplicate a file already on `_main`. A scoped manual `ALTER` on one tenant unblocks your own
|
|
115
145
|
testing but leaves the other 30 broken; say so when you do it.
|
|
@@ -205,6 +235,14 @@ blocked by the id-divergence hazard in item 2 above, which is why both go to jca
|
|
|
205
235
|
|
|
206
236
|
## Change history
|
|
207
237
|
|
|
238
|
+
- 2026-08-17 (later pass) — Extended the fan-out gap: it **also hits `_modules/<module>/`**, not just
|
|
239
|
+
`Client/`. Beta `Client_Aig` had never run `_modules/netsuite/2026-07-10a - UnitInventoryFields.sql`
|
|
240
|
+
despite `_modules.txt` listing `netsuite`, so `Units` lacked six `c_` columns prod has → 1054/EV-12
|
|
241
|
+
on nested unit writes. Note it is the **same date** as the missing `Client/2026-07-10a`, which
|
|
242
|
+
favors "specific dates were skipped" over "beta is behind as of date X". Added the **diagnostic
|
|
243
|
+
trap**: that 1054 was first mis-diagnosed as a cross-tenant `$_modelCache` leak because model
|
|
244
|
+
classes were read from a **local `_underscore` checkout on branch `TRUE-80282`** while beta deploys
|
|
245
|
+
`_sandbox-dev` — verify the environment's built branch before blaming code. (mhammontree)
|
|
208
246
|
- 2026-08-17 — TRUE-80562: documented that the **`Client/` fan-out is skipped independently of
|
|
209
247
|
`Core/`** on dev-sandbox — 31 of 33 tenant DBs lack `Entitlements.serviceAddressId` (only
|
|
210
248
|
`Client_Rate`/`Client_Elite` have it) while Core carries the RecordField (2763), so **entitlement
|
|
@@ -214,32 +214,55 @@ one of them to the maximum.
|
|
|
214
214
|
(`Logs_Aig.Api` and the core/writer `Logs`), keyed by contract number. **Confirm that before
|
|
215
215
|
assuming the loss is permanent.**
|
|
216
216
|
|
|
217
|
-
##
|
|
217
|
+
## RESOLVED — the `Units` 1054 was a missing module migration, NOT a cross-tenant leak
|
|
218
218
|
|
|
219
|
-
|
|
219
|
+
**Supersedes the earlier "cross-tenant field list" open question. There is no leak — do not repeat
|
|
220
|
+
that hypothesis.** An AIG POST carrying `entitlementUnits` produced an `INSERT INTO Units` including
|
|
220
221
|
`c_itemDescription`, `c_lastModified`, `c_chargingBrick`, `c_deviceStatus`, `c_chargingCable`,
|
|
221
|
-
`c_grade` → **MySQL 1054
|
|
222
|
-
|
|
223
|
-
**
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
222
|
+
`c_grade` → **MySQL 1054**. Those are **AIG's own fields**:
|
|
223
|
+
|
|
224
|
+
- `dbchanges2/_modules/netsuite/2026-07-10a - UnitInventoryFields.sql` adds all six columns **and**
|
|
225
|
+
their `CustomRecordFields` rows to every client whose `_modules.txt` lists `netsuite`.
|
|
226
|
+
**`Client_Aig/_modules.txt` contains exactly `netsuite`** — AIG is opted in.
|
|
227
|
+
- **Prod `Client_Aig.Units` has all 7 custom columns.** Beta had never run the module file — the same
|
|
228
|
+
execution gap as `Entitlements.serviceAddressId`. After running it manually on beta `Client_Aig`,
|
|
229
|
+
`Units` gained all 7 columns and the nested unit write succeeded with those fields present.
|
|
230
|
+
- The wrong conclusion came from comparing beta runtime behavior against a **local `_underscore`
|
|
231
|
+
checkout on branch `TRUE-80282`** while beta deploys `_sandbox-dev`. See
|
|
232
|
+
[per-client database connections](../../../2.0/apps/_underscore/features/per-client-database-connections.md)
|
|
233
|
+
and [non-prod metadata drift repair](../../../2.0/apps/dbchanges2/workflows/nonprod-metadata-drift-repair.md).
|
|
234
|
+
|
|
235
|
+
## Verified end-to-end with the full real-shaped payload (TRUE-80562, 2026-08-17)
|
|
236
|
+
|
|
237
|
+
`POST /v2/entitlements` on `api.beta.togahub.com` returned **201 with `entitlementUnits`
|
|
238
|
+
INCLUDED** — 1 unit, 3 coverage types (`C001`/`C002`/`C039`), 1 fulfillment method, 3 contact email
|
|
239
|
+
addresses.
|
|
240
|
+
|
|
241
|
+
> **This supersedes the earlier partial verification.** Removing `entitlementUnits` was **only a
|
|
242
|
+
> diagnostic isolation step**. **AIG must keep sending it — the unit is the covered device.** Their
|
|
243
|
+
> payload was never wrong.
|
|
244
|
+
|
|
245
|
+
## `Entitlements.number` carries a UNIQUE key — a replay 400s, it is not idempotent
|
|
246
|
+
|
|
247
|
+
Re-POSTing an already-created contract number returns **EV-10 / MySQL 1062** *"Duplicate entry … for
|
|
248
|
+
key `Entitlements.number`"* as an **HTTP 400**. This matters against AIG's documented
|
|
249
|
+
[scheduled re-send model](#feed-is-a-scheduled-re-send-batch-not-one-shot--self-healing): the feed is
|
|
250
|
+
safe **only while it re-sends unacknowledged contracts**. A bulk replay or back-fill over
|
|
251
|
+
already-created contracts will 400 per row — plan for that before running one.
|
|
252
|
+
|
|
253
|
+
## Beta test data left behind by TRUE-80562 (clean-up list)
|
|
254
|
+
|
|
255
|
+
For whoever cleans up beta `Client_Aig` (row ids only — do not copy the QA contact PII anywhere):
|
|
256
|
+
|
|
257
|
+
- `Entitlements` id **838** (number `1000045957545`) and the 10:41 entitlement (number
|
|
258
|
+
`1000045957546`).
|
|
259
|
+
- `Contacts` **854** and **856**, with the cross-linked primary email row **250** / phone row **18**
|
|
260
|
+
described in
|
|
261
|
+
[nested-relationship writes](../../../2.0/apps/api2/features/nested-relationship-writes.md).
|
|
262
|
+
- `Items` id **1** — `modelNumber` polluted with a test value; **revert to `NULL`**.
|
|
263
|
+
- **Beta `Client_Aig` had two migrations applied MANUALLY this session** (`Client/2026-07-22a`
|
|
264
|
+
`serviceAddressId` ALTERs, and the `_modules/netsuite/2026-07-10a` module file), so this tenant is
|
|
265
|
+
now **ahead of whatever the beta executor believes it has applied**.
|
|
243
266
|
|
|
244
267
|
## Business Unit (`UserDefined2`) — answered and DROPPED (nothing to build)
|
|
245
268
|
|
|
@@ -463,6 +486,18 @@ this interceptor or use this dual-purpose Items pattern.
|
|
|
463
486
|
|
|
464
487
|
## Change history
|
|
465
488
|
|
|
489
|
+
- 2026-08-17 (later pass) — **Correction + final verification, supersedes two points in the entry
|
|
490
|
+
below.** (1) The nested-`Units` "cross-tenant field list" open question is **WITHDRAWN** — the six
|
|
491
|
+
`c_` columns are AIG's **own**, added by `_modules/netsuite/2026-07-10a` to every client whose
|
|
492
|
+
`_modules.txt` lists `netsuite` (AIG's does); prod has all 7 columns, beta had never run the module
|
|
493
|
+
file. Running it on beta fixed the write. The wrong conclusion came from comparing beta runtime
|
|
494
|
+
against a **local `_underscore` checkout on branch `TRUE-80282`** vs beta's `_sandbox-dev`. (2)
|
|
495
|
+
Verified end-to-end with the **full real-shaped payload including `entitlementUnits`** — 201 with
|
|
496
|
+
1 unit / 3 coverage types / 1 fulfillment method / 3 emails; removing `entitlementUnits` earlier was
|
|
497
|
+
only an isolation step and **AIG must keep sending it**. Also recorded: `Entitlements.number` has a
|
|
498
|
+
**UNIQUE key**, so a replay of an already-created contract returns EV-10/1062 as HTTP 400 (matters
|
|
499
|
+
for any bulk back-fill); and a **beta clean-up list** of the rows/manual migrations this session
|
|
500
|
+
left behind. (mhammontree)
|
|
466
501
|
- 2026-08-17 — TRUE-80562 (beta EV-12 on entitlement intake). The TRUE-79978 **manual beta patch did
|
|
467
502
|
not survive a tenant reseed** — same EV-12 recurred 2026-07-21 (`Logs_Aig.Api` 93/95/98); the fix
|
|
468
503
|
now ships as `dbchanges2/Client_Aig/2026-08-17a`. Recorded the **scope rule** (only correct NULL
|
package/package.json
CHANGED