toga-ai 1.0.590 → 1.0.592

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -6,7 +6,7 @@ project: Library
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-08-04
9
+ updated: 2026-08-17
10
10
  owners: ["jcardinal", "mhammontree"]
11
11
  files:
12
12
  - library/app/error/capture.php
@@ -177,6 +177,15 @@ exists in `api2`/`worker2`. So 1.0 reads the AL1 container config at
177
177
 
178
178
  ## Gotchas / known issues
179
179
 
180
+ - **⚠ A grouped `Logs.Issue` row can show a STALE error id — read the live api2 transaction log for
181
+ the current error.** Two facts combine to mislead: (1) `Event.errorMessage` is **`varchar(255)`**,
182
+ so a long API response is **truncated** — often before the trailing error id; and (2)
183
+ `Logs.Issue.subject` / `Issue.errorMessage` hold the **first-occurrence** text, so a long-lived
184
+ issue keeps showing the error id from whenever it was first seen even after the per-call error has
185
+ changed. Observed 2026-08-17: issue `3H` still displayed `EO-1` / `3G-6` (the original
186
+ undefined-method fatal) while the current per-call failure was actually `EV-11` (an FK 1451 on a
187
+ DELETE). **To see what is failing *now*, read the live api2 transaction log
188
+ (`Logs_<client>.Api.responsePayload`) for the specific call — not the grouped `Issue` row.**
180
189
  - **Any PHP warning inside capture code would kill the request.** 1.0's
181
190
  `App_Error::handleError()` routes **every** PHP warning into `handleException()`, which renders
182
191
  the red box and calls `exit()` — so a single warning raised while capturing (an unreachable IMDS
@@ -220,6 +229,12 @@ exists in `api2`/`worker2`. So 1.0 reads the AL1 container config at
220
229
 
221
230
  ## Change history
222
231
 
232
+ - 2026-08-17 — Recorded a diagnostic trap: a grouped `Logs.Issue` row can show a **stale error id**
233
+ because `Event.errorMessage` is `varchar(255)` (truncates a long API response, often before the
234
+ error id) and `Issue.subject`/`errorMessage` keep the **first-occurrence** text — observed issue
235
+ `3H` still showing `EO-1`/`3G-6` while the live per-call error was `EV-11` (FK 1451). Read the live
236
+ `Logs_<client>.Api.responsePayload` for the actual current error. Surfaced diagnosing the Compass
237
+ USA NetSuite SO sync freeze. (jcardinal)
223
238
  - 2026-08-05 — Recorded the legacy `DELETE_DOA_PRE-SENTRY_error.php` handler leaking the full
224
239
  config array (prod DB / SES / NetSuite / Payeezy / SSL Store credentials + ticket PII) on a
225
240
  live towfoundation.togaview.com error page, meaning that host evaluates as dev mode in
@@ -6,7 +6,7 @@ project: Library
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-08-10
9
+ updated: 2026-08-17
10
10
  owners: [jcardinal, mhammontree, bala]
11
11
  files:
12
12
  - library/app/api/toga2.php
@@ -328,6 +328,16 @@ enable flags** and an optional `$monitorTogadeskDepartmentIds[]`:
328
328
  **and** the consuming code. (The general 2.0-side metadata behavior — reverse relations resolved via
329
329
  the child model's `FOREIGNKEY_MODEL` and named as the camelCase plural of the child model — is a
330
330
  framework-level fact captured separately.)
331
+ - **⚠ Depth alone does NOT return every reverse-hasMany collection — `GET /sales-orders` at depth 4
332
+ omits `itemFulfillmentItems` and `invoiceItems`.** Verified 2026-08-17 against prod
333
+ `Logs_Compass.Api` responses (no fields whitelist): a depth-4 sales-order GET returns
334
+ `salesOrderItems` and the **PO-link bridge** collections
335
+ (`salesOrderItemPurchaseOrderItems`, `purchaseOrderItemSalesOrderItems`) but **not**
336
+ `itemFulfillmentItems` or `invoiceItems`. So any importer code that guards on
337
+ `$salesOrderItem->itemFulfillmentItems` / `->invoiceItems` from that GET is a **silent no-op** —
338
+ same reverse-hasMany trap as above, now with a concrete consequence: the NetSuite SO renumber fix
339
+ keys off a recorded "could-not-delete" set instead of trusting those collections (see
340
+ [NetSuite → TOGa Supply per-client sync](../../worker/features/netsuite-togasupply-per-client-sync.md)).
331
341
  - **`TOGA_AIG.Customers.emailAddress` is `varchar(255)`** (V1 legacy, latin1_swedish_ci) — wide
332
342
  enough to hold several `'; '`-joined emails, so TRUE-79401 needed **no** schema/dbchanges migration
333
343
  (the earlier fear that it was ~VARCHAR(55) was wrong) — but **it is a real ceiling: ~6–7
@@ -386,6 +396,13 @@ enable flags** and an optional `$monitorTogadeskDepartmentIds[]`:
386
396
 
387
397
  ## Change history
388
398
 
399
+ - 2026-08-17 — Recorded that **depth-4 `GET /sales-orders` omits the `itemFulfillmentItems` and
400
+ `invoiceItems` reverse-hasMany collections** (it does return `salesOrderItems` and the PO-link
401
+ bridges), verified against prod `Logs_Compass.Api` — so code guarding on those collections from
402
+ that GET is a silent no-op, reinforcing the reverse-hasMany-fields trap. Surfaced fixing the
403
+ NetSuite SO renumber freeze (Compass USA), which now keys off a recorded "could-not-delete" set;
404
+ see [NetSuite → TOGa Supply per-client sync](../../worker/features/netsuite-togasupply-per-client-sync.md).
405
+ (jcardinal)
389
406
  - 2026-08-12 — Recorded the **`send()` return contract**: it always returns a response carrying
390
407
  `isSuccess` (early-returns on true, falls through to `return $response` on false, throws when the
391
408
  property is absent), so `empty($response->isSuccess)` is a safe failure check on anything it
@@ -6,7 +6,7 @@ project: Worker
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-08-12
9
+ updated: 2026-08-17
10
10
  owners: ["dfranks", "bala", "jcardinal", "mhammontree", "snaredla"]
11
11
  files:
12
12
  - worker/crons/toga2/netsuite/common_sync_togasupply.php
@@ -87,13 +87,28 @@ its IDLE write. **There is no timeout, memory cap, or resource limit anywhere in
87
87
  never *tune the window size down*. (Corollary: a window that has grown back to the cap is positive
88
88
  evidence the section is completing.)
89
89
 
90
- ### ⚠ A per-record API failure does NOT stop the cursor — the record is skipped forever
91
-
92
- The cursor write (`LAST_SYNC_DATETIME_*`) and `finishModeIteration()` both run **unconditionally
93
- after** the record loop. A record that throws mid-loop is captured to Sentry and **stepped over**,
94
- while the section still reports success and advances the cursor past its window. So
95
- **"section IDLE + cursor advancing" is not proof of a complete import** — it only proves the run
96
- didn't die.
90
+ ### ⚠ Per-record isolation is SECTION-SPECIFIC — and SALES_ORDERS has NONE (corrected 2026-08-17)
91
+
92
+ Whether a bad record is "stepped over" or freezes the section depends entirely on whether **that
93
+ section's** record loop wraps the `sync*FromNetsuite()` call in a per-record `try/catch`. It is not
94
+ uniform across the six sections, and the earlier blanket claim here ("a per-record failure does not
95
+ stop the cursor — the record is skipped forever") is **false for the SALES_ORDERS section**.
96
+
97
+ - **Where a section DOES isolate per record:** the cursor write (`LAST_SYNC_DATETIME_*`) and
98
+ `finishModeIteration()` run after the loop, so a caught record is skipped and the section still
99
+ reports IDLE and advances. There, **"section IDLE + cursor advancing" is not proof of a complete
100
+ import** — it only proves the run didn't die (recovery = rewind the cursor; re-import is
101
+ idempotent).
102
+ - **⚠ SALES_ORDERS has NO per-record `try/catch`.** The `SALES_ORDERS` foreach in
103
+ `common_sync_togasupply.php` (~L637-673) calls `App_Api_Toga2::syncSalesOrderFromNetsuite()`
104
+ **bare**. `send()` throws on any non-success by default, so **one poison order throws, propagates,
105
+ and aborts the whole section BEFORE** the cursor PUT + `finishModeIteration()`. The mode stays
106
+ `1-RUNNING`, the watermark freezes, the window shrinks ÷3 each run, and the **same** order is
107
+ re-tried and re-throws every 5 minutes — a permanent freeze, not a skip. So a stuck
108
+ `SALES_ORDERS` at `1-RUNNING` with a days-frozen watermark means **it is throwing on a specific
109
+ order** (go read the live api2 transaction log for that client — see the poison-order gotcha
110
+ below). The "shrinking window is a bug signal" rule still holds; the "record is stepped over"
111
+ corollary does **not** for this section.
97
112
 
98
113
  Proven concretely on Elite: sales order **169441** (NetSuite internal id 4624062, trandate
99
114
  2024-04-15, customer PO `BEGGS/04152024/…`) was simply absent from `Client_Elite.SalesOrders`
@@ -361,9 +376,46 @@ Parameters are stored **per client DB** but accessed **through the TOGa2 API**,
361
376
  a new line is actually created. Full root cause, the prod cleanup migration, and the
362
377
  deploy-fix-before-cleanup sequencing live in the
363
378
  [Compass ODP Duplicate PO-Line Cleanup](../../../clients/compass-usa/workflows/odp-duplicate-po-line-cleanup.md).
379
+ - **⚠ A single un-deletable renumbered line could freeze the entire SALES_ORDERS section (fixed
380
+ 2026-08-17).** When NetSuite renumbers a line, `syncSalesOrderFromNetsuite` renumbers by
381
+ **DELETE `/sales-order-items/{uuid}` then re-create in the per-item loop**. Its "pinned row" guard
382
+ only skipped rows carrying `itemFulfillmentItems` / `salesOrderItemPurchaseOrderItems` /
383
+ `purchaseOrderItemSalesOrderItems`; an **INVOICED** line
384
+ (`InvoiceItems.salesOrderItemId`, `ON DELETE RESTRICT`) was **not** guarded, so the DELETE failed
385
+ with MySQL **1451** → api2 **EV-11** 400, and — because the SALES_ORDERS loop has no per-record
386
+ `try/catch` (see the corrected gotcha above) — the throw aborted the section, freezing Compass USA
387
+ for ~5 days (`NETSUITE_EXECUTION_MODE_SALES_ORDERS='1-RUNNING'`, watermark stuck at
388
+ `2026-08-12 06:13:26`) while it re-tried the same poison order every run (SO **7447** /
389
+ `SalesOrderItem` 47828, invoiced by `InvoiceItem` 25515). **Fix (library-only, PHP 7.2-safe, no
390
+ signature changes)** in `syncSalesOrderFromNetsuite`: (1) the renumber-DELETE now passes
391
+ `$throwExceptionOnApiError=false`, checks `empty($resp->isSuccess)`, and records any row it cannot
392
+ delete in a new `$pinnedRenumberLineUniqueKeys` set (keyed by `c_netsuiteLineUniqueKey`); (2) the
393
+ per-item PUT loop treats a row in that set as **pinned** — it keeps its current `lineNumber` (no
394
+ `UNIQUE(salesOrderId,lineNumber)` collision) and its PUT is made non-throwing + `error_log` **for
395
+ pinned rows only** (normal rows keep throw-on-error so real failures still surface); (3) the
396
+ trailing "remove dropped items" DELETE now skips `invoiceItems` and checks `isSuccess`. Net: **no
397
+ single order can freeze the section again regardless of which `RESTRICT` child blocks a delete**;
398
+ the invoiced line simply stays at its existing `lineNumber` (a documented unresolvable
399
+ NetSuite-vs-invoiced case). Deploy `library` only — no SQL, no other repos — then the section
400
+ completes, mode flips IDLE, and the backlog backfills idempotently. Note the collection guard added
401
+ for `invoiceItems` is **best-effort only**: depth-4 `GET /sales-orders` does **not** return the
402
+ `itemFulfillmentItems` or `invoiceItems` reverse-hasMany collections (only the PO-link bridges), so
403
+ the fix keys off the recorded "could-not-delete" set rather than trusting those collections — see
404
+ [App_Api_Toga2 client & bridge](../../library/features/toga2-api-client-and-bridge.md).
364
405
 
365
406
  ## Change history
366
407
 
408
+ - 2026-08-17 — Fixed a Compass USA ~5-day SALES_ORDERS freeze: an **invoiced** line that NetSuite
409
+ renumbered could not be DELETEd (FK `InvoiceItems.salesOrderItemId` `RESTRICT` → MySQL 1451 →
410
+ api2 EV-11), and because the SALES_ORDERS loop has **no per-record `try/catch`** the throw aborted
411
+ the whole section (mode stuck `1-RUNNING`, watermark frozen, poison order SO 7447 re-tried every
412
+ run). `syncSalesOrderFromNetsuite` now makes the renumber-DELETE non-throwing, records
413
+ un-deletable rows in a `$pinnedRenumberLineUniqueKeys` set, and keeps a pinned row at its existing
414
+ `lineNumber` with a non-throwing PUT (normal rows still throw on error); the trailing dropped-item
415
+ DELETE now skips `invoiceItems` and checks `isSuccess`. **Corrected** the prior gotcha that claimed
416
+ a per-record failure never stops the cursor — that is **section-specific and false for
417
+ SALES_ORDERS**, which has no per-record isolation. Library-only deploy; no SQL/other repos.
418
+ Diagnosis touched worker/api2/_underscore (read-only). (jcardinal)
367
419
  - 2026-08-12 — TRUE-80499 (Elite onboarding, shared-engine work). **Fixed** the sparse-client
368
420
  `TypeError`: the two per-client item lookups are now initialised at
369
421
  `common_sync_togasupply.php:74-75` instead of only inside the items loop, so a client whose items
@@ -178,41 +178,47 @@ here — they live in `Config/*.ini`.)
178
178
  (`db_logs`) — the laptop trap there is documented separately in the worker NetSuite bootstrap
179
179
  notes.
180
180
 
181
- ## ⚠ OPEN QUESTION — does the alias-keyed `$_modelCache` leak across tenants? (UNPROVEN)
181
+ ## WITHDRAWN — the "alias-keyed `$_modelCache` cross-tenant leak" hypothesis (2026-08-17)
182
182
 
183
- **Do not repeat this as an established leak, and do not dismiss it either.** It needs a focused
184
- security review. Observed 2026-08-17 (TRUE-80562):
183
+ **There is no known cross-tenant leak. Do not cite this as a security concern.** An earlier entry in
184
+ this doc (same day, TRUE-80562) raised an *unproven* hypothesis that `_Database::$_modelCache`,
185
+ being keyed by the shared alias `'Client'`, was emitting one tenant's `c_` columns on another
186
+ tenant's `INSERT`. **That is not what happened.** The trigger was an AIG `Units` write failing MySQL
187
+ 1054 on six `c_` columns; the real cause was a **missing migration on beta**, not a cache collision:
185
188
 
186
- An **AIG** `POST /v2/entitlements` carrying nested `entitlementUnits` emitted an
187
- `INSERT INTO Units` containing `c_itemDescription`, `c_lastModified`, `c_chargingBrick`,
188
- `c_deviceStatus`, `c_chargingCable`, `c_grade` → **MySQL 1054**, surfaced as EV-12 with a
189
- `modelSaveError`.
189
+ - `dbchanges2/_modules/netsuite/2026-07-10a - UnitInventoryFields.sql` adds all six columns
190
+ (`c_itemDescription`, `c_lastModified`, `c_chargingBrick`, `c_deviceStatus`, `c_chargingCable`,
191
+ `c_grade`) **and** their `CustomRecordFields` rows to every client whose `_modules.txt` lists
192
+ `netsuite`. **`Client_Aig/_modules.txt` contains exactly `netsuite`** — AIG is opted in, so those
193
+ are **AIG's own fields**, not another tenant's.
194
+ - **Prod `Client_Aig.Units` has all 7 custom columns.** Beta simply never ran that module file — the
195
+ same execution gap as `Entitlements.serviceAddressId`. After the module migration was run manually
196
+ on beta `Client_Aig`, `Units` gained all 7 columns and the nested unit write succeeded.
190
197
 
191
- **Verified facts:**
198
+ **Methodology lesson — why the wrong conclusion was reached.** The model-class comparison that
199
+ produced the leak hypothesis was made against the **local `_underscore` checkout, which was on branch
200
+ `TRUE-80282`, while beta deploys `_sandbox-dev`.** Comparing beta *runtime* behavior against a local
201
+ checkout on a different branch is **not evidence**. api2's architecture doc already warns that
202
+ `_underscore` is cloned at build time from a moving branch
203
+ ([environment-variable-drives-underscore-branch](../../api2/features/environment-variable-drives-underscore-branch.md));
204
+ this is that hazard biting in practice. Before attributing a runtime field list to a model class,
205
+ confirm which branch the environment actually built from.
192
206
 
193
- - `Client_Aig.Units` has **only** `c_netsuiteInternalInventoryAssignmentId`, and
194
- `Client_Aig.CustomRecordFields` declares only that one field for `Units` (recordId 31) — AIG's
195
- **schema and metadata agree**.
196
- - Those six columns exist **only** in `Client_Adyen`, `Client_Elite`, `Client_Growrk`,
197
- `Client_Prudential`. Five are declared **only** on `_Model_Growrk_Unit`; `_Model_Aig_Unit` extends
198
- `_Model_Client_Unit` and declares none. `c_lastModified` is declared in **no** model anywhere in
199
- `_underscore`.
200
- - So the field list is **neither** AIG's schema, **nor** AIG's metadata, **nor** any model class AIG
201
- uses — it is exactly **four other tenants'** column set.
202
-
203
- **Hypothesis (not proven):** `_underscore/Model.php` (~676–687) caches query results as
204
- `_Database::$_modelCache[$this::DATABASE][$sql]`, and `DATABASE` is the shared **alias** `'Client'`
205
- — *not* the tenant schema name — so two tenants collide on the same cache key (the same alias-keying
206
- already documented in the gotcha above). **Unknown: whether that static survives across requests
207
- under PHP-FPM.** That single unknown is the difference between a harmless within-request artifact and
208
- a **cross-tenant data-leak vector**, which is why it must be answered rather than assumed.
209
-
210
- **Where to look:** `_underscore/Model.php` ~676–687, `api2/Component/Api/V2/V2.php` ~8918–8931,
211
- `_underscore/Model/Growrk/Unit.php`, `_underscore/Model/Aig/Unit.php`. Context:
212
- [AIG entitlement intake](../../../clients/aig/features/entitlement-intake.md).
207
+ **The real generalizable fact** — the beta migration fan-out gap is **not limited to `Client/`; it
208
+ also hits `_modules/<module>/`** — is recorded in
209
+ [non-prod metadata drift repair](../../dbchanges2/workflows/nonprod-metadata-drift-repair.md).
213
210
 
214
211
  ## Change history
215
212
 
213
+ - 2026-08-17 (later pass) — **WITHDRAWN, supersedes the entry below.** The alias-keyed
214
+ `$_modelCache` cross-tenant-leak hypothesis is **not supported** and is no longer a live security
215
+ concern. The six `c_` columns are **AIG's own**: `_modules/netsuite/2026-07-10a -
216
+ UnitInventoryFields.sql` adds them to every client whose `_modules.txt` lists `netsuite`, and
217
+ `Client_Aig/_modules.txt` contains exactly that; prod `Client_Aig.Units` has all 7 custom columns
218
+ and beta had simply never run the module file. Running it on beta fixed the write. The wrong
219
+ conclusion came from comparing beta runtime behavior against a **local `_underscore` checkout on
220
+ branch `TRUE-80282`** while beta deploys `_sandbox-dev` — the moving-branch hazard the api2
221
+ architecture doc already warns about. (mhammontree)
216
222
  - 2026-08-17 — Recorded an **unproven open question** (TRUE-80562): an AIG nested `Units` write
217
223
  emitted four *other* tenants' `c_` columns (1054 → EV-12) although AIG's schema, its
218
224
  `CustomRecordFields` metadata, and its model class all declare exactly one custom field. Suspected
@@ -6,8 +6,8 @@ project: API
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-08-14
10
- owners: ["mhammontree", "dfranks", "bala", "snaredla"]
9
+ updated: 2026-08-17
10
+ owners: ["mhammontree", "dfranks", "bala", "snaredla", "jcardinal"]
11
11
  files:
12
12
  - api2/Component/Api/V2/V2.php
13
13
  - api2/Controller/Index.php
@@ -448,6 +448,13 @@ and the failing environment**. It is a small table, and the drift is usually exa
448
448
 
449
449
  ## Change history
450
450
 
451
+ - 2026-08-17 — Corroborated that the unguarded-dispatch fatal is **systemic, not client-specific**:
452
+ the prod "undefined method" issue bucket (`Logs.Issue` ref 3G / id 59) grouped **both**
453
+ `_Model_Compass_Usa_SalesOrderItem::postPost()` **and** `_Model_Elite_SalesOrder::postPut()` — two
454
+ different tenants, one shared fragility. (This was the original Compass USA failure on 2026-08-12,
455
+ before it was mitigated on the 2.0 side and the client's NetSuite sync progressed to an unrelated
456
+ FK failure downstream.) Reinforces the standing fix direction: guard all four `V2.php` call sites
457
+ with `method_exists()`. Investigation only, no code change. (jcardinal)
451
458
  - 2026-08-14 — TRUE-80489, from building the Compass inactive-standalone-item order guard.
452
459
  (1) **A PRE interceptor on a PARENT record fires on the nested child POST, with the CHILD as the
453
460
  payload.** `POST /sales-orders/{uuid}/sales-order-items` resolves `SalesOrders` as the
@@ -94,6 +94,60 @@ fixing. Until then the caller-side workaround is a follow-up identifier-only `PU
94
94
  self-heal that detects the null/mismatched echo (shipped on the BDR funnel side — see
95
95
  web-funnel-app.md).
96
96
 
97
+ ## ⚠ CREATE path: a nested singular FK is matched TENANT-WIDE by VALUE and RE-PARENTED
98
+
99
+ The forward-singular-FK back-reference machinery has a second, worse manifestation — on the
100
+ **CREATE** path, and the damage is a **stolen child row** rather than a null pointer.
101
+
102
+ A nested `primaryContactEmailAddress` / `primaryContactPhoneNumber` object is matched **across the
103
+ whole tenant by its VALUE** (`emailAddress` / `phoneNumber`) — **not** by `uuid` — and the matched
104
+ child's back-reference `contactId` is then **flipped to the newest writer**, silently detaching it
105
+ from its previous owner.
106
+
107
+ **Measured on beta `Client_Aig` 2026-08-17 (TRUE-80562).** Two sequential POSTs each created a new
108
+ Contact (**854** at 10:19, **856** at 10:41). Afterwards:
109
+
110
+ - Contacts **854 AND 856** both have `primaryContactEmailAddressId = 250` and
111
+ `primaryContactPhoneNumberId = 18`.
112
+ - `ContactEmailAddresses` row **250** has `contactId = 856`.
113
+ - So contact **854 still points at a child row that now belongs to contact 856** — 854's primary
114
+ email is effectively **detached**.
115
+
116
+ **Singular and collection paths DIVERGE — only the forward singular FK re-parents.** The collection
117
+ members were **not** stolen: the nested `contactEmailAddresses` entries produced rows **251/252**
118
+ owned by 854 and **separate new rows 255/256** owned by 856. Same payload, same request — different
119
+ behavior per path.
120
+
121
+ > **Production implication.** Any two contacts sharing an email address or phone number — spouses, a
122
+ > shared household line, a corporate address — **collapse onto one child row whose owner flips to
123
+ > whoever wrote last**. For AIG this compounds with the `c_aigCustomerId` integer overflow (314 of
124
+ > 353 prod contacts clamped to a single identifier value), so **AIG contact identity on prod is not
125
+ > reliably distinct**. See
126
+ > [AIG entitlement intake](../../../clients/aig/features/entitlement-intake.md).
127
+
128
+ Same back-reference machinery as the
129
+ [UPDATE-path back-reference injection](#update-path-back-reference-injection-defeats-the-uuid-forced-match-primary-pointer-bug)
130
+ above; treat them as one defect with two symptoms when a platform fix is scoped.
131
+
132
+ ## Matching is "link-AND-UPDATE", not "link" — a matched row is mutated from the payload
133
+
134
+ A nested related object that **matches** an existing record does not merely link to it — the engine
135
+ also **writes the payload's fields onto that existing row**. So partner-supplied data can **mutate
136
+ shared catalog rows**.
137
+
138
+ **Measured (beta `Client_Aig`, 2026-08-17):** the nested `unit.item` matched `Client_Aig.Items`
139
+ id **1** by `partNumber`, and the payload's `modelNumber` was then written onto it — `Items` id 1
140
+ `modelNumber` went from `NULL` to a test value, `dtUpdated 2026-08-17 10:41:17`.
141
+
142
+ > **"Link vs. create" is really "link-and-update vs. create". Matching is not read-only.**
143
+
144
+ **Caveat on this example — do not over-read it.** The test payload put the **sale item** part number
145
+ in `unit.item`, whereas AIG's real payload sends the **device** part number there. Per the AIG data
146
+ model those are two different row kinds in the dual-purpose `Items` table (sale item =
147
+ `itemCategoryId` NULL; unit/device item = `itemCategoryId` set), so **real traffic would not mutate
148
+ this particular row** — but it would do exactly the same thing to the **device item's** row. The
149
+ behavior is general; the specific row in the measurement is an artifact of the test payload.
150
+
97
151
  ## Per-API overrides (`Apis_RecordFields`)
98
152
 
99
153
  The base identifier flags above (`Core.RecordFields.isIdentifier`) can be **overridden per API**
@@ -206,6 +260,19 @@ anywhere — no `messages[]`, nothing in the client `Logs` or the core/writer `L
206
260
 
207
261
  ## Change history
208
262
 
263
+ - 2026-08-17 (later pass) — Two new measured behaviors from the TRUE-80562 verification run on beta
264
+ `Client_Aig`. (1) **CREATE-path re-parenting:** a nested forward singular FK
265
+ (`primaryContactEmailAddress` / `primaryContactPhoneNumber`) is matched **tenant-wide by VALUE**,
266
+ not by uuid, and the matched child's back-reference `contactId` is **flipped to the newest writer**
267
+ — Contacts 854 and 856 both point at `ContactEmailAddresses` 250, which now belongs to 856, so
268
+ 854's primary email is detached. **Collection members are NOT stolen** (rows 251/252 stayed with
269
+ 854; 856 got new rows 255/256), so the singular and collection paths diverge. Prod implication: any
270
+ two contacts sharing an email/phone collapse onto one child row, compounding the AIG
271
+ `c_aigCustomerId` overflow. Same back-reference machinery as the 2026-07-20 UPDATE-path bug, but on
272
+ CREATE and with a stolen child instead of a null pointer. (2) **Matching is link-AND-UPDATE:** a
273
+ matched nested object is **mutated from the payload** — `unit.item` matched `Items` id 1 by
274
+ `partNumber` and wrote the payload's `modelNumber` onto it, so partner data can mutate shared
275
+ catalog rows. (mhammontree)
209
276
  - 2026-08-17 — TRUE-80562: the TRUE-79978 **manual one-row beta patch did not survive** — the same
210
277
  EV-12 recurred on beta 2026-07-21 (`Logs_Aig.Api` ids 93/95/98) after a tenant reseed (log rows
211
278
  call `apiId 2` "AIG Integrations"; beta `Client_Aig.Apis` now names it "Agilant"). Identifier
@@ -7,4 +7,4 @@
7
7
  | [2.0 New-Client Onboarding (manual process)](workflows/client-onboarding.md) | > **A local browser wizard now automates this.** Steps 2–9 below (create DBs, generate Core/API > inserts, append to `Clients_Db.txt`) — plus the dbchanges2 bla | Client/, Client_<Tenant>/, Core/, Logs_Client/ |
8
8
  | [Auditing a client DB that drifted from its models (partially applied module migration)](workflows/client-schema-drift-audit.md) | A recurring 2.0 failure mode: **one client's database drifts from what the PHP models declare**, usually because a `_modules/<module>/` migration was applied to | dbchanges2/Client_Growrk/2026-05-28.sql, dbchanges2/Client_Growrk/2026-08-10c - GrowrkServiceRequestCustomFieldsCatchUp.sql, dbchanges2/Client_Growrk/2026-08-10d - GrowrkServiceRequestTypeAndDispositionSeeds.sql, dbchanges2/_modules/netsuite/2026-07-10a - UnitInventoryFields.sql, dbchanges2/Client_Growrk/2026-08-10 - GrowrkUnitInventoryFieldsCatchUp.sql, dbchanges2/Client_Growrk/2026-08-10b - GrowrkUnitItemDescriptionAcl.sql, dbchanges2/Client_Growrk/_modules.txt |
9
9
  | [Local vs prod MySQL config parity — why “it passed locally” is not evidence](workflows/local-vs-prod-mysql-config-parity.md) | Several migration failures that look like "prod-only bugs" are actually **per-machine MySQL server-configuration differences**. | |
10
- | [Repairing non-prod metadata drift (works in prod, broken in beta/dev-sandbox)](workflows/nonprod-metadata-drift-repair.md) | Almost all 2.0 platform behavior is **metadata** — `Core.Records`/`RecordFields`, `Core.RecordScripts`, `Core.ApiPayloadInterceptors`, and per-client `Acl*` row | dbchanges2/Core/2026-06-30a - ItemFulfillmentStageDefaultInterceptor.sql, dbchanges2/Client_Compass/2026-08-06 - RemoveBrokenSalesOrderItemPostPostInterceptor.sql, dbchanges2/Core/2026-07-16a - TrackingNumberSignatureTypeRecordField.sql, dbchanges2/Client/2026-07-22c - TrackingNumberMeasureIdsFieldPermission.sql, api2/Config/beta.ini, api2/Config/sandbox-dev.ini, dbchanges2/Client/2026-07-22a - EntitlementServiceAddressId.sql, dbchanges2/Client/2026-07-23a - EntitlementServiceAddressIdFieldPermission.sql |
10
+ | [Repairing non-prod metadata drift (works in prod, broken in beta/dev-sandbox)](workflows/nonprod-metadata-drift-repair.md) | Almost all 2.0 platform behavior is **metadata** — `Core.Records`/`RecordFields`, `Core.RecordScripts`, `Core.ApiPayloadInterceptors`, and per-client `Acl*` row | dbchanges2/Core/2026-06-30a - ItemFulfillmentStageDefaultInterceptor.sql, dbchanges2/Client_Compass/2026-08-06 - RemoveBrokenSalesOrderItemPostPostInterceptor.sql, dbchanges2/Core/2026-07-16a - TrackingNumberSignatureTypeRecordField.sql, dbchanges2/_modules/netsuite/2026-07-10a - UnitInventoryFields.sql, dbchanges2/Client_Aig/_modules.txt, dbchanges2/Client/2026-07-22c - TrackingNumberMeasureIdsFieldPermission.sql, api2/Config/beta.ini, api2/Config/sandbox-dev.ini, dbchanges2/Client/2026-07-22a - EntitlementServiceAddressId.sql, dbchanges2/Client/2026-07-23a - EntitlementServiceAddressIdFieldPermission.sql |
@@ -12,6 +12,8 @@ files:
12
12
  - dbchanges2/Core/2026-06-30a - ItemFulfillmentStageDefaultInterceptor.sql
13
13
  - dbchanges2/Client_Compass/2026-08-06 - RemoveBrokenSalesOrderItemPostPostInterceptor.sql
14
14
  - dbchanges2/Core/2026-07-16a - TrackingNumberSignatureTypeRecordField.sql
15
+ - dbchanges2/_modules/netsuite/2026-07-10a - UnitInventoryFields.sql
16
+ - dbchanges2/Client_Aig/_modules.txt
15
17
  - dbchanges2/Client/2026-07-22c - TrackingNumberMeasureIdsFieldPermission.sql
16
18
  - api2/Config/beta.ini
17
19
  - api2/Config/sandbox-dev.ini
@@ -110,6 +112,34 @@ debugging. That migration's own header states the column must exist for all tena
110
112
  > A skipped `07-22a` sits **beside a `07-22b` that ran**. Check file-by-file; a date cutoff will
111
113
  > mislead you.
112
114
 
115
+ ### The gap is NOT limited to `Client/` — `_modules/<module>/` is skipped too
116
+
117
+ Verified 2026-08-17 on beta `Client_Aig`: **`_modules/netsuite/2026-07-10a - UnitInventoryFields.sql`
118
+ had never run** either, so `Client_Aig.Units` was missing six `c_` columns that **production has**.
119
+ `Client_Aig/_modules.txt` contains `netsuite`, so the client *is* opted in — the file was simply not
120
+ applied. Symptom is identical: **MySQL 1054 on a nested write**, surfaced as EV-12 with a
121
+ `modelSaveError`.
122
+
123
+ > **Audit the module folders as well as `Client/`.** For each tenant, read its `_modules.txt` and
124
+ > diff every `_modules/<module>/` file against the tenant's actual schema. A `Client/`-only audit
125
+ > reports a clean bill of health on a tenant that is still broken.
126
+
127
+ **This strengthens the "specific dates were skipped" reading.** The missing module file is dated
128
+ **`2026-07-10a`** — the **same date** as the already-recorded missing
129
+ `Client/2026-07-10a` (`ItemFulfillments.returnAddressId`). Two different fan-out folders both missing
130
+ the same date is far better explained by *that date's run failing* than by "beta is behind as of
131
+ date X."
132
+
133
+ **⚠ Diagnostic trap: do not attribute an unexpected runtime field list to a model class you read
134
+ locally.** During this session the missing-module 1054 was initially mis-diagnosed as a *cross-tenant
135
+ `$_modelCache` leak*, because the model-class comparison was made against a **local `_underscore`
136
+ checkout on branch `TRUE-80282`** while beta deploys **`_sandbox-dev`**. Beta runtime behavior
137
+ compared to a local checkout on a different branch is **not evidence** —
138
+ `_underscore` is cloned at build time from a moving branch
139
+ ([environment-variable-drives-underscore-branch](../../api2/features/environment-variable-drives-underscore-branch.md)).
140
+ Check the environment's built branch **before** concluding anything about model classes; the boring
141
+ "a migration never ran" explanation is the one that keeps being true.
142
+
113
143
  **Fix direction: run the existing pending `Client/` files — do NOT author a new migration**, which
114
144
  would duplicate a file already on `_main`. A scoped manual `ALTER` on one tenant unblocks your own
115
145
  testing but leaves the other 30 broken; say so when you do it.
@@ -205,6 +235,14 @@ blocked by the id-divergence hazard in item 2 above, which is why both go to jca
205
235
 
206
236
  ## Change history
207
237
 
238
+ - 2026-08-17 (later pass) — Extended the fan-out gap: it **also hits `_modules/<module>/`**, not just
239
+ `Client/`. Beta `Client_Aig` had never run `_modules/netsuite/2026-07-10a - UnitInventoryFields.sql`
240
+ despite `_modules.txt` listing `netsuite`, so `Units` lacked six `c_` columns prod has → 1054/EV-12
241
+ on nested unit writes. Note it is the **same date** as the missing `Client/2026-07-10a`, which
242
+ favors "specific dates were skipped" over "beta is behind as of date X". Added the **diagnostic
243
+ trap**: that 1054 was first mis-diagnosed as a cross-tenant `$_modelCache` leak because model
244
+ classes were read from a **local `_underscore` checkout on branch `TRUE-80282`** while beta deploys
245
+ `_sandbox-dev` — verify the environment's built branch before blaming code. (mhammontree)
208
246
  - 2026-08-17 — TRUE-80562: documented that the **`Client/` fan-out is skipped independently of
209
247
  `Core/`** on dev-sandbox — 31 of 33 tenant DBs lack `Entitlements.serviceAddressId` (only
210
248
  `Client_Rate`/`Client_Elite` have it) while Core carries the RecordField (2763), so **entitlement
@@ -214,32 +214,55 @@ one of them to the maximum.
214
214
  (`Logs_Aig.Api` and the core/writer `Logs`), keyed by contract number. **Confirm that before
215
215
  assuming the loss is permanent.**
216
216
 
217
- ## Open question — cross-tenant field list on nested `Units` writes (mechanism UNPROVEN)
217
+ ## RESOLVED — the `Units` 1054 was a missing module migration, NOT a cross-tenant leak
218
218
 
219
- An AIG POST carrying `entitlementUnits` produced an `INSERT INTO Units` that included
219
+ **Supersedes the earlier "cross-tenant field list" open question. There is no leak — do not repeat
220
+ that hypothesis.** An AIG POST carrying `entitlementUnits` produced an `INSERT INTO Units` including
220
221
  `c_itemDescription`, `c_lastModified`, `c_chargingBrick`, `c_deviceStatus`, `c_chargingCable`,
221
- `c_grade` → **MySQL 1054**, surfaced as EV-12 with a `modelSaveError`.
222
-
223
- **Verified facts:**
224
-
225
- - `Client_Aig.Units` has **only** `c_netsuiteInternalInventoryAssignmentId`, and
226
- `Client_Aig.CustomRecordFields` declares only that same one field for `Units` (recordId 31) —
227
- AIG's schema and metadata **agree**.
228
- - Those six columns exist only in `Client_Adyen`, `Client_Elite`, `Client_Growrk`,
229
- `Client_Prudential`. Five are declared **only** on `_Model_Growrk_Unit`; `_Model_Aig_Unit` extends
230
- `_Model_Client_Unit` and declares none. `c_lastModified` is declared in **no** model anywhere in
231
- `_underscore`.
232
- - So the field list is neither AIG's schema, nor AIG's metadata, nor any model class AIG uses — it is
233
- **exactly four other tenants' column set**.
234
-
235
- **Hypothesis — NOT PROVEN, do not repeat as fact.** `_underscore/Model.php` (~676–687) caches query
236
- results as `_Database::$_modelCache[$this::DATABASE][$sql]`, where `DATABASE` is the shared **alias**
237
- (`'Client'`), not the tenant schema name. It is **unknown** whether that static survives across
238
- requests under PHP-FPM. That distinction is the difference between a harmless within-request artifact
239
- and a **cross-tenant data-leak vector**, so it needs a focused security review before anyone dismisses
240
- it. Entry points: `_underscore/Model.php` ~676–687, `api2/Component/Api/V2/V2.php` ~8918–8931,
241
- `_underscore/Model/Growrk/Unit.php`, `_underscore/Model/Aig/Unit.php`. See
242
- [per-client database connections](../../../2.0/apps/_underscore/features/per-client-database-connections.md).
222
+ `c_grade` → **MySQL 1054**. Those are **AIG's own fields**:
223
+
224
+ - `dbchanges2/_modules/netsuite/2026-07-10a - UnitInventoryFields.sql` adds all six columns **and**
225
+ their `CustomRecordFields` rows to every client whose `_modules.txt` lists `netsuite`.
226
+ **`Client_Aig/_modules.txt` contains exactly `netsuite`** — AIG is opted in.
227
+ - **Prod `Client_Aig.Units` has all 7 custom columns.** Beta had never run the module file — the same
228
+ execution gap as `Entitlements.serviceAddressId`. After running it manually on beta `Client_Aig`,
229
+ `Units` gained all 7 columns and the nested unit write succeeded with those fields present.
230
+ - The wrong conclusion came from comparing beta runtime behavior against a **local `_underscore`
231
+ checkout on branch `TRUE-80282`** while beta deploys `_sandbox-dev`. See
232
+ [per-client database connections](../../../2.0/apps/_underscore/features/per-client-database-connections.md)
233
+ and [non-prod metadata drift repair](../../../2.0/apps/dbchanges2/workflows/nonprod-metadata-drift-repair.md).
234
+
235
+ ## Verified end-to-end with the full real-shaped payload (TRUE-80562, 2026-08-17)
236
+
237
+ `POST /v2/entitlements` on `api.beta.togahub.com` returned **201 with `entitlementUnits`
238
+ INCLUDED** — 1 unit, 3 coverage types (`C001`/`C002`/`C039`), 1 fulfillment method, 3 contact email
239
+ addresses.
240
+
241
+ > **This supersedes the earlier partial verification.** Removing `entitlementUnits` was **only a
242
+ > diagnostic isolation step**. **AIG must keep sending it — the unit is the covered device.** Their
243
+ > payload was never wrong.
244
+
245
+ ## `Entitlements.number` carries a UNIQUE key — a replay 400s, it is not idempotent
246
+
247
+ Re-POSTing an already-created contract number returns **EV-10 / MySQL 1062** *"Duplicate entry … for
248
+ key `Entitlements.number`"* as an **HTTP 400**. This matters against AIG's documented
249
+ [scheduled re-send model](#feed-is-a-scheduled-re-send-batch-not-one-shot--self-healing): the feed is
250
+ safe **only while it re-sends unacknowledged contracts**. A bulk replay or back-fill over
251
+ already-created contracts will 400 per row — plan for that before running one.
252
+
253
+ ## Beta test data left behind by TRUE-80562 (clean-up list)
254
+
255
+ For whoever cleans up beta `Client_Aig` (row ids only — do not copy the QA contact PII anywhere):
256
+
257
+ - `Entitlements` id **838** (number `1000045957545`) and the 10:41 entitlement (number
258
+ `1000045957546`).
259
+ - `Contacts` **854** and **856**, with the cross-linked primary email row **250** / phone row **18**
260
+ described in
261
+ [nested-relationship writes](../../../2.0/apps/api2/features/nested-relationship-writes.md).
262
+ - `Items` id **1** — `modelNumber` polluted with a test value; **revert to `NULL`**.
263
+ - **Beta `Client_Aig` had two migrations applied MANUALLY this session** (`Client/2026-07-22a`
264
+ `serviceAddressId` ALTERs, and the `_modules/netsuite/2026-07-10a` module file), so this tenant is
265
+ now **ahead of whatever the beta executor believes it has applied**.
243
266
 
244
267
  ## Business Unit (`UserDefined2`) — answered and DROPPED (nothing to build)
245
268
 
@@ -463,6 +486,18 @@ this interceptor or use this dual-purpose Items pattern.
463
486
 
464
487
  ## Change history
465
488
 
489
+ - 2026-08-17 (later pass) — **Correction + final verification, supersedes two points in the entry
490
+ below.** (1) The nested-`Units` "cross-tenant field list" open question is **WITHDRAWN** — the six
491
+ `c_` columns are AIG's **own**, added by `_modules/netsuite/2026-07-10a` to every client whose
492
+ `_modules.txt` lists `netsuite` (AIG's does); prod has all 7 columns, beta had never run the module
493
+ file. Running it on beta fixed the write. The wrong conclusion came from comparing beta runtime
494
+ against a **local `_underscore` checkout on branch `TRUE-80282`** vs beta's `_sandbox-dev`. (2)
495
+ Verified end-to-end with the **full real-shaped payload including `entitlementUnits`** — 201 with
496
+ 1 unit / 3 coverage types / 1 fulfillment method / 3 emails; removing `entitlementUnits` earlier was
497
+ only an isolation step and **AIG must keep sending it**. Also recorded: `Entitlements.number` has a
498
+ **UNIQUE key**, so a replay of an already-created contract returns EV-10/1062 as HTTP 400 (matters
499
+ for any bulk back-fill); and a **beta clean-up list** of the rows/manual migrations this session
500
+ left behind. (mhammontree)
466
501
  - 2026-08-17 — TRUE-80562 (beta EV-12 on entitlement intake). The TRUE-79978 **manual beta patch did
467
502
  not survive a tenant reseed** — same EV-12 recurred 2026-07-21 (`Logs_Aig.Api` 93/95/98); the fix
468
503
  now ships as `dbchanges2/Client_Aig/2026-08-17a`. Recorded the **scope rule** (only correct NULL
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "toga-ai",
3
- "version": "1.0.590",
3
+ "version": "1.0.592",
4
4
  "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
5
5
  "keywords": [
6
6
  "claude",