toga-ai 1.0.463 → 1.0.465

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -6,8 +6,8 @@ project: Library
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-06-23
10
- owners: [jcardinal]
9
+ updated: 2026-07-28
10
+ owners: [jcardinal, mhammontree]
11
11
  files:
12
12
  - library/app/api/toga2.php
13
13
  - worker/crons/toga2/aig/sync_togasupply_aig.php
@@ -90,6 +90,17 @@ bool $enabled2to1, bool $enabled1to2)`. When `enabled2to1`:
90
90
  - Pages 2.0 **`/contacts`** (for AIG, only those with `c_togaCustomerId IS NULL`) and upserts each
91
91
  into the 1.0 client DB `Customers`/`Addresses` via `syncContactToToga1Customer`, then **writes
92
92
  the new 1.0 id back** to the 2.0 contact's `c_togaCustomerId` (PUT `/contacts/{uuid}`).
93
+ - **Email storage (`syncContactToToga1Customer`, TRUE-79401):** the 1.0 `Customers.emailAddress`
94
+ column stores the contact's **full** list of 2.0 email addresses joined by the class constant
95
+ `CUSTOMER_EMAIL_DELIMITER = '; '`, not just the primary. The list is built from
96
+ `array_column((array)($togaContact->contactEmailAddresses ?? []), 'emailAddress')` →
97
+ trim / `array_filter` / `array_unique` → `implode` → `App_Database::sqlEscape` (escape **after**
98
+ imploding); an empty collection falls back to the single primary email (no regression). A
99
+ separately-escaped `$primaryEmailAddress` is retained **only** for the customer-dedupe lookup
100
+ (`WHERE Customers.emailAddress LIKE '$primaryEmailAddress'`) so storing a list doesn't break
101
+ record matching — dedupe is authoritatively keyed on **`c_togaCustomerId`**, with the primary-email
102
+ `LIKE` as a legacy fallback. Consuming the `contactEmailAddresses.emailAddress` collection required
103
+ adding it to the `/contacts` fetch `fields` whitelist (see gotcha).
93
104
  - Pages 2.0 **`/entitlements`** without a `c_togaServiceRequestId` and inserts 1.0
94
105
  `ServiceRequests` (+ a `Contacts` row if needed) via `syncEntitlementToToga1ServiceRequest`.
95
106
  - The 1→2 direction (`enabled1to2`) is a stub (not yet implemented).
@@ -154,11 +165,37 @@ enable flags** and an optional `$monitorTogadeskDepartmentIds[]`:
154
165
  the `App_Model` layer; follow the surrounding escaping discipline when modifying.
155
166
  - **Per-record error isolation** in the TOGaDesk sync is via `try/catch` → `\Sentry\captureException`;
156
167
  a thrown exception elsewhere (transport, checkpoint read) still aborts the whole run.
168
+ - **A reverse hasMany collection is only returned if you add it to the fetch `fields` whitelist.**
169
+ The `/contacts` fetch requests an **explicit** `fields` list from the 2.0 metadata API; a reverse
170
+ hasMany collection (e.g. `contactEmailAddresses.emailAddress`) is **not** returned unless it is
171
+ added to that list — so consuming a related collection means editing **both** the fetch field list
172
+ **and** the consuming code. (The general 2.0-side metadata behavior — reverse relations resolved via
173
+ the child model's `FOREIGNKEY_MODEL` and named as the camelCase plural of the child model — is a
174
+ framework-level fact captured separately.)
175
+ - **`TOGA_AIG.Customers.emailAddress` is `varchar(255)`** (V1 legacy, latin1_swedish_ci) — wide
176
+ enough to hold several `'; '`-joined emails, so TRUE-79401 needed **no** schema/dbchanges migration
177
+ (the earlier fear that it was ~VARCHAR(55) was wrong). `TOGA_AIG` is the 1.0 legacy DB;
178
+ `Client_Aig` is the 2.0 prod tenant.
179
+ - **The multi-email path is not yet exercised (open, TRUE-79401).** As of 2026-07-28 all 20
180
+ `Client_Aig.ContactEmailAddresses` rows are one-per-contact, and **no** api2/`_underscore` code
181
+ references `UserDefined3`/`UserDefined4` or creates `ContactEmailAddresses` rows. Whether an api2
182
+ intake mapping is still needed depends on how Staples / SA.com sends the extra emails: if they POST
183
+ nested `contactEmailAddresses` records to the V2 API the library change is complete; if they send
184
+ flat `UserDefined3`/`UserDefined4` fields expecting us to map them into email rows, an api2 intake
185
+ mapping must be built. Open question owned by **Paulina**.
157
186
  - **PHP 7.2** target (prod worker/library) — no arrow functions, typed properties, `??=`, `match`.
158
187
  Lint with `C:\xampp7\php\php.exe -l`.
159
188
 
160
189
  ## Change history
161
190
 
191
+ - 2026-07-28 — TRUE-79401: `syncContactToToga1Customer` now stores a contact's **full** email list in
192
+ 1.0 `TOGA_AIG.Customers.emailAddress` as a `CUSTOMER_EMAIL_DELIMITER` (`'; '`)-joined, deduped,
193
+ escaped string (was primary-only) for AIG/Staples support lookup. Added
194
+ `contactEmailAddresses.emailAddress` to the `/contacts` fetch `fields` whitelist; kept a separate
195
+ escaped primary email for the dedupe `LIKE` (dedupe stays keyed on `c_togaCustomerId`); empty list
196
+ falls back to primary. No migration needed (`Customers.emailAddress` is `varchar(255)`). Scope stayed
197
+ library-only. Multi-email path not yet exercised — payload shape (nested `contactEmailAddresses` vs
198
+ flat `UserDefined3/4`) is open, owner Paulina. (mhammontree)
162
199
  - 2026-06-23 — Initial documentation of the `App_Api_Toga2` transport (auth/token caching, options
163
200
  DSL) and the 1.0↔2.0 bridge routines `syncWithToga` (Contacts/Entitlements → Customers/ServiceRequests)
164
201
  and `syncWithTogadesk` (bi-directional ticket/repair-order/people/asset sync with dual checkpoint
@@ -18,5 +18,6 @@
18
18
  | [TableView field/column metadata (TableViewFields, hidden projected columns)](features/tableview-field-metadata.md) | The columns of a 2.0 table view are defined by DB metadata, not code. | _underscore/Model/Client/TableView.php, api2/Component/Api/V2/V2.php, dbchanges2/Client/2026-07-20 - ItemsUuidForPurchaseOrderItemsTableView.sql |
19
19
  | [Tickets API (/v2/tickets)](features/tickets-api.md) | The generic ticket endpoint of the 2.0 REST API. | Component/Api/V2/V2.php |
20
20
  | [V2 API error/message codes (EV/EZ troubleshooting map)](features/v2-api-error-codes.md) | The V2 JSON engine (`Component/Api/V2/V2.php`) returns short **message codes** in the response `error` field, grouped by family: `EN-*` authentication, `EZ-*` a | api2/Component/Api/V2/V2.php, _underscore/Model/Client/TrackingNumber.php |
21
+ | [V2 reverse hasMany collections must be named in the fetch fields whitelist](features/v2-reverse-hasmany-fields-whitelist.md) | In the V2 JSON engine, a **reverse hasMany** relationship — the collection of child records that foreign-key back to a parent (e.g. | api2/Component/Api/V2/V2.php |
21
22
  | [AWS CodePipeline Deployment via CodeConnections (GitHub → Elastic Beanstalk)](workflows/codepipeline-codeconnections-deploy.md) | 2.0 apps (`api2`, `_underscore`) are deployed through **AWS CodePipeline**. | api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, api2/ebs/register_instance_to_shared_application_load_balancer.php |
22
23
  | [New Environment Configuration & Provisioning (api2)](workflows/environment-configuration-and-provisioning.md) | What it takes for a 2.0 API environment (e.g. | api2/Config/<environment>.ini, api2/Controller/Index.php, dbchanges2/Core/2026-06-16a - DatabaseHosts for new QA QC stage demo environments.sql, dbchanges2/Logs/, _underscore/Route.php |
@@ -239,6 +239,18 @@ they are the known sharp edges. Do not re-discover these from scratch.
239
239
  [worker2 does](../worker2/features/alb-target-group-auto-registration.md). The same script's
240
240
  IMDSv2 handling (token on the `curl` command line, silent IMDSv1 fallback) must be fixed with it.
241
241
  See the Security note above.
242
+ 10. **Cross-client page-number paging is O(page), and the deepen chunk size is an untuned
243
+ performance dial.** Serving a deep page of a
244
+ [cross-client listing](features/cross-client-data-retrieval.md) requires materializing every
245
+ preceding row into the Cache cluster — page 4430 means 110,747 rows. Adaptive chunking
246
+ (`DEEPEN_CHUNK_MAX_RECORDS = 500`) makes it cheaper, not cheap. The durable fix is exposing
247
+ **keyset cursors to the caller** so cost is independent of depth; V2 already supports
248
+ `seekValues` + `seekUuid` correctly, so the mechanism exists but the caller-facing contract is
249
+ **unbuilt and not yet scoped**. Related dial: chunk size trades round trips against
250
+ sub-response size — a narrowed `fields` request stays small at any chunk, but an unrestricted
251
+ listing expands every row through `getFullModelData()` at the requested depth, which is the case
252
+ where 500 may need lowering. Watch it against `CURL_TIMEOUT_SECONDS = 30`. Do not raise the
253
+ chunk constant without checking sub-response size for wide, deep listings.
242
254
 
243
255
  ## When making changes here
244
256
 
@@ -254,8 +266,13 @@ they are the known sharp edges. Do not re-discover these from scratch.
254
266
  the duplicate check.
255
267
  - Preserve the commit-logs / rollback-data-on-failure invariant when editing the controller
256
268
  or `execute()`.
269
+ - **A code path with exactly one caller gets zero incidental coverage.** V2's keyset-pagination
270
+ branch emitted syntactically invalid SQL for an entire development cycle because only the
271
+ cross-client deepen path ever sets `$isKeysetMode`. In a ~2,000-line untested monolith, treat
272
+ single-caller branches in `V2.php` as unverified until exercised directly.
257
273
 
258
274
  ## Change history
275
+ - 2026-07-28 — Added Known issue #10: cross-client page-number paging is inherently O(page) (deep pages materialize every preceding row into the Cache cluster), with caller-facing keyset cursors identified as the durable fix but left unbuilt/unscoped, plus the `DEEPEN_CHUNK_MAX_RECORDS = 500` tuning tradeoff against `getFullModelData()` expansion and `CURL_TIMEOUT_SECONDS = 30`. Added a change-guidance bullet that single-caller branches in the untested `V2.php` monolith get zero incidental coverage (the keyset `LIMIT` syntax error shipped invisibly for a full cycle). (jcardinal)
259
276
  - 2026-07-28 — Documented the previously unrecorded `060_register_instance_to_shared_application_load_balancer` postdeploy hook pair in the Deployment section (non-prod self-registration into the same-named ALB target group; production skipped), and recorded a **committed IAM access key** in `ebs/register_instance_to_shared_application_load_balancer.php` as a security note + Known issue #9 — location, line range, commit subject, and remediation only (rotate, audit CloudTrail, move to the instance profile; history rewrite is a separate sign-off). Flagged api2's copy as the unhardened original vs. the new worker2 reference implementation. (jcardinal)
260
277
  - 2026-07-28 — Added a consolidated **Known issues / accepted risks** section (8 items), absorbing the previously free-floating deferred raw-exception-disclosure follow-up as item 1, so the tier's sharp edges (unrotated committed secrets, pre-execute phase still outside the main guard, local Logs DB name mismatch, permissive CORS, unpinned `_underscore` build clone, untested `V2.php` monolith, JWT rotation overlap window) are in one place instead of scattered. Recorded that `DB_CACHE` is resolved by name (`Databases.name = 'Cache'`), never by a hardcoded id, which differs per Core instance. (jcardinal)
261
278
  - 2026-07-28 — Added gotcha: the request-logger's auto-generated `Api.transactionId` (millisecond timestamp `Y-m-d H:i:s.v`, UNIQUE) collides under concurrent same-millisecond nested writes → MySQL 1062 → HTTP 500; platform-wide, observed on the Compass/Veyer ASN feed (`sourceIp 34.232.23.158`). Distinct from the client-supplied `transactionId`/EV-5 uniqueness contract. Fix direction: uuid the logged id or retry-on-1062. (bala)
@@ -42,10 +42,11 @@ is a **scatter-gather** engine: it fans out to each entitled client's own V2 API
42
42
  streams with a watermark, and caches the merged result per-user/per-query on a dedicated Cache
43
43
  cluster.
44
44
 
45
- > **Status: working end to end on local dev** (multi-client fan-out, ACL-filtered `client` object,
46
- > typed multi-column sort, keyset paging, narrowed `fields`, per-query cache isolation).
47
- > **Not yet deployed** — next stop is external testing. See
48
- > [api2 architecture](../architecture.md) for the accepted risks carried into that test.
45
+ > **Status: working end to end on local dev AND on a real deployed environment** — first
46
+ > deployed test 2026-07-28 on the **dev sandbox** (`api.beta.togahub.com`, dev-sandbox cluster).
47
+ > That test exposed a page-2-and-beyond failure whose root cause and four follow-on defects are
48
+ > now fixed and verified (see *Deep paging*, *Gotchas* and the change history). See
49
+ > [api2 architecture](../architecture.md) for the accepted risks carried alongside it.
49
50
 
50
51
  ## How it works
51
52
 
@@ -107,6 +108,73 @@ deepening and returned empty pages.
107
108
  the watermark cannot make any row safe, so querying it is a wasted round trip. At 1000 clients that
108
109
  is ~1 sub-request per pass instead of 1000 — the single biggest efficiency property of the design.
109
110
 
111
+ **Worked example (the canonical shape to reason about).** Sorting `-dateOrder` DESC across Compass
112
+ + NYCHH: Compass has 170 rows sharing `dateOrder = 2026-07-27` while NYCHH's *newest* row is
113
+ `2026-06-18`. Compass therefore holds the watermark for many consecutive pages, and NYCHH must
114
+ **not** be re-queried until Compass's cursor passes `2026-06-18`. Verified against live data: after
115
+ the 25-row page-1 cursor, 145 of the 170 same-date rows remain (25 + 145 = 170), i.e. the
116
+ multi-column seek predicate is correct across a large tie block.
117
+
118
+ ### The merge never serves past the watermark (`isDepthUnproven`)
119
+
120
+ `servePage()` used to do a blind `OFFSET`/`LIMIT` over the result table. When deepening **could not**
121
+ cover the requested page — a client failed, a cursor stalled, or `MAX_DEEPEN_ITERATIONS` ran out —
122
+ it still served a full page of rows that a lagging client may legitimately displace, i.e. it
123
+ presented a mis-ordered page as authoritative. One failed sub-request was enough to return 25 wrong
124
+ rows.
125
+
126
+ The page is now **clamped to `Tables.safeRecordCount`**, and any shortfall raises the new public
127
+ `$isDepthUnproven`, which V2 turns into a **WARNING** message telling the caller to retry to
128
+ continue deepening. Two new meta fields: `meta.crossClient.safeRecordCount` and
129
+ `meta.crossClient.isDepthUnproven`. `needsDeepening()` and the flag are deduped onto one new
130
+ `hasUnexhaustedClients()` helper.
131
+
132
+ **Subtlety — the flag is only raised while `hasUnexhaustedClients()` is true.** Once every client is
133
+ exhausted, `safeRecordCount` *is* the true end of the result set, so a short final page is correct
134
+ and must **not** warn.
135
+
136
+ ### Adaptive deepen chunking (sub-request page size ≠ caller's page size)
137
+
138
+ Because only the watermark client is deepened per pass, and each pass used to fetch the **caller's**
139
+ `recordsPerPage`, reaching page N cost ~N sequential sub-requests — and `MAX_DEEPEN_ITERATIONS = 50`
140
+ capped one HTTP request at ~1250 new rows. Requesting page 254 returned an *empty* page plus the
141
+ new warning.
142
+
143
+ The internal fetch size is now decoupled from the caller's:
144
+
145
+ ```
146
+ chunkSize = min(DEEPEN_CHUNK_MAX_RECORDS /* 500 */,
147
+ max(callerRecordsPerPage, target - safeRowCount()))
148
+ ```
149
+
150
+ Page 1 of 25 still fetches exactly 25; page 254 fetches 500 per pass — ~13 round trips instead of
151
+ 254. `fanOut()` takes `chunkSize` instead of `recordsPerPage`.
152
+
153
+ The caller's `recordsPerPage` still governs `servePage()`'s `LIMIT`, `needsDeepening()`'s target,
154
+ `reportedCounts()`, and the **cache identity** (`Tables.recordsPerPage` + query hash). Only the
155
+ internal fetch size changed — **no schema change**.
156
+
157
+ **Chunk-size tuning.** Chunk size trades round trips against sub-response size. A narrowed `fields`
158
+ request stays small at any chunk; an *unrestricted* listing expands every row through
159
+ `getFullModelData()` at the requested depth, and that is the case where 500 may need lowering.
160
+ Watch it against `CURL_TIMEOUT_SECONDS = 30`.
161
+
162
+ ### Only a cursor-less sub-response may set a client's total
163
+
164
+ Once the seek predicate is in the `WHERE` clause, V2's found-rows count is the **remainder after the
165
+ cursor**, not the client's total. `collectRows()` therefore tracks an `$isFirstFetch` map and only a
166
+ usable, cursor-less (offset-mode) sub-response establishes `totalRecordCount`; later deepen passes
167
+ leave the recorded total standing. Without this guard, fixing the keyset `LIMIT` bug would have made
168
+ reported totals silently **shrink** on every deepen pass.
169
+
170
+ ### Diagnosing a fan-out failure from the client logs
171
+
172
+ In `Logs_<Client>.Api`, a sub-request that logged its `POST /v2/auth/encrypted-user-uuid` (201) but
173
+ has **no corresponding GET row** proves the data sub-request died **before** the logging step (a
174
+ fatal) — which is what distinguishes it from a clean 0-row response. That single observation
175
+ isolated the keyset `LIMIT` root cause. Cross-check the *other* client's log too: the **absence** of
176
+ any row there confirms watermark-only deepening queried only the intended client.
177
+
110
178
  ### Per-row `client` object (ACL-filtered)
111
179
 
112
180
  Every returned row carries a `client` object filtered by the caller's **CORE** roles
@@ -157,12 +225,45 @@ cron.
157
225
  in client queries — because the PHP-side watermark merge compares bytes. Any linguistic collation
158
226
  would order differently and the merge could certify rows complete that MySQL then re-orders.
159
227
  - **Never use the raw stored row count** to decide paging depth; only `safeRecordCount`.
228
+ - **Never serve rows past `safeRecordCount`.** A short page + `isDepthUnproven` + a WARNING is
229
+ correct; a full page of possibly-displaceable rows is not.
230
+ - **A failed client is UNKNOWN, never exhausted.** Only a client that provably ran out of rows may
231
+ be marked `exhausted` — the watermark math treats exhausted clients as having no remainder.
232
+ - **Only a cursor-less sub-response may set a client's `totalRecordCount`** — a seek-filtered
233
+ response counts the remainder, not the total.
234
+ - **Anything compared against a sub-request's page size must compare against `chunkSize`**, not the
235
+ caller's `recordsPerPage`, now that the two differ.
160
236
  - **Every outbound sub-request needs its own globally-unique `transactionId`.**
161
237
  - **Forward the caller's original query string verbatim** — parsed `$httpOptions` do not round-trip
162
238
  (see the url-decode gotcha below). `MAX_CONCURRENT_CLIENT_FETCHES = 25` bounds fan-out.
163
239
 
164
240
  ## Gotchas
165
241
 
242
+ - **V2's keyset `LIMIT` had no leading newline — every seek sub-request was a SQL syntax error.**
243
+ In `processRoutePairs()` (`V2.php`, keyset branch, ~line 4113) the keyset path did
244
+ `$sql .= ('LIMIT ' . $recordsPerPage)` while the OFFSET path had a leading newline. The `ORDER BY`
245
+ immediately above ends in a bare `ASC`, so the generated SQL read
246
+ `` CAST(`uuid` AS BINARY) ASCLIMIT 25 ``. It hid for a whole development cycle because **only the
247
+ cross-client deepen path ever sets `$isKeysetMode`** — page 1 uses OFFSET and nothing else in api2
248
+ exercises keyset mode. This was the root cause of the entire "page 2+ returns wrong results" report.
249
+ *Lesson: a code path with exactly one caller gets zero incidental coverage.*
250
+ - **`emptyClientResult()` returning `exhausted => true` turned a transient error into a permanently
251
+ wrong cached answer.** `safeRowCount()` skips exhausted clients when computing the watermark, so a
252
+ failed client dropped out of the watermark → every already-cached row looked provably safe →
253
+ `safeRecordCount` was committed too high → `needsDeepening()` returned false for the **life of the
254
+ cache entry**, permanently excluding that client from every future deepen pass. It is now `false`
255
+ (unknown remainder). No runaway risk: the existing `$newRowCount === 0` stall guard still breaks
256
+ the loop.
257
+ - **`persistCursors()` erased the reported totals of every client it did not fetch this pass.** It
258
+ `DELETE`s all `Tables_Clients` rows then re-`INSERT`s with `$counts[$clientId] ?? null`, but a
259
+ deepen pass only populates `$counts` for the single watermark-holding client. Every other client's
260
+ `totalRecordCount`/`prohibitedRecordCount` became NULL → `meta.totalRecordCount` and
261
+ `totalPageCount` went to 0 → and because `nextPage` derives from `totalPageCount`, `nextPage`
262
+ became `null`, dead-ending any caller that pages by following `nextPage`. Fix: `SELECT` the
263
+ existing counts before the `DELETE` and carry them forward **per field**.
264
+ - **The exhaustion test had to move to `chunkSize` in the same commit as adaptive chunking.**
265
+ Comparing a full 500-row block against a caller page size of 25 declares the client "exhausted" and
266
+ silently reintroduces the `emptyClientResult` class of bug above.
166
267
  - **V2 NEVER url-decodes its query string.** It splits `QUERY_STRING` on `&` and `=` and consumes
167
268
  the raw values. Anything containing `,`, `[`, `"`, `%`, `+` or a space arrives mangled — this bit
168
269
  the feature twice (the JSON `seekValues` array, and a comma-separated `fields` list corrupted into
@@ -200,7 +301,35 @@ cron.
200
301
  - **`curl_multi` busy-spin guard** — both multi loops `usleep(100)` when
201
302
  `curl_multi_select() === -1`.
202
303
 
304
+ ## Known limitation — deep page-number paging is inherently O(page)
305
+
306
+ Serving page 4430 means materializing all 110,747 preceding rows into the cache. Adaptive chunking
307
+ makes that cheaper, not cheap: cost still grows with depth, because a page **number** cannot be
308
+ resolved without knowing everything before it.
309
+
310
+ The durable answer is **exposing keyset cursors to the caller** so cost is independent of depth — the
311
+ mechanism already exists (V2 now supports `seekValues` + `seekUuid` correctly). This was raised with
312
+ the developer on 2026-07-28 and **deliberately left unbuilt / not yet scoped**: record it as a known
313
+ limitation and a candidate direction, not as pending work.
314
+
203
315
  ## Change history
316
+ - 2026-07-28 — **First test on a real deployed environment** (dev sandbox,
317
+ `api.beta.togahub.com`), which reported wrong results on page 2+. Root cause: V2's keyset `LIMIT`
318
+ was concatenated without a leading newline, so **every** seek sub-request was a SQL syntax error —
319
+ invisible until now because only the cross-client deepen path sets `$isKeysetMode`. Fixed, plus the
320
+ four defects it masked: `emptyClientResult()` marking a *failed* client `exhausted` (which
321
+ permanently over-committed `safeRecordCount` and excluded that client from all future deepening);
322
+ `persistCursors()` NULL-ing the totals of every client not fetched in the pass (killing
323
+ `totalRecordCount`/`totalPageCount`/`nextPage`); totals being taken from cursor-filtered
324
+ sub-responses (now only a cursor-less first fetch sets a client's total); and `servePage()` serving
325
+ blind `OFFSET`/`LIMIT` past the watermark (now clamped to `safeRecordCount`, with a new public
326
+ `$isDepthUnproven` → V2 WARNING, `hasUnexhaustedClients()` helper, and new
327
+ `meta.crossClient.safeRecordCount` / `isDepthUnproven`). Added **adaptive deepen chunking**
328
+ (`DEEPEN_CHUNK_MAX_RECORDS = 500`; per-pass `chunkSize`, `fanOut()` and the exhaustion test both
329
+ switched to it) so deep pages cost ~page/500 round trips instead of ~page — no schema change, cache
330
+ identity unchanged. Recorded the Compass/NYCHH tie-block worked example, the log-based fan-out
331
+ failure diagnostic, and deep page-number paging being O(page) as a known limitation (caller-facing
332
+ keyset cursors left unbuilt). Verified working on dev sandbox. (jcardinal)
204
333
  - 2026-07-28 — Took the feature from never-executed to working end to end. Row identity moved from
205
334
  the numeric id to `rowUuid`/`seekUuid` (a V2 LIST never returns `id`); per-query
206
335
  `TableResults_<id>` tables with typed, direction-indexed sort columns; multi-column `seekValues`
@@ -0,0 +1,77 @@
1
+ ---
2
+ title: V2 reverse hasMany collections must be named in the fetch fields whitelist
3
+ framework: "2.0"
4
+ repo: api2
5
+ project: API
6
+ client: shared
7
+ type: feature
8
+ status: active
9
+ updated: 2026-07-28
10
+ owners: [mhammontree]
11
+ files:
12
+ - api2/Component/Api/V2/V2.php
13
+ related:
14
+ - nested-fk-acl-embedding.md
15
+ - ../../../../1.0/apps/library/features/toga2-api-client-and-bridge.md
16
+ ---
17
+
18
+ ## What it is
19
+
20
+ In the V2 JSON engine, a **reverse hasMany** relationship — the collection of child records that
21
+ foreign-key back to a parent (e.g. a Contact's many `ContactEmailAddresses`) — is **not** included
22
+ in a response by default. It is only serialized when the caller **explicitly names it in the fetch
23
+ `fields` list**. A parent GET that does not list the reverse collection comes back with that key
24
+ absent — no error, no warning `message`, still HTTP 200 / `isSuccess: true`. This is a framework-level
25
+ fact about how the 2.0 metadata layer resolves and names reverse relations; it is the general form of
26
+ the AIG-specific `contactEmailAddresses.emailAddress` gotcha in the 1.0 `App_Api_Toga2` bridge.
27
+
28
+ ## How it works
29
+
30
+ - **Reverse relations are metadata-resolved, not implicit.** The V2 engine builds the set of
31
+ embeddable relations for a model from the model metadata. A **forward** FK (parent points at one
32
+ child) resolves from the parent model's own FK field. A **reverse** hasMany (many children point
33
+ back at this parent) is resolved by scanning **child** models for a `FOREIGNKEY_MODEL` constant that
34
+ targets this model — the engine walks the child side to discover "who points at me".
35
+ - **Naming convention — camelCase plural of the child model.** A discovered reverse collection is
36
+ exposed under the **camelCase plural of the child model name**. So children modeled as
37
+ `ContactEmailAddress` (with `FOREIGNKEY_MODEL` → `Contact`) surface on the parent as
38
+ `contactEmailAddresses`; a nested field of that collection is dotted, e.g.
39
+ `contactEmailAddresses.emailAddress`. Get the pluralization/casing exactly right — a mismatched name
40
+ is treated as an unknown field and simply yields nothing rather than an error.
41
+ - **Whitelist-gated serialization.** The engine only serializes a reverse collection that appears in
42
+ the requested `fields` list (for the library options DSL, the nested-array form
43
+ `['contactEmailAddresses' => ['emailAddress']]`). Omit it and the collection is silently skipped —
44
+ it is opt-in, not opt-out. There is no default depth that pulls reverse hasMany collections in for
45
+ you.
46
+ - **Consuming a reverse collection is therefore a two-part change.** Any code that wants to read a
47
+ related collection off a parent must (1) add the collection (and the specific child fields) to the
48
+ **fetch `fields` whitelist**, and (2) add the **consuming code** that reads it. Adding only the
49
+ consumer leaves the key absent at runtime; adding only the field silently fetches data nobody uses.
50
+
51
+ ## Relationship to ACL embedding
52
+
53
+ This whitelist gate is **orthogonal to** the child-record ACL gate documented in
54
+ [nested-fk-acl-embedding.md](nested-fk-acl-embedding.md). Naming a reverse collection in `fields` is
55
+ necessary but not sufficient: each embedded child record is still re-checked against its own
56
+ `AclRecordPermissions`, and an ungranted child is silently dropped. A missing nested collection can be
57
+ caused by **either** an omitted `fields` entry **or** a missing child-record ACL grant — check both.
58
+
59
+ ## Gotchas
60
+
61
+ - **Silent, not erroring.** An unlisted (or misspelled) reverse collection produces no `messages[]`
62
+ entry — the symptom is purely a missing key in `data`. Confirm the exact camelCase-plural name from
63
+ the child model before assuming a data problem.
64
+ - **The 1.0 bridge hit this concretely.** TRUE-79401 needed
65
+ `contactEmailAddresses.emailAddress` added to the `/contacts` fetch `fields` list in
66
+ `App_Api_Toga2` before the 1.0 sync could read a contact's full email list — see
67
+ [App_Api_Toga2 — TOGa2 API Client & bridge](../../../../1.0/apps/library/features/toga2-api-client-and-bridge.md).
68
+
69
+ ## Change history
70
+
71
+ - 2026-07-28 — Captured the framework-level rule (split out of the TRUE-79401 library gotcha): V2
72
+ reverse hasMany collections are resolved via the child model's `FOREIGNKEY_MODEL`, exposed under the
73
+ camelCase plural of the child model, and are **only** serialized when explicitly named in the fetch
74
+ `fields` whitelist — otherwise the key is silently absent (200 / `isSuccess:true`, no message).
75
+ Orthogonal to, and stacks with, the child-record ACL embedding gate. (mhammontree)
76
+ </content>
77
+ </invoke>
@@ -19,7 +19,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
19
19
 
20
20
  - **_underscore** (_Underscore) _(framework core)_ — 40 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
21
21
  - **worker2** (Worker) — 34 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
22
- - **api2** (API) — 18 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
22
+ - **api2** (API) — 19 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
23
23
  - **dbchanges2** (Database Changes) _(framework core)_ — 3 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
24
24
  - **toga2-supply** (TOGa Supply) — 3 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)
25
25
  - **saml** (SAML SSO Gateway) — 3 doc(s) → [2.0/apps/saml/INDEX.md](2.0/apps/saml/INDEX.md)
@@ -5,11 +5,12 @@ apps:
5
5
  - _underscore
6
6
  - api2
7
7
  - dbchanges2
8
+ - library
8
9
  project: API
9
10
  client: aig
10
11
  type: profile
11
12
  status: active
12
- updated: 2026-06-19
13
+ updated: 2026-07-28
13
14
  owners: ["mhammontree"]
14
15
  files: []
15
16
  related:
@@ -35,6 +36,10 @@ go out under the Staples Protection Plan brand and link to `staplesprotection.to
35
36
  NetSuite item/SO/invoice traits).
36
37
  - **`dbchanges2`** — `Client_Aig/` schema + reference-data migrations (e.g. the SaleItem code
37
38
  catalog in `Client_Aig.Items`).
39
+ - **`library` (1.0)** — the `App_Api_Toga2` bridge syncs 2.0 AIG contacts/entitlements down into the
40
+ legacy 1.0 `TOGA_AIG` DB (`Customers`/`ServiceRequests`), run by a worker cron every ~10 min. Note
41
+ the two AIG databases: **`TOGA_AIG`** = V1 legacy sync target; **`Client_Aig`** = V2 prod tenant. See
42
+ [App_Api_Toga2 bridge](../../1.0/apps/library/features/toga2-api-client-and-bridge.md).
38
43
 
39
44
  See [entitlement-intake.md](features/entitlement-intake.md) for how a payload becomes an
40
45
  entitlement and how to load new sale-item codes.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "toga-ai",
3
- "version": "1.0.463",
3
+ "version": "1.0.465",
4
4
  "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
5
5
  "keywords": [
6
6
  "claude",