toga-ai 1.0.463 → 1.0.465
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/1.0/apps/library/features/toga2-api-client-and-bridge.md +39 -2
- package/knowledge/2.0/apps/api2/INDEX.md +1 -0
- package/knowledge/2.0/apps/api2/architecture.md +17 -0
- package/knowledge/2.0/apps/api2/features/cross-client-data-retrieval.md +133 -4
- package/knowledge/2.0/apps/api2/features/v2-reverse-hasmany-fields-whitelist.md +77 -0
- package/knowledge/INDEX.md +1 -1
- package/knowledge/clients/aig/profile.md +6 -1
- package/package.json +1 -1
|
@@ -6,8 +6,8 @@ project: Library
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-
|
|
10
|
-
owners: [jcardinal]
|
|
9
|
+
updated: 2026-07-28
|
|
10
|
+
owners: [jcardinal, mhammontree]
|
|
11
11
|
files:
|
|
12
12
|
- library/app/api/toga2.php
|
|
13
13
|
- worker/crons/toga2/aig/sync_togasupply_aig.php
|
|
@@ -90,6 +90,17 @@ bool $enabled2to1, bool $enabled1to2)`. When `enabled2to1`:
|
|
|
90
90
|
- Pages 2.0 **`/contacts`** (for AIG, only those with `c_togaCustomerId IS NULL`) and upserts each
|
|
91
91
|
into the 1.0 client DB `Customers`/`Addresses` via `syncContactToToga1Customer`, then **writes
|
|
92
92
|
the new 1.0 id back** to the 2.0 contact's `c_togaCustomerId` (PUT `/contacts/{uuid}`).
|
|
93
|
+
- **Email storage (`syncContactToToga1Customer`, TRUE-79401):** the 1.0 `Customers.emailAddress`
|
|
94
|
+
column stores the contact's **full** list of 2.0 email addresses joined by the class constant
|
|
95
|
+
`CUSTOMER_EMAIL_DELIMITER = '; '`, not just the primary. The list is built from
|
|
96
|
+
`array_column((array)($togaContact->contactEmailAddresses ?? []), 'emailAddress')` →
|
|
97
|
+
trim / `array_filter` / `array_unique` → `implode` → `App_Database::sqlEscape` (escape **after**
|
|
98
|
+
imploding); an empty collection falls back to the single primary email (no regression). A
|
|
99
|
+
separately-escaped `$primaryEmailAddress` is retained **only** for the customer-dedupe lookup
|
|
100
|
+
(`WHERE Customers.emailAddress LIKE '$primaryEmailAddress'`) so storing a list doesn't break
|
|
101
|
+
record matching — dedupe is authoritatively keyed on **`c_togaCustomerId`**, with the primary-email
|
|
102
|
+
`LIKE` as a legacy fallback. Consuming the `contactEmailAddresses.emailAddress` collection required
|
|
103
|
+
adding it to the `/contacts` fetch `fields` whitelist (see gotcha).
|
|
93
104
|
- Pages 2.0 **`/entitlements`** without a `c_togaServiceRequestId` and inserts 1.0
|
|
94
105
|
`ServiceRequests` (+ a `Contacts` row if needed) via `syncEntitlementToToga1ServiceRequest`.
|
|
95
106
|
- The 1→2 direction (`enabled1to2`) is a stub (not yet implemented).
|
|
@@ -154,11 +165,37 @@ enable flags** and an optional `$monitorTogadeskDepartmentIds[]`:
|
|
|
154
165
|
the `App_Model` layer; follow the surrounding escaping discipline when modifying.
|
|
155
166
|
- **Per-record error isolation** in the TOGaDesk sync is via `try/catch` → `\Sentry\captureException`;
|
|
156
167
|
a thrown exception elsewhere (transport, checkpoint read) still aborts the whole run.
|
|
168
|
+
- **A reverse hasMany collection is only returned if you add it to the fetch `fields` whitelist.**
|
|
169
|
+
The `/contacts` fetch requests an **explicit** `fields` list from the 2.0 metadata API; a reverse
|
|
170
|
+
hasMany collection (e.g. `contactEmailAddresses.emailAddress`) is **not** returned unless it is
|
|
171
|
+
added to that list — so consuming a related collection means editing **both** the fetch field list
|
|
172
|
+
**and** the consuming code. (The general 2.0-side metadata behavior — reverse relations resolved via
|
|
173
|
+
the child model's `FOREIGNKEY_MODEL` and named as the camelCase plural of the child model — is a
|
|
174
|
+
framework-level fact captured separately.)
|
|
175
|
+
- **`TOGA_AIG.Customers.emailAddress` is `varchar(255)`** (V1 legacy, latin1_swedish_ci) — wide
|
|
176
|
+
enough to hold several `'; '`-joined emails, so TRUE-79401 needed **no** schema/dbchanges migration
|
|
177
|
+
(the earlier fear that it was ~VARCHAR(55) was wrong). `TOGA_AIG` is the 1.0 legacy DB;
|
|
178
|
+
`Client_Aig` is the 2.0 prod tenant.
|
|
179
|
+
- **The multi-email path is not yet exercised (open, TRUE-79401).** As of 2026-07-28 all 20
|
|
180
|
+
`Client_Aig.ContactEmailAddresses` rows are one-per-contact, and **no** api2/`_underscore` code
|
|
181
|
+
references `UserDefined3`/`UserDefined4` or creates `ContactEmailAddresses` rows. Whether an api2
|
|
182
|
+
intake mapping is still needed depends on how Staples / SA.com sends the extra emails: if they POST
|
|
183
|
+
nested `contactEmailAddresses` records to the V2 API the library change is complete; if they send
|
|
184
|
+
flat `UserDefined3`/`UserDefined4` fields expecting us to map them into email rows, an api2 intake
|
|
185
|
+
mapping must be built. Open question owned by **Paulina**.
|
|
157
186
|
- **PHP 7.2** target (prod worker/library) — no arrow functions, typed properties, `??=`, `match`.
|
|
158
187
|
Lint with `C:\xampp7\php\php.exe -l`.
|
|
159
188
|
|
|
160
189
|
## Change history
|
|
161
190
|
|
|
191
|
+
- 2026-07-28 — TRUE-79401: `syncContactToToga1Customer` now stores a contact's **full** email list in
|
|
192
|
+
1.0 `TOGA_AIG.Customers.emailAddress` as a `CUSTOMER_EMAIL_DELIMITER` (`'; '`)-joined, deduped,
|
|
193
|
+
escaped string (was primary-only) for AIG/Staples support lookup. Added
|
|
194
|
+
`contactEmailAddresses.emailAddress` to the `/contacts` fetch `fields` whitelist; kept a separate
|
|
195
|
+
escaped primary email for the dedupe `LIKE` (dedupe stays keyed on `c_togaCustomerId`); empty list
|
|
196
|
+
falls back to primary. No migration needed (`Customers.emailAddress` is `varchar(255)`). Scope stayed
|
|
197
|
+
library-only. Multi-email path not yet exercised — payload shape (nested `contactEmailAddresses` vs
|
|
198
|
+
flat `UserDefined3/4`) is open, owner Paulina. (mhammontree)
|
|
162
199
|
- 2026-06-23 — Initial documentation of the `App_Api_Toga2` transport (auth/token caching, options
|
|
163
200
|
DSL) and the 1.0↔2.0 bridge routines `syncWithToga` (Contacts/Entitlements → Customers/ServiceRequests)
|
|
164
201
|
and `syncWithTogadesk` (bi-directional ticket/repair-order/people/asset sync with dual checkpoint
|
|
@@ -18,5 +18,6 @@
|
|
|
18
18
|
| [TableView field/column metadata (TableViewFields, hidden projected columns)](features/tableview-field-metadata.md) | The columns of a 2.0 table view are defined by DB metadata, not code. | _underscore/Model/Client/TableView.php, api2/Component/Api/V2/V2.php, dbchanges2/Client/2026-07-20 - ItemsUuidForPurchaseOrderItemsTableView.sql |
|
|
19
19
|
| [Tickets API (/v2/tickets)](features/tickets-api.md) | The generic ticket endpoint of the 2.0 REST API. | Component/Api/V2/V2.php |
|
|
20
20
|
| [V2 API error/message codes (EV/EZ troubleshooting map)](features/v2-api-error-codes.md) | The V2 JSON engine (`Component/Api/V2/V2.php`) returns short **message codes** in the response `error` field, grouped by family: `EN-*` authentication, `EZ-*` a | api2/Component/Api/V2/V2.php, _underscore/Model/Client/TrackingNumber.php |
|
|
21
|
+
| [V2 reverse hasMany collections must be named in the fetch fields whitelist](features/v2-reverse-hasmany-fields-whitelist.md) | In the V2 JSON engine, a **reverse hasMany** relationship — the collection of child records that foreign-key back to a parent (e.g. | api2/Component/Api/V2/V2.php |
|
|
21
22
|
| [AWS CodePipeline Deployment via CodeConnections (GitHub → Elastic Beanstalk)](workflows/codepipeline-codeconnections-deploy.md) | 2.0 apps (`api2`, `_underscore`) are deployed through **AWS CodePipeline**. | api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, api2/ebs/register_instance_to_shared_application_load_balancer.php |
|
|
22
23
|
| [New Environment Configuration & Provisioning (api2)](workflows/environment-configuration-and-provisioning.md) | What it takes for a 2.0 API environment (e.g. | api2/Config/<environment>.ini, api2/Controller/Index.php, dbchanges2/Core/2026-06-16a - DatabaseHosts for new QA QC stage demo environments.sql, dbchanges2/Logs/, _underscore/Route.php |
|
|
@@ -239,6 +239,18 @@ they are the known sharp edges. Do not re-discover these from scratch.
|
|
|
239
239
|
[worker2 does](../worker2/features/alb-target-group-auto-registration.md). The same script's
|
|
240
240
|
IMDSv2 handling (token on the `curl` command line, silent IMDSv1 fallback) must be fixed with it.
|
|
241
241
|
See the Security note above.
|
|
242
|
+
10. **Cross-client page-number paging is O(page), and the deepen chunk size is an untuned
|
|
243
|
+
performance dial.** Serving a deep page of a
|
|
244
|
+
[cross-client listing](features/cross-client-data-retrieval.md) requires materializing every
|
|
245
|
+
preceding row into the Cache cluster — page 4430 means 110,747 rows. Adaptive chunking
|
|
246
|
+
(`DEEPEN_CHUNK_MAX_RECORDS = 500`) makes it cheaper, not cheap. The durable fix is exposing
|
|
247
|
+
**keyset cursors to the caller** so cost is independent of depth; V2 already supports
|
|
248
|
+
`seekValues` + `seekUuid` correctly, so the mechanism exists but the caller-facing contract is
|
|
249
|
+
**unbuilt and not yet scoped**. Related dial: chunk size trades round trips against
|
|
250
|
+
sub-response size — a narrowed `fields` request stays small at any chunk, but an unrestricted
|
|
251
|
+
listing expands every row through `getFullModelData()` at the requested depth, which is the case
|
|
252
|
+
where 500 may need lowering. Watch it against `CURL_TIMEOUT_SECONDS = 30`. Do not raise the
|
|
253
|
+
chunk constant without checking sub-response size for wide, deep listings.
|
|
242
254
|
|
|
243
255
|
## When making changes here
|
|
244
256
|
|
|
@@ -254,8 +266,13 @@ they are the known sharp edges. Do not re-discover these from scratch.
|
|
|
254
266
|
the duplicate check.
|
|
255
267
|
- Preserve the commit-logs / rollback-data-on-failure invariant when editing the controller
|
|
256
268
|
or `execute()`.
|
|
269
|
+
- **A code path with exactly one caller gets zero incidental coverage.** V2's keyset-pagination
|
|
270
|
+
branch emitted syntactically invalid SQL for an entire development cycle because only the
|
|
271
|
+
cross-client deepen path ever sets `$isKeysetMode`. In a ~2,000-line untested monolith, treat
|
|
272
|
+
single-caller branches in `V2.php` as unverified until exercised directly.
|
|
257
273
|
|
|
258
274
|
## Change history
|
|
275
|
+
- 2026-07-28 — Added Known issue #10: cross-client page-number paging is inherently O(page) (deep pages materialize every preceding row into the Cache cluster), with caller-facing keyset cursors identified as the durable fix but left unbuilt/unscoped, plus the `DEEPEN_CHUNK_MAX_RECORDS = 500` tuning tradeoff against `getFullModelData()` expansion and `CURL_TIMEOUT_SECONDS = 30`. Added a change-guidance bullet that single-caller branches in the untested `V2.php` monolith get zero incidental coverage (the keyset `LIMIT` syntax error shipped invisibly for a full cycle). (jcardinal)
|
|
259
276
|
- 2026-07-28 — Documented the previously unrecorded `060_register_instance_to_shared_application_load_balancer` postdeploy hook pair in the Deployment section (non-prod self-registration into the same-named ALB target group; production skipped), and recorded a **committed IAM access key** in `ebs/register_instance_to_shared_application_load_balancer.php` as a security note + Known issue #9 — location, line range, commit subject, and remediation only (rotate, audit CloudTrail, move to the instance profile; history rewrite is a separate sign-off). Flagged api2's copy as the unhardened original vs. the new worker2 reference implementation. (jcardinal)
|
|
260
277
|
- 2026-07-28 — Added a consolidated **Known issues / accepted risks** section (8 items), absorbing the previously free-floating deferred raw-exception-disclosure follow-up as item 1, so the tier's sharp edges (unrotated committed secrets, pre-execute phase still outside the main guard, local Logs DB name mismatch, permissive CORS, unpinned `_underscore` build clone, untested `V2.php` monolith, JWT rotation overlap window) are in one place instead of scattered. Recorded that `DB_CACHE` is resolved by name (`Databases.name = 'Cache'`), never by a hardcoded id, which differs per Core instance. (jcardinal)
|
|
261
278
|
- 2026-07-28 — Added gotcha: the request-logger's auto-generated `Api.transactionId` (millisecond timestamp `Y-m-d H:i:s.v`, UNIQUE) collides under concurrent same-millisecond nested writes → MySQL 1062 → HTTP 500; platform-wide, observed on the Compass/Veyer ASN feed (`sourceIp 34.232.23.158`). Distinct from the client-supplied `transactionId`/EV-5 uniqueness contract. Fix direction: uuid the logged id or retry-on-1062. (bala)
|
|
@@ -42,10 +42,11 @@ is a **scatter-gather** engine: it fans out to each entitled client's own V2 API
|
|
|
42
42
|
streams with a watermark, and caches the merged result per-user/per-query on a dedicated Cache
|
|
43
43
|
cluster.
|
|
44
44
|
|
|
45
|
-
> **Status: working end to end on local dev
|
|
46
|
-
>
|
|
47
|
-
>
|
|
48
|
-
>
|
|
45
|
+
> **Status: working end to end on local dev AND on a real deployed environment** — first
|
|
46
|
+
> deployed test 2026-07-28 on the **dev sandbox** (`api.beta.togahub.com`, dev-sandbox cluster).
|
|
47
|
+
> That test exposed a page-2-and-beyond failure whose root cause and four follow-on defects are
|
|
48
|
+
> now fixed and verified (see *Deep paging*, *Gotchas* and the change history). See
|
|
49
|
+
> [api2 architecture](../architecture.md) for the accepted risks carried alongside it.
|
|
49
50
|
|
|
50
51
|
## How it works
|
|
51
52
|
|
|
@@ -107,6 +108,73 @@ deepening and returned empty pages.
|
|
|
107
108
|
the watermark cannot make any row safe, so querying it is a wasted round trip. At 1000 clients that
|
|
108
109
|
is ~1 sub-request per pass instead of 1000 — the single biggest efficiency property of the design.
|
|
109
110
|
|
|
111
|
+
**Worked example (the canonical shape to reason about).** Sorting `-dateOrder` DESC across Compass
|
|
112
|
+
+ NYCHH: Compass has 170 rows sharing `dateOrder = 2026-07-27` while NYCHH's *newest* row is
|
|
113
|
+
`2026-06-18`. Compass therefore holds the watermark for many consecutive pages, and NYCHH must
|
|
114
|
+
**not** be re-queried until Compass's cursor passes `2026-06-18`. Verified against live data: after
|
|
115
|
+
the 25-row page-1 cursor, 145 of the 170 same-date rows remain (25 + 145 = 170), i.e. the
|
|
116
|
+
multi-column seek predicate is correct across a large tie block.
|
|
117
|
+
|
|
118
|
+
### The merge never serves past the watermark (`isDepthUnproven`)
|
|
119
|
+
|
|
120
|
+
`servePage()` used to do a blind `OFFSET`/`LIMIT` over the result table. When deepening **could not**
|
|
121
|
+
cover the requested page — a client failed, a cursor stalled, or `MAX_DEEPEN_ITERATIONS` ran out —
|
|
122
|
+
it still served a full page of rows that a lagging client may legitimately displace, i.e. it
|
|
123
|
+
presented a mis-ordered page as authoritative. One failed sub-request was enough to return 25 wrong
|
|
124
|
+
rows.
|
|
125
|
+
|
|
126
|
+
The page is now **clamped to `Tables.safeRecordCount`**, and any shortfall raises the new public
|
|
127
|
+
`$isDepthUnproven`, which V2 turns into a **WARNING** message telling the caller to retry to
|
|
128
|
+
continue deepening. Two new meta fields: `meta.crossClient.safeRecordCount` and
|
|
129
|
+
`meta.crossClient.isDepthUnproven`. `needsDeepening()` and the flag are deduped onto one new
|
|
130
|
+
`hasUnexhaustedClients()` helper.
|
|
131
|
+
|
|
132
|
+
**Subtlety — the flag is only raised while `hasUnexhaustedClients()` is true.** Once every client is
|
|
133
|
+
exhausted, `safeRecordCount` *is* the true end of the result set, so a short final page is correct
|
|
134
|
+
and must **not** warn.
|
|
135
|
+
|
|
136
|
+
### Adaptive deepen chunking (sub-request page size ≠ caller's page size)
|
|
137
|
+
|
|
138
|
+
Because only the watermark client is deepened per pass, and each pass used to fetch the **caller's**
|
|
139
|
+
`recordsPerPage`, reaching page N cost ~N sequential sub-requests — and `MAX_DEEPEN_ITERATIONS = 50`
|
|
140
|
+
capped one HTTP request at ~1250 new rows. Requesting page 254 returned an *empty* page plus the
|
|
141
|
+
new warning.
|
|
142
|
+
|
|
143
|
+
The internal fetch size is now decoupled from the caller's:
|
|
144
|
+
|
|
145
|
+
```
|
|
146
|
+
chunkSize = min(DEEPEN_CHUNK_MAX_RECORDS /* 500 */,
|
|
147
|
+
max(callerRecordsPerPage, target - safeRowCount()))
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
Page 1 of 25 still fetches exactly 25; page 254 fetches 500 per pass — ~13 round trips instead of
|
|
151
|
+
254. `fanOut()` takes `chunkSize` instead of `recordsPerPage`.
|
|
152
|
+
|
|
153
|
+
The caller's `recordsPerPage` still governs `servePage()`'s `LIMIT`, `needsDeepening()`'s target,
|
|
154
|
+
`reportedCounts()`, and the **cache identity** (`Tables.recordsPerPage` + query hash). Only the
|
|
155
|
+
internal fetch size changed — **no schema change**.
|
|
156
|
+
|
|
157
|
+
**Chunk-size tuning.** Chunk size trades round trips against sub-response size. A narrowed `fields`
|
|
158
|
+
request stays small at any chunk; an *unrestricted* listing expands every row through
|
|
159
|
+
`getFullModelData()` at the requested depth, and that is the case where 500 may need lowering.
|
|
160
|
+
Watch it against `CURL_TIMEOUT_SECONDS = 30`.
|
|
161
|
+
|
|
162
|
+
### Only a cursor-less sub-response may set a client's total
|
|
163
|
+
|
|
164
|
+
Once the seek predicate is in the `WHERE` clause, V2's found-rows count is the **remainder after the
|
|
165
|
+
cursor**, not the client's total. `collectRows()` therefore tracks an `$isFirstFetch` map and only a
|
|
166
|
+
usable, cursor-less (offset-mode) sub-response establishes `totalRecordCount`; later deepen passes
|
|
167
|
+
leave the recorded total standing. Without this guard, fixing the keyset `LIMIT` bug would have made
|
|
168
|
+
reported totals silently **shrink** on every deepen pass.
|
|
169
|
+
|
|
170
|
+
### Diagnosing a fan-out failure from the client logs
|
|
171
|
+
|
|
172
|
+
In `Logs_<Client>.Api`, a sub-request that logged its `POST /v2/auth/encrypted-user-uuid` (201) but
|
|
173
|
+
has **no corresponding GET row** proves the data sub-request died **before** the logging step (a
|
|
174
|
+
fatal) — which is what distinguishes it from a clean 0-row response. That single observation
|
|
175
|
+
isolated the keyset `LIMIT` root cause. Cross-check the *other* client's log too: the **absence** of
|
|
176
|
+
any row there confirms watermark-only deepening queried only the intended client.
|
|
177
|
+
|
|
110
178
|
### Per-row `client` object (ACL-filtered)
|
|
111
179
|
|
|
112
180
|
Every returned row carries a `client` object filtered by the caller's **CORE** roles
|
|
@@ -157,12 +225,45 @@ cron.
|
|
|
157
225
|
in client queries — because the PHP-side watermark merge compares bytes. Any linguistic collation
|
|
158
226
|
would order differently and the merge could certify rows complete that MySQL then re-orders.
|
|
159
227
|
- **Never use the raw stored row count** to decide paging depth; only `safeRecordCount`.
|
|
228
|
+
- **Never serve rows past `safeRecordCount`.** A short page + `isDepthUnproven` + a WARNING is
|
|
229
|
+
correct; a full page of possibly-displaceable rows is not.
|
|
230
|
+
- **A failed client is UNKNOWN, never exhausted.** Only a client that provably ran out of rows may
|
|
231
|
+
be marked `exhausted` — the watermark math treats exhausted clients as having no remainder.
|
|
232
|
+
- **Only a cursor-less sub-response may set a client's `totalRecordCount`** — a seek-filtered
|
|
233
|
+
response counts the remainder, not the total.
|
|
234
|
+
- **Anything compared against a sub-request's page size must compare against `chunkSize`**, not the
|
|
235
|
+
caller's `recordsPerPage`, now that the two differ.
|
|
160
236
|
- **Every outbound sub-request needs its own globally-unique `transactionId`.**
|
|
161
237
|
- **Forward the caller's original query string verbatim** — parsed `$httpOptions` do not round-trip
|
|
162
238
|
(see the url-decode gotcha below). `MAX_CONCURRENT_CLIENT_FETCHES = 25` bounds fan-out.
|
|
163
239
|
|
|
164
240
|
## Gotchas
|
|
165
241
|
|
|
242
|
+
- **V2's keyset `LIMIT` had no leading newline — every seek sub-request was a SQL syntax error.**
|
|
243
|
+
In `processRoutePairs()` (`V2.php`, keyset branch, ~line 4113) the keyset path did
|
|
244
|
+
`$sql .= ('LIMIT ' . $recordsPerPage)` while the OFFSET path had a leading newline. The `ORDER BY`
|
|
245
|
+
immediately above ends in a bare `ASC`, so the generated SQL read
|
|
246
|
+
`` CAST(`uuid` AS BINARY) ASCLIMIT 25 ``. It hid for a whole development cycle because **only the
|
|
247
|
+
cross-client deepen path ever sets `$isKeysetMode`** — page 1 uses OFFSET and nothing else in api2
|
|
248
|
+
exercises keyset mode. This was the root cause of the entire "page 2+ returns wrong results" report.
|
|
249
|
+
*Lesson: a code path with exactly one caller gets zero incidental coverage.*
|
|
250
|
+
- **`emptyClientResult()` returning `exhausted => true` turned a transient error into a permanently
|
|
251
|
+
wrong cached answer.** `safeRowCount()` skips exhausted clients when computing the watermark, so a
|
|
252
|
+
failed client dropped out of the watermark → every already-cached row looked provably safe →
|
|
253
|
+
`safeRecordCount` was committed too high → `needsDeepening()` returned false for the **life of the
|
|
254
|
+
cache entry**, permanently excluding that client from every future deepen pass. It is now `false`
|
|
255
|
+
(unknown remainder). No runaway risk: the existing `$newRowCount === 0` stall guard still breaks
|
|
256
|
+
the loop.
|
|
257
|
+
- **`persistCursors()` erased the reported totals of every client it did not fetch this pass.** It
|
|
258
|
+
`DELETE`s all `Tables_Clients` rows then re-`INSERT`s with `$counts[$clientId] ?? null`, but a
|
|
259
|
+
deepen pass only populates `$counts` for the single watermark-holding client. Every other client's
|
|
260
|
+
`totalRecordCount`/`prohibitedRecordCount` became NULL → `meta.totalRecordCount` and
|
|
261
|
+
`totalPageCount` went to 0 → and because `nextPage` derives from `totalPageCount`, `nextPage`
|
|
262
|
+
became `null`, dead-ending any caller that pages by following `nextPage`. Fix: `SELECT` the
|
|
263
|
+
existing counts before the `DELETE` and carry them forward **per field**.
|
|
264
|
+
- **The exhaustion test had to move to `chunkSize` in the same commit as adaptive chunking.**
|
|
265
|
+
Comparing a full 500-row block against a caller page size of 25 declares the client "exhausted" and
|
|
266
|
+
silently reintroduces the `emptyClientResult` class of bug above.
|
|
166
267
|
- **V2 NEVER url-decodes its query string.** It splits `QUERY_STRING` on `&` and `=` and consumes
|
|
167
268
|
the raw values. Anything containing `,`, `[`, `"`, `%`, `+` or a space arrives mangled — this bit
|
|
168
269
|
the feature twice (the JSON `seekValues` array, and a comma-separated `fields` list corrupted into
|
|
@@ -200,7 +301,35 @@ cron.
|
|
|
200
301
|
- **`curl_multi` busy-spin guard** — both multi loops `usleep(100)` when
|
|
201
302
|
`curl_multi_select() === -1`.
|
|
202
303
|
|
|
304
|
+
## Known limitation — deep page-number paging is inherently O(page)
|
|
305
|
+
|
|
306
|
+
Serving page 4430 means materializing all 110,747 preceding rows into the cache. Adaptive chunking
|
|
307
|
+
makes that cheaper, not cheap: cost still grows with depth, because a page **number** cannot be
|
|
308
|
+
resolved without knowing everything before it.
|
|
309
|
+
|
|
310
|
+
The durable answer is **exposing keyset cursors to the caller** so cost is independent of depth — the
|
|
311
|
+
mechanism already exists (V2 now supports `seekValues` + `seekUuid` correctly). This was raised with
|
|
312
|
+
the developer on 2026-07-28 and **deliberately left unbuilt / not yet scoped**: record it as a known
|
|
313
|
+
limitation and a candidate direction, not as pending work.
|
|
314
|
+
|
|
203
315
|
## Change history
|
|
316
|
+
- 2026-07-28 — **First test on a real deployed environment** (dev sandbox,
|
|
317
|
+
`api.beta.togahub.com`), which reported wrong results on page 2+. Root cause: V2's keyset `LIMIT`
|
|
318
|
+
was concatenated without a leading newline, so **every** seek sub-request was a SQL syntax error —
|
|
319
|
+
invisible until now because only the cross-client deepen path sets `$isKeysetMode`. Fixed, plus the
|
|
320
|
+
four defects it masked: `emptyClientResult()` marking a *failed* client `exhausted` (which
|
|
321
|
+
permanently over-committed `safeRecordCount` and excluded that client from all future deepening);
|
|
322
|
+
`persistCursors()` NULL-ing the totals of every client not fetched in the pass (killing
|
|
323
|
+
`totalRecordCount`/`totalPageCount`/`nextPage`); totals being taken from cursor-filtered
|
|
324
|
+
sub-responses (now only a cursor-less first fetch sets a client's total); and `servePage()` serving
|
|
325
|
+
blind `OFFSET`/`LIMIT` past the watermark (now clamped to `safeRecordCount`, with a new public
|
|
326
|
+
`$isDepthUnproven` → V2 WARNING, `hasUnexhaustedClients()` helper, and new
|
|
327
|
+
`meta.crossClient.safeRecordCount` / `isDepthUnproven`). Added **adaptive deepen chunking**
|
|
328
|
+
(`DEEPEN_CHUNK_MAX_RECORDS = 500`; per-pass `chunkSize`, `fanOut()` and the exhaustion test both
|
|
329
|
+
switched to it) so deep pages cost ~page/500 round trips instead of ~page — no schema change, cache
|
|
330
|
+
identity unchanged. Recorded the Compass/NYCHH tie-block worked example, the log-based fan-out
|
|
331
|
+
failure diagnostic, and deep page-number paging being O(page) as a known limitation (caller-facing
|
|
332
|
+
keyset cursors left unbuilt). Verified working on dev sandbox. (jcardinal)
|
|
204
333
|
- 2026-07-28 — Took the feature from never-executed to working end to end. Row identity moved from
|
|
205
334
|
the numeric id to `rowUuid`/`seekUuid` (a V2 LIST never returns `id`); per-query
|
|
206
335
|
`TableResults_<id>` tables with typed, direction-indexed sort columns; multi-column `seekValues`
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: V2 reverse hasMany collections must be named in the fetch fields whitelist
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
repo: api2
|
|
5
|
+
project: API
|
|
6
|
+
client: shared
|
|
7
|
+
type: feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-07-28
|
|
10
|
+
owners: [mhammontree]
|
|
11
|
+
files:
|
|
12
|
+
- api2/Component/Api/V2/V2.php
|
|
13
|
+
related:
|
|
14
|
+
- nested-fk-acl-embedding.md
|
|
15
|
+
- ../../../../1.0/apps/library/features/toga2-api-client-and-bridge.md
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
## What it is
|
|
19
|
+
|
|
20
|
+
In the V2 JSON engine, a **reverse hasMany** relationship — the collection of child records that
|
|
21
|
+
foreign-key back to a parent (e.g. a Contact's many `ContactEmailAddresses`) — is **not** included
|
|
22
|
+
in a response by default. It is only serialized when the caller **explicitly names it in the fetch
|
|
23
|
+
`fields` list**. A parent GET that does not list the reverse collection comes back with that key
|
|
24
|
+
absent — no error, no warning `message`, still HTTP 200 / `isSuccess: true`. This is a framework-level
|
|
25
|
+
fact about how the 2.0 metadata layer resolves and names reverse relations; it is the general form of
|
|
26
|
+
the AIG-specific `contactEmailAddresses.emailAddress` gotcha in the 1.0 `App_Api_Toga2` bridge.
|
|
27
|
+
|
|
28
|
+
## How it works
|
|
29
|
+
|
|
30
|
+
- **Reverse relations are metadata-resolved, not implicit.** The V2 engine builds the set of
|
|
31
|
+
embeddable relations for a model from the model metadata. A **forward** FK (parent points at one
|
|
32
|
+
child) resolves from the parent model's own FK field. A **reverse** hasMany (many children point
|
|
33
|
+
back at this parent) is resolved by scanning **child** models for a `FOREIGNKEY_MODEL` constant that
|
|
34
|
+
targets this model — the engine walks the child side to discover "who points at me".
|
|
35
|
+
- **Naming convention — camelCase plural of the child model.** A discovered reverse collection is
|
|
36
|
+
exposed under the **camelCase plural of the child model name**. So children modeled as
|
|
37
|
+
`ContactEmailAddress` (with `FOREIGNKEY_MODEL` → `Contact`) surface on the parent as
|
|
38
|
+
`contactEmailAddresses`; a nested field of that collection is dotted, e.g.
|
|
39
|
+
`contactEmailAddresses.emailAddress`. Get the pluralization/casing exactly right — a mismatched name
|
|
40
|
+
is treated as an unknown field and simply yields nothing rather than an error.
|
|
41
|
+
- **Whitelist-gated serialization.** The engine only serializes a reverse collection that appears in
|
|
42
|
+
the requested `fields` list (for the library options DSL, the nested-array form
|
|
43
|
+
`['contactEmailAddresses' => ['emailAddress']]`). Omit it and the collection is silently skipped —
|
|
44
|
+
it is opt-in, not opt-out. There is no default depth that pulls reverse hasMany collections in for
|
|
45
|
+
you.
|
|
46
|
+
- **Consuming a reverse collection is therefore a two-part change.** Any code that wants to read a
|
|
47
|
+
related collection off a parent must (1) add the collection (and the specific child fields) to the
|
|
48
|
+
**fetch `fields` whitelist**, and (2) add the **consuming code** that reads it. Adding only the
|
|
49
|
+
consumer leaves the key absent at runtime; adding only the field silently fetches data nobody uses.
|
|
50
|
+
|
|
51
|
+
## Relationship to ACL embedding
|
|
52
|
+
|
|
53
|
+
This whitelist gate is **orthogonal to** the child-record ACL gate documented in
|
|
54
|
+
[nested-fk-acl-embedding.md](nested-fk-acl-embedding.md). Naming a reverse collection in `fields` is
|
|
55
|
+
necessary but not sufficient: each embedded child record is still re-checked against its own
|
|
56
|
+
`AclRecordPermissions`, and an ungranted child is silently dropped. A missing nested collection can be
|
|
57
|
+
caused by **either** an omitted `fields` entry **or** a missing child-record ACL grant — check both.
|
|
58
|
+
|
|
59
|
+
## Gotchas
|
|
60
|
+
|
|
61
|
+
- **Silent, not erroring.** An unlisted (or misspelled) reverse collection produces no `messages[]`
|
|
62
|
+
entry — the symptom is purely a missing key in `data`. Confirm the exact camelCase-plural name from
|
|
63
|
+
the child model before assuming a data problem.
|
|
64
|
+
- **The 1.0 bridge hit this concretely.** TRUE-79401 needed
|
|
65
|
+
`contactEmailAddresses.emailAddress` added to the `/contacts` fetch `fields` list in
|
|
66
|
+
`App_Api_Toga2` before the 1.0 sync could read a contact's full email list — see
|
|
67
|
+
[App_Api_Toga2 — TOGa2 API Client & bridge](../../../../1.0/apps/library/features/toga2-api-client-and-bridge.md).
|
|
68
|
+
|
|
69
|
+
## Change history
|
|
70
|
+
|
|
71
|
+
- 2026-07-28 — Captured the framework-level rule (split out of the TRUE-79401 library gotcha): V2
|
|
72
|
+
reverse hasMany collections are resolved via the child model's `FOREIGNKEY_MODEL`, exposed under the
|
|
73
|
+
camelCase plural of the child model, and are **only** serialized when explicitly named in the fetch
|
|
74
|
+
`fields` whitelist — otherwise the key is silently absent (200 / `isSuccess:true`, no message).
|
|
75
|
+
Orthogonal to, and stacks with, the child-record ACL embedding gate. (mhammontree)
|
|
76
|
+
</content>
|
|
77
|
+
</invoke>
|
package/knowledge/INDEX.md
CHANGED
|
@@ -19,7 +19,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
19
19
|
|
|
20
20
|
- **_underscore** (_Underscore) _(framework core)_ — 40 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
|
|
21
21
|
- **worker2** (Worker) — 34 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
|
|
22
|
-
- **api2** (API) —
|
|
22
|
+
- **api2** (API) — 19 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
|
|
23
23
|
- **dbchanges2** (Database Changes) _(framework core)_ — 3 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
|
|
24
24
|
- **toga2-supply** (TOGa Supply) — 3 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)
|
|
25
25
|
- **saml** (SAML SSO Gateway) — 3 doc(s) → [2.0/apps/saml/INDEX.md](2.0/apps/saml/INDEX.md)
|
|
@@ -5,11 +5,12 @@ apps:
|
|
|
5
5
|
- _underscore
|
|
6
6
|
- api2
|
|
7
7
|
- dbchanges2
|
|
8
|
+
- library
|
|
8
9
|
project: API
|
|
9
10
|
client: aig
|
|
10
11
|
type: profile
|
|
11
12
|
status: active
|
|
12
|
-
updated: 2026-
|
|
13
|
+
updated: 2026-07-28
|
|
13
14
|
owners: ["mhammontree"]
|
|
14
15
|
files: []
|
|
15
16
|
related:
|
|
@@ -35,6 +36,10 @@ go out under the Staples Protection Plan brand and link to `staplesprotection.to
|
|
|
35
36
|
NetSuite item/SO/invoice traits).
|
|
36
37
|
- **`dbchanges2`** — `Client_Aig/` schema + reference-data migrations (e.g. the SaleItem code
|
|
37
38
|
catalog in `Client_Aig.Items`).
|
|
39
|
+
- **`library` (1.0)** — the `App_Api_Toga2` bridge syncs 2.0 AIG contacts/entitlements down into the
|
|
40
|
+
legacy 1.0 `TOGA_AIG` DB (`Customers`/`ServiceRequests`), run by a worker cron every ~10 min. Note
|
|
41
|
+
the two AIG databases: **`TOGA_AIG`** = V1 legacy sync target; **`Client_Aig`** = V2 prod tenant. See
|
|
42
|
+
[App_Api_Toga2 bridge](../../1.0/apps/library/features/toga2-api-client-and-bridge.md).
|
|
38
43
|
|
|
39
44
|
See [entitlement-intake.md](features/entitlement-intake.md) for how a payload becomes an
|
|
40
45
|
entitlement and how to load new sale-item codes.
|
package/package.json
CHANGED