toga-ai 1.0.464 → 1.0.465

package/knowledge/2.0/apps/api2/architecture.md

@@ -239,6 +239,18 @@ they are the known sharp edges. Do not re-discover these from scratch.
    [worker2 does](../worker2/features/alb-target-group-auto-registration.md). The same script's
    IMDSv2 handling (token on the `curl` command line, silent IMDSv1 fallback) must be fixed with it.
    See the Security note above.
+10. **Cross-client page-number paging is O(page), and the deepen chunk size is an untuned
+    performance dial.** Serving a deep page of a
+    [cross-client listing](features/cross-client-data-retrieval.md) requires materializing every
+    preceding row into the Cache cluster — page 4430 means 110,747 rows. Adaptive chunking
+    (`DEEPEN_CHUNK_MAX_RECORDS = 500`) makes it cheaper, not cheap. The durable fix is exposing
+    **keyset cursors to the caller** so cost is independent of depth; V2 already supports
+    `seekValues` + `seekUuid` correctly, so the mechanism exists but the caller-facing contract is
+    **unbuilt and not yet scoped**. Related dial: chunk size trades round trips against
+    sub-response size — a narrowed `fields` request stays small at any chunk, but an unrestricted
+    listing expands every row through `getFullModelData()` at the requested depth, which is the case
+    where 500 may need lowering. Watch it against `CURL_TIMEOUT_SECONDS = 30`. Do not raise the
+    chunk constant without checking sub-response size for wide, deep listings.
 
 ## When making changes here
 
@@ -254,8 +266,13 @@ they are the known sharp edges. Do not re-discover these from scratch.
   the duplicate check.
 - Preserve the commit-logs / rollback-data-on-failure invariant when editing the controller
   or `execute()`.
+- **A code path with exactly one caller gets zero incidental coverage.** V2's keyset-pagination
+  branch emitted syntactically invalid SQL for an entire development cycle because only the
+  cross-client deepen path ever sets `$isKeysetMode`. In a ~2,000-line untested monolith, treat
+  single-caller branches in `V2.php` as unverified until exercised directly.
 
 ## Change history
+- 2026-07-28 — Added Known issue #10: cross-client page-number paging is inherently O(page) (deep pages materialize every preceding row into the Cache cluster), with caller-facing keyset cursors identified as the durable fix but left unbuilt/unscoped, plus the `DEEPEN_CHUNK_MAX_RECORDS = 500` tuning tradeoff against `getFullModelData()` expansion and `CURL_TIMEOUT_SECONDS = 30`. Added a change-guidance bullet that single-caller branches in the untested `V2.php` monolith get zero incidental coverage (the keyset `LIMIT` syntax error shipped invisibly for a full cycle). (jcardinal)
 - 2026-07-28 — Documented the previously unrecorded `060_register_instance_to_shared_application_load_balancer` postdeploy hook pair in the Deployment section (non-prod self-registration into the same-named ALB target group; production skipped), and recorded a **committed IAM access key** in `ebs/register_instance_to_shared_application_load_balancer.php` as a security note + Known issue #9 — location, line range, commit subject, and remediation only (rotate, audit CloudTrail, move to the instance profile; history rewrite is a separate sign-off). Flagged api2's copy as the unhardened original vs. the new worker2 reference implementation. (jcardinal)
 - 2026-07-28 — Added a consolidated **Known issues / accepted risks** section (8 items), absorbing the previously free-floating deferred raw-exception-disclosure follow-up as item 1, so the tier's sharp edges (unrotated committed secrets, pre-execute phase still outside the main guard, local Logs DB name mismatch, permissive CORS, unpinned `_underscore` build clone, untested `V2.php` monolith, JWT rotation overlap window) are in one place instead of scattered. Recorded that `DB_CACHE` is resolved by name (`Databases.name = 'Cache'`), never by a hardcoded id, which differs per Core instance. (jcardinal)
 - 2026-07-28 — Added gotcha: the request-logger's auto-generated `Api.transactionId` (millisecond timestamp `Y-m-d H:i:s.v`, UNIQUE) collides under concurrent same-millisecond nested writes → MySQL 1062 → HTTP 500; platform-wide, observed on the Compass/Veyer ASN feed (`sourceIp 34.232.23.158`). Distinct from the client-supplied `transactionId`/EV-5 uniqueness contract. Fix direction: uuid the logged id or retry-on-1062. (bala)

package/knowledge/2.0/apps/api2/features/cross-client-data-retrieval.md

@@ -42,10 +42,11 @@ is a **scatter-gather** engine: it fans out to each entitled client's own V2 API
 streams with a watermark, and caches the merged result per-user/per-query on a dedicated Cache
 cluster.
 
-> **Status: working end to end on local dev** (multi-client fan-out, ACL-filtered `client` object,
-> typed multi-column sort, keyset paging, narrowed `fields`, per-query cache isolation).
-> **Not yet deployed** — next stop is external testing. See
-> [api2 architecture](../architecture.md) for the accepted risks carried into that test.
+> **Status: working end to end on local dev AND on a real deployed environment** — first
+> deployed test 2026-07-28 on the **dev sandbox** (`api.beta.togahub.com`, dev-sandbox cluster).
+> That test exposed a page-2-and-beyond failure whose root cause and four follow-on defects are
+> now fixed and verified (see *Deep paging*, *Gotchas* and the change history). See
+> [api2 architecture](../architecture.md) for the accepted risks carried alongside it.
 
 ## How it works
 
@@ -107,6 +108,73 @@ deepening and returned empty pages.
 the watermark cannot make any row safe, so querying it is a wasted round trip. At 1000 clients that
 is ~1 sub-request per pass instead of 1000 — the single biggest efficiency property of the design.
 
+**Worked example (the canonical shape to reason about).** Sorting `-dateOrder` DESC across Compass
++ NYCHH: Compass has 170 rows sharing `dateOrder = 2026-07-27` while NYCHH's *newest* row is
+`2026-06-18`. Compass therefore holds the watermark for many consecutive pages, and NYCHH must
+**not** be re-queried until Compass's cursor passes `2026-06-18`. Verified against live data: after
+the 25-row page-1 cursor, 145 of the 170 same-date rows remain (25 + 145 = 170), i.e. the
+multi-column seek predicate is correct across a large tie block.
+
+### The merge never serves past the watermark (`isDepthUnproven`)
+
+`servePage()` used to do a blind `OFFSET`/`LIMIT` over the result table. When deepening **could not**
+cover the requested page — a client failed, a cursor stalled, or `MAX_DEEPEN_ITERATIONS` ran out —
+it still served a full page of rows that a lagging client may legitimately displace, i.e. it
+presented a mis-ordered page as authoritative. One failed sub-request was enough to return 25 wrong
+rows.
+
+The page is now **clamped to `Tables.safeRecordCount`**, and any shortfall raises the new public
+`$isDepthUnproven`, which V2 turns into a **WARNING** message telling the caller to retry to
+continue deepening. Two new meta fields: `meta.crossClient.safeRecordCount` and
+`meta.crossClient.isDepthUnproven`. `needsDeepening()` and the flag are deduped onto one new
+`hasUnexhaustedClients()` helper.
+
+**Subtlety — the flag is only raised while `hasUnexhaustedClients()` is true.** Once every client is
+exhausted, `safeRecordCount` *is* the true end of the result set, so a short final page is correct
+and must **not** warn.
+
+### Adaptive deepen chunking (sub-request page size ≠ caller's page size)
+
+Because only the watermark client is deepened per pass, and each pass used to fetch the **caller's**
+`recordsPerPage`, reaching page N cost ~N sequential sub-requests — and `MAX_DEEPEN_ITERATIONS = 50`
+capped one HTTP request at ~1250 new rows. Requesting page 254 returned an *empty* page plus the
+new warning.
+
+The internal fetch size is now decoupled from the caller's:
+
+```
+chunkSize = min(DEEPEN_CHUNK_MAX_RECORDS /* 500 */,
+                max(callerRecordsPerPage, target - safeRowCount()))
+```
+
+Page 1 of 25 still fetches exactly 25; page 254 fetches 500 per pass — ~13 round trips instead of
+254. `fanOut()` takes `chunkSize` instead of `recordsPerPage`.
+
+The caller's `recordsPerPage` still governs `servePage()`'s `LIMIT`, `needsDeepening()`'s target,
+`reportedCounts()`, and the **cache identity** (`Tables.recordsPerPage` + query hash). Only the
+internal fetch size changed — **no schema change**.
+
+**Chunk-size tuning.** Chunk size trades round trips against sub-response size. A narrowed `fields`
+request stays small at any chunk; an *unrestricted* listing expands every row through
+`getFullModelData()` at the requested depth, and that is the case where 500 may need lowering.
+Watch it against `CURL_TIMEOUT_SECONDS = 30`.
+
+### Only a cursor-less sub-response may set a client's total
+
+Once the seek predicate is in the `WHERE` clause, V2's found-rows count is the **remainder after the
+cursor**, not the client's total. `collectRows()` therefore tracks an `$isFirstFetch` map and only a
+usable, cursor-less (offset-mode) sub-response establishes `totalRecordCount`; later deepen passes
+leave the recorded total standing. Without this guard, fixing the keyset `LIMIT` bug would have made
+reported totals silently **shrink** on every deepen pass.
+
+### Diagnosing a fan-out failure from the client logs
+
+In `Logs_<Client>.Api`, a sub-request that logged its `POST /v2/auth/encrypted-user-uuid` (201) but
+has **no corresponding GET row** proves the data sub-request died **before** the logging step (a
+fatal) — which is what distinguishes it from a clean 0-row response. That single observation
+isolated the keyset `LIMIT` root cause. Cross-check the *other* client's log too: the **absence** of
+any row there confirms watermark-only deepening queried only the intended client.
+
 ### Per-row `client` object (ACL-filtered)
 
 Every returned row carries a `client` object filtered by the caller's **CORE** roles
@@ -157,12 +225,45 @@ cron.
   in client queries — because the PHP-side watermark merge compares bytes. Any linguistic collation
   would order differently and the merge could certify rows complete that MySQL then re-orders.
 - **Never use the raw stored row count** to decide paging depth; only `safeRecordCount`.
+- **Never serve rows past `safeRecordCount`.** A short page + `isDepthUnproven` + a WARNING is
+  correct; a full page of possibly-displaceable rows is not.
+- **A failed client is UNKNOWN, never exhausted.** Only a client that provably ran out of rows may
+  be marked `exhausted` — the watermark math treats exhausted clients as having no remainder.
+- **Only a cursor-less sub-response may set a client's `totalRecordCount`** — a seek-filtered
+  response counts the remainder, not the total.
+- **Anything compared against a sub-request's page size must compare against `chunkSize`**, not the
+  caller's `recordsPerPage`, now that the two differ.
 - **Every outbound sub-request needs its own globally-unique `transactionId`.**
 - **Forward the caller's original query string verbatim** — parsed `$httpOptions` do not round-trip
   (see the url-decode gotcha below). `MAX_CONCURRENT_CLIENT_FETCHES = 25` bounds fan-out.
 
 ## Gotchas
 
+- **V2's keyset `LIMIT` had no leading newline — every seek sub-request was a SQL syntax error.**
+  In `processRoutePairs()` (`V2.php`, keyset branch, ~line 4113) the keyset path did
+  `$sql .= ('LIMIT ' . $recordsPerPage)` while the OFFSET path had a leading newline. The `ORDER BY`
+  immediately above ends in a bare `ASC`, so the generated SQL read
+  `` CAST(`uuid` AS BINARY) ASCLIMIT 25 ``. It hid for a whole development cycle because **only the
+  cross-client deepen path ever sets `$isKeysetMode`** — page 1 uses OFFSET and nothing else in api2
+  exercises keyset mode. This was the root cause of the entire "page 2+ returns wrong results" report.
+  *Lesson: a code path with exactly one caller gets zero incidental coverage.*
+- **`emptyClientResult()` returning `exhausted => true` turned a transient error into a permanently
+  wrong cached answer.** `safeRowCount()` skips exhausted clients when computing the watermark, so a
+  failed client dropped out of the watermark → every already-cached row looked provably safe →
+  `safeRecordCount` was committed too high → `needsDeepening()` returned false for the **life of the
+  cache entry**, permanently excluding that client from every future deepen pass. It is now `false`
+  (unknown remainder). No runaway risk: the existing `$newRowCount === 0` stall guard still breaks
+  the loop.
+- **`persistCursors()` erased the reported totals of every client it did not fetch this pass.** It
+  `DELETE`s all `Tables_Clients` rows then re-`INSERT`s with `$counts[$clientId] ?? null`, but a
+  deepen pass only populates `$counts` for the single watermark-holding client. Every other client's
+  `totalRecordCount`/`prohibitedRecordCount` became NULL → `meta.totalRecordCount` and
+  `totalPageCount` went to 0 → and because `nextPage` derives from `totalPageCount`, `nextPage`
+  became `null`, dead-ending any caller that pages by following `nextPage`. Fix: `SELECT` the
+  existing counts before the `DELETE` and carry them forward **per field**.
+- **The exhaustion test had to move to `chunkSize` in the same commit as adaptive chunking.**
+  Comparing a full 500-row block against a caller page size of 25 declares the client "exhausted" and
+  silently reintroduces the `emptyClientResult` class of bug above.
 - **V2 NEVER url-decodes its query string.** It splits `QUERY_STRING` on `&` and `=` and consumes
   the raw values. Anything containing `,`, `[`, `"`, `%`, `+` or a space arrives mangled — this bit
   the feature twice (the JSON `seekValues` array, and a comma-separated `fields` list corrupted into
@@ -200,7 +301,35 @@ cron.
 - **`curl_multi` busy-spin guard** — both multi loops `usleep(100)` when
   `curl_multi_select() === -1`.
 
+## Known limitation — deep page-number paging is inherently O(page)
+
+Serving page 4430 means materializing all 110,747 preceding rows into the cache. Adaptive chunking
+makes that cheaper, not cheap: cost still grows with depth, because a page **number** cannot be
+resolved without knowing everything before it.
+
+The durable answer is **exposing keyset cursors to the caller** so cost is independent of depth — the
+mechanism already exists (V2 now supports `seekValues` + `seekUuid` correctly). This was raised with
+the developer on 2026-07-28 and **deliberately left unbuilt / not yet scoped**: record it as a known
+limitation and a candidate direction, not as pending work.
+
 ## Change history
+- 2026-07-28 — **First test on a real deployed environment** (dev sandbox,
+  `api.beta.togahub.com`), which reported wrong results on page 2+. Root cause: V2's keyset `LIMIT`
+  was concatenated without a leading newline, so **every** seek sub-request was a SQL syntax error —
+  invisible until now because only the cross-client deepen path sets `$isKeysetMode`. Fixed, plus the
+  four defects it masked: `emptyClientResult()` marking a *failed* client `exhausted` (which
+  permanently over-committed `safeRecordCount` and excluded that client from all future deepening);
+  `persistCursors()` NULL-ing the totals of every client not fetched in the pass (killing
+  `totalRecordCount`/`totalPageCount`/`nextPage`); totals being taken from cursor-filtered
+  sub-responses (now only a cursor-less first fetch sets a client's total); and `servePage()` serving
+  blind `OFFSET`/`LIMIT` past the watermark (now clamped to `safeRecordCount`, with a new public
+  `$isDepthUnproven` → V2 WARNING, `hasUnexhaustedClients()` helper, and new
+  `meta.crossClient.safeRecordCount` / `isDepthUnproven`). Added **adaptive deepen chunking**
+  (`DEEPEN_CHUNK_MAX_RECORDS = 500`; per-pass `chunkSize`, `fanOut()` and the exhaustion test both
+  switched to it) so deep pages cost ~page/500 round trips instead of ~page — no schema change, cache
+  identity unchanged. Recorded the Compass/NYCHH tie-block worked example, the log-based fan-out
+  failure diagnostic, and deep page-number paging being O(page) as a known limitation (caller-facing
+  keyset cursors left unbuilt). Verified working on dev sandbox. (jcardinal)
 - 2026-07-28 — Took the feature from never-executed to working end to end. Row identity moved from
   the numeric id to `rowUuid`/`seekUuid` (a V2 LIST never returns `id`); per-query
   `TableResults_<id>` tables with typed, direction-indexed sort columns; multi-column `seekValues`

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.464",
+  "version": "1.0.465",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",