toga-ai 1.0.464 → 1.0.466

package/knowledge/1.0/apps/tools/features/talos-kb-documents-admin.md

@@ -6,7 +6,7 @@ project: Tools
 client: shared
 type: feature
 status: active
-updated: 2026-07-16
+updated: 2026-07-28
 owners: [jcardinal]
 files:
   - tools/mvc/talos/kb-documents/get.php
@@ -108,7 +108,7 @@ Vectors** (the canonical design — see the architecture note below), and the fl
 3. **S3 Vectors store** — create a dedicated vector bucket + index via
    `App_Talos_Bedrock::createVectorStore()`, capturing the authoritative **`indexArn`**.
 4. **VECTOR KB** — `createKnowledgeBase($bedrock, $slug, $indexArn)` (VECTOR + `S3_VECTORS`
-   storage).
+   storage), **tagged** with the mandatory `App_Talos_Bedrock::KB_TAGS` pair (see below).
 5. **wait for `ACTIVE`** (`waitForKnowledgeBaseActive`).
 6. **S3 data source** — plain `S3` connector + `BEDROCK_DATA_AUTOMATION` parsing.
 7. **`Team.KnowledgeBases`** (`db_team`, Team MySQL cluster).
@@ -163,8 +163,16 @@ orphaned KB).
   float32, euclidean), then **polls `getIndex` for the authoritative index ARN** with a short
   `NotFound` retry; returns an actionable error on `Conflict`/`AlreadyExists` from an orphaned
   prior attempt.
+- **`KB_TAGS`** — the **mandatory** tag pair stamped on every new KB:
+  `['tenant' => 'True', 'model_role' => 'knowledge_base']`. Passed as the top-level `'tags'`
+  parameter of `CreateKnowledgeBase`. Casing is load-bearing: capital-T **`True`**, snake_case
+  **`model_role`**. Verified against the live `development-team-toga-desk` KB (`ZOLCCQZZL1`) with
+  `aws bedrock-agent list-tags-for-resource --resource-arn
+  arn:aws:bedrock:us-east-1:654654170868:knowledge-base/ZOLCCQZZL1 --region us-east-1`. These tags
+  drive **cost allocation and model-inventory reporting** in the togaiq account — an untagged KB is
+  invisible to those reports.
 - `createKnowledgeBase($bedrock, string $slug, string $indexArn)` — creates the VECTOR KB with
-  `S3_VECTORS` storage (signature now takes the index ARN).
+  `S3_VECTORS` storage (signature takes the index ARN) and applies `KB_TAGS`.
 - `waitForKnowledgeBaseActive()` polls `getKnowledgeBase` every `WAIT_INTERVAL_SECONDS` (3s)
   up to **`WAIT_MAX_SECONDS` = 300**. Exits immediately on `ACTIVE`/`FAILED`. 300s sits well
   under the 900s Apache/PHP ceiling.
@@ -191,6 +199,12 @@ orphaned KB).
   `iam:PassRole` on the shared role. Its bedrock create actions were already covered by
   `AmazonBedrockFullAccess` and S3 by `AmazonS3FullAccess`. **`s3vectors` is a SEPARATE service
   NOT covered by `AmazonBedrockFullAccess`** — that was the real missing grant.
+- **⚠ `bedrock:TagResource` (new, NOT yet verified as granted).** AWS authorizes
+  **tagging-on-create separately from the create action itself**. Now that `createKnowledgeBase()`
+  sends `tags`, the `bedrock-model-accessor` user must also hold **`bedrock:TagResource`** on the
+  knowledge-base resource, or `CreateKnowledgeBase` fails with `AccessDeniedException` — even
+  though the *identical call without tags* would have succeeded. Confirm this grant with devops
+  **before the next real KB creation** (see the orphaned-vector-bucket gotcha).
 
 ## Helpers
 
@@ -277,6 +291,17 @@ orphaned KB).
   `Conflict`/`AlreadyExists` on the vector bucket (surfaced as an actionable error) or creates a
   **duplicate** Bedrock KB (Bedrock allows same-name KBs with new ids). After a failed create:
   delete the orphaned vector store + KB and re-run clean, or finish the DB rows manually.
+- **Tagging on create needs its own IAM permission, and failing it orphans a vector bucket.**
+  `bedrock:TagResource` is authorized separately from `CreateKnowledgeBase`. Because the KB is
+  created at **step 4 — after** the S3 Vectors bucket + index (step 3) — a missing `TagResource`
+  grant aborts the flow leaving an **orphaned `bedrock-knowledge-base-{slug}` vector bucket with
+  no KB and no DB rows**, and retrying the same slug then hits `Conflict`/`AlreadyExists`. If a
+  create suddenly starts failing `AccessDeniedException` right after the vector store succeeds,
+  suspect this permission first — not the tag values.
+- **Only the Bedrock KB is tagged — the per-KB S3 Vectors bucket is deliberately NOT.** Scope
+  decision (2026-07-28): `createVectorStore()` leaves `bedrock-knowledge-base-{slug}` untagged.
+  Known gap: the vector bucket is a **separate billable resource**, so vector-storage cost is not
+  attributed by `tenant`/`model_role`. Revisit if cost reporting needs it.
 - **Two prior KBs use the WRONG (fully-MANAGED) type.** `development-team-craftex` and
   `development-team-toga-desk` were created by the earlier MANAGED code and were **intentionally
   left as-is** (this session's scope was code-only). They should be **recreated later** to match
@@ -296,6 +321,14 @@ orphaned KB).
 
 ## Change history
 
+- 2026-07-28 — **Every new Bedrock KB is now tagged** `tenant=True` + `model_role=knowledge_base`
+  via the new `App_Talos_Bedrock::KB_TAGS` constant, passed as the `tags` parameter of
+  `CreateKnowledgeBase` (required for togaiq cost allocation / model-inventory reporting; values
+  and casing verified against the live `toga-desk` KB `ZOLCCQZZL1`). **New IAM prerequisite:**
+  tagging-on-create needs `bedrock:TagResource` on the `bedrock-model-accessor` user — *not yet
+  verified as granted*; without it `CreateKnowledgeBase` returns `AccessDeniedException` and, with
+  no rollback, orphans the already-created S3 Vectors bucket. Scoped decision: the per-KB S3
+  Vectors bucket is left **untagged**, so vector-storage cost is not attributed. (jcardinal)
 - 2026-07-16 — **Corrected the KB-create architecture from MANAGED to VECTOR + S3 Vectors**
   (the canonical design, from `development-team-dunn`; the earlier code was modeled on the WRONG
   `craftex`/MANAGED KB). Rewrote `App_Talos_Bedrock`: KB is now `VECTOR` + `S3_VECTORS` storage;

package/knowledge/2.0/apps/api2/architecture.md

@@ -239,6 +239,18 @@ they are the known sharp edges. Do not re-discover these from scratch.
    [worker2 does](../worker2/features/alb-target-group-auto-registration.md). The same script's
    IMDSv2 handling (token on the `curl` command line, silent IMDSv1 fallback) must be fixed with it.
    See the Security note above.
+10. **Cross-client page-number paging is O(page), and the deepen chunk size is an untuned
+    performance dial.** Serving a deep page of a
+    [cross-client listing](features/cross-client-data-retrieval.md) requires materializing every
+    preceding row into the Cache cluster — page 4430 means 110,747 rows. Adaptive chunking
+    (`DEEPEN_CHUNK_MAX_RECORDS = 500`) makes it cheaper, not cheap. The durable fix is exposing
+    **keyset cursors to the caller** so cost is independent of depth; V2 already supports
+    `seekValues` + `seekUuid` correctly, so the mechanism exists but the caller-facing contract is
+    **unbuilt and not yet scoped**. Related dial: chunk size trades round trips against
+    sub-response size — a narrowed `fields` request stays small at any chunk, but an unrestricted
+    listing expands every row through `getFullModelData()` at the requested depth, which is the case
+    where 500 may need lowering. Watch it against `CURL_TIMEOUT_SECONDS = 30`. Do not raise the
+    chunk constant without checking sub-response size for wide, deep listings.
 
 ## When making changes here
 
@@ -254,8 +266,13 @@ they are the known sharp edges. Do not re-discover these from scratch.
   the duplicate check.
 - Preserve the commit-logs / rollback-data-on-failure invariant when editing the controller
   or `execute()`.
+- **A code path with exactly one caller gets zero incidental coverage.** V2's keyset-pagination
+  branch emitted syntactically invalid SQL for an entire development cycle because only the
+  cross-client deepen path ever sets `$isKeysetMode`. In a ~2,000-line untested monolith, treat
+  single-caller branches in `V2.php` as unverified until exercised directly.
 
 ## Change history
+- 2026-07-28 — Added Known issue #10: cross-client page-number paging is inherently O(page) (deep pages materialize every preceding row into the Cache cluster), with caller-facing keyset cursors identified as the durable fix but left unbuilt/unscoped, plus the `DEEPEN_CHUNK_MAX_RECORDS = 500` tuning tradeoff against `getFullModelData()` expansion and `CURL_TIMEOUT_SECONDS = 30`. Added a change-guidance bullet that single-caller branches in the untested `V2.php` monolith get zero incidental coverage (the keyset `LIMIT` syntax error shipped invisibly for a full cycle). (jcardinal)
 - 2026-07-28 — Documented the previously unrecorded `060_register_instance_to_shared_application_load_balancer` postdeploy hook pair in the Deployment section (non-prod self-registration into the same-named ALB target group; production skipped), and recorded a **committed IAM access key** in `ebs/register_instance_to_shared_application_load_balancer.php` as a security note + Known issue #9 — location, line range, commit subject, and remediation only (rotate, audit CloudTrail, move to the instance profile; history rewrite is a separate sign-off). Flagged api2's copy as the unhardened original vs. the new worker2 reference implementation. (jcardinal)
 - 2026-07-28 — Added a consolidated **Known issues / accepted risks** section (8 items), absorbing the previously free-floating deferred raw-exception-disclosure follow-up as item 1, so the tier's sharp edges (unrotated committed secrets, pre-execute phase still outside the main guard, local Logs DB name mismatch, permissive CORS, unpinned `_underscore` build clone, untested `V2.php` monolith, JWT rotation overlap window) are in one place instead of scattered. Recorded that `DB_CACHE` is resolved by name (`Databases.name = 'Cache'`), never by a hardcoded id, which differs per Core instance. (jcardinal)
 - 2026-07-28 — Added gotcha: the request-logger's auto-generated `Api.transactionId` (millisecond timestamp `Y-m-d H:i:s.v`, UNIQUE) collides under concurrent same-millisecond nested writes → MySQL 1062 → HTTP 500; platform-wide, observed on the Compass/Veyer ASN feed (`sourceIp 34.232.23.158`). Distinct from the client-supplied `transactionId`/EV-5 uniqueness contract. Fix direction: uuid the logged id or retry-on-1062. (bala)

package/knowledge/2.0/apps/api2/features/cross-client-data-retrieval.md

@@ -42,10 +42,11 @@ is a **scatter-gather** engine: it fans out to each entitled client's own V2 API
 streams with a watermark, and caches the merged result per-user/per-query on a dedicated Cache
 cluster.
 
-> **Status: working end to end on local dev** (multi-client fan-out, ACL-filtered `client` object,
-> typed multi-column sort, keyset paging, narrowed `fields`, per-query cache isolation).
-> **Not yet deployed** — next stop is external testing. See
-> [api2 architecture](../architecture.md) for the accepted risks carried into that test.
+> **Status: working end to end on local dev AND on a real deployed environment** — first
+> deployed test 2026-07-28 on the **dev sandbox** (`api.beta.togahub.com`, dev-sandbox cluster).
+> That test exposed a page-2-and-beyond failure whose root cause and four follow-on defects are
+> now fixed and verified (see *Deep paging*, *Gotchas* and the change history). See
+> [api2 architecture](../architecture.md) for the accepted risks carried alongside it.
 
 ## How it works
 
@@ -107,6 +108,73 @@ deepening and returned empty pages.
 the watermark cannot make any row safe, so querying it is a wasted round trip. At 1000 clients that
 is ~1 sub-request per pass instead of 1000 — the single biggest efficiency property of the design.
 
+**Worked example (the canonical shape to reason about).** Sorting `-dateOrder` DESC across Compass
++ NYCHH: Compass has 170 rows sharing `dateOrder = 2026-07-27` while NYCHH's *newest* row is
+`2026-06-18`. Compass therefore holds the watermark for many consecutive pages, and NYCHH must
+**not** be re-queried until Compass's cursor passes `2026-06-18`. Verified against live data: after
+the 25-row page-1 cursor, 145 of the 170 same-date rows remain (25 + 145 = 170), i.e. the
+multi-column seek predicate is correct across a large tie block.
+
+### The merge never serves past the watermark (`isDepthUnproven`)
+
+`servePage()` used to do a blind `OFFSET`/`LIMIT` over the result table. When deepening **could not**
+cover the requested page — a client failed, a cursor stalled, or `MAX_DEEPEN_ITERATIONS` ran out —
+it still served a full page of rows that a lagging client may legitimately displace, i.e. it
+presented a mis-ordered page as authoritative. One failed sub-request was enough to return 25 wrong
+rows.
+
+The page is now **clamped to `Tables.safeRecordCount`**, and any shortfall raises the new public
+`$isDepthUnproven`, which V2 turns into a **WARNING** message telling the caller to retry to
+continue deepening. Two new meta fields: `meta.crossClient.safeRecordCount` and
+`meta.crossClient.isDepthUnproven`. `needsDeepening()` and the flag are deduped onto one new
+`hasUnexhaustedClients()` helper.
+
+**Subtlety — the flag is only raised while `hasUnexhaustedClients()` is true.** Once every client is
+exhausted, `safeRecordCount` *is* the true end of the result set, so a short final page is correct
+and must **not** warn.
+
+### Adaptive deepen chunking (sub-request page size ≠ caller's page size)
+
+Because only the watermark client is deepened per pass, and each pass used to fetch the **caller's**
+`recordsPerPage`, reaching page N cost ~N sequential sub-requests — and `MAX_DEEPEN_ITERATIONS = 50`
+capped one HTTP request at ~1250 new rows. Requesting page 254 returned an *empty* page plus the
+new warning.
+
+The internal fetch size is now decoupled from the caller's:
+
+```
+chunkSize = min(DEEPEN_CHUNK_MAX_RECORDS /* 500 */,
+                max(callerRecordsPerPage, target - safeRowCount()))
+```
+
+Page 1 of 25 still fetches exactly 25; page 254 fetches 500 per pass — ~13 round trips instead of
+254. `fanOut()` takes `chunkSize` instead of `recordsPerPage`.
+
+The caller's `recordsPerPage` still governs `servePage()`'s `LIMIT`, `needsDeepening()`'s target,
+`reportedCounts()`, and the **cache identity** (`Tables.recordsPerPage` + query hash). Only the
+internal fetch size changed — **no schema change**.
+
+**Chunk-size tuning.** Chunk size trades round trips against sub-response size. A narrowed `fields`
+request stays small at any chunk; an *unrestricted* listing expands every row through
+`getFullModelData()` at the requested depth, and that is the case where 500 may need lowering.
+Watch it against `CURL_TIMEOUT_SECONDS = 30`.
+
+### Only a cursor-less sub-response may set a client's total
+
+Once the seek predicate is in the `WHERE` clause, V2's found-rows count is the **remainder after the
+cursor**, not the client's total. `collectRows()` therefore tracks an `$isFirstFetch` map and only a
+usable, cursor-less (offset-mode) sub-response establishes `totalRecordCount`; later deepen passes
+leave the recorded total standing. Without this guard, fixing the keyset `LIMIT` bug would have made
+reported totals silently **shrink** on every deepen pass.
+
+### Diagnosing a fan-out failure from the client logs
+
+In `Logs_<Client>.Api`, a sub-request that logged its `POST /v2/auth/encrypted-user-uuid` (201) but
+has **no corresponding GET row** proves the data sub-request died **before** the logging step (a
+fatal) — which is what distinguishes it from a clean 0-row response. That single observation
+isolated the keyset `LIMIT` root cause. Cross-check the *other* client's log too: the **absence** of
+any row there confirms watermark-only deepening queried only the intended client.
+
 ### Per-row `client` object (ACL-filtered)
 
 Every returned row carries a `client` object filtered by the caller's **CORE** roles
@@ -157,12 +225,45 @@ cron.
   in client queries — because the PHP-side watermark merge compares bytes. Any linguistic collation
   would order differently and the merge could certify rows complete that MySQL then re-orders.
 - **Never use the raw stored row count** to decide paging depth; only `safeRecordCount`.
+- **Never serve rows past `safeRecordCount`.** A short page + `isDepthUnproven` + a WARNING is
+  correct; a full page of possibly-displaceable rows is not.
+- **A failed client is UNKNOWN, never exhausted.** Only a client that provably ran out of rows may
+  be marked `exhausted` — the watermark math treats exhausted clients as having no remainder.
+- **Only a cursor-less sub-response may set a client's `totalRecordCount`** — a seek-filtered
+  response counts the remainder, not the total.
+- **Anything compared against a sub-request's page size must compare against `chunkSize`**, not the
+  caller's `recordsPerPage`, now that the two differ.
 - **Every outbound sub-request needs its own globally-unique `transactionId`.**
 - **Forward the caller's original query string verbatim** — parsed `$httpOptions` do not round-trip
   (see the url-decode gotcha below). `MAX_CONCURRENT_CLIENT_FETCHES = 25` bounds fan-out.
 
 ## Gotchas
 
+- **V2's keyset `LIMIT` had no leading newline — every seek sub-request was a SQL syntax error.**
+  In `processRoutePairs()` (`V2.php`, keyset branch, ~line 4113) the keyset path did
+  `$sql .= ('LIMIT ' . $recordsPerPage)` while the OFFSET path had a leading newline. The `ORDER BY`
+  immediately above ends in a bare `ASC`, so the generated SQL read
+  `` CAST(`uuid` AS BINARY) ASCLIMIT 25 ``. It hid for a whole development cycle because **only the
+  cross-client deepen path ever sets `$isKeysetMode`** — page 1 uses OFFSET and nothing else in api2
+  exercises keyset mode. This was the root cause of the entire "page 2+ returns wrong results" report.
+  *Lesson: a code path with exactly one caller gets zero incidental coverage.*
+- **`emptyClientResult()` returning `exhausted => true` turned a transient error into a permanently
+  wrong cached answer.** `safeRowCount()` skips exhausted clients when computing the watermark, so a
+  failed client dropped out of the watermark → every already-cached row looked provably safe →
+  `safeRecordCount` was committed too high → `needsDeepening()` returned false for the **life of the
+  cache entry**, permanently excluding that client from every future deepen pass. It is now `false`
+  (unknown remainder). No runaway risk: the existing `$newRowCount === 0` stall guard still breaks
+  the loop.
+- **`persistCursors()` erased the reported totals of every client it did not fetch this pass.** It
+  `DELETE`s all `Tables_Clients` rows then re-`INSERT`s with `$counts[$clientId] ?? null`, but a
+  deepen pass only populates `$counts` for the single watermark-holding client. Every other client's
+  `totalRecordCount`/`prohibitedRecordCount` became NULL → `meta.totalRecordCount` and
+  `totalPageCount` went to 0 → and because `nextPage` derives from `totalPageCount`, `nextPage`
+  became `null`, dead-ending any caller that pages by following `nextPage`. Fix: `SELECT` the
+  existing counts before the `DELETE` and carry them forward **per field**.
+- **The exhaustion test had to move to `chunkSize` in the same commit as adaptive chunking.**
+  Comparing a full 500-row block against a caller page size of 25 declares the client "exhausted" and
+  silently reintroduces the `emptyClientResult` class of bug above.
 - **V2 NEVER url-decodes its query string.** It splits `QUERY_STRING` on `&` and `=` and consumes
   the raw values. Anything containing `,`, `[`, `"`, `%`, `+` or a space arrives mangled — this bit
   the feature twice (the JSON `seekValues` array, and a comma-separated `fields` list corrupted into
@@ -200,7 +301,35 @@ cron.
 - **`curl_multi` busy-spin guard** — both multi loops `usleep(100)` when
   `curl_multi_select() === -1`.
 
+## Known limitation — deep page-number paging is inherently O(page)
+
+Serving page 4430 means materializing all 110,747 preceding rows into the cache. Adaptive chunking
+makes that cheaper, not cheap: cost still grows with depth, because a page **number** cannot be
+resolved without knowing everything before it.
+
+The durable answer is **exposing keyset cursors to the caller** so cost is independent of depth — the
+mechanism already exists (V2 now supports `seekValues` + `seekUuid` correctly). This was raised with
+the developer on 2026-07-28 and **deliberately left unbuilt / not yet scoped**: record it as a known
+limitation and a candidate direction, not as pending work.
+
 ## Change history
+- 2026-07-28 — **First test on a real deployed environment** (dev sandbox,
+  `api.beta.togahub.com`), which reported wrong results on page 2+. Root cause: V2's keyset `LIMIT`
+  was concatenated without a leading newline, so **every** seek sub-request was a SQL syntax error —
+  invisible until now because only the cross-client deepen path sets `$isKeysetMode`. Fixed, plus the
+  four defects it masked: `emptyClientResult()` marking a *failed* client `exhausted` (which
+  permanently over-committed `safeRecordCount` and excluded that client from all future deepening);
+  `persistCursors()` NULL-ing the totals of every client not fetched in the pass (killing
+  `totalRecordCount`/`totalPageCount`/`nextPage`); totals being taken from cursor-filtered
+  sub-responses (now only a cursor-less first fetch sets a client's total); and `servePage()` serving
+  blind `OFFSET`/`LIMIT` past the watermark (now clamped to `safeRecordCount`, with a new public
+  `$isDepthUnproven` → V2 WARNING, `hasUnexhaustedClients()` helper, and new
+  `meta.crossClient.safeRecordCount` / `isDepthUnproven`). Added **adaptive deepen chunking**
+  (`DEEPEN_CHUNK_MAX_RECORDS = 500`; per-pass `chunkSize`, `fanOut()` and the exhaustion test both
+  switched to it) so deep pages cost ~page/500 round trips instead of ~page — no schema change, cache
+  identity unchanged. Recorded the Compass/NYCHH tie-block worked example, the log-based fan-out
+  failure diagnostic, and deep page-number paging being O(page) as a known limitation (caller-facing
+  keyset cursors left unbuilt). Verified working on dev sandbox. (jcardinal)
 - 2026-07-28 — Took the feature from never-executed to working end to end. Row identity moved from
   the numeric id to `rowUuid`/`seekUuid` (a V2 LIST never returns `id`); per-query
   `TableResults_<id>` tables with typed, direction-indexed sort columns; multi-column `seekValues`

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.464",
+  "version": "1.0.466",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",