toga-ai 1.0.354 → 1.0.355

package/knowledge/1.0/apps/tools/features/talos-kb-documents-admin.md

@@ -6,7 +6,7 @@ project: Tools
 client: shared
 type: feature
 status: active
-updated: 2026-07-13
+updated: 2026-07-16
 owners: [jcardinal]
 files:
   - tools/mvc/talos/kb-documents/get.php
@@ -97,36 +97,100 @@ false success.
 ### `POST /talos/knowledge-bases` — create a new KB (synchronous provisioning)
 
 Admin action that provisions a brand-new Talos knowledge base end to end in **one
-synchronous request**: creates the S3 prefix, creates the Bedrock KB and **waits for it to
-reach `ACTIVE`**, creates the Bedrock data source (`MANAGED_KNOWLEDGE_BASE_CONNECTOR`), and
-inserts rows across **5 systems with no shared transaction**: `Team.KnowledgeBases` (`db_team`,
-Team MySQL cluster), `Client_True.VectorIndexes` (`db_true`, **Client MySQL cluster** — not the
-Team cluster), and 2 Talos **Postgres** rows (`public.knowledge_bases` +
-`public.assistant_knowledge_bases`, the latter linking a hard-coded assistant id). Step order:
-validate name (≤255) + derive slug via **`App_Talos_S3::slugify()`** (kebab-case, `iconv` ASCII
-translit; rejects a duplicate slug — `slug` is UNIQUE); create 3 S3 folder markers
-`development-team/{slug}/{approved,archive,upload}/` via **`App_Talos_S3::putFolderMarker()`**;
-create the Bedrock KB and wait for `ACTIVE`; create the data source; then the DB inserts. This flow
-**stays synchronous by design** — moving it to worker2/SQS async was explicitly rejected;
-the fix path is to raise timeouts, never to background the work.
+synchronous request**. The KB is a **classic VECTOR knowledge base backed by Amazon S3
+Vectors** (the canonical design — see the architecture note below), and the flow is now
+**9 steps** across systems with **no shared transaction**:
+
+1. **validate slug** (≤255) + derive via **`App_Talos_S3::slugify()`** (kebab-case, `iconv`
+   ASCII translit; rejects a duplicate slug — `slug` is UNIQUE).
+2. **S3 folders** — 3 markers `development-team/{slug}/{approved,archive,upload}/` via
+   **`App_Talos_S3::putFolderMarker()`**.
+3. **S3 Vectors store** — create a dedicated vector bucket + index via
+   `App_Talos_Bedrock::createVectorStore()`, capturing the authoritative **`indexArn`**.
+4. **VECTOR KB** — `createKnowledgeBase($bedrock, $slug, $indexArn)` (VECTOR + `S3_VECTORS`
+   storage).
+5. **wait for `ACTIVE`** (`waitForKnowledgeBaseActive`).
+6. **S3 data source** — plain `S3` connector + `BEDROCK_DATA_AUTOMATION` parsing.
+7. **`Team.KnowledgeBases`** (`db_team`, Team MySQL cluster).
+8. **`Client_True.VectorIndexes`** (`db_true`, **Client MySQL cluster** — not the Team cluster).
+9. **2 Talos Postgres rows** (`public.knowledge_bases` + `public.assistant_knowledge_bases`,
+   the latter linking a hard-coded assistant id).
+
+The create branch sets `set_time_limit(CREATE_MAX_EXECUTION_SECONDS = 600)` up top because the
+flow blocks on multi-minute AWS waits (S3 Vectors index readiness + KB `ACTIVE`, up to ~300s).
+This only prevents the PHP fatal — it does **not** lift the EB Apache `ProxyTimeout` ceiling
+(handled separately in `.platform`, see below). This flow **stays synchronous by design** —
+moving it to worker2/SQS async was explicitly rejected; the fix path is to raise timeouts,
+never to background the work.
+
+### Bedrock KB architecture — VECTOR + S3 Vectors (canonical)
+
+The canonical Talos KB config (from **`development-team-dunn`**, confirmed identical across
+`development-team-ninja-rmm` and `development-team-warehouse`) is a **classic VECTOR** KB, **not**
+the fully-MANAGED type:
+- **KnowledgeBase:** `knowledgeBaseConfiguration.type = VECTOR`;
+  `vectorKnowledgeBaseConfiguration.embeddingModelArn =
+  arn:aws:bedrock:us-east-1::foundation-model/amazon.titan-embed-text-v2:0`;
+  `embeddingModelConfiguration.bedrockEmbeddingModelConfiguration.embeddingDataType = FLOAT32`;
+  `storageConfiguration.type = S3_VECTORS` pointing at the per-KB S3 Vectors index ARN.
+- **S3 Vectors index** (one dedicated vector bucket + index **per KB**): `dimension = 1024`,
+  `dataType = float32`, `distanceMetric = euclidean`,
+  `metadataConfiguration.nonFilterableMetadataKeys = [AMAZON_BEDROCK_TEXT, AMAZON_BEDROCK_METADATA]`,
+  encryption `sseType = AES256` (SSE-S3). Vector bucket name mirrors the console:
+  `bedrock-knowledge-base-{slug}`; index name `bedrock-knowledge-base-default-index`.
+- **DataSource:** `dataSourceConfiguration.type = S3` (plain S3 connector);
+  `s3Configuration.bucketArn = arn:aws:s3:::togaiq`;
+  `inclusionPrefixes = ["development-team/{slug}/approved/"]`;
+  `vectorIngestionConfiguration.parsingConfiguration.parsingStrategy = BEDROCK_DATA_AUTOMATION`;
+  `dataDeletionPolicy = DELETE`.
+
+Account `654654170868`, region `us-east-1`. Verified working end-to-end in production
+(2026-07-16).
 
 The handler is **best-effort with no rollback**: on a mid-flow failure it stops and returns
 a `steps[]` report **without tearing down** already-created AWS resources (see Gotchas —
 orphaned KB).
 
-**`App_Talos_Bedrock`** (`_/app/talos/bedrock.php`) — Bedrock Agent client:
+**`App_Talos_Bedrock`** (`_/app/talos/bedrock.php`) — Bedrock Agent + S3 Vectors clients:
+- **`ROLE_ARN`** — the single **shared** execution role
+  `arn:aws:iam::654654170868:role/AmazonBedrockExecutionRoleForKnowledgeBase-talos`, hard-coded
+  (replaced the old per-KB `_tonej` role). See the IAM prerequisite below.
+- `s3VectorsClient()` — `Aws\S3Vectors\S3VectorsClient`, api version `2025-07-15`, using the same
+  `[talos]` credential resolution as `App_Talos_S3::client()`.
+- `vectorBucketName()` — derives `bedrock-knowledge-base-{slug}` and enforces the S3 Vectors
+  bucket-name length limit (3–63 chars).
+- `createVectorStore()` — creates the vector bucket + index sized to Titan v2 (dim 1024,
+  float32, euclidean), then **polls `getIndex` for the authoritative index ARN** with a short
+  `NotFound` retry; returns an actionable error on `Conflict`/`AlreadyExists` from an orphaned
+  prior attempt.
+- `createKnowledgeBase($bedrock, string $slug, string $indexArn)` — creates the VECTOR KB with
+  `S3_VECTORS` storage (signature now takes the index ARN).
 - `waitForKnowledgeBaseActive()` polls `getKnowledgeBase` every `WAIT_INTERVAL_SECONDS` (3s)
-  up to **`WAIT_MAX_SECONDS` = 300** (raised from 90; MANAGED KBs were observed taking ~3 min
-  to leave `CREATING`). Exits immediately on `ACTIVE`/`FAILED`, so fast activations are
-  unaffected. 300s sits well under the 900s Apache/PHP ceiling.
-- `createDataSource()` — `connectorParameters` for `managedKnowledgeBaseConnectorConfiguration`
-  is a **`Document` shape (free-form JSON)** in the AWS PHP SDK model and **must be passed as a
-  native PHP array** so the SDK serializes a JSON *object*. Passing a pre-`json_encode()`d
-  **string** fails `ValidationException "Invalid connector parameters format"`. Working object
-  shape (matches the craftex data source):
-  `{type:S3, filterConfiguration:{maxFileSizeInMegaBytes:"500",
-  inclusionPrefixes:["development-team/{slug}/approved/"]}, connectionConfiguration:{bucketName:togaiq,
-  bucketOwnerAccountId:654654170868, bucketArn:arn:aws:s3:::togaiq}, aclEnabled:false, version:"1"}`.
+  up to **`WAIT_MAX_SECONDS` = 300**. Exits immediately on `ACTIVE`/`FAILED`. 300s sits well
+  under the 900s Apache/PHP ceiling.
+- `createDataSource()` — plain **`S3`** connector + `BEDROCK_DATA_AUTOMATION` parsing (rewritten
+  from the old `MANAGED_KNOWLEDGE_BASE_CONNECTOR`). See the architecture note above for the exact
+  shape.
+- The vendored `aws/aws-sdk-php` (`^3.387`) already ships both the `S3Vectors` and `BedrockAgent`
+  clients and the `S3_VECTORS` `storageConfiguration` shape (verified against the SDK model). The
+  now-dead `BUCKET_NAME` / `BUCKET_OWNER_ACCOUNT_ID` constants were removed.
+
+**IAM prerequisites (one-time, manual — the app does NOT provision IAM):**
+- **Shared execution role** `AmazonBedrockExecutionRoleForKnowledgeBase-talos` — one role for
+  ALL Talos KBs (decision: do **not** create a role+policies per KB, which would require granting
+  the web app IAM write powers — rejected as a security expansion). Has 4 inline policies
+  mirroring dunn's but **wildcarded**: s3vectors data-plane on
+  `bucket/bedrock-knowledge-base-*/index/*`; `s3:GetObject` on
+  `togaiq/development-team/*/approved/*` (+ `s3:ListBucket` on `togaiq`); `bedrock:InvokeModel`
+  on `titan-embed-text-v2:0` (+ marketplace); BDA `GetDataAutomationStatus` +
+  `InvokeDataAutomationAsync`. Trust policy: `bedrock.amazonaws.com` assume-role with
+  `SourceAccount 654654170868` and `SourceArn knowledge-base/*`.
+- **App credential grant** — the `[talos]` app credentials are IAM user
+  `arn:aws:iam::654654170868:user/bedrock-model-accessor`. It was granted (new inline policy
+  **`TalosKbProvisioning`**) `s3vectors:CreateVectorBucket/CreateIndex/GetIndex` and
+  `iam:PassRole` on the shared role. Its bedrock create actions were already covered by
+  `AmazonBedrockFullAccess` and S3 by `AmazonS3FullAccess`. **`s3vectors` is a SEPARATE service
+  NOT covered by `AmazonBedrockFullAccess`** — that was the real missing grant.
 
 ## Helpers
 
@@ -200,20 +264,23 @@ orphaned KB).
 - **php-fpm `request_terminate_timeout` can still kill the request.** If
   `/etc/php-fpm.d/www.conf`'s `request_terminate_timeout` is ever set to 60 it terminates the
   request regardless of Apache — it must be `0` or `>= 900`.
-- **`get-data-source` echoes `connectorParameters` back as a quoted STRING** even though the
-  create input must be a JSON **object** — the GET representation is misleading and does NOT
-  indicate the required input shape.
-- **KB create has no rollback → orphaned KB risk.** A failure at the data-source step leaves an
-  orphaned Bedrock KB with **no DB rows** (`Team.KnowledgeBases`/`VectorIndexes`/Postgres never
-  inserted). Blindly re-running with the same name creates a **duplicate** Bedrock KB (Bedrock
-  allows same-name KBs with new ids). After a failed create: either
-  `aws bedrock-agent delete-knowledge-base` the orphan and re-run clean, or finish the DB rows
-  manually.
-- **Newest KBs use Bedrock's fully-MANAGED type** (`knowledgeBaseConfiguration.type=MANAGED`,
-  `managedKnowledgeBaseConfiguration.embeddingModelType=MANAGED`) — **AWS owns the vector store**,
-  so there is **no S3 Vectors index, no `embeddingModelArn`, and no `storageConfiguration`** to
-  build. Config was replicated from the existing `development-team-craftex` KB. Account
-  `654654170868`, region `us-east-1`, execution role `…ForKnowledgeBase_ofl5a`.
+- **The Bedrock API does NOT auto-provision the S3 Vectors bucket/index** (the console does).
+  They must be created first (`s3vectors CreateVectorBucket` + `CreateIndex`) and the resulting
+  index ARN passed into `CreateKnowledgeBase`. `createVectorStore()` handles this before KB
+  creation.
+- **`s3vectors` is a separate AWS service** — it is **not** covered by `AmazonBedrockFullAccess`.
+  The app credentials need explicit `s3vectors:*` grants (see IAM prerequisite above); this was
+  the real missing permission.
+- **KB create has no rollback → orphaned KB risk.** A failure mid-flow can leave an orphaned
+  S3 Vectors bucket/index and/or Bedrock KB with **no DB rows** (`Team.KnowledgeBases`/
+  `VectorIndexes`/Postgres never inserted). Blindly re-running with the same slug hits
+  `Conflict`/`AlreadyExists` on the vector bucket (surfaced as an actionable error) or creates a
+  **duplicate** Bedrock KB (Bedrock allows same-name KBs with new ids). After a failed create:
+  delete the orphaned vector store + KB and re-run clean, or finish the DB rows manually.
+- **Two prior KBs use the WRONG (fully-MANAGED) type.** `development-team-craftex` and
+  `development-team-toga-desk` were created by the earlier MANAGED code and were **intentionally
+  left as-is** (this session's scope was code-only). They should be **recreated later** to match
+  the canonical VECTOR + S3 Vectors design (dunn).
 - **KB-list source differs per page.** The new `/talos/knowledge-bases` page lists from
   `Team.KnowledgeBases` (`db_team`); `/talos/kb-documents` lists from live S3 folders. Don't
   assume one reflects the other.
@@ -222,15 +289,27 @@ orphaned KB).
   `Client_True.VectorIndexes` (Client cluster): `id, uuid`(UNIQUE)`, name, docSlug`(UNIQUE,
   nullable)`, sectionSlug, chunkSlug, description` — create sets `docSlug=sectionSlug=chunkSlug=
   bedrockKbId`.
-- **Not yet run end-to-end in production.** The whole feature is code-complete, `php -l` clean,
-  and reviewed, but the create flow has not been exercised in prod pending the `pdo_pgsql`
-  deploy plus network/IAM prerequisites (the developer is handling those).
+- **Verified working end-to-end in production (2026-07-16)** with the VECTOR + S3 Vectors design.
 - No new secret literals introduced; credentials live in the config sections above (documented by
   section/account, never by value). AWS account id `654654170868`, bucket `togaiq`, and the KB
   execution-role ARN are non-secret resource identifiers.
 
 ## Change history
 
+- 2026-07-16 — **Corrected the KB-create architecture from MANAGED to VECTOR + S3 Vectors**
+  (the canonical design, from `development-team-dunn`; the earlier code was modeled on the WRONG
+  `craftex`/MANAGED KB). Rewrote `App_Talos_Bedrock`: KB is now `VECTOR` + `S3_VECTORS` storage;
+  added `s3VectorsClient()`, `vectorBucketName()`, and `createVectorStore()` (create vector
+  bucket + index sized to Titan v2, poll `getIndex` for the ARN); `createKnowledgeBase()` now
+  takes the index ARN; `createDataSource()` rewritten to plain `S3` + `BEDROCK_DATA_AUTOMATION`;
+  removed dead `BUCKET_NAME`/`BUCKET_OWNER_ACCOUNT_ID`. Wired a new **S3 Vectors provisioning
+  step** into `post.php` before KB creation (flow is now 9 steps) and added
+  `set_time_limit(CREATE_MAX_EXECUTION_SECONDS=600)`. **Decided on ONE shared IAM execution role**
+  (`…ForKnowledgeBase-talos`, hard-coded `ROLE_ARN`) instead of per-KB roles, to avoid granting
+  the app IAM write powers; granted the `[talos]` user (`bedrock-model-accessor`) the missing
+  `s3vectors` + `iam:PassRole` permissions (`s3vectors` is NOT in `AmazonBedrockFullAccess`).
+  **Verified working end-to-end in production.** `craftex` and `toga-desk` remain on the old
+  MANAGED type (left as-is; recreate later). (jcardinal)
 - 2026-07-13 — Added the **`/talos/knowledge-bases` list + inline-rename page** (`get.php`;
   lists from `Team.KnowledgeBases`, rename-only, no delete, existence-check before update),
   registered first in the `talos-kb` nav group. Added the **`App_Pg`** tools-only PDO `pgsql`

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.354",
+  "version": "1.0.355",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",