toga-ai 1.0.319 → 1.0.321

package/knowledge/1.0/apps/tools/INDEX.md

@@ -7,6 +7,6 @@
 | [Tools MVC — Routing, CSRF & App_Database Access Patterns](features/mvc-data-access-patterns.md) | The load-bearing 1.0 (`App_`) framework conventions a developer needs when adding a page to the Tools app — URL routing, CSRF, and DB access through `App_Databa | tools/_/app/nav.php, tools/mvc/get.php |
 | [Tools Persona-Gated Navigation (App_Nav)](features/persona-gated-navigation.md) | `App_Nav` is the Tools app's two-level, **persona-gated** navigation. | tools/_/app/nav.php, tools/mvc/get.php |
 | [Tools SAML SSO Consumer & Persona-Gated Auth (App_Auth)](features/saml-sso-auth.md) | `App_Auth` is the Tools app's authentication layer: it consumes the SAML gateway `?saml=` handoff (see the 2.0 SAML downstream integration contract), establishe | tools/_/app/auth.php, tools/mvc/sso/initiate/get.php, tools/mvc/sso/get.php, tools/mvc/login/get.php, tools/mvc/login/post.php, tools/mvc/logout/get.php, tools/mvc/get.php, tools/config.production.ini, tools/config.local.ini |
-| [Talos Knowledge Base Admin UI (KB Documents + Vocabulary)](features/talos-kb-documents-admin.md) | A Tools (1.0) admin UI to browse/fix the Talos knowledge-base documents and manage the transcript-cleanup vocabulary — without a deploy. | tools/mvc/talos/kb-documents/get.php, tools/mvc/talos/kb-documents/post.php, tools/mvc/talos/vocabulary/get.php, tools/mvc/talos/vocabulary/post.php, tools/_/app/talos/s3.php, tools/_/app/worker.php, tools/_/app/nav.php, tools/config.production.ini, tools/config.alpha.ini |
+| [Talos Knowledge Base Admin UI (KB Documents + Vocabulary)](features/talos-kb-documents-admin.md) | A Tools (1.0) admin UI to browse/fix the Talos knowledge-base documents and manage the transcript-cleanup vocabulary — without a deploy. | tools/mvc/talos/kb-documents/get.php, tools/mvc/talos/kb-documents/post.php, tools/mvc/talos/knowledge-bases/get.php, tools/mvc/talos/knowledge-bases/post.php, tools/mvc/talos/vocabulary/get.php, tools/mvc/talos/vocabulary/post.php, tools/_/app/talos/s3.php, tools/_/app/talos/bedrock.php, tools/_/app/pg.php, tools/_/app/worker.php, tools/_/app/nav.php, tools/config.production.ini, tools/config.alpha.ini, tools/.platform/httpd/conf.d/timeouts.conf, tools/.platform/hooks/prebuild/01-install-php-pgsql.sh, tools/.platform/hooks/postdeploy/01-restart-php.sh |
 | [Talos Pricing UI (Onboarding, Dashboard, Benchmarks, Cost Factors + Estimator)](features/talos-pricing-ui.md) | The 1.0 (tools app) face of the **Talos Pricing Platform** — a "Talos Pricing" nav folder with four pages plus a client-side estimate engine. | tools/_/app/nav.php, tools/_/app/talos/estimator.php, tools/mvc/talos/onboarding/get.php, tools/mvc/talos/onboarding/post.php, tools/mvc/talos/pricing/get.php, tools/mvc/talos/benchmarks/get.php, tools/mvc/talos/factors/get.php, tools/mvc/talos/factors/post.php, tools/assets/css/style.css |
 | [Deploying Tools to Elastic Beanstalk (PHP 8.5 / Amazon Linux 2023)](workflows/deploy-to-elastic-beanstalk-al2023.md) | How the **Tools** 1.0 app boots on Elastic Beanstalk running `PHP 8.5 on 64bit Amazon Linux 2023/4.13.1 (aarch64)`. | tools/.ebextensions/004_http_to_https.config, tools/.ebextensions/006_mount-s3fs.config, tools/.ebextensions/007_setup_export_cache_folders.config, tools/.ebextensions/008_setup_ldap.config, tools/.ebextensions/009_setup_phpini.config, tools/.ebextensions/020_setup_git_libraries.config, tools/.ebextensions/050_register_instance_to_shared_application_load_balancer.config, tools/ebs/git.json |

package/knowledge/1.0/apps/tools/features/talos-kb-documents-admin.md

@@ -6,18 +6,25 @@ project: Tools
 client: shared
 type: feature
 status: active
-updated: 2026-07-09
+updated: 2026-07-13
 owners: [jcardinal]
 files:
   - tools/mvc/talos/kb-documents/get.php
   - tools/mvc/talos/kb-documents/post.php
+  - tools/mvc/talos/knowledge-bases/get.php
+  - tools/mvc/talos/knowledge-bases/post.php
   - tools/mvc/talos/vocabulary/get.php
   - tools/mvc/talos/vocabulary/post.php
   - tools/_/app/talos/s3.php
+  - tools/_/app/talos/bedrock.php
+  - tools/_/app/pg.php
   - tools/_/app/worker.php
   - tools/_/app/nav.php
   - tools/config.production.ini
   - tools/config.alpha.ini
+  - tools/.platform/httpd/conf.d/timeouts.conf
+  - tools/.platform/hooks/prebuild/01-install-php-pgsql.sh
+  - tools/.platform/hooks/postdeploy/01-restart-php.sh
 related:
   - ./mvc-data-access-patterns.md
   - ./persona-gated-navigation.md
@@ -31,7 +38,8 @@ A Tools (1.0) admin UI to browse/fix the Talos knowledge-base documents and mana
 transcript-cleanup vocabulary — without a deploy. It is the human front-end to the worker2
 [Talos Transcript Ingestion](../../../2.0/apps/worker2/features/talos-transcript-ingestion.md)
 pipeline. A new **"Talos Knowledge Base"** nav group (personas *Development Team* / *TOGa
-Technology*) exposes two pages.
+Technology*) exposes the pages below. **"Knowledge Bases"** is registered as the **first**
+action in the `talos-kb` nav group (above "KB Documents") and inherits the group's personas.
 
 ## Pages
 
@@ -73,6 +81,53 @@ so they displayed **double-encoded** (e.g. "Grand & Toy"). `vocabulary/post.
 `html_entity_decode()`s values on save so the DB stores **raw** text; `vocabulary/get.php`
 already escapes once at display. Do not re-add encoding on save.
 
+### `/talos/knowledge-bases` — KB list + inline rename
+
+The list/admin page (`get.php`) for Talos knowledge bases. Unlike KB Documents (whose KB
+list is derived from **live S3 folders**), this page's list comes from the
+**`Team.KnowledgeBases` MySQL table** (via `db_team`) — the two are distinct sources of
+truth. Columns: Slug, Name, Bedrock KB ID, Created On (`dtCreated`, converted UTC→
+America/Chicago). **Name is the only editable field** and is edited **in place** (Edit→Save);
+there is **no delete option**. Server strings are escaped before `innerHTML` (XSS guard).
+
+Rename posts `action=rename` → `UPDATE Team.KnowledgeBases SET name WHERE slug` (`db_team`),
+preceded by an **existence SELECT** so a stale slug reports **"not found"** instead of a
+false success.
+
+### `POST /talos/knowledge-bases` — create a new KB (synchronous provisioning)
+
+Admin action that provisions a brand-new Talos knowledge base end to end in **one
+synchronous request**: creates the S3 prefix, creates the Bedrock KB and **waits for it to
+reach `ACTIVE`**, creates the Bedrock data source (`MANAGED_KNOWLEDGE_BASE_CONNECTOR`), and
+inserts rows across **5 systems with no shared transaction**: `Team.KnowledgeBases` (`db_team`,
+Team MySQL cluster), `Client_True.VectorIndexes` (`db_true`, **Client MySQL cluster** — not the
+Team cluster), and 2 Talos **Postgres** rows (`public.knowledge_bases` +
+`public.assistant_knowledge_bases`, the latter linking a hard-coded assistant id). Step order:
+validate name (≤255) + derive slug via **`App_Talos_S3::slugify()`** (kebab-case, `iconv` ASCII
+translit; rejects a duplicate slug — `slug` is UNIQUE); create 3 S3 folder markers
+`development-team/{slug}/{approved,archive,upload}/` via **`App_Talos_S3::putFolderMarker()`**;
+create the Bedrock KB and wait for `ACTIVE`; create the data source; then the DB inserts. This flow
+**stays synchronous by design** — moving it to worker2/SQS async was explicitly rejected;
+the fix path is to raise timeouts, never to background the work.
+
+The handler is **best-effort with no rollback**: on a mid-flow failure it stops and returns
+a `steps[]` report **without tearing down** already-created AWS resources (see Gotchas —
+orphaned KB).
+
+**`App_Talos_Bedrock`** (`_/app/talos/bedrock.php`) — Bedrock Agent client:
+- `waitForKnowledgeBaseActive()` polls `getKnowledgeBase` every `WAIT_INTERVAL_SECONDS` (3s)
+  up to **`WAIT_MAX_SECONDS` = 300** (raised from 90; MANAGED KBs were observed taking ~3 min
+  to leave `CREATING`). Exits immediately on `ACTIVE`/`FAILED`, so fast activations are
+  unaffected. 300s sits well under the 900s Apache/PHP ceiling.
+- `createDataSource()` — `connectorParameters` for `managedKnowledgeBaseConnectorConfiguration`
+  is a **`Document` shape (free-form JSON)** in the AWS PHP SDK model and **must be passed as a
+  native PHP array** so the SDK serializes a JSON *object*. Passing a pre-`json_encode()`d
+  **string** fails `ValidationException "Invalid connector parameters format"`. Working object
+  shape (matches the craftex data source):
+  `{type:S3, filterConfiguration:{maxFileSizeInMegaBytes:"500",
+  inclusionPrefixes:["development-team/{slug}/approved/"]}, connectionConfiguration:{bucketName:togaiq,
+  bucketOwnerAccountId:654654170868, bucketArn:arn:aws:s3:::togaiq}, aclEnabled:false, version:"1"}`.
+
 ## Helpers
 
 - **`App_Talos_S3`** (`_/app/talos/s3.php`) — S3 client; list KB slugs; list/count approved
@@ -85,6 +140,13 @@ already escapes once at display. Do not re-add encoding on save.
   the SDK default chain — because `App_Talos_S3` only ever talks to **togaiq**, so it needs the
   togaiq-account key (the tools `[aws]` key is the toga-private key, which can cross-account *read*
   togaiq but not *write* it).
+- **`App_Pg`** (`_/app/pg.php`) — a **tools-only** minimal PDO `pgsql` wrapper for the Talos
+  Postgres DB (host `writer.talos.database.togahub.com:5432`, db `tenant_true`, user
+  `talosadmin`). Deliberately **not** added to the shared 1.0 `library` (the App_ framework has
+  no Postgres support). The password is **read from the existing `[database_team]` config
+  section** (shared with the MySQL DBs) — **no new secret introduced**. `execute()` uses real
+  PDO prepared statements (bound params). If the `pdo_pgsql` extension is missing it returns
+  `ok=false` with a generic message (detail goes only to `error_log`, never to the client).
 - **`App_Worker::enqueue()`** (`_/app/worker.php`) — inserts a `Core.WorkerJobs` row and sends
   the `{"workerJobId": id}` SQS message, so a 1.0 app can enqueue worker2 jobs. Requires the
   `[worker]` (`queue_url`/`queue_region`) and `[aws]` (main-account key for SQS `SendMessage`)
@@ -100,6 +162,12 @@ already escapes once at display. Do not re-add encoding on save.
   - **`[aws]`** — main-account key for SQS `SendMessage` (toga-private key; cross-account reads
     togaiq but cannot write it — that's why `[talos]` exists).
   - `[database_toga2core]` — writable 2.0 Core (already present).
+- **`pdo_pgsql` PHP extension** must be installed on the tools Elastic Beanstalk environment
+  (the create flow writes to Talos Postgres via `App_Pg`). Provided by two `.platform` hooks
+  (mirroring worker2's): `hooks/prebuild/01-install-php-pgsql.sh` (installs `php-pgsql`,
+  idempotent, **fails the deploy loudly** if `pgsql` won't load) and
+  `hooks/postdeploy/01-restart-php.sh` (restarts php-fpm/httpd). **EB runs `.platform/hooks`
+  regardless of the git exec bit** — worker2's equivalents are committed `100644`.
 - Apply the dbchanges2 `Team/2026-06-30a..e` + `Core/2026-06-30a` migrations in order.
 - Verify the Bedrock KB region / data-source name against the live account (TODOs in code).
 
@@ -116,11 +184,70 @@ already escapes once at display. Do not re-add encoding on save.
   within the same bucket; move/edit copy via get+put.
 - **Move/edit re-sync must be one `SyncKb("source,dest")` job**, not two parallel jobs — Bedrock
   allows only one in-flight ingestion per KB and parallel syncs race.
+- **KB create is synchronous and long (~minutes).** It must run under a raised timeout — see
+  the EB Apache 504 gotcha below. Do **not** re-architect it to worker2/SQS (explicitly
+  rejected).
+- **EB AL2023 PHP platform → 60s default proxy timeout → 504.** The tools app runs on Elastic
+  Beanstalk PHP 8.5 / Amazon Linux 2023, where **Apache proxies to php-fpm via
+  `mod_proxy_fcgi`** and the proxy read timeout defaults to `Timeout` = **60s**. A long
+  synchronous request (KB create) returns **504 after exactly 60s**. Raising the **ALB idle
+  timeout does not help** (it's upstream of Apache), and PHP `set_time_limit(900)` is
+  irrelevant (Apache emits the 504, and on Linux `max_execution_time` does not count time
+  blocked in network I/O). Fix: an EB platform Apache drop-in at
+  **`.platform/httpd/conf.d/timeouts.conf`** setting `Timeout 900` and **`ProxyTimeout 900`**
+  (`ProxyTimeout` is the `mod_proxy_fcgi` read timeout; matches the framework's
+  `set_time_limit(900)`). Applied on deploy.
+- **php-fpm `request_terminate_timeout` can still kill the request.** If
+  `/etc/php-fpm.d/www.conf`'s `request_terminate_timeout` is ever set to 60 it terminates the
+  request regardless of Apache — it must be `0` or `>= 900`.
+- **`get-data-source` echoes `connectorParameters` back as a quoted STRING** even though the
+  create input must be a JSON **object** — the GET representation is misleading and does NOT
+  indicate the required input shape.
+- **KB create has no rollback → orphaned KB risk.** A failure at the data-source step leaves an
+  orphaned Bedrock KB with **no DB rows** (`Team.KnowledgeBases`/`VectorIndexes`/Postgres never
+  inserted). Blindly re-running with the same name creates a **duplicate** Bedrock KB (Bedrock
+  allows same-name KBs with new ids). After a failed create: either
+  `aws bedrock-agent delete-knowledge-base` the orphan and re-run clean, or finish the DB rows
+  manually.
+- **Newest KBs use Bedrock's fully-MANAGED type** (`knowledgeBaseConfiguration.type=MANAGED`,
+  `managedKnowledgeBaseConfiguration.embeddingModelType=MANAGED`) — **AWS owns the vector store**,
+  so there is **no S3 Vectors index, no `embeddingModelArn`, and no `storageConfiguration`** to
+  build. Config was replicated from the existing `development-team-craftex` KB. Account
+  `654654170868`, region `us-east-1`, execution role `…ForKnowledgeBase_ofl5a`.
+- **KB-list source differs per page.** The new `/talos/knowledge-bases` page lists from
+  `Team.KnowledgeBases` (`db_team`); `/talos/kb-documents` lists from live S3 folders. Don't
+  assume one reflects the other.
+- **Schemas.** `Team.KnowledgeBases`: `id, uuid`(UNIQUE)`, slug`(UNIQUE)`, name, isGeneral,
+  bedrockKbId`(nullable)`, isActive, dtCreated`(datetime default `CURRENT_TIMESTAMP`)`, dtUpdated`.
+  `Client_True.VectorIndexes` (Client cluster): `id, uuid`(UNIQUE)`, name, docSlug`(UNIQUE,
+  nullable)`, sectionSlug, chunkSlug, description` — create sets `docSlug=sectionSlug=chunkSlug=
+  bedrockKbId`.
+- **Not yet run end-to-end in production.** The whole feature is code-complete, `php -l` clean,
+  and reviewed, but the create flow has not been exercised in prod pending the `pdo_pgsql`
+  deploy plus network/IAM prerequisites (the developer is handling those).
 - No new secret literals introduced; credentials live in the config sections above (documented by
-  section/account, never by value).
+  section/account, never by value). AWS account id `654654170868`, bucket `togaiq`, and the KB
+  execution-role ARN are non-secret resource identifiers.
 
 ## Change history
 
+- 2026-07-13 — Added the **`/talos/knowledge-bases` list + inline-rename page** (`get.php`;
+  lists from `Team.KnowledgeBases`, rename-only, no delete, existence-check before update),
+  registered first in the `talos-kb` nav group. Added the **`App_Pg`** tools-only PDO `pgsql`
+  wrapper (password reused from `[database_team]`; not added to shared library) and the
+  **`.platform` pdo_pgsql install/restart hooks**. Documented the create flow's 5-system fan-out
+  (incl. `Client_True.VectorIndexes` on the **Client** cluster + 2 Postgres rows), the
+  `slugify()`/`putFolderMarker()` helpers, the fully-**MANAGED** Bedrock KB type (AWS-owned vector
+  store, no storageConfiguration), and the two DB schemas. Feature is code-complete/reviewed but
+  not yet run end-to-end in prod. (jcardinal)
+- 2026-07-13 — Added the **synchronous KB-create flow** (`POST /talos/knowledge-bases`) via new
+  `App_Talos_Bedrock` helper. Fixed the **504 at 60s**: EB AL2023 PHP = Apache→php-fpm over
+  `mod_proxy_fcgi` with a 60s default proxy read timeout; added
+  `.platform/httpd/conf.d/timeouts.conf` (`Timeout 900` + `ProxyTimeout 900`). Raised
+  `WAIT_MAX_SECONDS` 90→300 (MANAGED KBs take ~3 min to go `ACTIVE`). Fixed `createDataSource`
+  `ValidationException` by passing `connectorParameters` as a native array (`Document` shape),
+  not a `json_encode`d string. Documented the no-rollback orphaned-KB caveat and the misleading
+  `get-data-source` string echo. Flow stays synchronous (worker2 async rejected). (jcardinal)
 - 2026-07-09 — Fixed `App_Talos_S3::sanitizeTitle()` to convert `/`,`\` to hyphens (was
   spaces) so a slashed meeting title no longer creates a stray S3 sub-folder — matches
   worker2's `filenameSafeTitle()`; `updateApprovedDocument()` already renames the S3 object and

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.319",
+  "version": "1.0.321",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",