toga-ai 1.0.154 → 1.0.156

package/knowledge/1.0/apps/test/INDEX.md

@@ -9,6 +9,7 @@
 | [Developer Generators (password, UUID)](features/dev-generators.md) | Two tiny **1.0 `App_` framework** convenience scripts for everyday developer needs. | test/team/generate_password.php, test/team/uuid.php |
 | [Forecast vs NetSuite Discrepancy Analysis](features/forecast-netsuite-discrepancy-analysis.md) | `team/forecast-netsuite/discrepancy_analysis.php` detects discrepancies between our **Forecast database** and **NetSuite** (the source of truth for all sales da | test/team/forecast-netsuite/discrepancy_analysis.php |
 | [TableView Builder (2.0 TableViews SQL generator)](features/tableview-builder.md) | `team/tableViewBuilder/` generates SQL `INSERT` statements for the **2.0 `TableViews`**, `TableViewFields`, and `TableViewJoins` tables from a plain SQL `SELECT | test/team/tableViewBuilder/TableViewGenerator.php, test/team/tableViewBuilder/index.php, test/team/tableViewBuilder/Instructions.md |
+| [Talos Knowledge Base Pipeline (Uploader + Processor)](features/talos-kb-pipeline.md) | `team/talos/` holds the two-script web tooling that feeds the **TOGa Talos** (TOGa IQ) AI knowledge bases. | test/team/talos/kb_uploader.php, test/team/talos/kb_processor.php, test/team/talos/kb_processor.ini |
 | [TOGa 2.0 Client Onboarding SQL Generator](features/toga2-client-onboarding-sql.md) | `team/generate_toga2_onboarding_sql.php` generates the SQL needed to **onboard a new client** onto the 2.0 platform. | test/team/generate_toga2_onboarding_sql.php |
 | [TOGa 2.0 User Cross-Client Access SQL Generator](features/toga2-user-cross-client-access-sql.md) | `team/generate_toga2_user_access_sql.php` generates SQL to grant an existing 2.0 user from a **home client** access to a **cross client**. | test/team/generate_toga2_user_access_sql.php |
 | [URL & Domain Markdown Document Builder](features/url-domain-markdown-document.md) | `team/build_url_domain_markdown_document.php` generates a **markdown document of our URLs and domains** by pulling environments and domains from the 2.0 platfor | test/team/build_url_domain_markdown_document.php |

package/knowledge/1.0/apps/test/architecture.md

@@ -6,7 +6,7 @@ project: Test
 client: shared
 type: architecture
 status: active
-updated: 2026-06-16
+updated: 2026-06-22
 owners: [jcardinal]
 files:
   - test/team/
@@ -20,6 +20,7 @@ related:
   - ./features/active-directory-auth-test.md
   - ./features/url-domain-markdown-document.md
   - ./features/dev-generators.md
+  - ./features/talos-kb-pipeline.md
 ---
 
 ## Summary
@@ -81,9 +82,12 @@ manually against the target database; they do not execute changes themselves.
 | `generate_toga2_onboarding_sql.php` | 1.0 | 2.0 client DBs | [toga2-client-onboarding-sql](./features/toga2-client-onboarding-sql.md) |
 | `generate_toga2_user_access_sql.php` | 1.0 | 2.0 client DBs | [toga2-user-cross-client-access-sql](./features/toga2-user-cross-client-access-sql.md) |
 | `generate_password.php`, `uuid.php` | 1.0 | dev utility | [dev-generators](./features/dev-generators.md) |
+| `talos/kb_uploader.php`, `talos/kb_processor.php` | standalone | S3 + AWS Bedrock KBs | [talos-kb-pipeline](./features/talos-kb-pipeline.md) |
 
-**Retired / not documented:** `team/DOA_gitbook/` and `team/DOA_talos/` are being retired.
+**Retired / not documented:** `team/DOA_gitbook/` is being retired.
 
 ## Change history
 
 - 2026-06-16 — Initial capture of the `team/` folder. Registered repo in `registry.json`.
+- 2026-06-22 — Un-retired `team/DOA_talos/` → `team/talos/`; documented its uploader +
+  processor in [talos-kb-pipeline](./features/talos-kb-pipeline.md).

package/knowledge/1.0/apps/test/features/talos-kb-pipeline.md

@@ -0,0 +1,145 @@
+---
+title: Talos Knowledge Base Pipeline (Uploader + Processor)
+framework: "1.0"
+repo: test
+project: Test
+client: shared
+type: feature
+status: active
+updated: 2026-06-22
+owners: [jcardinal]
+files:
+  - test/team/talos/kb_uploader.php
+  - test/team/talos/kb_processor.php
+  - test/team/talos/kb_processor.ini
+related:
+  - ../architecture.md
+---
+
+## Summary
+
+`team/talos/` holds the two-script web tooling that feeds the **TOGa Talos** (TOGa IQ) AI
+knowledge bases. Content flows through S3 folders and ends up in **AWS Bedrock** knowledge
+bases, which back the internal AI agent that answers developer questions.
+
+- **`kb_uploader.php`** — the *intake* page. A user pastes a meeting transcript or uploads a
+  document; it lands in the S3 `upload/` folder of a chosen knowledge base.
+- **`kb_processor.php`** — the *review/approve* console. A reviewer cleans transcripts (AI +
+  automated text rules), approves files into the `approved/` folder, writes a Bedrock metadata
+  sidecar, archives the raw source, and triggers a Bedrock ingestion (sync) job.
+
+Both are **standalone PHP pages** — they do **not** bootstrap the 1.0 `App_` framework. They
+load the AWS SDK from `../../vendor/autoload.php` (Composer `aws/aws-sdk-php`) and read all
+settings from a shared `kb_processor.ini`. Target PHP 7.2+.
+
+> History: this folder was previously named `DOA_talos` (the `DOA_` prefix marked it as
+> deprecated-but-live). It was un-retired and renamed to `talos` on 2026-06-22.
+
+## S3 layout
+
+All paths are relative to `S3_BUCKET` (`togaiq`) under `S3_ROOT_PATH` (`development-team/`).
+Each **knowledge base** is a folder one level under the root (e.g. `development-team/<kb>/`),
+discovered by listing common prefixes. Within each KB:
+
+| Folder (config key) | Default | Role |
+|---|---|---|
+| `S3_UPLOAD_FOLDER`   | `upload`   | Raw intake — where the uploader drops files |
+| `S3_APPROVED_FOLDER` | `approved` | Reviewed/approved content (+ `.metadata.json` sidecars) — the Bedrock data source |
+| `S3_ARCHIVE_FOLDER`  | `archive`  | Raw source files after they've been approved |
+
+A KB named **`general`** is treated specially: it is never auto-synced on approve/delete;
+instead it is synced in bulk via the "Sync All General" flow.
+
+## kb_uploader.php — intake
+
+Single page with two POST forms (toggled by a client-side mode switcher):
+
+1. **Meeting Transcription** (`submit_meeting`) — fields: date, meeting name, KB, raw
+   transcript. The meeting name is slash-sanitized; the transcript's repeated blank lines are
+   collapsed; the file is named **`YYYY-MM-DD - {MEETING_NAME}.txt`** and `putObject`'d to
+   `<root>/<kb>/upload/`. This filename pattern is what the processor later recognizes as a
+   meeting (`isMeetingTranscript()` regex).
+2. **File Upload** (`submit_file`) — fields: KB, document name, file. Extension is validated
+   against `ALLOWED_EXTENSIONS`; the file is stored as `{documentName}.{ext}` in the same
+   `upload/` folder via `putObject` with `SourceFile`.
+
+Knowledge-base options come from `getKnowledgeBases()`. All output is escaped with
+`htmlspecialchars()`.
+
+## kb_processor.php — review & approve
+
+The page lists everything in every KB's `upload/` folder, split into **meetings** (filename
+matches `^\d{4}-\d{2}-\d{2} - .+\.txt$`) and **documents**. It exposes AJAX actions (POST
+`action=...`, JSON responses) plus an **Approved Files Browser** modal. Key actions:
+
+| `action` | Purpose |
+|---|---|
+| `get_file_content` | Return an S3 object's text (transcript editor / file viewer) |
+| `prepare_transcript` | AI clean → automated clean → prepend `date - title` header |
+| `save_to_kb` | Approve a file (see flow below) |
+| `download_file` | Stream any S3 key to the browser as an attachment (`Content-Disposition`) |
+| `delete_file` / `delete_approved_file` | Delete from `upload/` / `approved/` (latter re-syncs the KB) |
+| `list_approved_files` / `count_approved_files` | Populate the Approved Files Browser |
+| `start_sync_all_general` / `sync_next_kb` / `get_sync_progress` | Session-backed batch sync of every KB's `general` data source, polled from the front-end |
+
+### Transcript cleaning (`prepare_transcript`)
+
+1. **AI cleaning** — `callAICleaningAPI()` POSTs to `AI_API_ENDPOINT` (the TOGa IQ AI service,
+   model `AI_MODEL_ID`, Bedrock Claude Haiku 4.5) with an `output_schema` requesting a single
+   `knowledge_doc` string. The lengthy `AI_SYSTEM_PROMPT` enforces a strict lossless plain-text
+   format (OVERVIEW + topical sections, verbatim code blocks, PascalCase tables / camelCase
+   columns). Retries twice with backoff on network errors and HTTP 429/5xx.
+2. **Automated cleaning** — `applyAutomatedCleaning()` strips markdown artifacts (backticks,
+   `> ` quoting, `# ` headings), normalizes blank lines around headings/lists, removes
+   `:contentReference[...]` markers, and applies the `REPLACEMENTS[...]` map from the ini
+   (mishear corrections, e.g. `Talus → Talos`). Replacement keys support `:c` (case-sensitive)
+   and `:w` (whole-word) flags.
+3. For meeting files, a `YYYY-MM-DD - Title` header is prepended.
+
+### Approve flow (`save_to_kb`)
+
+1. Write the approved content to `<root>/<newKb>/approved/<filename>` — uploading the cleaned
+   text for meetings, or copying the original S3 object directly for documents.
+2. Write a **Bedrock metadata sidecar** `<filename>.metadata.json`
+   (`buildApprovedMetadataJson()`): `metadataAttributes` with `slug`, `client` (the KB name),
+   `state=approved`, `approved_at`, `filename`, each `includeForEmbedding: false`.
+3. **Archive** the raw source: move `upload/<file>` → `archive/<file>` (copy + delete).
+4. **Sync** the Bedrock knowledge base unless the KB is `general`:
+   `syncKnowledgeBaseDataSource()` resolves the KB by name (`KB_PREFIX + kb`, e.g.
+   `development-team-<kb>`), finds the matching data source, and `startIngestionJob()`s it with
+   a UUID client token. Retries on `ConflictException` / throttling up to `KB_CONFLICT_MAX_RETRIES`.
+
+### Approved Files Browser
+
+Modal listing the chosen KB's `approved/` files with size/modified, plus per-row **view**
+(text types open in a viewer via `get_file_content`), **download** (⬇️ — form-POSTs
+`download_file` so the browser saves the file locally), and **delete** (re-syncs the KB).
+Added 2026-06-22: the per-row download button (`downloadApprovedFile()`), reusing the
+existing `download_file` handler.
+
+## Configuration (`kb_processor.ini`)
+
+Shared by both scripts via `parse_ini_file(..., true)`. Sections: AWS creds/region, S3 bucket
++ folder names, Bedrock (`KB_PREFIX`, sync delays, conflict-retry tuning), `ALLOWED_EXTENSIONS`,
+AI settings (`AI_API_ENDPOINT`, `AI_MODEL_ID`, `AI_API_KEY`, temperature/tokens/timeout, the
+system prompt), and the `REPLACEMENTS[...]` mishear-correction map.
+
+> **⚠️ Security:** `kb_processor.ini` currently stores **live AWS access/secret keys and the AI
+> API key in plaintext, committed to the repo.** These should be rotated and moved out of source
+> (per-developer config or environment) — do not copy these values into other files or docs.
+
+## Gotchas
+
+- **Not framework code** — no `_.php` / `App_Framework_Sandbox`. Don't assume `App_*` helpers
+  are available here; everything goes through the AWS SDK and the ini.
+- **`general` KB is special-cased** everywhere — approve/delete skip its per-file sync; use the
+  batch "Sync All General" flow instead.
+- **Filename convention is load-bearing**: the `YYYY-MM-DD - Name.txt` pattern is the only thing
+  that distinguishes a meeting (gets AI cleaning) from a document (copied as-is).
+- Batch sync state lives in `$_SESSION['sync_progress']` and is driven by front-end polling
+  (`sync_next_kb`), one KB per request with a `KB_SYNC_DELAY_SECONDS` pause between.
+
+## Change history
+
+- 2026-06-22 — Un-retired the folder (`DOA_talos` → `talos`); added a per-file **Download**
+  button to the Approved Files Browser; initial knowledge capture of the uploader + processor.

package/knowledge/INDEX.md

@@ -10,7 +10,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 - **togaview** (TOGa View) — 6 doc(s) → [1.0/apps/togaview/INDEX.md](1.0/apps/togaview/INDEX.md)
 - **webhook** (Webhook) — 1 doc(s) → [1.0/apps/webhook/INDEX.md](1.0/apps/webhook/INDEX.md)
 - **walmarttechservices** (Walmart Tech Services) — 1 doc(s) → [1.0/apps/walmarttechservices/INDEX.md](1.0/apps/walmarttechservices/INDEX.md)
-- **test** (Test) — 10 doc(s) → [1.0/apps/test/INDEX.md](1.0/apps/test/INDEX.md)
+- **test** (Test) — 11 doc(s) → [1.0/apps/test/INDEX.md](1.0/apps/test/INDEX.md)
 
 ## 2.0 framework
 

package/knowledge/clients/compass-usa/INDEX.md

@@ -5,5 +5,6 @@
 | [Compass ASN → ItemFulfillment Auto-Creation](features/asn-to-item-fulfillment.md) | 2.0 | For Compass USA, posting an AdvanceShippingNotice (ASN) auto-creates the ItemFulfillment (IF) on the upstream SalesOrder. | _underscore/Model/Compass/AdvanceShippingNotice.php, _underscore/Model/Compass/PurchaseOrder.php, api2/Component/Api/Cxml/Cxml.php, dbchanges2/Client_Compass/2026-06-11 - AsnItemTrackingNumberAcl.sql, dbchanges2/Client_Compass/2026-06-15b - BackfillSA132781ItemFulfillmentTracking.sql, dbchanges2/Client_Compass/2026-06-16 - CleanupSA132763CrossLineTracking.sql, dbchanges2/Client_Compass/2026-06-16b - CleanupSA132743CrossLineTracking.sql, dbchanges2/Client_Compass/2026-06-16c - BackfillSA132763C40QYUCTracking.sql, dbchanges2/Client_Compass/2026-06-18a - CleanupSA132898DuplicateTracking.sql, dbchanges2/Client_Compass/2026-06-18b - CleanupSA132881DuplicateTracking.sql |
 | [Compass: Item-Fulfillment TableViews (for-sales-order-items & for-sales-orders, tracking via bridge)](features/item-fulfillment-tracking-tableview.md) | 2.0 | Two sibling Compass TableViews in `Client_Compass` display fulfilled items in toga2-supply, both driven by `TableViews` / `TableViewJoins` / `TableViewFields` c | dbchanges2/Client_Compass/2026-06-10 - ItemFulfillmentsForSalesOrderItemsTableView.sql, dbchanges2/Client_Compass/2026-06-11 - ItemFulfillmentsForSalesOrdersTableView.sql, dbchanges2/Client_Compass/2026-06-15a - FixItemFulfillmentTrackingNumberJoins.sql |
 | [Compass MITS PO → SO Item Linking](features/mits-po-to-so-item-linking.md) | 2.0 | MITS sends Compass inbound Purchase Orders (`POST /v2/purchase-orders`) against a Sales Order (`mitsSalesOrder`). | _underscore/Model/Compass/PurchaseOrder.php, worker/crons/toga2/compass/workflow/3a_import_office_depot_purchase_orders.php |
+| [Compass MITS PO Transmission to Vendors](features/mits-po-transmission-to-vendors.md) | 2.0 | The 1.0 worker cron `2_transmit_mits_purchase_orders_to_vendors.php` transmits Compass PurchaseOrders to their vendors (Office Depot, Strategic Systems, Compass | worker/crons/toga2/compass/workflow/2_transmit_mits_purchase_orders_to_vendors.php, library/app/client/compass.php |
 | [Compass USA](profile.md) | 2.0 | Compass USA is a TOGA client running a multi-tier supply-chain commerce operation. |  |
 | [Compass Order Lifecycle & Data-Integrity Invariants](workflows/order-lifecycle-and-data-integrity.md) | 2.0 | End-to-end map of how a Compass order flows through the `Client_Compass` (2.0) database and the **expected raw-data shape** at each link/ASN/IF level. |  |

package/knowledge/clients/compass-usa/features/mits-po-to-so-item-linking.md

@@ -13,6 +13,7 @@ files:
   - worker/crons/toga2/compass/workflow/3a_import_office_depot_purchase_orders.php
 related:
   - asn-to-item-fulfillment.md
+  - mits-po-transmission-to-vendors.md
 ---
 
 ## Summary

package/knowledge/clients/compass-usa/features/mits-po-transmission-to-vendors.md

@@ -0,0 +1,89 @@
+---
+title: Compass MITS PO Transmission to Vendors
+framework: "2.0"
+project: _Underscore
+client: compass-usa
+type: client-feature
+status: active
+updated: 2026-06-22
+owners: ["rgirish"]
+files:
+  - worker/crons/toga2/compass/workflow/2_transmit_mits_purchase_orders_to_vendors.php
+  - library/app/client/compass.php
+related:
+  - mits-po-to-so-item-linking.md
+  - order-lifecycle-and-data-integrity.md
+---
+
+## Summary
+The 1.0 worker cron `2_transmit_mits_purchase_orders_to_vendors.php` transmits Compass
+PurchaseOrders to their vendors (Office Depot, Strategic Systems, Compass, Presidio). Each
+vendor's `Integration` decides the channel: **CXML** (HTTP POST of a cXML OrderRequest) or
+**EMAIL** (a generated PDF emailed via `App_Email`). On success it writes "transmitted to
+Vendor" log rows and stamps `PurchaseOrders.dtSubmitted`.
+
+## Key files / entry points
+- `worker/crons/toga2/compass/workflow/2_transmit_mits_purchase_orders_to_vendors.php` — the cron.
+- `library/app/client/compass.php` — `App_Client_Compass` constants:
+  `CUSTOMER_ID__COMPASS = 2`, `CUSTOMER_ID__OFFICE_DEPOT = 1`, `CUSTOMER_ID__AGILANT = 3`;
+  `VENDOR_ID__OFFICE_DEPOT = 1`, `VENDOR_ID__STRATEGIC_SYSTEMS = 25`, `VENDOR_ID__COMPASS = 26`,
+  `VENDOR_ID__PRESIDIO = 4`.
+
+## How it works
+1. **Outer query** selects POs to send: `customerId = CUSTOMER_ID__COMPASS (2)`,
+   `dtSubmitted IS NULL`, and `vendorId IN (1, 25, 26, 4)`. (So it only ever transmits
+   `customerId = 2` POs — `customerId = 1` ODP-as-customer POs are out of scope here.)
+2. **Per PO**, an inner query loads `PurchaseOrderItems` and groups them by integration
+   (`VendorItems.overridePoSubmissionIntegrationId` else the vendor's
+   `defaultPoSubmissionIntegrationId`).
+3. **Per integration group** (the `foreach`): build + send the cXML POST or the PDF email,
+   then INSERT two `Logs_Compass.Record` rows
+   (`"Purchase Order <number> transmitted to Vendor via <CXML|EMAIL>. Part numbers: ..."`,
+   recordId 17 = PO and 14 = SO).
+4. **After the integration loop**, `UPDATE PurchaseOrders SET dtSubmitted = NOW()` and `sleep(10)`.
+
+Routing note: Office Depot (vendor 1) `defaultPoSubmissionIntegrationId = 2`, which is an
+**EMAIL** integration ("Office Depot / Yes") — so ODP POs are emailed PDFs by default, not
+cXML, unless a `VendorItem` overrides it. There is also a CXML integration (id 1,
+"Office Depot / No") that is NOT the default.
+
+## Data model
+- `Client_Compass.PurchaseOrders.dtSubmitted` — set by step 4; the outer query's `IS NULL`
+  guard means a PO is only ever picked up once.
+- `Logs_Compass.Record` — `recordId` 17 (PurchaseOrder) / 14 (SalesOrder), `primaryKeyId` =
+  the PO/SO `id`. The "transmitted to Vendor" note is the success marker.
+
+## Client variations
+Compass Canada has a parallel `compasscanada/workflow/2_transmit_...` keyed on
+`CUSTOMER_ID__COMPASS_CANADA`.
+
+## Gotchas / known issues
+- **Item-less PO is silently marked submitted without being sent (the "logs show not sent
+  but dtSubmitted is set" symptom).** The `dtSubmitted = NOW()` UPDATE runs **unconditionally
+  after** the per-integration `foreach`, not inside it. If a PO has **zero
+  `PurchaseOrderItems`**, the inner items query returns nothing, `$itemsToSubmitByIntegration`
+  stays empty, the `foreach` body never runs — so **nothing is transmitted and no
+  "transmitted to Vendor" `Record` row is written** — yet `dtSubmitted` still gets stamped.
+  The PO then never re-qualifies for transmission (`dtSubmitted IS NULL` no longer matches).
+  **Diagnostic signature:** PO has `dtSubmitted` set but there is NO
+  `Logs_Compass.Record` "transmitted to Vendor" note (only "received from MITS") → the PO had
+  no line items. Confirmed on MR241298 / PO 50298575-1 (2026-06-21): `dtSubmitted` 23:20:04,
+  zero PurchaseOrderItems, no transmit log, no outbound Api log.
+- **Root cause is upstream at MITS, not this cron.** MITS created the order header-only:
+  `POST /v2/sales-orders` and `POST /v2/purchase-orders` were both received with
+  `"salesOrderItems":[]` / `"purchaseOrderItems":[]` (both 201). The V2 API accepts an order
+  with no items (no empty-items rejection), TOGA stored exactly that, and the cron then masked
+  it. To confirm what MITS actually sent, read the inbound `requestPayload` in `Logs_Compass.Api`
+  (`direction = 'IN'`, `route` `/v2/purchase-orders` or `/v2/sales-orders`, `apiId = 2`).
+- **No try/catch around the log INSERTs or the dtSubmitted UPDATE**, and they write to
+  different DBs (`Logs_Compass` vs `Client_Compass`) — a `Logs_Compass` failure can still leave
+  `dtSubmitted` stamped without a log (separate, previously-observed failure mode).
+
+## Change history
+- 2026-06-22 — Documented the cron and the unconditional-`dtSubmitted` gotcha: item-less POs
+  get marked submitted without transmitting or logging; diagnosed via MR241298 inbound MITS
+  payload showing empty item arrays. No code change made (diagnosis only). (rgirish)
+
+## Related docs
+- [Compass MITS PO → SO Item Linking](mits-po-to-so-item-linking.md)
+- [Compass Order Lifecycle & Data-Integrity Invariants](../workflows/order-lifecycle-and-data-integrity.md)

package/knowledge/clients/compass-usa/workflows/order-lifecycle-and-data-integrity.md

@@ -13,6 +13,7 @@ related:
   - clients/compass-usa/features/asn-to-item-fulfillment.md
   - clients/compass-usa/features/mits-po-to-so-item-linking.md
   - clients/compass-usa/features/item-fulfillment-tracking-tableview.md
+  - clients/compass-usa/features/mits-po-transmission-to-vendors.md
 ---
 
 ## Summary

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.154",
+  "version": "1.0.156",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",