toga-ai 1.0.155 → 1.0.157

package/knowledge/1.0/apps/test/INDEX.md

@@ -9,6 +9,7 @@
 | [Developer Generators (password, UUID)](features/dev-generators.md) | Two tiny **1.0 `App_` framework** convenience scripts for everyday developer needs. | test/team/generate_password.php, test/team/uuid.php |
 | [Forecast vs NetSuite Discrepancy Analysis](features/forecast-netsuite-discrepancy-analysis.md) | `team/forecast-netsuite/discrepancy_analysis.php` detects discrepancies between our **Forecast database** and **NetSuite** (the source of truth for all sales da | test/team/forecast-netsuite/discrepancy_analysis.php |
 | [TableView Builder (2.0 TableViews SQL generator)](features/tableview-builder.md) | `team/tableViewBuilder/` generates SQL `INSERT` statements for the **2.0 `TableViews`**, `TableViewFields`, and `TableViewJoins` tables from a plain SQL `SELECT | test/team/tableViewBuilder/TableViewGenerator.php, test/team/tableViewBuilder/index.php, test/team/tableViewBuilder/Instructions.md |
+| [Talos Knowledge Base Pipeline (Uploader + Processor)](features/talos-kb-pipeline.md) | `team/talos/` holds the two-script web tooling that feeds the **TOGa Talos** (TOGa IQ) AI knowledge bases. | test/team/talos/kb_uploader.php, test/team/talos/kb_processor.php, test/team/talos/kb_processor.ini |
 | [TOGa 2.0 Client Onboarding SQL Generator](features/toga2-client-onboarding-sql.md) | `team/generate_toga2_onboarding_sql.php` generates the SQL needed to **onboard a new client** onto the 2.0 platform. | test/team/generate_toga2_onboarding_sql.php |
 | [TOGa 2.0 User Cross-Client Access SQL Generator](features/toga2-user-cross-client-access-sql.md) | `team/generate_toga2_user_access_sql.php` generates SQL to grant an existing 2.0 user from a **home client** access to a **cross client**. | test/team/generate_toga2_user_access_sql.php |
 | [URL & Domain Markdown Document Builder](features/url-domain-markdown-document.md) | `team/build_url_domain_markdown_document.php` generates a **markdown document of our URLs and domains** by pulling environments and domains from the 2.0 platfor | test/team/build_url_domain_markdown_document.php |

package/knowledge/1.0/apps/test/architecture.md

@@ -6,7 +6,7 @@ project: Test
 client: shared
 type: architecture
 status: active
-updated: 2026-06-16
+updated: 2026-06-22
 owners: [jcardinal]
 files:
   - test/team/
@@ -20,6 +20,7 @@ related:
   - ./features/active-directory-auth-test.md
   - ./features/url-domain-markdown-document.md
   - ./features/dev-generators.md
+  - ./features/talos-kb-pipeline.md
 ---
 
 ## Summary
@@ -81,9 +82,12 @@ manually against the target database; they do not execute changes themselves.
 | `generate_toga2_onboarding_sql.php` | 1.0 | 2.0 client DBs | [toga2-client-onboarding-sql](./features/toga2-client-onboarding-sql.md) |
 | `generate_toga2_user_access_sql.php` | 1.0 | 2.0 client DBs | [toga2-user-cross-client-access-sql](./features/toga2-user-cross-client-access-sql.md) |
 | `generate_password.php`, `uuid.php` | 1.0 | dev utility | [dev-generators](./features/dev-generators.md) |
+| `talos/kb_uploader.php`, `talos/kb_processor.php` | standalone | S3 + AWS Bedrock KBs | [talos-kb-pipeline](./features/talos-kb-pipeline.md) |
 
-**Retired / not documented:** `team/DOA_gitbook/` and `team/DOA_talos/` are being retired.
+**Retired / not documented:** `team/DOA_gitbook/` is being retired.
 
 ## Change history
 
 - 2026-06-16 — Initial capture of the `team/` folder. Registered repo in `registry.json`.
+- 2026-06-22 — Un-retired `team/DOA_talos/` → `team/talos/`; documented its uploader +
+  processor in [talos-kb-pipeline](./features/talos-kb-pipeline.md).

package/knowledge/1.0/apps/test/features/talos-kb-pipeline.md

@@ -0,0 +1,145 @@
+---
+title: Talos Knowledge Base Pipeline (Uploader + Processor)
+framework: "1.0"
+repo: test
+project: Test
+client: shared
+type: feature
+status: active
+updated: 2026-06-22
+owners: [jcardinal]
+files:
+  - test/team/talos/kb_uploader.php
+  - test/team/talos/kb_processor.php
+  - test/team/talos/kb_processor.ini
+related:
+  - ../architecture.md
+---
+
+## Summary
+
+`team/talos/` holds the two-script web tooling that feeds the **TOGa Talos** (TOGa IQ) AI
+knowledge bases. Content flows through S3 folders and ends up in **AWS Bedrock** knowledge
+bases, which back the internal AI agent that answers developer questions.
+
+- **`kb_uploader.php`** — the *intake* page. A user pastes a meeting transcript or uploads a
+  document; it lands in the S3 `upload/` folder of a chosen knowledge base.
+- **`kb_processor.php`** — the *review/approve* console. A reviewer cleans transcripts (AI +
+  automated text rules), approves files into the `approved/` folder, writes a Bedrock metadata
+  sidecar, archives the raw source, and triggers a Bedrock ingestion (sync) job.
+
+Both are **standalone PHP pages** — they do **not** bootstrap the 1.0 `App_` framework. They
+load the AWS SDK from `../../vendor/autoload.php` (Composer `aws/aws-sdk-php`) and read all
+settings from a shared `kb_processor.ini`. Target PHP 7.2+.
+
+> History: this folder was previously named `DOA_talos` (the `DOA_` prefix marked it as
+> deprecated-but-live). It was un-retired and renamed to `talos` on 2026-06-22.
+
+## S3 layout
+
+All paths are relative to `S3_BUCKET` (`togaiq`) under `S3_ROOT_PATH` (`development-team/`).
+Each **knowledge base** is a folder one level under the root (e.g. `development-team/<kb>/`),
+discovered by listing common prefixes. Within each KB:
+
+| Folder (config key) | Default | Role |
+|---|---|---|
+| `S3_UPLOAD_FOLDER`   | `upload`   | Raw intake — where the uploader drops files |
+| `S3_APPROVED_FOLDER` | `approved` | Reviewed/approved content (+ `.metadata.json` sidecars) — the Bedrock data source |
+| `S3_ARCHIVE_FOLDER`  | `archive`  | Raw source files after they've been approved |
+
+A KB named **`general`** is treated specially: it is never auto-synced on approve/delete;
+instead it is synced in bulk via the "Sync All General" flow.
+
+## kb_uploader.php — intake
+
+Single page with two POST forms (toggled by a client-side mode switcher):
+
+1. **Meeting Transcription** (`submit_meeting`) — fields: date, meeting name, KB, raw
+   transcript. The meeting name is slash-sanitized; the transcript's repeated blank lines are
+   collapsed; the file is named **`YYYY-MM-DD - {MEETING_NAME}.txt`** and `putObject`'d to
+   `<root>/<kb>/upload/`. This filename pattern is what the processor later recognizes as a
+   meeting (`isMeetingTranscript()` regex).
+2. **File Upload** (`submit_file`) — fields: KB, document name, file. Extension is validated
+   against `ALLOWED_EXTENSIONS`; the file is stored as `{documentName}.{ext}` in the same
+   `upload/` folder via `putObject` with `SourceFile`.
+
+Knowledge-base options come from `getKnowledgeBases()`. All output is escaped with
+`htmlspecialchars()`.
+
+## kb_processor.php — review & approve
+
+The page lists everything in every KB's `upload/` folder, split into **meetings** (filename
+matches `^\d{4}-\d{2}-\d{2} - .+\.txt$`) and **documents**. It exposes AJAX actions (POST
+`action=...`, JSON responses) plus an **Approved Files Browser** modal. Key actions:
+
+| `action` | Purpose |
+|---|---|
+| `get_file_content` | Return an S3 object's text (transcript editor / file viewer) |
+| `prepare_transcript` | AI clean → automated clean → prepend `date - title` header |
+| `save_to_kb` | Approve a file (see flow below) |
+| `download_file` | Stream any S3 key to the browser as an attachment (`Content-Disposition`) |
+| `delete_file` / `delete_approved_file` | Delete from `upload/` / `approved/` (latter re-syncs the KB) |
+| `list_approved_files` / `count_approved_files` | Populate the Approved Files Browser |
+| `start_sync_all_general` / `sync_next_kb` / `get_sync_progress` | Session-backed batch sync of every KB's `general` data source, polled from the front-end |
+
+### Transcript cleaning (`prepare_transcript`)
+
+1. **AI cleaning** — `callAICleaningAPI()` POSTs to `AI_API_ENDPOINT` (the TOGa IQ AI service,
+   model `AI_MODEL_ID`, Bedrock Claude Haiku 4.5) with an `output_schema` requesting a single
+   `knowledge_doc` string. The lengthy `AI_SYSTEM_PROMPT` enforces a strict lossless plain-text
+   format (OVERVIEW + topical sections, verbatim code blocks, PascalCase tables / camelCase
+   columns). Retries twice with backoff on network errors and HTTP 429/5xx.
+2. **Automated cleaning** — `applyAutomatedCleaning()` strips markdown artifacts (backticks,
+   `> ` quoting, `# ` headings), normalizes blank lines around headings/lists, removes
+   `:contentReference[...]` markers, and applies the `REPLACEMENTS[...]` map from the ini
+   (mishear corrections, e.g. `Talus → Talos`). Replacement keys support `:c` (case-sensitive)
+   and `:w` (whole-word) flags.
+3. For meeting files, a `YYYY-MM-DD - Title` header is prepended.
+
+### Approve flow (`save_to_kb`)
+
+1. Write the approved content to `<root>/<newKb>/approved/<filename>` — uploading the cleaned
+   text for meetings, or copying the original S3 object directly for documents.
+2. Write a **Bedrock metadata sidecar** `<filename>.metadata.json`
+   (`buildApprovedMetadataJson()`): `metadataAttributes` with `slug`, `client` (the KB name),
+   `state=approved`, `approved_at`, `filename`, each `includeForEmbedding: false`.
+3. **Archive** the raw source: move `upload/<file>` → `archive/<file>` (copy + delete).
+4. **Sync** the Bedrock knowledge base unless the KB is `general`:
+   `syncKnowledgeBaseDataSource()` resolves the KB by name (`KB_PREFIX + kb`, e.g.
+   `development-team-<kb>`), finds the matching data source, and `startIngestionJob()`s it with
+   a UUID client token. Retries on `ConflictException` / throttling up to `KB_CONFLICT_MAX_RETRIES`.
+
+### Approved Files Browser
+
+Modal listing the chosen KB's `approved/` files with size/modified, plus per-row **view**
+(text types open in a viewer via `get_file_content`), **download** (⬇️ — form-POSTs
+`download_file` so the browser saves the file locally), and **delete** (re-syncs the KB).
+Added 2026-06-22: the per-row download button (`downloadApprovedFile()`), reusing the
+existing `download_file` handler.
+
+## Configuration (`kb_processor.ini`)
+
+Shared by both scripts via `parse_ini_file(..., true)`. Sections: AWS creds/region, S3 bucket
++ folder names, Bedrock (`KB_PREFIX`, sync delays, conflict-retry tuning), `ALLOWED_EXTENSIONS`,
+AI settings (`AI_API_ENDPOINT`, `AI_MODEL_ID`, `AI_API_KEY`, temperature/tokens/timeout, the
+system prompt), and the `REPLACEMENTS[...]` mishear-correction map.
+
+> **⚠️ Security:** `kb_processor.ini` currently stores **live AWS access/secret keys and the AI
+> API key in plaintext, committed to the repo.** These should be rotated and moved out of source
+> (per-developer config or environment) — do not copy these values into other files or docs.
+
+## Gotchas
+
+- **Not framework code** — no `_.php` / `App_Framework_Sandbox`. Don't assume `App_*` helpers
+  are available here; everything goes through the AWS SDK and the ini.
+- **`general` KB is special-cased** everywhere — approve/delete skip its per-file sync; use the
+  batch "Sync All General" flow instead.
+- **Filename convention is load-bearing**: the `YYYY-MM-DD - Name.txt` pattern is the only thing
+  that distinguishes a meeting (gets AI cleaning) from a document (copied as-is).
+- Batch sync state lives in `$_SESSION['sync_progress']` and is driven by front-end polling
+  (`sync_next_kb`), one KB per request with a `KB_SYNC_DELAY_SECONDS` pause between.
+
+## Change history
+
+- 2026-06-22 — Un-retired the folder (`DOA_talos` → `talos`); added a per-file **Download**
+  button to the Approved Files Browser; initial knowledge capture of the uploader + processor.

package/knowledge/2.0/apps/_underscore/INDEX.md

@@ -8,3 +8,4 @@
 | [Per-Client Database Connections & the Local Logs Trap](features/per-client-database-connections.md) | When `_underscore` serves a request for a client it opens **three distinct per-client database connections**, not one. | _underscore/Database.php, _underscore/ApiRequest.php, _underscore/Model/Client/Logs/Api.php |
 | [Recursive Item Fulfillments (upstream mirroring)](features/recursive-item-fulfillments.md) | In a multi-tier supply chain a sales order (SO) spawns a purchase order (PO) that becomes another SO downstream, and so on. | _underscore/Model/Client/ItemFulfillment.php, _underscore/Model/Client/ItemFulfillmentItem.php, _underscore/Model/Client/ItemFulfillmentItemUnit.php, _underscore/Model/Client/ItemFulfillmentPackage.php, _underscore/Model/Compass/AdvanceShippingNotice.php, dbchanges2/Core/2026-02-13 - 75601 - RecursiveItemFulfillmentCreation.sql, dbchanges2/Core/2026-06-04 - RecursiveItemFulfillmentPut.sql |
 | [Tracking-Number Bridge Migration (ASN / Item Fulfillment / Item Receipt)](features/tracking-number-bridges.md) | Shipment tracking numbers used to live as **scalar FK columns** (`trackingNumberId`, `returnTrackingNumberId`) directly on the lowest-level "unit"/"item" tables | api2/Component/Api/V2/V2.php, _underscore/Model/Client/AdvanceShippingNoticeItemUnit.php, _underscore/Model/Client/AdvanceShippingNoticeItemUnits/TrackingNumber.php, _underscore/Model/Client/ItemFulfillmentItemUnits/TrackingNumber.php, _underscore/Model/Client/ItemFulfillment.php, _underscore/Model/Prudential/AdvanceShippingNotice.php, _underscore/Model/Compass/AdvanceShippingNotice.php, _underscore/Trait/Netsuite/ItemFulfillment.php, api2/Component/Api/Cxml/Cxml.php, dbchanges2/Client/2026-06-10 - TrackingNumberBridges.sql, dbchanges2/Core/2026-06-10 - TrackingNumberBridges.sql, dbchanges2/Client_Prudential/2026-06-15 - ItemFulfillmentTrackingNumberAclLogicGroups.sql, dbchanges2/Client_Quad/2026-06-18a - ItemFulfillmentItemReceiptTrackingNumberAclLogicGroups.sql, dbchanges2/Client_Nychh/2026-06-19b - ItemFulfillmentItemReceiptTrackingNumberAclLogicGroups.sql, dbchanges2/Client_Nychh/2026-06-19a - FixItemFulfillmentTableViewTrackingAndRoot.sql, dbchanges2/Client_Quad/2026-06-19a - FixItemFulfillmentTableViewTrackingAndRoot.sql |
+| [Units for Items for Purchase Orders — Data Structure](features/units-for-items-for-purchase-orders.md) | Describes how unit (serialized inventory) data is linked to sales-order and purchase-order line items behind the `units-for-items-for-purchase-orders` TableView |  |

package/knowledge/2.0/apps/_underscore/features/units-for-items-for-purchase-orders.md

@@ -0,0 +1,92 @@
+---
+title: Units for Items for Purchase Orders — Data Structure
+framework: "2.0"
+repo: _underscore
+project: _Underscore
+client: shared
+type: feature
+status: active
+updated: 2026-06-22
+owners: ["bala"]
+files: []
+related: []
+---
+
+## Summary
+Describes how unit (serialized inventory) data is linked to sales-order and purchase-order
+line items behind the `units-for-items-for-purchase-orders` TableView. A unit only becomes
+visible in this view if it can be walked from the sales-order item out to a unit record. There
+are **two independent linkage paths** in the schema, and the view traverses only one of them —
+which is the root cause of "units not showing" cases.
+
+## How it works
+A `Unit` can be tied to an order line through either of two chains:
+
+**1. Receipt path (what this view follows):**
+```
+SalesOrderItems
+  -> SalesOrderItems_PurchaseOrderItems   (item-level bridge: salesOrderItemId, purchaseOrderItemId)
+  -> PurchaseOrderItems                   (PurchaseOrderItems.id = bridge.purchaseOrderItemId)
+  -> ItemReceiptItems                     (ItemReceiptItems.purchaseOrderItemId = PurchaseOrderItems.id)
+  -> ItemReceiptItemUnits                 (ItemReceiptItemUnits.itemReceiptItemId = ItemReceiptItems.id)
+  -> Units                                (ItemReceiptItemUnits.unitId)
+```
+
+**2. Fulfillment path (direct; NOT followed by this view):**
+```
+SalesOrderItems
+  -> ItemFulfillmentItems                 (ItemFulfillmentItems.salesOrderItemId = SalesOrderItems.id)
+  -> ItemFulfillmentItemUnits             (ItemFulfillmentItemUnits.itemFulfillmentItemId = ItemFulfillmentItems.id)
+  -> Units                                (ItemFulfillmentItemUnits.unitId)
+```
+
+The receipt path requires both the item-level bridge **and** an item receipt against the
+bridged purchase-order line. The fulfillment path reaches `SalesOrderItems` directly by
+`salesOrderItemId` with no purchase order or bridge involved.
+
+## Data model
+Header-level vs item-level bridges, and the level each transaction is anchored at:
+
+- **`PurchaseOrders_SalesOrders`** — *header* bridge: links a customer PO to a sales order
+  (`purchaseOrderId`, `salesOrderId`).
+- **`SalesOrderItems_PurchaseOrderItems`** — *item* bridge: links a sales-order line to a
+  vendor purchase-order line (`salesOrderItemId`, `purchaseOrderItemId`). This is the bridge
+  the receipt path depends on.
+- **Item receipts are PO-anchored only**: `ItemReceipts.purchaseOrderId`,
+  `ItemReceiptItems.purchaseOrderItemId` — there is **no sales-order FK on a receipt**. The
+  only way a received unit reaches a sales order is through the item bridge.
+- **Item fulfillments are SO-anchored**: `ItemFulfillments.salesOrderId`,
+  `ItemFulfillmentItems.salesOrderItemId`.
+- Both customer POs and vendor POs live in the same `PurchaseOrders` table, distinguished by
+  which bridge references them (header bridge = customer PO; item bridge via
+  `PurchaseOrderItems` = vendor PO).
+- `Items.inventoryType` governs whether units exist at all (see Gotchas).
+
+## Client variations
+None — uniform. The linkage structure is framework-level and identical across clients.
+
+## Gotchas / known issues
+- **The view only walks the receipt path.** A line whose units exist solely via the
+  fulfillment path (shipped from stock, never received against the bridged PO line) shows
+  **0 units** even though units physically exist. No data backfill can surface these — only a
+  resolver/view change that also walks the fulfillment path will.
+- **Missing item bridge → no units**, even when units were received. Without a
+  `SalesOrderItems_PurchaseOrderItems` row, the receipt path has no starting link.
+- **A present bridge is not sufficient.** If the bridged PO line has no
+  `ItemReceiptItemUnits`, the receipt path still resolves to nothing.
+- **`inventoryType` determines unit existence**: `SERIALIZED` and `HYBRID` items can produce
+  unit records; `NON_SERIALIZED` (bulk/consumable) never do. `HYBRID` items fulfilled as a
+  plain quantity (no serial capture) also produce no units — legitimately blank.
+- **Receipt-path counts are PO-line-wide.** Units surfaced reflect everything received against
+  the bridged PO line, which may exceed or differ from what was actually fulfilled to a given
+  sales order (over/under-attribution), because attribution is by PO line, not by the
+  fulfillment that shipped the unit.
+
+## Change history
+- 2026-06-22 — Documented the two-path units linkage model behind the
+  units-for-items-for-purchase-orders view and why the receipt-only traversal hides
+  fulfillment-only units. (bala)
+
+## Related docs
+- [[recursive-item-fulfillments]]
+- [[tracking-number-bridges]]

package/knowledge/INDEX.md

@@ -10,11 +10,11 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 - **togaview** (TOGa View) — 6 doc(s) → [1.0/apps/togaview/INDEX.md](1.0/apps/togaview/INDEX.md)
 - **webhook** (Webhook) — 1 doc(s) → [1.0/apps/webhook/INDEX.md](1.0/apps/webhook/INDEX.md)
 - **walmarttechservices** (Walmart Tech Services) — 1 doc(s) → [1.0/apps/walmarttechservices/INDEX.md](1.0/apps/walmarttechservices/INDEX.md)
-- **test** (Test) — 10 doc(s) → [1.0/apps/test/INDEX.md](1.0/apps/test/INDEX.md)
+- **test** (Test) — 11 doc(s) → [1.0/apps/test/INDEX.md](1.0/apps/test/INDEX.md)
 
 ## 2.0 framework
 
-- **_underscore** (_Underscore) _(framework core)_ — 9 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
+- **_underscore** (_Underscore) _(framework core)_ — 10 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
 - **worker2** (Worker) — 8 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
 - **api2** (API) — 4 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
 - **dbchanges2** (Database Changes) _(framework core)_ — 1 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.155",
+  "version": "1.0.157",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",