toga-ai 1.0.65 → 1.0.67

package/knowledge/clients/compass-usa/INDEX.md

@@ -6,3 +6,4 @@
 | [Compass: Item-Fulfillment TableViews (for-sales-order-items & for-sales-orders, tracking via bridge)](features/item-fulfillment-tracking-tableview.md) | 2.0 | Two sibling Compass TableViews in `Client_Compass` display fulfilled items in toga2-supply, both driven by `TableViews` / `TableViewJoins` / `TableViewFields` c | dbchanges2/Client_Compass/2026-06-10 - ItemFulfillmentsForSalesOrderItemsTableView.sql, dbchanges2/Client_Compass/2026-06-11 - ItemFulfillmentsForSalesOrdersTableView.sql |
 | [Compass MITS PO → SO Item Linking](features/mits-po-to-so-item-linking.md) | 2.0 | MITS sends Compass inbound Purchase Orders (`POST /v2/purchase-orders`) against a Sales Order (`mitsSalesOrder`). | _underscore/Model/Compass/PurchaseOrder.php |
 | [Compass USA](profile.md) | 2.0 | Compass USA is a TOGA client running a multi-tier supply-chain commerce operation. |  |
+| [Compass Order Lifecycle & Data-Integrity Invariants](workflows/order-lifecycle-and-data-integrity.md) | 2.0 | End-to-end map of how a Compass order flows through the `Client_Compass` (2.0) database and the **expected raw-data shape** at each link/ASN/IF level. |  |

package/knowledge/clients/compass-usa/workflows/order-lifecycle-and-data-integrity.md

@@ -0,0 +1,117 @@
+---
+title: Compass Order Lifecycle & Data-Integrity Invariants
+framework: "2.0"
+project: _Underscore
+client: compass-usa
+type: workflow
+status: active
+updated: 2026-06-12
+owners: ["jcardinal"]
+files: []
+related:
+  - clients/compass-usa/profile.md
+  - clients/compass-usa/features/asn-to-item-fulfillment.md
+  - clients/compass-usa/features/mits-po-to-so-item-linking.md
+  - clients/compass-usa/features/item-fulfillment-tracking-tableview.md
+---
+
+## Summary
+
+End-to-end map of how a Compass order flows through the `Client_Compass` (2.0) database and the
+**expected raw-data shape** at each link/ASN/IF level. This is the reference for diagnosing and
+retroactively repairing the order data, where 50+ historical bugs have left links, ASNs, and item
+fulfillments out of the shape the workflow should produce. All facts below were verified against the
+live `prod` / `Client_Compass` schema.
+
+## The chain (supply direction, by customerId / vendorId)
+
+```
+Compass customer SO  (SalesOrders.customerId = 2, number SA/MA/MR)
+  --SalesOrders_PurchaseOrders-->     MITS PO   (PurchaseOrders.customerId = 2, vendorId IN (1,26,25))
+    --PurchaseOrders_SalesOrders-->   Office Depot SO (customerId = 1)
+      --SalesOrders_PurchaseOrders--> Office Depot PO (vendorId = 1)
+        --PurchaseOrders_SalesOrders--> Agilant/NetSuite SO (customerId = 3)
+```
+
+- **SA** orders are user-created in Compass then sent to MITS; **MR** orders are created
+  retroactively from the inbound MITS PO (they do not originate as Compass-entered SOs); MA exists too.
+  Structurally MR uses the same bridges as SA.
+- One Compass SO item can split into multiple PO items (bundles / `c_isConfiguration`,
+  `parentSalesOrderItemId`, `bundleId`).
+
+## Item-link bridges & the purity rule
+
+- `SalesOrderItems_PurchaseOrderItems(salesOrderItemId, purchaseOrderItemId)` — a SO item → its PO item.
+- `PurchaseOrderItems_SalesOrderItems(purchaseOrderItemId, salesOrderItemId)` — a PO item → its downstream SO item.
+- Item identity: `PurchaseOrderItems.vendorItemId → VendorItems.itemId`. **`PurchaseOrderItems` has NO
+  `salesOrderItemId` column** — POI↔SOI is resolvable ONLY through these two bridges.
+- **PURE link ⇔ `SalesOrderItems.itemId = linked POI's VendorItems.itemId`. SPURIOUS ⇔ they differ.**
+  Spurious links (the lineNumber-match bug, see mits-po-to-so-item-linking) cause phantom over-fulfillment.
+- **Many-to-one is legitimate**, not spurious — several SOIs may link to one POI
+  (e.g. `PurchaseOrderItems_SalesOrderItems.purchaseOrderItemId = 174808` has 6 SOIs). Never treat
+  multiplicity as corruption; the only spuriousness test is the itemId mismatch.
+
+## Two fulfillment flows (different expected IF shapes)
+
+- **Flow A — direct ASN→IF** (MITS PO vendor in {25,26}, e.g. Strategic Systems; no Agilant leg):
+  an ASN against the MITS PO produces an ItemFulfillment on the Compass SO. Serialized units come
+  from the ASN units.
+- **Flow B — Office Depot / NetSuite** (MITS PO vendor 1, with ODP SO/PO + Agilant SO): the
+  fulfillment **originates on the Agilant IF** (created by NetSuite sync; `c_netsuiteInternalItemFulfillmentId`
+  set) and **mirrors UP the chain**. Units come from NetSuite inventory assignment, NOT the ASN
+  (ODP ASN units are frequently tracking-only, `unitId` NULL).
+
+### Upstream mirror direction (verified from real rows)
+The **Compass-SO IF is the TOP** (`upstreamItemFulfillmentId = NULL`). The ODP IF's
+`upstreamItemFulfillmentId` points to the Compass IF; the Agilant IF's points to the ODP IF. The
+back-link chain climbs **Agilant → ODP → Compass**. Item back-links use
+`ItemFulfillmentItems.upstreamItemFulfillmentItemId` the same direction and may be many-downstream →
+one-upstream for bundles.
+
+## Expected raw-data invariants (for detect-and-repair)
+
+1. **Links:** every SOI that should be procured has exactly one PURE bridge to the matching POI on the
+   correct PO; spurious (itemId-mismatch) bridges should not exist; the full header chain
+   (SalesOrders_PurchaseOrders / PurchaseOrders_SalesOrders) should be intact at each present layer.
+2. **Never over-fulfill:** `SUM(ItemFulfillmentItems.quantity)` per SOI must be `≤ SalesOrderItems.quantity`.
+   A shipped POI fanning out to many linked SOIs must not produce IF items totaling more than ordered.
+3. **ASN/IF tracking is maintained at all three levels and rolls up:** a tracking number on a unit must
+   also be linked at the item level and the header level — on BOTH the ASN side
+   (`AdvanceShippingNoticeItemUnits_TrackingNumbers` → `AdvanceShippingNoticeItems_TrackingNumbers` →
+   `AdvanceShippingNotices_TrackingNumbers`) and the IF side
+   (`ItemFulfillmentItemUnits_TrackingNumbers` → `ItemFulfillmentItems_TrackingNumbers` →
+   `ItemFulfillments_TrackingNumbers`). Always link existing `TrackingNumbers`; never create them.
+4. **Legacy stage:** a shipped IF (has items) with `itemFulfillmentStageId IS NULL` should be set to 3
+   ("Shipped"). Stage 3 = Shipped; stages 1/2 are unused in prod.
+5. **Mirror back-links:** every IF on a non-source SO (ODP/Agilant) should have
+   `upstreamItemFulfillmentId` set, and each of its items `upstreamItemFulfillmentItemId` set.
+
+## Verified schema corrections (older code/scripts assume these wrongly)
+- **`ItemFulfillmentPackages` does NOT exist** — header IF tracking is `ItemFulfillments_TrackingNumbers`.
+- `ItemFulfillmentItemUnits` has **no `trackingNumberId`** — unit tracking is the bridge
+  `ItemFulfillmentItemUnits_TrackingNumbers`.
+- ASN unit table is `AdvanceShippingNoticeItemUnits` (`unitId` nullable = tracking-only).
+- `ItemFulfillments.number` is UNIQUE NOT NULL; `dateItemFulfillment` NOT NULL.
+
+## Systems involved
+`Client_Compass` (2.0) tables; the 1.0 worker crons under `worker/crons/toga2/compass/workflow/`
+(transmit-to-MITS, transmit-to-vendors, ODP EDI import, Strategic Systems ASN import, NetSuite sync);
+`api2` cXML ShipNotice handler; the `_underscore` recursive item-fulfillment engine.
+
+## Retrofix tooling
+A standalone repair script lives in the worker 1.0 test area (`test/@jeff/compass/retrofix.php`,
+machine-local path tracked in dev memory). It sweeps Compass SOs newest-first, detects deviations from
+the invariants above, and repairs via direct SQL on `db_prod2_compass`. Safety rules baked in: rehearsal
+mode (logs intended writes, rolls back), per-phase gating, a hard block on any write to the
+`TrackingNumbers` table, and the never-over-fulfill cap. It is a diagnostic/repair tool, not part of the
+runtime workflow.
+
+## Edge cases & escalation
+- Incomplete orders (no PO/ASN/Agilant leg yet) are normal — repair only the layers that exist.
+- Logs for confirming original payloads: `Logs_Compass.Api` (inbound MITS POs apiId=2; inbound cXML
+  ShipNotices), legacy `Logs.API` (outbound transmits), legacy `Logs.FileLog` (`ODP_EDI`, ~30-day retention).
+- High-multiplier over-fulfillment (5×–20×) does not fit the split-PO spurious-link pattern — separate cause.
+
+## Change history
+- 2026-06-12 — Documented full lifecycle, both fulfillment flows, verified upstream-mirror direction,
+  the expected-shape invariants, and the retrofix tool (jcardinal)

package/knowledge/clients/tow-foundation/features/receipt-processing.md

@@ -5,7 +5,7 @@ project: Worker
 client: tow-foundation
 type: client-feature
 status: active
-updated: 2026-06-11
+updated: 2026-06-12
 owners: ["rgirish"]
 files:
   - worker2/Worker/Client/TowFoundation.php
@@ -35,15 +35,13 @@ Optional filter parameters: `limit` (int), `year` (string e.g. `"2026"`), `perso
 ### SharePoint folder structure
 ```
 Credit Card Receipts/
+  3. QB Excel/                 ← generated Excel files for ALL persons uploaded here after each run
   {Person}/
     {Year}/
       {BillingCycleFolder}/   ← receipt files live here (e.g. "Amex ending in 06-03-2026")
       {Person} reports/        ← .xlsx billing statements for cross-verification
-      Archive/
-        {Person}/              ← successfully processed receipts land here (renamed)
-        exception/
-          {Person}/            ← failed receipts land here (original name preserved)
-      3. QB Excel/             ← generated Excel files uploaded here after each run
+      Archive/                 ← successfully processed receipts land here (renamed, flat)
+      Archive/exception/       ← failed receipts land here (original name preserved, flat)
 ```
 
 ### Processing flow
@@ -55,18 +53,18 @@ Credit Card Receipts/
    and parsed upfront via PhpSpreadsheet (auto-detect header row by scanning for
    description/amount keywords)
 5. **Pass 1 — Extract** — for each receipt:
-   - Enforce size limit (4 MB images, 10 MB documents)
+   - Enforce type + size limit: unsupported mime types (`application/octet-stream`) throw immediately; 4 MB cap for images, 10 MB for documents
    - Download file bytes from SharePoint
    - POST to Talos AI `/api/ai/generate` → structured `{vendor_name, invoice_date, total, payment_memo, category, ...}`
    - Cross-verify amount ± $0.01 against parsed statement; if matched, override `payment_memo` with statement description
    - Build base filename (no suffix yet) and store in `$pendingRenames`
-   - On extract failure → move to `Archive/exception/{Person}/` immediately
+   - On extract failure → move to `Archive/exception/` immediately
 6. **Collision detection** — count base name occurrences across all pending renames
 7. **Pass 2 — Move** — for each pending rename:
    - If base name appears more than once, assign `_a`, `_b`, `_c`... suffix to **all** colliding files (including the first)
-   - PATCH SharePoint to rename + move to `Archive/{Person}/`
+   - PATCH SharePoint to rename + move to `Archive/` (flat — no person subfolder)
 8. **Excel generation** — one `.xlsx` per person (PhpSpreadsheet); columns: Row #, Account Name, QB Vendor, Payment Amount, Date, Payment Method, Payment Memo, QB Description, Class, Category, Payment Account, Ref No.
-9. **SharePoint upload** — each Excel uploaded to `{Person}/{Year}/3. QB Excel/` via Graph API PUT
+9. **SharePoint upload** — each Excel uploaded to root-level `Credit Card Receipts/3. QB Excel/` via Graph API PUT (shared folder for all persons)
 10. **Summary email** — sent with Excel files attached; To: Jheanelle, CC: Magdalena, BCC: devteam@togatech.com
 
 ### Renamed file format
@@ -124,7 +122,18 @@ Fatal errors send only to `NOTIFY_EMAIL_DEV` (no CC/BCC).
 - **No client DB** — there is no audit trail in MySQL. All state lives in SharePoint folder
   structure and email. If a run is interrupted mid-way, some receipts may be archived
   without a corresponding Excel row.
+- **Unsupported file types** — macOS `.textClipping` files and any file resolving to
+  `application/octet-stream` are rejected with a `ReceiptProcessingException` at the
+  size-check stage (before download). They land in `Archive/exception/` and appear in the
+  summary email so Jheanelle can clean them up manually.
+- **Archive folders are flat** — processed files land directly in `Archive/` and failed
+  files in `Archive/exception/` with no extra person subfolder. The Archive is already
+  scoped inside `{Person}/{Year}/` so the extra nesting is redundant.
+- **QB Excel goes to the root `3. QB Excel/` folder** — all persons' Excels land in
+  `Credit Card Receipts/3. QB Excel/` (not inside each person's subfolder). This folder
+  exists at the root of `Credit Card Receipts/` alongside the person folders.
 
 ## Change history
 
+- 2026-06-12 — QB Excel upload moved to root-level `3. QB Excel/`; Archive/exception folders flattened (no person subfolder); unsupported mime types (`application/octet-stream`) now throw ReceiptProcessingException instead of being silently skipped (rgirish)
 - 2026-06-11 — Added CC (Magdalena), BCC (devteam@togatech.com), per-cardholder Archive subfolders, Excel upload to "3. QB Excel" SharePoint folder, two-pass rename with alphabetical duplicate suffix, rename format changed to `YYYY.MM.DD Full Name_Vendor_Amount` (rgirish)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.65",
+  "version": "1.0.67",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",