toga-ai 1.0.204 → 1.0.205

package/knowledge/clients/tow-foundation/features/receipt-processing.md

@@ -5,7 +5,7 @@ project: Worker
 client: tow-foundation
 type: client-feature
 status: active
-updated: 2026-06-23
+updated: 2026-06-25
 owners: ["rgirish"]
 files:
   - worker2/Worker/Client/TowFoundation.php
@@ -57,8 +57,11 @@ Credit Card Receipts/
 5. **Pass 1 — Extract** — for each receipt:
    - Enforce type + size limit: unsupported mime types (`application/octet-stream`) throw immediately; 4 MB cap for images, 10 MB for documents
    - Download file bytes from SharePoint
-   - POST to Talos AI `/api/ai/generate` → structured `{vendor_name, invoice_date, total, payment_memo, category, ...}`
+   - **`.docx` files bypass Talos** and route to `extractDocxData()` (Talos returns HTTP 500 on a `.docx` MIME type — it only accepts PDF/image). That method opens the docx as a ZIP, extracts `word/document.xml`, strips tags, and regex-parses `MM/DD/YY: description` lines into `line_items[]`
+   - All other types POST to Talos AI `/api/ai/generate` → structured `{vendor_name, invoice_date, total, payment_memo, category, ...}`
+   - **Year guard on AI dates** — if Talos returns an `invoice_date` whose year is more than 1 year from the current year (AI hallucination on two-digit year inputs, e.g. `5/13/76` → 1976, `5/28/28` → 2028), the year is clamped to the current year while month/day are preserved
    - Cross-verify amount ± $0.01 against parsed statement; if matched, override `payment_memo` with statement description
+   - **Zero-total fallback** — if the extracted/docx total is 0, `lookupStatementTotalByVendor()` derives the amount from the billing statement (see *Statement structure* below)
    - Build base filename (no suffix yet) and store in `$pendingRenames`
    - On extract failure → move to `Archive/exception/` immediately
 6. **Deduplication** — after Pass 1, receipts sharing identical `(personName, invoice_date, vendor_name, total)` are deduplicated; only the first is kept, the rest are logged as `DUPLICATE skipped` and dropped. This catches the same receipt uploaded twice to SharePoint.
@@ -66,6 +69,7 @@ Credit Card Receipts/
 8. **Pass 2 — Move** — for each pending rename:
    - If base name appears more than once, assign `_a`, `_b`, `_c`... suffix to **all** colliding files (including the first)
    - PATCH SharePoint to rename + move to `Archive/` (flat — no person subfolder)
+   - **Multi-entry docx expansion** — a `.docx` whose `extractDocxData()` produced multiple `line_items[]` (e.g. a subway-rides document with 14 dated trip entries) is expanded into one Excel row per entry
 8. **Excel generation** — one `.xlsx` per person (PhpSpreadsheet); columns: Row #, Account Name, QB Vendor, Payment Amount, Date, Payment Method, Payment Memo, QB Description, Class, Category, Payment Account, Ref No.
 9. **SharePoint upload** — each Excel uploaded to root-level `Credit Card Receipts/3. QB Excel/` via Graph API PUT (shared folder for all persons)
 10. **Summary email** — sent with Excel files attached; To: Jheanelle, CC: Magdalena, BCC: devteam@togatech.com
@@ -83,6 +87,28 @@ Amex billing cycle runs 3rd-to-3rd. `computeRefNo()` returns the cycle-end date
 - Charge on or before the 3rd → cycle ends on the 3rd of the same month
 - Charge after the 3rd → cycle ends on the 3rd of the next month
 
+### Billing statement structure (Amex xlsx)
+
+The Amex statement Emily submits is a **multi-sheet** xlsx — one sheet per weekly billing
+period (e.g. `"05.04.2026 - 05.11.2026"`, `"05.11.2026 - 05.18.2026"`, …). `loadStatementExcel()`
+must iterate **`getAllSheets()`** and accumulate rows from every sheet — using `getActiveSheet()`
+reads only sheet 0 and silently drops charges that live on later sheets.
+
+Each sheet's columns: `Date | Receipt | Notes | Description | Amount | Extended Details |
+Appears On Your Statement As | Address | City/State | Zip Code | Country | Reference | Category`.
+
+- The **`Receipt` column (col B)** is a human-readable label Emily fills in to group charges
+  (e.g. `"1"`, `"2"`, `"Subway Rides"`). All transit charges (`NYCT PAYGO`) get the label
+  `"Subway Rides"` across all sheets. `loadStatementExcel()` must detect and capture this column.
+- `lookupStatementTotalByVendor()` resolves a statement total for a vendor in two passes:
+  1. **Receipt-label sum (first)** — sum the `Amount` of **all** rows whose `Receipt` label
+     contains the vendor keyword. Transit is one `$3.00` row per ride (e.g. 14 rows tagged
+     `"Subway Rides"`), so the total is the **sum** ($42.00), not a single row.
+  2. **Description-keyword match (fallback)** — for non-transit vendors, match the keyword
+     against the `Description` column. (The description for subway is `"AplPay NYCT PAYGO
+     NEW YORK"`, which contains no `"subway"` — so the receipt-label pass is what makes
+     transit resolve at all.)
+
 ### QB vendor matching
 `TowFoundationVendors.php` holds 2,994 vendor names exported verbatim from Tow Foundation's
 QuickBooks vendor list (June 2026). `matchQbVendor()` runs three passes against this list:
@@ -180,8 +206,9 @@ Fatal errors send only to `NOTIFY_EMAIL_DEV` (no CC/BCC).
   exists at the root of `Credit Card Receipts/` alongside the person folders.
 
 - **Person folder name normalization** — SharePoint folders are named `"Brent Peterkin CC receipts"` but `CLASS_MAP` / `PAYMENT_ACCOUNT_MAP` keys are just `"Brent Peterkin"`. The ` CC receipts` suffix is stripped via regex in `walkReceiptsFolder()`. Without this, Class and Payment Account columns are blank for those persons.
+- **Archive paths must use the actual SharePoint folder name, not the normalized name** — the normalized person name (` CC receipts` stripped, spaces replaced) is for **map lookups only**. When building the SharePoint archive path, use the *actual* folder name (e.g. `"Emily Tow CC receipts"`), not the normalized `"Emily Tow"` — otherwise the PATCH move 404s because the path segment does not exist. Keep the normalized name and the real folder name as separate values.
 - **`billingCycle` filter is suffix-match, not exact** — always pass just the date portion (`"06-03-2026"`), not the full folder name. Passing the full name (`"Amex ending in 06-03-2026"`) also works but would miss Mastercard folders.
-- **Ole Mole not in QB vendor list** — as of June 2026 this vendor is not in Tow Foundation's QuickBooks, so `qb_vendor` will be blank for Ole Mole charges until they add it to QB and the vendor list is refreshed.
+- **Vendors not in QB vendor list** — as of June 2026 these vendors are not in Tow Foundation's QuickBooks, so `qb_vendor` is blank for their charges until the client adds them to QB and the vendor list is refreshed: **Ole Mole**, **AMORE PIZZA CAFE**, **Green & Tonic (New Canaan)**, **NCFP (National Center for Philanthropy)**, **Rippling**, **Langan's**. This is a client action, not a code fix.
 - **Duplicate detection is within-run only** — if the same receipt was processed in a prior run and archived, it won't be caught. The dedup only covers receipts present in the current run's `$pendingRenames`.
 - **Unmapped AI categories pass through** — if the AI returns a category string not in `TowFoundationCategories.php`, `mapCategory()` logs a warning and writes the raw AI string to the Excel. Check error logs after a run if the Category column looks odd; add the new value to the map file to fix it.
 - **Short vendor acronyms (< 4 chars) require whole-word match** — the substring pass uses a word-boundary regex for needles under 4 chars to avoid false positives (e.g. `"UPS"` inside `"USPS"`). If a short vendor name is not matching, check that the QB vendor list entry starts or ends with the acronym as a whole word.
@@ -189,6 +216,7 @@ Fatal errors send only to `NOTIFY_EMAIL_DEV` (no CC/BCC).
 
 ## Change history
 
+- 2026-06-25 — Statement parsing hardened for Emily Tow's May 2026 Amex xlsx: `loadStatementExcel()` now iterates `getAllSheets()` (4 weekly-period sheets) instead of `getActiveSheet()` and captures the `Receipt` column; `lookupStatementTotalByVendor()` now **sums** all receipt-label-matched rows (14 × $3.00 "Subway Rides" = $42.00) before falling back to description-keyword match — fixes subway totals reading $0/$9.00. Added `extractDocxData()` so `.docx` receipts bypass Talos (Talos 500s on docx), parsing `MM/DD/YY: description` line items into per-entry Excel rows. Added a year guard clamping AI-hallucinated `invoice_date` years (e.g. 1976, 2028) to the current year. Fixed archive-path bug: SharePoint move now uses the actual folder name (`"Emily Tow CC receipts"`) not the normalized name, fixing 404 on archive moves. 5 more vendors found missing from the QB vendor list (client action). (rgirish)
 - 2026-06-23 — `TowFoundationCategories.php` added: AI category values now mapped to exact QB account strings via `mapCategory()`; `"Transportation"` → `"6610 Travel expense"` etc. (19 mappings). QB vendor substring match minimum lowered from 4 to 3 chars with whole-word boundary guard for short needles — fixes `"CVS"` → `"CVS Pharmacy"`, `"MTA"` → `"MTA Metro card"`, prevents `"UPS"` → `"USPS"` false positive. (rgirish)
 - 2026-06-18 — QB vendor matching via local fuzzy match against `TowFoundationVendors.php` (2,994 vendors); AI category prompt updated to return specific types (Dining, Travel, etc.); duplicate receipt detection added (person+date+vendor+amount dedup); `billingCycle` filter parameter added (suffix-match — card-type agnostic); person folder ` CC receipts` suffix stripped so CLASS_MAP/PAYMENT_ACCOUNT_MAP resolve correctly (rgirish)
 - 2026-06-12 — QB Excel upload moved to root-level `3. QB Excel/`; Archive/exception folders flattened (no person subfolder); unsupported mime types (`application/octet-stream`) now throw ReceiptProcessingException instead of being silently skipped (rgirish)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.204",
+  "version": "1.0.205",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",