toga-ai 1.0.299 → 1.0.300

package/knowledge/clients/tow-foundation/features/receipt-processing.md

@@ -5,7 +5,7 @@ project: Worker
 client: tow-foundation
 type: client-feature
 status: active
-updated: 2026-06-29
+updated: 2026-07-09
 owners: ["rgirish"]
 files:
   - worker2/Worker/Client/TowFoundation.php
@@ -57,14 +57,15 @@ Credit Card Receipts/
 5. **Pass 1 — Extract** — for each receipt:
    - Enforce type + size limit: unsupported mime types (`application/octet-stream`) throw immediately; 4 MB cap for images, 10 MB for documents
    - Download file bytes from SharePoint
-   - **`.docx` files bypass Talos** and route to `extractDocxData()` (Talos returns HTTP 500 on a `.docx` MIME type — it only accepts PDF/image). That method opens the docx as a ZIP, extracts `word/document.xml`, strips tags, and regex-parses `MM/DD/YY: description` lines into `line_items[]`
+   - **`.docx` files bypass Talos** and route to `extractDocxData()` (Talos returns HTTP 500 on a `.docx` MIME type — it only accepts PDF/image). That method opens the docx as a ZIP, extracts `word/document.xml`, strips tags, and regex-parses ride lines into `line_items[]`. The ride-line regex (in both `extractDocxData()` and `parseDocxRideLines()`) accepts **slash or dot** date separators (`[\/.]` — e.g. `05/06/2025` and `05.06.2025`) and treats the `[:\-]?` description delimiter as **optional**. Dash-separated dates (`05-06-2025`) are intentionally **not** supported — a dash would clash with the `:`/`-` description delimiter. A docx that matches zero ride lines throws `ReceiptProcessingException` and produces no Excel, so a new date format silently zero-outputs — widen the regex when a cardholder introduces one
    - All other types POST to Talos AI `/api/ai/generate` → structured `{vendor_name, invoice_date, total, payment_memo, category, ...}`
    - **Year guard on AI dates** — if Talos returns an `invoice_date` whose year is more than 1 year from the current year (AI hallucination on two-digit year inputs, e.g. `5/13/76` → 1976, `5/28/28` → 2028), the year is clamped to the current year while month/day are preserved
-   - Cross-verify amount ± $0.01 against parsed statement; if matched, the statement description **may** override `payment_memo`. The statement memo is the **authoritative source for activity-purpose headings** (Grantee meeting, Peer meeting, Grantee event, etc.) — but only when it actually *is* such a heading. The Amex statement's `Description` column for ride-share/taxi charges holds raw bank strings (e.g. `"AplPay LYFT 855-280-0278 CA"`, `"UBER"`), not activity headings. **Priority order (enforced):** (1) statement memo wins only if it is a proper activity heading; (2) otherwise the AI memo is kept — including when the statement only has a raw bank string, because the AI's descriptive memo (e.g. `"Transportation: Lyft ride from East Side at 105th to 925 9th Ave"`) is more useful than a raw Apple Pay string. The override condition is `empty($extracted->payment_memo) || self::isActivityHeadingMemo($matched)`. `isActivityHeadingMemo()` returns true only when the statement memo (left-trimmed, lowercased) starts with a recognised activity-purpose prefix followed by a colon (`grantee meeting:`, `grantee event:`, `peer meeting:`, `colleague meeting:`, `staff theater:`, `professional development:`, `dining:`, `travel:`). This **replaces** the earlier `isBareTranportationMemo()` approach, which fired whenever the AI memo started with `"Transportation:"` and so let the raw bank string overwrite a useful AI memo on ride-share rows (degraded rows 1, 17, 18, 21, 26, 31, 32, 34 on Emily Tow's statement).
+   - **Payment memo — the statement `Notes` column is authoritative.** `loadStatementExcel()` detects a `Notes` column (exact-match `"notes"`/`"note"` first; a substring fallback runs only if no exact header was found, guarded by `$notesExact`) and attaches a `notes` field to each parsed row. In `Run()`, if the matching statement row has a **non-empty `notes` value**, it is used **verbatim** as `payment_memo` — this is the cardholder's own hand-written memo (e.g. `"Postage: Board materials shipped to Frank Tow"`) and beats any AI-inferred memo. Only if the row has no notes does processing fall back to the prior AI/description behavior. This **supersedes** the earlier `isActivityHeadingMemo()` allow-list, which dropped the cardholder's Notes entirely (it only detected columns containing `"description"`/`"memo"`) and rejected valid headings like `Communications:`, `Postage:`, `Office Supplies:`, `RF:` — shipping AI memos in their place.
+   - **Statement row matching.** `matchStatementCandidate()` returns the whole matched statement row (± $0.01 amount, then date tie-break). `matchStatementRow()` (description fallback, best-effort) and `matchStatementNotes()` (returns the row's `notes`) both delegate to it. `matchStatementCandidate($strict)`: in **strict** mode, when multiple rows share an amount and none resolves by date, it returns `null` and logs — rather than guessing an arbitrary first candidate. `matchStatementNotes()` uses `strict=true` (attaching the wrong cardholder's memo is worse than none); the description fallback stays non-strict/best-effort.
    - **Zero-total fallback** — if the extracted/docx total is 0, `lookupStatementTotalByVendor()` derives the amount from the billing statement (see *Statement structure* below)
    - Build base filename (no suffix yet) and store in `$pendingRenames`
    - On extract failure → move to `Archive/exception/` immediately
-6. **Deduplication** — after Pass 1, receipts sharing identical `(personName, invoice_date, vendor_name, total)` are deduplicated; only the first is kept, the rest are logged as `DUPLICATE skipped` and dropped. This catches the same receipt uploaded twice to SharePoint.
+6. **Deduplication** — after Pass 1, receipts are deduplicated on the **SharePoint drive-item `fileId` only** (a true re-upload is the same drive item; distinct files are distinct items). Duplicates are logged as `DUPLICATE skipped` and dropped. **Do not** key dedup on `(personName, invoice_date, vendor_name, total)` — that collided genuinely distinct same-day/same-amount charges (e.g. two $26 Metro-North legs, two $3 subway swipes) and silently dropped one, so 2 of Michael's 5 charges never reached the sheet. Distinct files that produce the same base filename are handled downstream by the `_a`/`_b` suffix logic.
 7. **Collision detection** — count base name occurrences across remaining pending renames
 8. **Pass 2 — Move** — for each pending rename:
    - If base name appears more than once, assign `_a`, `_b`, `_c`... suffix to **all** colliding files (including the first)
@@ -72,7 +73,19 @@ Credit Card Receipts/
    - **Multi-entry docx expansion** — a `.docx` whose `extractDocxData()` produced multiple `line_items[]` (e.g. a subway-rides document with 14 dated trip entries) is expanded into one Excel row per entry
 8. **Excel generation** — one `.xlsx` per person (PhpSpreadsheet); columns: Row #, Account Name, QB Vendor, Payment Amount, Date, Payment Method, Payment Memo, QB Description, Class, Category, Payment Account, Ref No.
 9. **SharePoint upload** — each Excel uploaded to root-level `Credit Card Receipts/3. QB Excel/` via Graph API PUT (shared folder for all persons)
-10. **Summary email** — sent with Excel files attached; To: Jheanelle, CC: Magdalena, BCC: devteam@togatech.com
+10. **Summary email** — sent with Excel files attached; To: Jheanelle, CC: Magdalena, BCC: devteam@togatech.com. `sendSummaryEmail()` takes a `$warnings` param and renders an **"Amounts to verify"** section listing rows whose amount could not be reconciled against the statement (see below).
+
+### Amount reconciliation (flag-only)
+A non-docx receipt's `payment_amount` comes **solely from AI OCR** and is **never** overridden by
+the statement `Amount` — the statement is consulted only for the memo. This is systemic for every
+JPG/PDF receipt (e.g. a Kellari Taverna charge OCR'd a handwritten total as `$548.11` vs. the
+statement's `$538.11`). To surface these without changing the number, `statementHasAmount()` checks
+whether the AI total matches any statement charge within a cent. In `Run()`, for **non-docx** receipts
+with `total > 0` **when a statement is loaded for that cycle**, if no statement amount matches, a
+non-fatal entry is added to `$warnings` (**the row is still emitted**) and rendered in the summary
+email's "Amounts to verify" section for human review before QuickBooks import. Docx receipts (amount
+sourced from the statement) and receipts with no loaded statement are excluded. Auto-overriding
+`payment_amount` from the statement was **deliberately deferred** as higher-risk.
 
 ### Renamed file format
 ```
@@ -87,6 +100,18 @@ Amex billing cycle runs 3rd-to-3rd. `computeRefNo()` returns the cycle-end date
 - Charge on or before the 3rd → cycle ends on the 3rd of the same month
 - Charge after the 3rd → cycle ends on the 3rd of the next month
 
+### Excel filename — derived from the billing-cycle end, not the charge date
+`buildExcelFileName(..., $billingCycleFolder)` uses the **cycle end**, not the first row's
+charge date, to pick the reporting month — so April charges that belong to the
+`"Amex ending in 05-03-2026"` cycle produce a `_05_2026` file that agrees with the per-row
+Ref No. (which is also cycle-end-based). `parseCycleEndDate()` parses `MM-YYYY` from the
+cycle folder name (anchored to `"ending in"`, with a bare `MM-DD-YYYY` fallback). If the
+cycle can't be resolved, it falls back to the charge date. `resolvePersonCycle()` picks a
+person's cycle from `$personCycles[person][cycleFolder]` (collected in Pass 2); if a run
+spans **multiple** cycles for one person it warns and uses the first (see open items). The
+resolved cycle is passed to all three `buildExcelFileName()` call sites (upload + both
+`sendSummaryEmail()` sites) so the uploaded filename and the email attachment always match.
+
 ### Billing statement structure (Amex xlsx)
 
 The Amex statement Emily submits is a **multi-sheet** xlsx — one sheet per weekly billing
@@ -207,9 +232,11 @@ Fatal errors send only to `NOTIFY_EMAIL_DEV` (no CC/BCC).
   names with special characters (spaces, dots). Always resolve to ID first.
 - **`ensureFolderPath()` for "3. QB Excel"** — the folder name contains a dot and space;
   `ensureFolderPath` handles this correctly via `rawurlencode` on each path segment.
-- **Statement matching tolerance** — `matchStatementRow()` uses `abs($row['amount'] - $amount) < 0.01`
-  for amount matching, then date tie-break for multiple same-amount rows. Returns first
-  candidate if date tie-break also ambiguous.
+- **Statement matching tolerance** — matching uses `abs($row['amount'] - $amount) < 0.01`, then a
+  date tie-break for multiple same-amount rows. When even the date tie-break is ambiguous, behavior
+  depends on the caller's `$strict` flag: **strict** (`matchStatementNotes()`) returns `null` and
+  logs rather than guessing (a wrong authoritative memo is worse than none); **non-strict**
+  (`matchStatementRow()` description fallback) returns the first candidate, best-effort.
 - **No client DB** — there is no audit trail in MySQL. All state lives in SharePoint folder
   structure and email. If a run is interrupted mid-way, some receipts may be archived
   without a corresponding Excel row.
@@ -228,14 +255,29 @@ Fatal errors send only to `NOTIFY_EMAIL_DEV` (no CC/BCC).
 - **Archive paths must use the actual SharePoint folder name, not the normalized name** — the normalized person name (` CC receipts` stripped, spaces replaced) is for **map lookups only**. When building the SharePoint archive path, use the *actual* folder name (e.g. `"Emily Tow CC receipts"`), not the normalized `"Emily Tow"` — otherwise the PATCH move 404s because the path segment does not exist. Keep the normalized name and the real folder name as separate values.
 - **`billingCycle` filter is suffix-match, not exact** — always pass just the date portion (`"06-03-2026"`), not the full folder name. Passing the full name (`"Amex ending in 06-03-2026"`) also works but would miss Mastercard folders.
 - **Vendors not in QB vendor list** — as of June 2026 these vendors are not in Tow Foundation's QuickBooks, so `qb_vendor` is blank for their charges until the client adds them to QB and the vendor list is refreshed: **Ole Mole**, **AMORE PIZZA CAFE**, **Green & Tonic (New Canaan)**, **NCFP (National Center for Philanthropy)**, **Rippling**, **Langan's**. This is a client action, not a code fix.
-- **Duplicate detection is within-run only** — if the same receipt was processed in a prior run and archived, it won't be caught. The dedup only covers receipts present in the current run's `$pendingRenames`.
+- **Duplicate detection is within-run and `fileId`-keyed only** — dedup keys on the SharePoint
+  drive-item `fileId`, so it catches only a literal re-upload of the same item present in the
+  current run. It deliberately does **not** dedup on person/date/vendor/amount (that dropped
+  distinct same-amount charges); genuinely distinct files with the same content are all kept.
 - **Unmapped AI categories pass through** — if the AI returns a category string not in `TowFoundationCategories.php`, `mapCategory()` logs a warning and writes the raw AI string to the Excel. Check error logs after a run if the Category column looks odd; add the new value to the map file to fix it.
 - **Short vendor acronyms (< 4 chars) require whole-word match** — the substring pass uses a word-boundary regex for needles under 4 chars to avoid false positives (e.g. `"UPS"` inside `"USPS"`). If a short vendor name is not matching, check that the QB vendor list entry starts or ends with the acronym as a whole word.
 - **Restoring archived receipts** — use the committed `MoveBack` action (see *MoveBack action* above), not a one-shot CLI script. It moves all non-Excel receipt files from a person's `Archive/` back into the named billing-cycle folder. (The earlier throwaway `/tmp/tow_restore_archives.php` script is obsolete.)
 - **Mixed-cycle receipts can collect in Archive** — receipts from a prior run can linger in a person's `Archive/` under a different billing cycle (e.g. January receipts found alongside May during an Emily Tow restore). `MoveBack` moves **everything** in `Archive/` back, so after a restore, verify the receipts belong to the cycle you intend to reprocess; stray off-cycle files should be moved (via Graph API) to their own billing-cycle folder so they are not reprocessed with the wrong cycle.
 
+## Open items / known-deferred
+
+- **Multi-cycle runs merge to one file** — if a single run spans multiple billing cycles for one
+  person, all rows still merge into one Excel (named after the first cycle) with a logged warning.
+  Full per-`(person, cycle)` output split is deferred.
+- **Dash-separated docx dates unsupported** — `05-06-2025` is intentionally not parsed (dash clashes
+  with the description delimiter); only slash and dot separators work.
+- **Amount reconciliation is flag-only** — mismatched OCR amounts are surfaced in the "Amounts to
+  verify" email section but never auto-corrected from the statement; auto-override was deferred as
+  higher-risk.
+
 ## Change history
 
+- 2026-07-09 — **Client QA pass on a production run — six fixes (code-only, not deployed).** (1) Statement `Notes` column is now the **authoritative** payment memo, used verbatim; `loadStatementExcel()` detects `Notes` (exact-match first, guarded substring fallback) — replaces the `isActivityHeadingMemo()` allow-list that dropped hand-written memos (fixed Angela, Emily, Katy, Meghan, Nadia). (2) `matchStatementCandidate($strict)` returns `null` on ambiguous same-amount matches for the Notes memo instead of guessing; `matchStatementNotes()` is strict, description fallback stays best-effort. (3) Dedup now keys on SharePoint `fileId` only — the old person|date|vendor|amount key dropped distinct same-amount charges (Michael lost 2 of 5). (4) docx ride-line regex now accepts dot dates (`[\/.]`) with optional `[:\-]?` delimiter — Jheanelle's dot-format subway docx previously matched 0 lines and produced no output. (5) `buildExcelFileName()` now derives the reporting month from the billing-cycle end (`parseCycleEndDate()`/`resolvePersonCycle()`) not the charge date, so the filename agrees with the Ref No. (Diane). (6) Systemic discovery: non-docx `payment_amount` is AI-OCR only, never reconciled against the statement `Amount` (Meghan $548.11 vs $538.11); added flag-only `statementHasAmount()` + `$warnings` "Amounts to verify" email section — auto-override deferred. (rgirish)
 - 2026-06-29 — **Reversed the over-aggressive memo override.** Replaced `isBareTranportationMemo()` (added earlier same day) with `isActivityHeadingMemo()`: the statement memo now overrides the AI memo **only** when it is a proper activity-purpose heading (e.g. `Grantee meeting:`, `Peer meeting:`), not whenever the AI memo started with `"Transportation:"`. The previous approach replaced semi-useful AI memos with raw bank strings (`"AplPay LYFT …"`, `"UBER"`) on ride-share/taxi rows where the statement has no activity heading — degraded 8 rows on Emily Tow's statement (1, 17, 18, 21, 26, 31, 32, 34). New priority: statement heading > AI memo, with AI memo kept for any non-heading statement description. (rgirish)
 - 2026-06-29 — `payment_memo` override now also fires on a **bare** `"Transportation:"` AI memo, not just an empty one: new `isBareTranportationMemo()` lets the cardholder's statement Excel (the authoritative source for activity-purpose headings like "Peer meeting:", "Grantee event:") override the AI's generic transportation heading. Fixes wrong headings on Emily Tow May 2026 transit rows. Added the committed `MoveBack` action (+ `collectFilesRecursiveRaw()`) to restore a person's archived receipts into a billing-cycle folder for reprocessing, replacing the throwaway `/tmp/tow_restore_archives.php` script. (rgirish)
 - 2026-06-25 — Statement parsing hardened for Emily Tow's May 2026 Amex xlsx: `loadStatementExcel()` now iterates `getAllSheets()` (4 weekly-period sheets) instead of `getActiveSheet()` and captures the `Receipt` column; `lookupStatementTotalByVendor()` now **sums** all receipt-label-matched rows (14 × $3.00 "Subway Rides" = $42.00) before falling back to description-keyword match — fixes subway totals reading $0/$9.00. Added `extractDocxData()` so `.docx` receipts bypass Talos (Talos 500s on docx), parsing `MM/DD/YY: description` line items into per-entry Excel rows. Added a year guard clamping AI-hallucinated `invoice_date` years (e.g. 1976, 2028) to the current year. Fixed archive-path bug: SharePoint move now uses the actual folder name (`"Emily Tow CC receipts"`) not the normalized name, fixing 404 on archive moves. 5 more vendors found missing from the QB vendor list (client action). (rgirish)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.299",
+  "version": "1.0.300",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",