toga-ai 1.0.307 → 1.0.308

package/knowledge/clients/tow-foundation/features/receipt-processing.md

@@ -41,7 +41,7 @@ Credit Card Receipts/
   {Person}/
     {Year}/
       {BillingCycleFolder}/   ← receipt files live here (e.g. "Amex ending in 06-03-2026")
-      {Person} reports/        ← .xlsx billing statements for cross-verification
+      {Person} report[s]/      ← .xlsx billing statements (folder name matched on singular "report")
       Archive/                 ← successfully processed receipts land here (renamed, flat)
       Archive/exception/       ← failed receipts land here (original name preserved, flat)
 ```
@@ -51,13 +51,23 @@ Credit Card Receipts/
 1. **Auth** — OAuth2 client-credentials token from Microsoft Graph
 2. **Walk** — `walkReceiptsFolder()` enumerates all receipt files and statement Excels
 3. **Filter** — optional `$year` / `$person` / `$billingCycle` / `$limit` applied to the receipt list. `$billingCycle` uses `str_ends_with()` so passing just the date (e.g. `"06-03-2026"`) matches any card type. Person folders named `"Brent Peterkin CC receipts"` are normalized to `"Brent Peterkin"` via regex strip of ` CC receipts` suffix before any map lookups.
-4. **Pre-load statements** — all billing-cycle `.xlsx` files under `reports/` are downloaded
-   and parsed upfront via PhpSpreadsheet (auto-detect header row by scanning for
-   description/amount keywords)
+4. **Pre-load statements** — all billing-cycle `.xlsx` files under a person's statements
+   folder are downloaded and parsed upfront via PhpSpreadsheet (auto-detect header row by
+   scanning for description/amount keywords). **Statements folder detection matches the
+   singular `report`.** `walkReceiptsFolder()` recognizes a person's statements folder via
+   `stripos($child->name,'report')` (singular). It previously matched `'reports'` (plural)
+   only, so cardholders whose folder is named `"{Name} report"` (singular — Angela Leis,
+   Kathryn Lockhart, Meghan Lowney, Eileen Wiseman) had their statement Excel **never
+   loaded**: `$parsedStatements[$person]` was unset and the entire `Notes`-memo override was
+   silently skipped (AI-inferred memos shipped instead). The singular match also covers
+   `"reports"`, `"Report"`, `"Amex reports"`. **Folder-name matching is the gate for the
+   whole Notes-memo feature** — verifying `matchStatementNotes()` in isolation is
+   insufficient; the statement must actually load during `Run()`, so verify against a real
+   run that `$parsedStatements` includes the person.
 5. **Pass 1 — Extract** — for each receipt:
    - Enforce type + size limit: unsupported mime types (`application/octet-stream`) throw immediately; 4 MB cap for images, 10 MB for documents
    - Download file bytes from SharePoint
-   - **`.docx` files bypass Talos** and route to `extractDocxData()` (Talos returns HTTP 500 on a `.docx` MIME type — it only accepts PDF/image). That method opens the docx as a ZIP, extracts `word/document.xml`, strips tags, and regex-parses ride lines into `line_items[]`. The ride-line regex (in both `extractDocxData()` and `parseDocxRideLines()`) accepts **slash or dot** date separators (`[\/.]` — e.g. `05/06/2025` and `05.06.2025`) and treats the `[:\-]?` description delimiter as **optional**. Dash-separated dates (`05-06-2025`) are intentionally **not** supported — a dash would clash with the `:`/`-` description delimiter. A docx that matches zero ride lines throws `ReceiptProcessingException` and produces no Excel, so a new date format silently zero-outputs — widen the regex when a cardholder introduces one
+   - **`.docx` files bypass Talos** and route to `extractDocxData()` (Talos returns HTTP 500 on a `.docx` MIME type — it only accepts PDF/image). That method opens the docx as a ZIP, extracts `word/document.xml`, strips tags, and regex-parses ride lines into `line_items[]`. **Tag-stripping — a run boundary is not a word boundary.** When flattening the XML to text, `</w:r>` (run end) must be replaced with `''` (empty string), **not** a space — Word splits a single word across multiple runs, so replacing `</w:r>` with a space injects spaces mid-word (`"Innocence"`->`"Innoc ence"`, `"Armstrong"`->`"Armstron g"`, `"2026.06.03"`->`"202 6 . 0 6 .0 3"`). Only `</w:p>` (paragraph end) is a real line break (`"\n"`); `<w:t>` text keeps its own spaces. Both `extractDocxData()` and `parseDocxRideLines()` do this replacement. **Header strip before ride parsing.** `extractDocxData()` first removes any statement header line via `preg_replace('/(?:AMEX|Visa)\s+Closing\s+Date\s+\d{4}[.\/-]\d{1,2}[.\/-]\d{1,2}/i','',$text)` — otherwise the ride regex greedily matches `"AMEX Closing Date 2026.06.03"` as the first ride (reading the `YYYY.MM.DD` as a date and swallowing the real first ride into its description, corrupting row 1 and dropping the last real ride). The header is stripped by keyword rather than by restricting ride dates to 2-digit years — narrowing ride dates would regress Jheanelle's 4-digit dot dates. The ride-line regex (in both `extractDocxData()` and `parseDocxRideLines()`) accepts **slash or dot** date separators (`[\/.]` — e.g. `05/06/2025` and `05.06.2025`) and treats the `[:\-]?` description delimiter as **optional**. Dash-separated dates (`05-06-2025`) are intentionally **not** supported — a dash would clash with the `:`/`-` description delimiter. A docx that matches zero ride lines throws `ReceiptProcessingException` and produces no Excel, so a new date format silently zero-outputs — widen the regex when a cardholder introduces one
    - All other types POST to Talos AI `/api/ai/generate` → structured `{vendor_name, invoice_date, total, payment_memo, category, ...}`
    - **Year guard on AI dates** — if Talos returns an `invoice_date` whose year is more than 1 year from the current year (AI hallucination on two-digit year inputs, e.g. `5/13/76` → 1976, `5/28/28` → 2028), the year is clamped to the current year while month/day are preserved
    - **Payment memo — the statement `Notes` column is authoritative.** `loadStatementExcel()` detects a `Notes` column (exact-match `"notes"`/`"note"` first; a substring fallback runs only if no exact header was found, guarded by `$notesExact`) and attaches a `notes` field to each parsed row. In `Run()`, if the matching statement row has a **non-empty `notes` value**, it is used **verbatim** as `payment_memo` — this is the cardholder's own hand-written memo (e.g. `"Postage: Board materials shipped to Frank Tow"`) and beats any AI-inferred memo. Only if the row has no notes does processing fall back to the prior AI/description behavior. This **supersedes** the earlier `isActivityHeadingMemo()` allow-list, which dropped the cardholder's Notes entirely (it only detected columns containing `"description"`/`"memo"`) and rejected valid headings like `Communications:`, `Postage:`, `Office Supplies:`, `RF:` — shipping AI memos in their place.
@@ -201,6 +211,22 @@ the old one-shot `/tmp/tow_restore_archives.php` CLI script.
 walks all subfolders of `Archive/` (both the processed root and `exception/`) and collects
 every non-folder item into a flat accumulator array.
 
+### Reprocessing a cycle locally (operational)
+To re-run one cardholder + cycle after a code fix, without deploying:
+1. **Restore receipts** — call `MoveBack(person, "Amex ending in MM-DD-YYYY", "2026")` to move
+   archived receipts back into the billing-cycle folder.
+2. **Run via CLI, not the HTTP endpoint** — invoke the action directly in PHP so the run
+   bypasses the Worker HTTP/`WorkerJobs` dispatch:
+   ```
+   ENVIRONMENT=dev-... php -r '... require "_underscore.php"; echo _Worker_Client_TowFoundation_ProcessReceipts::Run(null,null,$person,$cycle);'
+   ```
+   The local worker HTTP endpoint returns **HTTP 500** on direct `{action,parameters}` calls
+   because the 2026-07-07 change made the direct-payload path require a `Core.WorkerJobs`
+   table that local dev DBs lack (MySQL Error #1146 `core.workerjobs doesn't exist`). The CLI
+   path avoids that dispatch entirely.
+3. **Email is safe locally** — each `Run()` sends the summary email, but with `debug_mode=1`
+   locally all mail is redirected to `send_debug_emails_to`, so no client contact is reached.
+
 ## Email routing
 
 Defined as constants in `TowFoundation.php`:
@@ -254,6 +280,15 @@ Fatal errors send only to `NOTIFY_EMAIL_DEV` (no CC/BCC).
 - **Person folder name normalization** — SharePoint folders are named `"Brent Peterkin CC receipts"` but `CLASS_MAP` / `PAYMENT_ACCOUNT_MAP` keys are just `"Brent Peterkin"`. The ` CC receipts` suffix is stripped via regex in `walkReceiptsFolder()`. Without this, Class and Payment Account columns are blank for those persons.
 - **Archive paths must use the actual SharePoint folder name, not the normalized name** — the normalized person name (` CC receipts` stripped, spaces replaced) is for **map lookups only**. When building the SharePoint archive path, use the *actual* folder name (e.g. `"Emily Tow CC receipts"`), not the normalized `"Emily Tow"` — otherwise the PATCH move 404s because the path segment does not exist. Keep the normalized name and the real folder name as separate values.
 - **`billingCycle` filter is suffix-match, not exact** — always pass just the date portion (`"06-03-2026"`), not the full folder name. Passing the full name (`"Amex ending in 06-03-2026"`) also works but would miss Mastercard folders.
+- **Missing per-cycle statement → no Notes memos (data dependency, not a bug)** — statements
+  are keyed by `cycleKey = filename-without-extension`, and the code looks up the statement
+  whose name matches the billing-cycle folder (e.g. `"Amex ending in 06-03-2026"`). If a
+  cardholder's report folder has no statement file for that cycle, **zero Notes are loaded**
+  and **all** of that person's memos fall back to AI-inferred (their hand-written `"RF:"` red
+  memos never appear). As of the 2026-07-09 run, **Nadia Alia's** `"Nadia's Amex reports"`
+  folder held Jan–May statements but no `"Amex ending in 06-03-2026.xlsx"` (and even her 05-03
+  file has no `Notes` column). Resolution is a **client action**: upload Nadia's 06-03-2026
+  statement with a `Notes` column, then re-run her. Not a code fix.
 - **Vendors not in QB vendor list** — as of June 2026 these vendors are not in Tow Foundation's QuickBooks, so `qb_vendor` is blank for their charges until the client adds them to QB and the vendor list is refreshed: **Ole Mole**, **AMORE PIZZA CAFE**, **Green & Tonic (New Canaan)**, **NCFP (National Center for Philanthropy)**, **Rippling**, **Langan's**. This is a client action, not a code fix.
 - **Duplicate detection is within-run and `fileId`-keyed only** — dedup keys on the SharePoint
   drive-item `fileId`, so it catches only a literal re-upload of the same item present in the
@@ -274,9 +309,16 @@ Fatal errors send only to `NOTIFY_EMAIL_DEV` (no CC/BCC).
 - **Amount reconciliation is flag-only** — mismatched OCR amounts are surfaced in the "Amounts to
   verify" email section but never auto-corrected from the statement; auto-override was deferred as
   higher-risk.
+- **Not yet reprocessed / undeployed (as of 2026-07-09)** — **Jheanelle** and **Diane** still need
+  a `MoveBack` + `Run` to pick up the docx run-boundary / Closing-Date-header fixes. All 2026-07-09
+  fixes are **local and uncommitted** — no deployment has been performed.
+- **Subway docx line-item memos come from the docx, by design** — the per-ride memos on a subway
+  `.docx` are sourced from the docx line-item description, not the statement `Notes` column. This is
+  intentional; do not "fix" it by routing them through `matchStatementNotes()`.
 
 ## Change history
 
+- 2026-07-09 — **Local reprocess-all-cardholders verification pass — three more fixes + two data/ops findings (code-only, uncommitted).** These fixes were found *after* the six-fix pass earlier the same day, while re-running every cardholder locally and inspecting the uploaded QB files. (1) **Statements-folder detection matched plural `reports` only** — `walkReceiptsFolder()` now matches `stripos(name,'report')` (singular), so `"{Name} report"` folders (Angela Leis, Kathryn Lockhart, Meghan Lowney, Eileen Wiseman) actually load their statement; without this the earlier Notes-memo fix silently did nothing for them in a real `Run()` (folder-name matching is the gate — verifying `matchStatementNotes()` in isolation is not enough). (2) **docx run-boundary word-splitting** — `</w:r>` was replaced with a space, injecting spaces mid-word (`"Innoc ence"`, `"Armstron g"`, `"202 6 . 0 6 .0 3"`) in Emily Tow's subway memos; now replaced with `''` at both sites (`extractDocxData`, `parseDocxRideLines`) — only `</w:p>` is a real line break. (3) **docx "Closing Date" header mis-parsed as first ride** — the ride regex read `"AMEX Closing Date 2026.06.03"` as a ride and swallowed the real first ride; now stripped upfront via `preg_replace('/(?:AMEX|Visa)\s+Closing\s+Date\s+\d{4}[.\/-]\d{1,2}[.\/-]\d{1,2}/i','',$text)` (kept 4-digit dot dates for Jheanelle rather than narrowing ride dates). Data/ops findings (no code change): **Nadia Alia's** June statement is absent from SharePoint → all her memos AI-inferred until the client uploads `"Amex ending in 06-03-2026.xlsx"` with a Notes column; local worker HTTP endpoint 500s on direct `{action,parameters}` calls (post 2026-07-07 `Core.WorkerJobs` dependency absent in dev) → reprocess a cycle via CLI + `MoveBack` (debug_mode redirects mail). Verified correct: Angela, Emily (incl. 14 subway line items), Kathryn, Meghan (Kellari $548.11 flagged), Michael (all 5 charges). Nadia pending client statement; Jheanelle + Diane not yet reprocessed. (rgirish)
 - 2026-07-09 — **Client QA pass on a production run — six fixes (code-only, not deployed).** (1) Statement `Notes` column is now the **authoritative** payment memo, used verbatim; `loadStatementExcel()` detects `Notes` (exact-match first, guarded substring fallback) — replaces the `isActivityHeadingMemo()` allow-list that dropped hand-written memos (fixed Angela, Emily, Katy, Meghan, Nadia). (2) `matchStatementCandidate($strict)` returns `null` on ambiguous same-amount matches for the Notes memo instead of guessing; `matchStatementNotes()` is strict, description fallback stays best-effort. (3) Dedup now keys on SharePoint `fileId` only — the old person|date|vendor|amount key dropped distinct same-amount charges (Michael lost 2 of 5). (4) docx ride-line regex now accepts dot dates (`[\/.]`) with optional `[:\-]?` delimiter — Jheanelle's dot-format subway docx previously matched 0 lines and produced no output. (5) `buildExcelFileName()` now derives the reporting month from the billing-cycle end (`parseCycleEndDate()`/`resolvePersonCycle()`) not the charge date, so the filename agrees with the Ref No. (Diane). (6) Systemic discovery: non-docx `payment_amount` is AI-OCR only, never reconciled against the statement `Amount` (Meghan $548.11 vs $538.11); added flag-only `statementHasAmount()` + `$warnings` "Amounts to verify" email section — auto-override deferred. (rgirish)
 - 2026-06-29 — **Reversed the over-aggressive memo override.** Replaced `isBareTranportationMemo()` (added earlier same day) with `isActivityHeadingMemo()`: the statement memo now overrides the AI memo **only** when it is a proper activity-purpose heading (e.g. `Grantee meeting:`, `Peer meeting:`), not whenever the AI memo started with `"Transportation:"`. The previous approach replaced semi-useful AI memos with raw bank strings (`"AplPay LYFT …"`, `"UBER"`) on ride-share/taxi rows where the statement has no activity heading — degraded 8 rows on Emily Tow's statement (1, 17, 18, 21, 26, 31, 32, 34). New priority: statement heading > AI memo, with AI memo kept for any non-heading statement description. (rgirish)
 - 2026-06-29 — `payment_memo` override now also fires on a **bare** `"Transportation:"` AI memo, not just an empty one: new `isBareTranportationMemo()` lets the cardholder's statement Excel (the authoritative source for activity-purpose headings like "Peer meeting:", "Grantee event:") override the AI's generic transportation heading. Fixes wrong headings on Emily Tow May 2026 transit rows. Added the committed `MoveBack` action (+ `collectFilesRecursiveRaw()`) to restore a person's archived receipts into a billing-cycle folder for reprocessing, replacing the throwaway `/tmp/tow_restore_archives.php` script. (rgirish)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.307",
+  "version": "1.0.308",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",