toga-ai 1.0.366 → 1.0.367

package/knowledge/clients/tow-foundation/features/receipt-processing.md

@@ -5,7 +5,7 @@ project: Worker
 client: tow-foundation
 type: client-feature
 status: active
-updated: 2026-07-13
+updated: 2026-07-17
 owners: ["rgirish"]
 files:
   - worker2/Worker/Client/TowFoundation.php
@@ -75,7 +75,11 @@ Credit Card Receipts/
    - All other types POST to Talos AI `/api/ai/generate` → structured `{vendor_name, invoice_date, total, payment_memo, category, ...}`
    - **Year guard on AI dates** — if Talos returns an `invoice_date` whose year is more than 1 year from the current year (AI hallucination on two-digit year inputs, e.g. `5/13/76` → 1976, `5/28/28` → 2028), the year is clamped to the current year while month/day are preserved
    - **Payment memo — the statement `Notes` column is authoritative.** `loadStatementExcel()` detects a `Notes` column (exact-match `"notes"`/`"note"` first; a substring fallback runs only if no exact header was found, guarded by `$notesExact`) and attaches a `notes` field to each parsed row. In `Run()`, if the matching statement row has a **non-empty `notes` value**, it is used **verbatim** as `payment_memo` — this is the cardholder's own hand-written memo (e.g. `"Postage: Board materials shipped to Frank Tow"`) and beats any AI-inferred memo. Only if the row has no notes does processing fall back to the prior AI/description behavior. This **supersedes** the earlier `isActivityHeadingMemo()` allow-list, which dropped the cardholder's Notes entirely (it only detected columns containing `"description"`/`"memo"`) and rejected valid headings like `Communications:`, `Postage:`, `Office Supplies:`, `RF:` — shipping AI memos in their place.
+   - **The AI memo prompt must transcribe VERBATIM and never invent.** For receipts where the memo comes from the AI (image/PDF, no statement Notes for that row), the `extractReceiptData()` system+user prompt and the `payment_memo` output-schema description are written to (a) transcribe a human-added note **word-for-word** — no paraphrasing, abbreviating, or reformatting — and (b) return **`null`** when the receipt has no human-added note, never inferring one. This fixes two reported client defects: paraphrase (`"Lucy Ball of Lone Pine Foundation"` shortened to `"LB LPFoundation"`) and pure invention (a fabricated Asana memo). The model was previously *told* to infer a memo when no note was present — that instruction was removed. When editing this prompt, keep the verbatim + null-when-absent rules intact; a blank memo is the **correct** output for a note-less receipt (flagged in the Excel — see *Excel `Review` column*).
+   - **Full-page annotation scanning.** The prompt instructs the model to scan the **entire page** — margins, corners, header, and beside the address block — for human-added notes that are visually distinct (highlighted, handwritten, or overlaid), with a worked example. Annotations placed in the top/side margin were being missed and returned blank: a highlighted `"Telephone & Internet"` box on an Optimum bill, a red `"Board Meeting … Remaining Deposit"` note on a Garelick invoice, and an orange `"IT Software License"` on a Lucid invoice.
+   - **Reviewer/cardholder initials prefix is stripped.** Per client request, a leading initials tag (`"RF:"`, `"NA:"`, `"ML:"` — 1–2 capital letters followed by a colon) is removed from the memo, keeping just the description. Enforced **twice**: as a prompt rule (a stated one-exception) *and* as a deterministic code backstop — a private `stripInitialsPrefix()` called at the end of `extractReceiptData()`. The backstop guards against stripping legitimate word prefixes like `"Postage:"` / `"Peer meeting:"` (3+ letters are left alone; only 1–2-caps + colon are stripped).
    - **Statement row matching.** `matchStatementCandidate()` returns the whole matched statement row (± $0.01 amount, then date tie-break). `matchStatementRow()` (description fallback, best-effort) and `matchStatementNotes()` (returns the row's `notes`) both delegate to it. `matchStatementCandidate($strict)`: in **strict** mode, when multiple rows share an amount and none resolves by date, it returns `null` and logs — rather than guessing an arbitrary first candidate. `matchStatementNotes()` uses `strict=true` (attaching the wrong cardholder's memo is worse than none); the description fallback stays non-strict/best-effort.
+   - **Month+day fallback for OCR year drift.** The date tie-break among same-amount candidates first tries exact `Y-m-d`, then falls back to a **month+day match that ignores the year**. This handles the AI OCR'ing the wrong *year* on a receipt (e.g. `2025` vs `2026`) — without the fallback the authoritative cardholder-written statement `Notes` memo failed to attach and silently fell back to AI inference, even though the correct row was present.
    - **Zero-total fallback** — if the extracted/docx total is 0, `lookupStatementTotalByVendor()` derives the amount from the billing statement (see *Statement structure* below)
    - Build base filename (no suffix yet) and store in `$pendingRenames`
    - On extract failure → move to `Archive/exception/` immediately
@@ -85,7 +89,7 @@ Credit Card Receipts/
    - If base name appears more than once, assign `_a`, `_b`, `_c`... suffix to **all** colliding files (including the first)
    - PATCH SharePoint to rename + move to `Archive/` (flat — no person subfolder)
    - **Multi-entry docx expansion** — a `.docx` whose `extractDocxData()` produced multiple `line_items[]` (e.g. a subway-rides document with 14 dated trip entries) is expanded into one Excel row per entry
-8. **Excel generation** — one `.xlsx` per person (PhpSpreadsheet); columns: Row #, Account Name, QB Vendor, Payment Amount, Date, Payment Method, Payment Memo, QB Description, Class, Category, Payment Account, Ref No.
+8. **Excel generation** — one `.xlsx` per person (PhpSpreadsheet); columns: Row #, Account Name, QB Vendor, Payment Amount, Date, Payment Method, Payment Memo, QB Description, Class, Category, Payment Account, Ref No., **Review** (in-sheet flag column — see *Excel `Review` column*).
 9. **SharePoint upload** — each person's Excel is uploaded to **that person's own**
    `Credit Card Receipts/{Person} CC receipts/{Year}/3. QB Excel/` folder via Graph API PUT (see *Per-person QB Excel upload location* below)
 10. **Summary email** — sent with Excel files attached; To: Jheanelle, CC: Magdalena, BCC: devteam@togatech.com. `sendSummaryEmail()` takes a `$warnings` param and renders an **"Amounts to verify"** section listing rows whose amount could not be reconciled against the statement (see below).
@@ -102,6 +106,19 @@ email's "Amounts to verify" section for human review before QuickBooks import. D
 sourced from the statement) and receipts with no loaded statement are excluded. Auto-overriding
 `payment_amount` from the statement was **deliberately deferred** as higher-risk.
 
+### Excel `Review` column
+The client reviews the generated **Excel**, not the summary email — so unverified amounts and
+deliberately-blank memos must surface **in the sheet**, not only in the email. Before this, an
+amount flagged only in the email looked (in the sheet) like an invented charge. `buildExcelRow()`
++ `generateExcel()` (header + row) add a `Review` column that flags:
+- `"VERIFY AMOUNT (no matching charge on statement)"` — the extracted total matches no statement
+  charge (same condition as the email's "Amounts to verify"). Real case: an **Ololo Safari**
+  receipt whose foreign-currency KES total (`$172,872`) was OCR'd instead of the USD charge
+  (`$1,340.09`).
+- `"ADD MEMO (no note found on receipt)"` — the memo is deliberately blank because no human-added
+  note was found (the correct outcome for a note-less receipt — see the verbatim/no-invention memo
+  rule).
+
 ### Renamed file format
 ```
 YYYY.MM.DD Name of Cardholder_VendorName_Amount[_a].ext
@@ -246,6 +263,31 @@ the old one-shot `/tmp/tow_restore_archives.php` CLI script.
 walks all subfolders of `Archive/` (both the processed root and `exception/`) and collects
 every non-folder item into a flat accumulator array.
 
+**Cycle-aware restore (`$scopeToCycle`, default `true`).** `MoveBack()` takes a
+`$scopeToCycle` param (default `true`) plus helpers `cycleDateWindow()` and `archivedFileDate()`.
+When scoped, restoring a billing cycle only moves files whose **archived-filename date**
+(`YYYY.MM.DD`, extracted by `archivedFileDate()`) falls inside that cycle's window
+(`cycleDateWindow()` — Amex cycle = the day after the prior 3rd through this 3rd, inclusive).
+Previously `MoveBack` dumped the **entire** Archive into one cycle folder, collapsing multiple
+cycles together (see the *Mixed-cycle receipts* gotcha, now largely addressed by default scoping).
+Pass `$scopeToCycle = false` to restore the whole Archive regardless of date.
+
+### DownloadCycleFiles action (read-only offline export)
+`DownloadCycleFiles(string $person, string $billingCycle, string $destDir, ?string $year = null, ?string $match = null, bool $includeArchive = ...): string`
+is a **read-only** scripted action that downloads a person's cycle receipts plus their statement
+Excel(s) to a **local directory** for offline review. It **moves nothing** on SharePoint and
+sends no email. `$destDir` is constrained to a path **inside `__DIR__`** (no arbitrary
+filesystem writes). It walks the `Archive/` too (`collectPersonArchiveFiles()`) since processed
+receipts get archived, and falls back to **all** of a person's statements when none matches the
+cycle name.
+
+```json
+{
+  "action": "Client/TowFoundation/ProcessReceipts/DownloadCycleFiles",
+  "parameters": { "person": "Nadia Alia", "billingCycle": "Amex ending in 07-03-2026", "destDir": "cycle_review", "year": "2026" }
+}
+```
+
 ### Reprocessing a cycle locally (operational)
 To re-run one cardholder + cycle after a code fix, without deploying:
 1. **Restore receipts** — call `MoveBack(person, "Amex ending in MM-DD-YYYY", "2026")` to move
@@ -387,6 +429,9 @@ Fatal errors send only to `NOTIFY_EMAIL_DEV` (no CC/BCC).
   distinct same-amount charges); genuinely distinct files with the same content are all kept.
 - **Unmapped AI categories pass through** — if the AI returns a category string not in `TowFoundationCategories.php`, `mapCategory()` logs a warning and writes the raw AI string to the Excel. Check error logs after a run if the Category column looks odd; add the new value to the map file to fix it.
 - **Short vendor acronyms (< 4 chars) require whole-word match** — the substring pass uses a word-boundary regex for needles under 4 chars to avoid false positives (e.g. `"UPS"` inside `"USPS"`). If a short vendor name is not matching, check that the QB vendor list entry starts or ends with the acronym as a whole word.
+- **Local `Run()` always ends in a non-fatal `Unknown database 'logs_towfoundation'` error** — running `Run()` locally (env `dev-rohan-mac`, vhost `http://worker2/`) always throws at the `_Email->send()` step because the local environment lacks the `Logs_Towfoundation` DB. **This is non-fatal to the run's real work**: the per-person Excel is generated **and** uploaded to SharePoint **before** the email step, so the processing succeeds even though the run reports an error at the very end. Do not treat this trailing error as a failed run locally.
+- **Nadia Alia's statements are raw Amex "Transaction Details" exports with no `Notes` column** — so there is **no** verbatim-note source for her, and blank memos flagged `ADD MEMO` are the **correct** outcome (not a bug). Her billing-cycle folders also don't align with her statement files, and no statement covers the `06-04..07-03` window. (Distinct from, and in addition to, the general missing-statement gotcha above.)
+- **Talos endpoint currently points at beta** — per a config comment, `/api/ai/generate` is pointed at `api.beta.togaiq.com` (beta), not production. Relevant if receipt-extraction behavior differs from prod Talos.
 - **Restoring archived receipts** — use the committed `MoveBack` action (see *MoveBack action* above), not a one-shot CLI script. It moves all non-Excel receipt files from a person's `Archive/` back into the named billing-cycle folder. (The earlier throwaway `/tmp/tow_restore_archives.php` script is obsolete.)
 - **Mixed-cycle receipts can collect in Archive** — receipts from a prior run can linger in a person's `Archive/` under a different billing cycle (e.g. January receipts found alongside May during an Emily Tow restore). `MoveBack` moves **everything** in `Archive/` back, so after a restore, verify the receipts belong to the cycle you intend to reprocess; stray off-cycle files should be moved (via Graph API) to their own billing-cycle folder so they are not reprocessed with the wrong cycle.
 
@@ -420,6 +465,7 @@ Fatal errors send only to `NOTIFY_EMAIL_DEV` (no CC/BCC).
 
 ## Change history
 
+- 2026-07-17 — **Memo fidelity overhaul + in-sheet Review column + cycle-aware MoveBack + offline export.** All in `ProcessReceipts.php`. (1) FIXED: the AI was paraphrasing/inventing the payment memo — rewrote the `extractReceiptData()` prompt + `payment_memo` schema to transcribe a human-added note **verbatim** and return **`null`** (never infer) when none exists; removed the old "infer a memo" instruction (client-reported: `"Lucy Ball of Lone Pine Foundation"`→`"LB LPFoundation"`, and a fabricated Asana memo). (2) FIXED: statement `Notes` override silently failed when AI OCR'd the wrong **year** — `matchStatementCandidate()` now falls back to a month+day (year-ignoring) date match so the authoritative Notes memo still wins despite year drift. (3) BUILT: in-sheet `Review` column (`buildExcelRow`+`generateExcel`) flags `"VERIFY AMOUNT (no matching charge on statement)"` (e.g. Ololo Safari KES `$172,872` OCR'd vs USD `$1,340.09`) and `"ADD MEMO (no note found on receipt)"` — the client reviews the Excel, not the email, so unverified/blank rows must surface in the sheet. (4) BUILT: prompt now scans the **entire page** (margins/corners/header/beside address) for visually-distinct human notes (missed a highlighted Optimum "Telephone & Internet" box, a red Garelick "Board Meeting … Remaining Deposit", an orange Lucid "IT Software License"). (5) BUILT: strip a leading 1–2-caps-plus-colon initials tag (`"RF:"`) from the memo — enforced in the prompt **and** as a deterministic `stripInitialsPrefix()` backstop that leaves 3+-letter prefixes (`"Postage:"`) alone. (6) BUILT: cycle-aware `MoveBack($scopeToCycle=true)` + `cycleDateWindow()`/`archivedFileDate()` — restore now moves only files whose archived-filename date falls in the cycle window (Amex = day-after-prior-3rd → this-3rd), instead of dumping the whole Archive into one cycle. (7) BUILT: read-only `DownloadCycleFiles(person, billingCycle, destDir, year, match, includeArchive)` — downloads a person's cycle receipts + statement Excel(s) to a local dir (constrained inside `__DIR__`), walking Archive via `collectPersonArchiveFiles()`, falling back to all statements when none matches. (8) DISCOVERED: local `Run()` always ends in a non-fatal `Unknown database 'logs_towfoundation'` at `_Email->send()` (Excel is generated + uploaded before that step, so work succeeds); Nadia Alia's statements are raw Amex "Transaction Details" exports with no `Notes` column (blank `ADD MEMO` memos are correct for her; no statement covers `06-04..07-03`); Talos is currently pointed at beta (`api.beta.togaiq.com`). Code changes remain **uncommitted** on the `_production` working tree (`fix/towfoundation-memo-verbatim` branch proposed, not yet created). (rgirish)
 - 2026-07-13 — **Read-only Preview action + Ligia map-key fix + July cycle findings.** (1) BUILT: `Preview(?year, ?person, ?billingCycle): string` — a true dry-run that reuses `walkReceiptsFolder()` and the same filters as `Run()` but only tallies (downloads/extracts/moves nothing, sends no email), returning pretty JSON with per-person receipt counts, cycles, fileTypes, up-to-3 sampleFiles, and hasStatement, plus a softer `personsWithNoReceipts` list. Read `persons[]` as the authoritative empty signal; `personsWithNoReceipts` is derived from statement-only folders and also surfaces stray top-level folders (e.g. "Processed") as cosmetic noise. (2) FIXED: `CLASS_MAP`/`PAYMENT_ACCOUNT_MAP` keys for Ligia Marroquin Soto were `"Ligia Marroquin"` (no "Soto") — never hit against the folder-derived `"Ligia Marroquin Soto"`, so her Class + Payment Account silently blanked (both maps fall back to `''` with no warning). Renamed keys to `"Ligia Marroquin Soto"` (Class ⇒ `Administration:Operations`, Payment Account ⇒ `AMEX Open Credit Card:Ligia Marroquin-Soto Amex CC`); broadened the person-folder-normalization gotcha — map keys MUST equal `folder − " CC receipts"`. (3) DISCOVERED: July Amex `"07-03-2026"` has 72 receipts across 9 people (Nadia Alia 43 …); Diane Sierpina, Brent Peterkin, Susan Ransden empty for July; Ryan Farrell + Magdalena Minta have July receipts but no statement → AI-inferred memos; **Michael Zuber Zander** has a receipts folder but is in neither map (and zero receipts) → left unmapped pending client-provided QB Class + Payment Account. Also recorded that SharePoint creds live in the `[sharepoint_towfoundation]` Config ini section, read via `_Config::sharepoint_towfoundation()`. (rgirish)
 - 2026-07-13 — **Per-person QB Excel location fix + two production findings.** (1) FIXED: `uploadExcelToSharePoint()` now writes each person's generated QB Excel to that person's own `Credit Card Receipts/{Person} CC receipts/{Year}/3. QB Excel/` subfolder (matching the docblock and client expectation) instead of a single shared root `3. QB Excel/`; signature is now `(accessToken, driveId, personFolderName, year, excelName, tmpFile)`, fed by a `$personFolders` map populated first-write-wins in Pass 1 with an `error_log` fallback to `{personName}/{currentYear}`. (2) DISCOVERED (production-critical gotcha): the statement-Notes lookup is EXACT string equality between the statement filename (`cycleKey`) and the receipt's billing-cycle FOLDER name — no date/fuzzy fallback — so a card whose folder has no identically-named statement silently skips the authoritative Notes memo and falls back to AI inference with zero signal (confirmed for Ryan Farrell's Mastercard folder vs. an Amex-named statement); also noted `computeRefNo()` hard-codes cycle-end day `03`, giving Mastercard rows a `…03…` Ref No. (pre-existing, unchanged). (3) DECIDED: monthly production procedure is two explicit per-card `Run` invocations (Amex + Ryan Farrell's Mastercard), NOT a "MM-YYYY month sweep" — the sweep prototype was deliberately reverted because `PAYMENT_ACCOUNT_MAP` is keyed per-person not per-card and would misattribute the QB Payment Account for anyone holding two cards in a swept month; recorded production-safety facts (no dry-run — `limit` still archives + emails; file moves reversible via `MoveBack` but email is not recallable; recipients are compile-time `NOTIFY_*` constants). (rgirish)
 - 2026-07-09 — **Local reprocess-all-cardholders verification pass — three more fixes + two data/ops findings (code-only, uncommitted).** These fixes were found *after* the six-fix pass earlier the same day, while re-running every cardholder locally and inspecting the uploaded QB files. (1) **Statements-folder detection matched plural `reports` only** — `walkReceiptsFolder()` now matches `stripos(name,'report')` (singular), so `"{Name} report"` folders (Angela Leis, Kathryn Lockhart, Meghan Lowney, Eileen Wiseman) actually load their statement; without this the earlier Notes-memo fix silently did nothing for them in a real `Run()` (folder-name matching is the gate — verifying `matchStatementNotes()` in isolation is not enough). (2) **docx run-boundary word-splitting** — `</w:r>` was replaced with a space, injecting spaces mid-word (`"Innoc ence"`, `"Armstron g"`, `"202 6 . 0 6 .0 3"`) in Emily Tow's subway memos; now replaced with `''` at both sites (`extractDocxData`, `parseDocxRideLines`) — only `</w:p>` is a real line break. (3) **docx "Closing Date" header mis-parsed as first ride** — the ride regex read `"AMEX Closing Date 2026.06.03"` as a ride and swallowed the real first ride; now stripped upfront via `preg_replace('/(?:AMEX|Visa)\s+Closing\s+Date\s+\d{4}[.\/-]\d{1,2}[.\/-]\d{1,2}/i','',$text)` (kept 4-digit dot dates for Jheanelle rather than narrowing ride dates). Data/ops findings (no code change): **Nadia Alia's** June statement is absent from SharePoint → all her memos AI-inferred until the client uploads `"Amex ending in 06-03-2026.xlsx"` with a Notes column; local worker HTTP endpoint 500s on direct `{action,parameters}` calls (post 2026-07-07 `Core.WorkerJobs` dependency absent in dev) → reprocess a cycle via CLI + `MoveBack` (debug_mode redirects mail). Verified correct: Angela, Emily (incl. 14 subway line items), Kathryn, Meghan (Kellari $548.11 flagged), Michael (all 5 charges). Nadia pending client statement; Jheanelle + Diane not yet reprocessed. (rgirish)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.366",
+  "version": "1.0.367",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",