toga-ai 1.0.307 → 1.0.309
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -6,7 +6,7 @@ project: Database Changes
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: architecture
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-07-
|
|
9
|
+
updated: 2026-07-09
|
|
10
10
|
owners: [jcardinal, mhammontree, bala]
|
|
11
11
|
files:
|
|
12
12
|
- Core/
|
|
@@ -168,35 +168,62 @@ its own header.)
|
|
|
168
168
|
5. Never edit or re-date an already-applied file — add a new dated file instead. Never put new
|
|
169
169
|
work in a `HISTORIC` folder.
|
|
170
170
|
|
|
171
|
-
## Bulk data loads —
|
|
171
|
+
## Bulk data loads — batch, and stage large sets in a temp table
|
|
172
172
|
|
|
173
|
-
When a change inserts many rows (reference-data / code-table loads), write it as
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
173
|
+
When a change inserts many rows (reference-data / code-table loads), write it as **batched
|
|
174
|
+
statements**, not one `INSERT` per row. The database is billed per query and the external
|
|
175
|
+
executor runs each statement as its own round-trip, so N separate inserts cost N times the
|
|
176
|
+
round-trips (and query cost) of a batched load.
|
|
177
177
|
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
178
|
+
To stay **idempotent** without a unique key to `INSERT IGNORE` against, anti-join the source
|
|
179
|
+
set to the target and insert only the misses.
|
|
180
|
+
|
|
181
|
+
**Small sets (a few hundred rows):** a single `INSERT ... VALUES (...),(...),...`, or an
|
|
182
|
+
`INSERT ... SELECT` over a `UNION ALL` derived row set, is fine:
|
|
182
183
|
|
|
183
184
|
```sql
|
|
184
185
|
INSERT INTO TargetTable (uuid, keyColumn, otherColumn)
|
|
185
|
-
SELECT
|
|
186
|
-
src.uuid,
|
|
187
|
-
src.keyColumn,
|
|
188
|
-
src.otherColumn
|
|
186
|
+
SELECT src.uuid, src.keyColumn, src.otherColumn
|
|
189
187
|
FROM (
|
|
190
188
|
SELECT '<uuid>' AS uuid, '<key>' AS keyColumn, '<value>' AS otherColumn
|
|
191
189
|
UNION ALL SELECT ...
|
|
192
190
|
) AS src
|
|
193
191
|
LEFT JOIN TargetTable existing ON existing.keyColumn = src.keyColumn
|
|
194
|
-
WHERE
|
|
195
|
-
|
|
192
|
+
WHERE existing.id IS NULL;
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
**⚠ Large sets — never build them as a long `UNION ALL` chain.** MySQL parses a `UNION ALL`
|
|
196
|
+
chain **recursively**, one parser-stack frame per link. Around ~2,000+ links overflow the
|
|
197
|
+
default 256 KB per-thread stack and the statement dies at **parse time** (before a single row
|
|
198
|
+
is processed) with:
|
|
199
|
+
|
|
200
|
+
ERROR 1436 (HY000): Thread stack overrun: <n> bytes used of a 262144 byte stack, and
|
|
201
|
+
20000 bytes needed. Use 'mysqld --thread_stack=#' to specify a bigger stack.
|
|
202
|
+
|
|
203
|
+
This is a parser-recursion limit — **not** a row-volume, `max_allowed_packet`, or
|
|
204
|
+
server-misconfiguration issue, and **do not** raise `thread_stack` on the server for a one-off
|
|
205
|
+
load. Instead, stage the rows in a **session-scoped temp table via a flat multi-row `VALUES`
|
|
206
|
+
list** (which the parser reads as a flat list, no recursion), then run the anti-join once from
|
|
207
|
+
the staging table:
|
|
208
|
+
|
|
209
|
+
```sql
|
|
210
|
+
CREATE TEMPORARY TABLE _stage (
|
|
211
|
+
uuid VARCHAR(36) NOT NULL, keyColumn VARCHAR(255) NOT NULL, otherColumn TEXT NULL);
|
|
212
|
+
INSERT INTO _stage (uuid, keyColumn, otherColumn)
|
|
213
|
+
VALUES ('<uuid>','<key>','<value>'), ('<uuid>','<key>','<value>'), ... ; -- flat, thousands of rows
|
|
214
|
+
INSERT INTO TargetTable (uuid, keyColumn, otherColumn)
|
|
215
|
+
SELECT src.uuid, src.keyColumn, src.otherColumn
|
|
216
|
+
FROM _stage AS src
|
|
217
|
+
LEFT JOIN TargetTable existing ON existing.keyColumn = src.keyColumn
|
|
218
|
+
WHERE existing.id IS NULL;
|
|
219
|
+
DROP TEMPORARY TABLE _stage;
|
|
196
220
|
```
|
|
197
221
|
|
|
198
|
-
|
|
199
|
-
|
|
222
|
+
The `TEMPORARY` table is session-scoped (safe on production, auto-dropped on disconnect even if
|
|
223
|
+
the run aborts). A multi-row `VALUES` load of a few thousand rows is a few hundred KB — well
|
|
224
|
+
under the 64 MB `max_allowed_packet` default. First hit in
|
|
225
|
+
`Client_Aig/2026-06-18a - TRUE-79534 AIG SaleItem codes.sql` (2,200 AIG SaleItem codes); see
|
|
226
|
+
`clients/aig/features/entitlement-intake.md` → *Uploading new codes*.
|
|
200
227
|
|
|
201
228
|
## Self-referencing DELETE — wrap the subquery in a derived table
|
|
202
229
|
|
|
@@ -88,17 +88,27 @@ When AIG sends a new "Active SaleItemID" spreadsheet (columns `SaleItemID`, `Des
|
|
|
88
88
|
3. Generate **fully-random UUIDs** (e.g. `uuid4`), never the time-based MySQL `UUID()` —
|
|
89
89
|
per the 2.0 standard. Bake literal UUIDs into the SQL since a migration can't call PHP's
|
|
90
90
|
`_String::generateUuid()`.
|
|
91
|
-
4.
|
|
92
|
-
|
|
93
|
-
can't help)
|
|
91
|
+
4. Feed the rows through a **session-scoped temp table populated by a flat multi-row
|
|
92
|
+
`VALUES` list**, then anti-join once into `Items` — idempotent because `Items` has **no
|
|
93
|
+
unique key on `partNumber`** (so `INSERT IGNORE` can't help). Do **not** assemble the rows
|
|
94
|
+
as a long `UNION ALL` derived table: at AIG's ~2,200-code scale that overflows MySQL's
|
|
95
|
+
parser stack and fails at **parse time** with **ERROR 1436 (HY000) Thread stack overrun**
|
|
96
|
+
(see the gotcha below and the dbchanges2 architecture bulk-load rule). A flat `VALUES`
|
|
97
|
+
list does not recurse and stays well under `max_allowed_packet`:
|
|
94
98
|
```sql
|
|
99
|
+
CREATE TEMPORARY TABLE _aig_saleitem_stage (
|
|
100
|
+
uuid VARCHAR(36) NOT NULL, partNumber VARCHAR(255) NOT NULL, description TEXT NULL);
|
|
101
|
+
INSERT INTO _aig_saleitem_stage (uuid, partNumber, description)
|
|
102
|
+
VALUES ('<uuid>','<part>','<desc>'), ('<uuid>','<part>','<desc>'), ... ; -- flat, ~2,200 rows
|
|
95
103
|
INSERT INTO Items (uuid, partNumber, description)
|
|
96
104
|
SELECT src.uuid, src.partNumber, src.description
|
|
97
|
-
FROM
|
|
98
|
-
UNION ALL SELECT ... ) AS src
|
|
105
|
+
FROM _aig_saleitem_stage AS src
|
|
99
106
|
LEFT JOIN Items existing ON existing.partNumber = src.partNumber
|
|
100
107
|
WHERE existing.id IS NULL;
|
|
108
|
+
DROP TEMPORARY TABLE _aig_saleitem_stage;
|
|
101
109
|
```
|
|
110
|
+
The `TEMPORARY` table is session-scoped (safe on production, auto-dropped on disconnect
|
|
111
|
+
even if the run aborts). Do **not** raise `thread_stack` on the server for a one-off load.
|
|
102
112
|
|
|
103
113
|
## Client variations
|
|
104
114
|
|
|
@@ -121,6 +131,13 @@ this interceptor or use this dual-purpose Items pattern.
|
|
|
121
131
|
codes are **not** in the SaleItemID spreadsheet and must be sourced separately if missing.
|
|
122
132
|
- **No unique key on `Items.partNumber`** — use an anti-join for idempotent loads, not
|
|
123
133
|
`INSERT IGNORE`.
|
|
134
|
+
- **A long `UNION ALL` derived table fails at ~2,200 rows with ERROR 1436 "Thread stack
|
|
135
|
+
overrun."** The AIG SaleItem load is exactly this scale. MySQL builds the `UNION ALL`
|
|
136
|
+
chain recursively, so ~2,200 links overflow the default 256 KB per-thread parser stack and
|
|
137
|
+
the statement dies at **parse time** (before any row is processed) — it is a parser-recursion
|
|
138
|
+
limit, **not** a row-volume, `max_allowed_packet`, or server-config problem. Stage the rows
|
|
139
|
+
in a temp table via a flat multi-row `VALUES` list and anti-join from there (see *Uploading
|
|
140
|
+
new codes* step 4). Do not raise `thread_stack` for a one-off load.
|
|
124
141
|
- **An `entitlementFulfillmentType` EV-12 VALIDATION error is NOT a bad client payload —
|
|
125
142
|
it's this interceptor's injected field failing to resolve.** AIG never sends that field;
|
|
126
143
|
the interceptor injects it as `{name: "Repair"}`. The error means the V2 resolver could not
|
|
@@ -136,6 +153,11 @@ this interceptor or use this dual-purpose Items pattern.
|
|
|
136
153
|
|
|
137
154
|
## Change history
|
|
138
155
|
|
|
156
|
+
- 2026-07-09 — TRUE-79534 SaleItem load reworked: the 2,200-code `INSERT ... SELECT FROM
|
|
157
|
+
(<2,199 `UNION ALL` links>)` derived table failed at parse time with ERROR 1436 (Thread
|
|
158
|
+
stack overrun). Rebuilt as a `TEMPORARY` staging table filled by a flat multi-row `VALUES`
|
|
159
|
+
list + the same anti-join into `Items` (identical rows/UUIDs, 0 duplicate partNumbers).
|
|
160
|
+
(mhammontree)
|
|
139
161
|
- 2026-07-09 — TRUE-79978: root-caused an `entitlementFulfillmentType` EV-12 on beta/QA. The
|
|
140
162
|
field is injected (name-only) by `prePost`; the beta `Client_Aig.Apis_RecordFields` override
|
|
141
163
|
row (apiId=2, recordFieldId=1202) had `overrideIsIdentifier=NULL` + `overrideChildPolicy=MATCH_CREATE`,
|
|
@@ -41,7 +41,7 @@ Credit Card Receipts/
|
|
|
41
41
|
{Person}/
|
|
42
42
|
{Year}/
|
|
43
43
|
{BillingCycleFolder}/ ← receipt files live here (e.g. "Amex ending in 06-03-2026")
|
|
44
|
-
{Person}
|
|
44
|
+
{Person} report[s]/ ← .xlsx billing statements (folder name matched on singular "report")
|
|
45
45
|
Archive/ ← successfully processed receipts land here (renamed, flat)
|
|
46
46
|
Archive/exception/ ← failed receipts land here (original name preserved, flat)
|
|
47
47
|
```
|
|
@@ -51,13 +51,23 @@ Credit Card Receipts/
|
|
|
51
51
|
1. **Auth** — OAuth2 client-credentials token from Microsoft Graph
|
|
52
52
|
2. **Walk** — `walkReceiptsFolder()` enumerates all receipt files and statement Excels
|
|
53
53
|
3. **Filter** — optional `$year` / `$person` / `$billingCycle` / `$limit` applied to the receipt list. `$billingCycle` uses `str_ends_with()` so passing just the date (e.g. `"06-03-2026"`) matches any card type. Person folders named `"Brent Peterkin CC receipts"` are normalized to `"Brent Peterkin"` via regex strip of ` CC receipts` suffix before any map lookups.
|
|
54
|
-
4. **Pre-load statements** — all billing-cycle `.xlsx` files under
|
|
55
|
-
and parsed upfront via PhpSpreadsheet (auto-detect header row by
|
|
56
|
-
description/amount keywords)
|
|
54
|
+
4. **Pre-load statements** — all billing-cycle `.xlsx` files under a person's statements
|
|
55
|
+
folder are downloaded and parsed upfront via PhpSpreadsheet (auto-detect header row by
|
|
56
|
+
scanning for description/amount keywords). **Statements folder detection matches the
|
|
57
|
+
singular `report`.** `walkReceiptsFolder()` recognizes a person's statements folder via
|
|
58
|
+
`stripos($child->name,'report')` (singular). It previously matched `'reports'` (plural)
|
|
59
|
+
only, so cardholders whose folder is named `"{Name} report"` (singular — Angela Leis,
|
|
60
|
+
Kathryn Lockhart, Meghan Lowney, Eileen Wiseman) had their statement Excel **never
|
|
61
|
+
loaded**: `$parsedStatements[$person]` was unset and the entire `Notes`-memo override was
|
|
62
|
+
silently skipped (AI-inferred memos shipped instead). The singular match also covers
|
|
63
|
+
`"reports"`, `"Report"`, `"Amex reports"`. **Folder-name matching is the gate for the
|
|
64
|
+
whole Notes-memo feature** — verifying `matchStatementNotes()` in isolation is
|
|
65
|
+
insufficient; the statement must actually load during `Run()`, so verify against a real
|
|
66
|
+
run that `$parsedStatements` includes the person.
|
|
57
67
|
5. **Pass 1 — Extract** — for each receipt:
|
|
58
68
|
- Enforce type + size limit: unsupported mime types (`application/octet-stream`) throw immediately; 4 MB cap for images, 10 MB for documents
|
|
59
69
|
- Download file bytes from SharePoint
|
|
60
|
-
- **`.docx` files bypass Talos** and route to `extractDocxData()` (Talos returns HTTP 500 on a `.docx` MIME type — it only accepts PDF/image). That method opens the docx as a ZIP, extracts `word/document.xml`, strips tags, and regex-parses ride lines into `line_items[]`. The ride-line regex (in both `extractDocxData()` and `parseDocxRideLines()`) accepts **slash or dot** date separators (`[\/.]` — e.g. `05/06/2025` and `05.06.2025`) and treats the `[:\-]?` description delimiter as **optional**. Dash-separated dates (`05-06-2025`) are intentionally **not** supported — a dash would clash with the `:`/`-` description delimiter. A docx that matches zero ride lines throws `ReceiptProcessingException` and produces no Excel, so a new date format silently zero-outputs — widen the regex when a cardholder introduces one
|
|
70
|
+
- **`.docx` files bypass Talos** and route to `extractDocxData()` (Talos returns HTTP 500 on a `.docx` MIME type — it only accepts PDF/image). That method opens the docx as a ZIP, extracts `word/document.xml`, strips tags, and regex-parses ride lines into `line_items[]`. **Tag-stripping — a run boundary is not a word boundary.** When flattening the XML to text, `</w:r>` (run end) must be replaced with `''` (empty string), **not** a space — Word splits a single word across multiple runs, so replacing `</w:r>` with a space injects spaces mid-word (`"Innocence"`->`"Innoc ence"`, `"Armstrong"`->`"Armstron g"`, `"2026.06.03"`->`"202 6 . 0 6 .0 3"`). Only `</w:p>` (paragraph end) is a real line break (`"\n"`); `<w:t>` text keeps its own spaces. Both `extractDocxData()` and `parseDocxRideLines()` do this replacement. **Header strip before ride parsing.** `extractDocxData()` first removes any statement header line via `preg_replace('/(?:AMEX|Visa)\s+Closing\s+Date\s+\d{4}[.\/-]\d{1,2}[.\/-]\d{1,2}/i','',$text)` — otherwise the ride regex greedily matches `"AMEX Closing Date 2026.06.03"` as the first ride (reading the `YYYY.MM.DD` as a date and swallowing the real first ride into its description, corrupting row 1 and dropping the last real ride). The header is stripped by keyword rather than by restricting ride dates to 2-digit years — narrowing ride dates would regress Jheanelle's 4-digit dot dates. The ride-line regex (in both `extractDocxData()` and `parseDocxRideLines()`) accepts **slash or dot** date separators (`[\/.]` — e.g. `05/06/2025` and `05.06.2025`) and treats the `[:\-]?` description delimiter as **optional**. Dash-separated dates (`05-06-2025`) are intentionally **not** supported — a dash would clash with the `:`/`-` description delimiter. A docx that matches zero ride lines throws `ReceiptProcessingException` and produces no Excel, so a new date format silently zero-outputs — widen the regex when a cardholder introduces one
|
|
61
71
|
- All other types POST to Talos AI `/api/ai/generate` → structured `{vendor_name, invoice_date, total, payment_memo, category, ...}`
|
|
62
72
|
- **Year guard on AI dates** — if Talos returns an `invoice_date` whose year is more than 1 year from the current year (AI hallucination on two-digit year inputs, e.g. `5/13/76` → 1976, `5/28/28` → 2028), the year is clamped to the current year while month/day are preserved
|
|
63
73
|
- **Payment memo — the statement `Notes` column is authoritative.** `loadStatementExcel()` detects a `Notes` column (exact-match `"notes"`/`"note"` first; a substring fallback runs only if no exact header was found, guarded by `$notesExact`) and attaches a `notes` field to each parsed row. In `Run()`, if the matching statement row has a **non-empty `notes` value**, it is used **verbatim** as `payment_memo` — this is the cardholder's own hand-written memo (e.g. `"Postage: Board materials shipped to Frank Tow"`) and beats any AI-inferred memo. Only if the row has no notes does processing fall back to the prior AI/description behavior. This **supersedes** the earlier `isActivityHeadingMemo()` allow-list, which dropped the cardholder's Notes entirely (it only detected columns containing `"description"`/`"memo"`) and rejected valid headings like `Communications:`, `Postage:`, `Office Supplies:`, `RF:` — shipping AI memos in their place.
|
|
@@ -201,6 +211,22 @@ the old one-shot `/tmp/tow_restore_archives.php` CLI script.
|
|
|
201
211
|
walks all subfolders of `Archive/` (both the processed root and `exception/`) and collects
|
|
202
212
|
every non-folder item into a flat accumulator array.
|
|
203
213
|
|
|
214
|
+
### Reprocessing a cycle locally (operational)
|
|
215
|
+
To re-run one cardholder + cycle after a code fix, without deploying:
|
|
216
|
+
1. **Restore receipts** — call `MoveBack(person, "Amex ending in MM-DD-YYYY", "2026")` to move
|
|
217
|
+
archived receipts back into the billing-cycle folder.
|
|
218
|
+
2. **Run via CLI, not the HTTP endpoint** — invoke the action directly in PHP so the run
|
|
219
|
+
bypasses the Worker HTTP/`WorkerJobs` dispatch:
|
|
220
|
+
```
|
|
221
|
+
ENVIRONMENT=dev-... php -r '... require "_underscore.php"; echo _Worker_Client_TowFoundation_ProcessReceipts::Run(null,null,$person,$cycle);'
|
|
222
|
+
```
|
|
223
|
+
The local worker HTTP endpoint returns **HTTP 500** on direct `{action,parameters}` calls
|
|
224
|
+
because the 2026-07-07 change made the direct-payload path require a `Core.WorkerJobs`
|
|
225
|
+
table that local dev DBs lack (MySQL Error #1146 `core.workerjobs doesn't exist`). The CLI
|
|
226
|
+
path avoids that dispatch entirely.
|
|
227
|
+
3. **Email is safe locally** — each `Run()` sends the summary email, but with `debug_mode=1`
|
|
228
|
+
locally all mail is redirected to `send_debug_emails_to`, so no client contact is reached.
|
|
229
|
+
|
|
204
230
|
## Email routing
|
|
205
231
|
|
|
206
232
|
Defined as constants in `TowFoundation.php`:
|
|
@@ -254,6 +280,15 @@ Fatal errors send only to `NOTIFY_EMAIL_DEV` (no CC/BCC).
|
|
|
254
280
|
- **Person folder name normalization** — SharePoint folders are named `"Brent Peterkin CC receipts"` but `CLASS_MAP` / `PAYMENT_ACCOUNT_MAP` keys are just `"Brent Peterkin"`. The ` CC receipts` suffix is stripped via regex in `walkReceiptsFolder()`. Without this, Class and Payment Account columns are blank for those persons.
|
|
255
281
|
- **Archive paths must use the actual SharePoint folder name, not the normalized name** — the normalized person name (` CC receipts` stripped, spaces replaced) is for **map lookups only**. When building the SharePoint archive path, use the *actual* folder name (e.g. `"Emily Tow CC receipts"`), not the normalized `"Emily Tow"` — otherwise the PATCH move 404s because the path segment does not exist. Keep the normalized name and the real folder name as separate values.
|
|
256
282
|
- **`billingCycle` filter is suffix-match, not exact** — always pass just the date portion (`"06-03-2026"`), not the full folder name. Passing the full name (`"Amex ending in 06-03-2026"`) also works but would miss Mastercard folders.
|
|
283
|
+
- **Missing per-cycle statement → no Notes memos (data dependency, not a bug)** — statements
|
|
284
|
+
are keyed by `cycleKey = filename-without-extension`, and the code looks up the statement
|
|
285
|
+
whose name matches the billing-cycle folder (e.g. `"Amex ending in 06-03-2026"`). If a
|
|
286
|
+
cardholder's report folder has no statement file for that cycle, **zero Notes are loaded**
|
|
287
|
+
and **all** of that person's memos fall back to AI-inferred (their hand-written `"RF:"` red
|
|
288
|
+
memos never appear). As of the 2026-07-09 run, **Nadia Alia's** `"Nadia's Amex reports"`
|
|
289
|
+
folder held Jan–May statements but no `"Amex ending in 06-03-2026.xlsx"` (and even her 05-03
|
|
290
|
+
file has no `Notes` column). Resolution is a **client action**: upload Nadia's 06-03-2026
|
|
291
|
+
statement with a `Notes` column, then re-run her. Not a code fix.
|
|
257
292
|
- **Vendors not in QB vendor list** — as of June 2026 these vendors are not in Tow Foundation's QuickBooks, so `qb_vendor` is blank for their charges until the client adds them to QB and the vendor list is refreshed: **Ole Mole**, **AMORE PIZZA CAFE**, **Green & Tonic (New Canaan)**, **NCFP (National Center for Philanthropy)**, **Rippling**, **Langan's**. This is a client action, not a code fix.
|
|
258
293
|
- **Duplicate detection is within-run and `fileId`-keyed only** — dedup keys on the SharePoint
|
|
259
294
|
drive-item `fileId`, so it catches only a literal re-upload of the same item present in the
|
|
@@ -274,9 +309,16 @@ Fatal errors send only to `NOTIFY_EMAIL_DEV` (no CC/BCC).
|
|
|
274
309
|
- **Amount reconciliation is flag-only** — mismatched OCR amounts are surfaced in the "Amounts to
|
|
275
310
|
verify" email section but never auto-corrected from the statement; auto-override was deferred as
|
|
276
311
|
higher-risk.
|
|
312
|
+
- **Not yet reprocessed / undeployed (as of 2026-07-09)** — **Jheanelle** and **Diane** still need
|
|
313
|
+
a `MoveBack` + `Run` to pick up the docx run-boundary / Closing-Date-header fixes. All 2026-07-09
|
|
314
|
+
fixes are **local and uncommitted** — no deployment has been performed.
|
|
315
|
+
- **Subway docx line-item memos come from the docx, by design** — the per-ride memos on a subway
|
|
316
|
+
`.docx` are sourced from the docx line-item description, not the statement `Notes` column. This is
|
|
317
|
+
intentional; do not "fix" it by routing them through `matchStatementNotes()`.
|
|
277
318
|
|
|
278
319
|
## Change history
|
|
279
320
|
|
|
321
|
+
- 2026-07-09 — **Local reprocess-all-cardholders verification pass — three more fixes + two data/ops findings (code-only, uncommitted).** These fixes were found *after* the six-fix pass earlier the same day, while re-running every cardholder locally and inspecting the uploaded QB files. (1) **Statements-folder detection matched plural `reports` only** — `walkReceiptsFolder()` now matches `stripos(name,'report')` (singular), so `"{Name} report"` folders (Angela Leis, Kathryn Lockhart, Meghan Lowney, Eileen Wiseman) actually load their statement; without this the earlier Notes-memo fix silently did nothing for them in a real `Run()` (folder-name matching is the gate — verifying `matchStatementNotes()` in isolation is not enough). (2) **docx run-boundary word-splitting** — `</w:r>` was replaced with a space, injecting spaces mid-word (`"Innoc ence"`, `"Armstron g"`, `"202 6 . 0 6 .0 3"`) in Emily Tow's subway memos; now replaced with `''` at both sites (`extractDocxData`, `parseDocxRideLines`) — only `</w:p>` is a real line break. (3) **docx "Closing Date" header mis-parsed as first ride** — the ride regex read `"AMEX Closing Date 2026.06.03"` as a ride and swallowed the real first ride; now stripped upfront via `preg_replace('/(?:AMEX|Visa)\s+Closing\s+Date\s+\d{4}[.\/-]\d{1,2}[.\/-]\d{1,2}/i','',$text)` (kept 4-digit dot dates for Jheanelle rather than narrowing ride dates). Data/ops findings (no code change): **Nadia Alia's** June statement is absent from SharePoint → all her memos AI-inferred until the client uploads `"Amex ending in 06-03-2026.xlsx"` with a Notes column; local worker HTTP endpoint 500s on direct `{action,parameters}` calls (post 2026-07-07 `Core.WorkerJobs` dependency absent in dev) → reprocess a cycle via CLI + `MoveBack` (debug_mode redirects mail). Verified correct: Angela, Emily (incl. 14 subway line items), Kathryn, Meghan (Kellari $548.11 flagged), Michael (all 5 charges). Nadia pending client statement; Jheanelle + Diane not yet reprocessed. (rgirish)
|
|
280
322
|
- 2026-07-09 — **Client QA pass on a production run — six fixes (code-only, not deployed).** (1) Statement `Notes` column is now the **authoritative** payment memo, used verbatim; `loadStatementExcel()` detects `Notes` (exact-match first, guarded substring fallback) — replaces the `isActivityHeadingMemo()` allow-list that dropped hand-written memos (fixed Angela, Emily, Katy, Meghan, Nadia). (2) `matchStatementCandidate($strict)` returns `null` on ambiguous same-amount matches for the Notes memo instead of guessing; `matchStatementNotes()` is strict, description fallback stays best-effort. (3) Dedup now keys on SharePoint `fileId` only — the old person|date|vendor|amount key dropped distinct same-amount charges (Michael lost 2 of 5). (4) docx ride-line regex now accepts dot dates (`[\/.]`) with optional `[:\-]?` delimiter — Jheanelle's dot-format subway docx previously matched 0 lines and produced no output. (5) `buildExcelFileName()` now derives the reporting month from the billing-cycle end (`parseCycleEndDate()`/`resolvePersonCycle()`) not the charge date, so the filename agrees with the Ref No. (Diane). (6) Systemic discovery: non-docx `payment_amount` is AI-OCR only, never reconciled against the statement `Amount` (Meghan $548.11 vs $538.11); added flag-only `statementHasAmount()` + `$warnings` "Amounts to verify" email section — auto-override deferred. (rgirish)
|
|
281
323
|
- 2026-06-29 — **Reversed the over-aggressive memo override.** Replaced `isBareTranportationMemo()` (added earlier same day) with `isActivityHeadingMemo()`: the statement memo now overrides the AI memo **only** when it is a proper activity-purpose heading (e.g. `Grantee meeting:`, `Peer meeting:`), not whenever the AI memo started with `"Transportation:"`. The previous approach replaced semi-useful AI memos with raw bank strings (`"AplPay LYFT …"`, `"UBER"`) on ride-share/taxi rows where the statement has no activity heading — degraded 8 rows on Emily Tow's statement (1, 17, 18, 21, 26, 31, 32, 34). New priority: statement heading > AI memo, with AI memo kept for any non-heading statement description. (rgirish)
|
|
282
324
|
- 2026-06-29 — `payment_memo` override now also fires on a **bare** `"Transportation:"` AI memo, not just an empty one: new `isBareTranportationMemo()` lets the cardholder's statement Excel (the authoritative source for activity-purpose headings like "Peer meeting:", "Grantee event:") override the AI's generic transportation heading. Fixes wrong headings on Emily Tow May 2026 transit rows. Added the committed `MoveBack` action (+ `collectFilesRecursiveRaw()`) to restore a person's archived receipts into a billing-cycle folder for reprocessing, replacing the throwaway `/tmp/tow_restore_archives.php` script. (rgirish)
|
package/package.json
CHANGED