toga-ai 1.0.308 → 1.0.310

package/knowledge/1.0/apps/togadesk/features/email-to-ticket-intake.md

@@ -7,7 +7,7 @@ client: shared
 type: feature
 status: active
 updated: 2026-07-09
-owners: ["jcardinal", "dfranks"]
+owners: ["jcardinal", "dfranks", "mhammontree"]
 files:
   - crons/tickets.php
   - crons/tickets_prod.php
@@ -40,6 +40,19 @@ TOGa Desk ingests support email into tickets through a cron-driven IMAP poller
 3. **Parse & route (`emailToTicket()`):** extract from/to/cc/subject/body (with encoding fixes);
    match an existing ticket via `In-Reply-To` + `Message-ID` headers and **plus-addressing**
    (`support+<ticketnum>@…`). Matched ⇒ `addReply()`; otherwise ⇒ `add()`.
+   - **Noise suppression (before any ticket/attachment is created):** both feeders drop known
+     automated no-noise mail via the shared helper
+     `Ticket::isSuppressedNoiseEmail(?string $from, ?string $subject): bool` (static, in
+     `class.ticket.php`). It returns true only when the sender matches `/^no-?reply@/` **AND**
+     the subject starts with `staples po ` — both required so a real person is never blocked, and
+     the sender check is **domain-agnostic** (matches `noreply@togatech.com` /
+     `noreply@goagilant.com` and survives future sender-domain flips; the Staples PO alert sender
+     flipped `goagilant.com → togatech.com` on 2026-06-02). Wired into BOTH intake feeders:
+     `crons/tickets.php` adds it as one more `&&`-clause in the existing IMAP intake guard
+     (alongside the delivery-status / `MAILER-DAEMON` / Compass auto-reply filters);
+     `crons/pipe.php` uses it as an early guard-and-`exit(0)` right after the `$from`/`$subject`
+     parse. Each feeder logs a "suppressed" line. (TRUE-79975 — ~815 such alerts since Jun 2024,
+     ~704 of them landing in the dept-1 "Support Desk" catch-all and being closed by hand.)
 4. **Side effects:** auto-create a user when the sender domain maps to an existing client; assign
    to a department via the department email lookup; save attachments locally then push to **S3**
    (`App_Cloud::copyFileToS3()`, bucket `asifiles`); optionally generate Talos/TogaIQ summaries.
@@ -49,7 +62,12 @@ TOGa Desk ingests support email into tickets through a cron-driven IMAP poller
 ## Data model
 Writes `tickets`, `tickets_replies`, attachment `files`; reads config from the `config` table
 (`tickets_defaultdepartment`, `tickets_defaultclient`, `auto_close_tickets`,
-`email_from_address`, `timezone`). Department routing keys on `tickets_departments.email`.
+`email_from_address`, `timezone`). Department routing keys on a `tickets_departments.email`
+`LIKE` match against the To/CC addresses; **anything unmatched falls through to config
+`tickets_defaultdepartment` (= dept 1 "Support Desk" catch-all) and `tickets_defaultclient`** —
+which is why unrouted/automated mail piles into "Support Desk". There is **no allowlist/blocklist
+config framework**: noise filtering is ad-hoc inline guards in the crons (the hardcoded Compass
+"Automatic reply:" filter and now `isSuppressedNoiseEmail`).
 
 ## Gotchas / known issues
 - **`$processName` guard:** `tickets.php` throws if executed directly instead of via an env
@@ -59,7 +77,13 @@ Writes `tickets`, `tickets_replies`, attachment `files`; reads config from the `
 - **Loop prevention** only excludes FROM==department-TO addresses; CC/reply-all across multiple
   department addresses can still loop.
 - **Auto-reply filters** are partly hardcoded (e.g. ignore `TechHub@compass-usa.com`
-  "Automatic reply:" messages).
+  "Automatic reply:" messages), and now the `isSuppressedNoiseEmail` Staples-PO filter — there
+  is no config-driven allow/block list, so each new noise source is another inline guard.
+- **`Ticket::extractEmail()` returned an undefined offset on a no-match** (bare `$matches[0]`).
+  With a sender-less/malformed email this codebase promotes the warning to a thrown error
+  (`E_ALL` + `App_Error::handleError`), so an intake cron could **halt**. Hardened to
+  `return $matches[0] ?? '';` (TRUE-79975) — important because `isSuppressedNoiseEmail` routes
+  nullable/malformed sender input through `extractEmail`, putting this latent bug on the hot path.
 - Attachments are deleted locally immediately after the S3 upload with **no retry** if S3 fails.
 - **`adminid` clobber on reply routing:** after `emailToTicket()` sets `$data['adminid']` from
   the FROM address (~line 1407), an asset lookup (~line 1437-1442) overwrites it with the
@@ -69,6 +93,10 @@ Writes `tickets`, `tickets_replies`, attachment `files`; reads config from the `
   classification. See the ticket-lifecycle doc.
 
 ## Change history
+- 2026-07-09 — added `Ticket::isSuppressedNoiseEmail()` noise suppression (no-reply sender + "Staples PO"
+  subject) wired into both feeders before ticket creation; hardened `extractEmail()` no-match to
+  `''`; documented the `tickets_defaultdepartment` → "Support Desk" catch-all fall-through
+  (TRUE-79975) (mhammontree)
 - 2026-07-09 — documented the `emailToTicket()` adminid-overwrite reply misclassification
   (TRUE-80114 planning investigation) (dfranks)
 - 2026-06-15 — documented from a source read of `crons/tickets*.php` and `emailToTicket()` (jcardinal)

package/knowledge/2.0/apps/dbchanges2/architecture.md

@@ -6,7 +6,7 @@ project: Database Changes
 client: shared
 type: architecture
 status: active
-updated: 2026-07-07
+updated: 2026-07-09
 owners: [jcardinal, mhammontree, bala]
 files:
   - Core/
@@ -168,35 +168,62 @@ its own header.)
 5. Never edit or re-date an already-applied file — add a new dated file instead. Never put new
    work in a `HISTORIC` folder.
 
-## Bulk data loads — one statement, not many
+## Bulk data loads — batch, and stage large sets in a temp table
 
-When a change inserts many rows (reference-data / code-table loads), write it as a **single
-bulk `INSERT`**, not one `INSERT` statement per row. The database is billed per query and the
-external executor runs each statement as its own round-trip, so N separate inserts cost N times
-the round-trips (and query cost) of one batched statement. This is a standing team preference.
+When a change inserts many rows (reference-data / code-table loads), write it as **batched
+statements**, not one `INSERT` per row. The database is billed per query and the external
+executor runs each statement as its own round-trip, so N separate inserts cost N times the
+round-trips (and query cost) of a batched load.
 
-- Use one `INSERT ... VALUES (...),(...),...` or `INSERT ... SELECT` over a derived
-  (`UNION ALL`) row set.
-- To stay **idempotent** without a unique key to `INSERT IGNORE` against, anti-join the source
-  set to the target and insert only the misses:
+To stay **idempotent** without a unique key to `INSERT IGNORE` against, anti-join the source
+set to the target and insert only the misses.
+
+**Small sets (a few hundred rows):** a single `INSERT ... VALUES (...),(...),...`, or an
+`INSERT ... SELECT` over a `UNION ALL` derived row set, is fine:
 
 ```sql
 INSERT INTO TargetTable (uuid, keyColumn, otherColumn)
-SELECT
-	src.uuid,
-	src.keyColumn,
-	src.otherColumn
+SELECT src.uuid, src.keyColumn, src.otherColumn
 FROM (
 	SELECT '<uuid>' AS uuid, '<key>' AS keyColumn, '<value>' AS otherColumn
 	UNION ALL SELECT ...
 ) AS src
 LEFT JOIN TargetTable existing ON existing.keyColumn = src.keyColumn
-WHERE
-	existing.id IS NULL;
+WHERE existing.id IS NULL;
+```
+
+**⚠ Large sets — never build them as a long `UNION ALL` chain.** MySQL parses a `UNION ALL`
+chain **recursively**, one parser-stack frame per link. Around ~2,000+ links overflow the
+default 256 KB per-thread stack and the statement dies at **parse time** (before a single row
+is processed) with:
+
+    ERROR 1436 (HY000): Thread stack overrun: <n> bytes used of a 262144 byte stack, and
+    20000 bytes needed. Use 'mysqld --thread_stack=#' to specify a bigger stack.
+
+This is a parser-recursion limit — **not** a row-volume, `max_allowed_packet`, or
+server-misconfiguration issue, and **do not** raise `thread_stack` on the server for a one-off
+load. Instead, stage the rows in a **session-scoped temp table via a flat multi-row `VALUES`
+list** (which the parser reads as a flat list, no recursion), then run the anti-join once from
+the staging table:
+
+```sql
+CREATE TEMPORARY TABLE _stage (
+	uuid VARCHAR(36) NOT NULL, keyColumn VARCHAR(255) NOT NULL, otherColumn TEXT NULL);
+INSERT INTO _stage (uuid, keyColumn, otherColumn)
+VALUES ('<uuid>','<key>','<value>'), ('<uuid>','<key>','<value>'), ... ;  -- flat, thousands of rows
+INSERT INTO TargetTable (uuid, keyColumn, otherColumn)
+SELECT src.uuid, src.keyColumn, src.otherColumn
+FROM _stage AS src
+LEFT JOIN TargetTable existing ON existing.keyColumn = src.keyColumn
+WHERE existing.id IS NULL;
+DROP TEMPORARY TABLE _stage;
 ```
 
-- Keep the single statement well under `max_allowed_packet` (64 MB default) — a few thousand
-  rows is comfortably fine.
+The `TEMPORARY` table is session-scoped (safe on production, auto-dropped on disconnect even if
+the run aborts). A multi-row `VALUES` load of a few thousand rows is a few hundred KB — well
+under the 64 MB `max_allowed_packet` default. First hit in
+`Client_Aig/2026-06-18a - TRUE-79534 AIG SaleItem codes.sql` (2,200 AIG SaleItem codes); see
+`clients/aig/features/entitlement-intake.md` → *Uploading new codes*.
 
 ## Self-referencing DELETE — wrap the subquery in a derived table
 

package/knowledge/clients/aig/features/entitlement-intake.md

@@ -88,17 +88,27 @@ When AIG sends a new "Active SaleItemID" spreadsheet (columns `SaleItemID`, `Des
 3. Generate **fully-random UUIDs** (e.g. `uuid4`), never the time-based MySQL `UUID()` —
    per the 2.0 standard. Bake literal UUIDs into the SQL since a migration can't call PHP's
    `_String::generateUuid()`.
-4. Use a **single bulk `INSERT`** (see the dbchanges2 architecture rule), made idempotent with
-   an anti-join because `Items` has **no unique key on `partNumber`** (so `INSERT IGNORE`
-   can't help):
+4. Feed the rows through a **session-scoped temp table populated by a flat multi-row
+   `VALUES` list**, then anti-join once into `Items` — idempotent because `Items` has **no
+   unique key on `partNumber`** (so `INSERT IGNORE` can't help). Do **not** assemble the rows
+   as a long `UNION ALL` derived table: at AIG's ~2,200-code scale that overflows MySQL's
+   parser stack and fails at **parse time** with **ERROR 1436 (HY000) Thread stack overrun**
+   (see the gotcha below and the dbchanges2 architecture bulk-load rule). A flat `VALUES`
+   list does not recurse and stays well under `max_allowed_packet`:
    ```sql
+   CREATE TEMPORARY TABLE _aig_saleitem_stage (
+       uuid VARCHAR(36) NOT NULL, partNumber VARCHAR(255) NOT NULL, description TEXT NULL);
+   INSERT INTO _aig_saleitem_stage (uuid, partNumber, description)
+   VALUES ('<uuid>','<part>','<desc>'), ('<uuid>','<part>','<desc>'), ... ;  -- flat, ~2,200 rows
    INSERT INTO Items (uuid, partNumber, description)
    SELECT src.uuid, src.partNumber, src.description
-   FROM ( SELECT '<uuid>' AS uuid, '<part>' AS partNumber, '<desc>' AS description
-          UNION ALL SELECT ... ) AS src
+   FROM _aig_saleitem_stage AS src
    LEFT JOIN Items existing ON existing.partNumber = src.partNumber
    WHERE existing.id IS NULL;
+   DROP TEMPORARY TABLE _aig_saleitem_stage;
    ```
+   The `TEMPORARY` table is session-scoped (safe on production, auto-dropped on disconnect
+   even if the run aborts). Do **not** raise `thread_stack` on the server for a one-off load.
 
 ## Client variations
 
@@ -121,6 +131,13 @@ this interceptor or use this dual-purpose Items pattern.
   codes are **not** in the SaleItemID spreadsheet and must be sourced separately if missing.
 - **No unique key on `Items.partNumber`** — use an anti-join for idempotent loads, not
   `INSERT IGNORE`.
+- **A long `UNION ALL` derived table fails at ~2,200 rows with ERROR 1436 "Thread stack
+  overrun."** The AIG SaleItem load is exactly this scale. MySQL builds the `UNION ALL`
+  chain recursively, so ~2,200 links overflow the default 256 KB per-thread parser stack and
+  the statement dies at **parse time** (before any row is processed) — it is a parser-recursion
+  limit, **not** a row-volume, `max_allowed_packet`, or server-config problem. Stage the rows
+  in a temp table via a flat multi-row `VALUES` list and anti-join from there (see *Uploading
+  new codes* step 4). Do not raise `thread_stack` for a one-off load.
 - **An `entitlementFulfillmentType` EV-12 VALIDATION error is NOT a bad client payload —
   it's this interceptor's injected field failing to resolve.** AIG never sends that field;
   the interceptor injects it as `{name: "Repair"}`. The error means the V2 resolver could not
@@ -136,6 +153,11 @@ this interceptor or use this dual-purpose Items pattern.
 
 ## Change history
 
+- 2026-07-09 — TRUE-79534 SaleItem load reworked: the 2,200-code `INSERT ... SELECT FROM
+  (<2,199 `UNION ALL` links>)` derived table failed at parse time with ERROR 1436 (Thread
+  stack overrun). Rebuilt as a `TEMPORARY` staging table filled by a flat multi-row `VALUES`
+  list + the same anti-join into `Items` (identical rows/UUIDs, 0 duplicate partNumbers).
+  (mhammontree)
 - 2026-07-09 — TRUE-79978: root-caused an `entitlementFulfillmentType` EV-12 on beta/QA. The
   field is injected (name-only) by `prePost`; the beta `Client_Aig.Apis_RecordFields` override
   row (apiId=2, recordFieldId=1202) had `overrideIsIdentifier=NULL` + `overrideChildPolicy=MATCH_CREATE`,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.308",
+  "version": "1.0.310",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",