toga-ai 1.0.308 → 1.0.309

package/knowledge/2.0/apps/dbchanges2/architecture.md

@@ -6,7 +6,7 @@ project: Database Changes
 client: shared
 type: architecture
 status: active
-updated: 2026-07-07
+updated: 2026-07-09
 owners: [jcardinal, mhammontree, bala]
 files:
   - Core/
@@ -168,35 +168,62 @@ its own header.)
 5. Never edit or re-date an already-applied file — add a new dated file instead. Never put new
    work in a `HISTORIC` folder.
 
-## Bulk data loads — one statement, not many
+## Bulk data loads — batch, and stage large sets in a temp table
 
-When a change inserts many rows (reference-data / code-table loads), write it as a **single
-bulk `INSERT`**, not one `INSERT` statement per row. The database is billed per query and the
-external executor runs each statement as its own round-trip, so N separate inserts cost N times
-the round-trips (and query cost) of one batched statement. This is a standing team preference.
+When a change inserts many rows (reference-data / code-table loads), write it as **batched
+statements**, not one `INSERT` per row. The database is billed per query and the external
+executor runs each statement as its own round-trip, so N separate inserts cost N times the
+round-trips (and query cost) of a batched load.
 
-- Use one `INSERT ... VALUES (...),(...),...` or `INSERT ... SELECT` over a derived
-  (`UNION ALL`) row set.
-- To stay **idempotent** without a unique key to `INSERT IGNORE` against, anti-join the source
-  set to the target and insert only the misses:
+To stay **idempotent** without a unique key to `INSERT IGNORE` against, anti-join the source
+set to the target and insert only the misses.
+
+**Small sets (a few hundred rows):** a single `INSERT ... VALUES (...),(...),...`, or an
+`INSERT ... SELECT` over a `UNION ALL` derived row set, is fine:
 
 ```sql
 INSERT INTO TargetTable (uuid, keyColumn, otherColumn)
-SELECT
-	src.uuid,
-	src.keyColumn,
-	src.otherColumn
+SELECT src.uuid, src.keyColumn, src.otherColumn
 FROM (
 	SELECT '<uuid>' AS uuid, '<key>' AS keyColumn, '<value>' AS otherColumn
 	UNION ALL SELECT ...
 ) AS src
 LEFT JOIN TargetTable existing ON existing.keyColumn = src.keyColumn
-WHERE
-	existing.id IS NULL;
+WHERE existing.id IS NULL;
+```
+
+**⚠ Large sets — never build them as a long `UNION ALL` chain.** MySQL parses a `UNION ALL`
+chain **recursively**, one parser-stack frame per link. Around ~2,000+ links overflow the
+default 256 KB per-thread stack and the statement dies at **parse time** (before a single row
+is processed) with:
+
+    ERROR 1436 (HY000): Thread stack overrun: <n> bytes used of a 262144 byte stack, and
+    20000 bytes needed. Use 'mysqld --thread_stack=#' to specify a bigger stack.
+
+This is a parser-recursion limit — **not** a row-volume, `max_allowed_packet`, or
+server-misconfiguration issue, and **do not** raise `thread_stack` on the server for a one-off
+load. Instead, stage the rows in a **session-scoped temp table via a flat multi-row `VALUES`
+list** (which the parser reads as a flat list, no recursion), then run the anti-join once from
+the staging table:
+
+```sql
+CREATE TEMPORARY TABLE _stage (
+	uuid VARCHAR(36) NOT NULL, keyColumn VARCHAR(255) NOT NULL, otherColumn TEXT NULL);
+INSERT INTO _stage (uuid, keyColumn, otherColumn)
+VALUES ('<uuid>','<key>','<value>'), ('<uuid>','<key>','<value>'), ... ;  -- flat, thousands of rows
+INSERT INTO TargetTable (uuid, keyColumn, otherColumn)
+SELECT src.uuid, src.keyColumn, src.otherColumn
+FROM _stage AS src
+LEFT JOIN TargetTable existing ON existing.keyColumn = src.keyColumn
+WHERE existing.id IS NULL;
+DROP TEMPORARY TABLE _stage;
 ```
 
-- Keep the single statement well under `max_allowed_packet` (64 MB default) — a few thousand
-  rows is comfortably fine.
+The `TEMPORARY` table is session-scoped (safe on production, auto-dropped on disconnect even if
+the run aborts). A multi-row `VALUES` load of a few thousand rows is a few hundred KB — well
+under the 64 MB `max_allowed_packet` default. First hit in
+`Client_Aig/2026-06-18a - TRUE-79534 AIG SaleItem codes.sql` (2,200 AIG SaleItem codes); see
+`clients/aig/features/entitlement-intake.md` → *Uploading new codes*.
 
 ## Self-referencing DELETE — wrap the subquery in a derived table
 

package/knowledge/clients/aig/features/entitlement-intake.md

@@ -88,17 +88,27 @@ When AIG sends a new "Active SaleItemID" spreadsheet (columns `SaleItemID`, `Des
 3. Generate **fully-random UUIDs** (e.g. `uuid4`), never the time-based MySQL `UUID()` —
    per the 2.0 standard. Bake literal UUIDs into the SQL since a migration can't call PHP's
    `_String::generateUuid()`.
-4. Use a **single bulk `INSERT`** (see the dbchanges2 architecture rule), made idempotent with
-   an anti-join because `Items` has **no unique key on `partNumber`** (so `INSERT IGNORE`
-   can't help):
+4. Feed the rows through a **session-scoped temp table populated by a flat multi-row
+   `VALUES` list**, then anti-join once into `Items` — idempotent because `Items` has **no
+   unique key on `partNumber`** (so `INSERT IGNORE` can't help). Do **not** assemble the rows
+   as a long `UNION ALL` derived table: at AIG's ~2,200-code scale that overflows MySQL's
+   parser stack and fails at **parse time** with **ERROR 1436 (HY000) Thread stack overrun**
+   (see the gotcha below and the dbchanges2 architecture bulk-load rule). A flat `VALUES`
+   list does not recurse and stays well under `max_allowed_packet`:
    ```sql
+   CREATE TEMPORARY TABLE _aig_saleitem_stage (
+       uuid VARCHAR(36) NOT NULL, partNumber VARCHAR(255) NOT NULL, description TEXT NULL);
+   INSERT INTO _aig_saleitem_stage (uuid, partNumber, description)
+   VALUES ('<uuid>','<part>','<desc>'), ('<uuid>','<part>','<desc>'), ... ;  -- flat, ~2,200 rows
    INSERT INTO Items (uuid, partNumber, description)
    SELECT src.uuid, src.partNumber, src.description
-   FROM ( SELECT '<uuid>' AS uuid, '<part>' AS partNumber, '<desc>' AS description
-          UNION ALL SELECT ... ) AS src
+   FROM _aig_saleitem_stage AS src
    LEFT JOIN Items existing ON existing.partNumber = src.partNumber
    WHERE existing.id IS NULL;
+   DROP TEMPORARY TABLE _aig_saleitem_stage;
    ```
+   The `TEMPORARY` table is session-scoped (safe on production, auto-dropped on disconnect
+   even if the run aborts). Do **not** raise `thread_stack` on the server for a one-off load.
 
 ## Client variations
 
@@ -121,6 +131,13 @@ this interceptor or use this dual-purpose Items pattern.
   codes are **not** in the SaleItemID spreadsheet and must be sourced separately if missing.
 - **No unique key on `Items.partNumber`** — use an anti-join for idempotent loads, not
   `INSERT IGNORE`.
+- **A long `UNION ALL` derived table fails at ~2,200 rows with ERROR 1436 "Thread stack
+  overrun."** The AIG SaleItem load is exactly this scale. MySQL builds the `UNION ALL`
+  chain recursively, so ~2,200 links overflow the default 256 KB per-thread parser stack and
+  the statement dies at **parse time** (before any row is processed) — it is a parser-recursion
+  limit, **not** a row-volume, `max_allowed_packet`, or server-config problem. Stage the rows
+  in a temp table via a flat multi-row `VALUES` list and anti-join from there (see *Uploading
+  new codes* step 4). Do not raise `thread_stack` for a one-off load.
 - **An `entitlementFulfillmentType` EV-12 VALIDATION error is NOT a bad client payload —
   it's this interceptor's injected field failing to resolve.** AIG never sends that field;
   the interceptor injects it as `{name: "Repair"}`. The error means the V2 resolver could not
@@ -136,6 +153,11 @@ this interceptor or use this dual-purpose Items pattern.
 
 ## Change history
 
+- 2026-07-09 — TRUE-79534 SaleItem load reworked: the 2,200-code `INSERT ... SELECT FROM
+  (<2,199 `UNION ALL` links>)` derived table failed at parse time with ERROR 1436 (Thread
+  stack overrun). Rebuilt as a `TEMPORARY` staging table filled by a flat multi-row `VALUES`
+  list + the same anti-join into `Items` (identical rows/UUIDs, 0 duplicate partNumbers).
+  (mhammontree)
 - 2026-07-09 — TRUE-79978: root-caused an `entitlementFulfillmentType` EV-12 on beta/QA. The
   field is injected (name-only) by `prePost`; the beta `Client_Aig.Apis_RecordFields` override
   row (apiId=2, recordFieldId=1202) had `overrideIsIdentifier=NULL` + `overrideChildPolicy=MATCH_CREATE`,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.308",
+  "version": "1.0.309",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",