toga-ai 1.0.1192 → 1.0.1193

package/knowledge/2.0/apps/worker2/architecture.md

@@ -6,8 +6,8 @@ project: Worker
 client: shared
 type: architecture
 status: active
-updated: 2026-10-21
-owners: [jcardinal, ajean, rgirish, bala]
+updated: 2026-10-29
+owners: [jcardinal, ajean, rgirish, bala, mhammontree]
 files:
   - worker2/Controller/Index.php
   - worker2/Worker/
@@ -66,7 +66,7 @@ Derived status: `pending` (`dtQueued IS NULL AND isSuccess IS NULL`) → `queued
 
 ### `WorkerJobs.output` is written by MULTIPLE producers — rename it everywhere at once
 
-The outcome column (renamed from `failureReason` → `output`, MEDIUMTEXT) is written by **four** places that must change together: the PHP worker check-out (both paths in `Controller/Index.php`), the `Retry()` reset (`Worker/Infrastructure/Worker.php`, which clears `output`), and the Python **JobScheduler watchdog** Lambda (`LambdaFunctions/JobScheduler/lambda_function.py`, Part B timeout UPDATE). Because this is a **rename, not add-and-keep**, old-code/new-column and new-code/old-column both break — so the `dbchanges2` `ALTER TABLE Core.WorkerJobs CHANGE failureReason output MEDIUMTEXT NULL` must land **simultaneously** with the code deploy, and the watchdog Lambda must be **repackaged/redeployed separately** (a Python zip redeploy, not just a code push) or its UPDATE errors on the missing column. The PHP `$failureReason` **local variables** were intentionally left unchanged (only the SQL column moved). Do **not** touch the same-named-but-unrelated `failureReason` fields on other tables: the `Email` table (`Worker/Infrastructure/Email/Send.php`) and `Team.TranscriptExports` (`Worker/Team/Transcripts.php`).
+The outcome column (renamed from `failureReason` → `output`, MEDIUMTEXT — prod DESCRIBE 2026-10-29 shows TEXT; see the shrink gotcha) is written by **four** places that must change together: the PHP worker check-out (both paths in `Controller/Index.php`), the `Retry()` reset (`Worker/Infrastructure/Worker.php`, which clears `output`), and the Python **JobScheduler watchdog** Lambda (`LambdaFunctions/JobScheduler/lambda_function.py`, Part B timeout UPDATE). Because this is a **rename, not add-and-keep**, old-code/new-column and new-code/old-column both break — so the `dbchanges2` `ALTER TABLE Core.WorkerJobs CHANGE failureReason output MEDIUMTEXT NULL` must land **simultaneously** with the code deploy, and the watchdog Lambda must be **repackaged/redeployed separately** (a Python zip redeploy, not just a code push) or its UPDATE errors on the missing column. The PHP `$failureReason` **local variables** were intentionally left unchanged (only the SQL column moved). Do **not** touch the same-named-but-unrelated `failureReason` fields on other tables: the `Email` table (`Worker/Infrastructure/Email/Send.php`) and `Team.TranscriptExports` (`Worker/Team/Transcripts.php`).
 
 ## The four Lambdas (`worker2/LambdaFunctions/`, Python)
 
@@ -157,10 +157,14 @@ For **programmatic** enqueue via `_Worker::runTask()`, the MySQL-first rule was
 ## Gotchas
 
 - **Never give `?controller=` URL test steps for worker2.** `_.php` sets `DEFAULT_CONTROLLER_METHOD` to `_Controller_Index::worker()`, so every URL routes there and shows a blank page; the "Access via: ?controller=…" comments in `Controller/*Test.php` are misleading. Prove pure logic with a CLI runner (stub `_Component`/`_Controller`, require the file, call `run()`) or a worker action POSTed as `{action, parameters}`.
-- **Shrinking `WorkerJobs.output` MEDIUMTEXT → TEXT (planned, not yet done).** TEXT holds 65,535 **bytes** (utf8mb4 = up to 4 bytes/char), not chars. Trim byte-safe before the ALTER, repeating until 0 rows: `UPDATE WorkerJobs SET output = LEFT(output, CHAR_LENGTH(output) - (LENGTH(output) - 65535)) WHERE LENGTH(output) > 65535 LIMIT 1000`. After the change, every producer must truncate first or the result UPDATE fails with 1406 Data too long (strict mode) and the job result is lost: PHP check-out (both paths in `Controller/Index.php`) and the JobScheduler watchdog Lambda UPDATE (repackage + redeploy it). The `MODIFY` must restate the column COMMENT. On MySQL 8 the type change rebuilds the table.
+- **Shrinking `WorkerJobs.output` MEDIUMTEXT → TEXT — appears done: prod DESCRIBE on 2026-10-29 shows `TEXT`.** Not checked: whether every producer truncates first. TEXT holds 65,535 **bytes** (utf8mb4 = up to 4 bytes/char), not chars. Trim byte-safe before the ALTER, repeating until 0 rows: `UPDATE WorkerJobs SET output = LEFT(output, CHAR_LENGTH(output) - (LENGTH(output) - 65535)) WHERE LENGTH(output) > 65535 LIMIT 1000`. After the change, every producer must truncate first or the result UPDATE fails with 1406 Data too long (strict mode) and the job result is lost: PHP check-out (both paths in `Controller/Index.php`) and the JobScheduler watchdog Lambda UPDATE (repackage + redeploy it). The `MODIFY` must restate the column COMMENT. On MySQL 8 the type change rebuilds the table.
 
 - **⚠ OBSERVED `Core.WorkerJobs` retention is ~30 days, not 90 — unexplained.** Measured 2026-09-15 in prod: `MIN(dtCreated)` across the **whole** table is **2026-08-16 02:00:46** (790,597 rows). Either the cleanup cron does not behave as described, or something else prunes the table. **Not investigated** — no cause is asserted.
 - **A missing `WorkerJobs` row is NOT proof the job never ran.** Any "there is no job for this record, so the webhook never fired" conclusion is invalid past the ~30-day floor. That exact wrong call was made and retracted on 2026-09-15. Same failure family as *an empty log table is not proof a code path never ran*.
 - **The watchdog does not stop the PHP process** — an over-budget action keeps running past a false failure row and typically loses its result UPDATE to MySQL 2006 (prod job 1075223). See JobScheduler Part B above.
 - **`_Worker::runTask()` SQS-first gotchas (2026-07-07):** `dtQueued` on these rows reflects worker pickup/row-creation time, **not** true SQS enqueue time — semantic drift for any queue-latency metric. Debug/legacy messages without a `uuid` get a random one, so dedupe does not apply to them — an HTTP-layer retry of a debug call could double-run.
 - **Known follow-up (not done):** the ~100-line log-DB-registration + `initialize()` blocks are duplicated between the `workerJobId` path and the direct-payload path — candidate for extraction to shared private helpers.
+
+## Change history
+
+- 2026-10-29 — Prod `Core.WorkerJobs.output` is TEXT (DESCRIBE); planned shrink appears done, producer truncation not checked (TRUE-83392). (mhammontree)

package/knowledge/2.0/apps/worker2/features/creating-worker-actions.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-10-22
+updated: 2026-10-29
 owners: [jcardinal, dfranks, mhammontree, tcox, bala, ajean, rgirish, snaredla]
 files:
   - worker2/Worker/
@@ -163,6 +163,17 @@ Use realistic placeholder values, not empty strings/nulls.
 - Needs the worker key in `Authorization` (never paste the value into docs, tickets or chat).
 - Runs **synchronously** and creates **no `Core.WorkerJobs` row** — the SQS path (`_Worker::runTask` → `WorkerProductionQueue`) is the only one that logs a job. Use it to re-run a single action (e.g. one service request); read the result from the HTTP reply.
 
+### Run an action in prod (queued and tracked)
+Proven 2026-10-29 (TRUE-83392, job 2950480): insert one row into prod `Core.WorkerJobs` with
+`dtQueued` NULL; JobScheduler picks it up within ~1 minute and the row records the result.
+```sql
+INSERT INTO WorkerJobs (uuid, jobType, action, parameters)
+VALUES ('<random v4 uuid>', 'ACTION', 'Client/Compass/PeopleFile/ProcessPeopleFiles', NULL);
+```
+- It is not scheduler-fired from a `CronJobs` row, so it gets the 300s action cap.
+- **POSTing `{"action": ...}` to api2 does NOT run a worker action** — it returns
+  `isSuccess:false` with empty data and creates no job (2026-10-29, `api-writer.togahub.com/v2`).
+
 ### The HTTP reply is PREFIXED — never `JSON.parse` it raw
 
 The dispatcher returns your action's return value **wrapped in the same composed success
@@ -553,6 +564,7 @@ Rules: rename **both sides in one PR and one deploy**; grep every repo for the o
   commit-before-SQS transaction pattern that the worker relies on.
 
 ## Change history
+- 2026-10-29 — Run an action in prod via a `Core.WorkerJobs` ACTION row; a POST to api2 does not run worker actions (TRUE-83392). (mhammontree)
 - 2026-10-22 — Debug mode: child-job output before `http_response_code(200)` reads as a false 500 (TRUE-82919). (jcardinal)
 - 2026-10-22 — Gotcha: vendor named args must match vendor param names; PHP 8.5 vendor tree. (snaredla)
 - 2026-10-20 — Debug-mode `runTask` capped at depth 5 (`X-Toga-Worker-Depth`), throws on non-2xx, 300 s timeout; local ini needs `[api] worker` (TRUE-82934). (jcardinal)

package/knowledge/2.0/apps/worker2/features/stranded-worker-jobs.md

@@ -40,7 +40,8 @@ Infrastructure/Errors/Escalate 4, plus 2 each for eight `Monitor/Compass/*` acti
 **CRON row** — mark it `isSuccess = 0` with an explanatory `output`.
 - Do **not** delete it: that destroys the audit record.
 - Do **not** re-queue by nulling `dtQueued`: that re-runs stale time-based work.
-- If the run is genuinely still needed, fire it deliberately as a fresh job.
+- If the run is genuinely still needed, fire it deliberately as a fresh job:
+  [run an action in prod](./creating-worker-actions.md#run-an-action-in-prod-queued-and-tracked).
 
 **WEBHOOK row** — the payload is still in `parameters`, so
 `_Worker_Infrastructure_Worker::Retry($workerJobId)` genuinely recovers it. Retry it.

package/knowledge/clients/compass-usa/features/people-file-import-admin-page.md

@@ -6,7 +6,7 @@ project: Tools
 client: compass-usa
 type: client-feature
 status: active
-updated: 2026-10-19
+updated: 2026-10-29
 owners: [mhammontree, bala]
 files:
   - tools/_/app/compass/peoplefile.php
@@ -48,6 +48,10 @@ worker2 side (`Worker/Client/Compass/PeopleFile.php`):
 **Deploy order: worker2 FIRST, then tools.** The page calls `ListFiles`, which does not exist
 until worker2 ships.
 
+**⚠ worker2 side NOT in prod as of 2026-10-29.** `origin/_production` `PeopleFile.php` has no
+`ListFiles` / `gatherPeopleFiles`, and prod runs still record `<no return message>`. Until it
+ships, queue runs by hand (next section).
+
 Reference: prod `Core.CronJobs` id **8** = "Compass - People File Import", schedule `15 9 * * *`,
 `maxExecutionTime` 3600, action `Client/Compass/PeopleFile/ProcessPeopleFiles`. A job enqueued
 from this page is **not** scheduler-fired, so it gets the 300s action cap, not 3600.
@@ -62,6 +66,18 @@ devteam@togatech.com and vburks@togatech.com. So after N missed days:
    after the [shared SFTP server](../../../2.0/apps/worker2/features/shared-sftp-server-source-ip-ban.md)
    banned worker2.
 
+### Recovering a file that fails to parse
+A bad file blocks every newer file ([why](./people-file-user-lifecycle.md#gotchas)). Worked
+2026-10-29 (TRUE-83392):
+1. Download the failing file from cgsftp; find the line(s) whose pipe-column count is not 19.
+2. Fix the line break in a copy; re-upload over the original, **same file name**.
+3. Queue one run with a prod `Core.WorkerJobs` row (action
+   `Client/Compass/PeopleFile/ProcessPeopleFiles`, parameters `NULL`) — see
+   [run an action in prod](../../../2.0/apps/worker2/features/creating-worker-actions.md#run-an-action-in-prod-queued-and-tracked).
+4. Confirm `isSuccess = 1` on the job and the `COMPASS PEOPLE File Import - <file>` email in
+   `Logs_Compass.Email` (job 2950480: 19.8s, email 151836).
+5. Tell Compass which row was broken.
+
 ## Gotchas
 
 - **A failed run does not mean nothing happened.** The importer is not all-or-nothing — see
@@ -70,5 +86,6 @@ devteam@togatech.com and vburks@togatech.com. So after N missed days:
 - **Persona-gated to Development Team.** It runs a production import; do not widen it casually.
 
 ## Change history
+- 2026-10-29 — TRUE-83392: worker2 side of TRUE-82250 found not deployed to prod; added the fix-and-requeue procedure for a file that fails to parse. (mhammontree)
 - 2026-10-19 — Catch-up procedure (one run per waiting file, check the import email) after Oct 17–19 were missed in the worker2 SFTP IP ban (TRUE-82898). (bala)
 - 2026-09-23 — Created with the page (TRUE-82250, tools 0cc6f99 / worker2 a81ad6c, branch TRUE-82250): shared `gatherPeopleFiles`, new `ListFiles` action, in-flight guard, and a JSON summary return on every path. (mhammontree)

package/knowledge/clients/compass-usa/features/people-file-user-lifecycle.md

@@ -6,7 +6,7 @@ project: Worker
 client: compass-usa
 type: client-feature
 status: active
-updated: 2026-10-27
+updated: 2026-10-29
 owners: [bala, ajean, mhammontree]
 files:
   - worker2/Worker/Client/Compass/PeopleFile.php
@@ -130,6 +130,9 @@ The insert-time hook designed 2026-08-12 was built and grew into a reusable engi
 - **⚠ A whole tenant run is ONE transaction, and US runs first — so a US exception means Canada never executes at all.** Any throw rolls back everything for that run: inserts, updates, reactivations, **deactivations**, personas, roles and approval reassignment. The 2026-09-22 FK failure blocked both tenants for a day this way. Rollback is clean (proven: a run that died on `Users_Personas` after the `Users` insert left 0 rows behind), and the exception aborts **before** the SFTP delete, so the input file survives and the next run reprocesses it.
 - **⚠ Read `LAST_INSERT_ID()` on the write connection, never through `_Query`.** `_Query` routes SELECTs to the read host, and `LAST_INSERT_ID()` is per connection — so it would return `0`, which the `contactId` FK now rejects. `insertContact()` uses `mysqli_query` + `mysqli_insert_id` on the transaction connection, throws if the id is `<= 0`, and calls `_Database::resetQueryCache($dbName)` by hand because it bypasses `_Query`'s automatic cache reset. Framework detail: [Per-client database connections](../../../2.0/apps/_underscore/features/per-client-database-connections.md).
 - **A "successful" run is not proof a file was processed.** The SFTP picker is FIFO (oldest first) and handles **one** file per run, and an **empty SFTP directory is a silent success** — the job just returns. The PEOPLE file is a full snapshot of the active roster (80,805 usernames vs 75,133 active users on 2026-09-23), not a delta. After N missed days, catch-up needs N runs: [admin page](./people-file-import-admin-page.md#catching-up-after-missed-days).
+- **⚠ One bad file blocks the whole queue.** The importer always takes the **oldest** `PEOPLE*.txt` and deletes it only on success. A file that fails is retried every day and every newer file waits behind it until someone fixes or replaces it. Fix: [admin page → recovering a file that fails to parse](./people-file-import-admin-page.md#recovering-a-file-that-fails-to-parse).
+- **⚠ `Invalid PEOPLE row: expected 19 columns, found N at data row #M` = a line break inside a row.** Thrown in `stageFromPeopleFile()`. The file is pipe-delimited, 19 columns, LF line endings; a stray LF in a field splits one row into two short lines (2026-10-29: user `EYERLC01`, break right after `supv_personnel_nbr`, 13 + 7 columns; the only bad row in 81,320 lines). **`#M` is NOT the file line number** — it counts only rows actually processed (skipped rows are not counted). Find the bad line by scanning the file for lines whose pipe-column count is not 19.
+- **⚠ The summary email is near its attachment size limit.** The email attaches the whole PEOPLE file (~16.0–16.1 MB; 16,082,923 bytes on 2026-10-29). `Logs_Compass.EmailAttachment.fileData` is `MEDIUMBLOB` (max 16,777,215 bytes). A small growth in the file will make the email step fail — after the data is committed (see the not-all-or-nothing gotcha above). Not fixed as of 2026-10-29; parked until it fails — watch for it.
 - **No summary email = the import never ran.** `connectSFTP()` (~line 181) throws before any DB work or the email step. `Error reading SSH identification string` there is the shared SFTP server banning worker2's IP, not a bad password. That box also serves Staples + NYCDOE, and NYCDOE load can lock Compass out (Oct 17-19 2026, TRUE-82932). Check steps + cause: [Shared SFTP server](../../../2.0/apps/worker2/features/shared-sftp-server-source-ip-ban.md).
 - **⚠ A stale local client DB silently invalidates a test of this cron.** Local `Client_Compass` had `contactId` NULLABLE, so the *old broken code passed locally* — a NULL FK is allowed; only prod's NOT NULL turns the missing column into a `0`. Local also lacked persona 35 and still called persona 1 "All". Drop and reload both tenants from prod, then re-verify the constraint survived the copy, before trusting any result. Local-run mechanics: [Running worker2 locally](../../../2.0/apps/worker2/workflows/running-worker2-locally.md).
 
@@ -144,6 +147,7 @@ The insert-time hook designed 2026-08-12 was built and grew into a reusable engi
 - [Users.email as login identity](../../../2.0/apps/_underscore/features/user-email-as-identity-and-password-reset.md)
 
 ## Change history
+- 2026-10-29 — TRUE-83392: 10/29 file failed on a split row (`EYERLC01`); fixed the line by hand, re-uploaded, queued one `ProcessPeopleFiles` run (job 2950480, success). Added the bad-row, queue-block and attachment-size gotchas. No code changed. Parked until the problem happens again (no backlog task, by developer choice): optional join-short-lines fix; the attachment size limit; `Client_Compass` user 93605 (`eyerlc01`) has `c_supportedByUserId` NULL after import although the file names supervisor `HILLB17` — not investigated. Also: tell Compass about the line break. (mhammontree)
 - 2026-10-27 — Hand-created app users get only Base + no location and are deactivated by the next run when not in the file (TRUE-83280). (bala)
 - 2026-10-21 — TRUE-82976: fixed the post-insert refresh (`AND isActive = 1 ORDER BY id`, worker2 `e22b3ec`) so new-user roles, personas and supervisor link always go to the newest active row. (bala)
 - 2026-10-21 — TRUE-82976: restored new-user Base + Compass Base role staging (`_needsUserId`), lost in `51cfcb9`; prod backfill 152 US / 24 CA `Users_Roles` rows; logged the open no-`ORDER BY` refresh risk. (bala)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.1192",
+  "version": "1.0.1193",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",