toga-ai 1.0.1132 → 1.0.1133

package/knowledge/1.0/apps/library/features/db-connection-lifecycle-and-reconnect.md

@@ -50,7 +50,7 @@ $lastErrorNumber = mysqli_errno($link);         // query.php:173  <- the 2006 ch
 
 `mysqli_report(MYSQLI_REPORT_STRICT)` is set in `App_Error::initialize()` (`library/app/error.php:252`). **STRICT raises mysqli exceptions only for CONNECTION errors** — everything else stays a warning. That single line is why a dead connection is a warning and not something you can catch.
 
-### The built-in 2006 reconnect (fixed 2026-10-23, TRUE-82812)
+### The built-in 2006 reconnect (fixed 2026-10-23, TRUE-82812 + TRUE-83163)
 
 `App_Query::execute()` retries MySQL error **2006** at `query.php:~176`, capped by `$connectionAttemptsToMake = 2` (`query.php:85`). Until `library` `96cffb23` it was broken:
 
@@ -58,7 +58,9 @@ $lastErrorNumber = mysqli_errno($link);         // query.php:173  <- the 2006 ch
 2. **Cache handed back the same handle (fixed).** Retry called `databaseClose()` + `registerDatabaseConnect()`; `databaseClose()` never clears `App_Database::$connections[$hostname]`, so it got the same **closed** mysqli → PHP Error `mysqli object is already closed` at `query.php:168` (1.8K events, worker1.5 `startech/import_supporting_records_pcmaticb2b.php` → `cronFinished()`). Now calls `App_Database::reconnect()`.
 3. **Stale `$link` (fixed).** `$link` is now re-read after the reconnect.
 
-**Open limits (not fixed):** links on the same DB host share one mysqli object, so after `reconnect()` other registry links on that host (e.g. `db_toga` vs `db_common`) still hold the closed object. `reconnect()` also does not clear the `_read` reader-host cache slot.
+4. **PHP 8 exception path (TRUE-83163).** `app/error.php` sets `MYSQLI_REPORT_STRICT`, so on PHP 8.1+ a 2006 can arrive as `mysqli_sql_exception`, which skipped the errno branch. The loop now also catches `mysqli_sql_exception` code 2006 and retries (private `reconnectAfterServerGone()`, main or `_read`). Other codes rethrow. Same "already closed" error also hit Compass OD sales orders and Prudential FedEx delivered crons.
+
+**Open limit (not fixed):** links on the same DB host share one mysqli object, so after `reconnect()` other registry links on that host (e.g. `db_toga` vs `db_common`) still hold the closed object. On a `_read` retry, TRUE-83163 drops the `_read` cache slot that held the stale link.
 
 ### A `try`/`catch` around a DB call cannot help
 
@@ -76,7 +78,7 @@ Canonical mechanics and the two sanctioned local opt-outs are in the [back-end s
 | `App_Database::reconnect($link)` | `database.php:446` | The connection is already presumed dead and you must get a working one (e.g. inside a shutdown handler). |
 | `App_Database::queryForked($sql, $link)` | `database.php:465` | A write you want to run in a **separate PHP process**, so the parent cron never holds its connection open across it. Falls back to a direct `query()` on Windows. |
 
-**`reconnect()` is the fix for cache bug #2 above** — it is the only thing that actually forces a new handle:
+**`reconnect()` is the only thing that forces a new handle** (cache bug #2 above):
 
 ```php
 self::databaseClose($linkRegistryString);
@@ -117,6 +119,7 @@ The alert pattern as first shipped in Compass ODP cron 5 (`worker` commit `afce3
   - `library/app/error.php:5-8` has `IMMEDIATELY_TERMINATE_INSTANCE_IF_ERROR_STRING_CONTAINS`, matched **on substring before severity**. A recoverable warning could terminate an EC2 instance.
 
 ## Change history
+- 2026-10-23 — TRUE-83163: 2006 retry also catches `mysqli_sql_exception` 2006 (PHP 8.1+ STRICT path) and drops the stale `_read` cache slot. (jcardinal)
 - 2026-10-23 — TRUE-82812: 2006 retry now uses `reconnect()` and re-reads `$link` (`96cffb23`); noted shared-handle and `_read` limits, `cronFinished()` catch gap. (snaredla)
 - 2026-10-08 — Shutdown alarm can fire falsely (warning inside the SOAP call exits too); Compass cron 5 replaced it with an externalId lookup (`fd426d9b`). Added the remote-id rule to the Summary. (bala)
 - 2026-09-25 — Corrected the cron 5 pattern: it stamps via `queryForked` with no `keepAlive`, and disarms the alarm before the forked write is confirmed; noted the monitor backstop. (rgirish)

package/knowledge/1.0/apps/library/features/php-8-5-runtime-compatibility.md

@@ -66,6 +66,10 @@ Fix patterns (TRUE-83059 `d28f9b23`, TRUE-83055 PR #905):
 | 8.2 | dynamic property | declare it on the class (TRUE-83055: `App_Api_Carrier_ShipmentRequest::$shipToAddressLine2`, `App_Email_AigTemplates_SendAigSupportTicketReplyNotification::$contact`) |
 | 8.1 | `SoapClient` overrides without return types (`app/soaptransaction.php`) | return-type attributes + 6th `$uriParserClass` arg |
 | 8.5 | `case 'x';` (semicolon after `case`) — vendored `php-imap2/src/Roundcube/ImapClient.php:2580` flooded worker1.5's `worker-cron.log` | `case 'x':` (library `53ea57f8`) |
+| 8.5 | warning "The float ... is not representable as an int, cast occurred" — `App_String::encrypt/decrypt`, `underscoreEncrypt/decryptUnderscoreEncrypt` key loop `$i += $i` turns float at ~56+ chars, then `$i % 255` (killed Walmart order reminder/processor crons) | private `keyByteModulo()`: wrap the float to signed 64-bit (`fmod` 2^64) before `% 255` — exact PHP 7/8.2 result, old links still decrypt (0 mismatches / 30,720 cases) (TRUE-83163) |
+| 8.x | `socket_set_timeout()` (`phpmailer/class.smtp.php`) | `stream_set_timeout()` (TRUE-83163) |
+| 8.1 | `null` into `strlen`/`trim`, null array key (`hjson/HJSONParser.php`, `catalog/content.php`, `App_Database::buildDoubleFieldArray` / `buildAssocArray`) | `(string)` cast / `?? ''`; `buildAssocArray` casts the key, so a wrong column index stays loud (TRUE-83163) |
+| 8.2 | dynamic property | declared: `App_ASNRaw` props, `App_ApiResult::$success`, NetSuite `SearchTextNumberField` / `SearchEnumMultiSelectField::$operatorSpecified` (TRUE-83163) |
 
 State on `_production`: `php -l` on PHP 8.4 shows no deprecations in `app/` or `browser/`.
 
@@ -91,20 +95,22 @@ ext-imap is gone from PHP 8.4+, so on 8.5 hosts (worker1.5) every `imap2_*` call
 
 ## Gotchas
 
-- **Still open — vendored third-party folders only:** `LightSaml/`, `simplesaml/`, `jpgraph/`, `mobile_detect/`, `invoiceprocessor/pdf/` raise 8.4 compile-time deprecations (implicit nullable, optional-before-required, `${var}`). Logged, not fatal.
+- **Still open:** vendored `phpseclib/`, `LightSaml/`, `simplesaml/`, `jpgraph/`, `mobile_detect/`, `invoiceprocessor/` raise 8.4 compile-time deprecations (implicit nullable, optional-before-required, `${var}`); `app/string.php:277` `mb_convert_encoding(..., 'HTML-ENTITIES')` (8.2) — no byte-identical 7.2-safe replacement found. Logged, not fatal.
+- **⚠ Never "fix" the `App_String` encrypt key loop with real modular math.** It changes the key for long strings, so every issued link (Walmart unsubscribe/register) stops decrypting. `keyByteModulo()` must reproduce the old 64-bit wraparound exactly.
 - **Warnings and notices still exit.** Only deprecations were softened — an "offset on null" or "undefined index" still kills the request on any PHP version.
 - **`file_get_contents` to IMDS or any optional endpoint:** the `ignore_errors` stream option covers only HTTP status — a refused or slow connection still warns, and App_Error exits. Wrap the call in a local `set_error_handler` / `restore_error_handler` and check for a `200` status line (togaview TRUE-83060).
 - **The `instance-data` hostname does not resolve on AL2023.** Use `169.254.169.254` only in new code. Old uses: `App_Error_Capture::AWS_METADATA_HOSTS` falls back to it when `169.254.169.254` returns non-200 (e.g. the tags path with instance metadata tags off), so the cron log shows `getaddrinfo for instance-data failed` warnings (noise, capture still works). `App_Cloud::getInstanceId()` (`app/cloud.php` ~L711) calls only `instance-data`, so it returns `''`, and `App_Framework::getInstanceId()` caches that `''` in `instanceId.txt`. That is why worker1.5 writes `instanceId = ''` to `Common.CronJobExecutions` and `Logs.Emails`. `app/error.php` ~L642 (Sentry AWS context) also uses it.
 - **PHP 7.2 safety was checked by reading, not running** (no 7.2 binary locally on PR #905). The guards use `PHP_VERSION_ID` (5.2.7+) and `?type` (7.1+). Run a 7.2 smoke test if a 7.2 app breaks after this change.
 - **Do not "repair" the removed `App_Edi::cleanText` block.** A working version would replace every character 0-127 (all letters and digits) with spaces and blank every EDI file.
 - Local dev on PHP 8.2 does not show 8.4/8.5 deprecations — test with an 8.5 binary.
-- **PHP 8 string-to-number comparison can turn a loop infinite.** `api` subscription: a non-numeric quantity compared to an int looped forever on 8.x. Cast `(int)` before the loop.
+- **PHP 8 string-to-number comparison changed (`"abc" != 0` is now true).** `api` subscription: a non-numeric quantity compared to an int looped forever. worker1.5 `get_new_detectprotectservices.php`: HTML junk rows passed a `$cols[0] != 0` check and inserted duplicate recNo 0. Cast `(int)` before comparing.
 - **OAuth IMAP (`imap2_open(..., OP_XOAUTH2)`) always uses the Roundcube path — even on PHP 7.2 with ext-imap.** So `HeaderInfo::fromMessage` runs on the 7.2 worker too. Gate every php-imap2 behavior change on `!IMAP2_RETROFIT_MODE` (= `function_exists('imap_open')` at bootstrap load) so 7.2 stays byte-identical.
 - **php-imap2 gaps still open (no-ext-imap only):** group end writes `grp: ` not `grp: ;`; `mutf7ToUtf8` / `utf8ToMutf7` / `utf8` return input; `mailCompose` returns `false`; `rfc822ParseHeaders` still needs zbateson (not on importer paths). `imap_binary` line width (60, CRLF) not compared to real ext-imap. 7.2 safety checked by reading only.
 - **Needs `mbstring` + `iconv`** on the 8.5 host for the Polyfill decode paths — check `php -m`.
 - **A library fix reaches worker1.5 only after a worker1.5 redeploy** ([worker1.5 architecture](../../worker1.5/architecture.md)).
 
 ## Change history
+- 2026-10-23 — `App_String` key float overflow fixed with exact old result; more deprecations cleared; open list updated (TRUE-83163) (jcardinal)
 - 2026-10-23 — php-imap2 works without ext-imap: Polyfill loaded + filled in; OAuth/Roundcube gotcha; open gaps (TRUE-83148, `4a2c363f`) (jcardinal)
 - 2026-10-23 — `instance-data` does not resolve on AL2023 (empty instance id, log noise); deprecations-continue behavior confirmed in the worker1.5 cron log (TRUE-83107). (bala)
 - 2026-10-23 — `case 'x';` 8.5 deprecation fixed in vendored php-imap2 (TRUE-83116). (bala)

package/knowledge/1.0/apps/worker/INDEX.md

@@ -2,7 +2,7 @@
 
 | Doc | Summary |
 |-----|---------|
-| [Worker (1.0 Framework) Architecture](architecture.md) | The 1.0 `App_` background-job tier (~550 CRON scripts); open for how crons are scheduled, how the 7-worker role election works, and its silent-death failure mod |
+| [Worker (1.0 Framework) Architecture](architecture.md) | RETIRED 1.0 background-job tier (history only); every cron now runs on worker1.5. |
 | [Canon received-PO → NetSuite sales order sync](features/canon-received-po-netsuite-order-sync.md) | Canon-only worker cron that reads Vision customer POs and creates the matching NetSuite sales order (bundles exploded, prefixes stripped); open before touching |
 | [Compass MA Sales Order Exception Report](features/compass-ma-sales-order-exception-report.md) | Worker cron emailing ops the "Compass Refresh Exception Report" of `MA%` Compass sales orders whose ODP NetSuite order isn't keyed yet; open for the two-hop bri |
 | [Compass Approval Reminder & 30-Day Auto-Cancel Crons (1.0 worker)](features/compass-manager-approval-reminder-emails.md) | Per-tenant Compass worker crons that remind approvers daily (8:00) and auto-cancel orders still unapproved after 30 days (8:30); open before touching approval a |

package/knowledge/1.0/apps/worker/architecture.md

@@ -5,7 +5,7 @@ repo: worker
 project: Worker
 client: shared
 type: architecture
-status: active
+status: deprecated
 updated: 2026-10-23
 owners: [jcardinal, sking, bala]
 files:
@@ -26,9 +26,9 @@ related:
   - ../worker1.5/architecture.md
 ---
 
-The 1.0 `App_` background-job tier (~550 CRON scripts); open for how crons are scheduled, how the 7-worker role election works, and its silent-death failure modes.
+RETIRED 1.0 background-job tier (history only); every cron now runs on worker1.5.
 
-> **Moved:** the worker 1.0 jobs now run on one worker1.5 server on PHP 8.5 (TRUE-83078, commit 58dab19). For how crons run now, see [../worker1.5/architecture.md](../worker1.5/architecture.md).
+> **Retired (2026-10-22):** the `worker` repo runs nothing. Every cron moved to worker1.5 (TRUE-83078). Make code fixes in worker1.5; read this doc for history only. See [../worker1.5/architecture.md](../worker1.5/architecture.md).
 
 ## Summary
 

package/knowledge/1.0/apps/worker1.5/INDEX.md

@@ -5,4 +5,4 @@
 | [Worker 1.5 (1.0 Framework) Architecture](architecture.md) | The single-server 1.0 cron tier that replaced worker's 7-role fleet; open for how the crontab is built from schedules/*.json, the overlap check, and what the cr |
 | [worker1.5 — Committed Secrets & Credentials in Logs (debt register)](features/committed-secrets-and-log-exposure.md) | Open security debt in worker1.5: a GitHub token committed in `ebs/git.json` and a cron that writes OptimumDesk passwords into the cron log; open before touching |
 | [worker1.5 PHP ini overrides on EB AL2023 (memory_limit 11G, zz-application.ini)](features/php-ini-memory-limit-eb.md) | How worker1.5 sets PHP ini values (memory_limit 11G) on EB `worker15-production`; open before changing any PHP ini setting or wondering why the EB console value |
-| [Diagnosing worker1.5 crons that stopped running (single-server 1.0 tier)](workflows/diagnosing-crons-not-running.md) | Steps to tell "worker1.5 crons are not running" from "a downstream system is failing", find the cause on the box, and recover; open when several 1.0 jobs or bac |
+| [Diagnosing worker1.5 crons that stopped running (single-server 1.0 tier)](workflows/diagnosing-crons-not-running.md) | Steps to tell "worker1.5 crons are not running" from "a downstream system is failing", find the cause on the box, and recover, plus how to triage failing crons |

package/knowledge/1.0/apps/worker1.5/architecture.md

@@ -30,7 +30,7 @@ The single-server 1.0 cron tier that replaced worker's 7-role fleet; open for ho
 
 ## Summary
 
-`worker1.5` is the 1.0 `App_` background-job tier on **one Elastic Beanstalk server, PHP 8.5, AL2023** (TRUE-83078). It is a copy of the legacy `worker` repo with the same folder layout (`crons/<area>/...`, `crons/toga2/<client>/`). The 7 role schedules were merged into one file. Role election, heartbeat, stale-worker termination and per-role Elastic IPs are gone: one server runs every active job. Builds on `library`.
+`worker1.5` is the 1.0 `App_` background-job tier on **one Elastic Beanstalk server, PHP 8.5, AL2023** (TRUE-83078). It replaced the legacy `worker` repo, which is retired: every worker 1.0 cron moved here on 2026-10-22. Same folder layout (`crons/<area>/...`, `crons/toga2/<client>/`). The 7 role schedules were merged into one file. Role election, heartbeat, stale-worker termination and per-role Elastic IPs are gone: one server runs every active job. Builds on `library`.
 
 **Critical rules:** the schedule `schedules/cron.<ENVIRONMENT>.json` is the source of truth. Production is `cron.worker.json`. A script not in it never runs. A bad `schedule` (not 5 fields) or a `%` in a command aborts the whole crontab install on deploy. Keep cron's default `/bin/sh`: `App_Framework::isProcessRunning()` only ignores `/bin/sh` parent rows, so another shell breaks the already-running check (TRUE-83121). Any PHP warning exits the request or cron (see the backend-php standard, *Any PHP warning terminates the request*). Host setup uses `.platform/hooks` (prebuild/predeploy/postdeploy), not `.ebextensions`; never apply `worker` (AL1) EB fixes here blindly. PHP ini values come from the repo (`ebs/setup_phpini.php` → `/etc/php.d/zz-application.ini`), not the EB console; memory_limit is 11G per process on a 16 GiB m7g.xlarge, so watch crons running at the same time. See [[php-ini-memory-limit-eb]].
 
@@ -59,4 +59,4 @@ No CloudWatch log group: every job's output is `/var/log/worker-cron.log` on the
 - **IMAP crons depend on library's php-imap2 patches.** No ext-imap on PHP 8.5; php-imap2 comes from the cloned `library`, not composer. Desk, Quad and ~20 Compass / Compass Canada IMAP crons crashed until library `4a2c363f` (see [PHP 8.5 § IMAP without ext-imap](../library/features/php-8-5-runtime-compatibility.md#imap-without-ext-imap-php-imap2)). togadesk is cloned as `ontrack` → `/var/www/html/ontrack`; the prod Desk importer is `php /var/www/html/crons/../ontrack/crons/tickets_prod.php` (TRUE-83148).
 - **Do not read files that are not in this repo on a 1.0 page.** The page read `schedules/cron.worker.catalog.json` (old worker only). The warning became `exit` and the page died about 474 times an hour (9,387 total), probably from a health check on `/`. Fixed in TRUE-83107 (c903b20).
 - **`App_Misc::getFileList` uses `readdir` and does not sort.** Any loop that keeps state per file must set that state on every file. The old loop set `$jsonContents` only for a non-empty category, so `cron.worker.json` reused the previous file's rows, or warned (and exited) if it came first.
-- **Cron log noise you can ignore:** `PHP Deprecated: ...` lines (the job continues) and `getaddrinfo for instance-data failed` warnings (see [PHP 8.5 compatibility § Gotchas](../library/features/php-8-5-runtime-compatibility.md)). Two open faults show up in the log: the OptimumDesk sync prints a credential row into its exception ([Error capture 1.0 § Gotchas](../library/features/error-capture-1-0.md)), and empty-From emails fail at SES every run ([Email queue § Gotchas](../library/features/email-queue-attachments.md)).
+- **Cron log noise you can ignore:** `PHP Deprecated: ...` lines (the job continues) and `getaddrinfo for instance-data failed` warnings (see [PHP 8.5 compatibility § Gotchas](../library/features/php-8-5-runtime-compatibility.md)). Two open faults show up in the log: the OptimumDesk sync prints a credential row into its exception ([Error capture 1.0 § Gotchas](../library/features/error-capture-1-0.md)), and empty-From emails fail at SES every run ([Email queue § Gotchas](../library/features/email-queue-attachments.md)). Triage steps and known entries: [diagnosing workflow § Triage](workflows/diagnosing-crons-not-running.md#triage-failing-crons-in-worker-cronlog).

package/knowledge/1.0/apps/worker1.5/workflows/diagnosing-crons-not-running.md

@@ -7,7 +7,7 @@ client: shared
 type: workflow
 status: active
 updated: 2026-10-23
-owners: [bala]
+owners: [bala, jcardinal]
 files:
   - worker1.5/ebs/cron.worker.php
   - worker1.5/ebs/setup_git_libraries.php
@@ -22,7 +22,7 @@ related:
   - ../../../clients/compass-usa/features/mits-sales-order-transmission-alerting.md
 ---
 
-Steps to tell "worker1.5 crons are not running" from "a downstream system is failing", find the cause on the box, and recover; open when several 1.0 jobs or backlogs stall at once.
+Steps to tell "worker1.5 crons are not running" from "a downstream system is failing", find the cause on the box, and recover, plus how to triage failing crons in `worker-cron.log`; open when several 1.0 jobs or backlogs stall at once, or a cron errors.
 
 ## Summary
 
@@ -45,14 +45,28 @@ Since TRUE-83078 (2026-10-22) every 1.0 cron runs on ONE server: EB env `worker1
    - Backlog crons (MITS transmit, SR→SO, email send, NetSuite syncs) drain by themselves. Watch the backlog fall; do not assume.
    - Fixed-time daily/weekly jobs do NOT catch up. Re-run each missed one by hand (`php /var/www/html/crons/<path>` on the box) and tell the people who expect it.
 
+## Triage failing crons in `worker-cron.log`
+
+1. Group the `PHP Warning` and `ERROR [..]` lines by message. `App_Error_Capture` adds the cron name to each.
+2. Skip known noise: `ImapClient.php:2580`, `instance-data` lookups, `PHP Deprecated` (job continues).
+3. For the caller, open prod `Logs.Issue` / `Event` by reference (e.g. `8F`); the Event trace gives script:line.
+
+Known entries (TRUE-83163, 2026-10-23):
+- `Allowed memory size of 268435456` = the HTTP 500 body from `api.beta.togahub.com` (php-fpm 256 MB on `GET /contacts/{uuid}/contact-addresses` depth=3, Issue `8F`), not a worker1.5 crash. That cron, "Generates sales and purchase orders for Prudential (Beta)", is set `active: 0`.
+- `togadesk desk/includes/functions.php:1017` `$ticket['clientid']` offset on false (~1,500 per log): Desk crons do not use App_Error, so noise, not a crash.
+- `get_new_detectprotectservices.php`: the remote feed returns HTML, not CSV, since ~2025-09-11. Remote issue; the cron now skips those rows.
+- `addDeviceTickets.php` (ManageEngine): fails loud with curl error / HTTP code / body snippet, 5 s timeout (`MANAGEENGINE_TIMEOUT_SECONDS`). ManageEngine host reachability from worker1.5 is still open.
+- NYC DOE `send_request_item_updates.php` is broken in git (undefined vars); see [ServiceNow integration § Gotchas](../../../../clients/nycdoe/features/servicenow-integration.md).
+
 ## Gotchas
 
 - **A script that calls `isProcessRunning()` itself fails half-way.** `crons/notifications/infrastructure/send_emails.php` passes `false` to `cronInitialization()` (so it checks in every run) but calls `isProcessRunning()` again at line 14 and exits. Under the bash shell it checked in every 3 minutes and sent nothing. A fresh `CronJobExecutions` row is not proof the job did its work.
-- **Schedules come from `worker1.5/schedules/cron.worker.json` only.** The old `worker` repo schedule files no longer drive prod.
+- **Schedules come from `worker1.5/schedules/cron.worker.json` only.** The old `worker` repo is retired (all its crons moved here 2026-10-22).
 
 ## Worked example (2026-10-22 16:56 → 2026-10-23 ~10:30 CT, TRUE-83121)
 
 TRUE-83078's installer wrote `SHELL=/bin/bash`. 222 of 234 active crons never checked in. Seen: 16 Compass SOs not sent to MITS (oldest 836 min, MITS returned 200 on every call it got, last call 16:10), all Compass 1.0 workflow crons, NetSuite toga2 syncs, Prudential SR→SO (12 REQs), 141 unsent `Common.EmailsQueued` ticket emails (RUMC, TOW, PC Matic, Apiary, Craftex), and NYC DOE "Send DOE Courier Installation Schedule (7:00 AM)" and "Daily Open and Closed DOE Report (1:00 AM)" missed on 2026-10-23. Fixed by dropping the `SHELL` line (worker1.5 PR #8, `ce0f0b2`). Backlogs drained by ~10:52 CT; missed DOE reports needed a manual re-run.
 
 ## Change history
+- 2026-10-23 — Added log triage steps and known log entries after the PHP 8.5 stabilization (TRUE-83163). (jcardinal)
 - 2026-10-23 — Created from the worker1.5 cron outage after the single-server move (TRUE-83121, TRUE-83116, TRUE-83138). (bala)

package/knowledge/1.0/standards/backend-php.md

@@ -527,7 +527,7 @@ Rules:
 
 ### Asynchronous / background processing
 
-- For long-running or background work, hand off to the legacy worker tier (the `worker` / `worker1.5` apps) rather than running long tasks inline in a web request. Use AJAX for front-end interactions.
+- For long-running or background work, hand off to the legacy worker tier (`worker1.5`; the old `worker` repo is retired) rather than running long tasks inline in a web request. Use AJAX for front-end interactions.
 
 ## API design
 

package/knowledge/INDEX.md

@@ -5,9 +5,9 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 ## 1.0 framework
 
 - **library** (Library) _(framework core)_ — 41 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
-- **worker** (Worker) — 43 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
+- **worker** (Worker) — 42 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
 - **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
-- **worker1.5** (Worker 1.5) — 6 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
+- **worker1.5** (Worker 1.5) — 7 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
 - **togadesk** (TOGa Desk) — 18 doc(s) → [1.0/apps/togadesk/INDEX.md](1.0/apps/togadesk/INDEX.md)
 - **togaview** (TOGa View) — 9 doc(s) → [1.0/apps/togaview/INDEX.md](1.0/apps/togaview/INDEX.md)
 - **webhook** (Webhook) — 1 doc(s) → [1.0/apps/webhook/INDEX.md](1.0/apps/webhook/INDEX.md)

package/knowledge/clients/aig/profile.md

@@ -8,11 +8,12 @@ apps:
   - library
   - worker
   - worker2
+  - worker1.5
 project: API
 client: aig
 type: profile
 status: active
-updated: 2026-09-16
+updated: 2026-10-23
 owners: ["mhammontree", "jcardinal"]
 files: []
 related:

package/knowledge/clients/compass-canada/profile.md

@@ -15,11 +15,12 @@ apps:
   - dbchanges2
   - library
   - tools
+  - worker1.5
 project: _Underscore
 client: compass-canada
 type: profile
 status: active
-updated: 2026-10-22
+updated: 2026-10-23
 owners: [jcardinal, bala, tcox, apeterson, ajean, mhammontree, rgirish]
 files: []
 related:

package/knowledge/clients/nycdoe/features/servicenow-integration.md

@@ -6,7 +6,7 @@ project: Worker
 client: nycdoe
 type: client-feature
 status: active
-updated: 2026-10-22
+updated: 2026-10-23
 owners: [mhammontree, sking, bala, jcardinal]
 files:
   - worker/crons/sync/nycdoe/import_asn.php
@@ -179,7 +179,7 @@ All **live** DOE business logic is 1.0. The only other 2.0 code is the in-progre
 ## Gotchas
 - **OPEN (seen 2026-10-22, not investigated) — `Undefined offset` at `library/app/asnprocessor/apple.php:74`** from `legacy_import_asn.php`; prod `Logs.Issue` #1115, new that day.
 - **⚠ OPEN BUG (found 2026-10-19, not fixed) — 1.0 Lexmark/CDW dates save as `1969-12-31`.** CDW sends `YYYY/MM/DD`, but `library/app/asnprocessor/lexmark.php` reads dates with `App_Date::convertMDYYYYtoSQLDate`. All 5,244 of 5,244 `App_ASNProcessor_Lexmark` queue rows since 2026-09-01 have `dateShipped` (and other dates) = 1969-12-31. 2.0 Parse reads `YYYY/MM/DD` correctly. Never trust a 1.0 Lexmark/Acer ship date.
-- **⚠ OUTAGE (fixed TRUE-82735, not deployed 2026-10-14) — the RITM and POD crons never finished for months.** Signs: `Common.CronJobExecutions.dtCheckOut` NULL on every run, zero ServiceNow calls. RITM cron: Feb–Mar 2026 refactors dropped the per-MSO inner loop → `Logs.Issue` 1R *"Undefined variable: row2"* every run since at least 2026-05-07 (last sane version `ce98cd2b5`). POD cron: the file query had **no `ORDER BY`**, so every run hit the same poison file (RITM1387575, already closed → empty-body 200 on `-7` → `Logs.Issue` 1S *"Empty response body from API (HTTP 200)"*), died, never set `dtTransmitted`, and re-uploaded that POD every 5 min; 6,718 files stuck, last sent 2026-09-04. Lesson: an unordered work queue + one poison row + a run-killing throw = a silent total outage; order the queue, isolate per row, and capture.
+- **⚠ OUTAGE (fixed TRUE-82735, not deployed 2026-10-14) — the RITM and POD crons never finished for months.** Signs: `Common.CronJobExecutions.dtCheckOut` NULL on every run, zero ServiceNow calls. RITM cron: Feb–Mar 2026 refactors dropped the per-MSO inner loop → `Logs.Issue` 1R *"Undefined variable: row2"* every run since at least 2026-05-07 (last sane version `ce98cd2b5`). POD cron: the file query had **no `ORDER BY`**, so every run hit the same poison file (RITM1387575, already closed → empty-body 200 on `-7` → `Logs.Issue` 1S *"Empty response body from API (HTTP 200)"*), died, never set `dtTransmitted`, and re-uploaded that POD every 5 min; 6,718 files stuck, last sent 2026-09-04. Lesson: an unordered work queue + one poison row + a run-killing throw = a silent total outage; order the queue, isolate per row, and capture. **Still broken on worker1.5 (2026-10-23, TRUE-83163):** its `crons/sync/nycdoe/send_request_item_updates.php` (copied in `58dab19`) has undefined `$row2` / `$locationSysId` / `$sendLocationUpdate` / `$ritmNumber` — a git bug, not PHP 8; left unfixed by choice.
 
 - **⚠ OPEN (parked, handed to Skyler 2026-10-16) — Stage 5 repair-order loop on RITM1320328.** MSO 317327 / PO `WR260255743` / ASN 24959 item 51224 (qty 282, 283 units, `installTicketStatus` `pending`, 18 units with no `togadeskRepairOrderId`). Since June `3_create_installation_ticket.php` creates a new INSTALLATION repair order every few hours: 637 ROs (1 COMPLETE, 23 ORDER_ASSIGNED_AWAITING_SCHEDULING, 613 ORDER_CREATED_AWAITING_REVIEW), all on serial `NXJJEAA0015281EAF72P00`. Each pass also PATCHes the RITM to `-12`, which ServiceNow refuses (empty-body 200). Stop lever: cancel `AdvanceShippingNoticeItems.installTicketStatus` with a reason, per `dbchanges/Core/SK/2026-07-30-dedupe-asn-install-units.sql` — a business call. Root cause in code not confirmed.
 - **OPEN (parked 2026-10-16) — POD cron PATCHes a BLANK ticket number.** `send_nycdoe_proof_of_delivery.php` with an empty `ritmRequestId` calls `/api/nycd3/data/tasks/` → HTTP 400, ~8 per 8h, in pairs every ~2h.
@@ -295,6 +295,7 @@ All **live** DOE business logic is 1.0. The only other 2.0 code is the in-progre
 - Local deep-dive (full WR260236464 case study with record IDs): `doe_service_now_integration.md` in Mark's WWW workspace root; also the `service-now-integration` skill (`asn-import-flow.md`)
 
 ## Change history
+- 2026-10-23 — RITM cron still broken in the worker1.5 copy (undefined vars); not fixed on TRUE-83163 (jcardinal)
 - 2026-10-22 — Open: new `apple.php:74` Undefined offset (`Logs.Issue` #1115), seen during TRUE-82983 (jcardinal)
 - 2026-10-22 — TRUE-83064: repaired customer PO `WR270089579` (two ASNs, Acer + Lexmark vendor) via dbchanges#270; open-bug gotcha — Stage 5 takes the manufacturer from the ASN vendor and keys the MSO on the ASN; cross-received serials dead-end silently; 152 mislabeled assets on Lexmark model `23782` remain. No code change. (sking)
 - 2026-10-19 — Gated delete live since ~10-11 and its re-release login storm (worker2 SFTP IP ban); file names matter to 1.0 only for FileLog/dedupeKey (rename = re-email); open Lexmark 1969-12-31 date bug. (bala)

package/knowledge/clients/pcmaticb2b/profile.md

@@ -5,9 +5,9 @@ project: TOGa
 client: pcmaticb2b
 type: client-feature
 status: active
-updated: 2026-09-16
+updated: 2026-10-23
 owners: [snaredla, jcardinal]
-apps: [_underscore, api2, worker, worker2, togadesk, library, dbchanges2]
+apps: [_underscore, api2, worker, worker2, togadesk, library, dbchanges2, worker1.5]
 files:
   - worker/crons/toga2/startech/sync_togadesk_startech_pcmaticb2b.php
   - togadesk/desk/includes/functions.php

package/knowledge/clients/prudential/features/device-information-import-and-contact-linking.md

@@ -1,15 +1,15 @@
 ---
 title: "Prudential: Device information import + unit→contact linking (import_device_information.php)"
 framework: "1.0"
-repo: worker
-project: Worker
+repo: worker1.5
+project: Worker 1.5
 client: prudential
 type: client-feature
 status: active
-updated: 2026-09-16
-owners: ["bala"]
+updated: 2026-10-23
+owners: ["bala", "jcardinal"]
 files:
-  - worker/crons/toga2/prudential/import_device_information.php
+  - worker1.5/crons/toga2/prudential/import_device_information.php
   - worker/crons/toga2/prudential/backfill_unit_contacts.php
   - dbchanges2/Client_Prudential/2026-07-07 - Contact Dedup Merge.sql
 related:
@@ -19,7 +19,7 @@ related:
 How the device-sync cron links a Unit to its employee Contact, the "null null" duplicate-contact incident, and the dedup+backfill remediation; open before touching device import or contact linking.
 
 ## Summary
-Prudential's **device-sync** cron (`worker/crons/toga2/prudential/import_device_information.php`) pulls device/asset records (from ServiceNow / the Dell CMDB feed) and PUTs them into the 2.0 platform (`Client_Prudential` via the TOGa2 API), writing `Units` and their owning `Contact`. Each unit's contact is keyed to an employee by the custom field **`c_employeeXid`**. This records how the cron links a unit to its real employee contact, the production incident from getting that wrong, and the two remediation pieces (a dedup migration and a null-contact backfill cron).
+Prudential's **device-sync** cron (`worker1.5/crons/toga2/prudential/import_device_information.php`; was `worker`, retired) pulls device/asset records (from ServiceNow / the Dell CMDB feed) and PUTs them into the 2.0 platform (`Client_Prudential` via the TOGa2 API), writing `Units` and their owning `Contact`. Each unit's contact is keyed to an employee by the custom field **`c_employeeXid`**. This records how the cron links a unit to its real employee contact, the production incident from getting that wrong, and the two remediation pieces (a dedup migration and a null-contact backfill cron).
 
 ## How it works
 - The cron loads **all** existing Prudential contacts once into an in-memory **`c_employeeXid` → keeper uuid** map (`loadPrudentialContactMap`).
@@ -51,6 +51,7 @@ Uses inline keeper subqueries — **no work table, no backup table** (the DBA ta
 Resumable, unit-driven. For each `Unit` with `contactId` NULL it queries ServiceNow by serial (`/gbts/v1/sn_pcaas/asset?serial_number=`) for the owner XID, resolves the contact uuid (reusing the map logic), and PUTs `/units {serialNumber, contact:{uuid}}`. Resumable via a `Parameters` pointer key **`SERVICE_NOW_UNIT_CONTACT_BACKFILL`** (upserts; generates a uuid4 on insert), time-bounded per run.
 
 ## Gotchas
+- **The unit lookup returns `units` as a list** (since the explicit serialNumber where/and filter, `worker` `e7926dd2`). Read `units[0]`; an empty list means no unit → create branch.
 - **⚠ Never link a related record by a non-identifier custom field.** `c_employeeXid` is not a match key; only `uuid` (or the numeric internal id) links an existing child. Passing the custom field alone silently creates a new empty record on every write.
 - **Ordering is load-bearing:** deploy the cron fix → run the dedup migration → run the backfill. Running dedup before the cron fix regenerates duplicates.
 - **⚠ Do NOT force NULL contactId to zero.** Null-contact units are frequently **legitimate**: Dell has not yet assigned the laptop to an employee, so the device is unassigned inventory (ServiceNow returns no `ownedBy`). Verified read-only on the prod copy: of **22,960** null-contact units, only **26** shipped on a Toga ASN (all 26 trace to a named order contact); the other ~22,934 never went through a Toga order and cluster in inventory dispositions/stages (only 15 "In Use"). **Caveat:** the device-import cron also pulls CMDB assets never ordered through Toga, so "not on an ASN" does not by itself prove unassigned — definitive ownership is **ServiceNow-by-serial only**. A residual of no-owner units after backfill is **expected and correct**.
@@ -60,6 +61,7 @@ Resumable, unit-driven. For each `Unit` with `contactId` NULL it queries Service
 - **⚠ `Client_Prudential.Users.contactId` is now NOT NULL — and do NOT blind email-create contacts for Prudential.** The `Client_Prudential/2026-09-21a` backfill linked ~14,298 null-contactId users to their `c_employeeXid` **keeper** contact (ROW_NUMBER: primary-email, then name, then oldest id), created a Contact+email for only ~20 residual, then set `contactId` NOT NULL. Prudential's Contacts were just deduped 893k→19.7k, so a blanket email-create backfill would **re-pollute** them — link by XID keeper, never create-per-user. Part of the platform-wide contactId initiative: [client-user-contactid-required](../../../2.0/apps/_underscore/features/client-user-contactid-required.md).
 
 ## Change history
+- 2026-10-23 — Unit lookup returns `units` as a list since the explicit serialNumber where/and filter (`worker` `e7926dd2`); cron reads `units[0]`, empty list → create. Moved to worker1.5 (TRUE-83163) (jcardinal)
 - 2026-09-21 — `Client_Prudential.Users.contactId` made NOT NULL after the `2026-09-21a` backfill linked null-contactId users to their XID keeper contact. Noted the "never blind email-create for Prudential" rule (would re-pollute the just-deduped Contacts) and cross-linked the platform contactId initiative. (jcardinal)
 
 ## Related

package/knowledge/clients/staples/profile.md

@@ -7,11 +7,12 @@ apps:
   - worker2
   - dbchanges
   - dbchanges2
+  - worker1.5
 project: Worker
 client: staples
 type: profile
 status: active
-updated: 2026-10-19
+updated: 2026-10-23
 owners: [jcardinal, bala, mhammontree]
 files:
   - worker/crons/sync/staples/sync_staples_cxml.php

package/knowledge/clients/walmart/profile.md

@@ -8,11 +8,12 @@ apps:
   - library
   - walmarttechservices
   - walmart-activate
+  - worker1.5
 client: walmart
 type: client-feature
 status: active
-updated: 2026-10-22
-owners: ["mhammontree", "rgirish"]
+updated: 2026-10-23
+owners: ["mhammontree", "rgirish", "jcardinal"]
 files:
   - library/app/client/walmart.php
   - library/app/email/walmartb2c.php

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.1132",
+  "version": "1.0.1133",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",