toga-ai 1.0.1126 → 1.0.1127

package/knowledge/1.0/apps/library/features/cron-execution-monitoring.md

@@ -6,15 +6,17 @@ project: Library
 client: shared
 type: feature
 status: active
-updated: 2026-10-01
+updated: 2026-10-23
 owners: [dfranks, bala, mhammontree]
 files:
   - library/app/framework.php
 related:
   - ../../worker/architecture.md
+  - ../../worker1.5/architecture.md
   - ./email-queue-attachments.md
   - ../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md
   - ../../worker/workflows/tracing-a-worker-cron-run-in-production.md
+  - ../../worker1.5/workflows/diagnosing-crons-not-running.md
 ---
 
 1.0 cron check-in/out lifecycle via `App_Framework::cronInitialization()`/`cronFinished()` → `Common.CronJobExecutions`; open to monitor a 1.0 cron or read its per-run evidence.
@@ -48,6 +50,8 @@ Two consequences:
 ## Gotchas
 
 - **⚠ A run skipped by the overlap guard leaves NO ROW — absence is ambiguous.** `cronInitialization()` calls `exitIfProcessRunning()` **before** the `CronJobExecutions` INSERT, and the `cronFinished(true)` that guard then calls does nothing because `cronLogId` was never set. So a blocked run is indistinguishable from a run that never launched: **a missing row for an expected slot means "skipped", not "hung"**, and a hung run looks different (it has `dtCheckIn` with no `dtCheckOut`). `isProcessRunning()` matches **any** `ps -ef` line containing the script path (it only excludes `/bin/sh` and its own pid), so a stray `tail -f` or editor on that path blocks the cron indefinitely.
+- **⚠ A non-`/bin/sh` crontab shell stops every guarded cron at once (TRUE-83121).** worker1.5's `ebs/cron.worker.php` wrote `SHELL=/bin/bash` into root's crontab, so each job's parent row was `/bin/bash -c "php .../crons/<job>.php ..."`. `isProcessRunning()` matched that parent, so every cron using the default `$exitIfProcessAlreadyRunning = true` (3rd arg of `cronInitialization()`) exited before its INSERT: **222 of 234 crons** silent from 2026-10-22 16:45 to 2026-10-23 10:35 Central (no Office Depot 850s, Staples, Prudential, NetSuite syncs). Only crons passing `false` ran. Fix: remove the `SHELL` line (worker1.5 PR #8). Rule: [worker1.5 Critical rules](../../worker1.5/architecture.md).
+- **Many crons with zero rows since a deploy, while only the `false`-3rd-arg crons still log = the guard matches every run** (shell or `ps` output changed), not real overlap. Steps: [Diagnosing worker1.5 crons that stopped running](../../worker1.5/workflows/diagnosing-crons-not-running.md). worker1.5 rows have `instanceId = ''` ([why](./php-8-5-runtime-compatibility.md)); rows with an EC2 id are the old worker fleet, last at 2026-10-22 16:56 Central.
 - **`CronJobExecutions` is huge (~120M rows) with no index on `job`.** Never query it per run (e.g. as a cron's watermark) — it scans the table. Use a small dedicated table instead.
 - **`db_log` has no `CronJobExecutions` table.** A `cronFinished()` UPDATE that runs against the `db_log` connection throws on **every** 1.0 cron job. Both the INSERT and the UPDATE must target `db_common`.
 - **A `git merge` can silently resurrect pre-refactor logic.** These functions were cleanly consolidated to a single `db_common`-only INSERT/UPDATE pair (commit `0a3a6f75`), but a later `git merge origin/_production` (`509614036605…`) reintroduced the old dual-path structure from `_production`'s still-unmigrated copy — leaving duplicate INSERT/UPDATE blocks and reverting one path's connection argument back to `db_log`. Automatic merge resolution updated the SQL text (renamed the table) but restored the wrong connection name, producing a platform-wide breaking bug disguised as a simple PR-feedback rework. **When a merge touches a function you already refactored on your branch and the base branch never got that refactor, diff against the merge-base — do not just eyeball the current file** — to catch resurrected logic. (Fixed 2026-07-06: consolidated back to one `db_common` INSERT/UPDATE pair.)
@@ -58,4 +62,5 @@ Two consequences:
 - [OneUptime worker2 monitoring](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md), [Tracing a worker cron run in production](../../worker/workflows/tracing-a-worker-cron-run-in-production.md).
 
 ## Change history
+- 2026-10-23 — Added the crontab-shell failure mode (`SHELL=/bin/bash` stopped 222 of 234 worker1.5 crons), the zero-rows tell, and the worker1.5 `instanceId = ''` note (TRUE-83121). (bala)
 - 2026-10-01 — Added the no-index / ~120M-row gotcha (rejected as a watermark source in TRUE-82180). (mhammontree)

package/knowledge/1.0/apps/library/features/email-queue-attachments.md

@@ -6,12 +6,13 @@ project: Library
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
+updated: 2026-10-23
 owners: ["mhammontree", "bala"]
 files:
   - library/app/email.php
   - library/phpmailer/class.phpmailer.php
   - worker/crons/notifications/infrastructure/send_emails.php
+  - worker1.5/crons/notifications/infrastructure/send_emails.php
   - worker/crons/notifications/covid/send_covid_pending_vaccination_approval.php
   - togadesk/desk/includes/functions.php
 related:
@@ -58,9 +59,11 @@ Therefore: **deploy `worker` before `togadesk`.** Roll back in the inverse order
 - **PHPMailer 5 hides a missing attachment.** `AddAttachment()` (`library/phpmailer/class.phpmailer.php` ~L1358) treats an unreadable path as `STOP_CONTINUE` with `$exceptions` off, and sends the message anyway. Absence of an error proves nothing.
 - **The drain cron's exclusion filter is dead code.** `send_emails.php` ~L34 carries `AND EmailsQueued.serializedEmailObject NOT LIKE '%App_Email%'`. The column holds base64, in which the literal `App_Email` never appears (verified with a PHP round-trip) — the filter matches and excludes nothing. **Not changed**: "fixing" it would stop TOGa Desk mail entirely. Needs a decision from whoever wrote it before anyone touches it.
 - **OPEN RISK — `max_allowed_packet` on the prod Common cluster is unverified.** Rows can now carry ~9.8 MB of base64. The column is LONGTEXT so it is fine, but MySQL 5.7 defaults `max_allowed_packet` to 4 MB (8.0: 64 MB). If prod is below the budget the INSERT fails as an `App_Query` exception on a live ticket reply — worse than a missing attachment. Check it, and lower `MAX_QUEUED_ATTACHMENT_BYTES` beneath it if needed (the oversized path already degrades to an inline send, the correct behavior).
+- **⚠ OPEN: an email with an empty From fails at SES every time.** `App_Email` puts `fromEmailAddress` into `$mailer->From` with no check, and SES rejects it: `The following From address failed: ... Invalid MAIL FROM address provided`. Seen since at least 2026-10-01 (308 failures after 2026-10-22 12:00) on Elite Helpdesk "Support Ticket #N: Elite - Elite Helpdesk - Contact Center" and Office Depot "Support Ticket #N: Office Depot - Talos Office Depot tickets" mail; ticket #14781 had 260 attempts to 20 recipients. The drain deletes the queue row after one try, so repeats mean the caller sends it again each run. The code that builds these emails is not in the local repos; not fixed. Not caused by worker1.5.
+- **Debug send failures in legacy `Logs.Emails`** (`dtSent`, `isSuccess`, `error`, `sourceJob`, `instanceId`, `subject`, `from`, `to`): group by `from`, `isSuccess`, `error`. On worker1.5 `instanceId` is `''` ([PHP 8.5 compatibility § Gotchas](./php-8-5-runtime-compatibility.md)).
 - Regression coverage: `test/@Mark/TRUE-80952/test_queued_email_attachments.php` — 26 assertions, no DB, no SMTP, confirmed failing against pre-fix HEAD.
 
 ## Related
 
-- [Branded HTML email templates](./email-templates.md), [App_Email side effects & test mode](./email-test-mode-and-write-free-sends.md).
+- [Branded HTML email templates](./email-templates.md), [App_Email side effects & test mode](./email-test-mode-and-write-free-sends.md), [SES sender identity](./email-ses-sender-identity.md).
 - [Library architecture](../architecture.md), [TOGaDesk notifications](../../togadesk/features/notifications.md).

package/knowledge/1.0/apps/library/features/error-capture-1-0.md

@@ -6,10 +6,11 @@ project: Library
 client: shared
 type: feature
 status: active
-updated: 2026-10-22
+updated: 2026-10-23
 owners: ["jcardinal", "mhammontree", "bala", "snaredla", "rgirish"]
 files:
   - library/app/error/capture.php
+  - worker1.5/crons/sync/retailservices/sync_retail_services_client_service.php
   - library/app/error.php
   - library/app/exception/business.php
   - library/app/exception/api.php
@@ -118,6 +119,8 @@ Two things are specifically 1.0:
 ## Gotchas
 
 - **Looking for a 1.0 cron fatal? Query `Logs.Issue` on environment `prod`, ordered by `dtLastOccurred`.** The legacy `Logs.Errors` and `Vision_Log.ErrorLog` tables do **not** carry 1.0 cron fatals — searching them wastes time (2026-09-18, DOE outbound-cron outage). Uncaught exceptions reach here via `App_Error::handleException` → `captureException()`, which returns the short quotable reference. **A `catch` BYPASSES capture** — if you catch per record, call `App_Error_Capture::captureException($e)` yourself, and put only the returned reference (never `$e->getMessage()`, which can carry a raw vendor response body) into an email or log.
+- **Did a fix stop an issue? Count its events per hour.** `Logs.Event` joined to `Logs.Issue` on `Event.issueId = Issue.id`, `WHERE Issue.reference = '<ref>'`, grouped by hour of `Event.dtCreated`. Example: worker1.5 issue `17P` ran 460 to 500 events an hour; the last one was 2026-10-23 10:28:20, one minute after the fix merged (TRUE-83107).
+- **⚠ OPEN: worker1.5 OptimumDesk sync puts a password into the error message.** `crons/sync/retailservices/sync_retail_services_client_service.php` L471 and L591 throw with `print_r($rowAccount, true)`. `$rowAccount` is the `$_ENV` the parent passes, which includes `ClientSuppliers.opDeskPassword`. The message is not redacted, so it lands in `/var/log/worker-cron.log` and in `Issue.subject` / `Event.errorMessage`. Fix: put only ids (`clientAccountId`, `api`) in the message. Rule: never `print_r` a row that carries credentials into an exception or log.
 - **App_Error writes the real error message nowhere on the server** — only to Sentry and `Logs.Issue`. On a new env where both sends fail (no `[database_toga2logs]`, Sentry blocked) the red box is all you get. Fix per app: wrap the App_Error handlers and also `error_log()` + `syslog(LOG_ERR)` the message; syslog reaches `/var/log/messages`, which EB log bundles include (togadesk `_/app/framework.php`, TRUE-83053 — [Desk deploy § Error visibility](../../togadesk/workflows/deploy-to-elastic-beanstalk-al2023.md#error-visibility)).
 - **Vision does not capture — use Sentry.** `vision/config.prod.ini` has no `[database_toga2logs]` section, so vision errors never reach `Logs.Issue`; only Sentry project `vision` has them. Do not search `Logs.Issue` for vision errors.
 - **`\Error` and `\Exception` declare `$message` separately — reflect the right base class.** `App_Error::handleException` (`library/app/error.php` ~646) reflected `\Exception::$message` on a PHP `\Error` (e.g. class not found) → `Cannot access protected property Error::$message`, so the error never reached Sentry, and the catch `debugVar()`-ed the full trace onto the prod page (TRUE-82793). Fixed on library branch TRUE-82959: reflect `'\Error'` when `instanceof \Error`, else `'\Exception'`; the catch now `error_log()`s and only `debugVar()`s under `App_Registry::inDevMode()`.
@@ -131,6 +134,7 @@ Two things are specifically 1.0:
 - **⚠ The legacy pre-Sentry handler dumps the entire config on any error it renders.** `library/app/DELETE_DOA_PRE-SENTRY_error.php` is a dead-named but still-reachable handler whose `htmlBacktrace()` does `print_r($entry['args'])` for **every** frame. On 2026-08-05 a real error page on **https://towfoundation.togaview.com** rendered ~26 pages containing the whole config array — production DB host/user/password, SES SMTP credentials, NetSuite credentials, Payeezy API key/secret/merchant token, SSL Store token — plus ticket PII. The current `library/app/error.php` is safe (it dumps only under `App_Registry::inDevMode()`), but **both** handlers gate on `inDevMode()`, so the affected host was **evaluating as dev mode in production**. Two actions: (1) verify `inDevMode()` per host in production before trusting any error page not to leak, and (2) delete the legacy handler. Rotation of every exposed credential **was escalated to Jeff on 2026-08-05; confirmation that it was completed is still owed** — treat those credentials as compromised until confirmed. (No values are recorded here or anywhere in the KB.)
 
 ## Change history
+- 2026-10-23 — Issue event-rate check; open OptimumDesk password leak in worker1.5 exception messages (TRUE-83107) (bala)
 - 2026-10-22 — Gotcha: App_Error never logs the message on the server; app-level syslog pattern (TRUE-83053) (rgirish)
 - 2026-10-22 — `setAccessible()` guarded for PHP 8.5 in `error.php` / `capture.php`; likely cause of the open handler crash (TRUE-83059) (rgirish)
 - 2026-10-20 — Fixed `\Error` reflection + prod trace dump in `handleException` (TRUE-82959) (snaredla)

package/knowledge/1.0/apps/library/features/php-8-5-runtime-compatibility.md

@@ -6,8 +6,8 @@ project: Library
 client: shared
 type: feature
 status: active
-updated: 2026-10-22
-owners: ["rgirish", "jcardinal"]
+updated: 2026-10-23
+owners: ["rgirish", "jcardinal", "bala"]
 files:
   - library/app/error.php
   - library/app/date.php
@@ -23,6 +23,9 @@ files:
   - library/app/api/cybersource.php
   - library/app/togadesk/reports.php
   - library/browser/datagrid.php
+  - library/app/framework.php
+  - library/app/edi.php
+  - library/netsuitetoolkit/NSPHPClient.php
 related:
   - error-capture-1-0.md
   - ../../walmart-activate/workflows/deploy-to-elastic-beanstalk-al2023.md
@@ -37,7 +40,7 @@ How library (1.0) handles PHP 8.4/8.5 deprecations (logged, not fatal), the PHP
 
 ## Summary
 
-Since library PR #905 (TRUE-83055, on `_production` 2026-10-22), `App_Error::handleError` logs `E_DEPRECATED` / `E_USER_DEPRECATED` once per file:line per request (`error_log`, via `logDeprecation`) and returns `true`. A deprecation no longer goes through `handleException`, so it does not roll back the open transaction or exit. Every other warning and notice still exits as before. library must stay PHP 7.2 safe (other 1.0 apps run 7.2): fixes are version guards or 7.1-valid syntax only.
+Since library PR #905 (TRUE-83055, on `_production` 2026-10-22), `App_Error::handleError` logs `E_DEPRECATED` / `E_USER_DEPRECATED` once per file:line per request (`error_log`, via `logDeprecation`) and returns `true`. A deprecation no longer goes through `handleException`, so it does not roll back the open transaction or exit (seen on worker1.5 prod: one Staples `sync_staples_cxml.php` run logged deprecations at L548, L786, L795, L835 and kept going). A deprecation masked out by `error_reporting()` is dropped silently. Every other warning and notice still exits as before. library must stay PHP 7.2 safe (other 1.0 apps run 7.2): fixes are version guards or 7.1-valid syntax only.
 
 ## How it works
 
@@ -47,12 +50,14 @@ Fix patterns (TRUE-83059 `d28f9b23`, TRUE-83055 PR #905):
 |---|---|---|
 | 8.5 | `curl_close()`, `imagedestroy()`, `xml_parser_free()` | call only when `PHP_VERSION_ID < 80000` (no-ops since 8.0) |
 | 8.5 | `ReflectionProperty::setAccessible()` | call only when `PHP_VERSION_ID < 80100` |
-| 8.4 | implicit nullable `string $x = null` | `?string $x = null` (valid since 7.1) |
+| 8.4 | implicit nullable `string $x = null` | `?string $x = null` (valid since 7.1); e.g. `NSPHPClient::setFields(?array $fieldArray = null)` (PR #908) |
+| 8.1 | `null` into `str_replace` from `App_Edi::cleanText`'s `json_decode` of an unquoted `\u0000...\u007f` string | block removed (PR #908): the decode was always `NULL`, so the replace never changed EDI text on any version; output byte-identical |
 | 8.2 | `utf8_encode($x)` / `utf8_decode($x)` | `mb_convert_encoding((string) $x, 'UTF-8', 'ISO-8859-1')` / `mb_convert_encoding((string) $x, 'ISO-8859-1', 'UTF-8')` — byte-identical for all 256 bytes, so the Cybersource HMAC is unchanged |
 | 8.5 | `$http_response_header` (`app/error/capture.php`) | library: `stream_get_meta_data()` (7.2 safe); 8.4+-only apps: `http_get_last_response_headers()` |
 | 8.x | `(boolean)` / `(integer)` casts | `(bool)` / `(int)` |
 | 8.2 | dynamic property | declare it on the class (TRUE-83055: `App_Api_Carrier_ShipmentRequest::$shipToAddressLine2`, `App_Email_AigTemplates_SendAigSupportTicketReplyNotification::$contact`) |
 | 8.1 | `SoapClient` overrides without return types (`app/soaptransaction.php`) | return-type attributes + 6th `$uriParserClass` arg |
+| 8.5 | `case 'x';` (semicolon after `case`) — vendored `php-imap2/src/Roundcube/ImapClient.php:2580` flooded worker1.5's `worker-cron.log` | `case 'x':` (library `53ea57f8`) |
 
 State on `_production`: `php -l` on PHP 8.4 shows no deprecations in `app/` or `browser/`.
 
@@ -70,11 +75,15 @@ Null guards (TRUE-83054, commit `04a2293c`, landed directly on `_production`)
 - **Still open — vendored third-party folders only:** `LightSaml/`, `simplesaml/`, `jpgraph/`, `mobile_detect/`, `invoiceprocessor/pdf/` raise 8.4 compile-time deprecations (implicit nullable, optional-before-required, `${var}`). Logged, not fatal.
 - **Warnings and notices still exit.** Only deprecations were softened — an "offset on null" or "undefined index" still kills the request on any PHP version.
 - **`file_get_contents` to IMDS or any optional endpoint:** the `ignore_errors` stream option covers only HTTP status — a refused or slow connection still warns, and App_Error exits. Wrap the call in a local `set_error_handler` / `restore_error_handler` and check for a `200` status line (togaview TRUE-83060).
+- **The `instance-data` hostname does not resolve on AL2023.** Use `169.254.169.254` only in new code. Old uses: `App_Error_Capture::AWS_METADATA_HOSTS` falls back to it when `169.254.169.254` returns non-200 (e.g. the tags path with instance metadata tags off), so the cron log shows `getaddrinfo for instance-data failed` warnings (noise, capture still works). `App_Cloud::getInstanceId()` (`app/cloud.php` ~L711) calls only `instance-data`, so it returns `''`, and `App_Framework::getInstanceId()` caches that `''` in `instanceId.txt`. That is why worker1.5 writes `instanceId = ''` to `Common.CronJobExecutions` and `Logs.Emails`. `app/error.php` ~L642 (Sentry AWS context) also uses it.
 - **PHP 7.2 safety was checked by reading, not running** (no 7.2 binary locally on PR #905). The guards use `PHP_VERSION_ID` (5.2.7+) and `?type` (7.1+). Run a 7.2 smoke test if a 7.2 app breaks after this change.
+- **Do not "repair" the removed `App_Edi::cleanText` block.** A working version would replace every character 0-127 (all letters and digits) with spaces and blank every EDI file.
 - Local dev on PHP 8.2 does not show 8.4/8.5 deprecations — test with an 8.5 binary.
 - **PHP 8 string-to-number comparison can turn a loop infinite.** `api` subscription: a non-numeric quantity compared to an int looped forever on 8.x. Cast `(int)` before the loop.
 
 ## Change history
+- 2026-10-23 — `instance-data` does not resolve on AL2023 (empty instance id, log noise); deprecations-continue behavior confirmed in the worker1.5 cron log (TRUE-83107). (bala)
+- 2026-10-23 — `case 'x';` 8.5 deprecation fixed in vendored php-imap2 (TRUE-83116). (bala)
 - 2026-10-22 — Dynamic-property fixes; app-level offset-on-false shim pattern from the Desk move (TRUE-83055, TRUE-83053) (rgirish)
 - 2026-10-22 — `App_Date` / `App_Database` null guards; 8.x break classes from the `api` move (TRUE-83054) (jcardinal)
 - 2026-10-22 — `$http_response_header` is an 8.5 deprecation; optional-endpoint warning pattern (TRUE-83060) (rgirish)

package/knowledge/1.0/apps/store/workflows/deploy-to-elastic-beanstalk-al2023.md

@@ -6,8 +6,8 @@ project: Store
 client: shared
 type: workflow
 status: active
-updated: 2026-10-22
-owners: ["rgirish"]
+updated: 2026-10-23
+owners: ["rgirish", "bala"]
 files:
   - store/.ebextensions/001_platform.config
   - store/.ebextensions/009_setup_phpini.config
@@ -48,12 +48,13 @@ store moved from PHP 7.2 / Amazon Linux 1 to PHP 8.5 / AL2023 on TRUE-83055 (sto
 
 - **The S3 `setup_git_libraries.php` clones nothing on AL2023** (it needs IMDSv1 and `/var/app/ondeck`). Use the bash clone. Never use `ignoreErrors` on the clone.
 - **PHP 8.x code fixes in store (TRUE-83055):**
-  - `case X;` → `case X:` (49 places: `common/spglobal/header.php`, `common/spglobal/home.php`, `common/nychh/home.php`). 8.5 prints a deprecation, and 1.0 exits on it.
+  - `case X;` → `case X:` (49 places: `common/spglobal/header.php`, `common/spglobal/home.php`, `common/nychh/home.php`). 8.5 prints a deprecation (fatal before library PR #905, now only logged).
   - `App_Database::fetchRow($res)[0]` → `(App_Database::fetchRow($res)[0] ?? null)` (18 places in `_/app/framework.php`, `mvc/users/impersonate/{put,delete}.php`). On no row it warns "offset on null" (since 7.4), and App_Error exits. Seen as Compass login failing when the user has no Location.
 - **Open: ~149 call sites pass a possibly-NULL DB value to `trim` / `strtolower` / `explode`** (8.1 deprecation). Not fatal: library `_production` logs deprecations instead of exiting (PR #905, see [PHP 8.5 runtime compatibility](../../library/features/php-8-5-runtime-compatibility.md)). Fix them from `error_log`.
 - **Open security issue (pre-existing, needs its own ticket):** committed `config.*.ini` in the store web root can be downloaded under Apache. Same issue as Walmart Activate; copy togaview's working `deny.conf` fix ([TOGa View deploy](../../togaview/workflows/deploy-to-elastic-beanstalk-al2023.md)). `ebs/git.json` holds a clone token — never copy it out.
 - **Not yet checked on the new environment:** `getenv('ENVIRONMENT')` under php-fpm behind Apache, and that `/var/app/staging` is removed after deploy (the `005` php-file-watch cron skips while it exists).
 
 ## Change history
+- 2026-10-23 — `case X;` note: deprecations are logged, not fatal, since PR #905 (TRUE-83107) (bala)
 - 2026-10-22 — Live on `togatech-legacy-store-production` (toga-prod); arm64 dart-sass in `021` (TRUE-83055) (rgirish)
 - 2026-10-22 — Created: store moved to PHP 8.5 / AL2023 (Apache proxy, bash clone, phpini composer, Redis via `ini_set`, 8.x code fixes) (TRUE-83055) (rgirish)

package/knowledge/1.0/apps/tools/features/github-audit.md

@@ -6,8 +6,8 @@ project: Tools
 client: shared
 type: feature
 status: active
-updated: 2026-10-09
-owners: [jcardinal]
+updated: 2026-10-23
+owners: [jcardinal, bala]
 files:
   - tools/_/app/github/audit.php
   - tools/mvc/developers/github-audit/get.php
@@ -85,7 +85,7 @@ Behind Tools SSO. `get.php` and `post.php` self-guard with `requireAuth` + `App_
 
 ## Gotchas
 - **The `hidden` attribute is beaten by a class `display:` rule.** The results/spinner box had the HTML `hidden` attribute but still showed on load (looked like the tool auto-ran), because an author `display:flex` class rule beats the UA `[hidden]{display:none}`. Fix: a scoped guard `.<scope> [hidden] { display:none !important; }` (or `:not([hidden])` on the display rule). General server-rendered-UI trap.
-- **`curl_close()` is fatal on PHP 8.5** in the 1.0 framework (deprecation → `ErrorException` → aborted request). The audit engine avoids `curl_close()` — same rule as the Design tool. See [Tools architecture](../architecture.md) and framework-rules.
+- **The audit engine avoids `curl_close()`** (deprecated in 8.5, a no-op since 8.0). It aborted the request until library PR #905 (2026-10-22); now deprecations are only logged ([PHP 8.5 compatibility](../../library/features/php-8-5-runtime-compatibility.md)).
 - **APCu lock is per-EB-instance**, so the single-run guard is per-instance, not org-wide.
 
 ## Related
@@ -95,4 +95,5 @@ Behind Tools SSO. `get.php` and `post.php` self-guard with `requireAuth` + `App_
 - [github_audit.php standalone script](../../test/features/github-audit-script.md)
 
 ## Change history
+- 2026-10-23 — `curl_close()` no longer fatal: library logs deprecations (TRUE-83107) (bala)
 - 2026-10-09 — `token()` now resolves its own `GITHUB_TOKEN`; no longer borrows `App_Design_Github::token()` (deleted when Design moved to S3, TRUE-82616) (jcardinal)

package/knowledge/1.0/apps/tools/features/legacy-email-notifier.md

@@ -6,7 +6,7 @@ project: Tools
 client: shared
 type: feature
 status: active
-updated: 2026-10-13
+updated: 2026-10-23
 owners: [bala, jcardinal]
 files:
   - tools/mvc/email-migration/notify/get.php
@@ -58,8 +58,8 @@ Prod `tools` runs **multiple Elastic Beanstalk instances behind a load balancer*
 ### `App_Email::send()` is unusable in the tools runtime; send via SES SMTP directly
 Every send failed (`sent:0`, all "Send failed") because `App_Email::send()` also does a `Defaults` lookup (Vision / `db_main`), a `Logs.Emails` write (`db_logs`), and an `EmailSuppressionList` check (Common / `db_common`) — that 1.0 email-stack DB wiring is **not** present in the tools app, so it threw for every recipient. Fix: send with **PHPMailer directly** using `App_Email`'s SES SMTP **constants** (`SMTP_HOSTNAME` / `SMTP_PORT` / `SMTP_SECURE` / `SMTP_USERNAME` / `SMTP_PASSWORD` — the same proven SES SMTP creds, `us-west-2`) and **no DB calls**; From `no-reply@togatech.com` / "TOGa IT". **Operational prerequisite:** the `togatech.com` sender identity/domain must be **verified in that us-west-2 SES account** or delivery is rejected. Any future tools-app feature that needs to send email hits the same wall — bypass `App_Email::send()` and go straight to SES.
 
-### PhpSpreadsheet 1.6.0 deprecations abort the request
-The vendored PhpSpreadsheet is **1.6.0** (library composer constraint `^1.6`, which supports only PHP 5.6/7.0) and emits `E_DEPRECATED` all over on PHP 8.x. A single deprecation during the xlsx parse aborts the request, because the 1.0 error handler promotes any PHP notice/deprecation to a thrown exception **and** (when Sentry is absent) echoes "Sentry is not installed…" into the response, corrupting the JSON. Fix: install a **scoped `set_error_handler`** at the top of the endpoint that swallows non-fatal diagnostics (returns `true`) for the duration of the action; real fatals and thrown exceptions still reach the `catch`. The framework mechanism is documented in [mvc-data-access-patterns](./mvc-data-access-patterns.md).
+### PhpSpreadsheet 1.6.0 deprecations aborted the request (before library PR #905)
+The vendored PhpSpreadsheet is **1.6.0** (library composer constraint `^1.6`, which supports only PHP 5.6/7.0) and emits `E_DEPRECATED` all over on PHP 8.x. Until library PR #905 (2026-10-22) a single deprecation during the xlsx parse aborted the request and (when Sentry is absent) echoed "Sentry is not installed…" into the JSON. Now deprecations are only logged ([PHP 8.5 compatibility](../../library/features/php-8-5-runtime-compatibility.md)); a warning or notice still aborts. Fix: install a **scoped `set_error_handler`** at the top of the endpoint that swallows non-fatal diagnostics (returns `true`) for the duration of the action; real fatals and thrown exceptions still reach the `catch`. The framework mechanism is documented in [mvc-data-access-patterns](./mvc-data-access-patterns.md).
 
 **Security:** 1.6 is not safe for untrusted xlsx (XXE scanner bypass, no zip-size limits). New upload features parse in worker2 instead — see [job bridge](./worker2-job-bridge.md).
 
@@ -73,4 +73,5 @@ The vendored PhpSpreadsheet is **1.6.0** (library composer constraint `^1.6`, wh
 - [Tools architecture](../architecture.md)
 
 ## Change history
+- 2026-10-23 — Deprecations no longer abort since library PR #905 (TRUE-83107) (bala)
 - 2026-10-13 — Noted PhpSpreadsheet 1.6 is unsafe for untrusted xlsx; new uploads parse in worker2 (bala)

package/knowledge/1.0/apps/tools/features/mvc-data-access-patterns.md

@@ -6,7 +6,7 @@ project: Tools
 client: shared
 type: feature
 status: active
-updated: 2026-10-06
+updated: 2026-10-23
 owners: [jcardinal, bala, ajean]
 files:
   - tools/_/app/nav.php
@@ -146,8 +146,8 @@ if (!headers_sent()) { http_response_code(401); }   // matches App_MVC::routeTo'
 
 `establishSession()` (auth.php) guards `session_regenerate_id` for the same reason; the SSO failure path hit this too — see [saml-sso-auth](./saml-sso-auth.md). The clean long-term fix is to handle status-setting routes **before any output** and `exit`.
 
-### The error handler promotes ANY PHP notice/deprecation to a thrown exception (and corrupts JSON)
-`App_Error::handleError` (`library/app/error.php`) **ignores the `error_reporting` mask** and promotes **any** PHP notice/warning/deprecation to a thrown `ErrorException`. When Sentry is not installed it **also echoes** "Sentry is not installed…" into the response — which **corrupts a JSON response body**. So a single `E_DEPRECATED` (e.g. an old vendored package like PhpSpreadsheet 1.6.0 running on PHP 8.x) aborts the whole request and, on an AJAX endpoint, breaks the JSON the browser is parsing. This is the same escalation behind the preloader-flush 500 above.
+### The error handler turns ANY PHP notice/warning into an abort (and corrupts JSON)
+`App_Error::handleError` (`library/app/error.php`) **ignores the `error_reporting` mask** for notices and warnings and sends them to `handleException`, which exits. When Sentry is not installed it **also echoes** "Sentry is not installed…" into the response — which **corrupts a JSON response body** on an AJAX endpoint. This is the same escalation behind the preloader-flush 500 above. Deprecations are the exception: since library PR #905 (2026-10-22) they are only logged ([PHP 8.5 compatibility](../../library/features/php-8-5-runtime-compatibility.md)).
 
 Fix for a page/endpoint that legitimately triggers non-fatal diagnostics: install a **scoped** `set_error_handler` at the top of the action that **returns `true`** to swallow non-fatal notices for the duration of the action; real fatals and thrown exceptions still reach your `catch`. Scope it tightly — do not disable diagnostics app-wide.
 
@@ -163,5 +163,6 @@ set_error_handler(fn(): bool => true); // swallow non-fatal notices/deprecations
 - [1.0 MVC page pattern & app skeleton](../../library/features/mvc-page-pattern-and-app-skeleton.md)
 
 ## Change history
+- 2026-10-23 — Deprecations are logged, not aborts, since library PR #905 (TRUE-83107) (bala)
 - 2026-10-06 — Added the `GET_LOCK` 64-char gotcha (hash the variable part). (ajean)
 - 2026-10-05 — Added the request-input xssProtect gotcha (JSON posts decode to NULL). (bala)

package/knowledge/1.0/apps/worker/workflows/running-a-worker-cron-locally.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: workflow
 status: active
-updated: 2026-10-16
+updated: 2026-10-23
 owners: ["bala"]
 files:
   - worker/_/app/framework.php
@@ -64,7 +64,8 @@ For logic-only tests with no bootstrap, use the
 
 ### Step 4 — run on PHP 7.2
 Use Laragon's PHP 7.2.33 build (matches prod). On PHP 8.1 the old PhpSpreadsheet in `library`
-hits an `Iterator` return-type deprecation, which `App_Error` turns into an exit.
+hits an `Iterator` return-type deprecation. Older library builds exited on it; since library PR #905
+(2026-10-22) it is only logged ([PHP 8.5 compatibility](../../library/features/php-8-5-runtime-compatibility.md)).
 
 ## Gotchas
 - **`App_Framework_Worker::initialize()` cannot run twice in one process** — the second
@@ -84,5 +85,6 @@ hits an `Iterator` return-type deprecation, which `App_Error` turns into an exit
 - [App_Email side effects & test mode](../../library/features/email-test-mode-and-write-free-sends.md)
 
 ## Change history
+- 2026-10-23 — Deprecations no longer exit since library PR #905 (TRUE-83107) (bala)
 - 2026-10-16 — Created from the cron 7 local test harness (prod-pointing configs, Sentry, SMTP,
   PHP 7.2, Windows `:` file names, NetSuite 500/unreachable test). (bala, TRUE-82888)

package/knowledge/1.0/apps/worker1.5/INDEX.md

@@ -3,4 +3,6 @@
 | Doc | Summary |
 |-----|---------|
 | [Worker 1.5 (1.0 Framework) Architecture](architecture.md) | The single-server 1.0 cron tier that replaced worker's 7-role fleet; open for how the crontab is built from schedules/*.json, the overlap check, and what the cr |
+| [worker1.5 — Committed Secrets & Credentials in Logs (debt register)](features/committed-secrets-and-log-exposure.md) | Open security debt in worker1.5: a GitHub token committed in `ebs/git.json` and a cron that writes OptimumDesk passwords into the cron log; open before touching |
 | [worker1.5 PHP ini overrides on EB AL2023 (memory_limit 11G, zz-application.ini)](features/php-ini-memory-limit-eb.md) | How worker1.5 sets PHP ini values (memory_limit 11G) on EB `worker15-production`; open before changing any PHP ini setting or wondering why the EB console value |
+| [Diagnosing worker1.5 crons that stopped running (single-server 1.0 tier)](workflows/diagnosing-crons-not-running.md) | Steps to tell "worker1.5 crons are not running" from "a downstream system is failing", find the cause on the box, and recover; open when several 1.0 jobs or bac |

package/knowledge/1.0/apps/worker1.5/architecture.md

@@ -15,11 +15,14 @@ files:
   - worker1.5/crons/
   - worker1.5/mvc/get.php
   - worker1.5/ebs/setup_phpini.php
+  - worker1.5/ebs/setup_git_libraries.php
 related:
   - features/php-ini-memory-limit-eb.md
   - ../worker/architecture.md
   - ../library/architecture.md
   - ../../standards/backend-php.md
+  - workflows/diagnosing-crons-not-running.md
+  - features/committed-secrets-and-log-exposure.md
 ---
 
 The single-server 1.0 cron tier that replaced worker's 7-role fleet; open for how the crontab is built from schedules/*.json, the overlap check, and what the cron list page is (display only).
@@ -32,13 +35,14 @@ The single-server 1.0 cron tier that replaced worker's 7-role fleet; open for ho
 
 ## How a job becomes a cron
 
-1. On deploy, `.platform/hooks/postdeploy/020_cron.sh` links `/var/www/html` to `/var/app/current`, reads `ENVIRONMENT` from EB config, and runs `php ebs/cron.worker.php`.
-2. `cron.worker.php` reads `schedules/cron.<ENVIRONMENT>.json` and writes root's crontab:
+1. On deploy, `ebs/setup_git_libraries.php` clones `library`, `_underscore`, `dbchanges`, `resources` and `togadesk` (branch `_production` for production, else `_<ENVIRONMENT>`) into the app; a failed clone fails the deploy. **A library fix reaches worker1.5 only on the next worker1.5 deploy.** Clone credentials: [committed secrets](features/committed-secrets-and-log-exposure.md).
+2. On deploy, `.platform/hooks/postdeploy/020_cron.sh` links `/var/www/html` to `/var/app/current`, reads `ENVIRONMENT` from EB config, and runs `php ebs/cron.worker.php`.
+3. `cron.worker.php` reads `schedules/cron.<ENVIRONMENT>.json` and writes root's crontab:
    - `cron` → `php /var/www/html/crons/<cron>`; `url` → `wget -q -O /dev/null http://localhost<url>`; `command` → used as is.
    - `active: 0` rows are skipped.
    - Every job appends to the one log file `/var/log/worker-cron.log` (logrotate `/etc/logrotate.d/worker-cron`).
    - It refuses to install if the file is missing, the JSON is bad, a row has no cron/url/command, or there are 0 active jobs.
-3. Timezone for the installer is `America/Chicago`.
+4. Timezone for the installer is `America/Chicago`.
 
 Schedule files: `cron.alpha/beta/demo/hotfix/sprint/stage/ui/worker.json`. Only `worker` (production) carries the real list. The others hold 1 to 2 rows.
 
@@ -46,6 +50,11 @@ Schedule files: `cron.alpha/beta/demo/hotfix/sprint/stage/ui/worker.json`. Only
 
 `/` renders the schedule files as a list plus a sidebar. **Display only.** Crons never go through it, so a page error does not stop crons. It reads and sorts the schedule files once into `$scheduleSections`. `cron.worker.json` (empty category) shows as section `misc`.
 
+## Operating it
+
+No CloudWatch log group: every job's output is `/var/log/worker-cron.log` on the box; reach it with SSM Session Manager. `Common.CronJobExecutions.instanceId` is `''` for this server. Crons stalled or many 1.0 backlogs at once: [Diagnosing worker1.5 crons that stopped running](workflows/diagnosing-crons-not-running.md).
+
 ## Gotchas
 - **Do not read files that are not in this repo on a 1.0 page.** The page read `schedules/cron.worker.catalog.json` (old worker only). The warning became `exit` and the page died about 474 times an hour (9,387 total), probably from a health check on `/`. Fixed in TRUE-83107 (c903b20).
 - **`App_Misc::getFileList` uses `readdir` and does not sort.** Any loop that keeps state per file must set that state on every file. The old loop set `$jsonContents` only for a non-empty category, so `cron.worker.json` reused the previous file's rows, or warned (and exited) if it came first.
+- **Cron log noise you can ignore:** `PHP Deprecated: ...` lines (the job continues) and `getaddrinfo for instance-data failed` warnings (see [PHP 8.5 compatibility § Gotchas](../library/features/php-8-5-runtime-compatibility.md)). Two open faults show up in the log: the OptimumDesk sync prints a credential row into its exception ([Error capture 1.0 § Gotchas](../library/features/error-capture-1-0.md)), and empty-From emails fail at SES every run ([Email queue § Gotchas](../library/features/email-queue-attachments.md)).

package/knowledge/1.0/apps/worker1.5/features/committed-secrets-and-log-exposure.md

@@ -0,0 +1,37 @@
+---
+title: "worker1.5 — Committed Secrets & Credentials in Logs (debt register)"
+framework: "1.0"
+repo: worker1.5
+project: Worker 1.5
+client: shared
+type: feature
+status: active
+updated: 2026-10-23
+owners: ["bala"]
+files:
+  - worker1.5/ebs/git.json
+  - worker1.5/ebs/setup_git_libraries.php
+  - worker1.5/crons/sync/retailservices/sync_retail_services_client_service.php
+related:
+  - ../architecture.md
+  - ../../worker/architecture.md
+  - ../../../2.0/apps/worker2/features/committed-secrets-and-config-debt.md
+  - ../../library/features/error-capture-1-0.md
+---
+
+Open security debt in worker1.5: a GitHub token committed in `ebs/git.json` and a cron that writes OptimumDesk passwords into the cron log; open before touching deploy cloning, rotating credentials, or reading `/var/log/worker-cron.log`.
+
+## Summary
+Two open risks found 2026-10-23 (TRUE-83121). Neither is fixed. Values are never recorded here.
+
+## How it works
+| Risk | Where | What to do |
+|---|---|---|
+| Plaintext GitHub token for account `agilantgit` committed on `_main` | `ebs/git.json` (used at deploy to clone `library`, `_underscore`, `resources`, `dbchanges`, `togadesk`) | Treat as leaked. Rotate it, then move it to SSM / EB env and read it at deploy. Same debt exists in `worker` (copied from it). |
+| OptimumDesk passwords in the error message | `crons/sync/retailservices/sync_retail_services_client_service.php` | Details and fix: [1.0 error capture § Gotchas](../../library/features/error-capture-1-0.md). Treat any `/var/log/worker-cron.log` or `Logs.Issue` row holding it as sensitive. |
+
+## Gotchas
+- `/var/log/worker-cron.log` is one shared file for every job on the server (see [architecture](../architecture.md)), so one leaking cron exposes the secret to anyone reading cron output.
+
+## Change history
+- 2026-10-23 — Created: GitHub token in `ebs/git.json`, OptimumDesk passwords in the cron log (TRUE-83121). (bala)

package/knowledge/1.0/apps/worker1.5/workflows/diagnosing-crons-not-running.md

@@ -0,0 +1,58 @@
+---
+title: Diagnosing worker1.5 crons that stopped running (single-server 1.0 tier)
+framework: "1.0"
+repo: worker1.5
+project: Worker 1.5
+client: shared
+type: workflow
+status: active
+updated: 2026-10-23
+owners: [bala]
+files:
+  - worker1.5/ebs/cron.worker.php
+  - worker1.5/ebs/setup_git_libraries.php
+  - worker1.5/schedules/cron.worker.json
+  - worker1.5/crons/notifications/infrastructure/send_emails.php
+  - library/app/framework.php
+related:
+  - ../architecture.md
+  - ../../library/features/cron-execution-monitoring.md
+  - ../../worker/workflows/tracing-a-worker-cron-run-in-production.md
+  - ../../library/features/email-queue-attachments.md
+  - ../../../clients/compass-usa/features/mits-sales-order-transmission-alerting.md
+---
+
+Steps to tell "worker1.5 crons are not running" from "a downstream system is failing", find the cause on the box, and recover; open when several 1.0 jobs or backlogs stall at once.
+
+## Summary
+
+Since TRUE-83078 (2026-10-22) every 1.0 cron runs on ONE server: EB env `worker15-production` (toga-production account, us-east-1, AL2023, PHP 8.5). So one bad crontab or library change stops every 1.0 job at once. Symptoms show up as many unrelated alarms (Compass MITS backlog, Prudential SRs not becoming SOs, NetSuite syncs idle, emails not sent, daily reports missing). Check "is the cron running at all" before debugging any single job.
+
+## Steps
+
+1. **Prove the cron is not running (DB, from your desk).**
+   - Legacy `Common.CronJobExecutions`: rows per `job` since the suspected boundary. Zero rows for most jobs = crons exit before check-in (step 3; reading rules in [Cron Execution Monitoring](../../library/features/cron-execution-monitoring.md)). `instanceId` is `''` for worker1.5 rows: host `instance-data` does not resolve on AL2023 ([PHP 8.5 compatibility § Gotchas](../../library/features/php-8-5-runtime-compatibility.md)). Not a cron fault.
+   - Business signals that separate "not running" from "downstream rejecting": legacy `Logs.API` last outbound call per endpoint (e.g. MITS), `Client_Compass.SalesOrders.c_dtTransmittedToCompass`, `Common.EmailsQueued` row count (a queue, should drain to ~0), `Logs.Issue` new events. No new calls + no new Issue events = not running.
+2. **Get on the box.** SSM Session Manager (instance shows Online). There is NO CloudWatch log group for worker15-production. Every job's output goes to `/var/log/worker-cron.log` on the box.
+3. **Check the overlap guard.**
+   - `crontab -l` as root: there must be no `SHELL=` line (cron must use `/bin/sh`). Why: the crontab-shell gotcha in [Cron Execution Monitoring](../../library/features/cron-execution-monitoring.md).
+   - `ps -ef | grep <script>`: any row holding the script path (a parent shell, a `tail -f`) blocks it.
+   - Tell: the same cron run by hand (`php /var/www/html/crons/<path>`) from an interactive shell works. "Most crons dead, only the `cronInitialization(x, y, false)` ones alive" also fits.
+4. **Ship the fix through the right deploy.**
+   - Crontab fix → `worker1.5/ebs/cron.worker.php` + worker1.5 deploy.
+   - A `library` fix reaches worker1.5 only on the NEXT worker1.5 deploy: `ebs/setup_git_libraries.php` clones `library`, `_underscore`, `dbchanges`, `resources`, `togadesk` at branch `_production` during each deploy. Merging library alone changes nothing on the box.
+5. **Recover after crons run again.**
+   - Backlog crons (MITS transmit, SR→SO, email send, NetSuite syncs) drain by themselves. Watch the backlog fall; do not assume.
+   - Fixed-time daily/weekly jobs do NOT catch up. Re-run each missed one by hand (`php /var/www/html/crons/<path>` on the box) and tell the people who expect it.
+
+## Gotchas
+
+- **A script that calls `isProcessRunning()` itself fails half-way.** `crons/notifications/infrastructure/send_emails.php` passes `false` to `cronInitialization()` (so it checks in every run) but calls `isProcessRunning()` again at line 14 and exits. Under the bash shell it checked in every 3 minutes and sent nothing. A fresh `CronJobExecutions` row is not proof the job did its work.
+- **Schedules come from `worker1.5/schedules/cron.worker.json` only.** The old `worker` repo schedule files no longer drive prod.
+
+## Worked example (2026-10-22 16:56 → 2026-10-23 ~10:30 CT, TRUE-83121)
+
+TRUE-83078's installer wrote `SHELL=/bin/bash`. 222 of 234 active crons never checked in. Seen: 16 Compass SOs not sent to MITS (oldest 836 min, MITS returned 200 on every call it got, last call 16:10), all Compass 1.0 workflow crons, NetSuite toga2 syncs, Prudential SR→SO (12 REQs), 141 unsent `Common.EmailsQueued` ticket emails (RUMC, TOW, PC Matic, Apiary, Craftex), and NYC DOE "Send DOE Courier Installation Schedule (7:00 AM)" and "Daily Open and Closed DOE Report (1:00 AM)" missed on 2026-10-23. Fixed by dropping the `SHELL` line (worker1.5 PR #8, `ce0f0b2`). Backlogs drained by ~10:52 CT; missed DOE reports needed a manual re-run.
+
+## Change history
+- 2026-10-23 — Created from the worker1.5 cron outage after the single-server move (TRUE-83121, TRUE-83116, TRUE-83138). (bala)

package/knowledge/1.0/standards/backend-php.md

@@ -5,8 +5,8 @@ project: Library
 client: shared
 type: standard
 status: active
-updated: 2026-10-06
-owners: [jcardinal, rgirish, mhammontree, ajean]
+updated: 2026-10-23
+owners: [jcardinal, rgirish, mhammontree, ajean, bala]
 files: []
 related:
   - ../apps/library/architecture.md
@@ -266,7 +266,7 @@ private static $memoryReserve = null;
 
 ### Any PHP warning terminates the request — how to opt out locally
 
-`App_Error::handleError()` routes **every** PHP warning into `handleException()`, which renders the error page and calls **`exit()`**. (2.0's equivalent *throws*, which a `catch` can absorb — 1.0 does not.) So a warning raised inside code that must not escalate — error capture, a monitoring ping, an IMDS probe where `file_get_contents` warns before returning `false`, a retry loop — **kills the request or cron mid-run while reporting an unrelated error.**
+`App_Error::handleError()` routes **every** PHP warning into `handleException()`, which renders the error page and calls **`exit()`**. Deprecations are the one exception: since library PR #905 (2026-10-22) they are logged once per call site and the code continues ([PHP 8.5 compatibility](../apps/library/features/php-8-5-runtime-compatibility.md)). (2.0's equivalent *throws*, which a `catch` can absorb — 1.0 does not.) So a warning raised inside code that must not escalate — error capture, a monitoring ping, an IMDS probe where `file_get_contents` warns before returning `false`, a retry loop — **kills the request or cron mid-run while reporting an unrelated error.**
 
 Two sanctioned patterns, in order of preference:
 

package/knowledge/1.0/standards/framework-rules.md

@@ -5,15 +5,15 @@ project: Library
 client: shared
 type: standard
 status: active
-updated: 2026-09-16
-owners: [jcardinal, ajean]
+updated: 2026-10-23
+owners: [jcardinal, ajean, bala]
 files: []
 related:
   - ../apps/library/architecture.md
   - ../apps/worker/architecture.md
   - backend-php.md
 ---
-1.0 `App_` framework conventions — class naming, inheritance, model registry, file mapping, dbchanges, PHP 8.x deprecation deaths, controller error handling; open for any 1.0 repo code.
+1.0 `App_` framework conventions — class naming, inheritance, model registry, file mapping, dbchanges, PHP 8.x deprecations (logged) vs warnings (exit), controller error handling; open for any 1.0 repo code.
 
 Applies to all 1.0 repos: `library` (core) and `worker` (app). Loaded on-demand by `/start` when a 1.0 repo is in scope (not always-on) — see rules-scoping in `CONVENTIONS.md`.
 
@@ -103,16 +103,16 @@ Applies regardless of which app repo the feature lives in. Even a column only `w
 
 Never create a `.sql` file inside a 1.0 app repo. The only SQL that belongs there is query strings embedded in PHP — not standalone migration files.
 
-## PHP 8.x deprecations abort the request (audit before running on 8.5)
+## PHP 8.x: deprecations are logged, warnings still abort (audit before running on 8.5)
 
-The 1.0 error handler **terminates** on a PHP deprecation notice (`App_Error::handleError` → `handleException` → `exit`; it does not throw a catchable `ErrorException`), so the request/job dies where the deprecation fires. On PHP 8.5 a single deprecated call in a live path is a hard, uncontainable failure — a surrounding `try/catch` will not save it. See the back-end standard's *"Any PHP warning terminates the request — how to opt out locally"* for the two sanctioned local opt-outs.
+Since library PR #905 (TRUE-83055, `cd91686e`, 2026-10-22) `App_Error::handleError` logs `E_DEPRECATED` / `E_USER_DEPRECATED` once per file:line per request (`PHP Deprecated: <msg> in <file>:<line>` via `error_log`) and the code keeps running. Every other notice and warning still goes `handleException` → `exit` (no catchable `ErrorException`, so a `try/catch` will not save it). See the back-end standard's *"Any PHP warning terminates the request — how to opt out locally"*.
 
-- When porting or running 1.0 code on PHP 8.x, **audit for deprecated-in-8.x calls** in every reachable path before deploy.
-- Known offender: **`curl_close()`** — deprecated as of PHP 8.5, a no-op since PHP 8.0. Remove all `curl_close()` calls; the cURL handle closes automatically. (Design Demo Admin incident — see framework-rules-cases.md.)
-- In `library`, guard deprecated calls by version (`PHP_VERSION_ID`) and use 7.1-valid syntax (`?string $x = null`) — other 1.0 apps still run PHP 7.2. Fixed/open list: [PHP 8.5 compatibility](../apps/library/features/php-8-5-runtime-compatibility.md).
-- Main 8.x break classes in 1.0 apps: `null` into internal string functions (`trim`, `base64_decode`, `stripos` — deprecated 8.1) and reading keys of an empty `fetchRow` / `fetchOne` result (warning since 8.0). Cast `(string)` / default the array at the call site; DB/JSON fields can be null.
-- When the root cause is a library helper (e.g. `App_Date`, `App_Database`), fix it in library, keeping it PHP 7.2 safe — see [PHP 8.5 compatibility](../apps/library/features/php-8-5-runtime-compatibility.md).
-- Do not suppress the deprecation with `@` (see coding-style rules) — remove/replace the deprecated call.
+- Still remove deprecated calls when porting to 8.x: they fill the log and break in a later PHP. Known: **`curl_close()`** (deprecated 8.5, no-op since 8.0).
+- An app deployed with a library build older than PR #905 still exits on a deprecation.
+- In `library`, guard deprecated calls by version (`PHP_VERSION_ID`) and use 7.1-valid syntax (`?string $x = null`): other 1.0 apps still run PHP 7.2. Fixed/open list: [PHP 8.5 compatibility](../apps/library/features/php-8-5-runtime-compatibility.md).
+- Main 8.x break classes in 1.0 apps: reading keys of an empty `fetchRow` / `fetchOne` result (a **warning** since 8.0, still fatal) and `null` into internal string functions (`trim`, `base64_decode`, `stripos`; a deprecation since 8.1, now logged). Default the array / cast `(string)` at the call site; DB/JSON fields can be null.
+- When the root cause is a library helper (e.g. `App_Date`, `App_Database`), fix it in library, keeping it PHP 7.2 safe.
+- Do not suppress with `@` (see coding-style rules); remove or replace the call.
 
 ## Error handling in controllers
 

package/knowledge/2.0/apps/api2/features/request-logging.md

@@ -6,7 +6,7 @@ project: API
 client: shared
 type: feature
 status: active
-updated: 2026-10-20
+updated: 2026-10-23
 owners: ["mhammontree", "dfranks", "bala", "jcardinal", "rgirish"]
 files:
   - api2/Component/Api/V2/V2.php
@@ -67,6 +67,7 @@ Do **not** reach for `_Exception_Business` merely to "make sure the rejection is
 The commit-the-logs / roll-back-the-data contract holds for the **handled** paths — a validation rejection or an exception the controller catches. It did **not** hold for an **uncaught `Error`** observed in production on 2026-08-04 (a `Call to undefined method` raised from an [API payload interceptor](api-payload-interceptors.md) dispatch): the fatal was reported to the error tracker, but **`Logs_Compass.Api` contained zero 5xx rows for the entire day and the failing request appears nowhere in it at all.** Suspected regression in the recent `Error.php` / Issue-Event table normalization work. Practical consequences:
 - **Do not conclude "the request never happened" from an empty log table.** For a fatal, absence of a log row is not evidence of absence of the request. Check the error tracker's per-event detail — that may be the *only* record.
 - **Diagnose from the DATA, not the log.** The data transaction rolls back while a *previous* successful request already committed its own rows, so a fatal on a multi-request write sequence leaves a detectable **orphan / zero-child parent**. Sweep for those instead of grepping logs — see [Compass Order Lifecycle & Data-Integrity Invariants](../../../clients/compass-usa/workflows/order-lifecycle-and-data-integrity.md#detecting-empty-shell-orders-left-by-a-fatal-mid-sequence).
+- **A huge response can OOM in the log write itself.** The api log INSERT inlines the whole JSON `responsePayload` (V2.php ~L2379-2391 → `_Model` save ~L345 → `_Query::setQuery()` L430, which makes a full `strtoupper` copy of the SQL). A multi-MB response → HTTP 500 `Allowed memory size of 268435456 bytes exhausted ... in _underscore/Query.php line 430`, with **no** `Logs_<Client>.Api` row and **no** `Logs.Issue`. Only the caller sees it (Prudential TRUE-83138: 1.0 App error "Failed to JSON decode API response (HTTP 500)" in worker1.5 `worker-cron.log`). Usual cause: an over-deep GET — see [V2 query contract § depth re-expands every child collection](v2-rest-query-contract.md).
 - **Orphan sweeps only find PARTIAL casualties.** An attempt that rolled back completely leaves no row in any table and, with no API log row, no trail at all.
 - **It applies to plain GETs too, and the response body comes back EMPTY.** Re-confirmed 2026-08-27: a `GET /v2/table-views/meta` that throws inside `_Model_Client_TableView::meta()` (a `TableViewJoins` row pointing at a deleted `Core` record) returns **500 with no body at all** — no `error` code, no `messages` — and `Logs_CompassCanada.Api` holds no row for it. **An empty 500 body plus a missing Api log row is a usable fingerprint** for "a fatal was thrown before the response was assembled". See [tableview-joins](../../_underscore/features/tableview-joins.md).
 
@@ -233,6 +234,7 @@ Both a **fallback recovery path** and a **scope diagnostic**. In the TRUE-80179
 - **`Logs_<Client>.Api` is pruned (once TRUE-81902 deploys)** — GET rows after 30 days, writes after 90. A missing older row is not proof the request never happened. See [Logs retention cleanup](../../worker2/features/logs-retention-cleanup.md).
 
 ## Change history
+- 2026-10-23 — A multi-MB response OOMs in the log INSERT (`Query.php` L430) and leaves no Api row or Issue (TRUE-83138). (bala)
 - 2026-10-20 — Gotcha: local `Logs.Api.sourceIp` NOT NULL (blank Core Logs template) vs nullable prod (TRUE-82934). (jcardinal)
 - 2026-10-16 — Gotchas: delegator/staff logins log under the target `userId` (trace a browser by XFF in `requestHeaders`); `dtStamp` is server-local; Network Errors leave no row. (bala)
 - 2026-10-16 — Timing gotcha: `dtStamp` = request start; GETs ~1.1s logged + ~1.5–1.9s unlogged between calls. (rgirish)

package/knowledge/2.0/apps/api2/features/v2-rest-query-contract.md

@@ -6,7 +6,7 @@ project: API
 client: shared
 type: feature
 status: active
-updated: 2026-10-19
+updated: 2026-10-23
 owners: [tcox, bala, apeterson, jcardinal, rgirish]
 files:
   - api2/Component/Api/V2/V2.php
@@ -186,6 +186,24 @@ The same route's **list GET was ~1s**, so this is the **write read-back**, not t
   `api2`/`_underscore` is an **open follow-up** (needs a `cto` review before touching the shared
   serializer). Lowering caller depth is a mitigation, not the fix.
 
+### ⚠ `depth` re-expands EVERY child collection for EVERY row — one fat collection blows the 256 MB limit (OPEN)
+
+`getFullModelData()` pulls each inherent child collection **in full** (V2.php ~L7514-7527) for every
+row it returns, at every level. The guard meant to stop re-expanding a parent never fires: it is
+written as `$skipModelsOfInherentChildren[$modelName][$primaryKeyId]` (~L7285, keyed by the row's
+OWN model + id) but read as `[$childModelName][$parentPrimaryKeyId]` (~L7483). So
+`GET /contacts/{uuid}/contact-addresses?depth=3` returns every address, and each address re-expands
+its contact's addresses, emails, phones, vendors, locations and opt-ins again.
+
+- Prudential (TRUE-83138, 2026-10-23): contact 54 had **2,530** ContactAddresses → HTTP 500
+  `Allowed memory size of 268435456 bytes exhausted ... in _underscore/Query.php line 430`. The
+  read itself is many small per-row queries; memory ran out writing the API log row (see
+  [request logging](request-logging.md), "uncaught PHP fatal" section).
+- **Caller rule:** on a route whose parent has a large child collection, send `depth: 1` (or 2) and
+  only the `fields` you read.
+- **Engine fix is OPEN:** correct the skip-guard key (shared serializer → `cto` review first).
+  Raising api2 `memory_limit` (done 2026-10-23) is a stopgap.
+
 ### Measured cost of a list GET — it is per ROW × per NAMED CALCULATED FIELD × per FK hop
 
 One `GET /v2/purchase-order-items` over the **same 1,238 rows** (NYCHH, `sandbox-client`), one curl
@@ -658,6 +676,7 @@ If you need the behaviour, loop uuids client-side. Point 3 is the durable, non-o
   [V2 request logging](request-logging.md).
 
 ## Change history
+- 2026-10-23 — `depth` re-expands every child collection per row (skip-guard key mismatch) → 256 MB OOM on a 2,530-address contact (TRUE-83138). (bala)
 - 2026-10-19 — `limit` on a list GET = 404 `EV-14`, use `recordsPerPage` (TRUE-82915). (jcardinal)
 - 2026-10-19 — Self-join `where` needs an `ojoin` alias (`Locations@p`); `_` field ACL bypass does not apply cross-client (TRUE-82911). (jcardinal)
 - 2026-10-18 — Grouped queries: stored root group key now selected + ACL-gated; known gaps (joined key, total count, `ORDER BY uuid` under ONLY_FULL_GROUP_BY) (TRUE-82876). (jcardinal)

package/knowledge/2.0/apps/worker2/features/oneuptime-worker2-monitoring.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-10-22
+updated: 2026-10-23
 owners: ["jcardinal", "mhammontree", "bala", "rgirish"]
 files:
   - worker2/Worker/Monitor/Compass.php
@@ -96,7 +96,8 @@ Each measures one Compass/Office Depot (or Compass Canada) integration metric an
   `c_dtTransmittedToCompass` NULL (not yet transmitted to MITS). Alarms on **either** `pendingCount > 10`
   **OR** `oldestPendingAgeMinutes > 180` (3h), so a single order stuck too long pages even under the
   count threshold — this catches the 1.0 cron's
-  [poison-order starvation](../../../clients/compass-usa/features/mits-sales-order-transmission-alerting.md).
+  [poison-order starvation](../../../clients/compass-usa/features/mits-sales-order-transmission-alerting.md)
+  **and** the cron not running at all (worker1.5 outage 2026-10-22/23, TRUE-83116 — check that first).
   **Age is measured from the LATER of the two approval decisions — `GREATEST(stage-1, stage-2)
   dtDecision`** — because an order is eligible to transmit only once **both** stages are approved.
   Measuring from **stage-2 alone is wrong** (source of a false HIGH): Compass **VIP auto-approve stamps
@@ -623,6 +624,7 @@ production deploy.**
 - [OneUptime 1.0 worker uptime monitoring](../../../1.0/apps/worker/features/oneuptime-worker-uptime-monitoring.md) — the 1.0 push-heartbeat predecessor
 
 ## Change history
+- 2026-10-23 — `CompassSalesOrderTransmission` also fires when the 1.0 transmit cron is not running (worker1.5 outage, TRUE-83116). (bala)
 - 2026-10-22 — Added `CompassCanadaSalesOrderTransmission` (cron 98); its first two prod runs hit the undefined-method gotcha (row inserted before deploy). (bala)
 - 2026-10-16 — Added same-class merge-conflict gotcha; linked NYCHH `TransferOrderEmailsOwed` (TRUE-82758). (mhammontree)
 - 2026-10-15 — Absence rule must exceed the longest gap between pushes for non-`*/5` monitors; linked NYCHH `AssetTagBackfill` (twice daily, 960 min). (mhammontree)

package/knowledge/INDEX.md

@@ -7,7 +7,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 - **library** (Library) _(framework core)_ — 41 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
 - **worker** (Worker) — 43 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
 - **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
-- **worker1.5** (Worker 1.5) — 2 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
+- **worker1.5** (Worker 1.5) — 4 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
 - **togadesk** (TOGa Desk) — 18 doc(s) → [1.0/apps/togadesk/INDEX.md](1.0/apps/togadesk/INDEX.md)
 - **togaview** (TOGa View) — 9 doc(s) → [1.0/apps/togaview/INDEX.md](1.0/apps/togaview/INDEX.md)
 - **webhook** (Webhook) — 1 doc(s) → [1.0/apps/webhook/INDEX.md](1.0/apps/webhook/INDEX.md)

package/knowledge/clients/compass-usa/features/mits-sales-order-transmission-alerting.md

@@ -5,11 +5,11 @@ project: _Underscore
 client: compass-usa
 type: client-feature
 status: active
-updated: 2026-10-22
+updated: 2026-10-23
 owners: ["dfranks", "bala", "jcardinal"]
 files:
-  - worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php
-  - worker/crons/toga2/compasscanada/workflow/1_transmit_compassCanada_sales_orders_to_mits.php
+  - worker1.5/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php
+  - worker1.5/crons/toga2/compasscanada/workflow/1_transmit_compassCanada_sales_orders_to_mits.php
   - worker2/Worker/Infrastructure/Errors.php
   - tools/mvc/errors/post.php
   - _underscore/Model/Core/Logs/Issue.php
@@ -25,6 +25,7 @@ related:
   - ../../../2.0/apps/_underscore/features/error-reporting-issue-event.md
   - ../../../1.0/apps/library/features/error-capture-1-0.md
   - ../../../1.0/apps/tools/features/errors-curation-console.md
+  - ../../../1.0/apps/worker1.5/workflows/diagnosing-crons-not-running.md
 ---
 How a `1_transmit_..._sales_orders_to_mits.php` failure reaches a human — USA rejections route through Logs Issue/Event, Canada still emails in-cron; open for "add someone to MITS error emails," a stuck transmit backlog, or the poison-order starvation.
 
@@ -49,7 +50,7 @@ Consequence for anyone handed "add <person> to the MITS error emails": the answe
 
 So the USA cron is a **hybrid**: its rejection path is issue-routed, but it still holds one hardcoded email for the no-items case. Only the rejection branch was migrated.
 
-Both crons are live — `worker/schedules/cron.worker.sync.json` schedules `toga2/compass/workflow/1_...` and `toga2/compasscanada/workflow/1_...`. (Per the [ODP pipeline doc's](../workflows/odp-order-pipeline-to-netsuite.md) rule, a grep hit under `crons/toga2/**` is not proof a script runs — these two were verified against the schedule.)
+Both crons are live on the single worker1.5 server (since TRUE-83078, 2026-10-22) — `worker1.5/schedules/cron.worker.json` schedules `toga2/compass/workflow/1_...` (`*/5`) and `toga2/compasscanada/workflow/1_...` (`*/10`). The old `worker` repo schedules no longer run in prod. (Per the [ODP pipeline doc's](../workflows/odp-order-pipeline-to-netsuite.md) rule, a grep hit under `crons/toga2/**` is not proof a script runs — these two were verified against the schedule.)
 
 **Current recipients on every in-cron email above:** `devteam@togatech.com` + `vburks@togatech.com` (business stakeholder, added 2026-08-31).
 
@@ -68,11 +69,14 @@ Both crons are live — `worker/schedules/cron.worker.sync.json` schedules `toga
 
 Configuration is **never retroactive**: `shouldNotify()` fires on the first occurrence, on escalation, or when the backed-off reminder interval elapses, and returns false for a window with zero events. The first email lands on the **next live rejection**, not on existing history.
 
+### First check on a backlog alarm: is the cron running at all?
+Before reading rejections, rule out a dead cron. 2026-10-22/23 (TRUE-83116): the `CompassSalesOrderTransmission` alarm fired (16 pending, oldest 836 min) with **zero** new `Logs.Issue` 24 events and MITS returning 200 on every call — the last `Logs.API` call to MITS was 16:10 the day before. The transmit cron was not running at all: the worker1.5 crontab shell made the overlap guard exit every run. Signs: no new Issue 24 events + stale last `Logs.API` call + `c_dtTransmittedToCompass` not moving. Steps: [Diagnosing worker1.5 crons that stopped running](../../../1.0/apps/worker1.5/workflows/diagnosing-crons-not-running.md).
+
 ### Why the backlog happens — poison-order starvation (root cause)
 The alert almost never fires on **volume**. It fires because one bad order **aborts the whole run** and every un-processed order behind it starves. Diagnosed 2026-08-31 against a live backlog (0 new, several >24h, one ~20 days old).
 - The cron loops approved-but-untransmitted orders and `throw`s on any MITS rejection (`App_Exception_Business` on a non-200, plain `Exception` on "No Items Found"). A throw **ends the entire run**, so every order not yet reached that run stays untransmitted.
 - The driver query uses **`ORDER BY RAND()`**, so *which* orders get processed before the bad one is random each run. A stuck "poison" order therefore blocks a **different** set of good orders every run — good orders can wait days behind one bad one.
-- Throughput ceiling: the cron sleeps **30s per order** (86400s time limit), scheduled every 3 min but overlap-guarded, so it is effectively one long-running instance — ~110 orders/hr.
+- Throughput ceiling: the cron sleeps **30s per order** (86400s time limit), scheduled every 5 min but overlap-guarded, so it is effectively one long-running instance — ~110 orders/hr.
 
 **Two rejection signatures seen in prod** (Logs.Issue id 24, issueKey `COMPASS_SALES_ORDER_MITS_TRANSMIT_REJECTED`, ~2329 occurrences since 2026-08-03):
 1. **HTTP 400 `The AssetType field is required`** — a line item's catalog Item has `assetTypeId = NULL`, so the payload `assetType` is null and MITS rejects the **whole order**. This was the active blocker (e.g. item `4U555A-S`; ~10 distinct NULL-assetType items blocked 9 of 11 pending orders).
@@ -102,6 +106,7 @@ The worker2 OneUptime monitor now catches this starvation directly: `CompassSale
 - As of 2026-08-04 prod `Logs.IssueEmailAddress` is empty, so this is the **first business-routed issue on the platform**: `processBusinessIssue()` will execute in production for the first time when this issue next fires. Watch that run rather than assuming it works.
 
 ## Change history
+- 2026-10-23 — Crons now on worker1.5 (`*/5` / `*/10`); backlog alarm can mean the cron is not running at all, check that first (TRUE-83116). (bala)
 - 2026-10-06 — Order query no longer joins `Users` on `contactId` (duplicate users caused double POST + false 403 `SalesOrder:Exists` alerts); noted `Logs.API` and the intentional kit config line. (bala)
 
 ## Related

package/knowledge/clients/compass-usa/features/odp-edi-file-retention-and-retry.md

@@ -6,7 +6,7 @@ project: Worker
 client: compass-usa
 type: client-feature
 status: active
-updated: 2026-09-16
+updated: 2026-10-23
 owners: ["bala"]
 files:
   - worker/crons/toga2/compass/workflow/3a_import_office_depot_purchase_orders.php

package/knowledge/clients/compass-usa/features/oneuptime-office-depot-inbound-files-monitor.md

@@ -6,7 +6,7 @@ project: Worker
 client: compass-usa
 type: client-feature
 status: active
-updated: 2026-10-13
+updated: 2026-10-23
 owners: ["bala"]
 files:
   - worker2/Worker/Monitor/Compass.php
@@ -43,12 +43,14 @@ Payload: `status`, `alarm` (`OK`/`DEGRADED`/`HIGH`), `reason`, `minutesSinceLast
 First live run 2026-10-13: last file 20 min ago, 214 in 24 h, 0 of 20 stuck, `alarm: OK`.
 
 ## Gotchas
-- **⚠ There is no `status: "error"` push.** No try/catch (developer standard), so an AWS failure (role, CloudWatch, S3) throws, nothing is pushed, and OneUptime's **"not received in 15 min"** criterion turns it red. A red monitor with no recent body = the checker died, not a backlog; read the worker2 `WorkerJobs` row for the exception.
+- **⚠ There is no `status: "error"` push.** No try/catch (developer standard), so an AWS failure (role, CloudWatch, S3) throws, nothing is pushed, and OneUptime's **"not received in 15 min"** criterion turns it red. A red monitor with no recent body = the checker died, not a backlog; read prod `Core.WorkerJobs.output` for the run output or exception.
+- **It also catches "1.0 crons not running".** The mover is a worker1.5 cron, so a stopped cron host shows as files stuck in `processed/`. Alarmed correctly on 2026-10-22/23 when worker1.5's crontab shell stopped most crons (TRUE-83121, [pipeline gotcha](../workflows/odp-order-pipeline-to-netsuite.md)).
 - **Not covered:** weekend and holiday silence never alarms; only the 20 newest files are checked, so an old file stuck behind 20 good ones is missed.
 - Reads the **prod** agreement stream only. The OfficeDepotTest agreement also writes to `/agilant-as2` and is ignored.
 - The OneUptime import JSON is **not** in git. It was removed from the PR before merge; import JSONs are handed over outside the repo ([runbook](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md#provisioning-a-monitor-runbook)).
 
 ## Change history
+- 2026-10-23 — Confirmed it catches a full 1.0 cron stop (worker1.5, TRUE-83121); run output is in `Core.WorkerJobs.output`. (bala)
 - 2026-10-13 — Created: CloudWatch AS2-Inbound + S3 existence check, business-hours silence alarm; added after the 2026-10-10 Lambda outage left `OfficeDepotEdiImportQueue` green (TRUE-82728). (bala)
 
 ## Related

package/knowledge/clients/compass-usa/profile.md

@@ -15,7 +15,6 @@ apps:
   - worker1.5
   - worker2
   - dbchanges2
-  - library
   - test
   - talos-backend
   - tools
@@ -23,7 +22,7 @@ project: _Underscore
 client: compass-usa
 type: profile
 status: active
-updated: 2026-10-22
+updated: 2026-10-23
 owners: [jcardinal, bala, tcox, apeterson, dfranks, akhokhani, ajean, rgirish]
 files: []
 related:

package/knowledge/clients/compass-usa/workflows/odp-edi-import-recovery.md

@@ -6,7 +6,7 @@ project: Worker
 client: compass-usa
 type: workflow
 status: active
-updated: 2026-10-13
+updated: 2026-10-23
 owners: ["dfranks", "bala"]
 files:
   - worker/crons/toga2/compass/workflow/3a_import_office_depot_purchase_orders.php
@@ -21,6 +21,7 @@ related:
   - ../../../2.0/apps/api2/features/request-logging.md
   - clients/compass-usa/profile.md
   - clients/compass-usa/features/oneuptime-office-depot-inbound-files-monitor.md
+  - ../../../1.0/apps/worker1.5/workflows/diagnosing-crons-not-running.md
 ---
 Runbook to recover Compass ODP EDI 850 imports: re-drop an x12 that cron 3a deleted after a failure, move AS2 files stuck in `processed/` when nothing reaches 3a, or build the order by SQL when 3a cannot run; open when ODP orders are missing in Supply or never reached NetSuite.
 
@@ -82,7 +83,19 @@ The files reach S3 but never get moved to `OfficeDepot/`, so 3a has nothing to i
 3. The AS2 inbound log group (`AS2-Inbound…`) — confirms ODP is still sending.
 4. If re-saving the S3 event notification fails with `Unable to validate the following destination configurations`, S3 can no longer invoke the Lambda (Lambda suspended or its permission gone).
 
-**Recover:** move the stuck `processed/*.net` files, **filtered by date window**, to the root of `OfficeDepot/` with the same name. Never move the 2024 test files in `processed/`. 3a skips POs that already exist, so replaying a file that was already imported is safe. The stop-gap cron `move_office_depot_as2_files_to_import_folder.php` does this every 5 minutes (live since 2026-10-12; the Lambda is still suspended as of 2026-10-13); turn it off once AWS restores the Lambda.
+**Recover:** move the stuck `processed/*.net` files, **filtered by date window**, to the root of `OfficeDepot/` with the same name. Never move the 2024 test files in `processed/`. 3a skips POs that already exist, so replaying a file that was already imported is safe. The stop-gap cron `move_office_depot_as2_files_to_import_folder.php` does this every 5 minutes (live since 2026-10-12, on worker1.5 since 2026-10-22; the Lambda is still broken as of 2026-10-23); turn it off once AWS restores the Lambda.
+
+**If the mover itself has no `Common.CronJobExecutions` rows**, the crons are not running at all (2026-10-22/23: worker1.5 crontab shell). Follow [Diagnosing worker1.5 crons that stopped running](../../../1.0/apps/worker1.5/workflows/diagnosing-crons-not-running.md); the backlog clears itself once 3a and the mover run.
+
+### After a long cron stop — deadlock rollback (PO missing, SO line present)
+When every cron restarts at once, 3a's PO save can deadlock (MySQL **1213**) with cron 2 (`2_transmit_mits_purchase_orders_to_vendors.php`) catching up its backlog. The PO is rolled back, but the file is still logged in `Logs.FileLog` and deleted from S3. Result: an ODP SalesOrder **with its line but no ODP PO**.
+
+Recover with Steps 1-2: rebuild the 850 from `Logs.FileLog.fileData` and drop it in the root of `s3://agilant-as2/OfficeDepot/`. It is safe:
+- 3a reuses the ODP SO by number + `customerId`, and matches the existing SO line by part number (no duplicate line).
+- The over-quantity guard does not count the rolled-back PO.
+- A second run of the same file skips, because the PO now exists.
+
+Verified 2026-10-23: ODP PO `42278391-5910`, ODP SO `486472538001` (id 128209), `Logs.FileLog` id 905447 → PO id 119085 created, linked to SO 128209, accept 855 sent.
 
 ### Last resort — manual import by SQL (3a cannot run)
 Prefer re-dropping the file (Steps 1-4): 3a also writes the `Logs_Compass` `Api` / `Record` / `RecordField` audit rows and an SQS event, which plain SQL does not. If the order must exist now, write exactly what 3a's api2 calls write for one SA/MR 850.
@@ -114,6 +127,7 @@ After the SQL import:
 - A **fully** rolled-back import leaves no row anywhere and no API log row, so it will not show up in an orphan sweep at all; only the error tracker's per-event detail can enumerate those.
 
 ## Change history
+- 2026-10-23 — Added the deadlock-rollback case after the worker1.5 cron stop (PO 42278391-5910 replayed from FileLog) and the "mover has no CronJobExecutions rows" check (TRUE-83121). (bala)
 - 2026-10-13 — Added the manual-SQL-import section (exact rows 3a's api2 calls write, hooks that do not run for apiId 1); linked the inbound files monitor; Lambda still suspended, stop-gap cron live. (bala)
 - 2026-10-12 — Added the "AS2 files stuck in `processed/`" case (Lambda `rename_edi_files` stopped 2026-10-10, 52 POs); corrected the inbound S3 filename to `<uuid>.<msgid>.net`. (bala)
 

package/knowledge/clients/compass-usa/workflows/odp-order-pipeline-to-netsuite.md

@@ -6,7 +6,7 @@ project: Worker
 client: compass-usa
 type: workflow
 status: active
-updated: 2026-10-21
+updated: 2026-10-23
 owners: ["rgirish", "bala", "dfranks", "jcardinal", "mhammontree"]
 files:
   - worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php
@@ -36,12 +36,16 @@ related:
   - 1.0/apps/library/features/app-cloud-aws-credentials.md
   - clients/compass-usa/features/oneuptime-office-depot-inbound-files-monitor.md
   - 2.0/apps/dbchanges2/features/netsuite-id-unique-keys.md
+  - 1.0/apps/worker1.5/workflows/diagnosing-crons-not-running.md
+  - 1.0/apps/worker1.5/architecture.md
 ---
 The end-to-end Compass ODP order → NetSuite pipeline through the numbered 1.0 worker crons, plus the "this order never reached NetSuite" diagnostic method; open for any ODP-to-NetSuite tracing or cron-mechanics question.
 
 ## Summary
 The end-to-end **Compass Office Depot (ODP) order → NetSuite** pipeline as it actually runs through the 1.0 `worker` crons under `worker/crons/toga2/compass/`, plus the **diagnostic method** for "this Compass order never reached NetSuite" tickets. The *cron mechanics* companion to the 2.0 [Compass Order Lifecycle & Data-Integrity Invariants](order-lifecycle-and-data-integrity.md) (raw-data shape / repair side). All facts verified against the live `prod` / `Client_Compass` schema and the cron source. The crons are `App_` (1.0) framework code; the data lives in the 2.0 `Client_Compass` / `Logs_Compass` schema.
 
+**Since 2026-10-22 (TRUE-83078) these crons run on [worker1.5](../../../1.0/apps/worker1.5/architecture.md)**, same `crons/toga2/compass/...` paths; the schedule is `worker1.5/schedules/cron.worker.json` (not `worker/schedules/cron.worker.sync.json`). The old `worker` fleet stopped 2026-10-22 16:56 Central.
+
 The pipeline (customerId / vendorId, in run order):
 ```
 Commerce → Compass SalesOrder            (SalesOrders.customerId = 2)
@@ -63,7 +67,7 @@ ODP SalesOrder (customerId = 1)
 1. **`workflow/1_transmit_compass_sales_orders_to_mits.php`** — transmits the Compass `SalesOrder` (customerId=2) to MITS via API.
 2. **`workflow/2_transmit_mits_purchase_orders_to_vendors.php`** — transmits the MITS-created PO to ODP. ODP integration is **cXML** (auth audience `"Office Depot (Cxml)"`); also supports an **email fallback** path per the vendor's integration type. Sets `PurchaseOrders.dtSubmitted` when transmitted (~2 min after PO creation). Item-less-PO gotcha + CXML-vs-EMAIL routing detail: [MITS PO Transmission to Vendors](../features/mits-po-transmission-to-vendors.md).
 3. **`workflow/3a_import_office_depot_purchase_orders.php`** — pulls the ODP **850** from S3 bucket `agilant-as2`, prefix `OfficeDepot/` (excluding `OUTBOX/` and `SENT/`; `:46` S3 read, `:54-58` prefix filters) and creates the downstream **ODP SalesOrder** (customerId=1 = Office Depot). **Deletes the S3 object after processing** — so an absent S3 object is expected once ingested, not evidence of a miss.
-   - **⚠ Corrected 2026-08-28 — the delete is now CONDITIONAL.** 3a used to delete **unconditionally**, so a failed import left nothing to retry and **silently destroyed the order**. It now deletes only when **every PO in the file imported end to end**; otherwise the file stays in `OfficeDepot/` and the next `*/5` run retries it. Which failures count, the reject-855 / error-email / file-log guards, and the retries-forever limitation: [ODP EDI File Retention & Import Retry](../features/odp-edi-file-retention-and-retry.md). **Not deployed as of 2026-08-28** — production still deletes unconditionally, so treat any pre-deploy incident as a lost file.
+   - **⚠ Corrected 2026-08-28 — the delete is now CONDITIONAL.** 3a used to delete **unconditionally**, so a failed import left nothing to retry and **silently destroyed the order**. It now deletes only when **every PO in the file imported end to end**; otherwise the file stays in `OfficeDepot/` and the next `*/5` run retries it. Which failures count, the reject-855 / error-email / file-log guards, and the retries-forever limitation: [ODP EDI File Retention & Import Retry](../features/odp-edi-file-retention-and-retry.md). **Still not deployed:** worker1.5 `_main` 3a deletes unconditionally (verified 2026-10-23), so treat any failed import as a lost file.
 4. **`workflow/4_transmit_office_depot_po_acknowledgements.php`** (every 15 min) sends the **accept** X12 **855** for imported ODP `PurchaseOrders` with `dtAcknowledged IS NULL`, then stamps `dtAcknowledged`. It delegates 855 construction to `App_Edi::buildOfficeDepotPurchaseOrderAcknowledgement855()` (accept mode). A **reject** 855 is instead sent **inline from cron 3a** when the over-quantity guard rejects a PO (a rejected PO is never persisted, so cron 4 can never see it). Cron 3a also runs the over-quantity guard (`officeDepotPurchaseOrderExceedsCompassDemand()`) after the same-PO-number de-dupe and before any SO/PO creation, rejecting a whole PO when ODP re-sends an already-processed Compass SO line under a **new** PO number. See [ODP EDI 855 Acknowledgement + Over-Quantity Guard](../features/odp-edi-855-acknowledgement-and-overquantity-guard.md).
 5. **`workflow/5_create_netsuite_sales_orders_from_office_depot_purchase_orders.php`** — scheduled **every 10 minutes** (`*/10`, verified against `worker/schedules/` on 2026-08-04; this doc previously said "hourly"); picks up ODP SalesOrders (`OfficeDepotSalesOrders.customerId = 1 AND c_dtTransmittedToNetsuite IS NULL`), creates the SO in NetSuite, and writes back `c_dtTransmittedToNetsuite` + `c_netsuiteInternalSalesOrderId` **onto the ODP SalesOrder (customerId=1), never the Compass SO (customerId=2)**. Before sending it checks NetSuite for the order by `externalId` so a lost stamp never creates a duplicate: [duplicate-proof send](#cron-5-duplicate-proof-send--externalid--lookup). Exclusion filters (an order legitimately waiting is not a bug):
    - `CompassSalesOrders.number NOT LIKE 'MA%'` — MA orders excluded. **They are keyed into NetSuite by hand** (~20 min each), and the worker2 `Monitor/Compass/OfficeDepotNetsuiteIntegration` monitor carries the **same** `NOT LIKE 'MA%'` clause — so MA orders are invisible to both. Coverage is [cron 7's exception report](../../../1.0/apps/worker/features/compass-ma-sales-order-exception-report.md) plus, since 2026-09-08, the dedicated [MA refresh-order monitor](../features/oneuptime-ma-refresh-order-monitor.md).
@@ -83,7 +87,7 @@ AWS account toga-legacy (`502614707982`), `us-west-2`:
 - **Not in the live path:** Transfer Family workflow `w-f8951fd4252a5b890` "OfficeDepot" (a COPY step to `OfficeDepot/`) is attached to the server OnUpload but has **never run** (no executions in a year of server logs). Do not debug it.
 - Agreement `a-f1a87e1edc5c442fa` "OfficeDepotTest" also writes to `/agilant-as2`. Canon AS2 uses a separate bucket, `agilant-canon-as2`.
 - `processed/` also holds old **2024 test `.net` files**. Never move them; any bulk move must filter by date.
-- **Stop-gap cron (2026-10-12):** `workflow/move_office_depot_as2_files_to_import_folder.php` (`*/5`, scheduled right after 3a) does the Lambda's job: moves `*.net` from `processed/` to `OfficeDepot/`, only files modified after `2026-10-10 12:30 UTC` and older than 120 s, and throws if the S3 list returns `null`. Committed on `worker` `_production` (`b3f57866`) and live; it authenticates through the `WorkloadsRuntime` role ([AWS credentials](../../../1.0/apps/library/features/app-cloud-aws-credentials.md)), first clean run 2026-10-12 17:00 CDT after the library deploy. **The Lambda is still suspended by AWS as of 2026-10-13** (last log 2026-10-10 12:29 UTC). **Turn the cron off (`"active": 0`) once AWS restores the Lambda.**
+- **Stop-gap cron (2026-10-12):** `workflow/move_office_depot_as2_files_to_import_folder.php` (`*/5`, scheduled right after 3a) does the Lambda's job: moves `*.net` from `processed/` to `OfficeDepot/`, only files modified after `2026-10-10 12:30 UTC` and older than 120 s, and throws if the S3 list returns `null`. Committed on `worker` `_production` (`b3f57866`) and live; it authenticates through the `WorkloadsRuntime` role ([AWS credentials](../../../1.0/apps/library/features/app-cloud-aws-credentials.md)), first clean run 2026-10-12 17:00 CDT after the library deploy. **The Lambda is still broken as of 2026-10-23** (last log 2026-10-10 12:29 UTC); the mover now runs from worker1.5. **Turn the cron off (`"active": 0`) once AWS restores the Lambda.**
 - **Monitored by** [`Monitor/Compass/OfficeDepotInboundFiles`](../features/oneuptime-office-depot-inbound-files-monitor.md) (cron 93): alarms when a received file is not moved out of `processed/`, or on business-hours silence. `OfficeDepotEdiImportQueue` cannot see this step.
 - Diagnosing and recovering stuck files: [Recovering a lost ODP EDI import](odp-edi-import-recovery.md#other-case--as2-files-stuck-in-processed-3as-inbox-is-empty).
 
@@ -197,6 +201,7 @@ The NetSuite writeback lands on the **downstream ODP SalesOrder (customerId=1)**
 - **⚠ `Client_Compass.SalesOrders.c_netsuiteInternalSalesOrderId` can never be UNIQUE.** The Compass order and its Office Depot order share one NetSuite SO id by design (this workflow's cron 5 stamp and the NetSuite SO sync write the same id). Every NetSuite-id key migration skips this column on Compass — see [NetSuite id UNIQUE keys](../../../2.0/apps/dbchanges2/features/netsuite-id-unique-keys.md). Do not dedupe Compass SOs by NetSuite id.
 - **⚠ Cron 3a linked every Compass PO line to the LAST ODP SO line (2026-04-28 → 2026-10-06; fixed `worker` `02cc2473`).** Commit `afb520c1` left a scalar `$officeDepotSalesOrderItemUuid` that the customer-PO link loop reused, so all `PurchaseOrderItems_SalesOrderItems` rows for a PO pointed at the last ODP SO line. The recursive IF walk then broke at the ODP→Compass tier and Compass orders got no fulfillment or tracking (proven on SA130861 / SA130762). Fix: link via `$lookupSalesOrderItemUuidFromPurchaseOrderLine[lineNumber]`; repeated vendor part numbers pair one-to-one (`array_shift` on the lookup). Zero bad links on ~440 orders since 2026-10-07. History is repaired by [retrofix2](../../../1.0/apps/test/features/compass-retrofix2-sql-generator.md) (bulk) or [fix_item_fulfillment_chain](../../../1.0/apps/test/features/compass-fix-item-fulfillment-chain.md) (one order). **Since 3a resolves 850 lines through ODP `VendorItems`, the ODP SO line is often the catalog-2 twin of the Compass item — match these tiers by part number, not `itemId`.**
 - **⚠ 2026-10-10 → 10-12: ODP POs stopped arriving because the Lambda stopped, not 3a.** An attacker used a leaked long-lived IAM key; AWS suspended services including Lambda, so `rename_edi_files` stopped at **07:29 CDT Sat 2026-10-10** and **52 POs** piled up in `processed/`. 67 `.net` files (850s + 997s) were moved by hand on 2026-10-12; 3a created all 52 POs (0 rejects) and cron 5 sent them to NetSuite. The leaked key was deleted the same day; 3a and the move cron now run on the `WorkloadsRuntime` role ([AWS credentials](../../../1.0/apps/library/features/app-cloud-aws-credentials.md)). All stuck POs imported. The `OfficeDepot/` backlog monitor stayed green throughout (its folder was empty); the [inbound files monitor](../features/oneuptime-office-depot-inbound-files-monitor.md) now covers that gap. Symptom of a dead mover: files pile up in `processed/`, `OfficeDepot/` stays empty, no new ODP SalesOrders.
+- **⚠ 2026-10-22 16:45 → 10-23 10:40 Central: no ODP 850 imported because worker1.5 crons were not running (TRUE-83121).** worker1.5's crontab set `SHELL=/bin/bash`, so the already-running guard stopped 3a, the mover, 5 and most other crons before check-in ([diagnosis steps](../../../1.0/apps/worker1.5/workflows/diagnosing-crons-not-running.md)). The [inbound files monitor](../features/oneuptime-office-depot-inbound-files-monitor.md) alarmed correctly (files stuck in `processed/`). Restarting every cron at once caused one 3a/cron 2 deadlock; recovery in [ODP EDI import recovery](odp-edi-import-recovery.md#after-a-long-cron-stop--deadlock-rollback-po-missing-so-line-present).
 - **⚠ `edi/1_download_edi_s3_create_po_toga.php` is DEAD CODE (correction 2026-08-04).** It appears in **no** schedule file. The scheduled entry *named* "Download EDI From S3 Files & Create PO - EDI Office Depot" (`*/5 * * * *`, `worker/schedules/cron.worker.sync.json` `:254-259`) actually points at **`toga2/compass/workflow/3a_import_office_depot_purchase_orders.php`**. The two scripts have **different idempotency and different API call shapes** — `edi/1`'s create branch POSTs the SO with *nested* items and a `customer.name`, while **3a POSTs the header alone (uuids) and then POSTs the items in a separate request**. Reasoning from `edi/1` gives wrong answers. **General rule: never infer which script runs from its filename or folder** — verify against `worker/schedules/cron.*.json`, and note the schedule entry's **name can be actively misleading**. The folders `compass/edi/`, `compass/DOA_edi_odp_flow/`, and `compass/DOA_Scripts/` are **legacy / inactive**; `compass/edi/3_send_edi_855(reject).php` is an abandoned sketch using **non-standard** reject codes (`BAK01=A1` / `BAK02=RE`) — **do not** follow it.
 - **⚠ The `OfficeDepot/` listing makes 3a runs overlap (NOT fixed).** An instrumented prod run 2026-08-25 showed the S3 listing returning **138,472 objects** and taking **32 seconds** before any PO was processed, each PO then ~40 seconds. A file-bearing run blows past the `*/5` schedule and the following runs are **silently skipped** by `App_Framework::exitIfProcessRunning()` — leaving **no `CronJobExecutions` row at all**, which reads as "the cron never fired". The object count is driven by `OfficeDepot/SENT/` and `OfficeDepot/OUTBOX/` accumulating: 3a skips those prefixes when *processing* but still *lists* them. Fix candidates (separate work): prune `SENT/`, or list with a tighter prefix. See [Tracing a worker cron run in production](../../1.0/apps/worker/workflows/tracing-a-worker-cron-run-in-production.md).
 - **⚠ `REF~QC` — not `BEG` — is the value to search on for the ODP SalesOrder.** Searching `PurchaseOrders` / `SalesOrders` for the `BEG` number finds nothing for the SO and wastes time.
@@ -233,6 +238,7 @@ The NetSuite writeback lands on the **downstream ODP SalesOrder (customerId=1)**
 - **Editing cron recipient lists — a grep hit is not a live hit.** Email recipients are hardcoded in the cron scripts, and **several dormant copies of the same script exist** under `crons/toga2/compass/edi/` and `crons/toga2/compass/DOA_edi_odp_flow/`. Before changing a recipient list, cross-check the file against `worker/schedules/*.json` for a referencing entry whose `active` is not `0`. Only the scheduled copies matter.
 
 ## Change history
+- 2026-10-23 — Crons now run on worker1.5 (schedule `cron.worker.json`); 2026-10-22/23 cron-stop outage; retention fix still not deployed; Lambda still broken (TRUE-83121). (bala)
 - 2026-10-21 — Gotcha: Compass SalesOrders NetSuite SO id is never UNIQUE (TRUE-82291 skips it). (rgirish)
 - 2026-10-15 — Cron 3a last-ODP-line link bug (since `afb520c1`) root-caused and fixed in `02cc2473`; re-drop does not repair a wrong link. (jcardinal)
 - 2026-10-14 — TRUE-82797 (pending deploy): cron 5 merged only one ODP SO per Compass SO, so siblings went out as a 2nd NetSuite SO (41 Compass SOs in 60 days). Fix merges all ODP SOs of a Compass SO, splits tablets by `TABLET_KIT_PART_NUMBERS`, externalId = lowest ODP SO id, duplicate-part guard emails devteam. Decided against category grouping. (rgirish)

package/knowledge/clients/nycdoe/profile.md

@@ -3,6 +3,7 @@ title: New York City Department of Education
 framework: "1.0"
 apps:
   - worker
+  - worker1.5
   - library
   - togadesk
   - dbchanges
@@ -13,7 +14,7 @@ project: Worker
 client: nycdoe
 type: profile
 status: active
-updated: 2026-10-19
+updated: 2026-10-23
 owners: [mhammontree, sking, bala]
 files: []
 related:
@@ -26,7 +27,7 @@ related:
 NYC DOE client profile (1.0 worker tier live; 2.0 ASN rewrite in progress), ServiceNow + EDI; open for DOE's platform scope, endpoints, API clients, and where its ticket/ASN/EDI logic lives.
 
 ## Summary
-NYC DOE — TOGA client whose **live** integration runs in the **1.0 worker tier** (~30 cron scripts under `worker/crons/sync/nycdoe/` and `worker/crons/notifications/nycdoe/`, on the `sync` and `notification` worker roles). ServiceNow is DOE's system of record for tickets (Incidents and RITMs); TOGA mirrors those locally, drives fulfillment through TOGa Desk and NetSuite, and pushes status / ETA / proof-of-delivery back. A parallel EDI relationship (850 POs in, invoices out) runs over DOE's SFTP. A **2.0 rewrite (TRUE-80519)** of the ASN intake is being built against the `Client_Nycdoe` tenant — not yet live.
+NYC DOE — TOGA client whose **live** integration runs in the **1.0 worker tier** (~30 cron scripts under `crons/sync/nycdoe/` and `crons/notifications/nycdoe/`; since 2026-10-22 they run from `worker1.5` on one server, schedule `worker1.5/schedules/cron.worker.json`). Fixed-time DOE reports missed during a cron outage do not catch up; re-run them by hand (see [Diagnosing worker1.5 crons that stopped running](../../1.0/apps/worker1.5/workflows/diagnosing-crons-not-running.md)). ServiceNow is DOE's system of record for tickets (Incidents and RITMs); TOGA mirrors those locally, drives fulfillment through TOGa Desk and NetSuite, and pushes status / ETA / proof-of-delivery back. A parallel EDI relationship (850 POs in, invoices out) runs over DOE's SFTP. A **2.0 rewrite (TRUE-80519)** of the ASN intake is being built against the `Client_Nycdoe` tenant — not yet live.
 
 ## Platforms & data
 - **1.0 worker tier** — all live business logic. Local mirror tables in `db_common` (`NYCDOETickets`, `NYCDOELocations`, `AdvanceShippingNotice*`).

package/knowledge/clients/prudential/profile.md

@@ -7,6 +7,7 @@ apps:
   - dbchanges2
   - websocket
   - worker
+  - worker1.5
   - worker2
   - library
   - toga2-commerce
@@ -17,7 +18,7 @@ project: _Underscore
 client: prudential
 type: profile
 status: active
-updated: 2026-10-22
+updated: 2026-10-23
 owners: ["jcardinal", "rgirish", "bala", "mhammontree"]
 files: []
 related:
@@ -40,8 +41,8 @@ related:
 ## Summary
 Prudential is a TOGA client whose device-fulfillment flow is driven by **Dell** via the Dell
 API (`Client_Prudential.Apis.id = 2`). Dell posts Advance Shipping Notices (with tracking and
-return-tracking per unit) into the 2.0 platform (DB `Client_Prudential`); the 1.0 worker tier
-(`crons/toga2/prudential/`) handles NetSuite sync, ServiceNow (RITM) close/ship updates, and
+return-tracking per unit) into the 2.0 platform (DB `Client_Prudential`); the 1.0 cron tier (since 2026-10-22 the single
+worker1.5 server, `worker1.5/crons/toga2/prudential/`) handles SR → SO/PO generation, NetSuite sync, ServiceNow (RITM) close/ship updates, and
 order-status transmissions.
 
 ## Integrations
@@ -119,11 +120,21 @@ The order-created cron branches on this id; the shipped/delivered crons branch o
   features/service-request-auth-rejection-alert.md.
 
 - **Daily "Prudential REQ's Processing Report" recipients are hardcoded.** Sent by 1.0
-  `worker/crons/notifications/reports/prudential_exception_report.php` (NOT the sibling
-  `reports/prudential/` folder), scheduled in `worker/schedules/cron.worker.notification.json` as
+  `worker1.5/crons/notifications/reports/prudential_exception_report.php` (NOT the sibling
+  `reports/prudential/` folder), scheduled in `worker1.5/schedules/cron.worker.json` as
   "Prudential Exception Report (9:00 AM)". Recipients = the `$emailTo` PHP array near the top of the
   file (a hardcoded list of togatech.com addresses, all TO, no CC); no DB table or setting. To add or
-  remove someone: edit that array, deploy worker `_production` (CodePipeline `Worker`). (TRUE-82925)
+  remove someone: edit that array, deploy worker1.5. (TRUE-82925)
+
+- **⚠ The SR → SO cron piles up ContactAddresses (OPEN, TRUE-83138).**
+  `worker1.5/crons/toga2/prudential/generate_sales_and_purchase_orders_from_service_requests.php`
+  (~L182-249) GETs `/contacts/{uuid}/contact-addresses?depth=3`, and when the REQ address differs from
+  the first saved one it POSTs a NEW `/contact-addresses` and PUTs the contact's
+  `primaryContactAddress` — every run, never reusing a match. Prod 2026-10-23: 15,379 rows for ~20.7k
+  contacts; contact 54 had 2,530. That GET then ran api2 out of memory (500, no Api log row; see
+  [V2 query contract § depth](../../2.0/apps/api2/features/v2-rest-query-contract.md)) and SRs stopped
+  becoming SOs. api2 memory was raised as a stopgap. Open: reuse a matching ContactAddress, and send
+  a lower `depth` on that GET.
 
 - **SR number race (TRUE-82209: ServiceNow bursts → 1062 `Duplicate entry 'SR…'` → 400 `EV-10`, SR lost).** Fixed by
   per-client record numbers from `RecordNumbers` (TRUE-82715; SR series continues); the `usleep` stopgap in `_Model_Prudential_ServiceRequest::prePost()`

package/knowledge/clients/true/profile.md

@@ -29,12 +29,13 @@ apps:
   - test
   - togaview
   - toga
+  - worker1.5
 project: _Underscore
 client: true
 type: profile
 status: active
-updated: 2026-10-22
-owners: [jcardinal, kyalamarthi, tcox, rgirish, mhammontree, akhokhani]
+updated: 2026-10-23
+owners: [jcardinal, kyalamarthi, tcox, rgirish, mhammontree, akhokhani, bala]
 files: []
 related:
   - features/toga-hd-voice-demo.md

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.1126",
+  "version": "1.0.1127",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",