toga-ai 1.0.645 → 1.0.647

package/knowledge/1.0/apps/library/features/cron-execution-monitoring.md

@@ -6,12 +6,13 @@ project: Library
 client: shared
 type: feature
 status: active
-updated: 2026-07-06
-owners: [dfranks]
+updated: 2026-08-25
+owners: [dfranks, bala]
 files:
   - library/app/framework.php
 related:
   - ../../worker/architecture.md
+  - ../../worker/workflows/tracing-a-worker-cron-run-in-production.md
 ---
 
 ## Summary
@@ -38,13 +39,28 @@ row in `db_log` and of the per-job Sentry check-in monitors.
   cast silently discards the sub-second precision even when the schema column is `decimal(10,3)`.
 - **Schema:** the table is provisioned by the dbchanges migration
   `dbchanges/Common/DF/2026-5-7 Cron Checkin.sql` (table `CronJobExecutions`,
-  `executionTimeSeconds decimal(10,3)`, no `note` column).
+  `executionTimeSeconds decimal(10,3)`). **Correction (2026-08-25): prod `Common.CronJobExecutions`
+  does have a `note` TEXT column** — verified by writing to it and reading it back. `App_Framework`
+  never populates it, which makes it the practical place to park **temporary** step tracing for a
+  cron you are diagnosing (worker crons have no readable stdout — see
+  [Tracing a worker cron run in production](../../worker/workflows/tracing-a-worker-cron-run-in-production.md)).
+  Remove the tracing when you are done, and never write payloads or credentials into it.
+- **Don't confuse the two `note`s.** `cronInitialization()` writes `note = 'Started execution'` to
+  the **`Log`** table on `db_log` (the older `CRON`/`recordType` row), not to `CronJobExecutions`.
 - **Design decision:** 1.0 records executions by writing **directly to the shared DB**
   (`db_common`), *not* by POSTing from 1.0 to a 2.0 API endpoint. Per Jeff Cardinal, 1.0 code
   must not POST to 2.0 code; the direct shared-DB write is the sanctioned path (TRUE-78182).
 
 ## Gotchas
 
+- **⚠ A run skipped by the overlap guard leaves NO ROW — absence is ambiguous.**
+  `cronInitialization()` calls `exitIfProcessRunning()` **before** the `CronJobExecutions` INSERT,
+  and the `cronFinished(true)` that guard then calls does nothing because `cronLogId` was never
+  set. So a blocked run is indistinguishable from a run that never launched: **a missing row for
+  an expected slot means "skipped", not "hung"**, and a hung run looks different (it has
+  `dtCheckIn` with no `dtCheckOut`). `isProcessRunning()` matches **any** `ps -ef` line containing
+  the script path (it only excludes `/bin/sh` and its own pid), so a stray `tail -f` or editor on
+  that path blocks the cron indefinitely.
 - **`db_log` has no `CronJobExecutions` table.** A `cronFinished()` UPDATE that runs against the
   `db_log` connection throws on **every** 1.0 cron job. Both the INSERT and the UPDATE must
   target `db_common`.
@@ -60,6 +76,12 @@ row in `db_log` and of the per-job Sentry check-in monitors.
   logic. (Fixed 2026-07-06: consolidated back to one `db_common` INSERT/UPDATE pair.)
 
 ## Change history
+- 2026-08-25 — Corrected the schema note (prod `CronJobExecutions` **does** carry a `note` TEXT
+  column, unused by `App_Framework`, usable for temporary cron tracing) and separated it from the
+  `note = 'Started execution'` row `cronInitialization()` writes to `Log` on `db_log`. Recorded
+  that the overlap guard runs **before** the INSERT, so a skipped run writes no row at all and
+  cannot be told apart from "never launched" — and that `isProcessRunning()`'s `ps -ef` substring
+  match lets any process holding the script path block the cron. No code change. (bala)
 - 2026-07-06 — Fixed a merge (`origin/_production`) that reintroduced duplicate
   `CronJobExecutions` writes and pointed one `cronFinished()` UPDATE at `db_log` (where the
   table doesn't exist), which would have thrown on every 1.0 cron. Consolidated to one

package/knowledge/1.0/apps/worker/INDEX.md

@@ -11,9 +11,10 @@
 | [NetSuite Sales Order Sales Rep Sourcing (Staples & ODP EDI orders)](features/netsuite-sales-order-sales-rep-sourcing.md) | How the **sales rep** on a NetSuite Sales Order is determined for the two 1.0 `worker` EDI order-creation integrations (Staples cXML and Compass/ODP EDI). | worker/crons/toga2/compass/workflow/5_create_netsuite_sales_orders_from_office_depot_purchase_orders.php, worker/crons/sync/staples/sync_staples_cxml.php, test/@Mark/NetSuite/TRUE_80451_customer_salesrep_diag.php |
 | [NetSuite → TOGa Supply Per-Client Sync (thin wrappers)](features/netsuite-togasupply-per-client-sync.md) | Syncs NetSuite transactions (sales orders, purchase orders, invoices, item receipts, item fulfillments, inventory adjustments) into each TOGa Supply (2.0) clien | worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_canon.php, worker/crons/toga2/netsuite/sync_togasupply_elite.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql, library/app/api/toga2.php, library/app/api/netsuite/rest.php, library/app/framework.php, library/app/systemmonitor/netsuiteintegration.php, test/@srija/Elite Testing/Service Requests/test_sync_togasupply_elite_section.php, test/@srija/Elite Testing/Service Requests/test_diagnose_togasupply_elite.php |
 | [OneUptime Server monitor + disk/memory hygiene on the 1.0 worker EB host](features/oneuptime-server-monitor-host-hygiene.md) | The 1.0 `agilant-worker` EB environment runs on the **legacy Amazon Linux 1 PHP 7.2 platform** (Apache httpd/prefork, s3fs mounts, cron) and repeatedly went dow | worker/.ebextensions/040_disk_memory_hygiene.config, worker/.ebextensions/045_oneuptime_agent.config, worker/ebs/cron.worker.php, worker/ebs/mount-s3fs-folders.php, worker/ebs/apache_settings.php, worker/ebs/setup_phpini.php |
-| [OneUptime external uptime monitoring for 1.0 workers](features/oneuptime-worker-uptime-monitoring.md) | Every 1.0 worker box self-reports its liveness to an external OneUptime monitor once per minute by curl-POSTing to a per-worker "Incoming Request" heartbeat URL | library/app/worker.php, worker/crons/worker/worker_heartbeat.php |
+| [OneUptime external uptime monitoring for 1.0 workers](features/oneuptime-worker-uptime-monitoring.md) | Every 1.0 worker box self-reports its liveness to an external OneUptime monitor once per minute by curl-POSTing to a per-worker "Incoming Request" heartbeat URL | library/app/worker.php, worker/crons/worker/worker_heartbeat.php, worker/ebs/cron.worker.php, worker/.ebextensions/045_oneuptime_agent.config |
 | [Prudential: Send Shipments for the Day report (daily cron)](features/send-shipments-for-the-day.md) | Daily cron (9:00 PM) that emails Prudential and Dell stakeholders an Excel report of all devices shipped that day, including tracking number, serial number, emp | worker/crons/notifications/reports/send_shipments_for_the_day.php |
 | [Staples cXML Order Import (SFTP → NetSuite)](features/staples-cxml-order-import.md) | `sync_staples_cxml.php` is an **hourly** cron (runs at **:45**) that imports Staples cXML purchase orders from SFTP into NetSuite as Sales Orders, then writes a | worker/crons/sync/staples/sync_staples_cxml.php |
 | [Diagnosing frozen 1.0 worker cron check-ins (Sentry "missed" flood)](workflows/diagnosing-frozen-cron-checkins.md) | When 1.0 worker cron timestamps freeze and Sentry project `worker1` fills with **`missed`** check-ins, the intuitive diagnosis — a wedged `App_Framework::isProc | worker/.ebextensions/cron.config, library/app/worker.php |
 | [isFulfillable Multi-Client Backfill (all togasupply clients)](workflows/isfulfillable-multi-client-backfill.md) | One-time backfill that catches up `Items.isFulfillable` on **existing** items across **all 17 togasupply clients** (AIG, Broward Sheriff, Canon, Endeavor Health | worker/crons/toga2/netsuite/backfill_isfulfillable_all_clients.php, library/app/api/toga2.php |
 | [Onboarding a Client to the NetSuite TOGa Supply Sync](workflows/onboarding-client-to-netsuite-togasupply-sync.md) | How to add a new TOGa 2 client to the per-client NetSuite → TOGa Supply importer (`worker/crons/toga2/netsuite/`). | worker/crons/toga2/netsuite/sync_togasupply.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_elite.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql, dbchanges2/_modules/netsuite/2026-08-05 - CLEAN NETSUITE CLINET.SQL |
+| [Tracing a 1.0 worker cron run in production (no stdout, silent skips)](workflows/tracing-a-worker-cron-run-in-production.md) | How to answer *"did this cron actually run, and what did it do?"* on the 1.0 `worker` tier, where **there is no usable stdout** and **a skipped run leaves no tr | worker/ebs/cron.worker.php, worker/schedules/cron.worker.sync.json, library/app/framework.php |

package/knowledge/1.0/apps/worker/architecture.md

@@ -6,8 +6,8 @@ project: Worker
 client: shared
 type: architecture
 status: active
-updated: 2026-07-29
-owners: [jcardinal, sking]
+updated: 2026-08-24
+owners: [jcardinal, sking, bala]
 files:
   - worker/index.php
   - worker/_/app/framework.php
@@ -20,6 +20,8 @@ files:
 related:
   - ../library/architecture.md
   - ../dbchanges/workflows/authoring-and-shipping-sql-files.md
+  - ./features/oneuptime-worker-uptime-monitoring.md
+  - ../../2.0/apps/worker2/features/worker-fleet-role-assignment-monitor.md
 ---
 
 ## Summary
@@ -44,6 +46,7 @@ which **self-elects a distinct role** (`notification`, `database`, `infrastructu
 under `crons/`; the schedule (cron registration) is the source of truth for what runs — a script
 that isn't scheduled never executes. Client integrations live under `crons/toga2/<client>/`.
 Use prepared statements for all SQL; never interpolate input.
+A worker that fails deploy-time role election runs NO crons while EB still reads healthy.
 
 ## How a job becomes a cron (the dispatch pipeline)
 
@@ -108,6 +111,32 @@ is the most important and least obvious part of the architecture.
 > Net effect: roles are a **claim-the-first-free-slot pool**, and a dead worker's role is
 > reclaimed by its replacement within minutes. There is no static instance→role mapping to edit.
 
+### The failure mode: NO NAME = NO CRONTAB (and it is silent)
+
+The election is the **single point of failure for an entire instance**, and the architecture gives
+it no second chance:
+
+- The claim runs **only at DEPLOY time** — `ebs/cron.worker.php` is written as the EB `appdeploy`
+  enact hook. **It never retries.** An instance that fails to claim stays nameless until the next
+  deploy or until something terminates it.
+- The claim writes `/etc/worker-role` and **only then** appends `cron.worker.<role>.json` to the
+  crontab. So a nameless instance has **no role crontab at all** — it runs *nothing*, not even
+  `worker_heartbeat.php`, while **EB reports the environment healthy**. The box is up; it just
+  does no work.
+- **The claim's DB connect is `@mysqli_connect(...)` — error-suppressed.** A boot-time connect
+  failure to `Vision_Log` therefore leaves the instance nameless **with no log line anywhere**.
+  This is the prime suspect for the 2026-08-24 incident (2 of 7 workers roleless for ~3 hours),
+  and it violates the no-`@`-suppression rule in `rules/toga/common/coding-style.md`. Removing the
+  `@` and logging the failure is the recommended fix.
+- `Workers` **only ever holds instances that SUCCEEDED**, so a failed claim leaves no row —
+  **absence is the only evidence**, and detecting it requires the AWS instance list.
+
+Because every OneUptime monitor on this tier is keyed by **role** rather than by instance, none of
+them can see this (see
+[the blind spot](./features/oneuptime-worker-uptime-monitoring.md)). External coverage now comes
+from the 2.0
+[Worker Fleet Role-Assignment Monitor](../../2.0/apps/worker2/features/worker-fleet-role-assignment-monitor.md).
+
 ## Anatomy of a cron script
 
 Every script is self-contained and follows this boilerplate:
@@ -208,6 +237,10 @@ web face (`mvc/` GET routes for login/logout/404); the tier's real work is the c
 
 ## Conventions & gotchas
 
+- **A worker that fails role election is silently dead, not degraded.** No name → no crontab →
+  the instance runs nothing while EB reads healthy, and the `@`-suppressed `mysqli_connect` in
+  `ebs/cron.worker.php` logs nothing. Never treat "EB is green" as evidence the fleet is working;
+  check `Vision_Log.Workers` against the actual EB instance list.
 - **`dbchanges` auto-apply is branch-gated, and `_production` is NOT auto-applied.**
   `crons/infrastructure/execute_dbchanges.php` runs every 2 minutes but only for branches matching
   `_%`, deriving the env by stripping the leading `_` and requiring `config.<env>.ini` in the

package/knowledge/1.0/apps/worker/features/oneuptime-worker-uptime-monitoring.md

@@ -6,12 +6,17 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-07-13
-owners: ["jcardinal"]
+updated: 2026-08-24
+owners: ["jcardinal", "bala"]
 files:
   - library/app/worker.php
   - worker/crons/worker/worker_heartbeat.php
-related: ["oneuptime-server-monitor-host-hygiene"]
+  - worker/ebs/cron.worker.php
+  - worker/.ebextensions/045_oneuptime_agent.config
+related:
+  - ./oneuptime-server-monitor-host-hygiene.md
+  - ../architecture.md
+  - ../../../2.0/apps/worker2/features/worker-fleet-role-assignment-monitor.md
 ---
 
 ## Summary
@@ -57,6 +62,31 @@ Database, Infrastructure, TOGa, TOGa Desk, and Catalog, then importing each into
 manually. To add a new worker's monitor, clone an existing monitor export the same way and
 register the resulting heartbeat URL in `App_Worker::$oneUptimeEndpoints`.
 
+## THE BLIND SPOT — every monitor here is keyed by ROLE, so a ROLELESS instance is invisible
+
+Load-bearing limitation, learned the hard way (2026-08-24: **2 of the 7 production workers ran
+roleless for ~3 hours undetected**). This layer, and the Server monitors in
+[host hygiene](./oneuptime-server-monitor-host-hygiene.md), are keyed by `workerName` — **14
+monitors, none keyed by instance**:
+
+- `worker_heartbeat.php` only pings `App_Worker::$oneUptimeEndpoints[$row['workerName']]` for
+  the row matching **its own** `instanceId`. No row → no ping target → it pushes nothing.
+- The `045_oneuptime_agent.config` Server agent reads `/etc/worker-role` and **exits clean when
+  it is empty.**
+
+So an instance that never claimed a role reports to nothing — and because every role is still
+owned by *someone*, **no role-keyed monitor is missing a ping either.** The fleet reads 100%
+healthy while a box does zero work (role assignment happens **only at deploy time**, and no role
+means the role's `cron.worker.<role>.json` is never appended to the crontab, so the instance runs
+**nothing**).
+
+`Vision_Log.Workers` only ever holds instances that **succeeded** in claiming a role, so the
+failure leaves no row and no log line — **absence is the only evidence**, and detecting an
+absence requires the AWS instance list, which nothing on this tier consults. That gap is what
+the 2.0
+[Worker Fleet Role-Assignment Monitor](../../../2.0/apps/worker2/features/worker-fleet-role-assignment-monitor.md)
+now closes from outside the fleet. **Do not try to close it with another role-keyed monitor.**
+
 ## Gotchas
 
 - Do **not** hardcode the heartbeat URLs/UUIDs anywhere but `App_Worker::$oneUptimeEndpoints`
@@ -67,6 +97,13 @@ register the resulting heartbeat URL in `App_Worker::$oneUptimeEndpoints`.
   pending cleanup — the live source of truth is the `App_Worker::$oneUptimeEndpoints` map.
 
 ## Change history
+- 2026-08-24 — Documented **the blind spot**: all 14 monitors on this tier are keyed by
+  `workerName`, so an instance that claimed **no** role pings nothing and leaves every role-keyed
+  monitor still green — which is how 2 of 7 production workers ran roleless for ~3 hours
+  undetected. Recorded that `Vision_Log.Workers` only holds successful claims (absence is the
+  only evidence) and that the gap is now covered from outside the fleet by the 2.0
+  [Worker Fleet Role-Assignment Monitor](../../../2.0/apps/worker2/features/worker-fleet-role-assignment-monitor.md).
+  (bala)
 - 2026-07-13 — Added external OneUptime heartbeat push for all 1.0 workers on top of the
   existing internal DB-heartbeat mechanism; endpoints keyed by workerName in
   `App_Worker::$oneUptimeEndpoints`. (jcardinal)

package/knowledge/1.0/apps/worker/workflows/tracing-a-worker-cron-run-in-production.md

@@ -0,0 +1,124 @@
+---
+title: Tracing a 1.0 worker cron run in production (no stdout, silent skips)
+framework: "1.0"
+repo: worker
+project: Worker
+client: shared
+type: workflow
+status: active
+updated: 2026-08-25
+owners: ["bala"]
+files:
+  - worker/ebs/cron.worker.php
+  - worker/schedules/cron.worker.sync.json
+  - library/app/framework.php
+related:
+  - ../architecture.md
+  - ./diagnosing-frozen-cron-checkins.md
+  - ../../library/features/cron-execution-monitoring.md
+  - ../../../1.0/standards/backend-php.md
+---
+
+## Summary
+How to answer *"did this cron actually run, and what did it do?"* on the 1.0 `worker` tier, where
+**there is no usable stdout** and **a skipped run leaves no trace at all**. Written from a
+production diagnostic session on
+`worker/crons/toga2/compass/workflow/3a_import_office_depot_purchase_orders.php`; the mechanics
+apply to every cron on the tier.
+
+Use this before you conclude a cron "never fired" — three different situations look identical
+from the outside.
+
+## Step 1 — get the real schedule from `schedules/*.json`, never from the file
+The `.php` file's own header comment is not authoritative and is routinely stale. So are skills,
+docs and tickets. `worker/schedules/cron.<env>[.<role>].json` is the only source of truth.
+
+Worked example: the entry *"Download EDI From S3 Files & Create PO - EDI Office Depot"* in
+`cron.worker.sync.json` is `*/5 * * * *` — **every 5 minutes**. The cron file's header said
+"EVERY HOUR" (corrected 2026-08-25) and at least one skill still says hourly. A wrong assumed
+frequency makes you read a normal gap as an outage.
+
+Also check whether the job is registered in the env you are testing in at all: this one is **not**
+in `cron.beta.json`, so beta never runs it on a schedule. Nothing on beta is evidence about it.
+
+## Step 2 — echo/print goes nowhere useful, so do not debug with it
+The crontab line is assembled in `worker/ebs/cron.worker.php`, and the two blocks that build it
+differ:
+
+| Schedule file | Command written | Where output goes |
+|---|---|---|
+| base `cron.<env>.json` | `php /var/www/html/crons/<script>` | **nowhere** — the ` > /var/www/cache/WORKER_ERROR_$RANDOM` redirect is **commented out** |
+| role `cron.worker.<role>.json` | `php /var/www/html/crons/<script> > /var/www/cache/WORKER_ERROR_$RANDOM` | a **randomly named** file under `/var/www/cache/` on whichever of the 7 instances ran it |
+
+Either way you cannot go and read it: base-schedule output is discarded outright, and a
+role-schedule run leaves an unpredictable filename on a box in an autoscaled fleet. **Treat
+worker cron stdout as write-only.**
+
+## Step 3 — put temporary tracing somewhere you can query
+What worked: write timestamped step markers into the **`note`** column of the run's
+`Common.CronJobExecutions` row (the row `App_Framework::cronInitialization()` already inserted),
+then read them back over the DB tooling from your desk. `dtCheckIn`, `dtCheckOut` and
+`executionTimeSeconds` on the same row give you the shape of the run for free.
+
+Rules: keep it to a few markers, and **remove the tracing once diagnosed** (it was removed in
+this session). Never write payloads or credentials into `note`.
+
+## Step 4 — read `Common.CronJobExecutions` to tell "ran", "hung" and "skipped" apart
+Query the legacy env's `Common.CronJobExecutions` filtering `job LIKE` the script path:
+
+- **`dtCheckIn` + `dtCheckOut` + `executionTimeSeconds`** — it ran and finished; the duration
+  tells you whether it fits inside its schedule slot.
+- **`dtCheckIn` with no `dtCheckOut`** — it started and never completed (crash, timeout, or the
+  instance went away).
+- **no row for a slot you expected** — the run was **skipped by the overlap guard**, not "never
+  fired". A blocked run writes **nothing**: `cronInitialization()` calls
+  `exitIfProcessRunning()` **before** the INSERT, and the `cronFinished(true)` it then calls is a
+  no-op because `cronLogId` is unset. See
+  [Cron Execution Monitoring](../../library/features/cron-execution-monitoring.md).
+
+### The overlap guard is a `ps -ef` substring match — anything can block it
+`App_Framework::isProcessRunning()` returns true for **any** `ps -ef` line containing the script
+path (it only excludes `/bin/sh` lines and its own pid). So a developer's `tail -f`, an editor, or
+any shell holding that path in its command line **blocks the cron indefinitely** while looking
+like a legitimate overlap. Check `ps -ef` for what is actually holding the name before assuming a
+long-running instance of the job itself.
+
+## Step 5 — if runs overlap, look at what the job does before its real work
+A job whose own runtime exceeds its schedule interval silently loses most of its slots to the
+guard. Measure the phases, not the total: in the ODP case an instrumented run showed the S3
+listing alone returning **138,472 objects and taking 32 seconds** before any PO was touched, with
+each PO then costing roughly 40 seconds — comfortably past a 5-minute schedule. Prefixes the job
+skips while processing are still listed, so accumulated `SENT/` and `OUTBOX/` objects inflate
+every run.
+
+## Deploy-time trap: a PHP 8-only syntax kills the whole cron with no visible error
+This tier runs **PHP 7.2**, so a **PHP 8 named argument** (`someFunction(items: $x)`) is a
+**parse error**, not a runtime warning:
+
+```
+PHP Parse error: syntax error, unexpected ':', expecting ',' or ')'
+```
+
+The script then does nothing at all on every tick — and per Steps 2 and 4 you will see no output
+and no `CronJobExecutions` row, i.e. it looks exactly like "the cron never fired". Typed
+parameters and return types (`: void`, `: array`, `: bool`, `?array`, `?object`, `object $x`) are
+PHP 7.0/7.1 and **are** safe here; they are already used in
+`worker/crons/toga2/compass/backfill_items_isfulfillable.php` and
+`update_item_fulfillments_in_netsuite.php`.
+
+`1.0/standards/backend-php.md` already forbids named arguments in 1.0 — but a workspace
+`CLAUDE.md` / `.github/copilot-instructions.md` that says *"use PHP 8 Named Parameters"* applies
+to **2.0 only**. The 1.0 standard wins in `library/`, `worker/` and every other 1.0 app.
+
+**Do not read a PHP 8 function call in existing code as proof the tier is PHP 8.**
+`library/app/systemmonitor/500error.php` uses `str_contains()`; that file is not a version
+signal, it is a latent bug.
+
+## Change history
+- 2026-08-25 — Documented from a prod diagnostic session on the Compass ODP 850 importer: the
+  schedule JSON (not the file header) is authoritative, worker cron stdout is unreadable (base
+  block's `WORKER_ERROR` redirect commented out, role block writes a random filename),
+  `Common.CronJobExecutions.note` is the practical place for temporary tracing, a run skipped by
+  the `ps -ef` overlap guard writes **no row at all** (guard runs before the INSERT), the guard
+  matches any process holding the script path, and a PHP 8 named argument parse-errors the whole
+  cron silently on this PHP 7.2 tier. (bala)

package/knowledge/2.0/apps/_underscore/features/email-template-sending.md

@@ -132,6 +132,16 @@ can keep using `sendEmail($api, ...)`.
     predated the commit. That is a **deploy gap, not a code bug** — a redeploy fixes it, and no code
     change should be made. Same known-issue as
     [api2 environment-variable-drives-underscore-branch](../../api2/features/environment-variable-drives-underscore-branch.md).
+- **⚠ An explicit `null` recipient is a 400, not a 500 (fixed 2026-08-25).** `$to`/`$cc`/`$bcc` on
+  `sendEmail()`/`send()`/`dispatch()` are typed `string|array|null` (a scripted-API caller can pass an
+  argument that is present-but-`null` — the `[]` default only applies when the arg is **omitted**).
+  Before the fix they were non-nullable `string|array`, so an explicit `null` `$to` threw
+  `Argument #3 ($to) must be of type array|string, null given` → **HTTP 500** — this was the platform's
+  **largest 5xx contributor** (prod `Logs.Issue` ref `2F`, 828 occurrences). `dispatch()` now normalizes
+  each recipient at the top (`(array)($x ?? [])`); after merging the template's stored
+  `EmailTemplateOutgoingEmailAddress` rows, if `$to` is **still empty** it throws `_Exception_Validation`
+  (→ **HTTP 400**) rather than a TypeError-500 or sending a recipientless email. A missing recipient is
+  client input error, not a server fault.
 - **`sendEmail()`'s signature is load-bearing for scripted APIs** — the Record Script engine
   (`api2/Component/Api/V2/V2.php`, ~line 3594) calls the method with `api` as a named
   argument, so the first param must stay `&$api`. Do not "clean it up" by removing it.
@@ -198,6 +208,13 @@ worker method) in-process instead.
   restored it — and that `sendEmail` has been variadic since 2024-12-24, so the spread was always
   correct and May was the regression. Documented the contract on `sendEmail()`'s docblock plus a marker
   comment at each of the four Quad call sites. (apeterson)
+- 2026-08-25 — Made `$to`/`$cc`/`$bcc` nullable (`string|array|null`) on `sendEmail()`/`send()`/
+  `dispatch()` and normalized each to an array at the top of `dispatch()` (`(array)($x ?? [])`),
+  fixing the platform's **largest 5xx contributor** (prod `Logs.Issue` ref `2F`, 828 occurrences): a
+  scripted-API caller passing an explicit `null` recipient hit `Argument #3 ($to) must be of type
+  array|string, null given` → HTTP 500. After merging the template's stored recipient addresses, an
+  empty `$to` now throws `_Exception_Validation` (HTTP 400) instead of a TypeError-500 or a
+  recipientless send. Added the corresponding gotcha. (jcardinal)
 - 2026-08-18 — Fixed the Quad order-approval/rejection emails' production 500
   (`str_replace(): Argument #2 must be string`, EO-1): the four `Model/Quad/ApprovalDecision.php`
   + `Model/Quad/SalesOrder.php` call sites passed the template-vars map as a bare positional

package/knowledge/2.0/apps/_underscore/features/per-client-database-connections.md

@@ -6,8 +6,8 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-08-17
-owners: ["dfranks", "jcardinal", "mhammontree", "apeterson", "kyalamarthi"]
+updated: 2026-08-24
+owners: ["dfranks", "jcardinal", "mhammontree", "apeterson", "kyalamarthi", "bala"]
 files:
   - _underscore/Database.php
   - _underscore/Model.php
@@ -177,6 +177,16 @@ here — they live in `Config/*.ini`.)
 - Related 1.0 analogue: the legacy `App_` worker has the same hazard writing to `Logs.API`
   (`db_logs`) — the laptop trap there is documented separately in the worker NetSuite bootstrap
   notes.
+- **`_Database::register()` is LAZY — it stores connection config and never opens a socket.**
+  The connect happens on first use. So adding a static alias registration to an app's `_.php`
+  bootstrap is **safe across every environment**, even one whose config group points at a
+  localhost that lacks the schema: nothing fails until something actually queries that alias.
+  This is what makes "register the alias globally, use it in one cron" a two-line change rather
+  than a per-environment config exercise (worked example: `DB_VISION_LOGS` in
+  [the Worker Fleet Role-Assignment Monitor](../../worker2/features/worker-fleet-role-assignment-monitor.md)).
+  The corollary is the trap the rest of this doc describes: because registration is free and
+  silent, a **bad** registration also stays silent until the request that needs it blows up with
+  `Unknown database`.
 
 ## WITHDRAWN — the "alias-keyed `$_modelCache` cross-tenant leak" hypothesis (2026-08-17)
 
@@ -210,6 +220,11 @@ also hits `_modules/<module>/`** — is recorded in
 
 ## Change history
 
+- 2026-08-24 — Recorded that **`_Database::register()` is lazy** (stores config, opens no
+  socket; the connect happens on first use), so a static alias registration in an app's `_.php`
+  is safe in every environment including a localhost config that lacks the schema — and that
+  this is also why a *bad* registration stays silent until first query. Verified while adding
+  `DB_VISION_LOGS` to worker2. (bala)
 - 2026-08-17 (later pass) — **WITHDRAWN, supersedes the entry below.** The alias-keyed
   `$_modelCache` cross-tenant-leak hypothesis is **not supported** and is no longer a live security
   concern. The six `c_` columns are **AIG's own**: `_modules/netsuite/2026-07-10a -

package/knowledge/2.0/apps/api2/architecture.md

@@ -6,7 +6,7 @@ project: API
 client: shared
 type: architecture
 status: active
-updated: 2026-08-20
+updated: 2026-08-25
 owners: [jcardinal, bala, mhammontree, dfranks]
 files:
   - api2/Controller/Index.php
@@ -114,14 +114,22 @@ One ~2,000-line `execute()` then `processRoutePairs()`:
 5. **Transaction logging** — every request logged (to client/core Logs DB, or as a JSONL
    line shipped by CloudWatch when `[api] log_filepath` is set).
 
-> **⚠ Auto-generated `Api.transactionId` collides under concurrency → 1062 → HTTP 500.**
+> **Auto-generated `Api.transactionId` used to collide under concurrency → 1062 → HTTP 500 (FIXED 2026-08-25).**
 > Separate from the *client-supplied* `transactionId` uniqueness check (EV-5, above): the inbound
-> request-logger inserts its `Api` log row with a UNIQUE `transactionId` set to a
-> millisecond-precision timestamp (`Y-m-d H:i:s.v`). Concurrent nested writes generated within the
-> same millisecond collide on that UNIQUE key → MySQL **1062** → **HTTP 500**. This breaks ingestion
-> for high-volume senders (seen on the Compass/Veyer ASN feed, `sourceIp 34.232.23.158`) and is
-> platform-wide. Fix direction: make the logged `transactionId` unique-enough (uuid) or
-> retry-on-1062. Until fixed, a burst of concurrent posts can intermittently 500 with no app-level cause.
+> request-logger inserted its `Api` log row with a UNIQUE `transactionId` (`Logs.Api.transactionId`,
+> `varchar(255) UNIQUE`, in both Core `Logs` and `Logs_<Client>`) set to a millisecond-precision
+> timestamp (`Y-m-d H:i:s.v`). Concurrent inserts in the same millisecond collided on that UNIQUE key
+> → MySQL **1062** → the logger threw and turned a *successful* request into **HTTP 500 (EO-1)**. This
+> broke ingestion for high-volume senders (seen on the Compass/Veyer ASN feed, `sourceIp 34.232.23.158`;
+> confirmed in prod `Logs.Issue` reference `1Z`, 304 occurrences, `Duplicate entry
+> '<ms-timestamp>' for key 'Api.transactionId'`, stack `/v2/users/me → _Model_Client_User::me() →
+> internalApiRequest() → _Model->save()`).
+> **Fix:** the two DB-insert log sites now set the log-row `transactionId` to a random uuid
+> (`_String::generateUuid()`) so it cannot collide — `V2.php` `internalApiRequest()` logger (~L2370)
+> and `Controller/Index.php` uncaught-`Throwable` error-recovery logger (~L456). The request's own base
+> `transactionId` was already a uuid when auto-generated (`V2.php` ~L2130); only the bare-ms log-row id
+> was at fault. The **sibling CloudWatch JSONL file-append logger** (`Controller/Index.php` ~L433) is
+> **intentionally left bare-ms** — a JSONL append has no UNIQUE key and cannot collide.
 
 ## CRUD engine — `processRoutePairs()`
 
@@ -216,6 +224,34 @@ IMDSv2 token on the `curl` command line (visible in `ps`/`/proc/<pid>/cmdline`,
 back silently to IMDSv1 — fix both alongside the key. *(Location + remediation only; no key material
 is recorded anywhere.)*
 
+## EB "Degraded" health alerts vs. the ALB's own view (monitoring — read before triaging)
+
+An Elastic Beanstalk enhanced-health transition — *"Environment health transitioned Ok→Degraded …
+One or more TargetGroups … in a reduced health state: awseb-AWSEB-<id> - Degraded"* — **does NOT
+mean the ALB marked a target unhealthy.** On `api-production-1` (AWS account `654654170868`), during
+such an event the ALB's own view was **fully healthy throughout**: `HealthyHostCount=2`,
+`UnHealthyHostCount=0`, ELB `5xx=0`, `TargetConnectionErrorCount=0`.
+
+The two subsystems measure different things and legitimately disagree during any transient:
+
+- **EB enhanced health** rates the target group from its **own per-instance request-latency/status
+  sampling on a point-in-time snapshot** — a short traffic burst (observed `RequestCount ~10x`
+  baseline, peaking ~1,936/min, with a `TargetResponseTime` max outlier ~7.7 s) is enough to flip it
+  to *Degraded*.
+- **The target-group console** shows **current ALB health**, which recovers within ~45 s (3 checks).
+  So when you open the console after the alert, it "looks perfectly healthy" — because it is, now.
+
+api2's EB target group health-checks **`/health`** (interval 15 s, timeout 5 s, unhealthy threshold
+5 ≈ **75 s to trip**, healthy threshold 3 ≈ **45 s to recover**). A transient burst is **expected
+behavior, not an ALB/target-group defect** — do not chase a phantom target failure.
+
+**The exception:** some *Degraded* events are instead genuine **"X% HTTP 5xx"** — those are real
+code bugs, not transient bursts. The 2026-08 batch traced to the `transactionId`-1062 collision
+(gotcha above), the platform-wide EmailTemplate null-recipient TypeError, and the Pcmaticb2b ticket
+interceptor's plain-`Exception` rejections — all now returning 4xx/fixed. Distinguish the two by
+reading the ALB 5xx metric and `Logs.Issue`: zero 5xx + a traffic spike = transient EB snapshot;
+a sustained 5xx rate = a code bug to fix.
+
 ## Known issues / accepted risks
 
 Open items a maintainer should know before changing this tier. None are "bugs to fix right now" —
@@ -289,6 +325,18 @@ they are the known sharp edges. Do not re-discover these from scratch.
   single-caller branches in `V2.php` as unverified until exercised directly.
 
 ## Change history
+- 2026-08-25 — **Resolved the auto-generated `Api.transactionId` 1062 collision** (was the 2026-07-28
+  gotcha): the two DB-insert log sites now uuid the log-row `transactionId` (`_String::generateUuid()`)
+  — `V2.php` `internalApiRequest()` logger (~L2370) and `Controller/Index.php` error-recovery logger
+  (~L456); the CloudWatch JSONL append (~L433) is intentionally left bare-ms (no UNIQUE key). Confirmed
+  the production symptom (`Logs.Issue` ref `1Z`, 304 occurrences, `/v2/users/me` → `_Model->save()`).
+  Also added a **monitoring section**: an EB enhanced-health *Ok→Degraded* "TargetGroups … reduced
+  health state" transition does **not** mean the ALB marked a target unhealthy — on `api-production-1`
+  (acct `654654170868`) the ALB view stayed fully healthy (HealthyHostCount=2, 5xx=0) while a short
+  ~10x traffic burst (~1,936/min, TargetResponseTime max ~7.7 s) flipped EB's point-in-time snapshot;
+  the target-group console recovers in ~45 s so it "looks healthy" after the fact. Recorded api2's
+  `/health` check timing (15 s / 5 s / trip 75 s / recover 45 s) and how to tell a transient burst
+  (0 ALB 5xx) from a genuine "X% HTTP 5xx" code bug. (jcardinal)
 - 2026-07-28 — Added Known issue #10: cross-client page-number paging is inherently O(page) (deep pages materialize every preceding row into the Cache cluster), with caller-facing keyset cursors identified as the durable fix but left unbuilt/unscoped, plus the `DEEPEN_CHUNK_MAX_RECORDS = 500` tuning tradeoff against `getFullModelData()` expansion and `CURL_TIMEOUT_SECONDS = 30`. Added a change-guidance bullet that single-caller branches in the untested `V2.php` monolith get zero incidental coverage (the keyset `LIMIT` syntax error shipped invisibly for a full cycle). (jcardinal)
 - 2026-07-28 — Documented the previously unrecorded `060_register_instance_to_shared_application_load_balancer` postdeploy hook pair in the Deployment section (non-prod self-registration into the same-named ALB target group; production skipped), and recorded a **committed IAM access key** in `ebs/register_instance_to_shared_application_load_balancer.php` as a security note + Known issue #9 — location, line range, commit subject, and remediation only (rotate, audit CloudTrail, move to the instance profile; history rewrite is a separate sign-off). Flagged api2's copy as the unhardened original vs. the new worker2 reference implementation. (jcardinal)
 - 2026-07-28 — Added a consolidated **Known issues / accepted risks** section (8 items), absorbing the previously free-floating deferred raw-exception-disclosure follow-up as item 1, so the tier's sharp edges (unrotated committed secrets, pre-execute phase still outside the main guard, local Logs DB name mismatch, permissive CORS, unpinned `_underscore` build clone, untested `V2.php` monolith, JWT rotation overlap window) are in one place instead of scattered. Recorded that `DB_CACHE` is resolved by name (`Databases.name = 'Cache'`), never by a hardcoded id, which differs per Core instance. (jcardinal)

package/knowledge/2.0/apps/worker2/INDEX.md

@@ -17,7 +17,7 @@
 | [ClickUp Work Type Automation (Committed / Conditional / Stretch)](features/clickup-work-type-automation.md) | The ClickUp webhook handler (`_Worker_Clickup`) automatically maintains each task's **Work Type** custom field — `Committed`, `Conditional`, or `Stretch` — base | worker2/Worker/Clickup.php, worker2/Tests/Worker/ClickupWorkTypeTest.php |
 | [Compass VIP Support Importer (worker2)](features/compass-vip-support-importer.md) | A worker2 action that ingests Compass's quarterly VIP spreadsheet and assigns each VIP user's support technician by setting `Users.c_supportedByUserId` in `Clie | worker2/Worker/Client/Compass/VipSupport.php |
 | [Creating Worker Actions](features/creating-worker-actions.md) | How to add a new callable Worker action — a PHP class whose `public static` methods are invoked as background jobs (via webhook, cron, or `_Worker::runTask()`). | worker2/Worker/, worker2/Controller/Index.php, _underscore/Worker.php |
-| [Cross-account AWS access for worker2 crons (_Component_Aws_Workloads)](features/cross-account-aws-access.md) | `_Component_Aws_Workloads` is the **standard, and only sanctioned, way any new worker2 cron obtains AWS access** — for any account, any region, any AWS SDK clie | worker2/Component/Aws/Workloads/Workloads.php, worker2/Config/production.ini, worker2/Worker/Infrastructure/CloudWatch.php |
+| [Cross-account AWS access for worker2 crons (_Component_Aws_Workloads)](features/cross-account-aws-access.md) | `_Component_Aws_Workloads` is the **standard, and only sanctioned, way any new worker2 cron obtains AWS access** — for any account, any region, any AWS SDK clie | worker2/Component/Aws/Workloads/Workloads.php, worker2/Config/production.ini, worker2/Worker/Infrastructure/CloudWatch.php, worker2/Worker/Monitor/Fleet.php, library/app/worker.php, library/app/cloud.php |
 | [Elastic Beanstalk health monitor → OneUptime (ElasticBeanstalkHealth)](features/elastic-beanstalk-health-monitor.md) | `_Worker_Infrastructure_CloudWatch::ElasticBeanstalkHealth()` is a worker2 cron that reads each Elastic Beanstalk (EB) environment's **enhanced-health** status | worker2/Worker/Infrastructure/CloudWatch.php |
 | [Elite Freshservice Sync (worker2)](features/elite-freshservice-sync.md) | `_Worker_Elite` processes Freshservice webhook events and syncs them into TOGA 2. | worker2/Worker/Elite.php, worker2/Config/dev-kmaramreddy-laptop.ini |
 | [Error Escalation Cron (Errors::Escalate → ClickUp / email)](features/error-escalation-cron.md) | `_Worker_Infrastructure_Errors::Escalate` (renamed from `SyncWithClickup`) is the sole owner of **escalation, de-escalation, ClickUp ticketing, reminders, busin | worker2/Worker/Infrastructure/Errors.php, worker2/Worker/Notification/Email.php, worker2/Worker/Notification/EmailTemplate.php, worker2/Worker/Client/True.php, worker2/Worker/Clickup/ErrorTask.php, worker2/Worker/Clickup.php, worker2/Controller/Index.php, worker2/Config/production.ini, _underscore/Model/Core/Logs/Issue.php, dbchanges2/Core/2026-07-30a - Error escalation cron job.sql, dbchanges2/Logs/2026-08-03a - Issue clickupPriority.sql, dbchanges2/Core/2026-08-04a - Error neglect digest cron job.sql |
@@ -47,6 +47,7 @@
 | [Centralized Tracking-Status Refresh (worker2 Sync cron, all clients)](features/tracking-status-refresh.md) | A single platform-wide cron keeps `TrackingNumbers.status` current for **every provisioned client** until each shipment reaches a terminal state, by polling Fed | worker2/Worker/Sync/TrackingNumbers.php, _underscore/Component/Library/Carriers/Fedex/Fedex.php, _underscore/Component/Library/Carriers/Usps/Usps.php, _underscore/Component/Library/Carriers/Ups/Ups.php, _underscore/Model/Client/TrackingNumber.php, dbchanges2/Client/2026-07-30 - TrackingNumbers_RefreshColumns.sql, dbchanges2/Client/2026-07-30 - Acl_TrackingStatus_Grant.sql, dbchanges2/Client_Tdsynnex/2026-07-30 - Apis_TdsynnexKey.sql, dbchanges2/Core/2026-07-30 - TrackingRefresh_Cron.sql |
 | [VAPI Webhook Handler (worker2 — AI-BDR end-of-call processing)](features/vapi-webhook-handler.md) | `_Worker_Vapi` ([worker2/Worker/Vapi.php](worker2/Worker/Vapi.php)) is the **PHP side of the AI-BDR call loop** — the webhook that receives VAPI's end-of-call r | worker2/Worker/Vapi.php, worker2/Worker/Ai/Bdr/Vapi.php, worker2/Controller/Index.php |
 | [WJE Freshservice Sync (worker2)](features/wje-freshservice-sync.md) | WJE ("WJE IT", helpdesk `wje.freshservice.com`) is a **Freshservice**-based help-desk client whose tickets, contacts, assets, groups, categories, and canned res | worker2/Worker/Wje.php, _underscore/Component/Api/Wje/Wje.php, _underscore/Model/Wje/Ticket.php, _underscore/Model/Wje/TicketNote.php, _underscore/Model/Wje/Contact.php, _underscore/Model/Wje/Unit.php, _underscore/Model/Wje/TicketTeam.php, _underscore/Model/Wje/TicketCategory.php, _underscore/Model/Wje/AssetType.php, _underscore/Model/Wje/PredefinedReply.php, library/app/api/wje.php, worker/crons/toga2/wje/import_supporting_records.php, worker/crons/toga2/wje/sync_togasupply_wje.php, worker/crons/notifications/reports/wje/wje_common.php, library/app/systemmonitor/wje.php, dbchanges2/Client_Wje/2024-10-04 - WjeOnboarding.sql |
+| [1.0 Worker Fleet Role-Assignment Monitor (Monitor/Fleet/RoleAssignment)](features/worker-fleet-role-assignment-monitor.md) | `_Worker_Monitor_Fleet::RoleAssignment()` is a **2.0 worker2 cron that watches the 1.0 `worker` fleet from the outside**. | worker2/Worker/Monitor/Fleet.php, worker2/_.php, dbchanges2/Core/2026-08-24 - Worker Role Assignment Monitor.sql, worker/ebs/cron.worker.php, worker/crons/worker/worker_heartbeat.php, library/app/worker.php |
 | [PHP Runtime Upgrade on Elastic Beanstalk (worker2 8.3 → 8.5 + PhpSpreadsheet 1.x → 3.x)](workflows/php-runtime-upgrade-dependency-audit.md) | The procedure used to move worker2 from **PHP 8.3 to PHP 8.5** on Elastic Beanstalk, and the dependency work that had to land first. | worker2/composer.json, worker2/composer.lock, worker2/Worker/Team/Sprint.php, worker2/Worker/Client/TowFoundation/ProcessReceipts.php, worker2/Worker/Forecast/Import.php |
 | [Running worker2 locally against real NetSuite (and what EB does instead)](workflows/running-worker2-locally.md) | How to boot **worker2** on a developer machine and run a real worker action against the **live NetSuite** account and a **local** `Forecast` database. | worker2/index.php, worker2/composer.json, worker2/.ebextensions/php_include_underscore.config, worker2/.ebextensions/git.json, worker2/.ebextensions/git.php, worker2/Config/production.ini, _underscore/Component/Api/Netsuite/Netsuite.php, dbchanges2/Logs_Client/2026-04-08_BLANK_CLIENT_LOGS_DATABASE.SQL |
 | [Ticket → ClickUp Pseudocode Planning (Talos-grounded)](workflows/ticket-to-pseudocode-planning.md) | A repeatable procedure for turning a ClickUp ticket into a reviewed, formatted implementation plan posted back to the ticket's `📝 Pseudocode` custom field. | test/@dave/clickup_md2delta.js |

package/knowledge/2.0/apps/worker2/features/cross-account-aws-access.md

@@ -6,18 +6,22 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-11
-owners: [jcardinal]
+updated: 2026-08-24
+owners: [jcardinal, bala]
 files:
   - worker2/Component/Aws/Workloads/Workloads.php
   - worker2/Config/production.ini
   - worker2/Worker/Infrastructure/CloudWatch.php
+  - worker2/Worker/Monitor/Fleet.php
+  - library/app/worker.php
+  - library/app/cloud.php
 related:
   - ../../_underscore/features/component-model-namespace-registration.md
   - ../../_underscore/features/config-group-access.md
   - ../../_underscore/features/cloud-s3-helpers.md
   - ./oneuptime-worker2-monitoring.md
   - ./elastic-beanstalk-health-monitor.md
+  - ./worker-fleet-role-assignment-monitor.md
 ---
 
 ## Summary
@@ -79,7 +83,8 @@ region is only a runtime argument to `client()`, never a reason for a second rol
 - **654654170868** — production hub. Home region `us-east-1`; also used in `us-west-2` and
   `eu-west-1`. worker2 runs here on Elastic Beanstalk; its instance profile role
   `ElasticBeanstalk-EC2-Instance-Profile` is the caller identity.
-- **502614707982** — legacy account, `us-west-2`.
+- **502614707982** — legacy account, `us-west-2`. **This is where the 1.0 `worker` fleet
+  actually runs** — see the account-confusion gotcha below.
 - **975050298201** — non-production account, `us-east-1`.
 
 Each `WorkloadsRuntime` role has broad service permissions (`elasticbeanstalk:*`, `ec2:*`,
@@ -144,8 +149,33 @@ code yet. New code must not adopt the static key.
 **Never record the access-key values in knowledge.** They exist in `production.ini`; that is
 their only home.
 
+### Which account the 1.0 worker fleet lives in (easy and expensive to get wrong)
+
+The 1.0 `worker` EC2 instances are in **502614707982**, `us-west-2` (EB environment
+`agilant-worker`, confirmed in the EC2 console).
+
+**`App_Worker::AWS_WORKER_QUEUE_URL` in `library/app/worker.php` points at 654654170868 and
+misleads** — that account holds only the **SQS queue the 1.0 workers read from**, not the
+instances. Anything that needs to *see* the worker boxes (EB, EC2, health) must target
+**502614707982**.
+
+worker 1.0 authenticates there with a **long-lived hardcoded IAM access key in
+`library/app/cloud.php`** — a committed credential that should be rotated and moved to a role
+(value deliberately not recorded here; see [Legacy static key](#legacy-static-key--being-retired)).
+worker2 does **not** use it: it assumes `WorkloadsRuntime` cross-account from its EB instance
+role, which is exactly the migration this component exists to enable. First worker2 consumer in
+that account is the
+[Worker Fleet Role-Assignment Monitor](./worker-fleet-role-assignment-monitor.md).
+
+**Scope EB reads by `EnvironmentName`, never account-wide.** 502614707982 also runs
+`agilant-worker-alpha` and `agilant-worker-beta` as **separate EB environments that have no role
+registry at all**. An account-level EC2 scan would sweep them in and misreport them.
+
 ## Gotchas / known issues
 
+- **Do not assume the 1.0 worker fleet is in the 868 production hub.** `AWS_WORKER_QUEUE_URL`
+  (654654170868) is the SQS queue only; the instances are in the legacy account 502614707982,
+  `us-west-2`. See [the section above](#which-account-the-10-worker-fleet-lives-in-easy-and-expensive-to-get-wrong).
 - Do not construct `StsClient` or pass explicit credentials by hand in new crons — the whole
   point is that the instance role is the caller and the component owns the assume-role step.
 - The instance-role-as-caller only works **on the EB tier**. Off-tier (e.g. a local CLI run)
@@ -161,6 +191,15 @@ their only home.
   see [creating-worker-actions.md](./creating-worker-actions.md).
 
 ## Change history
+- 2026-08-24 — Pinned down **which account the 1.0 `worker` fleet actually runs in**:
+  **502614707982** (`us-west-2`, EB env `agilant-worker`), **not** the 654654170868 production hub
+  that `App_Worker::AWS_WORKER_QUEUE_URL` implies — that account holds only the SQS queue the 1.0
+  workers read from. Recorded that worker 1.0 reaches it with the hardcoded static key in
+  `library/app/cloud.php` (rotate + move to a role) while worker2 assumes `WorkloadsRuntime` from
+  its instance role, and that EB reads there must be scoped by `EnvironmentName` because
+  `agilant-worker-alpha`/`-beta` are separate registry-less environments in the same account.
+  Found while building the
+  [Worker Fleet Role-Assignment Monitor](./worker-fleet-role-assignment-monitor.md). (bala)
 - 2026-08-11 — Noted that `ElasticBeanstalkHealth`'s `$environments` must be typed `object|array`
   (not `array`): the nested JSON object arrives as `stdClass` because the dispatcher casts only
   the top level of `CronJobs.parameters`, so a strict `array` throws a `TypeError` at dispatch.