toga-ai 1.0.645 → 1.0.646

package/knowledge/1.0/apps/library/features/cron-execution-monitoring.md

@@ -6,12 +6,13 @@ project: Library
 client: shared
 type: feature
 status: active
-updated: 2026-07-06
-owners: [dfranks]
+updated: 2026-08-25
+owners: [dfranks, bala]
 files:
   - library/app/framework.php
 related:
   - ../../worker/architecture.md
+  - ../../worker/workflows/tracing-a-worker-cron-run-in-production.md
 ---
 
 ## Summary
@@ -38,13 +39,28 @@ row in `db_log` and of the per-job Sentry check-in monitors.
   cast silently discards the sub-second precision even when the schema column is `decimal(10,3)`.
 - **Schema:** the table is provisioned by the dbchanges migration
   `dbchanges/Common/DF/2026-5-7 Cron Checkin.sql` (table `CronJobExecutions`,
-  `executionTimeSeconds decimal(10,3)`, no `note` column).
+  `executionTimeSeconds decimal(10,3)`). **Correction (2026-08-25): prod `Common.CronJobExecutions`
+  does have a `note` TEXT column** — verified by writing to it and reading it back. `App_Framework`
+  never populates it, which makes it the practical place to park **temporary** step tracing for a
+  cron you are diagnosing (worker crons have no readable stdout — see
+  [Tracing a worker cron run in production](../../worker/workflows/tracing-a-worker-cron-run-in-production.md)).
+  Remove the tracing when you are done, and never write payloads or credentials into it.
+- **Don't confuse the two `note`s.** `cronInitialization()` writes `note = 'Started execution'` to
+  the **`Log`** table on `db_log` (the older `CRON`/`recordType` row), not to `CronJobExecutions`.
 - **Design decision:** 1.0 records executions by writing **directly to the shared DB**
   (`db_common`), *not* by POSTing from 1.0 to a 2.0 API endpoint. Per Jeff Cardinal, 1.0 code
   must not POST to 2.0 code; the direct shared-DB write is the sanctioned path (TRUE-78182).
 
 ## Gotchas
 
+- **⚠ A run skipped by the overlap guard leaves NO ROW — absence is ambiguous.**
+  `cronInitialization()` calls `exitIfProcessRunning()` **before** the `CronJobExecutions` INSERT,
+  and the `cronFinished(true)` that guard then calls does nothing because `cronLogId` was never
+  set. So a blocked run is indistinguishable from a run that never launched: **a missing row for
+  an expected slot means "skipped", not "hung"**, and a hung run looks different (it has
+  `dtCheckIn` with no `dtCheckOut`). `isProcessRunning()` matches **any** `ps -ef` line containing
+  the script path (it only excludes `/bin/sh` and its own pid), so a stray `tail -f` or editor on
+  that path blocks the cron indefinitely.
 - **`db_log` has no `CronJobExecutions` table.** A `cronFinished()` UPDATE that runs against the
   `db_log` connection throws on **every** 1.0 cron job. Both the INSERT and the UPDATE must
   target `db_common`.
@@ -60,6 +76,12 @@ row in `db_log` and of the per-job Sentry check-in monitors.
   logic. (Fixed 2026-07-06: consolidated back to one `db_common` INSERT/UPDATE pair.)
 
 ## Change history
+- 2026-08-25 — Corrected the schema note (prod `CronJobExecutions` **does** carry a `note` TEXT
+  column, unused by `App_Framework`, usable for temporary cron tracing) and separated it from the
+  `note = 'Started execution'` row `cronInitialization()` writes to `Log` on `db_log`. Recorded
+  that the overlap guard runs **before** the INSERT, so a skipped run writes no row at all and
+  cannot be told apart from "never launched" — and that `isProcessRunning()`'s `ps -ef` substring
+  match lets any process holding the script path block the cron. No code change. (bala)
 - 2026-07-06 — Fixed a merge (`origin/_production`) that reintroduced duplicate
   `CronJobExecutions` writes and pointed one `cronFinished()` UPDATE at `db_log` (where the
   table doesn't exist), which would have thrown on every 1.0 cron. Consolidated to one

package/knowledge/1.0/apps/worker/INDEX.md

@@ -11,9 +11,10 @@
 | [NetSuite Sales Order Sales Rep Sourcing (Staples & ODP EDI orders)](features/netsuite-sales-order-sales-rep-sourcing.md) | How the **sales rep** on a NetSuite Sales Order is determined for the two 1.0 `worker` EDI order-creation integrations (Staples cXML and Compass/ODP EDI). | worker/crons/toga2/compass/workflow/5_create_netsuite_sales_orders_from_office_depot_purchase_orders.php, worker/crons/sync/staples/sync_staples_cxml.php, test/@Mark/NetSuite/TRUE_80451_customer_salesrep_diag.php |
 | [NetSuite → TOGa Supply Per-Client Sync (thin wrappers)](features/netsuite-togasupply-per-client-sync.md) | Syncs NetSuite transactions (sales orders, purchase orders, invoices, item receipts, item fulfillments, inventory adjustments) into each TOGa Supply (2.0) clien | worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_canon.php, worker/crons/toga2/netsuite/sync_togasupply_elite.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql, library/app/api/toga2.php, library/app/api/netsuite/rest.php, library/app/framework.php, library/app/systemmonitor/netsuiteintegration.php, test/@srija/Elite Testing/Service Requests/test_sync_togasupply_elite_section.php, test/@srija/Elite Testing/Service Requests/test_diagnose_togasupply_elite.php |
 | [OneUptime Server monitor + disk/memory hygiene on the 1.0 worker EB host](features/oneuptime-server-monitor-host-hygiene.md) | The 1.0 `agilant-worker` EB environment runs on the **legacy Amazon Linux 1 PHP 7.2 platform** (Apache httpd/prefork, s3fs mounts, cron) and repeatedly went dow | worker/.ebextensions/040_disk_memory_hygiene.config, worker/.ebextensions/045_oneuptime_agent.config, worker/ebs/cron.worker.php, worker/ebs/mount-s3fs-folders.php, worker/ebs/apache_settings.php, worker/ebs/setup_phpini.php |
-| [OneUptime external uptime monitoring for 1.0 workers](features/oneuptime-worker-uptime-monitoring.md) | Every 1.0 worker box self-reports its liveness to an external OneUptime monitor once per minute by curl-POSTing to a per-worker "Incoming Request" heartbeat URL | library/app/worker.php, worker/crons/worker/worker_heartbeat.php |
+| [OneUptime external uptime monitoring for 1.0 workers](features/oneuptime-worker-uptime-monitoring.md) | Every 1.0 worker box self-reports its liveness to an external OneUptime monitor once per minute by curl-POSTing to a per-worker "Incoming Request" heartbeat URL | library/app/worker.php, worker/crons/worker/worker_heartbeat.php, worker/ebs/cron.worker.php, worker/.ebextensions/045_oneuptime_agent.config |
 | [Prudential: Send Shipments for the Day report (daily cron)](features/send-shipments-for-the-day.md) | Daily cron (9:00 PM) that emails Prudential and Dell stakeholders an Excel report of all devices shipped that day, including tracking number, serial number, emp | worker/crons/notifications/reports/send_shipments_for_the_day.php |
 | [Staples cXML Order Import (SFTP → NetSuite)](features/staples-cxml-order-import.md) | `sync_staples_cxml.php` is an **hourly** cron (runs at **:45**) that imports Staples cXML purchase orders from SFTP into NetSuite as Sales Orders, then writes a | worker/crons/sync/staples/sync_staples_cxml.php |
 | [Diagnosing frozen 1.0 worker cron check-ins (Sentry "missed" flood)](workflows/diagnosing-frozen-cron-checkins.md) | When 1.0 worker cron timestamps freeze and Sentry project `worker1` fills with **`missed`** check-ins, the intuitive diagnosis — a wedged `App_Framework::isProc | worker/.ebextensions/cron.config, library/app/worker.php |
 | [isFulfillable Multi-Client Backfill (all togasupply clients)](workflows/isfulfillable-multi-client-backfill.md) | One-time backfill that catches up `Items.isFulfillable` on **existing** items across **all 17 togasupply clients** (AIG, Broward Sheriff, Canon, Endeavor Health | worker/crons/toga2/netsuite/backfill_isfulfillable_all_clients.php, library/app/api/toga2.php |
 | [Onboarding a Client to the NetSuite TOGa Supply Sync](workflows/onboarding-client-to-netsuite-togasupply-sync.md) | How to add a new TOGa 2 client to the per-client NetSuite → TOGa Supply importer (`worker/crons/toga2/netsuite/`). | worker/crons/toga2/netsuite/sync_togasupply.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_elite.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql, dbchanges2/_modules/netsuite/2026-08-05 - CLEAN NETSUITE CLINET.SQL |
+| [Tracing a 1.0 worker cron run in production (no stdout, silent skips)](workflows/tracing-a-worker-cron-run-in-production.md) | How to answer *"did this cron actually run, and what did it do?"* on the 1.0 `worker` tier, where **there is no usable stdout** and **a skipped run leaves no tr | worker/ebs/cron.worker.php, worker/schedules/cron.worker.sync.json, library/app/framework.php |

package/knowledge/1.0/apps/worker/architecture.md

@@ -6,8 +6,8 @@ project: Worker
 client: shared
 type: architecture
 status: active
-updated: 2026-07-29
-owners: [jcardinal, sking]
+updated: 2026-08-24
+owners: [jcardinal, sking, bala]
 files:
   - worker/index.php
   - worker/_/app/framework.php
@@ -20,6 +20,8 @@ files:
 related:
   - ../library/architecture.md
   - ../dbchanges/workflows/authoring-and-shipping-sql-files.md
+  - ./features/oneuptime-worker-uptime-monitoring.md
+  - ../../2.0/apps/worker2/features/worker-fleet-role-assignment-monitor.md
 ---
 
 ## Summary
@@ -44,6 +46,7 @@ which **self-elects a distinct role** (`notification`, `database`, `infrastructu
 under `crons/`; the schedule (cron registration) is the source of truth for what runs — a script
 that isn't scheduled never executes. Client integrations live under `crons/toga2/<client>/`.
 Use prepared statements for all SQL; never interpolate input.
+A worker that fails deploy-time role election runs NO crons while EB still reads healthy.
 
 ## How a job becomes a cron (the dispatch pipeline)
 
@@ -108,6 +111,32 @@ is the most important and least obvious part of the architecture.
 > Net effect: roles are a **claim-the-first-free-slot pool**, and a dead worker's role is
 > reclaimed by its replacement within minutes. There is no static instance→role mapping to edit.
 
+### The failure mode: NO NAME = NO CRONTAB (and it is silent)
+
+The election is the **single point of failure for an entire instance**, and the architecture gives
+it no second chance:
+
+- The claim runs **only at DEPLOY time** — `ebs/cron.worker.php` is written as the EB `appdeploy`
+  enact hook. **It never retries.** An instance that fails to claim stays nameless until the next
+  deploy or until something terminates it.
+- The claim writes `/etc/worker-role` and **only then** appends `cron.worker.<role>.json` to the
+  crontab. So a nameless instance has **no role crontab at all** — it runs *nothing*, not even
+  `worker_heartbeat.php`, while **EB reports the environment healthy**. The box is up; it just
+  does no work.
+- **The claim's DB connect is `@mysqli_connect(...)` — error-suppressed.** A boot-time connect
+  failure to `Vision_Log` therefore leaves the instance nameless **with no log line anywhere**.
+  This is the prime suspect for the 2026-08-24 incident (2 of 7 workers roleless for ~3 hours),
+  and it violates the no-`@`-suppression rule in `rules/toga/common/coding-style.md`. Removing the
+  `@` and logging the failure is the recommended fix.
+- `Workers` **only ever holds instances that SUCCEEDED**, so a failed claim leaves no row —
+  **absence is the only evidence**, and detecting it requires the AWS instance list.
+
+Because every OneUptime monitor on this tier is keyed by **role** rather than by instance, none of
+them can see this (see
+[the blind spot](./features/oneuptime-worker-uptime-monitoring.md)). External coverage now comes
+from the 2.0
+[Worker Fleet Role-Assignment Monitor](../../2.0/apps/worker2/features/worker-fleet-role-assignment-monitor.md).
+
 ## Anatomy of a cron script
 
 Every script is self-contained and follows this boilerplate:
@@ -208,6 +237,10 @@ web face (`mvc/` GET routes for login/logout/404); the tier's real work is the c
 
 ## Conventions & gotchas
 
+- **A worker that fails role election is silently dead, not degraded.** No name → no crontab →
+  the instance runs nothing while EB reads healthy, and the `@`-suppressed `mysqli_connect` in
+  `ebs/cron.worker.php` logs nothing. Never treat "EB is green" as evidence the fleet is working;
+  check `Vision_Log.Workers` against the actual EB instance list.
 - **`dbchanges` auto-apply is branch-gated, and `_production` is NOT auto-applied.**
   `crons/infrastructure/execute_dbchanges.php` runs every 2 minutes but only for branches matching
   `_%`, deriving the env by stripping the leading `_` and requiring `config.<env>.ini` in the

package/knowledge/1.0/apps/worker/features/oneuptime-worker-uptime-monitoring.md

@@ -6,12 +6,17 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-07-13
-owners: ["jcardinal"]
+updated: 2026-08-24
+owners: ["jcardinal", "bala"]
 files:
   - library/app/worker.php
   - worker/crons/worker/worker_heartbeat.php
-related: ["oneuptime-server-monitor-host-hygiene"]
+  - worker/ebs/cron.worker.php
+  - worker/.ebextensions/045_oneuptime_agent.config
+related:
+  - ./oneuptime-server-monitor-host-hygiene.md
+  - ../architecture.md
+  - ../../../2.0/apps/worker2/features/worker-fleet-role-assignment-monitor.md
 ---
 
 ## Summary
@@ -57,6 +62,31 @@ Database, Infrastructure, TOGa, TOGa Desk, and Catalog, then importing each into
 manually. To add a new worker's monitor, clone an existing monitor export the same way and
 register the resulting heartbeat URL in `App_Worker::$oneUptimeEndpoints`.
 
+## THE BLIND SPOT — every monitor here is keyed by ROLE, so a ROLELESS instance is invisible
+
+Load-bearing limitation, learned the hard way (2026-08-24: **2 of the 7 production workers ran
+roleless for ~3 hours undetected**). This layer, and the Server monitors in
+[host hygiene](./oneuptime-server-monitor-host-hygiene.md), are keyed by `workerName` — **14
+monitors, none keyed by instance**:
+
+- `worker_heartbeat.php` only pings `App_Worker::$oneUptimeEndpoints[$row['workerName']]` for
+  the row matching **its own** `instanceId`. No row → no ping target → it pushes nothing.
+- The `045_oneuptime_agent.config` Server agent reads `/etc/worker-role` and **exits clean when
+  it is empty.**
+
+So an instance that never claimed a role reports to nothing — and because every role is still
+owned by *someone*, **no role-keyed monitor is missing a ping either.** The fleet reads 100%
+healthy while a box does zero work (role assignment happens **only at deploy time**, and no role
+means the role's `cron.worker.<role>.json` is never appended to the crontab, so the instance runs
+**nothing**).
+
+`Vision_Log.Workers` only ever holds instances that **succeeded** in claiming a role, so the
+failure leaves no row and no log line — **absence is the only evidence**, and detecting an
+absence requires the AWS instance list, which nothing on this tier consults. That gap is what
+the 2.0
+[Worker Fleet Role-Assignment Monitor](../../../2.0/apps/worker2/features/worker-fleet-role-assignment-monitor.md)
+now closes from outside the fleet. **Do not try to close it with another role-keyed monitor.**
+
 ## Gotchas
 
 - Do **not** hardcode the heartbeat URLs/UUIDs anywhere but `App_Worker::$oneUptimeEndpoints`
@@ -67,6 +97,13 @@ register the resulting heartbeat URL in `App_Worker::$oneUptimeEndpoints`.
   pending cleanup — the live source of truth is the `App_Worker::$oneUptimeEndpoints` map.
 
 ## Change history
+- 2026-08-24 — Documented **the blind spot**: all 14 monitors on this tier are keyed by
+  `workerName`, so an instance that claimed **no** role pings nothing and leaves every role-keyed
+  monitor still green — which is how 2 of 7 production workers ran roleless for ~3 hours
+  undetected. Recorded that `Vision_Log.Workers` only holds successful claims (absence is the
+  only evidence) and that the gap is now covered from outside the fleet by the 2.0
+  [Worker Fleet Role-Assignment Monitor](../../../2.0/apps/worker2/features/worker-fleet-role-assignment-monitor.md).
+  (bala)
 - 2026-07-13 — Added external OneUptime heartbeat push for all 1.0 workers on top of the
   existing internal DB-heartbeat mechanism; endpoints keyed by workerName in
   `App_Worker::$oneUptimeEndpoints`. (jcardinal)

package/knowledge/1.0/apps/worker/workflows/tracing-a-worker-cron-run-in-production.md

@@ -0,0 +1,124 @@
+---
+title: Tracing a 1.0 worker cron run in production (no stdout, silent skips)
+framework: "1.0"
+repo: worker
+project: Worker
+client: shared
+type: workflow
+status: active
+updated: 2026-08-25
+owners: ["bala"]
+files:
+  - worker/ebs/cron.worker.php
+  - worker/schedules/cron.worker.sync.json
+  - library/app/framework.php
+related:
+  - ../architecture.md
+  - ./diagnosing-frozen-cron-checkins.md
+  - ../../library/features/cron-execution-monitoring.md
+  - ../../../1.0/standards/backend-php.md
+---
+
+## Summary
+How to answer *"did this cron actually run, and what did it do?"* on the 1.0 `worker` tier, where
+**there is no usable stdout** and **a skipped run leaves no trace at all**. Written from a
+production diagnostic session on
+`worker/crons/toga2/compass/workflow/3a_import_office_depot_purchase_orders.php`; the mechanics
+apply to every cron on the tier.
+
+Use this before you conclude a cron "never fired" — three different situations look identical
+from the outside.
+
+## Step 1 — get the real schedule from `schedules/*.json`, never from the file
+The `.php` file's own header comment is not authoritative and is routinely stale. So are skills,
+docs and tickets. `worker/schedules/cron.<env>[.<role>].json` is the only source of truth.
+
+Worked example: the entry *"Download EDI From S3 Files & Create PO - EDI Office Depot"* in
+`cron.worker.sync.json` is `*/5 * * * *` — **every 5 minutes**. The cron file's header said
+"EVERY HOUR" (corrected 2026-08-25) and at least one skill still says hourly. A wrong assumed
+frequency makes you read a normal gap as an outage.
+
+Also check whether the job is registered in the env you are testing in at all: this one is **not**
+in `cron.beta.json`, so beta never runs it on a schedule. Nothing on beta is evidence about it.
+
+## Step 2 — echo/print goes nowhere useful, so do not debug with it
+The crontab line is assembled in `worker/ebs/cron.worker.php`, and the two blocks that build it
+differ:
+
+| Schedule file | Command written | Where output goes |
+|---|---|---|
+| base `cron.<env>.json` | `php /var/www/html/crons/<script>` | **nowhere** — the ` > /var/www/cache/WORKER_ERROR_$RANDOM` redirect is **commented out** |
+| role `cron.worker.<role>.json` | `php /var/www/html/crons/<script> > /var/www/cache/WORKER_ERROR_$RANDOM` | a **randomly named** file under `/var/www/cache/` on whichever of the 7 instances ran it |
+
+Either way you cannot go and read it: base-schedule output is discarded outright, and a
+role-schedule run leaves an unpredictable filename on a box in an autoscaled fleet. **Treat
+worker cron stdout as write-only.**
+
+## Step 3 — put temporary tracing somewhere you can query
+What worked: write timestamped step markers into the **`note`** column of the run's
+`Common.CronJobExecutions` row (the row `App_Framework::cronInitialization()` already inserted),
+then read them back over the DB tooling from your desk. `dtCheckIn`, `dtCheckOut` and
+`executionTimeSeconds` on the same row give you the shape of the run for free.
+
+Rules: keep it to a few markers, and **remove the tracing once diagnosed** (it was removed in
+this session). Never write payloads or credentials into `note`.
+
+## Step 4 — read `Common.CronJobExecutions` to tell "ran", "hung" and "skipped" apart
+Query the legacy env's `Common.CronJobExecutions` filtering `job LIKE` the script path:
+
+- **`dtCheckIn` + `dtCheckOut` + `executionTimeSeconds`** — it ran and finished; the duration
+  tells you whether it fits inside its schedule slot.
+- **`dtCheckIn` with no `dtCheckOut`** — it started and never completed (crash, timeout, or the
+  instance went away).
+- **no row for a slot you expected** — the run was **skipped by the overlap guard**, not "never
+  fired". A blocked run writes **nothing**: `cronInitialization()` calls
+  `exitIfProcessRunning()` **before** the INSERT, and the `cronFinished(true)` it then calls is a
+  no-op because `cronLogId` is unset. See
+  [Cron Execution Monitoring](../../library/features/cron-execution-monitoring.md).
+
+### The overlap guard is a `ps -ef` substring match — anything can block it
+`App_Framework::isProcessRunning()` returns true for **any** `ps -ef` line containing the script
+path (it only excludes `/bin/sh` lines and its own pid). So a developer's `tail -f`, an editor, or
+any shell holding that path in its command line **blocks the cron indefinitely** while looking
+like a legitimate overlap. Check `ps -ef` for what is actually holding the name before assuming a
+long-running instance of the job itself.
+
+## Step 5 — if runs overlap, look at what the job does before its real work
+A job whose own runtime exceeds its schedule interval silently loses most of its slots to the
+guard. Measure the phases, not the total: in the ODP case an instrumented run showed the S3
+listing alone returning **138,472 objects and taking 32 seconds** before any PO was touched, with
+each PO then costing roughly 40 seconds — comfortably past a 5-minute schedule. Prefixes the job
+skips while processing are still listed, so accumulated `SENT/` and `OUTBOX/` objects inflate
+every run.
+
+## Deploy-time trap: a PHP 8-only syntax kills the whole cron with no visible error
+This tier runs **PHP 7.2**, so a **PHP 8 named argument** (`someFunction(items: $x)`) is a
+**parse error**, not a runtime warning:
+
+```
+PHP Parse error: syntax error, unexpected ':', expecting ',' or ')'
+```
+
+The script then does nothing at all on every tick — and per Steps 2 and 4 you will see no output
+and no `CronJobExecutions` row, i.e. it looks exactly like "the cron never fired". Typed
+parameters and return types (`: void`, `: array`, `: bool`, `?array`, `?object`, `object $x`) are
+PHP 7.0/7.1 and **are** safe here; they are already used in
+`worker/crons/toga2/compass/backfill_items_isfulfillable.php` and
+`update_item_fulfillments_in_netsuite.php`.
+
+`1.0/standards/backend-php.md` already forbids named arguments in 1.0 — but a workspace
+`CLAUDE.md` / `.github/copilot-instructions.md` that says *"use PHP 8 Named Parameters"* applies
+to **2.0 only**. The 1.0 standard wins in `library/`, `worker/` and every other 1.0 app.
+
+**Do not read a PHP 8 function call in existing code as proof the tier is PHP 8.**
+`library/app/systemmonitor/500error.php` uses `str_contains()`; that file is not a version
+signal, it is a latent bug.
+
+## Change history
+- 2026-08-25 — Documented from a prod diagnostic session on the Compass ODP 850 importer: the
+  schedule JSON (not the file header) is authoritative, worker cron stdout is unreadable (base
+  block's `WORKER_ERROR` redirect commented out, role block writes a random filename),
+  `Common.CronJobExecutions.note` is the practical place for temporary tracing, a run skipped by
+  the `ps -ef` overlap guard writes **no row at all** (guard runs before the INSERT), the guard
+  matches any process holding the script path, and a PHP 8 named argument parse-errors the whole
+  cron silently on this PHP 7.2 tier. (bala)

package/knowledge/2.0/apps/_underscore/features/per-client-database-connections.md

@@ -6,8 +6,8 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-08-17
-owners: ["dfranks", "jcardinal", "mhammontree", "apeterson", "kyalamarthi"]
+updated: 2026-08-24
+owners: ["dfranks", "jcardinal", "mhammontree", "apeterson", "kyalamarthi", "bala"]
 files:
   - _underscore/Database.php
   - _underscore/Model.php
@@ -177,6 +177,16 @@ here — they live in `Config/*.ini`.)
 - Related 1.0 analogue: the legacy `App_` worker has the same hazard writing to `Logs.API`
   (`db_logs`) — the laptop trap there is documented separately in the worker NetSuite bootstrap
   notes.
+- **`_Database::register()` is LAZY — it stores connection config and never opens a socket.**
+  The connect happens on first use. So adding a static alias registration to an app's `_.php`
+  bootstrap is **safe across every environment**, even one whose config group points at a
+  localhost that lacks the schema: nothing fails until something actually queries that alias.
+  This is what makes "register the alias globally, use it in one cron" a two-line change rather
+  than a per-environment config exercise (worked example: `DB_VISION_LOGS` in
+  [the Worker Fleet Role-Assignment Monitor](../../worker2/features/worker-fleet-role-assignment-monitor.md)).
+  The corollary is the trap the rest of this doc describes: because registration is free and
+  silent, a **bad** registration also stays silent until the request that needs it blows up with
+  `Unknown database`.
 
 ## WITHDRAWN — the "alias-keyed `$_modelCache` cross-tenant leak" hypothesis (2026-08-17)
 
@@ -210,6 +220,11 @@ also hits `_modules/<module>/`** — is recorded in
 
 ## Change history
 
+- 2026-08-24 — Recorded that **`_Database::register()` is lazy** (stores config, opens no
+  socket; the connect happens on first use), so a static alias registration in an app's `_.php`
+  is safe in every environment including a localhost config that lacks the schema — and that
+  this is also why a *bad* registration stays silent until first query. Verified while adding
+  `DB_VISION_LOGS` to worker2. (bala)
 - 2026-08-17 (later pass) — **WITHDRAWN, supersedes the entry below.** The alias-keyed
   `$_modelCache` cross-tenant-leak hypothesis is **not supported** and is no longer a live security
   concern. The six `c_` columns are **AIG's own**: `_modules/netsuite/2026-07-10a -

package/knowledge/2.0/apps/worker2/INDEX.md

@@ -17,7 +17,7 @@
 | [ClickUp Work Type Automation (Committed / Conditional / Stretch)](features/clickup-work-type-automation.md) | The ClickUp webhook handler (`_Worker_Clickup`) automatically maintains each task's **Work Type** custom field — `Committed`, `Conditional`, or `Stretch` — base | worker2/Worker/Clickup.php, worker2/Tests/Worker/ClickupWorkTypeTest.php |
 | [Compass VIP Support Importer (worker2)](features/compass-vip-support-importer.md) | A worker2 action that ingests Compass's quarterly VIP spreadsheet and assigns each VIP user's support technician by setting `Users.c_supportedByUserId` in `Clie | worker2/Worker/Client/Compass/VipSupport.php |
 | [Creating Worker Actions](features/creating-worker-actions.md) | How to add a new callable Worker action — a PHP class whose `public static` methods are invoked as background jobs (via webhook, cron, or `_Worker::runTask()`). | worker2/Worker/, worker2/Controller/Index.php, _underscore/Worker.php |
-| [Cross-account AWS access for worker2 crons (_Component_Aws_Workloads)](features/cross-account-aws-access.md) | `_Component_Aws_Workloads` is the **standard, and only sanctioned, way any new worker2 cron obtains AWS access** — for any account, any region, any AWS SDK clie | worker2/Component/Aws/Workloads/Workloads.php, worker2/Config/production.ini, worker2/Worker/Infrastructure/CloudWatch.php |
+| [Cross-account AWS access for worker2 crons (_Component_Aws_Workloads)](features/cross-account-aws-access.md) | `_Component_Aws_Workloads` is the **standard, and only sanctioned, way any new worker2 cron obtains AWS access** — for any account, any region, any AWS SDK clie | worker2/Component/Aws/Workloads/Workloads.php, worker2/Config/production.ini, worker2/Worker/Infrastructure/CloudWatch.php, worker2/Worker/Monitor/Fleet.php, library/app/worker.php, library/app/cloud.php |
 | [Elastic Beanstalk health monitor → OneUptime (ElasticBeanstalkHealth)](features/elastic-beanstalk-health-monitor.md) | `_Worker_Infrastructure_CloudWatch::ElasticBeanstalkHealth()` is a worker2 cron that reads each Elastic Beanstalk (EB) environment's **enhanced-health** status | worker2/Worker/Infrastructure/CloudWatch.php |
 | [Elite Freshservice Sync (worker2)](features/elite-freshservice-sync.md) | `_Worker_Elite` processes Freshservice webhook events and syncs them into TOGA 2. | worker2/Worker/Elite.php, worker2/Config/dev-kmaramreddy-laptop.ini |
 | [Error Escalation Cron (Errors::Escalate → ClickUp / email)](features/error-escalation-cron.md) | `_Worker_Infrastructure_Errors::Escalate` (renamed from `SyncWithClickup`) is the sole owner of **escalation, de-escalation, ClickUp ticketing, reminders, busin | worker2/Worker/Infrastructure/Errors.php, worker2/Worker/Notification/Email.php, worker2/Worker/Notification/EmailTemplate.php, worker2/Worker/Client/True.php, worker2/Worker/Clickup/ErrorTask.php, worker2/Worker/Clickup.php, worker2/Controller/Index.php, worker2/Config/production.ini, _underscore/Model/Core/Logs/Issue.php, dbchanges2/Core/2026-07-30a - Error escalation cron job.sql, dbchanges2/Logs/2026-08-03a - Issue clickupPriority.sql, dbchanges2/Core/2026-08-04a - Error neglect digest cron job.sql |
@@ -47,6 +47,7 @@
 | [Centralized Tracking-Status Refresh (worker2 Sync cron, all clients)](features/tracking-status-refresh.md) | A single platform-wide cron keeps `TrackingNumbers.status` current for **every provisioned client** until each shipment reaches a terminal state, by polling Fed | worker2/Worker/Sync/TrackingNumbers.php, _underscore/Component/Library/Carriers/Fedex/Fedex.php, _underscore/Component/Library/Carriers/Usps/Usps.php, _underscore/Component/Library/Carriers/Ups/Ups.php, _underscore/Model/Client/TrackingNumber.php, dbchanges2/Client/2026-07-30 - TrackingNumbers_RefreshColumns.sql, dbchanges2/Client/2026-07-30 - Acl_TrackingStatus_Grant.sql, dbchanges2/Client_Tdsynnex/2026-07-30 - Apis_TdsynnexKey.sql, dbchanges2/Core/2026-07-30 - TrackingRefresh_Cron.sql |
 | [VAPI Webhook Handler (worker2 — AI-BDR end-of-call processing)](features/vapi-webhook-handler.md) | `_Worker_Vapi` ([worker2/Worker/Vapi.php](worker2/Worker/Vapi.php)) is the **PHP side of the AI-BDR call loop** — the webhook that receives VAPI's end-of-call r | worker2/Worker/Vapi.php, worker2/Worker/Ai/Bdr/Vapi.php, worker2/Controller/Index.php |
 | [WJE Freshservice Sync (worker2)](features/wje-freshservice-sync.md) | WJE ("WJE IT", helpdesk `wje.freshservice.com`) is a **Freshservice**-based help-desk client whose tickets, contacts, assets, groups, categories, and canned res | worker2/Worker/Wje.php, _underscore/Component/Api/Wje/Wje.php, _underscore/Model/Wje/Ticket.php, _underscore/Model/Wje/TicketNote.php, _underscore/Model/Wje/Contact.php, _underscore/Model/Wje/Unit.php, _underscore/Model/Wje/TicketTeam.php, _underscore/Model/Wje/TicketCategory.php, _underscore/Model/Wje/AssetType.php, _underscore/Model/Wje/PredefinedReply.php, library/app/api/wje.php, worker/crons/toga2/wje/import_supporting_records.php, worker/crons/toga2/wje/sync_togasupply_wje.php, worker/crons/notifications/reports/wje/wje_common.php, library/app/systemmonitor/wje.php, dbchanges2/Client_Wje/2024-10-04 - WjeOnboarding.sql |
+| [1.0 Worker Fleet Role-Assignment Monitor (Monitor/Fleet/RoleAssignment)](features/worker-fleet-role-assignment-monitor.md) | `_Worker_Monitor_Fleet::RoleAssignment()` is a **2.0 worker2 cron that watches the 1.0 `worker` fleet from the outside**. | worker2/Worker/Monitor/Fleet.php, worker2/_.php, dbchanges2/Core/2026-08-24 - Worker Role Assignment Monitor.sql, worker/ebs/cron.worker.php, worker/crons/worker/worker_heartbeat.php, library/app/worker.php |
 | [PHP Runtime Upgrade on Elastic Beanstalk (worker2 8.3 → 8.5 + PhpSpreadsheet 1.x → 3.x)](workflows/php-runtime-upgrade-dependency-audit.md) | The procedure used to move worker2 from **PHP 8.3 to PHP 8.5** on Elastic Beanstalk, and the dependency work that had to land first. | worker2/composer.json, worker2/composer.lock, worker2/Worker/Team/Sprint.php, worker2/Worker/Client/TowFoundation/ProcessReceipts.php, worker2/Worker/Forecast/Import.php |
 | [Running worker2 locally against real NetSuite (and what EB does instead)](workflows/running-worker2-locally.md) | How to boot **worker2** on a developer machine and run a real worker action against the **live NetSuite** account and a **local** `Forecast` database. | worker2/index.php, worker2/composer.json, worker2/.ebextensions/php_include_underscore.config, worker2/.ebextensions/git.json, worker2/.ebextensions/git.php, worker2/Config/production.ini, _underscore/Component/Api/Netsuite/Netsuite.php, dbchanges2/Logs_Client/2026-04-08_BLANK_CLIENT_LOGS_DATABASE.SQL |
 | [Ticket → ClickUp Pseudocode Planning (Talos-grounded)](workflows/ticket-to-pseudocode-planning.md) | A repeatable procedure for turning a ClickUp ticket into a reviewed, formatted implementation plan posted back to the ticket's `📝 Pseudocode` custom field. | test/@dave/clickup_md2delta.js |

package/knowledge/2.0/apps/worker2/features/cross-account-aws-access.md

@@ -6,18 +6,22 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-11
-owners: [jcardinal]
+updated: 2026-08-24
+owners: [jcardinal, bala]
 files:
   - worker2/Component/Aws/Workloads/Workloads.php
   - worker2/Config/production.ini
   - worker2/Worker/Infrastructure/CloudWatch.php
+  - worker2/Worker/Monitor/Fleet.php
+  - library/app/worker.php
+  - library/app/cloud.php
 related:
   - ../../_underscore/features/component-model-namespace-registration.md
   - ../../_underscore/features/config-group-access.md
   - ../../_underscore/features/cloud-s3-helpers.md
   - ./oneuptime-worker2-monitoring.md
   - ./elastic-beanstalk-health-monitor.md
+  - ./worker-fleet-role-assignment-monitor.md
 ---
 
 ## Summary
@@ -79,7 +83,8 @@ region is only a runtime argument to `client()`, never a reason for a second rol
 - **654654170868** — production hub. Home region `us-east-1`; also used in `us-west-2` and
   `eu-west-1`. worker2 runs here on Elastic Beanstalk; its instance profile role
   `ElasticBeanstalk-EC2-Instance-Profile` is the caller identity.
-- **502614707982** — legacy account, `us-west-2`.
+- **502614707982** — legacy account, `us-west-2`. **This is where the 1.0 `worker` fleet
+  actually runs** — see the account-confusion gotcha below.
 - **975050298201** — non-production account, `us-east-1`.
 
 Each `WorkloadsRuntime` role has broad service permissions (`elasticbeanstalk:*`, `ec2:*`,
@@ -144,8 +149,33 @@ code yet. New code must not adopt the static key.
 **Never record the access-key values in knowledge.** They exist in `production.ini`; that is
 their only home.
 
+### Which account the 1.0 worker fleet lives in (easy and expensive to get wrong)
+
+The 1.0 `worker` EC2 instances are in **502614707982**, `us-west-2` (EB environment
+`agilant-worker`, confirmed in the EC2 console).
+
+**`App_Worker::AWS_WORKER_QUEUE_URL` in `library/app/worker.php` points at 654654170868 and
+misleads** — that account holds only the **SQS queue the 1.0 workers read from**, not the
+instances. Anything that needs to *see* the worker boxes (EB, EC2, health) must target
+**502614707982**.
+
+worker 1.0 authenticates there with a **long-lived hardcoded IAM access key in
+`library/app/cloud.php`** — a committed credential that should be rotated and moved to a role
+(value deliberately not recorded here; see [Legacy static key](#legacy-static-key--being-retired)).
+worker2 does **not** use it: it assumes `WorkloadsRuntime` cross-account from its EB instance
+role, which is exactly the migration this component exists to enable. First worker2 consumer in
+that account is the
+[Worker Fleet Role-Assignment Monitor](./worker-fleet-role-assignment-monitor.md).
+
+**Scope EB reads by `EnvironmentName`, never account-wide.** 502614707982 also runs
+`agilant-worker-alpha` and `agilant-worker-beta` as **separate EB environments that have no role
+registry at all**. An account-level EC2 scan would sweep them in and misreport them.
+
 ## Gotchas / known issues
 
+- **Do not assume the 1.0 worker fleet is in the 868 production hub.** `AWS_WORKER_QUEUE_URL`
+  (654654170868) is the SQS queue only; the instances are in the legacy account 502614707982,
+  `us-west-2`. See [the section above](#which-account-the-10-worker-fleet-lives-in-easy-and-expensive-to-get-wrong).
 - Do not construct `StsClient` or pass explicit credentials by hand in new crons — the whole
   point is that the instance role is the caller and the component owns the assume-role step.
 - The instance-role-as-caller only works **on the EB tier**. Off-tier (e.g. a local CLI run)
@@ -161,6 +191,15 @@ their only home.
   see [creating-worker-actions.md](./creating-worker-actions.md).
 
 ## Change history
+- 2026-08-24 — Pinned down **which account the 1.0 `worker` fleet actually runs in**:
+  **502614707982** (`us-west-2`, EB env `agilant-worker`), **not** the 654654170868 production hub
+  that `App_Worker::AWS_WORKER_QUEUE_URL` implies — that account holds only the SQS queue the 1.0
+  workers read from. Recorded that worker 1.0 reaches it with the hardcoded static key in
+  `library/app/cloud.php` (rotate + move to a role) while worker2 assumes `WorkloadsRuntime` from
+  its instance role, and that EB reads there must be scoped by `EnvironmentName` because
+  `agilant-worker-alpha`/`-beta` are separate registry-less environments in the same account.
+  Found while building the
+  [Worker Fleet Role-Assignment Monitor](./worker-fleet-role-assignment-monitor.md). (bala)
 - 2026-08-11 — Noted that `ElasticBeanstalkHealth`'s `$environments` must be typed `object|array`
   (not `array`): the nested JSON object arrives as `stdClass` because the dispatcher casts only
   the top level of `CronJobs.parameters`, so a strict `array` throws a `TypeError` at dispatch.

package/knowledge/2.0/apps/worker2/features/oneuptime-worker2-monitoring.md

@@ -139,6 +139,35 @@ POSTed JSON body is addressed via the `requestBody` prefix — e.g.
 - Docs: https://oneuptime.com/docs/monitor/incident-alert-templating and
   https://oneuptime.com/docs/en/monitor/incoming-request-monitor
 
+### The monitor IMPORT JSON — exact shape (verified 2026-08-24)
+
+Monitors are provisioned by importing a **resource-export JSON**. The importer is strict and
+its error messages are the only documentation, so the verified shape:
+
+```
+{ fileType: "oneuptime-resource-export", schemaVersion: 1, resourceType: "Monitor", items: [ … ] }
+```
+
+- **`monitorType` must be the UI DISPLAY name, with a space — `"Incoming Request"`, NOT
+  `"IncomingRequest"`.** The importer rejects the concatenated form and helpfully lists the
+  valid values in the error.
+- Criteria nest deeply, each level carrying a `_type` tag:
+  `monitorSteps{_type:"MonitorSteps", value:{ monitorStepsInstanceArray:[ {_type:"MonitorStep",
+  value:{ monitorCriteria:{_type:"MonitorCriteria", value:{ monitorCriteriaInstanceArray:[ … ] }}}} ]}}`
+- Each **criterion instance** carries `monitorStatusId`, `filterCondition` (`All`/`Any`),
+  `filters[]`, `incidents[]`, `isEnabled`, `name`, `description`.
+- **Timing filters use `checkOn: "Incoming Request"`** with `filterType`
+  `"Not Recieved In Minutes"` / `"Recieved In Minutes"` (OneUptime's own misspelling) —
+  **not** `checkOn: "Request Body"`, which is only for the string-token matching above.
+- `incidentSeverityId` is wrapped: `{_type: "ObjectID", value: "…"}`.
+- There is **no `projectId`** in the export, and **status/severity ObjectIDs are
+  project-specific** — clone them out of an existing monitor's export rather than inventing them.
+
+**Practical fix for the [activation-ordering trap](#gotchas--known-issues):** instead of
+importing the monitor twice (body criteria first, absence criteria after the first push), import
+the absence criteria **in the same file with `isEnabled: false`** and simply flip them on once a
+push has landed.
+
 ### Heartbeat / cron cadence timing
 
 The 10-min-Degraded / 15-min-Offline heartbeat thresholds pair with a **5-minute** cron
@@ -327,6 +356,16 @@ check for another client.
   monitors must pass the region (see [Cloud S3 helpers](../../_underscore/features/cloud-s3-helpers.md)).
 
 ## Change history
+- 2026-08-24 — Documented the **monitor import-JSON schema** the runbook previously described
+  only conceptually: the `oneuptime-resource-export` envelope, `monitorType` must be the
+  spaced display name `"Incoming Request"`, the `_type`-tagged
+  `monitorSteps → monitorStepsInstanceArray → monitorCriteria → monitorCriteriaInstanceArray`
+  nesting, per-criterion fields, timing filters on `checkOn: "Incoming Request"` (not
+  `"Request Body"`), the `ObjectID`-wrapped `incidentSeverityId`, and that status/severity
+  ObjectIDs are project-specific with no `projectId` in the export. Added the practical dodge for
+  the activation-ordering trap: import the absence criteria with `isEnabled: false` and flip them
+  on after the first push, instead of importing twice. Verified while provisioning the
+  [Worker Fleet Role-Assignment Monitor](./worker-fleet-role-assignment-monitor.md). (bala)
 - 2026-08-20 — Added the **SFTP monitor** shape (phpseclib3 `rawlist` + age-cutoff count +
   decided token; login/list failure → `status:error` connectivity check) and its two
   runtime gotchas: `rawlist` returns **arrays** not objects, and the SFTP type constants are

package/knowledge/2.0/apps/worker2/features/worker-fleet-role-assignment-monitor.md

@@ -0,0 +1,175 @@
+---
+title: 1.0 Worker Fleet Role-Assignment Monitor (Monitor/Fleet/RoleAssignment)
+framework: "2.0"
+repo: worker2
+project: Worker
+client: shared
+type: feature
+status: active
+updated: 2026-08-24
+owners: ["bala"]
+files:
+  - worker2/Worker/Monitor/Fleet.php
+  - worker2/_.php
+  - dbchanges2/Core/2026-08-24 - Worker Role Assignment Monitor.sql
+  - worker/ebs/cron.worker.php
+  - worker/crons/worker/worker_heartbeat.php
+  - library/app/worker.php
+related:
+  - ./oneuptime-worker2-monitoring.md
+  - ./cross-account-aws-access.md
+  - ./elastic-beanstalk-health-monitor.md
+  - ../../../1.0/apps/worker/architecture.md
+  - ../../../1.0/apps/worker/features/oneuptime-worker-uptime-monitoring.md
+---
+
+## Summary
+
+`_Worker_Monitor_Fleet::RoleAssignment()` is a **2.0 worker2 cron that watches the 1.0
+`worker` fleet from the outside**. Every 5 minutes it lists the EC2 instances Elastic
+Beanstalk reports for the `agilant-worker` environment and **compares that list against the
+`Vision_Log.Workers` role registry**, then pushes three findings to a OneUptime Incoming
+Request monitor:
+
+| Finding | Meaning |
+|---|---|
+| `namelessInstances` | An instance is running in AWS but owns **no** row in `Workers` — it claimed no role, so it is running **no crons at all**. |
+| `unownedRoles` | A role in `App_Worker::$desiredWorkers` that **nobody** currently owns. |
+| `staleRoles` | A role whose `dtHeartbeat` has stopped advancing. |
+
+**Why it exists:** 2 of the 7 production workers ran **roleless for roughly 3 hours with
+nobody noticing**. Every pre-existing monitor is keyed by *role*, so an instance that never
+got a role is invisible to all of them **by design** (see
+[the blind spot](#the-blind-spot-why-none-of-the-14-existing-monitors-could-see-this)).
+Detection is now about 10 minutes.
+
+It follows the standard team token contract from
+[OneUptime push-metric monitors](./oneuptime-worker2-monitoring.md): the worker decides,
+OneUptime string-matches. `status = reporting | error`, `alarm = HIGH | OK`.
+
+## Key files / entry points
+
+- `worker2/Worker/Monitor/Fleet.php` — `_Worker_Monitor_Fleet`, action
+  **`Monitor/Fleet/RoleAssignment`** (routing verified: `Monitor/Fleet/RoleAssignment` maps to
+  `_Worker_Monitor_Fleet::RoleAssignment`).
+- `worker2/_.php` — adds `const DB_VISION_LOGS = 'Vision_Log'` and its
+  `_Database::register()` call on the **`[database1]`** connection group.
+- `dbchanges2/Core/2026-08-24 - Worker Role Assignment Monitor.sql` — the `Core.CronJobs`
+  row: `*/5 * * * *`, `parameters` NULL, **`isActive = 0`**.
+- AWS access is obtained by assuming `WorkloadsRuntime` via
+  `_Component_Aws_Workloads::client(ElasticBeanstalkClient::class, ...)` — see
+  [Cross-account AWS access](./cross-account-aws-access.md). No static keys.
+
+All tunables are **class constants with cron-parameter overrides**, so the schedule and the
+thresholds live in the `CronJobs` row rather than in a deploy.
+
+## How it works
+
+1. Assume `WorkloadsRuntime` in the **legacy** account (`502614707982`, `us-west-2`) and call
+   **`describeInstancesHealth`** on the EB environment **`agilant-worker`**.
+2. Read `Vision_Log.Workers` (`workerInstanceId` UNIQUE, `workerName` UNIQUE, `dtHeartbeat`)
+   through the new `DB_VISION_LOGS` alias.
+3. Diff the two sets in both directions, plus a heartbeat-freshness pass, producing the three
+   findings above.
+4. Apply the **grace period** (default 10 min): an instance that is still booting has not had
+   a chance to claim a role yet and must not page.
+5. POST the decided tokens to the OneUptime monitor, non-fatal.
+
+**Fail-safe:** a failed AWS call *or* a failed registry read pushes **`status = error`**, so a
+blind checker is never mistaken for a healthy fleet. This is the same rule as
+[the EB health monitor](./elastic-beanstalk-health-monitor.md) — a read that could not happen
+is never reported as "fine."
+
+### Why `Vision_Log` needed no new plumbing
+
+worker2's **`[database1]` group already points at the legacy production cluster**
+(cluster id `clwbyqvxdm4q`, `us-west-2`) — it is what already backs the `TOGaDeskSupport`
+alias. `Vision_Log` lives on that **same single all-in-one legacy cluster** (verified: `Core`,
+`TOGaDeskSupport`, `Vision` and `Vision_Log` are all present on it). So the whole database
+change was **two lines in `_.php`**: no new credentials, no security-group change, no
+`Core.Databases` rows.
+
+Two facts that make that safe, both verified this session:
+
+- All **9** `worker2/Config/*.ini` files define `[database1]`, so the registration resolves in
+  every environment.
+- **`_Database::register()` is lazy** — it stores connection config and never opens a socket —
+  so a static registration in `_.php` costs nothing and cannot break a dev config whose
+  `[database1]` points at localhost.
+
+## The blind spot (why none of the 14 existing monitors could see this)
+
+This is the durable "why" the incident exposed, and the reason this monitor had to reach for
+AWS rather than the database alone.
+
+- 1.0 role assignment happens in `worker/ebs/cron.worker.php` and **only at DEPLOY time** (it
+  is written as the EB `appdeploy` enact hook). The script claims a free name, writes
+  `/etc/worker-role`, and **only then** appends that role's `cron.worker.<name>.json` entries.
+- Therefore **no name means no crontab**. A roleless instance runs *nothing*, while EB happily
+  reports the environment healthy — the box is up, it just does no work.
+- **All 14 existing OneUptime monitors are keyed by ROLE, not by instance.**
+  `crons/worker/worker_heartbeat.php` only pings the endpoint for the row matching **its own**
+  `instanceId`, and the `045_oneuptime_agent.config` infrastructure agent reads
+  `/etc/worker-role` and **exits clean when it is empty**. A roleless instance therefore
+  reports to nothing, and no role-keyed monitor is missing a ping.
+- `Vision_Log.Workers` **only ever holds instances that SUCCEEDED** in claiming a role. The
+  failure leaves no row and no log line, so **absence is the only evidence** — and seeing an
+  absence requires the AWS instance list. That is precisely why this monitor exists in this
+  shape.
+
+## Design history — built in worker2, not in worker 1.0
+
+It was **first built in 1.0** (`worker` owns the `Vision_Log` connection natively), then moved
+to worker2 and **the 1.0 version was deleted**. Recorded so it is not resurrected:
+
+- worker2 already reaches the same cluster (above), so the 1.0 home bought nothing.
+- worker2 uses the **EB instance role**; 1.0 would have used the hardcoded static AWS key.
+- Schedule and parameters live in a **`CronJobs` row**, not in a deploy.
+- Most importantly, it watches the 1.0 fleet **from outside it** instead of from a member of it.
+
+The obvious objection — "a monitor running on the fleet it watches has a blind spot" — is
+covered either way by OneUptime's own **absence criteria** (10 min Degraded / 15 min Offline),
+which catch the monitor itself dying. The other three reasons decided it.
+
+## Verification performed at build time
+
+- **24/24 logic tests pass** against the real `Fleet.php` (private methods driven by
+  reflection): today's healthy fleet, a **replay of the actual incident**, the grace period,
+  stale heartbeats, a NULL-name spare row, a ghost row for a terminated instance, and an empty
+  registry.
+- `Core.CronJobs` columns, uuid uniqueness, and the absence of a clashing `Monitor/Fleet%`
+  action all confirmed.
+- **Unverified at capture time:** whether `WorkloadsRuntime` exists in `502614707982` with
+  `elasticbeanstalk:DescribeInstancesHealth`. If it does not, the monitor pushes
+  `status = error` — loud, not silent.
+
+## Gotchas / known issues
+
+- **The `CronJobs` row ships `isActive = 0` and must be flipped on after worker2 deploys.**
+  Follow the activation ordering in
+  [OneUptime push-metric monitors](./oneuptime-worker2-monitoring.md) — confirm a push has
+  landed before enabling the absence criteria.
+- **Never widen the AWS scope to an account-level EC2 scan.** The same account also runs
+  `agilant-worker-alpha` and `agilant-worker-beta` as **separate EB environments with no role
+  registry at all**. `describeInstancesHealth` is scoped by `EnvironmentName`, which correctly
+  excludes them; an account-wide scan would flag every alpha/beta box as roleless.
+- **Known false-positive risk, deliberately NOT guarded yet.**
+  `worker_heartbeat.php` terminates a stale instance and **then** deletes its `Workers` row. If
+  EB still lists that instance in `describeInstancesHealth` for a minute or two after the row is
+  gone, this monitor sees an instance with no role and raises `alarm = HIGH` — a possible false
+  page during a deploy or a recycle. It was not pre-solved because **how long EB keeps a
+  terminating instance in the health list is unverified**. If it shows up in practice, the fix is
+  a one-line filter on the instance's health status.
+- **The OneUptime push URL is a credential** — never log it, never record its value.
+- The class lives under `Worker/Monitor/` (singular), matching the OneUptime push monitors —
+  not the `Worker/Monitors/` (plural) orchestrator framework.
+
+## Change history
+- 2026-08-24 — Created. Added `_Worker_Monitor_Fleet::RoleAssignment()`
+  (`Monitor/Fleet/RoleAssignment`, `*/5 * * * *`, `isActive = 0` pending deploy) reporting
+  `namelessInstances` / `unownedRoles` / `staleRoles` to OneUptime, after 2 of 7 production
+  workers ran roleless for ~3 hours undetected. Registered `DB_VISION_LOGS` on the existing
+  `[database1]` legacy-cluster group (2 lines, no new credentials). Recorded the role-keyed
+  monitoring blind spot that made the incident invisible, the decision to build in worker2
+  rather than 1.0 (and delete the 1.0 version), and the unguarded terminating-instance
+  false-positive risk. worker2 `_production` e75690f, dbchanges2 `_main` 17732e2. (bala)

package/knowledge/INDEX.md

@@ -5,7 +5,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 ## 1.0 framework
 
 - **library** (Library) _(framework core)_ — 19 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
-- **worker** (Worker) — 26 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
+- **worker** (Worker) — 28 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
 - **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
 - **worker1.5** (Worker 1.5) — 0 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
 - **togadesk** (TOGa Desk) — 13 doc(s) → [1.0/apps/togadesk/INDEX.md](1.0/apps/togadesk/INDEX.md)
@@ -19,7 +19,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 ## 2.0 framework
 
 - **_underscore** (_Underscore) _(framework core)_ — 62 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
-- **worker2** (Worker) — 55 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
+- **worker2** (Worker) — 56 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
 - **api2** (API) — 25 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
 - **dbchanges2** (Database Changes) _(framework core)_ — 8 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
 - **toga2-supply** (TOGa Supply) — 7 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)

package/knowledge/clients/compass-usa/INDEX.md

@@ -12,6 +12,7 @@
 | [Compass MITS PO rejection — zero line items (400 + Issue + PM alert email)](features/mits-po-zero-line-item-rejection.md) | 2.0 | MITS transmits Compass purchase orders that carry **`"purchaseOrderItems": []`** (real example: PO `50309196-1`, `mitsSalesOrder` `MR244630`). | _underscore/Model/Compass/PurchaseOrder.php, _underscore/Model/Compass/Usa/PurchaseOrder.php, _underscore/Model/Compass/Canada/PurchaseOrder.php, worker2/Worker/Client/Compass/reports/PurchaseOrderRejectionEmail.php |
 | [Compass MITS Sales-Order Transmission — Rejection Alerting (Issue/Event, not email-in-cron)](features/mits-sales-order-transmission-alerting.md) | 2.0 | When MITS **rejects** a Compass sales order transmitted by the 1.0 cron `1_transmit_compass_sales_orders_to_mits.php`, the alert is no longer an email built ins | worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php, worker2/Worker/Infrastructure/Errors.php, _underscore/Model/Core/Logs/Issue.php, _underscore/Model/Core/Logs/IssueEmailAddress.php, dbchanges2/Logs/2026-08-04a - MITS rejection business recipients.sql |
 | [Compass MR/MA Order Auto-Approval & Status Gate](features/mr-ma-order-approval-and-status.md) | 2.0 | Compass **MR** and **MA** sales orders are system-generated from the MITS / Office Depot EDI pipeline (they do not originate as user-entered SA orders) and must | _underscore/Model/Compass/SalesOrder.php, _underscore/Model/Compass/PurchaseOrder.php, worker/crons/toga2/compass/workflow/3a_import_office_depot_purchase_orders.php |
+| [Compass ODP EDI 850 Line-Item Resolution (VA part number to IN SKU fallback)](features/odp-edi-850-item-resolution.md) | 1.0 | How each **PO1 line** on an inbound Office Depot (ODP) **EDI 850** is resolved to a real `Client_Compass` catalog item before cron `3a_import_office_depot_purch | worker/crons/toga2/compass/workflow/3a_import_office_depot_purchase_orders.php, library/app/Edi.php |
 | [Compass Office Depot EDI 855 Acknowledgement + Over-Quantity PO Guard](features/odp-edi-855-acknowledgement-and-overquantity-guard.md) | 1.0 | How Compass acknowledges Office Depot (ODP) inbound **EDI 850** purchase orders with an **X12 855**, and the **over-quantity guard** that rejects a duplicate PO | worker/crons/toga2/compass/workflow/3a_import_office_depot_purchase_orders.php, worker/crons/toga2/compass/workflow/4_transmit_office_depot_po_acknowledgements.php, library/app/edi.php |
 | [Compass PEOPLE-File User Lifecycle (duplicate accounts, reactivation grace window, raw-SQL deactivation)](features/people-file-user-lifecycle.md) | 2.0 | The nightly **PEOPLE** file cron (`_Worker_Client_Compass_PeopleFile`, `worker2/Worker/Client/Compass/PeopleFile.php`) owns the whole `Users` row lifecycle for | worker2/Worker/Client/Compass/PeopleFile.php |
 | [Persona Model & Levy-Sector Gating (worker2 PEOPLE cron)](features/persona-model-and-levy-gating.md) | 2.0 | Compass USA catalogue visibility is driven by **personas** in `Client_Compass`. | worker2/Worker/Client/Compass/PeopleFile.php |

package/knowledge/clients/compass-usa/features/odp-edi-850-item-resolution.md

@@ -0,0 +1,179 @@
+---
+title: Compass ODP EDI 850 Line-Item Resolution (VA part number to IN SKU fallback)
+framework: "1.0"
+repo: worker
+project: Worker
+client: compass-usa
+type: client-feature
+status: active
+updated: 2026-08-25
+owners: ["bala"]
+files:
+  - worker/crons/toga2/compass/workflow/3a_import_office_depot_purchase_orders.php
+  - library/app/Edi.php
+related:
+  - odp-edi-855-acknowledgement-and-overquantity-guard.md
+  - ../workflows/odp-order-pipeline-to-netsuite.md
+  - ../workflows/odp-duplicate-po-line-cleanup.md
+  - ../workflows/odp-edi-import-recovery.md
+  - ../profile.md
+---
+
+## Summary
+How each **PO1 line** on an inbound Office Depot (ODP) **EDI 850** is resolved to a real
+`Client_Compass` catalog item before cron `3a_import_office_depot_purchase_orders.php` writes
+anything. ODP sends **two** identifiers per line and only one of them is reliable:
+
+| PO1 element | Qualifier | Parsed into | Reliability |
+|---|---|---|---|
+| `elements[7]` | `VA` | `partNumber` | **free text that drifts** — a mistyped character arrives verbatim |
+| `elements[9]` | `IN` | `vendorPartNumber` (ODP's own SKU) | **correct every time observed** |
+
+(Mapping verified in `App_Edi::translatePurchaseOrder850()`, the `case 'PO1'` block. So on a
+Compass PO the `VendorItems.vendorPartNumber` value **is** ODP's SKU, and 3a's later
+PO-item-to-SO-item matching already keyed on it — only sales-order-item creation trusted the VA
+text.)
+
+Before 2026-08-25 the importer created the ODP sales-order item from the **VA text only**, so a
+single wrong character killed the whole PO import. Resolution now runs **first**, for **every
+line**, and nothing is written unless all lines resolve.
+
+## The resolution ladder (`resolveOfficeDepotEdiItemPartNumbers()`)
+Runs on the parsed 850 **before** the duplicate-PO check, the over-quantity guard, and any
+SalesOrder / PurchaseOrder / item creation.
+
+1. **VA part number matches the Compass catalog** — use it as-is
+   (`partNumberSource = 'ediPartNumber'`).
+2. **Fall back to ODP's own SKU** — `VendorItems.vendorPartNumber = <IN value>` with
+   `vendorId = App_Client_Compass::VENDOR_ID__OFFICE_DEPOT` gives `Items.partNumber`, which is
+   then **confirmed to be in the Compass catalog** (`partNumberSource = 'vendorSku'`).
+   Auto-resolves **only when exactly ONE catalog candidate remains.**
+3. **Otherwise the line is unresolved** — it is reported with a plain-English reason and
+   **nothing at all is imported for that PO**.
+
+Two api2 `GET`s back it, both batched per file (all lines' values in one `in` clause):
+`/items` joined to `Catalogs` filtered on `Catalogs.uuid = UUID_ITEM_CATALOG`
+(`961dbcbc-1943-11ef-8e3b-0aae2980db55`, the "Compass" catalog the cron posts against), and
+`/vendor-items` joined to `Items` filtered on `vendorPartNumber IN (...)` plus the ODP vendor id.
+All matching is **case-insensitive** (keys upper-cased, original casing returned).
+
+### Why SKU resolution and not "clean up the string"
+The real failures were not a single normalisable pattern:
+
+| ODP PO | VA part number sent | Correct part number | ODP SKU (IN) |
+|---|---|---|---|
+| `41762948-1170` (2026-08-25) | `US-AND-CROTHALL-POM` (extra `M`) | `US-AND-CROTHALL-PO` | `4976158` |
+| `41242800-1170`, `41243306-1170` (2026-06-17) | `ODTABKITTINGOPT3` (missing `P`) | `ODPTABKITTINGOPT3` | `4900941` |
+
+A character-stripping / fuzzy-normalising rule fixes the first and **not** the June pair. The
+SKU is the only identifier that was right in every case, so the SKU is the fallback key.
+
+## Never create an item to match a bad inbound string
+Hand-patching these failures in the past created **duplicate junk items in the Agilant catalog**
+— e.g. `ODTABKITTINGOPT3` (item id 2715) now sits alongside the real `ODPTABKITTINGOPT3`
+(id 2738). Creating an item so a mangled part number "matches" is the wrong fix: it splits one
+physical SKU across two `Items` rows, which is the same shape as the
+[dual-catalog duplicate PO-line bug](../workflows/odp-duplicate-po-line-cleanup.md). Fix the
+`VendorItems` mapping (or have ODP correct their part number) instead.
+
+## ODP VendorItems span two catalogs, and a few SKUs are ambiguous
+Verified on prod `Client_Compass`, `VendorItems.vendorId = 1`: rows point at items in
+**catalogId 1 ("Compass")**, **catalogId 2 ("Agilant")**, and **22 rows carry a null catalog**.
+Three SKUs map to **more than one catalog-1 item**:
+
+| ODP SKU | Catalog-1 candidates |
+|---|---|
+| `7128204` | `AW5M5UT` / `AW5M5UT-1` |
+| `7423324` | `INC019BTBK` / `INC019BTBK-NS` |
+| `8030904` | `C30705092` / `C30705092-DS` |
+
+This is why step 2 requires **exactly one** candidate. An ambiguous SKU is **reported, never
+guessed** — picking one silently would ship the wrong item. A SKU whose only match sits outside
+catalog 1 is treated the same way (unresolved), which is what the extra catalog-confirmation
+lookup is for.
+
+## Nothing is written until every line resolves
+The gate is a single `$shouldProcessOrder = empty($itemResolution['unresolvedItems'])` check at
+the top of the per-PO loop, ahead of all writes.
+
+**What it prevents:** the importer used to create the ODP sales order **first** and add items one
+at a time, so a bad line left a **half-built order in production** — with no PO and no 855.
+Exactly what happened to ODP SO `480238825001` on 2026-08-20: lines 1-3 created, line 4 failed,
+no PurchaseOrder, no acknowledgement. Now a failed line leaves the PO completely untouched, so
+re-dropping the 850 is a clean retry (see
+[import recovery](../workflows/odp-edi-import-recovery.md)).
+
+The resolved part numbers are also what the downstream checks now see:
+- the existing `SKIP_IF_CONTAINS_PART_NUMBERS` skip list, and
+- the **over-quantity guard** (`officeDepotPurchaseOrderExceedsCompassDemand()`), which keys on
+  `partNumber` and **silently skips any part it cannot match** — so before this change a mangled
+  part number bypassed duplicate-quantity protection entirely. See
+  [855 Acknowledgement + Over-Quantity Guard](odp-edi-855-acknowledgement-and-overquantity-guard.md).
+
+## `ediPartNumber` — what ODP sent is kept alongside what we resolved
+Every parsed item now carries **both**: `ediPartNumber` (ODP's original VA text) and `partNumber`
+(the resolved catalog value), plus `partNumberSource` (`ediPartNumber` | `vendorSku` | `null`).
+An EDI acknowledgement must reflect the partner's own values, so the **reject 855 is built from
+`ediPartNumber`**.
+
+> **The accept 855 still echoes OUR corrected part number.** Cron
+> `4_transmit_office_depot_po_acknowledgements.php` reads `Items.partNumber` off the stored PO,
+> so a SKU-corrected line acknowledges the **real Compass part number**, not the string ODP sent
+> (PO109 still carries their SKU). Accepted deliberately on 2026-08-25. To change it,
+> `ediPartNumber` has to be carried onto the PurchaseOrder record — it is not persisted today.
+
+## A failed lookup degrades to the OLD behaviour, it does not block
+If either api2 lookup returns `isSuccess = false`, `buildItemsUnchangedAfterFailedLookup()`
+returns every line **exactly as ODP sent it** (`ediPartNumber` populated, `partNumber`
+untouched, no unresolved lines) and the import proceeds as it did before this feature existed.
+
+**Why:** blocking on a lookup failure would turn a one-line problem into *no ODP PO imports at
+all*. Worst case now equals the old behaviour, never worse. The failed lookup still emails.
+
+## The failure email (one sender, no template in the cron)
+`sendErrorNotification(string $message, ?object $response = null)` is the **only** sender —
+subject `Office Depot PO Import error: <PO numbers>`, the raw EDI attached as
+`EDI_<PO numbers>.txt`. The API-response `print_r` dump is appended **only when a response is
+passed**, so a business-readable message is no longer buried in a dump.
+
+`buildItemResolutionErrorMessage()` builds that body: the Compass PO / Compass SO / ODP SO, then
+per failed line the **ODP part number, the ODP SKU, the qty and the exact reason**, closing with
+the fix (set the SKU up as an ODP vendor item against the correct Compass item, or have ODP
+correct the part number, then re-drop the attached file into `OfficeDepot/` on `agilant-as2`).
+All interpolated values pass through `htmlspecialchars()`.
+
+**No email template belongs in the cron** — `App_Email_Agilant` already wraps the body via
+`App_Email_Template::renderHtml()`. A second sender and a cron-local template were both
+explicitly rejected.
+
+## Helper map (all in cron 3a)
+| Function | Job |
+|---|---|
+| `resolveOfficeDepotEdiItemPartNumbers()` | the ladder; returns `['items' => ..., 'unresolvedItems' => ...]` |
+| `fetchCatalogPartNumbersByPartNumber()` | `/items` + `Catalogs.uuid` to an upper-cased part-number lookup (`null` = call failed) |
+| `fetchPartNumbersByVendorSku()` | `/vendor-items` + ODP vendor id to SKU then **every** candidate part number |
+| `filterToCatalogPartNumbers()` | keeps only candidates confirmed in the Compass catalog |
+| `buildItemsUnchangedAfterFailedLookup()` | the degrade-to-old-behaviour path |
+| `buildItemResolutionFailureReason()` | one plain sentence per failed line |
+| `buildItemResolutionErrorMessage()` | the full email body |
+| `uniqueNonEmptyValues()` | trim / drop blanks / de-dupe before an `in` clause |
+
+## Verification (2026-08-25)
+`php -l` clean. A standalone harness ran the real resolver functions against prod-shaped lookup
+data — 20 checks covering the `41762948-1170` fix, the June `ODTABKITTINGOPT3` pair, an
+ambiguous SKU, a SKU pointing outside the Compass catalog, a line with no SKU, and the
+failed-lookup degradation. Prod after deploy: ODP `PurchaseOrders` id 112094 number
+`41762948-1170` created 13:32:54 with all 4 items correct, `SalesOrderItems` line 4 id 613590
+created as `US-AND-CROTHALL-PO`, lines 1-3 reused (not duplicated), `FileLog` 878902.
+
+## Change history
+- 2026-08-25 — Built the VA-to-IN resolution ladder so a drifted ODP part number no longer kills
+  a PO import: exact catalog match first, then ODP's own SKU through `VendorItems` (vendorId 1)
+  confirmed in the Compass catalog, auto-resolving only on a single candidate and reporting
+  ambiguity otherwise. Moved resolution **ahead of all writes** (a bad line used to leave a
+  half-built ODP SO — `480238825001`, 2026-08-20), fed the resolved part numbers to the skip list
+  and the over-quantity guard (which silently skips unmatched parts, so a mangled part number had
+  been bypassing duplicate protection), added `ediPartNumber` so the reject 855 echoes ODP's own
+  value, and made a failed lookup degrade to the pre-change behaviour rather than block. One
+  error sender only, no template in the cron. Recorded the dual-catalog `VendorItems` spread and
+  the 3 ambiguous SKUs. Deployed to prod and verified (commit `fc23bb46`). (bala)

package/knowledge/clients/compass-usa/features/odp-edi-855-acknowledgement-and-overquantity-guard.md

@@ -6,13 +6,14 @@ project: Worker
 client: compass-usa
 type: client-feature
 status: active
-updated: 2026-08-13
-owners: ["jcardinal"]
+updated: 2026-08-25
+owners: ["jcardinal", "bala"]
 files:
   - worker/crons/toga2/compass/workflow/3a_import_office_depot_purchase_orders.php
   - worker/crons/toga2/compass/workflow/4_transmit_office_depot_po_acknowledgements.php
   - library/app/edi.php
 related:
+  - odp-edi-850-item-resolution.md
   - ../workflows/odp-order-pipeline-to-netsuite.md
   - ../workflows/odp-duplicate-po-line-cleanup.md
   - mits-po-transmission-to-vendors.md
@@ -65,6 +66,25 @@ The demand relay it walks is the standard ODP chain — Compass SO (top, e.g. `M
 Compass PO → ODP SO → ODP PO (leaf) — via the `SalesOrders_PurchaseOrders` and
 `PurchaseOrders_SalesOrders` bridges (the same 6-table join the acknowledgement cron uses).
 
+### ⚠ The guard only works on RESOLVED part numbers (2026-08-25)
+Because the guard keys on `partNumber` and **ignores any part it cannot match**, a part number
+that ODP mistyped used to slip past duplicate-quantity protection entirely — the guard saw an
+unknown part and passed the PO. Cron 3a now resolves every 850 line to a real Compass catalog part
+number **before** the guard runs, and feeds the guard the resolved values. Anything that fails to
+resolve stops the import outright, so an unmatched part can no longer reach the guard as a silent
+pass. Ladder, ambiguity rules and the degrade-on-lookup-failure policy:
+[ODP EDI 850 Line-Item Resolution](odp-edi-850-item-resolution.md).
+
+## Which part number each 855 echoes
+An acknowledgement must reflect the partner's own values, so the two paths now differ:
+
+- **Reject 855 (from 3a)** — built from **`ediPartNumber`**, the raw `VA` part number ODP sent,
+  which the resolver preserves on every parsed item alongside the corrected `partNumber`.
+- **Accept 855 (cron 4)** — reads `Items.partNumber` off the **stored** PO, so a SKU-corrected
+  line echoes back **our** part number, not ODP's. `PO109` still carries their SKU, so the line is
+  still identifiable to them. Accepted deliberately; carrying `ediPartNumber` onto the
+  PurchaseOrder record is what it would take to change (it is not persisted today).
+
 ## Why the reject is sent inline, not by cron 4
 A rejected PO is **never persisted**, so cron 4 — which keys off `PurchaseOrders` rows with
 `dtAcknowledged IS NULL` — can never see it. The importer therefore **builds and sends the
@@ -127,6 +147,13 @@ A legacy, abandoned sketch (`compass/edi/3_send_edi_855(reject).php`) used **non
 codes `BAK01 = A1` / `BAK02 = RE` — that path was **deliberately NOT followed**.
 
 ## Change history
+- 2026-08-25 — The over-quantity guard is now fed **resolved** part numbers (it ignores parts it
+  cannot match, so a mistyped ODP part number had been bypassing duplicate protection), the
+  **reject 855 echoes `ediPartNumber`** (what ODP sent) instead of our corrected value, and the
+  accept 855's use of our own `Items.partNumber` was recorded as a known, accepted difference.
+  Cron 3a's `sendErrorNotification()` became `(string $message, ?object $response = null)` so a
+  business-readable failure email is not buried in an API `print_r` dump. Details in the new
+  [ODP EDI 850 Line-Item Resolution](odp-edi-850-item-resolution.md). (bala)
 - 2026-08-13 — Rejections now raise a business alert: cron `3a` throws
   `App_Exception_Business` (`COMPASS_OFFICE_DEPOT_PO_IMPORT_REJECTED_OVER_QUANTITY`,
   `URGENCY_HIGH`) so every over-quantity/duplicate reject surfaces as a Logs.Issue instead of

package/knowledge/clients/compass-usa/profile.md

@@ -18,7 +18,7 @@ project: _Underscore
 client: compass-usa
 type: profile
 status: active
-updated: 2026-08-20
+updated: 2026-08-25
 owners: [jcardinal, bala, tcox, apeterson, dfranks]
 files: []
 related:
@@ -34,6 +34,7 @@ related:
   - features/stranded-approval-reassignment.md
   - workflows/cross-kit-bundle-corruption.md
   - workflows/odp-duplicate-po-line-cleanup.md
+  - features/odp-edi-850-item-resolution.md
   - features/odp-edi-855-acknowledgement-and-overquantity-guard.md
   - ../../2.0/apps/worker2/features/compass-vip-support-importer.md
   - ../../2.0/apps/toga2-commerce/features/expedited-shipping-gating.md
@@ -99,6 +100,14 @@ separate, related client (see its own profile).
   rejects the whole PO and sends a reject **EDI 855**; ODP POs are acknowledged with an 855
   (accept via cron 4, reject inline from 3a) built by a shared `App_Edi` method. See
   [ODP EDI 855 Acknowledgement + Over-Quantity Guard](features/odp-edi-855-acknowledgement-and-overquantity-guard.md).
+- **ODP's part numbers on the 850 drift; their SKU does not (2026-08-25).** A `PO1` line carries a
+  free-text `VA` part number **and** ODP's own `IN` SKU. A single mistyped character in the `VA`
+  text used to kill the whole PO import (and half-build the ODP sales order). Cron 3a now resolves
+  every line — catalog match first, then the SKU through `VendorItems` (vendorId 1) — before it
+  writes anything, and never guesses on an ambiguous SKU:
+  [ODP EDI 850 Line-Item Resolution](features/odp-edi-850-item-resolution.md).
+  **Never create an `Items` row so a mangled inbound part number "matches"** — that is what put
+  duplicate junk items in the Agilant catalog (`ODTABKITTINGOPT3` vs `ODPTABKITTINGOPT3`).
 - ASN ingestion entry points: cXML to the V2 API (logged in `Logs_Compass.Api`) and
   `worker/crons/toga2/compass/workflow/3b_import_strategic_systems_advance_shipping_notices.php`.
 

package/knowledge/clients/compass-usa/workflows/odp-order-pipeline-to-netsuite.md

@@ -6,7 +6,7 @@ project: Worker
 client: compass-usa
 type: workflow
 status: active
-updated: 2026-08-21
+updated: 2026-08-25
 owners: ["rgirish", "bala", "dfranks", "jcardinal", "mhammontree"]
 files:
   - worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php
@@ -19,7 +19,9 @@ files:
   - library/app/client/compass.php
 related:
   - 1.0/apps/worker/features/netsuite-sales-order-sales-rep-sourcing.md
+  - clients/compass-usa/features/odp-edi-850-item-resolution.md
   - clients/compass-usa/features/odp-edi-855-acknowledgement-and-overquantity-guard.md
+  - 1.0/apps/worker/workflows/tracing-a-worker-cron-run-in-production.md
   - clients/compass-usa/workflows/odp-edi-import-recovery.md
   - clients/compass-usa/workflows/order-lifecycle-and-data-integrity.md
   - clients/compass-usa/features/mits-po-transmission-to-vendors.md
@@ -97,6 +99,27 @@ duplicates and backfills whatever is missing**:
 `customerPurchaseOrder` stays `NULL` on a repaired order and must be set manually. See
 [Recovering a lost ODP EDI import](odp-edi-import-recovery.md).
 
+### Every 850 line must resolve to a catalog item before anything is written (2026-08-25)
+3a resolves **all** PO1 lines to real `Client_Compass` catalog part numbers as the **first** thing
+it does per PO — ahead of the duplicate-PO check, the over-quantity guard and every write. If any
+line fails, **nothing is created for that PO** and the failure is emailed with the EDI attached.
+
+Before this, 3a created the ODP SalesOrder first and added items one at a time, so one bad line
+left a **half-built order in prod** with no PurchaseOrder and no 855 (ODP SO `480238825001`,
+2026-08-20). Combined with the idempotency table above, a failed import is now always a clean
+re-drop rather than a repair job. Ladder, ambiguity rules and the failed-lookup fallback:
+[ODP EDI 850 Line-Item Resolution](../features/odp-edi-850-item-resolution.md).
+
+### ⚠ Known issue (NOT fixed): the `OfficeDepot/` listing makes 3a runs overlap
+An instrumented prod run on 2026-08-25 showed the S3 listing returning **138,472 objects** and
+taking **32 seconds** before any PO was processed, with each PO then costing roughly 40 seconds.
+A file-bearing run therefore blows past the `*/5` schedule and the following runs are **silently
+skipped** by `App_Framework::exitIfProcessRunning()` — leaving **no `CronJobExecutions` row at
+all**, which reads as "the cron never fired". The object count is driven by `OfficeDepot/SENT/`
+and `OfficeDepot/OUTBOX/` accumulating: 3a skips those prefixes when *processing* but still
+*lists* them. Fix candidates (separate work): prune `SENT/`, or list with a tighter prefix. See
+[Tracing a worker cron run in production](../../1.0/apps/worker/workflows/tracing-a-worker-cron-run-in-production.md).
+
 ### The EDI 850 → TOGa identifier mapping (not guessable — this is the key to tracing an ODP order)
 
 | 850 segment | Meaning | Where it lands |
@@ -105,7 +128,14 @@ duplicates and backfills whatever is missing**:
 | **`REF~QC`** | the **ODP SALES ORDER number** (e.g. `475390815001`) | `SalesOrders.number` with `customerId = 1` |
 | **`REF~EU` / `REF~PO`** | the **Compass MITS PO number** (e.g. `50310002-1`) | its `c_mitsSalesOrder` field holds the ODP SO number |
 | **`REF~LU`** | the **Compass sales order** (e.g. `SA135273`) | `SalesOrders` with `customerId = 2` |
-| **`PO1`** | the line | qty, UOM, unit price, vendor part, ODP item id |
+| **`PO1`** | the line | qty, UOM, unit price, and **two** item identifiers (below) |
+
+**`PO1` carries two item identifiers and only one of them is trustworthy:** `elements[7]` (the
+`VA` qualifier) is parsed into **`partNumber`** and is **free text that drifts**; `elements[9]`
+(the `IN` qualifier) is parsed into **`vendorPartNumber`** and is **ODP's own SKU**, correct in
+every observed case. So on a Compass PO the `VendorItems.vendorPartNumber` value *is* ODP's SKU.
+Cron 3a resolves each line through that SKU when the `VA` text does not match the catalog — see
+[ODP EDI 850 Line-Item Resolution](../features/odp-edi-850-item-resolution.md).
 
 **⚠ `REF~QC` — not `BEG` — is the value to search on for the ODP SalesOrder.** Searching
 `PurchaseOrders` / `SalesOrders` for the `BEG` number finds nothing for the SO and wastes time.
@@ -242,6 +272,13 @@ wrong record makes every order look "stuck." Use the **join**, never a number ma
   transmission feature.)
 
 ## Change history
+- 2026-08-25 — Cron 3a now **resolves every 850 line to a catalog item before it writes anything**
+  (a bad line used to leave a half-built ODP SO); documented the `PO1` two-identifier detail
+  (`elements[7]` = drifting `VA` text, `elements[9]` = ODP's reliable `IN` SKU) behind the new
+  [850 item-resolution feature](../features/odp-edi-850-item-resolution.md). Flagged the
+  **unfixed** `OfficeDepot/` listing problem: 138,472 objects / 32 s per run pushes 3a past its
+  `*/5` slot, so following runs are skipped by the overlap guard with **no `CronJobExecutions`
+  row** to show it. (bala)
 - 2026-08-21 — **Cron 5 sales rep** no longer hardcodes employee 34877: it reads the end
   customer's native NetSuite `salesRep` RecordRef (`App_NetSuite::getCustomer()`, cached per
   run, failures logged and treated as "no rep", left unset when absent), with the end-customer

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.645",
+  "version": "1.0.646",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",