toga-ai 1.0.1203 → 1.0.1204

package/knowledge/2.0/apps/worker2/features/oneuptime-worker2-monitoring.md

@@ -125,6 +125,8 @@ Each measures one Compass/Office Depot (or Compass Canada) integration metric an
   count threshold — this catches the 1.0 cron's
   [poison-order starvation](../../../clients/compass-usa/features/mits-sales-order-transmission-alerting.md)
   **and** the cron not running at all (worker1.5 outage 2026-10-22/23, TRUE-83116 — check that first).
+  The incident title says "backlog is too high" for both rules, even when one order is stuck; triage
+  steps live in the [MITS alerting doc](../../../../clients/compass-usa/features/mits-sales-order-transmission-alerting.md#triage-a-compasssalesordertransmission-alarm).
   **Age is measured from the LATER of the two approval decisions — `GREATEST(stage-1, stage-2)
   dtDecision`** — because an order is eligible to transmit only once **both** stages are approved.
   Measuring from **stage-2 alone is wrong** (source of a false HIGH): Compass **VIP auto-approve stamps
@@ -650,6 +652,10 @@ production deploy.**
   11 pre-existing instances across `Monitor/Compass.php`, `Monitor/Aig.php`, `Monitor/Operations.php`);
   never log it and never write its value into a doc. Moving these to `Config/production.ini` is a
   recognized future improvement, not yet done.
+- **⚠ Never run a monitor method from a local worker2.** The push URL is hardcoded in the method, with
+  no environment check, so a local run (`ENVIRONMENT=dev-<name>` via Apache `SetEnv`,
+  `databaseClient=localhost`) pushes **local DB numbers into the real prod monitor** — it can clear or
+  fake a live incident. To check a monitor's numbers, run its SQL read-only against prod instead.
 - **Registering the cron before the deploy lands fails every run.** A `Core.CronJobs` row for a method
   that is not yet on the instance throws `Call to undefined method _Worker_Monitor_Compass::<Name>()`
   from `Controller/Index.php:375`, because the box still runs the old `/var/app/current/` copy. Register
@@ -687,6 +693,7 @@ production deploy.**
 - [OneUptime 1.0 worker uptime monitoring](../../../1.0/apps/worker/features/oneuptime-worker-uptime-monitoring.md) — the 1.0 push-heartbeat predecessor
 
 ## Change history
+- 2026-10-29 — Gotcha: never run a monitor from local worker2, the hardcoded push URL feeds local numbers into the prod monitor; `CompassSalesOrderTransmission` triage now in the MITS alerting doc (TRUE-83407). (bala)
 - 2026-10-29 — Every OneUptime push retries once (`setAutoRetry`, 1 retry, 2s; `PUSH_RETRY_*` consts on every monitor class); worker1.5 heartbeat too (TRUE-83398). (bala)
 - 2026-10-29 — Gotcha: "stopped reporting" incidents from failed OneUptime pushes (`OneUptime push FAILED.` + `alarm=OK` in `WorkerJobs.output`) and how to tell them from a real outage (TRUE-83380). (bala)
 - 2026-10-27 — `EmailedAsnImport` moved off php-imap2 to `_Component_Email_Inbound_Imap` (webklex), TLS now validated, no ext-imap (TRUE-82933). (jcardinal)

package/knowledge/clients/compass-usa/features/mits-sales-order-transmission-alerting.md

@@ -5,10 +5,11 @@ project: _Underscore
 client: compass-usa
 type: client-feature
 status: active
-updated: 2026-10-23
+updated: 2026-10-29
 owners: ["dfranks", "bala", "jcardinal"]
 files:
   - worker1.5/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php
+  - worker2/Worker/Monitor/Compass.php
   - worker1.5/crons/toga2/compasscanada/workflow/1_transmit_compassCanada_sales_orders_to_mits.php
   - worker2/Worker/Infrastructure/Errors.php
   - tools/mvc/errors/post.php
@@ -27,7 +28,7 @@ related:
   - ../../../1.0/apps/tools/features/errors-curation-console.md
   - ../../../1.0/apps/worker1.5/workflows/diagnosing-crons-not-running.md
 ---
-How a `1_transmit_..._sales_orders_to_mits.php` failure reaches a human — USA rejections route through Logs Issue/Event, Canada still emails in-cron; open for "add someone to MITS error emails," a stuck transmit backlog, or the poison-order starvation.
+How a `1_transmit_..._sales_orders_to_mits.php` failure reaches a human — USA rejections route through Logs Issue/Event, Canada still emails in-cron; open for "add someone to MITS error emails," a `CompassSalesOrderTransmission` alarm / stuck transmit backlog, or a MITS rejection (AssetType, SalesOrder:Exists, ShippingEmail).
 
 ## Summary
 How a failure in the `1_transmit_..._sales_orders_to_mits.php` cron reaches a human — for **both** Compass tenants, because **they no longer work the same way**.
@@ -69,6 +70,13 @@ Both crons are live on the single worker1.5 server (since TRUE-83078, 2026-10-22
 
 Configuration is **never retroactive**: `shouldNotify()` fires on the first occurrence, on escalation, or when the backed-off reminder interval elapses, and returns false for a window with zero events. The first email lands on the **next live rejection**, not on existing history.
 
+### Triage a `CompassSalesOrderTransmission` alarm
+The OneUptime incident title says "Compass sales order transmission backlog is too high" for **both** alarm rules (`pendingCount > 10` OR oldest pending > 180 min). One order stuck past 180 min fires it with a count of 2 (2026-10-29, TRUE-83407). So:
+1. Run the monitor's SQL (`_Worker_Monitor_Compass::CompassSalesOrderTransmission`, `worker2/Worker/Monitor/Compass.php`) **read-only on prod** to get the stuck order numbers. Never run the monitor itself locally: it pushes local numbers into the real prod monitor ([gotcha](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md#gotchas)).
+2. Rule out a dead cron (next section).
+3. Read the per-order reason in `Logs.Event.errorMessage` for issueKey `COMPASS_SALES_ORDER_MITS_TRANSMIT_REJECTED`. `Logs.Issue.errorMessage` keeps only the **first** message ever raised, not the latest.
+4. Match the reason to a signature below.
+
 ### First check on a backlog alarm: is the cron running at all?
 Before reading rejections, rule out a dead cron. 2026-10-22/23 (TRUE-83116): the `CompassSalesOrderTransmission` alarm fired (16 pending, oldest 836 min) with **zero** new `Logs.Issue` 24 events and MITS returning 200 on every call — the last `Logs.API` call to MITS was 16:10 the day before. The transmit cron was not running at all: the worker1.5 crontab shell made the overlap guard exit every run. Signs: no new Issue 24 events + stale last `Logs.API` call + `c_dtTransmittedToCompass` not moving. Steps: [Diagnosing worker1.5 crons that stopped running](../../../1.0/apps/worker1.5/workflows/diagnosing-crons-not-running.md).
 
@@ -78,9 +86,13 @@ The alert almost never fires on **volume**. It fires because one bad order **abo
 - The driver query uses **`ORDER BY RAND()`**, so *which* orders get processed before the bad one is random each run. A stuck "poison" order therefore blocks a **different** set of good orders every run — good orders can wait days behind one bad one.
 - Throughput ceiling: the cron sleeps **30s per order** (86400s time limit), scheduled every 5 min but overlap-guarded, so it is effectively one long-running instance — ~110 orders/hr.
 
-**Two rejection signatures seen in prod** (Logs.Issue id 24, issueKey `COMPASS_SALES_ORDER_MITS_TRANSMIT_REJECTED`, ~2329 occurrences since 2026-08-03):
+**Rejection signatures seen in prod** (Logs.Issue id 24, issueKey `COMPASS_SALES_ORDER_MITS_TRANSMIT_REJECTED`, ~2329 occurrences since 2026-08-03):
 1. **HTTP 400 `The AssetType field is required`** — a line item's catalog Item has `assetTypeId = NULL`, so the payload `assetType` is null and MITS rejects the **whole order**. This was the active blocker (e.g. item `4U555A-S`; ~10 distinct NULL-assetType items blocked 9 of 11 pending orders).
 2. **HTTP 403 `SalesOrder:Exists`** — MITS already created the order on a prior attempt, but our side never recorded `c_dtTransmittedToCompass`, so the cron **retries it forever**. **Now handled (2026-09-02):** on a 403 `SalesOrder:Exists` the cron stamps `SalesOrders.c_dtTransmittedToCompass`, writes a `Record`, and **continues** to the next order instead of throwing — so an already-created order no longer starves the queue. Root cause of the original loop (why we never stamped the first time) was the `App_ApiTransaction` decode fatal — see the gotcha below.
+3. **HTTP 400 `Your request is not valid! ... The ShippingEmail ...`** — the order's contact has no email. The cron sends `shippingEmail` = the contact's primary email (`Contacts.primaryContactEmailAddressId` → `ContactEmailAddresses`), else the email of the first `Users` row with that `contactId`. Both empty → MITS rejects, the cron throws every 5 min for that order. Seen 2026-10-29: SA139813 and SA139936 (contact 334468). Usual source: the PEOPLE file had no email, so the import made a new user + contact with none while the person's old inactive user still holds it (see [PEOPLE-file user lifecycle](people-file-user-lifecycle.md)). **Fix is data, not code:**
+   1. Confirm the right address with the Compass team (Vanessa). An older inactive `Users` row of the same person often has it.
+   2. Add a `ContactEmailAddresses` row for the contact (`name = 'WORK'`), then set `Contacts.primaryContactEmailAddressId` to it. Example: `dbchanges2/Client_Compass/2026-10-29a - SA139936ContactEmail.sql`.
+   3. The next `*/5` run sends the order. The nightly PEOPLE import does not undo the fix while the file still has no email for that person.
 
 **Deeper root cause of the 2026-08-31 outage — a wrong catalog merge, not just missing data.** The NULL-assetType items were largely created by an ad-hoc cross-client dedup that folded Compass's **199 catalog-1 (store) items into their Agilant (catalog-2) canonical**, so Compass order lines resolved to Agilant items that have no store classification. The same merge fused their VendorItems, which triggered the SQL 1242 abort (see gotchas). Full root cause + reversal playbook: [catalog-fold reversal](../workflows/catalog-fold-reversal.md). Compass keeps two intentional catalogs and the customer→catalog rule is load-bearing — see [Compass item catalogs & duplicate items](./item-catalogs-and-duplicate-items.md).
 
@@ -106,6 +118,7 @@ The worker2 OneUptime monitor now catches this starvation directly: `CompassSale
 - As of 2026-08-04 prod `Logs.IssueEmailAddress` is empty, so this is the **first business-routed issue on the platform**: `processBusinessIssue()` will execute in production for the first time when this issue next fires. Watch that run rather than assuming it works.
 
 ## Change history
+- 2026-10-29 — Alarm triage steps (incident title says "backlog" even for one stuck order; latest reason is in `Logs.Event`); third rejection signature, HTTP 400 `ShippingEmail` from a contact with no email, fixed by data (TRUE-83407). (bala)
 - 2026-10-23 — Crons now on worker1.5 (`*/5` / `*/10`); backlog alarm can mean the cron is not running at all, check that first (TRUE-83116). (bala)
 - 2026-10-06 — Order query no longer joins `Users` on `contactId` (duplicate users caused double POST + false 403 `SalesOrder:Exists` alerts); noted `Logs.API` and the intentional kit config line. (bala)
 

package/knowledge/clients/compass-usa/features/people-file-user-lifecycle.md

@@ -120,7 +120,8 @@ The insert-time hook designed 2026-08-12 was built and grew into a reusable engi
 - **`Client_Compass.Users` is ~176,000 rows with no index on `c_hrEmpPersonnelNbr`, `email`, `c_hrEmpUsername`, or `isActive`.** Any per-row lookup against those columns is a full scan — `EXPLAIN` returns `type=ALL`, 176,018 rows, no possible keys (re-confirmed prod 2026-09-02, same on `Client_CompassCanada`). An index migration is recommended and **still not done**. **The blank-client template is not a safe proxy for this schema**: `dbchanges2/Client/2026-06-03- BLANK_CLIENT_DATABASE.sql` declares `UNIQUE KEY email (email)`, and Compass has diverged from it. Check the live tenant, never the template.
 - **⚠ There is NO UNIQUE KEY on `Users.email` in either tenant — the database will silently accept a duplicate email.** Prod-verified 2026-09-04. Complete index list on `Users` in **both** `Client_Compass` and `Client_CompassCanada`: `PRIMARY(id)`, `uuid`, `Users_UNIQUE(clientId, homeClientUserId)`, `contactId`, `Users_Users_FK(supervisorUserId)`, `Users_Addresses_FK(addressId)`, `Users_Images_FK(thumbnailImageId)`, `dtCreated`, `dtUpdated`, `Users_c_supportedByUserId`, `fk_users_defaultCostCenterId`. Also the proof of the missing-index gotcha above. **Consequence: any code that writes an email must do its own cross-row duplicate check** — and because email is a login/reset identity in 2.0, a duplicate is a security problem, not just messy data. See [Users.email is a login identity](../../../2.0/apps/_underscore/features/user-email-as-identity-and-password-reset.md).
 - **⚠ The `Users` ↔ `Contacts` email/phone mirror is reached through the PRIMARY pointers, not through `name = 'WORK'`.** `PeopleFile.php:300-310` joins on `Contacts.primaryContactEmailAddressId` / `Contacts.primaryContactPhoneNumberId`. `name = 'WORK'` is only set at **insert**, and there is **no unique key on `(contactId, name)`** — so a contact can hold several `WORK` rows and "the WORK row" is **not deterministic**. Follow the primary pointers; never match on the name.
-- **⚠ The importer re-syncs the `Contacts` child rows UNCONDITIONALLY for every existing user.** `PeopleFile.php:630-632` runs **outside** the `hasDiff` gate, so a `Contacts` email/phone edit made by hand or by a tool is reverted on the next run **even when the `Users` row itself does not diff**. Any out-of-band contact edit is temporary in exactly the same way an out-of-band activation is.
+- **⚠ The importer re-syncs the `Contacts` child rows UNCONDITIONALLY for every existing user.** `PeopleFile.php:630-632` runs **outside** the `hasDiff` gate, so a `Contacts` email/phone edit made by hand or by a tool is reverted on the next run **even when the `Users` row itself does not diff** — **but only when the file carries a value**. `stageContactsForUpdate()` inserts/updates the email only for a non-empty file email, and the cleanup keeps contacts a `Users` row references, so a hand-added email on a person the file sends **without** an email survives.
+- **⚠ A re-hire with no email in the file gets a new user + contact with NO email.** The old inactive user (outside the 5-day window) still holds the address. 2026-10-27: new user 311624 (`southa02`) + contact 334468 with no email, old user 16333 (`southa01`) had it. Downstream break: the MITS transmit cron has no `shippingEmail` and MITS rejects the order — fix and steps in [MITS alerting](mits-sales-order-transmission-alerting.md) (signature 3).
 - **⚠ `executeForTenant()`'s inactivation sweep is the ONLY place in the file that writes `isActive = 0`.** Every other `isActive` path in the ~2,200-line class only forces it back to **1**. Two consequences: (a) **an activation made outside this cron is temporary** — the next run deactivates the person again unless they are in that day's PEOPLE file — while (b) **a deactivation is durable**, because nothing here ever reactivates anyone on its own. This makes the [Tools Compass admin pages](../../../1.0/apps/tools/features/compass-user-persona-admin.md) a one-way-durable tool.
 - **⚠ A user created by hand in the Compass app (by an Admin or Manager) is temporary.** It gets only the **Base** persona and **no `locationId`**, so it sees only Base content (no sector personas, no segment persona). If the person is not in the PEOPLE file, the next 09:16 run sets it inactive (3 cases seen 2026-10-26/27). Fix is to get them into the PEOPLE file, not to re-activate or hand-add personas.
 - **`ensureCrossRegionUsersActive()` runs `UPDATE Users SET isActive = 1 WHERE email = X` with no `isActive` filter**, so it reactivates **every** row carrying a protected address, dead duplicates included. Verified prod 2026-09-02 for the original four, and 2026-09-16 for the first two CA-only addresses: each resolves to exactly **one** row per database, so nothing is being resurrected today — but the query would happily do it if a duplicate appeared. That hardcoded list is also the **only** durable way to hold a cross-region user active, and changing it needs a code change and a deploy per request. **Recommended (not done): scope it to a `Users.id` list instead of matching on email** — matching on email means an email edit silently moves a user in or out of the protected set, which is why the [Tools Compass admin](../../../1.0/apps/tools/features/compass-user-persona-admin.md) blocks the protected addresses from being edited at all.
@@ -147,6 +148,7 @@ The insert-time hook designed 2026-08-12 was built and grew into a reusable engi
 - [Users.email as login identity](../../../2.0/apps/_underscore/features/user-email-as-identity-and-password-reset.md)
 
 ## Change history
+- 2026-10-29 — TRUE-83407: re-hire with no file email gets a new user + contact with no email (breaks MITS `shippingEmail`); a hand-added email survives while the file has none. (bala)
 - 2026-10-29 — TRUE-83392: 10/29 file failed on a split row (`EYERLC01`); fixed the line by hand, re-uploaded, queued one `ProcessPeopleFiles` run (job 2950480, success). Added the bad-row, queue-block and attachment-size gotchas. No code changed. Parked until the problem happens again (no backlog task, by developer choice): optional join-short-lines fix; the attachment size limit; `Client_Compass` user 93605 (`eyerlc01`) has `c_supportedByUserId` NULL after import although the file names supervisor `HILLB17` — not investigated. Also: tell Compass about the line break. (mhammontree)
 - 2026-10-27 — Hand-created app users get only Base + no location and are deactivated by the next run when not in the file (TRUE-83280). (bala)
 - 2026-10-21 — TRUE-82976: fixed the post-insert refresh (`AND isActive = 1 ORDER BY id`, worker2 `e22b3ec`) so new-user roles, personas and supervisor link always go to the newest active row. (bala)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.1203",
+  "version": "1.0.1204",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",