toga-ai 1.0.773 → 1.0.774

package/knowledge/1.0/apps/worker/INDEX.md

@@ -6,7 +6,7 @@
 | [Canon received-PO → NetSuite sales order sync](features/canon-received-po-netsuite-order-sync.md) | A 1.0 `worker` cron that reads Canon customer purchase orders out of the **Vision** databases and creates the matching **sales order in NetSuite**. | worker/crons/sync/sync_received_po.php, library/netsuitetoolkit/NSPHPClient.php |
 | [Compass MA Sales Order Exception Report](features/compass-ma-sales-order-exception-report.md) | A worker cron that emails operations the "Compass Refresh Exception Report" — Compass `MA%` sales orders whose corresponding Office Depot (ODP) sales order has | worker/crons/toga2/compass/workflow/7_generate_ma_sales_order_exception_report.php, worker/schedules/cron.worker.sync.json |
 | [Compass Manager Approval Reminder Emails (1.0 worker crons)](features/compass-manager-approval-reminder-emails.md) | Two 1.0 worker crons nag approvers about sales orders still waiting on a decision — one per Compass tenant. | worker/crons/toga2/compass/compass_email_reminders.php, worker/crons/toga2/compasscanada/compass_email_reminders.php, worker/schedules/cron.worker.sync.json, worker1.5/crons/toga2/compass/compass_email_reminders.php, worker1.5/schedules/cron.worker.json |
-| [Compass Partial In-Transit & Delivered Emails (per package)](features/compass-partial-in-transit-delivered-emails.md) | Compass USA and Compass Canada send a **per-package** in-transit email (and a matching delivered email) instead of one email listing the whole order. | worker/crons/toga2/compass/update_salesorder_status_from_odp.php, worker/crons/toga2/compass/send_delivered_email.php, worker/crons/toga2/compasscanada/update_salesorder_status_from_odp.php, worker/crons/toga2/compasscanada/workflow/3_update_salesorder_status_from_grand_and_toy.php, worker/crons/toga2/compasscanada/send_delivered_email.php, worker/crons/toga2/compasscanada/workflow/test_partial_in_transit_email.php, worker/crons/toga2/compasscanada/workflow/test_partial_delivered_email.php, library/app/client/compasscanada.php |
+| [Compass Partial In-Transit & Delivered Emails (per package)](features/compass-partial-in-transit-delivered-emails.md) | Compass USA and Compass Canada send a **per-package** in-transit email (and a matching delivered email) instead of one email listing the whole order. | worker/crons/toga2/compass/update_salesorder_status_from_odp.php, worker/crons/toga2/compass/send_delivered_email.php, worker/crons/toga2/compasscanada/update_salesorder_status_from_g&t.php, worker/crons/toga2/compasscanada/send_delivered_email.php, worker/crons/toga2/compasscanada/workflow/test_partial_in_transit_email.php, worker/crons/toga2/compasscanada/workflow/test_partial_delivered_email.php, library/app/client/compasscanada.php, worker/schedules/cron.worker.sync.json, worker2/Worker/Monitor/Compass.php, dbchanges2/Client_Compass/2026-09-03 - compass_intransit_email_backlog_backfill.sql |
 | [Elite TOGA 2.0 → TOGaDeskSupport Standalone Attachment Sync](features/elite-togadesk-attachment-sync.md) | `sync_togadesk_elite_attachments.php` is a standalone cron (every 5 minutes) that syncs file attachments from TOGA 2.0 into TOGaDeskSupport for Elite. | worker/crons/toga2/elite/sync_togadesk_elite_attachments.php, worker/crons/toga2/elite/test_sync_togadesk_elite_attachments.php |
 | [Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling](features/forecast2-netsuite-reconciliation.md) | CLI tools to **audit** and **repair** drift between the production `Forecast` DB (core2) and NetSuite. | test/@dave/checker.php, worker2/Component/Forecast/SaleImport/SaleImport.php, test/@dave/looper.php, test/@dave/reconcile_netsuite_totals.php, test/@dave/fixer.php, tools/bin/forecast/fixer.php, tools/bin/forecast/checker.php, test/@dave/analyze_netsuite_forecast_diff.php, test/@dave/trueup_sales.php, test/@dave/reconcile_drift_2023plus.php, test/@dave/probe_invoice_gap_2026.php, test/@dave/probe_creditmemo_gap_detail.php, test/@dave/trueup_open_orders.php, test/@dave/loop_trueup_open_orders.php, test/@dave/trueup_opportunities.php, test/@dave/probe_sales_gap_direct.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_profit_invoices.php, test/@dave/probe_profit_gap.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_open_orders.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/schedules/cron.worker.infrastructure.json |
 | [Forecast2 Supporting-Records Nightly Import (1.0 cron) — Customers & ConsolidatedCustomers](features/forecast2-supporting-records-import.md) | The nightly **1.0** pull that keeps the Forecast2 lookup/dimension tables (`Forecast.Accounts`, `Classifications`, `Customers`, `ConsolidatedCustomers`, `Employ | worker/crons/toga2/forecast2/import_supporting_records.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker/schedules/cron.worker.infrastructure.json, worker/.ebextensions/009_setup_phpini.config, worker/crons/sync/netsuite/netsuite_customers.php, library/app/api/netsuite/rest.php, library/app/model/forecast2/customer.php, library/app/model/forecast2/consolidatedcustomer.php |

package/knowledge/1.0/apps/worker/features/compass-partial-in-transit-delivered-emails.md

@@ -6,18 +6,23 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-26
+updated: 2026-09-04
 owners: ["bala"]
 files:
   - worker/crons/toga2/compass/update_salesorder_status_from_odp.php
   - worker/crons/toga2/compass/send_delivered_email.php
-  - worker/crons/toga2/compasscanada/update_salesorder_status_from_odp.php
-  - worker/crons/toga2/compasscanada/workflow/3_update_salesorder_status_from_grand_and_toy.php
+  - worker/crons/toga2/compasscanada/update_salesorder_status_from_g&t.php
   - worker/crons/toga2/compasscanada/send_delivered_email.php
   - worker/crons/toga2/compasscanada/workflow/test_partial_in_transit_email.php
   - worker/crons/toga2/compasscanada/workflow/test_partial_delivered_email.php
   - library/app/client/compasscanada.php
+  - worker/schedules/cron.worker.sync.json
+  - worker2/Worker/Monitor/Compass.php
+  - dbchanges2/Client_Compass/2026-09-03 - compass_intransit_email_backlog_backfill.sql
 related:
+  - ../../../clients/compass-usa/features/contact-email-resolution.md
+  - ../../../2.0/apps/worker2/features/tracking-status-refresh.md
+  - ../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md
   - ../../../clients/compass-canada/features/french-order-email-localization.md
   - ../../../clients/compass-usa/features/order-fulfillment-status-per-line.md
   - ../../../2.0/apps/_underscore/features/item-fulfillment-stage-lifecycle-and-order-status.md
@@ -34,18 +39,28 @@ and earlier packages). The dynamic HTML is injected into a stored `EmailTemplate
 
 ## Key files / entry points
 
-- `worker/crons/toga2/compass/update_salesorder_status_from_odp.php` — Compass USA prod cron.
-- `worker/crons/toga2/compasscanada/workflow/3_update_salesorder_status_from_grand_and_toy.php`
-  — Compass Canada prod cron.
+- `worker/crons/toga2/compass/update_salesorder_status_from_odp.php` — Compass USA prod cron,
+  registered in `worker/schedules/cron.worker.sync.json` as **"Update Compass SalesOrder Status
+  From ODP CXML"**, schedule **`0 */4 * * *`** — it fires at 00/04/08/12/16/20 **US/Central**
+  (the prod DB timezone). Every threshold below (batch cap, staleness, monitor alarms) is
+  calibrated to that 4-hour cadence.
+- `worker/crons/toga2/compasscanada/update_salesorder_status_from_g&t.php` — Compass Canada
+  prod cron, registered as "Update CompassCanada SalesOrder Status From G&T CXML".
+  **⚠ This filename contains `&` and therefore cannot run — see the gotcha below.** It was
+  renamed from `update_salesorder_status_from_odp.php` on 2026-09-03; the duplicate
+  `workflow/3_update_salesorder_status_from_grand_and_toy.php` was **deleted** in the same pass and
+  its stale `schedules/cron.worker.sync.json` entry removed.
 - Shared helper set in each cron: `getOrderFulfillmentData`, `buildItemRowsHtml`,
   `buildSectionHeaderHtml`, `buildPackageBlockHtml`, `buildTrackingUrl`, `buildOrderItemsHtml`.
 
 ## How it works
 
 1. **Driving query** finds shipped sales orders and their tracking numbers (`computedStatusSlug
-   = 'shipped'`, `c_dtInTransitEmailSent IS NULL`, date cutoff) via the ASN chain
-   (`AdvanceShippingNoticeItemUnits_TrackingNumbers`), one row per `TrackingNumbers.number`.
-2. Per tracking number: skip if the carrier API already reports delivered.
+   = 'shipped'`, `c_dtInTransitEmailSent IS NULL`, rolling date window) via the ASN chain
+   (`AdvanceShippingNoticeItemUnits_TrackingNumbers`), one row per `TrackingNumbers.number`,
+   **newest first and capped** — see *Draining the candidate set* below.
+2. Per tracking number: skip if the shipment is already finished — checked first against the
+   stored `TrackingNumbers.status` (free), and only then with a live carrier call.
 3. **`getOrderFulfillmentData`** returns the order's packages (grouped by tracking number,
    ordered by `dtCreated`) and ordered items, sourced from the **`ItemFulfillmentItems`** line
    bridge (`ItemFulfillmentItems` → `ItemFulfillments` (scoped to the buyer SO) →
@@ -58,6 +73,73 @@ and earlier packages). The dynamic HTML is injected into a stored `EmailTemplate
    `App_Api_Toga2::send(..., 'POST', '/email-templates/sendEmail', $payload, [], true)`.
 6. On a successful (non-throwing) send, set `c_dtInTransitEmailSent = NOW()`.
 
+### Draining the candidate set: two-pass, newest-first, bounded
+
+**This is the load-bearing part of the cron. Getting it wrong took Compass USA in-transit emails to
+near zero for a week** (2026-08-27 → 2026-09-03: 130–190 a day before, then 7 / 1 / 6 / 0 / 1).
+Customers were not told their order had shipped.
+
+The original shape was: select **every** unstamped `TrackingNumbers` row since a hardcoded
+`'2026-06-15'`, then call UPS/FedEx **live, one row at a time**. When a shipment turned out to be
+already delivered the loop `continue`d **without stamping `c_dtInTransitEmailSent`** — so the same
+row came back on the next run, forever. The candidate set grew to **2,379** and the run hit its
+`App_Framework::cronInitialization(3600)` one-hour limit while still inside the June rows. It never
+reached the day's new ASNs. Nothing errored; the cron just ran out of time on every tick.
+
+Four changes fix it, and they are the pattern to copy for any *"poll an external API per row, then
+mark the row done"* cron:
+
+1. **Two passes.** Pass 1 walks the candidates and **stamps** every already-finished row so it
+   retires from the candidate set permanently. Pass 2 sends. A row can never again be examined
+   forever without being retired.
+2. **Newest first, hard cap.** `ORDER BY MAX(TrackingNumbers.dtCreated) DESC` +
+   `LIMIT IN_TRANSIT_EMAIL_BATCH_SIZE` (**400**). Today's shipments are served before any backlog,
+   so a backlog can delay old mail but can never starve new mail.
+3. **Rolling window, not a fixed date.** `IN_TRANSIT_EMAIL_LOOKBACK_DAYS = 14` →
+   `dtCreated >= DATE_SUB(NOW(), INTERVAL 14 DAY)`. A hardcoded cutoff date guarantees an
+   ever-growing scan; a rolling window is self-limiting.
+4. **Trust the stored status before paying for a carrier call.** `isTrackingStatusFinal(?string)`
+   short-circuits when `TrackingNumbers.status` is already `DELIVERED` or `RETURNED_TO_SENDER`
+   (`TRACKING_FINAL_STATUSES`). worker2's
+   [centralized tracking-status refresh](../../../2.0/apps/worker2/features/tracking-status-refresh.md)
+   keeps that column current for every client, so this 1.0 cron should not re-ask the carrier about
+   a finished shipment.
+
+### Resolving the recipient: a `Users` row is NOT required (USA)
+
+The To address originally came only from `Users.email`, via `INNER JOIN Users ON Users.contactId =
+…`. **EDI-created MR orders never get a `Users` row**, so those orders were silently absent from
+the driving query — 12 MR orders with shipped parcels were never emailed and never even showed up
+as a backlog.
+
+The USA cron now resolves the address in this precedence:
+
+```
+LEFT OUTER JOIN Users
+    ON Users.contactId = ShippedSalesOrders.contactId
+    AND Users.email IS NOT NULL AND Users.email <> ''
+INNER JOIN Contacts
+    ON Contacts.id = ShippedSalesOrders.contactId
+LEFT OUTER JOIN ContactEmailAddresses
+    ON ContactEmailAddresses.id = Contacts.primaryContactEmailAddressId
+    AND ContactEmailAddresses.contactId = Contacts.id      # ownership guard
+…
+IFNULL(Users.email, ContactEmailAddresses.emailAddress) AS contactEmail
+```
+
+The same `IFNULL` resolves the display name, and the `WHERE` requires the resolved address to be
+non-empty.
+
+**The `ContactEmailAddresses.contactId = Contacts.id` predicate is not optional.** About 14% of
+Compass contacts point at an email row owned by a *different* contact, so without it the cron mails
+the wrong person — see
+[Compass contact email resolution](../../../clients/compass-usa/features/contact-email-resolution.md).
+
+**Regression-checked, not assumed.** Over 60 days and **4,419 SA tracking rows** the old and new
+queries both select **4,416**: 0 newly included, 0 lost, resolved address and name identical on all
+4,416. The `Users` address always wins because it is first in the `IFNULL`, so the change is
+strictly additive — it only reaches orders the old query could not see at all.
+
 ### Tracking-URL building (`buildTrackingUrl`)
 
 The "Track Shipment" button and each per-package link resolve in this precedence:
@@ -100,7 +182,22 @@ the tracking number as emailed (it retries next run).
 
 ## Client variations
 
-- **Compass USA** — English only; `update_salesorder_status_from_odp.php`.
+- **Compass USA** — English only; `update_salesorder_status_from_odp.php`. Has the full
+  2026-09-03 hardening: two-pass drain, 400-row cap, 14-day window, `isTrackingStatusFinal()`,
+  **and** the `Users` → `Contacts`/`ContactEmailAddresses` recipient fallback with the ownership
+  guard.
+- **⚠ Canada is only PARTLY at parity (verified 2026-09-04).**
+  `compasscanada/update_salesorder_status_from_g&t.php` has the batching changes
+  (`IN_TRANSIT_EMAIL_BATCH_SIZE = 400`, `IN_TRANSIT_EMAIL_LOOKBACK_DAYS = 14`,
+  `ORDER BY MAX(TrackingNumbers.dtCreated) DESC`, `isTrackingStatusFinal()`) but **still resolves
+  the recipient with `INNER JOIN Users … Users.email AS contactEmail`** — no `Contacts` /
+  `ContactEmailAddresses` fallback and no ownership guard. Canada therefore still cannot email a
+  contact that has no `Users` row. Port the recipient block if Canada ever gets EDI-created MR
+  orders.
+- **Canada keeps its own retry behaviour on a carrier throw.** It leaves the row **unstamped** when
+  the carrier lookup throws, so the next run retries it; only rows confirmed *final* are stamped.
+  That is deliberate and must survive any future parity pass — stamping on a throw would silently
+  drop the email.
 - **Compass Canada** — EN/FR by `UserGlobalSettings.settingId = 2` (FR when the value starts
   with `fr`); French section labels and a French (HTML-entity) status message; uses
   `htmlentities` in the section header builder and a 7-param French-aware `buildPackageBlockHtml`.
@@ -113,6 +210,33 @@ the tracking number as emailed (it retries next run).
 
 ## Gotchas / known issues
 
+- **⚠ A cron filename containing `&` cannot run at all.** `worker/ebs/cron.worker.php:41` builds
+  the crontab command by plain unquoted concatenation
+  (`'php /var/www/html/crons/' . $cron->cron`), so cron reads `&` as *"background this and start a
+  new command"*: `update_salesorder_status_from_g&t.php` executes as
+  `…/update_salesorder_status_from_g` plus a stray `t.php`, and the real script never runs —
+  silently, on every tick. **The Canada cron is in this state as of 2026-09-04 and must be renamed
+  (e.g. `update_salesorder_status_from_gt.php`) with `schedules/cron.worker.sync.json` repointed.**
+  Never put `&` (or any shell metacharacter) in a cron filename. See
+  [tracing a 1.0 worker cron run](../workflows/tracing-a-worker-cron-run-in-production.md).
+- **⚠ Invoice tracking numbers are NOT in scope and must never be counted as in-transit
+  backlog.** `POST /v2/invoice-tracking-numbers` creates `TrackingNumbers` rows bridged through the
+  **`InvoiceTrackingNumbers`** table — never the ASN chain. On 2026-09-03 alone, 98 were created
+  (11:15–11:40) and **zero** of them touch `AdvanceShippingNoticeItemUnits_TrackingNumbers`.
+  Anything that walks the ASN unit bridge (this cron, and the OneUptime queue monitor) correctly
+  ignores them. Their shape is distinctive: **`shippingCarrierId` set, `shippingMethodId` NULL**.
+  A naive `TrackingNumbers WHERE c_dtInTransitEmailSent IS NULL` count will read these as a backlog
+  that never drains.
+- **Clear an in-transit backlog by STAMPING, not by emailing — and refresh statuses first.**
+  Decided 2026-09-03 while draining the outage backlog
+  (`dbchanges2/Client_Compass/2026-09-03 - compass_intransit_email_backlog_backfill.sql`):
+  `c_dtInTransitEmailSent` was stamped on **64,132** rows older than the 14-day window and **421**
+  in-window rows already on a final status, leaving **68** genuinely still moving. Sending an
+  "in transit" email for a June order that arrived two months ago confuses the customer, so old
+  rows retire silently. **Ordering matters:** let worker2 `Sync/TrackingNumbers` refresh any
+  recently-unparked rows **before** running the backfill, otherwise a row with a stale status gets
+  a wrong in-transit email. (A parked row holds a stale status indefinitely — see
+  [tracking-status refresh](../../../2.0/apps/worker2/features/tracking-status-refresh.md).)
 - **Canada now uses the `ItemFulfillmentItems` line bridge, not the ASN chain**, for
   `getOrderFulfillmentData` — same query as Compass USA. Canada was originally on the ASN chain
   because it had **zero `ItemFulfillmentItems`**; if that line-bridge data is not populated for a
@@ -138,7 +262,7 @@ the tracking number as emailed (it retries next run).
   language even though the mail goes to the **manager**. Fixed 2026-08-03.
 - **⚠ Each cron keeps its OWN duplicated copy of the 2.0 `_status` CASE — a standing divergence
   risk.** `compass/update_salesorder_status_from_odp.php`,
-  `compasscanada/update_salesorder_status_from_odp.php` and
+  `compasscanada/update_salesorder_status_from_g&t.php` and
   `compass/workflow/test_partial_in_transit_email.php` each re-implement the model's status SQL as
   `computedStatusSlug`. Today that is safe by luck of scope: the copies only ever produce canceled /
   pendingApprovalUnknown / pendingFulfillment / shipped and they filter on
@@ -161,6 +285,27 @@ the tracking number as emailed (it retries next run).
 
 ## Change history
 
+- 2026-09-04 — **Fixed the Compass USA in-transit email outage (2026-08-27 → 2026-09-03, 130–190
+  emails/day → near zero).** Root cause: already-delivered rows were `continue`d without stamping
+  `c_dtInTransitEmailSent`, so the candidate set grew to 2,379 and every run burned its
+  `cronInitialization(3600)` hour inside rows dating back to a hardcoded `'2026-06-15'`, never
+  reaching new ASNs. Fix: two-pass drain (stamp finished rows, then send), newest-first
+  `ORDER BY MAX(dtCreated) DESC` with `IN_TRANSIT_EMAIL_BATCH_SIZE = 400`, rolling
+  `IN_TRANSIT_EMAIL_LOOKBACK_DAYS = 14` replacing the fixed date, and `isTrackingStatusFinal()`
+  short-circuiting the carrier call on `DELIVERED`/`RETURNED_TO_SENDER` using the status worker2
+  `Sync/TrackingNumbers` already maintains. Also **widened the recipient**: `LEFT OUTER JOIN Users`
+  + `Contacts`/`ContactEmailAddresses` with an ownership guard and
+  `IFNULL(Users.email, ContactEmailAddresses.emailAddress)`, which recovered 12 EDI-created MR
+  orders that have no `Users` row (regression-checked: 4,416 of 4,419 SA rows selected by both old
+  and new queries, 0 gained, 0 lost, identical resolved address/name). Backlog cleared by
+  **stamping** 64,132 out-of-window + 421 final-status rows rather than emailing them. Recorded
+  that **invoice tracking numbers are not in scope** (bridged via `InvoiceTrackingNumbers`, never
+  the ASN chain) and that a **`&` in a cron filename stops the cron dead**. Canada got the batching
+  changes only (renamed to `update_salesorder_status_from_g&t.php`; duplicate
+  `workflow/3_update_salesorder_status_from_grand_and_toy.php` deleted); its recipient block is
+  **still `INNER JOIN Users`**. New OneUptime monitor `Monitor/Compass/InTransitEmailQueue` now
+  watches this queue — see
+  [OneUptime push-metric monitors](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md). (bala)
 - 2026-08-26 — Flagged the **duplicated `_status` CASE** in the three crons as a standing divergence
   risk after the 2.0 Compass fulfillment rule was rewritten. Verified the crons are unaffected: their
   copies only emit canceled / pendingApprovalUnknown / pendingFulfillment / shipped and filter on

package/knowledge/1.0/apps/worker/workflows/tracing-a-worker-cron-run-in-production.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: workflow
 status: active
-updated: 2026-09-01
+updated: 2026-09-04
 owners: ["bala", "kyalamarthi"]
 files:
   - worker/ebs/cron.worker.php
@@ -170,7 +170,28 @@ to **2.0 only**. The 1.0 standard wins in `library/`, `worker/` and every other
 `library/app/systemmonitor/500error.php` uses `str_contains()`; that file is not a version
 signal, it is a latent bug.
 
+## Deploy-time trap: a `&` (or any shell metacharacter) in the cron FILENAME silently kills it
+`ebs/cron.worker.php:41` builds each crontab line by plain string concatenation
+(`'php /var/www/html/crons/' . $cron->cron`) with **no quoting and no escaping**. Cron then parses
+that line as a shell command, so a `&` in the filename is read as *"run everything before this in
+the background and start a new command."*
+
+Worked example (found 2026-09-03): a developer renamed the Compass Canada status cron to
+`crons/toga2/compasscanada/update_salesorder_status_from_g&t.php` and repointed
+`schedules/cron.worker.sync.json` at it. The crontab line becomes two commands —
+`php .../update_salesorder_status_from_g` (a file that does not exist) and a stray `t.php` — so the
+real script **never executes**, on every tick, with no error anywhere. It looks exactly like
+"the cron never fired".
+
+**Rule: a cron filename must be plain `[a-z0-9_]` plus `.php`.** No `&`, no spaces, no `;`, `|`,
+`$`, `(`, `)`, `*`, `?`, or quotes. The *display name* in `schedules/*.json` is free text and may
+contain anything; only the `cron` path is shell-parsed.
+
 ## Change history
+- 2026-09-04 — Added the **`&`-in-a-cron-filename trap**: `ebs/cron.worker.php:41` concatenates the
+  path into the crontab line unquoted, so a shell metacharacter splits the command and the script
+  never runs while leaving no trace. Found via the renamed Compass Canada
+  `update_salesorder_status_from_g&t.php`. (bala)
 - 2026-09-01 — Added **Step 6: check the email log, not the inbox** — a 1.0 cron's sends land in
   the **legacy** env's `Logs.Emails` (schema `Logs`, table `Emails`), never prod `Logs` or
   `Logs_<Client>`; gave the verification query, and noted that a missing row usually means the job

package/knowledge/2.0/apps/worker2/features/all-client-email-queue-monitor.md

@@ -6,8 +6,8 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-07-30
-owners: ["jcardinal"]
+updated: 2026-09-04
+owners: ["jcardinal", "bala"]
 files:
   - worker2/Worker/Monitor/Operations.php
   - _underscore/Database.php
@@ -111,6 +111,15 @@ matter are the **log database name** and whether the `Email` migration has been
 
 ## Gotchas / known issues
 
+- **⚠ This monitor is blind to an email that was never queued.** It reads `Logs_<Client>.Email`,
+  so it can only see a send that already produced a row. If the producing cron dies, stalls or
+  never selects the work, this monitor reports a **healthy empty queue**. That is exactly what
+  happened during the Compass USA in-transit outage (2026-08-27 → 2026-09-03): the 1.0 sending
+  cron timed out before creating a single email row and this monitor stayed green all week. Pair
+  any email-queue monitor with a **backlog monitor on the producer's input** — first instance
+  `Monitor/Compass/InTransitEmailQueue`, see
+  [OneUptime push-metric monitors](./oneuptime-worker2-monitoring.md).
+
 - **A client's database name is NOT derivable from `clientIdentifier`.** `Compass_Usa` →
   `Logs_Compass`. Always resolve `Clients.logDatabaseId` → `Databases.name`. Building
   `'Logs_' . $clientIdentifier` silently targets a nonexistent database. See
@@ -150,6 +159,10 @@ matter are the **log database name** and whether the `Email` migration has been
   against framework source only, not against real rows.
 
 ## Change history
+- 2026-09-04 — Recorded the monitor's **structural blind spot**: it only sees emails that reached
+  `Logs_<Client>.Email`, so a producer that never queues anything reads as healthy (proved by the
+  week-long Compass USA in-transit outage). Points at the new
+  `Monitor/Compass/InTransitEmailQueue` input-backlog monitor as the complement. (bala)
 
 - 2026-07-30 — Refactored `EmailQueue()` from Compass-only (hardcoded `Logs_Compass` via
   `initialize()` + `const DB_LOGS_COMPASS`, both removed) to an all-client fan-out

package/knowledge/2.0/apps/worker2/features/oneuptime-worker2-monitoring.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-09-03
+updated: 2026-09-04
 owners: ["jcardinal", "mhammontree", "bala"]
 files:
   - worker2/Worker/Monitor/Compass.php
@@ -23,6 +23,7 @@ files:
   - worker2/Config/sandbox-dev.ini
 related:
   - ./oneuptime-incident-clickup-sync.md
+  - ../../../1.0/apps/worker/features/compass-partial-in-transit-delivered-emails.md
   - ./monitoring-framework.md
   - ./all-client-email-queue-monitor.md
   - ../../_underscore/features/cloud-s3-helpers.md
@@ -36,7 +37,7 @@ A second, **OneUptime-reporting** monitoring pattern for the 2.0 worker2 tier, p
 the 1.0 `App_SystemMonitor_Compass` monitors. A worker2 cron action runs a self-contained
 check, decides pass/fail itself, and POSTs a JSON metric body to a OneUptime "Incoming
 Request" monitor — replacing the 1.0 tier's email alerts. The class is
-`abstract class _Worker_Monitor_Compass` and now carries a **suite of 9 monitors** covering
+`abstract class _Worker_Monitor_Compass` and now carries a **suite of 10 monitors** covering
 the Compass USA / Office Depot integration (S3, DB, and mailbox checks). Designed to be
 **client-agnostic** — a reusable template for future monitors and clients.
 
@@ -59,7 +60,7 @@ that reports straight to OneUptime. Note the folder difference: these live under
 - `_underscore/Cloud.php` — `_Cloud::getS3Objects()` list helper the S3 monitors rely on
   (see [Cloud S3 helpers](../../_underscore/features/cloud-s3-helpers.md)).
 
-## The 9 monitors
+## The 10 monitors
 
 Each measures one Compass/Office Depot integration metric and pushes a decided alarm token.
 - **OfficeDepotEdiImportQueue** (S3) — counts files >10 min old in
@@ -106,6 +107,11 @@ Each measures one Compass/Office Depot integration metric and pushes a decided a
   logic is superseded by this corrected definition.)
 - **OfficeDepotAs2Outbox** (S3) — counts ONLY top-level files in
   `s3://agilant-as2/OfficeDepot/OUTBOX/` (skips subfolders + directory placeholders).
+- **InTransitEmailQueue** (DB, added 2026-09-04) — watches the Compass USA **in-transit
+  shipping-notification queue**: `TrackingNumbers` rows on the ASN unit bridge with
+  `c_dtInTransitEmailSent IS NULL` inside a 14-day window (the window the 1.0 sending cron uses).
+  Status-page name *"Shipping Notification Emails (USA, In Transit)"*. See
+  [why it exists](#watch-the-step-before-the-log-row-exists) below.
 
 ## How it works
 
@@ -185,12 +191,58 @@ its error messages are the only documentation, so the verified shape:
 - `incidentSeverityId` is wrapped: `{_type: "ObjectID", value: "…"}`.
 - There is **no `projectId`** in the export, and **status/severity ObjectIDs are
   project-specific** — clone them out of an existing monitor's export rather than inventing them.
+- **⚠ Every description in the file is capped at 500 characters.** The importer rejects the whole
+  file with `description length cannot be more than 500 characters`. It applies to the **monitor**
+  description **and** to every **criteria** and **incident** description in the same file, so a
+  chatty runbook paragraph inside a criterion silently blocks the import. Keep each description
+  short and put the detail in the knowledge base.
 
 **Practical fix for the [activation-ordering trap](#gotchas--known-issues):** instead of
 importing the monitor twice (body criteria first, absence criteria after the first push), import
 the absence criteria **in the same file with `isEnabled: false`** and simply flip them on once a
 push has landed.
 
+### Watch the step BEFORE the log row exists
+
+A queue monitor that reads a **log** table can only see work that already produced a row. The
+all-client [Email Queue Monitor](./all-client-email-queue-monitor.md) watches
+`Logs_<Client>.Email` for `FAILED` and stale `PENDING` — excellent once an email row exists, and
+**structurally blind before that**. During the Compass USA in-transit outage (2026-08-27 →
+2026-09-03) the sending cron never got far enough to create a single email row, so that monitor
+reported a perfectly healthy, empty queue for a week while customers were not being told their
+orders had shipped.
+
+**Rule: for any producer→log pipeline, monitor the producer's INPUT backlog, not only its output
+log.** `InTransitEmailQueue` is the worked example — it counts unsent candidates in the source
+table, so the alarm fires even when nothing downstream ever runs. Its thresholds
+(`Monitor/Compass/InTransitEmailQueue`, `*/5 * * * *`):
+
+| Signal | DEGRADED | HIGH |
+|---|---|---|
+| `queuedRows` (unsent, in-window) | > 400 | > 800 |
+| `oldestQueuedHours` | > 5 | > 9 |
+| `lastStampHoursAgo` while `queuedRows > 0` | — | > 9 |
+| heartbeat not received | 10 min | 15 min |
+
+The row thresholds come straight from the sending cron's `IN_TRANSIT_EMAIL_BATCH_SIZE = 400`
+(past 400 it cannot catch up in one pass; 800 is two passes behind) and the hour thresholds from its
+`0 */4 * * *` schedule (worst normal wait is just under 4h, so 5h means a shipment missed its run
+and 9h means two). This is the *"calibrate against the tier that runs the business logic"* rule
+applied literally — see below. A DB read failure pushes `"status":"error"` so "we cannot see"
+never reads as healthy.
+
+**It reports `unroutableRows` (candidates with no emailable recipient) as a plain number that
+deliberately does not drive the alarm** — otherwise the monitor would sit red forever on rows the
+cron structurally cannot send. That is the *"backlog monitors must exclude known-permanently-stranded
+rows"* rule.
+
+> **⚠ Known divergence (2026-09-04):** the monitor decides routable/unroutable from the
+> `Users` join **only**, while the sending cron now also falls back to
+> `Contacts` / `ContactEmailAddresses`. So some rows counted as `unroutableRows` are in fact
+> sendable, and they are excluded from `queuedRows`. Port the cron's recipient fallback into the
+> monitor query to bring the two definitions back together — see
+> [Compass in-transit emails](../../../1.0/apps/worker/features/compass-partial-in-transit-delivered-emails.md).
+
 ### Heartbeat / cron cadence timing
 
 The 10-min-Degraded / 15-min-Offline heartbeat thresholds pair with a **5-minute** cron
@@ -317,9 +369,9 @@ Testing **recovery** is the step people skip and the most valuable one:
 ### CronJob scheduling for these monitors
 
 Run **all hours**, `*/5 * * * *` (every 5 min) — **not** the 1.0 `6-20` business-hours
-window. Integration backlogs must be caught outside business hours too. The nine
-`Monitor/Compass/*` actions are `Core.CronJobs` **ids 27–35** (EmailedAsnImport = id 31),
-each `maxExecutionTime` **120s** — the reason any in-run retry budget must stay well under 120s.
+window. Integration backlogs must be caught outside business hours too. The original nine
+`Monitor/Compass/*` actions are `Core.CronJobs` **ids 27–35** (EmailedAsnImport = id 31), with
+`InTransitEmailQueue` added on the same `*/5` cadence, each `maxExecutionTime` **120s** — the reason any in-run retry budget must stay well under 120s.
 
 ## DB monitor conventions
 
@@ -438,6 +490,16 @@ check for another client.
   monitors must pass the region (see [Cloud S3 helpers](../../_underscore/features/cloud-s3-helpers.md)).
 
 ## Change history
+- 2026-09-04 — Added a **10th monitor, `InTransitEmailQueue`** (Compass USA shipping-notification
+  backlog: unsent `TrackingNumbers` on the ASN unit bridge inside the cron's 14-day window;
+  DEGRADED >400 rows / >5h oldest, HIGH >800 rows / >9h oldest / >9h with nothing stamped while
+  rows wait; thresholds derived from the 1.0 cron's 400-row cap and `0 */4 * * *` schedule;
+  `unroutableRows` reported but never alarmed; DB failure pushes `status:"error"`). Wrote up the
+  general lesson it embodies — **monitor the producer's input backlog, not only its output log** —
+  after the all-client Email Queue Monitor showed a healthy empty queue through a week-long outage
+  in which no email row was ever created. Recorded the **500-character cap on every description**
+  in a OneUptime resource-import file (monitor, criteria and incident descriptions alike), and the
+  monitor-vs-cron `unroutableRows` divergence. (bala)
 - 2026-09-03 — **Fixed a false HIGH age alarm** in `CompassSalesOrderTransmission()`.
   `oldestPendingAgeMinutes` was measured from **stage-2 approval only**
   (`TIMESTAMPDIFF(MINUTE, ApprovalDecisionsStage2.dtDecision, NOW())`), but stage 2 is **not**

package/knowledge/2.0/apps/worker2/features/tracking-status-refresh.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-24
+updated: 2026-09-04
 owners: ["bala"]
 files:
   - worker2/Worker/Sync/TrackingNumbers.php
@@ -169,6 +169,42 @@ to slug **`beta`** or **`sandbox-dev`**, **not `dev`**, and those enumerate a **
 client set than `dev`. Confirm the box's actual `ENVIRONMENT` value before trusting the dispatcher's
 client enumeration on beta.
 
+### ⚠ A PARKED row holds a STALE status forever — and downstream code now trusts that status
+
+A row parked after repeated carrier failures gets `c_dtTrackingParked` /
+`c_trackingParkReason = 'MAX_FAILURES'` (20 failed lookups) and its `c_dtTrackingCheckNext` cursor
+stops being advanced, so **the row is never re-checked and `TrackingNumbers.status` freezes at
+whatever it last was** — typically an early `IN_TRANSIT`. Parking is silent: nothing distinguishes
+a stale status from a fresh one at the point of read.
+
+Measured 2026-09-03: **189 Compass rows created 2026-08-20 → 2026-08-29 were parked on
+`MAX_FAILURES`.** Unparking restores them:
+
+```sql
+UPDATE TrackingNumbers
+SET
+	c_dtTrackingParked = NULL,
+	c_trackingParkReason = NULL,
+	c_trackingCheckFailures = 0,
+	c_dtTrackingCheckNext = NOW()
+WHERE
+	c_trackingParkReason = 'MAX_FAILURES'
+```
+
+The refresh cron is **`Core.CronJobs` id 69, action `Sync/TrackingNumbers/Refresh`, every 10
+minutes**, so the unparked rows pick up a real status within a couple of ticks.
+
+**Why this now matters more than it used to.** The 1.0 Compass in-transit email cron short-circuits
+its carrier call when `TrackingNumbers.status` is already `DELIVERED` / `RETURNED_TO_SENDER`, i.e.
+it **trusts this column**. A parked row therefore keeps a shipment in the email queue (or, in the
+opposite direction, can drive a wrong decision) with no visible error. **Before any bulk operation
+that reads `TrackingNumbers.status` — a backfill, a reconciliation, a mass stamp — unpark and let
+the refresh catch up first.** See
+[Compass in-transit emails](../../../1.0/apps/worker/features/compass-partial-in-transit-delivered-emails.md).
+
+Open question: nothing today re-tries or reports parked rows. A periodic unpark (or a monitor on
+parked-row count) is unbuilt work.
+
 ### ⚠ Conflict with the 1.0 Compass delivered-email cron
 The in-transit / delivered emails **today** come from a separate **1.0** Compass worker cron
 (`worker/crons/toga2/compass/send_delivered_email.php`) that polls UPS itself, sends via api2
@@ -190,6 +226,14 @@ Plan (not built): move the email logic into a Compass `postPut` interceptor and
   check, and **resolve the PHP-8.5 worker → api2 crash** (the gate).
 
 ## Change history
+- 2026-09-04 — Recorded that **parked rows hold a stale status indefinitely**: 189 Compass rows
+  (created 2026-08-20 → 08-29) sat on `c_trackingParkReason = 'MAX_FAILURES'` after 20 failed
+  carrier lookups, with `c_dtTrackingCheckNext` no longer advancing, so `TrackingNumbers.status`
+  froze silently. Added the unpark recipe and noted the cron identity (`Core.CronJobs` id 69,
+  `Sync/TrackingNumbers/Refresh`, `*/10`). This got sharper because the 1.0 Compass in-transit
+  email cron now **trusts** `TrackingNumbers.status` to skip a carrier call, so any bulk read of
+  that column must unpark and let the refresh catch up first. No periodic unpark or parked-row
+  monitor exists yet. (bala)
 - 2026-08-24 — Built the centralized tracking-status refresh: dispatcher + per-client child
   (`_Worker_Sync_TrackingNumbers`), per-row `c_dtTrackingCheckNext` re-check cursor (+1h imminent /
   +6h else, jitter, terminal → NULL, 45-day cutoff), `PAGE_CAP=50`/`SOFT_TIME_BUDGET_SECONDS=40`

package/knowledge/INDEX.md

@@ -18,7 +18,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 
 ## 2.0 framework
 
-- **_underscore** (_Underscore) _(framework core)_ — 80 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
+- **_underscore** (_Underscore) _(framework core)_ — 81 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
 - **worker2** (Worker) — 59 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
 - **api2** (API) — 25 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
 - **dbchanges2** (Database Changes) _(framework core)_ — 15 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)

package/knowledge/clients/compass-canada/INDEX.md

@@ -3,8 +3,8 @@
 | Doc | Framework | Summary | Files |
 |-----|-----------|---------|-------|
 | [French (fr-CA) Item Feature Translations (Compass Canada)](features/french-item-feature-translations.md) | 2.0 | Renders item **feature** text on the Compass Canada French storefront — feature names, feature-group headers (e.g. | _underscore/Model/Compass/Canada/Feature.php, _underscore/Model/Compass/Canada/ItemCategoryFeatureGroup.php, _underscore/Model/Client/FeatureTranslation.php, _underscore/Model/Client/ItemCategoryFeatureGroupTranslation.php, _underscore/Model/Client/ItemFeatureTranslation.php, worker2/Worker/Etilize/ItemTranslations.php, dbchanges2/Client/2026-07-13a - FeatureTranslations.sql, dbchanges2/Client/2026-07-13b - FeatureTranslationsAcl.sql, dbchanges2/Core/2026-07-13 - FeatureTranslationsRecord.sql, dbchanges2/Client_CompassCanada/2026-07-13 - FeatureAttributeCustomFields.sql, dbchanges2/Client_CompassCanada/2026-07-13 - DedupeItemFeaturesAndGroups.sql, dbchanges2/Client_CompassCanada/2026-07-13 - SeedFrenchFeatureTranslations.sql |
-| [French (fr-CA) Order Email Localization (Compass Canada)](features/french-order-email-localization.md) | 2.0 | Compass Canada order emails (order requested, manager-approval request, approved/rejected, in-transit, delivered, reminders) are sent in **each recipient's own | _underscore/Model/Compass/SalesOrder.php, _underscore/Model/Compass/ApprovalDecision.php, _underscore/Component/Api/Togaiq/Togaiq.php, _underscore/String.php, library/app/client/compasscanada.php, worker/crons/toga2/compasscanada/send_delivered_email.php, worker/crons/toga2/compasscanada/compass_email_reminders.php, worker/crons/toga2/compasscanada/compass_cancel_pending_approval_orders.php, worker/crons/toga2/compasscanada/update_salesorder_status_from_odp.php, worker/crons/toga2/compasscanada/workflow/3_update_salesorder_status_from_grand_and_toy.php, worker/crons/toga2/compasscanada/workflow/4_import_grand_and_toy_advance_shipping_notices.php, worker/crons/toga2/compasscanada/workflow/test_partial_in_transit_email.php, worker/crons/toga2/compasscanada/workflow/test_partial_delivered_email.php, worker/crons/toga2/compasscanada/workflow/test_email_previews_prod.php |
-| [Grand & Toy ASN Import (Compass Canada)](features/grand-and-toy-asn-import.md) | 2.0 | Imports Grand & Toy (G&T) Advance Shipping Notices for Compass Canada. | api2/Component/Api/Cxml/Cxml.php, worker/crons/toga2/compasscanada/workflow/4_import_grand_and_toy_advance_shipping_notices.php, worker/crons/toga2/compasscanada/workflow/import_grand_and_toy_asn_from_file.php, worker/crons/toga2/compasscanada/workflow_beta/import_grand_and_toy_asn_from_file.php, worker/schedules/cron.worker.sync.json, _underscore/Model/Compass/AdvanceShippingNotice.php, _underscore/Model/Compass/Canada/AdvanceShippingNotice.php, dbchanges2/Client_CompassCanada/ |
+| [French (fr-CA) Order Email Localization (Compass Canada)](features/french-order-email-localization.md) | 2.0 | Compass Canada order emails (order requested, manager-approval request, approved/rejected, in-transit, delivered, reminders) are sent in **each recipient's own | _underscore/Model/Compass/SalesOrder.php, _underscore/Model/Compass/ApprovalDecision.php, _underscore/Component/Api/Togaiq/Togaiq.php, _underscore/String.php, library/app/client/compasscanada.php, worker/crons/toga2/compasscanada/send_delivered_email.php, worker/crons/toga2/compasscanada/compass_email_reminders.php, worker/crons/toga2/compasscanada/compass_cancel_pending_approval_orders.php, worker/crons/toga2/compasscanada/update_salesorder_status_from_g&t.php, worker/crons/toga2/compasscanada/workflow/3_update_salesorder_status_from_grand_and_toy.php, worker/crons/toga2/compasscanada/workflow/3_import_grand_and_toy_advance_shipping_notices.php, worker/crons/toga2/compasscanada/workflow/test_partial_in_transit_email.php, worker/crons/toga2/compasscanada/workflow/test_partial_delivered_email.php, worker/crons/toga2/compasscanada/workflow/test_email_previews_prod.php |
+| [Grand & Toy ASN Import (Compass Canada)](features/grand-and-toy-asn-import.md) | 2.0 | Imports Grand & Toy (G&T) Advance Shipping Notices for Compass Canada. | api2/Component/Api/Cxml/Cxml.php, worker/crons/toga2/compasscanada/workflow/3_import_grand_and_toy_advance_shipping_notices.php, worker/crons/toga2/compasscanada/workflow/import_grand_and_toy_asn_from_file.php, worker/crons/toga2/compasscanada/workflow_beta/import_grand_and_toy_asn_from_file.php, worker/schedules/cron.worker.sync.json, _underscore/Model/Compass/AdvanceShippingNotice.php, _underscore/Model/Compass/Canada/AdvanceShippingNotice.php, dbchanges2/Client_CompassCanada/ |
 | [Compass Canada — Item-Fulfillment TableViews (dead-Core-id 500, rebuilt onto the USA shape)](features/item-fulfillment-tableviews.md) | 2.0 | `Client_CompassCanada`'s two item-fulfillment table views — **`item-fulfillments-for-sales-orders`** (`TableViews.id 12`) and **`item-fulfillments-for-sales-ord | dbchanges2/Client_CompassCanada/2026-08-27a - ItemFulfillmentTableViewsRebuild.sql, dbchanges2/Client_CompassCanada/2026-08-26 - EnableTableViewFieldFilters.sql, _underscore/Model/Client/TableView.php |
 | [Compass Canada — Fulfilled vs Partially Fulfilled (per-line, ASN bridge only)](features/order-fulfillment-status-per-line.md) | 2.0 | TOGa Supply showed **Fulfilled** on Compass Canada orders that were only partly shipped. | _underscore/Model/Compass/Canada/SalesOrder.php, _underscore/Model/Compass/SalesOrder.php |
 | [Compass Canada](profile.md) | 2.0 | Compass Canada is the Canadian arm of the Compass account — a separate TOGA tenant, related to but distinct from Compass USA. |  |

package/knowledge/clients/compass-canada/features/french-order-email-localization.md

@@ -16,9 +16,9 @@ files:
   - worker/crons/toga2/compasscanada/send_delivered_email.php
   - worker/crons/toga2/compasscanada/compass_email_reminders.php
   - worker/crons/toga2/compasscanada/compass_cancel_pending_approval_orders.php
-  - worker/crons/toga2/compasscanada/update_salesorder_status_from_odp.php
+  - worker/crons/toga2/compasscanada/update_salesorder_status_from_g&t.php
   - worker/crons/toga2/compasscanada/workflow/3_update_salesorder_status_from_grand_and_toy.php
-  - worker/crons/toga2/compasscanada/workflow/4_import_grand_and_toy_advance_shipping_notices.php
+  - worker/crons/toga2/compasscanada/workflow/3_import_grand_and_toy_advance_shipping_notices.php
   - worker/crons/toga2/compasscanada/workflow/test_partial_in_transit_email.php
   - worker/crons/toga2/compasscanada/workflow/test_partial_delivered_email.php
   - worker/crons/toga2/compasscanada/workflow/test_email_previews_prod.php

package/knowledge/clients/compass-canada/features/grand-and-toy-asn-import.md

@@ -9,7 +9,7 @@ updated: 2026-08-20
 owners: ["bala"]
 files:
   - api2/Component/Api/Cxml/Cxml.php
-  - worker/crons/toga2/compasscanada/workflow/4_import_grand_and_toy_advance_shipping_notices.php
+  - worker/crons/toga2/compasscanada/workflow/3_import_grand_and_toy_advance_shipping_notices.php
   - worker/crons/toga2/compasscanada/workflow/import_grand_and_toy_asn_from_file.php
   - worker/crons/toga2/compasscanada/workflow_beta/import_grand_and_toy_asn_from_file.php
   - worker/schedules/cron.worker.sync.json
@@ -65,7 +65,7 @@ serial numbers and the reference for the field mapping.
 > non-prod patch also evaporates on the next tenant reseed — the same lesson as AIG TRUE-80562.)
 
 ## Key files / entry points
-- `worker/crons/toga2/compasscanada/workflow/4_import_grand_and_toy_advance_shipping_notices.php`
+- `worker/crons/toga2/compasscanada/workflow/3_import_grand_and_toy_advance_shipping_notices.php`
   — the scheduled cron (1.0 worker tier). Reads mailbox `compasscanada.status@togatech.com`
   (OAuth2 creds — currently hardcoded with a TODO; must move to `App_Registry::get('config')['compasscanada']['gt_asn_oauth2']`
   before production), parses the 24-column CSV, posts to api2 `/advance-shipping-notices`.

package/knowledge/clients/compass-canada/features/order-fulfillment-status-per-line.md

@@ -160,7 +160,7 @@ them correctly read **pendingFulfillment**.
 - **The badge is display-only and never stored.** The status is computed live, so there is no
   backfill and reverting the code fully reverts the behaviour. **No cron, email or report reads the
   fulfilled vs partiallyFulfilled distinction.** Canada's in-transit email cron
-  (`worker/crons/toga2/compasscanada/update_salesorder_status_from_odp.php`) keeps its **own
+  (`worker/crons/toga2/compasscanada/update_salesorder_status_from_g&t.php`) keeps its **own
   duplicated copy** of the `_status` CASE, but it only ever produces canceled /
   pendingApprovalUnknown / pendingFulfillment / shipped and filters on
   `computedStatusSlug = 'shipped'`. Note the Canada pendingFulfillment gate is **no longer

package/knowledge/clients/compass-usa/INDEX.md

@@ -4,6 +4,7 @@
 |-----|-----------|---------|-------|
 | [Compass Approval-Decision Flow (Notifications & Manager Reassignment)](features/approval-decision-flow.md) | 2.0 | Compass's sales-order approval flow — approval/notification email lists, **manager reassignment**, VIP auto-approve, and EN/FR localization — lives **entirely i | _underscore/Model/Compass/ApprovalDecision.php, _underscore/Model/Compass/SalesOrder.php, _underscore/Model/Compass/Usa/ApprovalDecision.php, _underscore/Model/Compass/Canada/ApprovalDecision.php, _underscore/Model/Client/ApprovalTemplateStage.php, _underscore/Model/Quad/SalesOrder.php |
 | [Compass ASN → ItemFulfillment Auto-Creation](features/asn-to-item-fulfillment.md) | 2.0 | For Compass USA, posting an AdvanceShippingNotice (ASN) auto-creates the ItemFulfillment (IF) on the upstream SalesOrder. | _underscore/Model/Compass/AdvanceShippingNotice.php, _underscore/Model/Compass/PurchaseOrder.php, api2/Component/Api/Cxml/Cxml.php, dbchanges2/Client_Compass/2026-06-11 - AsnItemTrackingNumberAcl.sql, dbchanges2/Client_Compass/2026-06-15b - BackfillSA132781ItemFulfillmentTracking.sql, dbchanges2/Client_Compass/2026-06-16 - CleanupSA132763CrossLineTracking.sql, dbchanges2/Client_Compass/2026-06-16b - CleanupSA132743CrossLineTracking.sql, dbchanges2/Client_Compass/2026-06-16c - BackfillSA132763C40QYUCTracking.sql, dbchanges2/Client_Compass/2026-06-18a - CleanupSA132898DuplicateTracking.sql, dbchanges2/Client_Compass/2026-06-18b - CleanupSA132881DuplicateTracking.sql, dbchanges2/Client_Compass/2026-07-02a - FixSA133377TrackingSerialAndDuplicateIF.sql |
+| [Compass Contact Email Resolution & Crossed `primaryContactEmailAddressId` Pointers](features/contact-email-resolution.md) | 2.0 | **Any Compass code that reads a contact's email address must also assert that the email row belongs to that contact.** Verified in prod on 2026-09-03: **42,926 | _underscore/Model/Compass/PurchaseOrder.php, worker/crons/toga2/compass/update_salesorder_status_from_odp.php |
 | [Cost Centers — Unit Locations, numeric-only policy](features/cost-centers.md) | 2.0 | A Compass "cost center" — the value a user picks in commerce and that lands on an order — is **not** a `CostCenters` row. | toga2-commerce/src/pages/Cart/api/CartApi.ts, worker1.5/crons/toga2/compass/import_locations.php, _underscore/Model/Compass/SalesOrder.php, api2/Component/Api/V2/V2.php, dbchanges2/Client_Compass/2026-07-06 - RemoveNonNumericCostCenters.sql |
 | [Compass isFulfillable — Data Quality & the Type-Derived Rule](features/isfulfillable-data-quality-and-type-rule.md) | 2.0 | A **prod-data investigation (2026-08-12, read-only)** into why so many Compass storefront lines still have a dead **Qty Fulfilled** cell. | _underscore/Model/Client/Item.php, library/app/api/toga2.php, worker/crons/toga2/netsuite/backfill_isfulfillable_all_clients.php, dbchanges2/Core/2026-07-17 - RegisterItemIsFulfillableInterceptors.sql, toga2-supply/src/pages/Orders/view/OrderView/components/sections/OrderItemsTableSection.tsx |
 | [Compass Item Catalogs (Compass vs Agilant) & the Duplicate Items the NetSuite Sync Creates](features/item-catalogs-and-duplicate-items.md) | 1.0 | `Client_Compass.Items` is **split across catalogs by which pipeline created the item**, and the two halves are routinely mistaken for one. | library/app/api/toga2.php, _underscore/Model/Compass/PurchaseOrder.php, _underscore/Model/Compass/Usa/PurchaseOrder.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/compass/workflow/3a_import_office_depot_purchase_orders.php, worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php |

package/knowledge/clients/compass-usa/features/contact-email-resolution.md

@@ -0,0 +1,149 @@
+---
+title: Compass Contact Email Resolution & Crossed `primaryContactEmailAddressId` Pointers
+framework: "2.0"
+repo: _underscore
+project: _Underscore
+client: compass-usa
+type: client-feature
+status: active
+updated: 2026-09-04
+owners: ["bala"]
+files:
+  - _underscore/Model/Compass/PurchaseOrder.php
+  - worker/crons/toga2/compass/update_salesorder_status_from_odp.php
+related:
+  - ./mits-po-to-so-item-linking.md
+  - ./people-file-user-lifecycle.md
+  - ../profile.md
+  - ../../../1.0/apps/worker/features/compass-partial-in-transit-delivered-emails.md
+---
+
+## Summary
+
+**Any Compass code that reads a contact's email address must also assert that the email row
+belongs to that contact.** Verified in prod on 2026-09-03: **42,926 of 313,660** Compass contacts
+(**~14%**) have `Contacts.primaryContactEmailAddressId` pointing at a `ContactEmailAddresses` row
+whose `contactId` is a **different** contact. Read the pointer without that guard and you mail the
+wrong human.
+
+A second, related defect sits one level up: on **MR** orders the sales order itself is sometimes
+attached to the **wrong contact**, while the MITS payload named the right person all along. So when
+a Compass contact email "looks wrong" on an MR order, the usual truth is *the email is correct and
+the SO-to-contact link is broken* — check the link before you touch any email data.
+
+## The correct way to read a Compass contact's email
+
+```sql
+INNER JOIN Contacts
+    ON Contacts.id = <someRecord>.contactId
+LEFT OUTER JOIN ContactEmailAddresses
+    ON ContactEmailAddresses.id = Contacts.primaryContactEmailAddressId
+    AND ContactEmailAddresses.contactId = Contacts.id      # <- the ownership guard
+```
+
+The `AND ContactEmailAddresses.contactId = Contacts.id` predicate is the whole point. Without it
+the join happily follows a pointer into another contact's mailbox. First production consumer to
+carry the guard: the Compass USA in-transit email cron
+(`worker/crons/toga2/compass/update_salesorder_status_from_odp.php`) — see
+[Compass partial in-transit & delivered emails](../../../1.0/apps/worker/features/compass-partial-in-transit-delivered-emails.md).
+
+Note the guard makes the join *fail closed*: a crossed contact resolves to **no** address rather
+than to a wrong one. That is the desired trade — a missing email is a visible gap, a wrong email is
+a customer-facing incident.
+
+## Why the pointers are crossed
+
+The Compass tenant carries a large population of **duplicate contacts** created by the MITS
+purchase-order import (e.g. **"Ben Berdan" exists 4 times**). Duplicates are created in adjacent id
+ranges, and the pointer/owner pairing drifts between them.
+
+The mechanism is visible in `_Model_Compass_PurchaseOrder::prePost()` (`handleMrOrder` path,
+`_underscore/Model/Compass/PurchaseOrder.php` ~lines 72–105). Before creating the SO it resolves the
+contact like this:
+
+```php
+$api->internalApiRequest('GET', '/contacts', null, [
+    'join'           => [['ContactEmailAddresses' => ['ContactEmailAddresses.id' => 'Contacts.primaryContactEmailAddressId']]],
+    'where'          => ['and' => [['ContactEmailAddresses.emailAddress' => ['=' => $contactEmail]]]],
+    'recordsPerPage' => 1,
+]);
+// no match -> POST /contacts creates a brand-new contact
+```
+
+Three properties of that lookup matter:
+
+1. **No ownership guard** — it joins on `ContactEmailAddresses.id = Contacts.primaryContactEmailAddressId`
+   only, so a contact that *points at* someone else's email row is a valid match for that email.
+2. **`recordsPerPage 1` with no `ORDER BY`** — when several contacts match, which one comes back is
+   arbitrary and can change between calls.
+3. **Miss → create** — a lookup that fails to see the existing person creates yet another duplicate,
+   which feeds the next round.
+
+## Consequence: MR sales orders linked to the wrong contact
+
+The MITS purchase-order payload names the real person in `contact._name` and
+`contact.primaryContactEmailAddress.emailAddress`, but the SO ends up attached to a **neighbouring
+contact id**. Confirmed cases (prod, 2026-09-03):
+
+| MR order | SO is linked to | MITS payload actually named |
+|---|---|---|
+| MR245526 | contact **205042** "Luis Cervantes" | "Rusty Evans" / `Rusty.Evans@compass-usa.com` = contact **205041** |
+| MR245513 | Eric Campbell | Benjamin Berdan |
+| MR245463, MR244893, MR244699, MR245221 | same shape | — |
+
+Note the adjacent ids (205041 / 205042) — the hallmark of duplicates created together by the import.
+
+**Triage rule: the payload is the source of truth.** `Logs_Compass.Api` holds the inbound MITS
+`POST /v2/purchase-orders` body; compare `contact.primaryContactEmailAddress.emailAddress` there
+against the SO's linked contact before concluding the email data is wrong.
+
+**10 such orders were deliberately left unfixed pending business input** — repointing a shipped SO
+to a different person is a business decision, not a data cleanup.
+
+## The 2026-09-03 data fix (32 safe MR cases)
+
+`compass_mr_contact_email_ownership_2026-09-03.sql` — run in prod, all 32 verified. The safe repair
+shape, for the cases where the crossed contact and the owner hold the **same address**:
+
+1. Insert a `ContactEmailAddresses` row **owned by the crossed contact**, carrying the same
+   `emailAddress`.
+2. Repoint that contact's `Contacts.primaryContactEmailAddressId` at its own new row.
+
+Nobody's address changes; each contact simply stops borrowing another contact's row. This is only
+valid when the two addresses already match — a differing address is a real identity question and
+must not be auto-resolved.
+
+The remaining ~42,900 crossed contacts were **not** mass-repaired. Consumers carry the ownership
+guard instead, which is safe today; a bulk cleanup is still open work.
+
+## Gotchas / known issues
+
+- **⚠ The pointer is crossed on ~14% of Compass contacts — treat an unguarded read as a bug.**
+  This applies to reports, exports, notification crons, api2 payload interceptors, anything.
+- **A `Users` row is a different lookup and does not have this problem.** `Users.email` is keyed by
+  `Users.contactId` directly. Where both exist, prefer `Users.email` and fall back to the guarded
+  `ContactEmailAddresses` read — that is the `IFNULL(Users.email, ContactEmailAddresses.emailAddress)`
+  precedence used by the in-transit cron. EDI-created MR orders have **no** `Users` row at all,
+  which is why the fallback exists.
+- **`_Model_Compass_PurchaseOrder`'s contact lookup still has no ownership guard** (as of
+  2026-09-04). Fixing it there stops *new* crossings; the existing 42,926 remain either way.
+- **Compass Canada shares the model** (`Model/Compass/Canada/PurchaseOrder.php` inherits the parent),
+  so the same lookup shape runs against `Client_CompassCanada`. Not measured there.
+- Duplicate-contact creation and duplicate-**user** creation are separate problems with separate
+  causes — the `Users` duplicates come from the PEOPLE-file reactivation window, see
+  [PEOPLE-file user lifecycle](./people-file-user-lifecycle.md).
+
+## Change history
+
+- 2026-09-04 — Documented that **42,926 / 313,660 (~14%) Compass contacts have
+  `Contacts.primaryContactEmailAddressId` pointing at another contact's `ContactEmailAddresses`
+  row** (verified prod 2026-09-03), and established the mandatory
+  `ContactEmailAddresses.contactId = Contacts.id` ownership guard for every email read. Traced the
+  cause to `_Model_Compass_PurchaseOrder`'s MR contact lookup (joins on the pointer only, no
+  ownership guard, `recordsPerPage 1` with no `ORDER BY`, miss creates a duplicate) and to the
+  resulting duplicate contacts from the MITS PO import. Recorded the related defect that **MR sales
+  orders can be linked to the wrong contact** while the MITS payload named the right person
+  (MR245526, MR245513, MR245463, MR244893, MR244699, MR245221), with the triage rule to trust the
+  logged payload. Fixed 32 safe MR cases via
+  `compass_mr_contact_email_ownership_2026-09-03.sql` (give each contact its own email row with the
+  same address, then repoint); 10 ambiguous orders left for business input. (bala)

package/knowledge/clients/compass-usa/features/order-fulfillment-status-per-line.md

@@ -204,7 +204,7 @@ Re-confirmed 2026-08-27: **no cron, email or report reads the distinction.**
   behaviour.
 - **In-transit / delivered email crons are unaffected.**
   `worker/crons/toga2/compass/update_salesorder_status_from_odp.php`,
-  `worker/crons/toga2/compasscanada/update_salesorder_status_from_odp.php` and
+  `worker/crons/toga2/compasscanada/update_salesorder_status_from_g&t.php` and
   `compass/workflow/test_partial_in_transit_email.php` each keep their **own duplicated copy** of the
   `_status` CASE, but those copies only ever produce canceled / pendingApprovalUnknown /
   pendingFulfillment / shipped and they filter on `computedStatusSlug = 'shipped'`. **That

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.773",
+  "version": "1.0.774",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",