toga-ai 1.0.817 → 1.0.819

package/knowledge/1.0/apps/store/features/saml-sso-certificate-and-keys.md

@@ -76,6 +76,23 @@ This mirrors the overlap the 2.0 `saml` gateway provides, though the gateway now
 a multi-cert list out of Secrets Manager rather than two files on disk — see the SP
 cert-rotation workflow.
 
+### Two ACS handlers, two SAML libraries — tell them apart from the stack trace
+
+`store/_/app/framework.php` has **two** separate `case 'acs'` handlers that use **different SAML
+libraries**. They are easy to confuse when reading a production stack trace, and they run on
+**different servers**:
+
+| Handler | Library | Host / EB environment |
+|---|---|---|
+| The `togahub` host block's `case 'acs'` (the `$samlKeyPairs` dual-keypair decrypt) | **LightSAML** (`decryptMultiAssertion`) | `prudential.togahub.com` → `prudential-store-prod` |
+| The `prudential` host block's `case 'acs'` | **OneLogin_Saml2_Auth** (`isAuthenticated()`, `getNameId()`, `getAttribute('givenname'/'surname'/'emailaddress')`) | `prudential.togacommerce.com` → `true-togacommerce-prod` |
+
+**Diagnostic rule:** a trace with `LightSaml\Model\Assertion\*` frames is the **togahub** block
+(`prudential-store-prod`). A trace with `OneLogin` frames is the **prudential** block
+(`true-togacommerce-prod`). This is how a production decrypt error gets pinned to the right
+server — and the two are deployed by different pipeline actions, so fixing the wrong one changes
+nothing. See the Prudential store SAML deploy-topology workflow.
+
 ### Private key loading
 
 The three inline `$settingsInfo` SP blocks read the key from disk:
@@ -108,6 +125,23 @@ key; it is a record of what we do, not a recommendation.
 - **Don't assume where the ACS runs.** The host that terminates `?acs` for a client may be a
   different EB environment from the storefront. See the Prudential store SAML deploy-topology
   workflow before deploying a cert change.
+- **A decrypt stack trace CANNOT tell you how many keys were tried.** In
+  `library/LightSaml/Model/Assertion/EncryptedElementReader.php`, `decryptMulti()` returns
+  `$this->decrypt($key)` from *inside* the `foreach`, and re-throws `$lastException` after the
+  loop. The re-thrown exception keeps the stack from where it was first thrown, so the captured
+  trace looks **identical** whether one key was tried or several all failed. Seeing
+  `decryptMultiAssertion` in a trace therefore does **not** prove the dual-keypair code is
+  deployed — the old single-key code called the same method with a one-item array. To tell them
+  apart, check the deployed EB `Deployment.VersionLabel`, or look for the new code's distinct log
+  line `SAML keypair not readable, skipping:`.
+- **A mismatched cert/key pair fails with an OAEP padding error, not a clear message.** The
+  symptom is `Failure decrypting Data (openssl private) -
+  RSA_padding_check_PKCS1_OAEP_mgf1: oaep decoding error` (also seen as
+  `RSA_EAY_PRIVATE_DECRYPT: padding check failed`) thrown out of the LightSAML
+  `EncryptedElementReader`. Confirm a suspected mismatch by comparing the **modulus** of the cert
+  and the key with `openssl x509 -noout -modulus` / `openssl rsa -noout -modulus` — identical
+  modulus means the pair matches. Compare the **committed git blobs**, not just the local working
+  files, or you will verify a pair that was never deployed.
 
 ## Client_True migration (deferred 2026-09-15)
 
@@ -136,6 +170,14 @@ What the migration must handle:
   architecture doc's Known issues.
 
 ## Change history
+- 2026-09-15 — Documented that `framework.php` has two `case 'acs'` handlers on two different
+  SAML libraries (LightSAML for `togahub` vs OneLogin for `prudential`) running on two different
+  EB environments, and how to read a stack trace to tell which one failed. Recorded that a
+  LightSAML `decryptMulti()` trace cannot reveal how many keys were tried (re-thrown exception
+  keeps the original stack), so `decryptMultiAssertion` in a trace does not prove the
+  dual-keypair build is deployed — check the EB VersionLabel or the
+  `SAML keypair not readable, skipping:` log line. Added the OAEP-padding-error signature of a
+  mismatched cert/key pair and the openssl-modulus check against committed blobs. (rgirish)
 - 2026-09-15 — Rotated the `togahub.com` SP cert to the ACM-issued `togahub_com_2027-03-31.crt`
   and added a dual-keypair fallback in `?acs` so both the old and new certs decrypt during the
   overlap (`decryptMultiAssertion`). Removed three hardcoded RSA private-key blobs from

package/knowledge/1.0/apps/worker/features/compass-partial-in-transit-delivered-emails.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-09-10
+updated: 2026-09-16
 owners: ["bala"]
 files:
   - worker/crons/toga2/compass/update_salesorder_status_from_odp.php
@@ -18,8 +18,10 @@ files:
   - library/app/client/compasscanada.php
   - worker/schedules/cron.worker.sync.json
   - worker2/Worker/Monitor/Compass.php
+  - worker/crons/toga2/compass/workflow/3b_import_strategic_systems_advance_shipping_notices.php
   - dbchanges2/Client_Compass/2026-09-03 - compass_intransit_email_backlog_backfill.sql
 related:
+  - ../../../clients/compass-usa/features/tracking-number-origin-and-asn-link.md
   - ../../../clients/compass-usa/features/contact-email-resolution.md
   - ../../../2.0/apps/worker2/features/tracking-status-refresh.md
   - ../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md
@@ -56,9 +58,11 @@ and earlier packages). The dynamic HTML is injected into a stored `EmailTemplate
 ## How it works
 
 1. **Driving query** finds shipped sales orders and their tracking numbers (`computedStatusSlug
-   = 'shipped'`, `c_dtInTransitEmailSent IS NULL`, rolling date window) via the ASN chain
+   = 'shipped'`, `c_dtInTransitEmailSent IS NULL`, rolling date window on
+   **`AdvanceShippingNotices.dtCreated`**) via the ASN chain
    (`AdvanceShippingNoticeItemUnits_TrackingNumbers`), one row per `TrackingNumbers.number`,
-   **newest first and capped** — see *Draining the candidate set* below.
+   **newest first and capped** — see *Draining the candidate set* and *Which clock the window
+   runs on* below.
 2. Per tracking number: skip if the shipment is already finished — checked first against the
    stored `TrackingNumbers.status` (free), and only then with a live carrier call.
 3. **`getOrderFulfillmentData`** returns the order's packages (grouped by tracking number,
@@ -73,6 +77,13 @@ and earlier packages). The dynamic HTML is injected into a stored `EmailTemplate
    `App_Api_Toga2::send(..., 'POST', '/email-templates/sendEmail', $payload, [], true)`.
 6. On a successful (non-throwing) send, set `c_dtInTransitEmailSent = NOW()`.
 
+**The two loops behave differently, on purpose.** Loop 1 stamps `c_dtInTransitEmailSent` on any
+row whose `TrackingNumbers.status` is in `TRACKING_FINAL_STATUSES` (`DELIVERED`,
+`RETURNED_TO_SENDER`) and **sends nothing**. Loop 2 emails the rest and stamps after a successful
+send. So a parcel that was already delivered by the time the cron reaches it is **silently
+retired, never announced** — expected behaviour, and the reason a genuine gap can show up as
+"22 rows missed, 1 email actually owed".
+
 ### Draining the candidate set: two-pass, newest-first, bounded
 
 **This is the load-bearing part of the cron. Getting it wrong took Compass USA in-transit emails to
@@ -95,9 +106,10 @@ mark the row done"* cron:
 2. **Newest first, hard cap.** `ORDER BY MAX(TrackingNumbers.dtCreated) DESC` +
    `LIMIT IN_TRANSIT_EMAIL_BATCH_SIZE` (**400**). Today's shipments are served before any backlog,
    so a backlog can delay old mail but can never starve new mail.
-3. **Rolling window, not a fixed date.** `IN_TRANSIT_EMAIL_LOOKBACK_DAYS = 14` →
-   `dtCreated >= DATE_SUB(NOW(), INTERVAL 14 DAY)`. A hardcoded cutoff date guarantees an
-   ever-growing scan; a rolling window is self-limiting.
+3. **Rolling window, not a fixed date.** `IN_TRANSIT_EMAIL_LOOKBACK_DAYS` →
+   `dtCreated >= DATE_SUB(NOW(), INTERVAL N DAY)`. A hardcoded cutoff date guarantees an
+   ever-growing scan; a rolling window is self-limiting. **Which `dtCreated` matters enormously —
+   see the next section.** (USA is at 7 days as of 2026-09-15; Canada is still at 14.)
 4. **Trust the stored status before paying for a carrier call.** `isTrackingStatusFinal(?string)`
    short-circuits when `TrackingNumbers.status` is already `DELIVERED` or `RETURNED_TO_SENDER`
    (`TRACKING_FINAL_STATUSES`). worker2's
@@ -105,6 +117,37 @@ mark the row done"* cron:
    keeps that column current for every client, so this 1.0 cron should not re-ask the carrier about
    a finished shipment.
 
+### Which clock the window runs on: `AdvanceShippingNotices.dtCreated`, never `TrackingNumbers.dtCreated`
+
+**This is the single easiest way to break Compass shipping emails, and it has now broken them
+twice.** For Compass a `TrackingNumbers` row is created by the **NetSuite invoice sync**, not by
+the ASN. The ASN shows up days or weeks later and is the only thing that links the tracking number
+to an order and therefore to a recipient. So `TrackingNumbers.dtCreated` is the **invoice** date,
+and a lookback anchored on it measures the wrong thing entirely. Full derivation:
+[where a Compass TrackingNumbers row comes from](../../../clients/compass-usa/features/tracking-number-origin-and-asn-link.md).
+
+- **Window (this cron):** filter on `AdvanceShippingNotices.dtCreated` — "how long has this been
+  *sendable*". Fixed 2026-09-16 at `update_salesorder_status_from_odp.php` line ~432;
+  `IN_TRANSIT_EMAIL_LOOKBACK_DAYS` stays at **7**, and the comment above the constant now says why.
+- **Age / alarm (the monitor):** use `GREATEST(TrackingNumbers.dtCreated,
+  AdvanceShippingNotices.dtCreated)`.
+
+**What went wrong.** On 2026-09-15 11:35 commit `81824325` ("Changing in transit email limit to 7")
+cut the lookback from 14 to 7 days. Because the window was anchored on the invoice date,
+**22 shipments** invoiced 2026-09-03 11:17–11:38 whose ASNs only landed 2026-09-15 16:00–17:00 fell
+outside the 7-day window and were never emailed: SA136625, SA136627, SA136643, SA136673, SA136680,
+SA136694, MR243688, MR244839, MR244893, MR244904, MR245219, MR245303, MR245337, MR245404, MR245409,
+MR245420, MR245424, MR245429, MR245441, MR245454, MR245462, MR245463. Since 2026-08-01, **53**
+tracking rows had their ASN land more than 7 days after the tracking row (max gap **13 days**), so
+the 14-day window had been masking the same design flaw all along.
+
+**Verified on prod before shipping:** the ASN-anchored window returns the same 22 rows, drops **0**
+rows the old window covered, and yields exactly **1** real catch-up email — MR245462 (tracking
+`1Z8696XA0392020614`, still `IN_TRANSIT`). The other 21 already read `DELIVERED`, so loop 1 stamps
+them and sends nothing.
+
+**Status: edited in the working tree, not yet committed or deployed (2026-09-16).**
+
 ### Resolving the recipient: a `Users` row is NOT required (USA)
 
 The To address originally came only from `Users.email`, via `INNER JOIN Users ON Users.contactId =
@@ -227,14 +270,30 @@ the tracking number as emailed (it retries next run).
   (e.g. `update_salesorder_status_from_gt.php`) with `schedules/cron.worker.sync.json` repointed.**
   Never put `&` (or any shell metacharacter) in a cron filename. See
   [tracing a 1.0 worker cron run](../workflows/tracing-a-worker-cron-run-in-production.md).
-- **⚠ Invoice tracking numbers are NOT in scope and must never be counted as in-transit
-  backlog.** `POST /v2/invoice-tracking-numbers` creates `TrackingNumbers` rows bridged through the
-  **`InvoiceTrackingNumbers`** table — never the ASN chain. On 2026-09-03 alone, 98 were created
-  (11:15–11:40) and **zero** of them touch `AdvanceShippingNoticeItemUnits_TrackingNumbers`.
-  Anything that walks the ASN unit bridge (this cron, and the OneUptime queue monitor) correctly
-  ignores them. Their shape is distinctive: **`shippingCarrierId` set, `shippingMethodId` NULL**.
-  A naive `TrackingNumbers WHERE c_dtInTransitEmailSent IS NULL` count will read these as a backlog
-  that never drains.
+- **⚠ CORRECTED 2026-09-16 — invoice tracking numbers ARE in scope; they are the normal origin.**
+  This doc previously said `POST /v2/invoice-tracking-numbers` rows are bridged only through
+  `InvoiceTrackingNumbers` and "never the ASN chain". **That was wrong.** For Compass the invoice
+  sync is what *creates* nearly every `TrackingNumbers` row (98 POSTs / 100 rows in the 2026-09-03
+  11:15–11:40 window, against 4 ASNs created all day); the ASN attaches to those same rows days or
+  weeks later. What survives is the narrow version: a tracking number that **never** receives an
+  ASN can never be emailed, so a naive `TrackingNumbers WHERE c_dtInTransitEmailSent IS NULL` count
+  with no ASN join reads as a backlog that never drains. And because the row is born at invoice
+  time, `TrackingNumbers.dtCreated` must never be used as the window clock — see
+  [tracking-number origin](../../../clients/compass-usa/features/tracking-number-origin-and-asn-link.md).
+
+- **⚠ Do NOT make `3b_import_strategic_systems_advance_shipping_notices.php` email for
+  pre-existing tracking numbers — tried 2026-09-16 and REVERTED.** 3b gates its in-transit email on
+  `$isTrackingNumberInserted` (line ~722 in `processOfficeDepotFile`, line ~1431 in
+  `processStrategicSystems`): it only emails when that run INSERTs the tracking row. When the
+  invoice sync already created the number, 3b still builds the ASN link but sends nothing, which
+  looks like the bug. Changing the gate to "has this customer been told"
+  (`c_dtInTransitEmailSent IS NULL`) was implemented and then backed out, because **3b only ever
+  sends the plain in-transit template** `e936aa4b-d5b6-41dd-be1a-938b2b27af9e`, while this catch-up
+  cron chooses between the **partial** template `d9b3f2a7-1c84-4e6d-9f50-3a7c1e8b2d46` (split order
+  / items still being prepared) and the plain one. Firing 3b for pre-existing tracking numbers would
+  **silently downgrade every split order from the partial email to the plain email**. 3b was also
+  never the real break: the catch-up cron already covers that case inside its window, so fixing the
+  window was the correct and far smaller change.
 - **Clear an in-transit backlog by STAMPING, not by emailing — and refresh statuses first.**
   Decided 2026-09-03 while draining the outage backlog
   (`dbchanges2/Client_Compass/2026-09-03 - compass_intransit_email_backlog_backfill.sql`):
@@ -292,6 +351,20 @@ the tracking number as emailed (it retries next run).
   interceptor and retire this 1.0 cron.
 
 ## Change history
+- 2026-09-16 — **Re-anchored the in-transit window on `AdvanceShippingNotices.dtCreated`** (was
+  `TrackingNumbers.dtCreated`, `update_salesorder_status_from_odp.php` line ~432);
+  `IN_TRANSIT_EMAIL_LOOKBACK_DAYS` stays 7 with a comment explaining why. Cause: commit `81824325`
+  (2026-09-15 11:35) cut the lookback 14 → 7 while the window still measured the **invoice** date,
+  so 22 shipments invoiced 2026-09-03 whose ASNs landed 2026-09-15 fell out of the window (only
+  MR245462 still owed a real email; the other 21 read DELIVERED). Verified on prod: 22 rows in,
+  0 rows lost, 1 catch-up email. **Corrected the old "invoice tracking numbers never touch the ASN
+  chain" gotcha** — the invoice sync is in fact the normal origin of Compass `TrackingNumbers` rows;
+  new doc
+  [tracking-number origin and the ASN link](../../../clients/compass-usa/features/tracking-number-origin-and-asn-link.md).
+  Recorded the **rejected** alternative of making 3b email for pre-existing tracking numbers (it
+  would downgrade split orders from the partial template to the plain one). Not yet committed or
+  deployed. (bala)
+
 - 2026-09-10 — Recorded that a zero-line-item ASN is invisible to this cron (candidates come from
   `AdvanceShippingNoticeItemUnits(_TrackingNumbers)`), the failure mode Compass Canada's G&T
   importer produced until 2026-09-08. (bala)

package/knowledge/2.0/apps/worker2/features/oneuptime-worker2-monitoring.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-09-08
+updated: 2026-09-16
 owners: ["jcardinal", "mhammontree", "bala"]
 files:
   - worker2/Worker/Monitor/Compass.php
@@ -23,6 +23,7 @@ files:
   - worker2/Config/sandbox-dev.ini
 related:
   - ./oneuptime-incident-clickup-sync.md
+  - ../../../clients/compass-usa/features/tracking-number-origin-and-asn-link.md
   - ../../../1.0/apps/worker/features/compass-partial-in-transit-delivered-emails.md
   - ./monitoring-framework.md
   - ./all-client-email-queue-monitor.md
@@ -110,7 +111,9 @@ Each measures one Compass/Office Depot integration metric and pushes a decided a
   `s3://agilant-as2/OfficeDepot/OUTBOX/` (skips subfolders + directory placeholders).
 - **InTransitEmailQueue** (DB, added 2026-09-04) — watches the Compass USA **in-transit
   shipping-notification queue**: `TrackingNumbers` rows on the ASN unit bridge with
-  `c_dtInTransitEmailSent IS NULL` inside a 14-day window (the window the 1.0 sending cron uses).
+  `c_dtInTransitEmailSent IS NULL` inside a **7-day window on `AdvanceShippingNotices.dtCreated`**
+  (exactly the window the 1.0 sending cron uses — see *Age the backlog on the clock the work
+  actually starts on* below).
   Status-page name *"Shipping Notification Emails (USA, In Transit)"*. See
   [why it exists](#watch-the-step-before-the-log-row-exists) below.
 - **MaRefreshOrderExceptions** (DB, added 2026-09-08) — Compass **`MA%` employee-refresh**
@@ -244,6 +247,36 @@ deliberately does not drive the alarm** — otherwise the monitor would sit red
 cron structurally cannot send. That is the *"backlog monitors must exclude known-permanently-stranded
 rows"* rule.
 
+### Age the backlog on the clock the work actually starts on
+
+`InTransitEmailQueue` originally did both of these off `TrackingNumbers.dtCreated`, and both were
+wrong for Compass, because a Compass `TrackingNumbers` row is created by the **NetSuite invoice
+sync** and only becomes sendable when the ASN lands days or weeks later
+([derivation](../../../clients/compass-usa/features/tracking-number-origin-and-asn-link.md)).
+Three corrections (a. committed as `82d614c` and deployed; b. and c. in the working tree, not yet
+deployed as of 2026-09-16):
+
+a. **`oldestQueuedHours` / `lateRows` now age from
+   `GREATEST(TrackingNumbers.dtCreated, AdvanceShippingNotices.dtCreated)`** — "how long has this
+   been *sendable*", not "how long since we invoiced it".
+b. **`$LOOKBACK_DAYS` 14 → 7**, to match the cron's `IN_TRANSIT_EMAIL_LOOKBACK_DAYS`.
+c. **The WHERE lookback filters on `AdvanceShippingNotices.dtCreated`**, so the monitor counts
+   exactly the set the cron will work on.
+
+**Why it matters — the monitor cried wolf, then muddied a real incident.** Two HIGH alarms fired on
+2026-09-15 (15:00:46 and 15:35:46) reporting "oldest 90h / 91h" for rows that had been sendable for
+*minutes*. Replaying both moments with the `GREATEST(...)` expression gives **1h** and **0h** with
+**0** late rows, i.e. OK. A third alarm on 2026-09-16 08:55 was a **real** gap, and the 14-vs-7 day
+mismatch made it confusing: the monitor was counting 22 rows the cron would never look at.
+
+**Rule: a backlog monitor must age and window on the same clock, and the same column, as the tier
+that does the work.** Any drift between the two turns the monitor into noise in both directions —
+false alarms on rows that are fine, and silence on rows the cron cannot see.
+
+Rows with no `Users` row and no `ContactEmailAddresses` fallback (typically EDI-created MR orders)
+can never be emailed at all; the monitor reports those as `unroutableRows` and deliberately never
+alarms on them.
+
 > **⚠ Known divergence (2026-09-04):** the monitor decides routable/unroutable from the
 > `Users` join **only**, while the sending cron now also falls back to
 > `Contacts` / `ContactEmailAddresses`. So some rows counted as `unroutableRows` are in fact
@@ -543,6 +576,15 @@ check for another client.
   monitors must pass the region (see [Cloud S3 helpers](../../_underscore/features/cloud-s3-helpers.md)).
 
 ## Change history
+- 2026-09-16 — **`InTransitEmailQueue` re-anchored onto the sendable clock.** `oldestQueuedHours` /
+  `lateRows` now age from `GREATEST(TrackingNumbers.dtCreated, AdvanceShippingNotices.dtCreated)`
+  (committed `82d614c`, deployed); `$LOOKBACK_DAYS` 14 → 7 and the WHERE lookback moved onto
+  `AdvanceShippingNotices.dtCreated` to match the 1.0 cron exactly (working tree, not deployed).
+  Cause: a Compass `TrackingNumbers` row is born from the NetSuite invoice sync, so the old age
+  measured invoice age — two false HIGH alarms on 2026-09-15 ("oldest 90h / 91h") replay as 1h / 0h
+  with 0 late rows. Added the general rule that a backlog monitor must age and window on the same
+  clock and column as the tier that does the work. (bala)
+
 - 2026-09-08 — Added an **11th monitor, `MaRefreshOrderExceptions`** (Compass `MA%`
   employee-refresh orders with no NetSuite order; `Core.CronJobs` id 74; DEGRADED >10 waiting /
   >32h, HIGH >20 / >72h) — full derivation in

package/knowledge/INDEX.md

@@ -5,7 +5,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 ## 1.0 framework
 
 - **library** (Library) _(framework core)_ — 26 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
-- **worker** (Worker) — 35 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
+- **worker** (Worker) — 36 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
 - **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
 - **worker1.5** (Worker 1.5) — 0 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
 - **togadesk** (TOGa Desk) — 13 doc(s) → [1.0/apps/togadesk/INDEX.md](1.0/apps/togadesk/INDEX.md)

package/knowledge/clients/compass-usa/INDEX.md

@@ -23,6 +23,7 @@
 | [Persona Model & Levy-Sector Gating (worker2 PEOPLE cron)](features/persona-model-and-levy-gating.md) | 2.0 | Compass USA catalogue visibility is driven by **personas** in `Client_Compass`. |
 | [Compass Sales-Order Line-Number Renumbering (and why it hangs off the SO hooks)](features/sales-order-line-renumbering.md) | 2.0 | Compass sales-order lines must stay numbered **1..N with no gaps** after any add, edit, or delete — downstream MITS/PO linking reads `lineNumber` as an identity |
 | [Stranded Approval Reassignment (repointing approvals off dead duplicate Compass Users rows)](features/stranded-approval-reassignment.md) | 2.0 | A Compass employee who leaves and comes back after more than `CONTACT_UNLINK_GRACE_DAYS` (5) is **inserted as a brand-new `Users` row** by the PEOPLE importer i |
+| [Compass: where a `TrackingNumbers` row comes from (invoice sync, NOT the ASN) and how it reaches a customer](features/tracking-number-origin-and-asn-link.md) | 1.0 | For Compass USA a `TrackingNumbers` row is **born from the NetSuite invoice sync**, not from the ASN. |
 | [Compass USA](profile.md) | 2.0 | Compass USA is a TOGA client running a multi-tier supply-chain commerce operation. |
 | [Compass USA — reversing the 199-item catalog fold (MITS outage recovery)](workflows/catalog-fold-reversal.md) | 2.0 | The **root cause** of the Compass MITS transmission outage (the `assetType`-NULL and duplicate-vendor symptoms) was an ad-hoc cross-client **duplicate-items mer |
 | [Changing what is IN a Compass kit (adding / swapping a BundleItems line)](workflows/changing-a-kit-line-item.md) | 2.0 | "Add this fee/component to every kit that has X" is a recurring Compass request, and it looks like one `INSERT` into `Client_Compass.BundleItems`. |

package/knowledge/clients/compass-usa/features/tracking-number-origin-and-asn-link.md

@@ -0,0 +1,115 @@
+---
+title: Compass: where a `TrackingNumbers` row comes from (invoice sync, NOT the ASN) and how it reaches a customer
+framework: "1.0"
+repo: worker
+project: Worker
+client: compass-usa
+type: client-feature
+status: active
+updated: 2026-09-16
+owners: ["bala"]
+files:
+  - library/app/api/toga2.php
+  - worker/crons/toga2/netsuite/common_sync_togasupply.php
+  - worker/crons/toga2/netsuite/sync_togasupply_compass_usa.php
+  - worker/crons/toga2/compass/update_salesorder_status_from_odp.php
+  - worker2/Worker/Monitor/Compass.php
+related:
+  - ../../../1.0/apps/worker/features/compass-partial-in-transit-delivered-emails.md
+  - ../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md
+  - ./asn-to-item-fulfillment.md
+  - ./contact-email-resolution.md
+  - ../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md
+---
+
+## Summary
+
+For Compass USA a `TrackingNumbers` row is **born from the NetSuite invoice sync**, not from the
+ASN. The ASN turns up days or weeks later and is the **only** thing that ties that tracking number
+to a sales order, and therefore to a customer's email address.
+
+The consequence, and the reason this doc exists: **`TrackingNumbers.dtCreated` is the invoice
+date.** It is the wrong clock to time any shipping notification, backlog age, or SLA from. Anything
+that asks *"when did we get this tracking number?"* and means *"when could we first have told the
+customer?"* is wrong for Compass by up to two weeks.
+
+## Key files / entry points
+
+- `worker/crons/toga2/netsuite/sync_togasupply_compass_usa.php` → `common_sync_togasupply.php` →
+  `App_Api_Toga2::syncInvoiceFromNetsuite()` (`library/app/api/toga2.php`, function opens at
+  line 3203; the tracking-number POST is at line 3662).
+- The POST is `POST /v2/invoice-tracking-numbers`, payload shaped as a nested object:
+  `{"invoice":{"uuid":"…"},"trackingNumber":{"number":"…","shippingCarrier":{…}}}`. api2 creates
+  the `TrackingNumbers` row **from that nested object** — so the row exists long before any ASN.
+- `worker/crons/toga2/compass/update_salesorder_status_from_odp.php` — the consumer that must use
+  the ASN clock, not this one.
+- `worker2/Worker/Monitor/Compass.php` → `InTransitEmailQueue` — the monitor over the same queue.
+
+## How it works
+
+1. The invoice sync imports a NetSuite invoice and POSTs each of its tracking numbers. That
+   **creates** `TrackingNumbers`, stamping `dtCreated` = the moment of the invoice import.
+2. The invoice does **not** carry the order link. `Invoices.salesOrderId` is NULL on all of the
+   affected invoices, and across Compass since 2026-08-01 it is NULL on **1,868 of 2,022**
+   invoices. So at this point the tracking number belongs to nobody.
+3. Later (hours to weeks) the ASN arrives, and **that** is what links the tracking number to a
+   customer. The only path is:
+
+   ```
+   TrackingNumbers
+     → AdvanceShippingNoticeItemUnits_TrackingNumbers
+     → AdvanceShippingNoticeItemUnits
+     → AdvanceShippingNoticeItems
+     → AdvanceShippingNotices.purchaseOrderId
+     → SalesOrders_PurchaseOrders
+     → SalesOrders
+     → Contacts / Users          (the email address)
+   ```
+
+4. Only once that chain exists can the in-transit email be built or sent.
+
+### How big the gap is (measured on prod)
+
+- **2026-09-03, 11:15:56–11:40:20** — `Logs_Compass.Api` shows **98** `POST /v2/invoice-tracking-numbers`
+  from the worker instances. **100** `TrackingNumbers` rows were created in that window, while only
+  **4** ASNs were created in the whole of 09-03.
+- Since **2026-08-01**, **53** tracking rows had their ASN land **more than 7 days** after the
+  tracking row, with a **maximum gap of 13 days**.
+- The concrete failure: 22 shipments invoiced 2026-09-03 11:17–11:38 whose ASNs only landed
+  2026-09-15 16:00–17:00.
+
+## Data model
+
+- `TrackingNumbers.dtCreated` — **invoice import time** for Compass. Not the ship time, not the
+  time the customer became notifiable.
+- `AdvanceShippingNotices.dtCreated` — the first moment a tracking number is attached to an order.
+  **This is the notifiable-from clock.**
+- `Invoices.salesOrderId` — usually NULL for Compass; do not rely on it to reach the order.
+- `InvoiceTrackingNumbers` — the invoice-side bridge. A row can sit on this bridge *and* later on
+  the ASN unit bridge; the two are not mutually exclusive.
+
+## Gotchas / known issues
+
+- **⚠ Never window, age, or alarm a Compass shipping feature on `TrackingNumbers.dtCreated`.** Use
+  `AdvanceShippingNotices.dtCreated` for a window, and `GREATEST(TrackingNumbers.dtCreated,
+  AdvanceShippingNotices.dtCreated)` when you need "how long has this been sendable". Getting this
+  wrong produced both a real missed-email gap and two false HIGH alarms — see
+  [Compass in-transit emails](../../../1.0/apps/worker/features/compass-partial-in-transit-delivered-emails.md).
+- **⚠ Correction to an earlier belief.** It was previously recorded that invoice-created tracking
+  numbers "never touch the ASN chain" and are therefore out of scope for in-transit email. That is
+  wrong. They are the *normal* origin for Compass tracking numbers; the ASN chain simply attaches
+  to them later. What is true is the narrower statement: a tracking number that **never** gets an
+  ASN can never be emailed, and a naive `TrackingNumbers WHERE c_dtInTransitEmailSent IS NULL`
+  count (with no ASN join) reads those as a backlog that never drains.
+- **A tracking number with no ASN has no customer.** There is no fallback route from invoice to
+  contact, because `Invoices.salesOrderId` is NULL. Waiting for the ASN is the design, not a bug.
+- Credentials for the sync ride on `App_Api_Toga2` class constants (e.g.
+  `App_Api_Toga2::CLIENT_UUID_COMPASS`) — reference the constant, never the value.
+
+## Change history
+- 2026-09-16 — First write-up. Established that Compass `TrackingNumbers` rows are created by the
+  NetSuite invoice sync (`syncInvoiceFromNetsuite` → `POST /v2/invoice-tracking-numbers`), that
+  `Invoices.salesOrderId` is NULL on 1,868 of 2,022 Compass invoices since 2026-08-01, and that the
+  ASN chain is the only route from a tracking number to a customer. Measured the invoice→ASN gap
+  (53 rows over 7 days since 2026-08-01, max 13 days) and corrected the earlier "invoice tracking
+  numbers never touch the ASN chain" claim. (bala)

package/knowledge/clients/prudential/workflows/store-saml-deploy-topology.md

@@ -23,6 +23,28 @@ opposite one from what you would guess. Read this before deploying anything SAML
 Prudential. It also records the EB deploy failure mode we hit on 2026-09-15 (a wedged command
 agent that times out after 14 minutes while the environment still shows Green).
 
+## The full login flow — the two legs run on two different servers
+
+The outbound (first) leg does **not** hop through togahub. It redirects the browser straight to
+Microsoft Entra:
+
+1. User hits `prudential.togacommerce.com` (**`true-togacommerce-prod`**). The `prudential`
+   host block in `store/_/app/framework.php` fires when the session host is `prudential`, the app
+   is not in test mode, and the session has no `accountId`.
+2. With no recognized `?ssoCommand`, it falls to the `default:` case, which calls
+   `$auth->login($originUrl)` on `OneLogin_Saml2_Auth`.
+3. That redirects to Prudential's Entra tenant:
+   `https://login.microsoftonline.com/d8fde2f5-9392-4260-8a03-0ad01f4746e9/saml2`.
+4. The SP block advertises entityId `https://prudential.togahub.com/?meta` and ACS
+   `https://prudential.togahub.com/?acs` — so **Entra POSTs the response to togahub**, i.e. the
+   inbound leg lands on **`prudential-store-prod`**, a different EB environment.
+5. `prudential-store-prod` decrypts the assertion (LightSAML), then redirects back to
+   `prudential.togacommerce.com/?hash=<base64 of xid:first:last:email>`.
+
+**Why this matters:** the outbound and inbound legs are different servers with different
+deploys. A deploy that succeeds on one does **not** fix the other. Decrypt/cert changes belong on
+`prudential-store-prod`.
+
 ## The routing — the pipeline action names are misleading
 
 | Hostname | CNAME → EB environment | Deployed by pipeline action |
@@ -92,7 +114,41 @@ Observed 2026-09-15: `prudential-store-prod` on `i-07ae1b81ee48df938` (t3.micro,
 A reboot restored the EB command channel on 2026-09-15, but SSM still did not report after 3
 minutes — EB working again is the signal to retry, not SSM.
 
+## Incident 2026-09-15 — OAEP decrypt error caused by the stuck deploy
+
+Production Sentry showed, from the togahub (LightSAML) ACS handler:
+
+```
+Failure decrypting Data (openssl private) - RSA_padding_check_PKCS1_OAEP_mgf1: oaep decoding error
+```
+(also seen as `RSA_EAY_PRIVATE_DECRYPT: padding check failed`)
+
+**Root cause: not a bad key — a deploy that never landed.** `prudential-store-prod` was still
+running the pre-rotation code (commit `275d16a9`, an Aug 13 build) because its deploy kept timing
+out with the wedged-agent failure above. That old code loaded a **mismatched cert/key pair**: the
+`togahub_com_2025-03-25.crt` cert with `toga_private_key.key`, which belongs to the 2026-02-27
+cert. Confirmed by comparing the two openssl moduli — they differ. The new dual-keypair fallback
+fixes this, but it was not on the box.
+
+The `library` repo was verified **clean and not the cause**: all three new files committed and
+pushed on `_production`, both keypairs (2027 active and 2026-02-27 fallback) verified as matching
+pairs against the **committed git blobs** (not just local files), both keys unencrypted (no
+`Proc-Type` / `DEK-Info`), and both loading via the exact call the code makes —
+`openssl_pkey_get_private($pem, "")`.
+
+**Lesson:** when a SAML decrypt error appears right after a cert rotation, check *what build is
+actually deployed* before you suspect the key material. Verify against committed blobs, and note
+that the stack trace alone cannot tell you which build is running (see the cert & keys feature
+doc's gotcha on `decryptMulti`).
+
 ## Change history
+- 2026-09-15 — Traced the outbound SAML leg: `prudential.togacommerce.com` redirects the browser
+  straight to Prudential's Entra tenant (no togahub hop), and Entra POSTs back to
+  `prudential.togahub.com/?acs` on a different EB environment — so a deploy to one leg does not
+  fix the other. Recorded the OAEP decrypt incident: the error came from `prudential-store-prod`
+  still serving the pre-rotation Aug 13 build (`275d16a9`) with a mismatched cert/key pair,
+  because its deploy kept timing out; the `library` repo was verified clean against committed
+  git blobs. (rgirish)
 - 2026-09-15 — Created while rotating the `togahub.com` SAML cert. Documented that
   `prudential.togahub.com` (SAML ACS) and `prudential.togacommerce.com` (storefront) are separate
   EB environments with swapped-sounding pipeline action names, the split AWS accounts/regions,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.817",
+  "version": "1.0.819",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",