toga-ai 1.0.620 → 1.0.622

package/knowledge/1.0/apps/library/INDEX.md

@@ -9,6 +9,7 @@
 | [Cron Execution Monitoring (App_Framework check-in/out → CronJobExecutions)](features/cron-execution-monitoring.md) | `App_Framework::cronInitialization()` / `App_Framework::cronFinished()` (in `library/app/framework.php`) give every 1.0 (`App_`) cron job a check-in/check-out l | library/app/framework.php |
 | [Diagnostic Dialog — View Recommended Services Routing](features/diagnostic-dialog-view-recommended-services.md) | Two "View Recommended Services" buttons exist in the TOGa Refresh 2026 SR view: 1. | library/app/model/toga/diagnostic.php, library/app/model/servicerequest.php |
 | [Elite Freshservice Sync (library)](features/elite-freshservice-sync.md) | `App_Api_Toga2` in `library/app/api/toga2.php` orchestrates bidirectional sync between TOGA 2 and TOGaDesk. | library/app/api/toga2.php |
+| [App_Email Queued Sending & Attachments (Common.EmailsQueued)](features/email-queue-attachments.md) | `App_Email::send()` can either send **inline** (PHPMailer talks to SES right there) or **queue** the message: `base64(serialize($this))` is inserted into `Commo | library/app/email.php, library/phpmailer/class.phpmailer.php, worker/crons/notifications/infrastructure/send_emails.php, worker/crons/notifications/covid/send_covid_pending_vaccination_approval.php, togadesk/desk/includes/functions.php |
 | [Branded HTML Email Templates (App_Email_Template)](features/email-templates.md) | `App_Email_Template` (`app/email/template.php`) is the base class for branded HTML emails in the 1.0 (`App_`) framework. | library/app/email/template.php, library/app/email/agilant.php |
 | [Error Capture in 1.0 (App_Error_Capture → shared 2.0 Logs DB)](features/error-capture-1-0.md) | The 1.0 side of the platform error-reporting pipeline (TRUE-78188). | library/app/error/capture.php, library/app/error.php, library/app/exception/business.php, library/app/api/toga2.php, library/app/cloud.php, worker/config.worker.ini, worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php |
 | [HTTP 500 Error Monitor (App_SystemMonitor_500Error) — and why its \"Error Type\" is not a diagnosis](features/http-500-error-monitor.md) | `App_SystemMonitor_500Error` (`library/app/systemmonitor/500error.php`, title **"HTTP 500 Error Alert"**) is the 1.0 system monitor that watches **`Logs.Api` fo | library/app/systemmonitor/500error.php, worker/crons/infrastructure/system_monitors.php, api2/Controller/Index.php, _underscore/Error.php |

package/knowledge/1.0/apps/library/features/email-queue-attachments.md

@@ -0,0 +1,112 @@
+---
+title: App_Email Queued Sending & Attachments (Common.EmailsQueued)
+framework: "1.0"
+repo: library
+project: Library
+client: shared
+type: feature
+status: active
+updated: 2026-08-20
+owners: ["mhammontree"]
+files:
+  - library/app/email.php
+  - library/phpmailer/class.phpmailer.php
+  - worker/crons/notifications/infrastructure/send_emails.php
+  - worker/crons/notifications/covid/send_covid_pending_vaccination_approval.php
+  - togadesk/desk/includes/functions.php
+related:
+  - ./email-templates.md
+  - ../architecture.md
+  - ../../togadesk/features/notifications.md
+---
+
+## Summary
+
+`App_Email::send()` can either send **inline** (PHPMailer talks to SES right there) or **queue**
+the message: `base64(serialize($this))` is inserted into `Common.EmailsQueued` and drained later
+by a **different app on a different host** — `worker/crons/notifications/infrastructure/send_emails.php`,
+scheduled `* * * * *`.
+
+Two facts dominate everything else here:
+
+1. **Attachments used to be queued as filesystem *paths*.** The drain host has no such path, and
+   PHPMailer 5 silently swallows an unreadable attachment, so the mail arrived with the body
+   intact and the attachment gone — **no error, no log**. Fixed 2026-08-20 (TRUE-80952) by
+   embedding the file **bytes** at queue time.
+2. **Queued email does not queue outside production.** `setQueueForSending()` is gated on
+   `!App_Registry::inTestMode()`, and every non-prod config has `internal.test_mode = 1`. So on
+   alpha/beta/local the message is sent **inline** off local disk and attachments always work —
+   any test there is a **false pass**.
+
+## How it works
+
+- `App_Email::setQueueForSending($bool)` (`library/app/email.php` ~L161):
+  `$this->queueForSending = (($bool && !App_Registry::inTestMode()) ? … : false);`
+- `send()` with queueing on serializes `$this` and inserts into `Common.EmailsQueued`
+  (`serializedEmailObject`, LONGTEXT).
+- Before serializing, `App_Email::embedAttachmentsForQueue()` (private, added TRUE-80952) reads
+  each attachment into **base64 bytes**. On drain, embedded entries go out via PHPMailer's
+  `AddStringAttachment()`; legacy bare-string path entries still use `AddAttachment()`.
+- **Size budget:** `MAX_QUEUED_ATTACHMENT_BYTES = 7340032` (7 MB raw ≈ 9.8 MB base64, under the
+  SES 10 MB message cap). Over budget, embedding returns false and `send()` **falls back to an
+  inline send** instead of queueing — except when outbound email is disabled, where there is no
+  inline fallback and the case is logged.
+- The file read is wrapped in a **local `set_error_handler` → `ErrorException` with a `finally`
+  restore**. This is mandatory, not stylistic: in 1.0 any PHP warning routes through
+  `App_Error::handleError` → `handleException` → `exit()`, so an unguarded `file_get_contents`
+  warning would kill the `send_emails` cron **mid-loop** and strand every queued email behind it.
+  `is_file()`/`is_readable()` are not sufficient — the file can vanish between check and read (TOCTOU).
+
+## Blast radius (verified 2026-08-20)
+
+Only **two** 1.0 call sites both queue *and* attach: TOGa Desk's `sendEmail()`
+(`togadesk/desk/includes/functions.php`) and
+`worker/crons/notifications/covid/send_covid_pending_vaccination_approval.php` (~L44 + L82, bare
+string path — it had the identical bug and is fixed by the same change). worker's other 16
+`setQueueForSending` sites pass `false`; togaview's two queueing sites (`mvc/signup/post.php`,
+`mvc/reset_password_success/post.php`) attach nothing; `tools` and `walmarttechservices` never
+attach. Inline sending and bare-string path attachments are unchanged.
+
+## Deploy ordering (asymmetric — and so is rollback)
+
+The queue payload format changed, and the queueing app and the draining app deploy separately.
+
+- worker **new** + togadesk **old** → togadesk queues paths, new worker still handles paths. Safe.
+- worker **old** + togadesk **new** → togadesk queues `['data' => …]`, the old worker looks for
+  `['path']`, finds none, and **drops the attachment** — same symptom, new cause.
+
+Therefore: **deploy `worker` before `togadesk`.** Roll back in the inverse order — roll back
+togadesk first, let the queue drain empty, *then* worker.
+
+## Gotchas / known issues
+
+- **You cannot reproduce or validate a queued-email bug outside production.** `test_mode = 1` on
+  alpha, beta and every dev config collapses `setQueueForSending(true)` to `false`. This is the
+  most likely reason TRUE-75813 and TRUE-79924 were "verified" and closed while prod stayed
+  broken — both corrected the upload path *inside togadesk*, which was already working; the file
+  was lost one hop later, at the queue. Locally the queueing app and the drain also share one
+  filesystem, so even with queueing forced on the path resolves — you must **delete the source
+  file between queue and drain** to simulate the drain host.
+- **PHPMailer 5 hides a missing attachment.** `AddAttachment()`
+  (`library/phpmailer/class.phpmailer.php` ~L1358) treats an unreadable path as `STOP_CONTINUE`
+  with `$exceptions` off, and sends the message anyway. Absence of an error proves nothing.
+- **The drain cron's exclusion filter is dead code.** `send_emails.php` ~L34 carries
+  `AND EmailsQueued.serializedEmailObject NOT LIKE '%App_Email%'`. The column holds base64, in
+  which the literal `App_Email` never appears (verified with a PHP round-trip) — the filter
+  matches and excludes nothing. **Not changed**: "fixing" it would stop TOGa Desk mail entirely.
+  Needs a decision from whoever wrote it before anyone touches it.
+- **OPEN RISK — `max_allowed_packet` on the prod Common cluster is unverified.** Rows can now
+  carry ~9.8 MB of base64. The column is LONGTEXT so it is fine, but MySQL 5.7 defaults
+  `max_allowed_packet` to 4 MB (8.0: 64 MB). If prod is below the budget the INSERT fails as an
+  `App_Query` exception on a live ticket reply — worse than a missing attachment. Check it, and
+  lower `MAX_QUEUED_ATTACHMENT_BYTES` beneath it if needed (the oversized path already degrades
+  to an inline send, which is the correct behavior).
+- Regression coverage: `test/@Mark/TRUE-80952/test_queued_email_attachments.php` — 26 assertions,
+  no DB, no SMTP, confirmed failing against pre-fix HEAD.
+
+## Change history
+- 2026-08-20 — TRUE-80952: queued attachments now embed base64 bytes (`embedAttachmentsForQueue()`
+  + `AddStringAttachment`) instead of paths that do not exist on the drain host; added
+  `MAX_QUEUED_ATTACHMENT_BYTES` with inline-send fallback and a scoped warning guard around the
+  file read; recorded the `test_mode` no-queue trap, the dead drain filter, and the
+  worker-before-togadesk deploy order (mhammontree)

package/knowledge/1.0/apps/test/INDEX.md

@@ -10,7 +10,7 @@
 | [Developer Generators (password, UUID)](features/dev-generators.md) | Two tiny **1.0 `App_` framework** convenience scripts for everyday developer needs. | test/team/generate_password.php, test/team/uuid.php |
 | [Forecast vs NetSuite Discrepancy Analysis](features/forecast-netsuite-discrepancy-analysis.md) | `team/forecast-netsuite/discrepancy_analysis.php` detects discrepancies between our **Forecast database** and **NetSuite** (the source of truth for all sales da | test/team/forecast-netsuite/discrepancy_analysis.php |
 | [@goagilant.com → @togatech.com Email-Domain Migration (1.0 + 2.0)](features/goagilant-to-togatech-email-migration.md) | Reference + technique for migrating the company email domain `@goagilant.com` → `@togatech.com` across **both** platforms. | migrate_goagilant_to_togatech_2026-06-26.sql, migrate_goagilant_to_togatech_LEGACY_2026-06-26.sql |
-| [Static (no-DB) Regression Harness for 1.0 Logic + Source Drift Guard](features/static-no-db-regression-harness.md) | 1.0 has **no PHPUnit**, and most of its business logic sits inside methods that also write SQL, so "just call it" means standing up a client database. | test/@Mark/AIG/test_multi_email.php, library/app/database.php, library/app/api/toga2.php |
+| [Static (no-DB) Regression Harness for 1.0 Logic + Source Drift Guard](features/static-no-db-regression-harness.md) | 1.0 has **no PHPUnit**, and most of its business logic sits inside methods that also write SQL, so "just call it" means standing up a client database. | test/@Mark/AIG/test_multi_email.php, test/@Mark/TRUE-80952/test_queued_email_attachments.php, library/app/database.php, library/app/api/toga2.php |
 | [TableView Builder (2.0 TableViews SQL generator)](features/tableview-builder.md) | `team/tableViewBuilder/` generates SQL `INSERT` statements for the **2.0 `TableViews`**, `TableViewFields`, and `TableViewJoins` tables from a plain SQL `SELECT | test/team/tableViewBuilder/TableViewGenerator.php, test/team/tableViewBuilder/index.php, test/team/tableViewBuilder/Instructions.md |
 | [Talos Knowledge Base Pipeline (Uploader + Processor)](features/talos-kb-pipeline.md) | `team/talos/` holds the two-script web tooling that feeds the **TOGa Talos** (TOGa IQ) AI knowledge bases. | test/team/talos/kb_uploader.php, test/team/talos/kb_processor.php, test/team/talos/kb_processor.ini |
 | [TOGa 2.0 Client Onboarding SQL Generator](features/toga2-client-onboarding-sql.md) | > **Superseded by the browser wizard.** The generation logic here was extracted into the reusable > `OnboardingSqlGenerator` class and wrapped in a local browse | test/team/generate_toga2_onboarding_sql.php |

package/knowledge/1.0/apps/test/features/static-no-db-regression-harness.md

@@ -6,10 +6,11 @@ project: Test
 client: shared
 type: feature
 status: active
-updated: 2026-08-03
+updated: 2026-08-20
 owners: ["mhammontree"]
 files:
   - test/@Mark/AIG/test_multi_email.php
+  - test/@Mark/TRUE-80952/test_queued_email_attachments.php
   - library/app/database.php
   - library/app/api/toga2.php
 related:
@@ -75,6 +76,17 @@ method**, which is a large share of `library`.
 entirely. That was out of scope because it changes production code; prefer it when you are
 already editing the method.
 
+## Second worked example — private methods via `ReflectionMethod`
+
+`test/@Mark/TRUE-80952/test_queued_email_attachments.php` (26 assertions, PHP 7.2, no DB, no SMTP)
+extends the same pattern to a **private** method: it `require_once`s `library/app/email.php`
+directly and invokes `App_Email::embedAttachmentsForQueue()` through `ReflectionMethod` with
+`setAccessible(true)`. It covers the embed + serialize round-trip **with the source file deleted**
+(the real production failure mode), a missing file, a directory path, an empty file, the legacy
+bare-string path form, idempotent re-embed, exact-limit and aggregate-limit budget checks, and
+that the scoped error handler is restored. Confirmed failing against pre-fix HEAD — a 1.0
+regression test is only credible if you have run it against the broken code.
+
 ## Gotchas / known issues
 
 - **A mirrored test proves the mirror, not production.** The drift guard is what makes it
@@ -102,6 +114,8 @@ already editing the method.
   **`worker2` / `_underscore` enforce 8.1** — a helper shared between the two must satisfy 7.2.
 
 ## Change history
+- 2026-08-20 — added the TRUE-80952 `ReflectionMethod` variant (private-method coverage, source
+  file deleted to simulate the queue-drain host) as a second worked example (mhammontree)
 
 - 2026-08-12 — Sharpened the 7.2 lint gotcha into a real verification procedure: the default CLI is
   PHP 8.x, so `php -l` there proves nothing; use an actual 7.2 binary, add `-n` if its `php.ini` is

package/knowledge/1.0/apps/togadesk/INDEX.md

@@ -6,7 +6,7 @@
 | [Email-to-Ticket Intake (crons/tickets.php)](features/email-to-ticket-intake.md) | TOGa Desk ingests support email into tickets through a cron-driven IMAP poller (`crons/tickets.php`) plus a postfix pipe variant (`crons/pipe.php`). | crons/tickets.php, crons/tickets_prod.php, crons/pipe.php, desk/includes/classes/class.ticket.php, vendor/classes/class.imap.php |
 | [Field-Service Dispatch (central / repair orders)](features/field-service-dispatch.md) | The **central** subsystem is TOGa Desk's field-service dispatch domain: repair-order lifecycle, technician scheduling, onsite vs depot service, parts, and shipm | desk/includes/controllers/actions/central/, desk/includes/classes/class.repair.php, desk/includes/classes/class.repairhistory.php, desk/_/browser/datatable/central.php, desk/template/pages/central.php, desk/template/pages/central/view.php, desk/template/modals/central/addTracking.php, library/app/model/togadesk/repairorder.php, library/app/model/togadesk/repairordertracking.php, library/app/api/carrier/ups.php |
 | [Managed Service Order Create Flow (New MSO modal → tickets/add)](features/managed-service-order-create.md) | The **Managed Service Order (MSO) create flow** is how a TOGa Desk user manually creates a managed service order: pick an End User (or a Client Location), choos | desk/includes/controllers/actions/tickets/add.php, desk/template/modals/tickets/addNew.php, desk/template/pages/managed.php, desk/template/pages/getinfo.php, desk/template/footer.php, library/app/model/togadesk/repairorder.php |
-| [Ticket Email Notifications (notifications table)](features/notifications.md) | Which TOGa Desk emails fire for a given client is driven **entirely by data**, not code: the `TOGaDeskSupport.notifications` table holds one row per `(clientid, | desk/includes/classes/class.notification.php, crons/tickets.php, crons/tickets_prod.php |
+| [Ticket Email Notifications (notifications table)](features/notifications.md) | Which TOGa Desk emails fire for a given client is driven **entirely by data**, not code: the `TOGaDeskSupport.notifications` table holds one row per `(clientid, | desk/includes/classes/class.notification.php, desk/includes/functions.php, desk/includes/classes/class.ticket.php, crons/tickets.php, crons/tickets_prod.php |
 | [Per-Client Hostname Restriction (getRestrictedClient)](features/per-client-host-restriction.md) | TOGa Desk supports **per-client branded URLs** (e.g. | desk/includes/functions.php, desk/template/header.php, _/browser/datatable/tickets, desk/includes/classes/class.ticket.php, ebs/setup_phpini.php, ebs/http_to_https.php, desk/template/modals/kb/viewDocument.php, desk/template/modals/files/aws-view.php, desk/template/modals/files/contract-view.php, template/pages/documents/view.php, template/pages/tasks/view.php |
 | [REST API (RPC-over-POST) & API-Key Auth](features/rest-api.md) | TOGa Desk exposes a programmatic API at `desk/api/`. | desk/api/index.php, desk/api/resources/tickets.php, desk/api/resources/assets.php, desk/api/resources/authenticate.php, desk/includes/classes/class.apikey.php, desk/includes/functions.php |
 | [SMB Contract Editing & the clientMspId Corruption Trap](features/smb-contract-editing.md) | The SMB contracts page (`/desk/?route=toga/smbcontracts&togaClientId=<id>`) edits `TOGA_*.SMBContracts` rows via a modal. | desk/template/modals/toga/smbcontracts/smbContract.php, desk/includes/controllers/modals/toga/smbcontracts/smbContract.php, desk/includes/controllers/actions/toga/smbcontracts/smbContract.php, desk/includes/controllers/actions/toga/smbcontracts/edit.php |

package/knowledge/1.0/apps/togadesk/features/notifications.md

@@ -6,13 +6,16 @@ project: TOGa Desk
 client: shared
 type: feature
 status: active
-updated: 2026-06-24
-owners: ["ajean"]
+updated: 2026-08-20
+owners: ["ajean", "mhammontree"]
 files:
   - desk/includes/classes/class.notification.php
+  - desk/includes/functions.php
+  - desk/includes/classes/class.ticket.php
   - crons/tickets.php
   - crons/tickets_prod.php
 related:
+  - 1.0/apps/library/features/email-queue-attachments.md
   - 1.0/apps/togadesk/features/ticket-lifecycle.md
   - 1.0/apps/togadesk/features/email-to-ticket-intake.md
   - 1.0/apps/togadesk/architecture.md
@@ -55,6 +58,20 @@ nothing if the `STAFF_*` row is absent.
 - `TOGaDeskSupport.people_departments` — peopleid ↔ departmentid mapping.
 
 ## Gotchas / known issues
+- **Outbound attachments were dropped at the email queue, not in TOGa Desk.** `sendEmail()`
+  queues via `App_Email` into `Common.EmailsQueued`; until TRUE-80952 attachments were serialized
+  as local **paths** that do not exist on the worker host that drains the queue, and PHPMailer 5
+  silently sent the mail without them. Two earlier tickets (TRUE-75813, TRUE-79924) "fixed" the
+  upload path inside TOGa Desk — which was never the problem — and passed testing because
+  **alpha/beta/local run `test_mode = 1`, which disables queueing entirely and sends inline**.
+  Never validate an attachment/email change for TOGa Desk on a non-prod environment. See
+  [App_Email queued sending & attachments](../../library/features/email-queue-attachments.md).
+- **The AIG branch of `sendEmail()` never attaches anything (open bug, separate from TRUE-80952).**
+  `desk/includes/functions.php` ~L961-975 and the AIG arm of `addReply()` in
+  `class.ticket.php` build `App_Email_AigTemplates_SendAigSupportTicketReplyNotification` and pass
+  `$attachments` only to `logEmail()` — **`addAttachment()` is never called**. Staples Protection
+  ticket replies therefore lose attachments for a different reason, and the TRUE-80952 fix does
+  not help them. Needs its own ticket.
 - **Silent disable by omission.** A missing `(clientid, type)` row produces no email and no log
   line — diagnose "missing notification" reports by querying `notifications` for that client's
   rows first, before suspecting recipient flags or code.
@@ -70,6 +87,9 @@ nothing if the `STAFF_*` row is absent.
   notification still depends on a `STAFF_NEW_TICKET` row existing for the ticket's client.
 
 ## Change history
+- 2026-08-20 — TRUE-80952: recorded that outbound attachments were lost at the `App_Email` queue
+  (not in TOGa Desk), that non-prod `test_mode` disables queueing and yields false passes, and the
+  separate AIG-arm bug where `$attachments` never reaches `addAttachment()` (mhammontree)
 - 2026-06-24 — documented the `notifications` `(clientid, type)` data model and staff-recipient
   resolution; recorded the clientid-178 missing-`STAFF_*`-rows incident (ajean)
 

package/knowledge/1.0/standards/backend-php.md

@@ -5,11 +5,12 @@ project: Library
 client: shared
 type: standard
 status: active
-updated: 2026-08-14
+updated: 2026-08-20
 owners: [jcardinal, rgirish, mhammontree, ajean]
 files: []
 related:
   - ../apps/library/architecture.md
+  - ../apps/library/features/email-queue-attachments.md
   - ../apps/library/features/error-capture-1-0.md
   - ../../2.0/standards/backend-php.md
 ---
@@ -306,6 +307,14 @@ Two sanctioned patterns, in order of preference:
 
 Never leave either one in effect beyond the block that needs it.
 
+**Worked example — reading a file you are about to serialize (TRUE-80952).**
+`App_Email::embedAttachmentsForQueue()` wraps its `file_get_contents` in a local
+`set_error_handler` → `ErrorException` with a `finally` restore. Guarding with
+`is_file()`/`is_readable()` alone is **not** sufficient: the file can vanish between the
+check and the read (TOCTOU), and the resulting warning would `exit()` the
+`send_emails` cron **mid-loop**, stranding every queued email behind it. Any file read
+inside a loop that processes a queue takes pattern 1.
+
 ### Start the session before App_Error's fatal handler — and set the handler in code, not INI
 
 On EB PHP 8.5 / AL2023, the `.so` extension and cookie flags aside, two rules make Redis
@@ -501,6 +510,25 @@ Additional verified detail:
 > (~L112-123), `clients/prudential/profile.md` (~L75). Prefer the local-handler pattern
 > from the canonical section above for anything new.
 
+### Test mode silently disables queued email — non-prod cannot validate a queue bug
+
+`App_Email::setQueueForSending()` (`library/app/email.php` ~L161) is gated on
+`!App_Registry::inTestMode()`:
+
+    $this->queueForSending = (($bool && !App_Registry::inTestMode()) ? … : false);
+
+`inTestMode()` is `config['internal']['test_mode'] > 0`, and **every non-prod config is 1**
+(verified for togadesk: prod 0; alpha 1; beta 1; all `config.dev-*.ini` 1). So outside
+production `setQueueForSending(true)` collapses to `false` and the email is sent **inline**,
+off the local filesystem, by the same process that built it. Anything that only breaks on the
+queue path — a serialization gap, a resource the drain host cannot reach — **passes on
+alpha/beta/local**. TRUE-75813 and TRUE-79924 were both verified and closed this way while
+production stayed broken.
+
+Rule: a change to queued email is not verified until it is verified in production, or against
+a harness that exercises the serialize → drain boundary with the queueing host's local state
+removed. Treat a non-prod pass on queue code as no evidence at all.
+
 ## Security Best Practices
 
 ### Input Validation and Sanitization

package/knowledge/2.0/apps/_underscore/features/tracking-number-bridges.md

@@ -6,7 +6,7 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-08-18
+updated: 2026-08-20
 owners: ["jcardinal", "mhammontree", "dfranks", "apeterson", "bala", "tcox"]
 files:
   - api2/Component/Api/V2/V2.php
@@ -29,6 +29,7 @@ files:
   - dbchanges2/Client_Elite/2026-08-17 - FulfillmentTableViews.sql
 related:
   - recursive-item-fulfillments.md
+  - ../../api2/features/cxml-shipnotice-gateway.md
 ---
 
 ## Summary
@@ -120,6 +121,20 @@ returnTrackingNumber: {...} }]`, and an IFIU's as `itemFulfillmentItemUnitTracki
 
 ## Gotchas / known issues
 
+- **⚠ The three ASN bridge `trackingNumberId` fields do NOT share a `childPolicy`, and a new client
+  must be provisioned per-API.** Core defaults: **1437** (unit bridge) `MATCH_CREATE`, **1441** (item
+  bridge) `MATCH`, **1445** (header bridge) `MATCH`. Only the unit bridge may mint a `TrackingNumbers`
+  row on its own. A producer that posts tracking at the item or header level (the cXML ShipNotice
+  translator does, since 2026-06-11) therefore needs `MATCH_CREATE` **overrides** in that tenant's
+  `Apis_RecordFields` for the writing API, or the write fails **EV-12 on field `trackingNumber`**.
+  Compass USA has all three overridden (`apiId 4`); Compass Canada had none, which rejected 100% of
+  its ship notices for 67 days (2026-06-11 → 2026-08-20). **Add these three override rows to the
+  onboarding checklist for any client whose ASNs arrive by cXML** — and ship them as a
+  `dbchanges2/Client_<Tenant>/` migration, since a manual patch dies on the next reseed. Do **not**
+  flip the Core defaults: Core ids are shared by every tenant (the same reason the IF-side 2184 flip
+  was rejected). Mechanics:
+  [nested-relationship writes](../../api2/features/nested-relationship-writes.md) and the
+  [cXML ShipNotice gateway](../../api2/features/cxml-shipnotice-gateway.md).
 - **Tracking is now an inherent child, not a scalar.** Any producer that sent a flat `trackingNumber`
   on a unit must nest it under the unit's `*TrackingNumbers` child array, or it is silently dropped.
 - **Pruning a unit/item means deleting its bridge rows FIRST — the bridge FKs are `RESTRICT`, not
@@ -238,6 +253,12 @@ returnTrackingNumber: {...} }]`, and an IFIU's as `itemFulfillmentItemUnitTracki
   for record 41.
 
 ## Change history
+- 2026-08-20 — Recorded the **`childPolicy` asymmetry across the three ASN tracking bridges** (1437
+  `MATCH_CREATE`, 1441/1445 `MATCH`) and the resulting per-API `Apis_RecordFields` provisioning
+  requirement for any client posting item- or header-level tracking. Compass Canada had zero override
+  rows, so every cXML ship notice was rejected EV-12 on `trackingNumber` from the 2026-06-11
+  ShipControl rework until 2026-08-20. Added to the new-client onboarding checklist; Core defaults
+  must not be flipped. (bala)
 - 2026-08-18 - **Measured which bridge each client actually populates, and it is not what "copy
   Compass" assumes.** Added the 318-vs-319 row-count table (Compass 4,312/10,144; CompassCanada
   169/138; Elite 0/109; Prudential 0/380; Quad 0/633; NYCHH 0/unit-level) and the rule to

package/knowledge/2.0/apps/api2/INDEX.md

@@ -5,6 +5,7 @@
 | [API (api2 / TOGa API v2) Architecture](architecture.md) | `api2` is the backend powering the public **TOGa 2.0 API**. | api2/Controller/Index.php, api2/Component/Api/V2/V2.php, api2/Component/Api/Cxml/Cxml.php, api2/Component/Api/V2/Response/Response.php, api2/Config/ |
 | [API Payload Interceptors (metadata-registered prePost/postPut model hooks)](features/api-payload-interceptors.md) | A `_Model`'s `prePost()` / `postPost()` / `prePut()` / `postPut()` hooks are **not** called by the model. | api2/Component/Api/V2/V2.php, api2/Controller/Index.php, _underscore/Model/Client/ItemFulfillment.php, _underscore/Model/Rate/Entitlement.php, _underscore/Model/Compass/PurchaseOrder.php, _underscore/Model/Elite/SalesOrder.php, _underscore/Model/Elite/ServiceRequest.php, _underscore/Model/Compass/SalesOrder.php, _underscore/Model/Compass/SalesOrderItem.php, dbchanges2/Core/2026-06-30a - ItemFulfillmentStageDefaultInterceptor.sql, dbchanges2/Core/2026-08-06a - Service request and sales order payload interceptors.sql, dbchanges2/Client_Compass/2026-08-06 - RemoveBrokenSalesOrderItemPostPostInterceptor.sql, dbchanges2/Client_Compass/2026-08-14a - AddSalesOrderInactiveStandaloneItemPreInterceptors.sql, dbchanges2/Client_CompassCanada/2026-08-14a - AddSalesOrderInactiveStandaloneItemPreInterceptors.sql |
 | [Multi-Client (Cross-Client) Data Retrieval](features/cross-client-data-retrieval.md) | A single authenticated V2 GET listing can return records across **many** clients (designed for 1000+) that the caller is entitled to, honoring **each target cli | api2/Component/Api/CrossClient/CrossClient.php, api2/Component/Api/V2/V2.php, api2/Controller/Index.php, _underscore/Model/Cache/Table.php, _underscore/Model/Cache/Tables/Client.php, _underscore/Model/Core/Record.php, worker2/Worker/Platform/Cache.php, worker2/Controller/Index.php, worker2/_.php, dbchanges2/Cache/2026-06-30a - MultiClientCacheTables.sql, dbchanges2/Core/2026-06-30b - CacheClusterRegistrationAndRecordTtl.sql, dbchanges2/Core/2026-07-27a - PlatformCacheCleanCron.sql |
+| [cXML ShipNotice Gateway (ASN ingestion, carrier resolution, per-client provisioning)](features/cxml-shipnotice-gateway.md) | `_Component_Api_Cxml` accepts a supplier `ShipNoticeRequest` and translates it into a `POST /v2/advance-shipping-notices` on the V2 JSON engine. | api2/Component/Api/Cxml/Cxml.php, api2/Component/Api/V2/V2.php, api2/Controller/Index.php |
 | [Encrypted-User-UUID Auth Handoff (/auth/encrypted-user-uuid)](features/encrypted-user-uuid-auth-handoff.md) | `POST /auth/encrypted-user-uuid` is the intended **cross-client / SSO-handoff identity mechanism**: given an encrypted `{client, user}` UUID pair, it mints a fr | api2/Component/Api/CrossClient/CrossClient.php |
 | [ENVIRONMENT (not the EB environment name) decides the _underscore branch and Config file](features/environment-variable-drives-underscore-branch.md) | An api2 Elastic Beanstalk instance decides **which `_underscore` branch it clones** and **which `Config/<env>.ini` it loads** from the EB environment property * | api2/.ebextensions/git.php, api2/.ebextensions/php_include_underscore.config, api2/.ebextensions/git.sandbox-dev.json, api2/.ebextensions/git.sandbox-client.json, api2/Config/beta.ini, api2/Config/sandbox-dev.ini, api2/Component/Api/V2/V2.php |
 | [Health-check endpoint (/health liveness short-circuit)](features/health-check-endpoint.md) | `_Controller_Index::api()` short-circuits **liveness/health-probe** requests to an HTTP 200 **before** any routing, DB bootstrap, or V2 engine work runs. | api2/Controller/Index.php |
@@ -25,4 +26,4 @@
 | [V2 REST query contract (params, where grammar, encoding, ACL behavior)](features/v2-rest-query-contract.md) | What an **HTTP client** has to get right to query the Toga v2 REST API: which query params are recognized, the exact `where` grammar, how the query string is (n | api2/Component/Api/V2/V2.php |
 | [V2 reverse hasMany collections must be named in the fetch fields whitelist](features/v2-reverse-hasmany-fields-whitelist.md) | In the V2 JSON engine, a **reverse hasMany** relationship — the collection of child records that foreign-key back to a parent (e.g. | api2/Component/Api/V2/V2.php |
 | [AWS CodePipeline Deployment via CodeConnections (GitHub → Elastic Beanstalk)](workflows/codepipeline-codeconnections-deploy.md) | 2.0 apps (`api2`, `_underscore`) are deployed through **AWS CodePipeline**. | api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, api2/ebs/register_instance_to_shared_application_load_balancer.php, api2/.platform/hooks/prebuild/git.sh, api2/.ebextensions/git.php, api2/.ebextensions/git.sandbox-dev.json |
-| [New Environment Configuration & Provisioning (api2)](workflows/environment-configuration-and-provisioning.md) | What it takes for a 2.0 API environment (e.g. | api2/Config/<environment>.ini, api2/Controller/Index.php, dbchanges2/Core/2026-06-16a - DatabaseHosts for new QA QC stage demo environments.sql, dbchanges2/Logs/, _underscore/Route.php |
+| [New Environment Configuration & Provisioning (api2)](workflows/environment-configuration-and-provisioning.md) | What it takes for a 2.0 API environment (e.g. | api2/Config/<environment>.ini, api2/composer.json, api2/composer.lock, api2/Controller/Index.php, dbchanges2/Core/2026-06-16a - DatabaseHosts for new QA QC stage demo environments.sql, dbchanges2/Logs/, _underscore/Route.php |

package/knowledge/2.0/apps/api2/architecture.md

@@ -6,7 +6,7 @@ project: API
 client: shared
 type: architecture
 status: active
-updated: 2026-08-03
+updated: 2026-08-20
 owners: [jcardinal, bala, mhammontree, dfranks]
 files:
   - api2/Controller/Index.php
@@ -14,7 +14,8 @@ files:
   - api2/Component/Api/Cxml/Cxml.php
   - api2/Component/Api/V2/Response/Response.php
   - api2/Config/
-related: []
+related:
+  - features/cxml-shipnotice-gateway.md
 ---
 
 ## Summary
@@ -146,8 +147,18 @@ Core `ClientApiIdentities` → dispatch: `OrderRequest` builds a Sales Order and
 `/sales-orders`; `ShipNoticeRequest` → `/advance-shipping-notices`; `PunchOutSetupRequest`
 creates a Vision (1.0) quote, Store (1.0) cart, or TOGa Commerce session. **cXML rides on
 top of the V2 JSON engine** (it authenticates and POSTs via `_ApiRequest`), inheriting ACL,
-interceptors, and logging. PunchOut paths write directly to legacy **Vision/Store** (1.0)
-DBs (`DB_VISION_1` / `DB_STORE_1`).
+interceptors, and logging. PunchOut paths write directly to legacy **Vision/Store**
+(1.0) DBs (`DB_VISION_1` / `DB_STORE_1`).
+
+**Critical rule — every internal call made from this component must run as the client resolved from
+the document's `Sender > Credential`, never as a hardcoded tenant.** A hardcoded client uuid/api
+credential here is a silent cross-tenant data leak: the read side uses `DB_CLIENT` and looks
+correct while writes land in the wrong client DB (found 2026-08-20 in
+`findOrCreateShippingCarrier()`, which wrote Compass Canada carriers into `Client_Compass`).
+Second gateway-wide trap: whether a nested child (e.g. an ASN `trackingNumber`) is **created** or
+only **looked up** is per-API DB provisioning — `Client_<x>.Apis_RecordFields.overrideChildPolicy` —
+so a fully-provisioned client can still reject 100% of its documents. Details:
+[cXML ShipNotice Gateway](features/cxml-shipnotice-gateway.md).
 
 ## Multi-database architecture
 

package/knowledge/2.0/apps/api2/features/cxml-shipnotice-gateway.md

@@ -0,0 +1,186 @@
+---
+title: cXML ShipNotice Gateway (ASN ingestion, carrier resolution, per-client provisioning)
+framework: "2.0"
+repo: api2
+project: API
+client: shared
+type: feature
+status: active
+updated: 2026-08-20
+owners: ["bala"]
+files:
+  - api2/Component/Api/Cxml/Cxml.php
+  - api2/Component/Api/V2/V2.php
+  - api2/Controller/Index.php
+related:
+  - ../architecture.md
+  - nested-relationship-writes.md
+  - ../../_underscore/features/tracking-number-bridges.md
+  - ../../../clients/compass-canada/features/grand-and-toy-asn-import.md
+  - ../../../clients/compass-usa/features/asn-to-item-fulfillment.md
+---
+
+## Summary
+
+`_Component_Api_Cxml` accepts a supplier `ShipNoticeRequest` and translates it into a
+`POST /v2/advance-shipping-notices` on the V2 JSON engine. Six clients use this gateway today
+(Compass USA, Compass Canada, PC Matic B2B, IM Houston and others). The translator resolves the
+authenticating client from the document's `Sender > Credential`, resolves/creates the shipping
+carrier, and builds the nested ASN payload including the three tracking bridges.
+
+Two things make this path fragile, and both bit production in 2026:
+
+1. **Nested tracking-number creation is per-API provisioning, not code.** Whether a nested
+   `trackingNumber` is *created* or only *looked up* comes from
+   `Client_<x>.Apis_RecordFields.overrideChildPolicy` for the authenticating API. A client can be
+   fully provisioned for ASNs and still reject **100%** of ship notices because that table is empty.
+2. **The gateway is multi-tenant but was not written multi-tenant everywhere.** One helper
+   authenticated its internal API call as a hardcoded tenant, so one client's traffic wrote rows
+   into another client's database.
+
+## Key files / entry points
+
+- `api2/Component/Api/Cxml/Cxml.php` — `execute()` authenticates the `Sender > Credential`,
+  dispatches `ShipNoticeRequest`, and builds the ASN payload.
+  `findOrCreateShippingCarrier()` resolves the carrier named in the document.
+- `api2/Component/Api/V2/V2.php` — `getChildPolicy()` (~8556) decides link-vs-create for each
+  nested object; see [nested-relationship-writes](nested-relationship-writes.md).
+- Host routing: `cxml` / `cxml1-3` / `cxml-writer` (see [architecture](../architecture.md)).
+
+## How it works
+
+1. The document arrives as `text/xml`. The `Sender > Credential` (Identity + SharedSecret) **is**
+   the authentication — there is no `Authorization` header on a cXML POST. So a stored document can
+   be replayed as-is without the caller handling any credential.
+2. The credential resolves to a client + API (`Core.ClientApiIdentities` and the tenant's `Apis`
+   row). Everything downstream must run **as that client**.
+3. Carrier: the header's `CarrierIdentifier` / `CarrierName` is matched against the tenant's
+   `ShippingCarriers` (`code = X OR name = X`), and created if absent.
+4. The ASN payload is POSTed to `/v2/advance-shipping-notices` through the V2 engine, so it inherits
+   ACL, interceptors and logging. The per-client ASN interceptor (Core record 55) then builds the
+   ItemFulfillment chain.
+5. The raw request and response are retained in full in `Logs_<Client>.Api`
+   (`direction = IN`, `route = '/'`, `method = POST`) — including **rejected** documents. That log is
+   the replay source for any backfill.
+
+## Per-client provisioning — the three tracking-bridge child policies
+
+A nested `trackingNumber` inside an ASN POST is only *created* when the owning bridge field's
+effective child policy is `MATCH_CREATE`. The `Core.RecordFields` defaults are **not uniform**:
+
+| Bridge field | RecordField | Core default `childPolicy` |
+|---|---|---|
+| ASN **unit** bridge `trackingNumberId` | 1437 | `MATCH_CREATE` |
+| ASN **item** bridge `trackingNumberId` | 1441 | `MATCH` |
+| ASN **header** bridge `trackingNumberId` | 1445 | `MATCH` |
+
+`getChildPolicy()` prefers `Client_<x>.Apis_RecordFields.overrideChildPolicy` for the
+authenticating API and only falls back to the Core default. So:
+
+- **Compass USA** cXML API (`apiId 4`) has all three overridden to `MATCH_CREATE` — its ASNs work.
+- **Compass Canada** "Grand & Toy (Cxml)" API (`apiId 3`) had **zero** rows, so 1441/1445 stayed
+  `MATCH`, the nested tracking number was looked up instead of created, and every ship notice was
+  rejected **EV-12 on field `trackingNumber`**. Fixed 2026-08-20 by inserting three rows
+  (`apiId 3`, `recordFieldId` 1437 / 1441 / 1445, `overrideChildPolicy = MATCH_CREATE`).
+
+> **Onboarding rule — this is the step that gets missed.** When enabling cXML ASN ingestion for a
+> new client, provisioning the ASN records and ACLs is **not enough**. Insert the three
+> `Apis_RecordFields` overrides for that client's cXML API, as a `dbchanges2/Client_<Tenant>/`
+> migration. Verify with
+> `SELECT apiId, recordFieldId, overrideChildPolicy FROM Client_<x>.Apis_RecordFields`.
+>
+> **Known gap today: PC Matic B2B** has the ASN records provisioned and **zero**
+> `Apis_RecordFields` rows, so it will fail the same way on its first ship notice. IM Houston is
+> worth the same check.
+
+Do **not** "fix" this by flipping `Core.RecordFields` 1441/1445 to `MATCH_CREATE`: Core ids are
+identical in every tenant, so that grants mint capability to every client at once. The equivalent
+change on the IF-side field (2184) was already proposed and rejected under CTO review. Per-API
+overrides are the sanctioned lever. Note also the sibling trap documented in
+[nested-relationship-writes](nested-relationship-writes.md): a row added **only** to set a child
+policy, leaving `overrideIsIdentifier` NULL, silently strips that field as an identifier.
+
+## Authenticate as the resolved Sender's client — never a hardcoded tenant
+
+`findOrCreateShippingCarrier()` had two defects (fixed 2026-08-20; code is in the working tree and
+**not yet committed**):
+
+- **Cross-tenant write.** The carrier *lookup* correctly used `DB_CLIENT`, but the carrier *create*
+  call was authenticated with a **hardcoded Compass USA client uuid plus api credentials**, so a
+  carrier created for any other tenant was written into `Client_Compass`. Evidence found in prod:
+  15 `PRECI`, 5 `GANDT` and 3 `XXXX` junk carrier rows in `Client_Compass` created by **Compass
+  Canada** traffic, plus 4 Canada tracking numbers left with a NULL carrier.
+- **Wrong response key.** It read the create response from `data->shippingCarrier` (singular) while
+  the API returns `data->shippingCarriers` (plural), so the new uuid came back `null` even on a
+  successful `201`.
+
+Fix: store the authenticated client uuid and api uuid/secret on the component when the cXML `Sender`
+credential is resolved, authenticate the internal call as **that** client, read the plural key with
+the singular kept as a fallback, and add `findShippingCarrierUuidByCode()` as a DB read-back if
+neither key parses. Verified locally end to end. Affects all six cXML gateway clients, not just
+Compass Canada.
+
+> **Rule for any new cXML helper that calls the API: take the client/api context from the resolved
+> Sender credential.** A hardcoded tenant constant in this component is a cross-tenant data leak,
+> and it fails silently, because the read side uses `DB_CLIENT` and therefore looks correct.
+
+## Carrier resolution constraints
+
+- `ShippingCarriers.code` holds **one** code per carrier, and the lookup is `code = X OR name = X`.
+  Several vendor codes therefore cannot map onto one carrier without duplicate rows — this is why
+  `Client_Compass` legitimately carries **7 separate FedEx rows**.
+- `TrackingNumbers.shippingMethodId` is **nullable** and cXML never sends a shipping method, so the
+  cXML path needs **no** `ShippingMethods` row per carrier. The "every carrier needs its own Ground
+  method" requirement came from the retired CSV cron only — do not carry it into cXML provisioning.
+
+## Gotchas / known issues
+
+- **⚠ No alerting on cXML gateway 4xx/5xx.** The gateway emails nobody on rejection. The retired
+  Compass Canada CSV cron emailed the team on success, failure, wrong column count and empty file;
+  the cXML path that replaced it is silent. This is why a **67-day, 100%-rejection outage went
+  unnoticed** on Compass Canada (see the
+  [G&T ASN import doc](../../../clients/compass-canada/features/grand-and-toy-asn-import.md)).
+  Highest-value remaining fix on this path: alert on any non-2xx `Logs_<Client>.Api` row for the
+  cXML route.
+- **`Apis.maxAuthsPerHour` throttles replays.** The gateway authenticates **once per document** and
+  the limit is `255`/hour, so a bulk cXML replay needs roughly **20-second spacing**. The JSON
+  `/v2/advance-shipping-notices` route reuses one Bearer token and needs no pacing.
+- **A rejected document is fully retained**, so any cXML outage is recoverable by replay from
+  `Logs_<Client>.Api` — dedupe on the vendor's shipment/order/tracking identifiers, never on the
+  log row id.
+
+## Testing the gateway locally (the only option while beta is down)
+
+Beta api2 is unusable at composer bootstrap (see
+[environment configuration and provisioning](../workflows/environment-configuration-and-provisioning.md)),
+so gateway changes are verified on a local Laragon machine:
+
+1. The api2 vhost already carries `ServerAlias cxml` and `*.cxml`, so `POST http://cxml/` serves the
+   local api2 checkout and the router's host parser resolves the host part to `cxml` and takes the
+   cXML branch.
+2. `Core.DatabaseHosts` maps the `dev` environment to `localhost`, and `[api] _` points at
+   `http://api2/v2`, so internal API calls stay local.
+3. The **local** `vendor/`'s `platform_check.php` requires only PHP 7.2.5, so it boots on Laragon's
+   PHP 8.1.x (unlike the deployed beta bundle).
+4. **Insert the client's `Apis_RecordFields` overrides locally first.** A fresh local
+   `Client_<x>.Apis_RecordFields` is empty, so without them the test fails EV-12 for the *original*
+   provisioning reason and the change under test is never exercised.
+
+## Change history
+- 2026-08-20 — Documented the cXML ShipNotice path as its own subject after a 67-day Compass Canada
+  outage. Root cause: nested tracking-number creation is governed by per-API
+  `Apis_RecordFields.overrideChildPolicy`, and Canada's cXML API had zero rows while the Core
+  defaults for 1441/1445 are `MATCH` (only 1437 is `MATCH_CREATE`) — every ASN rejected EV-12 on
+  `trackingNumber`. Also fixed two cross-tenant defects in `findOrCreateShippingCarrier()` (hardcoded
+  Compass USA auth wrote carriers into `Client_Compass`; the response was read from the singular
+  `shippingCarrier` key instead of `shippingCarriers`). Recorded the onboarding provisioning rule
+  (PC Matic B2B has the same gap today), the carrier `code` and nullable-`shippingMethodId`
+  constraints, the missing 4xx/5xx alerting, the `maxAuthsPerHour` replay pacing, and the local
+  Laragon test procedure. (bala)
+
+## Related docs
+- [api2 architecture](../architecture.md) — host dispatch and the cXML branch.
+- [Nested-relationship writes and child matching](nested-relationship-writes.md) — the
+  `Apis_RecordFields` override layer in full.
+- [Tracking-number bridges](../../_underscore/features/tracking-number-bridges.md) — the bridge
+  records these policies apply to.

package/knowledge/2.0/apps/api2/features/nested-relationship-writes.md

@@ -6,13 +6,14 @@ project: API
 client: shared
 type: feature
 status: active
-updated: 2026-08-17
+updated: 2026-08-20
 owners: ["bala", "mhammontree", "tcox"]
 files:
   - api2/Component/Api/V2/V2.php
   - _underscore/Model/Client/ContactEmailAddress.php
 related:
   - ../architecture.md
+  - cxml-shipnotice-gateway.md
   - ../../../clients/aig/features/entitlement-intake.md
   - ../../ai-bdr/features/web-funnel-app.md
   - ../../worker2/features/vapi-webhook-handler.md
@@ -169,6 +170,49 @@ columns matter:
 > Correct the data with an explicit `overrideIsIdentifier = 1`, or (systemic fix, not yet done)
 > change the code so only an explicit `0` removes an identifier and `NULL` means "no override".
 
+### `overrideChildPolicy` decides create-vs-lookup — and a MISSING row is a silent 100% outage
+
+`getChildPolicy()` (`V2.php` ~8556) resolves the effective policy as
+**`Client_<x>.Apis_RecordFields.overrideChildPolicy` for the authenticating API, falling back to
+`Core.RecordFields.childPolicy`**. Two consequences that are easy to miss:
+
+- **The same payload behaves differently per API within one tenant.** Provisioning a record and its
+  ACLs is not enough; the API doing the writing needs its own override rows.
+- **`Core` defaults are not uniform across sibling fields**, so "it works for the other client" is
+  not evidence the metadata is right. The ASN tracking bridges are the canonical example:
+
+| Bridge field | RecordField | Core default |
+|---|---|---|
+| ASN **unit** bridge `trackingNumberId` | 1437 | `MATCH_CREATE` |
+| ASN **item** bridge `trackingNumberId` | 1441 | `MATCH` |
+| ASN **header** bridge `trackingNumberId` | 1445 | `MATCH` |
+
+Compass USA's cXML API (`apiId 4`) overrides all three to `MATCH_CREATE`; Compass Canada's cXML API
+(`apiId 3`) had **zero** rows, so its nested `trackingNumber` was looked up instead of created and
+**100% of its ship notices were rejected EV-12 on field `trackingNumber` for 67 days** (root-caused
+2026-08-20). Fixed by inserting the three override rows for `apiId 3`.
+
+**Do not "fix" this class of bug by editing `Core.RecordFields`.** Core ids are identical in every
+client DB, so flipping a base `childPolicy` to `MATCH_CREATE` grants mint capability to every tenant
+at once — the reason the equivalent proposal on the IF-side field (2184) was rejected under CTO
+review. Per-API overrides are the sanctioned lever. Full ASN case:
+[cXML ShipNotice Gateway](cxml-shipnotice-gateway.md).
+
+### `MATCH_UPSERT` on a nested FK creates the related record — including deep grandchildren
+
+`MATCH_UPSERT` means "match on an identifier, otherwise create". So a nested object two levels deep
+is created by the same write. Measured on an ASN POST (prod, 2026-08-20): RecordField **335**
+(`advance-shipping-notice-item-units.unitId`) is `MATCH_UPSERT`, so a nested `unit` that matches
+nothing is **created** and linked to the ASN item unit — no separate `/units` call is needed, and
+interceptor 55 then propagates it to `ItemFulfillmentItemUnits`.
+
+The trap is one level deeper: **`units.itemId` is itself `MATCH_UPSERT`**. Referencing the item by a
+non-identifier value (e.g. a `partNumber`) or by a mistyped uuid **silently creates a junk `Items`
+row** rather than failing. Always reference an existing parent record **by `uuid`**, and resolve
+business keys to uuids in the caller. (Also note `Units.itemId` is NOT NULL, so the item cannot be
+omitted; and the `Units` identifier set is `uuid`/`itemId`/`serialNumber`/`assetTag`/`vendorId`/
+`macAddress`, so item + serialNumber is what gives find-or-create on a device.)
+
 ### Fix it with a migration — a manual one-row data patch on non-prod DOES NOT SURVIVE
 
 The TRUE-79978 beta fix was applied by hand as a single `UPDATE` on 2026-07-09. The **identical
@@ -260,6 +304,16 @@ anywhere — no `messages[]`, nothing in the client `Logs` or the core/writer `L
 
 ## Change history
 
+- 2026-08-20 — Documented that **`overrideChildPolicy` (not just `overrideIsIdentifier`) is per-API
+  provisioning**, and that a *missing* `Apis_RecordFields` row is a silent 100%-rejection outage:
+  Compass Canada's cXML API had zero rows and the Core defaults for the ASN **item**/**header**
+  tracking bridges (1441/1445) are `MATCH` while the **unit** bridge (1437) is `MATCH_CREATE`, so
+  every ship notice failed EV-12 on `trackingNumber` for 67 days. Added the rule that this class of
+  bug must be fixed with per-API overrides, never by editing `Core.RecordFields` (Core ids are shared
+  by every tenant). Also documented `MATCH_UPSERT` deep creation — a nested `unit` on an ASN POST is
+  created outright (RecordField 335), and because `units.itemId` is *also* `MATCH_UPSERT`, a
+  non-identifier or mistyped item reference silently creates a junk `Items` row, so parents must be
+  referenced by `uuid`. (bala)
 - 2026-08-17 (later pass) — Two new measured behaviors from the TRUE-80562 verification run on beta
   `Client_Aig`. (1) **CREATE-path re-parenting:** a nested forward singular FK
   (`primaryContactEmailAddress` / `primaryContactPhoneNumber`) is matched **tenant-wide by VALUE**,

package/knowledge/2.0/apps/api2/workflows/environment-configuration-and-provisioning.md

@@ -6,10 +6,12 @@ project: API
 client: shared
 type: workflow
 status: active
-updated: 2026-07-20
-owners: ["jcardinal"]
+updated: 2026-08-20
+owners: ["jcardinal", "bala"]
 files:
   - api2/Config/<environment>.ini
+  - api2/composer.json
+  - api2/composer.lock
   - api2/Controller/Index.php
   - dbchanges2/Core/2026-06-16a - DatabaseHosts for new QA QC stage demo environments.sql
   - dbchanges2/Logs/
@@ -103,7 +105,38 @@ Diagnostic order:
 `Logs` schema was the *only* thing missing. Don't chase seed/config/connectivity when only base `Logs`
 is absent.
 
+## Edge case — a deployed `vendor/` built against the wrong PHP version kills the whole env
+
+**Symptom (beta api2, live 2026-08-20): every request fails, with no routing at all.**
+`vendor/composer/platform_check.php` demands **PHP >= 8.5.0** while the instance runs **8.1.12**, so
+it fatals inside the composer autoload at `Controller/Index.php:41` — **before** any routing, logging
+or error envelope. No Sentry event that looks like a request, nothing in `Logs.Api`.
+
+**It is not a code bug and not a config bug — the deployed artifact does not match the repo.**
+`composer.json` pins `php >= 8.2`, and `composer.lock` is **byte-identical** on `_beta` and
+`_production` (same content-hash, `phpmailer v7.1.1`, nothing requiring 8.5). So the deployed
+`vendor/` was **not built from the committed lock**.
+
+Diagnosis order when an environment fails with *no* log line at all:
+1. `head vendor/composer/platform_check.php` on the instance and compare its required version
+   against `php -v`.
+2. Compare `composer.lock` content-hash across branches; if they match, suspect the build, not the
+   repo.
+
+**Fix is DevOps, not application code:** raise the EB platform to at least PHP 8.2 **and** rebuild
+`vendor/` with `composer install` (never `composer update`, which would re-resolve against the
+builder's PHP and can re-introduce an 8.5 constraint). Until beta is repaired, gateway/API changes
+have to be verified locally — see
+[cXML ShipNotice Gateway → testing locally](../features/cxml-shipnotice-gateway.md).
+
 ## Change history
+- 2026-08-20 — Recorded that **beta api2 is down at composer bootstrap**:
+  `vendor/composer/platform_check.php` requires PHP >= 8.5.0 while the instance runs 8.1.12, fataling
+  in `Controller/Index.php:41` before any routing, so all beta api2 requests fail with nothing
+  logged. `composer.json` pins `php >= 8.2` and `composer.lock` is byte-identical on `_beta` and
+  `_production`, so the deployed `vendor/` was not built from the committed lock. Fix is DevOps
+  (raise the EB platform to >= 8.2 and rebuild with `composer install`, not `update`). Added the
+  no-log-at-all diagnosis order. (bala)
 - 2026-07-20 — `sandbox-client` API 500'd on every request (Route.php:525 view fatal). Root cause: the
   base `Logs` schema (`DB_LOGS`, `Core.Databases` id 11; tables Api/Error/Webhook) was missing on the
   all-in-one cluster although every `Logs_<Client>` existed. `_Controller_Index::api()` queries

package/knowledge/INDEX.md

@@ -4,7 +4,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 
 ## 1.0 framework
 
-- **library** (Library) _(framework core)_ — 18 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
+- **library** (Library) _(framework core)_ — 19 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
 - **worker** (Worker) — 25 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
 - **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
 - **worker1.5** (Worker 1.5) — 0 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
@@ -20,7 +20,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 
 - **_underscore** (_Underscore) _(framework core)_ — 60 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
 - **worker2** (Worker) — 53 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
-- **api2** (API) — 24 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
+- **api2** (API) — 25 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
 - **dbchanges2** (Database Changes) _(framework core)_ — 8 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
 - **toga2-supply** (TOGa Supply) — 7 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)
 - **saml** (SAML SSO Gateway) — 3 doc(s) → [2.0/apps/saml/INDEX.md](2.0/apps/saml/INDEX.md)

package/knowledge/clients/compass-canada/INDEX.md

@@ -4,5 +4,6 @@
 |-----|-----------|---------|-------|
 | [French (fr-CA) Item Feature Translations (Compass Canada)](features/french-item-feature-translations.md) | 2.0 | Renders item **feature** text on the Compass Canada French storefront — feature names, feature-group headers (e.g. | _underscore/Model/Compass/Canada/Feature.php, _underscore/Model/Compass/Canada/ItemCategoryFeatureGroup.php, _underscore/Model/Client/FeatureTranslation.php, _underscore/Model/Client/ItemCategoryFeatureGroupTranslation.php, _underscore/Model/Client/ItemFeatureTranslation.php, worker2/Worker/Etilize/ItemTranslations.php, dbchanges2/Client/2026-07-13a - FeatureTranslations.sql, dbchanges2/Client/2026-07-13b - FeatureTranslationsAcl.sql, dbchanges2/Core/2026-07-13 - FeatureTranslationsRecord.sql, dbchanges2/Client_CompassCanada/2026-07-13 - FeatureAttributeCustomFields.sql, dbchanges2/Client_CompassCanada/2026-07-13 - DedupeItemFeaturesAndGroups.sql, dbchanges2/Client_CompassCanada/2026-07-13 - SeedFrenchFeatureTranslations.sql |
 | [French (fr-CA) Order Email Localization (Compass Canada)](features/french-order-email-localization.md) | 2.0 | Compass Canada order emails (order requested, manager-approval request, approved/rejected, in-transit, delivered, reminders) are sent in **each recipient's own | _underscore/Model/Compass/SalesOrder.php, _underscore/Model/Compass/ApprovalDecision.php, _underscore/Component/Api/Togaiq/Togaiq.php, _underscore/String.php, library/app/client/compasscanada.php, worker/crons/toga2/compasscanada/send_delivered_email.php, worker/crons/toga2/compasscanada/compass_email_reminders.php, worker/crons/toga2/compasscanada/compass_cancel_pending_approval_orders.php, worker/crons/toga2/compasscanada/update_salesorder_status_from_odp.php, worker/crons/toga2/compasscanada/workflow/3_update_salesorder_status_from_grand_and_toy.php, worker/crons/toga2/compasscanada/workflow/4_import_grand_and_toy_advance_shipping_notices.php, worker/crons/toga2/compasscanada/workflow/test_partial_in_transit_email.php, worker/crons/toga2/compasscanada/workflow/test_partial_delivered_email.php, worker/crons/toga2/compasscanada/workflow/test_email_previews_prod.php |
-| [Grand & Toy ASN Import (Compass Canada)](features/grand-and-toy-asn-import.md) | 2.0 | Imports Grand & Toy (G&T) Advance Shipping Notices for Compass Canada. | worker/crons/toga2/compasscanada/workflow/4_import_grand_and_toy_advance_shipping_notices.php, worker/crons/toga2/compasscanada/workflow/import_grand_and_toy_asn_from_file.php, worker/schedules/cron.worker.sync.json, _underscore/Model/Compass/AdvanceShippingNotice.php, _underscore/Model/Compass/Canada/AdvanceShippingNotice.php, dbchanges2/Client_CompassCanada/ |
+| [Grand & Toy ASN Import (Compass Canada)](features/grand-and-toy-asn-import.md) | 2.0 | Imports Grand & Toy (G&T) Advance Shipping Notices for Compass Canada. | api2/Component/Api/Cxml/Cxml.php, worker/crons/toga2/compasscanada/workflow/4_import_grand_and_toy_advance_shipping_notices.php, worker/crons/toga2/compasscanada/workflow/import_grand_and_toy_asn_from_file.php, worker/crons/toga2/compasscanada/workflow_beta/import_grand_and_toy_asn_from_file.php, worker/schedules/cron.worker.sync.json, _underscore/Model/Compass/AdvanceShippingNotice.php, _underscore/Model/Compass/Canada/AdvanceShippingNotice.php, dbchanges2/Client_CompassCanada/ |
 | [Compass Canada](profile.md) | 2.0 | Compass Canada is the Canadian arm of the Compass account — a separate TOGA tenant, related to but distinct from Compass USA. |  |
+| [Grand & Toy ASN Backfill (cXML replay + CSV-to-JSON)](workflows/grand-and-toy-asn-backfill.md) | 2.0 | How to recover Compass Canada shipments whose ASN never landed — used on 2026-08-20 to backfill the 67-day Grand & Toy outage (**518 shipments**: 171 via cXML r | worker/crons/toga2/compasscanada/workflow_beta/import_grand_and_toy_asn_from_file.php, api2/Component/Api/Cxml/Cxml.php, _underscore/Model/Compass/AdvanceShippingNotice.php |

package/knowledge/clients/compass-canada/features/grand-and-toy-asn-import.md

@@ -5,17 +5,22 @@ project: _Underscore
 client: compass-canada
 type: client-feature
 status: active
-updated: 2026-06-22
+updated: 2026-08-20
 owners: ["bala"]
 files:
+  - api2/Component/Api/Cxml/Cxml.php
   - worker/crons/toga2/compasscanada/workflow/4_import_grand_and_toy_advance_shipping_notices.php
   - worker/crons/toga2/compasscanada/workflow/import_grand_and_toy_asn_from_file.php
+  - worker/crons/toga2/compasscanada/workflow_beta/import_grand_and_toy_asn_from_file.php
   - worker/schedules/cron.worker.sync.json
   - _underscore/Model/Compass/AdvanceShippingNotice.php
   - _underscore/Model/Compass/Canada/AdvanceShippingNotice.php
   - dbchanges2/Client_CompassCanada/
 related:
   - ../profile.md
+  - ../workflows/grand-and-toy-asn-backfill.md
+  - ../../../2.0/apps/api2/features/cxml-shipnotice-gateway.md
+  - ../../compass-usa/features/asn-to-item-fulfillment.md
 ---
 
 ## Summary
@@ -25,6 +30,40 @@ G&T's ASN CSV from a mailbox, posts each shipped line to the 2.0 API as an Advan
 an in-transit email in English or French. It mirrors the Compass USA ODP / Strategic-Systems ASN
 flow, adapted for Canadian carriers and bilingual email.
 
+**The live path today is cXML, not the CSV cron.** G&T transmits ship notices to the api2 cXML
+gateway (`cxml-writer`), which translates them into the same `POST /v2/advance-shipping-notices`.
+The 24-column CSV cron below is **retired** — it remains documented because it is the only source of
+serial numbers and the reference for the field mapping.
+
+> ### Outage 2026-03 to 2026-08-20 — two stacked defects, 518 shipments lost
+>
+> **100% of G&T ship notices were rejected from 2026-06-11**, and most since March. Nothing alerted,
+> because the cXML gateway emails nobody on a 4xx (the retired CSV cron did).
+>
+> 1. **Before 2026-06-11** the translator sent G&T's carrier **CODE** in a field matched against
+>    `ShippingCarriers.NAME`. Only `UPS` and `DHL` resolved (code happens to equal the name); every
+>    other carrier failed **EV-12 on field `shippingCarrier`**. So the feed was already losing most
+>    shipments while looking healthy.
+> 2. **On 2026-06-11** the ShipControl rework (api2 commits `9259a10`, `8c987b0`) moved tracking onto
+>    the item/header bridges (RecordFields 1441/1445) and dropped the carrier from the header
+>    fallback. Those two bridge fields default to `childPolicy = MATCH` in Core and Compass Canada's
+>    cXML API had **no** `Apis_RecordFields` overrides, so the nested `trackingNumber` was looked up
+>    instead of created and every ASN failed **EV-12 on field `trackingNumber`**. Failure became
+>    total. Root-cause mechanics:
+>    [cXML ShipNotice Gateway](../../../2.0/apps/api2/features/cxml-shipnotice-gateway.md).
+>
+> Recovery procedure: [G&T ASN Backfill](../workflows/grand-and-toy-asn-backfill.md).
+
+> ### ⚠ MIGRATION DEBT — the 2026-08-20 fixes exist ONLY in production
+>
+> The three `Client_CompassCanada.Apis_RecordFields` rows (`apiId 3`, `recordFieldId` 1437/1441/1445,
+> `overrideChildPolicy = MATCH_CREATE`) and the carrier code/row changes were applied **directly to
+> the production database**. There is **no `dbchanges2/Client_CompassCanada/` migration yet**, so
+> beta, dev-sandbox, and any rebuilt or reseeded environment still fail for the original reason.
+> Proven this session: the local `Client_CompassCanada.Apis_RecordFields` is empty and reproduced the
+> June failure exactly. Write the migration before anything else touches this feed. (A manual
+> non-prod patch also evaporates on the next tenant reseed — the same lesson as AIG TRUE-80562.)
+
 ## Key files / entry points
 - `worker/crons/toga2/compasscanada/workflow/4_import_grand_and_toy_advance_shipping_notices.php`
   — the scheduled cron (1.0 worker tier). Reads mailbox `compasscanada.status@togatech.com`
@@ -74,11 +113,84 @@ flow, adapted for Canadian carriers and bilingual email.
 - Interceptor wiring: `Core.RecordScripts` + `Core.ApiPayloadInterceptors`, `recordId 55`
   (AdvanceShippingNotice). The `sendEmail` POST record script is `Core.RecordScripts recordId 202`.
 
+## The cXML documents G&T actually sends
+
+Measured across **all 1,074** stored documents in `Logs_CompassCanada.Api` — the shape never varies:
+
+- **No `ShipControl` block at all.** The ShipControl parser never fires for this vendor. Carrier and
+  tracking arrive on `ShipNoticeHeader` as `CarrierIdentifier` > `CarrierName` and `TrackingID`.
+- **No `Extrinsic` of any kind** — so no per-line tracking and no `TrackingURL`.
+- **No serial numbers and no asset tags.**
+- **All `operation="new"`.**
+- **One `TrackingID` even when `TotalPackages > 1`.**
+
+**Consequence:** the retired CSV feed was the only source of serial numbers, so ongoing cXML ASNs
+create **no `Units`**. If Compass Canada expect per-device serial visibility, that needs a business
+decision — it cannot be recovered from the cXML.
+
+## Carriers (cXML)
+
+G&T's complete carrier code set, harvested from all 1,074 documents:
+
+| Code | Carrier |
+|---|---|
+| `UPS` | UPS |
+| `FDXH` | FedEx |
+| `PUROL` | Purolator |
+| `PRECI` | Precision |
+| `NATEX` | Nationex |
+| `GANDT` | Grand & Toy own van |
+| `DHL` | DHL |
+| `XXXX` | placeholder — its `TrackingID` is the literal string `DIRECT` |
+
+Applied 2026-08-20: added the missing `code` values `PRECI` (Precision) and `NATEX` (Nationex), and
+created a **Grand & Toy** carrier row in `Client_CompassCanada.ShippingCarriers`.
+
+- `ShippingCarriers.code` holds **one** code per carrier and the lookup matches `code = X OR
+  name = X`, so several vendor codes cannot map onto one carrier without duplicate rows (this is why
+  `Client_Compass` has 7 separate FedEx rows).
+- **No Ground `ShippingMethod` is needed for the cXML path.** `TrackingNumbers.shippingMethodId` is
+  nullable and cXML never sends a method. The "each carrier needs its own Ground method" rule below
+  came from the retired CSV cron only.
+
+## Serial numbers in the same ASN POST (no separate /units call)
+
+A nested `unit` object inside the ASN POST creates the `Units` row outright: RecordField **335**
+(`advance-shipping-notice-item-units.unitId`) is `MATCH_UPSERT`, so a unit that does not match is
+created and linked to the ASNIU, and interceptor **55** then propagates it to
+`ItemFulfillmentItemUnits`. Verified in prod (Units row created + linked, IFIU populated).
+
+Constraints:
+- `Units.itemId` is **NOT NULL**, so the item must be supplied.
+- The `Units` identifiers are `uuid`, `itemId`, `serialNumber`, `assetTag`, `vendorId`, `macAddress`
+  — so **item + serialNumber** gives find-or-create on that pair.
+- `units.itemId` is itself `MATCH_UPSERT`, so **reference the item by `uuid`, never by
+  `partNumber`** — a typo silently creates a junk `Items` row.
+- Resolve the item **PO-scoped**: `PurchaseOrderItems` → `VendorItems.vendorPartNumber` →
+  `Items.uuid`, because the G&T "Part Number" column holds the **vendor** part number.
+
 ## Client variations
 Compass-Canada-specific; parallels Compass USA's ODP / Strategic-Systems ASN import. Canada sends
 a bilingual EN/FR in-transit email (USA is English only) and uses Canadian carriers.
 
 ## Gotchas / known issues
+- **⚠ `PurchaseOrderItems_SalesOrderItems` is EMPTY across ALL 865 Compass Canada purchase orders —
+  this is NORMAL and must NOT be "repaired".** `_Model_Compass_AdvanceShippingNotice::postPost`
+  resolves PO to SO without that bridge for this client. Inserting bridge rows to make one ASN
+  succeed would change fulfilment behaviour for that PO. If an ASN fails, the cause is elsewhere
+  (child policy, carrier, or the already-tracked-line `EO-1` below).
+- **⚠ Nothing alerts on a rejected ship notice.** The cXML gateway emails nobody on 4xx/5xx; the
+  retired CSV cron emailed `vburks@togatech.com` + `devTeam@togatech.com` on success, failure, wrong
+  column count and empty file. That asymmetry is why this outage ran 67 days unnoticed, and adding
+  alerting is the highest-value remaining fix. See the
+  [cXML gateway doc](../../../2.0/apps/api2/features/cxml-shipnotice-gateway.md).
+- **A line whose only `ItemFulfillmentItem` already carries item-level tracking makes the ASN POST
+  return 500 `EO-1`, not a validation error.** `remaining = 0` routes to the reconcile path,
+  `itemFulfillmentItemHasTracking()` skips the IFI, and the handler then fails instead of completing
+  cleanly. Deterministic (reproduced three times, both transports; PO `50305097-1` line 4, Sentry
+  `AV-30` / `69-68`). It is in the **shared Compass parent**, so Office Depot re-shipments on
+  already-tracked lines hit it too — see
+  [Compass ASN to ItemFulfillment](../../compass-usa/features/asn-to-item-fulfillment.md).
 - **CSV "Part Number" is the VENDOR part number (`VendorItems.vendorPartNumber`), NOT
   `Items.partNumber` directly.** Resolve the item through the `VendorItems` join. (e.g. vendor
   `IMFP3260` → `Items.partNumber` `C2CY5UC#ABA`.)
@@ -116,6 +228,22 @@ a bilingual EN/FR in-transit email (USA is English only) and uses Canadian carri
   before `fetchOne()`.
 
 ## Change history
+- 2026-08-20 — **Root-caused and fixed a 67-day, 100%-rejection outage on the G&T cXML ASN feed and
+  backfilled 518 shipments.** Root cause: nested tracking-number creation is governed by the
+  per-API `Client_<x>.Apis_RecordFields.overrideChildPolicy`, and Canada's "Grand & Toy (Cxml)" API
+  (`apiId 3`) had **zero** rows while the Core defaults for RecordFields 1441 (item bridge) and 1445
+  (header bridge) are `MATCH` (only 1437 is `MATCH_CREATE`) — so every ASN failed EV-12 on
+  `trackingNumber` after the 2026-06-11 ShipControl rework moved tracking onto those bridges. Before
+  that date a separate defect sent the carrier CODE into a NAME match, so only UPS and DHL resolved.
+  Applied to prod: three `Apis_RecordFields` `MATCH_CREATE` rows, the missing `PRECI`/`NATEX` carrier
+  codes, and a Grand & Toy carrier row. Also fixed the cross-tenant `findOrCreateShippingCarrier()`
+  defect in api2 (carriers for other tenants were written into `Client_Compass`). Backfilled 518
+  shipments (536 ASNs / 512 TrackingNumbers / 406 ItemFulfillments / 332 Units). Documented G&T's
+  invariant cXML shape (no ShipControl, no Extrinsic, no serials, one TrackingID per shipment), the
+  in-POST serial-number path (RecordField 335 `MATCH_UPSERT`), the empty
+  `PurchaseOrderItems_SalesOrderItems` invariant, the missing gateway alerting, and the outstanding
+  `EO-1` already-tracked-line bug. **⚠ All DB changes are prod-only — no dbchanges2 migration yet.**
+  (bala)
 - 2026-06-22 — Full variable rename (no abbreviations); moved PO/carrier/shipping-method lookup maps
   outside the CSV row loop (was causing N full-table queries per file); fixed ASNIU→TN link to run
   unconditionally for new ASNIUs (not only when TN was new); added PHP 8 null-coalesce on carrier→
@@ -127,3 +255,6 @@ a bilingual EN/FR in-transit email (USA is English only) and uses Canadian carri
 
 ## Related docs
 - [Compass Canada profile](../profile.md)
+- [G&T ASN Backfill workflow](../workflows/grand-and-toy-asn-backfill.md)
+- [cXML ShipNotice Gateway (api2)](../../../2.0/apps/api2/features/cxml-shipnotice-gateway.md)
+- [Compass ASN to ItemFulfillment (shared handler)](../../compass-usa/features/asn-to-item-fulfillment.md)

package/knowledge/clients/compass-canada/profile.md

@@ -16,11 +16,14 @@ project: _Underscore
 client: compass-canada
 type: profile
 status: active
-updated: 2026-08-17
+updated: 2026-08-20
 owners: [jcardinal, bala, tcox, apeterson]
 files: []
 related:
   - ../compass-usa/profile.md
+  - features/grand-and-toy-asn-import.md
+  - workflows/grand-and-toy-asn-backfill.md
+  - ../../2.0/apps/api2/features/cxml-shipnotice-gateway.md
   - ../compass-usa/features/approval-decision-flow.md
   - ../compass-usa/features/isfulfillable-data-quality-and-type-rule.md
   - features/french-order-email-localization.md
@@ -47,8 +50,21 @@ to but distinct from Compass USA. Like Compass USA it spans the **2.0** commerce
 - **Grand & Toy (G&T)** — primary hardware vendor. SOs flow toga → MITS → PO to G&T; G&T sends
   back ASNs. ASN ingestion (email CSV + the auto-created ItemFulfillment chain + bilingual
   in-transit email) is documented in [Grand & Toy ASN Import](features/grand-and-toy-asn-import.md).
-  Vendor uuid: `App_Client_CompassCanada::UUID_VENDOR__GRAND_TOY`. Canadian carriers in use:
-  UPS, FedEx, Purolator, Precision, Nationex, ATSL (each with a Ground `ShippingMethod`).
+  Vendor uuid: `App_Client_CompassCanada::UUID_VENDOR__GRAND_TOY`. **The live ASN path is cXML**
+  (api2 `cxml-writer`), not the retired 24-column CSV cron; carrier codes G&T sends are `UPS`,
+  `FDXH`, `PUROL`, `PRECI`, `NATEX`, `GANDT`, `DHL`, `XXXX` (placeholder, TrackingID = `DIRECT`).
+  The cXML path needs **no** Ground `ShippingMethod` per carrier (`TrackingNumbers.shippingMethodId`
+  is nullable and cXML sends no method) — that requirement came from the CSV cron only.
+- **⚠ The G&T ASN feed rejected 100% of ship notices for 67 days (2026-06-11 → 2026-08-20) and
+  nothing alerted.** Root cause: nested tracking-number creation is governed by
+  `Client_CompassCanada.Apis_RecordFields.overrideChildPolicy` per API, and the "Grand & Toy (Cxml)"
+  API (`apiId 3`) had zero rows. 518 shipments were backfilled. **The fix is prod-only — there is no
+  `dbchanges2/Client_CompassCanada/` migration yet, so beta / dev-sandbox / any reseed still fails.**
+  See [G&T ASN Import](features/grand-and-toy-asn-import.md) and the
+  [backfill workflow](workflows/grand-and-toy-asn-backfill.md).
+- **`PurchaseOrderItems_SalesOrderItems` is empty for all 865 Canada POs — that is normal for this
+  client.** `postPost` resolves PO to SO without it; never insert bridge rows to "repair" an ASN
+  failure here.
 - Worker crons for the G&T flow live under `worker/crons/toga2/compasscanada/workflow/`
   (`1_…` transmit SOs to MITS, `2_…` transmit POs to vendors, `3_…` status from G&T cXML,
   `4_…` import G&T ASNs).

package/knowledge/clients/compass-canada/workflows/grand-and-toy-asn-backfill.md

@@ -0,0 +1,101 @@
+---
+title: Grand & Toy ASN Backfill (cXML replay + CSV-to-JSON)
+framework: "2.0"
+project: _Underscore
+client: compass-canada
+type: workflow
+status: active
+updated: 2026-08-20
+owners: ["bala"]
+files:
+  - worker/crons/toga2/compasscanada/workflow_beta/import_grand_and_toy_asn_from_file.php
+  - api2/Component/Api/Cxml/Cxml.php
+  - _underscore/Model/Compass/AdvanceShippingNotice.php
+related:
+  - ../features/grand-and-toy-asn-import.md
+  - ../profile.md
+  - ../../../2.0/apps/api2/features/cxml-shipnotice-gateway.md
+---
+
+## Summary
+
+How to recover Compass Canada shipments whose ASN never landed — used on 2026-08-20 to backfill the
+67-day Grand & Toy outage (**518 shipments**: 171 via cXML replay + 347 via JSON; result **536 ASNs,
+512 TrackingNumbers, 406 ItemFulfillments, 332 Units**). Two complementary sources, because neither
+alone is complete: the stored cXML documents cover what G&T actually transmitted, and the G&T CSV
+covers everything else (and is the only source of serial numbers).
+
+**Run the root-cause fix first.** A backfill against an unprovisioned tenant just re-creates the
+original EV-12 rejections — see
+[G&T ASN Import](../features/grand-and-toy-asn-import.md) and the
+[cXML gateway provisioning rule](../../../2.0/apps/api2/features/cxml-shipnotice-gateway.md).
+
+## Route A — replay the stored cXML
+
+1. **Source.** Every rejected document is retained complete in `Logs_CompassCanada.Api`
+   (`direction = IN`, `route = '/'`, `method = POST`). Nothing was lost by the outage.
+2. **Dedupe** on `shipmentID` + `orderID` + `TrackingID`. G&T re-sends, and the log holds every
+   attempt.
+3. **Re-POST** the document body verbatim to `cxml-writer.togahub.com` as `text/xml`.
+   The `Sender` credential is **inside the document**, so no `Authorization` header is needed or read
+   — never extract or handle the credential.
+4. **Pace it ~20 seconds per document.** `Apis.maxAuthsPerHour` is `255` and the gateway
+   authenticates once per document.
+
+## Route B — build JSON ASNs from the G&T CSV
+
+1. POST to `/v2/advance-shipping-notices` with a Bearer token (one token for the whole run; no
+   pacing needed, the auth limit does not apply).
+2. **The payload must carry all three tracking bridges** or the partial in-transit email loses its
+   per-package item breakdown:
+   - `advanceShippingNoticeTrackingNumbers` (header)
+   - `advanceShippingNoticeItemTrackingNumbers` (item)
+   - `advanceShippingNoticeItemUnits[].advanceShippingNoticeItemUnitTrackingNumbers` (unit)
+3. Serial numbers can be created **inside the same POST** (nested `unit` object) — no separate
+   `/units` call. Resolve the item **PO-scoped** through `PurchaseOrderItems` →
+   `VendorItems.vendorPartNumber` → `Items.uuid`, because the G&T "Part Number" column holds the
+   **vendor** part number, and reference the item **by uuid only**. See the ASN-import doc's
+   serial-number section for the constraints.
+
+## Dedupe and safety guards
+
+- **`TrackingNumbers.number` carries a UNIQUE key**, so a replayed tracking number `MATCH`es rather
+  than duplicating. **But a repeated document still creates a duplicate ASN.** Always skip any
+  shipment whose tracking number already exists before posting.
+- **Suppress stale customer emails.** Stamp `TrackingNumbers.c_dtInTransitEmailSent`, keyed off
+  `AdvanceShippingNotices.dateShipped` older than **3 days**, so a genuinely fresh shipment still
+  emails the customer while months-old backfilled shipments stay silent.
+- Verify by row count afterwards (ASNs / TrackingNumbers / ItemFulfillments / Units), never by HTTP
+  status — a 201 only proves the parent was written.
+
+## Systems involved
+
+- `Logs_CompassCanada.Api` (cXML source of truth), `cxml-writer.togahub.com` (gateway),
+  `/v2/advance-shipping-notices` (JSON route), `Client_CompassCanada` (ASN/IF/Units/TrackingNumbers),
+  the shared `_Model_Compass_AdvanceShippingNotice::postPost` interceptor.
+- Script: `worker/crons/toga2/compasscanada/workflow_beta/import_grand_and_toy_asn_from_file.php`.
+
+## Edge cases
+
+- **A line whose only ItemFulfillmentItem already carries item-level tracking makes `postPost`
+  return 500 `EO-1`** (not a validation error) — the reconcile path skips the IFI and the handler
+  then fails instead of completing. Deterministic; hit three times across both transports (PO
+  `50305097-1` line 4; Sentry `AV-30`, `69-68`). Skip and record those shipments; the open bug is
+  tracked on the [shared Compass ASN handler doc](../../compass-usa/features/asn-to-item-fulfillment.md).
+- **G&T `XXXX` carrier code** carries the literal `DIRECT` as its `TrackingID` — it is a placeholder,
+  not a real tracking number.
+- **One `TrackingID` even when `TotalPackages > 1`** — do not synthesize per-package numbers.
+
+## Change history
+- 2026-08-20 — Documented the two-route backfill after recovering the 67-day G&T outage (518
+  shipments / 536 ASNs / 512 TrackingNumbers / 406 ItemFulfillments / 332 Units). Recorded the cXML
+  replay source and the ~20s `maxAuthsPerHour` pacing, the three-bridge JSON payload requirement,
+  the tracking-number-exists dedupe guard, the `c_dtInTransitEmailSent` + 3-day email suppression,
+  and the `EO-1` already-tracked-line edge case. (bala)
+
+## Related docs
+- [Grand & Toy ASN Import](../features/grand-and-toy-asn-import.md) — the ongoing feed and its root
+  cause history.
+- [cXML ShipNotice Gateway](../../../2.0/apps/api2/features/cxml-shipnotice-gateway.md) — provisioning
+  and replay mechanics.
+</content>

package/knowledge/clients/compass-usa/features/asn-to-item-fulfillment.md

@@ -5,7 +5,7 @@ project: _Underscore
 client: compass-usa
 type: client-feature
 status: active
-updated: 2026-07-02
+updated: 2026-08-20
 owners: [jcardinal, bala]
 files:
   - _underscore/Model/Compass/AdvanceShippingNotice.php
@@ -20,6 +20,8 @@ files:
   - dbchanges2/Client_Compass/2026-06-18b - CleanupSA132881DuplicateTracking.sql
   - dbchanges2/Client_Compass/2026-07-02a - FixSA133377TrackingSerialAndDuplicateIF.sql
 related:
+  - ../../../2.0/apps/api2/features/cxml-shipnotice-gateway.md
+  - ../../compass-canada/features/grand-and-toy-asn-import.md
   - ../../../2.0/apps/_underscore/features/recursive-item-fulfillments.md
   - ../../../2.0/apps/_underscore/features/item-fulfillment-stage-lifecycle-and-order-status.md
   - ../../../2.0/apps/_underscore/features/model-save-parent-cascade-stored-field-deadlock.md
@@ -111,6 +113,29 @@ Compass Canada (`Model/Compass/Canada/`) is a separate sub-client. The Compass c
 (Core `ClientApiIdentities` identity `153531108`, clientId 2) holds roles 1 and 3.
 
 ## Gotchas / known issues
+- **⚠ OPEN BUG — an already-tracked, fully-fulfilled line makes `postPost` return 500 `EO-1`
+  (2026-08-20).** When an ASN item resolves to a SalesOrderItem whose **only**
+  `ItemFulfillmentItem` already carries item-level tracking, `remaining = 0` routes to the reconcile
+  path, `itemFulfillmentItemHasTracking()` (added 2026-06-18) skips that IFI, and the handler then
+  **fails** instead of completing cleanly — so the caller gets an operation error, not the intended
+  no-op, and the ASN is lost. Deterministic: reproduced three times across both the cXML and the JSON
+  transports (repro: PO `50305097-1` line 4; Sentry ids `AV-30`, `69-68`). Found on Compass Canada
+  but it lives in the **shared Compass parent**, so Office Depot re-shipments on already-tracked
+  lines hit it too. This recurs whenever a vendor sends a second package for a line that already has
+  tracking. Fix direction: the skip should complete the request successfully (nothing to attach), not
+  fall through to the failure path.
+- **The cXML API needs `MATCH_CREATE` overrides on all three ASN tracking bridges.** Compass USA's
+  cXML API (`apiId 4`) has `Client_Compass.Apis_RecordFields` rows overriding RecordFields
+  **1437/1441/1445** to `MATCH_CREATE`. Only 1437 defaults to `MATCH_CREATE` in Core, so without the
+  1441/1445 overrides the nested `trackingNumber` is looked up instead of created and every ship
+  notice fails EV-12 — exactly what happened to Compass Canada for 67 days. Do not delete or
+  "normalize" these rows. See
+  [cXML ShipNotice Gateway](../../../2.0/apps/api2/features/cxml-shipnotice-gateway.md).
+- **`findOrCreateShippingCarrier()` was writing other tenants' carriers into `Client_Compass`
+  (fixed 2026-08-20, not yet committed).** The cXML helper authenticated its carrier-create call with
+  hardcoded Compass USA credentials, so Compass **Canada** traffic minted carriers here: 15 `PRECI`,
+  5 `GANDT` and 3 `XXXX` junk `ShippingCarriers` rows in `Client_Compass` are **not** Compass USA
+  data and should be cleaned up.
 - **ASN arriving AFTER the SO is already fulfilled (fixed 2026-06-15):** when an SO is fulfilled
   out-of-band (NetSuite/toga2-supply creates `F`-numbered IFs) **before** its ASN arrives,
   `postPost` previously found zero unfulfilled ASN items and skipped the **entire** propagation
@@ -221,6 +246,15 @@ Compass Canada (`Model/Compass/Canada/`) is a separate sub-client. The Compass c
 
 ## Change history
 Dated one-liners, newest first.
+- 2026-08-20 — Recorded an **open, deterministic 500 `EO-1`** in `postPost`: an ASN item whose
+  SalesOrderItem has one already-tracked `ItemFulfillmentItem` takes the reconcile path, the IFI is
+  skipped by `itemFulfillmentItemHasTracking()`, and the handler then fails rather than completing —
+  so a vendor's second package on an already-tracked line errors instead of no-op'ing. Found while
+  fixing the Compass Canada G&T cXML feed (shared parent, so Office Depot is exposed too). Also
+  recorded that Compass USA's cXML API (`apiId 4`) relies on `Apis_RecordFields` `MATCH_CREATE`
+  overrides for RecordFields 1437/1441/1445 (Core defaults 1441/1445 to `MATCH`), and that the
+  cross-tenant `findOrCreateShippingCarrier()` defect left 23 junk `Client_Compass.ShippingCarriers`
+  rows created by Compass Canada traffic. (bala)
 - 2026-08-13 — Diagnosed the recurring MySQL 1213 deadlocks on `POST /v2/advance-shipping-notices`
   (Office Depot cXML feed): 100% were the `_total` UPDATE emitted by the `_Model::save()` parent
   cascade re-saving the PurchaseOrder on each ASN insert, deadlocking under concurrent same-PO ASN

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.620",
+  "version": "1.0.622",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",