toga-ai 1.0.842 → 1.0.844

package/knowledge/1.0/apps/worker/features/staples-cxml-order-import.md

@@ -6,8 +6,8 @@ project: Worker
 client: staples
 type: client-feature
 status: active
-updated: 2026-09-16
-owners: [jcardinal, mhammontree]
+updated: 2026-09-18
+owners: [jcardinal, mhammontree, bala]
 files:
   - worker/crons/sync/staples/sync_staples_cxml.php
 related:
@@ -15,99 +15,77 @@ related:
   - 1.0/apps/worker/features/netsuite-sales-order-sales-rep-sourcing.md
 ---
 
-Hourly cron importing Staples cXML POs from SFTP into NetSuite Sales Orders (writes 855 ack back); open for the per-file flow, saved-search-8412 Location resolution, new-account onboarding, and the silent add-failure trap.
+Hourly cron importing Staples cXML POs from SFTP into NetSuite Sales Orders (writes 855 ack back); open for the per-file flow, saved-search-8412 Location resolution, new-account onboarding, how to diagnose an order that never appeared, and the silent add-failure trap.
 
 ## Summary
 
-`sync_staples_cxml.php` is an **hourly** cron (runs at **:45**) that imports Staples cXML
-purchase orders from SFTP into NetSuite as Sales Orders, then writes an 855 acknowledgement
-back. Staples is NetSuite parent entity/customer **internalId 30057**.
+`sync_staples_cxml.php` is an hourly cron (`45 * * * *`, `schedules/cron.worker.sync.json`) that imports Staples cXML purchase orders from SFTP into NetSuite as Sales Orders, then writes an 855 acknowledgement back. Staples is NetSuite parent entity/customer **internalId 30057**.
 
 ## How it works
 
-Flow per run: log into SFTP `sftp.goagilant.com` as `staplessftp`, list `/in`, and for each
-cXML order file:
-
-1. Replace a known part-number escaping edge case in the raw file.
-2. Parse via `App_Xml`.
-3. Log the raw file to **FileLog** (schema `Logs`, table `FileLog`, job
-   `STAPLES_IMPORT_ORDER`). `App_Model_Logs_FileLog` sets
-   `_databaseNameOverride = 'Logs'`, so this writes to the legacy/V1 prod schema `Logs`.
-   The full inbound cXML is stored in `FileLog.fileData`.
-4. Dedupe against existing NetSuite sales orders by customer PO
-   (`App_NetSuite::getSalesOrdersByCustomerPo`).
-5. Build and add a NetSuite `SalesOrder` under parent customer internalId 30057.
-6. On successful add: write an **855 acknowledgement** cXML to SFTP `/out`, archive the
-   source file to `/archive/archive_in`, and delete it from `/in`.
+Per run: log into SFTP `sftp.goagilant.com` as `staplessftp`, list `/in`, and for each cXML order file:
+
+1. Replace a known part-number escaping edge case in the raw file; parse via `App_Xml`.
+2. Log the raw file to **FileLog** (schema `Logs`, job `STAPLES_IMPORT_ORDER`) — written **before** any NetSuite call. `App_Model_Logs_FileLog` sets `_databaseNameOverride = 'Logs'`, so this writes to the legacy/V1 prod schema `Logs`. The full inbound cXML is stored in `FileLog.fileData`.
+3. Dedupe against existing NetSuite sales orders by customer PO (`App_NetSuite::getSalesOrdersByCustomerPo`).
+4. Build and add a NetSuite `SalesOrder` under parent customer 30057.
+5. **On success only:** write the 855 acknowledgement cXML to `/out` as `855_<orderId>_<ts>`, archive the source file to `/archive/archive_in`, and delete it from `/in`.
+
+Step 5 running only on success is the key diagnostic lever: no 855 and no archived PO means the sales order was never created.
 
 ### Line-item Location resolution
 
-Each line-item's **Location** is resolved by matching the order's Staples `CustomerID`
-(from `<Extrinsic name="CustomerID">`) against NetSuite **saved search 8412**:
+Each line-item's **Location** comes from the order's `<Extrinsic name="CustomerID">` account code matched against NetSuite **saved search 8412** (`ns_runSavedSearch`, recordType `customer`):
 
 - match key: `custentity_external_customer_id`
 - location value: `custentity_customer_location`
 
-If no match is found, `$internalClientLocationId` stays **NULL**, the line-item Location is
-empty, and **NetSuite rejects the whole add** with `USER_ERROR` "Please enter value(s) for:
-Location".
+No match — **or a match whose Location is blank** — leaves `$internalClientLocationId` **NULL**, the line-item Location empty, and **NetSuite rejects the whole add** with `USER_ERROR` "Please enter value(s) for: Location". This is the most common cause of a Staples PO that never reaches NetSuite, and it produces no log line (see gotchas).
 
 ### Onboarding rule — new Staples buying account
 
-A Staples buying account is a **NetSuite child customer** of parent 30057. Before that
-account's first order arrives it must exist in NetSuite with:
+A Staples buying account is a NetSuite **child customer of 30057**. Set it up *before* its first order arrives, or every order for that account jams on SFTP `/in`, retrying hourly forever with no error surfaced:
+
+- **Parent** = 30057. A customer not under 30057 does not appear in saved search 8412.
+- **`custentity_external_customer_id`** = the account code from the cXML `CustomerID`.
+- **`custentity_customer_location`** = one Location. The field is a **multi-select** and must hold **exactly one** entry.
+- Staples Locations are children of Location **36** "New York : New York Warehouse : Staples". Each end customer normally gets its own child Location there; some reuse 36 directly.
+
+Worked example (2026-09-18): account code `0ZZZMICR`, customer **54606** "Staples, Inc. : Ingram Micro" (entity id 4679:7752) existed with the account code but a **blank** Location, so its orders failed every run. Creating Location **209** "Ingram Micro" (parent 36) and setting it on 54606 made the next run create SO **290453** for PO 955KVL, write `855_955KVL_1789744201` to `/out`, and archive the PO. Known-good reference row: customer 38626 Advance Auto, location 79.
+
+## Diagnosing an order that never appeared
 
-- **External Customer ID** = the account's Staples `CustomerID`, and
-- a valid **Customer Location**
+The account code lives only in the **file body**, never in the filename, so start from FileLog.
 
-so it appears in saved search 8412. If it is missing (or has no Customer Location), every
-order for that account fails the NetSuite add and the file sits on SFTP `/in` retrying
-hourly forever — **with no error surfaced** (see gotchas). Coordinate with the NetSuite
-business admins to create the customer before go-live for any new account.
+1. **Count imports** — legacy `Logs`.`FileLog`, job `STAPLES_IMPORT_ORDER`, filtered `fileData LIKE '%<accountCode>%'`. Repeated hourly rows of the **same `fileName`** = the file is stuck in `/in` and the NetSuite add is failing silently.
+2. **Check SFTP** — an `855_<orderId>_*` in `/out` or `/archive`, and the PO in `/archive/archive_in`, are written only on success. Absent = never created.
+3. **Check NetSuite by SuiteQL** — `SELECT ... FROM transaction WHERE otherrefnum = '<orderId>'`. The Staples orderID lands in `otherRefNum`, which **Global Search does not index**, so a UI search proves nothing.
+4. **Read the mapping** — run saved search 8412 and read that account code's row's **Location Internal ID**. Blank Location = confirmed root cause. No row at all = customer missing, or not a child of 30057.
+
+This also separates the two failure modes: FileLog is written before the NetSuite call, and a PHP warning **hard-exits** a 1.0 cron (`App_Error` → `exit`), so **few** rows for a run means it died early, while **one row per file** means it processed every file and the adds failed. Because the full cXML lives in `FileLog.fileData`, ship-to identity and `CustomerID` can be recovered from the legacy `Logs` DB without touching SFTP.
 
 ## Gotchas / known issues
 
-**Silent-swallow of failed NetSuite adds (durable warning).** The
-`if ($response->writeResponse->status->isSuccess)` block (~line 888) has **no `else`**. A
-failed add logs nothing, emails no one, and leaves the file on `/in`. A file only leaves
-`/in` on (a) add success — ack + archive + delete — or (b) the "already processed" dedupe
-branch — delete. A failed add never removes it, so a pure data gap (e.g. an un-set-up
-customer) is invisible and retries indefinitely. This is why a real backlog went unnoticed
-for ~6 days.
-
-**`getSavedSearch(8412)` is called once per order inside the file loop** — expensive
-(~20s/file). A future refactor should fetch/cache it once per run.
-
-**The `salesRep` assignment on this cron is dead code.** `$salesOrder->salesRep = 'Aaron Myall';`
-(~line 138) assigns a plain **string** to a field NetSuite types as a **RecordRef**; NetSuite
-silently discards it. Proven empirically: SO **286680** came out with **Paulina Szeliga** — the
-parent customer 30057's rep — not Aaron Myall. Because `entity` is always the parent, NetSuite
-sources the parent's rep and the custom `custbody_end_customer` never drives it. Separate-ticket
-candidate; left in place as of 2026-08-21. Also: the comment `//sales rep` above the
-`custbody_ctc_inside_sales_transaction` block is **wrong** — that block sets **Inside Sales
-Support** (hardcoded Nicholas Vance, internalId 497). Full mechanics in
-[NetSuite Sales Order Sales Rep Sourcing](netsuite-sales-order-sales-rep-sourcing.md).
+**Silent-swallow of failed NetSuite adds (durable warning).** The `if ($response->writeResponse->status->isSuccess)` block (~line 888) has **no `else`**. A failed add logs nothing, emails no one, and leaves the file on `/in`. A file leaves `/in` only on add success (ack + archive + delete) or on the "already processed" dedupe branch. Real cost: order **435KNN** (Ingram, May 2026) retried **674 times** between 13 May and 10 Jun, was never created, and the file was eventually deleted by hand — that order is lost. **955KVL** retried 28 times over 17–18 Sep. A temporary diagnostic logging the NetSuite rejection under FileLog job `STAPLES_ADD_FAIL_DIAG` (run 13 Aug 2026) captured the exact "Please enter value(s) for: Location" error, but that code is **not** in the committed cron.
 
-**Recommended hardening (NOT yet implemented):**
-- Add the missing `else` to email + log add failures (mirror the existing "missing items"
-  email) and leave the file for retry.
-- Pre-guard: if `$internalClientLocationId` is NULL, skip the doomed add and alert
-  "Staples customer `<CustomerID>` not set up in NetSuite", naming the exact id.
+**Dedupe does hold under resends.** Staples resent 955KVL twice while it was failing, leaving three copies in `/in`; when it finally succeeded only **one** sales order was created.
 
-### Diagnostic technique — early PHP-warning exit vs. silent add-failure
+**Duplicate `custentity_external_customer_id` across customers.** The cron keys its lookup array by the external customer id, so if two customers share a code the **last row returned by the saved search wins** — silently pointing orders at the wrong Location. Seen: `0ZZZMICR` on both the old standalone INGRAM MICRO (21608 — not a child of 30057, so not in search 8412, and therefore harmless) and the new sub-client 54606; `AIM71009` on two customers. Worth cleaning up.
 
-In Worker 1.0 a PHP warning/notice **hard-exits** the cron (`App_Error` → `exit`), so
-nothing after the failure point runs. This cron writes its **FileLog** row (job
-`STAPLES_IMPORT_ORDER`) **before** any NetSuite call. So counting fresh FileLog rows for a
-run distinguishes the two failure modes:
+**`getSavedSearch(8412)` is called once per order inside the file loop** — expensive (~20s/file). A future refactor should fetch/cache it once per run.
 
-- **few** rows → it died early (a PHP warning aborted the run partway), vs.
-- **one row per file** → it processed every file but the NetSuite adds failed.
+**The `salesRep` assignment on this cron is dead code.** `$salesOrder->salesRep = 'Aaron Myall';` (~line 138) assigns a plain **string** to a field NetSuite types as a **RecordRef**; NetSuite silently discards it. SO **286680** came out with **Paulina Szeliga**, parent 30057's rep, because `entity` is always the parent. Also, the `//sales rep` comment above the `custbody_ctc_inside_sales_transaction` block is **wrong** — that block sets **Inside Sales Support** (hardcoded Nicholas Vance, internalId 497). Full mechanics in [NetSuite Sales Order Sales Rep Sourcing](netsuite-sales-order-sales-rep-sourcing.md).
 
-Because the full inbound cXML lives in `FileLog.fileData`, ship-to identity and `CustomerID`
-can be recovered directly from the legacy `Logs` DB without touching SFTP.
+**Recommended hardening (NOT yet implemented):**
+- Add the missing `else` to log the NetSuite `statusDetail` and email add failures (mirror the existing "missing items" email), leaving the file for retry.
+- Pre-guard: if `$internalClientLocationId` is NULL, skip the doomed add and alert "Staples customer `<CustomerID>` not set up in NetSuite", naming the exact id.
 
 ## Related
 
 - [Staples profile](../../../clients/staples/profile.md)
 - [NetSuite Sales Order Sales Rep Sourcing](netsuite-sales-order-sales-rep-sourcing.md)
+
+## Change history
+- 2026-09-18 — Production debug of account `0ZZZMICR` (Ingram Micro) confirmed the blank `custentity_customer_location` root cause end to end. Added the diagnostic steps (FileLog `fileData LIKE`, success-only 855/archive, SuiteQL on `otherrefnum`, saved search 8412), the multi-select single-entry + parent-Location-36 setup rule, the duplicate external-customer-id failure mode, and real retry counts (435KNN x674, order lost; 955KVL x28, dedupe held). No code changed. (bala)
+- 2026-08-21 — Recorded that `$salesOrder->salesRep = 'Aaron Myall'` is dead code and that the adjacent `//sales rep` comment actually describes Inside Sales Support (Nicholas Vance, 497). Removed a departing employee from the sibling cron's recipient list. No behavior change. (mhammontree)
+- 2026-08-13 — Documented feature (first KB entry). Diagnosed a ~6-day, ~30-order backlog stuck on `/in`: new child customer INTEGRA PARTNERS INC. (CustomerID 03716185) was not set up, so no saved-search-8412 match → NULL Location. 28 of 30 files were that account; 2 were CustomerID 05611619 (existed, missing Customer Location). Business admins created/fixed the customers; re-running drained the backlog. (jcardinal)

package/knowledge/2.0/apps/_underscore/features/calculated-column-filter-options.md

@@ -16,6 +16,8 @@ related:
   - sales-order-status-filter-surface.md
   - tableview-joins.md
   - ../../toga-blox/features/table.md
+  - ../../api2/features/environment-variable-drives-underscore-branch.md
+  - ../../toga25-supply/features/persisted-query-cache.md
   - ../../api2/features/tableview-field-metadata.md
 ---
 
@@ -72,23 +74,44 @@ It is a **no-op for every existing column**: nothing else declares a `<field>Fil
 - Verified locally: 13 options for `Client_Compass`, 5 for `Client_CompassCanada`. Compass Canada
   also has a role `Agilant - Administrators`, correctly excluded by the exact `name = 'Admin'` match.
 
-**Internal TOGA staff are excluded (2026-09-18).** Admin-role holders on `@togatech.com`
-(Dev Team, internal employees) were showing up as assignable to Compass users. Added
-`const INTERNAL_EMAIL_DOMAIN = 'togatech.com'` plus a WHERE condition. The condition is
-**NULL-safe on purpose**:
+**Two exclusion rules, both applied (2026-09-18).** Admin-role holders were showing up as
+assignable to Compass users when they should not be:
+
+1. **Internal TOGA staff, by domain** — `const INTERNAL_EMAIL_DOMAIN = 'togatech.com'`.
+2. **Named individuals on a CLIENT domain** — `const EXCLUDED_ADMIN_EMAILS = ['vanessa.burks@compass-usa.com'];`
+   built into a `NOT IN (...)`. The domain rule alone could never catch her — her email is a
+   `@compass-usa.com` address. Assume any hide list will eventually need both.
+
+Both conditions are **NULL-safe on purpose**:
 
 ```sql
 (Users.email IS NULL OR Users.email NOT LIKE '%@togatech.com')
+(Users.email IS NULL OR Users.email NOT IN ('...'))
 ```
 
-A bare `NOT LIKE` yields NULL for a NULL email and would silently drop those admins.
-
-Two alternatives were considered and rejected:
-1. Removing the `Admin` role from those users in `Users_Roles` — the team will not strip admin
-   roles from real accounts.
-2. A `c_isHiddenFromAdminFilter` custom column on `Client_Compass.Users` (the `c_isVip`
-   precedent), making a hide a plain UPDATE with no deploy. Rejected for the simpler hardcoded
-   domain rule — but this is the natural next step if per-user control is ever needed.
+A bare `NOT LIKE` / `NOT IN` yields NULL for a NULL email and would silently drop those admins.
+Each list value is escaped with `_Database::escape()`, and the `NOT IN` is **skipped entirely when
+the array is empty** — an empty list builds `NOT IN ()`, a MySQL syntax error.
+
+**Why a constant and not a `c_isHiddenFromAdminFilter` column on `Client_Compass.Users`** (proposed
+twice, rejected twice — do not re-propose): these same people **must still appear in other admin
+lists** in the product. A `c_` flag reads as a property of the person ("hidden from admin lists"),
+so the next developer reuses it for a different list and hides them there too. A constant inside
+`_assignedAdminFilterOptions()` cannot leak — the exclusion stays scoped to this one dropdown.
+Accepted trade: adding a name needs a deploy. Rationale is repeated in the method docblock.
+The other rejected option: stripping the `Admin` role in `Users_Roles` — the team will not strip
+admin roles from real accounts.
+
+**ACL cannot do this**, for two independent reasons. Row-level ACL does exist in 2.0
+(`Core.AclRecordExpressions.sqlExpression`, wired to a role via `AclLogicGroups` /
+`AclLogicGroupExpressions`), so the instinct is reasonable, but:
+1. `_assignedAdminFilterOptions()` runs a raw `new _Query(...)` straight against `DB_CLIENT` and
+   never passes through the `_Model` / V2 ACL layer, so no ACL row changes its result;
+2. even if it did, an `AclRecordExpression` filters **every** read of that record for that role —
+   hiding the person from user lists and lookups everywhere, far wider than one dropdown.
+
+**General rule: a filter-options static that builds its own `_Query` is outside ACL by
+construction.** Whatever it must exclude, it excludes in its own SQL.
 
 ### Why a model static and not a config column
 
@@ -119,6 +142,15 @@ generic version at roughly **4–5** such columns, driven by real examples.
 - **The dropdown is backend-driven — do not look in toga25-supply.** The front end only renders
   what `_assignedAdminFilterOptions()` returns. Any change to which admins appear goes in
   `_underscore/Model/Compass/SalesOrder.php`.
+- **⚠ "My change had no effect" is usually stale cache or an undeployed branch, not this code.**
+  Work the ladder cheapest-first: clear the `supply-chain-query-cache` localStorage key (stale table
+  meta is served with **nothing in the network tab**), then read the meta request host in DevTools to
+  learn which api2 tier answered, then check branch/deploy state — see
+  [ENVIRONMENT drives the _underscore branch](../../api2/features/environment-variable-drives-underscore-branch.md).
+  The client-model class chain is **not** a likely cause: `_Model_Client_TableView::meta()` swaps
+  `_Model_Client_` → `_Model_<clientIdentifier>_` (`TableView.php` ~L266), and
+  `_Model_Compass_Usa_SalesOrder` is an empty subclass of `_Model_Compass_SalesOrder`, so the
+  override is inherited. Rule it out in one minute.
 - **Key on the FIELD name, not the column slug.** The slug is `assigned-admin` (hyphen); the model
   field is `_assignedAdmin`. The seam uses the field name.
 - **Type `STRING` (not STATUS/SELECT) is what lets the branch fire** — the existing `switch` in

package/knowledge/2.0/apps/api2/features/environment-variable-drives-underscore-branch.md

@@ -6,7 +6,7 @@ project: API
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
+updated: 2026-09-18
 owners: ["bala", "mhammontree", "apeterson", "jcardinal", "tcox"]
 files:
   - api2/.ebextensions/git.php
@@ -90,6 +90,12 @@ So on `api.beta.togahub.com` and `api.dev.sandbox.togahub.com` the code fataled
 
 ## Gotchas
 
+- **⚠ Merged locally ≠ live. A `_underscore` change reaches a tier only when the branch is PUSHED *and* api2 has been REDEPLOYED since.** The clone happens in the api2 prebuild hook, so an unpushed (or un-redeployed) branch leaves the tier running the old framework while your local checkout and `git log` both look correct. Seen 2026-09-18: a model change was committed and merged locally but never pushed; `_sandbox-client` served the old code.
+  - Diagnosis ladder, cheapest first, when "my `_underscore` change did nothing":
+    1. Delete the `supply-chain-query-cache` localStorage key. toga25-supply persists the React Query cache, so stale table meta is served with **nothing in the network tab** — which reads as a backend fault. See [persisted query cache](../../toga25-supply/features/persisted-query-cache.md).
+    2. Read the meta request **host** in DevTools to learn which tier actually answered (`api.client.sandbox.togahub.com` = the `_sandbox-client` tier).
+    3. Only then suspect branch/deploy state — `git ls-remote origin <branch>` (is it pushed?) then the pipeline source action.
+
 - **⚠ Cross-repo change ordering: land `_underscore` BEFORE deploying api2.** Because api2's EB deploy **clones `_underscore` (branch `_<ENVIRONMENT>`, e.g. `_production`) at BUILD time** (`.platform/hooks/prebuild/git.sh`), a change spanning both repos must have its `_underscore` side merged to `_<ENVIRONMENT>` **first** — otherwise the freshly-built api2 calls a framework method/class not on the box and fatals. Landing `_underscore` first is **safe on its own**: the old api2 controller doesn't use the new framework code yet. Order: merge `_underscore` → (verify it's on the branch) → deploy api2.
   - **🚨 Worked instance — `client-sandbox` was 100% DOWN on this exact violation (2026-08-31, since fixed).** Every request returned 500 `Call to undefined method _Model_Core_AclFieldPermission::resolveFieldPermissions()`. `api2@_sandbox-client` (`Component/Api/V2/V2.php:6198`, commit `0cc37fe` "Delegate ACL field perms to shared resolver", jcardinal 2026-08-14) delegates to a resolver whose `_underscore` half (`a5e79448` "Surface field binding and ACL field resolver", same author/date) exists **only on `_qa-alpha` and `toga25-desk`** — never merged to **`_underscore@_sandbox-client`**. Resolved the same day (branches matched + redeployed). **Why a half-shipped ACL refactor is a TOTAL outage:** field-ACL resolution runs on essentially every authenticated GET, so there is no degraded mode — the whole tier 500s and every unrelated feature under test looks broken. When a whole sandbox goes dark, check the two-repo pairing before debugging your own feature.
 - **⚠ A "Could not find required file for `_Model_...`" fatal on prod can be a STALE-BUILD artifact, not a missing file.** `Logs.Issue #476` ("Could not find required file for '_Model_Client_ItemFulfillments_InventoryAdjustment'") fired because that model file **was** committed to `_production` but `api-production-1` was running an **older build**; a **redeploy** (re-pulls `_production`) resolved it. Before assuming a model file is absent, check the **running build's `_underscore` against `_production`** (file presence on the box; `git merge-base --is-ancestor`).

package/knowledge/2.0/apps/worker2/INDEX.md

@@ -47,6 +47,7 @@
 | [QA/QC Branch Automation (auto-mirror + rebuild-from-ledger revert)](features/qa-qc-branch-automation.md) | worker2-centralized GitHub branch automation (mirror, ledger-revert, the single guarded door to `_production`); open when touching QA/QC branch merges or revert |
 | [QA/QC Review Pipeline (ClickUp review chain, QC batch + defect/client-review)](features/qa-qc-review-pipeline.md) | ClickUp-driven QA/QC review + promotion chain (Dev→QA→QC→Client Review→Stage→Prod) for worker2; open when touching a review gate, the QC/Stage crons, or the Fea |
 | [S3 folder retention cleanup worker action (CleanS3Folder)](features/s3-folder-retention-cleanup.md) | Cron-parameterized worker action that deletes S3 objects under one folder older than a retention window; open when adding/tuning an S3 retention sweep or its de |
+| [The two sandbox-dev worker2 environments (and how `_underscore` reaches a worker box)](features/sandbox-dev-worker-environments.md) | Two Elastic Beanstalk environments both call themselves "sandbox-dev" worker, they run **different `ENVIRONMENT` values**, they clone **different `_underscore` |
 | [Service Request → Sales Order → Purchase Order generation (Sync/ServiceRequest)](features/service-request-sales-order-generation.md) | Worker `_Worker_Sync_ServiceRequest` that turns a Service Request → Sales Order → one PO per vendor for any tenant; open for the SR→SO→PO chain, its interceptor |
 | [SSO Stability Monitor (Monitor/Operations/SsoStability)](features/sso-stability-monitor.md) | Reporter cron that GETs the saml `/healthcheck` and pushes an SSO heartbeat to a OneUptime monitor; open when working on the SSO uptime probe, its tokens, or it |
 | [Startech Webhook Handler (worker2)](features/startech-webhook-handler.md) | Inbound Startech (Easeedesk) webhook handler that creates/updates the matching TOGA 2.0 ticket; open for the webhook flow, its payload, or ticket-type/stage res |

package/knowledge/2.0/apps/worker2/features/sandbox-dev-worker-environments.md

@@ -0,0 +1,101 @@
+---
+title: "The two sandbox-dev worker2 environments (and how `_underscore` reaches a worker box)"
+framework: "2.0"
+repo: worker2
+project: Worker
+client: shared
+type: feature
+status: active
+updated: 2026-09-18
+owners: ["bala"]
+files:
+  - worker2/.ebextensions/git.php
+  - worker2/.ebextensions/git.json
+  - worker2/.ebextensions/php_include_underscore.config
+  - worker2/Config/beta.ini
+  - worker2/Config/sandbox-dev.ini
+  - _underscore/Worker.php
+related:
+  - ../architecture.md
+  - ./creating-worker-actions.md
+  - ../../api2/features/environment-variable-drives-underscore-branch.md
+  - ../../api2/workflows/codepipeline-codeconnections-deploy.md
+---
+
+## Summary
+
+Two Elastic Beanstalk environments both call themselves "sandbox-dev" worker, they run **different
+`ENVIRONMENT` values**, they clone **different `_underscore` branches**, and **only one of them is
+deployed by CodePipeline** — yet both write the same database. Testing on one and assuming the other
+behaves the same wastes a deploy cycle and hides stale code.
+
+| EB environment | Tier | `ENVIRONMENT` | `_underscore` branch | Deployed by |
+|---|---|---|---|---|
+| `worker-invoke-sandbox-dev` | **Worker** (SQS) | `sandbox-dev` | `_sandbox-dev` | **by hand only** |
+| `worker-sandbox-dev` | **WebServer** | `beta` | `_beta` | CodePipeline |
+
+- `worker.beta.togahub.com` resolves to the same IPs as
+  `worker-sandbox-dev.us-east-1.elasticbeanstalk.com`, so an HTTP invoke tests the **WebServer** box.
+- **The cron runs on `worker-invoke-sandbox-dev`.** CodePipeline never touches it, so it keeps
+  running old code until someone deploys it manually.
+- `worker2/Config/beta.ini` and `Config/sandbox-dev.ini` both point at
+  `sandbox-dev.cluster-cb860gg2qw2n.us-east-1.rds.amazonaws.com`, so **finding your row in the
+  database proves nothing about which box wrote it.**
+
+## `_underscore` is NOT in the deployment zip — it is cloned at deploy time
+
+`worker2/.ebextensions/git.php` clones `_underscore` on the instance from branch
+**`'_' . strtolower(ENVIRONMENT)`**, into the path named in `.ebextensions/git.json`, and
+`php_include_underscore.config` puts that path on `include_path`. This is the same mechanism api2
+uses — see
+[ENVIRONMENT drives the `_underscore` branch](../../api2/features/environment-variable-drives-underscore-branch.md)
+for the full rules, the `branch`-pin trap and the committed-PAT warning. Do not restate them here.
+
+**The practical consequence for worker2:**
+
+- An **`_underscore`-only** change needs only a **redeploy of the same worker2 application
+  version** — the clone re-runs and picks up the branch tip.
+- A **worker2** change needs a **fresh pipeline build**; redeploying the existing version ships the
+  old PHP.
+
+Getting that backwards costs a full deploy cycle for a change that was never going to appear.
+
+## `_Worker::runTask` does not always use SQS — check `debug_mode` on the PRODUCER
+
+`_underscore/Worker.php` branches on `_Environment::isDebugMode()`: true → a synchronous POST to
+`_Config::api('worker')`; false → an SQS send using `_Config::cloud('aws_worker_queue_url')`. The
+general rule and its read-after-write hazard live in
+[creating worker actions](./creating-worker-actions.md).
+
+The concrete config that surprises people:
+
+- **`api2/Config/sandbox-dev.ini` sets `debug_mode = 1` and defines no `aws_worker_queue_url` at
+  all.** So a job queued from api2 on that environment travels over **HTTP to
+  `worker.beta.togahub.com`** — the WebServer box — and never enters SQS.
+- **`api2/Config/production.ini` sets `debug_mode = 0`** and does define the queue keys, so
+  production really is SQS.
+
+Before you go looking for a missing SQS message on a sandbox, confirm the producer's `debug_mode`.
+
+## Gotchas / known issues
+
+- **Staleness is per-instance.** Take the failing job's `Core.WorkerJobs.instanceId` and verify
+  *that* box received the deploy. "I deployed beta" is not a verification — see the
+  [CodePipeline deploy workflow](../../api2/workflows/codepipeline-codeconnections-deploy.md).
+- **Never infer the environment from the EB environment name.** `worker-sandbox-dev` is
+  `ENVIRONMENT=beta`; the name and the value disagree on purpose-built tiers more often than not.
+- **The clone deletes `.git`**, so the instance carries no commit id. Verify by file presence.
+
+## Change history
+
+- 2026-09-18 — Initial capture, from the NYCHH transfer-order-email work. Recorded that
+  `worker-invoke-sandbox-dev` (Worker tier, `ENVIRONMENT=sandbox-dev`, branch `_sandbox-dev`) runs
+  the crons and is **deployed by hand only**, while `worker-sandbox-dev` (WebServer tier,
+  `ENVIRONMENT=beta`, branch `_beta`) is the CodePipeline target behind
+  `worker.beta.togahub.com` — and that both `Config/beta.ini` and `Config/sandbox-dev.ini` point at
+  the same `sandbox-dev` cluster, so the data gives no clue which box ran. Recorded that
+  `_underscore` is cloned by `.ebextensions/git.php` at deploy time rather than zipped, so a
+  framework-only change needs a redeploy of the same version while a worker2 change needs a fresh
+  build. Recorded that `api2/Config/sandbox-dev.ini` sets `debug_mode = 1` with **no**
+  `aws_worker_queue_url`, so `_Worker::runTask` there POSTs synchronously to
+  `worker.beta.togahub.com` instead of using SQS. (bala)

package/knowledge/INDEX.md

@@ -19,8 +19,8 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 
 ## 2.0 framework
 
-- **_underscore** (_Underscore) _(framework core)_ — 88 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
-- **worker2** (Worker) — 69 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
+- **_underscore** (_Underscore) _(framework core)_ — 87 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
+- **worker2** (Worker) — 71 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
 - **api2** (API) — 26 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
 - **dbchanges2** (Database Changes) _(framework core)_ — 19 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
 - **toga2-supply** (TOGa Supply) — 9 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)

package/knowledge/clients/nychh/features/transfer-order-notification-emails.md

@@ -1,18 +1,17 @@
 ---
 title: "NYCHH transfer-order notification emails (Submitted / Packed / Shipped)"
 framework: "2.0"
-repo: _underscore
-project: _Underscore
+repo: worker2
+project: Worker
 client: nychh
 type: client-feature
 status: active
-updated: 2026-09-17
+updated: 2026-09-18
 owners: ["bala"]
 files:
-  - _underscore/Trait/Nychh/TransferOrderEmail.php
+  - worker2/Worker/Client/Nychh/TransferOrderEmails.php
   - _underscore/Model/Nychh/TransferOrder.php
   - _underscore/Model/Nychh/ItemFulfillment.php
-  - worker2/Worker/Client/Nychh/TransferOrderEmails.php
   - dbchanges2/Client_Nychh/2026-09-15a - TransferOrderEmails.sql
   - dbchanges2/Client_Nychh/2026-09-15b - TransferOrderEmailGoLiveRecipient.sql
   - dbchanges2/Core/2026-09-16a - NychhTransferOrderEmailsCron.sql
@@ -22,8 +21,10 @@ related:
   - ../../../2.0/apps/_underscore/features/email-template-sending.md
   - ../../../2.0/apps/_underscore/features/email-send-pipeline.md
   - ../../../2.0/apps/_underscore/features/autoincrement-record-numbering.md
+  - ../../../2.0/apps/_underscore/features/database-alias-repointing.md
   - ../../../2.0/apps/api2/features/api-payload-interceptors.md
   - ../../../2.0/apps/worker2/features/creating-worker-actions.md
+  - ../../../2.0/apps/worker2/features/sandbox-dev-worker-environments.md
   - ../profile.md
 ---
 
@@ -34,35 +35,44 @@ NYC Health + Hospitals (H+H) gets three notification emails for a transfer order
 All three send from `HealthcareteamNYCHHC@togatech.com` through the normal
 [client email-template](../../../2.0/apps/_underscore/features/email-template-sending.md) path.
 
-Two things carry most of the risk and are the reason this doc exists:
+**The whole feature is ONE worker2 file.** `worker2/Worker/Client/Nychh/TransferOrderEmails.php`
+holds the two queued entry points, the backstop cron, every query and the HTML builder. The
+`_underscore` per-client models only declare the `c_` marker fields and queue the job — they build
+nothing. That is deliberate: app-level over core, and a slow send must not sit inside an api2 save.
+
+Three rules carry most of the risk:
 
 1. **Submitted fires on `postPut`, not `postPost`** — on the POST the order still holds the
    `TA…` autoincrement placeholder, not the NetSuite number.
 2. **Only orders raised in TOGa Supply may be announced.** `TransferOrders.createdByUserId` is the
    discriminator; without it H+H would be mailed about every order the NetSuite sync ever imported.
+3. **A failed send throws — it is never swallowed**, so nothing is ever marked sent that did not go.
 
 Recipient is `DevTeam@togatech.com` while testing. `2026-09-15b` swaps it to
 `EndUserDeviceLifecycle@nychhc.org` — run that file only after sign-off.
 
 ## Key files / entry points
 
-- **`_underscore/Trait/Nychh/TransferOrderEmail.php`** — all the logic: guards, the SQL reads, the
-  HTML builders, the escaping, the size budget, and the send. Both models and the worker2 cron call
-  into it, so there is exactly one implementation.
-  - `isTransferOrderReadyToAnnounce()` / `readTransferOrderForEmail()` — eligibility + data read.
-  - `sendTransferOrderEmail(): bool` — sends and **returns whether it really sent**.
-  - `buildTrackingSectionHtml()`, `buildAssetTagTableHtml()`, `escapeForEmailHtml()`,
-    `maximumEmailBodyBytes()`.
-- **`_underscore/Model/Nychh/TransferOrder.php`** — `postPut` hook → Submitted; declares
-  `c_dtTransferOrderSubmittedEmailSent`.
-- **`_underscore/Model/Nychh/ItemFulfillment.php`** — `postPost` / `postPut` → Packed / Shipped;
-  declares `c_transferOrderEmailStageSent` + `c_dtTransferOrderEmailSent`.
-- **`worker2/Worker/Client/Nychh/TransferOrderEmails.php`** — action
-  `Client/Nychh/TransferOrderEmails/SendPending`, the backstop cron (`*/5`).
+- **`worker2/Worker/Client/Nychh/TransferOrderEmails.php`** — the entire feature.
+  - `SendSubmitted(string $transferOrderUuid)` — queued from the TransferOrder `postPut`.
+  - `SendFulfillment(string $itemFulfillmentUuid)` — queued from the ItemFulfillment hooks.
+  - `SendPending(bool $dryRun = true)` — the `*/5` backstop cron.
+  - `initialize()` registers the client DB before any query runs.
+  - Builders: `buildItemRowsHtml()`, `buildTrackingSectionHtml()`, `buildAssetTagTableHtml()`,
+    `escapeForEmailHtml()`; budget constant `MAXIMUM_EMAIL_BODY_BYTES`.
+- **`_underscore/Model/Nychh/TransferOrder.php`** — declares
+  `c_dtTransferOrderSubmittedEmailSent`; `postPut` calls `_Worker::runTask` with action
+  `Client/Nychh/TransferOrderEmails/SendSubmitted` and `{"transferOrderUuid": "…"}`.
+- **`_underscore/Model/Nychh/ItemFulfillment.php`** — declares `c_transferOrderEmailStageSent` +
+  `c_dtTransferOrderEmailSent`; `postPost` / `postPut` call action
+  `Client/Nychh/TransferOrderEmails/SendFulfillment` with `{"itemFulfillmentUuid": "…"}`.
+- **Template uuids** (rows in `Client_Nychh.EmailTemplates`, seeded by `2026-09-15a`):
+  - submitted `0f4a6c21-9d3e-4b87-a15c-6e2b8f70d914`
+  - packed `7b1d9e54-3c62-48af-b0d7-25f4a8c31e6b`
+  - shipped `c58f3a07-6b94-4d21-8e3f-914ad7b06c52`
 - **`nychh_email_templates/*.html`** — ⚠ the HTML source of the three templates. **These files are
   in no git repo** — they live only in the working folder. The dbchanges2 seed SQL is generated
-  *from* them, so right now the `EmailTemplates` rows in `2026-09-15a` are the only durable copy.
-  Get them into a repo before anyone else has to edit a template.
+  *from* them, so the `EmailTemplates` rows in `2026-09-15a` are the only durable copy.
 
 ## ⚠ Submitted must fire on `postPut` — the POST number is a placeholder
 
@@ -93,20 +103,25 @@ Measured in production: **all 1,936 sync orders have `createdByUserId` NULL and
 have it set.** That test is the only thing standing between H+H and a mailshot covering every
 historic order the importer has pulled in.
 
-The test appears in **four** places and they must stay in step: the trait's
-`isTransferOrderReadyToAnnounce()` and `readTransferOrderForEmail()`, and both worker2 cron queries.
+`TransferOrders.createdByUserId IS NOT NULL` appears in **four** queries in the worker file and
+they must stay in step: the Submitted guard, the order read, and both `SendPending` sweeps.
 
 ## How it works
 
-1. **Submitted** — `_Model_Nychh_TransferOrder::postPut` calls the trait. If the order is
-   supply-raised and `c_dtTransferOrderSubmittedEmailSent` is empty, it builds the body, sends, then
-   stamps the marker.
-2. **Packed / Shipped** — `_Model_Nychh_ItemFulfillment::postPost` / `postPut` call the trait. The
-   stage reached is recorded on `c_transferOrderEmailStageSent`, with `c_dtTransferOrderEmailSent`
-   as the timestamp, so a fulfillment cannot re-announce the same stage.
+1. **Submitted** — `_Model_Nychh_TransferOrder::postPut` queues `SendSubmitted` with the order
+   uuid. The worker re-checks the supply-order test and the empty
+   `c_dtTransferOrderSubmittedEmailSent` marker, sends, then stamps the marker.
+2. **Packed / Shipped** — the ItemFulfillment `postPost` / `postPut` queue `SendFulfillment` with
+   the fulfillment uuid. The stage reached is recorded on `c_transferOrderEmailStageSent`, with
+   `c_dtTransferOrderEmailSent` as the timestamp, so a fulfillment cannot re-announce a stage.
 3. The **backstop cron** sweeps anything the hooks missed (below).
-4. The trait renders the stored `EmailTemplates` body, fills `{orderItems}` and `{trackingSection}`
-   itself, and sends through `_Model_Client_EmailTemplate`.
+4. All three paths share the same markers, so nothing sends twice.
+5. The worker renders the stored `EmailTemplates` body, fills `{orderItems}` and
+   `{trackingSection}` itself, and sends through `_Model_Client_EmailTemplate`.
+
+Queuing rather than sending inline also means the api2 save returns immediately. See
+[creating worker actions](../../../2.0/apps/worker2/features/creating-worker-actions.md) for how the
+string-keyed parameters arrive as PHP named arguments and why `initialize()` runs first.
 
 ### The backstop cron — and why it is not optional
 
@@ -119,8 +134,8 @@ PUT follows.** Fulfillment **2704** in production: header row at 20:33:09, lines
 
 - Its fulfillment query requires **`EXISTS` on `ItemFulfillmentItems`**, so a header row written
   seconds ago with no lines yet is left for the next run instead of being mailed empty.
-- It also picks up any send that failed.
-- Caps at **25 of each kind per run**.
+- It also picks up any send that threw (the marker was never written).
+- Caps at **25 of each kind per run** (`MAXIMUM_EMAILS_PER_RUN`).
 
 ## Schema, registration and deploy order
 
@@ -138,22 +153,26 @@ A custom field needs **all three** pieces, matching how `Model/Client/TrackingNu
 the column, the `CustomRecordFields` row, the `AclCustomFieldPermissions` grant for **Base and
 API** — plus a public property on the per-client model (`FIELD_DATETIME` / `FIELD_CHAR`).
 
-- **⚠ Run `2026-09-15a` BEFORE the `_underscore` deploy.** The models now declare the three
+- **⚠ Run `2026-09-15a` BEFORE the `_underscore` deploy.** The models declare the three
   properties, so if the columns are missing every NYCHH transfer-order read breaks.
+- **⚠ The two repos reach a box by different routes.** An `_underscore`-only change needs only a
+  **redeploy of the same worker2 application version**; a change to `TransferOrderEmails.php` needs
+  a **fresh pipeline build**. See
+  [worker2 sandbox-dev environments](../../../2.0/apps/worker2/features/sandbox-dev-worker-environments.md).
 - **⚠ Cloning prod `Client_Nychh` into a sandbox WIPES this feature.** Prod does not have any of it
   yet, so a clone removes the three marker columns, the three `EmailTemplates` rows, the recipient
   rows, the custom-field rows and the PUT interceptor. **Re-run `2026-09-15a` after any clone.**
 
-## The item block lives in the trait, not in the `EmailTemplates` row
+## The item block lives in the worker, not in the `EmailTemplates` row
 
 `_Model_Client_EmailTemplate::replaceTemplateVariables()` is **one `str_replace` per key, with no
 loop, and it skips non-scalar values outright**. A list whose length varies per order therefore
-cannot live in the stored template. `{orderItems}` is a placeholder the trait fills with built HTML
-— the same split Compass already uses for its order items.
+cannot live in the stored template. `{orderItems}` is a placeholder the worker fills with built
+HTML.
 
 Cost of the split, stated so nobody is surprised later:
 
-- a change to the **row markup** needs an `_underscore` deploy;
+- a change to the **row markup** needs a worker2 deploy;
 - a change to the **chrome** only needs the SQL re-run;
 - the SQL is **generated from the `nychh_email_templates/*.html` sources**, so it must be
   regenerated after every template edit.
@@ -163,25 +182,34 @@ Cost of the split, stated so nobody is surprised later:
 - **Escape every scalar.** `replaceTemplateVariables()` does no escaping and the Send worker forces
   HTML mode, so a site name, contact name, address or PO number containing `<` landed as live
   markup. All **seven** scalar values go through `escapeForEmailHtml()`. `orderItems` and
-  `trackingSection` stay raw — they are HTML the trait built and already escaped.
-- **Tracking is built per order, not templated.** In production **1,484 of 2,479** shipped
-  transfer-order fulfillments carry **no tracking number at all**, and **322** of the rest have no
-  carrier on the number. As three flat placeholders those cases rendered `Tracking:` beside an empty
-  `img src` and a *Track Shipment* button pointing nowhere. `buildTrackingSectionHtml()` now drops
-  the whole block when there is no number, and drops the button when there is no URL. Placeholders
-  changed from `{trackingNumber}` `{trackingUrl}` `{trackingCarrierLogo}` to
+  `trackingSection` stay raw — they are HTML the worker built and already escaped.
+- **Tracking is per PACKAGE, not per fulfillment.** `readTrackingForEmail()` returns **every** row;
+  the section prints one *Tracking* line per package and the *Track Shipment* button uses the first
+  link. It had `LIMIT 1` and returned a single row, so a multi-package shipment showed the customer
+  only one number. Measured 2026-09-18: of **1,495** shipped fulfillments carrying tracking,
+  **1,203** have one number and **292** have two or more, **up to 25**.
+- **Many shipped fulfillments have no tracking at all** — 1,484 of 2,479 measured 2026-09-17, and
+  322 of the rest have no carrier on the number. `buildTrackingSectionHtml()` drops the whole block
+  when there is no number and drops the button when there is no URL. Placeholders are
   **`{trackingIntroSuffix}`** and **`{trackingSection}`**.
-- **Gmail clips a message past about 102 KB**, and one fulfillment line can carry hundreds of units
-  (fulfillment **2287** in prod has 300). `maximumEmailBodyBytes()` sets a **92,000-byte** budget,
-  measured against the items HTML as it is built; the remainder is summarised as *"and N more"*
-  rather than listed. Declaring the type once on the serial table and letting cells inherit cut a
-  row from about 340 bytes to about 140. Measured worst case: **94,864 bytes with 502 units
-  listed**, under the 102,400 limit.
+- **Gmail clips a message past about 102,400 bytes.** `MAXIMUM_EMAIL_BODY_BYTES = 80000` budgets the
+  **item block only**; the remainder is summarised as *"and N more"*. 80,000 (not the earlier
+  92,000) because the stored shipped template is **11,695 bytes** on its own and a 25-package
+  tracking block adds about **9,500** more. Declaring the type once on the serial table and letting
+  cells inherit cut a row from about 340 bytes to about 140.
 - **Item titles need a fallback chain and a trim.** NYCHH `Items` rows often have an empty `title`,
   so the read uses
   `COALESCE(NULLIF(Items.title,''), NULLIF(Items.description,''), Items.partNumber)`. Titles and
   part numbers also carry trailing whitespace (item `210-BPPS` is stored with three trailing tabs),
   so both are `trim()`ed before rendering.
+- **Item image fallback is the NYC H+H mark** (`FALLBACK_ITEM_IMAGE_URL`, the H+H logo SVG on
+  `toga-public`), not the Compass no-product-image placeholder it was inheriting.
+- **The image cell is `width="50"`, matching the 50px image.** It was `width="70"` with a 20px right
+  padding, so the gap rendered roughly double the design — the padding is meant to be the whole gap.
+- **Fonts: `'Plus Jakarta Sans'` everywhere EXCEPT the item block.** Only the item title, P/N, Qty
+  and the asset-tag table use **Inter** (`ITEM_FONT_STACK`). The message paragraphs in all three
+  templates and the tracking line are Plus Jakarta Sans. Outlook gets Arial inside an
+  `<!--[if mso]-->` block because it cannot load web fonts.
 - **Detail box:** *Site Contact* falls back to the person who raised the order — 3 of the 6 supply
   orders in prod have no contact, and that is the same fallback the greeting already used. And
   `CONCAT_WS` skips NULL but **not** an empty string, so a blank address line 2 produced
@@ -189,17 +217,20 @@ Cost of the split, stated so nobody is surprised later:
 
 ## Gotchas / known issues
 
-- **⚠ `_Database::register()` takes the REAL database name first and the ALIAS last.** The cron
-  originally passed `_underscore::DB_CLIENT` (`'Client'`) as `$database` and `'Client_Nychh'` as
-  `$alias`, which registers a database literally called **`Client`** and re-points the
-  `Client_Nychh` alias at it — breaking the parent registration too. Correct form:
-  `database: self::DB_CLIENT_NYCHH, alias: _underscore::DB_CLIENT`. The trait reads through
+- **⚠ There is NO `try`/`catch` anywhere in this feature, by design.**
+  `sendTransferOrderEmail()` returns `void` and **throws `_Exception`** when
+  `_Model_Client_EmailTemplate::send()` returns `false` (which happens only when the template row is
+  inactive). Both callers write the sent marker on the line *after* the send, so a throw means
+  nothing is marked sent and the cron retries. Do not "harden" this with a catch: a swallowed
+  failure loses the email permanently **and** hides the row from the backstop cron. (This reverses
+  the earlier bool-returning design — the throw is now the contract.)
+- **⚠ `_Database::register()` takes the REAL database name first and the ALIAS last.**
+  `initialize()` originally passed `_underscore::DB_CLIENT` (`'Client'`) as `$database` and
+  `'Client_Nychh'` as `$alias`, which registers a database literally called **`Client`** and
+  re-points the `Client_Nychh` alias at it. Correct form:
+  `database: self::DB_CLIENT_NYCHH, alias: _underscore::DB_CLIENT` — the queries read through
   `_underscore::DB_CLIENT` because that is the alias every `_Model_Client_*` uses. See
   [DB alias re-pointing](../../../2.0/apps/_underscore/features/database-alias-repointing.md).
-- **A failed send must not be marked as sent.** `sendTransferOrderEmail()` returned `void` and the
-  caller wrote the marker unconditionally, so a single mail failure lost that email permanently
-  **and** hid the row from the backstop cron. It returns `bool` now, and the marker only lands on a
-  real send.
 - **`_Email::send()` only queues.** The finished email HTML can be read straight out of
   `Logs_Nychh.Email` without an inbox — that is how these were verified. See
   [the email send pipeline](../../../2.0/apps/_underscore/features/email-send-pipeline.md).
@@ -207,32 +238,36 @@ Cost of the split, stated so nobody is surprised later:
 
 ## Testing
 
-A local end-to-end harness (scratchpad, **not committed**) runs **82 scenarios, all passing**. The
-approach is worth reusing for any email feature:
-
-- It loads the **real** trait and the **real** worker2 cron class, not copies.
-- It runs against a throwaway local schema built with `CREATE TABLE LIKE` from `Client_Nychh`.
-- It stubs **only** `_Query` / `_Database` / `_Model_Client_EmailTemplate`.
-- It renders the **real** template bodies, loaded out of the dbchanges2 file, so an unresolved
-  `{placeholder}` fails a test.
-- Coverage: the send guards, the detail box, the item and serial tables, the tracking block, mail
-  failure, and the cron.
-
-The reversed `_Database::register()` arguments, the lost-on-failure marker, the tracking block, the
-unescaped values and both detail-box defects were all found here, not in production.
+A local end-to-end harness (scratchpad, **not committed**) runs the **real** worker class against a
+throwaway local schema built with `CREATE TABLE LIKE` from `Client_Nychh`, stubbing **only**
+`_Query` / `_Database` / `_Model_Client_EmailTemplate`, and rendering the **real** template bodies
+loaded out of the dbchanges2 file so an unresolved `{placeholder}` fails a test. The reversed
+`_Database::register()` arguments, the marker-written-on-failure bug, the tracking block, the
+unescaped values and both detail-box defects were all found there, not in production.
 
 ## Change history
 
+- 2026-09-18 — **Moved the whole feature out of `_underscore` into worker2.** Deleted
+  `_Trait_Nychh_TransferOrderEmail`; `worker2/Worker/Client/Nychh/TransferOrderEmails.php` now holds
+  the queued `SendSubmitted` / `SendFulfillment` entry points, the `SendPending` cron, every query
+  and the HTML builder, and the two per-client models only declare their `c_` fields and call
+  `_Worker::runTask` with the record uuid. Rationale: app-level over core, and a slow send must not
+  run inside an api2 save. Fixed the shipped email listing only the **first** tracking number
+  (`readTrackingForEmail` had `LIMIT 1`; 292 of 1,495 shipped fulfillments with tracking carry two or
+  more, up to 25). Lowered `MAXIMUM_EMAIL_BODY_BYTES` 92,000 → **80,000** so the 11,695-byte shipped
+  template and a ~9,500-byte 25-package tracking block still fit under Gmail's ~102,400 clip.
+  Replaced the inherited Compass no-product-image placeholder with the NYC H+H mark, and narrowed the
+  image cell `width="70"` → `width="50"`. Recorded the typeface rule (Inter for the item block only,
+  Plus Jakarta Sans everywhere else, Arial for Outlook) and the three `EmailTemplates` uuids.
+  **Reversed the earlier bool-return decision:** `sendTransferOrderEmail()` returns `void` and
+  **throws**, and there is deliberately no `try`/`catch` in the feature. (bala)
 - 2026-09-17 — Initial capture. Built the three H+H transfer-order emails (Submitted / Packed /
-  Shipped) as `_Trait_Nychh_TransferOrderEmail` plus model hooks, the `*/5` backstop cron
-  `Client/Nychh/TransferOrderEmails/SendPending`, and the `2026-09-15a` / `2026-09-15b` /
+  Shipped) plus model hooks, the `*/5` backstop cron, and the `2026-09-15a` / `2026-09-15b` /
   `2026-09-16a` migrations. Recorded the two load-bearing rules — **Submitted fires on `postPut`**
   because the POST still carries the `TA` placeholder (transfer order 1948: POST 10:05:34
   `TA100000`, PUT 10:05:46 `290007`), and **`createdByUserId` separates supply orders from sync
   orders** (1,936 sync rows NULL vs 6 supply rows set). Fixed a reversed `_Database::register()`
-  database/alias pair, a failed send still being marked sent, an unescaped HTML body, a broken
-  tracking block (1,484 of 2,479 fulfillments have no tracking number), a blank Site Contact and a
-  `CONCAT_WS` empty-string address gap. Added a 92,000-byte body budget for Gmail's ~102 KB clip
-  (worst case measured 94,864 bytes / 502 units). Flagged that a `Client_Nychh` clone wipes the
-  feature and that `nychh_email_templates/*.html` lives in no git repo. Verified by an 82-scenario
-  local harness running the real trait and cron. (bala)
+  database/alias pair, an unescaped HTML body, a broken tracking block (1,484 of 2,479 fulfillments
+  have no tracking number), a blank Site Contact and a `CONCAT_WS` empty-string address gap. Flagged
+  that a `Client_Nychh` clone wipes the feature and that `nychh_email_templates/*.html` lives in no
+  git repo. (bala)

package/knowledge/clients/staples/profile.md

@@ -9,7 +9,7 @@ project: Worker
 client: staples
 type: profile
 status: active
-updated: 2026-09-16
+updated: 2026-09-18
 owners: [jcardinal, bala, mhammontree]
 files:
   - worker/crons/sync/staples/sync_staples_cxml.php
@@ -24,13 +24,13 @@ Staples — headless Worker-1.0 integration client (two SFTP crons); open for it
 
 ## Summary
 
-Staples is a **headless** integration client — no UI, Worker 1.0 only, depending on the `library` (1.0 `App_`) core. Its entire footprint is two crons under `worker/crons/sync/staples/`:
+Staples is a **headless** integration client — no UI, Worker 1.0 only, depending on the `library` (1.0 `App_`) core. Its entire footprint is two crons under `worker/crons/sync/staples/`, both scheduled in `schedules/cron.worker.sync.json`:
 
-1. **`sync_staples_cxml.php`** — hourly inbound: pulls cXML purchase orders off SFTP, creates NetSuite Sales Orders, and writes 855 acknowledgements back. The documented feature — see **`1.0/apps/worker/features/staples-cxml-order-import.md`**.
-2. **`staples_asn_netsuite.php`** — outbound ASN (Advance Ship Notice) from NetSuite back to Staples. (Not yet documented in detail.)
+1. **`sync_staples_cxml.php`** — `45 * * * *`, "Create Sales Orders in Netsuite From Staples". Inbound: pulls cXML purchase orders off SFTP, creates NetSuite Sales Orders, and writes 855 acknowledgements back. The documented feature — see **`1.0/apps/worker/features/staples-cxml-order-import.md`**.
+2. **`staples_asn_netsuite.php`** — `50 * * * *`, "Create ASN from net suite to staples". Outbound ASN (Advance Ship Notice) from NetSuite back to Staples. (Not yet documented in detail.)
 
 **NetSuite identity:** Staples is the NetSuite parent entity/customer **internalId 30057**. Individual Staples buying accounts are NetSuite **child customers** matched by their Staples `CustomerID` against **saved search 8412** (match key `custentity_external_customer_id`; location from `custentity_customer_location`).
 
-**Onboarding rule (load-bearing):** any new Staples buying account must be set up in NetSuite — as a child customer of 30057, with its **External Customer ID** = the Staples `CustomerID` **and** a valid **Customer Location** — *before* its first order arrives. Miss either and inbound orders for that account silently jam on SFTP with no error (see the feature doc's gotchas).
+**Onboarding rule (load-bearing):** any new Staples buying account must be set up in NetSuite — as a child customer of 30057, with its **External Customer ID** = the Staples `CustomerID` **and** exactly one **Customer Location** (a child of Location 36 "New York : New York Warehouse : Staples") — *before* its first order arrives. Miss either and inbound orders for that account silently jam on SFTP `/in`, retrying hourly with nothing logged. Full setup rule and the diagnostic steps are in the [cXML import feature doc](../../1.0/apps/worker/features/staples-cxml-order-import.md).
 
 **Sales rep on Staples sales orders:** the SO `entity` is always parent **30057**, so NetSuite sources **30057's** rep (currently Paulina Szeliga, 50695) regardless of which child account bought — see [NetSuite Sales Order Sales Rep Sourcing](../../1.0/apps/worker/features/netsuite-sales-order-sales-rep-sourcing.md).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.842",
+  "version": "1.0.844",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",