toga-ai 1.0.773 → 1.0.775

package/knowledge/1.0/apps/webhook/INDEX.md

@@ -2,4 +2,4 @@
 
 | Doc | Summary | Files |
 |-----|---------|-------|
-| [NetSuite → ClickUp Opportunity Sync](features/netsuite-clickup-opportunity-sync.md) | NetSuite Opportunities are pushed into the ClickUp **Opportunities** list (list id `901111987449`, in the Opportunity/RFP Hub space `90113928591`) as tasks name | webhook/_/webhook/netsuite.php, webhook/api.php, api2/Webhook/Netsuite.php, worker2/Worker/Netsuite.php, api2/Controller/Index.php |
+| [NetSuite → ClickUp Opportunity Sync](features/netsuite-clickup-opportunity-sync.md) | > **⚠ CORRECTED 2026-09-04 — parts of this doc were factually wrong and actively misleading.** > Two claims below have been retracted in place: that > `worker2/ | webhook/_/webhook/netsuite.php, webhook/api.php, api2/Webhook/Netsuite.php, worker2/Worker/Netsuite.php, api2/Controller/Index.php |

package/knowledge/1.0/apps/webhook/features/netsuite-clickup-opportunity-sync.md

@@ -6,17 +6,33 @@ project: Webhook
 client: shared
 type: feature
 status: active
-updated: 2026-07-17
-owners: ["dfranks"]
+updated: 2026-09-04
+owners: ["dfranks", "ajean"]
 files:
   - webhook/_/webhook/netsuite.php
   - webhook/api.php
   - api2/Webhook/Netsuite.php
   - worker2/Worker/Netsuite.php
   - api2/Controller/Index.php
-related: []
+related:
+  - ../../../2.0/apps/worker2/features/netsuite-opportunity-sync.md
 ---
 
+> **⚠ CORRECTED 2026-09-04 — parts of this doc were factually wrong and actively misleading.**
+> Two claims below have been retracted in place: that
+> `worker2/Worker/Netsuite/Opportunity.php` and `maybeCreateClickupTask()` **do not exist**, and that
+> the integration has **"no dedup of any kind."** Both files exist and are git-active, and the 2.0
+> path carries an **atomic DB claim** on `Forecast.Opportunities.clickupTaskId` plus folder-wide
+> dedup. Those two claims are why a later session opened with a contradiction about which app owns
+> the receiver. Canonical detail:
+> [NetSuite ↔ ClickUp Opportunity Sync (worker2)](../../../2.0/apps/worker2/features/netsuite-opportunity-sync.md).
+>
+> **Scope caveat on everything else here:** the 1.0 `webhook` repo is **not checked out** on the
+> machine used for the 2026-09-04 review and could not be found anywhere on disk, so the claims in
+> this doc about the *live 1.0 receiver* are **unverified as of that date**. The NetSuite-side
+> content (scheduled script **2412**, UserEvent **2493**, the `OPP <id>` stub root cause and its fix)
+> was **not** in question and stands.
+
 ## Summary
 NetSuite Opportunities are pushed into the ClickUp **Opportunities** list (list id `901111987449`,
 in the Opportunity/RFP Hub space `90113928591`) as tasks named `<oppNumber> — <customerNumber>
@@ -74,23 +90,32 @@ the assignment fires 2493 in `edit` (full-record) context → proper name (see G
 None — single internal Agilant/True integration.
 
 ## Gotchas / known issues
-- **Create-only, NO dedup of any kind (confirmed 2026-07-17).** `Webhook_NetSuite::post()`
-  json-decodes the payload and calls `createTask()`, which POSTs a **new** task to ClickUp
-  list `901111987449` **unconditionally on every delivery** (writing link `db_client`). There
-  is no idempotency check whatsoever — so it double-creates a task on **any** repeat or
-  concurrent delivery, not only concurrent ones.
+- **Create-only with no dedup *in this 1.0 handler* — but the claim "NO dedup of any kind" was
+  WRONG and is retracted (2026-09-04).** As originally written (2026-07-17) this bullet asserted the
+  integration as a whole had "no idempotency check whatsoever." That is false at the platform level:
+  the **2.0** path (`worker2/Worker/Netsuite/Opportunity.php`) carries a **race-safe atomic claim** —
+  a guarded `UPDATE Opportunities SET clickupTaskId=… WHERE … AND clickupTaskId IS NULL` — plus
+  folder-wide dedup by `Opportunity #`. What remains true, scoped correctly: **`Webhook_NetSuite::post()`
+  in the 1.0 `webhook` repo** json-decodes the payload and calls `createTask()`, which POSTs a new
+  task **unconditionally on every delivery** (writing link `db_client`), so *that* handler
+  double-creates on any repeat or concurrent delivery. Unverified since 2026-09-04 (repo not checked
+  out).
 - **Live payload carries no immutable opportunity identifier (confirmed 2026-07-17).** The
   inbound body has **only**: `name`, `description`, `presalesLead`, `salesRepText`, `token`.
   It has **no NetSuite opportunity internal id and no opportunity/customer number**, so there
   is no stable key available in this handler to dedup on today. Any dedup fix must first get
   NetSuite (script 2493) to include an immutable id in the payload, or derive one another way.
-- **Where the swapped Opp#/Cust# mapping actually lives.** The
-  `OPPORTUNITY_NUMBER ← customerNumber` / `CUSTOMER_NUMBER ← opportunityNumber` swap is in the
-  **dormant 2.0 port** `worker2/Worker/Netsuite.php` (class `_Worker_NetSuite`, ~lines 116–121),
-  which is **NOT live**. The live `webhook/_/webhook/netsuite.php` payload doesn't even carry
-  those number fields (see above). There is **no** `worker2/Worker/Netsuite/Opportunity.php` and
-  **no** `maybeCreateClickupTask()` method anywhere — earlier plans referencing those paths were
-  citing files that do not exist.
+- **Where the swapped Opp#/Cust# mapping lives — with the "these files don't exist" claim
+  RETRACTED (2026-09-04).** The `OPPORTUNITY_NUMBER ← customerNumber` /
+  `CUSTOMER_NUMBER ← opportunityNumber` swap is in the **dormant 2.0 port**
+  `worker2/Worker/Netsuite.php` (class `_Worker_NetSuite`, ~lines 116–121). That part stands.
+  **What was wrong:** this bullet claimed there is "**no** `worker2/Worker/Netsuite/Opportunity.php`
+  and **no** `maybeCreateClickupTask()` method anywhere," and that plans citing them were citing
+  fictional files. **Both exist and are git-active.** `_Worker_Netsuite_Opportunity` is the current
+  2.0 handler — it does the Forecast upsert, holds the atomic `clickupTaskId` claim, and
+  `maybeCreateClickupTask()` is its ClickUp entry point. It also **fixed** the swap (2026-06-16).
+  Do not use this bullet to conclude the 2.0 path is vapour; read
+  [the worker2 doc](../../../2.0/apps/worker2/features/netsuite-opportunity-sync.md).
 - **`OPP <internalId>` degenerate tasks (root-caused & confirmed 2026-06-15).** A task named just
   `OPP 7147699` with no description/fields. The name is built by NetSuite and relayed verbatim;
   the opps are **fully populated** in NetSuite (so it's not missing data, not a TOGA bug). Cause:
@@ -148,6 +173,16 @@ Done 2026-06-15 for opps 7147699/7147698/7147472/6382879 → 4 proper tasks crea
 (task ids `868k0tr41`, `868k0tr16`, `868k0rwtp`, `868k0h44m`) were archived in ClickUp afterward.
 
 ## Change history
+- 2026-09-04 — **Corrected two factually wrong claims (retracted in place; no code changed).** This
+  doc asserted that `worker2/Worker/Netsuite/Opportunity.php` and `maybeCreateClickupTask()` "do not
+  exist" and that the integration has "no dedup of any kind." Both files exist and are git-active,
+  and the 2.0 path carries an **atomic DB claim** on `Forecast.Opportunities.clickupTaskId` plus
+  folder-wide dedup; the create-only/no-dedup statement is true only of the **1.0 `webhook` handler**
+  and has been rescoped to it. These two claims were the reason a later session opened with a
+  contradiction about which app owns the receiver. Added a scope caveat: the 1.0 `webhook` repo is
+  **not checked out** on the reviewing machine and could not be found on disk, so this doc's claims
+  about the live 1.0 receiver are **unverified as of 2026-09-04**; the NetSuite-side script 2412/2493
+  content was never in question and stands. (ajean)
 - 2026-07-17 — Confirmed against live code: `createTask()` POSTs to list `901111987449`
   **unconditionally with no dedup** (double-creates on any repeat/concurrent delivery); the
   live payload carries only `name`/`description`/`presalesLead`/`salesRepText`/`token` — **no

package/knowledge/1.0/apps/worker/INDEX.md

@@ -6,9 +6,9 @@
 | [Canon received-PO → NetSuite sales order sync](features/canon-received-po-netsuite-order-sync.md) | A 1.0 `worker` cron that reads Canon customer purchase orders out of the **Vision** databases and creates the matching **sales order in NetSuite**. | worker/crons/sync/sync_received_po.php, library/netsuitetoolkit/NSPHPClient.php |
 | [Compass MA Sales Order Exception Report](features/compass-ma-sales-order-exception-report.md) | A worker cron that emails operations the "Compass Refresh Exception Report" — Compass `MA%` sales orders whose corresponding Office Depot (ODP) sales order has | worker/crons/toga2/compass/workflow/7_generate_ma_sales_order_exception_report.php, worker/schedules/cron.worker.sync.json |
 | [Compass Manager Approval Reminder Emails (1.0 worker crons)](features/compass-manager-approval-reminder-emails.md) | Two 1.0 worker crons nag approvers about sales orders still waiting on a decision — one per Compass tenant. | worker/crons/toga2/compass/compass_email_reminders.php, worker/crons/toga2/compasscanada/compass_email_reminders.php, worker/schedules/cron.worker.sync.json, worker1.5/crons/toga2/compass/compass_email_reminders.php, worker1.5/schedules/cron.worker.json |
-| [Compass Partial In-Transit & Delivered Emails (per package)](features/compass-partial-in-transit-delivered-emails.md) | Compass USA and Compass Canada send a **per-package** in-transit email (and a matching delivered email) instead of one email listing the whole order. | worker/crons/toga2/compass/update_salesorder_status_from_odp.php, worker/crons/toga2/compass/send_delivered_email.php, worker/crons/toga2/compasscanada/update_salesorder_status_from_odp.php, worker/crons/toga2/compasscanada/workflow/3_update_salesorder_status_from_grand_and_toy.php, worker/crons/toga2/compasscanada/send_delivered_email.php, worker/crons/toga2/compasscanada/workflow/test_partial_in_transit_email.php, worker/crons/toga2/compasscanada/workflow/test_partial_delivered_email.php, library/app/client/compasscanada.php |
+| [Compass Partial In-Transit & Delivered Emails (per package)](features/compass-partial-in-transit-delivered-emails.md) | Compass USA and Compass Canada send a **per-package** in-transit email (and a matching delivered email) instead of one email listing the whole order. | worker/crons/toga2/compass/update_salesorder_status_from_odp.php, worker/crons/toga2/compass/send_delivered_email.php, worker/crons/toga2/compasscanada/update_salesorder_status_from_g&t.php, worker/crons/toga2/compasscanada/send_delivered_email.php, worker/crons/toga2/compasscanada/workflow/test_partial_in_transit_email.php, worker/crons/toga2/compasscanada/workflow/test_partial_delivered_email.php, library/app/client/compasscanada.php, worker/schedules/cron.worker.sync.json, worker2/Worker/Monitor/Compass.php, dbchanges2/Client_Compass/2026-09-03 - compass_intransit_email_backlog_backfill.sql |
 | [Elite TOGA 2.0 → TOGaDeskSupport Standalone Attachment Sync](features/elite-togadesk-attachment-sync.md) | `sync_togadesk_elite_attachments.php` is a standalone cron (every 5 minutes) that syncs file attachments from TOGA 2.0 into TOGaDeskSupport for Elite. | worker/crons/toga2/elite/sync_togadesk_elite_attachments.php, worker/crons/toga2/elite/test_sync_togadesk_elite_attachments.php |
-| [Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling](features/forecast2-netsuite-reconciliation.md) | CLI tools to **audit** and **repair** drift between the production `Forecast` DB (core2) and NetSuite. | test/@dave/checker.php, worker2/Component/Forecast/SaleImport/SaleImport.php, test/@dave/looper.php, test/@dave/reconcile_netsuite_totals.php, test/@dave/fixer.php, tools/bin/forecast/fixer.php, tools/bin/forecast/checker.php, test/@dave/analyze_netsuite_forecast_diff.php, test/@dave/trueup_sales.php, test/@dave/reconcile_drift_2023plus.php, test/@dave/probe_invoice_gap_2026.php, test/@dave/probe_creditmemo_gap_detail.php, test/@dave/trueup_open_orders.php, test/@dave/loop_trueup_open_orders.php, test/@dave/trueup_opportunities.php, test/@dave/probe_sales_gap_direct.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_profit_invoices.php, test/@dave/probe_profit_gap.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_open_orders.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/schedules/cron.worker.infrastructure.json |
+| [Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling](features/forecast2-netsuite-reconciliation.md) | CLI tools to **audit** and **repair** drift between the production `Forecast` DB (core2) and NetSuite. | test/@dave/checker.php, worker2/Component/Forecast/SaleImport/SaleImport.php, test/@dave/looper.php, test/@dave/reconcile_netsuite_totals.php, test/@dave/fixer.php, tools/bin/forecast/fixer.php, tools/bin/forecast/checker.php, test/@dave/analyze_netsuite_forecast_diff.php, test/@dave/trueup_sales.php, test/@dave/reconcile_drift_2023plus.php, test/@dave/probe_invoice_gap_2026.php, test/@dave/probe_creditmemo_gap_detail.php, test/@dave/trueup_open_orders.php, test/@dave/loop_trueup_open_orders.php, test/@dave/trueup_opportunities.php, test/@dave/probe_sales_gap_direct.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_profit_invoices.php, test/@dave/probe_profit_gap.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_open_orders.php, worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_opportunities.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/schedules/cron.worker.infrastructure.json |
 | [Forecast2 Supporting-Records Nightly Import (1.0 cron) — Customers & ConsolidatedCustomers](features/forecast2-supporting-records-import.md) | The nightly **1.0** pull that keeps the Forecast2 lookup/dimension tables (`Forecast.Accounts`, `Classifications`, `Customers`, `ConsolidatedCustomers`, `Employ | worker/crons/toga2/forecast2/import_supporting_records.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker/schedules/cron.worker.infrastructure.json, worker/.ebextensions/009_setup_phpini.config, worker/crons/sync/netsuite/netsuite_customers.php, library/app/api/netsuite/rest.php, library/app/model/forecast2/customer.php, library/app/model/forecast2/consolidatedcustomer.php |
 | [NetSuite Sales Order Sales Rep Sourcing (Staples & ODP EDI orders)](features/netsuite-sales-order-sales-rep-sourcing.md) | How the **sales rep** on a NetSuite Sales Order is determined for the two 1.0 `worker` EDI order-creation integrations (Staples cXML and Compass/ODP EDI). | worker/crons/toga2/compass/workflow/5_create_netsuite_sales_orders_from_office_depot_purchase_orders.php, worker/crons/sync/staples/sync_staples_cxml.php, test/@Mark/NetSuite/TRUE_80451_customer_salesrep_diag.php |
 | [NetSuite → TOGa Supply Per-Client Sync (thin wrappers)](features/netsuite-togasupply-per-client-sync.md) | Syncs NetSuite transactions (sales orders, purchase orders, invoices, item receipts, item fulfillments, inventory adjustments) into each TOGa Supply (2.0) clien | worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_canon.php, worker/crons/toga2/netsuite/sync_togasupply_elite.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql, library/app/api/toga2.php, library/app/api/netsuite/rest.php, library/app/query.php, library/app/database.php, library/app/framework.php, library/app/systemmonitor/netsuiteintegration.php, dbchanges2/Client/2026-09-02b - NetsuiteSyncCursorRename.sql, test/@srija/Elite Testing/Service Requests/test_sync_togasupply_elite_section.php, test/@srija/Elite Testing/Service Requests/test_diagnose_togasupply_elite.php |

package/knowledge/1.0/apps/worker/features/compass-partial-in-transit-delivered-emails.md

@@ -6,18 +6,23 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-26
+updated: 2026-09-04
 owners: ["bala"]
 files:
   - worker/crons/toga2/compass/update_salesorder_status_from_odp.php
   - worker/crons/toga2/compass/send_delivered_email.php
-  - worker/crons/toga2/compasscanada/update_salesorder_status_from_odp.php
-  - worker/crons/toga2/compasscanada/workflow/3_update_salesorder_status_from_grand_and_toy.php
+  - worker/crons/toga2/compasscanada/update_salesorder_status_from_g&t.php
   - worker/crons/toga2/compasscanada/send_delivered_email.php
   - worker/crons/toga2/compasscanada/workflow/test_partial_in_transit_email.php
   - worker/crons/toga2/compasscanada/workflow/test_partial_delivered_email.php
   - library/app/client/compasscanada.php
+  - worker/schedules/cron.worker.sync.json
+  - worker2/Worker/Monitor/Compass.php
+  - dbchanges2/Client_Compass/2026-09-03 - compass_intransit_email_backlog_backfill.sql
 related:
+  - ../../../clients/compass-usa/features/contact-email-resolution.md
+  - ../../../2.0/apps/worker2/features/tracking-status-refresh.md
+  - ../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md
   - ../../../clients/compass-canada/features/french-order-email-localization.md
   - ../../../clients/compass-usa/features/order-fulfillment-status-per-line.md
   - ../../../2.0/apps/_underscore/features/item-fulfillment-stage-lifecycle-and-order-status.md
@@ -34,18 +39,28 @@ and earlier packages). The dynamic HTML is injected into a stored `EmailTemplate
 
 ## Key files / entry points
 
-- `worker/crons/toga2/compass/update_salesorder_status_from_odp.php` — Compass USA prod cron.
-- `worker/crons/toga2/compasscanada/workflow/3_update_salesorder_status_from_grand_and_toy.php`
-  — Compass Canada prod cron.
+- `worker/crons/toga2/compass/update_salesorder_status_from_odp.php` — Compass USA prod cron,
+  registered in `worker/schedules/cron.worker.sync.json` as **"Update Compass SalesOrder Status
+  From ODP CXML"**, schedule **`0 */4 * * *`** — it fires at 00/04/08/12/16/20 **US/Central**
+  (the prod DB timezone). Every threshold below (batch cap, staleness, monitor alarms) is
+  calibrated to that 4-hour cadence.
+- `worker/crons/toga2/compasscanada/update_salesorder_status_from_g&t.php` — Compass Canada
+  prod cron, registered as "Update CompassCanada SalesOrder Status From G&T CXML".
+  **⚠ This filename contains `&` and therefore cannot run — see the gotcha below.** It was
+  renamed from `update_salesorder_status_from_odp.php` on 2026-09-03; the duplicate
+  `workflow/3_update_salesorder_status_from_grand_and_toy.php` was **deleted** in the same pass and
+  its stale `schedules/cron.worker.sync.json` entry removed.
 - Shared helper set in each cron: `getOrderFulfillmentData`, `buildItemRowsHtml`,
   `buildSectionHeaderHtml`, `buildPackageBlockHtml`, `buildTrackingUrl`, `buildOrderItemsHtml`.
 
 ## How it works
 
 1. **Driving query** finds shipped sales orders and their tracking numbers (`computedStatusSlug
-   = 'shipped'`, `c_dtInTransitEmailSent IS NULL`, date cutoff) via the ASN chain
-   (`AdvanceShippingNoticeItemUnits_TrackingNumbers`), one row per `TrackingNumbers.number`.
-2. Per tracking number: skip if the carrier API already reports delivered.
+   = 'shipped'`, `c_dtInTransitEmailSent IS NULL`, rolling date window) via the ASN chain
+   (`AdvanceShippingNoticeItemUnits_TrackingNumbers`), one row per `TrackingNumbers.number`,
+   **newest first and capped** — see *Draining the candidate set* below.
+2. Per tracking number: skip if the shipment is already finished — checked first against the
+   stored `TrackingNumbers.status` (free), and only then with a live carrier call.
 3. **`getOrderFulfillmentData`** returns the order's packages (grouped by tracking number,
    ordered by `dtCreated`) and ordered items, sourced from the **`ItemFulfillmentItems`** line
    bridge (`ItemFulfillmentItems` → `ItemFulfillments` (scoped to the buyer SO) →
@@ -58,6 +73,73 @@ and earlier packages). The dynamic HTML is injected into a stored `EmailTemplate
    `App_Api_Toga2::send(..., 'POST', '/email-templates/sendEmail', $payload, [], true)`.
 6. On a successful (non-throwing) send, set `c_dtInTransitEmailSent = NOW()`.
 
+### Draining the candidate set: two-pass, newest-first, bounded
+
+**This is the load-bearing part of the cron. Getting it wrong took Compass USA in-transit emails to
+near zero for a week** (2026-08-27 → 2026-09-03: 130–190 a day before, then 7 / 1 / 6 / 0 / 1).
+Customers were not told their order had shipped.
+
+The original shape was: select **every** unstamped `TrackingNumbers` row since a hardcoded
+`'2026-06-15'`, then call UPS/FedEx **live, one row at a time**. When a shipment turned out to be
+already delivered the loop `continue`d **without stamping `c_dtInTransitEmailSent`** — so the same
+row came back on the next run, forever. The candidate set grew to **2,379** and the run hit its
+`App_Framework::cronInitialization(3600)` one-hour limit while still inside the June rows. It never
+reached the day's new ASNs. Nothing errored; the cron just ran out of time on every tick.
+
+Four changes fix it, and they are the pattern to copy for any *"poll an external API per row, then
+mark the row done"* cron:
+
+1. **Two passes.** Pass 1 walks the candidates and **stamps** every already-finished row so it
+   retires from the candidate set permanently. Pass 2 sends. A row can never again be examined
+   forever without being retired.
+2. **Newest first, hard cap.** `ORDER BY MAX(TrackingNumbers.dtCreated) DESC` +
+   `LIMIT IN_TRANSIT_EMAIL_BATCH_SIZE` (**400**). Today's shipments are served before any backlog,
+   so a backlog can delay old mail but can never starve new mail.
+3. **Rolling window, not a fixed date.** `IN_TRANSIT_EMAIL_LOOKBACK_DAYS = 14` →
+   `dtCreated >= DATE_SUB(NOW(), INTERVAL 14 DAY)`. A hardcoded cutoff date guarantees an
+   ever-growing scan; a rolling window is self-limiting.
+4. **Trust the stored status before paying for a carrier call.** `isTrackingStatusFinal(?string)`
+   short-circuits when `TrackingNumbers.status` is already `DELIVERED` or `RETURNED_TO_SENDER`
+   (`TRACKING_FINAL_STATUSES`). worker2's
+   [centralized tracking-status refresh](../../../2.0/apps/worker2/features/tracking-status-refresh.md)
+   keeps that column current for every client, so this 1.0 cron should not re-ask the carrier about
+   a finished shipment.
+
+### Resolving the recipient: a `Users` row is NOT required (USA)
+
+The To address originally came only from `Users.email`, via `INNER JOIN Users ON Users.contactId =
+…`. **EDI-created MR orders never get a `Users` row**, so those orders were silently absent from
+the driving query — 12 MR orders with shipped parcels were never emailed and never even showed up
+as a backlog.
+
+The USA cron now resolves the address in this precedence:
+
+```
+LEFT OUTER JOIN Users
+    ON Users.contactId = ShippedSalesOrders.contactId
+    AND Users.email IS NOT NULL AND Users.email <> ''
+INNER JOIN Contacts
+    ON Contacts.id = ShippedSalesOrders.contactId
+LEFT OUTER JOIN ContactEmailAddresses
+    ON ContactEmailAddresses.id = Contacts.primaryContactEmailAddressId
+    AND ContactEmailAddresses.contactId = Contacts.id      # ownership guard
+…
+IFNULL(Users.email, ContactEmailAddresses.emailAddress) AS contactEmail
+```
+
+The same `IFNULL` resolves the display name, and the `WHERE` requires the resolved address to be
+non-empty.
+
+**The `ContactEmailAddresses.contactId = Contacts.id` predicate is not optional.** About 14% of
+Compass contacts point at an email row owned by a *different* contact, so without it the cron mails
+the wrong person — see
+[Compass contact email resolution](../../../clients/compass-usa/features/contact-email-resolution.md).
+
+**Regression-checked, not assumed.** Over 60 days and **4,419 SA tracking rows** the old and new
+queries both select **4,416**: 0 newly included, 0 lost, resolved address and name identical on all
+4,416. The `Users` address always wins because it is first in the `IFNULL`, so the change is
+strictly additive — it only reaches orders the old query could not see at all.
+
 ### Tracking-URL building (`buildTrackingUrl`)
 
 The "Track Shipment" button and each per-package link resolve in this precedence:
@@ -100,7 +182,22 @@ the tracking number as emailed (it retries next run).
 
 ## Client variations
 
-- **Compass USA** — English only; `update_salesorder_status_from_odp.php`.
+- **Compass USA** — English only; `update_salesorder_status_from_odp.php`. Has the full
+  2026-09-03 hardening: two-pass drain, 400-row cap, 14-day window, `isTrackingStatusFinal()`,
+  **and** the `Users` → `Contacts`/`ContactEmailAddresses` recipient fallback with the ownership
+  guard.
+- **⚠ Canada is only PARTLY at parity (verified 2026-09-04).**
+  `compasscanada/update_salesorder_status_from_g&t.php` has the batching changes
+  (`IN_TRANSIT_EMAIL_BATCH_SIZE = 400`, `IN_TRANSIT_EMAIL_LOOKBACK_DAYS = 14`,
+  `ORDER BY MAX(TrackingNumbers.dtCreated) DESC`, `isTrackingStatusFinal()`) but **still resolves
+  the recipient with `INNER JOIN Users … Users.email AS contactEmail`** — no `Contacts` /
+  `ContactEmailAddresses` fallback and no ownership guard. Canada therefore still cannot email a
+  contact that has no `Users` row. Port the recipient block if Canada ever gets EDI-created MR
+  orders.
+- **Canada keeps its own retry behaviour on a carrier throw.** It leaves the row **unstamped** when
+  the carrier lookup throws, so the next run retries it; only rows confirmed *final* are stamped.
+  That is deliberate and must survive any future parity pass — stamping on a throw would silently
+  drop the email.
 - **Compass Canada** — EN/FR by `UserGlobalSettings.settingId = 2` (FR when the value starts
   with `fr`); French section labels and a French (HTML-entity) status message; uses
   `htmlentities` in the section header builder and a 7-param French-aware `buildPackageBlockHtml`.
@@ -113,6 +210,33 @@ the tracking number as emailed (it retries next run).
 
 ## Gotchas / known issues
 
+- **⚠ A cron filename containing `&` cannot run at all.** `worker/ebs/cron.worker.php:41` builds
+  the crontab command by plain unquoted concatenation
+  (`'php /var/www/html/crons/' . $cron->cron`), so cron reads `&` as *"background this and start a
+  new command"*: `update_salesorder_status_from_g&t.php` executes as
+  `…/update_salesorder_status_from_g` plus a stray `t.php`, and the real script never runs —
+  silently, on every tick. **The Canada cron is in this state as of 2026-09-04 and must be renamed
+  (e.g. `update_salesorder_status_from_gt.php`) with `schedules/cron.worker.sync.json` repointed.**
+  Never put `&` (or any shell metacharacter) in a cron filename. See
+  [tracing a 1.0 worker cron run](../workflows/tracing-a-worker-cron-run-in-production.md).
+- **⚠ Invoice tracking numbers are NOT in scope and must never be counted as in-transit
+  backlog.** `POST /v2/invoice-tracking-numbers` creates `TrackingNumbers` rows bridged through the
+  **`InvoiceTrackingNumbers`** table — never the ASN chain. On 2026-09-03 alone, 98 were created
+  (11:15–11:40) and **zero** of them touch `AdvanceShippingNoticeItemUnits_TrackingNumbers`.
+  Anything that walks the ASN unit bridge (this cron, and the OneUptime queue monitor) correctly
+  ignores them. Their shape is distinctive: **`shippingCarrierId` set, `shippingMethodId` NULL**.
+  A naive `TrackingNumbers WHERE c_dtInTransitEmailSent IS NULL` count will read these as a backlog
+  that never drains.
+- **Clear an in-transit backlog by STAMPING, not by emailing — and refresh statuses first.**
+  Decided 2026-09-03 while draining the outage backlog
+  (`dbchanges2/Client_Compass/2026-09-03 - compass_intransit_email_backlog_backfill.sql`):
+  `c_dtInTransitEmailSent` was stamped on **64,132** rows older than the 14-day window and **421**
+  in-window rows already on a final status, leaving **68** genuinely still moving. Sending an
+  "in transit" email for a June order that arrived two months ago confuses the customer, so old
+  rows retire silently. **Ordering matters:** let worker2 `Sync/TrackingNumbers` refresh any
+  recently-unparked rows **before** running the backfill, otherwise a row with a stale status gets
+  a wrong in-transit email. (A parked row holds a stale status indefinitely — see
+  [tracking-status refresh](../../../2.0/apps/worker2/features/tracking-status-refresh.md).)
 - **Canada now uses the `ItemFulfillmentItems` line bridge, not the ASN chain**, for
   `getOrderFulfillmentData` — same query as Compass USA. Canada was originally on the ASN chain
   because it had **zero `ItemFulfillmentItems`**; if that line-bridge data is not populated for a
@@ -138,7 +262,7 @@ the tracking number as emailed (it retries next run).
   language even though the mail goes to the **manager**. Fixed 2026-08-03.
 - **⚠ Each cron keeps its OWN duplicated copy of the 2.0 `_status` CASE — a standing divergence
   risk.** `compass/update_salesorder_status_from_odp.php`,
-  `compasscanada/update_salesorder_status_from_odp.php` and
+  `compasscanada/update_salesorder_status_from_g&t.php` and
   `compass/workflow/test_partial_in_transit_email.php` each re-implement the model's status SQL as
   `computedStatusSlug`. Today that is safe by luck of scope: the copies only ever produce canceled /
   pendingApprovalUnknown / pendingFulfillment / shipped and they filter on
@@ -161,6 +285,27 @@ the tracking number as emailed (it retries next run).
 
 ## Change history
 
+- 2026-09-04 — **Fixed the Compass USA in-transit email outage (2026-08-27 → 2026-09-03, 130–190
+  emails/day → near zero).** Root cause: already-delivered rows were `continue`d without stamping
+  `c_dtInTransitEmailSent`, so the candidate set grew to 2,379 and every run burned its
+  `cronInitialization(3600)` hour inside rows dating back to a hardcoded `'2026-06-15'`, never
+  reaching new ASNs. Fix: two-pass drain (stamp finished rows, then send), newest-first
+  `ORDER BY MAX(dtCreated) DESC` with `IN_TRANSIT_EMAIL_BATCH_SIZE = 400`, rolling
+  `IN_TRANSIT_EMAIL_LOOKBACK_DAYS = 14` replacing the fixed date, and `isTrackingStatusFinal()`
+  short-circuiting the carrier call on `DELIVERED`/`RETURNED_TO_SENDER` using the status worker2
+  `Sync/TrackingNumbers` already maintains. Also **widened the recipient**: `LEFT OUTER JOIN Users`
+  + `Contacts`/`ContactEmailAddresses` with an ownership guard and
+  `IFNULL(Users.email, ContactEmailAddresses.emailAddress)`, which recovered 12 EDI-created MR
+  orders that have no `Users` row (regression-checked: 4,416 of 4,419 SA rows selected by both old
+  and new queries, 0 gained, 0 lost, identical resolved address/name). Backlog cleared by
+  **stamping** 64,132 out-of-window + 421 final-status rows rather than emailing them. Recorded
+  that **invoice tracking numbers are not in scope** (bridged via `InvoiceTrackingNumbers`, never
+  the ASN chain) and that a **`&` in a cron filename stops the cron dead**. Canada got the batching
+  changes only (renamed to `update_salesorder_status_from_g&t.php`; duplicate
+  `workflow/3_update_salesorder_status_from_grand_and_toy.php` deleted); its recipient block is
+  **still `INNER JOIN Users`**. New OneUptime monitor `Monitor/Compass/InTransitEmailQueue` now
+  watches this queue — see
+  [OneUptime push-metric monitors](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md). (bala)
 - 2026-08-26 — Flagged the **duplicated `_status` CASE** in the three crons as a standing divergence
   risk after the 2.0 Compass fulfillment rule was rewritten. Verified the crons are unaffected: their
   copies only emit canceled / pendingApprovalUnknown / pendingFulfillment / shipped and filter on

package/knowledge/1.0/apps/worker/features/forecast2-netsuite-reconciliation.md

@@ -6,8 +6,8 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-28
-owners: [dfranks, jcardinal, kyalamarthi]
+updated: 2026-09-04
+owners: [dfranks, jcardinal, kyalamarthi, ajean]
 files:
   - test/@dave/checker.php
   - worker2/Component/Forecast/SaleImport/SaleImport.php
@@ -32,6 +32,7 @@ files:
   - test/@dave/probe_profit_gap.php
   - worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php
   - worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_open_orders.php
+  - worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_opportunities.php
   - worker/crons/toga2/forecast2/import_open_orders.php
   - worker/schedules/cron.worker.infrastructure.json
 related:
@@ -373,6 +374,50 @@ have broken **every insert the cron makes**. It was caught only by rendering the
 stub harness before any run. **`php -l` cannot catch malformed SQL string assembly**; when you edit a
 query *builder*, print the built query and read it.
 
+### ⚠ The opportunities discrepancy-fix cron DELETES rows that carry a ClickUp link (2026-09-04)
+
+**This is a live data-loss path — treat it as the highest-priority item in this doc.**
+`worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_opportunities.php` (209 lines,
+`0 4 * * *`, **active**) and the same code path in `tools/bin/forecast/fixer.php`
+(`fixOpportunities()`) both build their delete set **without ever selecting or checking
+`clickupTaskId`**.
+
+**Blast radius, measured 2026-09-04: 746 of 26,766 in-window rows carried a ClickUp task id**, and
+one row was already stranded at the `PENDING` sentinel. Deleting such a row **orphans the ClickUp
+task** and, because the 2.0 webhook's atomic claim keys on `Forecast.Opportunities.clickupTaskId`,
+the next NetSuite event for that opportunity finds no claim and **creates a duplicate task**. The
+trigger does not require anything exotic: **a single truncated `listOpportunities()` page makes every
+unreturned opportunity look stale**, so an API hiccup is enough to mass-delete live rows.
+Claim semantics and the duplicate-creation consequence:
+[NetSuite ↔ ClickUp Opportunity Sync](../../../../2.0/apps/worker2/features/netsuite-opportunity-sync.md).
+
+Three further defects found in the same file:
+
+1. **⚠ `MAINTENANCE_MODE = true` gates LOGGING ONLY — it does NOT stop writes.** Its sibling
+   `periodic_forecast_discrepancy_fix_open_orders.php` uses the same-named constant as a genuine
+   **dry run**. So the reasonable assumption — "same constant, same meaning, safe to leave on while I
+   watch it" — is **wrong here, and it deletes**. This is a genuine trap, not a style inconsistency:
+   verify per file before relying on it.
+2. **The INSERT omits seven columns:** `forecastCategoryId`, `salesStageId`,
+   `percentToCloseStatusId`, `projectedProfit`, `leadSource`, `clickupTaskId`, and now
+   `endCustomerName`. Rows the cron recreates therefore come back stripped of all seven — the same
+   delete-and-reinsert failure shape as the `locationId` case above.
+3. **Empty `sql_mode` in prod turns omission #2 into silent corruption.** `projectedProfit` is
+   NOT NULL with **no default**, so instead of rejecting the INSERT, MySQL silently stores `0.00`.
+   **This is the root cause of the perpetual checker↔fixer churn** this doc has tracked for months:
+   the checker flags a profit delta, the fixer "repairs" the row, the cron zeroes it again the next
+   night.
+
+**And the reconciler cannot compute `projectedProfit` at all.** It fetches **no line items** — zero
+references to `itemList` or `costEstimate` anywhere in the file — while opportunity profit is
+Σ(amount − costEstimate) over qualifying lines (the formula recorded above). So it cannot fix the
+column it breaks, and there is no version of "just add `projectedProfit` to the INSERT" that works
+without adding line-item fetching first.
+
+**Agreed direction (2026-09-04): the reconciler should STOP INSERTing and report instead.** A
+component that cannot compute a NOT-NULL column should not be authoring rows. Detail in
+[the bidirectional opportunity-sync design](../../../../2.0/apps/worker2/features/netsuite-clickup-opportunity-bidirectional-design.md).
+
 ## Data model
 
 `Forecast.Sales`, `Forecast.OpenOrderItems` on the **core2** cluster
@@ -654,6 +699,21 @@ None — Forecast2 is a single shared dataset.
   sequence the rotation against whatever else reads that replica.
 
 ## Change history
+- 2026-09-04 — **Investigation only; no code changed (a prototype fix was written and deliberately
+  reverted).** Recorded a **live data-loss path**: the nightly
+  `periodic_forecast_discrepancy_fix_opportunities.php` (`0 4 * * *`, active) and
+  `tools/bin/forecast/fixer.php::fixOpportunities()` build their delete set **without checking
+  `clickupTaskId`** — **746 of 26,766 in-window rows carried a task id** (one already stranded at
+  `PENDING`), so a delete **orphans the ClickUp task and causes duplicate task creation**, and a
+  single truncated `listOpportunities()` page is enough to make every unreturned opportunity look
+  stale. Three more defects in the same file: **`MAINTENANCE_MODE = true` gates logging only, NOT
+  writes** (unlike its `..._open_orders.php` sibling, where it *is* a dry run — a genuine trap); the
+  INSERT **omits** `forecastCategoryId`, `salesStageId`, `percentToCloseStatusId`, `projectedProfit`,
+  `leadSource`, `clickupTaskId` and `endCustomerName`; and because prod `sql_mode` is **empty**,
+  `projectedProfit` (NOT NULL, no default) silently lands as **`0.00`** — the **root cause of the
+  long-standing checker↔fixer churn**. Also established that the reconciler **cannot compute
+  `projectedProfit` at all** (it fetches no line items — zero `itemList`/`costEstimate` references),
+  so the agreed direction is that it **stops INSERTing and reports** instead. (ajean)
 
 - 2026-08-28 — **⚠ Security finding (not fixed):** the `tools` repo copies of these scripts —
   `tools/bin/forecast/fixer.php:1424` and `tools/bin/forecast/checker.php:678` — hardcode **live

package/knowledge/1.0/apps/worker/workflows/tracing-a-worker-cron-run-in-production.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: workflow
 status: active
-updated: 2026-09-01
+updated: 2026-09-04
 owners: ["bala", "kyalamarthi"]
 files:
   - worker/ebs/cron.worker.php
@@ -170,7 +170,28 @@ to **2.0 only**. The 1.0 standard wins in `library/`, `worker/` and every other
 `library/app/systemmonitor/500error.php` uses `str_contains()`; that file is not a version
 signal, it is a latent bug.
 
+## Deploy-time trap: a `&` (or any shell metacharacter) in the cron FILENAME silently kills it
+`ebs/cron.worker.php:41` builds each crontab line by plain string concatenation
+(`'php /var/www/html/crons/' . $cron->cron`) with **no quoting and no escaping**. Cron then parses
+that line as a shell command, so a `&` in the filename is read as *"run everything before this in
+the background and start a new command."*
+
+Worked example (found 2026-09-03): a developer renamed the Compass Canada status cron to
+`crons/toga2/compasscanada/update_salesorder_status_from_g&t.php` and repointed
+`schedules/cron.worker.sync.json` at it. The crontab line becomes two commands —
+`php .../update_salesorder_status_from_g` (a file that does not exist) and a stray `t.php` — so the
+real script **never executes**, on every tick, with no error anywhere. It looks exactly like
+"the cron never fired".
+
+**Rule: a cron filename must be plain `[a-z0-9_]` plus `.php`.** No `&`, no spaces, no `;`, `|`,
+`$`, `(`, `)`, `*`, `?`, or quotes. The *display name* in `schedules/*.json` is free text and may
+contain anything; only the `cron` path is shell-parsed.
+
 ## Change history
+- 2026-09-04 — Added the **`&`-in-a-cron-filename trap**: `ebs/cron.worker.php:41` concatenates the
+  path into the crontab line unquoted, so a shell metacharacter splits the command and the script
+  never runs while leaving no trace. Found via the renamed Compass Canada
+  `update_salesorder_status_from_g&t.php`. (bala)
 - 2026-09-01 — Added **Step 6: check the email log, not the inbox** — a 1.0 cron's sends land in
   the **legacy** env's `Logs.Emails` (schema `Logs`, table `Emails`), never prod `Logs` or
   `Logs_<Client>`; gave the verification query, and noted that a missing row usually means the job

package/knowledge/2.0/apps/_underscore/INDEX.md

@@ -7,7 +7,7 @@
 | [ACL Permission Chain (Record & Field Authorization)](features/acl-permission-chain.md) | Authorization in the 2.0 API is **metadata-driven**: whether a role may Create/Read/Update/Delete a record is decided by rows across **four linked tables**, not | api2/Component/Api/V2/V2.php, dbchanges2/Core/2026-08-27a - TransferOrderUuidSearchableIdentifier.sql, _underscore/Model/Core/Page.php, _underscore/Model/Core/Surface.php, _underscore/Model/Client/TrackingNumber.php, dbchanges2/Client/2026-06-03- BLANK_CLIENT_DATABASE.sql, dbchanges2/Client/2026-06-23b - ItemTranslationsAcl.sql, dbchanges2/Client/2026-07-02c - TrackingNumberNeedsReturnLabelFieldPermission.sql, dbchanges2/Client/2026-07-22c - TrackingNumberMeasureIdsFieldPermission.sql, dbchanges2/Client_Quad/2026-08-24 - Quad Multi Currency Item Pricing.sql, dbchanges2/Client/2026-08-29 - ItemReceiptTransferOrderIdFieldPermission.sql, dbchanges2/Client/2026-08-29b - ItemReceiptItemTransferOrderItemIdFieldPermission.sql, dbchanges2/Client/2026-09-01a - PurchaseOrderItemQtyFieldsApiRoleRead.sql |
 | [Address Uniqueness Normalization (unit identifier + 5-digit ZIP comparison)](features/address-uniqueness-normalization.md) | When a business rule says *"only one X per physical address"*, comparing address rows field-for-field does **not** work: the same dwelling is spelled many diffe | _underscore/Model/Rate/Entitlement.php |
 | [Address Validation (carrier waterfall + validateAddress scripted endpoint)](features/address-validation.md) | `_Model_Client_Address::validateAddress` verifies a US address against a **carrier waterfall (USPS → FedEx → UPS)** and returns a single canonical, carrier-norm | _underscore/Model/Client/Address.php, _underscore/Component/Library/Carriers/Usps/Usps.php |
-| [_ApiRequest — JSON encode/decode & api-logging behavior](features/apirequest-json-content-type.md) | `_ApiRequest` is the 2.0 outbound HTTP client. | _underscore/ApiRequest.php |
+| [_ApiRequest — JSON encode/decode & api-logging behavior](features/apirequest-json-content-type.md) | `_ApiRequest` is the 2.0 outbound HTTP client. | _underscore/ApiRequest.php, _underscore/Model/Client/Logs/Api.php, _underscore/Component/Api/Clickup/Clickup.php |
 | [Assortment Name Translation (AssortmentTranslations sidecar)](features/assortment-name-translation.md) | Serves Assortment (product-grouping) **names** in multiple languages by adding a per-language **sidecar** table `AssortmentTranslations`, reusing the platform's | _underscore/Model/Client/AssortmentTranslation.php, dbchanges2/Client/2026-06-26a - AssortmentTranslations.sql, dbchanges2/Core/2026-06-26a - AssortmentTranslationsRecord.sql, dbchanges2/Client/2026-06-26b - AssortmentTranslationsAcl.sql |
 | [Asynchronous Query Execution (writes-only, via Worker)](features/async-query-execution.md) | `_Query` can run a **write** query asynchronously so a long/slow write does not hold a request-scoped DB connection open long enough to hit **"MySQL server has | _underscore/Query.php, worker2/Worker/Infrastructure/Database.php, worker2/Worker/Team/Transcripts.php |
 | [FIELD_AUTOINCREMENT record numbering (SA / TA prefixes, NULL-only trigger, no padding)](features/autoincrement-record-numbering.md) | A human-facing record number (`SA100001`, `TA100000`) is generated by the ORM, not by MySQL. | _underscore/Model.php, _underscore/Model/Client/SalesOrder.php, _underscore/Model/Client/TransferOrder.php |

package/knowledge/2.0/apps/_underscore/architecture.md

@@ -6,8 +6,8 @@ project: _Underscore
 client: shared
 type: architecture
 status: active
-updated: 2026-08-26
-owners: ["jcardinal", "rgirish", "mhammontree"]
+updated: 2026-09-04
+owners: ["jcardinal", "rgirish", "mhammontree", "ajean"]
 files:
   - _underscore/_underscore.php
   - _underscore/Loader.php
@@ -31,7 +31,10 @@ Minimum PHP 8.1 (enforced in `_underscore.php`). The current architecture is
 **Critical rules:** framework classes use a leading-underscore prefix (`_Model_*`, `_Controller_*`,
 `_Worker_*`). Always use parameterized queries / the `_Db` layer — never interpolate input into SQL.
 Beware the **lazy-transaction gotcha**: writes issued outside an explicitly committed transaction
-can be silently dropped — confirm commit semantics before relying on a write.
+can be silently dropped — confirm commit semantics before relying on a write. Two corollaries:
+**never hold a DB transaction open across an external HTTP call**, and **an empty log table is not
+proof a code path never ran** — uncommitted rows vanish, so absence of log rows means "no committed
+write", not "no execution".
 The Surface presentation layer replaces `Page::meta()` but `meta()` must NOT be deleted until
 every page is cut over (per-surface, with a parity diff); config describes, PHP decides —
 business logic never moves into Surface config.
@@ -369,6 +372,18 @@ multi-file UI components (`.php`/`.html`/`.css`/`.js`) invoked as `<_ComponentNa
   pin is `Addresses.isValidated` and `Entitlements.serviceAddressId` both-or-neither. Relevant code:
   `_underscore/Model/Rate/Entitlement.php`, `api2/Component/Api/V2/V2.php::getFullModelData`.)
 - **`_Database::register()` auto-starts a lazy transaction (since Apr 2 2026, commit `fa7835ed`).** Any code that calls `register()` and then writes to that DB must call `_Database::transactionCommit()` before the request ends — otherwise MySQL silently rolls back all writes when the connection closes. Lazy transactions only materialise on the first write, so read-only callers are unaffected. See `_underscore/Database.php:48`. First discovered when Rate SAML user provisioning silently discarded all new user INSERTs (Jun 2026).
+  - **Never hold a DB transaction open across an external HTTP round-trip.** Beyond pinning a
+    connection for the duration of a third-party call, it means the log/telemetry rows you are
+    writing *about* that call are still uncommitted while it runs. Live instance found 2026-09-04:
+    `_Worker_Clickup_Opportunity::logged()` opens `DB_LOGS`, never commits, and holds the
+    transaction across every ClickUp/NetSuite call.
+  - **An empty log table is NOT proof that a code path never ran.** This is the diagnostic trap the
+    gotcha creates, and it has already produced one wrong conclusion in this knowledge base: a
+    zero-row `Logs.Api` census was read as "this integration has been dead since it shipped," when
+    the writes were simply never committed. Before concluding a path is dead from absent rows,
+    confirm the writer commits. Correct pattern:
+    `worker2/Worker/Netsuite/SalesOrder.php:441`. Case detail:
+    [netsuite-opportunity-sync](../worker2/features/netsuite-opportunity-sync.md).
 - **`_Model::load()` with a collapsed WHERE clause silently full-table-scans → OOM (guarded 2026-08-26, `Model.php`).** `_Model::search()` builds `SELECT <pk> FROM <table> [WHERE …] ORDER BY <pk>` and emits the WHERE **only if at least one search term survives**. `buildSqlFieldValue()` **drops** a term (returns false) when a set field's value is `null` and its type is a plain `FIELD_CHAR` — i.e. not one of the special-cased types (`AUTOINCREMENT`, `DATETIME_CREATED`, `DATETIME_UPDATED`, `SQL`, `STORAGE`, `CHAR_UUID`). If the only set search field collapses this way, the WHERE vanishes and `search()` selects the **whole table**, constructing one model per row → memory exhaustion on a large table. Because `load()` calls `search()` first and only then checks `count() == 1`, it **OOMs building the result array before it ever counts** — the empty-WHERE case never reaches the "not exactly one" branch. **`load()` means "find exactly one by criteria"; an empty WHERE is always a bug, never a full-table scan.** **Fix (commit `0b944a51`, `_production`):** `load()` now guards at the top — if no set search field yields a WHERE term (via `buildSqlFieldValue`), it `error_log`s and returns `false` (or throws when `throwExceptionIfNotFound`) instead of scanning. `search()` is unchanged, so intentional "list all rows" callers of `search()` are unaffected. **Triage:** a PHP OOM stack trace of `Model.php search() → __construct` that is *shallow* (a single search→construct, not deeply repeated) is a runaway result set from a collapsed WHERE, **not** recursion — confirm by checking whether the single set search field is a `FIELD_CHAR` whose value is null. (Root cause of a recurring prod OOM: `Logs.Issue` ref `2L` (#38), first seen 2026-08-03, 136 occurrences, trace `Model.php search()→__construct`.)
 - **PHP "Unclosed '{'" parse errors report a MISLEADING line number.** When a `.php` file loaded by the autoloader (`Loader.php`) has a dropped/unbalanced brace, PHP reports `Unclosed '{' on line N` where N is the **outermost `class X {` line** and fails at EOF — NOT at the true location of the missing `}`. Worse, because the file loads lazily via the SPL autoloader, the runtime trace points at the **caller** that triggered the autoload (e.g. a `new _Email()` call site), not the broken file. **Triage rule:** for an "Unclosed '{'" error, the real culprit is a missing `}` somewhere between the reported line and EOF of the file that failed to load — run `php -l <file>` (it reports the EOF line) and scan the whole file. **Merge-conflict resolutions are a common source of a single dropped brace** — review the entire merge, not just the one file the error appears to name. (First hit: production 500 EO-1, Jul 2026 — a `}` dropped from `_Email::send()` during merge `685e4a14` surfaced as a trace pointing at the `new _Email()` caller.)