toga-ai 1.0.976 → 1.0.977

package/knowledge/1.0/apps/library/INDEX.md

@@ -24,6 +24,7 @@
 | [NetSuite classification → ItemClasses sync (opt-in per client)](features/netsuite-item-class-sync.md) | `App_Api_Toga2::getCreateItemClass()` mirrors NetSuite's **classification tree** into the client's `ItemClasses` table and stamps `Items.itemClassId`. |
 | [Items.description from the NetSuite item record (Sales Description), refreshed every sync](features/netsuite-item-description-sync.md) | How the NetSuite sync sets and refreshes `Items.description` (all NetSuite clients); open when a toga item description is blank, stale, or differs from NetSuite |
 | [isFulfillable from NetSuite during Item Sync (Phase 1)](features/netsuite-item-isfulfillable-sync.md) | 1.0 Phase-1 stamping of the NetSuite `isfulfillable` flag onto the Agilant source item during sync; open before changing fulfillability sync or a client overrid |
+| [NetSuite SOAP Toolkit (NetSuiteService) — search result shape, status, externalId lookup](features/netsuite-soap-toolkit-search.md) | How 1.0 SOAP searches (`new NetSuiteService()`, `TransactionSearch`) really return data: array/null result shape, empty `orderStatus`, class loading, tranId vs |
 | [NetSuite SuiteQL/REST API Reference](features/netsuite-suiteql-api-reference.md) | Working reference for the Agilant NetSuite REST/SuiteQL integration (`App_Api_Netsuite_Rest`): auth, SuiteQL mechanics, table schemas, type/status codes, perfor |
 | [NetSuite SuiteQL/REST Shim — Field Semantics](features/netsuite-suiteql-rest-shim.md) | Non-obvious NetSuite SuiteQL/REST field semantics behind `App_Api_Netsuite_Rest` shims (signs, `tl.id`==line, `iscogs`, ShipItem cost, `createdFrom`, shim-shape |
 | [NetSuite Sync Alert Monitor (App_SystemMonitor_NetSuiteIntegration)](features/netsuite-sync-alert-monitor.md) | 1.0 monitor `App_SystemMonitor_NetSuiteIntegration` — flags stale per-client NetSuite sync checkpoints (>48h) to ClickUp; open when a sync-stale alert is missin |

package/knowledge/1.0/apps/library/features/db-connection-lifecycle-and-reconnect.md

@@ -6,8 +6,8 @@ project: Library
 client: shared
 type: feature
 status: active
-updated: 2026-09-25
-owners: ["rgirish", "jcardinal"]
+updated: 2026-10-08
+owners: ["rgirish", "jcardinal", "bala"]
 files:
   - library/app/database.php
   - library/app/query.php
@@ -29,7 +29,7 @@ How a 1.0 mysqli connection dies during a long external call ("MySQL server has
 
 That shape produces the worst class of bug we have: **the remote system did the work, our stamp never landed, and the next run redoes it.** Two confirmed incidents — the Compass ODP duplicate NetSuite sales orders (2026-09-22) and the NetSuite → TOGa Supply item-fulfillment doom loop (2026-09-02).
 
-**Rule for any long-running cron:** a slow external call between two DB writes needs `App_Database::keepAlive()` around it. Do not rely on `App_Query`'s reconnect, and do not rely on `try`/`catch`.
+**Rule for any long-running cron:** a slow external call between two DB writes needs `App_Database::keepAlive()` around it. Do not rely on `App_Query`'s reconnect, and do not rely on `try`/`catch`. For create-then-stamp jobs, the strongest guard is an id you set on the remote record (e.g. NetSuite `externalId`) plus a lookup before sending, so an unstamped record is found, not re-created (Compass ODP cron 5, 2026-10-08).
 
 **`wait_timeout` on the production client cluster is 180 seconds** — three minutes, not the MySQL default of 8 hours. Verified 2026-09-22. Assume any external call that can exceed three minutes will kill the connection.
 
@@ -87,7 +87,9 @@ It returns **`false`** (rather than throwing) when the link has no config entry,
 
 When a cron creates a record in an external system and then stamps our DB, a dead connection at the stamp step produces a **silent duplicate**. `register_shutdown_function()` is the **only** code that still runs after `App_Error::handleException()` calls `exit`, so it is the only place an alert can fire.
 
-The alert pattern as shipped in Compass ODP cron 5 (`worker` commit `afce32ca`). Cron 5 has **no `keepAlive()` call** — it stamps via `App_Database::queryForked($sql, $clientToga2DatabaseLink)` (corrected 2026-09-25):
+**⚠ Prefer a remote-side lookup over this alarm (2026-10-08).** The alarm only knows "the process died while armed". A warning *inside* the external call (e.g. `SoapClient` SSL "Connection reset by peer") also `exit`s, so the alarm fires when nothing was created. Compass cron 5 hit exactly that on SA138624 and removed the alarm in `worker` `fd426d9b` for an externalId lookup ([design](../../../../clients/compass-usa/workflows/odp-order-pipeline-to-netsuite.md#cron-5-duplicate-proof-send--externalid--lookup)). If you still use the alarm, never let its message tell people to stamp by hand.
+
+The alert pattern as first shipped in Compass ODP cron 5 (`worker` commit `afce32ca`, since removed). Cron 5 has **no `keepAlive()` call** — it stamps via `App_Database::queryForked($sql, $clientToga2DatabaseLink)` (corrected 2026-09-25):
 
 1. **Arm** an `$unstampedNetsuiteOrder` array immediately *before* the external `add()`.
 2. **Clear** it right after the stamp is handed to `queryForked()`, and also when the remote system *rejects* the order (nothing was created, so nothing is orphaned). **⚠ The forked write is not confirmed** (code comment says so): a failed fork leaves the order sent, unstamped, and the alarm already off → possible duplicate on the next run. Cron 5 relies on the [NetSuite duplicate SO monitor](../../../../clients/compass-usa/features/oneuptime-netsuite-duplicate-sales-orders-monitor.md) as backstop. If you need a confirmed stamp, clear only after a synchronous write succeeds.
@@ -111,5 +113,6 @@ The alert pattern as shipped in Compass ODP cron 5 (`worker` commit `afce32ca`).
   - `library/app/error.php:5-8` has `IMMEDIATELY_TERMINATE_INSTANCE_IF_ERROR_STRING_CONTAINS`, matched **on substring before severity**. A recoverable warning could terminate an EC2 instance.
 
 ## Change history
+- 2026-10-08 — Shutdown alarm can fire falsely (warning inside the SOAP call exits too); Compass cron 5 replaced it with an externalId lookup (`fd426d9b`). Added the remote-id rule to the Summary. (bala)
 - 2026-09-25 — Corrected the cron 5 pattern: it stamps via `queryForked` with no `keepAlive`, and disarms the alarm before the forked write is confirmed; noted the monitor backstop. (rgirish)
 - 2026-09-22 — Doc created. Root-caused the "gone away" class: warning fires at `mysqli_select_db` before the 2006 check, `databaseClose()` never cleared `$connections`, `$link` captured above the retry loop. Added `App_Database::reconnect()` + `queryForked()` (`library` commit `60fdb888`) and documented the shutdown-handler alert pattern. Decided against raising `wait_timeout` from 180 s. (rgirish)

package/knowledge/1.0/apps/library/features/netsuite-soap-toolkit-search.md

@@ -0,0 +1,46 @@
+---
+title: "NetSuite SOAP Toolkit (NetSuiteService) — search result shape, status, externalId lookup"
+framework: "1.0"
+repo: library
+project: Library
+client: shared
+type: feature
+status: active
+updated: 2026-10-08
+owners: ["bala"]
+files:
+  - library/netsuitetoolkit/NSPHPClient.php
+  - library/app/netsuite.php
+  - worker/crons/toga2/compass/workflow/5_create_netsuite_sales_orders_from_office_depot_purchase_orders.php
+related:
+  - ./netsuite-suiteql-rest-shim.md
+  - ./db-connection-lifecycle-and-reconnect.md
+  - ../../worker/features/canon-received-po-netsuite-order-sync.md
+  - ../../../../clients/compass-usa/workflows/odp-order-pipeline-to-netsuite.md
+---
+How 1.0 SOAP searches (`new NetSuiteService()`, `TransactionSearch`) really return data: array/null result shape, empty `orderStatus`, class loading, tranId vs internalId, and how to look a record up by `externalId`; open before writing or reading any SOAP search in `library` / `worker`.
+
+## Summary
+- 1.0 talks to NetSuite SOAP through the vendored SuiteTalk toolkit (`library/netsuitetoolkit/`) and the `App_NetSuite` wrappers (`library/app/netsuite.php`). The REST/SuiteQL path is separate: [SuiteQL/REST shim](./netsuite-suiteql-rest-shim.md).
+- Each fact below was a wrong assumption on 2026-10-08 (Compass ODP cron 5 duplicate-proof send). Code that ignores them reads empty values or crashes, and in 1.0 a PHP warning is a fatal `exit` ([why](./db-connection-lifecycle-and-reconnect.md)).
+
+## How it works
+- **Result shape.** `NSPHPClient` sets `SOAP_SINGLE_ELEMENT_ARRAYS` (`NSPHPClient.php:200`), so `$response->searchResult->recordList->record` is an **array even for one hit**, and **null for zero hits**. Read it as `->record ?? []`. `App_NetSuite::getSalesOrdersByCustomerPo()` returns that raw value (array or null), first page only.
+- **Classes load in the constructor.** `SearchStringField`, `TransactionSearchBasic`, the `*Operator` enums etc. exist only after `new NetSuiteService()` has run. Construct the service first, then build the search.
+- **Lookup by externalId.** `TransactionSearchBasic->externalIdString` = `SearchStringField` with `operator = SearchStringFieldOperator::is`, `operatorSpecified = true`, plus `type` = `[TransactionType::_salesOrder]`. Working example: `findNetsuiteSalesOrderInternalIdsByExternalId()` in Compass cron 5.
+- **Lookup by customer PO.** `App_NetSuite::getSalesOrdersByCustomerPo($otherRefNum, $customerInternalIds)` searches `otherRefNum` `equalTo`, optionally restricted to `entity` anyOf.
+
+## Gotchas
+- **`orderStatus` is NULL on basic `TransactionSearch` results.** Use the `status` label string instead: `'Pending Fulfillment'`, `'Billed'`, `'Closed'`, `'Cancelled'`. Compare against named constants.
+- **tranId is not internalId.** `tranId` is the SO number people see in NetSuite (e.g. `292731`); `internalId` is the record key (`7586315`). Store and stamp `internalId`; put `tranId` in any email or message ops will read.
+- **`memo` is unreliable as a match key.** It is often NULL or hand-edited in NetSuite. Use it only as a tie-breaker (cron 5 uses it only when `otherRefNum` was cut at 45 chars), never as the only key.
+- **An `otherRefNum` list is not a stable key.** Prefer an `externalId` you set yourself on create. See the [Compass cron 5 design](../../../../clients/compass-usa/workflows/odp-order-pipeline-to-netsuite.md#cron-5-duplicate-proof-send--externalid--lookup).
+
+## Change history
+- 2026-10-08 — Doc created from the Compass cron 5 externalId work: result shape, NULL `orderStatus`, class loading, tranId vs internalId, memo unreliability, externalId search recipe. (bala)
+
+## Related
+- [SuiteQL/REST shim](./netsuite-suiteql-rest-shim.md)
+- [1.0 MySQL connection lifecycle](./db-connection-lifecycle-and-reconnect.md)
+- [Canon received-PO sync](../../worker/features/canon-received-po-netsuite-order-sync.md) (same array-shape fact, ruled out there)
+- [Compass ODP pipeline to NetSuite](../../../../clients/compass-usa/workflows/odp-order-pipeline-to-netsuite.md)

package/knowledge/INDEX.md

@@ -4,7 +4,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 
 ## 1.0 framework
 
-- **library** (Library) _(framework core)_ — 33 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
+- **library** (Library) _(framework core)_ — 34 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
 - **worker** (Worker) — 42 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
 - **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
 - **worker1.5** (Worker 1.5) — 0 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)

package/knowledge/clients/compass-usa/workflows/odp-order-pipeline-to-netsuite.md

@@ -6,7 +6,7 @@ project: Worker
 client: compass-usa
 type: workflow
 status: active
-updated: 2026-09-25
+updated: 2026-10-08
 owners: ["rgirish", "bala", "dfranks", "jcardinal", "mhammontree"]
 files:
   - worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php
@@ -31,6 +31,7 @@ related:
   - clients/compass-usa/features/mits-po-to-so-item-linking.md
   - clients/compass-usa/profile.md
   - clients/compass-usa/features/oneuptime-netsuite-duplicate-sales-orders-monitor.md
+  - 1.0/apps/library/features/netsuite-soap-toolkit-search.md
 ---
 The end-to-end Compass ODP order → NetSuite pipeline through the numbered 1.0 worker crons, plus the "this order never reached NetSuite" diagnostic method; open for any ODP-to-NetSuite tracing or cron-mechanics question.
 
@@ -59,7 +60,7 @@ ODP SalesOrder (customerId = 1)
 3. **`workflow/3a_import_office_depot_purchase_orders.php`** — pulls the ODP **850** from S3 bucket `agilant-as2`, prefix `OfficeDepot/` (excluding `OUTBOX/` and `SENT/`; `:46` S3 read, `:54-58` prefix filters) and creates the downstream **ODP SalesOrder** (customerId=1 = Office Depot). **Deletes the S3 object after processing** — so an absent S3 object is expected once ingested, not evidence of a miss.
    - **⚠ Corrected 2026-08-28 — the delete is now CONDITIONAL.** 3a used to delete **unconditionally**, so a failed import left nothing to retry and **silently destroyed the order**. It now deletes only when **every PO in the file imported end to end**; otherwise the file stays in `OfficeDepot/` and the next `*/5` run retries it. Which failures count, the reject-855 / error-email / file-log guards, and the retries-forever limitation: [ODP EDI File Retention & Import Retry](../features/odp-edi-file-retention-and-retry.md). **Not deployed as of 2026-08-28** — production still deletes unconditionally, so treat any pre-deploy incident as a lost file.
 4. **`workflow/4_transmit_office_depot_po_acknowledgements.php`** (every 15 min) sends the **accept** X12 **855** for imported ODP `PurchaseOrders` with `dtAcknowledged IS NULL`, then stamps `dtAcknowledged`. It delegates 855 construction to `App_Edi::buildOfficeDepotPurchaseOrderAcknowledgement855()` (accept mode). A **reject** 855 is instead sent **inline from cron 3a** when the over-quantity guard rejects a PO (a rejected PO is never persisted, so cron 4 can never see it). Cron 3a also runs the over-quantity guard (`officeDepotPurchaseOrderExceedsCompassDemand()`) after the same-PO-number de-dupe and before any SO/PO creation, rejecting a whole PO when ODP re-sends an already-processed Compass SO line under a **new** PO number. See [ODP EDI 855 Acknowledgement + Over-Quantity Guard](../features/odp-edi-855-acknowledgement-and-overquantity-guard.md).
-5. **`workflow/5_create_netsuite_sales_orders_from_office_depot_purchase_orders.php`** — scheduled **every 10 minutes** (`*/10`, verified against `worker/schedules/` on 2026-08-04; this doc previously said "hourly"); picks up ODP SalesOrders (`OfficeDepotSalesOrders.customerId = 1 AND c_dtTransmittedToNetsuite IS NULL`), creates the SO in NetSuite, and writes back `c_dtTransmittedToNetsuite` + `c_netsuiteInternalSalesOrderId` **onto the ODP SalesOrder (customerId=1), never the Compass SO (customerId=2)**. Exclusion filters (an order legitimately waiting is not a bug):
+5. **`workflow/5_create_netsuite_sales_orders_from_office_depot_purchase_orders.php`** — scheduled **every 10 minutes** (`*/10`, verified against `worker/schedules/` on 2026-08-04; this doc previously said "hourly"); picks up ODP SalesOrders (`OfficeDepotSalesOrders.customerId = 1 AND c_dtTransmittedToNetsuite IS NULL`), creates the SO in NetSuite, and writes back `c_dtTransmittedToNetsuite` + `c_netsuiteInternalSalesOrderId` **onto the ODP SalesOrder (customerId=1), never the Compass SO (customerId=2)**. Before sending it checks NetSuite for the order by `externalId` so a lost stamp never creates a duplicate: [duplicate-proof send](#cron-5-duplicate-proof-send--externalid--lookup). Exclusion filters (an order legitimately waiting is not a bug):
    - `CompassSalesOrders.number NOT LIKE 'MA%'` — MA orders excluded. **They are keyed into NetSuite by hand** (~20 min each), and the worker2 `Monitor/Compass/OfficeDepotNetsuiteIntegration` monitor carries the **same** `NOT LIKE 'MA%'` clause — so MA orders are invisible to both. Coverage is [cron 7's exception report](../../../1.0/apps/worker/features/compass-ma-sales-order-exception-report.md) plus, since 2026-09-08, the dedicated [MA refresh-order monitor](../features/oneuptime-ma-refresh-order-monitor.md).
    - the two-part **release gate** below (usually releases on the **next 10-minute run**; the 30-minute timeout is the fallback, not the normal path).
    - **computer-kit orders** (Bundles 187–196) wait for a **2nd PO** before syncing. This one is **not** in the `WHERE` clause — it is a per-row check run in PHP *after* the query (`Bundles.number IN (192, 187, 188, 195, 193, 189, 194, 190, 191, 196)`, then a count of that Compass SO's POs), so such an order is selected and then skipped.
@@ -111,13 +112,25 @@ Two different Compass ODP error emails exist, from **two different pipeline stag
 1. **`Item not found in NetSuite: <partNumber>`** — the SKU resolved to no NetSuite internal id.
 2. **NetSuite `add()` rejection** — NetSuite refused the SalesOrder (the Ext Price root cause below, the common case).
 
-**Root cause of the recurring "Please enter a value for Ext Price" rejection:** in cron 5's item loop, `$soItem->rate` (NetSuite's "Ext Price") is set **only** when the SKU resolves as a NetSuite **item group** via `App_NetSuite::getItemGroupInternalIdFromPartNumber()`. For a **plain item** resolved via `App_NetSuite::getItemInternalIdFromPartNumber()`, no rate is ever set, so NetSuite's `add()` rejects the whole order with USER_ERROR **"Please enter a value for Ext Price."** The order then stays stuck: cron 5 only picks ODP SalesOrders with `c_dtTransmittedToNetsuite IS NULL` and **re-attempts (and re-emails) every hour** until a success stamps `c_dtTransmittedToNetsuite` + `c_netsuiteInternalSalesOrderId`. A repeating hourly "SO failed in NetSuite" email for the same order is this loop, not a new failure each time.
+**Root cause of the recurring "Please enter a value for Ext Price" rejection:** in cron 5's item loop, `$soItem->rate` (NetSuite's "Ext Price") is set **only** when the SKU resolves as a NetSuite **item group** via `App_NetSuite::getItemGroupInternalIdFromPartNumber()`. For a **plain item** resolved via `App_NetSuite::getItemInternalIdFromPartNumber()`, no rate is ever set, so NetSuite's `add()` rejects the whole order with USER_ERROR **"Please enter a value for Ext Price."** The order then stays stuck: cron 5 only picks ODP SalesOrders with `c_dtTransmittedToNetsuite IS NULL` and **re-attempts (and re-emails) every `*/10` run** until a success stamps `c_dtTransmittedToNetsuite` + `c_netsuiteInternalSalesOrderId`. A repeating "SO failed in NetSuite" email for the same order is this loop, not a new failure each time.
 
 Error-email rework (2026-08-03, cron 5):
 - New helper **`buildNetsuiteErrorMessage(AddResponse $response)`** walks `writeResponse->status->statusDetail[]` and returns `NetSuite rejected the order - <CODE>: <message>` (multiple details joined by `; `) — replacing the old whole-`AddResponse` `print_r` dump.
 - The email body appends the **no-rate item part numbers** (collected in `$itemsWithoutRate` whenever `$soItem->rate` is unset) after the error line — the Ext Price culprits.
 - An attachment **`NetSuiteError_<SO>.txt`** (written to `sys_get_temp_dir()`) carries the Compass SO/PO, the Office Depot PO, and the full NetSuite request (SalesOrder) + response dump, attached via `App_Email_Agilant->addAttachment()`.
 
+### Cron 5 duplicate-proof send — externalId + lookup
+Shipped in `worker` `fd426d9b` (2026-10-08, `_production`); **pending EB deploy**. Replaces the shutdown alert as the duplicate guard. Per ODP SalesOrder:
+1. Set `externalId = 'toga2-compass-odso-<ODP SalesOrders.id>'` (`NETSUITE_EXTERNAL_ID_PREFIX`). Keyed on the ODP SO id because it is stable. `otherRefNum` is **not**: it is a `GROUP_CONCAT` of the still-unsent sibling ODP POs cut to 45 chars, so it changes as siblings send.
+2. `findNetsuiteSalesOrderInternalIdsByExternalId()` (`externalIdString`, operator `is`). Hit → stamp that id, do not send.
+3. No hit → fallback for orders sent before externalId existed: `findNetsuiteSalesOrderInternalIdsByPurchaseOrder()` → `App_NetSuite::getSalesOrdersByCustomerPo(otherRefNum, 35581)`. Keeps only SOs with **no** externalId and status label not `Closed`/`Cancelled`; if the PO list was cut at 45 chars, the Compass PO memo must also match. Hit → stamp, and email ops to confirm the match (on a PO match or more than one hit).
+4. Neither → `add()`. If it fails or returns no usable internalId, search by externalId again (a rejected add can mean "already exists") and stamp if found; otherwise the error email.
+- Stamp = `stampSalesOrderAsTransmittedToNetsuite()`, a forked `UPDATE` (`queryForked`) of `c_dtTransmittedToNetsuite` + `c_netsuiteInternalSalesOrderId`. Still unconfirmed, but a lost stamp now heals on the next run via step 2. The id goes into `$skippedSalesOrderIds` so the same run's loop does not re-pick it before the fork lands.
+- Removed: the `register_shutdown_function` alert and the `try`/`catch` around `add()`. An exception now stops the run; the next run recovers via the lookup.
+- **Decision 2026-10-08 (independent CTO opinion): `add()` + lookup, NOT `upsert()`.** `upsert()` by externalId would overwrite SOs ops edited or that are already fulfilled/billed. Supersedes the 2026-09-25 "monitor only" decision.
+- **Open at capture:** NetSuite sandbox check that a second `add()` with the same externalId is rejected; the ops-email text in the cron still names the Toga SO id and internalId (reword per the ops-email rule in Gotchas).
+- SOAP search facts this relies on (array/null results, NULL `orderStatus`, tranId vs internalId): [NetSuite SOAP toolkit](../../../1.0/apps/library/features/netsuite-soap-toolkit-search.md).
+
 ### Constants & identities (`library/app/client/compass.php`)
 - `App_Client_Compass::VENDOR_ID__OFFICE_DEPOT = 1` — `Vendors.id = 1` = "OFFICE DEPOT".
 - `App_Client_Compass::CUSTOMER_ID__OFFICE_DEPOT = 1` — `Customers.id = 1` = "Office Depot".
@@ -146,9 +159,11 @@ The NetSuite writeback lands on the **downstream ODP SalesOrder (customerId=1)**
   - **Mechanism.** Cron 5 holds one MySQL connection across the slow NetSuite SOAP `add()`. `wait_timeout` on the prod client cluster is **180 seconds**. A kit order's `add()` runs past that, so the connection is dead when the stamp `UPDATE SalesOrders SET c_dtTransmittedToNetsuite ...` runs (`crons/.../5_create_netsuite_sales_orders_from_office_depot_purchase_orders.php:433`). The order exists in NetSuite but is **unstamped**, so the next `*/10` run re-sends it.
   - **Proof, not inference.** `Logs.Issue` **769** (reference `W0`), *"mysqli_select_db(): MySQL server has gone away"*, first seen 2026-09-15, 33 occurrences; trace `query.php:168 → database.php:431 → cron 5 :433`. **All 15** duplicate NetSuite SOs are followed by a gone-away event 8-10 s later, zero misses; the 10 SOs that stuck have no matching error. **Logs are CDT, NetSuite is EST — apply the 1-hour offset before comparing timestamps.**
   - **Why kits.** `PC-KIT-OPT2` (10 expanded lines, all $12.58) was 10 of the 11 duplicated POs. The part number does not cause this — kits are the slow path, so they are the orders whose connection times out.
-  - **Fix (deployed in cron 5, `afce32ca`).** (a) The stamp runs via `App_Database::queryForked($sql, $clientToga2DatabaseLink)` in a separate process — **there is NO `keepAlive()` call** (corrected 2026-09-25; the first write-up said there was). (b) A `register_shutdown_function()` that emails devteam if the process dies with an order created in NetSuite but unstamped. Full mechanics: [1.0 MySQL Connection Lifecycle](../../../1.0/apps/library/features/db-connection-lifecycle-and-reconnect.md).
-  - **⚠ Remaining gap: the forked stamp is NOT confirmed** (the code comment says so), and the shutdown alarm is disarmed right after the fork. A failed forked write leaves the order in NetSuite, unstamped, with no alert → possible duplicate on the next run. The [NetSuite duplicate SO monitor](../features/oneuptime-netsuite-duplicate-sales-orders-monitor.md) is the backstop.
-  - **Decision 2026-09-25 — monitor only, no more prevention in cron 5.** Untaken option: set a NetSuite `externalId` on the SO so NetSuite rejects a second create. Declined because the script already has fixes for this.
+  - **Fix (cron 5, `afce32ca`).** The stamp runs via `App_Database::queryForked($sql, $clientToga2DatabaseLink)` in a separate process; **there is NO `keepAlive()` call**. The shutdown alert added with it was removed in `fd426d9b` (false alarm, see below). The unconfirmed-stamp gap is now closed by the [externalId lookup](#cron-5-duplicate-proof-send--externalid--lookup); the [NetSuite duplicate SO monitor](../features/oneuptime-netsuite-duplicate-sales-orders-monitor.md) stays as backstop. Mechanics: [1.0 MySQL Connection Lifecycle](../../../1.0/apps/library/features/db-connection-lifecycle-and-reconnect.md).
+- **⚠ False "SO created in NetSuite but NOT stamped" alert (2026-10-07, Compass SO `SA138624`).** `SoapClient::__doRequest(): SSL: Connection reset by peer` fired **inside** `add()`; 1.0 `App_Error::handleError` turned the warning into `exit`, so no `catch` ran. The shutdown alarm, armed before `add()`, then claimed the order was created and suggested a manual `UPDATE` stamp. NetSuite had created **nothing**; a later run created it once (NetSuite SO `292731`). Running that UPDATE would have marked a non-existent order as sent, and the order would have been lost. **Lesson:** an alarm armed around an external call cannot tell "created" from "connection dropped". Never tell ops to stamp by hand; ask NetSuite (externalId lookup). Alarm removed in `fd426d9b`.
+- **⚠ Never test cron 5 on beta or any non-prod box.** `$inProduction = true` is hardcoded (`:39`) and the NetSuite config points at the **prod** account `1095849`, so any run creates real prod NetSuite orders. Test lookup logic with a read-only script: search only, no `add()`, no stamp.
+- **⚠ "MySQL server has gone away" still kills runs (known, out of scope).** `Logs.Issue` `W0` reached **153** occurrences by 2026-10-07: after a slow NetSuite call the client link is dead and the run dies after ~1 order. With the lookup the order is not duplicated, only delayed to the next run.
+- **Ops emails from this cron:** name the Compass SO, the Office Depot PO and the NetSuite SO number (`tranId`, e.g. `292731`, not internalId `7586315`); say what happened and the one action. Never include SQL, externalIds or Toga internal ids.
 - **⚠ The try/catch cron 5 added on Sep 21 to prevent exactly this is DEAD CODE.** Commit `fe4d110d` wrapped the stamp in a `try`/`catch` specifically to stop duplicates. It cannot fire: 1.0 turns the "gone away" **warning** into a fatal `exit` inside the error handler, so it never unwinds to a `catch`. Evidence: 33 `Logs.Issue` 769 occurrences, **zero** matching rows in `Logs_Compass.Email`. Only the shutdown handler runs after that `exit`.
 - **⚠ The Sep 21 push is what made a latent bug visible.** `fe4d110d` flipped this cron from `"active": 0` to `"active": 1` in `schedules/cron.worker.sync.json`, and `85eb8784` turned it into a 450-second bounded loop (`$startTime = time() + 450`) running every 10 minutes. The stamp race had existed since 2026-09-15 (9 POs duplicated Sep 15–17 even while the schedule was off); two days of scheduled running then produced most of the rest (22 POs total, see above). **Before re-enabling any long-dormant cron that writes to an external system, check its stamp path.**
 - **⚠ The ODP PO number lands in `otherRefNum`, which NetSuite global search does NOT index.** Cron 5 sets `$salesOrder->otherRefNum = substr($rowSalesOrders['PurchaseOrderNumbers'], 0, 45)` (`:194`). Typing the Office Depot PO number into NetSuite's **global search box returns nothing** — the field is populated, it is simply not searchable. That silence reads exactly like "the order never reached NetSuite," and it is not. **Open the sales order by internal id instead:** take `c_netsuiteInternalSalesOrderId` from the **Office Depot** `SalesOrders` row (`customerId = 1`). Verified 2026-08-31 on ODP POs `41849836-1079` / `41849745-1127` / `41849655-5125` → SO internal ids `7437998` / `7437898` / `7438098`.
@@ -163,6 +178,7 @@ The NetSuite writeback lands on the **downstream ODP SalesOrder (customerId=1)**
 - **Editing cron recipient lists — a grep hit is not a live hit.** Email recipients are hardcoded in the cron scripts, and **several dormant copies of the same script exist** under `crons/toga2/compass/edi/` and `crons/toga2/compass/DOA_edi_odp_flow/`. Before changing a recipient list, cross-check the file against `worker/schedules/*.json` for a referencing entry whose `active` is not `0`. Only the scheduled copies matter.
 
 ## Change history
+- 2026-10-08 — Cron 5 duplicate-proof send (`fd426d9b`): externalId `toga2-compass-odso-<ODP SO id>` + lookup before/after `add()`, PO fallback for older orders; removed the shutdown alert (false alarm on SA138624) and the try/catch around `add()`; chose add+lookup over upsert (CTO). Added no-beta-testing and ops-email gotchas. (bala)
 - 2026-09-25 — Corrected the fix description (forked stamp via `queryForked`, no `keepAlive`; stamp unconfirmed); widened duplicate scope to 22 POs Sep 15–22; flagged 9 all-Billed POs as open; recorded monitor-only decision and the `NetsuiteDuplicateSalesOrders` backstop. (rgirish)
 - 2026-09-22 — Root-caused the duplicate NetSuite SOs (15 across 11 POs, Sep 21-22) as the dead-MySQL-connection stamp race, not a Toga duplicate; fixed cron 5 with a forked stamp + a `register_shutdown_function` alert (`worker` `afce32ca`). Recorded that the Sep 21 try/catch is dead code and that `fe4d110d` re-enabling the cron is what surfaced the latent bug. (rgirish)
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.976",
+  "version": "1.0.977",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",