toga-ai 1.0.548 → 1.0.549

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -14,7 +14,7 @@
14
14
  | [Running 2.0 code from a bare CLI script (bootstrap + transactions)](features/cli-script-bootstrap.md) | A throwaway CLI script (a data check, a backfill dry-run, a render harness) that wants the real 2.0 framework — `_Model`, `_Query`, `_Database` — is **not** the | _underscore/Database.php, _underscore/Environment.php, api2/Initialize.php |
15
15
  | [_Cloud S3 helpers (copy / get / delete / list)](features/cloud-s3-helpers.md) | `_Cloud` centralizes AWS SDK S3 usage for the 2.0 stack so the `S3Client` never leaks into workers or app code. | _underscore/Cloud.php |
16
16
  | [_Component_*/_Model_* project-namespace registration (autoloader) & backslash-qualify traps](features/component-model-namespace-registration.md) | Every **project-local** `_Component_*` and `_Model_*` class in a 2.0 app **must declare the project namespace** at the top of the file: ```php namespace <NAMESP | _underscore/Loader.php, worker2/_.php, api2/_.php, worker2/Component/Forecast/Db/Db.php, worker2/Component/Forecast/SaleImport/SaleImport.php, worker2/Component/Api/Oneuptime/Oneuptime.php, api2/Component/Api/Netsuite/Netsuite.php, _underscore/Component/Api/Paypal/Paypal.php |
17
- | [_Config group access — the two-argument "optional" form does NOT rescue a missing GROUP](features/config-group-access.md) | `_Config::<group>('<property>')` reads a value out of the active `Config/<ENVIRONMENT>.ini`, and **it throws when the requested group is absent from the ini.** | _underscore/Config.php, _underscore/Component/Api/Paypal/Paypal.php |
17
+ | [_Config group access — the two-argument "optional" form does NOT rescue a missing GROUP](features/config-group-access.md) | `_Config::<group>('<property>')` reads a value out of the active `Config/<ENVIRONMENT>.ini`, and **it throws when the requested group is absent from the ini.** | _underscore/Config.php, _underscore/Component/Api/Paypal/Paypal.php, _underscore/Worker.php, api2/Config/beta.ini, api2/Config/sandbox-dev.ini |
18
18
  | [Re-pointing a DB alias mid-request (_Database::register park/restore)](features/database-alias-repointing.md) | `_Database` keys **all live per-database runtime state by the connection ALIAS** (`Client` / `_underscore::DB_CLIENT`, `ClientLogs`, `Archive`), **not** by the | _underscore/Database.php, _underscore/Query.php, api2/Component/Api/V2/V2.php, api2/Component/Api/CrossClient/CrossClient.php |
19
19
  | [2.0 Email Send Pipeline (queue + Send worker)](features/email-send-pipeline.md) | In 2.0, `_Email::send()` **does not transmit** — it queues the message. | _underscore/Email.php, _underscore/String.php, worker2/Worker/Infrastructure/Email/Send.php |
20
20
  | [Client Email Template Sending](features/email-template-sending.md) | `_Model_Client_EmailTemplate` sends a stored, client-defined email template by UUID. | _underscore/Model/Client/EmailTemplate.php, _underscore/Model/Client/EmailTemplateOutgoingEmailAddress.php, _underscore/Email.php |
@@ -23,7 +23,7 @@
23
23
  | [Forecast.Sales NetSuite import engine (real-time webhook)](features/forecast-sale-import.md) | Real-time importer that takes a NetSuite **sale** record and writes its lines into `Forecast.Sales` (the Forecast2 revenue table). | worker2/Component/Forecast/SaleImport/SaleImport.php, worker2/Component/Forecast/Db/Db.php, _underscore/Component/Api/Netsuite/Netsuite.php, worker2/Worker/Netsuite/Invoice.php, worker2/Worker/Netsuite/CashSale.php, worker2/Worker/Netsuite/CreditMemo.php, worker2/Worker/Netsuite/CashRefund.php, worker2/Worker/Netsuite/JournalEntry.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Worker/Netsuite/SalesOrder.php, dbchanges2/Forecast/2026-06-26a - Add journalEntry to Sales transaction type enum.sql, test/@dave/test_invoice_lifecycle.php, test/@dave/test_je_lifecycle.php, test/@dave/test_creditmemo_lifecycle.php, test/@dave/test_cashsale_lifecycle.php, test/@dave/test_cashrefund_lifecycle.php, test/@dave/test_fetchrecord_routes.php, test/@dave/verify_je_classification.php, test/@dave/probe_je_accounts.php, test/@dave/probe_je_shape.php, test/@dave/fixer.php, test/@dave/Junk Drawer/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/Junk Drawer/NetSuite/api-message-queue/dev_ue_api_msg_queue_enqueue.js |
24
24
  | [isFulfillable Propagation Up the SO↔PO Chain](features/fulfillable-item-propagation.md) | `Items.isFulfillable` is a boolean that gates whether a storefront line's **Qty Fulfilled** cell is actionable. | _underscore/Model/Client/Item.php, _underscore/Model/Compass/Item.php, dbchanges2/Core/2026-07-17 - Items isFulfillable RecordField.sql, dbchanges2/Core/2026-07-17 - RegisterItemIsFulfillableInterceptors.sql, dbchanges2/Client/2026-07-17 - ItemsisFulfillable.sql |
25
25
  | [Item-Fulfillment Stage Lifecycle (picked/packed/shipped) & Order Status](features/item-fulfillment-stage-lifecycle-and-order-status.md) | Every ItemFulfillment (IF) now carries an explicit **stage** — picked → packed → shipped — resolved through `ItemFulfillmentStages → ItemFulfillmentStatuses` (m | _underscore/Model/Client/SalesOrder.php, _underscore/Model/Quad/SalesOrder.php, _underscore/Model/Compass/SalesOrder.php, _underscore/Model/Compass/SalesOrderStatus.php, _underscore/Model/Client/SalesOrderItem.php, _underscore/Model/Client/Item.php, _underscore/Model/Client/PurchaseOrderItem.php, library/app/api/toga2.php, dbchanges2/Client/2026-06-30a - BackfillNullStageItemFulfillmentsToShipped.sql, dbchanges2/Client/2026-06-30b - SalesOrderStatusesPickedPacked.sql, dbchanges2/Client/2026-06-30c - ItemFulfillmentStageIdNotNull.sql, dbchanges2/Client_CompassCanada/2026-06-30a - ItemFulfillmentLifecycleAndShippedBackfill.sql |
26
- | [DB-free unit testing for _underscore model interceptors](features/model-interceptor-unit-testing.md) | `_underscore` shipped with **no** PHPUnit setup (no `composer.json`/`phpunit`; only vendored PhpOffice tests existed). | _underscore/Test/bootstrap.php, _underscore/Test/Prudential/ServiceRequestTest.php |
26
+ | [DB-free unit testing for _underscore model interceptors](features/model-interceptor-unit-testing.md) | `_underscore` shipped with **no** PHPUnit setup (no `composer.json`/`phpunit`; only vendored PhpOffice tests existed). | _underscore/Test/bootstrap.php, _underscore/Test/Prudential/ServiceRequestTest.php, test/@Bala/tests/netsuite_salesorder_payload_tests.php |
27
27
  | [_Model magic-field access (__get without __isset)](features/model-magic-field-access.md) | `_Model` exposes DB columns as "magic" properties via `__get()`, but it defines **no** `__isset()`. | _underscore/Model/Core/Model.php, _underscore/Model.php, _underscore/Model/Rate/Subscription.php |
28
28
  | [_Model::save() vs raw _Query — no atomic conditional update](features/model-save-vs-query-atomic-update.md) | `_Model::save()` is a plain load-then-write ORM primitive and **cannot express an atomic conditional update** (an optimistic-concurrency / row-claim guard such | _underscore/Model.php, _underscore/Query.php, _underscore/Model/Rate/Subscription.php |
29
29
  | [NetSuite REST Client (_Component_Api_Netsuite) — record writes & SuiteQL](features/netsuite-rest-client.md) | `_Component_Api_Netsuite` is the **2.0 `_underscore` NetSuite REST client** — the shared primitive every worker2/api2 NetSuite caller uses for record GETs, Suit | _underscore/Component/Api/Netsuite/Netsuite.php |
@@ -6,11 +6,14 @@ project: _Underscore
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-08-06
10
- owners: ["mhammontree"]
9
+ updated: 2026-08-10
10
+ owners: ["mhammontree", "bala"]
11
11
  files:
12
12
  - _underscore/Config.php
13
13
  - _underscore/Component/Api/Paypal/Paypal.php
14
+ - _underscore/Worker.php
15
+ - api2/Config/beta.ini
16
+ - api2/Config/sandbox-dev.ini
14
17
  related:
15
18
  - ./cli-script-bootstrap.md
16
19
  - ./model-magic-field-access.md
@@ -72,6 +75,23 @@ environment without a `[paypal]` section returned **`EO-1` with a stack trace to
72
75
  retail customer** instead of failing cleanly with a business message. Guarding with
73
76
  `isGroupSet()` turned it into a controlled `_Exception_Business`.
74
77
 
78
+ ## A missing **key inside a group that exists** throws too — and framework internals read keys you didn't
79
+
80
+ The table above is easy to read as "worry about missing groups." The single-argument form
81
+ **throws on a missing property as well** (`Config.php:79`), and the keys at risk are often ones
82
+ **your code never mentions** because a framework helper reads them.
83
+
84
+ Concrete case (2026-08-10, Elite sales-order push): `_Worker::runTask`'s non-debug path reads
85
+ `[cloud] aws_worker_queue_url` and `aws_worker_queue_region`. `api2/Config/beta.ini` and
86
+ `sandbox-dev.ini` both **had a `[cloud]` group but neither key**; `production.ini` had both. A
87
+ `postPost` interceptor calling `runTask` therefore produced an **uncaught throw → HTTP 500 `EO-1`
88
+ with a null envelope and no log row** — a failure that looks nothing like "config". Fixed by adding
89
+ the two keys to `beta.ini` and `sandbox-dev.ini`.
90
+
91
+ **Rule:** when you introduce a call into a framework helper (`_Worker::runTask`, a carrier client, an
92
+ S3 helper) from a new tier, **diff that tier's ini against `production.ini` for the keys the helper
93
+ reads** — the group existing proves nothing.
94
+
75
95
  ## Gotchas / known issues
76
96
 
77
97
  - **`_Config::<group>('prop', false)` is NOT a null-safe read.** It is property-optional only.
@@ -86,6 +106,13 @@ retail customer** instead of failing cleanly with a business message. Guarding w
86
106
 
87
107
  ## Change history
88
108
 
109
+ - 2026-08-10 — Recorded that a **missing key inside an existing group throws as well**
110
+ (`Config.php:79`), and that the risky keys are the ones **framework helpers** read on your behalf:
111
+ `_Worker::runTask`'s non-debug path needs `[cloud] aws_worker_queue_url` /
112
+ `aws_worker_queue_region`, which `api2/Config/beta.ini` and `sandbox-dev.ini` lacked (both had a
113
+ `[cloud]` group; `production.ini` had the keys) — so an interceptor calling `runTask` returned
114
+ **`EO-1` with a null envelope and no log row**. Keys added to both files; the durable rule is to
115
+ diff a tier's ini against `production.ini` for the keys a newly-called helper reads. (bala)
89
116
  - 2026-08-06 — TRUE-80282 testing: documented that `_Config::<group>()` **throws when the group is
90
117
  absent** and that the two-argument "optional" form does **not** rescue a missing group —
91
118
  `_Config::__callStatic` (`Config.php:64-87`) tests `$properties[0]` for the optional flag
@@ -6,13 +6,15 @@ project: _Underscore
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-07-24
9
+ updated: 2026-08-10
10
10
  owners: ["bala"]
11
11
  files:
12
12
  - _underscore/Test/bootstrap.php
13
13
  - _underscore/Test/Prudential/ServiceRequestTest.php
14
+ - test/@Bala/tests/netsuite_salesorder_payload_tests.php
14
15
  related:
15
16
  - ../../../../clients/prudential/features/service-request-address-validation.md
17
+ - ../../worker2/features/netsuite-salesorder-outbound-push.md
16
18
  ---
17
19
 
18
20
  ## Summary
@@ -56,11 +58,22 @@ Verified 6/6 passing against the real code.
56
58
  making them public just to test them.
57
59
  - **Stub only the persistence base, load the real logic.** The value of the pattern is that the
58
60
  production validator runs unchanged; do not reimplement the validator in the stub.
61
+ - **The pattern travels beyond `_underscore` models.** The same stub-the-dependencies +
62
+ reflect-into-privates technique was applied to a **worker2 action** from the `test` repo
63
+ (`test/@Bala/tests/netsuite_salesorder_payload_tests.php`, 76 tests, plain PHP rather than
64
+ PHPUnit): it `require`s the real `worker2/Worker/Netsuite/SalesOrder.php` with stubbed framework
65
+ dependencies and drives its private builders/resolvers by reflection. Use it as the second worked
66
+ example — see
67
+ [Toga → NetSuite sales-order push](../../worker2/features/netsuite-salesorder-outbound-push.md).
59
68
  - There is still no `composer.json`/autoloader for the repo — tests rely on the explicit
60
69
  `--bootstrap` file to wire up requires. Adding more model tests means extending
61
70
  `Test/bootstrap.php` with the stubs that model needs.
62
71
 
63
72
  ## Change history
73
+ - 2026-08-10 — Noted that the stub + reflect-into-privates pattern also carries to **worker2 actions**
74
+ driven from the `test` repo (`test/@Bala/tests/netsuite_salesorder_payload_tests.php`, 76 tests,
75
+ plain PHP, loading the real `Worker/Netsuite/SalesOrder.php`) — a second worked example of testing
76
+ shipped code rather than a copy. (bala)
64
77
  - 2026-07-24 — Created: first PHPUnit scaffold in `_underscore` (`Test/bootstrap.php` +
65
78
  `Test/Prudential/ServiceRequestTest.php`), a DB-free pattern that stubs the model's persistence
66
79
  base and reflects into private interceptor validators; seeded with 6 regression tests for the
@@ -6,12 +6,13 @@ project: _Underscore
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-08-05
9
+ updated: 2026-08-10
10
10
  owners: ["dfranks", "jcardinal", "bala"]
11
11
  files:
12
12
  - _underscore/Component/Api/Netsuite/Netsuite.php
13
13
  related:
14
14
  - ../../worker2/features/netsuite-salesorder-open-orders-sync.md
15
+ - ../../worker2/features/netsuite-salesorder-outbound-push.md
15
16
  - ../../worker2/features/netsuite-opportunity-sync.md
16
17
  - ../architecture.md
17
18
  ---
@@ -86,17 +87,21 @@ through `_ApiRequest` directly (mirroring `send()`'s auth/endpoint/header setup)
86
87
  numeric id tolerating a query string or fragment.
87
88
  - Throws on non-2xx, on no `Location` header, and on no id parsed.
88
89
 
89
- > **⚠ BUG (live, not yet fixed):** the trailing-id regex was written as `'#/(\d+)(?:[?#]|$)#'`,
90
- > which uses `#` as the PCRE **delimiter** *and* puts `#` inside the `[?#]` character class — so PHP
91
- > reads the class-`#` as a premature closing delimiter and the whole pattern **always throws
92
- > `preg_match(): Unknown modifier ']'`**. This fires *after* the record is already created, so the
93
- > create succeeds in NetSuite but `createRecord()` raises and the caller never gets the new id
94
- > it breaks the entire outbound create/push path **for any record type** (observed on `journalentry`
95
- > and again on `invoice` create). **Fix direction:** change the delimiter so `#`
96
- > isn't both delimiter and class member, e.g. `~/(\d+)(?:[?#]|$)~`. Interim workaround used in
97
- > probes and in `test/@dave/test_invoice_lifecycle.php`: parse the new internalId off the `Location`
98
- > value carried in the thrown exception message (the lifecycle harness ships its own correct
99
- > `~`-delimited parse to sidestep this without editing the framework). Confirmed live 2026-06-26.
90
+ > **✅ FIXED 2026-08-10 (`Netsuite.php` ~L198) re-delimited to `~`.** The trailing-id regex was
91
+ > written as `'#/(\d+)(?:[?#]|$)#'`, which used `#` as the PCRE **delimiter** *and* put `#` inside
92
+ > the `[?#]` character class — PHP read the class-`#` as a premature closing delimiter, treated
93
+ > `]|$)` as modifiers, and `preg_match()` returned `false` with **`Unknown modifier ']'`**. So
94
+ > `createRecord()` could **never** return: it broke the outbound create/push path for **every**
95
+ > record type (observed on `journalentry`, `invoice`, and `salesOrder`). The fix is simply the `~`
96
+ > delimiter. The old interim workaround parsing the id out of the thrown exception message, as
97
+ > `test/@dave/test_invoice_lifecycle.php` does is no longer needed for new code.
98
+ >
99
+ > **⚠ Keep the consequence in mind, it outlives the bug:** the failure happened **after** the HTTP
100
+ > 201 *and* after the `Location` header was read NetSuite had already created the record. Any
101
+ > "create succeeded but we crashed after" fault in this class therefore leaves an **orphan in
102
+ > NetSuite with nothing in Toga pointing at it** (this bug orphaned two Elite sales orders in live
103
+ > account 1095849). **Never blind-retry a failed `createRecord()` — look the record up in NetSuite
104
+ > first.**
100
105
 
101
106
  ### Read — `fetchRecord(string $route, string $label): array`
102
107
 
@@ -163,12 +168,22 @@ doc.)
163
168
  - **204-with-empty-body is success, not failure.** Both create (204 + `Location`) and update (204,
164
169
  no body) return no payload; treat a 2xx with empty body as success and key off the status, not the
165
170
  body.
166
- - **`createRecord()`'s trailing-id regex is a live `#`-delimiter bug** (`'#/(\d+)(?:[?#]|$)#'`
167
- always `Unknown modifier ']'`), thrown *after* the record is created see the note under
168
- *Create* for the cause and the `~…~`-delimiter fix.
171
+ - **A `createRecord()` failure does NOT mean the record wasn't created.** The id parse happens after
172
+ the 201 + `Location` read, so a post-create fault leaves an **orphan in NetSuite**. Check NetSuite
173
+ before retrying. (The `#`-delimiter regex bug that caused this class of orphan was fixed
174
+ 2026-08-10 — see the note under *Create*.)
169
175
 
170
176
  ## Change history
171
177
 
178
+ - 2026-08-10 — **Fixed the `createRecord()` trailing-id regex** (`Netsuite.php` ~L198): re-delimited
179
+ `'#/(\d+)(?:[?#]|$)#'` → `~`, so `preg_match` no longer returns `false` with
180
+ *"Unknown modifier ']'"* and the method can actually return an internalId. Recorded the durable
181
+ consequence — the throw fired **after** the 201 and the `Location` read, so NetSuite had already
182
+ created the record: this bug left **two orphaned Elite sales orders in live account 1095849**, and
183
+ any post-create fault in this class must be diagnosed by looking the record up in NetSuite rather
184
+ than by retrying. Surfaced building the
185
+ [Toga → NetSuite sales-order push](../../worker2/features/netsuite-salesorder-outbound-push.md).
186
+ (bala)
172
187
  - 2026-08-05 — Recorded the **SuiteQL custom-column filter trap**: a NetSuite custom column
173
188
  (e.g. `inventorynumber.custitemnumber_asset_tag`/`_mac_address`) can be `SELECT`ed but **not**
174
189
  filtered on — `WHERE custitemnumber_asset_tag IS NOT NULL` returns 0 rows with no error. Drive
@@ -3,7 +3,7 @@
3
3
  | Doc | Summary | Files |
4
4
  |-----|---------|-------|
5
5
  | [API (api2 / TOGa API v2) Architecture](architecture.md) | `api2` is the backend powering the public **TOGa 2.0 API**. | api2/Controller/Index.php, api2/Component/Api/V2/V2.php, api2/Component/Api/Cxml/Cxml.php, api2/Component/Api/V2/Response/Response.php, api2/Config/ |
6
- | [API Payload Interceptors (metadata-registered prePost/postPut model hooks)](features/api-payload-interceptors.md) | A `_Model`'s `prePost()` / `postPost()` / `prePut()` / `postPut()` hooks are **not** called by the model. | api2/Component/Api/V2/V2.php, _underscore/Model/Client/ItemFulfillment.php, dbchanges2/Core/2026-06-30a - ItemFulfillmentStageDefaultInterceptor.sql |
6
+ | [API Payload Interceptors (metadata-registered prePost/postPut model hooks)](features/api-payload-interceptors.md) | A `_Model`'s `prePost()` / `postPost()` / `prePut()` / `postPut()` hooks are **not** called by the model. | api2/Component/Api/V2/V2.php, api2/Controller/Index.php, _underscore/Model/Client/ItemFulfillment.php, _underscore/Model/Elite/SalesOrder.php, dbchanges2/Core/2026-06-30a - ItemFulfillmentStageDefaultInterceptor.sql |
7
7
  | [Multi-Client (Cross-Client) Data Retrieval](features/cross-client-data-retrieval.md) | A single authenticated V2 GET listing can return records across **many** clients (designed for 1000+) that the caller is entitled to, honoring **each target cli | api2/Component/Api/CrossClient/CrossClient.php, api2/Component/Api/V2/V2.php, api2/Controller/Index.php, _underscore/Model/Cache/Table.php, _underscore/Model/Cache/Tables/Client.php, _underscore/Model/Core/Record.php, worker2/Worker/Platform/Cache.php, worker2/Controller/Index.php, worker2/_.php, dbchanges2/Cache/2026-06-30a - MultiClientCacheTables.sql, dbchanges2/Core/2026-06-30b - CacheClusterRegistrationAndRecordTtl.sql, dbchanges2/Core/2026-07-27a - PlatformCacheCleanCron.sql |
8
8
  | [Encrypted-User-UUID Auth Handoff (/auth/encrypted-user-uuid)](features/encrypted-user-uuid-auth-handoff.md) | `POST /auth/encrypted-user-uuid` is the intended **cross-client / SSO-handoff identity mechanism**: given an encrypted `{client, user}` UUID pair, it mints a fr | api2/Component/Api/CrossClient/CrossClient.php |
9
9
  | [ENVIRONMENT (not the EB environment name) decides the _underscore branch and Config file](features/environment-variable-drives-underscore-branch.md) | An api2 Elastic Beanstalk instance decides **which `_underscore` branch it clones** and **which `Config/<env>.ini` it loads** from the EB environment property * | api2/.ebextensions/git.php, api2/.ebextensions/php_include_underscore.config, api2/.ebextensions/git.sandbox-dev.json, api2/Config/beta.ini, api2/Config/sandbox-dev.ini, api2/Component/Api/V2/V2.php |
@@ -6,17 +6,22 @@ project: API
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-08-04
10
- owners: ["mhammontree", "dfranks"]
9
+ updated: 2026-08-10
10
+ owners: ["mhammontree", "dfranks", "bala"]
11
11
  files:
12
12
  - api2/Component/Api/V2/V2.php
13
+ - api2/Controller/Index.php
13
14
  - _underscore/Model/Client/ItemFulfillment.php
15
+ - _underscore/Model/Elite/SalesOrder.php
14
16
  - dbchanges2/Core/2026-06-30a - ItemFulfillmentStageDefaultInterceptor.sql
15
17
  related:
16
18
  - ./record-scripts.md
17
19
  - ../architecture.md
18
20
  - ../../_underscore/features/acl-permission-chain.md
19
21
  - ../../_underscore/features/recursive-item-fulfillments.md
22
+ - ../../_underscore/features/config-group-access.md
23
+ - ../../worker2/features/creating-worker-actions.md
24
+ - ../../../clients/elite/features/salesorder-netsuite-push.md
20
25
  ---
21
26
 
22
27
  ## Summary
@@ -126,6 +131,78 @@ per-region subclass does *not* need its own copy. Do not add duplicate hooks per
126
131
  advance-shipping-notices, **179** approval-decisions) all map 1:1 to real methods — **the row whose
127
132
  record has no corresponding method is the anomaly.**
128
133
 
134
+ ## Queueing background work from a post interceptor — the rules that make it work
135
+
136
+ A `postPost`/`postPut` hook that hands work to worker2 (`_Worker::runTask`) sits in the hardest spot
137
+ in the stack: it runs **inside api2's open transaction**, its payload is shallow, and its reads
138
+ default to a **read replica**. All five rules below were paid for in a single build (the Elite
139
+ sales-order push, 2026-08-10) and every one of them fails **silently or only in production**.
140
+
141
+ ### 1. `$api->internalApiRequest()` returns records **FLAT** — there is no `data` wrapper
142
+
143
+ It returns `(object) processRoutePairs(...)`, so the shape is **`$response->salesOrders`**, *not*
144
+ `$response->data->salesOrders`. The `data` envelope exists only on a real **HTTP** response.
145
+ In-tree proof: `_underscore/Model/Client/SalesOrder.php:373` reads
146
+ `$templateResponse->approvalTemplates`.
147
+
148
+ This cost a full debugging cycle: the wrong path returned `null` every time, so the hook logged and
149
+ returned **without queueing** — producing a clean `201` with no worker job and no visible error.
150
+
151
+ ### 2. `ApiPayloadInterceptors.minDepth` does **not** deepen a POST interceptor's payload
152
+
153
+ In `api2/Component/Api/V2/V2.php` the POST response is serialized by `getFullModelData` at
154
+ **~L5407**, but the `minDepth` bump for **post** interceptors happens at **~L5892-5900** — *after*.
155
+ So raising `minDepth` cannot give a `postPost` hook a deeper payload. **When a post interceptor
156
+ needs depth, use `internalApiRequest`.** (Depth matters more than it looks: at depth 3 an order's
157
+ `shipToAddress.state.country` is absent entirely — country sits four levels down.)
158
+
159
+ ### 3. The hook runs inside api2's transaction — another process cannot see the new row
160
+
161
+ The commit is at **`api2/Controller/Index.php:427`**, gated on `$api->response->isSuccess`, **after**
162
+ `execute()` returns. A worker that starts before that commit reads nothing: the symptom is
163
+ *"SalesOrder id=NNN not found"* from a job that was queued correctly.
164
+
165
+ ### 4. …and the reader connection cannot see it **at all** — this fails 100% in prod, 0% on beta
166
+
167
+ `_Database::$_isReadHostEnabled` defaults to **TRUE**, so `SELECT`s go to the reader
168
+ (`Query.php:273`). A reader is a **different connection**, so an uncommitted row is invisible to it —
169
+ this is **not replication lag**, and no amount of waiting fixes it. `Core.DatabaseHosts` is what
170
+ makes the bug environment-dependent:
171
+
172
+ | Environment | hostWriter | hostReader |
173
+ |---|---|---|
174
+ | production (env 1) | `writer.client.database.togahub.com` | `reader1/2/3.*` |
175
+ | dev-sandbox (env 4) | *same endpoint* | *same endpoint* |
176
+
177
+ So this entire class of bug is **invisible on beta/dev-sandbox and fails on every production
178
+ request.** Never conclude "it works on beta" for an in-request read-after-write.
179
+
180
+ ### 5. The pattern that works
181
+
182
+ - Read the record **inside the same request** with `internalApiRequest`, wrapped in
183
+ `getIsReadHostEnabled()` / `setIsReadHostEnabled(false)` and restored in a **`finally`** (pin the
184
+ read to the writer, then put the flag back for everyone else).
185
+ - **Pass the resulting object along in the job parameters** so the worker never re-reads it. (The
186
+ worker action must accept it — see
187
+ [Toga → NetSuite sales-order push](../../worker2/features/netsuite-salesorder-outbound-push.md),
188
+ whose `?object $salesOrder` parameter exists for exactly this.)
189
+ - Pass **`false`** for `internalApiRequest`'s 5th argument (`$throwExceptionsOnError`). Its throw is
190
+ guarded by `!empty($this->response->messages)` on the **shared** response object, and
191
+ `json_encode([])` is the **truthy** string `"[]"` — so it throws even with **zero real errors**
192
+ once any message (even a warning) is present anywhere in the request.
193
+
194
+ ### Two more failure modes to expect on this path
195
+
196
+ - **Config drift kills it before it starts.** `_Worker::runTask`'s non-debug path reads
197
+ `[cloud] aws_worker_queue_url` / `aws_worker_queue_region`, and `_Config::__callStatic` **throws on
198
+ a missing key even inside a group that exists** — an uncaught throw from `postPost` surfaces as
199
+ **HTTP 500 `EO-1` with a null envelope and no log row**. See
200
+ [_Config group access](../../_underscore/features/config-group-access.md).
201
+ - **`debug_mode` changes how `runTask` dispatches**, and the debug path runs the worker *inside* your
202
+ open transaction — so it always fails a read-back. Also, an **absent `WorkerJobs` row is
203
+ ambiguous** ("never queued" vs "queued, never consumed"). Both are covered in
204
+ [Creating Worker Actions](../../worker2/features/creating-worker-actions.md).
205
+
129
206
  ## Worked example — the EV-10 that was not a code bug
130
207
 
131
208
  `_Model_Client_ItemFulfillment::prePost()` defaults `itemFulfillmentStageId` to the shipped stage on
@@ -158,6 +235,15 @@ and the failing environment**. It is a small table, and the drift is usually exa
158
235
  metadata — "works in prod, not in beta/dev-sandbox" is the signature of drift, not of a bad
159
236
  deploy. See the
160
237
  [non-prod metadata drift repair workflow](../../dbchanges2/workflows/nonprod-metadata-drift-repair.md).
238
+ - **⚠ An in-request read-after-write in an interceptor works on beta and fails 100% in production.**
239
+ Reads default to the **reader** connection, which cannot see the request's own uncommitted row;
240
+ dev-sandbox points writer and reader at the same endpoint, so it hides the bug. Pin with
241
+ `setIsReadHostEnabled(false)` (restore in `finally`).
242
+ - **`internalApiRequest` has no `data` envelope** (`$response->salesOrders`, not
243
+ `$response->data->salesOrders`), and `minDepth` cannot deepen a POST interceptor's payload — the
244
+ bump happens after serialization. A wrong read path here is a **silent no-queue with a clean 201**.
245
+ - **Anything a post interceptor queues must survive the transaction boundary** — pass the record
246
+ snapshot in the job parameters rather than letting the worker re-read it.
161
247
  - **Never emit output from an interceptor.** A stray `echo`/`print_r` corrupts the `/v2`
162
248
  `{isSuccess,…,data}` envelope (this really happened in `ItemFulfillment.php` — see the
163
249
  carrier-shipping-labels doc).
@@ -166,6 +252,22 @@ and the failing environment**. It is a small table, and the drift is usually exa
166
252
 
167
253
  ## Change history
168
254
 
255
+ - 2026-08-10 — Added **Queueing background work from a post interceptor**, five verified rules from
256
+ the Elite sales-order push: (1) `internalApiRequest` returns records **flat** —
257
+ `$response->salesOrders`, not `$response->data->salesOrders` (the `data` envelope exists only on a
258
+ real HTTP response; in-tree proof `_underscore/Model/Client/SalesOrder.php:373`) — and getting it
259
+ wrong yields a clean 201 with no job and no error; (2) `minDepth` **cannot** deepen a POST
260
+ interceptor's payload (serialization at `V2.php` ~L5407 precedes the post-interceptor bump at
261
+ ~L5892-5900); (3) hooks run inside api2's transaction, committed at `Controller/Index.php:427`
262
+ after `execute()`, so another process sees *"id not found"*; (4) reads default to the **reader**
263
+ connection (`_Database::$_isReadHostEnabled`, `Query.php:273`), which cannot see the uncommitted
264
+ row **at all** — and because dev-sandbox's `Core.DatabaseHosts` points writer and reader at one
265
+ endpoint while production splits them, the bug is **invisible on beta and fails 100% in prod**;
266
+ (5) the working pattern — writer-pinned `internalApiRequest` (`setIsReadHostEnabled(false)`,
267
+ restored in `finally`), snapshot passed in the job parameters, and `$throwExceptionsOnError =
268
+ false` because the throw is guarded on the shared `response->messages` where `json_encode([])` is
269
+ the truthy `"[]"`. Cross-linked the `[cloud] aws_worker_queue_*` config-drift `EO-1` and the
270
+ `debug_mode`/absent-`WorkerJobs`-row traps. (bala)
169
271
  - 2026-08-04 — Documented the **inverse failure mode**: the derived-name dispatch is **unguarded at
170
272
  all four `V2.php` call sites** (`~L8655` lookup, `~L3148` pre, `~L5694` post, `~L7411`/`~L7437`
171
273
  alternate pre) and there is **no `phpMethod` column in production**, so one row naming a
@@ -6,8 +6,8 @@ project: API
6
6
  client: shared
7
7
  type: workflow
8
8
  status: active
9
- updated: 2026-07-31
10
- owners: ["jcardinal", "mhammontree"]
9
+ updated: 2026-08-10
10
+ owners: ["jcardinal", "mhammontree", "bala"]
11
11
  files:
12
12
  - api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh
13
13
  - api2/ebs/register_instance_to_shared_application_load_balancer.php
@@ -142,6 +142,42 @@ Production is deliberately skipped and still registers by the existing process.
142
142
  which is also the reference implementation — **api2's copy has known security defects**, including
143
143
  a committed IAM access key.
144
144
 
145
+ ### Gotcha: staleness is **per-INSTANCE**, not per-branch — verify the box that CONSUMES THE QUEUE
146
+
147
+ Two `worker2` instances can serve the **same Core database while running different code**, and the
148
+ one you test through is not the one that runs your queued jobs. Seen 2026-08-10 on the Elite
149
+ sales-order push:
150
+
151
+ | Instance | Role | Result |
152
+ |---|---|---|
153
+ | `i-0262cfdc6cb97a3d2` | web tier behind `worker.beta.togahub.com` | current code + NetSuite credentials → **succeeded** |
154
+ | `i-0be8ef2e550dec05b` | **SQS consumer** (env `e-rimmbihkge`, `WorkerSandboxDevQueue`) | rejected the identical parameter with *"Unknown named parameter `$salesOrderUuid`"* and separately failed NetSuite auth |
155
+
156
+ The EB bundle logs proved the consumer's last `app_source_bundle` extraction was **7 days earlier**.
157
+ Note how the symptom lies: *"Unknown named parameter"* reads like a caller/contract bug, and an auth
158
+ failure reads like a credentials problem — both were just **old code on one box**.
159
+
160
+ **Procedure:** take the failing job's **`Core.WorkerJobs.instanceId`**, and verify *that* instance
161
+ (and the EB environment that consumes the queue) actually received the deploy. "I deployed beta" is
162
+ not a verification — a direct POST to the worker URL only proves the **web** instance.
163
+
164
+ ### Where each tier's `error_log()` output actually goes
165
+
166
+ `worker2` ships `/var/log/php-fpm/www-error.log` to the CloudWatch log group **`worker`** via
167
+ `.platform/hooks/postdeploy/020-cloudwatch-www-error.sh`. **`api2` has no equivalent hook** — so
168
+ api2 `error_log()` output (including an interceptor's silent-skip messages) is **not in CloudWatch**
169
+ and must be pulled from the EB request/full logs. Do not conclude "the code never logged" from an
170
+ empty CloudWatch search on the api2 tier.
171
+
172
+ ### Credentials live in the committed `Config/*.ini` files (finding, 2026-08-10)
173
+
174
+ Both `worker2` and `api2` carry **live secrets in `Config/*.ini` files that are committed to those
175
+ repos** — NetSuite OAuth `clientId`/`certificateId`, AWS keys, database passwords, a Talos agent key.
176
+ Document only **where** they live, never the values. **Recommendation (not actioned):** move them to
177
+ **EB environment properties or SSM Parameter Store** and treat anything currently committed as
178
+ rotate-on-exposure. Related: api2's ALB auto-registration copy also carries a committed IAM access
179
+ key (see the auto-registration note above).
180
+
145
181
  ### On-instance composer install (when a dep is missing post-deploy)
146
182
  If a Composer dep is missing on the running instance (e.g. `Class "FPDF" not found` because
147
183
  `composer.lock` wasn't committed), you can install it on the box over SSH/PuTTY — but
@@ -164,6 +200,17 @@ aws codeconnections get-connection --connection-arn "<CONN_ARN>" --region "$REGI
164
200
  ```
165
201
 
166
202
  ## Change history
203
+ - 2026-08-10 — Added three deploy-verification facts from the Elite sales-order push: (1) **staleness
204
+ is per-INSTANCE** — the worker2 web instance behind `worker.beta.togahub.com`
205
+ (`i-0262cfdc6cb97a3d2`) ran current code while the **SQS consumer** (`i-0be8ef2e550dec05b`, env
206
+ `e-rimmbihkge`, `WorkerSandboxDevQueue`) was 7 days stale and failed the same job with *"Unknown
207
+ named parameter"* + a NetSuite auth error, so verify by `Core.WorkerJobs.instanceId` and by the
208
+ environment that **consumes the queue**, not by "I deployed beta"; (2) worker2 ships
209
+ `www-error.log` to the CloudWatch group `worker` via
210
+ `.platform/hooks/postdeploy/020-cloudwatch-www-error.sh` but **api2 has no such hook**, so api2
211
+ `error_log()` output is only in the EB request/full logs; (3) recorded that live credentials sit in
212
+ the **committed** `worker2`/`api2` `Config/*.ini` files (locations only — no values) with the
213
+ recommendation to move them to EB environment properties / SSM. (bala)
167
214
  - 2026-07-31 — TRUE-80487: added the **full cross-tier "beta" environment map** after it cost two
168
215
  wasted deploys. `api.beta.togahub.com` is EB env **`API-Sandbox-Dev`** on the **`dev-sandbox`**
169
216
  database, cloning `_underscore` **`_sandbox-dev`**; `toga2-view`'s Amplify target **named
@@ -26,6 +26,7 @@
26
26
  | [NetSuite Integrations Monitor (Monitor/Operations/NetsuiteIntegrations)](features/netsuite-integrations-monitor.md) | `_Worker_Monitor_Operations::NetsuiteIntegrations()` is a cross-client health check that detects **stuck NetSuite ↔ 2.0 integrations**. | worker2/Worker/Monitor/Operations.php, dbchanges2/Core/2026-08-10a - Netsuite Integrations Monitor.sql, _underscore/Database.php, _underscore/Query.php |
27
27
  | [NetSuite ↔ ClickUp / TOGA Opportunity Sync (API Message Queue + worker2 webhook)](features/netsuite-opportunity-sync.md) | Outbound sync from NetSuite to TOGA for the record types the Forecast2 importer pulls (opportunities first; sales/items/etc. | worker2/Worker/Netsuite.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Worker/Clickup.php, worker2/Worker/Clickup/Opportunity.php, worker2/Controller/Index.php, _underscore/Worker.php, test/@dave/NetSuite/api-message-queue/lib_amq_queue.js, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/NetSuite/api-message-queue/ue_amq_drain.js, test/@dave/NetSuite/api-message-queue/ss_amq_drain.js, test/@dave/NetSuite/api-message-queue/DEPLOY_RUNBOOK.md, test/@dave/clickup/backfill_opportunity_numbers.php, test/@dave/clickup/probe_opportunity_fields.php, test/@dave/probe_clickup_desc_match.php, test/@dave/test_model_load_behavior.php, dbchanges2/Forecast/2026-06-25a - Add unique index on Opportunities netsuiteOpportunityInternalId.sql, _underscore/Model/Forecast/Opportunity.php, test/@dave/approach/TRUE-80044.md, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
28
28
  | [NetSuite → Forecast Open-Orders Sync (salesOrder webhook → OpenOrderItems)](features/netsuite-salesorder-open-orders-sync.md) | Webhook-driven, single-record port of the legacy open-orders importer (TRUE-79142). | worker2/Worker/Netsuite/SalesOrder.php, worker2/Worker/Netsuite.php, worker2/Component/Forecast/Db/Db.php, worker2/Worker/Netsuite/Location.php, test/@dave/Junk Drawer/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/Junk Drawer/NetSuite/api-message-queue/ue_amq_invoice_resync_salesorder.js, test/@dave/probe_salesorder_rest_shape.php, test/@dave/probe_open_order_lines.php, test/@dave/check_so_status.php, test/@dave/check_so_history.php, test/@dave/probe_so_rest_lines.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_open_order_gating.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
29
+ | [Toga → NetSuite Sales-Order Push (multi-client, mapping-driven worker)](features/netsuite-salesorder-outbound-push.md) | The **outbound** half of `worker2/Worker/Netsuite/SalesOrder.php` (everything from the `OUTBOUND PUSH (REST)` banner down) pushes a Toga sales order **into** Ne | worker2/Worker/Netsuite/SalesOrder.php, test/@Bala/tests/netsuite_salesorder_payload_tests.php, worker/crons/toga2/prudential/transmissions_to_netsuite.php |
29
30
  | [NetSuite Supporting-Record Webhook Importer (the reusable recipe)](features/netsuite-supporting-record-webhook-importer.md) | A single **repeatable recipe** for porting a legacy daily-pull NetSuite *supporting-record* importer (the lookup/dimension tables behind Forecast2 — Employees, | worker2/Worker/Netsuite/Employee.php, worker2/Worker/Netsuite/Account.php, worker2/Worker/Netsuite/Classification.php, worker2/Worker/Netsuite/Customer.php, worker2/Worker/Netsuite/Item.php, worker2/Worker/Netsuite.php, _underscore/Model/Forecast/Employee.php, _underscore/Model/Forecast/Account.php, _underscore/Model/Forecast/Classification.php, _underscore/Component/Forecast/Db/Db.php, test/@dave/test_employee_lifecycle.php, test/@dave/test_account_lifecycle.php, test/@dave/test_classification_lifecycle.php, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, worker/crons/toga2/forecast2/import_supporting_records.php |
30
31
  | [Background Email-Template Worker (_Worker_Notification_EmailTemplate)](features/notification-email-template.md) | `_Worker_Notification_EmailTemplate::Send(...)` dispatches a **stored, client-defined `EmailTemplates` row off-thread** as a background WorkerJob. | worker2/Worker/Notification/EmailTemplate.php, worker2/Worker/Client/True.php, _underscore/Model/Client/EmailTemplate.php |
31
32
  | [DB-Driven Notification (Internal) Email](features/notification-email.md) | Internal/notification emails (merge-conflict alerts, ops notices — anything system-generated, not client-facing transactional mail) are sent through one worker | worker2/Worker/Notification/Email.php, _underscore/Model/Client/EmailTemplate.php, dbchanges2/Client/2026-06-23a - EmailTemplateWrapper.sql, dbchanges2/Client_True/2026-06-23a - EmailTemplateWrapper.sql |
@@ -6,14 +6,15 @@ project: Worker
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-07-29
10
- owners: [jcardinal, dfranks, mhammontree, tcox]
9
+ updated: 2026-08-10
10
+ owners: [jcardinal, dfranks, mhammontree, tcox, bala]
11
11
  files:
12
12
  - worker2/Worker/
13
13
  - worker2/Controller/Index.php
14
14
  - _underscore/Worker.php
15
15
  related:
16
16
  - ../architecture.md
17
+ - ../../api2/features/api-payload-interceptors.md
17
18
  ---
18
19
 
19
20
  ## Summary
@@ -197,6 +198,31 @@ So **do not design "throw so it retries" — it won't.** To actually retry / sel
197
198
  Rule of thumb: `throw` to surface a genuine failure; `runTask`/in-process loop when the work should
198
199
  be reattempted.
199
200
 
201
+ ## `debug_mode` changes how `runTask` dispatches — and the debug path can't do read-after-write
202
+
203
+ `_Worker::runTask` has two completely different behaviours:
204
+
205
+ | `debug_mode` | What happens |
206
+ |---|---|
207
+ | `1` | a **synchronous** `_ApiRequest` POST to `_Config::api('worker')` — the action runs **inside the caller's open transaction** |
208
+ | `0` | the job is **enqueued to SQS** and runs later, in another process, on its own connection |
209
+
210
+ So under debug mode any action that reads back a row the caller just wrote **always fails** (the
211
+ caller hasn't committed). Two related debug-mode side effects to expect: `_Component_Api_Toga` calls
212
+ `$api->debug(true)`, and `ApiRequest.php` **echoes an HTML dump** — which produces
213
+ *"http_response_code(): Cannot set response code - headers already sent"* inside the worker.
214
+
215
+ ### An ABSENT `WorkerJobs` row is ambiguous — don't read it as "never queued"
216
+
217
+ On the SQS path `runTask` deliberately does **no MySQL work** (stale-connection safety); the
218
+ **worker tier** INSERTs the `WorkerJobs` row on its own connection
219
+ (`worker2/Controller/Index.php` ~L498), using the client-side `uuid` as an idempotency key behind a
220
+ UNIQUE index on `WorkerJobs.uuid`. See [architecture.md](../architecture.md) for the full path.
221
+
222
+ Debugging consequence: **no row means either "the caller never queued it" or "the message was queued
223
+ and never consumed."** Those have completely different fixes. Distinguish them with the **SQS queue
224
+ metrics**, not the database.
225
+
200
226
  ## Gotchas
201
227
 
202
228
  - Class must be **`abstract`** and methods **`public static`** or routing fails.
@@ -233,6 +259,14 @@ be reattempted.
233
259
  commit-before-SQS transaction pattern that the worker relies on.
234
260
 
235
261
  ## Change history
262
+ - 2026-08-10 — Documented that **`debug_mode` changes `runTask`'s dispatch** (1 = synchronous
263
+ `_ApiRequest` POST to `_Config::api('worker')`, running the action inside the caller's open
264
+ transaction so any read-back fails; 0 = SQS enqueue), plus the debug-mode side effects
265
+ (`_Component_Api_Toga` calls `$api->debug(true)` and `ApiRequest.php` echoes an HTML dump →
266
+ *"headers already sent"* in the worker). Added the debugging rule that an **absent `WorkerJobs`
267
+ row is ambiguous** — the worker tier writes the row (`Controller/Index.php` ~L498, uuid as
268
+ idempotency key), so "never queued" and "queued but never consumed" look identical in the DB and
269
+ must be told apart via **SQS queue metrics**. Surfaced building the Elite sales-order push. (bala)
236
270
  - 2026-08-06 — Added the **`[database]` vs `[databaseClient]`** gotcha: `Client_*` schemas live on
237
271
  the tenant cluster configured in worker2's `Config/*.ini` `[databaseClient]` section, not the core
238
272
  `[database]` one (the symptom is a misleading `Unknown database 'Client_Compass'`), and prod's
@@ -6,8 +6,8 @@ project: Worker
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-07-24
10
- owners: ["dfranks"]
9
+ updated: 2026-08-10
10
+ owners: ["dfranks", "bala"]
11
11
  files:
12
12
  - worker2/Worker/Netsuite/SalesOrder.php
13
13
  - worker2/Worker/Netsuite.php
@@ -28,6 +28,7 @@ files:
28
28
  - worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php
29
29
  related:
30
30
  - ./netsuite-opportunity-sync.md
31
+ - ./netsuite-salesorder-outbound-push.md
31
32
  - ../../_underscore/features/netsuite-rest-client.md
32
33
  - ../architecture.md
33
34
  ---
@@ -78,45 +79,25 @@ separate build (see *Related work*).
78
79
  deletes any row NetSuite no longer returns as open (orphans + now-closed lines). Commits
79
80
  `DB_FORECAST` (lazy-transaction discipline).
80
81
 
81
- ## Outbound push (Toga → NetSuite) — REST-only
82
-
83
- The reverse direction (`CreateNetSuite` / `UpdateNetSuite` / `Sync`) pushes a Toga sales order
84
- *into* NetSuite. As of 2026-06-24 it is **REST-only** via the `_underscore`
85
- [`_Component_Api_Netsuite`](../../_underscore/features/netsuite-rest-client.md) client the SOAP
86
- `NetSuiteService` was fully removed from this file.
87
-
88
- - **Item lookup** — SOAP `ItemSearchBasic/search()` → SuiteQL `SELECT id FROM item WHERE itemid =
89
- '<part>'`.
90
- - **Customer resolution** new `resolveNetSuiteCustomerId()`: prefers mapped
91
- `c_netsuiteInternalCustomerId`, else SuiteQL `SELECT id FROM customer WHERE companyname =
92
- '<name>'`; **throws on 0 or >1 matches** (never attach the wrong customer — REST has no SOAP
93
- RecordRef-by-name resolution).
94
- - **`buildNetSuiteOrder()`** returns a REST JSON body: `entity:{id}`,
95
- `item:{items:[{item:{id},quantity,rate}]}`, **date-only** `tranDate` (`YYYY-MM-DD`; SOAP needed
96
- full ISO 8601), `shippingAddress` via `array_filter` dropping null/`''`. `amount` is **omitted**
97
- (NetSuite computes it).
98
- - **Create** → `createRecord()` (reads the new internalId from the 204 `Location` header);
99
- **Update** → `send('PATCH', RECORD_SALES_ORDER.'/'.$id, $body)` (204, no body, throws on non-2xx).
100
- The SOAP `getNetSuiteErrorDetail()` was removed — REST errors surface as exceptions from
101
- `send()`/`createRecord()`.
102
- - **SuiteQL string escaping** uses a private `escapeSuiteQl()` (`str_replace("'", "''", $value)`),
103
- **not** `_Database::escape()` — SuiteQL doubles the quote, so backslash escaping would corrupt
104
- `O'Brien`-style names. (See the [REST client doc](../../_underscore/features/netsuite-rest-client.md).)
82
+ ## Outbound push (Toga → NetSuite) — documented separately
83
+
84
+ The reverse direction (`CreateNetSuite` / `UpdateNetSuite` / `Sync`) lives in the **same file** but is
85
+ a **separate subject with its own doc**:
86
+ [Toga → NetSuite Sales-Order Push](./netsuite-salesorder-outbound-push.md). It went REST-only on
87
+ 2026-06-24 (SOAP `NetSuiteService` fully removed from this file) and was rebuilt on 2026-08-10 as a
88
+ multi-client, mapping-driven worker.
89
+
90
+ **The two halves share no code, constants or helpers** (verified 2026-08-10: this inbound half's
91
+ 2,231 executable tokens are byte-identical to `origin/_production` after the outbound rebuild). Read
92
+ the outbound doc before touching anything below the `OUTBOUND PUSH (REST)` banner and don't assume
93
+ a change there affects the import.
94
+
105
95
  - **Blast radius isolated.** Two other worker2 files still use SOAP and each construct their own
106
96
  `NetSuiteService` (`Worker/Ai/Bdr/Netsuite.php`, `Worker/Client/Aig/ClosedClaims.php`) — untouched;
107
97
  `library/netsuitetoolkit` stays loaded for worker2.
108
98
 
109
- ### Refactor (2026-06-24)
110
-
111
- - `fetchSalesOrder()` 4 positional params → `(array $clientContext, int $salesOrderId)` per the
112
- TOGA 4-param rule (4+ positional must become an assoc-array/named-params signature; PHP named
113
- *arguments* at the call site do **not** satisfy the rule — it targets the **signature**).
114
- - **`Sync()` double-fetch fixed.** `Sync` previously re-ran `getClientContext()` + `fetchSalesOrder()`
115
- and then called Create/Update, which fetched **again** (2× Core DB + 2× Toga API per Sync). It now
116
- fetches **once** and delegates to private `doCreate(array,object,int)` / `doUpdate(object,int)`;
117
- the public `CreateNetSuite`/`UpdateNetSuite` are thin fetch-then-delegate wrappers (public
118
- action-router contract unchanged). Closes a small **TOCTOU** — the open/close decision and the
119
- write now use the same fetched record.
99
+ The 2026-06-24 outbound refactor notes (the `fetchSalesOrder()` 4-param signature rule and the
100
+ `Sync()` double-fetch/TOCTOU fix) moved to that doc as well.
120
101
 
121
102
  ## Data model
122
103
 
@@ -415,6 +396,12 @@ test fixture (it surfaced the stale SO 7181316 above).
415
396
 
416
397
  ## Change history
417
398
 
399
+ - 2026-08-10 — **Split the outbound push (Toga → NetSuite) out into its own doc**
400
+ ([netsuite-salesorder-outbound-push](./netsuite-salesorder-outbound-push.md)) when it was rebuilt as
401
+ a multi-client mapping-driven worker; this doc now covers the **inbound** import only. Verified the
402
+ inbound half is byte-identical to `origin/_production` (2,231 executable tokens) and that the two
403
+ halves share no code, constants or helpers. The 2026-06-24 outbound refactor notes moved with it.
404
+ (bala)
418
405
  - 2026-07-24 — **Resolved the open questions on the OpenOrderItems location / amountDue Power-BI
419
406
  backfill (investigation + planning).** `locationId` is NULL because `Forecast.Locations` is
420
407
  unpopulated and `resolveLocationId` (`worker2/Component/Forecast/Db/Db.php`) has no self-heal on a
@@ -0,0 +1,159 @@
1
+ ---
2
+ title: Toga → NetSuite Sales-Order Push (multi-client, mapping-driven worker)
3
+ framework: "2.0"
4
+ repo: worker2
5
+ project: Worker
6
+ client: shared
7
+ type: feature
8
+ status: active
9
+ updated: 2026-08-10
10
+ owners: ["bala"]
11
+ files:
12
+ - worker2/Worker/Netsuite/SalesOrder.php
13
+ - test/@Bala/tests/netsuite_salesorder_payload_tests.php
14
+ - worker/crons/toga2/prudential/transmissions_to_netsuite.php
15
+ related:
16
+ - ./netsuite-salesorder-open-orders-sync.md
17
+ - ./creating-worker-actions.md
18
+ - ../../_underscore/features/netsuite-rest-client.md
19
+ - ../../api2/features/api-payload-interceptors.md
20
+ - ../../../clients/elite/features/salesorder-netsuite-push.md
21
+ ---
22
+
23
+ ## Summary
24
+
25
+ The **outbound** half of `worker2/Worker/Netsuite/SalesOrder.php` (everything from the
26
+ `OUTBOUND PUSH (REST)` banner down) pushes a Toga sales order **into** NetSuite over REST. As of
27
+ 2026-08-10 it is a **single multi-client worker**: every NetSuite id it needs is resolved by
28
+ following the order's own foreign keys to that record's `c_netsuite*` mapping column, so **nothing
29
+ is hardcoded per client**. It replaces the 1.0 per-client cron pattern
30
+ (`worker/crons/toga2/prudential/transmissions_to_netsuite.php`), which hardcoded one client's
31
+ ids and ran on a timer.
32
+
33
+ The **inbound** NetSuite → Forecast open-orders import lives in the *same file* but is a separate
34
+ subject and was not changed — see
35
+ [NetSuite → Forecast Open-Orders Sync](./netsuite-salesorder-open-orders-sync.md). The two halves
36
+ share no code, constants or helpers (verified 2026-08-10: the inbound half's 2,231 executable
37
+ tokens are identical to `origin/_production`). **Edit one without assuming anything about the
38
+ other.**
39
+
40
+ ## Key files / entry points
41
+
42
+ | Action | Method | Purpose |
43
+ |---|---|---|
44
+ | `Netsuite/SalesOrder/CreateNetSuite` | `CreateNetSuite` | create the order in NetSuite, write the id back |
45
+ | `Netsuite/SalesOrder/UpdateNetSuite` | `UpdateNetSuite` | PATCH an already-synced order |
46
+ | `Netsuite/SalesOrder/Sync` | `Sync` | create-or-update in one call |
47
+
48
+ All three share one signature — **the parameter names are the queue contract** (the dispatcher
49
+ spreads `parameters` as named arguments):
50
+
51
+ ```
52
+ (string $clientIdentifier, string $salesOrderUuid, bool $dryRun = false, ?object $salesOrder = null)
53
+ ```
54
+
55
+ Supporting privates: `getClientContext`, `fetchSalesOrder`, `buildNetSuiteOrder`,
56
+ `resolveNetSuiteCustomerId`, `resolveNetSuiteItemId`, `runSuiteQl`, `escapeSuiteQl`,
57
+ `assertOrderMatchesUuid`, `doCreate`, `doUpdate`. Constants: `SUITEQL_ITEM_TYPE__GROUP = 'Group'`,
58
+ `TOGA_FETCH_DEPTH = 4`.
59
+
60
+ ## How it works
61
+
62
+ 1. `getClientContext($clientIdentifier)` resolves the tenant — the client is a **parameter**, not a
63
+ per-client class or cron.
64
+ 2. The order is either **carried in the job** (`$salesOrder`, see below) or fetched by uuid via
65
+ `fetchSalesOrder` at `TOGA_FETCH_DEPTH = 4`.
66
+ 3. `assertOrderMatchesUuid()` runs **first** in both `doCreate` and `doUpdate` — before the dry-run
67
+ branch and before the already-synced check — so a job carrying the wrong order is reported as a
68
+ mismatch rather than as a misleading "already synced".
69
+ 4. `buildNetSuiteOrder()` assembles the REST body. Every NetSuite id — **customer, item, location,
70
+ shipping method** — is resolved by walking the order's own foreign keys to that related record's
71
+ `c_netsuite*` mapping column; where a value must be looked up in NetSuite instead,
72
+ `runSuiteQl()` does it (string literals escaped with `escapeSuiteQl()` — SuiteQL **doubles** the
73
+ quote, `''`, it never backslashes it).
74
+ 5. **A missing customer / item / location mapping aborts the whole order** rather than sending a
75
+ partial one. A missing **shipping method** is simply omitted so NetSuite applies its default.
76
+ 6. Create → `_Component_Api_Netsuite::createRecord()` (id from the 204/201 `Location` header);
77
+ update → `send('PATCH', …)`. The new internalId is written back to
78
+ `SalesOrders.c_netsuiteInternalSalesOrderId`.
79
+ 7. `dryRun = true` builds and returns the payload without calling NetSuite.
80
+
81
+ ### The job carries the order snapshot — the worker does not re-read it
82
+
83
+ `$salesOrder` lets the **caller** (an api2 post interceptor) pass the already-read order object
84
+ along in the job parameters, so the worker never re-reads it from MySQL. This is deliberate and
85
+ load-bearing: the interceptor runs inside api2's still-open transaction, so a worker in another
86
+ process (on a **reader** connection) cannot see the new row at all. See
87
+ [API Payload Interceptors → queueing background work](../../api2/features/api-payload-interceptors.md).
88
+
89
+ **Rejected alternatives (2026-08-10):**
90
+ - **A bounded retry loop in the worker** — fragile and timing-dependent; it papers over the
91
+ visibility problem instead of removing it.
92
+ - **Copying Prudential's "pass `$payload` straight through" pattern** — the POST body contains only
93
+ uuids, while NetSuite needs the resolved `c_netsuite*` ids; and at depth 3
94
+ `shipToAddress.state.country` is omitted entirely (country sits four levels down: order →
95
+ shipToAddress → state → country), which breaks non-US orders. Hence `TOGA_FETCH_DEPTH = 4`.
96
+ - **Pushing synchronously from the interceptor** — a NetSuite create takes **~12s** (measured); it
97
+ can never be atomic with the MySQL commit anyway, and holding a transaction open across a 12s
98
+ external call risks row locks and dropped connections. The push stays **async**.
99
+
100
+ ## Earlier history worth keeping (2026-06-24, moved from the inbound doc)
101
+
102
+ - **REST-only since 2026-06-24** — the SOAP `NetSuiteService` was fully removed from this file. Item
103
+ lookup went from SOAP `ItemSearchBasic/search()` to SuiteQL `SELECT id FROM item WHERE itemid = …`;
104
+ `tranDate` is **date-only** (`YYYY-MM-DD`; SOAP needed full ISO 8601); `amount` is **omitted** so
105
+ NetSuite computes it; `shippingAddress` is `array_filter`ed to drop null/`''`. SOAP's
106
+ `getNetSuiteErrorDetail()` is gone — REST errors surface as exceptions from
107
+ `send()`/`createRecord()`.
108
+ - **`fetchSalesOrder()` was reduced from 4 positional params** to `(array $clientContext, int
109
+ $salesOrderId)` per the TOGA 4-param rule — 4+ positional parameters must become an
110
+ assoc-array/named-params **signature**; PHP named *arguments* at the call site do **not** satisfy
111
+ the rule.
112
+ - **`Sync()`'s double fetch was removed:** it used to re-run `getClientContext()` +
113
+ `fetchSalesOrder()` and then call Create/Update, which fetched again (2× Core DB + 2× Toga API per
114
+ Sync). It now fetches once and delegates to the private `doCreate`/`doUpdate`, with
115
+ `CreateNetSuite`/`UpdateNetSuite` as thin wrappers — which also closed a TOCTOU, since the decision
116
+ and the write now use the same fetched record.
117
+
118
+ ## Test harness (76 tests, all passing)
119
+
120
+ `test/@Bala/tests/netsuite_salesorder_payload_tests.php` loads the **real** worker file with
121
+ stubbed dependencies and drives the private methods by **reflection**, so it tests shipped code
122
+ rather than a copy (same technique as
123
+ [DB-free interceptor unit testing](../../_underscore/features/model-interceptor-unit-testing.md)).
124
+ Coverage: header fields, mapping resolution, SuiteQL quote-doubling, missing-mapping aborts, dry
125
+ run, create, update, the job-carried-snapshot path, and the uuid-match guard.
126
+
127
+ ## Gotchas / known issues
128
+
129
+ - **⚠ A crash *after* the NetSuite create leaves an orphan.** `createRecord()` reads the `Location`
130
+ header after NetSuite has already created the record, so any failure past that point (the
131
+ `#`-delimiter regex bug did exactly this — see the
132
+ [REST client doc](../../_underscore/features/netsuite-rest-client.md)) leaves a live NetSuite
133
+ order with **nothing in Toga pointing at it**. Two Elite orders were orphaned in NetSuite account
134
+ 1095849 this way. **Never blind-retry a failed create** — check NetSuite for the record first.
135
+ - **This action does not retry itself.** Worker actions have no DLQ and no auto-retry: a throw
136
+ records `isSuccess = 0` and stops. Unlike the 1.0 cron it replaces (which re-selected unsent
137
+ orders every tick and was therefore self-healing), an event-triggered push fires **once**. Pair
138
+ it with a monitor on orders whose `c_netsuiteInternalSalesOrderId` is still NULL — see the
139
+ [Elite client-feature](../../../clients/elite/features/salesorder-netsuite-push.md).
140
+ - **`$dryRun` and `$salesOrder` are optional but positional-by-name.** Callers must pass them as
141
+ named keys in `parameters`; a caller that sends a differently-named key gets
142
+ *"Unknown named parameter"* from the dispatcher — which on a stale instance looks like a code bug
143
+ rather than a deploy problem (see the
144
+ [deploy workflow](../../api2/workflows/codepipeline-codeconnections-deploy.md)).
145
+ - **The inbound importer in this file is untouched and unrelated.** Do not "tidy" shared-looking
146
+ helpers across the banner; there are none.
147
+
148
+ ## Change history
149
+
150
+ - 2026-08-10 — **Rebuilt the outbound push as a single multi-client, mapping-driven worker**
151
+ (`CreateNetSuite` / `UpdateNetSuite` / `Sync`, signature
152
+ `(clientIdentifier, salesOrderUuid, dryRun, salesOrder)`), replacing the per-client 1.0 cron
153
+ pattern: every NetSuite id resolves through the order's own FKs to `c_netsuite*` columns, a
154
+ missing customer/item/location mapping **aborts** the order, a missing shipping method is
155
+ omitted. Added the **job-carried order snapshot** (`?object $salesOrder`, `TOGA_FETCH_DEPTH = 4`
156
+ so `shipToAddress.state.country` is present) and the `assertOrderMatchesUuid()` guard from code
157
+ review. Recorded the rejected alternatives (worker retry loop, Prudential's payload
158
+ pass-through, synchronous push — a NetSuite create measures ~12s) and the 76-test reflection
159
+ harness. Split out of the inbound open-orders doc, which now points here. (bala)
@@ -6,8 +6,8 @@ project: Worker
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-06-30
10
- owners: ["dfranks"]
9
+ updated: 2026-08-10
10
+ owners: ["dfranks", "bala"]
11
11
  files:
12
12
  - worker2/Worker/Netsuite/Employee.php
13
13
  - worker2/Worker/Netsuite/Account.php
@@ -114,9 +114,18 @@ recipe and its two recurring variants so a new one is a fill-in-the-blanks job,
114
114
  no migration needed.
115
115
  - **`save()` does not diff** — see recipe step 2: the change-detect guard before `save()` is what makes
116
116
  a no-change webhook a true no-op.
117
+ - **⚠ Unrelated observation, production, 2026-08-10: three of these job types are at 0% success and
118
+ nobody is watching.** `Core.WorkerJobs` in prod shows `Netsuite/InventoryItem/post` (266 jobs),
119
+ `Netsuite/InventoryItem/put` (1,014) and `Netsuite/ItemGroup/post` (5) with **zero successes**, and
120
+ they are still firing daily. Recorded here as a flag for whoever owns these importers — it was not
121
+ investigated (surfaced incidentally while auditing WorkerJobs for the Elite sales-order push).
117
122
 
118
123
  ## Change history
119
124
 
125
+ - 2026-08-10 — Flagged a **production health observation** (not investigated): prod `Core.WorkerJobs`
126
+ has `Netsuite/InventoryItem/post` (266), `Netsuite/InventoryItem/put` (1,014) and
127
+ `Netsuite/ItemGroup/post` (5) at **0% success**, still running daily and apparently unmonitored.
128
+ Surfaced incidentally while auditing WorkerJobs for the Elite sales-order push. (bala)
120
129
  - 2026-06-30 — Captured the supporting-record webhook-importer recipe and its two variants
121
130
  (self-ref RESOLVE-OR-NULL parent FK + child-orphan delete-guard; Accounts `accttype` enum map; leaf
122
131
  segment naming) after the third+ application (TRUE-79870 Employees, TRUE-79871 Accounts, TRUE-79872
@@ -18,8 +18,8 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
18
18
 
19
19
  ## 2.0 framework
20
20
 
21
- - **_underscore** (_Underscore) _(framework core)_ — 49 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
22
- - **worker2** (Worker) — 44 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
21
+ - **_underscore** (_Underscore) _(framework core)_ — 50 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
22
+ - **worker2** (Worker) — 45 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
23
23
  - **api2** (API) — 22 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
24
24
  - **dbchanges2** (Database Changes) _(framework core)_ — 8 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
25
25
  - **toga2-supply** (TOGa Supply) — 5 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)
@@ -2,6 +2,7 @@
2
2
 
3
3
  | Doc | Framework | Summary | Files |
4
4
  |-----|-----------|---------|-------|
5
+ | [Elite SalesOrder → NetSuite Push (postPost/postPut interceptors → worker2)](features/salesorder-netsuite-push.md) | 2.0 | Elite orders created in Toga are pushed into NetSuite **event-driven**, not on a cron. | _underscore/Model/Elite/SalesOrder.php, worker2/Worker/Netsuite/SalesOrder.php, dbchanges2/Client_Elite/2026-08-10 - SalesOrderNetsuiteInterceptors.sql |
5
6
  | [Elite — supply2 frontend scope (Inventory + Service Requests, both built)](features/supply2-scope.md) | 2.0 | Scope for onboarding Elite to the `toga2-supply` frontend (host `ELITE`). | toga2-supply/ELITE-CLIENT-TASK-NOTES.md, toga2-supply/src/pages/Inventory/viewModel/FIELDS/DUMMYGROUPOPTIONS.ts, toga2-supply/src/pages/Inventory/viewModel/FIELDS/INVENTORYPAGEFIELDS.ts, toga2-supply/src/pages/Inventory/viewModel/index.ts, toga2-supply/src/pages/Inventory/listing/InventoryPage.tsx, toga2-supply/src/pages/Inventory/listing/InventoryRouter.tsx, toga2-supply/src/pages/Inventory/listing/InventorySubTablePage.tsx, toga2-supply/src/utils/resolveClientHostName.ts, toga2-supply/src/utils/convertConstructorColumnTitles.ts, toga2-supply/src/pages/Orders/OrdersPage.tsx, toga2-supply/src/pages/Orders/viewModel/FIELDS/ELITE/BASE.json, toga2-supply/src/pages/Orders/viewModel/useOrdersPageViewModel.ts, toga2-supply/src/pages/Orders/api/OrdersApi.ts, toga2-supply/src/pages/Orders/view/OrderView/viewModel/useOrderDetailsViewModel.ts, toga2-supply/src/components/ui/Tables/PrimaryTable/helperFunctions/renderModalContent.tsx, toga2-supply/src/components/layout/SlideMenu/SlideMenu.tsx, toga2-supply/package.json |
6
7
  | [Elite — stale TableView config (11 dead Core.RecordFields across 9 views)](features/supply2-tableview-config-drift.md) | 2.0 | `Client_Elite`'s `TableViewJoins` predate **two** platform bridge-table migrations and still reference **11 deleted `Core.RecordFields` ids (211, 321, 932, 358, | dbchanges2/Client_Elite/, dbchanges2/Client_Nychh/2026-06-19a - FixItemFulfillmentTableViewTrackingAndRoot.sql |
7
- | [Elite](profile.md) | 2.0 | Elite is a managed-services client that uses **Freshservice** as their helpdesk platform. | worker2/Worker/Elite.php, library/app/api/toga2.php |
8
+ | [Elite](profile.md) | 2.0 | Elite is a managed-services client that uses **Freshservice** as their helpdesk platform. | worker2/Worker/Elite.php, library/app/api/toga2.php, _underscore/Model/Elite/SalesOrder.php |
@@ -0,0 +1,121 @@
1
+ ---
2
+ title: "Elite SalesOrder → NetSuite Push (postPost/postPut interceptors → worker2)"
3
+ framework: "2.0"
4
+ repo: _underscore
5
+ project: _Underscore
6
+ client: elite
7
+ type: client-feature
8
+ status: active
9
+ updated: 2026-08-10
10
+ owners: ["bala"]
11
+ files:
12
+ - _underscore/Model/Elite/SalesOrder.php
13
+ - worker2/Worker/Netsuite/SalesOrder.php
14
+ - dbchanges2/Client_Elite/2026-08-10 - SalesOrderNetsuiteInterceptors.sql
15
+ related:
16
+ - ../../../2.0/apps/worker2/features/netsuite-salesorder-outbound-push.md
17
+ - ../../../2.0/apps/api2/features/api-payload-interceptors.md
18
+ - ../../../2.0/apps/_underscore/features/netsuite-rest-client.md
19
+ - ../profile.md
20
+ ---
21
+
22
+ ## Summary
23
+
24
+ Elite orders created in Toga are pushed into NetSuite **event-driven**, not on a cron.
25
+ `_Model_Elite_SalesOrder::postPost` queues the worker action
26
+ `Netsuite/SalesOrder/CreateNetSuite` when an order is created, and `postPut` queues
27
+ `UpdateNetSuite` when the **shipping method** changes. The worker itself is the shared,
28
+ client-agnostic push documented in
29
+ [Toga → NetSuite Sales-Order Push](../../../2.0/apps/worker2/features/netsuite-salesorder-outbound-push.md) —
30
+ Elite contributes only the trigger and the interceptor registration data.
31
+
32
+ This is Elite's override of the shared push path; it is the second client on that path (Rate uses a
33
+ different shape — a CashSale built inline in `postPost`).
34
+
35
+ ## How it works
36
+
37
+ 1. **`postPost`** calls `parent::postPost()` **first** (so approval records are still created),
38
+ then reads the order for NetSuite and queues `Netsuite/SalesOrder/CreateNetSuite`.
39
+ 2. **`postPut`** does the same and queues `Netsuite/SalesOrder/UpdateNetSuite`, but only when the
40
+ shipping method changed.
41
+ 3. **Both bail immediately when `$api->httpPayload->c_netsuiteInternalSalesOrderId` is set** — such
42
+ an order arrived **from** NetSuite via the inbound sync and must not be echoed back.
43
+ 4. The private `readSalesOrderForNetsuite()` reads the order **inside the same request** via
44
+ `$api->internalApiRequest()`, pinned to the writer, and the resulting order object is passed
45
+ along in the job parameters so the worker never re-reads it. The full set of timing /
46
+ read-replica / envelope rules that make this correct is documented once, in
47
+ [API Payload Interceptors → queueing background work from a post interceptor](../../../2.0/apps/api2/features/api-payload-interceptors.md).
48
+ 5. `runTask` is wrapped in `try/catch` + `error_log`, so a broken queue can never fail the sales
49
+ order POST itself.
50
+
51
+ ## Interceptor registration is per-environment DATA — and prod has none
52
+
53
+ `Client_Elite.ApiPayloadInterceptors` is **EMPTY in production**. Without rows, `postPost`/`postPut`
54
+ **never run**: the order saves, nothing else happens, and there is no error anywhere. The
55
+ migration `dbchanges2/Client_Elite/2026-08-10 - SalesOrderNetsuiteInterceptors.sql` registers
56
+ `POST`/`POST` and `POST`/`PUT` for **`recordId 14` (sales-orders)**, guarded with
57
+ `WHERE NOT EXISTS` so a replay cannot create duplicates — **a duplicate row would queue the same
58
+ NetSuite job twice for one order.** (Check the file carries dbchanges2's mandatory
59
+ `YYYY-MM-DD<letter>` suffix before it merges.)
60
+
61
+ `recordId 14` and the `Apis` ids (**1 = Agilant, 2 = Elite**) happen to match between dev-sandbox
62
+ and production — **verify them per environment rather than assuming.**
63
+
64
+ ## Current production state (2026-08-10)
65
+
66
+ - **264 Elite sales orders; 258 carry a NetSuite id, 6 do not.**
67
+ - **No order has yet completed the full interceptor → SQS → worker → NetSuite path end to end.**
68
+ The furthest run (`WorkerJobs 22743`) queued correctly **with** the order snapshot and with
69
+ `shipToAddress.state.country = CA` present, then failed on a **stale SQS-consumer instance**
70
+ (see the [deploy workflow](../../../2.0/apps/api2/workflows/codepipeline-codeconnections-deploy.md)).
71
+ - One order was created in NetSuite successfully by **direct worker invocation**
72
+ (`WorkerJobs 22452`): NetSuite id **7358493** written back to `SalesOrders.259`.
73
+ - **Two orphaned Elite orders exist in live NetSuite account 1095849** from the `createRecord()`
74
+ regex bug — created in NetSuite with nothing in Toga pointing at them.
75
+
76
+ ## What the existing production precedent does and does not prove
77
+
78
+ The interceptor → `runTask` → SQS → worker2 chain **is** proven at volume in production
79
+ (`Core.WorkerJobs`): `Notification/EmailTemplate/Send` 1091/1091 succeeded and
80
+ `Client/Prudential/reports/ReqRejectionEmail/sendRejectionEmail` 20/20 — both queued from
81
+ interceptors. worker2 ↔ NetSuite is healthy too: `Netsuite/SalesOrder/put` 35,259 jobs / 35,220
82
+ succeeded; `Netsuite/Webhook` 46,273 / 46,207.
83
+
84
+ **What Prudential does NOT prove:** it passes `$payload` straight through and never reads back, so
85
+ the **in-transaction read + writer-pinning** this feature depends on has **no production
86
+ precedent**. Treat the first prod Elite order as a real first run.
87
+
88
+ ## Gotchas / known issues
89
+
90
+ - **⚠ No retry safety net — this is the structural difference from the 1.0 cron.** The 1.0
91
+ Prudential cron selected on `PurchaseOrders.dtSubmitted IS NULL`, so a failed run healed itself
92
+ on the next tick. An interceptor fires **once**; a failed push leaves an order with
93
+ `c_netsuiteInternalSalesOrderId IS NULL` and **nothing watching**.
94
+ **Recommended follow-up (NOT built): an alert-only monitor** on orders older than ~15 minutes
95
+ that are still NULL. A **blind retry sweeper was deliberately rejected** — a previous attempt may
96
+ have created the NetSuite order and only failed the write-back, so retrying would duplicate.
97
+ - **⚠ A missing `ApiPayloadInterceptors` row is silent.** "The push didn't happen" in a given
98
+ environment is a **data** question first (`SELECT` the client's interceptor rows), never a code
99
+ question.
100
+ - **api2's `error_log()` output is not in CloudWatch.** worker2 ships
101
+ `/var/log/php-fpm/www-error.log` to the `worker` log group via a postdeploy hook; **api2 has no
102
+ equivalent hook**, so the interceptor's silent-skip messages must be pulled from the EB
103
+ request/full logs.
104
+ - **Orders that arrived from NetSuite must not be pushed back** — the
105
+ `c_netsuiteInternalSalesOrderId`-on-payload bail is what prevents an echo loop. Keep it in any
106
+ new hook on this model.
107
+
108
+ ## Change history
109
+
110
+ - 2026-08-10 — **Built the Elite push trigger**: `_Model_Elite_SalesOrder::postPost` queues
111
+ `Netsuite/SalesOrder/CreateNetSuite`, `postPut` queues `UpdateNetSuite` on a shipping-method
112
+ change; both call `parent::postPost()` first, bail when the payload carries
113
+ `c_netsuiteInternalSalesOrderId` (order came from the inbound sync), read the order in-request via
114
+ `internalApiRequest` (writer-pinned) and pass the snapshot in the job, and wrap `runTask` in
115
+ try/catch so a queue failure cannot fail the POST. Added
116
+ `dbchanges2/Client_Elite/2026-08-10 - SalesOrderNetsuiteInterceptors.sql` (recordId 14, POST/POST
117
+ + POST/PUT, `NOT EXISTS`-guarded) because **`Client_Elite.ApiPayloadInterceptors` is empty in
118
+ prod**. Recorded the missing retry safety net vs. the 1.0 cron (alert-only monitor recommended;
119
+ blind retry sweeper rejected — duplicate risk), the two orphaned NetSuite orders in account
120
+ 1095849, and the current state: 264 orders / 258 with a NetSuite id, no end-to-end run yet
121
+ (NetSuite 7358493 created by direct worker invocation). (bala)
@@ -6,20 +6,24 @@ apps:
6
6
  - library
7
7
  - toga2-supply
8
8
  - dbchanges2
9
+ - _underscore
10
+ - api2
9
11
  project: Worker
10
12
  client: elite
11
13
  type: profile
12
14
  status: active
13
15
  updated: 2026-08-10
14
- owners: [snaredla, apeterson, tcox]
16
+ owners: [snaredla, apeterson, tcox, bala]
15
17
  files:
16
18
  - worker2/Worker/Elite.php
17
19
  - library/app/api/toga2.php
20
+ - _underscore/Model/Elite/SalesOrder.php
18
21
  related:
19
22
  - 2.0/apps/worker2/features/elite-freshservice-sync.md
20
23
  - 1.0/apps/library/features/elite-freshservice-sync.md
21
24
  - features/supply2-scope.md
22
25
  - features/supply2-tableview-config-drift.md
26
+ - features/salesorder-netsuite-push.md
23
27
  ---
24
28
 
25
29
  ## Summary
@@ -42,6 +46,12 @@ view is built on (joining Tickets / Customers / ServiceRequestTypes / SalesOrder
42
46
  dev-sandbox have no linked sales order** — whether that is a legitimate lifecycle state is an open
43
47
  question for backend/product (see supply2-scope).
44
48
 
49
+ **Elite orders are pushed to NetSuite event-driven (2026-08).** `_Model_Elite_SalesOrder`'s
50
+ `postPost`/`postPut` interceptors queue the shared worker2 push. Elite's
51
+ `ApiPayloadInterceptors` table is **empty in production**, so the registration migration is the
52
+ go-live gate — see [salesorder-netsuite-push](features/salesorder-netsuite-push.md). This work adds
53
+ **`_underscore`** and **`api2`** to Elite's app scope.
54
+
45
55
  **Tenant/test-account notes.** `Core.Domains` rows for Elite already exist in **every**
46
56
  environment (dev `http://elite.togasupply`, beta, production, sandbox-dev) with no dbchanges2 Core
47
57
  migration creating them, and Elite has **no `ClientAuthentications` row** — supply2 login is
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "toga-ai",
3
- "version": "1.0.548",
3
+ "version": "1.0.549",
4
4
  "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
5
5
  "keywords": [
6
6
  "claude",