toga-ai 1.0.623 → 1.0.625

package/knowledge/2.0/apps/_underscore/features/netsuite-rest-client.md

@@ -6,7 +6,7 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-08-10
+updated: 2026-08-20
 owners: ["dfranks", "jcardinal", "bala"]
 files:
   - _underscore/Component/Api/Netsuite/Netsuite.php
@@ -127,6 +127,23 @@ Updates do **not** need a new helper. A NetSuite record PATCH returns 204 with n
 `send()` already throws on non-2xx — so the update path is just `send('PATCH', RECORD.'/'.$id,
 $body)`. Only the create path (which must read the `Location` header) required the new primitive.
 
+### Idempotent creates — stamp an `externalId`, look it up with `eid:`
+
+`createRecord()` has no idempotency of its own, so a caller that must not create twice should put an
+**`externalId`** on the record it creates. Two things follow, and both are cheap:
+
+- **NetSuite itself rejects a duplicate `externalId`** within the account, so it is a backstop even
+  when the caller's own pre-check misses.
+- **A record can be read back by external id** with `GET /record/v1/<record>/eid:<externalId>` — no
+  SuiteQL involved. That matters because whether SuiteQL can **filter** on the native
+  `transaction.externalid` column in account 1095849 is **not yet verified** (see the custom-column
+  trap below for why a silent zero-row filter is a real risk here). `eid:` is the fallback.
+
+Account 1095849 is **shared by every client**, so an external id must be globally unique and
+prefixed — never a per-client record number. Worked example:
+[Toga → NetSuite sales-order push](../../worker2/features/netsuite-salesorder-outbound-push.md)
+(`toga-so-<salesOrder uuid>`).
+
 ## SuiteQL string-literal escaping — double the quote, never backslash
 
 SuiteQL is **ANSI SQL sent to NetSuite as the JSON `q` string** — it is **not** executed against
@@ -175,6 +192,13 @@ doc.)
 
 ## Change history
 
+- 2026-08-20 — Added **idempotent creates via `externalId`**: `createRecord()` has no idempotency, so
+  stamp an `externalId` (NetSuite rejects a duplicate within the account) and read the record back
+  with `GET /record/v1/<record>/eid:<externalId>` — the non-SuiteQL fallback, which matters because
+  **filtering** on the native `transaction.externalid` column is still unverified in account 1095849.
+  Noted that the account is shared by every client, so external ids must be globally unique and
+  prefixed. (bala)
+
 - 2026-08-10 — **Fixed the `createRecord()` trailing-id regex** (`Netsuite.php` ~L198): re-delimited
   `'#/(\d+)(?:[?#]|$)#'` → `~`, so `preg_match` no longer returns `false` with
   *"Unknown modifier ']'"* and the method can actually return an internalId. Recorded the durable

package/knowledge/2.0/apps/_underscore/features/record-change-audit-log.md

@@ -6,7 +6,7 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-08-13
+updated: 2026-08-20
 owners: [bala]
 files:
   - _underscore/Model/Client/Logs/Record.php
@@ -15,6 +15,9 @@ files:
 related:
   - ../../api2/features/request-logging.md
   - ../../../../clients/compass-usa/features/approval-decision-flow.md
+  - ../../../../clients/compass-usa/features/stranded-approval-reassignment.md
+  - ../../api2/features/api-payload-interceptors.md
+  - ../../worker2/features/creating-worker-actions.md
 ---
 
 ## Summary
@@ -90,6 +93,19 @@ single record: the milestone notes are the chapter headings, the field changes a
   automatically. Real example:
   [Compass PEOPLE-file user lifecycle](../../../../clients/compass-usa/features/people-file-user-lifecycle.md),
   whose deactivation batch is a raw `UPDATE Users SET isActive = 0 WHERE id IN (...)`.
+- **⚠⚠ `_Model_Client_Logs_Record::addNote()` IS A SILENT NO-OP OUTSIDE api2.** `addNote()` only
+  **stages** rows into the class's static `$_notes` array. The **only** thing that ever flushes that
+  array is api2's `Component/Api/V2/V2.php`. Nothing in `_underscore` and nothing in `worker2`
+  flushes it. So **any worker or cron that calls `addNote()` loses the note entirely** — no error, no
+  warning, no row. This is the most dangerous failure mode on this page, because the calling code
+  reads as if it audited itself.
+
+  A worker that needs a note must either drain `_Model_Client_Logs_Record::$_notes` itself and INSERT
+  into `Logs_<Tenant>.Record`, or write the `Record` rows directly. Note that
+  **`Logs_<Tenant>.Record.clientId` is `NOT NULL`**, and a worker generally does not have an `$api`
+  to read the client id from — resolve it from `Core` (`Clients` JOIN `Databases` on `name`). Known
+  ids: `Client_Compass` = **2** (`Logs_Compass`), `Client_CompassCanada` = **43**
+  (`Logs_CompassCanada`). This applies to **every** client, not just Compass.
 - **⚠ `Logs_*` and `Core` are on different database clusters — you cannot join them in one query.**
   `Record.recordId` → `Core.Records` and `RecordField.recordFieldId` → `Core.RecordFields` are
   logical foreign keys across a cluster boundary. Resolve the `Core` ids in a **separate** query and
@@ -104,6 +120,14 @@ single record: the milestone notes are the chapter headings, the field changes a
 
 ## Change history
 
+- 2026-08-20 — Added the **`addNote()` is a silent no-op outside api2** gotcha: `addNote()` only
+  stages into the static `$_notes` array and **only api2's `Component/Api/V2/V2.php` ever flushes
+  it**, so every `addNote()` call from `_underscore` or `worker2` loses its note with no error. A
+  worker must drain `$_notes` itself or write `Record` rows directly, and must resolve the
+  `NOT NULL` `Record.clientId` from `Core` (`Clients` JOIN `Databases` on `name`) because it has no
+  `$api` — `Client_Compass` = 2, `Client_CompassCanada` = 43. Found while building the Compass
+  [stranded approval reassignment](../../../../clients/compass-usa/features/stranded-approval-reassignment.md)
+  engine, but it is framework-wide and client-agnostic. (bala)
 - 2026-08-13 — Documented the `Logs_<Tenant>.Record` / `RecordField` field-change audit trail as its
   own subject: the two row shapes (field-change rows with `RecordField` children vs. milestone rows
   carrying only a `note`), scoping by `Record.recordId` = the **`Core.Records` model id**
@@ -119,3 +143,7 @@ single record: the milestone notes are the chapter headings, the field changes a
   request log, the other half of "what happened to this record".
 - [Compass Approval-Decision Flow](../../../../clients/compass-usa/features/approval-decision-flow.md)
   — the investigation this technique was generalized from.
+- [Stranded Approval Reassignment](../../../../clients/compass-usa/features/stranded-approval-reassignment.md)
+  — the worker that hit the `addNote()` no-op and had to write its own `Record` rows.
+- [API Payload Interceptors](../../api2/features/api-payload-interceptors.md) — why api2 is the only
+  tier that flushes staged notes.

package/knowledge/2.0/apps/api2/INDEX.md

@@ -3,7 +3,7 @@
 | Doc | Summary | Files |
 |-----|---------|-------|
 | [API (api2 / TOGa API v2) Architecture](architecture.md) | `api2` is the backend powering the public **TOGa 2.0 API**. | api2/Controller/Index.php, api2/Component/Api/V2/V2.php, api2/Component/Api/Cxml/Cxml.php, api2/Component/Api/V2/Response/Response.php, api2/Config/ |
-| [API Payload Interceptors (metadata-registered prePost/postPut model hooks)](features/api-payload-interceptors.md) | A `_Model`'s `prePost()` / `postPost()` / `prePut()` / `postPut()` hooks are **not** called by the model. | api2/Component/Api/V2/V2.php, api2/Controller/Index.php, _underscore/Model/Client/ItemFulfillment.php, _underscore/Model/Rate/Entitlement.php, _underscore/Model/Compass/PurchaseOrder.php, _underscore/Model/Elite/SalesOrder.php, _underscore/Model/Elite/ServiceRequest.php, _underscore/Model/Compass/SalesOrder.php, _underscore/Model/Compass/SalesOrderItem.php, dbchanges2/Core/2026-06-30a - ItemFulfillmentStageDefaultInterceptor.sql, dbchanges2/Core/2026-08-06a - Service request and sales order payload interceptors.sql, dbchanges2/Client_Compass/2026-08-06 - RemoveBrokenSalesOrderItemPostPostInterceptor.sql, dbchanges2/Client_Compass/2026-08-14a - AddSalesOrderInactiveStandaloneItemPreInterceptors.sql, dbchanges2/Client_CompassCanada/2026-08-14a - AddSalesOrderInactiveStandaloneItemPreInterceptors.sql |
+| [API Payload Interceptors (metadata-registered prePost/postPut model hooks)](features/api-payload-interceptors.md) | A `_Model`'s `prePost()` / `postPost()` / `prePut()` / `postPut()` hooks are **not** called by the model. | api2/Component/Api/V2/V2.php, api2/Controller/Index.php, _underscore/Model/Client/ItemFulfillment.php, _underscore/Model/Rate/Entitlement.php, _underscore/Model/Compass/PurchaseOrder.php, _underscore/Model/Client/SalesOrder.php, _underscore/Model/Elite/SalesOrder.php, _underscore/Model/Elite/ServiceRequest.php, _underscore/Model/Compass/SalesOrder.php, _underscore/Model/Compass/SalesOrderItem.php, _underscore/Model/Compass/ApprovalDecision.php, dbchanges2/Core/2026-06-30a - ItemFulfillmentStageDefaultInterceptor.sql, dbchanges2/Core/2026-08-06a - Service request and sales order payload interceptors.sql, dbchanges2/Client_Compass/2026-08-06 - RemoveBrokenSalesOrderItemPostPostInterceptor.sql, dbchanges2/Client_Compass/2026-08-14a - AddSalesOrderInactiveStandaloneItemPreInterceptors.sql, dbchanges2/Client_CompassCanada/2026-08-14a - AddSalesOrderInactiveStandaloneItemPreInterceptors.sql |
 | [Multi-Client (Cross-Client) Data Retrieval](features/cross-client-data-retrieval.md) | A single authenticated V2 GET listing can return records across **many** clients (designed for 1000+) that the caller is entitled to, honoring **each target cli | api2/Component/Api/CrossClient/CrossClient.php, api2/Component/Api/V2/V2.php, api2/Controller/Index.php, _underscore/Model/Cache/Table.php, _underscore/Model/Cache/Tables/Client.php, _underscore/Model/Core/Record.php, worker2/Worker/Platform/Cache.php, worker2/Controller/Index.php, worker2/_.php, dbchanges2/Cache/2026-06-30a - MultiClientCacheTables.sql, dbchanges2/Core/2026-06-30b - CacheClusterRegistrationAndRecordTtl.sql, dbchanges2/Core/2026-07-27a - PlatformCacheCleanCron.sql |
 | [cXML ShipNotice Gateway (ASN ingestion, carrier resolution, per-client provisioning)](features/cxml-shipnotice-gateway.md) | `_Component_Api_Cxml` accepts a supplier `ShipNoticeRequest` and translates it into a `POST /v2/advance-shipping-notices` on the V2 JSON engine. | api2/Component/Api/Cxml/Cxml.php, api2/Component/Api/V2/V2.php, api2/Controller/Index.php |
 | [Encrypted-User-UUID Auth Handoff (/auth/encrypted-user-uuid)](features/encrypted-user-uuid-auth-handoff.md) | `POST /auth/encrypted-user-uuid` is the intended **cross-client / SSO-handoff identity mechanism**: given an encrypted `{client, user}` UUID pair, it mints a fr | api2/Component/Api/CrossClient/CrossClient.php |

package/knowledge/2.0/apps/api2/features/api-payload-interceptors.md

@@ -6,7 +6,7 @@ project: API
 client: shared
 type: feature
 status: active
-updated: 2026-08-18
+updated: 2026-08-20
 owners: ["mhammontree", "dfranks", "bala", "snaredla", "jcardinal"]
 files:
   - api2/Component/Api/V2/V2.php
@@ -14,10 +14,12 @@ files:
   - _underscore/Model/Client/ItemFulfillment.php
   - _underscore/Model/Rate/Entitlement.php
   - _underscore/Model/Compass/PurchaseOrder.php
+  - _underscore/Model/Client/SalesOrder.php
   - _underscore/Model/Elite/SalesOrder.php
   - _underscore/Model/Elite/ServiceRequest.php
   - _underscore/Model/Compass/SalesOrder.php
   - _underscore/Model/Compass/SalesOrderItem.php
+  - _underscore/Model/Compass/ApprovalDecision.php
   - dbchanges2/Core/2026-06-30a - ItemFulfillmentStageDefaultInterceptor.sql
   - dbchanges2/Core/2026-08-06a - Service request and sales order payload interceptors.sql
   - dbchanges2/Client_Compass/2026-08-06 - RemoveBrokenSalesOrderItemPostPostInterceptor.sql
@@ -32,6 +34,9 @@ related:
   - ../../_underscore/features/recursive-item-fulfillments.md
   - ../../_underscore/features/config-group-access.md
   - ../../worker2/features/creating-worker-actions.md
+  - ../../../clients/compass-usa/features/stranded-approval-reassignment.md
+  - ../../_underscore/features/record-change-audit-log.md
+  - ../../_underscore/features/email-template-sending.md
   - ../../../clients/elite/features/salesorder-netsuite-push.md
   - ../../../clients/compass-usa/features/sales-order-line-renumbering.md
 ---
@@ -394,7 +399,10 @@ request.** Never conclude "it works on beta" for an in-request read-after-write.
 - Pass **`false`** for `internalApiRequest`'s 5th argument (`$throwExceptionsOnError`). Its throw is
   guarded by `!empty($this->response->messages)` on the **shared** response object, and
   `json_encode([])` is the **truthy** string `"[]"` — so it throws even with **zero real errors**
-  once any message (even a warning) is present anywhere in the request.
+  once any message (even a warning) is present anywhere in the request. **This is not theoretical:**
+  it took Elite's production `POST /v2/sales-orders` down on 2026-08-20, from the *parent* model's
+  own `internalApiRequest` (see [the inherited-chain section](#-activating-a-clients-first-interceptor-row-also-switches-on-the-whole-inherited-chain)).
+  The throw site is `V2.php:2416`.
 
 ### Two more failure modes to expect on this path
 
@@ -408,6 +416,83 @@ request.** Never conclude "it works on beta" for an in-request read-after-write.
   ambiguous** ("never queued" vs "queued, never consumed"). Both are covered in
   [Creating Worker Actions](../../worker2/features/creating-worker-actions.md).
 
+## ⚠ Activating a client's FIRST interceptor row also switches on the whole INHERITED chain
+
+A client whose `ApiPayloadInterceptors` set is **empty** has never run `postPost` at all — which
+means it has also never run the **parent** `_Model_Client_<Record>::postPost()` that a client
+override calls via `parent::`. Inserting the first row does not merely enable the code you wrote; it
+enables **every inherited statement above it**, for the first time, in that environment, against that
+client's data.
+
+**Worked example — Elite, production, 2026-08-20** (Sentry `API-1Z8`, 12 events, escalating).
+Activating two `sales-orders` rows made `POST /v2/sales-orders` return **`EO-1` 500 with the order
+rolled back**, and the exception message was literally **`[]`**:
+
+1. Elite's `postPost` calls `parent::postPost()` first (correctly — approvals must still be created);
+2. the parent, `_underscore/Model/Client/SalesOrder.php:359`, reads `/approval-templates` via
+   `internalApiRequest` with **throwing left ON**;
+3. Elite has **zero `ApprovalTemplates`** rows, so that nested GET returns empty and puts a message
+   on the **shared** response object;
+4. `V2.php:2416` does `json_encode(array_filter($messages, ...ERROR...))` → the string `"[]"` →
+   **truthy** → throw, with no real error in it;
+5. the controller catches it, returns `EO-1`, and **rolls back** the write.
+
+**Reading the signature:** an exception whose **message is `[]`** on a write route means *this* — a
+message-array truthiness throw. Nothing is wrong with the payload, and the client's own interceptor
+code may never have executed.
+
+**Two fixes, both in shared code:**
+
+- pass **`false`** as `internalApiRequest`'s 5th argument in the **parent** (narrow, per-caller); or
+- fix **`V2.php:2416`** to test the **filtered array** rather than `json_encode()` of it — which
+  fixes **every** caller on this path. Senior-owned: raise it rather than patching one caller.
+
+**Before activating a row for a client that has none:** walk the parent chain for the derived method
+and confirm the data each inherited step depends on actually exists for that tenant (here: at least
+one `ApprovalTemplates` row). This is the same reason activation must be staged `isActive = 0` first
+— see the gotchas.
+
+## Calling a real interceptor FROM a worker — the minimal `$api` shim
+
+The reverse direction of the section above: not "queue a worker from an interceptor" but **"run the
+production interceptor from a worker"**, so a backfill or repair job reuses the business rules
+instead of reimplementing them. This is the right instinct — a duplicated auto-approve threshold or
+email-routing rule will drift from the real one.
+
+**The technique: measure the actual `$api` surface, then fake exactly that.** `$api` looks
+un-fakeable because it is huge, but any single interceptor chain touches very little of it. Trace the
+whole chain reachable from the entry point and you usually find a handful of properties.
+
+Worked measurement (2026-08-20, `_Model_Compass_ApprovalDecision::postPut()`): the **entire** `$api`
+surface reachable from that method is **three properties** — `$api->client`, `$api->httpPayload`,
+`$api->recordUuid`. So a worker can build a 3-property shim and call the genuine `prePut`/`postPut`.
+The Compass repair engine does exactly this; see
+[Stranded Approval Reassignment](../../../clients/compass-usa/features/stranded-approval-reassignment.md).
+
+Rules for doing it safely:
+
+- **Trace the chain, per class — do not generalize the surface from a sibling.**
+  `$api->internalApiRequest` and `$api->route` *do* appear in
+  `_underscore/Model/Compass/SalesOrder.php`, but only inside
+  `_Model_Compass_SalesOrder::postPut()` — a **different class**, not in the `ApprovalDecision`
+  chain. Assuming "the Compass models need `internalApiRequest`" would have produced a shim that was
+  both wrong and larger than needed. Re-measure for every entry point.
+- **A shim is a hard-fail contract.** If the interceptor later starts reading a fourth property, the
+  worker gets a fatal on a `stdClass` with no such property. That is the correct outcome (loud, not
+  silent), but it means the shim must be built where a reviewer will see it, not buried.
+- **Emails still go out.** The interceptors send real mail through the real pipeline, so a worker
+  that runs them is a sending path — cap it, add a `dryRun`, and commit per unit of work, because a
+  sent email cannot be rolled back.
+- **Do not use the interceptor path just to send an email.** For a plain send there is a documented
+  non-API entry point: `_Model_Client_EmailTemplate::send(string $clientIdentifier, ...)`, versus the
+  `$api`-based `sendEmail()`. See
+  [Email Template Sending](../../_underscore/features/email-template-sending.md). Faking `$api` is
+  for reusing *business logic*, not for reaching the mailer.
+- **Notes will not be logged.** `_Model_Client_Logs_Record::addNote()` is flushed only by
+  `Component/Api/V2/V2.php`, so notes staged by the interceptor you just ran are **dropped** in a
+  worker. See
+  [Record Change Audit Log](../../_underscore/features/record-change-audit-log.md).
+
 ## Worked example — the EV-10 that was not a code bug
 
 `_Model_Client_ItemFulfillment::prePost()` defaults `itemFulfillmentStageId` to the shipped stage on
@@ -442,6 +527,11 @@ and the failing environment**. It is a small table, and the drift is usually exa
   `Core.Records` row, so every change is direct SQL and nothing records who made it. Snapshot the
   client's row set into the ticket before and after any change — that dump is your only rollback
   and your only history.
+- **⚠ The FIRST row for a client activates INHERITED code, not just yours.** An empty
+  `ApiPayloadInterceptors` set means the shared `_Model_Client_<Record>::postPost()` has never run
+  for that tenant either. Elite's first two prod rows took `POST /v2/sales-orders` down through the
+  **parent's** `/approval-templates` read — exception message literally `[]`. Check the parent chain
+  and its data prerequisites before flipping a client's first row on.
 - **⚠ A migration that enables an interceptor must insert `isActive = 0`.** Activation is a *data*
   change that switches on a *code* path; if the PHP defining the method is not confirmed deployed to
   that environment, the row takes the endpoint down. Insert inactive, verify the deploy, then flip
@@ -505,6 +595,26 @@ and the failing environment**. It is a small table, and the drift is usually exa
 
 ## Change history
 
+- 2026-08-20 (later pass) — Recorded that **activating a client's FIRST interceptor row switches on
+  the entire INHERITED chain**, with the Elite production outage as the worked example: two new
+  `sales-orders` rows made every `POST /v2/sales-orders` return `EO-1` 500 with the order rolled
+  back, because the *parent* `_Model_Client_SalesOrder::postPost()` — running for the first time for
+  that tenant — reads `/approval-templates` with `internalApiRequest` throwing **on**, Elite has zero
+  `ApprovalTemplates`, and `V2.php:2416`'s `json_encode(array_filter(...))` yields the **truthy**
+  string `"[]"`. Documented the diagnostic signature (an exception whose message is literally `[]`)
+  and the two shared-code fixes (parent passes `false`, or `V2.php:2416` tests the filtered
+  **array** — senior-owned, not changed). (bala)
+- 2026-08-20 — Added **"Calling a real interceptor FROM a worker — the minimal `$api` shim"**: the
+  reverse of the queue-from-interceptor path. A worker can reuse production interceptor logic by
+  measuring the `$api` surface an interceptor chain actually touches and faking only that — verified
+  that the **entire** surface reachable from `_Model_Compass_ApprovalDecision::postPut()` is three
+  properties (`client`, `httpPayload`, `recordUuid`). Recorded that the surface must be re-measured
+  **per class** (`$api->internalApiRequest` / `$api->route` appear in
+  `_Model_Compass_SalesOrder::postPut()`, a different class not in that chain), that the shim is a
+  deliberately loud hard-fail contract, that running interceptors really does send email, that
+  `_Model_Client_EmailTemplate::send($clientIdentifier, …)` is the right tool when you only want a
+  mail, and that staged `addNote()` notes are silently dropped outside api2. (bala)
+
 - 2026-08-18 — TRUE-81049: documented that a **POST-processing interceptor cannot persist by
   mutating `$payload`** — `V2.php:5934` passes `$outData`, the outbound response, which the engine
   returns and never saves, so the caller sees the value while the column stays NULL. Added the

package/knowledge/2.0/apps/api2/features/v2-api-error-codes.md

@@ -6,7 +6,7 @@ project: API
 client: shared
 type: feature
 status: active
-updated: 2026-08-18
+updated: 2026-08-20
 owners: [mhammontree, tcox, jcardinal, ajean, bala]
 files:
   - api2/Component/Api/V2/V2.php
@@ -48,6 +48,7 @@ migration instead of re-deriving it. All field/script ACL rows live in the **CLI
 | **EO-1** | Operation failed — identifier "There is no field called 'X' in the '_Model_Client_Y' model" | The DB column **and** `Core.RecordFields` exist, but the **generated model class** `_underscore/Model/Client/<Name>.php` doesn't declare the field. This is the **4th** requirement beyond the 3-file migration — and most often it's a **cross-repo git branch mismatch** (`_underscore` on a branch whose generated model lacks a field the DB/RecordFields already carry) | Declare the field in the generated model class (`public $field = self::FIELD_*`) and put all related repos (`_underscore`, `api2`, `dbchanges2`, `toga2-supply`) on the **same** feature branch — see the ACL doc's writable-field recipe |
 | **EO-1** | Operation failed — **surfaced from a PHP warning/notice, not a real op error** (e.g. "Attempt to read property 'id' on bool", undefined variable) | A latent PHP warning escalates to a 500 because **Sentry's `ErrorHandler` in api2 promotes warnings/notices into thrown exceptions** (see diagnosis note 5). The known instance: in `V2.php::processRoutePairs()` an **unresolved route** leaves the local `$record = false`, and the post-processing payload interceptor layer then dereferenced `$record->id` → warning → 500 | Guard before dereferencing an unresolved record. The fix added a guard clause `if (!$record) return [$rawRequestedRouteName => $outData];` **before** the interceptor/logging layer (so a bad route returns a clean envelope, not a fatal), and initializes `$record = false;` at the top of the `foreach ($lookupByRouteNames ...)` loop so the invalid-HTTP-method / null-`$action` branch can't leave `$record` undefined (an undefined-variable warning would itself escalate to a 500) |
 | **EO-1** | Operation failed — MySQL **1054 "Unknown column 'X' in 'field list'"** | The **exact reverse** of the row above: the generated model (or a **trait** it `use`s) *declares* the field but the client table has **no such column**. `_Model` builds its SELECT from **every declared property**, so this fires on **any** load of that record on **any** route — including when the caller never asked for that field. Usual cause: a client DB whose module migration only **partially** applied | Add the missing column to that client DB — plus its `CustomRecordFields` row and `AclCustomFieldPermissions` grant if it is a `c_` field. Audit the rest of the drift first: [client schema-drift audit](../../dbchanges2/workflows/client-schema-drift-audit.md) |
+| **EO-1** | Operation failed — the exception message is literally **`[]`** (an empty JSON array) | Not an operation error at all, but a **message-array truthiness throw**. `internalApiRequest`'s throw is guarded on `!empty($this->response->messages)`, and `V2.php:2416` builds that as `json_encode(array_filter($messages, ...ERROR...))` — with zero real errors it is the **string `"[]"`, which PHP treats as truthy**. So any message (even a warning) anywhere in the request throws — including a nested GET that legitimately returns nothing. Live case (2026-08-20): Elite's `postPost` → `parent::postPost()` → `/approval-templates` with **no `ApprovalTemplates` rows** → every `POST /v2/sales-orders` 500s and the order **rolls back** | Short term, pass `false` as `internalApiRequest`'s 5th argument (`$throwExceptionsOnError`) in the calling model. Real fix: `V2.php:2416` tests the filtered **array** instead of its JSON string — that fixes every caller. Background: [API payload interceptors](api-payload-interceptors.md) |
 | **EO-2** | *"invalid configuration / A required database could not be reached while initializing the request"* — **almost always a lie** | The `catch (Throwable $dbBootstrapError)` in `api2/Controller/Index.php` (~L364) wraps the **entire request dispatch**, not just the DB bootstrap, so **any** Throwable that escapes query processing is relabelled EO-2. Real causes seen: MySQL 1054 on a missing column, a promoted PHP warning, a model fatal | **Ignore the message text.** Get the real exception from Sentry (`captureException`) or, on newer builds, from `identifiers.message` in the envelope; then resolve `error.id` per diagnosis note 9 |
 | **EZ-1** | Unauthorized record/script dispatch | Missing **`AclRecordScripts`** (scripted APIs) or the **`AclRecordPermissions`** four-table chain (records) for the caller's role | Grant `AclRecordScripts` (scripts) or complete the record-CRUD chain |
 | **EZ-2** | Field-level authorization denied (READ) | The field is registered (`Core.RecordFields` present → no `EV-8`) but the caller's role has no **`AclFieldPermissions`** grant to **read** it. The read-side counterpart of `EV-9` (write). **This applies to the `id` field too:** fetching a record by its numeric `id` (`GET /v2/<record>?fields=id,...`) 403s `EZ-2` if `id` has no read grant — the `id` field is ACL-gated like any other, not implicitly readable | Add the `AclFieldPermissions` row for the role (`isWritable=0` if the field is server-written). For a **custom** `c_` field use `AclCustomFieldPermissions` instead — see the ACL doc's standard-vs-custom table |
@@ -272,6 +273,12 @@ Full mechanics:
 - The response-envelope shape and code families: see [api2 architecture](../architecture.md).
 
 ## Change history
+- 2026-08-20 — Added a **fourth `EO-1` case: an exception message that is literally `[]`.** It is a
+  message-array truthiness throw, not an operation error — `internalApiRequest`'s throw is guarded on
+  the **shared** `response->messages`, and `V2.php:2416`'s `json_encode(array_filter(...))` is the
+  truthy string `"[]"` when there are **zero** real errors, so a nested GET that legitimately returns
+  nothing 500s the whole write and rolls it back (live case: Elite `/approval-templates` with no
+  `ApprovalTemplates` rows). (bala)
 - 2026-08-18 - **Corrected the EV-8 remedy.** "Register the field in `Core.RecordFields`" is not the
   only route: a **calculated (`_`-prefixed) field** declared on a shared `_Model_Client_*` model is
   fully exposed by a **client-scoped `CustomRecordFields` row + `AclCustomFieldPermissions` grant,

package/knowledge/2.0/apps/worker2/features/netsuite-salesorder-outbound-push.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-10
+updated: 2026-08-20
 owners: ["bala"]
 files:
   - worker2/Worker/Netsuite/SalesOrder.php
@@ -54,8 +54,10 @@ spreads `parameters` as named arguments):
 
 Supporting privates: `getClientContext`, `fetchSalesOrder`, `buildNetSuiteOrder`,
 `resolveNetSuiteCustomerId`, `resolveNetSuiteItemId`, `runSuiteQl`, `escapeSuiteQl`,
-`assertOrderMatchesUuid`, `doCreate`, `doUpdate`. Constants: `SUITEQL_ITEM_TYPE__GROUP = 'Group'`,
-`TOGA_FETCH_DEPTH = 4`.
+`assertOrderMatchesUuid`, `doCreate`, `doUpdate`, `buildExternalId`,
+`findNetSuiteOrderByExternalId`, `writeBackNetSuiteId`. Constants:
+`SUITEQL_ITEM_TYPE__GROUP = 'Group'`, `TOGA_FETCH_DEPTH = 4`,
+`NETSUITE_EXTERNAL_ID_PREFIX = 'toga-so-'`.
 
 ## How it works
 
@@ -73,9 +75,12 @@ Supporting privates: `getClientContext`, `fetchSalesOrder`, `buildNetSuiteOrder`
    quote, `''`, it never backslashes it).
 5. **A missing customer / item / location mapping aborts the whole order** rather than sending a
    partial one. A missing **shipping method** is simply omitted so NetSuite applies its default.
-6. Create → `_Component_Api_Netsuite::createRecord()` (id from the 204/201 `Location` header);
-   update → `send('PATCH', …)`. The new internalId is written back to
-   `SalesOrders.c_netsuiteInternalSalesOrderId`.
+6. Create → `_Component_Api_Netsuite::createRecord()` (id from the 204/201 `Location` header),
+   with an **`externalId` stamped on the record** so the create is idempotent — `doCreate` now has
+   four outcomes (`created` / `adopted` / `recovered` / `dryRun`), see
+   [Idempotency](#idempotency--the-netsuite-externalid) below; update → `send('PATCH', …)`.
+   The internalId is written back to `SalesOrders.c_netsuiteInternalSalesOrderId` by the shared
+   `writeBackNetSuiteId()` (three paths now save it).
 7. `dryRun = true` builds and returns the payload without calling NetSuite.
 
 ### The job carries the order snapshot — the worker does not re-read it
@@ -97,6 +102,87 @@ process (on a **reader** connection) cannot see the new row at all. See
   can never be atomic with the MySQL commit anyway, and holding a transaction open across a 12s
   external call risks row locks and dropped connections. The push stays **async**.
 
+## Idempotency — the NetSuite `externalId`
+
+Every order this worker creates is stamped with a NetSuite **`externalId`**, so *"have I already
+sent this order?"* is answerable **from NetSuite** rather than from Toga's own write-back. That
+turns the orphan failure below from a manual-reconciliation job into a self-healing one.
+
+**Format: `toga-so-<salesOrder uuid>`** — `NETSUITE_EXTERNAL_ID_PREFIX` + the order uuid, built by
+`buildExternalId()`; 44 chars, well inside NetSuite's 255 limit.
+
+- **The uuid, NOT `SalesOrders.number`.** NetSuite account **1095849 is shared by every client**,
+  while `number` is only unique within one client's database — Elite's `SA100005` and Compass's
+  `SA100005` can both exist. Keying on `number` would eventually make NetSuite refuse a legitimate
+  order because a **different client's** order already claimed that external id.
+- **The `toga-so-` prefix** because `externalId` is shared with other integrations in that same
+  account; a bare uuid says nothing about where the record came from.
+- **⚠ The format must not change once orders carry it.** Orders already in NetSuite would not be
+  found under a new format, so the adopt/recover paths below would silently miss them and create
+  duplicates. Treat `buildExternalId()` as a frozen contract.
+
+### `doCreate` has four outcomes
+
+| Result | When | What it does |
+|---|---|---|
+| `created` | nothing in NetSuite carries this external id | creates the record with `externalId` stamped on it |
+| `adopted` | `findNetSuiteOrderByExternalId()` found one | **no second create** — writes that id back |
+| `recovered` | `createRecord()` threw but the record exists anyway | claims the id, keeps the original error in a `createError` field |
+| `dryRun` | `$dryRun = true` | reports `externalId` and `alreadyInNetSuite`, creates nothing |
+
+If the create genuinely failed **and** nothing exists in NetSuite, the original exception is
+**rethrown untouched** — a real failure must never be swallowed by the recovery path.
+
+**Two independent layers, deliberately:**
+
+1. the SuiteQL pre-check — `SELECT id FROM transaction WHERE externalid = '<escaped>'`; **more than
+   one row throws** rather than guessing which record is the order; and
+2. `externalId` on the record itself, so **NetSuite** refuses the duplicate even if the lookup misses.
+
+### Write-back failure no longer asks for manual reconciliation
+
+The old error on a failed write-back said *"manual reconciliation required"*. It now names **both**
+the internal id and the external id and says a later run will find the record — because with the
+external id that is now true. The regression test asserting the old wording was updated
+**deliberately**: the contract improved, so the assertion moved with it (not a string-match fix).
+
+### Verifying the lookup — you have to fake the orphan first
+
+`CreateNetSuite` refuses at the top when `c_netsuiteInternalSalesOrderId` is already set, and that
+guard runs **before the dry-run branch**, so the external-id lookup is unreachable on an
+already-synced order. To exercise it (or the orphan-recovery path) against real data:
+
+1. `UPDATE SalesOrders SET c_netsuiteInternalSalesOrderId = NULL` for an order whose NetSuite record
+   already carries the external id;
+2. **dry-run** it — `alreadyInNetSuite` returning the id proves the SuiteQL filter works, `null`
+   means it does not;
+3. a real (non-dry) run then returns `adopted` and restores the id, so the test cleans up after
+   itself.
+
+### ⚠ Still unverified: can SuiteQL FILTER on `transaction.externalid` in this account?
+
+A `created` result **cannot distinguish** *"the lookup ran and found nothing"* from *"the lookup is
+broken and always finds nothing."* This project has already hit a NetSuite column that could be
+`SELECT`ed but silently returned zero rows when filtered on (see the
+[REST client doc](../../_underscore/features/netsuite-rest-client.md)), so the live check above is
+still owed. **Fallback if the filter does not work:** `GET /record/v1/salesOrder/eid:<externalId>`,
+which does not depend on SuiteQL at all.
+
+### Proven against live NetSuite (2026-08-20)
+
+Four real orders were created and written back by **direct worker invocation** (not the
+interceptor): NetSuite **7358493, 7359455, 7359555, 7366038**, each taking **~11–13s** — which is
+why this push is asynchronous. The duplicate guard was proven by a real second attempt on an
+already-synced order being refused. All four ran with `hasSnapshot = 0` (direct calls), so they do
+**not** exercise the interceptor-queued path — see the
+[Elite client-feature](../../../clients/elite/features/salesorder-netsuite-push.md).
+
+**⚠ These four orders reached PRODUCTION client data.** There is no NetSuite sandbox: every
+environment points at live account 1095849, so the inbound NetSuite → Toga sync imported these
+dev-sandbox test orders into **production** `Client_Elite` as real sales orders. The row-level
+cleanup caveat is in the Elite doc; the general rule is in
+[running worker2 locally](../workflows/running-worker2-locally.md).
+
 ## Earlier history worth keeping (2026-06-24, moved from the inbound doc)
 
 - **REST-only since 2026-06-24** — the SOAP `NetSuiteService` was fully removed from this file. Item
@@ -115,14 +201,26 @@ process (on a **reader** connection) cannot see the new row at all. See
   `CreateNetSuite`/`UpdateNetSuite` as thin wrappers — which also closed a TOCTOU, since the decision
   and the write now use the same fetched record.
 
-## Test harness (76 tests, all passing)
+## Test harness (85 tests, all passing)
 
 `test/@Bala/tests/netsuite_salesorder_payload_tests.php` loads the **real** worker file with
 stubbed dependencies and drives the private methods by **reflection**, so it tests shipped code
 rather than a copy (same technique as
 [DB-free interceptor unit testing](../../_underscore/features/model-interceptor-unit-testing.md)).
 Coverage: header fields, mapping resolution, SuiteQL quote-doubling, missing-mapping aborts, dry
-run, create, update, the job-carried-snapshot path, and the uuid-match guard.
+run, create, update, and the job-carried-snapshot path, plus:
+
+- **the uuid-match guard** — mismatch refused, refused on a dry run too, refused when the carried
+  order has no uuid, checked on update as well as create, reported **before** the already-synced
+  check, and a matching order still accepted;
+- **the external id** — built and stamped on the create payload, adoption skips the create,
+  adoption still writes back, recovery after a failed create, a genuine failure still surfaces the
+  **real** error and writes nothing, two matching records refuses to guess, the query really does
+  filter on `externalid`, dry run reports `externalId` + `alreadyInNetSuite`, and updates never
+  touch external ids.
+
+Stub additions this needed: `$throwOnCreate` and `$createPayloadsSeen` on the NetSuite stub, and a
+`$nextRows` queue on the `_Query` stub so `getClientContext()` resolves.
 
 ## Gotchas / known issues
 
@@ -131,7 +229,11 @@ run, create, update, the job-carried-snapshot path, and the uuid-match guard.
   `#`-delimiter regex bug did exactly this — see the
   [REST client doc](../../_underscore/features/netsuite-rest-client.md)) leaves a live NetSuite
   order with **nothing in Toga pointing at it**. Two Elite orders were orphaned in NetSuite account
-  1095849 this way. **Never blind-retry a failed create** — check NetSuite for the record first.
+  1095849 this way. **As of 2026-08-20 this is recoverable:** a re-run finds the record by
+  `externalId` and returns `adopted`/`recovered` instead of creating a second one (see
+  [Idempotency](#idempotency--the-netsuite-externalid)). That only holds for orders created **with**
+  an external id — the two pre-existing orphans carry none, so they still need manual linking, and
+  a blind retry on any order created before this change can still duplicate.
 - **This action does not retry itself.** Worker actions have no DLQ and no auto-retry: a throw
   records `isSuccess = 0` and stops. Unlike the 1.0 cron it replaces (which re-selected unsent
   orders every tick and was therefore self-healing), an event-triggered push fires **once**. Pair
@@ -142,6 +244,13 @@ run, create, update, the job-carried-snapshot path, and the uuid-match guard.
   *"Unknown named parameter"* from the dispatcher — which on a stale instance looks like a code bug
   rather than a deploy problem (see the
   [deploy workflow](../../api2/workflows/codepipeline-codeconnections-deploy.md)).
+- **⚠ Commit before any file-wide rewrite — a clean `git status` is evidence about HEAD, not
+  evidence your work exists.** The entire `externalId` implementation in this file was **lost once**
+  and had to be re-applied: it had only ever existed in the working tree, and a file-wide comment
+  rewrite overwrote it. `git status` reported the file **clean** while the code was missing, and
+  `grep` found the new constant in **no** branch (`TRUE-80497`, `_beta`, `_sandbox-dev`). What
+  surfaced it was the **test suite** — 8 failures accurately describing what the code no longer did.
+  Commit before a file-wide edit or a branch change, and never read "clean" as "saved".
 - **The inbound importer in this file is untouched and unrelated.** Do not "tidy" shared-looking
   helpers across the banner; there are none.
 - **⚠ `getClientContext()` is copy-pasted across three workers and has already DRIFTED.** The same
@@ -156,6 +265,23 @@ run, create, update, the job-carried-snapshot path, and the uuid-match guard.
 
 ## Change history
 
+- 2026-08-20 — **Made the create idempotent with a NetSuite `externalId`**
+  (`NETSUITE_EXTERNAL_ID_PREFIX = 'toga-so-'`, `buildExternalId()`,
+  `findNetSuiteOrderByExternalId()`, and `writeBackNetSuiteId()` extracted because three paths now
+  save the id): `doCreate` returns `created` / `adopted` / `recovered` / `dryRun`, a genuine failure
+  still rethrows untouched, and the guard is two-layer (SuiteQL pre-check **and** `externalId` on the
+  record so NetSuite itself refuses a duplicate). **Decided** the format is
+  `toga-so-<salesOrder uuid>` and is frozen — the uuid, not `SalesOrders.number`, because NetSuite
+  account 1095849 is shared by every client and `number` is only unique per client DB. Reworded the
+  write-back failure away from "manual reconciliation required" (it is now self-healing). Recorded
+  that the already-synced guard runs **before** the dry-run branch, so verifying the lookup needs
+  `c_netsuiteInternalSalesOrderId` NULLed first; that **whether SuiteQL can filter on
+  `transaction.externalid` is still unverified** (fallback `GET /record/v1/salesOrder/eid:<id>`); the
+  first live end-to-end proof (7358493 / 7359455 / 7359555 / 7366038, ~11–13s each, direct
+  invocation — and they round-tripped into **production** Elite data); the harness at **85** tests;
+  and the lesson that a clean `git status` does not mean uncommitted work survived a file-wide
+  rewrite. (bala)
+
 - 2026-08-12 — Recorded that `getClientContext()` is duplicated across three workers
   (`Netsuite/SalesOrder`, `Sync/ServiceRequest`, `Sync/SalesOrderStatus`) and has **drifted**: only
   the Sync/ServiceRequest copy orders the `Apis` credential lookup (`ORDER BY id ASC`), so the other

package/knowledge/2.0/apps/worker2/features/service-request-sales-order-generation.md

@@ -6,8 +6,8 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-12
-owners: ["snaredla"]
+updated: 2026-08-20
+owners: ["snaredla", "bala"]
 files:
   - worker2/Worker/Sync/ServiceRequest.php
   - _underscore/Model/Elite/ServiceRequest.php
@@ -156,6 +156,15 @@ have it. See the `c_`-column convention in `2.0/standards/framework-rules.md`.
 - **The NetSuite actions in this chain are `Netsuite/SalesOrder/CreateNetSuite` and
   `Netsuite/SalesOrder/UpdateNetSuite`.** There is **no** `Netsuite/SalesOrder/Push` — do not write
   that string into a migration, a test or a doc.
+- **⚠ It does not set `locationId` — which blocks the NetSuite push downstream (open, 2026-08-20).**
+  Orders this worker creates land with `locationId` **NULL** (prod Elite `SalesOrders` **279** /
+  `SA100000`, from `serviceRequestId 3`), and `Netsuite/SalesOrder/CreateNetSuite` then fails loudly:
+  *"no NetSuite location mapped on line 1 or on the order — map
+  `Locations.c_netsuiteInternalLocationId`"*. **The mapping data is complete** (prod Elite
+  `Locations` 1 and 2 → 196, 3 → 5), so this is a missing **field**, not a missing mapping. The fix
+  belongs **here, not in the push**: an order should record where it ships from regardless of
+  NetSuite, and `buildNetSuiteOrder()` deliberately refuses to guess a warehouse when the data does
+  not say. See [the NetSuite push](./netsuite-salesorder-outbound-push.md).
 
 ## Verifying the whole chain — `test_elite_chain_toga2.php`
 
@@ -188,6 +197,11 @@ Recorded so nobody re-debugs them as bugs. All four were confirmed against Elite
 
 ## Change history
 
+- 2026-08-20 — Recorded an **open defect: the worker leaves `locationId` NULL**, so the downstream
+  `Netsuite/SalesOrder/CreateNetSuite` push fails loudly on every order it generates (prod Elite
+  `SalesOrders` 279 / `SA100000`). The prod Elite `Locations` → NetSuite mappings are complete, so
+  this is a missing field rather than missing config, and the push will **not** default it — guessing
+  a warehouse is what its fail-loud check exists to prevent. (bala)
 - 2026-08-12 (later pass) — Recorded three deployment/verification facts: the action rename
   `GenerateSalesOrder` → `GenerateSalesOrderFromServiceRequest` is a **two-repo, single-deploy**
   change (the `runTask` string lives in `_underscore/Model/Elite/ServiceRequest.php:44`, and a

package/knowledge/2.0/apps/worker2/workflows/running-worker2-locally.md

@@ -6,8 +6,8 @@ project: Worker
 client: shared
 type: workflow
 status: active
-updated: 2026-08-19
-owners: ["kyalamarthi"]
+updated: 2026-08-20
+owners: ["kyalamarthi", "bala"]
 files:
   - worker2/index.php
   - worker2/composer.json
@@ -85,6 +85,15 @@ through `_Worker::runTask` locally: the dev config's `[cloud] aws_worker_queue_u
   appears not to have been applied to dev-sandbox since roughly February 2026. A dry run there dies
   on the missing column, which reads like a code bug. Use a local fixture instead, and see
   [repairing non-prod drift](../../dbchanges2/workflows/nonprod-metadata-drift-repair.md).
+- **⚠ There is NO NetSuite sandbox — every environment writes to the LIVE account (1095849).**
+  A "beta" or dev-sandbox test that creates a NetSuite record creates it in **production NetSuite**,
+  and the inbound NetSuite → Toga sync then imports it straight back into **production Toga** as real
+  client data. Four dev-sandbox Elite test orders (NetSuite **7358493 / 7359455 / 7359555 /
+  7366038**) arrived in prod `Client_Elite.SalesOrders` **261–266** exactly this way. Before running
+  any NetSuite **write** test, decide up front how you will identify and clean up the records in
+  **both** systems — a stable `externalId` makes them findable, see the
+  [sales-order push](../features/netsuite-salesorder-outbound-push.md) — and never bulk-delete a row
+  range afterwards: unrelated records land in the same range (7369129 and 7374431 did).
 - **⚠ SECURITY — live credentials are committed in this repo; they are locations to fix, not
   settings to copy.** Do not paste any of these values anywhere, including a knowledge doc:
   - `worker2/.ebextensions/git.json` carries a **live GitHub personal access token**, used so
@@ -101,6 +110,11 @@ through `_Worker::runTask` locally: the dev config's `[cloud] aws_worker_queue_u
 
 ## Change history
 
+- 2026-08-20 — Recorded that **there is no NetSuite sandbox**: every environment points at live
+  account 1095849, so a dev-sandbox/beta write test creates **production** NetSuite records **and**
+  the inbound sync round-trips them into **production Toga** client data (four Elite test orders
+  landed in prod `Client_Elite.SalesOrders` 261–266). Plan the two-system cleanup before testing a
+  NetSuite write, and never bulk-delete the resulting row range. (bala)
 - 2026-08-19 — Created while testing the customer account-type sync (TRUE-80206) with no usable
   shared environment: recorded the five-step local boot (composer install with the `ext-zip` opt-out,
   `set_include_path()` for `_underscore` vs. EB's `php_include_underscore.config`, `ENVIRONMENT` →