toga-ai 1.0.698 → 1.0.700

package/knowledge/1.0/apps/library/INDEX.md

@@ -14,6 +14,7 @@
 | [Error Capture in 1.0 (App_Error_Capture → shared 2.0 Logs DB)](features/error-capture-1-0.md) | The 1.0 side of the platform error-reporting pipeline (TRUE-78188). | library/app/error/capture.php, library/app/error.php, library/app/exception/business.php, library/app/api/toga2.php, library/app/cloud.php, worker/config.worker.ini, worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php |
 | [HTTP 500 Error Monitor (App_SystemMonitor_500Error) — and why its \"Error Type\" is not a diagnosis](features/http-500-error-monitor.md) | `App_SystemMonitor_500Error` (`library/app/systemmonitor/500error.php`, title **"HTTP 500 Error Alert"**) is the 1.0 system monitor that watches **`Logs.Api` fo | library/app/systemmonitor/500error.php, worker/crons/infrastructure/system_monitors.php, api2/Controller/Index.php, _underscore/Error.php |
 | [1.0 MVC Page Pattern & New-App Skeleton](features/mvc-page-pattern-and-app-skeleton.md) | This is the **reusable recipe for standing up a new 1.0 (`App_`) application** and for adding pages to one — the folder-based MVC routing, the page lifecycle, t | library/app/framework.php, library/app/frameworkindex.php, library/app/mvc.php, library/app/database.php, library/app/model.php, library/app/config.php |
+| [NetSuite File Cabinet Content Retrieval via RESTlet (fetchInvoiceFile)](features/netsuite-filecabinet-restlet.md) | How 1.0 pulls **File Cabinet binary content** (invoice PDFs) out of NetSuite over REST. | library/app/api/netsuite/rest.php, worker/crons/toga2/netsuite/netsuite_restlet_filecontent.js, worker/config.worker.ini |
 | [isFulfillable from NetSuite during Item Sync (Phase 1)](features/netsuite-item-isfulfillable-sync.md) | This is the **1.0 (Phase 1)** half of the `isFulfillable` feature: reading the NetSuite `isfulfillable` flag during item sync and stamping it onto the **Agilant | library/app/netsuite.php, library/app/api/toga2.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/backfill_isfulfillable_jul5.php |
 | [NetSuite SuiteQL/REST API Reference](features/netsuite-suiteql-api-reference.md) | General working reference for the Agilant NetSuite integration: how to authenticate, how SuiteQL behaves, and the confirmed schema of the tables/columns/codes w | library/app/api/netsuite/rest.php, library/ssl/netsuite_ec_key.pem, test/@dave/Junk Drawer/nsq.php |
 | [NetSuite SuiteQL/REST Shim — Field Semantics](features/netsuite-suiteql-rest-shim.md) | `App_Api_Netsuite_Rest` is the REST/SuiteQL replacement for the deprecated NetSuite SOAP toolkit. | library/app/api/netsuite/rest.php, library/app/netsuite.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |

package/knowledge/1.0/apps/library/features/netsuite-filecabinet-restlet.md

@@ -0,0 +1,122 @@
+---
+title: NetSuite File Cabinet Content Retrieval via RESTlet (fetchInvoiceFile)
+framework: "1.0"
+repo: library
+project: Library
+client: shared
+type: feature
+status: active
+updated: 2026-08-29
+owners: [jcardinal]
+files:
+  - library/app/api/netsuite/rest.php
+  - worker/crons/toga2/netsuite/netsuite_restlet_filecontent.js
+  - worker/config.worker.ini
+related:
+  - netsuite-suiteql-api-reference.md
+  - netsuite-suiteql-rest-shim.md
+  - ../../worker/features/netsuite-togasupply-per-client-sync.md
+---
+
+## Summary
+
+How 1.0 pulls **File Cabinet binary content** (invoice PDFs) out of NetSuite over REST. The
+short version: **there is no token-only, code-only REST path to File Cabinet binary content in
+this account.** The `/record/v1/file` Record Service is not enabled, SuiteQL exposes only file
+*metadata* (no content column), and the `media.nl` UI URL is not fetchable with an OAuth token.
+The only REST mechanism is a **SuiteScript 2.1 RESTlet** that loads the file server-side and
+returns its bytes base64-encoded. `App_Api_Netsuite_Rest::fetchInvoiceFile()` uses exactly that.
+
+This replaced the SOAP `App_NetSuite::getInvoiceFile` as part of completing the SOAP→REST
+migration of the togasupply sync path (see the
+[per-client-sync doc](../../worker/features/netsuite-togasupply-per-client-sync.md)).
+
+## Why there is no token-only REST path (verified live, 2026-08-29)
+
+Three routes were probed against this account and all three are dead ends for binary content:
+
+- **REST Record Service `GET /record/v1/file/{id}` → HTTP 404 `NONEXISTENT_ID`**, *"Record type
+  'file' does not exist."* The `file` record type is simply not in this account's REST Record
+  Service. (The earlier `fetchInvoiceFile` design assumed this endpoint worked — it never did.)
+- **SuiteQL `file` table exposes ONLY metadata** — `id, name, filetype, filesize, folder, url,
+  createddate, lastmodifieddate` and `is*` flags. There is **no binary/content column**. Good for
+  *finding* the file id by name; useless for its bytes.
+- **The SuiteQL `url` is a `media.nl` UI link, not an API resource.** It is a
+  `/core/media/media.nl?...&c=<acct>&h=<hash>` link on the `<acct>.app.netsuite.com` **UI domain**.
+  Fetching it — anonymous OR with an OAuth Bearer — is rejected (redirects to `page_not_found`),
+  because invoice PDFs are **not** flagged "Available Without Login" and the OAuth token does not
+  authorize the UI media servlet.
+
+Conclusion: to get File Cabinet bytes over REST you must run server-side script. That is the
+RESTlet.
+
+## How it works
+
+### NetSuite side — the RESTlet
+
+`worker/crons/toga2/netsuite/netsuite_restlet_filecontent.js` is a **SuiteScript 2.1 RESTlet**
+that does `N/file.load(id).getContents()` and returns the content **base64-encoded**. Deployed in
+NetSuite as script **`customscript2829`** / deployment **`customdeploy1`** (non-secret identifiers),
+with the **RESTlets** OAuth scope enabled on the **"TOGA OAuth 2.0"** integration record.
+
+### Client side
+
+- **`App_Api_Netsuite_Rest::callRestlet()`** — GETs
+  `<acct>.restlets.api.netsuite.com/app/site/hosting/restlet.nl?script=<id>&deploy=<id>` (note the
+  **`restlets.api.netsuite.com`** host, distinct from the SuiteQL/REST `suitetalk` host).
+- **`fetchInvoiceFile()`** — SuiteQL `file`-table lookup by name to get the file **id** →
+  `callRestlet()` → `base64_decode()` → raw bytes. **Fail-loud preserved:** no file row → `null`;
+  a file found but unretrievable, or a RESTlet error → **throws** (never silently returns empty).
+- **Config:** the script/deploy ids live in `worker/config.worker.ini` `[netsuite]` as
+  `restletScriptId` / `restletDeployId`.
+
+### Validated byte-identical to SOAP
+
+RESTlet-retrieved PDFs were confirmed **byte-identical** (matching length + md5) to what SOAP
+`getInvoiceFile` returned, across 3 Compass invoices (~265 KB each). Zero `fetchInvoiceFile` /
+`INVALID_LOGIN` errors after deploy. This also unblocked Endeavor Health's whole pipeline — its
+INVOICES section had been throwing on the broken file fetch and collateral-freezing
+IR/IF/InvAdj.
+
+## Gotchas / known issues
+
+- **⚠ NetSuite issues ONE active OAuth 2.0 token per client credential — a second token with a
+  DIFFERENT scope INVALIDATES the first.** Fetching a separately-scoped `restlets` token caused the
+  very next SuiteQL call to fail with **HTTP 400 `INVALID_LOGIN`**. **Fix: request ONE
+  combined-scope token** — a single `OAUTH_SCOPE` of `rest_webservices restlets` — never two
+  separate tokens. This requires the **RESTlets scope enabled on the integration record**. The
+  initially-tried "isolated separate restlets token" design is what caused the `INVALID_LOGIN` and
+  was removed. (This is a general OAuth fact: any future added scope must be folded into the single
+  combined-scope request, not fetched as a second token.)
+- **⚠ A RESTlet whose function returns an object requires the REQUEST to carry
+  `Content-Type: application/json`** — otherwise NetSuite rejects it with HTTP 400
+  `INVALID_RETURN_DATA_FORMAT` (*"Invalid data format. You should return text."*). Set the request
+  content type even though `callRestlet` is a GET with no meaningful body.
+- **PHP 7.2 prod constraint** applies to `rest.php` as everywhere — lint with
+  `C:\xampp7\php\php.exe -l` before deploying (a single 7.4+ token parse-errors the whole class;
+  see the [api reference](netsuite-suiteql-api-reference.md) blast-radius gotcha).
+
+## Change history
+
+- 2026-08-29 — Created. Migrated invoice-PDF retrieval off SOAP `getInvoiceFile` to a SuiteScript
+  2.1 RESTlet (`customscript2829`/`customdeploy1`) after verifying live that **no token-only REST
+  path to File Cabinet binary exists** in this account: `/record/v1/file/{id}` 404s
+  (`NONEXISTENT_ID`, file record not in the REST Record Service), SuiteQL `file` is metadata-only
+  (no content column), and the `media.nl` UI url is not OAuth-fetchable. Added
+  `App_Api_Netsuite_Rest::callRestlet()` + rewrote `fetchInvoiceFile` (SuiteQL id lookup →
+  callRestlet → base64_decode, fail-loud). Recorded two RESTlet/OAuth gotchas: one active token per
+  client credential so scopes must be combined (`rest_webservices restlets` in a single request,
+  else the next SuiteQL call 400s `INVALID_LOGIN`), and an object-returning RESTlet needs request
+  `Content-Type: application/json` (else 400 `INVALID_RETURN_DATA_FORMAT`). RESTlet PDFs verified
+  byte-identical to SOAP across 3 Compass invoices; unblocked Endeavor Health's pipeline. (jcardinal)
+
+## Related docs
+
+- [`netsuite-suiteql-api-reference.md`](netsuite-suiteql-api-reference.md) — auth (OAuth2/JWT),
+  SuiteQL mechanics, the `fetch*ById()` helper family that `fetchInvoiceFile` belongs to.
+- [`netsuite-suiteql-rest-shim.md`](netsuite-suiteql-rest-shim.md) — REST-shim field/shape
+  semantics.
+- [per-client-sync](../../worker/features/netsuite-togasupply-per-client-sync.md) — the togasupply
+  engine whose INVOICES section consumes `fetchInvoiceFile`.
+</content>
+</invoke>

package/knowledge/1.0/apps/library/features/netsuite-suiteql-api-reference.md

@@ -14,6 +14,7 @@ files:
   - test/@dave/Junk Drawer/nsq.php
 related:
   - netsuite-suiteql-rest-shim.md
+  - netsuite-filecabinet-restlet.md
   - ../architecture.md
   - ../../worker/features/forecast2-netsuite-reconciliation.md
   - ../../../clients/nychh/features/netsuite-inventory-adjustment-fulfillment-link.md
@@ -42,6 +43,12 @@ This doc is the **broad map** (mechanics, schemas, codes, performance). For the
   a placeholder `CERTIFICATE_ID_HERE` that `initialize()` explicitly rejects.
 - `readConfig()` supports two framework contexts: 1.0 via `App_Registry::get('config')['netsuite']`,
   and 2.0 via `_Config::netsuite()`.
+- **⚠ ONE active OAuth token per client credential — combine scopes, never fetch a second token.**
+  NetSuite invalidates the first token when a second is requested with a *different* scope: after a
+  separately-scoped `restlets` token was fetched, the next SuiteQL call 400'd `INVALID_LOGIN`. Request
+  a single combined `OAUTH_SCOPE` (currently `rest_webservices restlets`, the latter enabled on the
+  "TOGA OAuth 2.0" integration for `fetchInvoiceFile`). Any future scope must be folded into this one
+  request. See [File Cabinet content via RESTlet](netsuite-filecabinet-restlet.md).
 
 ## Bootstrapping the client from a standalone script
 
@@ -349,6 +356,35 @@ Used for fields SuiteQL doesn't expose on `transaction`: `shippingCost`, `taxTot
   when `fetchItemFulfillmentById` returned zero lines — see
   [NYCHH inventory-adjustment → item-fulfillment linking](../../../clients/nychh/features/netsuite-inventory-adjustment-fulfillment-link.md).
 
+### `fetch*ById()` single-record helpers — the REST replacements for the SOAP `App_NetSuite::get*`
+
+The togasupply importer fetches individual records by internal id through a family of `fetch*ById()`
+helpers on `App_Api_Netsuite_Rest`, each assembling a SOAP-shaped shim from a SuiteQL header/lines
+query plus (where needed) a REST record GET. As of **2026-08-29 the togasupply sync path uses these
+exclusively — zero `App_NetSuite::` SOAP network calls remain** in `common_sync_togasupply.php` or the
+`*FromNetsuite()` methods:
+
+- **`fetchItemReceiptById`** — header + lines + serials (serials via SuiteQL
+  `inventoryassignment ⋈ inventorynumber`, grouped per product). Replaces SOAP
+  `getItemReceipt`/`getInventoryNumber*`/`getItemIsSerialized`.
+- **`fetchPurchaseOrderById`** / **`fetchSalesOrderByIdFull`** — replace SOAP `getPurchaseOrder` /
+  `getSalesOrder`.
+- **`fetchItemFlagsByIds`** — batch item serialized/fulfillable flags; replaces
+  `getItemSerializedAndFulfillable`.
+- **`fetchInventoryAdjustmentById`** — replaces the `diagnose_hh` script's SOAP `getObject`.
+- **`fetchInvoiceFile`** — SuiteQL `file`-table lookup for the id → **RESTlet** call
+  (`callRestlet()`) → `base64_decode` → raw bytes. Replaces SOAP `getInvoiceFile`. **NOT** a REST
+  record GET — `GET /record/v1/file/{id}` 404s (`NONEXISTENT_ID`; the `file` record is not in this
+  account's REST Record Service), SuiteQL `file` is metadata-only, and the `media.nl` url is not
+  OAuth-fetchable, so a SuiteScript RESTlet is the only REST path to File Cabinet binary. Full
+  mechanics + the two OAuth/RESTlet gotchas: [File Cabinet content via
+  RESTlet](netsuite-filecabinet-restlet.md).
+
+`listCustomers()` also gained a **`companyName`** field on its shim (the `netsuite_customers.php` cron
+moved from `listAllCustomers` → `listCustomers`). REST-shim *shape* traps that bite these helpers (an
+absent nested list crashes 1.0; `scriptId` case-sensitivity on `customFieldList`) are in
+[`netsuite-suiteql-rest-shim.md`](netsuite-suiteql-rest-shim.md).
+
 ## `list*()` helpers — behavior & cost
 
 `listSales()`, `listSalesOrders()`, `listOpportunities()` all:
@@ -418,6 +454,21 @@ numbers. Budget hours for multi-year runs and launch them under `nohup`/`tmux`.
 
 ## Change history
 
+- 2026-08-29 — Corrected `fetchInvoiceFile` (deploy/verify session): it is **RESTlet-based**, not a
+  REST record GET — `GET /record/v1/file/{id}` 404s (`file` record not in this account's REST Record
+  Service), SuiteQL `file` is metadata-only, `media.nl` isn't OAuth-fetchable, so a SuiteScript 2.1
+  RESTlet is the only REST path to File Cabinet binary; new
+  [File Cabinet content via RESTlet](netsuite-filecabinet-restlet.md) doc holds the mechanics. Added the
+  **one-active-token-per-credential** auth rule to Authentication (combine scopes into a single
+  `rest_webservices restlets` request; a separate second token 400s the next SuiteQL call
+  `INVALID_LOGIN`). (jcardinal)
+- 2026-08-29 — Documented the **`fetch*ById()` single-record helper family** (`fetchItemReceiptById`,
+  `fetchPurchaseOrderById`, `fetchSalesOrderByIdFull`, `fetchItemFlagsByIds`,
+  `fetchInventoryAdjustmentById`, `fetchInvoiceFile`) that replaced the SOAP `App_NetSuite::get*` calls,
+  and the `listCustomers` `companyName` addition — the togasupply sync path now makes **zero
+  `App_NetSuite::` SOAP network calls**. Added while root-causing the 5 stuck togasupply integrations as
+  REST-shim shape mismatches (see the
+  [per-client-sync doc](../../worker/features/netsuite-togasupply-per-client-sync.md)). (jcardinal)
 - 2026-08-29 — Recorded two NetSuite REST/SuiteQL facts from the NYCHH inventory-adjustment →
   item-fulfillment **REST migration**: **REST record sublist endpoints
   (`/record/v1/{type}/{id}/{sublist}`) reject `limit`/`offset` with a 400 `INVALID_PARAMETER`** (a

package/knowledge/1.0/apps/library/features/netsuite-suiteql-rest-shim.md

@@ -6,7 +6,7 @@ project: Library
 client: shared
 type: feature
 status: active
-updated: 2026-08-27
+updated: 2026-08-29
 owners: [dfranks, jcardinal, kyalamarthi]
 files:
   - library/app/api/netsuite/rest.php
@@ -112,6 +112,43 @@ Forecast2 tables already store.
   TransferOrders). A custom field absent from the record reads as null downstream. See
   [NYCHH NetSuite → TransferOrders import](../../../clients/nychh/features/netsuite-transfer-order-import.md).
 
+- **⚠ `scriptId` in a `customFieldList` def is CASE-SENSITIVE on lookup — a case skew silently
+  returns the default (2026-08-29).** `custbody_stocking_order` was **tagged** in the shim with the
+  uppercase scriptId `CUSTBODY_STOCKING_ORDER` but **read** by the importer as lowercase
+  `custbody_stocking_order`, so the lookup never matched and always fell back to the default (unchecked)
+  — which mis-routed **every** NYCHH order to a Transfer Order. Fixed by lowercasing the scriptId in
+  **both** `buildSalesOrderShims()` and `fetchSalesOrderById()` defs; `END_CUSTOMER` is intentionally
+  left uppercase because it is resolved by internalId `3149`, not by scriptId. Nothing errors on a case
+  skew — the field just reads as its default — so verify the exact scriptId string on both the write and
+  the read side. See [NYCHH TransferOrders import](../../../clients/nychh/features/netsuite-transfer-order-import.md).
+
+- **`fetchPurchaseOrderById()` omitted `entity` from its SELECT and never set `$shim->entity` — so
+  every PO shim had a null vendor and the fail-loud guard fired every run (2026-08-29).** Unlike its
+  sibling `listPurchaseOrders` (which DOES select `entity` + `BUILTIN.DF(entity)` and populates the
+  shim's vendor), `fetchPurchaseOrderById`'s SuiteQL header query left `entity` out and the shim never
+  built `$shim->entity`. The item-receipt path (`syncItemReceiptFromNetsuite` →
+  `syncPurchaseOrderFromNetsuite`) therefore always saw a null entity, and the fail-loud guard
+  `if (empty($nsPurchaseOrder->entity)) throw 'has no vendor/entity'` threw on **every** run — freezing
+  **Canon** and **Endeavor Health** ITEM_RECEIPTS at `1-RUNNING` (window collapsed ÷3 to 1s), watermarks
+  frozen ~8 and ~5 days. **Fix:** added `entity, BUILTIN.DF(entity) AS entity_name` to the header SELECT
+  and populated `$shim->entity` as `{internalId, name}` (matching the other shims); a genuinely
+  vendorless PO still throws. Same class as the `tl.location` regression below and the missing-entity
+  guard was correct — the SELECT was the defect. **When adding a shim, every field the guard/consumer
+  reads must actually be in the SELECT.**
+
+- **A shim must produce the SOAP object's SHAPE, not just its scalar fields — an absent nested list
+  crashes 1.0 (2026-08-29).** `fetchCustomerById()`'s `addressbookList` was where the AIG lookup phase
+  crashed: a REST customer with no/odd address made the shim omit the nested `addressbook` structure the
+  old SOAP object always carried, and the downstream unguarded read raised a warning → `App_Error`
+  `exit()` (see the `getCustomFieldValue` warning-safety gotcha below — same class). It now builds
+  `addressbookList` via a `shimAddress()` helper that normalizes `country` to the `_unitedStates` enum
+  and returns an **empty `addressbook: []`** when there is no address, so the shape always matches;
+  two helpers were extracted for reuse — `countryToSoapEnum()` (country string → SOAP enum) and
+  `refString()`. This is the concrete "REST shim returns null/absent where SOAP had a value" failure
+  the [per-client-sync doc](../../worker/features/netsuite-togasupply-per-client-sync.md) diagnoses via
+  `Logs.Issue`/`Logs.Event`. **When adding or editing a shim, reproduce the full SOAP shape (including
+  empty nested lists), not just the fields you happen to need.**
+
 ## Transaction status sourcing (1.0) — and the 2026.2 REST `status.id` change
 
 1.0 reaches NetSuite over **two transports, and neither reads status off a REST record GET
@@ -192,6 +229,25 @@ None — uniform across clients (NetSuite is a single shared account).
 
 ## Change history
 
+- 2026-08-29 — Recorded `fetchPurchaseOrderById()`'s **missing `entity` SELECT** (deploy/verify
+  session): its header query omitted `entity`/`BUILTIN.DF(entity)` and the shim never set
+  `$shim->entity` (unlike `listPurchaseOrders`), so the PO vendor was always null and the
+  `syncPurchaseOrderFromNetsuite` fail-loud missing-entity guard threw every run — the actual cause of
+  the **Canon + Endeavor Health ITEM_RECEIPTS** freeze at `1-RUNNING` (watermarks ~8/~5 days). Fixed
+  additively (add `entity, BUILTIN.DF(entity) AS entity_name`; populate `$shim->entity` as
+  `{internalId,name}`); same "a field the shim consumes must be in the SELECT" class as `tl.location`.
+  Deployed ~12:52; both sections self-healed RUNNING→IDLE and resumed advancing. (jcardinal)
+- 2026-08-29 — Recorded two shim-shape facts from root-causing the 5 stuck togasupply integrations
+  (which were **shim-shape mismatches, not dead SOAP** — see the
+  [per-client-sync doc](../../worker/features/netsuite-togasupply-per-client-sync.md)): a
+  `customFieldList` **`scriptId` is case-sensitive on lookup** — `CUSTBODY_STOCKING_ORDER` (write) vs
+  `custbody_stocking_order` (read) silently returned the default and mis-routed every NYCHH order to a
+  Transfer Order (fixed by lowercasing in both `buildSalesOrderShims`/`fetchSalesOrderById`;
+  `END_CUSTOMER` stays uppercase, resolved by internalId 3149); and a shim must reproduce the SOAP
+  object's **full shape** — `fetchCustomerById()`'s `addressbookList` omitting the nested `addressbook`
+  crashed the AIG lookup phase (unguarded null read → warning → `exit()`), now built via `shimAddress()`
+  (country → `_unitedStates` enum, empty → `addressbook: []`) with extracted `countryToSoapEnum()` /
+  `refString()` helpers. (jcardinal)
 - 2026-08-27 — **Traced the missing `tl.location` SELECT to its cause — a merge, not an omission — and
   recorded what restoring it re-animates** (forecast2 open-orders, TRUE-79162). TRUE-79078 (31fadd77)
   originally added `tl.location` + `BUILTIN.DF(tl.location)`; merge c3caa11a (2026-06-24) resolved in

package/knowledge/1.0/apps/library/features/toga2-api-client-and-bridge.md

@@ -6,7 +6,7 @@ project: Library
 client: shared
 type: feature
 status: active
-updated: 2026-08-17
+updated: 2026-08-29
 owners: [jcardinal, mhammontree, bala]
 files:
   - library/app/api/toga2.php
@@ -396,6 +396,15 @@ enable flags** and an optional `$monitorTogadeskDepartmentIds[]`:
 
 ## Change history
 
+- 2026-08-29 — The `*FromNetsuite()` importer methods in this class were **completed onto REST** — the
+  item-receipt and inventory-adjustment paths dropped their last `App_NetSuite::` SOAP calls in favor of
+  `App_Api_Netsuite_Rest::fetch*ById()`, so the togasupply importer now makes **zero SOAP network
+  calls**. Also **corrects the 2026-08-27 entry below**: the private `isStockingOrder()` custom-field
+  read had a `scriptId` case-mismatch (`CUSTBODY_STOCKING_ORDER` vs `custbody_stocking_order`) that
+  silently defaulted and mis-routed every NYCHH order — now fixed. Method-level detail and the full
+  per-client fix set live on
+  [NetSuite → TOGa Supply per-client sync](../../worker/features/netsuite-togasupply-per-client-sync.md)
+  (2026-08-29). (jcardinal)
 - 2026-08-27 — `send()`'s throw sites were **reverted to plain `throw new Exception(...)`** (both
   sites) as part of removing the sync engine's swallow-and-skip data-loss bug — the importer must
   propagate a failed write, not catch it (see

package/knowledge/1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-28
+updated: 2026-08-29
 owners: ["dfranks", "bala", "jcardinal", "mhammontree", "snaredla"]
 files:
   - worker/crons/toga2/netsuite/common_sync_togasupply.php
@@ -531,8 +531,149 @@ Parameters are stored **per client DB** but accessed **through the TOGa2 API**,
   direct-FK-column change) on
   [NYCHH inventory-adjustment → item-fulfillment linking](../../../clients/nychh/features/netsuite-inventory-adjustment-fulfillment-link.md).
 
+- **⚠ 2026-08-29: the 5 stuck togasupply integrations were REST-shim SHAPE mismatches, NOT dead
+  SOAP — and how to tell them apart.** Five clients had a section frozen at `RUNNING` and the first
+  hypothesis was "SOAP is down." **SOAP was up.** The real cause was introduced by the earlier
+  SOAP→REST list-method conversion: a REST shim returns `null`/absent where the old SOAP object
+  carried a value, and in 1.0 **any unguarded read of that null raises a PHP warning that
+  `App_Error` escalates to `exit()`** (see the 1.0 backend-php standard *Any PHP warning terminates
+  the request*, and the [rest-shim warning-safety gotcha](../../library/features/netsuite-suiteql-rest-shim.md)).
+  The `exit()` mid-section leaves that record-type's `NETSUITE_EXECUTION_MODE_*` stuck at `RUNNING`
+  with the watermark frozen — i.e. **a shim-shape mismatch presents identically to a poison order.**
+  So a stuck section is never "SOAP is dead"; it is one specific record hitting one specific
+  unguarded shim field.
+  **Diagnosis playbook (durable):** query prod **`Logs.Issue` + `Logs.Event`** (the
+  `App_Error_Capture` tables, schema `Logs`, via the toga-db MCP) for the client's `clientId` — each
+  stuck section maps to an exact crash **file:line** in `common_sync_togasupply.php` or `toga2.php`.
+  The `RUNNING` mode + frozen watermark **is** the stuck signal (see the window/watermark gotchas
+  above); `Logs.Issue`/`Event` turns it into a line number, far faster than reading the cron blind.
+  **Per-client fixes shipped (all in `library`, `php -l` clean, and fail-loud — a genuinely
+  unresolvable case now throws rather than silently continuing):**
+  - **Item receipts — Canon + Endeavor Health** (`syncItemReceiptFromNetsuite`): migrated off SOAP to
+    new REST `fetchItemReceiptById` (header + lines + serials via SuiteQL
+    `inventoryassignment ⋈ inventorynumber`, serials grouped per product) + `fetchPurchaseOrderById`
+    + `fetchItemFlagsByIds`; removed all `App_NetSuite::` SOAP calls (`getItemReceipt` /
+    `getPurchaseOrder` / `getItemIsSerialized` / `getInventoryNumber*`). Guards added;
+    **`RtnAuth` (return-authorization) item receipts are intentionally NOT imported** (explicit
+    documented skip), and an **unknown `createdFrom` type throws**.
+  - **Purchase orders** (`syncPurchaseOrderFromNetsuite`): a missing PO entity now **throws** (was an
+    unguarded crash); `getSalesOrder` → REST `fetchSalesOrderByIdFull`; a `2400` ACL-strip that used
+    to skip now **throws**. Threaded a new `array &$lookupWarehouseByNetsuiteLocationInternalId` param
+    from **both** callers — which also fixed a **null `arg13` `TypeError`** into
+    `syncSalesOrderFromNetsuite`.
+  - **Transfer orders — GroWrk `SALES_ORDERS`** (`syncTransferOrderFromNetsuite`): fixed an
+    all-catalog lookup that read `->uuid` off the array instead of `[0]`; an **unmapped NetSuite TO
+    status now throws** (was default-to-pending + log).
+  - **Item fulfillments — Elite (also Quad / S&P Global / Prudential-EDI)**
+    (`syncItemFulfillmentFromNetsuite`): the "Undefined index" on service/shipping lines with an
+    **empty `orderLine`** (`$lookupNsSalesOrderItemsByOrderLine['']`) is now guarded so it falls
+    through to the existing reconstruct path.
+  - **NYCHH stocking-order routing** — the `isStockingOrder` scriptId **case-mismatch** (full detail
+    in the [rest-shim doc](../../library/features/netsuite-suiteql-rest-shim.md)): the custom field was
+    tagged `CUSTBODY_STOCKING_ORDER` but looked up lowercase → always returned the default →
+    **every NYCHH order was mis-routed to a Transfer Order.** Fixed by lowercasing the scriptId in
+    both shim defs.
+  - **AIG lookup-phase crash** — `fetchCustomerById`'s `addressbookList` is now built via `shimAddress`
+    (country normalized to the `_unitedStates` enum, empty → `addressbook: []`); extracted
+    `countryToSoapEnum()` + `refString()` helpers, plus a belt-and-suspenders guard in
+    `common_sync_togasupply.php`.
+
+- **SOAP→REST is now COMPLETE for the togasupply sync path (2026-08-29).** With the item-receipt and
+  (2026-08-29) inventory-adjustment migrations plus the new `fetch*ById` methods, **zero
+  `App_NetSuite::` (SOAP) network calls remain** anywhere in the `common_sync_togasupply.php` engine
+  or the `*FromNetsuite()` methods it calls — so the whole tier-wide-freeze class above is now driven
+  by REST-shim shape, not SOAP availability. New REST completeness pieces built in `library`:
+  `fetchInvoiceFile` (SuiteQL `file` id lookup + **RESTlet** content fetch → `base64_decode` → raw
+  bytes; replaces `getInvoiceFile` — the `/record/v1/file/{id}` approach 404s, so a SuiteScript RESTlet
+  is the only REST path — see [File Cabinet content via RESTlet](../../library/features/netsuite-filecabinet-restlet.md)),
+  `getItemSerializedAndFulfillable` → `fetchItemFlagsByIds`,
+  the `diagnose_hh` script's `getObject` → `fetchInventoryAdjustmentById`, and
+  `worker/crons/sync/netsuite/netsuite_customers.php` `listAllCustomers` → `listCustomers` (with
+  `companyName` added to the `listCustomers` shim).
+
+- **⚠ SALES_ORDERS line renumber is now a validate-before-mutate TWO-PHASE UPDATE (2026-08-29) —
+  supersedes the delete-recreate + pinned-skip approach.** `syncSalesOrderFromNetsuite`'s renumber no
+  longer DELETEs and re-creates lines (nor keeps un-deletable/invoiced rows at their old number — see
+  the 2026-08-17 pinned-skip gotcha above, now historical). It **parks** every drifting line at a
+  temporary `lineNumber` (base = `max(current, target lineNumbers) + headroom`, headroom const
+  **1000**), then **finalizes** each to its target — so `UNIQUE(salesOrderId, lineNumber)` is never
+  transiently violated and pinned FK-RESTRICT children (invoiced/fulfilled lines) renumber accurately
+  via `UPDATE` instead of being stranded at a stale number. It **throws BEFORE any PUT** if a kept or
+  undeletable row occupies a needed target, so a failure never leaves the order in a partial state.
+  Note `SalesOrderItems.lineNumber` is a **`smallint`**, so the parked base must stay within smallint
+  range for the client's line counts (headroom 1000 is safe at observed volumes).
+
+- **⚠ Canon + Endeavor Health ITEM_RECEIPTS froze because `fetchPurchaseOrderById` never populated the
+  PO vendor (deployed/verified 2026-08-29).** The PO shim's `->entity` was always null (its SuiteQL
+  SELECT omitted `entity`, unlike `listPurchaseOrders`), so the fail-loud missing-entity guard in
+  `syncPurchaseOrderFromNetsuite` (reached via `syncItemReceiptFromNetsuite`) threw **every** run —
+  freezing Canon (clientId 12) and Endeavor Health (clientId 11) ITEM_RECEIPTS at `1-RUNNING`,
+  watermarks frozen ~8/~5 days. Fix is in the shim
+  ([rest-shim doc](../../library/features/netsuite-suiteql-rest-shim.md)): add
+  `entity, BUILTIN.DF(entity) AS entity_name` and build `$shim->entity`. `library`-only, additive; a
+  genuinely vendorless PO still throws. Deployed ~12:52 — both sections self-healed RUNNING→IDLE and
+  resumed advancing. Same class as the missing `tl.location` SELECT.
+
+- **⚠ Diagnostic refinement — the stuck signal is specifically `1-RUNNING`, NOT any `*-RUNNING`.** A
+  section frozen because it throws every run collapses its window ÷3 all the way to **1 second**, so its
+  `NETSUITE_EXECUTION_MODE_*` reads exactly **`1-RUNNING`**. A **large-window** `<big>-RUNNING` (e.g.
+  `864000-RUNNING`) is usually just a healthy section caught **mid-execution** at the query instant —
+  not stuck. And a large-window value with a *years-old* watermark can be a legitimate long **backfill**
+  (e.g. NYCHH backfilling from 2020/2021), also not a bug. Read the window magnitude, not just the
+  RUNNING state. **Sweep method:** read every client's `Client_<X>.Parameters` for
+  `NETSUITE_EXECUTION_MODE_*` / `NETSUITE_LAST_SYNC_DATETIME_*`; a cross-schema `UNION` over many client
+  DBs needs `CONVERT(<col> USING utf8mb4)` on the value column or MySQL raises *"Illegal mix of
+  collations."* Then map each stuck client/section to an exact crash **file:line** via prod
+  `Logs.Event` — note **`Logs.Issue` has no `clientId` column but `Logs.Event` does**, so filter
+  `Logs.Event` by `clientId`.
+
+- **OPEN, NOT FIXED (observed 2026-08-29, low priority — separate tickets):**
+  - **AIG (clientId 32, held account)** crashes in the **LOOKUPS** phase at
+    `common_sync_togasupply.php:463` with `Undefined property: stdClass::$companyName` — a
+    customer-shim shape gap that **survived** the earlier `fetchCustomerById` `addressbookList` fix.
+    (AIG remains a held/non-supply account — see the AIG gotcha above.)
+  - **Adyen (clientId 48)** crashes at `common_sync_togasupply.php:99` *"Invalid API Response"* from
+    `App_Api_Toga2::send()` at **setup** — it is **stuck-IDLE** (crashes before any section writes
+    RUNNING, so the `1-RUNNING` sweep misses it — same class as a LOOKUPS/setup abort). Likely an
+    **improperly onboarded** client (cf. AIG "listed ≠ functioning"). No knowledge-base client profile
+    exists for Adyen; no supply work has been done for it — confirm it's a real supply client before
+    onboarding effort.
+  Both are pre-existing and unrelated to this session's fixes.
+
 ## Change history
 
+- 2026-08-29 — **Deploy/verify/stabilize session for the SOAP→REST conversion.** Root-caused the
+  **Canon + Endeavor Health ITEM_RECEIPTS** freeze: `fetchPurchaseOrderById` never populated the PO
+  vendor (`entity` missing from its SuiteQL SELECT), so `syncPurchaseOrderFromNetsuite`'s fail-loud
+  missing-entity guard threw every run — fixed in the shim (add `entity`/`BUILTIN.DF(entity)`, build
+  `$shim->entity`); deployed ~12:52, both sections self-healed. Migrated **invoice-PDF retrieval to a
+  RESTlet** (`/record/v1/file/{id}` 404s — no token-only REST path to File Cabinet binary exists), which
+  unblocked Endeavor's whole pipeline; new [File Cabinet content via
+  RESTlet](../../library/features/netsuite-filecabinet-restlet.md) doc. Recorded the **`1-RUNNING`
+  (window collapsed to 1s) = stuck** vs large-window `*-RUNNING` (healthy/mid-run or legit backfill)
+  diagnostic, the cross-schema `Parameters` sweep needing `CONVERT(... USING utf8mb4)`, and that
+  `Logs.Event` (not `Logs.Issue`) carries `clientId`. Logged two OPEN, unfixed items: **AIG** (clientId
+  32) LOOKUPS crash at `:463` (`Undefined property $companyName`, customer-shim gap that survived the
+  addressbook fix) and **Adyen** (clientId 48) setup crash at `:99` (stuck-IDLE, likely improperly
+  onboarded). (jcardinal)
+- 2026-08-29 — **Root-caused the 5 stuck togasupply integrations as REST-shim SHAPE mismatches, not
+  dead SOAP, and fixed them.** SOAP was up; the earlier SOAP→REST conversion left shims returning
+  null/absent where SOAP had values, and a 1.0 unguarded read of that null → PHP warning →
+  `App_Error` `exit()` → the section frozen at `RUNNING`/watermark stuck (presents identically to a
+  poison order). Recorded the durable diagnosis playbook (map each stuck `clientId`/section to a crash
+  file:line via prod `Logs.Issue`/`Logs.Event`, the `App_Error_Capture` tables). Shipped per-client
+  `library` fixes, all fail-loud: item-receipts SOAP→REST (`fetchItemReceiptById` +
+  `fetchPurchaseOrderById` + `fetchItemFlagsByIds`, RtnAuth skip, unknown `createdFrom` throws — Canon
+  + Endeavor Health); PO missing-entity throw + `arg13` warehouse-lookup TypeError fix +
+  `fetchSalesOrderByIdFull` + 2400-ACL throw; TransferOrder all-catalog `[0]`/`->uuid` fix + unmapped
+  status throw (GroWrk); IF empty-`orderLine` "Undefined index" guard (Elite/Quad/S&P/Prudential-EDI);
+  the `isStockingOrder` scriptId case-mismatch that mis-routed **every** NYCHH order to a Transfer
+  Order; and the AIG `fetchCustomerById` address-shim crash. Also **completed SOAP→REST for the whole
+  togasupply path** (zero `App_NetSuite::` network calls remain; new `fetchInvoiceFile` /
+  `fetchItemFlagsByIds` / `fetchInventoryAdjustmentById`, `listAllCustomers`→`listCustomers`+companyName),
+  and replaced the SALES_ORDERS renumber with a **validate-before-mutate two-phase UPDATE** (park at
+  temp lineNumbers, then finalize; throws before any PUT) superseding the delete-recreate + pinned-skip
+  approach. (jcardinal)
 - 2026-08-29 — The inventory-adjustment sync (and the on-demand fulfillment / sales-order import it
   triggers) in `syncInventoryAdjustmentFromNetsuite` was **migrated SOAP → REST/SuiteQL**, closing the
   SOAP round-trip cost flagged 2026-08-28; the remaining error-log-and-swallow `try/catch` in the

package/knowledge/1.0/standards/backend-php.md

@@ -5,13 +5,14 @@ project: Library
 client: shared
 type: standard
 status: active
-updated: 2026-08-21
+updated: 2026-08-29
 owners: [jcardinal, rgirish, mhammontree, ajean]
 files: []
 related:
   - ../apps/library/architecture.md
   - ../apps/library/features/email-queue-attachments.md
   - ../apps/library/features/error-capture-1-0.md
+  - ../apps/worker/features/netsuite-togasupply-per-client-sync.md
   - ../../2.0/standards/backend-php.md
 ---
 
@@ -507,9 +508,12 @@ Additional verified detail:
 - **Diagnostic fingerprint:** an occurrence count that is an exact multiple of the job's
   scheduled frequency (e.g. 20/hour for `*/3 * * * *`) means **one** poisoned item retried,
   not many distinct failures.
-- **In job code, prevention beats containment.** Guard optional data with `isset()` before
-  reading it, and consume/ack the item in a `finally` (or under a `catch (Throwable)`) so a
-  genuine exception cannot strand it. You cannot wrap your way out of a notice.
+- **In job code, prevention beats containment for a warning.** Guard optional data with
+  `isset()` before reading it — you cannot wrap your way out of a notice. This is about the
+  uncatchable-warning mechanics above; it is **not** license to ack-and-continue past a genuine
+  (catchable) exception in a reconciliation loop. See *Fail-loud is the default for
+  import/sync/cron code* below before reaching for a `finally`/`catch (Throwable)` ack — that
+  pattern caused a real data-loss incident in this exact engine.
 
 > **Existing guidance under review.** An earlier version of this section recommended
 > wrapping fragile I/O in `try { ... } catch (\ErrorException $e)` relying on the GLOBAL
@@ -521,6 +525,59 @@ Additional verified detail:
 > (~L112-123), `clients/prudential/profile.md` (~L75). Prefer the local-handler pattern
 > from the canonical section above for anything new.
 
+### Fail-loud is the default for import/sync/cron code — never catch-and-continue or log-and-skip a record
+
+**This is the standing policy for reconciliation jobs; it supersedes the "consume/ack in a
+`finally`" framing in the section above wherever the two would conflict.** For import/sync/cron
+code that reconciles data from an external system into ours — the NetSuite → TOGa Supply sync
+(`worker/crons/toga2/netsuite/`) is the canonical example — the rule is: **if anything prevents
+the job from producing accurate data, let it throw.** The exception must propagate and abort the
+run. Do not:
+
+- Wrap a per-record dispatch in `try { … } catch (Throwable) { error_log(...) }` and move on to
+  the next record.
+- Wrap a section/job in `try { … } finally { <advance the checkpoint> }` so the checkpoint still
+  moves even when the try body threw.
+- Downgrade a hard failure to a log line and keep going "because most records succeeded."
+
+**Why.** These jobs track progress with an execution-mode flag plus a watermark/cursor (e.g.
+`NETSUITE_EXECUTION_MODE_*` / `NETSUITE_LAST_SYNC_DATETIME_*`). A thrown, unhandled exception
+leaves that state at **RUNNING** with the watermark **not advanced** past the failure — visible
+proof the run failed, the record still owes a retry, and a developer needs to look. Catching the
+exception and continuing does the opposite: it drops the record, silently advances the checkpoint
+past it (the loss becomes **permanent**, not retried), and can even reset the mode back to
+**IDLE**, telling the next run "everything's fine" when it isn't. **Data accuracy is prioritized
+over availability here — even for a failure that is not auto-retriable** (an ACL/permission
+error, a malformed upstream record, a constraint violation). A run stuck at RUNNING that needs a
+developer to reconcile it is the correct, intended outcome; a clean-looking IDLE run that quietly
+dropped data is not.
+
+**The one legitimate exception is a true message/inbox queue** — a loop reading items one at a
+time from something like an IMAP inbox or an SQS queue, where an item that is never acknowledged
+blocks every item behind it forever. There, and only there, consuming/acking the poison item so
+the queue can move past it is defensible — and even then, **prefer surfacing** the failure
+(dead-letter it, alert on it, record it against the item) over silently discarding it. A
+reconciliation cron with its own per-client cursor is **not** this case: nothing blocks behind a
+RUNNING watermark except that one client's own next run, which is exactly what the window
+grow/shrink behavior is designed around.
+
+**Real incident.** On 2026-08-20 a change wrapped every per-order dispatch across all six
+NetSuite → TOGa Supply sections in `catch (Throwable) { error_log }` and wrapped each section in
+`try { … } finally { finishModeIteration }`. It looked like an improvement — "don't let one bad
+order jam the sync" — but it silently dropped any record whose write failed, advanced the
+watermark past it, and reset the mode to IDLE, so a failed run looked identical to a successful
+one. This was found and reverted 2026-08-27 (all per-order catches and section-level
+`try/finally` removed — a failure now aborts the section with the mode left RUNNING), and a
+related error-log-and-swallow path in the same engine's inventory-adjustment section was removed
+again 2026-08-29 during its SOAP→REST migration. Full detail: [NetSuite → TOGa Supply Per-Client
+Sync](../apps/worker/features/netsuite-togasupply-per-client-sync.md).
+
+This does not relax the "any PHP warning terminates the request" mechanics above, or the two
+sanctioned local opt-outs for a **notice/warning** you must genuinely contain (a monitoring ping,
+a defensively-guarded optional read). Those exist to survive an *uncatchable warning* without
+crashing something that must not crash. This rule is about a **catchable exception** in a
+reconciliation loop, where surviving it quietly is exactly the wrong behavior.
+
 ### Test mode silently disables queued email — non-prod cannot validate a queue bug
 
 `App_Email::setQueueForSending()` (`library/app/email.php` ~L161) is gated on

package/knowledge/1.0/standards/framework-rules.md

@@ -5,12 +5,13 @@ project: Library
 client: shared
 type: standard
 status: active
-updated: 2026-08-06
+updated: 2026-08-29
 owners: [jcardinal, ajean]
 files: []
 related:
   - ../apps/library/architecture.md
   - ../apps/worker/architecture.md
+  - backend-php.md
 ---
 
 # Framework 1.0 (App_) Rules
@@ -150,3 +151,12 @@ public function processOrder(): void {
     $this->response->success($result);
 }
 ```
+
+**This pattern is scoped to a single request/response unit of work — it does not extend to
+import/sync/cron code.** Above, catching stops the *whole* action and returns one error response;
+nothing after the catch runs "successfully" on partial data. A batch/reconciliation job (a cron
+looping over many external records, advancing a cursor/watermark) is a different shape: catching
+per-record there and continuing to the next one is exactly the anti-pattern that caused a
+data-loss incident in the NetSuite → TOGa Supply sync. For that class of code, see
+`1.0/standards/backend-php.md` § *Fail-loud is the default for import/sync/cron code* — the job
+must throw and abort, not catch-and-continue.

package/knowledge/2.0/standards/backend-php.md

@@ -5,13 +5,15 @@ project: _Underscore
 client: shared
 type: standard
 status: active
-updated: 2026-08-25
+updated: 2026-08-29
 owners: [jcardinal, mhammontree, dfranks, bala]
 files: []
 related:
   - ../apps/_underscore/architecture.md
   - ../apps/api2/architecture.md
   - ../apps/worker2/architecture.md
+  - framework-rules.md
+  - ../../1.0/standards/backend-php.md
 ---
 
 ## General Principals
@@ -677,6 +679,14 @@ $c = 'Hello ' . $world;
 ### PHP Error Handling
 
 * Always use exceptions to handle unexpected conditions.
+* **Import/sync/cron code that reconciles external data must fail loud — never catch-and-continue
+  or log-and-skip a record.** A mid-loop failure must throw and abort the job, leaving its
+  checkpoint/watermark unmoved for retry/investigation, rather than being caught, logged, and
+  stepped over. See `2.0/standards/framework-rules.md` § *The above is for a per-entity job — a
+  reconciliation/import worker that loops over many records is different, and must fail loud* and
+  `1.0/standards/backend-php.md` § *Fail-loud is the default for import/sync/cron code* (the
+  policy this corrects, after a per-order catch-and-continue change caused a data-loss incident in
+  the NetSuite → TOGa Supply sync).
 
 ### Choosing an exception class is a ROUTING decision, not style
 
@@ -946,6 +956,10 @@ See: 2.0/apps/worker2/features/cross-account-aws-access.md
 
 ## Change history
 
+- 2026-08-29 — Added a fail-loud cross-reference under **PHP Error Handling**: import/sync/cron
+  code that reconciles external data must never catch-and-continue or log-and-skip a record;
+  points to the full policy and the NetSuite → TOGa Supply sync data-loss incident recorded in
+  `framework-rules.md` and `1.0/standards/backend-php.md`. (jcardinal)
 - 2026-08-25 — Clarified the exception-routing rule: `_Exception_Validation` is a valid 400 throw
   site from a **scripted-API method** too, not only a pre/post interceptor or model op. Verified at
   the outer `execute()`/`api()` catch in `api2/Controller/Index.php` (~L317) where

package/knowledge/2.0/standards/framework-rules.md

@@ -5,7 +5,7 @@ project: _Underscore
 client: shared
 type: standard
 status: active
-updated: 2026-08-28
+updated: 2026-08-29
 owners: [jcardinal, ajean, apeterson]
 files: []
 related:
@@ -14,6 +14,8 @@ related:
   - ../apps/api2/architecture.md
   - ../apps/dbchanges2/architecture.md
   - ../apps/dbchanges2/features/surface-layer-schema.md
+  - backend-php.md
+  - ../../1.0/standards/backend-php.md
 ---
 
 # Framework 2.0 (_underscore) Rules
@@ -298,6 +300,23 @@ returns nothing (or returns only when it succeeds) leaves a blind spot in the au
 only for genuinely retriable conditions, where the queue's retry is the intended behaviour; a
 handled non-retriable failure should return a result describing what happened rather than throwing.
 
+### The above is for a per-entity job — a reconciliation/import worker that loops over many records is different, and must fail loud
+
+`ProcessOrder(int $orderId)` above is scoped to **one** record: the job either succeeds or fails
+as a whole, and "log and return a failure result" for a non-retriable error means *this job*
+(this one order) is done — no other record is silently dropped alongside it. Do not generalize
+that shape to a worker action that **internally loops over many records to reconcile external
+data** (a NetSuite import, a bulk sync, anything advancing its own cursor/watermark across a
+batch). For that shape, catching a per-record failure inside the loop and moving on to the next
+record is the same anti-pattern documented in `1.0/standards/backend-php.md` § *Fail-loud is the
+default for import/sync/cron code* — it silently drops the record and can leave the job's own
+progress-tracking (checkpoint/cursor) looking healthy when it isn't. A batch/reconciliation worker
+must let a mid-loop failure **throw and abort the job**, leaving its checkpoint unmoved so the
+queue's retry (or a developer) picks up where it actually left off — never catch-per-record and
+continue. Data accuracy outranks job "completion," even when the failure is not retriable (e.g. an
+ACL error on one record) — a developer reconciling a stalled job is the intended outcome, not a
+job that reports success having quietly skipped records.
+
 ## `c_`-prefixed custom columns are framework-dynamic
 
 Columns prefixed `c_` are **custom/dynamic** columns that the `_underscore` model layer handles
@@ -317,6 +336,14 @@ and not how the convention works. Use a non-`c_` column when you *do* want the f
 ACL-governed field treatment.
 
 ## Change history
+- 2026-08-29 — Added a distinction under **Error handling in workers**: the documented
+  `ProcessOrder`-style per-entity pattern (catch non-retriable, log, return a failure result) does
+  **not** extend to a worker action that loops over many records to reconcile external data (a
+  NetSuite import, a bulk sync). That shape must fail loud — a mid-loop failure throws and aborts
+  the job so its checkpoint/cursor stays unmoved — matching the fail-loud policy corrected in
+  `1.0/standards/backend-php.md` after a per-order catch-and-continue change caused a silent
+  data-loss incident in the NetSuite → TOGa Supply sync (found 2026-08-20, reverted 2026-08-27,
+  final swallow removed 2026-08-29). (jcardinal)
 - 2026-08-28 — Extended § Schema migrations with the production-state rules. Recorded that cluster
   isolation is **physical** (verified: the prod Core reader exposes only Core/Forecast/
   Forecast_Archive/Team, the client reader only `Client_*`), so a cross-database reference cannot

package/knowledge/INDEX.md

@@ -4,7 +4,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 
 ## 1.0 framework
 
-- **library** (Library) _(framework core)_ — 21 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
+- **library** (Library) _(framework core)_ — 22 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
 - **worker** (Worker) — 31 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
 - **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
 - **worker1.5** (Worker 1.5) — 0 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
@@ -46,9 +46,11 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 ## Clients
 
 - **AIG (Staples Protection Plan)** (`aig`) → [clients/aig/INDEX.md](clients/aig/INDEX.md)
+- **Canon** (`canon`) → [clients/canon/INDEX.md](clients/canon/INDEX.md)
 - **Compass Canada** (`compass-canada`) → [clients/compass-canada/INDEX.md](clients/compass-canada/INDEX.md)
 - **Compass USA** (`compass-usa`) → [clients/compass-usa/INDEX.md](clients/compass-usa/INDEX.md)
 - **Elite** (`elite`) → [clients/elite/INDEX.md](clients/elite/INDEX.md)
+- **Endeavor Health** (`endeavor-health`) → [clients/endeavor-health/INDEX.md](clients/endeavor-health/INDEX.md)
 - **Fordham University** (`fordham`) → [clients/fordham/INDEX.md](clients/fordham/INDEX.md)
 - **GroWrk** (`growrk`) → [clients/growrk/INDEX.md](clients/growrk/INDEX.md)
 - **Northwell Health** (`northwell`) → [clients/northwell/INDEX.md](clients/northwell/INDEX.md)
@@ -60,6 +62,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 - **Quad Graphics** (`quad`) → [clients/quad/INDEX.md](clients/quad/INDEX.md)
 - **Rate** (`rate`) → [clients/rate/INDEX.md](clients/rate/INDEX.md)
 - **Richmond University Medical Center** (`rumcsi`) → [clients/rumcsi/INDEX.md](clients/rumcsi/INDEX.md)
+- **S&P Global** (`spglobal`) → [clients/spglobal/INDEX.md](clients/spglobal/INDEX.md)
 - **Staples** (`staples`) → [clients/staples/INDEX.md](clients/staples/INDEX.md)
 - **Tow Foundation** (`tow-foundation`) → [clients/tow-foundation/INDEX.md](clients/tow-foundation/INDEX.md)
 - **TOGA Technology** (`true`) → [clients/true/INDEX.md](clients/true/INDEX.md)

package/knowledge/clients/aig/profile.md

@@ -6,13 +6,14 @@ apps:
   - api2
   - dbchanges2
   - library
+  - worker
   - worker2
 project: API
 client: aig
 type: profile
 status: active
-updated: 2026-08-17
-owners: ["mhammontree"]
+updated: 2026-08-29
+owners: ["mhammontree", "jcardinal"]
 files: []
 related:
   - clients/aig/features/entitlement-intake.md

package/knowledge/clients/canon/INDEX.md

@@ -0,0 +1,5 @@
+# Client: Canon `canon`
+
+| Doc | Framework | Summary | Files |
+|-----|-----------|---------|-------|
+| [Canon](profile.md) | 2.0 | Canon is a **TOGa-Supply NetSuite integration client** on the shared 1.0 NetSuite → TOGa Supply per-client sync (`worker` + `library`). |  |

package/knowledge/clients/canon/profile.md

@@ -0,0 +1,34 @@
+---
+title: "Canon"
+framework: "2.0"
+apps:
+  - library
+  - worker
+project: Worker
+client: canon
+type: profile
+status: active
+updated: 2026-08-29
+owners: ["jcardinal"]
+files: []
+related:
+  - ../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md
+---
+
+## Summary
+
+Canon is a **TOGa-Supply NetSuite integration client** on the shared 1.0 NetSuite → TOGa Supply
+per-client sync (`worker` + `library`). Its thin wrapper
+`worker/crons/toga2/netsuite/sync_togasupply_canon.php` is the **canonical example** referenced by
+the shared-engine doc — copy it when onboarding a new client. NetSuite transactions (sales orders,
+POs, invoices, item receipts, item fulfillments, inventory adjustments) sync into Canon's 2.0
+`Client_Canon` tenant through the TOGa2 API.
+
+This profile was seeded during the 2026-08-29 root-cause work that fixed Canon's stuck
+`ITEM_RECEIPTS` section — the item-receipt path was migrated SOAP → REST (`fetchItemReceiptById`);
+full mechanics and the diagnosis playbook live in
+[NetSuite → TOGa Supply per-client sync](../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md).
+
+## Integration touchpoints
+
+- **`worker` / `library` (1.0)** — the shared togasupply sync engine + `App_Api_Toga2` bridge.

package/knowledge/clients/endeavor-health/INDEX.md

@@ -0,0 +1,5 @@
+# Client: Endeavor Health `endeavor-health`
+
+| Doc | Framework | Summary | Files |
+|-----|-----------|---------|-------|
+| [Endeavor Health](profile.md) | 2.0 | Endeavor Health is a **TOGa-Supply NetSuite integration client** on the shared 1.0 NetSuite → TOGa Supply per-client sync (`worker` + `library`). |  |

package/knowledge/clients/endeavor-health/profile.md

@@ -0,0 +1,32 @@
+---
+title: "Endeavor Health"
+framework: "2.0"
+apps:
+  - library
+  - worker
+project: Worker
+client: endeavor-health
+type: profile
+status: active
+updated: 2026-08-29
+owners: ["jcardinal"]
+files: []
+related:
+  - ../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md
+---
+
+## Summary
+
+Endeavor Health is a **TOGa-Supply NetSuite integration client** on the shared 1.0 NetSuite → TOGa
+Supply per-client sync (`worker` + `library`). NetSuite transactions sync into Endeavor's 2.0
+`Client_<Id>` tenant through the TOGa2 API via a thin
+`worker/crons/toga2/netsuite/sync_togasupply_*.php` wrapper.
+
+This profile was seeded during the 2026-08-29 root-cause work that fixed Endeavor's stuck
+`ITEM_RECEIPTS` section — the item-receipt path was migrated SOAP → REST (`fetchItemReceiptById`);
+full mechanics and the diagnosis playbook live in
+[NetSuite → TOGa Supply per-client sync](../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md).
+
+## Integration touchpoints
+
+- **`worker` / `library` (1.0)** — the shared togasupply sync engine + `App_Api_Toga2` bridge.

package/knowledge/clients/spglobal/INDEX.md

@@ -0,0 +1,5 @@
+# Client: S&P Global `spglobal`
+
+| Doc | Framework | Summary | Files |
+|-----|-----------|---------|-------|
+| [S&P Global](profile.md) | 2.0 | S&P Global is a **TOGa-Supply NetSuite integration client** on the shared 1.0 NetSuite → TOGa Supply per-client sync (`worker` + `library`). |  |

package/knowledge/clients/spglobal/profile.md

@@ -0,0 +1,32 @@
+---
+title: "S&P Global"
+framework: "2.0"
+apps:
+  - library
+  - worker
+project: Worker
+client: spglobal
+type: profile
+status: active
+updated: 2026-08-29
+owners: ["jcardinal"]
+files: []
+related:
+  - ../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md
+---
+
+## Summary
+
+S&P Global is a **TOGa-Supply NetSuite integration client** on the shared 1.0 NetSuite → TOGa Supply
+per-client sync (`worker` + `library`). NetSuite transactions sync into its 2.0 `Client_<Id>` tenant
+through the TOGa2 API via a thin `worker/crons/toga2/netsuite/sync_togasupply_*.php` wrapper.
+
+This profile was seeded during the 2026-08-29 root-cause work on the stuck togasupply integrations:
+S&P Global was one of the clients exposed to the item-fulfillment "Undefined index" on an empty
+`orderLine` (`syncItemFulfillmentFromNetsuite`), fixed in shared code. Full mechanics and the
+diagnosis playbook live in
+[NetSuite → TOGa Supply per-client sync](../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md).
+
+## Integration touchpoints
+
+- **`worker` / `library` (1.0)** — the shared togasupply sync engine + `App_Api_Toga2` bridge.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.698",
+  "version": "1.0.700",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",