toga-ai 1.0.949 → 1.0.951

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -43,7 +43,11 @@ UMA students reach our system through a **daily SFTP file**, not an API. `togade
43
43
 
44
44
  Nothing is sent at escalation time — an escalated ticket waits for the next run.
45
45
 
46
- **Ticket sync schedule** (`schedules/cron.worker.togadesk.json`, server time = Central): **once per day** — Mon–Thu `0 15 * * 1-4`, Fri `30 11 * * 5`, Sat `0 17 * * 6`, Mon `0 1 * * 1`. No Sunday run (~32 h gap Sat 17:00 → Mon 01:00). Runs take ~2 min. Fix if asked: restore an hourly (or tighter) range.
46
+ **Ticket sync schedule** (`schedules/cron.worker.togadesk.json`, server time = Central): **once per day**, 4 separate entries kept on purpose (do not merge):
47
+ - Mon–Thu `30 7 * * 1-4` (7:30 AM CT) — UMA wants escalations in the morning (Jonathan, TRUE-82504). **Pending deploy as of 2026-10-05**; prod still runs `0 15 * * 1-4` until the next worker deploy.
48
+ - Fri `30 11 * * 5`, Sat `0 17 * * 6`, Mon `0 1 * * 1`.
49
+
50
+ No Sunday run (~32 h gap Sat 17:00 → Mon 01:00). Runs take ~2 min. Near-real-time escalation = a separate decision (restore an hourly range).
47
51
 
48
52
  ## Health check (read-only prod, env `legacy`)
49
53
 
@@ -63,9 +67,10 @@ Nothing is sent at escalation time — an escalated ticket waits for the next ru
63
67
  - **PII.** Files hold DOB and SSN last 4. When investigating, pull only dates/file names from FileLog, never `fileData`.
64
68
  - **Hardcoded credentials — treat as leaked, rotate.** SFTP login is hardcoded in `sync_uma_customers.php`; the Ivanti API key is hardcoded in `library/app/api/uma.php`. Do not copy them anywhere.
65
69
  - Does not call `App_Database::closeConnections()`.
66
- - **Escalations batch once a day.** Cron entry names still say windows like "9:00–21:00" — misleading; the cron expressions are single hours. Cause: `d53e1ab35` (TRUE-61196, 2025-03-10, UTC→CST) replaced hourly ranges (`0 3-15 * * 2-5`, `0 0-15 * * 6`, `0 0-17 * * 0`, `0 1-15 * * 1`) with single hours; later TZ shifts `9732b7b50` (TRUE-74692) and `9f16ad1de` (TRUE-77365) kept single hours.
70
+ - **Escalations batch once a day.** Only `schedule` controls run time; `name` is cosmetic (`ebs/cron.worker.php` uses it only in error messages). Old names said hourly windows like "9:00–21:00" and the Mon 1 AM entry was labeled "Sun" — TRUE-82504 renamed all 4 to their real CT/ET times (pending deploy as of 2026-10-05). Keep names in sync when changing a schedule. Cause of once-a-day: `d53e1ab35` (TRUE-61196, 2025-03-10, UTC→CST) replaced hourly ranges (`0 3-15 * * 2-5`, `0 0-15 * * 6`, `0 0-17 * * 0`, `0 1-15 * * 1`) with single hours; later TZ shifts `9732b7b50` (TRUE-74692) and `9f16ad1de` (TRUE-77365) kept single hours.
67
71
  - **`tickets.dtSynced` is not the send time.** On push the code sets it to the ticket's created timestamp; some rows show the run time instead.
68
72
 
69
73
  ## Change history
70
74
  - 2026-09-28 — Documented (TRUE-82352 investigation). Open incident: no UMA file since 2026-09-20; runs 9-21..9-27 clean with empty SFTP; togadesk worker instance changed that day but SFTP login still works from the new box — likely UMA stopped uploading, awaiting UMA confirmation. No wrong deactivations; new students and their tickets missing since 9-20. (mhammontree)
71
75
  - 2026-10-05 — TRUE-82504 escalation-batch investigation: escalations reach UMA's Ivanti once a day (prod: tickets created 9/29 20:17 → 9/30 08:21 all sent 9/30 15:01–15:02 CT; repeated 10/1). Findings delivered, no fix requested. (mhammontree)
76
+ - 2026-10-05 — TRUE-82504 follow-up: Mon–Thu ticket sync moved `0 15` → `30 7` (morning escalations, per Jonathan); all 4 entry names fixed to real times. Branch TRUE-82504, not deployed. (mhammontree)
@@ -6,7 +6,7 @@ project: API
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-09-16
9
+ updated: 2026-10-05
10
10
  owners: ["mhammontree", "dfranks", "bala", "snaredla", "jcardinal"]
11
11
  files:
12
12
  - api2/Component/Api/V2/V2.php
@@ -210,7 +210,7 @@ A `postPost`/`postPut` hook that hands work to worker2 (`_Worker::runTask`) sits
210
210
  So this entire class of bug is **invisible on beta/dev-sandbox and fails on every production request.** Never conclude "it works on beta" for an in-request read-after-write.
211
211
  5. **The pattern that works:**
212
212
  - Read the record **inside the same request** with `internalApiRequest`, wrapped in `getIsReadHostEnabled()` / `setIsReadHostEnabled(false)` and restored in a **`finally`** (pin the read to the writer, then put the flag back for everyone else).
213
- - **Pass the resulting object along in the job parameters** so the worker never re-reads it. (The worker action must accept it — see [Toga → NetSuite sales-order push](../../worker2/features/netsuite-salesorder-outbound-push.md), whose `?object $salesOrder` parameter exists for exactly this.)
213
+ - **Superseded 2026-10-05 for queueing: pass only the uuid, never the object** — a nested record can exceed the SQS 256KB cap and the job is lost. The worker reads the record itself with a bounded wait for the commit. Rule + reference: [runTask parameters are ids only](../../worker2/features/creating-worker-actions.md#-runtask-parameters-are-idsuuids-only--never-record-objects). (The Elite sales-order push still carries `?object $salesOrder` — legacy.) The writer-pinned read stays correct when the hook itself needs the record.
214
214
  - Pass **`false`** for `internalApiRequest`'s 5th argument (`$throwExceptionsOnError`). Its throw is guarded by `!empty($this->response->messages)` on the **shared** response object, and `json_encode([])` is the **truthy** string `"[]"` — so it throws even with **zero real errors** once any message (even a warning) is present anywhere in the request. **Not theoretical:** it took Elite's production `POST /v2/sales-orders` down on 2026-08-20, from the *parent* model's own `internalApiRequest` (see [the inherited-chain section](#-activating-a-clients-first-interceptor-row-also-switches-on-the-whole-inherited-chain)). The throw site is `V2.php:2416`.
215
215
 
216
216
  **Two more failure modes on this path:**
@@ -267,7 +267,7 @@ The reverse direction: not "queue a worker from an interceptor" but **"run the p
267
267
  - **Registration is per-environment.** Like `RecordFields` / `RecordScripts` / ACL rows, this is metadata — "works in prod, not in beta/dev-sandbox" is the signature of drift, not of a bad deploy. See the [non-prod metadata drift repair workflow](../../dbchanges2/workflows/nonprod-metadata-drift-repair.md).
268
268
  - **⚠ An in-request read-after-write in an interceptor works on beta and fails 100% in production.** Reads default to the **reader** connection, which cannot see the request's own uncommitted row; dev-sandbox points writer and reader at the same endpoint, so it hides the bug. Pin with `setIsReadHostEnabled(false)` (restore in `finally`).
269
269
  - **`internalApiRequest` has no `data` envelope** (`$response->salesOrders`, not `$response->data->salesOrders`), and `minDepth` cannot deepen a POST interceptor's payload — the bump happens after serialization. A wrong read path here is a **silent no-queue with a clean 201**.
270
- - **Anything a post interceptor queues must survive the transaction boundary** — pass the record snapshot in the job parameters rather than letting the worker re-read it.
270
+ - **Anything a post interceptor queues must survive the transaction boundary** — queue only the uuid; the worker reads with a bounded wait-and-retry until the commit lands. A `catch` around `runTask` must `_Error::captureException` (not only `error_log`). See [runTask parameters are ids only](../../worker2/features/creating-worker-actions.md#-runtask-parameters-are-idsuuids-only--never-record-objects).
271
271
  - **Never emit output from an interceptor.** A stray `echo`/`print_r` corrupts the `/v2` `{isSuccess,…,data}` envelope (this really happened in `ItemFulfillment.php` — see the carrier-shipping-labels doc).
272
272
  - **Interceptors are a client-behavior extension point.** Per api2's architecture doc, client specifics belong in `_Model_<Slug>_X` overrides + interceptor hooks, **not** in `V2.php`.
273
273
 
@@ -20,6 +20,7 @@ related:
20
20
  - ./netsuite-opportunity-client-labels.md
21
21
  - ../../api2/features/api-payload-interceptors.md
22
22
  - ../../_underscore/features/outbound-api-rate-budget.md
23
+ - ./netsuite-transferorder-outbound-push.md
23
24
  ---
24
25
 
25
26
  ## Summary
@@ -409,6 +410,17 @@ deploy that did not include the new file). Check the file is on the box before d
409
410
  **Worth fixing:** moving the `method_exists` call **inside** the `try` would turn this into a normal
410
411
  failed job with a real error message instead of a silent half-row.
411
412
 
413
+ ## ⚠ `runTask` parameters are ids/uuids only — never record objects
414
+
415
+ Team rule (jcardinal, 2026-10-05): `_Worker::runTask` parameters are **ids/uuids + small flags only**. Never put a record object in them — SQS messages must stay small.
416
+
417
+ - `runTask` **throws** above `_Worker::SQS_MAX_MESSAGE_BYTES = 262144`. A record read with nested children (depth 4–6) grows with the data around it and crosses the cap only on busy records — passes every test, fails later in prod. A "slimmed snapshot" is not an exception.
418
+ - **Producer queues before its own commit** (post interceptor, anything inside an open transaction): the worker reads the record itself with a **bounded wait-and-retry** (fixed attempts × fixed wait, inside the 300s budget), then throws. Reference: `fetchTransferOrderWhenCommitted()` in `worker2/Worker/Netsuite/SalesOrder.php`.
419
+ - **A producer `catch` around `runTask` must call `_Error::captureException`**, not only `error_log` — `error_log` leaves nothing in `Logs.Issue`/`Logs.Event`, so a dropped job is invisible.
420
+ - Legacy: the Elite sales-order push still carries `?object $salesOrder` (see [api payload interceptors](../../api2/features/api-payload-interceptors.md)) — same cap risk.
421
+
422
+ Incident: [transfer-order push SQS-cap](./netsuite-transferorder-outbound-push.md#-the-job-carries-only-the-uuid--never-the-order-sqs-cap-incident-2026-10-05).
423
+
412
424
  ## ⚠ Renaming an action is a CROSS-REPO rename — deploy both sides together
413
425
 
414
426
  The action name is a **string** held by the producers, not a symbol the compiler can check. Renaming
@@ -521,6 +533,7 @@ Rules: rename **both sides in one PR and one deploy**; grep every repo for the o
521
533
  commit-before-SQS transaction pattern that the worker relies on.
522
534
 
523
535
  ## Change history
536
+ - 2026-10-05 — Rule: `runTask` parameters are ids/uuids only; worker reads with bounded wait; producer catch must `_Error::captureException`. (jcardinal)
524
537
  - 2026-10-05 — Added read-host read-after-write gotcha for worker actions (TRUE-82185) (mhammontree)
525
538
  - 2026-09-17 — Added the **half-written `WorkerJobs` row** tell: a row with `dtStarted` but no
526
539
  `dtCompleted`/`isSuccess`/`output` means the **action class is not on the box**, not a hung job.
@@ -6,7 +6,7 @@ project: Worker
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-10-02
9
+ updated: 2026-10-05
10
10
  owners: ["bala", "jcardinal"]
11
11
  files:
12
12
  - worker2/Worker/Netsuite/SalesOrder.php
@@ -25,6 +25,7 @@ related:
25
25
  - ../../../../clients/nychh/features/transfer-order-netsuite-push.md
26
26
  - ../../../../clients/nychh/features/netsuite-transfer-order-import.md
27
27
  - ../../../../clients/elite/features/salesorder-netsuite-push.md
28
+ - ../../../../clients/nychh/features/oneuptime-transfer-order-netsuite-push-monitor.md
28
29
  ---
29
30
 
30
31
  Transfer-order → NetSuite push (a $0 sales order IS the transfer order) through the shared outbound route in `SalesOrder.php`; open for the transfer route, its customer/ship-to/tax gotchas, or onboarding a client.
@@ -33,12 +34,13 @@ Transfer-order → NetSuite push (a $0 sales order IS the transfer order) throug
33
34
 
34
35
  A transfer order created in TOGa Supply is pushed to NetSuite as a **zero-value sales order**. NetSuite has no transfer-order record for this account, so **a $0 sales order *is* the transfer order** (confirmed by the developer 2026-09-02) — the same representation the inbound importer relies on ([NYCHH NetSuite → TransferOrders import](../../../../clients/nychh/features/netsuite-transfer-order-import.md)).
35
36
 
36
- **Not a new worker.** Reuses the outbound push in `worker2/Worker/Netsuite/SalesOrder.php` ([Toga → NetSuite Sales-Order Push](./netsuite-salesorder-outbound-push.md)); read that doc first. Idempotency, the job-carried snapshot, the ~12s create, and the four `doCreate` outcomes are inherited unchanged. Everything here is the transfer-order route through that same file.
37
+ **Not a new worker.** Reuses the outbound push in `worker2/Worker/Netsuite/SalesOrder.php` ([Toga → NetSuite Sales-Order Push](./netsuite-salesorder-outbound-push.md)); read that doc first. **Unlike that push, the job carries only the uuid — the worker reads the order itself** (2026-10-05). Idempotency, the ~12s create, and the four `doCreate` outcomes are inherited unchanged. Everything here is the transfer-order route through that same file.
37
38
 
38
39
  **Status: LIVE, verified in production.** SO **289187** (internal id 7463537) was created from this path (2026-09-08). Deployed 2026-09-14 and confirmed working end to end. Landmark corrections:
39
40
 
40
41
  - 2026-09-08 (worker2 `cbafd25` on `_production`): the NetSuite customer is the CLIENT, not the destination location; `description` now sent as NetSuite `memo`.
41
42
  - 2026-09-09 → live 2026-09-14: the push was sending no ship-to address, so NetSuite used the customer's default site on every order — fixed by reading the destination Location's address (needs a depth-6 read).
43
+ - 2026-10-05: orders on busy POs were **never queued** (job over the SQS size cap, failure only `error_log`ged). Job now carries only the uuid; worker reads the order. See [the SQS-cap gotcha](#-the-job-carries-only-the-uuid--never-the-order-sqs-cap-incident-2026-10-05).
42
44
  - 2026-09-09→2026-09-14: sending that address then broke every create with an HTTP 400 tax error. Cause was NetSuite customer data, not code — the customer carried retired tax group **-8**. See [the tax-item section](#-the-customers-netsuite-tax-item-must-be-avatax-or-every-create-400s) before touching anything tax-related or onboarding a new client.
43
45
 
44
46
  ## How it works
@@ -49,7 +51,7 @@ A transfer order created in TOGa Supply is pushed to NetSuite as a **zero-value
49
51
  |---|---|---|
50
52
  | `Netsuite/SalesOrder/CreateNetSuiteFromTransferOrder` | `CreateNetSuiteFromTransferOrder` | push a transfer order as a $0 NetSuite sales order |
51
53
 
52
- Queue parameters (dispatcher spreads `parameters` as named args — **names are the contract**): `clientIdentifier`, `transferOrderUuid`, `transferOrder`, `dryRun`.
54
+ Queue parameters (dispatcher spreads `parameters` as named args — **names are the contract**): `clientIdentifier`, `transferOrderUuid`, `dryRun`. `?object $transferOrder` is kept **only** so jobs queued before 2026-10-05 still run — never send it.
53
55
 
54
56
  **A separate public entry point was chosen over an `isTransferOrder` flag on `CreateNetSuite`** — so the queue parameter names read correctly (`transferOrderUuid`, not `salesOrderUuid` carrying a transfer order). `CreateNetSuite` / `UpdateNetSuite` / `Sync` keep their original signatures untouched, which keeps Elite and Compass provably unaffected (developer's explicit requirement: *do not change the logic for the Elite service request*).
55
57
 
@@ -126,14 +128,9 @@ Live evidence: NetSuite order **7467799** (North Central Bronx in supply) shippe
126
128
  | sales order | `order → shipToAddress → state → country` | **4** (`TOGA_FETCH_DEPTH`, unchanged) |
127
129
  | transfer order | `order → destinationLocation → primaryLocationAddress → address → state → country` | **6** |
128
130
 
129
- At depth 4 the address comes back **missing with no error** — the max-depth omission rule in [V2 REST query contract](../../api2/features/v2-rest-query-contract.md). The order is read in **two** places, so both depths were raised:
131
+ At depth 4 the address comes back **missing with no error** — the max-depth omission rule in [V2 REST query contract](../../api2/features/v2-rest-query-contract.md). The worker reads transfer orders at `TOGA_FETCH_DEPTH__TRANSFER_ORDER = 6` (chosen by `$isTransferOrder` in `readSalesOrder()`). **Sales orders still read at 4.** Since 2026-10-05 this is the only read — `postPost` no longer reads the order (`NETSUITE_READ_DEPTH` removed).
130
132
 
131
- - `_Model_Client_TransferOrder::NETSUITE_READ_DEPTH` **4 → 6** — the `postPost` read done before queueing. The order travels with the job, so this is the read that matters in practice.
132
- - new `TOGA_FETCH_DEPTH__TRANSFER_ORDER = 6` in the worker, chosen by the `$isTransferOrder` flag in `fetchSalesOrder()` — the fallback path when no order is carried. **Sales orders still read at 4.**
133
-
134
- `NETSUITE_READ_DEPTH` stays a class constant (not INI) — it is a fact about the read shape, and the shape just got deeper.
135
-
136
- **Depth 6 is verified against the raw `Core.WorkerJobs.parameters` blob of a real job:** at 6 the full `destinationLocation.primaryLocationAddress.address.state.country` branch comes through. **Decoy:** the same payload also carries `contact.contactLocations[].location...address`, and *that* branch runs out of depth and has **no `state`**. If debugging a missing state/country, check which branch you are reading before raising depth again — the destination branch is complete.
133
+ **Decoy at depth 6:** the destination branch `destinationLocation.primaryLocationAddress.address.state.country` is complete, but `contact.contactLocations[].location...address` runs out of depth and has **no `state`**. Debugging a missing state/country → check which branch you are reading before raising depth.
137
134
 
138
135
  ### Idempotency — `toga-to-<transfer order uuid>`, never `number`
139
136
 
@@ -157,20 +154,20 @@ The uuid is the only value on the row unique everywhere and never moving. `exter
157
154
  - **Must NOT call `parent::postPost()`.** `_Model_Client_TransferOrder` extends `_Model_True`, which declares no `postPost`, so a parent call is fatal. The **opposite** of the Elite `postPost`, which must call its parent. Check the parent chain before adding the call by habit.
158
155
  - **Only does anything useful for a client whose model uses `_Trait_Netsuite_TransferOrder`** — that trait declares `c_netsuiteInternalSalesOrderId`. NYCHH is the only such client today, so **any other client given a `transfer-orders` interceptor row would start pushing to NetSuite** and break on the missing field. The row is the switch; treat adding one as a code change.
159
156
 
160
- The hook follows the five rules from [queueing background work from a post interceptor](../../api2/features/api-payload-interceptors.md): read the order back in-request with `internalApiRequest` at depth **4** (`NETSUITE_READ_DEPTH`), pinned to the writer (`setIsReadHostEnabled(false)`, restored in `finally`), `$throwExceptionsOnError = false`, and pass the order object along in job parameters so the worker never re-reads it. `runTask` is wrapped in **try/catch**, so a queue failure can never fail the transfer-order POST.
157
+ **`postPost` queues ONLY `{clientIdentifier, transferOrderUuid}`** — no read-back, no order object (developer rule, firm: SQS messages stay small; see [the SQS-cap gotcha](#-the-job-carries-only-the-uuid--never-the-order-sqs-cap-incident-2026-10-05)). `runTask` is wrapped in **try/catch** so a queue failure never fails the POST, and the catch now calls **`_Error::captureException`** (lands in `Logs.Issue`), not only `error_log`.
158
+
159
+ **The worker reads the order itself — with a bounded wait.** `runTask` is SQS-first and fires inside the api2 POST **before** the request's single commit (`api2/Controller/Index.php` `transactionStart` ~L316, commit ~L544), so the first read can miss the row. `fetchTransferOrderWhenCommitted()` retries `readSalesOrder()` `TRANSFER_ORDER_READ_ATTEMPTS = 6` times, `TRANSFER_ORDER_READ_WAIT_SECONDS = 5` apart (~25s, inside the 300s cap), then throws. No partial-read window: header + nested children commit in one transaction. `fetchSalesOrder()` is now a throwing wrapper over nullable `readSalesOrder()`; sales-order paths unchanged.
161
160
 
162
161
  **Two independent echo-loop guards** — an order that came *from* NetSuite must never be pushed back:
163
162
 
164
163
  1. The 1.0 importer's POST always carries `c_netsuiteInternalSalesOrderId` (`library/app/api/toga2.php` ~L4280), so `isset($api->httpPayload->c_netsuiteInternalSalesOrderId)` returns early.
165
- 2. The read-back order is re-checked for the same field.
164
+ 2. An order that already holds a NetSuite id now **raises an Issue in `doCreate`** (was a silent skip in `postPost`).
166
165
 
167
166
  Also non-firing by construction: the importer's line items go to record **313** (no interceptor row), and the importer's updates are **PUTs** while only **POST** is registered.
168
167
 
169
- **`NETSUITE_READ_DEPTH = 4` stays a class constant — rejected moving it to INI.** CodeRabbit flagged the constant as configuration belonging in `Config/<environment>.ini`. **Rejected** — it is a fact about the read shape: the PO number sits three hops out (`transferOrder` → `purchaseOrdersTransferOrders` → `purchaseOrder` → `number`), so 4 is the minimum depth that returns it — at 3 the value comes back **missing with no error**. Full reasoning (incl. why the suggested `_Config` call would have taken the write endpoint down): [_Config group access](../../_underscore/features/config-group-access.md).
170
-
171
168
  ### Metadata this push DEPENDS ON (`Core.InherentRecordChildren`)
172
169
 
173
- The depth-4 read must return **both** the lines and the PO number, so **two** inherent-child rows must exist for record **312** (`transfer-orders`):
170
+ The worker's read must return **both** the lines and the PO number, so **two** inherent-child rows must exist for record **312** (`transfer-orders`):
174
171
 
175
172
  | Parent | Child | Why |
176
173
  |---|---|---|
@@ -179,27 +176,41 @@ The depth-4 read must return **both** the lines and the PO number, so **two** in
179
176
 
180
177
  **325 is the header bridge (`_Model_Client_PurchaseOrders_TransferOrder`). 329 (`purchase-order-items-transfer-order-items`) is the ITEM-level bridge and is NOT what the PO number is read through** — an easy and expensive mix-up. Per-environment state, literals to copy, and the FK error a missing `Core.Records` row produces: [repairing non-prod metadata drift](../../dbchanges2/workflows/nonprod-metadata-drift-repair.md); the mechanism (what nests by default): [V2 REST query contract](../../api2/features/v2-rest-query-contract.md).
181
178
 
182
- ### Test harnesses — 68 tests, all passing, no database
179
+ ### Test harnesses — no database
183
180
 
184
181
  Both load the **real** shipped file with stubbed dependencies and drive private methods by **reflection**, same technique as `netsuite_salesorder_payload_tests.php` and [DB-free interceptor unit testing](../../_underscore/features/model-interceptor-unit-testing.md).
185
182
 
186
183
  - **`test/@Bala/tests/netsuite_transferorder_payload_tests.php` — 48 tests.** destination→customer / origin→warehouse mapping (**the test that would have caught the backwards design**), `rate` 0, PO number from the bridge and from the column and first-row-wins, single-object vs list for both bridge and lines, every abort path (no origin / no destination / no lines / no PO number / unresolvable item / origin with no warehouse id), `toga-to-` vs `toga-so-`, create / adopt / recover / dry run, write-back to `/transfer-orders/{uuid}`, that the **TA number is never sent to NetSuite**, and that a sales order still writes to `/sales-orders` and still takes NetSuite's number.
187
- - **`test/@Bala/tests/transferorder_interceptor_tests.php` — 20 tests.** correct action name and `transferOrderUuid` (not `salesOrderUuid`), one job per order in a **batch** POST, no queue when the request carried `c_netsuiteInternalSalesOrderId`, no queue when the order already holds an id, the POST never fails when the queue or read throws, the payload is not replaced, and the read is writer-pinned then restored (**including on a throw**) at depth 4 with `$throwExceptionsOnError = false`.
184
+ - **`test/@Bala/tests/transferorder_interceptor_tests.php` — STALE since 2026-10-05.** Still asserts the removed read-back (writer-pinned depth-4 read, order object in parameters, no queue when the order holds an id). Rewrite to assert: parameters are exactly `{clientIdentifier, transferOrderUuid}`, one job per order in a batch POST, no queue when the request carried `c_netsuiteInternalSalesOrderId`, queue failure → `_Error::captureException` and the POST still succeeds.
188
185
 
189
186
  Stub detail: `_Model_True` is stubbed with **only** the `FIELD_*` / `FIELDOPT_*` constants the model declares, because the model is never constructed.
190
187
 
191
188
  ### Open items
192
189
 
193
190
  - **No NetSuite sandbox — account 1095849 is live for every environment**, so a non-dry run from any tier writes production NetSuite data. (End-to-end run closed 2026-09-08 — SO 289187 / id 7463537.)
194
- - **The Create Transfer Order modal must (a) drop `"number"` from the POST and (b) send `transferOrderItems` NESTED in the same POST.** If it posts lines in separate calls afterwards, `postPost` reads **zero lines** and the push aborts — the trigger would then have to move. See [Transfer Orders page](../../toga25-supply/features/transfer-orders-page.md).
191
+ - **The Create Transfer Order modal must (a) drop `"number"` from the POST and (b) send `transferOrderItems` NESTED in the same POST.** If it posts lines in separate calls afterwards, the worker can read the header before the lines exist and abort on **zero lines** — the trigger would then have to move. See [Transfer Orders page](../../toga25-supply/features/transfer-orders-page.md).
195
192
  - **dev-sandbox metadata is incomplete** (record 325 + its RecordFields + both inherent-child rows).
196
193
  - **The -8 tax-group list is a snapshot (2026-09-14).** Re-run the check before onboarding any new client; **34283 Yale New Haven Health** and **2661 Endeavor Health** will fail on their first order.
197
- - **Delivery contact → `attention`: shipped 2026-09-25** (row above). In `Core.WorkerJobs.parameters` the FK arrives expanded as `transferOrder.contact` {firstName, lastName} — **no `contactId` key, `contact._name` never present**; no ACL/depth change was needed. toga25-supply's Create Transfer Order modal sets `TransferOrders.contactId` on every order it places. No length cap on attention (NetSuite ~99 chars) — same as other fields.
194
+ - **Delivery contact → `attention`: shipped 2026-09-25** (row above). In the worker's depth-6 read the FK arrives expanded as `transferOrder.contact` {firstName, lastName} — **no `contactId` key, `contact._name` never present**; no ACL/depth change was needed. toga25-supply's Create Transfer Order modal sets `TransferOrders.contactId` on every order it places. No length cap on attention (NetSuite ~99 chars) — same as other fields.
198
195
  - **Undecided: disable the customer name-search fallback for transfer orders?** `resolveNetSuiteCustomerId()` falls back to a SuiteQL search by company **name** when the destination has no customer id, and NYCHH's `Health & Hospitals` vs `NYC Health + Hospitals` are close enough to attach an order to the wrong NetSuite account.
199
196
  - **Decided and shipped 2026-09-08:** `description` goes out as `memo`, renders as "ORDER DESCRIPTION" on the NYCHH form. Still undecided: the TA number / `customerIdentificationNumber` into custom fields. **`contact` and `createdByUser` cannot be sent** — they need a NetSuite employee/contact id, which NYCHH `Contacts` do not carry.
200
197
 
201
198
  ## Gotchas
202
199
 
200
+ ### ⚠ The job carries only the uuid — never the order (SQS-cap incident, 2026-10-05)
201
+
202
+ **Symptom.** 8 `Client_Nychh.TransferOrders` (TA100000–TA100007, 2026-09-29 + 2026-10-05) had no `c_netsuiteInternalSalesOrderId` and **no `Core.WorkerJobs` row at all**; their POSTs took 5–27s (normal ~1s). Nothing in `Logs.Issue`.
203
+
204
+ **Cause.** `postPost` put the depth-6 order in the job. Depth 6 embeds the whole customer PO — every PO line, every sibling TO via `purchaseOrdersTransferOrders`, and each line's full Item (one was 111KB). Busy POs (EIT0011563: 55 lines / 174 TOs) pushed the message over `_Worker::SQS_MAX_MESSAGE_BYTES = 262144`; `runTask` threw; the catch only `error_log`ged → silent loss. Successful jobs already ran to 248,701 bytes, and it grows as POs are reused.
205
+
206
+ **Fix.** Job = `{clientIdentifier, transferOrderUuid}` only; worker reads with the bounded wait (see *The trigger*). A "slimmed snapshot" was tried and **rejected** by the developer. Team rule: [runTask parameters are ids only](./creating-worker-actions.md#-runtask-parameters-are-idsuuids-only--never-record-objects).
207
+
208
+ **Proving it:** compare `MAX(LENGTH(parameters))` in `Core.WorkerJobs` for this action against 262144. A missing `WorkerJobs` row + slow POSTs is the tell.
209
+
210
+ **Re-queue a missed order:** INSERT a `Core.WorkerJobs` row — `jobType` ACTION, `action` `Netsuite/SalesOrder/CreateNetSuiteFromTransferOrder`, `parameters` `{"clientIdentifier":"Nychh","transferOrderUuid":"<uuid>"}` (add `"dryRun":true` first), `dtQueued` NULL → JobScheduler picks it up. Safe to resend: the worker adopts by `externalId` `toga-to-<uuid>`. Backstop: [NYCHH push monitor](../../../../clients/nychh/features/oneuptime-transfer-order-netsuite-push-monitor.md).
211
+
212
+ **Accepted trade-offs:** local `debug_mode` (synchronous `runTask`) now fails the push after ~25s — row not committed yet; prod unaffected. A POST rolled back after queueing leaves a job that fails loud after ~25s.
213
+
203
214
  - **Stale test (2026-09-24):** `test/@Bala/tests/netsuite_transferorder_payload_tests.php` "PO number from the column" case now fails — the column is gone; rewrite it against the bridge.
204
215
 
205
216
  ### ⚠ The customer's NetSuite tax item must be AVATAX, or every create 400s
@@ -254,12 +265,14 @@ When the payload omits the ship-to address the create **succeeds**, so it looks
254
265
  ### Other traps
255
266
 
256
267
  - **The interceptor row is authored `isActive = 0` per the standard but shipped `isActive = 1`.** `_underscore` **must** be deployed before that migration runs, or every transfer-order POST 500s with *"Call to undefined method"* — the dispatch is unguarded (see [API payload interceptors](../../api2/features/api-payload-interceptors.md), and the Elite production incident for what that looks like live).
257
- - **`postPost` reads the order back at depth 4 — a shallower read silently loses the PO number**, and a lost PO number aborts the push. Do not "optimise" the depth.
268
+ - **The worker's transfer-order read must stay at depth 6** — shallower silently loses the PO number (aborts the push) and the ship-to address. Do not "optimise" the depth.
269
+ - **TA placeholder numbers get reused** after write-back replaces them (TA100001 was TO 1970, later TO 1975). Harmless — only unsent orders keep a TA number; never key on it.
258
270
  - **This action does not retry itself**, exactly like the sales-order push: a throw records `isSuccess = 0` and stops. An event-triggered push fires once.
259
271
  - **A NetSuite HTTP 400 naming a tax code is a CUSTOMER data problem, not a payload problem.** Check the customer's `taxitem` first; do not pin the tax fields in the payload (ignored) and do not drop the address to make the error go away (silently ships to the wrong site).
260
272
  - **`git diff --stat` after any programmatic full-file write to `_underscore`.** A full-file write with an editor default flipped CRLF → LF on `_underscore/Model/Nychh/TransferOrder.php` — content identical, git reported it modified with no content hunks. Restored with `git checkout --`. Standing rule: [2.0 backend PHP standards](../../standards/backend-php.md).
261
273
 
262
274
  ## Change history
275
+ - 2026-10-05 — Job carries only `{clientIdentifier, transferOrderUuid}`; worker reads the order with a bounded wait; queue failure → `_Error::captureException`. Fixes 8 NYCHH orders silently never queued (SQS 256KB cap). `NETSUITE_READ_DEPTH` + read-back removed; interceptor tests stale. (jcardinal)
263
276
  - 2026-10-02 — Push sends the TO's own `shipToAddress` when set (new `TransferOrders.shipToAddressId`, RecordField 2682); destination Location address is the fallback. (jcardinal)
264
277
  - 2026-09-24 — `resolvePurchaseOrderNumber()` bridge-only (column 2143 dropped); multiple POs joined with `' + '`. (jcardinal)
265
278
  - 2026-09-25 — delivery contact name sent as `shippingAddress.attention`. (jcardinal)
@@ -20,7 +20,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
20
20
  ## 2.0 framework
21
21
 
22
22
  - **_underscore** (_Underscore) _(framework core)_ — 94 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
23
- - **worker2** (Worker) — 83 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
23
+ - **worker2** (Worker) — 84 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
24
24
  - **api2** (API) — 26 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
25
25
  - **dbchanges2** (Database Changes) _(framework core)_ — 22 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
26
26
  - **toga2-supply** (TOGa Supply) — 9 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)
@@ -5,6 +5,7 @@
5
5
  | [NYCHH transfer-order destinations — location shipping addresses + delivery contacts](features/location-shipping-addresses.md) | 2.0 | NYCHH per-location shipping addresses + delivery contacts for transfer orders; open when a transfer's ship-to/target-location/contact looks wrong or shared. |
6
6
  | [NYCHH inventory-adjustment → item-fulfillment linking (many bridges, REST)](features/netsuite-inventory-adjustment-fulfillment-link.md) | 1.0 | NYCHH import of the NetSuite inventory-adjustment→item-fulfillment link (direct FK, REST); open when a NYCHH adjustment's FK link is missing or the INVENTORY_AD |
7
7
  | [NYCHH NetSuite → TransferOrders import (stocking-flag routing, PO bridges, origin location)](features/netsuite-transfer-order-import.md) | 1.0 | The 1.0 NetSuite→`Client_Nychh.TransferOrders` importer (stocking-flag routing, PO bridge links, origin/destination resolution, status map); open for any NYCHH |
8
+ | [NYCHH: transfer-order NetSuite push monitor (Monitor/Nychh/TransferOrderNetsuitePush)](features/oneuptime-transfer-order-netsuite-push-monitor.md) | 2.0 | worker2 OneUptime push monitor that alarms when a `Client_Nychh` transfer order has no NetSuite id 15+ minutes after creation; open when it fires, or when chang |
8
9
  | [NYCHH PO links are UPSTREAM — the downstream SO→PO route returns empty](features/po-number-upstream-direction.md) | 2.0 | Why NYCHH PO Number is empty on the downstream SO→PO route/`_purchaseOrders` field (links are upstream); open when a NYCHH SO shows a PO in the list but not the |
9
10
  | [NYCHH Talos AI Assistant Enablement (the only granted tenant)](features/talos-assistant-enablement.md) | 2.0 | How Talos AI assistant is enabled for NYCHH (the only granted tenant) via SurfaceOverrides; open when granting/changing NYCHH's chips or its assistant behaves w |
10
11
  | [NYCHH transfer-order 2.0 model — inventory quantities + V2 field enablement](features/transfer-order-inventory-quantities.md) | 2.0 | NYCHH 2.0 transfer-order model — two-branch inventory quantities + the model/ACL work that lets a TransferOrder POST; open when NYCHH `_qty*` reads 0 or a Trans |
@@ -0,0 +1,43 @@
1
+ ---
2
+ title: "NYCHH: transfer-order NetSuite push monitor (Monitor/Nychh/TransferOrderNetsuitePush)"
3
+ framework: "2.0"
4
+ repo: worker2
5
+ project: Worker
6
+ client: nychh
7
+ type: client-feature
8
+ status: active
9
+ updated: 2026-10-05
10
+ owners: ["jcardinal"]
11
+ files:
12
+ - worker2/Worker/Monitor/Nychh.php
13
+ - dbchanges2/Core/2026-10-05c - NYCHH Transfer Order NetSuite Push Monitor.sql
14
+ related:
15
+ - ../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md
16
+ - ../../../2.0/apps/worker2/features/netsuite-transferorder-outbound-push.md
17
+ - ./transfer-order-netsuite-push.md
18
+ - ../profile.md
19
+ ---
20
+ worker2 OneUptime push monitor that alarms when a `Client_Nychh` transfer order has no NetSuite id 15+ minutes after creation; open when it fires, or when changing the NYCHH transfer-order push.
21
+
22
+ ## Summary
23
+ `Monitor/Nychh/TransferOrderNetsuitePush` (`_Worker_Monitor_Nychh::TransferOrderNetsuitePush()`) is the backstop for the [transfer-order push](../../../2.0/apps/worker2/features/netsuite-transferorder-outbound-push.md), which fires once and never retries. Built after 8 orders were silently never queued (2026-10-05). Follows the shared [OneUptime push-metric pattern](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md).
24
+
25
+ - Live 2026-10-05. `Core.CronJobs` uuid `77e1af97-fa61-46f0-a2d8-c55db08b3e55`, `*/5 * * * *`, `maxExecutionTime` 120, `NOT EXISTS`-guarded; shipped `isActive = 0`, then activated.
26
+ - `initialize()` registers the client DB via `_Worker_Client_Nychh`.
27
+ - Push URL = in-method `$ONEUPTIME_URL` local (credential; never record the value).
28
+
29
+ ## How it works
30
+ - Query: `Client_Nychh.TransferOrders` with `c_netsuiteInternalSalesOrderId IS NULL` and `dtCreated` older than `$GRACE_MINUTES = 15` (push takes ~12s). Imported-from-NetSuite orders carry the id from the start, so they never match.
31
+ - Rows found → `"alarm":"DEGRADED"`, `pendingCount`, `transferOrderNumbers` (first 25, `MAX_LISTED_TRANSFER_ORDERS`), `oldestDtCreated`. None → `"alarm":"OK"`.
32
+ - Query failure → `"status":"error"`, `error:"db_query_failed"` (also degraded).
33
+ - Returns a JSON result on every path (recorded in `WorkerJobs.output`).
34
+
35
+ OneUptime: Incoming Request monitor, DEGRADED status id `8a160361-0b6a-4cbb-ad10-99ddd252637c`, operational `59b5ab80-63f6-4ffd-b3df-31c6ad127695`, offline after 20 min with no heartbeat. Import JSON is local only (`D:\STAGE\OneUptime Monitors\nychh-transfer-order-netsuite-push.json`), not committed.
36
+
37
+ ## Gotchas
38
+ - **When it fires:** the order was never queued, or its push failed. Check `Core.WorkerJobs` for the action + uuid; no row = never queued (look in `Logs.Issue` for the `captureException` from `_Model_Client_TransferOrder::postPost`). Re-queue recipe: [SQS-cap gotcha](../../../2.0/apps/worker2/features/netsuite-transferorder-outbound-push.md#-the-job-carries-only-the-uuid--never-the-order-sqs-cap-incident-2026-10-05).
39
+ - A listed number may be a `TA` placeholder; TA numbers are reused after write-back, so identify the order by uuid/id, not number.
40
+ - No exclusion for cancelled orders — a cancelled TO with no NetSuite id keeps it DEGRADED.
41
+
42
+ ## Change history
43
+ - 2026-10-05 — Created; monitor live. (jcardinal)
@@ -6,7 +6,7 @@ project: _Underscore
6
6
  client: nychh
7
7
  type: client-feature
8
8
  status: active
9
- updated: 2026-10-02
9
+ updated: 2026-10-05
10
10
  owners: ["bala", "jcardinal"]
11
11
  files:
12
12
  - _underscore/Model/Client/TransferOrder.php
@@ -23,6 +23,7 @@ related:
23
23
  - ./transfer-order-notification-emails.md
24
24
  - ./location-shipping-addresses.md
25
25
  - ./po-number-upstream-direction.md
26
+ - ./oneuptime-transfer-order-netsuite-push-monitor.md
26
27
  - ../profile.md
27
28
  ---
28
29
 
@@ -44,6 +45,13 @@ NetSuite SO **289187** (internal id 7463537) was created from this path
44
45
  and is correct. The location-role table below still stands, but **the destination location is NOT
45
46
  the NetSuite customer** — that was corrected the same day; see the next section.
46
47
 
48
+ **2026-10-05: 8 orders were never queued** (TO ids 1969, 1975, 1977, 1978, 1980, 1981, 2009, 2011 —
49
+ TA100000–TA100007, created 2026-09-29 + 2026-10-05). Busy customer POs (EIT0011563: 55 lines / 174 TOs;
50
+ EIT0012129: 71 TOs) made the job exceed the SQS 256KB cap. Fixed by sending only the uuid; all 8
51
+ re-queued by `Core.WorkerJobs` INSERT and pushed. Mechanics, proof query and re-queue recipe:
52
+ [the SQS-cap gotcha](../../../2.0/apps/worker2/features/netsuite-transferorder-outbound-push.md#-the-job-carries-only-the-uuid--never-the-order-sqs-cap-incident-2026-10-05).
53
+ Backstop alarm: [NYCHH transfer-order push monitor](./oneuptime-transfer-order-netsuite-push-monitor.md).
54
+
47
55
  ## ⚠ Location roles decide which `c_netsuite*` column a location carries
48
56
 
49
57
  For a NYCHH transfer order the **DESTINATION location is the NetSuite CUSTOMER** and the **ORIGIN
@@ -214,7 +222,7 @@ email, which must fire *after* this push writes `number` back. The `POST` row ab
214
222
  - Base (role 1) + API (role 3) ACL for 325/329 already complete (record CRUD, expressions 302/303, field
215
223
  write on uuid + both FKs). Duplicate Base `AclRecordPermissions` (325/331, 328/334) — harmless.
216
224
  - `TransferOrderStages` prod: 1 Pending, 2 In Transit, 3 Received, 4 Canceled, 5 Closed.
217
- - **`Core.InherentRecordChildren` for record 312** — the depth-4 read needs both the lines and the
225
+ - **`Core.InherentRecordChildren` for record 312** — the worker's read needs both the lines and the
218
226
  header PO bridge:
219
227
  - **prod** — was 312→313 and 312→**329** (the *item-level* bridge); repointed by the developer, now
220
228
  312→313 and 312→**325**. Correct.
@@ -225,6 +233,7 @@ email, which must fire *after* this push writes `number` back. The `POST` row ab
225
233
  It may also need record ACL rows for 325 — untested; an `EZ-1` would be the symptom.
226
234
 
227
235
  ## Change history
236
+ - 2026-10-05 — 8 TOs never queued (job over SQS 256KB cap on busy POs); job now uuid-only, all 8 re-queued + pushed; OneUptime monitor added. (jcardinal)
228
237
  - 2026-10-02 — Push uses the TO-owned `shipToAddress` when set; location-2 Jacobi-address note superseded. (jcardinal)
229
238
  - 2026-09-24 — PO # bridge-only (purchaseOrderId column dropped); 326/330 no longer dead (downstream vendor POs). (jcardinal)
230
239
 
@@ -18,7 +18,7 @@ project: _Underscore
18
18
  client: nychh
19
19
  type: profile
20
20
  status: active
21
- updated: 2026-10-02
21
+ updated: 2026-10-05
22
22
  owners: ["jcardinal", "apeterson", "bala", "akhokhani"]
23
23
  files:
24
24
  - dbchanges2/Client_Nychh/2026-09-02a - TransferOrderNetsuitePushInterceptor.sql
@@ -72,7 +72,7 @@ TOGA 2.0 client on `_underscore`, prod schema `Client_Nychh`. Supply-chain / ful
72
72
  - 🚩 **Two `Countries` rows share `c_netsuiteCountry = '_unitedStates'`** (id 1 US; id 2 stray `_t`, NY only, ~38k addresses). Froze ITEM_FULFILLMENTS 2026-10-02 (issue 935); hotfixed by `findStateUuidForNetsuiteAddress()` fallback. Stray `_t` cleanup OPEN.
73
73
  - **All 24 shipping locations shared ONE address until 2026-09-09 — the head-office Accounts Payable bill-to (50 Water St).** Each location had its own `Addresses` row (261-303) but they held identical values, and locations 1 + 2 shared `Addresses.id = 1`. `Client_Nychh/2026-09-09c - UpdateLocationShippingAddresses.sql` backfilled each from its NetSuite customer's `defaultshipping` address (applied + verified in prod: 22 distinct addresses over 24 locations). It only ever showed on the **transfer-order** screens, because `TransferOrders` has no ship-to column while `SalesOrders` carries its own `shipToAddressId`. Catch-all **location 2 "NYC Health + Hospitals"** now at 55 Water St 10041; its big TO count was mostly an import overwrite bug (fix 2026-09-24, see [NetSuite TO import](./features/netsuite-transfer-order-import.md)), and Bellevue/Jacobi carry NetSuite customer ids (29275 / 31917) that no longer exist. See [location shipping addresses](./features/location-shipping-addresses.md).
74
74
  - 🚨 **sandbox-client (`Core` + `Client_Nychh`) was reset from PRODUCTION again on 2026-09-04** — second reset in two days (first, 2026-09-03, over the stale `purchase-orders` TableView). Every "verified on <date>" sandbox-client claim in this profile dated **before 2026-09-04 now describes whatever prod had**. The 2026-09-01 → 2026-09-03 `dbchanges2` migrations may need re-running on sandbox-client. Re-read the environment before trusting any sandbox state; procedure + re-run order in [Targeting the right database](../../2.0/apps/dbchanges2/workflows/targeting-the-right-database.md).
75
- - **Transfer orders push OUT to NetSuite as $0 sales orders - LIVE and verified 2026-09-08.** NYCHH is the first and only client on that path; switched on purely by its `transfer-orders` (record 312) `POST`/`POST` interceptor row. NetSuite SO **289187** (id 7463537) was created from it. **The NetSuite customer is the CLIENT - 4490 / 28908 - on every transfer order, never the destination location** (a first cut sent the destination and put SO 7462031 on sub-customer 33674 "CHS"; `Locations` 1-26 are "Shipping" type mapping to sub-customers under 4490, and the bug hid because almost every order picks location 2, whose id *is* 28908). Origin location is still the NetSuite **warehouse**, and `TransferOrders.description` now goes out as the NetSuite `memo`, shown as **ORDER DESCRIPTION**. See [TransferOrder → NetSuite push](./features/transfer-order-netsuite-push.md).
75
+ - **Transfer orders push OUT to NetSuite as $0 sales orders - LIVE and verified 2026-09-08.** NYCHH is the first and only client on that path; switched on purely by its `transfer-orders` (record 312) `POST`/`POST` interceptor row. NetSuite SO **289187** (id 7463537) was created from it. **The NetSuite customer is the CLIENT - 4490 / 28908 - on every transfer order, never the destination location** (a first cut sent the destination and put SO 7462031 on sub-customer 33674 "CHS"; `Locations` 1-26 are "Shipping" type mapping to sub-customers under 4490, and the bug hid because almost every order picks location 2, whose id *is* 28908). Origin location is still the NetSuite **warehouse**, and `TransferOrders.description` now goes out as the NetSuite `memo`, shown as **ORDER DESCRIPTION**. See [TransferOrder → NetSuite push](./features/transfer-order-netsuite-push.md). Backstop alarm (any TO with no NetSuite id after 15 min): [push monitor](./features/oneuptime-transfer-order-netsuite-push-monitor.md).
76
76
  - **Transfer orders closed in NetSuite never reached the Closed stage - 18 stuck on `Pending`, oldest 2022 (fix written 2026-09-08, NOT deployed).** The 1.0 sync skips a `Closed` order before the transfer-order branch, so the importer's `case 'Closed'` was dead code. Two NYCHH gates make this confusing to verify: the transfer-order status refresh is nested inside `IS_ENABLED_INTEGRATION_SALES_ORDERS` (Jeff enabled the feeds 2026-08-28 `b063f05f` and disabled sales orders, POs, invoices and item receipts on 2026-09-03 `2b321e9d` - where every NYCHH cursor stops), and the seek cursor never re-reads a passed order. 🚨 **Their `NETSUITE_LAST_SYNC_CURSOR_ITEM_RECEIPTS` is stale at 2024-08-06**, so re-enabling that feed replays ~2 years. See [NetSuite → TransferOrders import](./features/netsuite-transfer-order-import.md).
77
77
  - **INVOICES mode stuck at `5-RUNNING` with no error (2026-09-24, worker fix NOT deployed).** Cursors were reset to 2026-09-24; INVOICES has a 24h hold-back and its skip path never wrote IDLE, so the window collapsed silently. After deploy set `NETSUITE_EXECUTION_MODE_INVOICES` to `864000-IDLE`; invoices modified before the reset need a rewind/re-import. See [per-client sync](../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md).
78
78
  - **Missing NetSuite records (item fulfillments + others) root-caused and fixed (2026-09-09).** Two separate bugs in the 1.0 NetSuite→TOGa Supply sync starved NYCHH imports, both deployed and verified in prod: (1) the resume cursor was **read on the wrong clock** (`strtotime()` in Chicago vs the Eastern-stored value), shifting the fetch window +1h and **silently skipping** any record in that 1-hour shadow — hit all 6 sections and all clients; and (2) the fulfillment `GET /sales-orders` ran at `depth=5` and **timed out at 900s**, freezing the section before the fulfillment row was written. Proven on IF `6138135` (Eastern 2025-09-04 17:26:58). Fixes + the standing "cursor is always Eastern" rule live in the [per-client sync feature doc](../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md).
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "toga-ai",
3
- "version": "1.0.949",
3
+ "version": "1.0.951",
4
4
  "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
5
5
  "keywords": [
6
6
  "claude",