toga-ai 1.0.696 → 1.0.698
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/1.0/apps/library/features/netsuite-suiteql-api-reference.md +38 -1
- package/knowledge/1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md +21 -0
- package/knowledge/2.0/apps/api2/architecture.md +29 -0
- package/knowledge/2.0/apps/api2/features/request-logging.md +16 -2
- package/knowledge/2.0/apps/api2/features/v2-rest-query-contract.md +31 -2
- package/knowledge/2.0/apps/api2/workflows/codepipeline-codeconnections-deploy.md +8 -0
- package/knowledge/2.0/apps/worker2/features/elastic-beanstalk-health-monitor.md +75 -10
- package/knowledge/clients/nychh/INDEX.md +1 -1
- package/knowledge/clients/nychh/features/netsuite-inventory-adjustment-fulfillment-link.md +156 -109
- package/package.json +1 -1
|
@@ -6,7 +6,7 @@ project: Library
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-08-
|
|
9
|
+
updated: 2026-08-29
|
|
10
10
|
owners: ["dfranks", "jcardinal", "kyalamarthi"]
|
|
11
11
|
files:
|
|
12
12
|
- library/app/api/netsuite/rest.php
|
|
@@ -16,6 +16,7 @@ related:
|
|
|
16
16
|
- netsuite-suiteql-rest-shim.md
|
|
17
17
|
- ../architecture.md
|
|
18
18
|
- ../../worker/features/forecast2-netsuite-reconciliation.md
|
|
19
|
+
- ../../../clients/nychh/features/netsuite-inventory-adjustment-fulfillment-link.md
|
|
19
20
|
---
|
|
20
21
|
|
|
21
22
|
## Summary
|
|
@@ -219,6 +220,22 @@ Other JE line facts (same probes):
|
|
|
219
220
|
> NetSuite custom field supplying those dimensions is created and populated (no `salesrep` column, no
|
|
220
221
|
> `item`). Re-evaluate only once that custom field exists — don't re-probe this from scratch.
|
|
221
222
|
|
|
223
|
+
### Item Fulfillment (`ItemShip`) lines — ~3 rows per fulfilled item, and `mainline` is NOT the filter
|
|
224
|
+
|
|
225
|
+
For an Item Fulfillment (`transaction.type = 'ItemShip'`), `transactionline` returns **~3 rows per
|
|
226
|
+
fulfilled item**: the order-linked line, a positive "wash" line, and an `isinventoryaffecting = 'T'`
|
|
227
|
+
line. **Only the order-linked line maps to a source-order line.** Keep exactly that one by requiring
|
|
228
|
+
a `previoustransactionlinelink` of type `SalesOrd`/`TrnfrOrd` (via an `EXISTS` correlated subquery),
|
|
229
|
+
and derive the source `orderLine` from that link's `previousline` (a correlated subquery — see the
|
|
230
|
+
`previousTransactionLineLink` section below).
|
|
231
|
+
|
|
232
|
+
- **⚠ `mainline = 'F'` is the WRONG filter for an ItemShip line.** NetSuite sometimes marks the
|
|
233
|
+
first real fulfilled line `mainline = 'T'`, so a `mainline='F'` predicate **silently drops it**.
|
|
234
|
+
Filter on the `SalesOrd`/`TrnfrOrd` line-link instead, not on `mainline`. (Same "extra posting
|
|
235
|
+
lines per item" shape as the credit-memo/cash-refund COGS lines in
|
|
236
|
+
`netsuite-suiteql-rest-shim.md`.) Surfaced building
|
|
237
|
+
[NYCHH inventory-adjustment → item-fulfillment linking](../../../clients/nychh/features/netsuite-inventory-adjustment-fulfillment-link.md).
|
|
238
|
+
|
|
222
239
|
## `previousTransactionLineLink` table
|
|
223
240
|
|
|
224
241
|
The bridge between a child line (e.g. invoice) and its originating parent line (e.g. SO).
|
|
@@ -322,6 +339,15 @@ Used for fields SuiteQL doesn't expose on `transaction`: `shippingCost`, `taxTot
|
|
|
322
339
|
`linesequencenumber` can diverge from REST `line` for SOAP-era records (confirmed empirically),
|
|
323
340
|
so keying on `linesequencenumber` risks duplicate Sales rows that can't be matched or deleted.
|
|
324
341
|
**`listSales()` keeps the per-id GET for this reason — do not eliminate it.**
|
|
342
|
+
- **⚠ REST record SUBLIST endpoints reject `limit`/`offset` (HTTP 400).**
|
|
343
|
+
`GET /record/v1/{type}/{id}/{sublist}` (e.g. `.../itemFulfillment/{id}/item?limit=1000&offset=N`)
|
|
344
|
+
returns **400 `INVALID_PARAMETER`** — *"Use one of: fields, expand, expandRecords,
|
|
345
|
+
expandSubResources, simpleEnumFormat, ids"*. `limit`/`offset` are valid only on **collection**
|
|
346
|
+
endpoints, not on a record's sublist. A shim that paginated a sublist this way silently got
|
|
347
|
+
**0 lines** and imported the record **header-only** with no error. Fetch line data via **SuiteQL
|
|
348
|
+
`transactionline`** instead (the pattern every other shim in `rest.php` already uses). Surfaced
|
|
349
|
+
when `fetchItemFulfillmentById` returned zero lines — see
|
|
350
|
+
[NYCHH inventory-adjustment → item-fulfillment linking](../../../clients/nychh/features/netsuite-inventory-adjustment-fulfillment-link.md).
|
|
325
351
|
|
|
326
352
|
## `list*()` helpers — behavior & cost
|
|
327
353
|
|
|
@@ -392,6 +418,17 @@ numbers. Budget hours for multi-year runs and launch them under `nohup`/`tmux`.
|
|
|
392
418
|
|
|
393
419
|
## Change history
|
|
394
420
|
|
|
421
|
+
- 2026-08-29 — Recorded two NetSuite REST/SuiteQL facts from the NYCHH inventory-adjustment →
|
|
422
|
+
item-fulfillment **REST migration**: **REST record sublist endpoints
|
|
423
|
+
(`/record/v1/{type}/{id}/{sublist}`) reject `limit`/`offset` with a 400 `INVALID_PARAMETER`** (a
|
|
424
|
+
paginated sublist fetch silently returns 0 lines → header-only import; use SuiteQL
|
|
425
|
+
`transactionline`), and an **Item Fulfillment (`ItemShip`) `transactionline` returns ~3 rows per
|
|
426
|
+
fulfilled item** (order-linked + positive "wash" + `isinventoryaffecting='T'`) — keep only the line
|
|
427
|
+
with a `previoustransactionlinelink` of type `SalesOrd`/`TrnfrOrd` and derive `orderLine` from its
|
|
428
|
+
`previousline`; **`mainline='F'` is the wrong filter** because NetSuite sometimes marks the first
|
|
429
|
+
real line `mainline='T'`. See
|
|
430
|
+
[NYCHH inventory-adjustment → item-fulfillment linking](../../../clients/nychh/features/netsuite-inventory-adjustment-fulfillment-link.md).
|
|
431
|
+
(jcardinal)
|
|
395
432
|
- 2026-08-19 — Added two SuiteQL mechanics that fail **silently**, both found porting the customer
|
|
396
433
|
account-type sync (TRUE-80206): **column aliases come back lower-cased** (`AS accountTypeLabel` →
|
|
397
434
|
`accounttypelabel`, so a camelCase read is `NULL` with no error — it made a backfill report success while
|
|
@@ -520,8 +520,29 @@ Parameters are stored **per client DB** but accessed **through the TOGa2 API**,
|
|
|
520
520
|
item-fulfillment lookup set through for the on-demand fulfillment import (lookups are passed
|
|
521
521
|
function-to-function, like the other sync functions).
|
|
522
522
|
|
|
523
|
+
- **⚠ Inventory-adjustment sync migrated SOAP → REST + now fails LOUD (2026-08-29).** The SOAP path
|
|
524
|
+
flagged just above (`App_NetSuite::getInventoryNumber*`, ~2 round-trips per serial) has been
|
|
525
|
+
**migrated to REST/SuiteQL** via `App_Api_Netsuite_Rest`, and the **on-demand fulfillment /
|
|
526
|
+
sales-order import** it triggers was converted to REST too — closing the ~1h-per-large-adjustment
|
|
527
|
+
cost that was the previously-planned next step. Also, the remaining **error-log-and-swallow
|
|
528
|
+
`try/catch` blocks in the NetSuite→DB sync path were removed** so a data-corruption failure surfaces
|
|
529
|
+
loudly (StarTech/Sentry catches left alone) — the same loud-failure policy as the 2026-08-27
|
|
530
|
+
per-order `try/catch` removal. Full detail (the REST-shim fixes it required, and the bridges →
|
|
531
|
+
direct-FK-column change) on
|
|
532
|
+
[NYCHH inventory-adjustment → item-fulfillment linking](../../../clients/nychh/features/netsuite-inventory-adjustment-fulfillment-link.md).
|
|
533
|
+
|
|
523
534
|
## Change history
|
|
524
535
|
|
|
536
|
+
- 2026-08-29 — The inventory-adjustment sync (and the on-demand fulfillment / sales-order import it
|
|
537
|
+
triggers) in `syncInventoryAdjustmentFromNetsuite` was **migrated SOAP → REST/SuiteQL**, closing the
|
|
538
|
+
SOAP round-trip cost flagged 2026-08-28; the remaining error-log-and-swallow `try/catch` in the
|
|
539
|
+
NetSuite→DB sync path were removed for loud failure (StarTech/Sentry catches untouched). The
|
|
540
|
+
migration required two `App_Api_Netsuite_Rest` shim fixes (a record-sublist `limit`/`offset` 400 and
|
|
541
|
+
the ItemShip 3-rows-per-item line filter) and moved the NYCHH adjustment↔fulfillment link off bridge
|
|
542
|
+
tables onto direct FK columns — see
|
|
543
|
+
[NYCHH inventory-adjustment → item-fulfillment linking](../../../clients/nychh/features/netsuite-inventory-adjustment-fulfillment-link.md).
|
|
544
|
+
(jcardinal)
|
|
545
|
+
|
|
525
546
|
- 2026-08-28 — Recorded two shared-engine inventory-adjustment fixes in
|
|
526
547
|
`syncInventoryAdjustmentFromNetsuite` (prune child `InventoryAdjustmentItemUnits` before the item —
|
|
527
548
|
FK RESTRICT/1451/EV-11; index the item lookup **one-level** `[$partNumber]` to match what the cron
|
|
@@ -302,6 +302,27 @@ the deploy/version timeline before hunting app bugs. **Fixed 2026-08-28:** all t
|
|
|
302
302
|
to **Rolling / batch-size 1** (≥1 healthy backend throughout a deploy), which eliminates the blip —
|
|
303
303
|
config + tradeoffs in the [deploy workflow](workflows/codepipeline-codeconnections-deploy.md).
|
|
304
304
|
|
|
305
|
+
**A fifth cause — a brief, self-healing "no-cause TargetGroup reduced health" flip on a
|
|
306
|
+
LOW-TRAFFIC, IDLE 2-instance env is EB Enhanced Health FLAPPINESS, not worker starvation.**
|
|
307
|
+
Distinct from the third cause (that one is the *busy* env, `api-production-1`, under Compass ASN
|
|
308
|
+
worker saturation). This is a single missed ALB `/health` check — a network/agent transient.
|
|
309
|
+
Fingerprint (verified `api-production-3`, eu-west-1, 2026-08-29 13:31 UTC `Severe`, static-`/health`
|
|
310
|
+
build deployed): self-heals in ~2–5 min; throughout, `HealthyHostCount` stays 2 (1-min minimum),
|
|
311
|
+
**0** target/ELB 5xx, sub-second latency, idle CPU, near-zero traffic; ALB target health already
|
|
312
|
+
reads "healthy" by the time you check. On an idle env there is no worker starvation, so **no
|
|
313
|
+
server-side app fix applies** — do not chase it. The mitigation is alert de-sensitization: the
|
|
314
|
+
worker2 EB health monitor now applies a **persistence-delay gate** (a non-5xx `Degraded`/`Severe`
|
|
315
|
+
must persist ≥ `nonFivexxPersistMinutes`, default 5, before it pages), which suppresses these flaps.
|
|
316
|
+
See [worker2 EB health monitor](../worker2/features/elastic-beanstalk-health-monitor.md).
|
|
317
|
+
|
|
318
|
+
**Triage aid — break an EB "% requests failing with HTTP 5xx" flag into ELB 502/503/504 to find
|
|
319
|
+
the cause:** **503** = no healthy backend (a deploy with no capacity); **504** = target timeout
|
|
320
|
+
(worker saturation); **502** = connection dropped mid-response (an `AllAtOnce` deploy killing
|
|
321
|
+
in-flight requests, or a PHP worker crash). Because **ELB-generated 5xx (any of these) never reach
|
|
322
|
+
PHP**, they leave **no row in `Logs.Api`** (api2 logs only *after* a completed response) — so an EB
|
|
323
|
+
"% 5xx" flag with an **empty `Logs.Api`** means an **ELB-level** 5xx, not an app 5xx. Discriminate
|
|
324
|
+
with CloudWatch `HTTPCode_ELB_502/503/504_Count` vs `HTTPCode_Target_5XX_Count`.
|
|
325
|
+
|
|
305
326
|
## Known issues / accepted risks
|
|
306
327
|
|
|
307
328
|
Open items a maintainer should know before changing this tier. None are "bugs to fix right now" —
|
|
@@ -379,6 +400,14 @@ they are the known sharp edges. Do not re-discover these from scratch.
|
|
|
379
400
|
single-caller branches in `V2.php` as unverified until exercised directly.
|
|
380
401
|
|
|
381
402
|
## Change history
|
|
403
|
+
- 2026-08-29 — Recorded a **fifth EB-`Degraded` cause**: a brief self-healing "no-cause reduced
|
|
404
|
+
health" flip on a low-traffic *idle* 2-instance env (e.g. `api-production-3`, eu-west-1) is EB
|
|
405
|
+
Enhanced Health flappiness (a single missed ALB `/health` check), distinct from the busy-env
|
|
406
|
+
PHP-`/health`-starvation cause — no server-side fix applies; mitigated by the worker2 monitor's
|
|
407
|
+
new non-5xx persistence-delay gate. Added an ELB **502/503/504 discrimination** triage aid
|
|
408
|
+
(503=no healthy backend, 504=target timeout/worker saturation, 502=dropped connection from an
|
|
409
|
+
AllAtOnce deploy or PHP crash); ELB-generated 5xx never reach PHP so they leave no `Logs.Api`
|
|
410
|
+
row — an EB "% 5xx" flag with an empty `Logs.Api` is ELB-level, not app 5xx. (jcardinal)
|
|
382
411
|
- 2026-08-28 — Recorded a **fourth EB-`Degraded` cause — the deploy-restart ELB 502**: under the old
|
|
383
412
|
`AllAtOnce`/100% policy a deploy restarts both instances of a 2-instance env at once, dropping
|
|
384
413
|
in-flight connections → an ELB-502 burst (43 in one minute on `api-production-1`, 2026-08-28) that
|
|
@@ -6,8 +6,8 @@ project: API
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-08-
|
|
10
|
-
owners: ["mhammontree", "dfranks", "bala"]
|
|
9
|
+
updated: 2026-08-29
|
|
10
|
+
owners: ["mhammontree", "dfranks", "bala", "jcardinal"]
|
|
11
11
|
files:
|
|
12
12
|
- api2/Component/Api/V2/V2.php
|
|
13
13
|
- api2/Controller/Index.php
|
|
@@ -175,6 +175,14 @@ There is no usable index on those columns. Do not start from a payload search.
|
|
|
175
175
|
|
|
176
176
|
The working pattern, in order:
|
|
177
177
|
|
|
178
|
+
> **`route` values are stored WITH the `/v2` prefix** (e.g. `/v2/inventory-adjustments/{uuid}`,
|
|
179
|
+
> `/v2/item-fulfillment-items`), so any narrowing predicate must filter `route LIKE '/v2/...'` — a
|
|
180
|
+
> bare `/inventory-adjustments` matches **nothing**. `route` also carries `requestPayload` +
|
|
181
|
+
> `responsePayload` + `responseCode` per row, so once narrowed the log gives the full request/response
|
|
182
|
+
> for a call **without NetSuite access** — it was the decisive tool for confirming a V2 PUT persisted
|
|
183
|
+
> a field its response echo omitted (see the [V2 query contract](v2-rest-query-contract.md) PUT-echo
|
|
184
|
+
> gotcha).
|
|
185
|
+
|
|
178
186
|
1. **Narrow with indexed / non-text columns only** — a `dtStamp` range plus `route`,
|
|
179
187
|
`method`, `responseCode`, `hostname`. A
|
|
180
188
|
`GROUP BY hostname, route, method, responseCode` over one day is cheap and is the fastest way
|
|
@@ -217,6 +225,12 @@ re-send.
|
|
|
217
225
|
|
|
218
226
|
## Change history
|
|
219
227
|
|
|
228
|
+
- 2026-08-29 — Noted that `Logs.Api.route` values are stored **with the `/v2` prefix** (e.g.
|
|
229
|
+
`/v2/inventory-adjustments/{uuid}`), so narrowing queries must filter `route LIKE '/v2/...'` — a
|
|
230
|
+
bare `/inventory-adjustments` predicate matches nothing. Reinforced that a narrowed row's
|
|
231
|
+
`requestPayload`/`responsePayload` is the decisive, NetSuite-access-free tool for diagnosing V2
|
|
232
|
+
write behavior — here, confirming a V2 PUT persisted a field its response echo omitted (NYCHH
|
|
233
|
+
inventory-adjustment → item-fulfillment linking). (jcardinal)
|
|
220
234
|
- 2026-08-13 — **Important limit on the "an HTTP 400 leaves a full log row" rule** (TRUE-80412):
|
|
221
235
|
`execute()` writes the transaction log near its **end** (~L2225-2271), *after* route processing
|
|
222
236
|
(~L1742-2043), so **a throw from a PRE interceptor produces no `Logs.Api` row at all** — the
|
|
@@ -6,8 +6,8 @@ project: API
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-08-
|
|
10
|
-
owners: [tcox, bala, apeterson]
|
|
9
|
+
updated: 2026-08-29
|
|
10
|
+
owners: [tcox, bala, apeterson, jcardinal]
|
|
11
11
|
files:
|
|
12
12
|
- api2/Component/Api/V2/V2.php
|
|
13
13
|
related:
|
|
@@ -108,6 +108,19 @@ calculated field is null on a nested object but correct when that object is fetc
|
|
|
108
108
|
Definition of the fields themselves:
|
|
109
109
|
[FIELD_SQL calculated fields](../../_underscore/features/calculated-sql-fields.md).
|
|
110
110
|
|
|
111
|
+
### ⚠ A non-null FK object at the MAX requested `depth` is OMITTED from the response
|
|
112
|
+
|
|
113
|
+
A GET serialized to `depth: N` renders FK objects down to level N, but a **non-null FK object that
|
|
114
|
+
sits at level N+1** — one hop *past* the max — is **dropped from the payload with no error**. It is
|
|
115
|
+
easy to miss because a **null** sibling FK still renders as `null`, so the omission reads like "that
|
|
116
|
+
FK is null" rather than "that FK was not serialized." Observed on `GET /item-fulfillment-items` at
|
|
117
|
+
`depth: 2`: the grandchild `item` (level 3 — `itemFulfillmentItem → salesOrderItem → item`) was
|
|
118
|
+
absent, so a product-match keyed on `salesOrderItem.item.uuid` came back empty and no link was
|
|
119
|
+
written. **Fix: request `depth: N+1`, or name the nested path explicitly in `fields`** (e.g.
|
|
120
|
+
`salesOrderItem.item.uuid` / `transferOrderItem.item.uuid`). Rule of thumb: the deepest FK you
|
|
121
|
+
actually read must sit at least one level *inside* `depth`, not on the boundary. Surfaced building
|
|
122
|
+
[NYCHH inventory-adjustment → item-fulfillment linking](../../../../clients/nychh/features/netsuite-inventory-adjustment-fulfillment-link.md).
|
|
123
|
+
|
|
111
124
|
### What nests by default is `InherentRecordChildren` — everything else needs the nested ROUTE
|
|
112
125
|
|
|
113
126
|
A record's one-to-many children arrive inside the record response **only** if `Core.
|
|
@@ -271,8 +284,24 @@ where=(field:op:value,LOGIC,field:op:value,(nested,OR,nested))
|
|
|
271
284
|
- **`gte`/`lte` silently are not operators** — use `ge`/`le`.
|
|
272
285
|
- **Do not trust a 200 with missing fields.** Without an explicit `fields=` list, ACL removal is
|
|
273
286
|
invisible; re-request with `fields=` to force `EZ-2` and see the denied list.
|
|
287
|
+
- **⚠ A PUT response ECHO can omit a field that DID persist — verify against the DB, not the echo.**
|
|
288
|
+
A `PUT /v2/inventory-adjustments/{uuid}` with `{"itemFulfillment":{"uuid":…}}` returned **200 with
|
|
289
|
+
an empty `messages[]`** and an echo that did **not** contain `itemFulfillment`, yet the FK column
|
|
290
|
+
**persisted correctly**. An absent field in the write-response echo is **not** proof the write
|
|
291
|
+
failed — confirm the field against the record's DB row (or the `Logs_<Client>.Api` `responsePayload`),
|
|
292
|
+
never from the write's own echoed body. See
|
|
293
|
+
[V2 request logging](request-logging.md).
|
|
274
294
|
|
|
275
295
|
## Change history
|
|
296
|
+
- 2026-08-29 — Recorded two response-shape facts from the NYCHH inventory-adjustment REST work: **a
|
|
297
|
+
non-null FK object one hop past the max `depth` is silently omitted** (a null sibling FK still
|
|
298
|
+
renders, so it reads as "null" not "missing") — `GET /item-fulfillment-items` at `depth:2` dropped
|
|
299
|
+
the level-3 `salesOrderItem.item`; fix with `depth+1` or an explicit nested `fields` path — and a
|
|
300
|
+
**PUT response echo can omit a field that persisted** (a `PUT /v2/inventory-adjustments/{uuid}`
|
|
301
|
+
returned 200 / empty-`messages` without echoing `itemFulfillment` though the FK saved), so verify a
|
|
302
|
+
write against the DB / `Logs_<Client>.Api`, not the echo. See
|
|
303
|
+
[NYCHH inventory-adjustment → item-fulfillment linking](../../../../clients/nychh/features/netsuite-inventory-adjustment-fulfillment-link.md).
|
|
304
|
+
(jcardinal)
|
|
276
305
|
- 2026-08-28 — Documented two response-shape facts hit while building the Transfer Orders modal.
|
|
277
306
|
**`calcDepth` defaults to 1**, so a `FIELD_SQL` calculated field on a record one FK hop out returns
|
|
278
307
|
`null` with **no** message (Location's `_address*` fields needed `calcDepth: 2`) — raise it one per
|
|
@@ -152,6 +152,10 @@ one 15 s interval (~45 s → 30 s) of per-batch health settle.
|
|
|
152
152
|
`BatchSize`); `aws:elasticbeanstalk:environment:process:default` (`HealthyThresholdCount`).
|
|
153
153
|
- Trade-off: Rolling is slower end-to-end than the old ~31 s `AllAtOnce`, and capacity briefly
|
|
154
154
|
drops to 1/2 instances per batch (fine at current traffic).
|
|
155
|
+
- **Confirmed working in production (2026-08-29).** The 2026-08-29 00:49 UTC deploy ran as
|
|
156
|
+
Rolling batches ("Batch 1 / Batch 2, one instance at a time, register and wait for healthy")
|
|
157
|
+
across all three API prod envs with **no post-deploy 5xx flip** — the `AllAtOnce`
|
|
158
|
+
deploy-restart 502 blip is eliminated in practice, not just in config.
|
|
155
159
|
|
|
156
160
|
### Deploy-policy guidance (don't reach for immutable expecting speed)
|
|
157
161
|
- **Immutable does NOT reduce deploy time.** It is the **slowest** policy — it boots a fresh
|
|
@@ -295,6 +299,10 @@ aws codeconnections get-connection --connection-arn "<CONN_ARN>" --region "$REGI
|
|
|
295
299
|
```
|
|
296
300
|
|
|
297
301
|
## Change history
|
|
302
|
+
- 2026-08-29 — Confirmed the Rolling/batch-1 policy works in production: the 2026-08-29
|
|
303
|
+
00:49 UTC deploy ran as Rolling batches (one instance at a time, register + wait for healthy)
|
|
304
|
+
on all three API prod envs with **no post-deploy 5xx flip** — the AllAtOnce deploy-restart
|
|
305
|
+
502 blip is eliminated. (jcardinal)
|
|
298
306
|
- 2026-08-28 — Recorded the **production deploy trigger + policy change**: the `api-production-1/2/3`
|
|
299
307
|
CodePipelines **auto-deploy on every push to `_production`** (WebhookV2 on the CodeConnections
|
|
300
308
|
connection, repos `_underscore` + `api2`), so a merge to `_production` **is** a three-region prod
|
|
@@ -6,7 +6,7 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-08-
|
|
9
|
+
updated: 2026-08-29
|
|
10
10
|
owners: [jcardinal]
|
|
11
11
|
files:
|
|
12
12
|
- worker2/Worker/Infrastructure/CloudWatch.php
|
|
@@ -28,10 +28,13 @@ OneUptime string-matches — OneUptime cannot compare numbers on a pushed body.
|
|
|
28
28
|
|
|
29
29
|
The load-bearing design decision this doc records is the **paging (alarm) criteria**, and
|
|
30
30
|
the key **2026-08-25** change: the 5xx page is now driven by a **CloudWatch windowed
|
|
31
|
-
request-rate gate**, fully **decoupled from EB's own `HealthStatus`**.
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
31
|
+
request-rate gate**, fully **decoupled from EB's own `HealthStatus`**. Whether a *5xx*
|
|
32
|
+
problem pages is decided by the true per-environment 5xx **rate over a trailing window**
|
|
33
|
+
(default 5 min), not by EB's ~10-second `ApplicationMetrics` snapshot. As of **2026-08-29**,
|
|
34
|
+
a non-5xx `Degraded`/`Severe` no longer pages immediately either — it must **persist** past a
|
|
35
|
+
threshold (`nonFivexxPersistMinutes`, default 5) before paging, which suppresses the brief
|
|
36
|
+
self-healing "reduced health" flips that EB Enhanced Health throws on low-traffic 2-instance
|
|
37
|
+
envs (see the persistence gate below).
|
|
35
38
|
|
|
36
39
|
**Why the change:** the old gate read EB `describeEnvironmentHealth`'s `ApplicationMetrics`,
|
|
37
40
|
a ~10-**second** window. At low traffic a single 500 in a near-empty bucket reads as 100%
|
|
@@ -52,6 +55,10 @@ itself runs non-fatal. CloudWatch read failures are caught **internally** and re
|
|
|
52
55
|
`_Worker::runTask('Infrastructure/CloudWatch/ElasticBeanstalkHealth', {awsAccountId,
|
|
53
56
|
environments, oneuptimeUrl, …thresholds})`.
|
|
54
57
|
- New private `fetchWindowedRequestCounts()` reads the CloudWatch metrics (below).
|
|
58
|
+
- New private `fetchNonOkDepartureAgeSeconds(ElasticBeanstalkClient $eb, string
|
|
59
|
+
$environmentName, int $lookbackSeconds): ?int` returns how long the current non-Ok episode
|
|
60
|
+
has persisted, from EB `describeEvents` (below). **Never throws** — catches `AwsException`
|
|
61
|
+
and returns `null`.
|
|
55
62
|
- AWS access is obtained through `_Component_Aws_Workloads::assumeCredentials()` (STS-assume
|
|
56
63
|
a read-only role per account, one assume reused across regions) — see
|
|
57
64
|
[Cross-account AWS access](./cross-account-aws-access.md), which also documents the
|
|
@@ -64,11 +71,14 @@ itself runs non-fatal. CloudWatch read failures are caught **internally** and re
|
|
|
64
71
|
|
|
65
72
|
Two independent paths now feed `alarm=HIGH`:
|
|
66
73
|
|
|
67
|
-
1. **EB-health path (non-5xx)
|
|
68
|
-
attributable solely to 5xx (latency, instances
|
|
69
|
-
no stated cause)
|
|
70
|
-
|
|
71
|
-
|
|
74
|
+
1. **EB-health path (non-5xx) — now gated on PERSISTENCE (2026-08-29).** EB `HealthStatus`
|
|
75
|
+
`Degraded` or `Severe` that is **not** attributable solely to 5xx (latency, instances
|
|
76
|
+
down, a failed deploy, a mixed cause, or no stated cause) pages **only once the current
|
|
77
|
+
non-Ok episode has persisted ≥ `nonFivexxPersistMinutes`** (new optional param, default 5)
|
|
78
|
+
— it no longer pages on the first snapshot. A sub-threshold blip is **reported but does not
|
|
79
|
+
page**. `Suspended` never pages. `causesAttributeSolelyTo5xx()` is what tells a solely-5xx
|
|
80
|
+
event apart from everything else. See the persistence-gate section below for how episode
|
|
81
|
+
age is measured and the fail-safe.
|
|
72
82
|
2. **CloudWatch windowed 5xx-rate path.** For a solely-5xx `Degraded`/`Severe`, the page is
|
|
73
83
|
decided **not** by EB but by the true windowed rate from CloudWatch. Given the window's
|
|
74
84
|
`requests` (ApplicationRequestsTotal) and `fivexx` (ApplicationRequests5xx), page when:
|
|
@@ -91,6 +101,7 @@ The four thresholds are **optional cron parameters** spread as **named arguments
|
|
|
91
101
|
| `minRequests` | `100` | Minimum windowed request count before the **share** gate applies |
|
|
92
102
|
| `min5xxPercent` | `20.0` | 5xx share (percent) at/above which a solely-5xx env pages, once `minRequests` is met |
|
|
93
103
|
| `min5xxAbsolute` | `20` | Absolute windowed 5xx count at/above which a solely-5xx env pages regardless of share |
|
|
104
|
+
| `nonFivexxPersistMinutes` | `5` | Minutes a **non-5xx** `Degraded`/`Severe` episode must persist before it pages (added 2026-08-29; a trailing optional param — existing `CronJobs.parameters` without the key still run) |
|
|
94
105
|
|
|
95
106
|
Removed this iteration: `shouldPageForHealth()` and the old
|
|
96
107
|
`HTTP_5XX_ALARM_RATIO` / `HTTP_5XX_ALARM_ABSOLUTE_COUNT` constants (superseded by the
|
|
@@ -123,6 +134,43 @@ A CloudWatch read *failure* (as opposed to empty data) sets **`probe=DEGRADED`**
|
|
|
123
134
|
gap), distinct from a measured-bad `alarm=HIGH` — same alarm-vs-probe split as the
|
|
124
135
|
multi-client monitors in [OneUptime push-metric monitors](./oneuptime-worker2-monitoring.md).
|
|
125
136
|
|
|
137
|
+
## Persistence gate on the non-5xx page (2026-08-29)
|
|
138
|
+
|
|
139
|
+
A non-5xx `Degraded`/`Severe` now pages only after the **current non-Ok episode has persisted
|
|
140
|
+
≥ `nonFivexxPersistMinutes`** (default 5). This exists because the brief "no-cause TargetGroup
|
|
141
|
+
reduced health" flips on low-traffic 2-instance envs are **EB Enhanced Health flappiness** —
|
|
142
|
+
a single missed ALB `/health` check (a network/agent transient), not a real app problem — and
|
|
143
|
+
they self-heal in ~2–5 min. A persistence delay suppresses those while a genuinely sustained
|
|
144
|
+
non-5xx degradation still pages, delayed only by the threshold. (The windowed 5xx-flood gate
|
|
145
|
+
is **unchanged** — it has its own, independent decision.)
|
|
146
|
+
|
|
147
|
+
**Fingerprint of the idle-env flap** (verified `api-production-3`, eu-west-1, 2026-08-29
|
|
148
|
+
13:31 UTC `Severe`, static-`/health` build deployed): self-heals in ~2–5 min; throughout,
|
|
149
|
+
`HealthyHostCount` stays 2 (1-min minimum), **0** target/ELB 5xx, sub-second latency, idle
|
|
150
|
+
CPU, near-zero traffic; ALB target health already reads "healthy" by the time you look. On an
|
|
151
|
+
idle env there is **no worker starvation**, so there is nothing to fix server-side — the
|
|
152
|
+
mitigation is this monitor persistence-delay gate, not more app code. **This is DISTINCT from
|
|
153
|
+
the PHP-`/health`-starvation cause** (that one is the *busy* env, `api-production-1`, under
|
|
154
|
+
ASN worker saturation, fixed by the static-`/health` file) — see
|
|
155
|
+
[api2 architecture — EB "Degraded" health](../../api2/architecture.md).
|
|
156
|
+
|
|
157
|
+
**Measuring episode age — `fetchNonOkDepartureAgeSeconds()`** reads EB `describeEvents`:
|
|
158
|
+
|
|
159
|
+
- The most recent **"transitioned from Ok to …"** event marks the current episode's start.
|
|
160
|
+
Once non-Ok, later transitions stay between non-Ok states until a to-Ok event; because
|
|
161
|
+
`describeEvents` returns **newest-first**, the first "from Ok to …" match wins.
|
|
162
|
+
- The query is **TIME-bounded**: `StartTime = now − (persistWindow +
|
|
163
|
+
HEALTH_EVENTS_LOOKBACK_BUFFER_SECONDS = 600)`, with a `MaxRecords = 100` cap — so a chatty
|
|
164
|
+
env's deploy/config events cannot push the departure event out of range.
|
|
165
|
+
- **Fail-safe:** an episode **older than the window** returns `null` → the caller **pages**.
|
|
166
|
+
That is correct: if the departure is beyond the lookback, the episode has already persisted
|
|
167
|
+
past the threshold, so paging is the safe outcome. A read failure also returns `null`
|
|
168
|
+
(the method catches `AwsException` and never throws) → same fail-safe.
|
|
169
|
+
|
|
170
|
+
**Auditable suppression:** a sub-threshold blip is reported, not paged, and the suppression is
|
|
171
|
+
made visible — `pendingPersistSeconds` is added to the OneUptime payload and a `pending=Ns`
|
|
172
|
+
annotation to the WorkerJobs summary — so a suppressed episode is never silent.
|
|
173
|
+
|
|
126
174
|
## Reading the windowed 5xx rate from CloudWatch (durable AWS reference)
|
|
127
175
|
|
|
128
176
|
- **EB application-request metrics are PER-INSTANCE ONLY.** `ApplicationRequestsTotal`,
|
|
@@ -214,6 +262,8 @@ the `probe` token; this is a OneUptime config change the developer applies.
|
|
|
214
262
|
`min5xxPercent`; absolute at `min5xxAbsolute-1` vs `min5xxAbsolute`; `requests` just below
|
|
215
263
|
`minRequests`; empty/zero window; mixed 5xx + non-5xx cause) — deferred pending a harness.
|
|
216
264
|
- **Deploy fails without degrading health** — still not caught (accepted residual).
|
|
265
|
+
- **`ElasticBeanstalkHealth()` is long** and grew with the persistence gate — php-reviewer
|
|
266
|
+
(0 critical) flagged it as a decomposition candidate; deferred pending the missing harness.
|
|
217
267
|
|
|
218
268
|
## Gotchas / known issues
|
|
219
269
|
|
|
@@ -235,6 +285,21 @@ the `probe` token; this is a OneUptime config change the developer applies.
|
|
|
235
285
|
record its value in a doc.
|
|
236
286
|
|
|
237
287
|
## Change history
|
|
288
|
+
- 2026-08-29 — Added a **persistence gate on the non-5xx page**: a non-5xx `Degraded`/`Severe`
|
|
289
|
+
no longer pages on the first snapshot — it must persist ≥ `nonFivexxPersistMinutes` (new
|
|
290
|
+
optional trailing param, default 5; existing `CronJobs.parameters` without the key still
|
|
291
|
+
run) before paging. Episode age comes from EB `describeEvents` via new never-throwing helper
|
|
292
|
+
`fetchNonOkDepartureAgeSeconds()` (most recent "from Ok to …" marks episode start; events
|
|
293
|
+
are newest-first so the first match wins; TIME-bounded `StartTime = now − (persistWindow +
|
|
294
|
+
HEALTH_EVENTS_LOOKBACK_BUFFER_SECONDS=600)`, `MaxRecords=100`, so an episode older than the
|
|
295
|
+
window returns `null` → the caller pages — correct fail-safe). Sub-threshold blips are
|
|
296
|
+
reported not paged, with `pendingPersistSeconds` (OneUptime payload) + `pending=Ns`
|
|
297
|
+
(WorkerJobs summary) so suppressions stay auditable. The windowed 5xx gate is unchanged.
|
|
298
|
+
Rationale: the brief self-healing "no-cause reduced health" flips on low-traffic 2-instance
|
|
299
|
+
envs are EB Enhanced Health flappiness (a single missed ALB `/health` check), not app bugs
|
|
300
|
+
— distinct from the busy-env PHP-`/health`-starvation cause. worker2-only; the param is
|
|
301
|
+
optional-with-default so no `_underscore`/dbchanges2 change. Not yet committed/deployed.
|
|
302
|
+
(jcardinal)
|
|
238
303
|
- 2026-08-25 — Replaced the EB `ApplicationMetrics` 5xx gate (a ~10s window that false-paged
|
|
239
304
|
"100% failing" on a lone 500 in a near-empty bucket — real worker-production incident: 16
|
|
240
305
|
500s / 516 req = 3.1% over 17 min) with a **CloudWatch windowed request-rate gate**. Reads
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
| Doc | Framework | Summary | Files |
|
|
4
4
|
|-----|-----------|---------|-------|
|
|
5
|
-
| [NYCHH inventory-adjustment → item-fulfillment linking (
|
|
5
|
+
| [NYCHH inventory-adjustment → item-fulfillment linking (direct FK columns, REST)](features/netsuite-inventory-adjustment-fulfillment-link.md) | 1.0 | For NYCHH stock adjustments, the NetSuite **inventory adjustment** carries a link to the **Item Fulfillment** it corrects, on the multi-select body custom field | library/app/api/toga2.php, library/app/api/netsuite/rest.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_hh.php, worker/crons/toga2/netsuite/diagnose_hh_import_inventory_adjustment.php |
|
|
6
6
|
| [NYCHH NetSuite → TransferOrders import (stocking-flag routing, PO bridges, origin location)](features/netsuite-transfer-order-import.md) | 1.0 | NYCHH's NetSuite **transfer orders are represented as NetSuite sales orders** (same as the GroWrk pattern), so the shared NetSuite → TOGa Supply importer (`App_ | library/app/api/toga2.php, library/app/api/netsuite/rest.php, library/app/netsuite.php, worker/crons/toga2/netsuite/sync_togasupply_hh.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/diagnose_hh_stuck_transfer_order.php, dbchanges2/Core/2026-08-27a - TransferOrderUuidSearchableIdentifier.sql, dbchanges2/Client/2026-08-28a - TransferOrderItemsTimestampColumns.sql, dbchanges2/Client/2026-08-28b - TransferOrderItemsTimestampColumnsReposition.sql |
|
|
7
7
|
| [NYCHH PO links are UPSTREAM — the downstream SO→PO route returns empty](features/po-number-upstream-direction.md) | 2.0 | NYCHH's sales orders are created **from the customer's purchase order**, so their SO↔PO links live in the **upstream** table `PurchaseOrders_SalesOrders` (route | _underscore/Model/Client/SalesOrder.php, _underscore/Trait/Netsuite/SalesOrder.php, toga25-supply/src/pages/SalesOrders/view/SalesOrderRecordModalLayout/hooks/usePurchaseOrderDetails.ts, dbchanges2/Client/2026-08-11b - SalesOrderPurchaseOrdersField.sql, dbchanges2/Client/2026-08-25 - SalesOrderPurchaseOrdersFieldAllClients.sql |
|
|
8
8
|
| [NYCHH transfer-order 2.0 model — inventory quantities + V2 field enablement](features/transfer-order-inventory-quantities.md) | 2.0 | The 2.0 (`_underscore`) side of NYCHH transfer-order support: a **two-branch inventory quantity model** on the NYCHH `Item` override, plus the **client-override | _underscore/Model/Nychh/Item.php, _underscore/Model/Nychh/TransferOrder.php, _underscore/Model/Nychh/TransferOrderStage.php, _underscore/Model/Client/TransferOrder.php, _underscore/Model/Client/TransferOrderItem.php, dbchanges2/Client_Nychh/2026-08-27a - TransferOrderCustomFieldWritePermission.sql, dbchanges2/Client_Nychh/2026-08-27b - TransferOrderStageStatusIdentifier.sql, dbchanges2/Client_Nychh/2026-08-28c - TransferOrderRecordAcl.sql, dbchanges2/Client/2026-08-28b - TransferOrderItemsTimestamps.sql |
|
|
@@ -1,29 +1,28 @@
|
|
|
1
1
|
---
|
|
2
|
-
title: "NYCHH inventory-adjustment → item-fulfillment linking (
|
|
2
|
+
title: "NYCHH inventory-adjustment → item-fulfillment linking (direct FK columns, REST)"
|
|
3
3
|
framework: "1.0"
|
|
4
4
|
repo: library
|
|
5
5
|
project: Library
|
|
6
6
|
client: nychh
|
|
7
7
|
type: client-feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-08-
|
|
9
|
+
updated: 2026-08-29
|
|
10
10
|
owners: [jcardinal]
|
|
11
11
|
files:
|
|
12
12
|
- library/app/api/toga2.php
|
|
13
|
+
- library/app/api/netsuite/rest.php
|
|
13
14
|
- worker/crons/toga2/netsuite/common_sync_togasupply.php
|
|
14
15
|
- worker/crons/toga2/netsuite/sync_togasupply_hh.php
|
|
15
16
|
- worker/crons/toga2/netsuite/diagnose_hh_import_inventory_adjustment.php
|
|
16
|
-
- _underscore/Model/Client/ItemFulfillments/InventoryAdjustment.php
|
|
17
|
-
- _underscore/Model/Client/ItemFulfillmentItems/InventoryAdjustmentItem.php
|
|
18
|
-
- dbchanges2/Core/2026-08-27b - ItemFulfillmentsInventoryAdjustmentsBridge.sql
|
|
19
|
-
- dbchanges2/Core/2026-08-28b - ItemFulfillmentItemsInventoryAdjustmentItemsBridge.sql
|
|
20
|
-
- dbchanges2/Client/2026-08-27a - ItemFulfillmentsInventoryAdjustmentsBridgeTable.sql
|
|
21
|
-
- dbchanges2/Client/2026-08-28c - ItemFulfillmentItemsInventoryAdjustmentItemsBridgeTable.sql
|
|
22
17
|
related:
|
|
23
18
|
- ./netsuite-transfer-order-import.md
|
|
24
19
|
- ./transfer-order-inventory-quantities.md
|
|
25
20
|
- ../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md
|
|
26
21
|
- ../../../1.0/apps/library/features/toga2-api-client-and-bridge.md
|
|
22
|
+
- ../../../1.0/apps/library/features/netsuite-suiteql-api-reference.md
|
|
23
|
+
- ../../../1.0/apps/library/features/netsuite-suiteql-rest-shim.md
|
|
24
|
+
- ../../../2.0/apps/api2/features/v2-rest-query-contract.md
|
|
25
|
+
- ../../../2.0/apps/api2/features/request-logging.md
|
|
27
26
|
- ../../../2.0/apps/_underscore/features/sales-order-purchase-order-bridge-direction.md
|
|
28
27
|
- ../profile.md
|
|
29
28
|
---
|
|
@@ -32,86 +31,134 @@ related:
|
|
|
32
31
|
|
|
33
32
|
For NYCHH stock adjustments, the NetSuite **inventory adjustment** carries a link to the **Item
|
|
34
33
|
Fulfillment** it corrects, on the multi-select body custom field **`custbody_stock_adjustment`**
|
|
35
|
-
(internalId **9735**). This feature imports that link into TOGa Supply
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
**
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
34
|
+
(internalId **9735**). This feature imports that link into TOGa Supply so H&H gets **adjustment →
|
|
35
|
+
item fulfillment → sales order → purchase order** traceability for their PO-specific fulfillment.
|
|
36
|
+
Per the **2026-08-27 TOGa Supply meeting**; the NetSuite custom field was implemented **<1 year
|
|
37
|
+
ago**, so older adjustments carry **no** link.
|
|
38
|
+
|
|
39
|
+
**Current design (2026-08-29): direct FK columns, over REST.** The link is now stored as **two
|
|
40
|
+
direct FK columns** — `InventoryAdjustments.itemFulfillmentId` and
|
|
41
|
+
`InventoryAdjustmentItems.itemFulfillmentItemId` — **replacing** the two bridge tables the
|
|
42
|
+
prior-session build used (`ItemFulfillments_InventoryAdjustments` /
|
|
43
|
+
`ItemFulfillmentItems_InventoryAdjustmentItems`). And the whole inventory-adjustment sync — plus the
|
|
44
|
+
on-demand fulfillment/sales-order import it triggers — was **migrated from SOAP to REST** for speed
|
|
45
|
+
(the SOAP path was ~2 NetSuite round-trips per serial, ~1h for a large serialized adjustment).
|
|
46
|
+
|
|
47
|
+
Like [NYCHH transfer-order import](./netsuite-transfer-order-import.md), the FK columns and models
|
|
48
|
+
are **all-client infrastructure** but the **attach is gated NYCHH-only** by
|
|
49
|
+
`IS_ENABLED_TRANSFER_ORDER_STOCKING_FLAG_ROUTING`.
|
|
50
|
+
|
|
51
|
+
## The two FK links (replaced the header/line bridge tables)
|
|
52
|
+
|
|
53
|
+
| Level | FK column | `Core.Records` | `Core.RecordFields` |
|
|
54
|
+
|---|---|---|---|
|
|
55
|
+
| Header | `InventoryAdjustments.itemFulfillmentId` | 154 (InventoryAdjustments) | reuses 2517 |
|
|
56
|
+
| Item | `InventoryAdjustmentItems.itemFulfillmentItemId` | 155 (InventoryAdjustmentItems) | reuses 2521 |
|
|
57
|
+
|
|
58
|
+
The FK columns **reuse the `RecordFields` already registered/granted from the earlier bridge work**
|
|
59
|
+
(2517 on record 154, 2521 on record 155), so this change needed **no new Core registration or ACL
|
|
60
|
+
grant** — the bridge tables (records 353/354) and their per-client copies are superseded. Writing a
|
|
61
|
+
link is now a plain PUT of the FK uuid onto the parent record, not a bridge-row insert.
|
|
60
62
|
|
|
61
63
|
## Attach code — `App_Api_Toga2::attachInventoryAdjustmentToItemFulfillment`
|
|
62
64
|
|
|
63
65
|
Lives in `library/app/api/toga2.php`, called from `syncInventoryAdjustmentFromNetsuite` **after the
|
|
64
|
-
header upsert**, gated on `IS_ENABLED_TRANSFER_ORDER_STOCKING_FLAG_ROUTING
|
|
65
|
-
`try/catch` — it is **best-effort and must never freeze the section** (a failed link is logged, the
|
|
66
|
-
adjustment import still completes). It:
|
|
66
|
+
header upsert**, gated on `IS_ENABLED_TRANSFER_ORDER_STOCKING_FLAG_ROUTING`. It:
|
|
67
67
|
|
|
68
68
|
1. reads `custbody_stock_adjustment` off the adjustment (the related Item Fulfillment),
|
|
69
69
|
2. resolves that fulfillment by `ItemFulfillments.c_netsuiteInternalItemFulfillmentId`,
|
|
70
|
-
3.
|
|
71
|
-
4.
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
70
|
+
3. writes the **header** FK (`InventoryAdjustments.itemFulfillmentId`), then
|
|
71
|
+
4. writes the **line** FK (`InventoryAdjustmentItems.itemFulfillmentItemId`) for each
|
|
72
|
+
same-product adjustment-item ↔ fulfillment-item pair, matched by product `item` uuid.
|
|
73
|
+
|
|
74
|
+
### ⚠ Reading the product uuid needs `depth=3` (the max-depth omission trap)
|
|
75
|
+
|
|
76
|
+
Step 4's product match reads `salesOrderItem->item->uuid` from `GET /item-fulfillment-items`. The
|
|
77
|
+
V2 read engine **omits a non-null FK object that sits at the max requested `depth`** — and a null
|
|
78
|
+
sibling FK still renders, so the omission is easy to miss. At `depth=2` the grandchild `item`
|
|
79
|
+
(level 3: `itemFulfillmentItem → salesOrderItem → item`) was dropped → the product-match map was
|
|
80
|
+
empty → **no `itemFulfillmentItemId` was written**. Fix: request **`depth=3`** *plus* explicit
|
|
81
|
+
nested fields `salesOrderItem.item.uuid` / `transferOrderItem.item.uuid`. Framework detail:
|
|
82
|
+
[V2 REST query contract — max-depth FK omission](../../../2.0/apps/api2/features/v2-rest-query-contract.md).
|
|
83
|
+
|
|
84
|
+
### ⚠ The FK write's PUT echo may not reflect what persisted
|
|
85
|
+
|
|
86
|
+
A `PUT /v2/inventory-adjustments/{uuid}` with `{"itemFulfillment":{"uuid":…}}` returned **200,
|
|
87
|
+
empty `messages[]`, and an echo that did NOT contain `itemFulfillment`** — yet the FK column
|
|
88
|
+
**persisted correctly**. Do not read an empty echo as a failed write; verify against the DB or the
|
|
89
|
+
`Logs_<Client>.Api` `responsePayload`. This was diagnosed entirely from `Logs_Nychh.Api` (routes
|
|
90
|
+
stored with the `/v2` prefix — filter `route LIKE '/v2/...'`), with **no NetSuite access needed**.
|
|
91
|
+
See [V2 request logging](../../../2.0/apps/api2/features/request-logging.md).
|
|
92
|
+
|
|
93
|
+
### Loud failure (best-effort try/catch removed)
|
|
94
|
+
|
|
95
|
+
The attach and the surrounding NetSuite→DB sync path **no longer swallow errors**: the
|
|
96
|
+
error-log-and-swallow `try/catch` blocks were removed so a data-corruption failure surfaces loudly
|
|
97
|
+
(StarTech/Sentry catches left alone). This is the same loud-failure policy as the 2026-08-27 removal
|
|
98
|
+
of the per-order `try/catch` skips across all six sync sections — see
|
|
99
|
+
[NetSuite → TOGa Supply per-client sync](../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md).
|
|
100
|
+
**Do not "soften" this back into a swallow** — a bridge/link miss must be visible, not silently
|
|
101
|
+
dropped.
|
|
102
|
+
|
|
103
|
+
## On-demand dependency import (fulfillment not yet in TOGA) — now REST
|
|
104
|
+
|
|
105
|
+
When the linked fulfillment has **not** been imported into TOGA yet, the attach **imports the
|
|
106
|
+
fulfillment on demand** via `syncItemFulfillmentFromNetsuite` (which itself cascades to the
|
|
107
|
+
originating sales order when missing), then **re-resolves and links**. This on-demand import was
|
|
108
|
+
**converted from SOAP to REST** this session.
|
|
109
|
+
|
|
110
|
+
Threading the on-demand import required the **full item-fulfillment lookup set** — customer,
|
|
111
|
+
country, state, location, manufacturer, item, all-catalog-items, vendor-item, shipping-carrier,
|
|
112
|
+
shipping-method, warehouse — to be passed through `syncInventoryAdjustmentFromNetsuite` (signature
|
|
113
|
+
**expanded 5 → 13 args**) and into `attachInventoryAdjustmentToItemFulfillment`, plus updating the
|
|
114
|
+
`common_sync_togasupply.php` `INVENTORY_ADJUSTMENTS` call to pass them. **Design rule (developer
|
|
115
|
+
instruction): all lookups are passed function-to-function**, like the other sync functions — not
|
|
116
|
+
rebuilt or fetched inside the callee.
|
|
117
|
+
|
|
118
|
+
## The REST-shim fixes this migration required (`library/app/api/netsuite/rest.php`)
|
|
119
|
+
|
|
120
|
+
Both are framework-level NetSuite facts — full detail on
|
|
121
|
+
[NetSuite SuiteQL/REST API Reference](../../../1.0/apps/library/features/netsuite-suiteql-api-reference.md):
|
|
122
|
+
|
|
123
|
+
- **Item-fulfillment REST shim returned zero line items.** `fetchItemFulfillmentById` fetched
|
|
124
|
+
fulfillment lines from the REST **record sublist** `GET /record/v1/itemFulfillment/{id}/item?limit=1000&offset=N`.
|
|
125
|
+
**REST record sublist endpoints reject `limit`/`offset`** (HTTP 400 `INVALID_PARAMETER`), so the
|
|
126
|
+
shim silently got 0 lines and every fulfillment imported **header-only** — starving all downstream
|
|
127
|
+
ItemFulfillmentItems and the line-level link. **Fix:** fetch lines via SuiteQL `transactionline`
|
|
128
|
+
(the pattern every other shim in the file already uses).
|
|
129
|
+
- **Fulfillment SuiteQL returned ~3 rows per fulfilled item.** An Item Fulfillment (`ItemShip`) has
|
|
130
|
+
three `transactionline` rows per fulfilled item: the order-linked line, a positive "wash" line, and
|
|
131
|
+
an `isinventoryaffecting='T'` line — only the order-linked line maps to a source-order line.
|
|
132
|
+
**Fix:** keep only lines with a `previoustransactionlinelink` of type `SalesOrd`/`TrnfrOrd` (via
|
|
133
|
+
`EXISTS`), and derive `orderLine` from that link's `previousline` (correlated subquery).
|
|
134
|
+
**`mainline='F'` is the WRONG filter** — NetSuite sometimes marks the first real fulfilled line
|
|
135
|
+
`mainline='T'`.
|
|
136
|
+
|
|
137
|
+
## Two shared-engine fixes in `syncInventoryAdjustmentFromNetsuite` (surfaced here, affect any client)
|
|
138
|
+
|
|
139
|
+
Both are in the shared library method — see also the
|
|
140
|
+
[per-client sync](../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md):
|
|
93
141
|
|
|
94
142
|
- **(a) Prune-order FK violation.** The stale-item prune deleted an `InventoryAdjustmentItem` while
|
|
95
143
|
its child `InventoryAdjustmentItemUnits` still referenced it (FK **RESTRICT**, no cascade) → MySQL
|
|
96
144
|
**1451** → `EV-11`. **Fix:** delete the child units (`/inventory-adjustment-item-units/{uuid}`)
|
|
97
|
-
**before** the item. (Same "delete
|
|
98
|
-
- **(b) Item-lookup indexing mismatch → duplicate items every run.** The item lookup was consumed
|
|
99
|
-
**two-level** `[$uuidManufacturer][$partNumber]`, but the cron passes it **one-level**
|
|
100
|
-
(populated in `common_sync_togasupply.php` ~L313)
|
|
101
|
-
|
|
102
|
-
(
|
|
103
|
-
(duplicates / `EV-10`). **Fix:** index one-level `[$partNumber]`.
|
|
145
|
+
**before** the item. (Same "delete child rows first" rule as the tracking-number bridges.)
|
|
146
|
+
- **(b) Item-lookup indexing mismatch → duplicate items every run.** The item lookup was consumed
|
|
147
|
+
**two-level** `[$uuidManufacturer][$partNumber]`, but the cron passes it **one-level**
|
|
148
|
+
`[$partNumber]` (populated in `common_sync_togasupply.php` ~L313); the two-level lookup is declared
|
|
149
|
+
but **never populated (dead)**. Existing items were never found, so items were **re-created every
|
|
150
|
+
run** (duplicates / `EV-10`). **Fix:** index one-level `[$partNumber]`.
|
|
104
151
|
|
|
105
152
|
## Diagnostic harness — `diagnose_hh_import_inventory_adjustment.php`
|
|
106
153
|
|
|
107
154
|
A **manual operator tool** in `worker/crons/toga2/netsuite/` (NOT in `cron.worker.sync.json`; does
|
|
108
155
|
**not** call `cronInitialization`, so no process lock / no cursor writes; installs a strict
|
|
109
156
|
`set_error_handler` that rethrows warnings as `ErrorException` for exact crash lines). It imports
|
|
110
|
-
**one** inventory adjustment
|
|
111
|
-
|
|
112
|
-
**
|
|
113
|
-
|
|
114
|
-
|
|
157
|
+
**one** inventory adjustment by calling the real `syncInventoryAdjustmentFromNetsuite`, seeds the
|
|
158
|
+
warehouse/item/manufacturer lookups (the item loop **filters** items to known warehouses and
|
|
159
|
+
**creates** items on a lookup-miss, so empty lookups drop or duplicate items), and has a
|
|
160
|
+
**read-only** diagnostic tail that dumps the NS item list + `custbody_stock_adjustment` + whether the
|
|
161
|
+
referenced fulfillment already exists in TOGA.
|
|
115
162
|
|
|
116
163
|
```
|
|
117
164
|
php crons/toga2/netsuite/diagnose_hh_import_inventory_adjustment.php <togaClient> <togaApi> <togaSecret> [adjustmentInternalId]
|
|
@@ -120,53 +167,53 @@ php crons/toga2/netsuite/diagnose_hh_import_inventory_adjustment.php <togaClient
|
|
|
120
167
|
|
|
121
168
|
## Go-live toggle
|
|
122
169
|
|
|
123
|
-
`IS_ENABLED_INTEGRATION_INVENTORY_ADJUSTMENTS`
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
## Open / next-session — migrate the inventory-adjustment sync from SOAP to REST
|
|
127
|
-
|
|
128
|
-
The inventory-adjustment sync is still **SOAP**: `App_NetSuite::getInventoryNumberFromSerialNumber` +
|
|
129
|
-
`getInventoryNumber` per serial (`toga2.php` ~5371/5374), plus `getObject('inventoryAdjustment')`,
|
|
130
|
-
`getItemDetails`, `detectItemType`, `getItemIsSerialized`, `getItemIsFulfillable`. For a **serialized**
|
|
131
|
-
adjustment this is ~2 NetSuite SOAP round-trips **per serial** → ~1 hour for a large adjustment; the
|
|
132
|
-
on-demand fulfillment import inherits the same cost. **Decision:** migrate the whole
|
|
133
|
-
inventory-adjustment sync to REST via `App_Api_Netsuite_Rest` (`rest.php` already has
|
|
134
|
-
`send`/`authenticate`/`suiteqlListAll`/`getCustomFieldValue`/`fetchItemById`/`listInventoryAdjustments`
|
|
135
|
-
(minimal) — the gaps are a full inventory-adjustment fetch with inventory detail/serials, and an
|
|
136
|
-
inventory-number/serial batch lookup). **This is the next session's task.**
|
|
170
|
+
`IS_ENABLED_INTEGRATION_INVENTORY_ADJUSTMENTS` in `sync_togasupply_hh.php` is the go-live switch for
|
|
171
|
+
this feature on the scheduled cron.
|
|
137
172
|
|
|
138
|
-
##
|
|
173
|
+
## Validation (end-to-end, prod)
|
|
139
174
|
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
175
|
+
- **Non-serialized:** linked adjustment **6994071** → fulfillment **6827251** — header FK + line FK
|
|
176
|
+
both populate.
|
|
177
|
+
- **Serialized:** linked adjustment **6871592** → fulfillment **6868763** (2 serials) — header FK,
|
|
178
|
+
line FK, **and 2 serial units on BOTH** the adjustment item and the fulfillment item all populate
|
|
179
|
+
correctly.
|
|
145
180
|
|
|
146
181
|
## Gotchas / known issues
|
|
147
182
|
|
|
148
183
|
- **Older adjustments have no link.** `custbody_stock_adjustment` was added in NetSuite <1 year ago;
|
|
149
|
-
adjustments predating it will never populate
|
|
150
|
-
- **
|
|
151
|
-
|
|
152
|
-
|
|
184
|
+
adjustments predating it will never populate an FK — not a bug.
|
|
185
|
+
- **Reading the product uuid at `depth=2` silently returns nothing** — the level-3 `item` object is
|
|
186
|
+
omitted at max depth. Use `depth=3` + explicit nested `fields` (see above).
|
|
187
|
+
- **A V2 PUT echo that omits `itemFulfillment` does NOT mean the write failed** — verify against the
|
|
188
|
+
DB / `Logs_<Client>.Api`, not the echoed body.
|
|
153
189
|
- **Empty lookups drop or duplicate items.** The item loop filters to known warehouses and creates
|
|
154
190
|
items on lookup-miss, so any manual/diagnostic invocation must seed the warehouse/item/manufacturer
|
|
155
191
|
lookups (the harness does).
|
|
156
192
|
|
|
157
193
|
## Change history
|
|
158
194
|
|
|
159
|
-
- 2026-08-
|
|
160
|
-
|
|
161
|
-
(`
|
|
162
|
-
|
|
163
|
-
`
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
195
|
+
- 2026-08-29 — **Migrated the feature to direct FK columns over REST, and made it fail loud.**
|
|
196
|
+
Replaced the two bridge tables with direct FK columns
|
|
197
|
+
(`InventoryAdjustments.itemFulfillmentId` reusing `RecordField` 2517/record 154;
|
|
198
|
+
`InventoryAdjustmentItems.itemFulfillmentItemId` reusing 2521/record 155 — no new Core
|
|
199
|
+
registration/ACL). Migrated `syncInventoryAdjustmentFromNetsuite` and its on-demand
|
|
200
|
+
fulfillment/sales-order import **SOAP → REST** (was ~2 round-trips/serial → ~1h for a large
|
|
201
|
+
serialized adjustment), and removed the error-log-and-swallow `try/catch` in the NetSuite→DB sync
|
|
202
|
+
path (StarTech/Sentry catches left alone). Fixed three read-path traps hit along the way: the REST
|
|
203
|
+
**record-sublist `limit`/`offset` 400** (`fetchItemFulfillmentById` returned 0 lines → header-only
|
|
204
|
+
import; now uses SuiteQL `transactionline`), the **ItemShip 3-rows-per-item** filter (keep the
|
|
205
|
+
`SalesOrd`/`TrnfrOrd` line-linked line, `mainline='F'` is wrong), and the **V2 max-depth FK
|
|
206
|
+
omission** (`depth=2` dropped `salesOrderItem.item` → now `depth=3` + explicit nested fields).
|
|
207
|
+
Discovered a **V2 PUT echo can omit a field that persisted** (verified via `Logs_Nychh.Api`, no
|
|
208
|
+
NetSuite access). Validated end-to-end in prod: non-serialized 6994071→6827251 and serialized
|
|
209
|
+
6871592→6868763 (header FK, line FK, and 2 serial units on both sides). (jcardinal)
|
|
210
|
+
- 2026-08-28 — Built the initial NYCHH inventory-adjustment → item-fulfillment linking off NetSuite
|
|
211
|
+
`custbody_stock_adjustment` (internalId 9735) as **two all-client bridge tables** (header record
|
|
212
|
+
353 / item record 354, Core-registered + Super-User/per-client ACL) with a gated best-effort
|
|
213
|
+
`App_Api_Toga2::attachInventoryAdjustmentToItemFulfillment` and **on-demand import** of a
|
|
214
|
+
not-yet-imported fulfillment via `syncItemFulfillmentFromNetsuite` (cascades to the SO) — which
|
|
215
|
+
expanded `syncInventoryAdjustmentFromNetsuite` 5→13 args to thread the full IF lookup set. Fixed the
|
|
216
|
+
two shared-engine bugs (prune child units before the item — FK RESTRICT/1451/EV-11; index the item
|
|
217
|
+
lookup one-level `[$partNumber]` — was two-level dead → duplicates/EV-10). Added the
|
|
218
|
+
`diagnose_hh_import_inventory_adjustment.php` operator harness. *(This bridge-table design was
|
|
219
|
+
superseded by the 2026-08-29 direct-FK / REST migration above.)* (jcardinal)
|
package/package.json
CHANGED