toga-ai 1.0.696 → 1.0.698

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -6,7 +6,7 @@ project: Library
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-08-19
9
+ updated: 2026-08-29
10
10
  owners: ["dfranks", "jcardinal", "kyalamarthi"]
11
11
  files:
12
12
  - library/app/api/netsuite/rest.php
@@ -16,6 +16,7 @@ related:
16
16
  - netsuite-suiteql-rest-shim.md
17
17
  - ../architecture.md
18
18
  - ../../worker/features/forecast2-netsuite-reconciliation.md
19
+ - ../../../clients/nychh/features/netsuite-inventory-adjustment-fulfillment-link.md
19
20
  ---
20
21
 
21
22
  ## Summary
@@ -219,6 +220,22 @@ Other JE line facts (same probes):
219
220
  > NetSuite custom field supplying those dimensions is created and populated (no `salesrep` column, no
220
221
  > `item`). Re-evaluate only once that custom field exists — don't re-probe this from scratch.
221
222
 
223
+ ### Item Fulfillment (`ItemShip`) lines — ~3 rows per fulfilled item, and `mainline` is NOT the filter
224
+
225
+ For an Item Fulfillment (`transaction.type = 'ItemShip'`), `transactionline` returns **~3 rows per
226
+ fulfilled item**: the order-linked line, a positive "wash" line, and an `isinventoryaffecting = 'T'`
227
+ line. **Only the order-linked line maps to a source-order line.** Keep exactly that one by requiring
228
+ a `previoustransactionlinelink` of type `SalesOrd`/`TrnfrOrd` (via an `EXISTS` correlated subquery),
229
+ and derive the source `orderLine` from that link's `previousline` (a correlated subquery — see the
230
+ `previousTransactionLineLink` section below).
231
+
232
+ - **⚠ `mainline = 'F'` is the WRONG filter for an ItemShip line.** NetSuite sometimes marks the
233
+ first real fulfilled line `mainline = 'T'`, so a `mainline='F'` predicate **silently drops it**.
234
+ Filter on the `SalesOrd`/`TrnfrOrd` line-link instead, not on `mainline`. (Same "extra posting
235
+ lines per item" shape as the credit-memo/cash-refund COGS lines in
236
+ `netsuite-suiteql-rest-shim.md`.) Surfaced building
237
+ [NYCHH inventory-adjustment → item-fulfillment linking](../../../clients/nychh/features/netsuite-inventory-adjustment-fulfillment-link.md).
238
+
222
239
  ## `previousTransactionLineLink` table
223
240
 
224
241
  The bridge between a child line (e.g. invoice) and its originating parent line (e.g. SO).
@@ -322,6 +339,15 @@ Used for fields SuiteQL doesn't expose on `transaction`: `shippingCost`, `taxTot
322
339
  `linesequencenumber` can diverge from REST `line` for SOAP-era records (confirmed empirically),
323
340
  so keying on `linesequencenumber` risks duplicate Sales rows that can't be matched or deleted.
324
341
  **`listSales()` keeps the per-id GET for this reason — do not eliminate it.**
342
+ - **⚠ REST record SUBLIST endpoints reject `limit`/`offset` (HTTP 400).**
343
+ `GET /record/v1/{type}/{id}/{sublist}` (e.g. `.../itemFulfillment/{id}/item?limit=1000&offset=N`)
344
+ returns **400 `INVALID_PARAMETER`** — *"Use one of: fields, expand, expandRecords,
345
+ expandSubResources, simpleEnumFormat, ids"*. `limit`/`offset` are valid only on **collection**
346
+ endpoints, not on a record's sublist. A shim that paginated a sublist this way silently got
347
+ **0 lines** and imported the record **header-only** with no error. Fetch line data via **SuiteQL
348
+ `transactionline`** instead (the pattern every other shim in `rest.php` already uses). Surfaced
349
+ when `fetchItemFulfillmentById` returned zero lines — see
350
+ [NYCHH inventory-adjustment → item-fulfillment linking](../../../clients/nychh/features/netsuite-inventory-adjustment-fulfillment-link.md).
325
351
 
326
352
  ## `list*()` helpers — behavior & cost
327
353
 
@@ -392,6 +418,17 @@ numbers. Budget hours for multi-year runs and launch them under `nohup`/`tmux`.
392
418
 
393
419
  ## Change history
394
420
 
421
+ - 2026-08-29 — Recorded two NetSuite REST/SuiteQL facts from the NYCHH inventory-adjustment →
422
+ item-fulfillment **REST migration**: **REST record sublist endpoints
423
+ (`/record/v1/{type}/{id}/{sublist}`) reject `limit`/`offset` with a 400 `INVALID_PARAMETER`** (a
424
+ paginated sublist fetch silently returns 0 lines → header-only import; use SuiteQL
425
+ `transactionline`), and an **Item Fulfillment (`ItemShip`) `transactionline` returns ~3 rows per
426
+ fulfilled item** (order-linked + positive "wash" + `isinventoryaffecting='T'`) — keep only the line
427
+ with a `previoustransactionlinelink` of type `SalesOrd`/`TrnfrOrd` and derive `orderLine` from its
428
+ `previousline`; **`mainline='F'` is the wrong filter** because NetSuite sometimes marks the first
429
+ real line `mainline='T'`. See
430
+ [NYCHH inventory-adjustment → item-fulfillment linking](../../../clients/nychh/features/netsuite-inventory-adjustment-fulfillment-link.md).
431
+ (jcardinal)
395
432
  - 2026-08-19 — Added two SuiteQL mechanics that fail **silently**, both found porting the customer
396
433
  account-type sync (TRUE-80206): **column aliases come back lower-cased** (`AS accountTypeLabel` →
397
434
  `accounttypelabel`, so a camelCase read is `NULL` with no error — it made a backfill report success while
@@ -520,8 +520,29 @@ Parameters are stored **per client DB** but accessed **through the TOGa2 API**,
520
520
  item-fulfillment lookup set through for the on-demand fulfillment import (lookups are passed
521
521
  function-to-function, like the other sync functions).
522
522
 
523
+ - **⚠ Inventory-adjustment sync migrated SOAP → REST + now fails LOUD (2026-08-29).** The SOAP path
524
+ flagged just above (`App_NetSuite::getInventoryNumber*`, ~2 round-trips per serial) has been
525
+ **migrated to REST/SuiteQL** via `App_Api_Netsuite_Rest`, and the **on-demand fulfillment /
526
+ sales-order import** it triggers was converted to REST too — closing the ~1h-per-large-adjustment
527
+ cost that was the previously-planned next step. Also, the remaining **error-log-and-swallow
528
+ `try/catch` blocks in the NetSuite→DB sync path were removed** so a data-corruption failure surfaces
529
+ loudly (StarTech/Sentry catches left alone) — the same loud-failure policy as the 2026-08-27
530
+ per-order `try/catch` removal. Full detail (the REST-shim fixes it required, and the bridges →
531
+ direct-FK-column change) on
532
+ [NYCHH inventory-adjustment → item-fulfillment linking](../../../clients/nychh/features/netsuite-inventory-adjustment-fulfillment-link.md).
533
+
523
534
  ## Change history
524
535
 
536
+ - 2026-08-29 — The inventory-adjustment sync (and the on-demand fulfillment / sales-order import it
537
+ triggers) in `syncInventoryAdjustmentFromNetsuite` was **migrated SOAP → REST/SuiteQL**, closing the
538
+ SOAP round-trip cost flagged 2026-08-28; the remaining error-log-and-swallow `try/catch` in the
539
+ NetSuite→DB sync path were removed for loud failure (StarTech/Sentry catches untouched). The
540
+ migration required two `App_Api_Netsuite_Rest` shim fixes (a record-sublist `limit`/`offset` 400 and
541
+ the ItemShip 3-rows-per-item line filter) and moved the NYCHH adjustment↔fulfillment link off bridge
542
+ tables onto direct FK columns — see
543
+ [NYCHH inventory-adjustment → item-fulfillment linking](../../../clients/nychh/features/netsuite-inventory-adjustment-fulfillment-link.md).
544
+ (jcardinal)
545
+
525
546
  - 2026-08-28 — Recorded two shared-engine inventory-adjustment fixes in
526
547
  `syncInventoryAdjustmentFromNetsuite` (prune child `InventoryAdjustmentItemUnits` before the item —
527
548
  FK RESTRICT/1451/EV-11; index the item lookup **one-level** `[$partNumber]` to match what the cron
@@ -302,6 +302,27 @@ the deploy/version timeline before hunting app bugs. **Fixed 2026-08-28:** all t
302
302
  to **Rolling / batch-size 1** (≥1 healthy backend throughout a deploy), which eliminates the blip —
303
303
  config + tradeoffs in the [deploy workflow](workflows/codepipeline-codeconnections-deploy.md).
304
304
 
305
+ **A fifth cause — a brief, self-healing "no-cause TargetGroup reduced health" flip on a
306
+ LOW-TRAFFIC, IDLE 2-instance env is EB Enhanced Health FLAPPINESS, not worker starvation.**
307
+ Distinct from the third cause (that one is the *busy* env, `api-production-1`, under Compass ASN
308
+ worker saturation). This is a single missed ALB `/health` check — a network/agent transient.
309
+ Fingerprint (verified `api-production-3`, eu-west-1, 2026-08-29 13:31 UTC `Severe`, static-`/health`
310
+ build deployed): self-heals in ~2–5 min; throughout, `HealthyHostCount` stays 2 (1-min minimum),
311
+ **0** target/ELB 5xx, sub-second latency, idle CPU, near-zero traffic; ALB target health already
312
+ reads "healthy" by the time you check. On an idle env there is no worker starvation, so **no
313
+ server-side app fix applies** — do not chase it. The mitigation is alert de-sensitization: the
314
+ worker2 EB health monitor now applies a **persistence-delay gate** (a non-5xx `Degraded`/`Severe`
315
+ must persist ≥ `nonFivexxPersistMinutes`, default 5, before it pages), which suppresses these flaps.
316
+ See [worker2 EB health monitor](../worker2/features/elastic-beanstalk-health-monitor.md).
317
+
318
+ **Triage aid — break an EB "% requests failing with HTTP 5xx" flag into ELB 502/503/504 to find
319
+ the cause:** **503** = no healthy backend (a deploy with no capacity); **504** = target timeout
320
+ (worker saturation); **502** = connection dropped mid-response (an `AllAtOnce` deploy killing
321
+ in-flight requests, or a PHP worker crash). Because **ELB-generated 5xx (any of these) never reach
322
+ PHP**, they leave **no row in `Logs.Api`** (api2 logs only *after* a completed response) — so an EB
323
+ "% 5xx" flag with an **empty `Logs.Api`** means an **ELB-level** 5xx, not an app 5xx. Discriminate
324
+ with CloudWatch `HTTPCode_ELB_502/503/504_Count` vs `HTTPCode_Target_5XX_Count`.
325
+
305
326
  ## Known issues / accepted risks
306
327
 
307
328
  Open items a maintainer should know before changing this tier. None are "bugs to fix right now" —
@@ -379,6 +400,14 @@ they are the known sharp edges. Do not re-discover these from scratch.
379
400
  single-caller branches in `V2.php` as unverified until exercised directly.
380
401
 
381
402
  ## Change history
403
+ - 2026-08-29 — Recorded a **fifth EB-`Degraded` cause**: a brief self-healing "no-cause reduced
404
+ health" flip on a low-traffic *idle* 2-instance env (e.g. `api-production-3`, eu-west-1) is EB
405
+ Enhanced Health flappiness (a single missed ALB `/health` check), distinct from the busy-env
406
+ PHP-`/health`-starvation cause — no server-side fix applies; mitigated by the worker2 monitor's
407
+ new non-5xx persistence-delay gate. Added an ELB **502/503/504 discrimination** triage aid
408
+ (503=no healthy backend, 504=target timeout/worker saturation, 502=dropped connection from an
409
+ AllAtOnce deploy or PHP crash); ELB-generated 5xx never reach PHP so they leave no `Logs.Api`
410
+ row — an EB "% 5xx" flag with an empty `Logs.Api` is ELB-level, not app 5xx. (jcardinal)
382
411
  - 2026-08-28 — Recorded a **fourth EB-`Degraded` cause — the deploy-restart ELB 502**: under the old
383
412
  `AllAtOnce`/100% policy a deploy restarts both instances of a 2-instance env at once, dropping
384
413
  in-flight connections → an ELB-502 burst (43 in one minute on `api-production-1`, 2026-08-28) that
@@ -6,8 +6,8 @@ project: API
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-08-28
10
- owners: ["mhammontree", "dfranks", "bala"]
9
+ updated: 2026-08-29
10
+ owners: ["mhammontree", "dfranks", "bala", "jcardinal"]
11
11
  files:
12
12
  - api2/Component/Api/V2/V2.php
13
13
  - api2/Controller/Index.php
@@ -175,6 +175,14 @@ There is no usable index on those columns. Do not start from a payload search.
175
175
 
176
176
  The working pattern, in order:
177
177
 
178
+ > **`route` values are stored WITH the `/v2` prefix** (e.g. `/v2/inventory-adjustments/{uuid}`,
179
+ > `/v2/item-fulfillment-items`), so any narrowing predicate must filter `route LIKE '/v2/...'` — a
180
+ > bare `/inventory-adjustments` matches **nothing**. `route` also carries `requestPayload` +
181
+ > `responsePayload` + `responseCode` per row, so once narrowed the log gives the full request/response
182
+ > for a call **without NetSuite access** — it was the decisive tool for confirming a V2 PUT persisted
183
+ > a field its response echo omitted (see the [V2 query contract](v2-rest-query-contract.md) PUT-echo
184
+ > gotcha).
185
+
178
186
  1. **Narrow with indexed / non-text columns only** — a `dtStamp` range plus `route`,
179
187
  `method`, `responseCode`, `hostname`. A
180
188
  `GROUP BY hostname, route, method, responseCode` over one day is cheap and is the fastest way
@@ -217,6 +225,12 @@ re-send.
217
225
 
218
226
  ## Change history
219
227
 
228
+ - 2026-08-29 — Noted that `Logs.Api.route` values are stored **with the `/v2` prefix** (e.g.
229
+ `/v2/inventory-adjustments/{uuid}`), so narrowing queries must filter `route LIKE '/v2/...'` — a
230
+ bare `/inventory-adjustments` predicate matches nothing. Reinforced that a narrowed row's
231
+ `requestPayload`/`responsePayload` is the decisive, NetSuite-access-free tool for diagnosing V2
232
+ write behavior — here, confirming a V2 PUT persisted a field its response echo omitted (NYCHH
233
+ inventory-adjustment → item-fulfillment linking). (jcardinal)
220
234
  - 2026-08-13 — **Important limit on the "an HTTP 400 leaves a full log row" rule** (TRUE-80412):
221
235
  `execute()` writes the transaction log near its **end** (~L2225-2271), *after* route processing
222
236
  (~L1742-2043), so **a throw from a PRE interceptor produces no `Logs.Api` row at all** — the
@@ -6,8 +6,8 @@ project: API
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-08-28
10
- owners: [tcox, bala, apeterson]
9
+ updated: 2026-08-29
10
+ owners: [tcox, bala, apeterson, jcardinal]
11
11
  files:
12
12
  - api2/Component/Api/V2/V2.php
13
13
  related:
@@ -108,6 +108,19 @@ calculated field is null on a nested object but correct when that object is fetc
108
108
  Definition of the fields themselves:
109
109
  [FIELD_SQL calculated fields](../../_underscore/features/calculated-sql-fields.md).
110
110
 
111
+ ### ⚠ A non-null FK object at the MAX requested `depth` is OMITTED from the response
112
+
113
+ A GET serialized to `depth: N` renders FK objects down to level N, but a **non-null FK object that
114
+ sits at level N+1** — one hop *past* the max — is **dropped from the payload with no error**. It is
115
+ easy to miss because a **null** sibling FK still renders as `null`, so the omission reads like "that
116
+ FK is null" rather than "that FK was not serialized." Observed on `GET /item-fulfillment-items` at
117
+ `depth: 2`: the grandchild `item` (level 3 — `itemFulfillmentItem → salesOrderItem → item`) was
118
+ absent, so a product-match keyed on `salesOrderItem.item.uuid` came back empty and no link was
119
+ written. **Fix: request `depth: N+1`, or name the nested path explicitly in `fields`** (e.g.
120
+ `salesOrderItem.item.uuid` / `transferOrderItem.item.uuid`). Rule of thumb: the deepest FK you
121
+ actually read must sit at least one level *inside* `depth`, not on the boundary. Surfaced building
122
+ [NYCHH inventory-adjustment → item-fulfillment linking](../../../../clients/nychh/features/netsuite-inventory-adjustment-fulfillment-link.md).
123
+
111
124
  ### What nests by default is `InherentRecordChildren` — everything else needs the nested ROUTE
112
125
 
113
126
  A record's one-to-many children arrive inside the record response **only** if `Core.
@@ -271,8 +284,24 @@ where=(field:op:value,LOGIC,field:op:value,(nested,OR,nested))
271
284
  - **`gte`/`lte` silently are not operators** — use `ge`/`le`.
272
285
  - **Do not trust a 200 with missing fields.** Without an explicit `fields=` list, ACL removal is
273
286
  invisible; re-request with `fields=` to force `EZ-2` and see the denied list.
287
+ - **⚠ A PUT response ECHO can omit a field that DID persist — verify against the DB, not the echo.**
288
+ A `PUT /v2/inventory-adjustments/{uuid}` with `{"itemFulfillment":{"uuid":…}}` returned **200 with
289
+ an empty `messages[]`** and an echo that did **not** contain `itemFulfillment`, yet the FK column
290
+ **persisted correctly**. An absent field in the write-response echo is **not** proof the write
291
+ failed — confirm the field against the record's DB row (or the `Logs_<Client>.Api` `responsePayload`),
292
+ never from the write's own echoed body. See
293
+ [V2 request logging](request-logging.md).
274
294
 
275
295
  ## Change history
296
+ - 2026-08-29 — Recorded two response-shape facts from the NYCHH inventory-adjustment REST work: **a
297
+ non-null FK object one hop past the max `depth` is silently omitted** (a null sibling FK still
298
+ renders, so it reads as "null" not "missing") — `GET /item-fulfillment-items` at `depth:2` dropped
299
+ the level-3 `salesOrderItem.item`; fix with `depth+1` or an explicit nested `fields` path — and a
300
+ **PUT response echo can omit a field that persisted** (a `PUT /v2/inventory-adjustments/{uuid}`
301
+ returned 200 / empty-`messages` without echoing `itemFulfillment` though the FK saved), so verify a
302
+ write against the DB / `Logs_<Client>.Api`, not the echo. See
303
+ [NYCHH inventory-adjustment → item-fulfillment linking](../../../../clients/nychh/features/netsuite-inventory-adjustment-fulfillment-link.md).
304
+ (jcardinal)
276
305
  - 2026-08-28 — Documented two response-shape facts hit while building the Transfer Orders modal.
277
306
  **`calcDepth` defaults to 1**, so a `FIELD_SQL` calculated field on a record one FK hop out returns
278
307
  `null` with **no** message (Location's `_address*` fields needed `calcDepth: 2`) — raise it one per
@@ -152,6 +152,10 @@ one 15 s interval (~45 s → 30 s) of per-batch health settle.
152
152
  `BatchSize`); `aws:elasticbeanstalk:environment:process:default` (`HealthyThresholdCount`).
153
153
  - Trade-off: Rolling is slower end-to-end than the old ~31 s `AllAtOnce`, and capacity briefly
154
154
  drops to 1/2 instances per batch (fine at current traffic).
155
+ - **Confirmed working in production (2026-08-29).** The 2026-08-29 00:49 UTC deploy ran as
156
+ Rolling batches ("Batch 1 / Batch 2, one instance at a time, register and wait for healthy")
157
+ across all three API prod envs with **no post-deploy 5xx flip** — the `AllAtOnce`
158
+ deploy-restart 502 blip is eliminated in practice, not just in config.
155
159
 
156
160
  ### Deploy-policy guidance (don't reach for immutable expecting speed)
157
161
  - **Immutable does NOT reduce deploy time.** It is the **slowest** policy — it boots a fresh
@@ -295,6 +299,10 @@ aws codeconnections get-connection --connection-arn "<CONN_ARN>" --region "$REGI
295
299
  ```
296
300
 
297
301
  ## Change history
302
+ - 2026-08-29 — Confirmed the Rolling/batch-1 policy works in production: the 2026-08-29
303
+ 00:49 UTC deploy ran as Rolling batches (one instance at a time, register + wait for healthy)
304
+ on all three API prod envs with **no post-deploy 5xx flip** — the AllAtOnce deploy-restart
305
+ 502 blip is eliminated. (jcardinal)
298
306
  - 2026-08-28 — Recorded the **production deploy trigger + policy change**: the `api-production-1/2/3`
299
307
  CodePipelines **auto-deploy on every push to `_production`** (WebhookV2 on the CodeConnections
300
308
  connection, repos `_underscore` + `api2`), so a merge to `_production` **is** a three-region prod
@@ -6,7 +6,7 @@ project: Worker
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-08-25
9
+ updated: 2026-08-29
10
10
  owners: [jcardinal]
11
11
  files:
12
12
  - worker2/Worker/Infrastructure/CloudWatch.php
@@ -28,10 +28,13 @@ OneUptime string-matches — OneUptime cannot compare numbers on a pushed body.
28
28
 
29
29
  The load-bearing design decision this doc records is the **paging (alarm) criteria**, and
30
30
  the key **2026-08-25** change: the 5xx page is now driven by a **CloudWatch windowed
31
- request-rate gate**, fully **decoupled from EB's own `HealthStatus`**. A non-5xx
32
- `Degraded`/`Severe` still pages immediately off EB health; but whether a *5xx* problem pages
33
- is decided by the true per-environment 5xx **rate over a trailing window** (default 5 min),
34
- not by EB's ~10-second `ApplicationMetrics` snapshot.
31
+ request-rate gate**, fully **decoupled from EB's own `HealthStatus`**. Whether a *5xx*
32
+ problem pages is decided by the true per-environment 5xx **rate over a trailing window**
33
+ (default 5 min), not by EB's ~10-second `ApplicationMetrics` snapshot. As of **2026-08-29**,
34
+ a non-5xx `Degraded`/`Severe` no longer pages immediately either — it must **persist** past a
35
+ threshold (`nonFivexxPersistMinutes`, default 5) before paging, which suppresses the brief
36
+ self-healing "reduced health" flips that EB Enhanced Health throws on low-traffic 2-instance
37
+ envs (see the persistence gate below).
35
38
 
36
39
  **Why the change:** the old gate read EB `describeEnvironmentHealth`'s `ApplicationMetrics`,
37
40
  a ~10-**second** window. At low traffic a single 500 in a near-empty bucket reads as 100%
@@ -52,6 +55,10 @@ itself runs non-fatal. CloudWatch read failures are caught **internally** and re
52
55
  `_Worker::runTask('Infrastructure/CloudWatch/ElasticBeanstalkHealth', {awsAccountId,
53
56
  environments, oneuptimeUrl, …thresholds})`.
54
57
  - New private `fetchWindowedRequestCounts()` reads the CloudWatch metrics (below).
58
+ - New private `fetchNonOkDepartureAgeSeconds(ElasticBeanstalkClient $eb, string
59
+ $environmentName, int $lookbackSeconds): ?int` returns how long the current non-Ok episode
60
+ has persisted, from EB `describeEvents` (below). **Never throws** — catches `AwsException`
61
+ and returns `null`.
55
62
  - AWS access is obtained through `_Component_Aws_Workloads::assumeCredentials()` (STS-assume
56
63
  a read-only role per account, one assume reused across regions) — see
57
64
  [Cross-account AWS access](./cross-account-aws-access.md), which also documents the
@@ -64,11 +71,14 @@ itself runs non-fatal. CloudWatch read failures are caught **internally** and re
64
71
 
65
72
  Two independent paths now feed `alarm=HIGH`:
66
73
 
67
- 1. **EB-health path (non-5xx).** EB `HealthStatus` `Degraded` or `Severe` that is **not**
68
- attributable solely to 5xx (latency, instances down, a failed deploy, a mixed cause, or
69
- no stated cause) **pages immediately** — unchanged, still driven off EB health.
70
- `Suspended` never pages. `causesAttributeSolelyTo5xx()` is what tells a solely-5xx event
71
- apart from everything else.
74
+ 1. **EB-health path (non-5xx) — now gated on PERSISTENCE (2026-08-29).** EB `HealthStatus`
75
+ `Degraded` or `Severe` that is **not** attributable solely to 5xx (latency, instances
76
+ down, a failed deploy, a mixed cause, or no stated cause) pages **only once the current
77
+ non-Ok episode has persisted ≥ `nonFivexxPersistMinutes`** (new optional param, default 5)
78
+ — it no longer pages on the first snapshot. A sub-threshold blip is **reported but does not
79
+ page**. `Suspended` never pages. `causesAttributeSolelyTo5xx()` is what tells a solely-5xx
80
+ event apart from everything else. See the persistence-gate section below for how episode
81
+ age is measured and the fail-safe.
72
82
  2. **CloudWatch windowed 5xx-rate path.** For a solely-5xx `Degraded`/`Severe`, the page is
73
83
  decided **not** by EB but by the true windowed rate from CloudWatch. Given the window's
74
84
  `requests` (ApplicationRequestsTotal) and `fivexx` (ApplicationRequests5xx), page when:
@@ -91,6 +101,7 @@ The four thresholds are **optional cron parameters** spread as **named arguments
91
101
  | `minRequests` | `100` | Minimum windowed request count before the **share** gate applies |
92
102
  | `min5xxPercent` | `20.0` | 5xx share (percent) at/above which a solely-5xx env pages, once `minRequests` is met |
93
103
  | `min5xxAbsolute` | `20` | Absolute windowed 5xx count at/above which a solely-5xx env pages regardless of share |
104
+ | `nonFivexxPersistMinutes` | `5` | Minutes a **non-5xx** `Degraded`/`Severe` episode must persist before it pages (added 2026-08-29; a trailing optional param — existing `CronJobs.parameters` without the key still run) |
94
105
 
95
106
  Removed this iteration: `shouldPageForHealth()` and the old
96
107
  `HTTP_5XX_ALARM_RATIO` / `HTTP_5XX_ALARM_ABSOLUTE_COUNT` constants (superseded by the
@@ -123,6 +134,43 @@ A CloudWatch read *failure* (as opposed to empty data) sets **`probe=DEGRADED`**
123
134
  gap), distinct from a measured-bad `alarm=HIGH` — same alarm-vs-probe split as the
124
135
  multi-client monitors in [OneUptime push-metric monitors](./oneuptime-worker2-monitoring.md).
125
136
 
137
+ ## Persistence gate on the non-5xx page (2026-08-29)
138
+
139
+ A non-5xx `Degraded`/`Severe` now pages only after the **current non-Ok episode has persisted
140
+ ≥ `nonFivexxPersistMinutes`** (default 5). This exists because the brief "no-cause TargetGroup
141
+ reduced health" flips on low-traffic 2-instance envs are **EB Enhanced Health flappiness** —
142
+ a single missed ALB `/health` check (a network/agent transient), not a real app problem — and
143
+ they self-heal in ~2–5 min. A persistence delay suppresses those while a genuinely sustained
144
+ non-5xx degradation still pages, delayed only by the threshold. (The windowed 5xx-flood gate
145
+ is **unchanged** — it has its own, independent decision.)
146
+
147
+ **Fingerprint of the idle-env flap** (verified `api-production-3`, eu-west-1, 2026-08-29
148
+ 13:31 UTC `Severe`, static-`/health` build deployed): self-heals in ~2–5 min; throughout,
149
+ `HealthyHostCount` stays 2 (1-min minimum), **0** target/ELB 5xx, sub-second latency, idle
150
+ CPU, near-zero traffic; ALB target health already reads "healthy" by the time you look. On an
151
+ idle env there is **no worker starvation**, so there is nothing to fix server-side — the
152
+ mitigation is this monitor persistence-delay gate, not more app code. **This is DISTINCT from
153
+ the PHP-`/health`-starvation cause** (that one is the *busy* env, `api-production-1`, under
154
+ ASN worker saturation, fixed by the static-`/health` file) — see
155
+ [api2 architecture — EB "Degraded" health](../../api2/architecture.md).
156
+
157
+ **Measuring episode age — `fetchNonOkDepartureAgeSeconds()`** reads EB `describeEvents`:
158
+
159
+ - The most recent **"transitioned from Ok to …"** event marks the current episode's start.
160
+ Once non-Ok, later transitions stay between non-Ok states until a to-Ok event; because
161
+ `describeEvents` returns **newest-first**, the first "from Ok to …" match wins.
162
+ - The query is **TIME-bounded**: `StartTime = now − (persistWindow +
163
+ HEALTH_EVENTS_LOOKBACK_BUFFER_SECONDS = 600)`, with a `MaxRecords = 100` cap — so a chatty
164
+ env's deploy/config events cannot push the departure event out of range.
165
+ - **Fail-safe:** an episode **older than the window** returns `null` → the caller **pages**.
166
+ That is correct: if the departure is beyond the lookback, the episode has already persisted
167
+ past the threshold, so paging is the safe outcome. A read failure also returns `null`
168
+ (the method catches `AwsException` and never throws) → same fail-safe.
169
+
170
+ **Auditable suppression:** a sub-threshold blip is reported, not paged, and the suppression is
171
+ made visible — `pendingPersistSeconds` is added to the OneUptime payload and a `pending=Ns`
172
+ annotation to the WorkerJobs summary — so a suppressed episode is never silent.
173
+
126
174
  ## Reading the windowed 5xx rate from CloudWatch (durable AWS reference)
127
175
 
128
176
  - **EB application-request metrics are PER-INSTANCE ONLY.** `ApplicationRequestsTotal`,
@@ -214,6 +262,8 @@ the `probe` token; this is a OneUptime config change the developer applies.
214
262
  `min5xxPercent`; absolute at `min5xxAbsolute-1` vs `min5xxAbsolute`; `requests` just below
215
263
  `minRequests`; empty/zero window; mixed 5xx + non-5xx cause) — deferred pending a harness.
216
264
  - **Deploy fails without degrading health** — still not caught (accepted residual).
265
+ - **`ElasticBeanstalkHealth()` is long** and grew with the persistence gate — php-reviewer
266
+ (0 critical) flagged it as a decomposition candidate; deferred pending the missing harness.
217
267
 
218
268
  ## Gotchas / known issues
219
269
 
@@ -235,6 +285,21 @@ the `probe` token; this is a OneUptime config change the developer applies.
235
285
  record its value in a doc.
236
286
 
237
287
  ## Change history
288
+ - 2026-08-29 — Added a **persistence gate on the non-5xx page**: a non-5xx `Degraded`/`Severe`
289
+ no longer pages on the first snapshot — it must persist ≥ `nonFivexxPersistMinutes` (new
290
+ optional trailing param, default 5; existing `CronJobs.parameters` without the key still
291
+ run) before paging. Episode age comes from EB `describeEvents` via new never-throwing helper
292
+ `fetchNonOkDepartureAgeSeconds()` (most recent "from Ok to …" marks episode start; events
293
+ are newest-first so the first match wins; TIME-bounded `StartTime = now − (persistWindow +
294
+ HEALTH_EVENTS_LOOKBACK_BUFFER_SECONDS=600)`, `MaxRecords=100`, so an episode older than the
295
+ window returns `null` → the caller pages — correct fail-safe). Sub-threshold blips are
296
+ reported not paged, with `pendingPersistSeconds` (OneUptime payload) + `pending=Ns`
297
+ (WorkerJobs summary) so suppressions stay auditable. The windowed 5xx gate is unchanged.
298
+ Rationale: the brief self-healing "no-cause reduced health" flips on low-traffic 2-instance
299
+ envs are EB Enhanced Health flappiness (a single missed ALB `/health` check), not app bugs
300
+ — distinct from the busy-env PHP-`/health`-starvation cause. worker2-only; the param is
301
+ optional-with-default so no `_underscore`/dbchanges2 change. Not yet committed/deployed.
302
+ (jcardinal)
238
303
  - 2026-08-25 — Replaced the EB `ApplicationMetrics` 5xx gate (a ~10s window that false-paged
239
304
  "100% failing" on a lone 500 in a near-empty bucket — real worker-production incident: 16
240
305
  500s / 516 req = 3.1% over 17 min) with a **CloudWatch windowed request-rate gate**. Reads
@@ -2,7 +2,7 @@
2
2
 
3
3
  | Doc | Framework | Summary | Files |
4
4
  |-----|-----------|---------|-------|
5
- | [NYCHH inventory-adjustment → item-fulfillment linking (custbody_stock_adjustment bridges)](features/netsuite-inventory-adjustment-fulfillment-link.md) | 1.0 | For NYCHH stock adjustments, the NetSuite **inventory adjustment** carries a link to the **Item Fulfillment** it corrects, on the multi-select body custom field | library/app/api/toga2.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_hh.php, worker/crons/toga2/netsuite/diagnose_hh_import_inventory_adjustment.php, _underscore/Model/Client/ItemFulfillments/InventoryAdjustment.php, _underscore/Model/Client/ItemFulfillmentItems/InventoryAdjustmentItem.php, dbchanges2/Core/2026-08-27b - ItemFulfillmentsInventoryAdjustmentsBridge.sql, dbchanges2/Core/2026-08-28b - ItemFulfillmentItemsInventoryAdjustmentItemsBridge.sql, dbchanges2/Client/2026-08-27a - ItemFulfillmentsInventoryAdjustmentsBridgeTable.sql, dbchanges2/Client/2026-08-28c - ItemFulfillmentItemsInventoryAdjustmentItemsBridgeTable.sql |
5
+ | [NYCHH inventory-adjustment → item-fulfillment linking (direct FK columns, REST)](features/netsuite-inventory-adjustment-fulfillment-link.md) | 1.0 | For NYCHH stock adjustments, the NetSuite **inventory adjustment** carries a link to the **Item Fulfillment** it corrects, on the multi-select body custom field | library/app/api/toga2.php, library/app/api/netsuite/rest.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_hh.php, worker/crons/toga2/netsuite/diagnose_hh_import_inventory_adjustment.php |
6
6
  | [NYCHH NetSuite → TransferOrders import (stocking-flag routing, PO bridges, origin location)](features/netsuite-transfer-order-import.md) | 1.0 | NYCHH's NetSuite **transfer orders are represented as NetSuite sales orders** (same as the GroWrk pattern), so the shared NetSuite → TOGa Supply importer (`App_ | library/app/api/toga2.php, library/app/api/netsuite/rest.php, library/app/netsuite.php, worker/crons/toga2/netsuite/sync_togasupply_hh.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/diagnose_hh_stuck_transfer_order.php, dbchanges2/Core/2026-08-27a - TransferOrderUuidSearchableIdentifier.sql, dbchanges2/Client/2026-08-28a - TransferOrderItemsTimestampColumns.sql, dbchanges2/Client/2026-08-28b - TransferOrderItemsTimestampColumnsReposition.sql |
7
7
  | [NYCHH PO links are UPSTREAM — the downstream SO→PO route returns empty](features/po-number-upstream-direction.md) | 2.0 | NYCHH's sales orders are created **from the customer's purchase order**, so their SO↔PO links live in the **upstream** table `PurchaseOrders_SalesOrders` (route | _underscore/Model/Client/SalesOrder.php, _underscore/Trait/Netsuite/SalesOrder.php, toga25-supply/src/pages/SalesOrders/view/SalesOrderRecordModalLayout/hooks/usePurchaseOrderDetails.ts, dbchanges2/Client/2026-08-11b - SalesOrderPurchaseOrdersField.sql, dbchanges2/Client/2026-08-25 - SalesOrderPurchaseOrdersFieldAllClients.sql |
8
8
  | [NYCHH transfer-order 2.0 model — inventory quantities + V2 field enablement](features/transfer-order-inventory-quantities.md) | 2.0 | The 2.0 (`_underscore`) side of NYCHH transfer-order support: a **two-branch inventory quantity model** on the NYCHH `Item` override, plus the **client-override | _underscore/Model/Nychh/Item.php, _underscore/Model/Nychh/TransferOrder.php, _underscore/Model/Nychh/TransferOrderStage.php, _underscore/Model/Client/TransferOrder.php, _underscore/Model/Client/TransferOrderItem.php, dbchanges2/Client_Nychh/2026-08-27a - TransferOrderCustomFieldWritePermission.sql, dbchanges2/Client_Nychh/2026-08-27b - TransferOrderStageStatusIdentifier.sql, dbchanges2/Client_Nychh/2026-08-28c - TransferOrderRecordAcl.sql, dbchanges2/Client/2026-08-28b - TransferOrderItemsTimestamps.sql |
@@ -1,29 +1,28 @@
1
1
  ---
2
- title: "NYCHH inventory-adjustment → item-fulfillment linking (custbody_stock_adjustment bridges)"
2
+ title: "NYCHH inventory-adjustment → item-fulfillment linking (direct FK columns, REST)"
3
3
  framework: "1.0"
4
4
  repo: library
5
5
  project: Library
6
6
  client: nychh
7
7
  type: client-feature
8
8
  status: active
9
- updated: 2026-08-28
9
+ updated: 2026-08-29
10
10
  owners: [jcardinal]
11
11
  files:
12
12
  - library/app/api/toga2.php
13
+ - library/app/api/netsuite/rest.php
13
14
  - worker/crons/toga2/netsuite/common_sync_togasupply.php
14
15
  - worker/crons/toga2/netsuite/sync_togasupply_hh.php
15
16
  - worker/crons/toga2/netsuite/diagnose_hh_import_inventory_adjustment.php
16
- - _underscore/Model/Client/ItemFulfillments/InventoryAdjustment.php
17
- - _underscore/Model/Client/ItemFulfillmentItems/InventoryAdjustmentItem.php
18
- - dbchanges2/Core/2026-08-27b - ItemFulfillmentsInventoryAdjustmentsBridge.sql
19
- - dbchanges2/Core/2026-08-28b - ItemFulfillmentItemsInventoryAdjustmentItemsBridge.sql
20
- - dbchanges2/Client/2026-08-27a - ItemFulfillmentsInventoryAdjustmentsBridgeTable.sql
21
- - dbchanges2/Client/2026-08-28c - ItemFulfillmentItemsInventoryAdjustmentItemsBridgeTable.sql
22
17
  related:
23
18
  - ./netsuite-transfer-order-import.md
24
19
  - ./transfer-order-inventory-quantities.md
25
20
  - ../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md
26
21
  - ../../../1.0/apps/library/features/toga2-api-client-and-bridge.md
22
+ - ../../../1.0/apps/library/features/netsuite-suiteql-api-reference.md
23
+ - ../../../1.0/apps/library/features/netsuite-suiteql-rest-shim.md
24
+ - ../../../2.0/apps/api2/features/v2-rest-query-contract.md
25
+ - ../../../2.0/apps/api2/features/request-logging.md
27
26
  - ../../../2.0/apps/_underscore/features/sales-order-purchase-order-bridge-direction.md
28
27
  - ../profile.md
29
28
  ---
@@ -32,86 +31,134 @@ related:
32
31
 
33
32
  For NYCHH stock adjustments, the NetSuite **inventory adjustment** carries a link to the **Item
34
33
  Fulfillment** it corrects, on the multi-select body custom field **`custbody_stock_adjustment`**
35
- (internalId **9735**). This feature imports that link into TOGa Supply as two new bridge tables
36
- (header + line), so H&H gets **adjustment → item fulfillment → sales order → purchase order**
37
- traceability for their PO-specific fulfillment. Per the **2026-08-27 TOGa Supply meeting**; the
38
- NetSuite custom field was implemented **<1 year ago**, so older adjustments carry **no** link.
39
-
40
- Like [NYCHH transfer-order import](./netsuite-transfer-order-import.md), the **tables and models are
41
- all-client infrastructure** (`dbchanges2/Client/` + shared `_Model_Client_*` bridge models) but the
42
- **attach is gated NYCHH-only** by `IS_ENABLED_TRANSFER_ORDER_STOCKING_FLAG_ROUTING`.
43
-
44
- ## The two bridges (both mirror the transfer-order bridge pattern)
45
-
46
- | Level | Bridge table | `Core.Records` id | `RecordFields` | Model |
47
- |---|---|---|---|---|
48
- | Header | `ItemFulfillments_InventoryAdjustments` | 353 | 2515–2518 | `_Model_Client_ItemFulfillments_InventoryAdjustment` |
49
- | Item | `ItemFulfillmentItems_InventoryAdjustmentItems` | 354 | 2519–2522 | `_Model_Client_ItemFulfillmentItems_InventoryAdjustmentItem` |
50
-
51
- Each shipped as: an all-client `Client/` bridge table, Core registration (`Records` + `RecordFields`),
52
- a Super-User ACL grant, and a per-client ACL **copied from the parent record**. The parent record
53
- uuids are already `isIdentifier = 1`, so — unlike the transfer-order bridges — **no `EV-12`
54
- uuid-identifier fix was needed** (contrast the 312/313 anomaly on
55
- [the ACL permission chain](../../../2.0/apps/_underscore/features/acl-permission-chain.md)).
56
-
57
- > **Bridge FK writes via a nested `{uuid}` need only record-level ACL, not field-level.** Verified
58
- > against the existing transfer-order bridges — the nested-write ACL check is at the record level for a
59
- > bridge link, so no `AclCustomFieldPermissions` grant is required on the FK columns.
34
+ (internalId **9735**). This feature imports that link into TOGa Supply so H&H gets **adjustment →
35
+ item fulfillment → sales order → purchase order** traceability for their PO-specific fulfillment.
36
+ Per the **2026-08-27 TOGa Supply meeting**; the NetSuite custom field was implemented **<1 year
37
+ ago**, so older adjustments carry **no** link.
38
+
39
+ **Current design (2026-08-29): direct FK columns, over REST.** The link is now stored as **two
40
+ direct FK columns** — `InventoryAdjustments.itemFulfillmentId` and
41
+ `InventoryAdjustmentItems.itemFulfillmentItemId` — **replacing** the two bridge tables the
42
+ prior-session build used (`ItemFulfillments_InventoryAdjustments` /
43
+ `ItemFulfillmentItems_InventoryAdjustmentItems`). And the whole inventory-adjustment sync — plus the
44
+ on-demand fulfillment/sales-order import it triggers — was **migrated from SOAP to REST** for speed
45
+ (the SOAP path was ~2 NetSuite round-trips per serial, ~1h for a large serialized adjustment).
46
+
47
+ Like [NYCHH transfer-order import](./netsuite-transfer-order-import.md), the FK columns and models
48
+ are **all-client infrastructure** but the **attach is gated NYCHH-only** by
49
+ `IS_ENABLED_TRANSFER_ORDER_STOCKING_FLAG_ROUTING`.
50
+
51
+ ## The two FK links (replaced the header/line bridge tables)
52
+
53
+ | Level | FK column | `Core.Records` | `Core.RecordFields` |
54
+ |---|---|---|---|
55
+ | Header | `InventoryAdjustments.itemFulfillmentId` | 154 (InventoryAdjustments) | reuses 2517 |
56
+ | Item | `InventoryAdjustmentItems.itemFulfillmentItemId` | 155 (InventoryAdjustmentItems) | reuses 2521 |
57
+
58
+ The FK columns **reuse the `RecordFields` already registered/granted from the earlier bridge work**
59
+ (2517 on record 154, 2521 on record 155), so this change needed **no new Core registration or ACL
60
+ grant** — the bridge tables (records 353/354) and their per-client copies are superseded. Writing a
61
+ link is now a plain PUT of the FK uuid onto the parent record, not a bridge-row insert.
60
62
 
61
63
  ## Attach code — `App_Api_Toga2::attachInventoryAdjustmentToItemFulfillment`
62
64
 
63
65
  Lives in `library/app/api/toga2.php`, called from `syncInventoryAdjustmentFromNetsuite` **after the
64
- header upsert**, gated on `IS_ENABLED_TRANSFER_ORDER_STOCKING_FLAG_ROUTING`, and wrapped in
65
- `try/catch` — it is **best-effort and must never freeze the section** (a failed link is logged, the
66
- adjustment import still completes). It:
66
+ header upsert**, gated on `IS_ENABLED_TRANSFER_ORDER_STOCKING_FLAG_ROUTING`. It:
67
67
 
68
68
  1. reads `custbody_stock_adjustment` off the adjustment (the related Item Fulfillment),
69
69
  2. resolves that fulfillment by `ItemFulfillments.c_netsuiteInternalItemFulfillmentId`,
70
- 3. idempotently creates the **header** link row, then
71
- 4. links **line items** by matching `itemId` — every same-product adjustment-item ↔ fulfillment-item
72
- pair is linked; the bridge's `UNIQUE(fk, fk)` dedupes re-runs.
73
-
74
- ## On-demand dependency import (fulfillment not yet in TOGA)
75
-
76
- When the linked fulfillment has **not** been imported into TOGA yet, the attach no longer skips and
77
- waits for the fulfillment backfill — it **imports the fulfillment on demand** via
78
- `syncItemFulfillmentFromNetsuite` (which itself cascades to the originating sales order when missing),
79
- then **re-resolves and links**.
80
-
81
- This required threading the **full item-fulfillment lookup set** — customer, country, state, location,
82
- manufacturer, item, all-catalog-items, vendor-item, shipping-carrier, shipping-method, warehouse —
83
- through `syncInventoryAdjustmentFromNetsuite` (signature **expanded 5 → 13 args**) and into
84
- `attachInventoryAdjustmentToItemFulfillment`, plus updating the `common_sync_togasupply.php`
85
- `INVENTORY_ADJUSTMENTS` call to pass them (the same set already passed to the item-fulfillment
86
- section). **Design rule (developer instruction): all lookups are passed function-to-function**, like
87
- the other sync functions — not rebuilt or fetched inside the callee.
88
-
89
- ## Two fixes in `syncInventoryAdjustmentFromNetsuite` (shared engine, surfaced by the harness)
90
-
91
- Both are in the shared library method, so they affect **any** client running inventory adjustments —
92
- see also the [per-client sync](../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md):
70
+ 3. writes the **header** FK (`InventoryAdjustments.itemFulfillmentId`), then
71
+ 4. writes the **line** FK (`InventoryAdjustmentItems.itemFulfillmentItemId`) for each
72
+ same-product adjustment-item ↔ fulfillment-item pair, matched by product `item` uuid.
73
+
74
+ ### ⚠ Reading the product uuid needs `depth=3` (the max-depth omission trap)
75
+
76
+ Step 4's product match reads `salesOrderItem->item->uuid` from `GET /item-fulfillment-items`. The
77
+ V2 read engine **omits a non-null FK object that sits at the max requested `depth`** — and a null
78
+ sibling FK still renders, so the omission is easy to miss. At `depth=2` the grandchild `item`
79
+ (level 3: `itemFulfillmentItem → salesOrderItem → item`) was dropped → the product-match map was
80
+ empty → **no `itemFulfillmentItemId` was written**. Fix: request **`depth=3`** *plus* explicit
81
+ nested fields `salesOrderItem.item.uuid` / `transferOrderItem.item.uuid`. Framework detail:
82
+ [V2 REST query contract — max-depth FK omission](../../../2.0/apps/api2/features/v2-rest-query-contract.md).
83
+
84
+ ### ⚠ The FK write's PUT echo may not reflect what persisted
85
+
86
+ A `PUT /v2/inventory-adjustments/{uuid}` with `{"itemFulfillment":{"uuid":…}}` returned **200,
87
+ empty `messages[]`, and an echo that did NOT contain `itemFulfillment`** — yet the FK column
88
+ **persisted correctly**. Do not read an empty echo as a failed write; verify against the DB or the
89
+ `Logs_<Client>.Api` `responsePayload`. This was diagnosed entirely from `Logs_Nychh.Api` (routes
90
+ stored with the `/v2` prefix — filter `route LIKE '/v2/...'`), with **no NetSuite access needed**.
91
+ See [V2 request logging](../../../2.0/apps/api2/features/request-logging.md).
92
+
93
+ ### Loud failure (best-effort try/catch removed)
94
+
95
+ The attach and the surrounding NetSuite→DB sync path **no longer swallow errors**: the
96
+ error-log-and-swallow `try/catch` blocks were removed so a data-corruption failure surfaces loudly
97
+ (StarTech/Sentry catches left alone). This is the same loud-failure policy as the 2026-08-27 removal
98
+ of the per-order `try/catch` skips across all six sync sections — see
99
+ [NetSuite → TOGa Supply per-client sync](../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md).
100
+ **Do not "soften" this back into a swallow** — a bridge/link miss must be visible, not silently
101
+ dropped.
102
+
103
+ ## On-demand dependency import (fulfillment not yet in TOGA) — now REST
104
+
105
+ When the linked fulfillment has **not** been imported into TOGA yet, the attach **imports the
106
+ fulfillment on demand** via `syncItemFulfillmentFromNetsuite` (which itself cascades to the
107
+ originating sales order when missing), then **re-resolves and links**. This on-demand import was
108
+ **converted from SOAP to REST** this session.
109
+
110
+ Threading the on-demand import required the **full item-fulfillment lookup set** — customer,
111
+ country, state, location, manufacturer, item, all-catalog-items, vendor-item, shipping-carrier,
112
+ shipping-method, warehouse — to be passed through `syncInventoryAdjustmentFromNetsuite` (signature
113
+ **expanded 5 → 13 args**) and into `attachInventoryAdjustmentToItemFulfillment`, plus updating the
114
+ `common_sync_togasupply.php` `INVENTORY_ADJUSTMENTS` call to pass them. **Design rule (developer
115
+ instruction): all lookups are passed function-to-function**, like the other sync functions — not
116
+ rebuilt or fetched inside the callee.
117
+
118
+ ## The REST-shim fixes this migration required (`library/app/api/netsuite/rest.php`)
119
+
120
+ Both are framework-level NetSuite facts — full detail on
121
+ [NetSuite SuiteQL/REST API Reference](../../../1.0/apps/library/features/netsuite-suiteql-api-reference.md):
122
+
123
+ - **Item-fulfillment REST shim returned zero line items.** `fetchItemFulfillmentById` fetched
124
+ fulfillment lines from the REST **record sublist** `GET /record/v1/itemFulfillment/{id}/item?limit=1000&offset=N`.
125
+ **REST record sublist endpoints reject `limit`/`offset`** (HTTP 400 `INVALID_PARAMETER`), so the
126
+ shim silently got 0 lines and every fulfillment imported **header-only** — starving all downstream
127
+ ItemFulfillmentItems and the line-level link. **Fix:** fetch lines via SuiteQL `transactionline`
128
+ (the pattern every other shim in the file already uses).
129
+ - **Fulfillment SuiteQL returned ~3 rows per fulfilled item.** An Item Fulfillment (`ItemShip`) has
130
+ three `transactionline` rows per fulfilled item: the order-linked line, a positive "wash" line, and
131
+ an `isinventoryaffecting='T'` line — only the order-linked line maps to a source-order line.
132
+ **Fix:** keep only lines with a `previoustransactionlinelink` of type `SalesOrd`/`TrnfrOrd` (via
133
+ `EXISTS`), and derive `orderLine` from that link's `previousline` (correlated subquery).
134
+ **`mainline='F'` is the WRONG filter** — NetSuite sometimes marks the first real fulfilled line
135
+ `mainline='T'`.
136
+
137
+ ## Two shared-engine fixes in `syncInventoryAdjustmentFromNetsuite` (surfaced here, affect any client)
138
+
139
+ Both are in the shared library method — see also the
140
+ [per-client sync](../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md):
93
141
 
94
142
  - **(a) Prune-order FK violation.** The stale-item prune deleted an `InventoryAdjustmentItem` while
95
143
  its child `InventoryAdjustmentItemUnits` still referenced it (FK **RESTRICT**, no cascade) → MySQL
96
144
  **1451** → `EV-11`. **Fix:** delete the child units (`/inventory-adjustment-item-units/{uuid}`)
97
- **before** the item. (Same "delete bridge/child rows first" rule as the tracking-number bridges.)
98
- - **(b) Item-lookup indexing mismatch → duplicate items every run.** The item lookup was consumed as a
99
- **two-level** `[$uuidManufacturer][$partNumber]`, but the cron passes it **one-level** `[$partNumber]`
100
- (populated in `common_sync_togasupply.php` ~L313). The two-level
101
- `$lookupItemByClientUuidAndManufacturerUuidAndPartNumberUpper` is declared but **never populated
102
- (dead)**. The mismatch meant existing items were never found, so items were **re-created every run**
103
- (duplicates / `EV-10`). **Fix:** index one-level `[$partNumber]`.
145
+ **before** the item. (Same "delete child rows first" rule as the tracking-number bridges.)
146
+ - **(b) Item-lookup indexing mismatch → duplicate items every run.** The item lookup was consumed
147
+ **two-level** `[$uuidManufacturer][$partNumber]`, but the cron passes it **one-level**
148
+ `[$partNumber]` (populated in `common_sync_togasupply.php` ~L313); the two-level lookup is declared
149
+ but **never populated (dead)**. Existing items were never found, so items were **re-created every
150
+ run** (duplicates / `EV-10`). **Fix:** index one-level `[$partNumber]`.
104
151
 
105
152
  ## Diagnostic harness — `diagnose_hh_import_inventory_adjustment.php`
106
153
 
107
154
  A **manual operator tool** in `worker/crons/toga2/netsuite/` (NOT in `cron.worker.sync.json`; does
108
155
  **not** call `cronInitialization`, so no process lock / no cursor writes; installs a strict
109
156
  `set_error_handler` that rethrows warnings as `ErrorException` for exact crash lines). It imports
110
- **one** inventory adjustment (default NS internal id **6516616**) by calling the real
111
- `syncInventoryAdjustmentFromNetsuite`, seeds the warehouse/item/manufacturer lookups (the item loop
112
- **filters** items to known warehouses and **creates** items on a lookup-miss, so empty lookups drop or
113
- duplicate items), and has a **read-only** diagnostic tail that dumps the NS item list +
114
- `custbody_stock_adjustment` + whether the referenced fulfillment already exists in TOGA.
157
+ **one** inventory adjustment by calling the real `syncInventoryAdjustmentFromNetsuite`, seeds the
158
+ warehouse/item/manufacturer lookups (the item loop **filters** items to known warehouses and
159
+ **creates** items on a lookup-miss, so empty lookups drop or duplicate items), and has a
160
+ **read-only** diagnostic tail that dumps the NS item list + `custbody_stock_adjustment` + whether the
161
+ referenced fulfillment already exists in TOGA.
115
162
 
116
163
  ```
117
164
  php crons/toga2/netsuite/diagnose_hh_import_inventory_adjustment.php <togaClient> <togaApi> <togaSecret> [adjustmentInternalId]
@@ -120,53 +167,53 @@ php crons/toga2/netsuite/diagnose_hh_import_inventory_adjustment.php <togaClient
120
167
 
121
168
  ## Go-live toggle
122
169
 
123
- `IS_ENABLED_INTEGRATION_INVENTORY_ADJUSTMENTS` is still **`false`** in `sync_togasupply_hh.php` —
124
- enabling it is the go-live switch for this feature on the scheduled cron.
125
-
126
- ## Open / next-session — migrate the inventory-adjustment sync from SOAP to REST
127
-
128
- The inventory-adjustment sync is still **SOAP**: `App_NetSuite::getInventoryNumberFromSerialNumber` +
129
- `getInventoryNumber` per serial (`toga2.php` ~5371/5374), plus `getObject('inventoryAdjustment')`,
130
- `getItemDetails`, `detectItemType`, `getItemIsSerialized`, `getItemIsFulfillable`. For a **serialized**
131
- adjustment this is ~2 NetSuite SOAP round-trips **per serial** → ~1 hour for a large adjustment; the
132
- on-demand fulfillment import inherits the same cost. **Decision:** migrate the whole
133
- inventory-adjustment sync to REST via `App_Api_Netsuite_Rest` (`rest.php` already has
134
- `send`/`authenticate`/`suiteqlListAll`/`getCustomFieldValue`/`fetchItemById`/`listInventoryAdjustments`
135
- (minimal) — the gaps are a full inventory-adjustment fetch with inventory detail/serials, and an
136
- inventory-number/serial batch lookup). **This is the next session's task.**
170
+ `IS_ENABLED_INTEGRATION_INVENTORY_ADJUSTMENTS` in `sync_togasupply_hh.php` is the go-live switch for
171
+ this feature on the scheduled cron.
137
172
 
138
- ## Known state — test adjustment 6516616
173
+ ## Validation (end-to-end, prod)
139
174
 
140
- NYCHH adjustment **6516616** (tranId **2136**) has **4 NON-serialized** items (adjustQtyBy
141
- 1800/1800/700/40) and `custbody_stock_adjustment` → Item Fulfillment **#289938** (NS internal
142
- 6507274). That fulfillment is **not yet imported** in TOGA, so the bridges stay empty until the
143
- on-demand import lands it. Import + item-sync of the adjustment itself works (4
144
- `InventoryAdjustmentItems`).
175
+ - **Non-serialized:** linked adjustment **6994071** → fulfillment **6827251** — header FK + line FK
176
+ both populate.
177
+ - **Serialized:** linked adjustment **6871592** → fulfillment **6868763** (2 serials) — header FK,
178
+ line FK, **and 2 serial units on BOTH** the adjustment item and the fulfillment item all populate
179
+ correctly.
145
180
 
146
181
  ## Gotchas / known issues
147
182
 
148
183
  - **Older adjustments have no link.** `custbody_stock_adjustment` was added in NetSuite <1 year ago;
149
- adjustments predating it will never populate a bridge — not a bug.
150
- - **The attach is best-effort by design.** It is gated NYCHH-only and wrapped in `try/catch`; a link
151
- failure logs and lets the adjustment import complete. Do not "harden" it into a throw — a bridge miss
152
- must not freeze `INVENTORY_ADJUSTMENTS`.
184
+ adjustments predating it will never populate an FK — not a bug.
185
+ - **Reading the product uuid at `depth=2` silently returns nothing** — the level-3 `item` object is
186
+ omitted at max depth. Use `depth=3` + explicit nested `fields` (see above).
187
+ - **A V2 PUT echo that omits `itemFulfillment` does NOT mean the write failed** — verify against the
188
+ DB / `Logs_<Client>.Api`, not the echoed body.
153
189
  - **Empty lookups drop or duplicate items.** The item loop filters to known warehouses and creates
154
190
  items on lookup-miss, so any manual/diagnostic invocation must seed the warehouse/item/manufacturer
155
191
  lookups (the harness does).
156
192
 
157
193
  ## Change history
158
194
 
159
- - 2026-08-28 — Built NYCHH inventory-adjustment → item-fulfillment linking off NetSuite
160
- `custbody_stock_adjustment` (internalId 9735): two all-client bridges
161
- (`ItemFulfillments_InventoryAdjustments` record 353 / `ItemFulfillmentItems_InventoryAdjustmentItems`
162
- record 354, Core-registered + Super-User/per-client ACL), a gated best-effort
163
- `App_Api_Toga2::attachInventoryAdjustmentToItemFulfillment` (resolves the fulfillment by
164
- `c_netsuiteInternalItemFulfillmentId`, idempotent header + itemId-matched line links), and **on-demand
165
- import** of a not-yet-imported fulfillment via `syncItemFulfillmentFromNetsuite` (cascades to the SO)
166
- — which expanded `syncInventoryAdjustmentFromNetsuite` 5→13 args to thread the full IF lookup set.
167
- Fixed two shared-engine bugs: the stale-item prune deleted an item before its child units (FK
168
- RESTRICT → 1451/EV-11 — now deletes units first), and the item lookup was consumed two-level while the
169
- cron populates it one-level `[$partNumber]` (re-created items every run → duplicates/EV-10 — now
170
- one-level). Added the `diagnose_hh_import_inventory_adjustment.php` operator harness. Go-live toggle
171
- `IS_ENABLED_INTEGRATION_INVENTORY_ADJUSTMENTS` still false. **Open (next session):** migrate the
172
- inventory-adjustment sync from SOAP (~2 round-trips/serial, ~1h for a large adjustment) to REST. (jcardinal)
195
+ - 2026-08-29 — **Migrated the feature to direct FK columns over REST, and made it fail loud.**
196
+ Replaced the two bridge tables with direct FK columns
197
+ (`InventoryAdjustments.itemFulfillmentId` reusing `RecordField` 2517/record 154;
198
+ `InventoryAdjustmentItems.itemFulfillmentItemId` reusing 2521/record 155 — no new Core
199
+ registration/ACL). Migrated `syncInventoryAdjustmentFromNetsuite` and its on-demand
200
+ fulfillment/sales-order import **SOAP → REST** (was ~2 round-trips/serial → ~1h for a large
201
+ serialized adjustment), and removed the error-log-and-swallow `try/catch` in the NetSuite→DB sync
202
+ path (StarTech/Sentry catches left alone). Fixed three read-path traps hit along the way: the REST
203
+ **record-sublist `limit`/`offset` 400** (`fetchItemFulfillmentById` returned 0 lines → header-only
204
+ import; now uses SuiteQL `transactionline`), the **ItemShip 3-rows-per-item** filter (keep the
205
+ `SalesOrd`/`TrnfrOrd` line-linked line, `mainline='F'` is wrong), and the **V2 max-depth FK
206
+ omission** (`depth=2` dropped `salesOrderItem.item` → now `depth=3` + explicit nested fields).
207
+ Discovered a **V2 PUT echo can omit a field that persisted** (verified via `Logs_Nychh.Api`, no
208
+ NetSuite access). Validated end-to-end in prod: non-serialized 6994071→6827251 and serialized
209
+ 6871592→6868763 (header FK, line FK, and 2 serial units on both sides). (jcardinal)
210
+ - 2026-08-28 — Built the initial NYCHH inventory-adjustment → item-fulfillment linking off NetSuite
211
+ `custbody_stock_adjustment` (internalId 9735) as **two all-client bridge tables** (header record
212
+ 353 / item record 354, Core-registered + Super-User/per-client ACL) with a gated best-effort
213
+ `App_Api_Toga2::attachInventoryAdjustmentToItemFulfillment` and **on-demand import** of a
214
+ not-yet-imported fulfillment via `syncItemFulfillmentFromNetsuite` (cascades to the SO) — which
215
+ expanded `syncInventoryAdjustmentFromNetsuite` 5→13 args to thread the full IF lookup set. Fixed the
216
+ two shared-engine bugs (prune child units before the item — FK RESTRICT/1451/EV-11; index the item
217
+ lookup one-level `[$partNumber]` — was two-level dead → duplicates/EV-10). Added the
218
+ `diagnose_hh_import_inventory_adjustment.php` operator harness. *(This bridge-table design was
219
+ superseded by the 2026-08-29 direct-FK / REST migration above.)* (jcardinal)
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "toga-ai",
3
- "version": "1.0.696",
3
+ "version": "1.0.698",
4
4
  "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
5
5
  "keywords": [
6
6
  "claude",