toga-ai 1.0.693 → 1.0.695

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -6,7 +6,7 @@ project: Worker
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-08-18
9
+ updated: 2026-08-28
10
10
  owners: ["dfranks", "bala", "jcardinal", "mhammontree", "snaredla"]
11
11
  files:
12
12
  - worker/crons/toga2/netsuite/common_sync_togasupply.php
@@ -504,8 +504,33 @@ Parameters are stored **per client DB** but accessed **through the TOGa2 API**,
504
504
  `NETSUITE_LAST_SYNC_DATETIME_*` frozen watermark), then read `Logs.Issue` — not `Logs_<Client>.Api`
505
505
  — for the throwing order.
506
506
 
507
+ - **⚠ Inventory-adjustment sync: two shared-engine bugs + it is still SOAP (2026-08-28).** Both fixed
508
+ in `syncInventoryAdjustmentFromNetsuite` (`library/app/api/toga2.php`) and reachable by any client
509
+ running inventory adjustments: (a) the stale-item prune deleted an `InventoryAdjustmentItem` before
510
+ its child `InventoryAdjustmentItemUnits` (FK **RESTRICT** → MySQL 1451 → `EV-11`) — now deletes the
511
+ child units first; (b) the item lookup was consumed **two-level**
512
+ `[$uuidManufacturer][$partNumber]` while the cron populates it **one-level** `[$partNumber]`
513
+ (`common_sync_togasupply.php` ~L313; the two-level lookup is declared but never populated — dead), so
514
+ existing items were never matched and **re-created every run** (duplicates / `EV-10`) — now indexed
515
+ one-level. Also: the section is **still SOAP** (`App_NetSuite::getInventoryNumber*` ~2 round-trips
516
+ **per serial** → ~1h for a large serialized adjustment); a REST migration via `App_Api_Netsuite_Rest`
517
+ is the planned next step. Surfaced building NYCHH's adjustment→fulfillment linking; full detail on
518
+ [NYCHH inventory-adjustment → item-fulfillment linking](../../../clients/nychh/features/netsuite-inventory-adjustment-fulfillment-link.md).
519
+ Note `syncInventoryAdjustmentFromNetsuite`'s signature was expanded **5 → 13 args** to thread the full
520
+ item-fulfillment lookup set through for the on-demand fulfillment import (lookups are passed
521
+ function-to-function, like the other sync functions).
522
+
507
523
  ## Change history
508
524
 
525
+ - 2026-08-28 — Recorded two shared-engine inventory-adjustment fixes in
526
+ `syncInventoryAdjustmentFromNetsuite` (prune child `InventoryAdjustmentItemUnits` before the item —
527
+ FK RESTRICT/1451/EV-11; index the item lookup **one-level** `[$partNumber]` to match what the cron
528
+ populates, was two-level dead → items re-created every run / duplicates / EV-10), the 5→13-arg
529
+ signature expansion that threads the full IF lookup set for on-demand fulfillment import, and that the
530
+ section is **still SOAP** (~2 round-trips/serial → ~1h large adjustment; REST migration planned).
531
+ Surfaced building NYCHH's adjustment→fulfillment linking — see
532
+ [NYCHH inventory-adjustment → item-fulfillment linking](../../../clients/nychh/features/netsuite-inventory-adjustment-fulfillment-link.md).
533
+ (jcardinal)
509
534
  - 2026-08-27 — **Removed a silent data-loss bug: all per-order `catch (Throwable)` skips and all
510
535
  section-level `try/finally` were deleted, so every section now fails LOUD.** A 2026-08-20 change had
511
536
  wrapped each order dispatch in `catch (Throwable) { error_log }` and each section in
@@ -4,7 +4,7 @@
4
4
  |-----|---------|-------|
5
5
  | [Proposed — git-sourced base+overlay JSON authoring for the Surface layer](architecture/surface-authoring-proposal.md) | A **proposal / handoff recommendation** (not implemented) that the Surface layer's *authoring* model move off hand-authored SQL against the `SurfaceOverrides` E | _underscore/Model/Core/Surface.php, _underscore/Model/Client/SurfaceOverride.php |
6
6
  | [_underscore Framework Architecture](architecture.md) | `_underscore` is the shared PHP backend framework for **all 2.0 applications**. | _underscore/_underscore.php, _underscore/Loader.php, _underscore/Framework.php, _underscore/Model.php, _underscore/Database.php, _underscore/Query.php, _underscore/Route.php, _underscore/Component.php |
7
- | [ACL Permission Chain (Record & Field Authorization)](features/acl-permission-chain.md) | Authorization in the 2.0 API is **metadata-driven**: whether a role may Create/Read/Update/Delete a record is decided by rows across **four linked tables**, not | api2/Component/Api/V2/V2.php, _underscore/Model/Core/Page.php, _underscore/Model/Core/Surface.php, _underscore/Model/Client/TrackingNumber.php, dbchanges2/Client/2026-06-03- BLANK_CLIENT_DATABASE.sql, dbchanges2/Client/2026-06-23b - ItemTranslationsAcl.sql, dbchanges2/Client/2026-07-02c - TrackingNumberNeedsReturnLabelFieldPermission.sql, dbchanges2/Client/2026-07-22c - TrackingNumberMeasureIdsFieldPermission.sql, dbchanges2/Client_Quad/2026-08-24 - Quad Multi Currency Item Pricing.sql |
7
+ | [ACL Permission Chain (Record & Field Authorization)](features/acl-permission-chain.md) | Authorization in the 2.0 API is **metadata-driven**: whether a role may Create/Read/Update/Delete a record is decided by rows across **four linked tables**, not | api2/Component/Api/V2/V2.php, dbchanges2/Core/2026-08-27a - TransferOrderUuidSearchableIdentifier.sql, _underscore/Model/Core/Page.php, _underscore/Model/Core/Surface.php, _underscore/Model/Client/TrackingNumber.php, dbchanges2/Client/2026-06-03- BLANK_CLIENT_DATABASE.sql, dbchanges2/Client/2026-06-23b - ItemTranslationsAcl.sql, dbchanges2/Client/2026-07-02c - TrackingNumberNeedsReturnLabelFieldPermission.sql, dbchanges2/Client/2026-07-22c - TrackingNumberMeasureIdsFieldPermission.sql, dbchanges2/Client_Quad/2026-08-24 - Quad Multi Currency Item Pricing.sql |
8
8
  | [Address Uniqueness Normalization (unit identifier + 5-digit ZIP comparison)](features/address-uniqueness-normalization.md) | When a business rule says *"only one X per physical address"*, comparing address rows field-for-field does **not** work: the same dwelling is spelled many diffe | _underscore/Model/Rate/Entitlement.php |
9
9
  | [Address Validation (carrier waterfall + validateAddress scripted endpoint)](features/address-validation.md) | `_Model_Client_Address::validateAddress` verifies a US address against a **carrier waterfall (USPS → FedEx → UPS)** and returns a single canonical, carrier-norm | _underscore/Model/Client/Address.php, _underscore/Component/Library/Carriers/Usps/Usps.php |
10
10
  | [_ApiRequest — JSON encode/decode & api-logging behavior](features/apirequest-json-content-type.md) | `_ApiRequest` is the 2.0 outbound HTTP client. | _underscore/ApiRequest.php |
@@ -10,6 +10,7 @@ updated: 2026-08-28
10
10
  owners: ["jcardinal", "mhammontree", "tcox", "bala"]
11
11
  files:
12
12
  - api2/Component/Api/V2/V2.php
13
+ - dbchanges2/Core/2026-08-27a - TransferOrderUuidSearchableIdentifier.sql
13
14
  - _underscore/Model/Core/Page.php
14
15
  - _underscore/Model/Core/Surface.php
15
16
  - _underscore/Model/Client/TrackingNumber.php
@@ -242,6 +243,18 @@ column + `CustomRecordFields` row + `AclCustomFieldPermissions` grant + model de
242
243
  2026-08-27 on NYCHH `c_netsuiteInternalTransferOrderStatus` (recordId 351): the field existed and was
243
244
  granted, but the nested transfer-order-stage lookup 400'd until `isIdentifier` was set to 1.
244
245
 
246
+ > **The same rule bites the base `uuid` field, not just `c_` fields.** When a nested write links its
247
+ > parent **by `uuid`** (the normal case for a `{uuid}` nested reference), that record's
248
+ > `RecordFields.uuid.isIdentifier` must be `1` or `searchableIdentifierFields` excludes `uuid` and the
249
+ > write 400s `EV-12`. Every record normally ships with `uuid.isIdentifier = 1` (e.g. SalesOrders record
250
+ > 14) — but **records 312 (`TransferOrders`) and 313 (`TransferOrderItems`) were the platform-wide
251
+ > anomaly**: their `uuid.isIdentifier` was `0`, so a nested `transferOrder:{uuid}` write on the
252
+ > transfer-order-items POST could not resolve its parent and froze the sync. Fixed platform-wide
253
+ > (`dbchanges2/Core/2026-08-27a - TransferOrderUuidSearchableIdentifier.sql`, `isIdentifier = 1` on
254
+ > records 312/313 `uuid`). When an `EV-12` mentions `searchableIdentifierFields:[id]` on a plain
255
+ > `{uuid}` nested write, check `Core.RecordFields.uuid.isIdentifier` for that record before anything
256
+ > else.
257
+
245
258
  **The `c_` declaration on a client-override model comes from its NetSuite trait.** The pattern for a
246
259
  NetSuite-synced record is `_Model_<Client>_X extends _Model_Client_X { use _Trait_Netsuite_X; }` — the
247
260
  trait is what declares the `c_` fields (so a client with only `extends`, no `use`, is missing them and
@@ -488,6 +501,11 @@ hardcoded `Core.RecordFields` id literals instead of a subselect.
488
501
  the space `Records.aclDatabase` selects — **`Core.Roles`** ids for a `CORE` record such as
489
502
  `surfaces` (`V2.php` ~L3090 / ~L6198), with `id.core.roles` derived from
490
503
  `Roles.coreRoleId` (~L1611-1627) and therefore usually just Base. (bala)
504
+ - 2026-08-28 — Extended the `isIdentifier` rule to the **base `uuid` field**: a nested `{uuid}` write
505
+ resolves its parent only if that record's `RecordFields.uuid.isIdentifier = 1`, else `EV-12`. Records
506
+ **312 (`TransferOrders`) / 313 (`TransferOrderItems`)** were the platform-wide anomaly (uuid
507
+ `isIdentifier = 0`), which froze the transfer-order-items POST; fixed platform-wide by
508
+ `dbchanges2/Core/2026-08-27a - TransferOrderUuidSearchableIdentifier.sql`. (jcardinal)
491
509
  - 2026-08-27 — Recorded two facts from wiring NYCHH transfer-order custom fields: **there is NO
492
510
  ACL/metadata cache** (`buildLookups()` re-reads per request, so a new grant/metadata row applies on
493
511
  the next request — no cache-bust exists; a whole debugging pass was wasted assuming one did), and a
@@ -7,8 +7,8 @@
7
7
  | [Multi-Client (Cross-Client) Data Retrieval](features/cross-client-data-retrieval.md) | A single authenticated V2 GET listing can return records across **many** clients (designed for 1000+) that the caller is entitled to, honoring **each target cli | api2/Component/Api/CrossClient/CrossClient.php, api2/Component/Api/V2/V2.php, api2/Controller/Index.php, _underscore/Model/Cache/Table.php, _underscore/Model/Cache/Tables/Client.php, _underscore/Model/Core/Record.php, worker2/Worker/Platform/Cache.php, worker2/Controller/Index.php, worker2/_.php, dbchanges2/Cache/2026-06-30a - MultiClientCacheTables.sql, dbchanges2/Core/2026-06-30b - CacheClusterRegistrationAndRecordTtl.sql, dbchanges2/Core/2026-07-27a - PlatformCacheCleanCron.sql |
8
8
  | [cXML ShipNotice Gateway (ASN ingestion, carrier resolution, per-client provisioning)](features/cxml-shipnotice-gateway.md) | `_Component_Api_Cxml` accepts a supplier `ShipNoticeRequest` and translates it into a `POST /v2/advance-shipping-notices` on the V2 JSON engine. | api2/Component/Api/Cxml/Cxml.php, api2/Component/Api/V2/V2.php, api2/Controller/Index.php |
9
9
  | [Encrypted-User-UUID Auth Handoff (/auth/encrypted-user-uuid)](features/encrypted-user-uuid-auth-handoff.md) | `POST /auth/encrypted-user-uuid` is the intended **cross-client / SSO-handoff identity mechanism**: given an encrypted `{client, user}` UUID pair, it mints a fr | api2/Component/Api/CrossClient/CrossClient.php |
10
- | [ENVIRONMENT (not the EB environment name) decides the _underscore branch and Config file](features/environment-variable-drives-underscore-branch.md) | An api2 Elastic Beanstalk instance decides **which `_underscore` branch it clones** and **which `Config/<env>.ini` it loads** from the EB environment property * | api2/.ebextensions/git.php, api2/.ebextensions/php_include_underscore.config, api2/.ebextensions/git.sandbox-dev.json, api2/.ebextensions/git.sandbox-client.json, api2/Config/beta.ini, api2/Config/sandbox-dev.ini, api2/Component/Api/V2/V2.php |
11
- | [Health-check endpoint (/health liveness short-circuit)](features/health-check-endpoint.md) | `_Controller_Index::api()` short-circuits **liveness/health-probe** requests to an HTTP 200 **before** any routing, DB bootstrap, or V2 engine work runs. | api2/Controller/Index.php |
10
+ | [ENVIRONMENT (not the EB environment name) decides the _underscore branch and Config file](features/environment-variable-drives-underscore-branch.md) | An api2 Elastic Beanstalk instance decides **which `_underscore` branch it clones** and **which `Config/<env>.ini` it loads** from the EB environment property * | api2/.ebextensions/git.php, api2/.platform/hooks/prebuild/git.sh, api2/.ebextensions/php_include_underscore.config, api2/.ebextensions/git.sandbox-dev.json, api2/.ebextensions/git.sandbox-client.json, api2/Config/beta.ini, api2/Config/sandbox-dev.ini, api2/Component/Api/V2/V2.php |
11
+ | [Health-check endpoint (/health liveness — static file + PHP short-circuit)](features/health-check-endpoint.md) | The EB/ALB liveness probe (`/health`) is served two ways, and the **load-bearing** one is now a **static Apache-served file**, not PHP: - **`/health` → a static | api2/health, api2/.platform/httpd/conf.d/health_probe.conf, api2/Controller/Index.php |
12
12
  | [Language Translation Layer (audience.language + sidecar tables)](features/language-translation-layer.md) | Serves the same TOGa data (Item title/description/longDescription, plus item **feature** text — `Features.name`, `ItemCategoryFeatureGroups.name`, `ItemFeatures | api2/Component/Api/V2/V2.php, api2/Component/Api/V2/Response/Response.php, _underscore/Model/Core/Setting.php, _underscore/Model/Core/RecordField.php, _underscore/Model/Core/DefaultGlobalSetting.php, _underscore/Model/Client/ItemTranslation.php, _underscore/Model/Client/FeatureTranslation.php, _underscore/Model/Client/ItemCategoryFeatureGroupTranslation.php, _underscore/Model/Client/ItemFeatureTranslation.php, dbchanges2/Client/2026-06-23a - ItemTranslations.sql, dbchanges2/Client/2026-06-23b - ItemTranslationsAcl.sql, dbchanges2/Client/2026-07-13a - FeatureTranslations.sql, dbchanges2/Client/2026-07-13b - FeatureTranslationsAcl.sql, dbchanges2/Core/2026-06-23a - RecordFieldsTranslationColumn.sql, dbchanges2/Core/2026-06-23b - ItemTranslationsRecord.sql, dbchanges2/Core/2026-07-13 - FeatureTranslationsRecord.sql |
13
13
  | [/auth/login resolves the client from the email domain, not the Bearer token (cross-client user path)](features/login-cross-client-user-resolution.md) | `POST /v2/auth/login` (email/password user login) can silently swap the target client mid-request. | api2/Component/Api/V2/V2.php, _underscore/String.php |
14
14
  | [Nested FK object embedding is gated by the CHILD record's own ACL](features/nested-fk-acl-embedding.md) | When the V2 JSON engine serializes a foreign-key field into a **nested object** (in `getFullModelData()`, ~V2.php L6016-6060), it re-checks the **child** record | api2/Component/Api/V2/V2.php, dbchanges2/Client_Compass/2026-07-23b - PurchaseOrdersRecordReadAcl.sql |
@@ -22,7 +22,7 @@
22
22
  | [TableView field/column metadata (TableViewFields, hidden projected columns)](features/tableview-field-metadata.md) | The columns of a 2.0 table view are defined by DB metadata, not code. | _underscore/Model/Client/TableView.php, api2/Component/Api/V2/V2.php, dbchanges2/Client/2026-07-20 - ItemsUuidForPurchaseOrderItemsTableView.sql, dbchanges2/Core/2026-08-17a - ItemFulfillmentQuantityFieldTypeNumber.sql, dbchanges2/Client/2026-08-17a - ItemFulfillmentColumnsCopyable.sql, dbchanges2/Client_Quad/2026-08-18c - SalesOrderListingSortByDateOrderDesc.sql, dbchanges2/Client_Quad/2026-08-24 - Quad Multi Currency Item Pricing.sql, dbchanges2/Client_Elite/2026-08-18 - InventoryUnitsItemColumns.sql, dbchanges2/Client_Nychh/2026-08-18 - InventoryUnitsItemColumns.sql, dbchanges2/Client_CompassCanada/2026-08-27a - ItemFulfillmentTableViewsRebuild.sql |
23
23
  | [Tickets API (/v2/tickets)](features/tickets-api.md) | The generic ticket endpoint of the 2.0 REST API. | Component/Api/V2/V2.php |
24
24
  | [V2 API error/message codes (EV/EZ troubleshooting map)](features/v2-api-error-codes.md) | The V2 JSON engine (`Component/Api/V2/V2.php`) returns short **message codes** in the response `error` field, grouped by family: `EN-*` authentication, `EZ-*` a | api2/Component/Api/V2/V2.php, api2/Component/Api/V2/Response/Response.php, api2/Component/Api/V2/Response/Oauth/Oauth.php, api2/Controller/Index.php, toga2-supply/src/globalTypes.ts, toga2-supply/src/pages/Orders/api/OrdersApi.ts, toga2-supply/src/api/toga.ts, _underscore/Model/Client/TrackingNumber.php |
25
- | [V2 request deadlock-retry — route-scoped in-process replay](features/v2-deadlock-retry.md) | api2's front controller can **detect a MySQL deadlock (1213) / lock-wait timeout (1205) and replay the whole request in-process**, so a transient lock collision | api2/Controller/Index.php, _underscore/Database.php, _underscore/Query.php |
25
+ | [V2 request retry (Prong A) — deadlock AND retriable unique-key race, route-scoped in-process replay](features/v2-deadlock-retry.md) | api2's front controller ("**Prong A**") can **detect a retriable concurrency failure and replay the whole request in-process**, so a transient collision self-re | api2/Controller/Index.php, api2/Component/Api/V2/V2.php, _underscore/Database.php, _underscore/Query.php, _underscore/Model/Compass/AdvanceShippingNotice.php |
26
26
  | [V2 REST query contract (params, where grammar, encoding, ACL behavior)](features/v2-rest-query-contract.md) | What an **HTTP client** has to get right to query the Toga v2 REST API: which query params are recognized, the exact `where` grammar, how the query string is (n | api2/Component/Api/V2/V2.php |
27
27
  | [V2 reverse hasMany collections must be named in the fetch fields whitelist](features/v2-reverse-hasmany-fields-whitelist.md) | In the V2 JSON engine, a **reverse hasMany** relationship — the collection of child records that foreign-key back to a parent (e.g. | api2/Component/Api/V2/V2.php |
28
28
  | [AWS CodePipeline Deployment via CodeConnections (GitHub → Elastic Beanstalk)](workflows/codepipeline-codeconnections-deploy.md) | 2.0 apps (`api2`, `_underscore`) are deployed through **AWS CodePipeline**. | api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, api2/ebs/register_instance_to_shared_application_load_balancer.php, api2/.platform/hooks/prebuild/git.sh, api2/.ebextensions/git.php, api2/.ebextensions/git.sandbox-dev.json |
@@ -6,7 +6,7 @@ project: API
6
6
  client: shared
7
7
  type: architecture
8
8
  status: active
9
- updated: 2026-08-26
9
+ updated: 2026-08-28
10
10
  owners: [jcardinal, bala, mhammontree, dfranks]
11
11
  files:
12
12
  - api2/Controller/Index.php
@@ -254,6 +254,27 @@ interceptor's plain-`Exception` rejections — all now returning 4xx/fixed. Dist
254
254
  reading the ALB 5xx metric and `Logs.Issue`: zero 5xx + a traffic spike = transient EB snapshot;
255
255
  a sustained 5xx rate = a code bug to fix.
256
256
 
257
+ **A third cause — a `Degraded`/`Severe` flip with NO stated cause is the PHP-served `/health`
258
+ probe being STARVED, not client traffic failing.** Confirmed on `api-production-1` (acct
259
+ `654654170868`): off-peak **bursts of slow, failing `POST /v2/advance-shipping-notices`**
260
+ (Compass ASN; 17–63 s each, ~100% HTTP 500) saturate the **PHP-FPM worker pool** on an
261
+ instance. The ALB `/health` check — historically **PHP-served, 5 s timeout** — then can't get a
262
+ worker and fails a few checks, so EB flags the target group "reduced health" **with no cause
263
+ cited**. Two reasons this looks invisible: (1) **health-check traffic does NOT appear in ALB
264
+ `RequestCount`/`5xx` metrics**, so the ALB view reads clean; (2) **`HealthyHostCount` never
265
+ drops**, because the probe failure is shorter than the **5-check / 75 s deregistration
266
+ threshold** — it flaps EB enhanced-health without ever deregistering a target. A separate, benign
267
+ contributor is genuine **sub-minute `/health` transients** on a single instance. Correlation
268
+ proof: a **06:00–06:08 CST ASN burst == the 11:05 UTC EB `Severe` flip**. Diagnosed via AWS CLI
269
+ (`describe-events`, `describe-configuration-settings`, CloudWatch `get-metric-data`) + prod
270
+ `Logs.Api`/`Logs.Issue`.
271
+
272
+ **Rule:** a "reduced health / no cause" flip on this 2-instance PHP env means the ALB `/health`
273
+ probe was starved of a PHP-FPM worker — do **not** chase client-traffic 5xx. **Fix shipped
274
+ (working tree):** `/health` is now served as a **static Apache file** (`api2/health`), decoupling
275
+ liveness from PHP-FPM worker availability so a slow-request burst can no longer flap it — see
276
+ [health-check endpoint](features/health-check-endpoint.md).
277
+
257
278
  ## Known issues / accepted risks
258
279
 
259
280
  Open items a maintainer should know before changing this tier. None are "bugs to fix right now" —
@@ -314,7 +335,11 @@ they are the known sharp edges. Do not re-discover these from scratch.
314
335
  class lives in `_underscore` (`Model/Client/...`).
315
336
  - **Client-specific behavior** belongs in `_Model_<ClientSlug>_X` overrides and
316
337
  `Client_ApiPayloadInterceptor` pre/post hooks — not in `V2.php`.
317
- - **Don't break `/health`** or weaken `enforce_https.conf`.
338
+ - **Don't break `/health`.** It is now a **static Apache-served file** (`api2/health`, body
339
+ `OK`) — **not** a PHP short-circuit — so the EB/ALB liveness probe is decoupled from PHP-FPM
340
+ worker availability. Do **not** route `/health` back through PHP or remove the file, and don't
341
+ weaken `enforce_https.conf`. (`/v2/health` remains PHP-handled.) See
342
+ [health-check endpoint](features/health-check-endpoint.md).
318
343
  - **504s:** verify `long_gateway_timeout.conf` on both this tier and the proxy
319
344
  (LB 3600s → proxy Apache 1800s → proxy cURL 900s → api Apache 1800s).
320
345
  - **`transactionId` uniqueness is load-bearing** — the idempotency/audit key; don't bypass
@@ -327,6 +352,7 @@ they are the known sharp edges. Do not re-discover these from scratch.
327
352
  single-caller branches in `V2.php` as unverified until exercised directly.
328
353
 
329
354
  ## Change history
355
+ - 2026-08-28 — Recorded the third EB-`Degraded` cause: a no-stated-cause "reduced health" flip on `api-production-1` is the **PHP-served `/health` probe starved of a PHP-FPM worker** by off-peak bursts of slow failing Compass ASN POSTs (17–63 s, ~100% 500) — invisible in ALB `RequestCount`/`5xx` and never dropping `HealthyHostCount` (flap < 75 s deregistration). Correlated a 06:00–06:08 CST ASN burst to the 11:05 UTC Severe flip. Noted the fix (static-file `/health` decouples liveness from PHP-FPM) and updated the "Don't break `/health`" rule to the static-file form. (jcardinal)
330
356
  - 2026-08-26 — Documented and fixed a `/v2/auth/public` OOM: a present-but-null client `transactionId` (`{"transactionId": null}`) flowed into the request-logger's `_Model::load()` dup-check; because `Logs.Api.transactionId` is `FIELD_CHAR`, a null value collapses the WHERE and full-table-scans prod `Logs.Api` → memory exhaustion (the `_underscore` collapsed-WHERE mechanism). V2 only auto-uuids an *absent* transactionId (~L543), so a present null fell through (~L553 read → null). Fix (commit `36ef31f`): coerce null/empty transactionId → uuid in the auth block, and skip `load()` in the final logger dup-check (~L2277) when null/empty. Added the "don't blame the latest deploy — check first-occurrence" triage note. (jcardinal)
331
357
  - 2026-08-25 — **Resolved the auto-generated `Api.transactionId` 1062 collision** (was the 2026-07-28
332
358
  gotcha): the two DB-insert log sites now uuid the log-row `transactionId` (`_String::generateUuid()`)
@@ -6,10 +6,11 @@ project: API
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-08-25
10
- owners: ["bala", "mhammontree", "apeterson"]
9
+ updated: 2026-08-28
10
+ owners: ["bala", "mhammontree", "apeterson", "jcardinal"]
11
11
  files:
12
12
  - api2/.ebextensions/git.php
13
+ - api2/.platform/hooks/prebuild/git.sh
13
14
  - api2/.ebextensions/php_include_underscore.config
14
15
  - api2/.ebextensions/git.sandbox-dev.json
15
16
  - api2/.ebextensions/git.sandbox-client.json
@@ -177,6 +178,21 @@ the tier trades a stale-code bug for a total outage. Full procedure:
177
178
 
178
179
  ## Gotchas / known issues
179
180
 
181
+ - **⚠ Cross-repo change ordering: land `_underscore` BEFORE deploying api2.** Because api2's EB
182
+ deploy **clones `_underscore` (branch `_<ENVIRONMENT>`, e.g. `_production`) at BUILD time**
183
+ (`.platform/hooks/prebuild/git.sh`), a change that spans both repos must have its `_underscore`
184
+ side merged to `_<ENVIRONMENT>` **first** — otherwise the freshly-built api2 will call a framework
185
+ method/class that isn't on the box and fatal. Landing `_underscore` first is **safe on its own**:
186
+ the old api2 controller simply doesn't use the new framework code yet. Order: merge `_underscore`
187
+ → (verify it's on the branch) → deploy api2.
188
+ - **⚠ A "Could not find required file for `_Model_...`" fatal on prod can be a STALE-BUILD
189
+ artifact, not a missing file.** `Logs.Issue #476`
190
+ ("Could not find required file for '_Model_Client_ItemFulfillments_InventoryAdjustment'") fired
191
+ because that model file **was** committed to `_production` but `api-production-1` was still running
192
+ an **older build**; a **redeploy** (which re-pulls `_production`) resolved it with no recurrence.
193
+ Before assuming a model file is absent, check the **running build's `_underscore` against
194
+ `_production`** (file presence on the box; `git merge-base --is-ancestor`) — the build-time clone
195
+ means a merged file may simply not be on that instance yet.
180
196
  - **⚠ Merging to the branch named after the EB environment can be a no-op.** Merge to
181
197
  `_` + `ENVIRONMENT`. If your change "deployed successfully" but the behavior is absent and no
182
198
  error appears, check this before debugging the code. A missing class also throws `Error`
@@ -241,6 +257,14 @@ the tier trades a stale-code bug for a total outage. Full procedure:
241
257
 
242
258
  ## Change history
243
259
 
260
+ - 2026-08-28 — Recorded two build-time-clone consequences: (1) **cross-repo deploy ordering** — api2
261
+ clones `_underscore` (`_<ENVIRONMENT>`) at build via `.platform/hooks/prebuild/git.sh`, so for a
262
+ change spanning both repos, `_underscore` must land on the deploy branch **before** api2 is
263
+ deployed (landing `_underscore` first is safe alone); and (2) a **"Could not find required file
264
+ for `_Model_...`" fatal can be a stale build**, not a missing file — `Logs.Issue #476` fired for
265
+ a model committed to `_production` while `api-production-1` ran an older build, and a redeploy
266
+ resolved it. Check the running build's `_underscore` vs `_production` before assuming the file is
267
+ absent. (jcardinal)
244
268
  - 2026-08-27 — Recorded the **inverse hazard**: moving a tier **forward** onto its correct
245
269
  `_underscore` branch is a **schema event**. The 2026-08-26 deploy moved `sandbox-client` off the
246
270
  `_production` framework (which the 2026-08-25 entry below explains it had been running for weeks)
@@ -1,14 +1,16 @@
1
1
  ---
2
- title: "Health-check endpoint (/health liveness short-circuit)"
2
+ title: "Health-check endpoint (/health liveness — static file + PHP short-circuit)"
3
3
  framework: "2.0"
4
4
  repo: api2
5
5
  project: API
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-07-24
9
+ updated: 2026-08-28
10
10
  owners: [jcardinal]
11
11
  files:
12
+ - api2/health
13
+ - api2/.platform/httpd/conf.d/health_probe.conf
12
14
  - api2/Controller/Index.php
13
15
  related:
14
16
  - ../architecture.md
@@ -17,13 +19,37 @@ related:
17
19
 
18
20
  ## Summary
19
21
 
20
- `_Controller_Index::api()` short-circuits **liveness/health-probe** requests to an HTTP 200
21
- **before** any routing, DB bootstrap, or V2 engine work runs. EB/LB (Elastic Beanstalk /
22
- load balancer) probes depend on this always returning 200 cheaply — the architecture doc's
23
- critical rule is literally "Don't break `/health`". This doc records the **matching contract**
24
- so a teammate doesn't reintroduce a brittle exact-string check.
22
+ The EB/ALB liveness probe (`/health`) is served two ways, and the **load-bearing** one is now
23
+ a **static Apache-served file**, not PHP:
25
24
 
26
- ## How it works
25
+ - **`/health` → a static file `api2/health`** (literal body `OK`), served by Apache **without
26
+ ever reaching PHP-FPM**. This is what the EB target group health-check hits. Decoupling the
27
+ probe from PHP means a PHP-FPM worker-pool exhaustion (e.g. a burst of slow ASN requests
28
+ saturating all workers) or a PHP transient can **no longer flap EB health** — the probe
29
+ answers from Apache regardless of app load.
30
+ - **`/v2/health` (and `/health` as a fallback) → PHP short-circuit** in
31
+ `_Controller_Index::api()`, an HTTP 200 returned **before** any routing, DB bootstrap, or V2
32
+ engine work. This still exists for callers that hit the PHP entrypoint directly.
33
+
34
+ This doc records the **matching contract** for both so a teammate doesn't reintroduce a brittle
35
+ exact-string check or accidentally route the liveness probe back through PHP.
36
+
37
+ ## The static probe (the one EB actually uses) — 2026-08-28
38
+
39
+ - **`api2/health`** — a one-line static file (`OK`). Apache serves it directly.
40
+ - **`api2/.platform/httpd/conf.d/health_probe.conf`** — comment-only documentation of why the
41
+ probe is static (no directives; it exists to explain the decision at the Apache layer).
42
+ - **Why it works without a rewrite fight:** `api2/.htaccess` only rewrites to `index.php` when
43
+ the request target is **not a real file** (`RewriteCond %{REQUEST_FILENAME} !-f`). Because
44
+ `health` is a real file on disk, the rewrite is skipped and Apache serves it straight.
45
+ - **Why `enforce_https` doesn't redirect it:** EB health checks arrive on **HTTP:80 with no
46
+ `X-Forwarded-Proto`**, and `enforce_https.conf` only redirects requests that carry the
47
+ forwarded-proto marker — so the probe is not bounced to HTTPS.
48
+ - **Do not** move `/health` back into a PHP-served route or rename/remove `api2/health` — that
49
+ re-couples EB liveness to PHP-FPM worker availability, which is the exact failure this fixed
50
+ (see [api2 architecture — EB "Degraded" health](../architecture.md)).
51
+
52
+ ## PHP short-circuit (for `/v2/health` and direct hits)
27
53
 
28
54
  The short-circuit lives at the top of `Index.php::api()`, before host dispatch and before the
29
55
  Core/Logs DB bootstrap. It matches against a class constant:
@@ -55,6 +81,15 @@ normalized form: lowercase, no trailing slash, no query) rather than re-adding a
55
81
  branch.
56
82
 
57
83
  ## Change history
84
+ - 2026-08-28 — Moved the **EB/ALB liveness probe off PHP**: `/health` is now a **static file**
85
+ (`api2/health`, body `OK`) served by Apache — it never reaches PHP-FPM, so a slow-request burst
86
+ or a PHP transient can no longer starve the probe and flap EB health. Works because
87
+ `.htaccess` only rewrites to `index.php` for non-files (`!-f`), and `enforce_https` skips the
88
+ probe (HTTP:80, no `X-Forwarded-Proto`). Added a comment-only
89
+ `.platform/httpd/conf.d/health_probe.conf` documenting the decision. `/v2/health` stays
90
+ PHP-handled and the `HEALTH_CHECK_ROUTES` PHP short-circuit remains as a fallback. Working tree
91
+ only (uncommitted). See the [EB "Degraded" health](../architecture.md) diagnosis this fixes.
92
+ (jcardinal)
58
93
  - 2026-07-24 — Replaced the brittle exact-string `REQUEST_URI == '/health'` check with a
59
94
  `HEALTH_CHECK_ROUTES` constant + normalized (lowercase / strip query / strip trailing slash)
60
95
  strict `in_array` match, so `/v2/health`, `/health/`, and `/health?x` all short-circuit to 200
@@ -1,38 +1,101 @@
1
1
  ---
2
- title: V2 request deadlock-retry — route-scoped in-process replay
2
+ title: V2 request retry (Prong A) — deadlock AND retriable unique-key race, route-scoped in-process replay
3
3
  framework: "2.0"
4
4
  repo: api2
5
5
  project: API
6
6
  client: shared
7
7
  type: feature
8
8
  status: draft
9
- updated: 2026-08-13
9
+ updated: 2026-08-28
10
10
  owners: [jcardinal]
11
11
  files:
12
12
  - api2/Controller/Index.php
13
+ - api2/Component/Api/V2/V2.php
13
14
  - _underscore/Database.php
14
15
  - _underscore/Query.php
16
+ - _underscore/Model/Compass/AdvanceShippingNotice.php
15
17
  related:
16
18
  - ../../_underscore/features/model-save-parent-cascade-stored-field-deadlock.md
19
+ - ../../../../clients/compass-usa/features/asn-to-item-fulfillment.md
17
20
  - ../architecture.md
18
21
  - ./v2-api-error-codes.md
19
22
  ---
20
23
 
21
24
  ## Summary
22
25
 
23
- api2's front controller can **detect a MySQL deadlock (1213) / lock-wait timeout (1205) and replay
24
- the whole request in-process**, so a transient lock collision self-recovers instead of returning an
25
- HTTP 400 to the caller. The replay is **route-scoped by an allowlist**
26
- (`DEADLOCK_RETRIABLE_ROUTES`) — currently only `/v2/advance-shipping-notices` — because whole-
27
- request in-process replay is **only safe for routes with no non-rollbackable external side effect**.
28
- This is the safety-net half of the ASN deadlock fix; the source-level fix (stop the collateral
29
- parent re-save) is in
30
- [_underscore's save-cascade doc](../../_underscore/features/model-save-parent-cascade-stored-field-deadlock.md).
26
+ api2's front controller ("**Prong A**") can **detect a retriable concurrency failure and replay
27
+ the whole request in-process**, so a transient collision self-recovers instead of returning an
28
+ error to the caller. Two failure classes now qualify:
31
29
 
32
- > Status (2026-08-13): **in the working tree only** — not committed or deployed. Pending sandbox
30
+ 1. **MySQL deadlock (1213) / lock-wait timeout (1205)** — the original case (a same-PO ASN write
31
+ deadlock; see the
32
+ [save-cascade doc](../../_underscore/features/model-save-parent-cascade-stored-field-deadlock.md)).
33
+ 2. **Retriable unique-key get-or-create race (1062)** — a concurrent same-key INSERT lost the
34
+ race; a whole-request replay on a fresh snapshot lets the retry's GET reuse the winner's
35
+ committed row (added 2026-08-28 for the Compass ASN data-loss race).
36
+
37
+ The replay is **route-scoped by an allowlist** (`DEADLOCK_RETRIABLE_ROUTES`) — currently only
38
+ `/v2/advance-shipping-notices` — because whole-request in-process replay is **only safe for routes
39
+ with no non-rollbackable external side effect**. The retry gate is now
40
+ **`wasDeadlock() || wasRetriableUniqueRace()`**.
41
+
42
+ > Status (2026-08-28): **in the working tree only** — not committed or deployed. Pending sandbox
33
43
  > validation + regression tests.
34
44
 
35
- ## How it works
45
+ ## The retriable unique-key race (1062) — Compass ASN data-loss (2026-08-28)
46
+
47
+ `_Model_Compass_AdvanceShippingNotice::postPost` **get-or-creates** an SO-numbered Item
48
+ Fulfillment (GET by number, else POST). `ItemFulfillments.number` is UNIQUE and vendors
49
+ re-transmit the same shipment file, so two concurrent same-number ASNs race: the loser's INSERT
50
+ trips the unique key. Pre-fix, `internalApiRequest` threw (default `throwExceptionsOnError=true`),
51
+ the 500 propagated out of `postPost`, and **the api2 controller rolled back the whole request —
52
+ losing the ASN row and its tracking numbers** (`Logs.Issue #239`, EV-10). This is a bug 1062
53
+ must NOT be treated like a normal error, because a replay recovers it cleanly.
54
+
55
+ **Fix (Approach B — a CTO review rejected a simpler in-place `FOR UPDATE` recovery, because an
56
+ interceptor cannot recover a poisoned response in place; see *V2 concurrency mechanics* below).**
57
+ A generalized retriable-unique-race path:
58
+
59
+ - **`_Database`** gains `const ERRNO_DUPLICATE_ENTRY = 1062`, a sticky
60
+ `public static bool $retriableUniqueRace` (reset per attempt in `clearLastError()`), and
61
+ `wasRetriableUniqueRace()` — true **only** when the caller-set flag is set **AND**
62
+ `lastErrorNumber === 1062`. The flag is caller-opt-in on purpose: a bare 1062 is a real
63
+ constraint violation and must **not** silently retry; only a caller that knows the 1062 is a
64
+ benign get-or-create race sets it.
65
+ - **The ASN model** calls the IF-create `internalApiRequest` with **`throwExceptionsOnError:false`**;
66
+ on an empty response with `lastErrorNumber === 1062` it sets `$retriableUniqueRace = true` and
67
+ returns null. `postPost` then **aborts immediately** (issues no further query).
68
+ - **The controller's Prong A gate** becomes `wasDeadlock() || wasRetriableUniqueRace()`. Replaying
69
+ the whole request on a fresh REPEATABLE-READ snapshot lets the retry's GET see the winner's
70
+ committed IF and reuse it.
71
+
72
+ Bounded by `MAX_DEADLOCK_ATTEMPTS = 3`, gated to `DEADLOCK_RETRIABLE_ROUTES =
73
+ ['/v2/advance-shipping-notices']`.
74
+
75
+ ## V2 concurrency mechanics (durable framework knowledge)
76
+
77
+ These are the V2-engine facts that decide **how any interceptor-level write race must be handled**
78
+ — they are why the fix is a whole-request replay and not an in-place recovery:
79
+
80
+ - **Response poisoning (load-bearing).** A failed nested save via `internalApiRequest` calls
81
+ `addDefinedMessage(EV-10)` on the **shared** `$api->response`, setting a non-2xx status **before**
82
+ control returns. So even if an interceptor catches the throw and "recovers," `execute()`
83
+ recomputes `isSuccess = false` (`V2.php` ~2154) and **self-rolls-back DB_CORE + DB_CLIENT**
84
+ (~2157-2159). **An interceptor therefore CANNOT recover a failed internal save in place** — the
85
+ only correct recovery is a whole-request replay. This same poisoning is what drives the Prong A
86
+ retry gate and, on exhaustion, produces the coherent EV-10 response.
87
+ - **`internalApiRequest($method, $route, $payload, $options, $throwExceptionsOnError = true)`** — a
88
+ **5th argument** controls throw-vs-return-on-error. Pass `false` to inspect `lastErrorNumber`
89
+ instead of unwinding.
90
+ - **`_Database::$lastErrorNumber` is sticky.** Set only on query failure, never cleared by a later
91
+ success; reset per attempt via `clearLastError()`. A recorded **deadlock (1213/1205) is protected
92
+ from overwrite** by a later error; a **1062 is NOT** protected — so read/act on a 1062 promptly.
93
+ - **The prod client cluster runs REPEATABLE-READ.** A same-transaction re-SELECT after a losing
94
+ concurrent INSERT is **blind to the winner's commit** — only a locking read or a **fresh-snapshot
95
+ whole-request replay** sees it. This is why in-place retry inside the transaction cannot work and
96
+ the replay must start a new transaction.
97
+
98
+ ## How it works — deadlock path
36
99
 
37
100
  **Detection primitive (`_underscore`, reusable).** `_Database` gains
38
101
  `public static ?int $lastErrorNumber`, the consts `ERRNO_DEADLOCK` (1213) and
@@ -42,7 +105,8 @@ driver errno **at its throw site**, and the recording is **sticky**: a later unr
42
105
  inside a model-save `try/catch` still be visible to the outer controller and trigger a retry.
43
106
 
44
107
  **Retry loop (`api2/Controller/Index.php`).** The v2 dispatch is wrapped in a bounded loop
45
- (`MAX_DEADLOCK_ATTEMPTS = 3`). On an unsuccessful response where `_Database::wasDeadlock()` is true
108
+ (`MAX_DEADLOCK_ATTEMPTS = 3`). On an unsuccessful response where
109
+ `_Database::wasDeadlock() || _Database::wasRetriableUniqueRace()` is true
46
110
  and the route is in `DEADLOCK_RETRIABLE_ROUTES`, the controller:
47
111
 
48
112
  1. rolls back all open transactions;
@@ -82,6 +146,22 @@ superglobal. Default is to leave a route off the list.
82
146
  not a substitute for removing the deadlock at its source.
83
147
 
84
148
  ## Change history
149
+ - 2026-08-28 — **Generalized Prong A to a second retriable class: the unique-key (1062)
150
+ get-or-create race**, fixing the Compass ASN data-loss bug (`Logs.Issue #239`, EV-10) where a
151
+ concurrent same-`ItemFulfillments.number` INSERT lost the race, `internalApiRequest` threw, and
152
+ the whole request rolled back — losing the ASN row and its tracking. Added
153
+ `_Database::ERRNO_DUPLICATE_ENTRY = 1062`, a sticky caller-opt-in `$retriableUniqueRace`
154
+ (reset in `clearLastError()`), and `wasRetriableUniqueRace()` (flag AND `lastErrorNumber===1062`);
155
+ the ASN model calls the IF-create with `throwExceptionsOnError:false`, sets the flag on an empty
156
+ 1062 response, and aborts; the controller gate became
157
+ `wasDeadlock() || wasRetriableUniqueRace()`. Chose the whole-request replay (Approach B) over an
158
+ in-place `FOR UPDATE` recovery after a CTO review, because V2 **response poisoning** (a failed
159
+ nested save flips the shared `$api->response` to non-2xx and `execute()` self-rolls-back
160
+ DB_CORE+DB_CLIENT, `V2.php` ~2154-2159) makes in-place interceptor recovery impossible, and the
161
+ prod cluster's REPEATABLE-READ isolation makes a same-transaction re-SELECT blind to the winner's
162
+ commit. Recorded the durable V2 concurrency mechanics (response poisoning, `internalApiRequest`
163
+ 5th arg, sticky-`lastErrorNumber` with 1213 protected / 1062 not, REPEATABLE-READ). Working tree
164
+ only, pending sandbox validation. (jcardinal)
85
165
  - 2026-08-13 — Added deadlock/lock-wait detection to `_Database`/`_Query` (`$lastErrorNumber`,
86
166
  `wasDeadlock()`, sticky errno recording) and a bounded, route-scoped in-process replay loop in
87
167
  `Controller/Index.php`, gated to `/v2/advance-shipping-notices` only. Documented why whole-request
@@ -223,6 +223,31 @@ corresponding committed file — are in
223
223
  [surface-layer-schema](../apps/dbchanges2/features/surface-layer-schema.md) and
224
224
  [surface-resolver](../apps/_underscore/features/surface-resolver.md).
225
225
 
226
+ ### Standard column ordering — `id`, `uuid`, `dtCreated`, `dtUpdated`, then the rest
227
+
228
+ Every table uses a fixed leading column order: `id` (primary key), then `uuid`, then **`dtCreated`
229
+ immediately after `uuid`**, then **`dtUpdated` immediately after `dtCreated`**, and the business
230
+ columns after that. This is a hard, fundamental team convention — it keeps the audit columns in the
231
+ same, predictable place on every table platform-wide.
232
+
233
+ - **New tables (`CREATE TABLE`):** declare the columns in that order.
234
+ - **Adding the timestamps to an existing table (`ALTER TABLE`):** position them explicitly with
235
+ `AFTER`, because a bare `ADD COLUMN` appends to the **end** of the table and violates this standard:
236
+
237
+ ```sql
238
+ ALTER TABLE MyTable
239
+ ADD COLUMN dtCreated DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP AFTER uuid,
240
+ ADD COLUMN dtUpdated DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP AFTER dtCreated;
241
+ ```
242
+
243
+ To reposition timestamps already appended at the end of an existing table, use
244
+ `MODIFY COLUMN dtCreated ... AFTER uuid` / `MODIFY COLUMN dtUpdated ... AFTER dtCreated`.
245
+
246
+ `dtCreated` / `dtUpdated` are `DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP` (with `dtUpdated` also
247
+ `ON UPDATE CURRENT_TIMESTAMP`), matching the model's `FIELD_DATETIME_CREATED` /
248
+ `FIELD_DATETIME_UPDATED` field types. Bridge tables that carry no timestamps (only `id`, `uuid`, and
249
+ the two FK columns) are exempt — the rule is about *where* the timestamps go when a table has them.
250
+
226
251
  ## Checking dependencies before touching shared code
227
252
 
228
253
  `dependsOn` in `knowledge/registry.json` means a repo extends or depends on another repo's classes. Before modifying a class in a dependency repo (e.g. `_underscore` core):
@@ -308,6 +333,11 @@ ACL-governed field treatment.
308
333
  did not write them back, so current state must be established with read-only queries against the
309
334
  target environment, and the cluster-isolation hook catches cross-database references but
310
335
  **cannot** catch state drift. (apeterson)
336
+ - 2026-08-28 — Documented the **standard leading column order** (`id`, `uuid`, then `dtCreated`
337
+ immediately after `uuid`, then `dtUpdated` immediately after `dtCreated`, then business columns),
338
+ including using `AFTER` on `ALTER TABLE ADD COLUMN` so timestamps are positioned rather than
339
+ appended. Prompted by the platform-wide `TransferOrderItems` drift where the audit columns were
340
+ missing entirely. (jcardinal)
311
341
  - 2026-07-29 — Documented the `c_`-prefixed framework-dynamic column convention (schema-only,
312
342
  no model-class/RecordFields/ACL declaration), generalized from the transcript AI-model routing
313
343
  work. (ajean)
@@ -4,7 +4,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
4
4
 
5
5
  ## 1.0 framework
6
6
 
7
- - **library** (Library) _(framework core)_ — 20 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
7
+ - **library** (Library) _(framework core)_ — 21 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
8
8
  - **worker** (Worker) — 31 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
9
9
  - **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
10
10
  - **worker1.5** (Worker 1.5) — 0 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
@@ -5,7 +5,7 @@ project: _Underscore
5
5
  client: compass-usa
6
6
  type: client-feature
7
7
  status: active
8
- updated: 2026-08-27
8
+ updated: 2026-08-28
9
9
  owners: [jcardinal, bala]
10
10
  files:
11
11
  - _underscore/Model/Compass/AdvanceShippingNotice.php
@@ -200,6 +200,20 @@ Compass Canada (`Model/Compass/Canada/`) is a separate sub-client. The Compass c
200
200
  (`$_model_skipParentCascadeOnSave`), and api2 retries the ASN route on a detected deadlock. Full
201
201
  mechanic:
202
202
  [save-cascade stored-field deadlock](../../../2.0/apps/_underscore/features/model-save-parent-cascade-stored-field-deadlock.md).
203
+ - **Concurrent same-SO-number ASNs lost the ASN + its tracking to a unique-key race (fixed
204
+ 2026-08-28, working tree).** `postPost` get-or-creates the SO-numbered IF (GET by number, else
205
+ POST). `ItemFulfillments.number` is UNIQUE and vendors re-transmit the same shipment file, so two
206
+ concurrent same-number ASNs race: the loser's IF INSERT hits the unique key (MySQL **1062**),
207
+ `internalApiRequest` threw (default `throwExceptionsOnError=true`), the 500 propagated out of
208
+ `postPost`, and the api2 controller **rolled back the whole request — losing the ASN row AND its
209
+ tracking numbers** (`Logs.Issue #239`, EV-10). Fix: the IF-create call now passes
210
+ `throwExceptionsOnError:false`; on an empty response with `lastErrorNumber===1062` the model sets
211
+ `_Database::$retriableUniqueRace = true` and returns null, `postPost` aborts, and api2's Prong A
212
+ replays the whole request on a fresh REPEATABLE-READ snapshot so the retry's GET reuses the
213
+ winner's committed IF. Distinct from the 1213 deadlock above (a different concurrency failure on
214
+ the same re-transmission traffic). Full mechanic + why in-place recovery is impossible (V2
215
+ response poisoning):
216
+ [V2 request retry — Prong A](../../../2.0/apps/api2/features/v2-deadlock-retry.md).
203
217
  - **Cross-line tracking contamination from bad SOI↔POI bridge rows (root cause, fixed
204
218
  forward 2026-06-11):** `postPost` resolves which IFI(s) a tracking number attaches to by
205
219
  joining `AdvanceShippingNoticeItems → SalesOrderItems_PurchaseOrderItems → SalesOrderItems`.
@@ -313,6 +327,13 @@ feed cannot currently support a line-level completeness test.
313
327
 
314
328
  ## Change history
315
329
  Dated one-liners, newest first.
330
+ - 2026-08-28 — Fixed a **data-loss unique-key race** on concurrent same-SO-number ASNs: the loser's
331
+ SO-numbered IF INSERT trips `ItemFulfillments.number` UNIQUE (1062), `internalApiRequest` threw,
332
+ and api2 rolled back the whole request, losing the ASN row + tracking (`Logs.Issue #239`, EV-10).
333
+ The model now creates the IF with `throwExceptionsOnError:false`, flags a retriable 1062
334
+ (`_Database::$retriableUniqueRace`) and aborts, and api2's Prong A replays the request on a fresh
335
+ snapshot so the retry reuses the winner's IF. Distinct from the 2026-08-13 1213 deadlock. Working
336
+ tree only; also swept 6 banned `throw new _Exception(...)` → `\Exception` in the file. (jcardinal)
316
337
  - 2026-08-27 — Quantified the **Office Depot duplicate-ASN** problem on prod and settled what it is:
317
338
  of ODP PO lines with more than one shipping notice, **10,012 exceed the PO line quantity and only 8
318
339
  are genuine split shipments**, so a second notice on an ODP line is a **duplicate**, not a partial.
@@ -5,6 +5,7 @@ apps:
5
5
  - _underscore
6
6
  - api2
7
7
  - worker2
8
+ - worker
8
9
  - toga2-supply
9
10
  - dbchanges2
10
11
  - library
@@ -12,8 +13,8 @@ project: _Underscore
12
13
  client: growrk
13
14
  type: profile
14
15
  status: active
15
- updated: 2026-08-10
16
- owners: ["rgirish", "mhammontree", "bala"]
16
+ updated: 2026-08-28
17
+ owners: ["rgirish", "mhammontree", "bala", "jcardinal"]
17
18
  files: []
18
19
  related:
19
20
  - clients/growrk/features/transfer-order-flow.md
@@ -56,6 +57,13 @@ module (`dbchanges2/Client_Growrk/_modules.txt`).
56
57
  2026-08-01, but it fails `POST /item-fulfillments` with **EV-12** and makes the Fulfill & Ship
57
58
  ship-to address arbitrary (`salesOrders[0]`). Needs its own ticket — see the
58
59
  [Fulfill & Ship gotchas](../../2.0/apps/toga2-supply/features/fulfill-and-ship.md).
60
+ - **GroWrk runs the shared 1.0 NetSuite→TOGa Supply sync (`worker`/`library`), and its transfer
61
+ orders share the transfer-order code path.** The 2026-08-28 `SALES_ORDERS` freeze on
62
+ `syncTransferOrderFromNetsuite` (the api2-omits-a-null-FK `$transferOrderStage` warning →
63
+ fatal, plus the platform-wide `TransferOrderItems` missing-timestamp-columns 1054) hit GroWrk too,
64
+ not just NYCHH; both were fixed in shared code. See
65
+ [NYCHH NetSuite → TransferOrders import](../nychh/features/netsuite-transfer-order-import.md) (freeze
66
+ chain) — the fixes are shared, only the routing gate is NYCHH-specific.
59
67
  - **`ShippingMethods` ids 11/12 ("Overnight Standard"/"Overnight Priority") have NULL `code`** — FedEx
60
68
  product names on UPS rows, duplicating Next Day Air. Selecting either fails at UPS (120500);
61
69
  awaiting a data-owner decision. Ids 3/4 were coded by
@@ -2,7 +2,8 @@
2
2
 
3
3
  | Doc | Framework | Summary | Files |
4
4
  |-----|-----------|---------|-------|
5
- | [NYCHH NetSuite → TransferOrders import (stocking-flag routing, PO bridges, origin location)](features/netsuite-transfer-order-import.md) | 1.0 | NYCHH's NetSuite **transfer orders are represented as NetSuite sales orders** (same as the GroWrk pattern), so the shared NetSuite → TOGa Supply importer (`App_ | library/app/api/toga2.php, library/app/api/netsuite/rest.php, library/app/netsuite.php, worker/crons/toga2/netsuite/sync_togasupply_hh.php, worker/crons/toga2/netsuite/common_sync_togasupply.php |
5
+ | [NYCHH inventory-adjustment → item-fulfillment linking (custbody_stock_adjustment bridges)](features/netsuite-inventory-adjustment-fulfillment-link.md) | 1.0 | For NYCHH stock adjustments, the NetSuite **inventory adjustment** carries a link to the **Item Fulfillment** it corrects, on the multi-select body custom field | library/app/api/toga2.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_hh.php, worker/crons/toga2/netsuite/diagnose_hh_import_inventory_adjustment.php, _underscore/Model/Client/ItemFulfillments/InventoryAdjustment.php, _underscore/Model/Client/ItemFulfillmentItems/InventoryAdjustmentItem.php, dbchanges2/Core/2026-08-27b - ItemFulfillmentsInventoryAdjustmentsBridge.sql, dbchanges2/Core/2026-08-28b - ItemFulfillmentItemsInventoryAdjustmentItemsBridge.sql, dbchanges2/Client/2026-08-27a - ItemFulfillmentsInventoryAdjustmentsBridgeTable.sql, dbchanges2/Client/2026-08-28c - ItemFulfillmentItemsInventoryAdjustmentItemsBridgeTable.sql |
6
+ | [NYCHH NetSuite → TransferOrders import (stocking-flag routing, PO bridges, origin location)](features/netsuite-transfer-order-import.md) | 1.0 | NYCHH's NetSuite **transfer orders are represented as NetSuite sales orders** (same as the GroWrk pattern), so the shared NetSuite → TOGa Supply importer (`App_ | library/app/api/toga2.php, library/app/api/netsuite/rest.php, library/app/netsuite.php, worker/crons/toga2/netsuite/sync_togasupply_hh.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/diagnose_hh_stuck_transfer_order.php, dbchanges2/Core/2026-08-27a - TransferOrderUuidSearchableIdentifier.sql, dbchanges2/Client/2026-08-28a - TransferOrderItemsTimestampColumns.sql, dbchanges2/Client/2026-08-28b - TransferOrderItemsTimestampColumnsReposition.sql |
6
7
  | [NYCHH PO links are UPSTREAM — the downstream SO→PO route returns empty](features/po-number-upstream-direction.md) | 2.0 | NYCHH's sales orders are created **from the customer's purchase order**, so their SO↔PO links live in the **upstream** table `PurchaseOrders_SalesOrders` (route | _underscore/Model/Client/SalesOrder.php, _underscore/Trait/Netsuite/SalesOrder.php, toga25-supply/src/pages/SalesOrders/view/SalesOrderRecordModalLayout/hooks/usePurchaseOrderDetails.ts, dbchanges2/Client/2026-08-11b - SalesOrderPurchaseOrdersField.sql, dbchanges2/Client/2026-08-25 - SalesOrderPurchaseOrdersFieldAllClients.sql |
7
8
  | [NYCHH transfer-order 2.0 model — inventory quantities + V2 field enablement](features/transfer-order-inventory-quantities.md) | 2.0 | The 2.0 (`_underscore`) side of NYCHH transfer-order support: a **two-branch inventory quantity model** on the NYCHH `Item` override, plus the **client-override | _underscore/Model/Nychh/Item.php, _underscore/Model/Nychh/TransferOrder.php, _underscore/Model/Nychh/TransferOrderStage.php, _underscore/Model/Client/TransferOrder.php, _underscore/Model/Client/TransferOrderItem.php, dbchanges2/Client_Nychh/2026-08-27a - TransferOrderCustomFieldWritePermission.sql, dbchanges2/Client_Nychh/2026-08-27b - TransferOrderStageStatusIdentifier.sql, dbchanges2/Client_Nychh/2026-08-28c - TransferOrderRecordAcl.sql, dbchanges2/Client/2026-08-28b - TransferOrderItemsTimestamps.sql |
8
9
  | [NYC Health & Hospitals](profile.md) | 2.0 | NYC Health & Hospitals (NYCHH) is a TOGA 2.0 client on the `_underscore` platform, prod schema `Client_Nychh`. | dbchanges2/Client_Nychh/2026-08-18 - InventoryUnitsItemColumns.sql, dbchanges2/Client_Nychh/2026-08-24 - FixInventoryGroupingsUnitsTopologyOverrideIds.sql, dbchanges2/Client_Nychh/2026-08-27a - TransferOrderCustomFieldWritePermission.sql, dbchanges2/Client_Nychh/2026-08-27b - TransferOrderStageStatusIdentifier.sql, dbchanges2/Client_Nychh/2026-08-28a - TransferOrdersTableView.sql, dbchanges2/Client_Nychh/2026-08-28b - TransferOrdersNavigationEnable.sql, dbchanges2/Client_Nychh/2026-08-28c - TransferOrderRecordAcl.sql, dbchanges2/Client/2026-08-28b - TransferOrderItemsTimestamps.sql |
@@ -0,0 +1,172 @@
1
+ ---
2
+ title: "NYCHH inventory-adjustment → item-fulfillment linking (custbody_stock_adjustment bridges)"
3
+ framework: "1.0"
4
+ repo: library
5
+ project: Library
6
+ client: nychh
7
+ type: client-feature
8
+ status: active
9
+ updated: 2026-08-28
10
+ owners: [jcardinal]
11
+ files:
12
+ - library/app/api/toga2.php
13
+ - worker/crons/toga2/netsuite/common_sync_togasupply.php
14
+ - worker/crons/toga2/netsuite/sync_togasupply_hh.php
15
+ - worker/crons/toga2/netsuite/diagnose_hh_import_inventory_adjustment.php
16
+ - _underscore/Model/Client/ItemFulfillments/InventoryAdjustment.php
17
+ - _underscore/Model/Client/ItemFulfillmentItems/InventoryAdjustmentItem.php
18
+ - dbchanges2/Core/2026-08-27b - ItemFulfillmentsInventoryAdjustmentsBridge.sql
19
+ - dbchanges2/Core/2026-08-28b - ItemFulfillmentItemsInventoryAdjustmentItemsBridge.sql
20
+ - dbchanges2/Client/2026-08-27a - ItemFulfillmentsInventoryAdjustmentsBridgeTable.sql
21
+ - dbchanges2/Client/2026-08-28c - ItemFulfillmentItemsInventoryAdjustmentItemsBridgeTable.sql
22
+ related:
23
+ - ./netsuite-transfer-order-import.md
24
+ - ./transfer-order-inventory-quantities.md
25
+ - ../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md
26
+ - ../../../1.0/apps/library/features/toga2-api-client-and-bridge.md
27
+ - ../../../2.0/apps/_underscore/features/sales-order-purchase-order-bridge-direction.md
28
+ - ../profile.md
29
+ ---
30
+
31
+ ## Summary
32
+
33
+ For NYCHH stock adjustments, the NetSuite **inventory adjustment** carries a link to the **Item
34
+ Fulfillment** it corrects, on the multi-select body custom field **`custbody_stock_adjustment`**
35
+ (internalId **9735**). This feature imports that link into TOGa Supply as two new bridge tables
36
+ (header + line), so H&H gets **adjustment → item fulfillment → sales order → purchase order**
37
+ traceability for their PO-specific fulfillment. Per the **2026-08-27 TOGa Supply meeting**; the
38
+ NetSuite custom field was implemented **<1 year ago**, so older adjustments carry **no** link.
39
+
40
+ Like [NYCHH transfer-order import](./netsuite-transfer-order-import.md), the **tables and models are
41
+ all-client infrastructure** (`dbchanges2/Client/` + shared `_Model_Client_*` bridge models) but the
42
+ **attach is gated NYCHH-only** by `IS_ENABLED_TRANSFER_ORDER_STOCKING_FLAG_ROUTING`.
43
+
44
+ ## The two bridges (both mirror the transfer-order bridge pattern)
45
+
46
+ | Level | Bridge table | `Core.Records` id | `RecordFields` | Model |
47
+ |---|---|---|---|---|
48
+ | Header | `ItemFulfillments_InventoryAdjustments` | 353 | 2515–2518 | `_Model_Client_ItemFulfillments_InventoryAdjustment` |
49
+ | Item | `ItemFulfillmentItems_InventoryAdjustmentItems` | 354 | 2519–2522 | `_Model_Client_ItemFulfillmentItems_InventoryAdjustmentItem` |
50
+
51
+ Each shipped as: an all-client `Client/` bridge table, Core registration (`Records` + `RecordFields`),
52
+ a Super-User ACL grant, and a per-client ACL **copied from the parent record**. The parent record
53
+ uuids are already `isIdentifier = 1`, so — unlike the transfer-order bridges — **no `EV-12`
54
+ uuid-identifier fix was needed** (contrast the 312/313 anomaly on
55
+ [the ACL permission chain](../../../2.0/apps/_underscore/features/acl-permission-chain.md)).
56
+
57
+ > **Bridge FK writes via a nested `{uuid}` need only record-level ACL, not field-level.** Verified
58
+ > against the existing transfer-order bridges — the nested-write ACL check is at the record level for a
59
+ > bridge link, so no `AclCustomFieldPermissions` grant is required on the FK columns.
60
+
61
+ ## Attach code — `App_Api_Toga2::attachInventoryAdjustmentToItemFulfillment`
62
+
63
+ Lives in `library/app/api/toga2.php`, called from `syncInventoryAdjustmentFromNetsuite` **after the
64
+ header upsert**, gated on `IS_ENABLED_TRANSFER_ORDER_STOCKING_FLAG_ROUTING`, and wrapped in
65
+ `try/catch` — it is **best-effort and must never freeze the section** (a failed link is logged, the
66
+ adjustment import still completes). It:
67
+
68
+ 1. reads `custbody_stock_adjustment` off the adjustment (the related Item Fulfillment),
69
+ 2. resolves that fulfillment by `ItemFulfillments.c_netsuiteInternalItemFulfillmentId`,
70
+ 3. idempotently creates the **header** link row, then
71
+ 4. links **line items** by matching `itemId` — every same-product adjustment-item ↔ fulfillment-item
72
+ pair is linked; the bridge's `UNIQUE(fk, fk)` dedupes re-runs.
73
+
74
+ ## On-demand dependency import (fulfillment not yet in TOGA)
75
+
76
+ When the linked fulfillment has **not** been imported into TOGA yet, the attach no longer skips and
77
+ waits for the fulfillment backfill — it **imports the fulfillment on demand** via
78
+ `syncItemFulfillmentFromNetsuite` (which itself cascades to the originating sales order when missing),
79
+ then **re-resolves and links**.
80
+
81
+ This required threading the **full item-fulfillment lookup set** — customer, country, state, location,
82
+ manufacturer, item, all-catalog-items, vendor-item, shipping-carrier, shipping-method, warehouse —
83
+ through `syncInventoryAdjustmentFromNetsuite` (signature **expanded 5 → 13 args**) and into
84
+ `attachInventoryAdjustmentToItemFulfillment`, plus updating the `common_sync_togasupply.php`
85
+ `INVENTORY_ADJUSTMENTS` call to pass them (the same set already passed to the item-fulfillment
86
+ section). **Design rule (developer instruction): all lookups are passed function-to-function**, like
87
+ the other sync functions — not rebuilt or fetched inside the callee.
88
+
89
+ ## Two fixes in `syncInventoryAdjustmentFromNetsuite` (shared engine, surfaced by the harness)
90
+
91
+ Both are in the shared library method, so they affect **any** client running inventory adjustments —
92
+ see also the [per-client sync](../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md):
93
+
94
+ - **(a) Prune-order FK violation.** The stale-item prune deleted an `InventoryAdjustmentItem` while
95
+ its child `InventoryAdjustmentItemUnits` still referenced it (FK **RESTRICT**, no cascade) → MySQL
96
+ **1451** → `EV-11`. **Fix:** delete the child units (`/inventory-adjustment-item-units/{uuid}`)
97
+ **before** the item. (Same "delete bridge/child rows first" rule as the tracking-number bridges.)
98
+ - **(b) Item-lookup indexing mismatch → duplicate items every run.** The item lookup was consumed as a
99
+ **two-level** `[$uuidManufacturer][$partNumber]`, but the cron passes it **one-level** `[$partNumber]`
100
+ (populated in `common_sync_togasupply.php` ~L313). The two-level
101
+ `$lookupItemByClientUuidAndManufacturerUuidAndPartNumberUpper` is declared but **never populated
102
+ (dead)**. The mismatch meant existing items were never found, so items were **re-created every run**
103
+ (duplicates / `EV-10`). **Fix:** index one-level `[$partNumber]`.
104
+
105
+ ## Diagnostic harness — `diagnose_hh_import_inventory_adjustment.php`
106
+
107
+ A **manual operator tool** in `worker/crons/toga2/netsuite/` (NOT in `cron.worker.sync.json`; does
108
+ **not** call `cronInitialization`, so no process lock / no cursor writes; installs a strict
109
+ `set_error_handler` that rethrows warnings as `ErrorException` for exact crash lines). It imports
110
+ **one** inventory adjustment (default NS internal id **6516616**) by calling the real
111
+ `syncInventoryAdjustmentFromNetsuite`, seeds the warehouse/item/manufacturer lookups (the item loop
112
+ **filters** items to known warehouses and **creates** items on a lookup-miss, so empty lookups drop or
113
+ duplicate items), and has a **read-only** diagnostic tail that dumps the NS item list +
114
+ `custbody_stock_adjustment` + whether the referenced fulfillment already exists in TOGA.
115
+
116
+ ```
117
+ php crons/toga2/netsuite/diagnose_hh_import_inventory_adjustment.php <togaClient> <togaApi> <togaSecret> [adjustmentInternalId]
118
+ ```
119
+ (creds = the `CLIENT_CONFIGURATION` client/api/secret from `sync_togasupply_hh.php`.)
120
+
121
+ ## Go-live toggle
122
+
123
+ `IS_ENABLED_INTEGRATION_INVENTORY_ADJUSTMENTS` is still **`false`** in `sync_togasupply_hh.php` —
124
+ enabling it is the go-live switch for this feature on the scheduled cron.
125
+
126
+ ## Open / next-session — migrate the inventory-adjustment sync from SOAP to REST
127
+
128
+ The inventory-adjustment sync is still **SOAP**: `App_NetSuite::getInventoryNumberFromSerialNumber` +
129
+ `getInventoryNumber` per serial (`toga2.php` ~5371/5374), plus `getObject('inventoryAdjustment')`,
130
+ `getItemDetails`, `detectItemType`, `getItemIsSerialized`, `getItemIsFulfillable`. For a **serialized**
131
+ adjustment this is ~2 NetSuite SOAP round-trips **per serial** → ~1 hour for a large adjustment; the
132
+ on-demand fulfillment import inherits the same cost. **Decision:** migrate the whole
133
+ inventory-adjustment sync to REST via `App_Api_Netsuite_Rest` (`rest.php` already has
134
+ `send`/`authenticate`/`suiteqlListAll`/`getCustomFieldValue`/`fetchItemById`/`listInventoryAdjustments`
135
+ (minimal) — the gaps are a full inventory-adjustment fetch with inventory detail/serials, and an
136
+ inventory-number/serial batch lookup). **This is the next session's task.**
137
+
138
+ ## Known state — test adjustment 6516616
139
+
140
+ NYCHH adjustment **6516616** (tranId **2136**) has **4 NON-serialized** items (adjustQtyBy
141
+ 1800/1800/700/40) and `custbody_stock_adjustment` → Item Fulfillment **#289938** (NS internal
142
+ 6507274). That fulfillment is **not yet imported** in TOGA, so the bridges stay empty until the
143
+ on-demand import lands it. Import + item-sync of the adjustment itself works (4
144
+ `InventoryAdjustmentItems`).
145
+
146
+ ## Gotchas / known issues
147
+
148
+ - **Older adjustments have no link.** `custbody_stock_adjustment` was added in NetSuite <1 year ago;
149
+ adjustments predating it will never populate a bridge — not a bug.
150
+ - **The attach is best-effort by design.** It is gated NYCHH-only and wrapped in `try/catch`; a link
151
+ failure logs and lets the adjustment import complete. Do not "harden" it into a throw — a bridge miss
152
+ must not freeze `INVENTORY_ADJUSTMENTS`.
153
+ - **Empty lookups drop or duplicate items.** The item loop filters to known warehouses and creates
154
+ items on lookup-miss, so any manual/diagnostic invocation must seed the warehouse/item/manufacturer
155
+ lookups (the harness does).
156
+
157
+ ## Change history
158
+
159
+ - 2026-08-28 — Built NYCHH inventory-adjustment → item-fulfillment linking off NetSuite
160
+ `custbody_stock_adjustment` (internalId 9735): two all-client bridges
161
+ (`ItemFulfillments_InventoryAdjustments` record 353 / `ItemFulfillmentItems_InventoryAdjustmentItems`
162
+ record 354, Core-registered + Super-User/per-client ACL), a gated best-effort
163
+ `App_Api_Toga2::attachInventoryAdjustmentToItemFulfillment` (resolves the fulfillment by
164
+ `c_netsuiteInternalItemFulfillmentId`, idempotent header + itemId-matched line links), and **on-demand
165
+ import** of a not-yet-imported fulfillment via `syncItemFulfillmentFromNetsuite` (cascades to the SO)
166
+ — which expanded `syncInventoryAdjustmentFromNetsuite` 5→13 args to thread the full IF lookup set.
167
+ Fixed two shared-engine bugs: the stale-item prune deleted an item before its child units (FK
168
+ RESTRICT → 1451/EV-11 — now deletes units first), and the item lookup was consumed two-level while the
169
+ cron populates it one-level `[$partNumber]` (re-created items every run → duplicates/EV-10 — now
170
+ one-level). Added the `diagnose_hh_import_inventory_adjustment.php` operator harness. Go-live toggle
171
+ `IS_ENABLED_INTEGRATION_INVENTORY_ADJUSTMENTS` still false. **Open (next session):** migrate the
172
+ inventory-adjustment sync from SOAP (~2 round-trips/serial, ~1h for a large adjustment) to REST. (jcardinal)
@@ -6,7 +6,7 @@ project: Library
6
6
  client: nychh
7
7
  type: client-feature
8
8
  status: active
9
- updated: 2026-08-27
9
+ updated: 2026-08-28
10
10
  owners: [jcardinal]
11
11
  files:
12
12
  - library/app/api/toga2.php
@@ -14,6 +14,10 @@ files:
14
14
  - library/app/netsuite.php
15
15
  - worker/crons/toga2/netsuite/sync_togasupply_hh.php
16
16
  - worker/crons/toga2/netsuite/common_sync_togasupply.php
17
+ - worker/crons/toga2/netsuite/diagnose_hh_stuck_transfer_order.php
18
+ - dbchanges2/Core/2026-08-27a - TransferOrderUuidSearchableIdentifier.sql
19
+ - dbchanges2/Client/2026-08-28a - TransferOrderItemsTimestampColumns.sql
20
+ - dbchanges2/Client/2026-08-28b - TransferOrderItemsTimestampColumnsReposition.sql
17
21
  related:
18
22
  - ../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md
19
23
  - ../../../1.0/apps/library/features/toga2-api-client-and-bridge.md
@@ -114,6 +118,48 @@ status was previously throwing (and being swallowed), which is why order 128829
114
118
  `App_NetSuite::getCustomFieldValue()` (`library/app/netsuite.php`) was hardened with `?? []` / `?? null`
115
119
  guards: 1.0 targets PHP 7.2 and any PHP warning on that path terminates the cron.
116
120
 
121
+ ## SALES_ORDERS freeze chain — three distinct causes, peeled in order (2026-08-28)
122
+
123
+ Once transfer-order routing was live, the NYCHH **and GroWrk** `SALES_ORDERS` section froze at
124
+ `1-RUNNING` on `syncTransferOrderFromNetsuite`. `SALES_ORDERS` has **no per-record `try/catch`** (see
125
+ [per-client sync](../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md)), so any
126
+ throw aborts the whole section and re-throws every run. Three separate faults sat one behind the next:
127
+
128
+ 1. **`Undefined property: stdClass::$transferOrderStage`** at the change-check in `toga2.php`
129
+ (`if ($transferOrder) {…}`). **api2 OMITS a null foreign-key object from a depth-2 GET entirely** —
130
+ it is *absent*, not null — so `$transferOrder->transferOrderStage` (and `originLocation` /
131
+ `destinationLocation`) raised a PHP warning that 1.0 `App_Error` escalates to fatal. **Fix:** guard
132
+ all three FK reads with `?? null`. (Lesson: on a depth-limited api2 GET, a nullable FK object may be
133
+ missing rather than null — always `?? null` before dereferencing.)
134
+ 2. **`EV-12` on the transfer-order-items POST** — the nested `transferOrder:{uuid}` link could not
135
+ resolve its parent because `RecordFields.uuid.isIdentifier` was **0** on records 312/313. Fixed
136
+ platform-wide (`dbchanges2/Core/2026-08-27a`). Full rule on the
137
+ [ACL permission chain](../../../2.0/apps/_underscore/features/acl-permission-chain.md).
138
+ 3. **MySQL 1054 `Unknown column 'dtCreated'` → api2 `EV-10`** one step further on.
139
+ `_Model_Client_TransferOrderItem` declares `dtCreated`/`dtUpdated`
140
+ (`FIELD_DATETIME_CREATED`/`_UPDATED`) and emits them on every INSERT, but the `TransferOrderItems`
141
+ **table lacked both columns in EVERY client DB** (a platform-wide drift — the header
142
+ `TransferOrders` table had them). **Fix (all-client, `dbchanges2/Client/` fan-out):** `2026-08-28a`
143
+ adds the two columns mirroring the header table (`DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP`,
144
+ `dtUpdated` also `ON UPDATE CURRENT_TIMESTAMP`); `2026-08-28b` repositions them (`AFTER uuid` /
145
+ `AFTER dtCreated`) per the new column-ordering standard —
146
+ [2.0 framework rules](../../../2.0/standards/framework-rules.md). Same failure class as GroWrk's
147
+ `_Model_Growrk_Unit` 1054: a declared model field with no physical column 1054s on every write.
148
+
149
+ ### Debugging lesson — `Logs.Issue` trace is stale; use a single-record harness for the real line
150
+
151
+ `Logs.Issue` aggregates by message + runner and captures its `trace` **once at first-seen, never
152
+ refreshed**. Issue 270's stale trace pointed at a bogus line and a spurious `$location` message and
153
+ sent debugging the wrong way for a long time. The reliable way to get the true crash line was
154
+ `worker/crons/toga2/netsuite/diagnose_hh_stuck_transfer_order.php` — a **manual operator tool** (not in
155
+ `cron.worker.sync.json`; does **not** call `cronInitialization`, so no process lock / no cursor
156
+ writes). It fetches a NetSuite sales/transfer order via `App_Api_Netsuite_Rest::listSalesOrders`
157
+ (NYCHH entity filter `listChildCustomers(28908)`, window in server tz `America/Chicago`), dumps its
158
+ shape, then re-runs the **real** `syncTransferOrderFromNetsuite` under a strict `set_error_handler`
159
+ that rethrows warnings as a catchable `ErrorException` with exact `file:line`. That is what found the
160
+ `transferOrderStage` crash after the stale `Logs.Issue` trace misdirected. **When a 1.0 cron freeze's
161
+ `Logs.Issue` trace looks wrong, trust a single-record harness over the aggregated trace.**
162
+
117
163
  ## Gotchas / known issues
118
164
 
119
165
  - **Order 128829 (no location anywhere) is unresolved** — see the origin-resolution section. Do not
@@ -128,6 +174,17 @@ guards: 1.0 targets PHP 7.2 and any PHP warning on that path terminates the cron
128
174
 
129
175
  ## Change history
130
176
 
177
+ - 2026-08-28 — Unfroze the `SALES_ORDERS` section (NYCHH + GroWrk) by peeling **three** stacked faults
178
+ on `syncTransferOrderFromNetsuite`: (1) `Undefined property $transferOrderStage` — api2 **omits a
179
+ null FK object** from a depth-2 GET, guarded `transferOrderStage`/`originLocation`/`destinationLocation`
180
+ with `?? null`; (2) `EV-12` because `RecordFields.uuid.isIdentifier` was 0 on records 312/313 (fixed
181
+ platform-wide, `dbchanges2/Core/2026-08-27a`); (3) MySQL 1054 → `EV-10` because `TransferOrderItems`
182
+ lacked the model-declared `dtCreated`/`dtUpdated` columns in every client DB (added all-client via
183
+ `dbchanges2/Client/2026-08-28a`, repositioned `AFTER uuid`/`AFTER dtCreated` per the new
184
+ column-ordering standard by `2026-08-28b`). Recorded the debugging lesson: `Logs.Issue`'s `trace` is
185
+ captured once and goes stale (Issue 270 misdirected), so a single-record harness
186
+ (`diagnose_hh_stuck_transfer_order.php`, strict `set_error_handler` → `ErrorException`) is the
187
+ reliable way to get the true crash line. (jcardinal)
131
188
  - 2026-08-27 — Built NYCHH transfer-order ingestion end-to-end. **Routing by `custbody_stocking_order`
132
189
  (id 7097), not dollar amount** (`isTransferOrder()` rewritten + new `isStockingOrder()` normalizer);
133
190
  the old `total===0 + hold-flag` heuristic was wrong for NYCHH (no hold flag set → everything landed
@@ -33,6 +33,7 @@ related:
33
33
  - ./features/po-number-upstream-direction.md
34
34
  - ./features/netsuite-transfer-order-import.md
35
35
  - ./features/transfer-order-inventory-quantities.md
36
+ - ./features/netsuite-inventory-adjustment-fulfillment-link.md
36
37
  - ../../2.0/apps/_underscore/features/sales-order-purchase-order-bridge-direction.md
37
38
  - ../../2.0/apps/toga25-supply/features/transfer-orders-page.md
38
39
  ---
@@ -89,6 +90,11 @@ table views. Client-specific DB change-sets live in `dbchanges2/Client_Nychh/`.
89
90
  fulfilled/backordered model on `_Model_Nychh_Item`, plus the client-override models + ACL rows that
90
91
  let a NetSuite TransferOrder POST succeed. See
91
92
  [NYCHH transfer-order inventory & quantities](./features/transfer-order-inventory-quantities.md).
93
+ - **Inventory-adjustment → item-fulfillment linking** — imports NetSuite `custbody_stock_adjustment`
94
+ (adjustment→fulfillment) into two all-client bridges for adjustment→fulfillment→SO→PO traceability;
95
+ gated NYCHH-only, best-effort, with on-demand fulfillment import. Go-live toggle
96
+ (`IS_ENABLED_INTEGRATION_INVENTORY_ADJUSTMENTS`) still off. See
97
+ [NYCHH inventory-adjustment → item-fulfillment linking](./features/netsuite-inventory-adjustment-fulfillment-link.md).
92
98
 
93
99
  - **PO links are UPSTREAM (`PurchaseOrders_SalesOrders`), not downstream.** The downstream
94
100
  `sales-order-purchase-orders` route returns an empty array for NYCHH by design, and the shared
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "toga-ai",
3
- "version": "1.0.693",
3
+ "version": "1.0.695",
4
4
  "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
5
5
  "keywords": [
6
6
  "claude",