toga-ai 1.0.693 → 1.0.695
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md +26 -1
- package/knowledge/2.0/apps/_underscore/INDEX.md +1 -1
- package/knowledge/2.0/apps/_underscore/features/acl-permission-chain.md +18 -0
- package/knowledge/2.0/apps/api2/INDEX.md +3 -3
- package/knowledge/2.0/apps/api2/architecture.md +28 -2
- package/knowledge/2.0/apps/api2/features/environment-variable-drives-underscore-branch.md +26 -2
- package/knowledge/2.0/apps/api2/features/health-check-endpoint.md +43 -8
- package/knowledge/2.0/apps/api2/features/v2-deadlock-retry.md +93 -13
- package/knowledge/2.0/standards/framework-rules.md +30 -0
- package/knowledge/INDEX.md +1 -1
- package/knowledge/clients/compass-usa/features/asn-to-item-fulfillment.md +22 -1
- package/knowledge/clients/growrk/profile.md +10 -2
- package/knowledge/clients/nychh/INDEX.md +2 -1
- package/knowledge/clients/nychh/features/netsuite-inventory-adjustment-fulfillment-link.md +172 -0
- package/knowledge/clients/nychh/features/netsuite-transfer-order-import.md +58 -1
- package/knowledge/clients/nychh/profile.md +6 -0
- package/package.json +1 -1
|
@@ -6,7 +6,7 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-08-
|
|
9
|
+
updated: 2026-08-28
|
|
10
10
|
owners: ["dfranks", "bala", "jcardinal", "mhammontree", "snaredla"]
|
|
11
11
|
files:
|
|
12
12
|
- worker/crons/toga2/netsuite/common_sync_togasupply.php
|
|
@@ -504,8 +504,33 @@ Parameters are stored **per client DB** but accessed **through the TOGa2 API**,
|
|
|
504
504
|
`NETSUITE_LAST_SYNC_DATETIME_*` frozen watermark), then read `Logs.Issue` — not `Logs_<Client>.Api`
|
|
505
505
|
— for the throwing order.
|
|
506
506
|
|
|
507
|
+
- **⚠ Inventory-adjustment sync: two shared-engine bugs + it is still SOAP (2026-08-28).** Both fixed
|
|
508
|
+
in `syncInventoryAdjustmentFromNetsuite` (`library/app/api/toga2.php`) and reachable by any client
|
|
509
|
+
running inventory adjustments: (a) the stale-item prune deleted an `InventoryAdjustmentItem` before
|
|
510
|
+
its child `InventoryAdjustmentItemUnits` (FK **RESTRICT** → MySQL 1451 → `EV-11`) — now deletes the
|
|
511
|
+
child units first; (b) the item lookup was consumed **two-level**
|
|
512
|
+
`[$uuidManufacturer][$partNumber]` while the cron populates it **one-level** `[$partNumber]`
|
|
513
|
+
(`common_sync_togasupply.php` ~L313; the two-level lookup is declared but never populated — dead), so
|
|
514
|
+
existing items were never matched and **re-created every run** (duplicates / `EV-10`) — now indexed
|
|
515
|
+
one-level. Also: the section is **still SOAP** (`App_NetSuite::getInventoryNumber*` ~2 round-trips
|
|
516
|
+
**per serial** → ~1h for a large serialized adjustment); a REST migration via `App_Api_Netsuite_Rest`
|
|
517
|
+
is the planned next step. Surfaced building NYCHH's adjustment→fulfillment linking; full detail on
|
|
518
|
+
[NYCHH inventory-adjustment → item-fulfillment linking](../../../clients/nychh/features/netsuite-inventory-adjustment-fulfillment-link.md).
|
|
519
|
+
Note `syncInventoryAdjustmentFromNetsuite`'s signature was expanded **5 → 13 args** to thread the full
|
|
520
|
+
item-fulfillment lookup set through for the on-demand fulfillment import (lookups are passed
|
|
521
|
+
function-to-function, like the other sync functions).
|
|
522
|
+
|
|
507
523
|
## Change history
|
|
508
524
|
|
|
525
|
+
- 2026-08-28 — Recorded two shared-engine inventory-adjustment fixes in
|
|
526
|
+
`syncInventoryAdjustmentFromNetsuite` (prune child `InventoryAdjustmentItemUnits` before the item —
|
|
527
|
+
FK RESTRICT/1451/EV-11; index the item lookup **one-level** `[$partNumber]` to match what the cron
|
|
528
|
+
populates, was two-level dead → items re-created every run / duplicates / EV-10), the 5→13-arg
|
|
529
|
+
signature expansion that threads the full IF lookup set for on-demand fulfillment import, and that the
|
|
530
|
+
section is **still SOAP** (~2 round-trips/serial → ~1h large adjustment; REST migration planned).
|
|
531
|
+
Surfaced building NYCHH's adjustment→fulfillment linking — see
|
|
532
|
+
[NYCHH inventory-adjustment → item-fulfillment linking](../../../clients/nychh/features/netsuite-inventory-adjustment-fulfillment-link.md).
|
|
533
|
+
(jcardinal)
|
|
509
534
|
- 2026-08-27 — **Removed a silent data-loss bug: all per-order `catch (Throwable)` skips and all
|
|
510
535
|
section-level `try/finally` were deleted, so every section now fails LOUD.** A 2026-08-20 change had
|
|
511
536
|
wrapped each order dispatch in `catch (Throwable) { error_log }` and each section in
|
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
|-----|---------|-------|
|
|
5
5
|
| [Proposed — git-sourced base+overlay JSON authoring for the Surface layer](architecture/surface-authoring-proposal.md) | A **proposal / handoff recommendation** (not implemented) that the Surface layer's *authoring* model move off hand-authored SQL against the `SurfaceOverrides` E | _underscore/Model/Core/Surface.php, _underscore/Model/Client/SurfaceOverride.php |
|
|
6
6
|
| [_underscore Framework Architecture](architecture.md) | `_underscore` is the shared PHP backend framework for **all 2.0 applications**. | _underscore/_underscore.php, _underscore/Loader.php, _underscore/Framework.php, _underscore/Model.php, _underscore/Database.php, _underscore/Query.php, _underscore/Route.php, _underscore/Component.php |
|
|
7
|
-
| [ACL Permission Chain (Record & Field Authorization)](features/acl-permission-chain.md) | Authorization in the 2.0 API is **metadata-driven**: whether a role may Create/Read/Update/Delete a record is decided by rows across **four linked tables**, not | api2/Component/Api/V2/V2.php, _underscore/Model/Core/Page.php, _underscore/Model/Core/Surface.php, _underscore/Model/Client/TrackingNumber.php, dbchanges2/Client/2026-06-03- BLANK_CLIENT_DATABASE.sql, dbchanges2/Client/2026-06-23b - ItemTranslationsAcl.sql, dbchanges2/Client/2026-07-02c - TrackingNumberNeedsReturnLabelFieldPermission.sql, dbchanges2/Client/2026-07-22c - TrackingNumberMeasureIdsFieldPermission.sql, dbchanges2/Client_Quad/2026-08-24 - Quad Multi Currency Item Pricing.sql |
|
|
7
|
+
| [ACL Permission Chain (Record & Field Authorization)](features/acl-permission-chain.md) | Authorization in the 2.0 API is **metadata-driven**: whether a role may Create/Read/Update/Delete a record is decided by rows across **four linked tables**, not | api2/Component/Api/V2/V2.php, dbchanges2/Core/2026-08-27a - TransferOrderUuidSearchableIdentifier.sql, _underscore/Model/Core/Page.php, _underscore/Model/Core/Surface.php, _underscore/Model/Client/TrackingNumber.php, dbchanges2/Client/2026-06-03- BLANK_CLIENT_DATABASE.sql, dbchanges2/Client/2026-06-23b - ItemTranslationsAcl.sql, dbchanges2/Client/2026-07-02c - TrackingNumberNeedsReturnLabelFieldPermission.sql, dbchanges2/Client/2026-07-22c - TrackingNumberMeasureIdsFieldPermission.sql, dbchanges2/Client_Quad/2026-08-24 - Quad Multi Currency Item Pricing.sql |
|
|
8
8
|
| [Address Uniqueness Normalization (unit identifier + 5-digit ZIP comparison)](features/address-uniqueness-normalization.md) | When a business rule says *"only one X per physical address"*, comparing address rows field-for-field does **not** work: the same dwelling is spelled many diffe | _underscore/Model/Rate/Entitlement.php |
|
|
9
9
|
| [Address Validation (carrier waterfall + validateAddress scripted endpoint)](features/address-validation.md) | `_Model_Client_Address::validateAddress` verifies a US address against a **carrier waterfall (USPS → FedEx → UPS)** and returns a single canonical, carrier-norm | _underscore/Model/Client/Address.php, _underscore/Component/Library/Carriers/Usps/Usps.php |
|
|
10
10
|
| [_ApiRequest — JSON encode/decode & api-logging behavior](features/apirequest-json-content-type.md) | `_ApiRequest` is the 2.0 outbound HTTP client. | _underscore/ApiRequest.php |
|
|
@@ -10,6 +10,7 @@ updated: 2026-08-28
|
|
|
10
10
|
owners: ["jcardinal", "mhammontree", "tcox", "bala"]
|
|
11
11
|
files:
|
|
12
12
|
- api2/Component/Api/V2/V2.php
|
|
13
|
+
- dbchanges2/Core/2026-08-27a - TransferOrderUuidSearchableIdentifier.sql
|
|
13
14
|
- _underscore/Model/Core/Page.php
|
|
14
15
|
- _underscore/Model/Core/Surface.php
|
|
15
16
|
- _underscore/Model/Client/TrackingNumber.php
|
|
@@ -242,6 +243,18 @@ column + `CustomRecordFields` row + `AclCustomFieldPermissions` grant + model de
|
|
|
242
243
|
2026-08-27 on NYCHH `c_netsuiteInternalTransferOrderStatus` (recordId 351): the field existed and was
|
|
243
244
|
granted, but the nested transfer-order-stage lookup 400'd until `isIdentifier` was set to 1.
|
|
244
245
|
|
|
246
|
+
> **The same rule bites the base `uuid` field, not just `c_` fields.** When a nested write links its
|
|
247
|
+
> parent **by `uuid`** (the normal case for a `{uuid}` nested reference), that record's
|
|
248
|
+
> `RecordFields.uuid.isIdentifier` must be `1` or `searchableIdentifierFields` excludes `uuid` and the
|
|
249
|
+
> write 400s `EV-12`. Every record normally ships with `uuid.isIdentifier = 1` (e.g. SalesOrders record
|
|
250
|
+
> 14) — but **records 312 (`TransferOrders`) and 313 (`TransferOrderItems`) were the platform-wide
|
|
251
|
+
> anomaly**: their `uuid.isIdentifier` was `0`, so a nested `transferOrder:{uuid}` write on the
|
|
252
|
+
> transfer-order-items POST could not resolve its parent and froze the sync. Fixed platform-wide
|
|
253
|
+
> (`dbchanges2/Core/2026-08-27a - TransferOrderUuidSearchableIdentifier.sql`, `isIdentifier = 1` on
|
|
254
|
+
> records 312/313 `uuid`). When an `EV-12` mentions `searchableIdentifierFields:[id]` on a plain
|
|
255
|
+
> `{uuid}` nested write, check `Core.RecordFields.uuid.isIdentifier` for that record before anything
|
|
256
|
+
> else.
|
|
257
|
+
|
|
245
258
|
**The `c_` declaration on a client-override model comes from its NetSuite trait.** The pattern for a
|
|
246
259
|
NetSuite-synced record is `_Model_<Client>_X extends _Model_Client_X { use _Trait_Netsuite_X; }` — the
|
|
247
260
|
trait is what declares the `c_` fields (so a client with only `extends`, no `use`, is missing them and
|
|
@@ -488,6 +501,11 @@ hardcoded `Core.RecordFields` id literals instead of a subselect.
|
|
|
488
501
|
the space `Records.aclDatabase` selects — **`Core.Roles`** ids for a `CORE` record such as
|
|
489
502
|
`surfaces` (`V2.php` ~L3090 / ~L6198), with `id.core.roles` derived from
|
|
490
503
|
`Roles.coreRoleId` (~L1611-1627) and therefore usually just Base. (bala)
|
|
504
|
+
- 2026-08-28 — Extended the `isIdentifier` rule to the **base `uuid` field**: a nested `{uuid}` write
|
|
505
|
+
resolves its parent only if that record's `RecordFields.uuid.isIdentifier = 1`, else `EV-12`. Records
|
|
506
|
+
**312 (`TransferOrders`) / 313 (`TransferOrderItems`)** were the platform-wide anomaly (uuid
|
|
507
|
+
`isIdentifier = 0`), which froze the transfer-order-items POST; fixed platform-wide by
|
|
508
|
+
`dbchanges2/Core/2026-08-27a - TransferOrderUuidSearchableIdentifier.sql`. (jcardinal)
|
|
491
509
|
- 2026-08-27 — Recorded two facts from wiring NYCHH transfer-order custom fields: **there is NO
|
|
492
510
|
ACL/metadata cache** (`buildLookups()` re-reads per request, so a new grant/metadata row applies on
|
|
493
511
|
the next request — no cache-bust exists; a whole debugging pass was wasted assuming one did), and a
|
|
@@ -7,8 +7,8 @@
|
|
|
7
7
|
| [Multi-Client (Cross-Client) Data Retrieval](features/cross-client-data-retrieval.md) | A single authenticated V2 GET listing can return records across **many** clients (designed for 1000+) that the caller is entitled to, honoring **each target cli | api2/Component/Api/CrossClient/CrossClient.php, api2/Component/Api/V2/V2.php, api2/Controller/Index.php, _underscore/Model/Cache/Table.php, _underscore/Model/Cache/Tables/Client.php, _underscore/Model/Core/Record.php, worker2/Worker/Platform/Cache.php, worker2/Controller/Index.php, worker2/_.php, dbchanges2/Cache/2026-06-30a - MultiClientCacheTables.sql, dbchanges2/Core/2026-06-30b - CacheClusterRegistrationAndRecordTtl.sql, dbchanges2/Core/2026-07-27a - PlatformCacheCleanCron.sql |
|
|
8
8
|
| [cXML ShipNotice Gateway (ASN ingestion, carrier resolution, per-client provisioning)](features/cxml-shipnotice-gateway.md) | `_Component_Api_Cxml` accepts a supplier `ShipNoticeRequest` and translates it into a `POST /v2/advance-shipping-notices` on the V2 JSON engine. | api2/Component/Api/Cxml/Cxml.php, api2/Component/Api/V2/V2.php, api2/Controller/Index.php |
|
|
9
9
|
| [Encrypted-User-UUID Auth Handoff (/auth/encrypted-user-uuid)](features/encrypted-user-uuid-auth-handoff.md) | `POST /auth/encrypted-user-uuid` is the intended **cross-client / SSO-handoff identity mechanism**: given an encrypted `{client, user}` UUID pair, it mints a fr | api2/Component/Api/CrossClient/CrossClient.php |
|
|
10
|
-
| [ENVIRONMENT (not the EB environment name) decides the _underscore branch and Config file](features/environment-variable-drives-underscore-branch.md) | An api2 Elastic Beanstalk instance decides **which `_underscore` branch it clones** and **which `Config/<env>.ini` it loads** from the EB environment property * | api2/.ebextensions/git.php, api2/.ebextensions/php_include_underscore.config, api2/.ebextensions/git.sandbox-dev.json, api2/.ebextensions/git.sandbox-client.json, api2/Config/beta.ini, api2/Config/sandbox-dev.ini, api2/Component/Api/V2/V2.php |
|
|
11
|
-
| [Health-check endpoint (/health liveness short-circuit)](features/health-check-endpoint.md) | `
|
|
10
|
+
| [ENVIRONMENT (not the EB environment name) decides the _underscore branch and Config file](features/environment-variable-drives-underscore-branch.md) | An api2 Elastic Beanstalk instance decides **which `_underscore` branch it clones** and **which `Config/<env>.ini` it loads** from the EB environment property * | api2/.ebextensions/git.php, api2/.platform/hooks/prebuild/git.sh, api2/.ebextensions/php_include_underscore.config, api2/.ebextensions/git.sandbox-dev.json, api2/.ebextensions/git.sandbox-client.json, api2/Config/beta.ini, api2/Config/sandbox-dev.ini, api2/Component/Api/V2/V2.php |
|
|
11
|
+
| [Health-check endpoint (/health liveness — static file + PHP short-circuit)](features/health-check-endpoint.md) | The EB/ALB liveness probe (`/health`) is served two ways, and the **load-bearing** one is now a **static Apache-served file**, not PHP: - **`/health` → a static | api2/health, api2/.platform/httpd/conf.d/health_probe.conf, api2/Controller/Index.php |
|
|
12
12
|
| [Language Translation Layer (audience.language + sidecar tables)](features/language-translation-layer.md) | Serves the same TOGa data (Item title/description/longDescription, plus item **feature** text — `Features.name`, `ItemCategoryFeatureGroups.name`, `ItemFeatures | api2/Component/Api/V2/V2.php, api2/Component/Api/V2/Response/Response.php, _underscore/Model/Core/Setting.php, _underscore/Model/Core/RecordField.php, _underscore/Model/Core/DefaultGlobalSetting.php, _underscore/Model/Client/ItemTranslation.php, _underscore/Model/Client/FeatureTranslation.php, _underscore/Model/Client/ItemCategoryFeatureGroupTranslation.php, _underscore/Model/Client/ItemFeatureTranslation.php, dbchanges2/Client/2026-06-23a - ItemTranslations.sql, dbchanges2/Client/2026-06-23b - ItemTranslationsAcl.sql, dbchanges2/Client/2026-07-13a - FeatureTranslations.sql, dbchanges2/Client/2026-07-13b - FeatureTranslationsAcl.sql, dbchanges2/Core/2026-06-23a - RecordFieldsTranslationColumn.sql, dbchanges2/Core/2026-06-23b - ItemTranslationsRecord.sql, dbchanges2/Core/2026-07-13 - FeatureTranslationsRecord.sql |
|
|
13
13
|
| [/auth/login resolves the client from the email domain, not the Bearer token (cross-client user path)](features/login-cross-client-user-resolution.md) | `POST /v2/auth/login` (email/password user login) can silently swap the target client mid-request. | api2/Component/Api/V2/V2.php, _underscore/String.php |
|
|
14
14
|
| [Nested FK object embedding is gated by the CHILD record's own ACL](features/nested-fk-acl-embedding.md) | When the V2 JSON engine serializes a foreign-key field into a **nested object** (in `getFullModelData()`, ~V2.php L6016-6060), it re-checks the **child** record | api2/Component/Api/V2/V2.php, dbchanges2/Client_Compass/2026-07-23b - PurchaseOrdersRecordReadAcl.sql |
|
|
@@ -22,7 +22,7 @@
|
|
|
22
22
|
| [TableView field/column metadata (TableViewFields, hidden projected columns)](features/tableview-field-metadata.md) | The columns of a 2.0 table view are defined by DB metadata, not code. | _underscore/Model/Client/TableView.php, api2/Component/Api/V2/V2.php, dbchanges2/Client/2026-07-20 - ItemsUuidForPurchaseOrderItemsTableView.sql, dbchanges2/Core/2026-08-17a - ItemFulfillmentQuantityFieldTypeNumber.sql, dbchanges2/Client/2026-08-17a - ItemFulfillmentColumnsCopyable.sql, dbchanges2/Client_Quad/2026-08-18c - SalesOrderListingSortByDateOrderDesc.sql, dbchanges2/Client_Quad/2026-08-24 - Quad Multi Currency Item Pricing.sql, dbchanges2/Client_Elite/2026-08-18 - InventoryUnitsItemColumns.sql, dbchanges2/Client_Nychh/2026-08-18 - InventoryUnitsItemColumns.sql, dbchanges2/Client_CompassCanada/2026-08-27a - ItemFulfillmentTableViewsRebuild.sql |
|
|
23
23
|
| [Tickets API (/v2/tickets)](features/tickets-api.md) | The generic ticket endpoint of the 2.0 REST API. | Component/Api/V2/V2.php |
|
|
24
24
|
| [V2 API error/message codes (EV/EZ troubleshooting map)](features/v2-api-error-codes.md) | The V2 JSON engine (`Component/Api/V2/V2.php`) returns short **message codes** in the response `error` field, grouped by family: `EN-*` authentication, `EZ-*` a | api2/Component/Api/V2/V2.php, api2/Component/Api/V2/Response/Response.php, api2/Component/Api/V2/Response/Oauth/Oauth.php, api2/Controller/Index.php, toga2-supply/src/globalTypes.ts, toga2-supply/src/pages/Orders/api/OrdersApi.ts, toga2-supply/src/api/toga.ts, _underscore/Model/Client/TrackingNumber.php |
|
|
25
|
-
| [V2 request
|
|
25
|
+
| [V2 request retry (Prong A) — deadlock AND retriable unique-key race, route-scoped in-process replay](features/v2-deadlock-retry.md) | api2's front controller ("**Prong A**") can **detect a retriable concurrency failure and replay the whole request in-process**, so a transient collision self-re | api2/Controller/Index.php, api2/Component/Api/V2/V2.php, _underscore/Database.php, _underscore/Query.php, _underscore/Model/Compass/AdvanceShippingNotice.php |
|
|
26
26
|
| [V2 REST query contract (params, where grammar, encoding, ACL behavior)](features/v2-rest-query-contract.md) | What an **HTTP client** has to get right to query the Toga v2 REST API: which query params are recognized, the exact `where` grammar, how the query string is (n | api2/Component/Api/V2/V2.php |
|
|
27
27
|
| [V2 reverse hasMany collections must be named in the fetch fields whitelist](features/v2-reverse-hasmany-fields-whitelist.md) | In the V2 JSON engine, a **reverse hasMany** relationship — the collection of child records that foreign-key back to a parent (e.g. | api2/Component/Api/V2/V2.php |
|
|
28
28
|
| [AWS CodePipeline Deployment via CodeConnections (GitHub → Elastic Beanstalk)](workflows/codepipeline-codeconnections-deploy.md) | 2.0 apps (`api2`, `_underscore`) are deployed through **AWS CodePipeline**. | api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, api2/ebs/register_instance_to_shared_application_load_balancer.php, api2/.platform/hooks/prebuild/git.sh, api2/.ebextensions/git.php, api2/.ebextensions/git.sandbox-dev.json |
|
|
@@ -6,7 +6,7 @@ project: API
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: architecture
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-08-
|
|
9
|
+
updated: 2026-08-28
|
|
10
10
|
owners: [jcardinal, bala, mhammontree, dfranks]
|
|
11
11
|
files:
|
|
12
12
|
- api2/Controller/Index.php
|
|
@@ -254,6 +254,27 @@ interceptor's plain-`Exception` rejections — all now returning 4xx/fixed. Dist
|
|
|
254
254
|
reading the ALB 5xx metric and `Logs.Issue`: zero 5xx + a traffic spike = transient EB snapshot;
|
|
255
255
|
a sustained 5xx rate = a code bug to fix.
|
|
256
256
|
|
|
257
|
+
**A third cause — a `Degraded`/`Severe` flip with NO stated cause is the PHP-served `/health`
|
|
258
|
+
probe being STARVED, not client traffic failing.** Confirmed on `api-production-1` (acct
|
|
259
|
+
`654654170868`): off-peak **bursts of slow, failing `POST /v2/advance-shipping-notices`**
|
|
260
|
+
(Compass ASN; 17–63 s each, ~100% HTTP 500) saturate the **PHP-FPM worker pool** on an
|
|
261
|
+
instance. The ALB `/health` check — historically **PHP-served, 5 s timeout** — then can't get a
|
|
262
|
+
worker and fails a few checks, so EB flags the target group "reduced health" **with no cause
|
|
263
|
+
cited**. Two reasons this looks invisible: (1) **health-check traffic does NOT appear in ALB
|
|
264
|
+
`RequestCount`/`5xx` metrics**, so the ALB view reads clean; (2) **`HealthyHostCount` never
|
|
265
|
+
drops**, because the probe failure is shorter than the **5-check / 75 s deregistration
|
|
266
|
+
threshold** — it flaps EB enhanced-health without ever deregistering a target. A separate, benign
|
|
267
|
+
contributor is genuine **sub-minute `/health` transients** on a single instance. Correlation
|
|
268
|
+
proof: a **06:00–06:08 CST ASN burst == the 11:05 UTC EB `Severe` flip**. Diagnosed via AWS CLI
|
|
269
|
+
(`describe-events`, `describe-configuration-settings`, CloudWatch `get-metric-data`) + prod
|
|
270
|
+
`Logs.Api`/`Logs.Issue`.
|
|
271
|
+
|
|
272
|
+
**Rule:** a "reduced health / no cause" flip on this 2-instance PHP env means the ALB `/health`
|
|
273
|
+
probe was starved of a PHP-FPM worker — do **not** chase client-traffic 5xx. **Fix shipped
|
|
274
|
+
(working tree):** `/health` is now served as a **static Apache file** (`api2/health`), decoupling
|
|
275
|
+
liveness from PHP-FPM worker availability so a slow-request burst can no longer flap it — see
|
|
276
|
+
[health-check endpoint](features/health-check-endpoint.md).
|
|
277
|
+
|
|
257
278
|
## Known issues / accepted risks
|
|
258
279
|
|
|
259
280
|
Open items a maintainer should know before changing this tier. None are "bugs to fix right now" —
|
|
@@ -314,7 +335,11 @@ they are the known sharp edges. Do not re-discover these from scratch.
|
|
|
314
335
|
class lives in `_underscore` (`Model/Client/...`).
|
|
315
336
|
- **Client-specific behavior** belongs in `_Model_<ClientSlug>_X` overrides and
|
|
316
337
|
`Client_ApiPayloadInterceptor` pre/post hooks — not in `V2.php`.
|
|
317
|
-
- **Don't break `/health
|
|
338
|
+
- **Don't break `/health`.** It is now a **static Apache-served file** (`api2/health`, body
|
|
339
|
+
`OK`) — **not** a PHP short-circuit — so the EB/ALB liveness probe is decoupled from PHP-FPM
|
|
340
|
+
worker availability. Do **not** route `/health` back through PHP or remove the file, and don't
|
|
341
|
+
weaken `enforce_https.conf`. (`/v2/health` remains PHP-handled.) See
|
|
342
|
+
[health-check endpoint](features/health-check-endpoint.md).
|
|
318
343
|
- **504s:** verify `long_gateway_timeout.conf` on both this tier and the proxy
|
|
319
344
|
(LB 3600s → proxy Apache 1800s → proxy cURL 900s → api Apache 1800s).
|
|
320
345
|
- **`transactionId` uniqueness is load-bearing** — the idempotency/audit key; don't bypass
|
|
@@ -327,6 +352,7 @@ they are the known sharp edges. Do not re-discover these from scratch.
|
|
|
327
352
|
single-caller branches in `V2.php` as unverified until exercised directly.
|
|
328
353
|
|
|
329
354
|
## Change history
|
|
355
|
+
- 2026-08-28 — Recorded the third EB-`Degraded` cause: a no-stated-cause "reduced health" flip on `api-production-1` is the **PHP-served `/health` probe starved of a PHP-FPM worker** by off-peak bursts of slow failing Compass ASN POSTs (17–63 s, ~100% 500) — invisible in ALB `RequestCount`/`5xx` and never dropping `HealthyHostCount` (flap < 75 s deregistration). Correlated a 06:00–06:08 CST ASN burst to the 11:05 UTC Severe flip. Noted the fix (static-file `/health` decouples liveness from PHP-FPM) and updated the "Don't break `/health`" rule to the static-file form. (jcardinal)
|
|
330
356
|
- 2026-08-26 — Documented and fixed a `/v2/auth/public` OOM: a present-but-null client `transactionId` (`{"transactionId": null}`) flowed into the request-logger's `_Model::load()` dup-check; because `Logs.Api.transactionId` is `FIELD_CHAR`, a null value collapses the WHERE and full-table-scans prod `Logs.Api` → memory exhaustion (the `_underscore` collapsed-WHERE mechanism). V2 only auto-uuids an *absent* transactionId (~L543), so a present null fell through (~L553 read → null). Fix (commit `36ef31f`): coerce null/empty transactionId → uuid in the auth block, and skip `load()` in the final logger dup-check (~L2277) when null/empty. Added the "don't blame the latest deploy — check first-occurrence" triage note. (jcardinal)
|
|
331
357
|
- 2026-08-25 — **Resolved the auto-generated `Api.transactionId` 1062 collision** (was the 2026-07-28
|
|
332
358
|
gotcha): the two DB-insert log sites now uuid the log-row `transactionId` (`_String::generateUuid()`)
|
|
@@ -6,10 +6,11 @@ project: API
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-08-
|
|
10
|
-
owners: ["bala", "mhammontree", "apeterson"]
|
|
9
|
+
updated: 2026-08-28
|
|
10
|
+
owners: ["bala", "mhammontree", "apeterson", "jcardinal"]
|
|
11
11
|
files:
|
|
12
12
|
- api2/.ebextensions/git.php
|
|
13
|
+
- api2/.platform/hooks/prebuild/git.sh
|
|
13
14
|
- api2/.ebextensions/php_include_underscore.config
|
|
14
15
|
- api2/.ebextensions/git.sandbox-dev.json
|
|
15
16
|
- api2/.ebextensions/git.sandbox-client.json
|
|
@@ -177,6 +178,21 @@ the tier trades a stale-code bug for a total outage. Full procedure:
|
|
|
177
178
|
|
|
178
179
|
## Gotchas / known issues
|
|
179
180
|
|
|
181
|
+
- **⚠ Cross-repo change ordering: land `_underscore` BEFORE deploying api2.** Because api2's EB
|
|
182
|
+
deploy **clones `_underscore` (branch `_<ENVIRONMENT>`, e.g. `_production`) at BUILD time**
|
|
183
|
+
(`.platform/hooks/prebuild/git.sh`), a change that spans both repos must have its `_underscore`
|
|
184
|
+
side merged to `_<ENVIRONMENT>` **first** — otherwise the freshly-built api2 will call a framework
|
|
185
|
+
method/class that isn't on the box and fatal. Landing `_underscore` first is **safe on its own**:
|
|
186
|
+
the old api2 controller simply doesn't use the new framework code yet. Order: merge `_underscore`
|
|
187
|
+
→ (verify it's on the branch) → deploy api2.
|
|
188
|
+
- **⚠ A "Could not find required file for `_Model_...`" fatal on prod can be a STALE-BUILD
|
|
189
|
+
artifact, not a missing file.** `Logs.Issue #476`
|
|
190
|
+
("Could not find required file for '_Model_Client_ItemFulfillments_InventoryAdjustment'") fired
|
|
191
|
+
because that model file **was** committed to `_production` but `api-production-1` was still running
|
|
192
|
+
an **older build**; a **redeploy** (which re-pulls `_production`) resolved it with no recurrence.
|
|
193
|
+
Before assuming a model file is absent, check the **running build's `_underscore` against
|
|
194
|
+
`_production`** (file presence on the box; `git merge-base --is-ancestor`) — the build-time clone
|
|
195
|
+
means a merged file may simply not be on that instance yet.
|
|
180
196
|
- **⚠ Merging to the branch named after the EB environment can be a no-op.** Merge to
|
|
181
197
|
`_` + `ENVIRONMENT`. If your change "deployed successfully" but the behavior is absent and no
|
|
182
198
|
error appears, check this before debugging the code. A missing class also throws `Error`
|
|
@@ -241,6 +257,14 @@ the tier trades a stale-code bug for a total outage. Full procedure:
|
|
|
241
257
|
|
|
242
258
|
## Change history
|
|
243
259
|
|
|
260
|
+
- 2026-08-28 — Recorded two build-time-clone consequences: (1) **cross-repo deploy ordering** — api2
|
|
261
|
+
clones `_underscore` (`_<ENVIRONMENT>`) at build via `.platform/hooks/prebuild/git.sh`, so for a
|
|
262
|
+
change spanning both repos, `_underscore` must land on the deploy branch **before** api2 is
|
|
263
|
+
deployed (landing `_underscore` first is safe alone); and (2) a **"Could not find required file
|
|
264
|
+
for `_Model_...`" fatal can be a stale build**, not a missing file — `Logs.Issue #476` fired for
|
|
265
|
+
a model committed to `_production` while `api-production-1` ran an older build, and a redeploy
|
|
266
|
+
resolved it. Check the running build's `_underscore` vs `_production` before assuming the file is
|
|
267
|
+
absent. (jcardinal)
|
|
244
268
|
- 2026-08-27 — Recorded the **inverse hazard**: moving a tier **forward** onto its correct
|
|
245
269
|
`_underscore` branch is a **schema event**. The 2026-08-26 deploy moved `sandbox-client` off the
|
|
246
270
|
`_production` framework (which the 2026-08-25 entry below explains it had been running for weeks)
|
|
@@ -1,14 +1,16 @@
|
|
|
1
1
|
---
|
|
2
|
-
title: "Health-check endpoint (/health liveness short-circuit)"
|
|
2
|
+
title: "Health-check endpoint (/health liveness — static file + PHP short-circuit)"
|
|
3
3
|
framework: "2.0"
|
|
4
4
|
repo: api2
|
|
5
5
|
project: API
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-
|
|
9
|
+
updated: 2026-08-28
|
|
10
10
|
owners: [jcardinal]
|
|
11
11
|
files:
|
|
12
|
+
- api2/health
|
|
13
|
+
- api2/.platform/httpd/conf.d/health_probe.conf
|
|
12
14
|
- api2/Controller/Index.php
|
|
13
15
|
related:
|
|
14
16
|
- ../architecture.md
|
|
@@ -17,13 +19,37 @@ related:
|
|
|
17
19
|
|
|
18
20
|
## Summary
|
|
19
21
|
|
|
20
|
-
`
|
|
21
|
-
**
|
|
22
|
-
load balancer) probes depend on this always returning 200 cheaply — the architecture doc's
|
|
23
|
-
critical rule is literally "Don't break `/health`". This doc records the **matching contract**
|
|
24
|
-
so a teammate doesn't reintroduce a brittle exact-string check.
|
|
22
|
+
The EB/ALB liveness probe (`/health`) is served two ways, and the **load-bearing** one is now
|
|
23
|
+
a **static Apache-served file**, not PHP:
|
|
25
24
|
|
|
26
|
-
|
|
25
|
+
- **`/health` → a static file `api2/health`** (literal body `OK`), served by Apache **without
|
|
26
|
+
ever reaching PHP-FPM**. This is what the EB target group health-check hits. Decoupling the
|
|
27
|
+
probe from PHP means a PHP-FPM worker-pool exhaustion (e.g. a burst of slow ASN requests
|
|
28
|
+
saturating all workers) or a PHP transient can **no longer flap EB health** — the probe
|
|
29
|
+
answers from Apache regardless of app load.
|
|
30
|
+
- **`/v2/health` (and `/health` as a fallback) → PHP short-circuit** in
|
|
31
|
+
`_Controller_Index::api()`, an HTTP 200 returned **before** any routing, DB bootstrap, or V2
|
|
32
|
+
engine work. This still exists for callers that hit the PHP entrypoint directly.
|
|
33
|
+
|
|
34
|
+
This doc records the **matching contract** for both so a teammate doesn't reintroduce a brittle
|
|
35
|
+
exact-string check or accidentally route the liveness probe back through PHP.
|
|
36
|
+
|
|
37
|
+
## The static probe (the one EB actually uses) — 2026-08-28
|
|
38
|
+
|
|
39
|
+
- **`api2/health`** — a one-line static file (`OK`). Apache serves it directly.
|
|
40
|
+
- **`api2/.platform/httpd/conf.d/health_probe.conf`** — comment-only documentation of why the
|
|
41
|
+
probe is static (no directives; it exists to explain the decision at the Apache layer).
|
|
42
|
+
- **Why it works without a rewrite fight:** `api2/.htaccess` only rewrites to `index.php` when
|
|
43
|
+
the request target is **not a real file** (`RewriteCond %{REQUEST_FILENAME} !-f`). Because
|
|
44
|
+
`health` is a real file on disk, the rewrite is skipped and Apache serves it straight.
|
|
45
|
+
- **Why `enforce_https` doesn't redirect it:** EB health checks arrive on **HTTP:80 with no
|
|
46
|
+
`X-Forwarded-Proto`**, and `enforce_https.conf` only redirects requests that carry the
|
|
47
|
+
forwarded-proto marker — so the probe is not bounced to HTTPS.
|
|
48
|
+
- **Do not** move `/health` back into a PHP-served route or rename/remove `api2/health` — that
|
|
49
|
+
re-couples EB liveness to PHP-FPM worker availability, which is the exact failure this fixed
|
|
50
|
+
(see [api2 architecture — EB "Degraded" health](../architecture.md)).
|
|
51
|
+
|
|
52
|
+
## PHP short-circuit (for `/v2/health` and direct hits)
|
|
27
53
|
|
|
28
54
|
The short-circuit lives at the top of `Index.php::api()`, before host dispatch and before the
|
|
29
55
|
Core/Logs DB bootstrap. It matches against a class constant:
|
|
@@ -55,6 +81,15 @@ normalized form: lowercase, no trailing slash, no query) rather than re-adding a
|
|
|
55
81
|
branch.
|
|
56
82
|
|
|
57
83
|
## Change history
|
|
84
|
+
- 2026-08-28 — Moved the **EB/ALB liveness probe off PHP**: `/health` is now a **static file**
|
|
85
|
+
(`api2/health`, body `OK`) served by Apache — it never reaches PHP-FPM, so a slow-request burst
|
|
86
|
+
or a PHP transient can no longer starve the probe and flap EB health. Works because
|
|
87
|
+
`.htaccess` only rewrites to `index.php` for non-files (`!-f`), and `enforce_https` skips the
|
|
88
|
+
probe (HTTP:80, no `X-Forwarded-Proto`). Added a comment-only
|
|
89
|
+
`.platform/httpd/conf.d/health_probe.conf` documenting the decision. `/v2/health` stays
|
|
90
|
+
PHP-handled and the `HEALTH_CHECK_ROUTES` PHP short-circuit remains as a fallback. Working tree
|
|
91
|
+
only (uncommitted). See the [EB "Degraded" health](../architecture.md) diagnosis this fixes.
|
|
92
|
+
(jcardinal)
|
|
58
93
|
- 2026-07-24 — Replaced the brittle exact-string `REQUEST_URI == '/health'` check with a
|
|
59
94
|
`HEALTH_CHECK_ROUTES` constant + normalized (lowercase / strip query / strip trailing slash)
|
|
60
95
|
strict `in_array` match, so `/v2/health`, `/health/`, and `/health?x` all short-circuit to 200
|
|
@@ -1,38 +1,101 @@
|
|
|
1
1
|
---
|
|
2
|
-
title: V2 request
|
|
2
|
+
title: V2 request retry (Prong A) — deadlock AND retriable unique-key race, route-scoped in-process replay
|
|
3
3
|
framework: "2.0"
|
|
4
4
|
repo: api2
|
|
5
5
|
project: API
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: draft
|
|
9
|
-
updated: 2026-08-
|
|
9
|
+
updated: 2026-08-28
|
|
10
10
|
owners: [jcardinal]
|
|
11
11
|
files:
|
|
12
12
|
- api2/Controller/Index.php
|
|
13
|
+
- api2/Component/Api/V2/V2.php
|
|
13
14
|
- _underscore/Database.php
|
|
14
15
|
- _underscore/Query.php
|
|
16
|
+
- _underscore/Model/Compass/AdvanceShippingNotice.php
|
|
15
17
|
related:
|
|
16
18
|
- ../../_underscore/features/model-save-parent-cascade-stored-field-deadlock.md
|
|
19
|
+
- ../../../../clients/compass-usa/features/asn-to-item-fulfillment.md
|
|
17
20
|
- ../architecture.md
|
|
18
21
|
- ./v2-api-error-codes.md
|
|
19
22
|
---
|
|
20
23
|
|
|
21
24
|
## Summary
|
|
22
25
|
|
|
23
|
-
api2's front controller can **detect a
|
|
24
|
-
the whole request in-process**, so a transient
|
|
25
|
-
|
|
26
|
-
(`DEADLOCK_RETRIABLE_ROUTES`) — currently only `/v2/advance-shipping-notices` — because whole-
|
|
27
|
-
request in-process replay is **only safe for routes with no non-rollbackable external side effect**.
|
|
28
|
-
This is the safety-net half of the ASN deadlock fix; the source-level fix (stop the collateral
|
|
29
|
-
parent re-save) is in
|
|
30
|
-
[_underscore's save-cascade doc](../../_underscore/features/model-save-parent-cascade-stored-field-deadlock.md).
|
|
26
|
+
api2's front controller ("**Prong A**") can **detect a retriable concurrency failure and replay
|
|
27
|
+
the whole request in-process**, so a transient collision self-recovers instead of returning an
|
|
28
|
+
error to the caller. Two failure classes now qualify:
|
|
31
29
|
|
|
32
|
-
|
|
30
|
+
1. **MySQL deadlock (1213) / lock-wait timeout (1205)** — the original case (a same-PO ASN write
|
|
31
|
+
deadlock; see the
|
|
32
|
+
[save-cascade doc](../../_underscore/features/model-save-parent-cascade-stored-field-deadlock.md)).
|
|
33
|
+
2. **Retriable unique-key get-or-create race (1062)** — a concurrent same-key INSERT lost the
|
|
34
|
+
race; a whole-request replay on a fresh snapshot lets the retry's GET reuse the winner's
|
|
35
|
+
committed row (added 2026-08-28 for the Compass ASN data-loss race).
|
|
36
|
+
|
|
37
|
+
The replay is **route-scoped by an allowlist** (`DEADLOCK_RETRIABLE_ROUTES`) — currently only
|
|
38
|
+
`/v2/advance-shipping-notices` — because whole-request in-process replay is **only safe for routes
|
|
39
|
+
with no non-rollbackable external side effect**. The retry gate is now
|
|
40
|
+
**`wasDeadlock() || wasRetriableUniqueRace()`**.
|
|
41
|
+
|
|
42
|
+
> Status (2026-08-28): **in the working tree only** — not committed or deployed. Pending sandbox
|
|
33
43
|
> validation + regression tests.
|
|
34
44
|
|
|
35
|
-
##
|
|
45
|
+
## The retriable unique-key race (1062) — Compass ASN data-loss (2026-08-28)
|
|
46
|
+
|
|
47
|
+
`_Model_Compass_AdvanceShippingNotice::postPost` **get-or-creates** an SO-numbered Item
|
|
48
|
+
Fulfillment (GET by number, else POST). `ItemFulfillments.number` is UNIQUE and vendors
|
|
49
|
+
re-transmit the same shipment file, so two concurrent same-number ASNs race: the loser's INSERT
|
|
50
|
+
trips the unique key. Pre-fix, `internalApiRequest` threw (default `throwExceptionsOnError=true`),
|
|
51
|
+
the 500 propagated out of `postPost`, and **the api2 controller rolled back the whole request —
|
|
52
|
+
losing the ASN row and its tracking numbers** (`Logs.Issue #239`, EV-10). This is a bug 1062
|
|
53
|
+
must NOT be treated like a normal error, because a replay recovers it cleanly.
|
|
54
|
+
|
|
55
|
+
**Fix (Approach B — a CTO review rejected a simpler in-place `FOR UPDATE` recovery, because an
|
|
56
|
+
interceptor cannot recover a poisoned response in place; see *V2 concurrency mechanics* below).**
|
|
57
|
+
A generalized retriable-unique-race path:
|
|
58
|
+
|
|
59
|
+
- **`_Database`** gains `const ERRNO_DUPLICATE_ENTRY = 1062`, a sticky
|
|
60
|
+
`public static bool $retriableUniqueRace` (reset per attempt in `clearLastError()`), and
|
|
61
|
+
`wasRetriableUniqueRace()` — true **only** when the caller-set flag is set **AND**
|
|
62
|
+
`lastErrorNumber === 1062`. The flag is caller-opt-in on purpose: a bare 1062 is a real
|
|
63
|
+
constraint violation and must **not** silently retry; only a caller that knows the 1062 is a
|
|
64
|
+
benign get-or-create race sets it.
|
|
65
|
+
- **The ASN model** calls the IF-create `internalApiRequest` with **`throwExceptionsOnError:false`**;
|
|
66
|
+
on an empty response with `lastErrorNumber === 1062` it sets `$retriableUniqueRace = true` and
|
|
67
|
+
returns null. `postPost` then **aborts immediately** (issues no further query).
|
|
68
|
+
- **The controller's Prong A gate** becomes `wasDeadlock() || wasRetriableUniqueRace()`. Replaying
|
|
69
|
+
the whole request on a fresh REPEATABLE-READ snapshot lets the retry's GET see the winner's
|
|
70
|
+
committed IF and reuse it.
|
|
71
|
+
|
|
72
|
+
Bounded by `MAX_DEADLOCK_ATTEMPTS = 3`, gated to `DEADLOCK_RETRIABLE_ROUTES =
|
|
73
|
+
['/v2/advance-shipping-notices']`.
|
|
74
|
+
|
|
75
|
+
## V2 concurrency mechanics (durable framework knowledge)
|
|
76
|
+
|
|
77
|
+
These are the V2-engine facts that decide **how any interceptor-level write race must be handled**
|
|
78
|
+
— they are why the fix is a whole-request replay and not an in-place recovery:
|
|
79
|
+
|
|
80
|
+
- **Response poisoning (load-bearing).** A failed nested save via `internalApiRequest` calls
|
|
81
|
+
`addDefinedMessage(EV-10)` on the **shared** `$api->response`, setting a non-2xx status **before**
|
|
82
|
+
control returns. So even if an interceptor catches the throw and "recovers," `execute()`
|
|
83
|
+
recomputes `isSuccess = false` (`V2.php` ~2154) and **self-rolls-back DB_CORE + DB_CLIENT**
|
|
84
|
+
(~2157-2159). **An interceptor therefore CANNOT recover a failed internal save in place** — the
|
|
85
|
+
only correct recovery is a whole-request replay. This same poisoning is what drives the Prong A
|
|
86
|
+
retry gate and, on exhaustion, produces the coherent EV-10 response.
|
|
87
|
+
- **`internalApiRequest($method, $route, $payload, $options, $throwExceptionsOnError = true)`** — a
|
|
88
|
+
**5th argument** controls throw-vs-return-on-error. Pass `false` to inspect `lastErrorNumber`
|
|
89
|
+
instead of unwinding.
|
|
90
|
+
- **`_Database::$lastErrorNumber` is sticky.** Set only on query failure, never cleared by a later
|
|
91
|
+
success; reset per attempt via `clearLastError()`. A recorded **deadlock (1213/1205) is protected
|
|
92
|
+
from overwrite** by a later error; a **1062 is NOT** protected — so read/act on a 1062 promptly.
|
|
93
|
+
- **The prod client cluster runs REPEATABLE-READ.** A same-transaction re-SELECT after a losing
|
|
94
|
+
concurrent INSERT is **blind to the winner's commit** — only a locking read or a **fresh-snapshot
|
|
95
|
+
whole-request replay** sees it. This is why in-place retry inside the transaction cannot work and
|
|
96
|
+
the replay must start a new transaction.
|
|
97
|
+
|
|
98
|
+
## How it works — deadlock path
|
|
36
99
|
|
|
37
100
|
**Detection primitive (`_underscore`, reusable).** `_Database` gains
|
|
38
101
|
`public static ?int $lastErrorNumber`, the consts `ERRNO_DEADLOCK` (1213) and
|
|
@@ -42,7 +105,8 @@ driver errno **at its throw site**, and the recording is **sticky**: a later unr
|
|
|
42
105
|
inside a model-save `try/catch` still be visible to the outer controller and trigger a retry.
|
|
43
106
|
|
|
44
107
|
**Retry loop (`api2/Controller/Index.php`).** The v2 dispatch is wrapped in a bounded loop
|
|
45
|
-
(`MAX_DEADLOCK_ATTEMPTS = 3`). On an unsuccessful response where
|
|
108
|
+
(`MAX_DEADLOCK_ATTEMPTS = 3`). On an unsuccessful response where
|
|
109
|
+
`_Database::wasDeadlock() || _Database::wasRetriableUniqueRace()` is true
|
|
46
110
|
and the route is in `DEADLOCK_RETRIABLE_ROUTES`, the controller:
|
|
47
111
|
|
|
48
112
|
1. rolls back all open transactions;
|
|
@@ -82,6 +146,22 @@ superglobal. Default is to leave a route off the list.
|
|
|
82
146
|
not a substitute for removing the deadlock at its source.
|
|
83
147
|
|
|
84
148
|
## Change history
|
|
149
|
+
- 2026-08-28 — **Generalized Prong A to a second retriable class: the unique-key (1062)
|
|
150
|
+
get-or-create race**, fixing the Compass ASN data-loss bug (`Logs.Issue #239`, EV-10) where a
|
|
151
|
+
concurrent same-`ItemFulfillments.number` INSERT lost the race, `internalApiRequest` threw, and
|
|
152
|
+
the whole request rolled back — losing the ASN row and its tracking. Added
|
|
153
|
+
`_Database::ERRNO_DUPLICATE_ENTRY = 1062`, a sticky caller-opt-in `$retriableUniqueRace`
|
|
154
|
+
(reset in `clearLastError()`), and `wasRetriableUniqueRace()` (flag AND `lastErrorNumber===1062`);
|
|
155
|
+
the ASN model calls the IF-create with `throwExceptionsOnError:false`, sets the flag on an empty
|
|
156
|
+
1062 response, and aborts; the controller gate became
|
|
157
|
+
`wasDeadlock() || wasRetriableUniqueRace()`. Chose the whole-request replay (Approach B) over an
|
|
158
|
+
in-place `FOR UPDATE` recovery after a CTO review, because V2 **response poisoning** (a failed
|
|
159
|
+
nested save flips the shared `$api->response` to non-2xx and `execute()` self-rolls-back
|
|
160
|
+
DB_CORE+DB_CLIENT, `V2.php` ~2154-2159) makes in-place interceptor recovery impossible, and the
|
|
161
|
+
prod cluster's REPEATABLE-READ isolation makes a same-transaction re-SELECT blind to the winner's
|
|
162
|
+
commit. Recorded the durable V2 concurrency mechanics (response poisoning, `internalApiRequest`
|
|
163
|
+
5th arg, sticky-`lastErrorNumber` with 1213 protected / 1062 not, REPEATABLE-READ). Working tree
|
|
164
|
+
only, pending sandbox validation. (jcardinal)
|
|
85
165
|
- 2026-08-13 — Added deadlock/lock-wait detection to `_Database`/`_Query` (`$lastErrorNumber`,
|
|
86
166
|
`wasDeadlock()`, sticky errno recording) and a bounded, route-scoped in-process replay loop in
|
|
87
167
|
`Controller/Index.php`, gated to `/v2/advance-shipping-notices` only. Documented why whole-request
|
|
@@ -223,6 +223,31 @@ corresponding committed file — are in
|
|
|
223
223
|
[surface-layer-schema](../apps/dbchanges2/features/surface-layer-schema.md) and
|
|
224
224
|
[surface-resolver](../apps/_underscore/features/surface-resolver.md).
|
|
225
225
|
|
|
226
|
+
### Standard column ordering — `id`, `uuid`, `dtCreated`, `dtUpdated`, then the rest
|
|
227
|
+
|
|
228
|
+
Every table uses a fixed leading column order: `id` (primary key), then `uuid`, then **`dtCreated`
|
|
229
|
+
immediately after `uuid`**, then **`dtUpdated` immediately after `dtCreated`**, and the business
|
|
230
|
+
columns after that. This is a hard, fundamental team convention — it keeps the audit columns in the
|
|
231
|
+
same, predictable place on every table platform-wide.
|
|
232
|
+
|
|
233
|
+
- **New tables (`CREATE TABLE`):** declare the columns in that order.
|
|
234
|
+
- **Adding the timestamps to an existing table (`ALTER TABLE`):** position them explicitly with
|
|
235
|
+
`AFTER`, because a bare `ADD COLUMN` appends to the **end** of the table and violates this standard:
|
|
236
|
+
|
|
237
|
+
```sql
|
|
238
|
+
ALTER TABLE MyTable
|
|
239
|
+
ADD COLUMN dtCreated DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP AFTER uuid,
|
|
240
|
+
ADD COLUMN dtUpdated DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP AFTER dtCreated;
|
|
241
|
+
```
|
|
242
|
+
|
|
243
|
+
To reposition timestamps already appended at the end of an existing table, use
|
|
244
|
+
`MODIFY COLUMN dtCreated ... AFTER uuid` / `MODIFY COLUMN dtUpdated ... AFTER dtCreated`.
|
|
245
|
+
|
|
246
|
+
`dtCreated` / `dtUpdated` are `DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP` (with `dtUpdated` also
|
|
247
|
+
`ON UPDATE CURRENT_TIMESTAMP`), matching the model's `FIELD_DATETIME_CREATED` /
|
|
248
|
+
`FIELD_DATETIME_UPDATED` field types. Bridge tables that carry no timestamps (only `id`, `uuid`, and
|
|
249
|
+
the two FK columns) are exempt — the rule is about *where* the timestamps go when a table has them.
|
|
250
|
+
|
|
226
251
|
## Checking dependencies before touching shared code
|
|
227
252
|
|
|
228
253
|
`dependsOn` in `knowledge/registry.json` means a repo extends or depends on another repo's classes. Before modifying a class in a dependency repo (e.g. `_underscore` core):
|
|
@@ -308,6 +333,11 @@ ACL-governed field treatment.
|
|
|
308
333
|
did not write them back, so current state must be established with read-only queries against the
|
|
309
334
|
target environment, and the cluster-isolation hook catches cross-database references but
|
|
310
335
|
**cannot** catch state drift. (apeterson)
|
|
336
|
+
- 2026-08-28 — Documented the **standard leading column order** (`id`, `uuid`, then `dtCreated`
|
|
337
|
+
immediately after `uuid`, then `dtUpdated` immediately after `dtCreated`, then business columns),
|
|
338
|
+
including using `AFTER` on `ALTER TABLE ADD COLUMN` so timestamps are positioned rather than
|
|
339
|
+
appended. Prompted by the platform-wide `TransferOrderItems` drift where the audit columns were
|
|
340
|
+
missing entirely. (jcardinal)
|
|
311
341
|
- 2026-07-29 — Documented the `c_`-prefixed framework-dynamic column convention (schema-only,
|
|
312
342
|
no model-class/RecordFields/ACL declaration), generalized from the transcript AI-model routing
|
|
313
343
|
work. (ajean)
|
package/knowledge/INDEX.md
CHANGED
|
@@ -4,7 +4,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
4
4
|
|
|
5
5
|
## 1.0 framework
|
|
6
6
|
|
|
7
|
-
- **library** (Library) _(framework core)_ —
|
|
7
|
+
- **library** (Library) _(framework core)_ — 21 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
|
|
8
8
|
- **worker** (Worker) — 31 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
|
|
9
9
|
- **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
|
|
10
10
|
- **worker1.5** (Worker 1.5) — 0 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
|
|
@@ -5,7 +5,7 @@ project: _Underscore
|
|
|
5
5
|
client: compass-usa
|
|
6
6
|
type: client-feature
|
|
7
7
|
status: active
|
|
8
|
-
updated: 2026-08-
|
|
8
|
+
updated: 2026-08-28
|
|
9
9
|
owners: [jcardinal, bala]
|
|
10
10
|
files:
|
|
11
11
|
- _underscore/Model/Compass/AdvanceShippingNotice.php
|
|
@@ -200,6 +200,20 @@ Compass Canada (`Model/Compass/Canada/`) is a separate sub-client. The Compass c
|
|
|
200
200
|
(`$_model_skipParentCascadeOnSave`), and api2 retries the ASN route on a detected deadlock. Full
|
|
201
201
|
mechanic:
|
|
202
202
|
[save-cascade stored-field deadlock](../../../2.0/apps/_underscore/features/model-save-parent-cascade-stored-field-deadlock.md).
|
|
203
|
+
- **Concurrent same-SO-number ASNs lost the ASN + its tracking to a unique-key race (fixed
|
|
204
|
+
2026-08-28, working tree).** `postPost` get-or-creates the SO-numbered IF (GET by number, else
|
|
205
|
+
POST). `ItemFulfillments.number` is UNIQUE and vendors re-transmit the same shipment file, so two
|
|
206
|
+
concurrent same-number ASNs race: the loser's IF INSERT hits the unique key (MySQL **1062**),
|
|
207
|
+
`internalApiRequest` threw (default `throwExceptionsOnError=true`), the 500 propagated out of
|
|
208
|
+
`postPost`, and the api2 controller **rolled back the whole request — losing the ASN row AND its
|
|
209
|
+
tracking numbers** (`Logs.Issue #239`, EV-10). Fix: the IF-create call now passes
|
|
210
|
+
`throwExceptionsOnError:false`; on an empty response with `lastErrorNumber===1062` the model sets
|
|
211
|
+
`_Database::$retriableUniqueRace = true` and returns null, `postPost` aborts, and api2's Prong A
|
|
212
|
+
replays the whole request on a fresh REPEATABLE-READ snapshot so the retry's GET reuses the
|
|
213
|
+
winner's committed IF. Distinct from the 1213 deadlock above (a different concurrency failure on
|
|
214
|
+
the same re-transmission traffic). Full mechanic + why in-place recovery is impossible (V2
|
|
215
|
+
response poisoning):
|
|
216
|
+
[V2 request retry — Prong A](../../../2.0/apps/api2/features/v2-deadlock-retry.md).
|
|
203
217
|
- **Cross-line tracking contamination from bad SOI↔POI bridge rows (root cause, fixed
|
|
204
218
|
forward 2026-06-11):** `postPost` resolves which IFI(s) a tracking number attaches to by
|
|
205
219
|
joining `AdvanceShippingNoticeItems → SalesOrderItems_PurchaseOrderItems → SalesOrderItems`.
|
|
@@ -313,6 +327,13 @@ feed cannot currently support a line-level completeness test.
|
|
|
313
327
|
|
|
314
328
|
## Change history
|
|
315
329
|
Dated one-liners, newest first.
|
|
330
|
+
- 2026-08-28 — Fixed a **data-loss unique-key race** on concurrent same-SO-number ASNs: the loser's
|
|
331
|
+
SO-numbered IF INSERT trips `ItemFulfillments.number` UNIQUE (1062), `internalApiRequest` threw,
|
|
332
|
+
and api2 rolled back the whole request, losing the ASN row + tracking (`Logs.Issue #239`, EV-10).
|
|
333
|
+
The model now creates the IF with `throwExceptionsOnError:false`, flags a retriable 1062
|
|
334
|
+
(`_Database::$retriableUniqueRace`) and aborts, and api2's Prong A replays the request on a fresh
|
|
335
|
+
snapshot so the retry reuses the winner's IF. Distinct from the 2026-08-13 1213 deadlock. Working
|
|
336
|
+
tree only; also swept 6 banned `throw new _Exception(...)` → `\Exception` in the file. (jcardinal)
|
|
316
337
|
- 2026-08-27 — Quantified the **Office Depot duplicate-ASN** problem on prod and settled what it is:
|
|
317
338
|
of ODP PO lines with more than one shipping notice, **10,012 exceed the PO line quantity and only 8
|
|
318
339
|
are genuine split shipments**, so a second notice on an ODP line is a **duplicate**, not a partial.
|
|
@@ -5,6 +5,7 @@ apps:
|
|
|
5
5
|
- _underscore
|
|
6
6
|
- api2
|
|
7
7
|
- worker2
|
|
8
|
+
- worker
|
|
8
9
|
- toga2-supply
|
|
9
10
|
- dbchanges2
|
|
10
11
|
- library
|
|
@@ -12,8 +13,8 @@ project: _Underscore
|
|
|
12
13
|
client: growrk
|
|
13
14
|
type: profile
|
|
14
15
|
status: active
|
|
15
|
-
updated: 2026-08-
|
|
16
|
-
owners: ["rgirish", "mhammontree", "bala"]
|
|
16
|
+
updated: 2026-08-28
|
|
17
|
+
owners: ["rgirish", "mhammontree", "bala", "jcardinal"]
|
|
17
18
|
files: []
|
|
18
19
|
related:
|
|
19
20
|
- clients/growrk/features/transfer-order-flow.md
|
|
@@ -56,6 +57,13 @@ module (`dbchanges2/Client_Growrk/_modules.txt`).
|
|
|
56
57
|
2026-08-01, but it fails `POST /item-fulfillments` with **EV-12** and makes the Fulfill & Ship
|
|
57
58
|
ship-to address arbitrary (`salesOrders[0]`). Needs its own ticket — see the
|
|
58
59
|
[Fulfill & Ship gotchas](../../2.0/apps/toga2-supply/features/fulfill-and-ship.md).
|
|
60
|
+
- **GroWrk runs the shared 1.0 NetSuite→TOGa Supply sync (`worker`/`library`), and its transfer
|
|
61
|
+
orders share the transfer-order code path.** The 2026-08-28 `SALES_ORDERS` freeze on
|
|
62
|
+
`syncTransferOrderFromNetsuite` (the api2-omits-a-null-FK `$transferOrderStage` warning →
|
|
63
|
+
fatal, plus the platform-wide `TransferOrderItems` missing-timestamp-columns 1054) hit GroWrk too,
|
|
64
|
+
not just NYCHH; both were fixed in shared code. See
|
|
65
|
+
[NYCHH NetSuite → TransferOrders import](../nychh/features/netsuite-transfer-order-import.md) (freeze
|
|
66
|
+
chain) — the fixes are shared, only the routing gate is NYCHH-specific.
|
|
59
67
|
- **`ShippingMethods` ids 11/12 ("Overnight Standard"/"Overnight Priority") have NULL `code`** — FedEx
|
|
60
68
|
product names on UPS rows, duplicating Next Day Air. Selecting either fails at UPS (120500);
|
|
61
69
|
awaiting a data-owner decision. Ids 3/4 were coded by
|
|
@@ -2,7 +2,8 @@
|
|
|
2
2
|
|
|
3
3
|
| Doc | Framework | Summary | Files |
|
|
4
4
|
|-----|-----------|---------|-------|
|
|
5
|
-
| [NYCHH
|
|
5
|
+
| [NYCHH inventory-adjustment → item-fulfillment linking (custbody_stock_adjustment bridges)](features/netsuite-inventory-adjustment-fulfillment-link.md) | 1.0 | For NYCHH stock adjustments, the NetSuite **inventory adjustment** carries a link to the **Item Fulfillment** it corrects, on the multi-select body custom field | library/app/api/toga2.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_hh.php, worker/crons/toga2/netsuite/diagnose_hh_import_inventory_adjustment.php, _underscore/Model/Client/ItemFulfillments/InventoryAdjustment.php, _underscore/Model/Client/ItemFulfillmentItems/InventoryAdjustmentItem.php, dbchanges2/Core/2026-08-27b - ItemFulfillmentsInventoryAdjustmentsBridge.sql, dbchanges2/Core/2026-08-28b - ItemFulfillmentItemsInventoryAdjustmentItemsBridge.sql, dbchanges2/Client/2026-08-27a - ItemFulfillmentsInventoryAdjustmentsBridgeTable.sql, dbchanges2/Client/2026-08-28c - ItemFulfillmentItemsInventoryAdjustmentItemsBridgeTable.sql |
|
|
6
|
+
| [NYCHH NetSuite → TransferOrders import (stocking-flag routing, PO bridges, origin location)](features/netsuite-transfer-order-import.md) | 1.0 | NYCHH's NetSuite **transfer orders are represented as NetSuite sales orders** (same as the GroWrk pattern), so the shared NetSuite → TOGa Supply importer (`App_ | library/app/api/toga2.php, library/app/api/netsuite/rest.php, library/app/netsuite.php, worker/crons/toga2/netsuite/sync_togasupply_hh.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/diagnose_hh_stuck_transfer_order.php, dbchanges2/Core/2026-08-27a - TransferOrderUuidSearchableIdentifier.sql, dbchanges2/Client/2026-08-28a - TransferOrderItemsTimestampColumns.sql, dbchanges2/Client/2026-08-28b - TransferOrderItemsTimestampColumnsReposition.sql |
|
|
6
7
|
| [NYCHH PO links are UPSTREAM — the downstream SO→PO route returns empty](features/po-number-upstream-direction.md) | 2.0 | NYCHH's sales orders are created **from the customer's purchase order**, so their SO↔PO links live in the **upstream** table `PurchaseOrders_SalesOrders` (route | _underscore/Model/Client/SalesOrder.php, _underscore/Trait/Netsuite/SalesOrder.php, toga25-supply/src/pages/SalesOrders/view/SalesOrderRecordModalLayout/hooks/usePurchaseOrderDetails.ts, dbchanges2/Client/2026-08-11b - SalesOrderPurchaseOrdersField.sql, dbchanges2/Client/2026-08-25 - SalesOrderPurchaseOrdersFieldAllClients.sql |
|
|
7
8
|
| [NYCHH transfer-order 2.0 model — inventory quantities + V2 field enablement](features/transfer-order-inventory-quantities.md) | 2.0 | The 2.0 (`_underscore`) side of NYCHH transfer-order support: a **two-branch inventory quantity model** on the NYCHH `Item` override, plus the **client-override | _underscore/Model/Nychh/Item.php, _underscore/Model/Nychh/TransferOrder.php, _underscore/Model/Nychh/TransferOrderStage.php, _underscore/Model/Client/TransferOrder.php, _underscore/Model/Client/TransferOrderItem.php, dbchanges2/Client_Nychh/2026-08-27a - TransferOrderCustomFieldWritePermission.sql, dbchanges2/Client_Nychh/2026-08-27b - TransferOrderStageStatusIdentifier.sql, dbchanges2/Client_Nychh/2026-08-28c - TransferOrderRecordAcl.sql, dbchanges2/Client/2026-08-28b - TransferOrderItemsTimestamps.sql |
|
|
8
9
|
| [NYC Health & Hospitals](profile.md) | 2.0 | NYC Health & Hospitals (NYCHH) is a TOGA 2.0 client on the `_underscore` platform, prod schema `Client_Nychh`. | dbchanges2/Client_Nychh/2026-08-18 - InventoryUnitsItemColumns.sql, dbchanges2/Client_Nychh/2026-08-24 - FixInventoryGroupingsUnitsTopologyOverrideIds.sql, dbchanges2/Client_Nychh/2026-08-27a - TransferOrderCustomFieldWritePermission.sql, dbchanges2/Client_Nychh/2026-08-27b - TransferOrderStageStatusIdentifier.sql, dbchanges2/Client_Nychh/2026-08-28a - TransferOrdersTableView.sql, dbchanges2/Client_Nychh/2026-08-28b - TransferOrdersNavigationEnable.sql, dbchanges2/Client_Nychh/2026-08-28c - TransferOrderRecordAcl.sql, dbchanges2/Client/2026-08-28b - TransferOrderItemsTimestamps.sql |
|
|
@@ -0,0 +1,172 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "NYCHH inventory-adjustment → item-fulfillment linking (custbody_stock_adjustment bridges)"
|
|
3
|
+
framework: "1.0"
|
|
4
|
+
repo: library
|
|
5
|
+
project: Library
|
|
6
|
+
client: nychh
|
|
7
|
+
type: client-feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-08-28
|
|
10
|
+
owners: [jcardinal]
|
|
11
|
+
files:
|
|
12
|
+
- library/app/api/toga2.php
|
|
13
|
+
- worker/crons/toga2/netsuite/common_sync_togasupply.php
|
|
14
|
+
- worker/crons/toga2/netsuite/sync_togasupply_hh.php
|
|
15
|
+
- worker/crons/toga2/netsuite/diagnose_hh_import_inventory_adjustment.php
|
|
16
|
+
- _underscore/Model/Client/ItemFulfillments/InventoryAdjustment.php
|
|
17
|
+
- _underscore/Model/Client/ItemFulfillmentItems/InventoryAdjustmentItem.php
|
|
18
|
+
- dbchanges2/Core/2026-08-27b - ItemFulfillmentsInventoryAdjustmentsBridge.sql
|
|
19
|
+
- dbchanges2/Core/2026-08-28b - ItemFulfillmentItemsInventoryAdjustmentItemsBridge.sql
|
|
20
|
+
- dbchanges2/Client/2026-08-27a - ItemFulfillmentsInventoryAdjustmentsBridgeTable.sql
|
|
21
|
+
- dbchanges2/Client/2026-08-28c - ItemFulfillmentItemsInventoryAdjustmentItemsBridgeTable.sql
|
|
22
|
+
related:
|
|
23
|
+
- ./netsuite-transfer-order-import.md
|
|
24
|
+
- ./transfer-order-inventory-quantities.md
|
|
25
|
+
- ../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md
|
|
26
|
+
- ../../../1.0/apps/library/features/toga2-api-client-and-bridge.md
|
|
27
|
+
- ../../../2.0/apps/_underscore/features/sales-order-purchase-order-bridge-direction.md
|
|
28
|
+
- ../profile.md
|
|
29
|
+
---
|
|
30
|
+
|
|
31
|
+
## Summary
|
|
32
|
+
|
|
33
|
+
For NYCHH stock adjustments, the NetSuite **inventory adjustment** carries a link to the **Item
|
|
34
|
+
Fulfillment** it corrects, on the multi-select body custom field **`custbody_stock_adjustment`**
|
|
35
|
+
(internalId **9735**). This feature imports that link into TOGa Supply as two new bridge tables
|
|
36
|
+
(header + line), so H&H gets **adjustment → item fulfillment → sales order → purchase order**
|
|
37
|
+
traceability for their PO-specific fulfillment. Per the **2026-08-27 TOGa Supply meeting**; the
|
|
38
|
+
NetSuite custom field was implemented **<1 year ago**, so older adjustments carry **no** link.
|
|
39
|
+
|
|
40
|
+
Like [NYCHH transfer-order import](./netsuite-transfer-order-import.md), the **tables and models are
|
|
41
|
+
all-client infrastructure** (`dbchanges2/Client/` + shared `_Model_Client_*` bridge models) but the
|
|
42
|
+
**attach is gated NYCHH-only** by `IS_ENABLED_TRANSFER_ORDER_STOCKING_FLAG_ROUTING`.
|
|
43
|
+
|
|
44
|
+
## The two bridges (both mirror the transfer-order bridge pattern)
|
|
45
|
+
|
|
46
|
+
| Level | Bridge table | `Core.Records` id | `RecordFields` | Model |
|
|
47
|
+
|---|---|---|---|---|
|
|
48
|
+
| Header | `ItemFulfillments_InventoryAdjustments` | 353 | 2515–2518 | `_Model_Client_ItemFulfillments_InventoryAdjustment` |
|
|
49
|
+
| Item | `ItemFulfillmentItems_InventoryAdjustmentItems` | 354 | 2519–2522 | `_Model_Client_ItemFulfillmentItems_InventoryAdjustmentItem` |
|
|
50
|
+
|
|
51
|
+
Each shipped as: an all-client `Client/` bridge table, Core registration (`Records` + `RecordFields`),
|
|
52
|
+
a Super-User ACL grant, and a per-client ACL **copied from the parent record**. The parent record
|
|
53
|
+
uuids are already `isIdentifier = 1`, so — unlike the transfer-order bridges — **no `EV-12`
|
|
54
|
+
uuid-identifier fix was needed** (contrast the 312/313 anomaly on
|
|
55
|
+
[the ACL permission chain](../../../2.0/apps/_underscore/features/acl-permission-chain.md)).
|
|
56
|
+
|
|
57
|
+
> **Bridge FK writes via a nested `{uuid}` need only record-level ACL, not field-level.** Verified
|
|
58
|
+
> against the existing transfer-order bridges — the nested-write ACL check is at the record level for a
|
|
59
|
+
> bridge link, so no `AclCustomFieldPermissions` grant is required on the FK columns.
|
|
60
|
+
|
|
61
|
+
## Attach code — `App_Api_Toga2::attachInventoryAdjustmentToItemFulfillment`
|
|
62
|
+
|
|
63
|
+
Lives in `library/app/api/toga2.php`, called from `syncInventoryAdjustmentFromNetsuite` **after the
|
|
64
|
+
header upsert**, gated on `IS_ENABLED_TRANSFER_ORDER_STOCKING_FLAG_ROUTING`, and wrapped in
|
|
65
|
+
`try/catch` — it is **best-effort and must never freeze the section** (a failed link is logged, the
|
|
66
|
+
adjustment import still completes). It:
|
|
67
|
+
|
|
68
|
+
1. reads `custbody_stock_adjustment` off the adjustment (the related Item Fulfillment),
|
|
69
|
+
2. resolves that fulfillment by `ItemFulfillments.c_netsuiteInternalItemFulfillmentId`,
|
|
70
|
+
3. idempotently creates the **header** link row, then
|
|
71
|
+
4. links **line items** by matching `itemId` — every same-product adjustment-item ↔ fulfillment-item
|
|
72
|
+
pair is linked; the bridge's `UNIQUE(fk, fk)` dedupes re-runs.
|
|
73
|
+
|
|
74
|
+
## On-demand dependency import (fulfillment not yet in TOGA)
|
|
75
|
+
|
|
76
|
+
When the linked fulfillment has **not** been imported into TOGA yet, the attach no longer skips and
|
|
77
|
+
waits for the fulfillment backfill — it **imports the fulfillment on demand** via
|
|
78
|
+
`syncItemFulfillmentFromNetsuite` (which itself cascades to the originating sales order when missing),
|
|
79
|
+
then **re-resolves and links**.
|
|
80
|
+
|
|
81
|
+
This required threading the **full item-fulfillment lookup set** — customer, country, state, location,
|
|
82
|
+
manufacturer, item, all-catalog-items, vendor-item, shipping-carrier, shipping-method, warehouse —
|
|
83
|
+
through `syncInventoryAdjustmentFromNetsuite` (signature **expanded 5 → 13 args**) and into
|
|
84
|
+
`attachInventoryAdjustmentToItemFulfillment`, plus updating the `common_sync_togasupply.php`
|
|
85
|
+
`INVENTORY_ADJUSTMENTS` call to pass them (the same set already passed to the item-fulfillment
|
|
86
|
+
section). **Design rule (developer instruction): all lookups are passed function-to-function**, like
|
|
87
|
+
the other sync functions — not rebuilt or fetched inside the callee.
|
|
88
|
+
|
|
89
|
+
## Two fixes in `syncInventoryAdjustmentFromNetsuite` (shared engine, surfaced by the harness)
|
|
90
|
+
|
|
91
|
+
Both are in the shared library method, so they affect **any** client running inventory adjustments —
|
|
92
|
+
see also the [per-client sync](../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md):
|
|
93
|
+
|
|
94
|
+
- **(a) Prune-order FK violation.** The stale-item prune deleted an `InventoryAdjustmentItem` while
|
|
95
|
+
its child `InventoryAdjustmentItemUnits` still referenced it (FK **RESTRICT**, no cascade) → MySQL
|
|
96
|
+
**1451** → `EV-11`. **Fix:** delete the child units (`/inventory-adjustment-item-units/{uuid}`)
|
|
97
|
+
**before** the item. (Same "delete bridge/child rows first" rule as the tracking-number bridges.)
|
|
98
|
+
- **(b) Item-lookup indexing mismatch → duplicate items every run.** The item lookup was consumed as a
|
|
99
|
+
**two-level** `[$uuidManufacturer][$partNumber]`, but the cron passes it **one-level** `[$partNumber]`
|
|
100
|
+
(populated in `common_sync_togasupply.php` ~L313). The two-level
|
|
101
|
+
`$lookupItemByClientUuidAndManufacturerUuidAndPartNumberUpper` is declared but **never populated
|
|
102
|
+
(dead)**. The mismatch meant existing items were never found, so items were **re-created every run**
|
|
103
|
+
(duplicates / `EV-10`). **Fix:** index one-level `[$partNumber]`.
|
|
104
|
+
|
|
105
|
+
## Diagnostic harness — `diagnose_hh_import_inventory_adjustment.php`
|
|
106
|
+
|
|
107
|
+
A **manual operator tool** in `worker/crons/toga2/netsuite/` (NOT in `cron.worker.sync.json`; does
|
|
108
|
+
**not** call `cronInitialization`, so no process lock / no cursor writes; installs a strict
|
|
109
|
+
`set_error_handler` that rethrows warnings as `ErrorException` for exact crash lines). It imports
|
|
110
|
+
**one** inventory adjustment (default NS internal id **6516616**) by calling the real
|
|
111
|
+
`syncInventoryAdjustmentFromNetsuite`, seeds the warehouse/item/manufacturer lookups (the item loop
|
|
112
|
+
**filters** items to known warehouses and **creates** items on a lookup-miss, so empty lookups drop or
|
|
113
|
+
duplicate items), and has a **read-only** diagnostic tail that dumps the NS item list +
|
|
114
|
+
`custbody_stock_adjustment` + whether the referenced fulfillment already exists in TOGA.
|
|
115
|
+
|
|
116
|
+
```
|
|
117
|
+
php crons/toga2/netsuite/diagnose_hh_import_inventory_adjustment.php <togaClient> <togaApi> <togaSecret> [adjustmentInternalId]
|
|
118
|
+
```
|
|
119
|
+
(creds = the `CLIENT_CONFIGURATION` client/api/secret from `sync_togasupply_hh.php`.)
|
|
120
|
+
|
|
121
|
+
## Go-live toggle
|
|
122
|
+
|
|
123
|
+
`IS_ENABLED_INTEGRATION_INVENTORY_ADJUSTMENTS` is still **`false`** in `sync_togasupply_hh.php` —
|
|
124
|
+
enabling it is the go-live switch for this feature on the scheduled cron.
|
|
125
|
+
|
|
126
|
+
## Open / next-session — migrate the inventory-adjustment sync from SOAP to REST
|
|
127
|
+
|
|
128
|
+
The inventory-adjustment sync is still **SOAP**: `App_NetSuite::getInventoryNumberFromSerialNumber` +
|
|
129
|
+
`getInventoryNumber` per serial (`toga2.php` ~5371/5374), plus `getObject('inventoryAdjustment')`,
|
|
130
|
+
`getItemDetails`, `detectItemType`, `getItemIsSerialized`, `getItemIsFulfillable`. For a **serialized**
|
|
131
|
+
adjustment this is ~2 NetSuite SOAP round-trips **per serial** → ~1 hour for a large adjustment; the
|
|
132
|
+
on-demand fulfillment import inherits the same cost. **Decision:** migrate the whole
|
|
133
|
+
inventory-adjustment sync to REST via `App_Api_Netsuite_Rest` (`rest.php` already has
|
|
134
|
+
`send`/`authenticate`/`suiteqlListAll`/`getCustomFieldValue`/`fetchItemById`/`listInventoryAdjustments`
|
|
135
|
+
(minimal) — the gaps are a full inventory-adjustment fetch with inventory detail/serials, and an
|
|
136
|
+
inventory-number/serial batch lookup). **This is the next session's task.**
|
|
137
|
+
|
|
138
|
+
## Known state — test adjustment 6516616
|
|
139
|
+
|
|
140
|
+
NYCHH adjustment **6516616** (tranId **2136**) has **4 NON-serialized** items (adjustQtyBy
|
|
141
|
+
1800/1800/700/40) and `custbody_stock_adjustment` → Item Fulfillment **#289938** (NS internal
|
|
142
|
+
6507274). That fulfillment is **not yet imported** in TOGA, so the bridges stay empty until the
|
|
143
|
+
on-demand import lands it. Import + item-sync of the adjustment itself works (4
|
|
144
|
+
`InventoryAdjustmentItems`).
|
|
145
|
+
|
|
146
|
+
## Gotchas / known issues
|
|
147
|
+
|
|
148
|
+
- **Older adjustments have no link.** `custbody_stock_adjustment` was added in NetSuite <1 year ago;
|
|
149
|
+
adjustments predating it will never populate a bridge — not a bug.
|
|
150
|
+
- **The attach is best-effort by design.** It is gated NYCHH-only and wrapped in `try/catch`; a link
|
|
151
|
+
failure logs and lets the adjustment import complete. Do not "harden" it into a throw — a bridge miss
|
|
152
|
+
must not freeze `INVENTORY_ADJUSTMENTS`.
|
|
153
|
+
- **Empty lookups drop or duplicate items.** The item loop filters to known warehouses and creates
|
|
154
|
+
items on lookup-miss, so any manual/diagnostic invocation must seed the warehouse/item/manufacturer
|
|
155
|
+
lookups (the harness does).
|
|
156
|
+
|
|
157
|
+
## Change history
|
|
158
|
+
|
|
159
|
+
- 2026-08-28 — Built NYCHH inventory-adjustment → item-fulfillment linking off NetSuite
|
|
160
|
+
`custbody_stock_adjustment` (internalId 9735): two all-client bridges
|
|
161
|
+
(`ItemFulfillments_InventoryAdjustments` record 353 / `ItemFulfillmentItems_InventoryAdjustmentItems`
|
|
162
|
+
record 354, Core-registered + Super-User/per-client ACL), a gated best-effort
|
|
163
|
+
`App_Api_Toga2::attachInventoryAdjustmentToItemFulfillment` (resolves the fulfillment by
|
|
164
|
+
`c_netsuiteInternalItemFulfillmentId`, idempotent header + itemId-matched line links), and **on-demand
|
|
165
|
+
import** of a not-yet-imported fulfillment via `syncItemFulfillmentFromNetsuite` (cascades to the SO)
|
|
166
|
+
— which expanded `syncInventoryAdjustmentFromNetsuite` 5→13 args to thread the full IF lookup set.
|
|
167
|
+
Fixed two shared-engine bugs: the stale-item prune deleted an item before its child units (FK
|
|
168
|
+
RESTRICT → 1451/EV-11 — now deletes units first), and the item lookup was consumed two-level while the
|
|
169
|
+
cron populates it one-level `[$partNumber]` (re-created items every run → duplicates/EV-10 — now
|
|
170
|
+
one-level). Added the `diagnose_hh_import_inventory_adjustment.php` operator harness. Go-live toggle
|
|
171
|
+
`IS_ENABLED_INTEGRATION_INVENTORY_ADJUSTMENTS` still false. **Open (next session):** migrate the
|
|
172
|
+
inventory-adjustment sync from SOAP (~2 round-trips/serial, ~1h for a large adjustment) to REST. (jcardinal)
|
|
@@ -6,7 +6,7 @@ project: Library
|
|
|
6
6
|
client: nychh
|
|
7
7
|
type: client-feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-08-
|
|
9
|
+
updated: 2026-08-28
|
|
10
10
|
owners: [jcardinal]
|
|
11
11
|
files:
|
|
12
12
|
- library/app/api/toga2.php
|
|
@@ -14,6 +14,10 @@ files:
|
|
|
14
14
|
- library/app/netsuite.php
|
|
15
15
|
- worker/crons/toga2/netsuite/sync_togasupply_hh.php
|
|
16
16
|
- worker/crons/toga2/netsuite/common_sync_togasupply.php
|
|
17
|
+
- worker/crons/toga2/netsuite/diagnose_hh_stuck_transfer_order.php
|
|
18
|
+
- dbchanges2/Core/2026-08-27a - TransferOrderUuidSearchableIdentifier.sql
|
|
19
|
+
- dbchanges2/Client/2026-08-28a - TransferOrderItemsTimestampColumns.sql
|
|
20
|
+
- dbchanges2/Client/2026-08-28b - TransferOrderItemsTimestampColumnsReposition.sql
|
|
17
21
|
related:
|
|
18
22
|
- ../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md
|
|
19
23
|
- ../../../1.0/apps/library/features/toga2-api-client-and-bridge.md
|
|
@@ -114,6 +118,48 @@ status was previously throwing (and being swallowed), which is why order 128829
|
|
|
114
118
|
`App_NetSuite::getCustomFieldValue()` (`library/app/netsuite.php`) was hardened with `?? []` / `?? null`
|
|
115
119
|
guards: 1.0 targets PHP 7.2 and any PHP warning on that path terminates the cron.
|
|
116
120
|
|
|
121
|
+
## SALES_ORDERS freeze chain — three distinct causes, peeled in order (2026-08-28)
|
|
122
|
+
|
|
123
|
+
Once transfer-order routing was live, the NYCHH **and GroWrk** `SALES_ORDERS` section froze at
|
|
124
|
+
`1-RUNNING` on `syncTransferOrderFromNetsuite`. `SALES_ORDERS` has **no per-record `try/catch`** (see
|
|
125
|
+
[per-client sync](../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md)), so any
|
|
126
|
+
throw aborts the whole section and re-throws every run. Three separate faults sat one behind the next:
|
|
127
|
+
|
|
128
|
+
1. **`Undefined property: stdClass::$transferOrderStage`** at the change-check in `toga2.php`
|
|
129
|
+
(`if ($transferOrder) {…}`). **api2 OMITS a null foreign-key object from a depth-2 GET entirely** —
|
|
130
|
+
it is *absent*, not null — so `$transferOrder->transferOrderStage` (and `originLocation` /
|
|
131
|
+
`destinationLocation`) raised a PHP warning that 1.0 `App_Error` escalates to fatal. **Fix:** guard
|
|
132
|
+
all three FK reads with `?? null`. (Lesson: on a depth-limited api2 GET, a nullable FK object may be
|
|
133
|
+
missing rather than null — always `?? null` before dereferencing.)
|
|
134
|
+
2. **`EV-12` on the transfer-order-items POST** — the nested `transferOrder:{uuid}` link could not
|
|
135
|
+
resolve its parent because `RecordFields.uuid.isIdentifier` was **0** on records 312/313. Fixed
|
|
136
|
+
platform-wide (`dbchanges2/Core/2026-08-27a`). Full rule on the
|
|
137
|
+
[ACL permission chain](../../../2.0/apps/_underscore/features/acl-permission-chain.md).
|
|
138
|
+
3. **MySQL 1054 `Unknown column 'dtCreated'` → api2 `EV-10`** one step further on.
|
|
139
|
+
`_Model_Client_TransferOrderItem` declares `dtCreated`/`dtUpdated`
|
|
140
|
+
(`FIELD_DATETIME_CREATED`/`_UPDATED`) and emits them on every INSERT, but the `TransferOrderItems`
|
|
141
|
+
**table lacked both columns in EVERY client DB** (a platform-wide drift — the header
|
|
142
|
+
`TransferOrders` table had them). **Fix (all-client, `dbchanges2/Client/` fan-out):** `2026-08-28a`
|
|
143
|
+
adds the two columns mirroring the header table (`DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP`,
|
|
144
|
+
`dtUpdated` also `ON UPDATE CURRENT_TIMESTAMP`); `2026-08-28b` repositions them (`AFTER uuid` /
|
|
145
|
+
`AFTER dtCreated`) per the new column-ordering standard —
|
|
146
|
+
[2.0 framework rules](../../../2.0/standards/framework-rules.md). Same failure class as GroWrk's
|
|
147
|
+
`_Model_Growrk_Unit` 1054: a declared model field with no physical column 1054s on every write.
|
|
148
|
+
|
|
149
|
+
### Debugging lesson — `Logs.Issue` trace is stale; use a single-record harness for the real line
|
|
150
|
+
|
|
151
|
+
`Logs.Issue` aggregates by message + runner and captures its `trace` **once at first-seen, never
|
|
152
|
+
refreshed**. Issue 270's stale trace pointed at a bogus line and a spurious `$location` message and
|
|
153
|
+
sent debugging the wrong way for a long time. The reliable way to get the true crash line was
|
|
154
|
+
`worker/crons/toga2/netsuite/diagnose_hh_stuck_transfer_order.php` — a **manual operator tool** (not in
|
|
155
|
+
`cron.worker.sync.json`; does **not** call `cronInitialization`, so no process lock / no cursor
|
|
156
|
+
writes). It fetches a NetSuite sales/transfer order via `App_Api_Netsuite_Rest::listSalesOrders`
|
|
157
|
+
(NYCHH entity filter `listChildCustomers(28908)`, window in server tz `America/Chicago`), dumps its
|
|
158
|
+
shape, then re-runs the **real** `syncTransferOrderFromNetsuite` under a strict `set_error_handler`
|
|
159
|
+
that rethrows warnings as a catchable `ErrorException` with exact `file:line`. That is what found the
|
|
160
|
+
`transferOrderStage` crash after the stale `Logs.Issue` trace misdirected. **When a 1.0 cron freeze's
|
|
161
|
+
`Logs.Issue` trace looks wrong, trust a single-record harness over the aggregated trace.**
|
|
162
|
+
|
|
117
163
|
## Gotchas / known issues
|
|
118
164
|
|
|
119
165
|
- **Order 128829 (no location anywhere) is unresolved** — see the origin-resolution section. Do not
|
|
@@ -128,6 +174,17 @@ guards: 1.0 targets PHP 7.2 and any PHP warning on that path terminates the cron
|
|
|
128
174
|
|
|
129
175
|
## Change history
|
|
130
176
|
|
|
177
|
+
- 2026-08-28 — Unfroze the `SALES_ORDERS` section (NYCHH + GroWrk) by peeling **three** stacked faults
|
|
178
|
+
on `syncTransferOrderFromNetsuite`: (1) `Undefined property $transferOrderStage` — api2 **omits a
|
|
179
|
+
null FK object** from a depth-2 GET, guarded `transferOrderStage`/`originLocation`/`destinationLocation`
|
|
180
|
+
with `?? null`; (2) `EV-12` because `RecordFields.uuid.isIdentifier` was 0 on records 312/313 (fixed
|
|
181
|
+
platform-wide, `dbchanges2/Core/2026-08-27a`); (3) MySQL 1054 → `EV-10` because `TransferOrderItems`
|
|
182
|
+
lacked the model-declared `dtCreated`/`dtUpdated` columns in every client DB (added all-client via
|
|
183
|
+
`dbchanges2/Client/2026-08-28a`, repositioned `AFTER uuid`/`AFTER dtCreated` per the new
|
|
184
|
+
column-ordering standard by `2026-08-28b`). Recorded the debugging lesson: `Logs.Issue`'s `trace` is
|
|
185
|
+
captured once and goes stale (Issue 270 misdirected), so a single-record harness
|
|
186
|
+
(`diagnose_hh_stuck_transfer_order.php`, strict `set_error_handler` → `ErrorException`) is the
|
|
187
|
+
reliable way to get the true crash line. (jcardinal)
|
|
131
188
|
- 2026-08-27 — Built NYCHH transfer-order ingestion end-to-end. **Routing by `custbody_stocking_order`
|
|
132
189
|
(id 7097), not dollar amount** (`isTransferOrder()` rewritten + new `isStockingOrder()` normalizer);
|
|
133
190
|
the old `total===0 + hold-flag` heuristic was wrong for NYCHH (no hold flag set → everything landed
|
|
@@ -33,6 +33,7 @@ related:
|
|
|
33
33
|
- ./features/po-number-upstream-direction.md
|
|
34
34
|
- ./features/netsuite-transfer-order-import.md
|
|
35
35
|
- ./features/transfer-order-inventory-quantities.md
|
|
36
|
+
- ./features/netsuite-inventory-adjustment-fulfillment-link.md
|
|
36
37
|
- ../../2.0/apps/_underscore/features/sales-order-purchase-order-bridge-direction.md
|
|
37
38
|
- ../../2.0/apps/toga25-supply/features/transfer-orders-page.md
|
|
38
39
|
---
|
|
@@ -89,6 +90,11 @@ table views. Client-specific DB change-sets live in `dbchanges2/Client_Nychh/`.
|
|
|
89
90
|
fulfilled/backordered model on `_Model_Nychh_Item`, plus the client-override models + ACL rows that
|
|
90
91
|
let a NetSuite TransferOrder POST succeed. See
|
|
91
92
|
[NYCHH transfer-order inventory & quantities](./features/transfer-order-inventory-quantities.md).
|
|
93
|
+
- **Inventory-adjustment → item-fulfillment linking** — imports NetSuite `custbody_stock_adjustment`
|
|
94
|
+
(adjustment→fulfillment) into two all-client bridges for adjustment→fulfillment→SO→PO traceability;
|
|
95
|
+
gated NYCHH-only, best-effort, with on-demand fulfillment import. Go-live toggle
|
|
96
|
+
(`IS_ENABLED_INTEGRATION_INVENTORY_ADJUSTMENTS`) still off. See
|
|
97
|
+
[NYCHH inventory-adjustment → item-fulfillment linking](./features/netsuite-inventory-adjustment-fulfillment-link.md).
|
|
92
98
|
|
|
93
99
|
- **PO links are UPSTREAM (`PurchaseOrders_SalesOrders`), not downstream.** The downstream
|
|
94
100
|
`sales-order-purchase-orders` route returns an empty array for NYCHH by design, and the shared
|
package/package.json
CHANGED