toga-ai 1.0.938 → 1.0.940

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -6,8 +6,8 @@ project: Library
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-09-30
10
- owners: ["rgirish"]
9
+ updated: 2026-10-05
10
+ owners: ["rgirish", "jcardinal"]
11
11
  files:
12
12
  - library/app/api/toga2.php
13
13
  - library/app/api/netsuite/rest.php
@@ -35,7 +35,7 @@ When the NetSuite → TOGa Supply sync drops (skips + deletes) an order (all cli
35
35
  - Every other case imports: some lines open (e.g. 9 open + 1 closed), all closed but shipped/invoiced/has a PO, all open, zero item lines.
36
36
  - **Why the related-records part:** NetSuite had 132 SOs since 2025 with every line closed but real shipments (Staples 43, GroWrk 21, ODP Veyer 14, Smoke Stacks 11, NYCDOE 5, Endeavor 5, Elite 2 — e.g. Elite SO 269622 → shipment 304666). Deleting them drops real stock movements.
37
37
  - Replaces 2026-09-29 "Closed status + related records = keep" and 2026-09-23 "delete every Closed".
38
- - Status: reviewed clean, **not yet deployed**.
38
+ - Status: deployed; per-client SO scope fix deployed 2026-10-05.
39
39
 
40
40
  ## How it works
41
41
  - **Item line** = `mainline = 'F'`, `item > 0`, not ShipItem/TaxItem/TaxGroup/Discount/EndGroup/BeginGroup/Subtotal/Description/Markup/Payment. One home: `App_Api_Netsuite_Rest::SALES_ORDER_ITEM_LINE_FILTER`.
@@ -44,6 +44,7 @@ When the NetSuite → TOGa Supply sync drops (skips + deletes) an order (all cli
44
44
  - `deleteClosedOrderFromNetsuite()` **KEEP guard**: re-checks live and keeps (logs `KEEP`, returns false) if `hasOpenLines` OR `hasRelatedTransactions`. Live re-check is needed because the bulk tool `delete_closed_netsuite_orders.php` (dry run unless `--execute`) passes an order with no lines.
45
45
  - The delete removes the SalesOrder/TransferOrder with its lines, invoices, fulfillments (units, tracking bridges) and bridge rows. Records on the other side of a bridge (PO, other SO, TrackingNumber, Unit) are never deleted — only the link row. Fails loud; a half-done delete is finished by the next run.
46
46
  - **Line field `isOpen`** on `SalesOrderItems` (after `preferredVendorId`) and `PurchaseOrderItems` (after `vendorItemId`): `TINYINT UNSIGNED NOT NULL DEFAULT 1`; model `[self::FIELD_BOOLEAN, self::FIELDOPT_DEFAULT => true]`. Core.RecordFields 2634 (record 15) / 2635 (record 18), BOOLEAN. Client ACL: API role read+write, Base role read, roles by name. The sync writes 0/1 from the NS line (`property_exists` + `(int)` change check). Old rows stay 1 until their order re-syncs.
47
+ - **Per-client SO scope:** private `closedOrderSalesOrderScope()` returns `[joins, extraEquals]` for `readAllForClosedOrderDelete()` — Compass `SalesOrders.locationId=615286` + `customerId=3`; Prudential `Customers` join + `Customers.uuid = CLIENT_UUID_PRUDENTIAL_CUSTOMER`; other clients unscoped. Same scope as SO sync (~L1012), invoice sync (~L3879), ~L6941.
47
48
  - Backfill shipped orders skipped under old rules: re-run their fulfillments with `process_netsuite_records.php` ([re-processing workflow](../../worker/workflows/reprocessing-one-netsuite-order.md)).
48
49
 
49
50
  ## Gotchas
@@ -51,8 +52,14 @@ When the NetSuite → TOGa Supply sync drops (skips + deletes) an order (all cli
51
52
  - **`Client/2026-09-30a` ALTER is not guarded** — a second run fails on duplicate column. The INSERTs are guarded. Deploy order + per-DB check: [verifying a migration ran](../../../2.0/apps/dbchanges2/workflows/verifying-a-migration-ran.md).
52
53
  - **The delete reads every route in `CLOSED_ORDER_DELETE_ROUTES` (toga2.php) — the client's API role (roleId 3) needs read on ALL of them:** record permission + an expression on each logic group + field permission (at least `uuid`). One gap throws in the SO loop and **SALES_ORDERS freezes**. Find gaps with `App_Api_Toga2::checkClosedOrderDeleteAccess()` (run by `delete_closed_netsuite_orders.php`; lists every gap in one run). ACL layers: [acl-permission-chain](../../../2.0/apps/_underscore/features/acl-permission-chain.md).
53
54
  - Compass, 2026-09-29: records 312/313 had logic groups with no expression; 257 and 254 had no `AclRecordPermissions`, then no `AclFieldPermissions` (`EZ-2` on `uuid`). Fix: `dbchanges2/Client_Compass/2026-09-29a - TransferOrderAclExpressions.sql`, then reset `NETSUITE_EXECUTION_MODE_SALES_ORDERS` to `864000-IDLE`.
55
+ - **⚠ Compass dual-SO trap — SO lookup by NetSuite id MUST use the per-client scope.** Compass has 2 `SalesOrders` per `c_netsuiteInternalSalesOrderId`: Office Depot order (`customerId=1`, stamped by cron 5) + NetSuite-synced order (`customerId=3`, `locationId=615286`). 2026-10-02 17:00 → 2026-10-05 SALES_ORDERS froze at `1-RUNNING` (cursor `2026-10-02 17:56:01`, `Logs.Issue` 940, throw at toga2.php:688 `send()`): closing NS SO 7566413 (TOGa 292235) also tried to delete Office Depot SO 481506511001; its fulfillment line `ItemFulfillmentItems` 335334 DELETE hit MySQL 1451 `fk_ItemFulfillmentItems_upstreamItemFulfillmentItemId` (downstream 335333 points up to it) → api2 `EV-11` → whole section aborted each run. Fixed by the scope above. 4th hit of this trap (SO sync, IF sync, invoice sync, closed-order delete) — apply the scope on every new lookup.
56
+ - Damage: 1 `ItemFulfillmentItemUnits` row on Office Depot fulfillment F116294 deleted 2026-10-02 16:59:53; not recoverable (no audit row for api deletes). Only other closed-order delete (2026-09-30, SO 285524 / NS 7354035) was correct.
57
+ - **OPEN (dev decision):** after dropping 292235, mirror fulfillments of the NS-deleted shipment remain on higher tiers — F116294 (id 91650, Office Depot SO) and 91651 (Compass customer order); Compass order may still show shipped. Not touched — never touch the Compass customer order.
58
+ - **OPEN low-risk:** the TransferOrder read in `deleteClosedOrderFromNetsuite()` and `retireStaleSalesOrderForTransferOrder` (~L5698) are still unscoped by NS id; for Compass they throw (fail loud), not delete.
59
+ - **Diagnosis tips:** `Logs_Compass.Api` has no audit for deletes; toga-db MCP rejects any SQL containing the word `DELETE` — filter `method LIKE 'DEL%'`; unbounded `LIKE` on `Api.responsePayload` over days times out — use tight `dtStamp`/`id` ranges.
54
60
  - Adding a route to `CLOSED_ORDER_DELETE_ROUTES` = a new ACL need on every client. Run the access check fleet-wide before deploy.
55
61
 
56
62
  ## Change history
57
63
  - 2026-09-30 — Created: Closed SO with related records kept; only Closed with none deleted. KEEP guard + Compass ACL gotcha. (rgirish)
58
64
  - 2026-09-30 — Rule v3: header status dropped; drop only when every item line `isclosed = 'T'` AND no related records. Added line `isOpen` field. (rgirish)
65
+ - 2026-10-05 — Per-client SO scope on the delete lookup (`closedOrderSalesOrderScope()`) after the Compass dual-SO freeze (Issue 940). Open items: higher-tier mirror fulfillments, unscoped TO lookups. (jcardinal)
@@ -6,7 +6,7 @@ project: Worker
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-10-01
9
+ updated: 2026-10-05
10
10
  owners: ["dfranks", "bala", "jcardinal", "mhammontree", "snaredla", "rgirish"]
11
11
  files:
12
12
  - worker/crons/toga2/netsuite/common_sync_togasupply.php
@@ -1966,6 +1966,7 @@ library (or vice versa) crashes GroWrk and Adyen on their next sync run.
1966
1966
  `customerId=3`; Prudential adds the `Customers` join + `CLIENT_UUID_PRUDENTIAL_CUSTOMER`; no other client
1967
1967
  changed, no else/default, and the guard still throws for a genuinely-missing SO. PHP 7.2-safe
1968
1968
  (`$options['join'] ?? []`), signature unchanged.
1969
+ - Same trap hit the closed-order delete 2026-10-05 (SALES_ORDERS freeze, Issue 940) — 4th path; see [closed-order rule](../../library/features/netsuite-closed-order-rule.md). Every SO-by-NS-id lookup needs this scope.
1969
1970
  - **⚠ OPEN follow-up (own ticket): six OTHER clients also keep two SalesOrders per NetSuite id, and the
1970
1971
  fix does NOT scope them.** A read-only sweep found the same dual-SO pattern in **Growrk (2789), Canon
1971
1972
  (2890), Endeavorhealth (4828), Spglobal (2893), Browardsheriff (160), Masonite (17)** (Nychh/Elite/Quad/Sbasite
@@ -2078,6 +2079,7 @@ library (or vice versa) crashes GroWrk and Adyen on their next sync run.
2078
2079
  [NetSuite Sync Alert Monitor](../../library/features/netsuite-sync-alert-monitor.md).
2079
2080
 
2080
2081
  ## Change history
2082
+ - 2026-10-05 — Closed-order delete now per-client SO scoped (Compass dual-SO freeze, Issue 940); details in [closed-order rule](../../library/features/netsuite-closed-order-rule.md). Library only. (jcardinal)
2081
2083
  - 2026-10-01 — `Items.description` now from the NetSuite item record, refreshed every touch ([doc](../../library/features/netsuite-item-description-sync.md); library + worker deploy together). Gotchas: invoice billing an SO older than the SO cursor (Adyen); receipt lines never rebuilt on an existing receipt (Elite). (rgirish)
2082
2084
  - 2026-10-01 — NYCHH diagnosis traps recorded: the wrapper is gated to **once a day** by
2083
2085
  `cronInitialization(86400)` despite its `*/5` schedule entry, and the live cursors/modes live in
@@ -10,7 +10,7 @@
10
10
  | [_ApiRequest — JSON encode/decode & api-logging behavior](features/apirequest-json-content-type.md) | `_ApiRequest`, the 2.0 outbound HTTP client — JSON encode/decode, retry, and api-logging traps; open before calling it or reading `Logs.Api`. |
11
11
  | [Assortment Name Translation (AssortmentTranslations sidecar)](features/assortment-name-translation.md) | Serves Assortment names in multiple languages via an `AssortmentTranslations` sidecar table on the metadata-driven translation layer; open when translating asso |
12
12
  | [Asynchronous Query Execution (writes-only, via Worker)](features/async-query-execution.md) | Run a slow WRITE asynchronously via the Worker tier so a long request write cannot hold a stale connection and hit "MySQL server has gone away"; open before usi |
13
- | [FIELD_AUTOINCREMENT record numbering (SA / TA prefixes, NULL-only trigger, no padding)](features/autoincrement-record-numbering.md) | ORM-generated human record numbers (`SA100001`, `TA100000`) via `FIELD_AUTOINCREMENT`; open when adding or debugging record-number prefixes. |
13
+ | [FIELD_AUTOINCREMENT record numbering (Parameters counter, race-safe, NULL-only trigger)](features/autoincrement-record-numbering.md) | ORM-generated record numbers (`SA100001`, `SR131021`) via `FIELD_AUTOINCREMENT` and the per-tenant `Parameters` counter that makes them race-safe; open when add |
14
14
  | [Calculated-column filter options (`<field>FilterOptions` model static → table meta)](features/calculated-column-filter-options.md) | Lets a calculated (`FIELD_SQL`) table column supply its own filter dropdown options via a `<field>FilterOptions` model static; open when adding a data-driven co |
15
15
  | [FIELD_SQL calculated fields — the underscore-prefix + same-name-method contract](features/calculated-sql-fields.md) | A `FIELD_SQL` (calculated) field on a `_Model` is bound by a **two-part contract that `_Model` enforces by throwing at model-construction time**, not by convent |
16
16
  | [Carrier Shipping Labels (UPS/FedEx) & NetSuite Item Fulfillment](features/carrier-shipping-labels.md) | Backend mechanics behind Fulfill & Ship: buying/persisting a carrier label (UPS/FedEx) and creating the NetSuite Item Fulfillment; open for label, reprint, retu |
@@ -1,29 +1,32 @@
1
1
  ---
2
- title: "FIELD_AUTOINCREMENT record numbering (SA / TA prefixes, NULL-only trigger, no padding)"
2
+ title: "FIELD_AUTOINCREMENT record numbering (Parameters counter, race-safe, NULL-only trigger)"
3
3
  framework: "2.0"
4
4
  repo: _underscore
5
5
  project: _Underscore
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-09-22
9
+ updated: 2026-10-02
10
10
  owners: ["bala", "rgirish"]
11
11
  files:
12
12
  - _underscore/Model.php
13
+ - _underscore/Database.php
13
14
  - _underscore/Model/Client/SalesOrder.php
14
15
  - _underscore/Model/Client/TransferOrder.php
16
+ - _underscore/Model/Prudential/ServiceRequest.php
15
17
  related:
16
18
  - ../architecture.md
17
19
  - ../../api2/features/v2-deadlock-retry.md
20
+ - ../../api2/features/environment-variable-drives-underscore-branch.md
18
21
  - ../../worker2/features/netsuite-transferorder-outbound-push.md
19
22
  - ../../worker2/features/netsuite-salesorder-outbound-push.md
20
23
  ---
21
24
 
22
- ORM-generated human record numbers (`SA100001`, `TA100000`) via `FIELD_AUTOINCREMENT`; open when adding or debugging record-number prefixes.
25
+ ORM-generated record numbers (`SA100001`, `SR131021`) via `FIELD_AUTOINCREMENT` and the per-tenant `Parameters` counter that makes them race-safe; open when adding a prefix, debugging a duplicate-number 1062, or load-testing creates.
23
26
 
24
27
  ## Summary
25
28
 
26
- A human-facing record number (`SA100001`, `TA100000`) is generated by the ORM, not by MySQL. A model declares it as `FIELD_AUTOINCREMENT` with two options:
29
+ A record number is generated by the ORM, not MySQL. A model declares:
27
30
 
28
31
  ```php
29
32
  public $number = [
@@ -33,36 +36,49 @@ public $number = [
33
36
  ];
34
37
  ```
35
38
 
36
- `_Model_Client_SalesOrder` uses `SA` / 100000; `_Model_Client_TransferOrder` was given `TA` / 100000 on 2026-09-02, on the shared client model, so every client's transfer orders get it.
39
+ 12 models use it (SalesOrder `SA`, PurchaseOrder `P`, ServiceRequest `SR`, TransferOrder `TA`, Ticket, ItemReceipt, InventoryAdjustment, ...). For `DB_CLIENT` models the next number comes from a **counter row in the tenant's `Parameters` table**, taken on its own connection, so concurrent creates never get the same number.
40
+
41
+ **Branch state (verified 2026-10-02):** counter code (TRUE-82209, `1760b58a` + caps-key follow-up `bf22215b`) is on `_underscore` `_beta`, `_sandbox-dev`, `TRUE-82209`. **Not** on `_production` or `_sandbox-client` — prod still runs the old `MAX()+1` path and the Prudential `usleep` stopgap.
37
42
 
38
43
  ## How it works
39
44
 
40
- Verified in `_underscore/Model.php` ~L546:
45
+ 1. **Fires only when the field is NULL.** A supplied number always wins (NetSuite importer's `tranId`, Compass PO `mitsSalesOrder`), and bypasses the counter. A front-end that posts `number` silently disables the generator.
46
+ 2. **Counter key:** `NEXT_NUMBER_<TABLE_IN_SNAKE_CASE>_<PREFIX><SUFFIX>`, all caps — e.g. `NEXT_NUMBER_SERVICE_REQUESTS_SR`, `NEXT_NUMBER_SALES_ORDERS_SA`, `NEXT_NUMBER_PURCHASE_ORDERS_P`, `NEXT_NUMBER_SALES_ORDERS_SAC` (Compass Canada). One row per table + prefix + suffix. `value` = last number issued.
47
+ 3. **Allocate** (`_Model::allocateAutoincrementNumber()`): `UPDATE Parameters SET value = LAST_INSERT_ID(CAST(value AS UNSIGNED)+1) WHERE key=...`, read back with `getInsertId()`, then **commit at once**. Runs on `_Database::DB_CLIENT_SEQUENCE` (`'ClientSequence'`), a separate connection — the api2 request transaction commits only at request end, so taking the row lock there would serialize every create.
48
+ 4. **`DB_CLIENT_SEQUENCE` is automatic.** `_Database::register()` points it at the same DB whenever the alias is `_underscore::DB_CLIENT` (covers api2, worker2, saml, cXML, Nychh worker), and drops its connection when DB_CLIENT is re-pointed to another tenant. No app change needed.
49
+ 5. **Self-seeding.** Row missing → run the old `MAX()` SQL on the sequence connection (writer, not read host), `INSERT ... ON DUPLICATE KEY UPDATE value = GREATEST(...)`, commit between each step. No per-tenant SQL or seed script.
50
+ 6. **Collision retry in `save()`.** If the INSERT fails 1062 **and** the message contains the number this save generated (a hand-set number took it), resync the counter past `MAX()` and retry, up to `MAX_AUTOINCREMENT_ATTEMPTS = 5`. A 1062 on any other unique key (uuid) is re-thrown. Any failed counter step rolls back the sequence connection.
51
+ 7. **Unchanged:** Core-DB models and `search()` keep the old `MAX()+1` path. **No zero padding** (prefix + integer). **Gaps are possible** (rolled-back request still used its number) — same as before.
41
52
 
42
- 1. **The generator only fires when the field arrives NULL** (`if (is_null($value))`). A supplied number always wins. That is what makes an importer that sends its own number (the NetSuite importer sends `number = $nsOrder->tranId`) completely unaffected by adding the option — and it is also why a front-end that still posts `"number": "286906"` means the generator never runs.
43
- 2. **The next value is `MAX(numeric part) + 1` scoped by `WHERE number LIKE '<prefix>%'`.** So pre-existing numbers in another format are ignored: NYCHH's 1,930 legacy transfer-order numbers (e.g. `286906`) do not participate, and the sequence starts clean at `TA100000`.
44
- 3. **There is NO zero padding.** The output is prefix + integer, so `TA00000` is not achievable without changing `Model.php`.
53
+ **Rejected alternatives (2026-10-02):**
54
+ - Whole-request replay via `DEADLOCK_RETRIABLE_ROUTES` for every numbered route — unsafe per tenant: Adyen / Elite / Growrk `postPost` hooks have external side effects. See [V2 request retry](../../api2/features/v2-deadlock-retry.md).
55
+ - Derive number from the AUTO_INCREMENT `id` — `number` is NOT NULL UNIQUE and hand-set numbers break it.
56
+ - `GET_LOCK` — the lock is held to request end, so creates serialize.
45
57
 
46
- **⚠ The generator is NOT concurrency-safe — read and insert can split.** `SELECT MAX(...)+1` and the `INSERT` are two separate statements with **no lock between them**. Two concurrent POSTs read the same MAX, pick the same number, and the loser gets **MySQL 1062** — a 400 to the caller, not a retry. Seen live on Prudential `/v2/service-requests` (`Duplicate entry 'SR131023' for key 'ServiceRequests.number'`).
58
+ ## Gotchas
47
59
 
48
- **Today's mitigation is a retry, not a fix.** api2 replays the whole request for two routes only (`/v2/advance-shipping-notices`, `/v2/service-requests`) — see [V2 request retry](../../api2/features/v2-deadlock-retry.md). **Every other `FIELD_AUTOINCREMENT` model has the same exposure and no mitigation**: SalesOrder, PurchaseOrder, Ticket, ItemReceipt, InventoryAdjustment, TransferOrder. A duplicate-number clash on those routes still returns 400. The durable answer is making the read and insert one operation in `Model.php`; that is a bigger, separate change and is **not done**.
60
+ - **⚠ Old path race (still live on prod until this ships):** `SELECT MAX+1` then a separate INSERT → concurrent creates pick one number → loser gets 1062 `Duplicate entry 'SR131021' for key 'ServiceRequests.number'` → HTTP 400 `EV-10`, record lost. The number in the error belongs to the **winner**.
61
+ - **⚠ A prefix on a SHARED model is not unique across clients** — every tenant starts at `TA100000`. Never key a cross-client id on `number`; NetSuite account 1095849 is shared, so pushes key `externalId` on uuid (`toga-so-<uuid>` / `toga-to-<uuid>`).
62
+ - **SA / TA numbers are placeholders** — overwritten by NetSuite `tranId` on write-back (~12s). Never build a key or match on `number`. Keeping a Toga number permanently needs **two** changes: a guard in `writeBackNetSuiteId()` **and** dropping `number` from the PUT in `library/app/api/toga2.php` (~L4295) and `_underscore/Trait/Netsuite/TransferOrder.php` (tried 2026-09-02, reverted).
63
+ - **`FIELD_AUTOINCREMENT` is special-cased in `buildSqlFieldValue()`**, so a null value does not collapse a `load()` WHERE the way a null `FIELD_CHAR` does — see [_underscore architecture](../architecture.md).
64
+ - **Adding the option renumbers nothing**; existing rows are untouched.
65
+ - **Error class depends on boot.** Real boot calls `mysqli_report(MYSQLI_REPORT_STRICT)` (`Error.php:217`), so a failed query returns false and `_Query` throws `_Exception_Database` (which `save()` catches). A test harness that skips this boot gets raw `mysqli_sql_exception` instead — boot the framework fully when testing.
66
+ - **Reproducing the race needs distinct rows per call.** Each concurrent test request must use a **different contact** (employee_xid / email) **and unit serialNumber**. Reusing one makes requests queue on that row lock and the race never shows (old code "passed" 10/10). Release all calls at once with the gate-file pattern ([V2 request retry](../../api2/features/v2-deadlock-retry.md)). Prudential `POST /v2/service-requests` also needs `customer.uuid`. Network latency is irrelevant.
67
+ - **Beta can't take big bursts:** sandbox-dev DB `max_connections = 90`; ~15+ simultaneous requests → `Too many connections` → 500 `EO-1`. Keep bursts at ~10. See [ENVIRONMENT decides the branch](../../api2/features/environment-variable-drives-underscore-branch.md).
68
+ - **Run `php -l Model.php Database.php` after resolving merge conflicts on env branches.** The TRUE-82209 merge into `_sandbox-dev`/`_beta` dropped `public static $_aliases` / `private static $_parked` in `Database.php` and duplicated `private $_model_generatedAutoincrementFields` in `Model.php` → beta outage (`Access to undeclared static property _Database::$_aliases`). The error page also dumped `$_SERVER`.
49
69
 
50
- ## Gotchas
70
+ ## Verification
51
71
 
52
- - **⚠ A prefix on a SHARED model is not unique across clients.** Because the option lives on `_Model_Client_TransferOrder`, every tenant starts at `TA100000` — the scan is per client database. Any cross-client identifier must therefore NOT be keyed on `number`: NetSuite account **1095849 is shared by every client** and enforces `externalId` uniqueness account-wide, which is why both pushes key their external id on the record **uuid** (`toga-so-<uuid>` / `toga-to-<uuid>`) — see [the transfer-order push](../../worker2/features/netsuite-transferorder-outbound-push.md).
53
- - **A generated number can be a PLACEHOLDER — decide its lifetime explicitly.** `SA`/`TA` numbers are overwritten by NetSuite's `tranId` on write-back, so the generated value lives about 12 seconds. That is the intended behaviour for both order types, but it makes `number` a moving value — never build a key, a match, or an external id on it.
54
- - **Keeping a Toga-side number permanently takes TWO changes, not one** (written 2026-09-02, then reverted at the developer's request — recorded so nobody rediscovers it): (1) a guard in `writeBackNetSuiteId()` so the push does not overwrite `number`, and (2) dropping `number` from the PUT in both `library/app/api/toga2.php` (~L4295) and `_underscore/Trait/Netsuite/TransferOrder.php`. Miss (2) and the 1.0 importer compares `number` and PUTs NetSuite's over the top on its next run, so the guard alone achieves nothing. Both files are back to untouched (verified with an empty `git diff`).
55
- - **`FIELD_AUTOINCREMENT` is one of the field types special-cased by `buildSqlFieldValue()`**, so a null value on it does NOT collapse a `load()` WHERE clause the way a null `FIELD_CHAR` does — see the collapsed-WHERE OOM note in [_underscore architecture](../architecture.md).
56
- - **Adding the option to a model does not renumber anything** and does not touch existing rows; it only affects inserts that arrive with `number` NULL.
57
- - **Concurrent inserts can pick the same number** (unlocked `MAX+1` then `INSERT`) → MySQL 1062 → 400. Only two api2 routes retry it; every other model is exposed. See [V2 request retry](../../api2/features/v2-deadlock-retry.md).
58
- - **A front-end that sends `number` silently disables the feature.** There is no error — the posted value is simply accepted. Check the actual POST body before concluding the generator is broken.
72
+ Beta (`api.beta.togahub.com` → sandbox-dev), 2026-10-02: before fix 8/10 concurrent creates failed 1062; after fix 100/100 (10 rounds of 10), 116 distinct numbers, counter matched `MAX()`.
59
73
 
60
74
  ## Related
61
75
 
62
76
  - [_underscore architecture](../architecture.md)
77
+ - [V2 request retry](../../api2/features/v2-deadlock-retry.md)
63
78
  - [NetSuite TransferOrder outbound push](../../worker2/features/netsuite-transferorder-outbound-push.md)
64
79
  - [NetSuite SalesOrder outbound push](../../worker2/features/netsuite-salesorder-outbound-push.md)
65
80
 
66
81
  ## Change history
67
- - 2026-09-22 — Recorded that the `MAX+1` generator is not concurrency-safe (unlocked read-then-insert → 1062), which routes retry it today, and that every other FIELD_AUTOINCREMENT model is still exposed. (rgirish)
82
+ - 2026-10-02 — Replaced unlocked `MAX()+1` with a per-tenant `Parameters` counter on its own committed connection (`DB_CLIENT_SEQUENCE`), self-seeding, 1062 resync retry; removed Prudential `usleep` stopgap. On `_beta`/`_sandbox-dev` only. (rgirish)
83
+ - 2026-09-22 — Recorded that the `MAX+1` generator is not concurrency-safe. (rgirish)
68
84
  - 2026-09-02 — `TA` / 100000 added to the shared `_Model_Client_TransferOrder`. (bala)
@@ -6,14 +6,15 @@ project: API
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-09-25
10
- owners: ["bala", "mhammontree", "apeterson", "jcardinal", "tcox"]
9
+ updated: 2026-10-02
10
+ owners: ["bala", "mhammontree", "apeterson", "jcardinal", "tcox", "rgirish"]
11
11
  files:
12
12
  - api2/.ebextensions/git.php
13
13
  - api2/.platform/hooks/prebuild/git.sh
14
14
  - api2/.ebextensions/php_include_underscore.config
15
15
  - api2/.ebextensions/git.sandbox-dev.json
16
16
  - api2/.ebextensions/git.sandbox-client.json
17
+ - api2/.ebextensions/git.beta.json
17
18
  - api2/Config/beta.ini
18
19
  - api2/Config/sandbox-dev.ini
19
20
  - api2/Component/Api/V2/V2.php
@@ -64,17 +65,18 @@ Checking only #2 (the one you can `grep`) is the trap: it can be perfectly corre
64
65
 
65
66
  Never infer the environment from: **the EB environment name** (`api-sandbox-dev` is `beta`); **the hostname / URL** (follows the EB name, not `ENVIRONMENT`); **which database the data landed in** (`Config/beta.ini` and `Config/sandbox-dev.ini` **both point at the same cluster, `dev.sandbox.database.togahub.com`**, so querying "the dev-sandbox database" and finding your row proves nothing about which tier wrote it).
66
67
 
67
- **`api.beta.togahub.com` — the concrete mapping, and the leftover files that mislead.** Independently re-confirmed 2026-08-06 (cost **two deploy cycles**):
68
- - `api.beta.togahub.com` runs **`ENVIRONMENT=beta`** → reads `Config/beta.ini` → clones `_underscore` branch **`_beta`**.
69
- - There is **deliberately NO `.ebextensions/git.beta.json`**, which is exactly why the branch falls through to the derived `'_' . strtolower($env)` = `_beta`.
70
- - **`Config/sandbox-dev.ini` AND `.ebextensions/git.sandbox-dev.json` both still exist** and the latter **pins `branch: _sandbox-dev`** — apparently left behind by a rename. They point at the same host and they are **not what serves it.** The single most misleading pair of files in the repo: they look authoritative and they are dead.
71
- - **Consequence: merging `_underscore` to `_sandbox-dev` does NOT reach that environment.**
68
+ **`api.beta.togahub.com` — the concrete mapping, and the leftover files that mislead.** Re-confirmed 2026-10-02:
69
+ - `api.beta.togahub.com` **and** `api.dev.sandbox.togahub.com` both hit EB env **`api-sandbox-dev`** (siblings `webhook-sandbox-dev`, `worker-sandbox-dev`), EB var **`ENVIRONMENT=beta`** → `Config/beta.ini`.
70
+ - With no `git.beta.json` the branch falls through to `_beta`. The CodePipeline `API-Sandbox-Dev` source/trigger is `_sandbox-dev`, but its `_underscore` artifact is **not** what gets deployed — the prebuild clone wins.
71
+ - **Fix (2026-10-02): `.ebextensions/git.beta.json` pinning `branch: _sandbox-dev`.** ⚠ As of 2026-10-02 it exists only in a local GitHub Desktop stash in api2 — **not committed to any branch**. Until it is committed to api2 `_beta` and redeployed, beta still clones `_underscore` `_beta`. Like every `git.*.json` it holds a credential — never copy the value.
72
+ - `Config/sandbox-dev.ini` + `git.sandbox-dev.json` exist but do **not** serve this host.
73
+ - **Switching `ENVIRONMENT` to `sandbox-dev` was considered and deferred** (goal: retire `beta` later). Blockers: `Config/sandbox-dev.ini` points FedEx at **live** `apis.fedex.com` and lacks PayPal sandbox / `startechPcmaticb2b`; tokens are env-scoped (`EN-5`); dev-sandbox Core has 142 `Domains` + 3 `ClientAuthentications` on environment `beta` (id 4) and none on `sandbox-dev` (id 23). Prudential's `beta` `Core.DatabaseHosts` row points at the sandbox-dev cluster.
72
74
 
73
75
  **⚠ Diagnostic: `Call to undefined method _Model_Client_<X>::<method>()` means the CLASS ISN'T DEPLOYED.** When a client model override is missing from the deployed `_underscore`, V2 does **not** error at resolution time. `class_exists` fails, so V2 quietly **declines the override and falls back to the BASE model** (`V2.php:3155-3162`). The base model has no client-specific method, so you get e.g. `Call to undefined method _Model_Client_Subscription::cancelApi()`. **Read this error as "the override class is not on the server," not "the code is wrong."** The `_Model_Client_*` (base) prefix — where you expected `_Model_<Slug>_*` — is the tell. Confirm by file presence, then check which branch that tier clones. Related: **the slug V2 uses to pick the override class is `Clients.clientIdentifier`** (`V2.php:1521`), taken from the JWT. `Core.Clients` has **no `slug` column** — do not go looking for one.
74
76
 
75
77
  **⚠ Moving a tier onto its correct branch is a SCHEMA event, not just a code event.** When a deploy moves an environment **forward** onto a newer `_underscore` branch — including a deploy that *fixes* a wrong branch pin — it fast-forwards across every model change made since the pin, and `_Model::initialize()` SELECTs **every declared property**. Any of those columns without a merged `dbchanges2` migration becomes an **environment-wide MySQL 1054 → HTTP 500 / `EO-1`**, for every tenant, immediately. **Concrete (2026-08-26):** the deploy moved `sandbox-client` off the `_production` framework (which it had been running for weeks) onto `_sandbox-client`, carrying ~2 weeks of model changes whose migrations were stranded on an unmerged `dbchanges2` branch. Table-meta requests 500'd on `RecordFields.maxLength`, then (once patched) on `TableViewFields.isGroupable`. **Rule: pair a branch-pin correction with a migration sweep** — enumerate what the new branch expects (`git log origin/_production..origin/<env-branch> --oneline -- Model/`) **before** redeploying, or the tier trades a stale-code bug for a total outage. Full procedure: [framework branch ahead of schema](../../dbchanges2/workflows/framework-branch-ahead-of-schema.md).
76
78
 
77
- **⚠ Never use a RECENTLY-ADDED `_underscore` symbol from api2 without checking branch containment.** "Land `_underscore` before deploying api2" is necessary but **not sufficient**, because every environment pins its **own** `_underscore` branch. A framework symbol merged to `_production` may be **absent from `_beta` and `_sandbox-dev` for weeks**, so app code that references it is fine on prod and **fatal on the sandboxes**. **Concrete (2026-09-01, `TRUE-81056`):** the first version of the duplicate-`transactionId` log fix used `\_Database::ERRNO_DUPLICATE_ENTRY` inside a `catch` block. That constant was added by `_underscore` commit **`6355d0f0`** and, verified with `git show <branch>:Database.php`:
79
+ **⚠ Never use a RECENTLY-ADDED `_underscore` symbol from api2 without checking branch containment.** (Example below is historical — `ERRNO_DUPLICATE_ENTRY` is on `_production`/`_beta`/`_sandbox-dev` as of 2026-10-02; the rule stands.) "Land `_underscore` before deploying api2" is necessary but **not sufficient**, because every environment pins its **own** `_underscore` branch. A framework symbol merged to `_production` may be **absent from `_beta` and `_sandbox-dev` for weeks**, so app code that references it is fine on prod and **fatal on the sandboxes**. **Concrete (2026-09-01, `TRUE-81056`):** the first version of the duplicate-`transactionId` log fix used `\_Database::ERRNO_DUPLICATE_ENTRY` inside a `catch` block. That constant was added by `_underscore` commit **`6355d0f0`** and, verified with `git show <branch>:Database.php`:
78
80
 
79
81
  | `_underscore` branch | `ERRNO_DUPLICATE_ENTRY` |
80
82
  |---|---|
@@ -107,9 +109,11 @@ So on `api.beta.togahub.com` and `api.dev.sandbox.togahub.com` the code fataled
107
109
  - **⚠ "grep says this framework function doesn't exist" is meaningless without naming the checked-out branch.** Because `_underscore` is cloned at build from a **moving per-environment branch**, a local checkout is routinely behind the branch a tier runs. Concrete (2026-08-24): a session concluded — with **three** corroborating pieces of code evidence — that `_Model_Core_Surface::_resolveLanguageId()` had never been implemented and the KB was wrong. It was not; the local tree was on a branch predating it. The method is present on `_sandbox-client`, `_sandbox-dev`, `surface-layer` and `origin/_beta` (merged into `_sandbox-client` 2026-07-22, `6809c6b9`). **Always run `git branch --show-current` (and check sibling branches) before declaring framework code absent** — corroborating reads from the same stale tree multiply the error.
108
110
  - **Deploy path is build-time, so a git push alone does not update running instances.** The `_underscore` clone happens in the prebuild hook at build time: commit the fix on the feature branch, merge into the deploy-channel branch (`_` + `ENVIRONMENT`, e.g. `_sandbox-client`), push, **then redeploy the EB environment** to re-run the clone.
109
111
  - **⚠ SECURITY — every `.ebextensions/git.*.json` carries a plaintext GitHub PAT** committed in the repo (used to clone `_underscore`). Pre-existing leak; treat the committed tokens as compromised, rotate, move to EB environment properties / SSM. Do **not** copy the token value anywhere. **Escalated 2026-08-24 — still not rotated, and exposure has WIDENED.** The token is present in **all ~20 per-environment `git.*.json` files**, and an attempted in-place redaction *increased* exposure: BSD `sed` on macOS does not support `\|` alternation, so the command failed in a way that **printed the token value into the session transcript**. Two durable takeaways: (a) rotation is overdue — treat it as an open action item; (b) **do not attempt ad-hoc redaction of a secret with a one-liner** — a failed redaction commonly echoes the value it was meant to hide (GNU-vs-BSD `sed` regex differences make macOS one-liners especially prone). Rotate at the source instead.
112
+ - **⚠ sandbox-dev DB `max_connections = 90`** (prod client cluster: 3000). Each api2 request opens ~6 connections (+1 for the autoincrement counter), so ~15+ simultaneous requests on beta / dev-sandbox fail `Too many connections` → HTTP 500 `EO-1`. Pre-existing (4322 such errors logged). Keep load tests at ~10 concurrent; a burst 500 there is not a code bug.
110
113
  - **Duplicate integration code is the trap this creates.** When the same capability is needed in both tiers it must exist twice (once per framework) — keep the credential/config location documented in both places so a rotation updates both.
111
114
 
112
115
  ## Change history
116
+ - 2026-10-02 — beta/dev-sandbox host → `api-sandbox-dev` mapping, `git.beta.json` pin to `_sandbox-dev` (uncommitted), deferred ENVIRONMENT rename and its blockers, sandbox-dev `max_connections` limit. (rgirish)
113
117
  - 2026-09-25 — renamed product reference from "TogaIQ" to "Talos" (rebrand; TOGa IQ is now the separate `toga25-iq` app) (jcardinal)
114
118
 
115
119
  ## Related
@@ -6,7 +6,7 @@ project: API
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-09-22
9
+ updated: 2026-10-02
10
10
  owners: [jcardinal, bala, ajean, rgirish]
11
11
  files:
12
12
  - api2/Controller/Index.php
@@ -34,9 +34,9 @@ api2's front controller ("**Prong A**") can **detect a retriable concurrency fai
34
34
 
35
35
  The replay is **route-scoped by an allowlist** (`DEADLOCK_RETRIABLE_ROUTES` — currently `['/v2/advance-shipping-notices', '/v2/service-requests']`) — because whole-request in-process replay is **only safe for routes with no non-rollbackable external side effect**. The retry gate is now **`wasDeadlock() || wasRetriableUniqueRace()`**.
36
36
 
37
- > **Status (2026-09-22).** Working on `_sandbox-dev` and **proven under real concurrency** (10/10 concurrent SR POSTs, zero duplicates). **Production still has NO race handling at all** for this — prod needs this fix plus the original `TRUE-82095` work. Prod evidence: 12 duplicate-number failures since 2026-08-18, latest `RITM6811124` / `REQ4510546`, which only succeeded on a manual resubmit as `SR131026`. The race is now **confirmed from production traffic**, not just the load test — see *Confirmed in production* below.
37
+ > **Status (2026-10-02).** The SR number race is now fixed at the source — a per-tenant `Parameters` counter in `_Model` (TRUE-82209, on `_underscore` `_beta`/`_sandbox-dev`), see [FIELD_AUTOINCREMENT record numbering](../../_underscore/features/autoincrement-record-numbering.md). The `/v2/service-requests` replay below (api2 `_sandbox-dev` only; `_beta`/`_production` list ASN only) is no longer the fix for numbers. **Extending replay to every numbered route was rejected**: Adyen / Elite / Growrk `postPost` hooks have external side effects. Prod still has neither fix.
38
38
 
39
- **Constants (verified 2026-09-22):** `MAX_DEADLOCK_ATTEMPTS = 5` (not 3), backoff 20ms-120ms widening per attempt, `DEADLOCK_RETRIABLE_ROUTES = ['/v2/advance-shipping-notices', '/v2/service-requests']`.
39
+ **Constants (verified 2026-10-02):** `MAX_DEADLOCK_ATTEMPTS = 5`, backoff 20ms-120ms widening per attempt. `DEADLOCK_RETRIABLE_ROUTES` = `['/v2/advance-shipping-notices', '/v2/service-requests']` on api2 `_sandbox-dev`; ASN only on `_beta` and `_production`.
40
40
 
41
41
  ## How it works
42
42
 
@@ -45,7 +45,7 @@ The replay is **route-scoped by an allowlist** (`DEADLOCK_RETRIABLE_ROUTES` —
45
45
  - **The ASN model** calls the IF-create `internalApiRequest` with **`throwExceptionsOnError:false`**; on an empty response with `lastErrorNumber === 1062` it sets `$retriableUniqueRace = true` and returns null. `postPost` then **aborts immediately** (issues no further query).
46
46
  - **The controller's Prong A gate** becomes `wasDeadlock() || wasRetriableUniqueRace()`. Replaying the whole request on a fresh REPEATABLE-READ snapshot lets the retry's GET see the winner's committed IF and reuse it.
47
47
 
48
- Bounded by `MAX_DEADLOCK_ATTEMPTS = 5`, gated to `DEADLOCK_RETRIABLE_ROUTES = ['/v2/advance-shipping-notices', '/v2/service-requests']`.
48
+ Bounded by `MAX_DEADLOCK_ATTEMPTS = 5`, gated to `DEADLOCK_RETRIABLE_ROUTES`.
49
49
 
50
50
  **V2 concurrency mechanics (durable framework knowledge)** — the V2-engine facts that decide **how any interceptor-level write race must be handled**, and why the fix is a whole-request replay and not an in-place recovery:
51
51
  - **Response poisoning (load-bearing).** A failed nested save via `internalApiRequest` calls `addDefinedMessage(EV-10)` on the **shared** `$api->response`, setting a non-2xx status **before** control returns. So even if an interceptor catches the throw and "recovers," `execute()` recomputes `isSuccess = false` (`V2.php` ~2154) and **self-rolls-back DB_CORE + DB_CLIENT** (~2157-2159). **An interceptor therefore CANNOT recover a failed internal save in place** — the only correct recovery is a whole-request replay. This same poisoning drives the Prong A retry gate and, on exhaustion, produces the coherent EV-10 response.
@@ -82,8 +82,8 @@ Two consequences of the typed exception worth knowing:
82
82
  **Symptom:** `/v2/service-requests` returned `400 EV-10` *"Duplicate entry 'SR131023' for key 'ServiceRequests.number'"* — 21 consecutive failures, all ~1.8s with no backoff visible. ServiceNow was **not** sending duplicates. The retry existed and was correctly wired; it just never fired.
83
83
 
84
84
  **The root-cause chain, in order:**
85
- 1. **`ServiceRequests.number` is not a MySQL auto-increment.** `_Model` `FIELD_AUTOINCREMENT` does a `SELECT MAX(number)+1` and then an `INSERT`, **with no lock between them** (see [FIELD_AUTOINCREMENT record numbering](../../_underscore/features/autoincrement-record-numbering.md)). Two concurrent POSTs read the same MAX, pick the same number, and the loser gets MySQL 1062.
86
- 2. **The retry was already built for exactly this.** `_underscore/Model.php` sets `_Database::$retriableUniqueRace` on that 1062, and `api2/Controller/Index.php` replays the whole request when `_Database::wasRetriableUniqueRace()` is true. That helper requires **both** the flag **and** `$lastErrorNumber === 1062`.
85
+ 1. **`ServiceRequests.number` is not a MySQL auto-increment.** The old `_Model` `FIELD_AUTOINCREMENT` did `SELECT MAX(number)+1` then an `INSERT`, **with no lock between them** (since replaced by a counter on `_beta`/`_sandbox-dev`) (see [FIELD_AUTOINCREMENT record numbering](../../_underscore/features/autoincrement-record-numbering.md)). Two concurrent POSTs read the same MAX, pick the same number, and the loser gets MySQL 1062.
86
+ 2. **The retry was already built for exactly this.** `_underscore/Model.php` set `_Database::$retriableUniqueRace` on that 1062 (removed by TRUE-82209 — no branch's `Model.php` sets it now), and `api2/Controller/Index.php` replays the whole request when `_Database::wasRetriableUniqueRace()` is true. That helper requires **both** the flag **and** `$lastErrorNumber === 1062`.
87
87
  3. **The logger destroyed the second half of that condition.** `_Component_Api_V2::saveApiLogUnderItsOwnTransactionId()` runs **between the failed INSERT and the retry check**, inside `execute()`. Its resend loop **deliberately causes its own 1062s** — it renames `transactionId` on collision and retries (see [V2 request logging](./request-logging.md)). Those bookkeeping 1062s **overwrote the caller's `$lastErrorNumber`**, so `wasRetriableUniqueRace()` returned false and the replay never ran.
88
88
 
89
89
  **Fix:** save and restore **both** `_Database::$lastErrorNumber` and `_Database::$retriableUniqueRace` in a `try/finally` **inside** `saveApiLogUnderItsOwnTransactionId()`. Scoped inside that method — *not* at the call site — so it only undoes that loop's own errors; deadlocks, the ASN route, and every other route behave exactly as before.
@@ -129,7 +129,6 @@ Two details that cost time:
129
129
  Both were considered and left alone on purpose. Do not "fix" either without re-checking the reasoning.
130
130
 
131
131
  - **The `V2.php` resend loop is unbounded** (`for ($resendNumber = 1; ; ...)`). Safe **today only because every pass changes a UNIQUE column**, so it cannot re-ask for a taken value.
132
- - **`_underscore/Model.php` marks ANY 1062 as a retriable race** without checking *which* unique key failed. `ServiceRequests` has three unique keys (`PRIMARY`, `uuid`, `number`) and only `number` is the benign race. Harmless today: `uuid` is a random uuid so a collision is effectively impossible, and `PRIMARY` cannot duplicate on INSERT.
133
132
 
134
133
  ### Rejected finding — a null `transactionId` does NOT break the first log save
135
134
 
@@ -143,9 +142,8 @@ Checked and **wrong**; recorded so it is not raised again. `Logs.Api.transaction
143
142
  - **Anything running between the failing query and the retry check must save/restore `_Database::$lastErrorNumber` + `$retriableUniqueRace`.** The API logger's own 1062s disarmed this retry completely for weeks; a 1062 is not protected from overwrite the way a deadlock is.
144
143
  - **Test concurrency with a gate file**, not a background loop — a loop spreads the requests out and can miss the race. Auth token is at `.data.tokens.access`.
145
144
  - **A branch cut from `_production` may have the CALL SITE but not the METHOD.** `saveApiLogUnderItsOwnTransactionId()` (commit `db94a26`) exists only on `_sandbox-dev`; branch `TRUE-82095` came off `_production` and would have fataled with *call to undefined method* on every DB-logged request. When branching off `_production`, verify the **callee** exists, not just the caller.
146
- - `ERRNO_DUPLICATE_ENTRY` is absent from `_beta`/`_sandbox-dev` — naming it directly fatals there; define a local `1062` constant and read `$lastErrorNumber` via `property_exists()`.
147
- - `DEADLOCK_RETRIABLE_ROUTES` is now `/v2/advance-shipping-notices` **and** `/v2/service-requests`; do not add a route without proving it is replay-safe (no irreversible side effect, no mutated superglobal).
148
- - **Every other `FIELD_AUTOINCREMENT` model has the same unlocked generator** — SalesOrder, PurchaseOrder, Ticket, ItemReceipt, InventoryAdjustment, TransferOrder — and **none is on `DEADLOCK_RETRIABLE_ROUTES`**, so a duplicate-number clash on those routes still returns 400. The durable answer is fixing the number generator so the read and the insert cannot split; that is a bigger, separate change.
145
+ - `ERRNO_DUPLICATE_ENTRY` is now on `_production`, `_beta`, `_sandbox-dev` (verified 2026-10-02); older branches still need the local-constant guard.
146
+ - Do not add a route to `DEADLOCK_RETRIABLE_ROUTES` without proving it is replay-safe (no irreversible side effect, no mutated superglobal). Number races are fixed in the generator, not here.
149
147
  - **The `SR…` number inside a `Duplicate entry` error belongs to the WINNER, not the failed request.** The loser never got a number. A "duplicate SR131023" report means one request was lost, not that two rows share a number.
150
148
  - **ServiceNow resends a failed SR on its own** (proven: `REQ4510546` failed, came back ~6h later as `SR131026`). After the prod deploy adds an internal retry, an outside resend could produce a **second SR for the same REQ** — confirm with the client first.
151
149
  - Do not reach for this protocol for an HTTP 429 — use `_ApiRequest::setAutoRetry()`.
@@ -160,6 +158,7 @@ Checked and **wrong**; recorded so it is not raised again. `Logs.Api.transaction
160
158
  - [ENVIRONMENT decides the `_underscore` branch](./environment-variable-drives-underscore-branch.md)
161
159
 
162
160
  ## Change history
161
+ - 2026-10-02 — SR number race now fixed in `_Model` (Parameters counter, TRUE-82209); replay-for-all-numbered-routes rejected; corrected per-branch route list and `ERRNO_DUPLICATE_ENTRY` availability. (rgirish)
163
162
  - 2026-09-22 — Added production confirmation of the race (3 same-second ServiceNow POSTs, `Logs_Prudential.Api` 8736482/8736483/8736484, 1062 on `SR131023`) and the open double-SR risk from ServiceNow's own resend. Investigation only, no code change. (rgirish)
164
163
  - 2026-09-22 — Found and fixed the reason this retry never fired on `/v2/service-requests`: the API logger's own 1062s overwrote `_Database::$lastErrorNumber`; now saved/restored in `try/finally` inside `saveApiLogUnderItsOwnTransactionId()`. Proven 10/10 under real concurrency on sandbox-dev. Also fixes ASN. Prod still unpatched. (rgirish)
165
164
  - 2026-09-22 — Corrected constants: `MAX_DEADLOCK_ATTEMPTS = 5` (was documented as 3); `DEADLOCK_RETRIABLE_ROUTES` now includes `/v2/service-requests`. (rgirish)
@@ -21,7 +21,7 @@ project: _Underscore
21
21
  client: compass-usa
22
22
  type: profile
23
23
  status: active
24
- updated: 2026-09-30
24
+ updated: 2026-10-05
25
25
  owners: [jcardinal, bala, tcox, apeterson, dfranks, akhokhani, ajean, rgirish]
26
26
  files: []
27
27
  related:
@@ -219,6 +219,7 @@ separate, related client (see its own profile).
219
219
  the ⚠ rule that **deactivation is raw bulk SQL, so no model hook ever fires on it**.
220
220
 
221
221
  ## Notes
222
+ - **⚠ SALES_ORDERS froze 2026-10-02 → 10-05 (Issue 940): closed-order delete matched the Office Depot twin SO** (dual-SO trap). Fixed in library with the per-client scope; open: higher-tier mirror fulfillments 91650/91651. Details: [closed-order rule](../../1.0/apps/library/features/netsuite-closed-order-rule.md).
222
223
  - **⚠ SALES_ORDERS froze 2026-09-29 on ACL gaps the closed-order delete hit** (transfer-orders 312/313, entitlement-sales-orders 257, sales-order-payments 254). Fixed by `dbchanges2/Client_Compass/2026-09-29a - TransferOrderAclExpressions.sql`, then mode reset to `864000-IDLE`. Details: [closed-order rule](../../1.0/apps/library/features/netsuite-closed-order-rule.md).
223
224
  - **⚠ The 1.0 NetSuite sync links item fulfillments ONLY to `customerId=3` / `locationId=615286`
224
225
  SalesOrders (2026-09-16).** This Compass-specific hardcoded scope filter lives in
@@ -17,7 +17,7 @@ project: _Underscore
17
17
  client: prudential
18
18
  type: profile
19
19
  status: active
20
- updated: 2026-09-29
20
+ updated: 2026-10-02
21
21
  owners: ["jcardinal", "rgirish", "bala", "mhammontree"]
22
22
  files: []
23
23
  related:
@@ -29,9 +29,7 @@ related:
29
29
  - features/dell-lch-iop-transmissions.md
30
30
  - features/device-information-import-and-contact-linking.md
31
31
  - workflows/http2-alb-workaround.md
32
- - features/service-request-auth-rejection-alert.md — the 15-minute refused-REQ (401/403) alert cron,
33
- where each code is logged, and how to tell "we refused it" from "they never sent it".
34
- - features/supply-orders-csv-export.md
32
+ - features/supply-orders-csv-export.md
35
33
  - features/order-email-pdf-attachments.md
36
34
  - features/itemclasses-device-tier-data.md
37
35
  - workflows/store-saml-deploy-topology.md
@@ -122,6 +120,12 @@ The order-created cron branches on this id; the shipped/delivered crons branch o
122
120
  The 15-minute `prudential_auth_rejection_alert.php` cron now alerts on them — see
123
121
  features/service-request-auth-rejection-alert.md.
124
122
 
123
+ - **SR number race (ServiceNow bursts → 1062 `Duplicate entry 'SR…'` → 400 `EV-10`, SR lost).** Fixed in
124
+ the framework by a `Parameters` counter (`NEXT_NUMBER_SERVICE_REQUESTS_SR`); the `usleep(0.5–2s)` stopgap in
125
+ `_Model_Prudential_ServiceRequest::prePost()` is removed. Both on `_underscore` `_beta`/`_sandbox-dev` only
126
+ (2026-10-02) — prod still has the race and the stopgap. Load-test with a distinct contact + serial per call and
127
+ `customer.uuid` set. See [FIELD_AUTOINCREMENT record numbering](../../2.0/apps/_underscore/features/autoincrement-record-numbering.md).
128
+
125
129
  ## Monitoring
126
130
  - `Monitor/Prudential/{SalesOrderGenerationBacklog,DellPurchaseOrderTransmissionBacklog,OrderShippedUpdateBacklog}`
127
131
  (worker2 → OneUptime) watch the order pipeline for stalls. Thresholds are DB-tunable and the
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "toga-ai",
3
- "version": "1.0.938",
3
+ "version": "1.0.940",
4
4
  "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
5
5
  "keywords": [
6
6
  "claude",