toga-ai 1.0.938 → 1.0.939
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/2.0/apps/_underscore/INDEX.md +1 -1
- package/knowledge/2.0/apps/_underscore/features/autoincrement-record-numbering.md +36 -20
- package/knowledge/2.0/apps/api2/features/environment-variable-drives-underscore-branch.md +12 -8
- package/knowledge/2.0/apps/api2/features/v2-deadlock-retry.md +9 -10
- package/knowledge/clients/prudential/profile.md +8 -4
- package/package.json +1 -1
|
@@ -10,7 +10,7 @@
|
|
|
10
10
|
| [_ApiRequest — JSON encode/decode & api-logging behavior](features/apirequest-json-content-type.md) | `_ApiRequest`, the 2.0 outbound HTTP client — JSON encode/decode, retry, and api-logging traps; open before calling it or reading `Logs.Api`. |
|
|
11
11
|
| [Assortment Name Translation (AssortmentTranslations sidecar)](features/assortment-name-translation.md) | Serves Assortment names in multiple languages via an `AssortmentTranslations` sidecar table on the metadata-driven translation layer; open when translating asso |
|
|
12
12
|
| [Asynchronous Query Execution (writes-only, via Worker)](features/async-query-execution.md) | Run a slow WRITE asynchronously via the Worker tier so a long request write cannot hold a stale connection and hit "MySQL server has gone away"; open before usi |
|
|
13
|
-
| [FIELD_AUTOINCREMENT record numbering (
|
|
13
|
+
| [FIELD_AUTOINCREMENT record numbering (Parameters counter, race-safe, NULL-only trigger)](features/autoincrement-record-numbering.md) | ORM-generated record numbers (`SA100001`, `SR131021`) via `FIELD_AUTOINCREMENT` and the per-tenant `Parameters` counter that makes them race-safe; open when add |
|
|
14
14
|
| [Calculated-column filter options (`<field>FilterOptions` model static → table meta)](features/calculated-column-filter-options.md) | Lets a calculated (`FIELD_SQL`) table column supply its own filter dropdown options via a `<field>FilterOptions` model static; open when adding a data-driven co |
|
|
15
15
|
| [FIELD_SQL calculated fields — the underscore-prefix + same-name-method contract](features/calculated-sql-fields.md) | A `FIELD_SQL` (calculated) field on a `_Model` is bound by a **two-part contract that `_Model` enforces by throwing at model-construction time**, not by convent |
|
|
16
16
|
| [Carrier Shipping Labels (UPS/FedEx) & NetSuite Item Fulfillment](features/carrier-shipping-labels.md) | Backend mechanics behind Fulfill & Ship: buying/persisting a carrier label (UPS/FedEx) and creating the NetSuite Item Fulfillment; open for label, reprint, retu |
|
|
@@ -1,29 +1,32 @@
|
|
|
1
1
|
---
|
|
2
|
-
title: "FIELD_AUTOINCREMENT record numbering (
|
|
2
|
+
title: "FIELD_AUTOINCREMENT record numbering (Parameters counter, race-safe, NULL-only trigger)"
|
|
3
3
|
framework: "2.0"
|
|
4
4
|
repo: _underscore
|
|
5
5
|
project: _Underscore
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-
|
|
9
|
+
updated: 2026-10-02
|
|
10
10
|
owners: ["bala", "rgirish"]
|
|
11
11
|
files:
|
|
12
12
|
- _underscore/Model.php
|
|
13
|
+
- _underscore/Database.php
|
|
13
14
|
- _underscore/Model/Client/SalesOrder.php
|
|
14
15
|
- _underscore/Model/Client/TransferOrder.php
|
|
16
|
+
- _underscore/Model/Prudential/ServiceRequest.php
|
|
15
17
|
related:
|
|
16
18
|
- ../architecture.md
|
|
17
19
|
- ../../api2/features/v2-deadlock-retry.md
|
|
20
|
+
- ../../api2/features/environment-variable-drives-underscore-branch.md
|
|
18
21
|
- ../../worker2/features/netsuite-transferorder-outbound-push.md
|
|
19
22
|
- ../../worker2/features/netsuite-salesorder-outbound-push.md
|
|
20
23
|
---
|
|
21
24
|
|
|
22
|
-
ORM-generated
|
|
25
|
+
ORM-generated record numbers (`SA100001`, `SR131021`) via `FIELD_AUTOINCREMENT` and the per-tenant `Parameters` counter that makes them race-safe; open when adding a prefix, debugging a duplicate-number 1062, or load-testing creates.
|
|
23
26
|
|
|
24
27
|
## Summary
|
|
25
28
|
|
|
26
|
-
A
|
|
29
|
+
A record number is generated by the ORM, not MySQL. A model declares:
|
|
27
30
|
|
|
28
31
|
```php
|
|
29
32
|
public $number = [
|
|
@@ -33,36 +36,49 @@ public $number = [
|
|
|
33
36
|
];
|
|
34
37
|
```
|
|
35
38
|
|
|
36
|
-
|
|
39
|
+
12 models use it (SalesOrder `SA`, PurchaseOrder `P`, ServiceRequest `SR`, TransferOrder `TA`, Ticket, ItemReceipt, InventoryAdjustment, ...). For `DB_CLIENT` models the next number comes from a **counter row in the tenant's `Parameters` table**, taken on its own connection, so concurrent creates never get the same number.
|
|
40
|
+
|
|
41
|
+
**Branch state (verified 2026-10-02):** counter code (TRUE-82209, `1760b58a` + caps-key follow-up `bf22215b`) is on `_underscore` `_beta`, `_sandbox-dev`, `TRUE-82209`. **Not** on `_production` or `_sandbox-client` — prod still runs the old `MAX()+1` path and the Prudential `usleep` stopgap.
|
|
37
42
|
|
|
38
43
|
## How it works
|
|
39
44
|
|
|
40
|
-
|
|
45
|
+
1. **Fires only when the field is NULL.** A supplied number always wins (NetSuite importer's `tranId`, Compass PO `mitsSalesOrder`), and bypasses the counter. A front-end that posts `number` silently disables the generator.
|
|
46
|
+
2. **Counter key:** `NEXT_NUMBER_<TABLE_IN_SNAKE_CASE>_<PREFIX><SUFFIX>`, all caps — e.g. `NEXT_NUMBER_SERVICE_REQUESTS_SR`, `NEXT_NUMBER_SALES_ORDERS_SA`, `NEXT_NUMBER_PURCHASE_ORDERS_P`, `NEXT_NUMBER_SALES_ORDERS_SAC` (Compass Canada). One row per table + prefix + suffix. `value` = last number issued.
|
|
47
|
+
3. **Allocate** (`_Model::allocateAutoincrementNumber()`): `UPDATE Parameters SET value = LAST_INSERT_ID(CAST(value AS UNSIGNED)+1) WHERE key=...`, read back with `getInsertId()`, then **commit at once**. Runs on `_Database::DB_CLIENT_SEQUENCE` (`'ClientSequence'`), a separate connection — the api2 request transaction commits only at request end, so taking the row lock there would serialize every create.
|
|
48
|
+
4. **`DB_CLIENT_SEQUENCE` is automatic.** `_Database::register()` points it at the same DB whenever the alias is `_underscore::DB_CLIENT` (covers api2, worker2, saml, cXML, Nychh worker), and drops its connection when DB_CLIENT is re-pointed to another tenant. No app change needed.
|
|
49
|
+
5. **Self-seeding.** Row missing → run the old `MAX()` SQL on the sequence connection (writer, not read host), `INSERT ... ON DUPLICATE KEY UPDATE value = GREATEST(...)`, commit between each step. No per-tenant SQL or seed script.
|
|
50
|
+
6. **Collision retry in `save()`.** If the INSERT fails 1062 **and** the message contains the number this save generated (a hand-set number took it), resync the counter past `MAX()` and retry, up to `MAX_AUTOINCREMENT_ATTEMPTS = 5`. A 1062 on any other unique key (uuid) is re-thrown. Any failed counter step rolls back the sequence connection.
|
|
51
|
+
7. **Unchanged:** Core-DB models and `search()` keep the old `MAX()+1` path. **No zero padding** (prefix + integer). **Gaps are possible** (rolled-back request still used its number) — same as before.
|
|
41
52
|
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
53
|
+
**Rejected alternatives (2026-10-02):**
|
|
54
|
+
- Whole-request replay via `DEADLOCK_RETRIABLE_ROUTES` for every numbered route — unsafe per tenant: Adyen / Elite / Growrk `postPost` hooks have external side effects. See [V2 request retry](../../api2/features/v2-deadlock-retry.md).
|
|
55
|
+
- Derive number from the AUTO_INCREMENT `id` — `number` is NOT NULL UNIQUE and hand-set numbers break it.
|
|
56
|
+
- `GET_LOCK` — the lock is held to request end, so creates serialize.
|
|
45
57
|
|
|
46
|
-
|
|
58
|
+
## Gotchas
|
|
47
59
|
|
|
48
|
-
|
|
60
|
+
- **⚠ Old path race (still live on prod until this ships):** `SELECT MAX+1` then a separate INSERT → concurrent creates pick one number → loser gets 1062 `Duplicate entry 'SR131021' for key 'ServiceRequests.number'` → HTTP 400 `EV-10`, record lost. The number in the error belongs to the **winner**.
|
|
61
|
+
- **⚠ A prefix on a SHARED model is not unique across clients** — every tenant starts at `TA100000`. Never key a cross-client id on `number`; NetSuite account 1095849 is shared, so pushes key `externalId` on uuid (`toga-so-<uuid>` / `toga-to-<uuid>`).
|
|
62
|
+
- **SA / TA numbers are placeholders** — overwritten by NetSuite `tranId` on write-back (~12s). Never build a key or match on `number`. Keeping a Toga number permanently needs **two** changes: a guard in `writeBackNetSuiteId()` **and** dropping `number` from the PUT in `library/app/api/toga2.php` (~L4295) and `_underscore/Trait/Netsuite/TransferOrder.php` (tried 2026-09-02, reverted).
|
|
63
|
+
- **`FIELD_AUTOINCREMENT` is special-cased in `buildSqlFieldValue()`**, so a null value does not collapse a `load()` WHERE the way a null `FIELD_CHAR` does — see [_underscore architecture](../architecture.md).
|
|
64
|
+
- **Adding the option renumbers nothing**; existing rows are untouched.
|
|
65
|
+
- **Error class depends on boot.** Real boot calls `mysqli_report(MYSQLI_REPORT_STRICT)` (`Error.php:217`), so a failed query returns false and `_Query` throws `_Exception_Database` (which `save()` catches). A test harness that skips this boot gets raw `mysqli_sql_exception` instead — boot the framework fully when testing.
|
|
66
|
+
- **Reproducing the race needs distinct rows per call.** Each concurrent test request must use a **different contact** (employee_xid / email) **and unit serialNumber**. Reusing one makes requests queue on that row lock and the race never shows (old code "passed" 10/10). Release all calls at once with the gate-file pattern ([V2 request retry](../../api2/features/v2-deadlock-retry.md)). Prudential `POST /v2/service-requests` also needs `customer.uuid`. Network latency is irrelevant.
|
|
67
|
+
- **Beta can't take big bursts:** sandbox-dev DB `max_connections = 90`; ~15+ simultaneous requests → `Too many connections` → 500 `EO-1`. Keep bursts at ~10. See [ENVIRONMENT decides the branch](../../api2/features/environment-variable-drives-underscore-branch.md).
|
|
68
|
+
- **Run `php -l Model.php Database.php` after resolving merge conflicts on env branches.** The TRUE-82209 merge into `_sandbox-dev`/`_beta` dropped `public static $_aliases` / `private static $_parked` in `Database.php` and duplicated `private $_model_generatedAutoincrementFields` in `Model.php` → beta outage (`Access to undeclared static property _Database::$_aliases`). The error page also dumped `$_SERVER`.
|
|
49
69
|
|
|
50
|
-
##
|
|
70
|
+
## Verification
|
|
51
71
|
|
|
52
|
-
|
|
53
|
-
- **A generated number can be a PLACEHOLDER — decide its lifetime explicitly.** `SA`/`TA` numbers are overwritten by NetSuite's `tranId` on write-back, so the generated value lives about 12 seconds. That is the intended behaviour for both order types, but it makes `number` a moving value — never build a key, a match, or an external id on it.
|
|
54
|
-
- **Keeping a Toga-side number permanently takes TWO changes, not one** (written 2026-09-02, then reverted at the developer's request — recorded so nobody rediscovers it): (1) a guard in `writeBackNetSuiteId()` so the push does not overwrite `number`, and (2) dropping `number` from the PUT in both `library/app/api/toga2.php` (~L4295) and `_underscore/Trait/Netsuite/TransferOrder.php`. Miss (2) and the 1.0 importer compares `number` and PUTs NetSuite's over the top on its next run, so the guard alone achieves nothing. Both files are back to untouched (verified with an empty `git diff`).
|
|
55
|
-
- **`FIELD_AUTOINCREMENT` is one of the field types special-cased by `buildSqlFieldValue()`**, so a null value on it does NOT collapse a `load()` WHERE clause the way a null `FIELD_CHAR` does — see the collapsed-WHERE OOM note in [_underscore architecture](../architecture.md).
|
|
56
|
-
- **Adding the option to a model does not renumber anything** and does not touch existing rows; it only affects inserts that arrive with `number` NULL.
|
|
57
|
-
- **Concurrent inserts can pick the same number** (unlocked `MAX+1` then `INSERT`) → MySQL 1062 → 400. Only two api2 routes retry it; every other model is exposed. See [V2 request retry](../../api2/features/v2-deadlock-retry.md).
|
|
58
|
-
- **A front-end that sends `number` silently disables the feature.** There is no error — the posted value is simply accepted. Check the actual POST body before concluding the generator is broken.
|
|
72
|
+
Beta (`api.beta.togahub.com` → sandbox-dev), 2026-10-02: before fix 8/10 concurrent creates failed 1062; after fix 100/100 (10 rounds of 10), 116 distinct numbers, counter matched `MAX()`.
|
|
59
73
|
|
|
60
74
|
## Related
|
|
61
75
|
|
|
62
76
|
- [_underscore architecture](../architecture.md)
|
|
77
|
+
- [V2 request retry](../../api2/features/v2-deadlock-retry.md)
|
|
63
78
|
- [NetSuite TransferOrder outbound push](../../worker2/features/netsuite-transferorder-outbound-push.md)
|
|
64
79
|
- [NetSuite SalesOrder outbound push](../../worker2/features/netsuite-salesorder-outbound-push.md)
|
|
65
80
|
|
|
66
81
|
## Change history
|
|
67
|
-
- 2026-
|
|
82
|
+
- 2026-10-02 — Replaced unlocked `MAX()+1` with a per-tenant `Parameters` counter on its own committed connection (`DB_CLIENT_SEQUENCE`), self-seeding, 1062 resync retry; removed Prudential `usleep` stopgap. On `_beta`/`_sandbox-dev` only. (rgirish)
|
|
83
|
+
- 2026-09-22 — Recorded that the `MAX+1` generator is not concurrency-safe. (rgirish)
|
|
68
84
|
- 2026-09-02 — `TA` / 100000 added to the shared `_Model_Client_TransferOrder`. (bala)
|
|
@@ -6,14 +6,15 @@ project: API
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-
|
|
10
|
-
owners: ["bala", "mhammontree", "apeterson", "jcardinal", "tcox"]
|
|
9
|
+
updated: 2026-10-02
|
|
10
|
+
owners: ["bala", "mhammontree", "apeterson", "jcardinal", "tcox", "rgirish"]
|
|
11
11
|
files:
|
|
12
12
|
- api2/.ebextensions/git.php
|
|
13
13
|
- api2/.platform/hooks/prebuild/git.sh
|
|
14
14
|
- api2/.ebextensions/php_include_underscore.config
|
|
15
15
|
- api2/.ebextensions/git.sandbox-dev.json
|
|
16
16
|
- api2/.ebextensions/git.sandbox-client.json
|
|
17
|
+
- api2/.ebextensions/git.beta.json
|
|
17
18
|
- api2/Config/beta.ini
|
|
18
19
|
- api2/Config/sandbox-dev.ini
|
|
19
20
|
- api2/Component/Api/V2/V2.php
|
|
@@ -64,17 +65,18 @@ Checking only #2 (the one you can `grep`) is the trap: it can be perfectly corre
|
|
|
64
65
|
|
|
65
66
|
Never infer the environment from: **the EB environment name** (`api-sandbox-dev` is `beta`); **the hostname / URL** (follows the EB name, not `ENVIRONMENT`); **which database the data landed in** (`Config/beta.ini` and `Config/sandbox-dev.ini` **both point at the same cluster, `dev.sandbox.database.togahub.com`**, so querying "the dev-sandbox database" and finding your row proves nothing about which tier wrote it).
|
|
66
67
|
|
|
67
|
-
**`api.beta.togahub.com` — the concrete mapping, and the leftover files that mislead.**
|
|
68
|
-
- `api.beta.togahub.com`
|
|
69
|
-
-
|
|
70
|
-
-
|
|
71
|
-
-
|
|
68
|
+
**`api.beta.togahub.com` — the concrete mapping, and the leftover files that mislead.** Re-confirmed 2026-10-02:
|
|
69
|
+
- `api.beta.togahub.com` **and** `api.dev.sandbox.togahub.com` both hit EB env **`api-sandbox-dev`** (siblings `webhook-sandbox-dev`, `worker-sandbox-dev`), EB var **`ENVIRONMENT=beta`** → `Config/beta.ini`.
|
|
70
|
+
- With no `git.beta.json` the branch falls through to `_beta`. The CodePipeline `API-Sandbox-Dev` source/trigger is `_sandbox-dev`, but its `_underscore` artifact is **not** what gets deployed — the prebuild clone wins.
|
|
71
|
+
- **Fix (2026-10-02): `.ebextensions/git.beta.json` pinning `branch: _sandbox-dev`.** ⚠ As of 2026-10-02 it exists only in a local GitHub Desktop stash in api2 — **not committed to any branch**. Until it is committed to api2 `_beta` and redeployed, beta still clones `_underscore` `_beta`. Like every `git.*.json` it holds a credential — never copy the value.
|
|
72
|
+
- `Config/sandbox-dev.ini` + `git.sandbox-dev.json` exist but do **not** serve this host.
|
|
73
|
+
- **Switching `ENVIRONMENT` to `sandbox-dev` was considered and deferred** (goal: retire `beta` later). Blockers: `Config/sandbox-dev.ini` points FedEx at **live** `apis.fedex.com` and lacks PayPal sandbox / `startechPcmaticb2b`; tokens are env-scoped (`EN-5`); dev-sandbox Core has 142 `Domains` + 3 `ClientAuthentications` on environment `beta` (id 4) and none on `sandbox-dev` (id 23). Prudential's `beta` `Core.DatabaseHosts` row points at the sandbox-dev cluster.
|
|
72
74
|
|
|
73
75
|
**⚠ Diagnostic: `Call to undefined method _Model_Client_<X>::<method>()` means the CLASS ISN'T DEPLOYED.** When a client model override is missing from the deployed `_underscore`, V2 does **not** error at resolution time. `class_exists` fails, so V2 quietly **declines the override and falls back to the BASE model** (`V2.php:3155-3162`). The base model has no client-specific method, so you get e.g. `Call to undefined method _Model_Client_Subscription::cancelApi()`. **Read this error as "the override class is not on the server," not "the code is wrong."** The `_Model_Client_*` (base) prefix — where you expected `_Model_<Slug>_*` — is the tell. Confirm by file presence, then check which branch that tier clones. Related: **the slug V2 uses to pick the override class is `Clients.clientIdentifier`** (`V2.php:1521`), taken from the JWT. `Core.Clients` has **no `slug` column** — do not go looking for one.
|
|
74
76
|
|
|
75
77
|
**⚠ Moving a tier onto its correct branch is a SCHEMA event, not just a code event.** When a deploy moves an environment **forward** onto a newer `_underscore` branch — including a deploy that *fixes* a wrong branch pin — it fast-forwards across every model change made since the pin, and `_Model::initialize()` SELECTs **every declared property**. Any of those columns without a merged `dbchanges2` migration becomes an **environment-wide MySQL 1054 → HTTP 500 / `EO-1`**, for every tenant, immediately. **Concrete (2026-08-26):** the deploy moved `sandbox-client` off the `_production` framework (which it had been running for weeks) onto `_sandbox-client`, carrying ~2 weeks of model changes whose migrations were stranded on an unmerged `dbchanges2` branch. Table-meta requests 500'd on `RecordFields.maxLength`, then (once patched) on `TableViewFields.isGroupable`. **Rule: pair a branch-pin correction with a migration sweep** — enumerate what the new branch expects (`git log origin/_production..origin/<env-branch> --oneline -- Model/`) **before** redeploying, or the tier trades a stale-code bug for a total outage. Full procedure: [framework branch ahead of schema](../../dbchanges2/workflows/framework-branch-ahead-of-schema.md).
|
|
76
78
|
|
|
77
|
-
**⚠ Never use a RECENTLY-ADDED `_underscore` symbol from api2 without checking branch containment.** "Land `_underscore` before deploying api2" is necessary but **not sufficient**, because every environment pins its **own** `_underscore` branch. A framework symbol merged to `_production` may be **absent from `_beta` and `_sandbox-dev` for weeks**, so app code that references it is fine on prod and **fatal on the sandboxes**. **Concrete (2026-09-01, `TRUE-81056`):** the first version of the duplicate-`transactionId` log fix used `\_Database::ERRNO_DUPLICATE_ENTRY` inside a `catch` block. That constant was added by `_underscore` commit **`6355d0f0`** and, verified with `git show <branch>:Database.php`:
|
|
79
|
+
**⚠ Never use a RECENTLY-ADDED `_underscore` symbol from api2 without checking branch containment.** (Example below is historical — `ERRNO_DUPLICATE_ENTRY` is on `_production`/`_beta`/`_sandbox-dev` as of 2026-10-02; the rule stands.) "Land `_underscore` before deploying api2" is necessary but **not sufficient**, because every environment pins its **own** `_underscore` branch. A framework symbol merged to `_production` may be **absent from `_beta` and `_sandbox-dev` for weeks**, so app code that references it is fine on prod and **fatal on the sandboxes**. **Concrete (2026-09-01, `TRUE-81056`):** the first version of the duplicate-`transactionId` log fix used `\_Database::ERRNO_DUPLICATE_ENTRY` inside a `catch` block. That constant was added by `_underscore` commit **`6355d0f0`** and, verified with `git show <branch>:Database.php`:
|
|
78
80
|
|
|
79
81
|
| `_underscore` branch | `ERRNO_DUPLICATE_ENTRY` |
|
|
80
82
|
|---|---|
|
|
@@ -107,9 +109,11 @@ So on `api.beta.togahub.com` and `api.dev.sandbox.togahub.com` the code fataled
|
|
|
107
109
|
- **⚠ "grep says this framework function doesn't exist" is meaningless without naming the checked-out branch.** Because `_underscore` is cloned at build from a **moving per-environment branch**, a local checkout is routinely behind the branch a tier runs. Concrete (2026-08-24): a session concluded — with **three** corroborating pieces of code evidence — that `_Model_Core_Surface::_resolveLanguageId()` had never been implemented and the KB was wrong. It was not; the local tree was on a branch predating it. The method is present on `_sandbox-client`, `_sandbox-dev`, `surface-layer` and `origin/_beta` (merged into `_sandbox-client` 2026-07-22, `6809c6b9`). **Always run `git branch --show-current` (and check sibling branches) before declaring framework code absent** — corroborating reads from the same stale tree multiply the error.
|
|
108
110
|
- **Deploy path is build-time, so a git push alone does not update running instances.** The `_underscore` clone happens in the prebuild hook at build time: commit the fix on the feature branch, merge into the deploy-channel branch (`_` + `ENVIRONMENT`, e.g. `_sandbox-client`), push, **then redeploy the EB environment** to re-run the clone.
|
|
109
111
|
- **⚠ SECURITY — every `.ebextensions/git.*.json` carries a plaintext GitHub PAT** committed in the repo (used to clone `_underscore`). Pre-existing leak; treat the committed tokens as compromised, rotate, move to EB environment properties / SSM. Do **not** copy the token value anywhere. **Escalated 2026-08-24 — still not rotated, and exposure has WIDENED.** The token is present in **all ~20 per-environment `git.*.json` files**, and an attempted in-place redaction *increased* exposure: BSD `sed` on macOS does not support `\|` alternation, so the command failed in a way that **printed the token value into the session transcript**. Two durable takeaways: (a) rotation is overdue — treat it as an open action item; (b) **do not attempt ad-hoc redaction of a secret with a one-liner** — a failed redaction commonly echoes the value it was meant to hide (GNU-vs-BSD `sed` regex differences make macOS one-liners especially prone). Rotate at the source instead.
|
|
112
|
+
- **⚠ sandbox-dev DB `max_connections = 90`** (prod client cluster: 3000). Each api2 request opens ~6 connections (+1 for the autoincrement counter), so ~15+ simultaneous requests on beta / dev-sandbox fail `Too many connections` → HTTP 500 `EO-1`. Pre-existing (4322 such errors logged). Keep load tests at ~10 concurrent; a burst 500 there is not a code bug.
|
|
110
113
|
- **Duplicate integration code is the trap this creates.** When the same capability is needed in both tiers it must exist twice (once per framework) — keep the credential/config location documented in both places so a rotation updates both.
|
|
111
114
|
|
|
112
115
|
## Change history
|
|
116
|
+
- 2026-10-02 — beta/dev-sandbox host → `api-sandbox-dev` mapping, `git.beta.json` pin to `_sandbox-dev` (uncommitted), deferred ENVIRONMENT rename and its blockers, sandbox-dev `max_connections` limit. (rgirish)
|
|
113
117
|
- 2026-09-25 — renamed product reference from "TogaIQ" to "Talos" (rebrand; TOGa IQ is now the separate `toga25-iq` app) (jcardinal)
|
|
114
118
|
|
|
115
119
|
## Related
|
|
@@ -6,7 +6,7 @@ project: API
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-
|
|
9
|
+
updated: 2026-10-02
|
|
10
10
|
owners: [jcardinal, bala, ajean, rgirish]
|
|
11
11
|
files:
|
|
12
12
|
- api2/Controller/Index.php
|
|
@@ -34,9 +34,9 @@ api2's front controller ("**Prong A**") can **detect a retriable concurrency fai
|
|
|
34
34
|
|
|
35
35
|
The replay is **route-scoped by an allowlist** (`DEADLOCK_RETRIABLE_ROUTES` — currently `['/v2/advance-shipping-notices', '/v2/service-requests']`) — because whole-request in-process replay is **only safe for routes with no non-rollbackable external side effect**. The retry gate is now **`wasDeadlock() || wasRetriableUniqueRace()`**.
|
|
36
36
|
|
|
37
|
-
> **Status (2026-
|
|
37
|
+
> **Status (2026-10-02).** The SR number race is now fixed at the source — a per-tenant `Parameters` counter in `_Model` (TRUE-82209, on `_underscore` `_beta`/`_sandbox-dev`), see [FIELD_AUTOINCREMENT record numbering](../../_underscore/features/autoincrement-record-numbering.md). The `/v2/service-requests` replay below (api2 `_sandbox-dev` only; `_beta`/`_production` list ASN only) is no longer the fix for numbers. **Extending replay to every numbered route was rejected**: Adyen / Elite / Growrk `postPost` hooks have external side effects. Prod still has neither fix.
|
|
38
38
|
|
|
39
|
-
**Constants (verified 2026-
|
|
39
|
+
**Constants (verified 2026-10-02):** `MAX_DEADLOCK_ATTEMPTS = 5`, backoff 20ms-120ms widening per attempt. `DEADLOCK_RETRIABLE_ROUTES` = `['/v2/advance-shipping-notices', '/v2/service-requests']` on api2 `_sandbox-dev`; ASN only on `_beta` and `_production`.
|
|
40
40
|
|
|
41
41
|
## How it works
|
|
42
42
|
|
|
@@ -45,7 +45,7 @@ The replay is **route-scoped by an allowlist** (`DEADLOCK_RETRIABLE_ROUTES` —
|
|
|
45
45
|
- **The ASN model** calls the IF-create `internalApiRequest` with **`throwExceptionsOnError:false`**; on an empty response with `lastErrorNumber === 1062` it sets `$retriableUniqueRace = true` and returns null. `postPost` then **aborts immediately** (issues no further query).
|
|
46
46
|
- **The controller's Prong A gate** becomes `wasDeadlock() || wasRetriableUniqueRace()`. Replaying the whole request on a fresh REPEATABLE-READ snapshot lets the retry's GET see the winner's committed IF and reuse it.
|
|
47
47
|
|
|
48
|
-
Bounded by `MAX_DEADLOCK_ATTEMPTS = 5`, gated to `DEADLOCK_RETRIABLE_ROUTES
|
|
48
|
+
Bounded by `MAX_DEADLOCK_ATTEMPTS = 5`, gated to `DEADLOCK_RETRIABLE_ROUTES`.
|
|
49
49
|
|
|
50
50
|
**V2 concurrency mechanics (durable framework knowledge)** — the V2-engine facts that decide **how any interceptor-level write race must be handled**, and why the fix is a whole-request replay and not an in-place recovery:
|
|
51
51
|
- **Response poisoning (load-bearing).** A failed nested save via `internalApiRequest` calls `addDefinedMessage(EV-10)` on the **shared** `$api->response`, setting a non-2xx status **before** control returns. So even if an interceptor catches the throw and "recovers," `execute()` recomputes `isSuccess = false` (`V2.php` ~2154) and **self-rolls-back DB_CORE + DB_CLIENT** (~2157-2159). **An interceptor therefore CANNOT recover a failed internal save in place** — the only correct recovery is a whole-request replay. This same poisoning drives the Prong A retry gate and, on exhaustion, produces the coherent EV-10 response.
|
|
@@ -82,8 +82,8 @@ Two consequences of the typed exception worth knowing:
|
|
|
82
82
|
**Symptom:** `/v2/service-requests` returned `400 EV-10` *"Duplicate entry 'SR131023' for key 'ServiceRequests.number'"* — 21 consecutive failures, all ~1.8s with no backoff visible. ServiceNow was **not** sending duplicates. The retry existed and was correctly wired; it just never fired.
|
|
83
83
|
|
|
84
84
|
**The root-cause chain, in order:**
|
|
85
|
-
1. **`ServiceRequests.number` is not a MySQL auto-increment.** `_Model` `FIELD_AUTOINCREMENT`
|
|
86
|
-
2. **The retry was already built for exactly this.** `_underscore/Model.php`
|
|
85
|
+
1. **`ServiceRequests.number` is not a MySQL auto-increment.** The old `_Model` `FIELD_AUTOINCREMENT` did `SELECT MAX(number)+1` then an `INSERT`, **with no lock between them** (since replaced by a counter on `_beta`/`_sandbox-dev`) (see [FIELD_AUTOINCREMENT record numbering](../../_underscore/features/autoincrement-record-numbering.md)). Two concurrent POSTs read the same MAX, pick the same number, and the loser gets MySQL 1062.
|
|
86
|
+
2. **The retry was already built for exactly this.** `_underscore/Model.php` set `_Database::$retriableUniqueRace` on that 1062 (removed by TRUE-82209 — no branch's `Model.php` sets it now), and `api2/Controller/Index.php` replays the whole request when `_Database::wasRetriableUniqueRace()` is true. That helper requires **both** the flag **and** `$lastErrorNumber === 1062`.
|
|
87
87
|
3. **The logger destroyed the second half of that condition.** `_Component_Api_V2::saveApiLogUnderItsOwnTransactionId()` runs **between the failed INSERT and the retry check**, inside `execute()`. Its resend loop **deliberately causes its own 1062s** — it renames `transactionId` on collision and retries (see [V2 request logging](./request-logging.md)). Those bookkeeping 1062s **overwrote the caller's `$lastErrorNumber`**, so `wasRetriableUniqueRace()` returned false and the replay never ran.
|
|
88
88
|
|
|
89
89
|
**Fix:** save and restore **both** `_Database::$lastErrorNumber` and `_Database::$retriableUniqueRace` in a `try/finally` **inside** `saveApiLogUnderItsOwnTransactionId()`. Scoped inside that method — *not* at the call site — so it only undoes that loop's own errors; deadlocks, the ASN route, and every other route behave exactly as before.
|
|
@@ -129,7 +129,6 @@ Two details that cost time:
|
|
|
129
129
|
Both were considered and left alone on purpose. Do not "fix" either without re-checking the reasoning.
|
|
130
130
|
|
|
131
131
|
- **The `V2.php` resend loop is unbounded** (`for ($resendNumber = 1; ; ...)`). Safe **today only because every pass changes a UNIQUE column**, so it cannot re-ask for a taken value.
|
|
132
|
-
- **`_underscore/Model.php` marks ANY 1062 as a retriable race** without checking *which* unique key failed. `ServiceRequests` has three unique keys (`PRIMARY`, `uuid`, `number`) and only `number` is the benign race. Harmless today: `uuid` is a random uuid so a collision is effectively impossible, and `PRIMARY` cannot duplicate on INSERT.
|
|
133
132
|
|
|
134
133
|
### Rejected finding — a null `transactionId` does NOT break the first log save
|
|
135
134
|
|
|
@@ -143,9 +142,8 @@ Checked and **wrong**; recorded so it is not raised again. `Logs.Api.transaction
|
|
|
143
142
|
- **Anything running between the failing query and the retry check must save/restore `_Database::$lastErrorNumber` + `$retriableUniqueRace`.** The API logger's own 1062s disarmed this retry completely for weeks; a 1062 is not protected from overwrite the way a deadlock is.
|
|
144
143
|
- **Test concurrency with a gate file**, not a background loop — a loop spreads the requests out and can miss the race. Auth token is at `.data.tokens.access`.
|
|
145
144
|
- **A branch cut from `_production` may have the CALL SITE but not the METHOD.** `saveApiLogUnderItsOwnTransactionId()` (commit `db94a26`) exists only on `_sandbox-dev`; branch `TRUE-82095` came off `_production` and would have fataled with *call to undefined method* on every DB-logged request. When branching off `_production`, verify the **callee** exists, not just the caller.
|
|
146
|
-
- `ERRNO_DUPLICATE_ENTRY` is
|
|
147
|
-
-
|
|
148
|
-
- **Every other `FIELD_AUTOINCREMENT` model has the same unlocked generator** — SalesOrder, PurchaseOrder, Ticket, ItemReceipt, InventoryAdjustment, TransferOrder — and **none is on `DEADLOCK_RETRIABLE_ROUTES`**, so a duplicate-number clash on those routes still returns 400. The durable answer is fixing the number generator so the read and the insert cannot split; that is a bigger, separate change.
|
|
145
|
+
- `ERRNO_DUPLICATE_ENTRY` is now on `_production`, `_beta`, `_sandbox-dev` (verified 2026-10-02); older branches still need the local-constant guard.
|
|
146
|
+
- Do not add a route to `DEADLOCK_RETRIABLE_ROUTES` without proving it is replay-safe (no irreversible side effect, no mutated superglobal). Number races are fixed in the generator, not here.
|
|
149
147
|
- **The `SR…` number inside a `Duplicate entry` error belongs to the WINNER, not the failed request.** The loser never got a number. A "duplicate SR131023" report means one request was lost, not that two rows share a number.
|
|
150
148
|
- **ServiceNow resends a failed SR on its own** (proven: `REQ4510546` failed, came back ~6h later as `SR131026`). After the prod deploy adds an internal retry, an outside resend could produce a **second SR for the same REQ** — confirm with the client first.
|
|
151
149
|
- Do not reach for this protocol for an HTTP 429 — use `_ApiRequest::setAutoRetry()`.
|
|
@@ -160,6 +158,7 @@ Checked and **wrong**; recorded so it is not raised again. `Logs.Api.transaction
|
|
|
160
158
|
- [ENVIRONMENT decides the `_underscore` branch](./environment-variable-drives-underscore-branch.md)
|
|
161
159
|
|
|
162
160
|
## Change history
|
|
161
|
+
- 2026-10-02 — SR number race now fixed in `_Model` (Parameters counter, TRUE-82209); replay-for-all-numbered-routes rejected; corrected per-branch route list and `ERRNO_DUPLICATE_ENTRY` availability. (rgirish)
|
|
163
162
|
- 2026-09-22 — Added production confirmation of the race (3 same-second ServiceNow POSTs, `Logs_Prudential.Api` 8736482/8736483/8736484, 1062 on `SR131023`) and the open double-SR risk from ServiceNow's own resend. Investigation only, no code change. (rgirish)
|
|
164
163
|
- 2026-09-22 — Found and fixed the reason this retry never fired on `/v2/service-requests`: the API logger's own 1062s overwrote `_Database::$lastErrorNumber`; now saved/restored in `try/finally` inside `saveApiLogUnderItsOwnTransactionId()`. Proven 10/10 under real concurrency on sandbox-dev. Also fixes ASN. Prod still unpatched. (rgirish)
|
|
165
164
|
- 2026-09-22 — Corrected constants: `MAX_DEADLOCK_ATTEMPTS = 5` (was documented as 3); `DEADLOCK_RETRIABLE_ROUTES` now includes `/v2/service-requests`. (rgirish)
|
|
@@ -17,7 +17,7 @@ project: _Underscore
|
|
|
17
17
|
client: prudential
|
|
18
18
|
type: profile
|
|
19
19
|
status: active
|
|
20
|
-
updated: 2026-
|
|
20
|
+
updated: 2026-10-02
|
|
21
21
|
owners: ["jcardinal", "rgirish", "bala", "mhammontree"]
|
|
22
22
|
files: []
|
|
23
23
|
related:
|
|
@@ -29,9 +29,7 @@ related:
|
|
|
29
29
|
- features/dell-lch-iop-transmissions.md
|
|
30
30
|
- features/device-information-import-and-contact-linking.md
|
|
31
31
|
- workflows/http2-alb-workaround.md
|
|
32
|
-
- features/
|
|
33
|
-
where each code is logged, and how to tell "we refused it" from "they never sent it".
|
|
34
|
-
- features/supply-orders-csv-export.md
|
|
32
|
+
- features/supply-orders-csv-export.md
|
|
35
33
|
- features/order-email-pdf-attachments.md
|
|
36
34
|
- features/itemclasses-device-tier-data.md
|
|
37
35
|
- workflows/store-saml-deploy-topology.md
|
|
@@ -122,6 +120,12 @@ The order-created cron branches on this id; the shipped/delivered crons branch o
|
|
|
122
120
|
The 15-minute `prudential_auth_rejection_alert.php` cron now alerts on them — see
|
|
123
121
|
features/service-request-auth-rejection-alert.md.
|
|
124
122
|
|
|
123
|
+
- **SR number race (ServiceNow bursts → 1062 `Duplicate entry 'SR…'` → 400 `EV-10`, SR lost).** Fixed in
|
|
124
|
+
the framework by a `Parameters` counter (`NEXT_NUMBER_SERVICE_REQUESTS_SR`); the `usleep(0.5–2s)` stopgap in
|
|
125
|
+
`_Model_Prudential_ServiceRequest::prePost()` is removed. Both on `_underscore` `_beta`/`_sandbox-dev` only
|
|
126
|
+
(2026-10-02) — prod still has the race and the stopgap. Load-test with a distinct contact + serial per call and
|
|
127
|
+
`customer.uuid` set. See [FIELD_AUTOINCREMENT record numbering](../../2.0/apps/_underscore/features/autoincrement-record-numbering.md).
|
|
128
|
+
|
|
125
129
|
## Monitoring
|
|
126
130
|
- `Monitor/Prudential/{SalesOrderGenerationBacklog,DellPurchaseOrderTransmissionBacklog,OrderShippedUpdateBacklog}`
|
|
127
131
|
(worker2 → OneUptime) watch the order pipeline for stalls. Thresholds are DB-tunable and the
|
package/package.json
CHANGED