toga-ai 1.0.730 → 1.0.732
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/1.0/apps/library/INDEX.md +1 -1
- package/knowledge/1.0/apps/library/features/email-test-mode-and-write-free-sends.md +43 -4
- package/knowledge/1.0/apps/library/features/netsuite-suiteql-rest-shim.md +22 -1
- package/knowledge/1.0/apps/worker/INDEX.md +2 -2
- package/knowledge/1.0/apps/worker/features/compass-ma-sales-order-exception-report.md +20 -3
- package/knowledge/1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md +41 -0
- package/knowledge/1.0/apps/worker/workflows/tracing-a-worker-cron-run-in-production.md +33 -1
- package/knowledge/2.0/apps/api2/features/environment-variable-drives-underscore-branch.md +52 -2
- package/knowledge/2.0/apps/api2/features/request-logging.md +133 -1
- package/knowledge/2.0/apps/api2/features/v2-deadlock-retry.md +27 -2
- package/knowledge/clients/elite/features/netsuite-togasupply-sync.md +13 -2
- package/package.json +1 -1
|
@@ -11,7 +11,7 @@
|
|
|
11
11
|
| [Elite Freshservice Sync (library)](features/elite-freshservice-sync.md) | `App_Api_Toga2` in `library/app/api/toga2.php` orchestrates bidirectional sync between TOGA 2 and TOGaDesk. | library/app/api/toga2.php |
|
|
12
12
|
| [App_Email Queued Sending & Attachments (Common.EmailsQueued)](features/email-queue-attachments.md) | `App_Email::send()` can either send **inline** (PHPMailer talks to SES right there) or **queue** the message: `base64(serialize($this))` is inserted into `Commo | library/app/email.php, library/phpmailer/class.phpmailer.php, worker/crons/notifications/infrastructure/send_emails.php, worker/crons/notifications/covid/send_covid_pending_vaccination_approval.php, togadesk/desk/includes/functions.php |
|
|
13
13
|
| [Branded HTML Email Templates (App_Email_Template)](features/email-templates.md) | `App_Email_Template` (`app/email/template.php`) is the base class for branded HTML emails in the 1.0 (`App_`) framework. | library/app/email/template.php, library/app/email/agilant.php |
|
|
14
|
-
| [App_Email Side Effects & Test Mode (how to send a real email that writes nothing)](features/email-test-mode-and-write-free-sends.md) | `App_Email::send()` is **not** side-effect free. | library/app/email.php, library/app/framework.php |
|
|
14
|
+
| [App_Email Side Effects & Test Mode (how to send a real email that writes nothing)](features/email-test-mode-and-write-free-sends.md) | `App_Email::send()` is **not** side-effect free. | library/app/email.php, library/app/model/logs/email.php, library/app/framework.php |
|
|
15
15
|
| [Error Capture in 1.0 (App_Error_Capture → shared 2.0 Logs DB)](features/error-capture-1-0.md) | The 1.0 side of the platform error-reporting pipeline (TRUE-78188). | library/app/error/capture.php, library/app/error.php, library/app/exception/business.php, library/app/api/toga2.php, library/app/cloud.php, worker/config.worker.ini, worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php |
|
|
16
16
|
| [HTTP 500 Error Monitor (App_SystemMonitor_500Error) — and why its \"Error Type\" is not a diagnosis](features/http-500-error-monitor.md) | `App_SystemMonitor_500Error` (`library/app/systemmonitor/500error.php`, title **"HTTP 500 Error Alert"**) is the 1.0 system monitor that watches **`Logs.Api` fo | library/app/systemmonitor/500error.php, worker/crons/infrastructure/system_monitors.php, api2/Controller/Index.php, _underscore/Error.php |
|
|
17
17
|
| [1.0 MVC Page Pattern & New-App Skeleton](features/mvc-page-pattern-and-app-skeleton.md) | This is the **reusable recipe for standing up a new 1.0 (`App_`) application** and for adding pages to one — the folder-based MVC routing, the page lifecycle, t | library/app/framework.php, library/app/frameworkindex.php, library/app/mvc.php, library/app/database.php, library/app/model.php, library/app/config.php |
|
|
@@ -6,10 +6,11 @@ project: Library
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-
|
|
9
|
+
updated: 2026-09-01
|
|
10
10
|
owners: ["bala"]
|
|
11
11
|
files:
|
|
12
12
|
- library/app/email.php
|
|
13
|
+
- library/app/model/logs/email.php
|
|
13
14
|
- library/app/framework.php
|
|
14
15
|
related:
|
|
15
16
|
- ./email-queue-attachments.md
|
|
@@ -32,7 +33,7 @@ Verified against `library/app/email.php` on 2026-08-31.
|
|
|
32
33
|
|
|
33
34
|
| Write | When | How to suppress |
|
|
34
35
|
|---|---|---|
|
|
35
|
-
| `Logs.
|
|
36
|
+
| `Logs.Emails` row (`App_Model_Logs_Email`) | every **inline** send, success or failure | `setIsLogEnabled(false)` |
|
|
36
37
|
| `Common.EmailsQueued` row (`App_Model_Common_EmailsQueued`) | `queueForSending` truthy, **or** outbound email is disabled and `sendEmailEvenIfOutboundEmailIsDisabled` is false | `setQueueForSending(false)` + `sendEmailEvenIfOutboundEmailIsDisabled = true` |
|
|
37
38
|
|
|
38
39
|
The queue-vs-send decision is a single condition (`email.php` ~L313):
|
|
@@ -46,6 +47,40 @@ if ($this->queueForSending || (!$this->sendEmailEvenIfOutboundEmailIsDisabled &&
|
|
|
46
47
|
a queue insert** with no code change on your side. The log write is inside the `else` branch, so a
|
|
47
48
|
queued message writes the queue row and **no** log row.
|
|
48
49
|
|
|
50
|
+
## Where the log lives — and how to verify a send actually happened
|
|
51
|
+
|
|
52
|
+
The log row is the **only** durable proof that a 1.0 script really sent a mail, so knowing where
|
|
53
|
+
it lands matters. `App_Model_Logs_Email` (`library/app/model/logs/email.php`) declares
|
|
54
|
+
`TABLENAME = 'Emails'`, `DATABASE = 'db_logs'` and `$_databaseNameOverride = 'Logs'`. Read that
|
|
55
|
+
literally:
|
|
56
|
+
|
|
57
|
+
- **Table is `Emails` (plural), schema `Logs`**, and it lives in the **legacy (v1) environment** —
|
|
58
|
+
the one `db_logs` points at for the 1.0 tier.
|
|
59
|
+
- It is **not** in the prod `Logs` schema, and **not** in a per-client `Logs_<Client>` schema.
|
|
60
|
+
Prod `Logs_Compass` has an `Email` (singular) table used by **2.0**, and it stays empty for
|
|
61
|
+
these 1.0 cron emails. Looking there is the usual wrong turn.
|
|
62
|
+
|
|
63
|
+
Columns worth reading: `dtSent`, `isSuccess`, `error`, `sourceJob`, `instanceId`, `subject`,
|
|
64
|
+
`from`, `to`, `cc`, `bcc`, `body`.
|
|
65
|
+
|
|
66
|
+
```sql
|
|
67
|
+
# legacy env, schema Logs
|
|
68
|
+
SELECT id, dtSent, isSuccess, error, sourceJob, subject, `to`
|
|
69
|
+
FROM Emails
|
|
70
|
+
WHERE subject LIKE '<subject prefix>%'
|
|
71
|
+
ORDER BY id DESC;
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
Two reading rules:
|
|
75
|
+
|
|
76
|
+
- **A row exists only if the code reached `->send()` and took the inline branch.** No row means
|
|
77
|
+
either the job never got that far (e.g. it had nothing to report), or the message was queued
|
|
78
|
+
instead (see the table above), or logging was disabled. It does **not** by itself mean a
|
|
79
|
+
failure.
|
|
80
|
+
- **`sourceJob` is `$_SERVER['SCRIPT_NAME']`, which tells you how it ran.** A full
|
|
81
|
+
`/var/www/html/crons/...` path is a scheduled run on the server; a bare filename is a manual
|
|
82
|
+
CLI run someone did by hand.
|
|
83
|
+
|
|
49
84
|
## The three switches for a genuinely write-free send
|
|
50
85
|
|
|
51
86
|
All three are required — any one of them left at its default reintroduces a write or swallows the
|
|
@@ -54,7 +89,7 @@ send:
|
|
|
54
89
|
1. `setQueueForSending(false)` — do not insert into `Common.EmailsQueued`.
|
|
55
90
|
2. `sendEmailEvenIfOutboundEmailIsDisabled = true` — do not fall into the queue branch just
|
|
56
91
|
because the `Defaults` flag is off.
|
|
57
|
-
3. `setIsLogEnabled(false)` — do not insert into `Logs.
|
|
92
|
+
3. `setIsLogEnabled(false)` — do not insert into `Logs.Emails`.
|
|
58
93
|
|
|
59
94
|
## `App_Email` only ever *narrows* the recipient list
|
|
60
95
|
|
|
@@ -114,12 +149,16 @@ php -d include_path=".;<library>;<_underscore>" <path/to/script>.php
|
|
|
114
149
|
queue path (see `email-queue-attachments.md`) and the recipient list. Two different behaviors
|
|
115
150
|
you cannot exercise locally.
|
|
116
151
|
- **Deleting the log write is not the same as being read-only.** `setIsLogEnabled(false)` only
|
|
117
|
-
suppresses `Logs.
|
|
152
|
+
suppresses `Logs.Emails`; the queue insert has its own switch and fires *instead of* sending.
|
|
118
153
|
- **Dry-run harness scripts are throwaway.** Keep them out of the repo — they carry hardcoded
|
|
119
154
|
recipient lists and bypass the real gating. Build one, use it, delete it; document the
|
|
120
155
|
*technique* here rather than committing the file.
|
|
121
156
|
|
|
122
157
|
## Change history
|
|
158
|
+
- 2026-09-01 — Corrected the log target: the table is **`Emails`** (plural) in schema **`Logs`**
|
|
159
|
+
on the **legacy** environment, not `Logs.Email` and not a per-client `Logs_<Client>` schema
|
|
160
|
+
(prod `Logs_Compass.Email` is the 2.0 table and stays empty for 1.0 cron mail). Added the
|
|
161
|
+
verification query and the `sourceJob` scheduled-vs-manual tell. (bala)
|
|
123
162
|
- 2026-08-31 — Documented that `App_Email::send()` writes `Logs.Email` unless
|
|
124
163
|
`setIsLogEnabled(false)`, that the queue branch also fires when `OUTBOUND_EMAIL_ENABLED` is off,
|
|
125
164
|
that `cronInitialization()` INSERTs `CronJobExecutions`, and that `send()` only ever narrows
|
|
@@ -6,7 +6,7 @@ project: Library
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-
|
|
9
|
+
updated: 2026-09-01
|
|
10
10
|
owners: [dfranks, jcardinal, kyalamarthi]
|
|
11
11
|
files:
|
|
12
12
|
- library/app/api/netsuite/rest.php
|
|
@@ -169,6 +169,19 @@ Forecast2 tables already store.
|
|
|
169
169
|
cross-checking item id, then number each line by the NetSuite `line` (fall back to `linesequencenumber`
|
|
170
170
|
if the item ids disagree). Self-heals on the next receipt sync (it re-imports the PO). Same "join a
|
|
171
171
|
NetSuite line on `line`, not `linesequencenumber`" lesson as the `tl.id` semantics above.
|
|
172
|
+
- **⚠ `listPurchaseOrders()` scoped POs by the VENDOR header `entity`, silently dropping every
|
|
173
|
+
drop-ship PO for all clients (fixed 2026-09-01).** It reused the shared `buildEntityInFilter()`
|
|
174
|
+
helper — correct for `listSalesOrders`/`listSales` (SO header `entity` = customer) but **wrong for
|
|
175
|
+
POs**, whose header `entity` is the **vendor**. The `AND entity IN (<customer ids>)` filter matched
|
|
176
|
+
ZERO customer POs, so the header query returned nothing and the section imported nothing — no error.
|
|
177
|
+
Replaced with a new `buildPurchaseOrderCustomerFilter()` that scopes by the customer on the LINES or
|
|
178
|
+
the end-customer body field:
|
|
179
|
+
`AND ( id IN (SELECT tl.transaction FROM transactionline tl WHERE tl.entity IN ($ids) AND tl.mainline = 'F') OR CUSTBODY_END_CUSTOMER IN ($ids) )`.
|
|
180
|
+
`buildEntityInFilter()` is unchanged. This is a **fetch-scope** bug (record never returned by the
|
|
181
|
+
`list*`), a class distinct from the per-id `fetchPurchaseOrderById` shim-shape gaps above — but the
|
|
182
|
+
same underlying trap of reusing a sales-order-shaped assumption on a purchase order. Full
|
|
183
|
+
engine-side impact + the tier-wide per-client backfill live on the
|
|
184
|
+
[per-client-sync doc](../../worker/features/netsuite-togasupply-per-client-sync.md).
|
|
172
185
|
- **A shim must produce the SOAP object's SHAPE, not just its scalar fields — an absent nested list
|
|
173
186
|
crashes 1.0 (2026-08-29).** `fetchCustomerById()`'s `addressbookList` was where the AIG lookup phase
|
|
174
187
|
crashed: a REST customer with no/odd address made the shim omit the nested `addressbook` structure the
|
|
@@ -262,6 +275,14 @@ None — uniform across clients (NetSuite is a single shared account).
|
|
|
262
275
|
|
|
263
276
|
## Change history
|
|
264
277
|
|
|
278
|
+
- 2026-09-01 — Recorded that **`listPurchaseOrders()` scoped POs by the VENDOR header `entity`** (via
|
|
279
|
+
the reused `buildEntityInFilter()`), silently dropping every drop-ship / special-order PO for all
|
|
280
|
+
togasupply clients since the SOAP→REST migration. A PO header `entity` is the vendor, not the
|
|
281
|
+
customer, so `AND entity IN (<customer ids>)` matched zero rows and the section imported nothing with
|
|
282
|
+
no error. Fixed by a new `buildPurchaseOrderCustomerFilter()` that scopes by the line-level customer
|
|
283
|
+
or `CUSTBODY_END_CUSTOMER` body field; `buildEntityInFilter()` untouched. Fetch-scope bug (distinct
|
|
284
|
+
from the per-id shim-shape gaps). Engine impact + per-client backfill on the per-client-sync doc.
|
|
285
|
+
(jcardinal)
|
|
265
286
|
- 2026-08-30 — Recorded that **`fetchPurchaseOrderById` numbered PO lines by `linesequencenumber` instead
|
|
266
287
|
of the NetSuite `line`** — a receipt for a post-gap line (PO with a deleted middle line, real lines
|
|
267
288
|
1,2,4) failed `EV-12` "unable to find a unique match" (GroWrk PO 168223). The PO record line sublist
|
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
| Doc | Summary | Files |
|
|
4
4
|
|-----|---------|-------|
|
|
5
5
|
| [Worker (1.0 Framework) Architecture](architecture.md) | `worker` is the legacy (**1.0** `App_` framework) **background-job tier**. | worker/index.php, worker/_/app/framework.php, worker/crons/, worker/schedules/, worker/ebs/cron.worker.php, worker/.ebextensions/035_cron.worker.config, worker/crons/infrastructure/execute_dbchanges.php, worker/.ebextensions/030_dbchanges.config |
|
|
6
|
-
| [Compass MA Sales Order Exception Report](features/compass-ma-sales-order-exception-report.md) | A worker cron that emails operations the "Compass Refresh Exception Report" — Compass `MA%` sales orders whose corresponding Office Depot (ODP) sales order has | worker/crons/toga2/compass/workflow/7_generate_ma_sales_order_exception_report.php |
|
|
6
|
+
| [Compass MA Sales Order Exception Report](features/compass-ma-sales-order-exception-report.md) | A worker cron that emails operations the "Compass Refresh Exception Report" — Compass `MA%` sales orders whose corresponding Office Depot (ODP) sales order has | worker/crons/toga2/compass/workflow/7_generate_ma_sales_order_exception_report.php, worker/schedules/cron.worker.sync.json |
|
|
7
7
|
| [Compass Manager Approval Reminder Emails (1.0 worker crons)](features/compass-manager-approval-reminder-emails.md) | Two 1.0 worker crons nag approvers about sales orders still waiting on a decision — one per Compass tenant. | worker/crons/toga2/compass/compass_email_reminders.php, worker/crons/toga2/compasscanada/compass_email_reminders.php, worker/schedules/cron.worker.sync.json, worker1.5/crons/toga2/compass/compass_email_reminders.php, worker1.5/schedules/cron.worker.json |
|
|
8
8
|
| [Compass Partial In-Transit & Delivered Emails (per package)](features/compass-partial-in-transit-delivered-emails.md) | Compass USA and Compass Canada send a **per-package** in-transit email (and a matching delivered email) instead of one email listing the whole order. | worker/crons/toga2/compass/update_salesorder_status_from_odp.php, worker/crons/toga2/compass/send_delivered_email.php, worker/crons/toga2/compasscanada/update_salesorder_status_from_odp.php, worker/crons/toga2/compasscanada/workflow/3_update_salesorder_status_from_grand_and_toy.php, worker/crons/toga2/compasscanada/send_delivered_email.php, worker/crons/toga2/compasscanada/workflow/test_partial_in_transit_email.php, worker/crons/toga2/compasscanada/workflow/test_partial_delivered_email.php, library/app/client/compasscanada.php |
|
|
9
9
|
| [Elite TOGA 2.0 → TOGaDeskSupport Standalone Attachment Sync](features/elite-togadesk-attachment-sync.md) | `sync_togadesk_elite_attachments.php` is a standalone cron (every 5 minutes) that syncs file attachments from TOGA 2.0 into TOGaDeskSupport for Elite. | worker/crons/toga2/elite/sync_togadesk_elite_attachments.php, worker/crons/toga2/elite/test_sync_togadesk_elite_attachments.php |
|
|
@@ -19,4 +19,4 @@
|
|
|
19
19
|
| [isFulfillable Multi-Client Backfill (all togasupply clients)](workflows/isfulfillable-multi-client-backfill.md) | One-time backfill that catches up `Items.isFulfillable` on **existing** items across **all 17 togasupply clients** (AIG, Broward Sheriff, Canon, Endeavor Health | worker/crons/toga2/netsuite/backfill_isfulfillable_all_clients.php, library/app/api/toga2.php |
|
|
20
20
|
| [Deduplicating NetSuite duplicate Items across client DBs (recursive MERGE)](workflows/netsuite-duplicate-items-dedup.md) | A one-time, per-client data cleanup that collapses **duplicate `Items` rows sharing a `partNumber`** in each NetSuite → TOGa Supply client DB down to a single c | library/app/api/toga2.php, worker/crons/toga2/netsuite/common_sync_togasupply.php |
|
|
21
21
|
| [Onboarding a Client to the NetSuite TOGa Supply Sync](workflows/onboarding-client-to-netsuite-togasupply-sync.md) | How to add a new TOGa 2 client to the per-client NetSuite → TOGa Supply importer (`worker/crons/toga2/netsuite/`). | worker/crons/toga2/netsuite/sync_togasupply.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_elite.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql, dbchanges2/_modules/netsuite/2026-08-05 - CLEAN NETSUITE CLINET.SQL |
|
|
22
|
-
| [Tracing a 1.0 worker cron run in production (no stdout, silent skips)](workflows/tracing-a-worker-cron-run-in-production.md) | How to answer *"did this cron actually run, and what did it do?"* on the 1.0 `worker` tier, where **there is no usable stdout** and **a skipped run leaves no tr | worker/ebs/cron.worker.php, worker/schedules/cron.worker.sync.json, worker/schedules/cron.worker.infrastructure.json, worker/.ebextensions/009_setup_phpini.config, library/app/framework.php |
|
|
22
|
+
| [Tracing a 1.0 worker cron run in production (no stdout, silent skips)](workflows/tracing-a-worker-cron-run-in-production.md) | How to answer *"did this cron actually run, and what did it do?"* on the 1.0 `worker` tier, where **there is no usable stdout** and **a skipped run leaves no tr | worker/ebs/cron.worker.php, worker/schedules/cron.worker.sync.json, worker/schedules/cron.worker.infrastructure.json, worker/.ebextensions/009_setup_phpini.config, library/app/framework.php, library/app/email.php, library/app/model/logs/email.php |
|
|
@@ -6,11 +6,14 @@ project: Worker
|
|
|
6
6
|
client: compass-usa
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-
|
|
9
|
+
updated: 2026-09-01
|
|
10
10
|
owners: ["bala", "mhammontree"]
|
|
11
11
|
files:
|
|
12
12
|
- worker/crons/toga2/compass/workflow/7_generate_ma_sales_order_exception_report.php
|
|
13
|
-
|
|
13
|
+
- worker/schedules/cron.worker.sync.json
|
|
14
|
+
related:
|
|
15
|
+
- ../workflows/tracing-a-worker-cron-run-in-production.md
|
|
16
|
+
- ../../library/features/email-test-mode-and-write-free-sends.md
|
|
14
17
|
---
|
|
15
18
|
|
|
16
19
|
## Summary
|
|
@@ -22,6 +25,9 @@ but haven't. Runs against the Compass client DB (`db_prod_compass` / `Client_Com
|
|
|
22
25
|
## Key files / entry points
|
|
23
26
|
- `worker/crons/toga2/compass/workflow/7_generate_ma_sales_order_exception_report.php` — the
|
|
24
27
|
whole job: one main SELECT, a per-row NetSuite recheck, and a PhpSpreadsheet Excel email.
|
|
28
|
+
- `worker/schedules/cron.worker.sync.json` — the schedule entry
|
|
29
|
+
**"Exception Report - EDI Office Depot (8am - 4pm)"**, `0 8-14 * * *` (hourly, on the hour;
|
|
30
|
+
the crontab runs **Central** time).
|
|
25
31
|
|
|
26
32
|
## How it works
|
|
27
33
|
1. **Main query** walks a two-hop bridge chain to connect the customer order to the vendor order:
|
|
@@ -37,7 +43,9 @@ but haven't. Runs against the Compass client DB (`db_prod_compass` / `Client_Com
|
|
|
37
43
|
(`CUSTOMER_ID__AGILANT` = 3) sales order tied to that ODP PO number. If found → it back-fills
|
|
38
44
|
`c_netsuiteInternalSalesOrderId` on the ODP SO and skips the row (drops it off the report
|
|
39
45
|
permanently). If not found → it writes the row to the Excel.
|
|
40
|
-
4. Emails the workbook to operations only if at least one data row was written
|
|
46
|
+
4. Emails the workbook to operations **only if at least one data row was written** — the send is
|
|
47
|
+
guarded by `if ($rowNumber > 2)`, where row 1 is the header. Subject is
|
|
48
|
+
`Compass Refresh Exception Report : <date>`.
|
|
41
49
|
|
|
42
50
|
## Data model
|
|
43
51
|
- `Client_Compass.SalesOrders` — the ODP sales order is the one keyed on (`id AS salesOrderId`,
|
|
@@ -69,10 +77,19 @@ Compass USA only — this is a Compass-specific integration cron.
|
|
|
69
77
|
- **Don't fake the NetSuite id to hide an order.** Writing a bogus `c_netsuiteInternalSalesOrderId`
|
|
70
78
|
removes it from the report but corrupts the field for anyone who reads it later. Use the stage
|
|
71
79
|
lever instead.
|
|
80
|
+
- **A silent hour is not a failure.** Because of the `$rowNumber > 2` guard, an hour with zero
|
|
81
|
+
exceptions sends nothing at all. Before treating a missing report as a broken cron, check the
|
|
82
|
+
run itself (`Common.CronJobExecutions`) and the send log — the legacy env's `Logs.Emails`
|
|
83
|
+
(schema `Logs`), `WHERE subject LIKE 'Compass Refresh Exception Report%'`. See
|
|
84
|
+
[Tracing a 1.0 worker cron run](../workflows/tracing-a-worker-cron-run-in-production.md).
|
|
72
85
|
- `$inProduction = true` switches the DB link to `db_prod_compass`; set false to test against
|
|
73
86
|
`db_beta_compass`.
|
|
74
87
|
|
|
75
88
|
## Change history
|
|
89
|
+
- 2026-09-01 — Recorded the schedule (`cron.worker.sync.json`, "Exception Report - EDI Office
|
|
90
|
+
Depot (8am - 4pm)", `0 8-14 * * *`) and the `$rowNumber > 2` send guard, so a silent hour reads
|
|
91
|
+
as "zero exceptions", not a fault; added how to prove a send via the legacy `Logs.Emails` table.
|
|
92
|
+
Verified 2026-09-01: recent runs all logged `isSuccess = 1`. (bala)
|
|
76
93
|
- 2026-08-21 — Removed the departing employee `mreich@togatech.com` from this cron's hardcoded
|
|
77
94
|
recipient list (TRUE-80451). Recipients are in-script, not DB-driven. (mhammontree)
|
|
78
95
|
- 2026-06-18 — Added stage-based exclusion so orders closed/canceled on ODP's end drop off the
|
|
@@ -389,6 +389,47 @@ Parameters are stored **per client DB** but accessed **through the TOGa2 API**,
|
|
|
389
389
|
skipped; and because the location filter is shared, receipts belonging to **other** clients at a
|
|
390
390
|
shared warehouse are returned and correctly skipped. Modelling receipts as "location-matched"
|
|
391
391
|
produces wrong diagnostics.
|
|
392
|
+
- **⚠ The PURCHASE_ORDERS section silently dropped EVERY drop-ship / special-order PO on ALL
|
|
393
|
+
togasupply clients — a PO header `entity` is the VENDOR, not the customer (fixed 2026-09-01).**
|
|
394
|
+
`App_Api_Netsuite_Rest::listPurchaseOrders()` (`library/app/api/netsuite/rest.php`) scoped the PO
|
|
395
|
+
HEADER query with `AND entity IN (<client customer ids>)` via the shared `buildEntityInFilter()`
|
|
396
|
+
helper. That helper is correct for **sales** orders (an SO header `entity` IS the customer) but
|
|
397
|
+
**wrong for POs** — a PO header `entity` is the **vendor**. So the customer-id filter matched ZERO
|
|
398
|
+
of a client's POs, the `foreach` never ran, and there was **no interest check, no throw, no log** —
|
|
399
|
+
a pure fetch-scope miss. Proven on Elite PO tranid 128238 (NS internal 5412515, customer 36443,
|
|
400
|
+
vendor TD Synnex 20202): `... AND entity IN (36443)` returned 0 rows; unfiltered returned 1;
|
|
401
|
+
filtering by LINE entity (`transactionline.entity IN (36443)`) returned 1.
|
|
402
|
+
- **Why drop-ship POs vanished ENTIRELY (two-path gap).** Two import paths reach a PO and both
|
|
403
|
+
excluded drop-ship: (a) the dedicated PURCHASE_ORDERS section, via the header-entity filter above;
|
|
404
|
+
(b) the nested `syncItemReceiptFromNetsuite → syncPurchaseOrderFromNetsuite` path (via
|
|
405
|
+
`fetchPurchaseOrderById`, which has **no** entity filter) never fires for a drop-ship PO because a
|
|
406
|
+
drop-ship PO produces item **fulfillments**, not item **receipts**. Stock POs still imported via
|
|
407
|
+
the item-receipt path — which is why Elite had 51 NS POs (all stock, from the 2026-08-12 backfill)
|
|
408
|
+
yet was missing drop-ship ones like 128238. Drop-ship POs carry `customForm.internalId = 148`
|
|
409
|
+
("ASI - Drop Ship Purchase Order"), the end customer on the transaction LINES
|
|
410
|
+
(`transactionline.entity`) and on `shipTo`, so the dedicated PURCHASE_ORDERS section is their
|
|
411
|
+
**only** import path.
|
|
412
|
+
- **Fix (library only, no SQL, no other repos):** added
|
|
413
|
+
`private static function buildPurchaseOrderCustomerFilter(array $entityInternalIds): string` and
|
|
414
|
+
switched `listPurchaseOrders()` to use it. It scopes POs by the customer on the LINES or the
|
|
415
|
+
end-customer body field instead of the vendor header:
|
|
416
|
+
`AND ( id IN (SELECT tl.transaction FROM transactionline tl WHERE tl.entity IN ($ids) AND tl.mainline = 'F') OR CUSTBODY_END_CUSTOMER IN ($ids) )`.
|
|
417
|
+
`buildEntityInFilter()` is left **untouched** (still correct for `listSalesOrders`/`listSales`/
|
|
418
|
+
`listItemFulfillments`, where header entity = customer). Empty-ids behavior preserved (returns
|
|
419
|
+
`''` = no scoping). Deployed; PO 128238 then imported for Elite (dev confirmed). **PO fetch must
|
|
420
|
+
scope by line-level customer / end-customer, never the vendor header entity — the opposite of the
|
|
421
|
+
sales-order shape, and the reason the reused helper silently lost data.** Cross-ref: the sibling
|
|
422
|
+
per-id path's own shim gaps are in the
|
|
423
|
+
[SuiteQL/REST shim doc](../../library/features/netsuite-suiteql-rest-shim.md).
|
|
424
|
+
- **⚠ Tier-wide backfill owed, per-client.** Because `listPurchaseOrders` is shared, EVERY
|
|
425
|
+
togasupply client has been missing its drop-ship POs since the SOAP→REST migration. Backfill each
|
|
426
|
+
client by rewinding `NETSUITE_LAST_SYNC_DATETIME_PURCHASE_ORDERS` (idempotent) — do it
|
|
427
|
+
**deliberately, one client at a time**, not all at once (Elite's first window 2024-04-15..04-18
|
|
428
|
+
alone held 269 POs).
|
|
429
|
+
- **Diagnostic lesson (durable):** a "silent, no-error, not-imported" record in this engine can be a
|
|
430
|
+
**fetch-scope mismatch** (the record is never returned by `list*`), distinct from a per-record
|
|
431
|
+
throw. The `$isInterested == 1` you see when feeding a single PO directly (or via the diagnose
|
|
432
|
+
script) can disagree with the scheduled section, which never fetched it.
|
|
392
433
|
- **`Units.locationId` is never populated by this sync — transaction location lives on the item
|
|
393
434
|
rows.** All 229 `Client_Elite.Units` have `locationId` NULL. `locationId` exists only on `Units`
|
|
394
435
|
and `InventoryAdjustmentItems` (not on the `*ItemUnits` bridge tables), and
|
|
@@ -6,7 +6,7 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: workflow
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-
|
|
9
|
+
updated: 2026-09-01
|
|
10
10
|
owners: ["bala", "kyalamarthi"]
|
|
11
11
|
files:
|
|
12
12
|
- worker/ebs/cron.worker.php
|
|
@@ -14,11 +14,14 @@ files:
|
|
|
14
14
|
- worker/schedules/cron.worker.infrastructure.json
|
|
15
15
|
- worker/.ebextensions/009_setup_phpini.config
|
|
16
16
|
- library/app/framework.php
|
|
17
|
+
- library/app/email.php
|
|
18
|
+
- library/app/model/logs/email.php
|
|
17
19
|
related:
|
|
18
20
|
- ../architecture.md
|
|
19
21
|
- ./diagnosing-frozen-cron-checkins.md
|
|
20
22
|
- ../features/forecast2-supporting-records-import.md
|
|
21
23
|
- ../../library/features/cron-execution-monitoring.md
|
|
24
|
+
- ../../library/features/email-test-mode-and-write-free-sends.md
|
|
22
25
|
- ../../../1.0/standards/backend-php.md
|
|
23
26
|
---
|
|
24
27
|
|
|
@@ -119,6 +122,31 @@ each PO then costing roughly 40 seconds — comfortably past a 5-minute schedule
|
|
|
119
122
|
skips while processing are still listed, so accumulated `SENT/` and `OUTBOX/` objects inflate
|
|
120
123
|
every run.
|
|
121
124
|
|
|
125
|
+
## Step 6 — if the cron's job is to email, check the email log, not the inbox
|
|
126
|
+
"Did the report actually go out?" is answered by the **legacy** environment's `Logs.Emails` table
|
|
127
|
+
(schema `Logs`, table `Emails` — plural). `App_Email::send()` writes one row per **inline** send,
|
|
128
|
+
success or failure. It is **not** in the prod `Logs` schema and **not** in `Logs_<Client>`; a
|
|
129
|
+
prod `Logs_Compass.Email` table exists but is the 2.0 one and stays empty for 1.0 cron mail.
|
|
130
|
+
|
|
131
|
+
```sql
|
|
132
|
+
# legacy env, schema Logs
|
|
133
|
+
SELECT id, dtSent, isSuccess, error, sourceJob, subject, `to`
|
|
134
|
+
FROM Emails
|
|
135
|
+
WHERE subject LIKE '<subject prefix>%'
|
|
136
|
+
ORDER BY id DESC;
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
- `isSuccess = 0` with `error` populated → the send was attempted and failed; that is a real fault.
|
|
140
|
+
- **No row is not automatically a fault.** Most reporting crons only call `->send()` when they have
|
|
141
|
+
something to report, so a silent slot usually means "nothing to report". Confirm the run itself
|
|
142
|
+
happened via Step 4's `CronJobExecutions` row before calling it a failure.
|
|
143
|
+
- `sourceJob` is `$_SERVER['SCRIPT_NAME']`: a full `/var/www/html/crons/...` path means a
|
|
144
|
+
**scheduled** server run, a bare filename means someone ran it **by hand**.
|
|
145
|
+
|
|
146
|
+
Detail on what else `send()` writes (the `Common.EmailsQueued` branch, `setIsLogEnabled(false)`,
|
|
147
|
+
`test_mode` recipient rewriting) is in
|
|
148
|
+
[App_Email Side Effects & Test Mode](../../library/features/email-test-mode-and-write-free-sends.md).
|
|
149
|
+
|
|
122
150
|
## Deploy-time trap: a PHP 8-only syntax kills the whole cron with no visible error
|
|
123
151
|
This tier runs **PHP 7.2**, so a **PHP 8 named argument** (`someFunction(items: $x)`) is a
|
|
124
152
|
**parse error**, not a runtime warning:
|
|
@@ -143,6 +171,10 @@ to **2.0 only**. The 1.0 standard wins in `library/`, `worker/` and every other
|
|
|
143
171
|
signal, it is a latent bug.
|
|
144
172
|
|
|
145
173
|
## Change history
|
|
174
|
+
- 2026-09-01 — Added **Step 6: check the email log, not the inbox** — a 1.0 cron's sends land in
|
|
175
|
+
the **legacy** env's `Logs.Emails` (schema `Logs`, table `Emails`), never prod `Logs` or
|
|
176
|
+
`Logs_<Client>`; gave the verification query, and noted that a missing row usually means the job
|
|
177
|
+
had nothing to report and that `sourceJob` distinguishes a scheduled run from a manual one. (bala)
|
|
146
178
|
- 2026-08-28 — Added **"which of the 7 boxes runs it — and in which timezone"** to Step 1, after a
|
|
147
179
|
production trace of the Forecast2 supporting-records cron (TRUE-81213). A `cron.worker.<role>.json`
|
|
148
180
|
job runs on exactly one dynamically-assigned instance: find it via `Vision_Log.Workers` or the EC2
|
|
@@ -6,7 +6,7 @@ project: API
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-
|
|
9
|
+
updated: 2026-09-01
|
|
10
10
|
owners: ["bala", "mhammontree", "apeterson", "jcardinal", "tcox"]
|
|
11
11
|
files:
|
|
12
12
|
- api2/.ebextensions/git.php
|
|
@@ -22,6 +22,8 @@ related:
|
|
|
22
22
|
- ../workflows/environment-configuration-and-provisioning.md
|
|
23
23
|
- ../workflows/codepipeline-codeconnections-deploy.md
|
|
24
24
|
- ../../_underscore/features/togaiq-gateway-client.md
|
|
25
|
+
- ./request-logging.md
|
|
26
|
+
- ./v2-deadlock-retry.md
|
|
25
27
|
- ../../dbchanges2/workflows/framework-branch-ahead-of-schema.md
|
|
26
28
|
---
|
|
27
29
|
|
|
@@ -176,6 +178,46 @@ patched) on `TableViewFields.isGroupable`.
|
|
|
176
178
|
the tier trades a stale-code bug for a total outage. Full procedure:
|
|
177
179
|
[framework branch ahead of schema](../../dbchanges2/workflows/framework-branch-ahead-of-schema.md).
|
|
178
180
|
|
|
181
|
+
## ⚠ Never use a RECENTLY-ADDED `_underscore` symbol from api2 without checking branch containment
|
|
182
|
+
|
|
183
|
+
The cross-repo ordering gotcha below says "land `_underscore` before deploying api2." That is
|
|
184
|
+
necessary but **not sufficient**, because every environment pins its **own** `_underscore` branch
|
|
185
|
+
(see the whole doc above). A framework symbol merged to `_production` may be **absent from `_beta`
|
|
186
|
+
and `_sandbox-dev` for weeks**, so app code that references it is fine on prod and **fatal on the
|
|
187
|
+
sandboxes**.
|
|
188
|
+
|
|
189
|
+
**Concrete (2026-09-01, `TRUE-81056`).** The first version of the duplicate-`transactionId` log fix
|
|
190
|
+
used `\_Database::ERRNO_DUPLICATE_ENTRY` inside a `catch` block. That constant was added by
|
|
191
|
+
`_underscore` commit **`6355d0f0`** and, verified with `git show <branch>:Database.php`:
|
|
192
|
+
|
|
193
|
+
| `_underscore` branch | `ERRNO_DUPLICATE_ENTRY` |
|
|
194
|
+
|---|---|
|
|
195
|
+
| `_production` | present |
|
|
196
|
+
| `_sandbox-client` | present |
|
|
197
|
+
| `_beta` | **absent** |
|
|
198
|
+
| `_sandbox-dev` | **absent** |
|
|
199
|
+
|
|
200
|
+
So on `api.beta.togahub.com` and `api.dev.sandbox.togahub.com` the code fataled with **`Undefined
|
|
201
|
+
constant _Database::ERRNO_DUPLICATE_ENTRY`** — *thrown from inside the catch block*, which means the
|
|
202
|
+
**real** error was replaced by a confusing one about the error handler. Confirmed live in
|
|
203
|
+
`Logs.Event` **7405** (beta) and **7404** (dev-sandbox).
|
|
204
|
+
|
|
205
|
+
**Hardening applied:** api2 now defines its own `MYSQL_ERRNO_DUPLICATE_ENTRY = 1062` and reads
|
|
206
|
+
`\_Database::$lastErrorNumber` through `property_exists()`, so an older framework **degrades
|
|
207
|
+
gracefully instead of fataling**.
|
|
208
|
+
|
|
209
|
+
**The rule, for any app repo (api2, worker2) calling into `_underscore`:**
|
|
210
|
+
|
|
211
|
+
- Before using a framework constant/method/property you (or a teammate) added recently, run
|
|
212
|
+
**`git branch -a --contains <commit>`** in `_underscore` and confirm it is on the branch of
|
|
213
|
+
**every** tier this code will run on — not just `_production`.
|
|
214
|
+
- When the symbol is not everywhere yet, either **wait for it to land on all tiers**, or **guard the
|
|
215
|
+
reference** (`property_exists()` / `defined()` / a local constant) so an older framework falls back
|
|
216
|
+
rather than fatals.
|
|
217
|
+
- **Never reference a not-yet-universal symbol from inside a `catch`/error path.** That is the worst
|
|
218
|
+
place for it: the fatal masks the very error you were trying to handle, and the incident is then
|
|
219
|
+
diagnosed against the wrong symptom.
|
|
220
|
+
|
|
179
221
|
## Gotchas / known issues
|
|
180
222
|
|
|
181
223
|
- **⚠ Cross-repo change ordering: land `_underscore` BEFORE deploying api2.** Because api2's EB
|
|
@@ -269,7 +311,15 @@ the tier trades a stale-code bug for a total outage. Full procedure:
|
|
|
269
311
|
documented in both places so a rotation updates both.
|
|
270
312
|
|
|
271
313
|
## Change history
|
|
272
|
-
|
|
314
|
+
- 2026-09-01 — Added the rule that **api2 must not reference a recently-added `_underscore` symbol
|
|
315
|
+
without checking branch containment**, since each tier pins its own framework branch. Concrete
|
|
316
|
+
(`TRUE-81056`): `\_Database::ERRNO_DUPLICATE_ENTRY` (added by `_underscore` `6355d0f0`) is present
|
|
317
|
+
only on `_production` and `_sandbox-client`, so referencing it in a `catch` block fataled with
|
|
318
|
+
`Undefined constant` on beta and dev-sandbox (`Logs.Event` 7405 / 7404) — and because the fatal was
|
|
319
|
+
*inside the catch*, it replaced the real error with a confusing one. Hardened api2 to define its own
|
|
320
|
+
`MYSQL_ERRNO_DUPLICATE_ENTRY = 1062` and read `\_Database::$lastErrorNumber` via `property_exists()`.
|
|
321
|
+
Rule: run `git branch -a --contains <commit>` on `_underscore` before using a new framework symbol,
|
|
322
|
+
and never reference a not-yet-universal symbol from an error path. (bala)
|
|
273
323
|
- 2026-08-31 — Recorded a **worked instance** of the cross-repo ordering rule: **`client-sandbox`
|
|
274
324
|
went entirely down** — every request 500'd with `Call to undefined method
|
|
275
325
|
_Model_Core_AclFieldPermission::resolveFieldPermissions()`. `api2@_sandbox-client` (`V2.php:6198`,
|
|
@@ -17,6 +17,8 @@ related:
|
|
|
17
17
|
- ../architecture.md
|
|
18
18
|
- ./api-payload-interceptors.md
|
|
19
19
|
- ./v2-api-error-codes.md
|
|
20
|
+
- ./v2-deadlock-retry.md
|
|
21
|
+
- ./environment-variable-drives-underscore-branch.md
|
|
20
22
|
- ../../_underscore/features/error-reporting-issue-event.md
|
|
21
23
|
- ../../../../clients/aig/features/entitlement-intake.md
|
|
22
24
|
- ../../../../clients/compass-usa/features/mits-po-to-so-item-linking.md
|
|
@@ -149,6 +151,116 @@ transaction no matter how many times apiproxy retried. That is why the retry att
|
|
|
149
151
|
there even in principle — the two logs are not two views of the same rows. Join the two by
|
|
150
152
|
`transactionId` when you need the full attempt sequence for one user action.
|
|
151
153
|
|
|
154
|
+
## ⚠ A duplicate `transactionId` must NOT fail the request — save the row, then rename on 1062
|
|
155
|
+
|
|
156
|
+
`Logs_<Client>.Api.transactionId` (and core `Logs.Api.transactionId`) carries a **UNIQUE** key, so
|
|
157
|
+
the log-save at the end of `execute()` is the one place where **two copies of the same request
|
|
158
|
+
collide**. Until 2026-09-01 that collision **turned an already-successful API response into an
|
|
159
|
+
HTTP 500** — the response was fully built, the work was done, and the *audit row* killed it.
|
|
160
|
+
|
|
161
|
+
**The old code was a check-then-insert** (`V2.php` ~L2283-2295 on `_production`):
|
|
162
|
+
|
|
163
|
+
```php
|
|
164
|
+
// PRE-FIX — structurally cannot work under concurrency
|
|
165
|
+
$existingLog = new $logsApiModelName();
|
|
166
|
+
$existingLog->transactionId = $transactionId;
|
|
167
|
+
if ($existingLog->load()) {
|
|
168
|
+
$transactionId .= ('-' . (new \DateTime())->format('Y-m-d H:i:s.v'));
|
|
169
|
+
$log->transactionId = $transactionId;
|
|
170
|
+
}
|
|
171
|
+
$log->save();
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
**Why the pre-check can never work.** The rival copy's row is still inside its **uncommitted
|
|
175
|
+
transaction**, so it is invisible to any `SELECT`. Both copies pass the check, both believe the id
|
|
176
|
+
is free, and the second `INSERT` hits the unique key with **MySQL 1062**. The exception escapes the
|
|
177
|
+
logger, unwinds to `Controller/Index.php`, and the caller gets a 500 for work that had already
|
|
178
|
+
succeeded. **No lookup can fix this** — a read cannot see a row that has not been committed yet.
|
|
179
|
+
|
|
180
|
+
**The fix (2026-09-01, `TRUE-81056` / commit `db94a26`)** — `saveApiLogUnderItsOwnTransactionId()`
|
|
181
|
+
and `buildResendTransactionId()` in `V2.php`: **save first, and let the database arbitrate.** On a
|
|
182
|
+
1062, rename the row to `<transactionId>-1`, then `-2`, and retry. Anything that is *not* a 1062 is
|
|
183
|
+
logged with `error_log()` and swallowed — **a log row must never be the reason a finished request
|
|
184
|
+
fails.** Properties worth knowing:
|
|
185
|
+
|
|
186
|
+
- The suffix is always built from the **originally requested** id, so the third copy is `-2`, never
|
|
187
|
+
`-1-2`. Every copy of one user action still sorts and searches together.
|
|
188
|
+
- The prefix is truncated to `TRANSACTION_ID_MAX_LENGTH` (255) minus the suffix, because a silent
|
|
189
|
+
MySQL column-width chop would re-collide.
|
|
190
|
+
- The loop is **deliberately unbounded and cannot spin**: every UNIQUE column changes per attempt
|
|
191
|
+
(`_Model` mints a fresh `uuid` inside each `save()`, `id` is auto-increment, the resend number only
|
|
192
|
+
climbs), so each try eliminates one taken value and never re-asks for the same one.
|
|
193
|
+
- **It also removes one `SELECT` per API request** against an ~11.5M-row table.
|
|
194
|
+
|
|
195
|
+
### The duplicate comes from BELOW the application — you cannot stop it, only tolerate it
|
|
196
|
+
|
|
197
|
+
Do not chase this as a double-click. The frontend axios interceptor
|
|
198
|
+
(`toga2-commerce/src/api/axiosInstance.ts`) mints a **fresh uuid per request**, so a genuine
|
|
199
|
+
double-click produces **two different ids**. Two requests sharing one id means the **identical bytes
|
|
200
|
+
were replayed by something under the app**:
|
|
201
|
+
|
|
202
|
+
- **2026-08-18 (production)** — the two copies arrived through **two different Zscaler proxy nodes**
|
|
203
|
+
(`136.226.49.71` and `136.226.49.80`) with **different ALB trace ids**.
|
|
204
|
+
- **2026-08-28** — the copies came over a plain path with **no proxy hop at all**.
|
|
205
|
+
|
|
206
|
+
A `GET` is safe to repeat, so browsers, corporate proxies and load balancers resend it on their own
|
|
207
|
+
when a connection drops — and they resend the original bytes, `transactionId` included. This is the
|
|
208
|
+
same mechanism already documented above for apiproxy's cross-region 5xx retry. **The server must
|
|
209
|
+
tolerate the repeat; the client cannot prevent it.**
|
|
210
|
+
|
|
211
|
+
### An UNAUTHENTICATED request still writes a log row — so the 500 was reachable with no token
|
|
212
|
+
|
|
213
|
+
A **401 response still reaches the log-save code** and writes a row carrying the caller's
|
|
214
|
+
`transactionId`. That is how the defect was proven from outside with no credentials, and it made the
|
|
215
|
+
500 a (minor) unauthenticated abuse vector. The fix closes it. When reproducing any log-write
|
|
216
|
+
behavior, remember you do **not** need a valid token to create a row.
|
|
217
|
+
|
|
218
|
+
### ⚠ Accepted exposure — removing the 500 also removed an ACCIDENTAL write-rollback
|
|
219
|
+
|
|
220
|
+
Recorded deliberately, because it is not obvious. On an uncaught `Throwable`,
|
|
221
|
+
`Controller/Index.php` (~L308) rolls back the request's data changes. The 1062 escaping the logger
|
|
222
|
+
was therefore **silently cancelling the losing duplicate's writes**. With the fix the loser no longer
|
|
223
|
+
throws, so **two concurrent duplicate POSTs would both commit.**
|
|
224
|
+
|
|
225
|
+
The intended guard is the **EV-5 client-`transactionId` replay check** (`V2.php` ~L1971-1977), but it
|
|
226
|
+
is a `load()` on `Logs_Client_Api.transactionId` and therefore has **exactly the same
|
|
227
|
+
uncommitted-read blind spot** as the code that was just removed — under true concurrency it does not
|
|
228
|
+
see the rival either. Every recorded occurrence of this defect has been a **GET**, so nothing has
|
|
229
|
+
been hit.
|
|
230
|
+
|
|
231
|
+
**Status: known, accepted exposure — not a regression to revert.** The correct long-term fix is to
|
|
232
|
+
**claim the `transactionId` at the START of the request instead of the end**, so the UNIQUE key
|
|
233
|
+
enforces idempotency atomically and the loser is rejected before it does any work.
|
|
234
|
+
|
|
235
|
+
### This is the fourth symptom of one root pattern — check-then-insert on `Logs.Api`
|
|
236
|
+
|
|
237
|
+
Treat any new `Logs.Api` incident as a member of this family before theorizing:
|
|
238
|
+
|
|
239
|
+
| Record | Symptom | What was done |
|
|
240
|
+
|---|---|---|
|
|
241
|
+
| `Logs.Issue 26` | Same defect on `/v2/users/me`, **312 occurrences** | "Fixed" 2026-08-25 (`84fd7a1`) by changing only the **fallback value** to a uuid — the race was left intact |
|
|
242
|
+
| `Logs.Issue 250` | The surviving `/v2/users` case | Same root cause |
|
|
243
|
+
| `Logs.Issue 2L` | Memory exhaustion on `/v2/auth/public` | `36ef31f` — the same pre-check full-table-scanned on a null/empty `transactionId` |
|
|
244
|
+
| `Logs.Issue CT` | 500 on a successful request (this one) | `db94a26` — pre-check removed entirely |
|
|
245
|
+
|
|
246
|
+
The 2026-08-25 work uuid'd the **auto-generated** log-row id, which is why the architecture doc's
|
|
247
|
+
gotcha reads "FIXED". It did not touch the **client-supplied** id, which still went through
|
|
248
|
+
check-then-insert. Both halves had to go.
|
|
249
|
+
|
|
250
|
+
### Verifying a concurrency fix — confirm the CONSTRAINT exists in the target environment
|
|
251
|
+
|
|
252
|
+
`Logs.Api.transactionId` is **UNIQUE on production, `sandbox-client` and `dev-sandbox`, but only a
|
|
253
|
+
NON-unique index locally.** A local pass therefore proves nothing: the race cannot fail where the
|
|
254
|
+
key does not exist. **Confirm the constraint in the environment you are testing before you trust a
|
|
255
|
+
green result.**
|
|
256
|
+
|
|
257
|
+
The method that worked:
|
|
258
|
+
|
|
259
|
+
1. **Reproduced on production first try** with two concurrent unauthenticated `curl`s sharing one
|
|
260
|
+
`transactionId`.
|
|
261
|
+
2. **Proven on `client-sandbox`**: three concurrent copies of the original failing request all
|
|
262
|
+
returned 401, wrote rows `<id>`, `<id>-1`, `<id>-2`, and produced **zero 500s**.
|
|
263
|
+
|
|
152
264
|
## Getting the REAL error behind a generic `EO-1` 500 (the three-hop chain)
|
|
153
265
|
|
|
154
266
|
A 500 whose only content is `EO-1` — *"Operation Failed: An internal server error occurred"* —
|
|
@@ -288,7 +400,27 @@ re-send.
|
|
|
288
400
|
> replay operationally, not by pasting payloads into the KB.
|
|
289
401
|
|
|
290
402
|
## Change history
|
|
291
|
-
|
|
403
|
+
- 2026-09-01 — **Fixed the duplicate-`transactionId` 500** (`Logs.Issue CT`, `TRUE-81056` /
|
|
404
|
+
`db94a26`): the log-save at the end of `execute()` did a **check-then-insert**, which cannot work
|
|
405
|
+
under concurrency because the rival copy's row is still uncommitted and therefore invisible to the
|
|
406
|
+
`SELECT` — both copies passed, the second `INSERT` hit the UNIQUE key with **1062**, and the
|
|
407
|
+
exception turned an **already-successful response into a 500**. Replaced with
|
|
408
|
+
`saveApiLogUnderItsOwnTransactionId()` / `buildResendTransactionId()`: save first, and on 1062
|
|
409
|
+
rename to `<id>-1`, `-2` and retry, so the database arbitrates; any non-1062 error is `error_log`'d
|
|
410
|
+
and swallowed (a log row must never fail a finished request). Also drops one `SELECT` per request
|
|
411
|
+
against an ~11.5M-row table. Recorded that the duplicate originates **below the app** — the
|
|
412
|
+
toga2-commerce axios interceptor mints a fresh uuid per request, so a double-click yields two
|
|
413
|
+
*different* ids; the 2026-08-18 production pair arrived via two Zscaler nodes
|
|
414
|
+
(`136.226.49.71`/`.80`) with different ALB trace ids, the 2026-08-28 pair with no proxy hop — a GET
|
|
415
|
+
is safe to repeat, so proxies and load balancers resend the original bytes on their own. Recorded
|
|
416
|
+
that a **401 still writes a log row**, which made the 500 reachable unauthenticated. **Accepted a
|
|
417
|
+
known exposure**: the 500 was accidentally triggering the `Controller/Index.php` (~L308)
|
|
418
|
+
`Throwable` rollback of the loser's writes, and the EV-5 replay guard (`V2.php` ~L1971-1977) shares
|
|
419
|
+
the same uncommitted-read blind spot — every recorded case is a GET, and the real fix is to claim
|
|
420
|
+
the `transactionId` at the **start** of the request. Linked the family (`Logs.Issue` 26 / 250 / 2L)
|
|
421
|
+
and the verification rule: `Logs.Api.transactionId` is UNIQUE on prod/`sandbox-client`/`dev-sandbox`
|
|
422
|
+
but **non-unique locally**, so confirm the constraint before trusting a green test. Reported by
|
|
423
|
+
Compass Group; the defect is platform-wide. (bala)
|
|
292
424
|
- 2026-09-01 — Added **Getting the REAL error behind a generic `EO-1` 500** — the three-hop chain
|
|
293
425
|
from an opaque 500 to the actual failing field. Key new fact: `internalApiRequest()` logs each
|
|
294
426
|
**nested** sub-request into `Logs_<Client>.Api` under a **randomly generated `transactionId`**
|
|
@@ -6,8 +6,8 @@ project: API
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: draft
|
|
9
|
-
updated: 2026-
|
|
10
|
-
owners: [jcardinal]
|
|
9
|
+
updated: 2026-09-01
|
|
10
|
+
owners: [jcardinal, bala]
|
|
11
11
|
files:
|
|
12
12
|
- api2/Controller/Index.php
|
|
13
13
|
- api2/Component/Api/V2/V2.php
|
|
@@ -19,6 +19,8 @@ related:
|
|
|
19
19
|
- ../../../../clients/compass-usa/features/asn-to-item-fulfillment.md
|
|
20
20
|
- ../architecture.md
|
|
21
21
|
- ./v2-api-error-codes.md
|
|
22
|
+
- ./request-logging.md
|
|
23
|
+
- ./environment-variable-drives-underscore-branch.md
|
|
22
24
|
---
|
|
23
25
|
|
|
24
26
|
## Summary
|
|
@@ -136,6 +138,22 @@ replay-safe and must stay off the list:
|
|
|
136
138
|
idempotent or transaction-scoped, with no irreversible side effect and no consumed/mutated
|
|
137
139
|
superglobal. Default is to leave a route off the list.
|
|
138
140
|
|
|
141
|
+
## ⚠ `_Database::ERRNO_DUPLICATE_ENTRY` has SHIPPED — but not to every tier
|
|
142
|
+
|
|
143
|
+
Verified 2026-09-01 (`git show <branch>:Database.php` in `_underscore`): the primitives added by
|
|
144
|
+
this work (commit **`6355d0f0`** — `ERRNO_DUPLICATE_ENTRY = 1062`, `$lastErrorNumber`,
|
|
145
|
+
`$retriableUniqueRace`, `wasRetriableUniqueRace()`) are on **`_production` and `_sandbox-client`
|
|
146
|
+
only**. They are **absent from `_beta` and `_sandbox-dev`**.
|
|
147
|
+
|
|
148
|
+
Because api2 clones `_underscore` from a **per-environment branch**, any app-side code that names
|
|
149
|
+
`\_Database::ERRNO_DUPLICATE_ENTRY` **fatals with `Undefined constant` on beta and dev-sandbox** —
|
|
150
|
+
and if the reference sits in a `catch` block, the fatal replaces the real error with a confusing one
|
|
151
|
+
(seen in `Logs.Event` 7405 / 7404 on 2026-09-01). App-side callers should define their own `1062`
|
|
152
|
+
constant and reach `$lastErrorNumber` through `property_exists()` until the framework side is on
|
|
153
|
+
every branch. See
|
|
154
|
+
[ENVIRONMENT decides the `_underscore` branch](./environment-variable-drives-underscore-branch.md)
|
|
155
|
+
and [V2 request logging](./request-logging.md).
|
|
156
|
+
|
|
139
157
|
## Gotchas / known issues
|
|
140
158
|
|
|
141
159
|
- Detection is built on the **MySQL errno** (1213/1205), not on matching the response text —
|
|
@@ -146,6 +164,13 @@ superglobal. Default is to leave a route off the list.
|
|
|
146
164
|
not a substitute for removing the deadlock at its source.
|
|
147
165
|
|
|
148
166
|
## Change history
|
|
167
|
+
- 2026-09-01 — Recorded the **deployment reach** of this work's `_underscore` primitives: commit
|
|
168
|
+
`6355d0f0` (`ERRNO_DUPLICATE_ENTRY`, `$lastErrorNumber`, `$retriableUniqueRace`,
|
|
169
|
+
`wasRetriableUniqueRace()`) is on **`_production` and `_sandbox-client` only** — **absent from
|
|
170
|
+
`_beta` and `_sandbox-dev`** — so api2 code naming `\_Database::ERRNO_DUPLICATE_ENTRY` fatals with
|
|
171
|
+
`Undefined constant` on those tiers (`Logs.Event` 7405 / 7404). App-side callers must define a local
|
|
172
|
+
1062 constant and read `$lastErrorNumber` via `property_exists()` until the branches catch up. No
|
|
173
|
+
code change to this feature. (bala)
|
|
149
174
|
- 2026-08-28 — **Generalized Prong A to a second retriable class: the unique-key (1062)
|
|
150
175
|
get-or-create race**, fixing the Compass ASN data-loss bug (`Logs.Issue #239`, EV-10) where a
|
|
151
176
|
concurrent same-`ItemFulfillments.number` INSERT lost the race, `internalApiRequest` threw, and
|
|
@@ -6,8 +6,8 @@ project: Worker
|
|
|
6
6
|
client: elite
|
|
7
7
|
type: client-feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-
|
|
10
|
-
owners: ["snaredla"]
|
|
9
|
+
updated: 2026-09-01
|
|
10
|
+
owners: ["snaredla", "jcardinal"]
|
|
11
11
|
files:
|
|
12
12
|
- worker/crons/toga2/netsuite/sync_togasupply_elite.php
|
|
13
13
|
- worker/crons/toga2/netsuite/common_sync_togasupply.php
|
|
@@ -125,6 +125,17 @@ the trait declares must exist as a column, a `CustomRecordFields` row **and** an
|
|
|
125
125
|
|
|
126
126
|
## Change history
|
|
127
127
|
|
|
128
|
+
- 2026-09-01 — **Drop-ship / special-order POs were never importing for Elite** (e.g. PO tranid
|
|
129
|
+
128238). Root cause was in shared code, not Elite: `listPurchaseOrders()` filtered the PO header by
|
|
130
|
+
the VENDOR `entity` (TD Synnex 20202) instead of the customer, so Elite's 36443-scoped filter matched
|
|
131
|
+
0 POs, and drop-ship POs — which make item fulfillments, not receipts — had no other import path.
|
|
132
|
+
Fixed engine-side (new `buildPurchaseOrderCustomerFilter()` scoping by line-level customer /
|
|
133
|
+
end-customer); PO 128238 then imported (dev confirmed). Full write-up on the
|
|
134
|
+
[engine doc](../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md) and
|
|
135
|
+
[SuiteQL/REST shim doc](../../../1.0/apps/library/features/netsuite-suiteql-rest-shim.md). Elite's
|
|
136
|
+
older stock-only PO count (51) will grow once its `NETSUITE_LAST_SYNC_DATETIME_PURCHASE_ORDERS` is
|
|
137
|
+
rewound to backfill drop-ship POs (idempotent; first 2024-04-15..04-18 window alone held 269 POs).
|
|
138
|
+
(jcardinal)
|
|
128
139
|
- 2026-08-12 — TRUE-80499: **onboarded Elite to the NetSuite → TOGa Supply importer.** Built the
|
|
129
140
|
wrapper (customer 36443, `isParentCustomer` false, 30-day window, invoices temporarily disabled),
|
|
130
141
|
the schedule entry, 18 `_Model_Elite_*` netsuite trait models, the `_modules/netsuite` opt-in, and
|
package/package.json
CHANGED