toga-ai 1.0.731 → 1.0.733

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -11,7 +11,7 @@
11
11
  | [Elite Freshservice Sync (library)](features/elite-freshservice-sync.md) | `App_Api_Toga2` in `library/app/api/toga2.php` orchestrates bidirectional sync between TOGA 2 and TOGaDesk. | library/app/api/toga2.php |
12
12
  | [App_Email Queued Sending & Attachments (Common.EmailsQueued)](features/email-queue-attachments.md) | `App_Email::send()` can either send **inline** (PHPMailer talks to SES right there) or **queue** the message: `base64(serialize($this))` is inserted into `Commo | library/app/email.php, library/phpmailer/class.phpmailer.php, worker/crons/notifications/infrastructure/send_emails.php, worker/crons/notifications/covid/send_covid_pending_vaccination_approval.php, togadesk/desk/includes/functions.php |
13
13
  | [Branded HTML Email Templates (App_Email_Template)](features/email-templates.md) | `App_Email_Template` (`app/email/template.php`) is the base class for branded HTML emails in the 1.0 (`App_`) framework. | library/app/email/template.php, library/app/email/agilant.php |
14
- | [App_Email Side Effects & Test Mode (how to send a real email that writes nothing)](features/email-test-mode-and-write-free-sends.md) | `App_Email::send()` is **not** side-effect free. | library/app/email.php, library/app/framework.php |
14
+ | [App_Email Side Effects & Test Mode (how to send a real email that writes nothing)](features/email-test-mode-and-write-free-sends.md) | `App_Email::send()` is **not** side-effect free. | library/app/email.php, library/app/model/logs/email.php, library/app/framework.php |
15
15
  | [Error Capture in 1.0 (App_Error_Capture → shared 2.0 Logs DB)](features/error-capture-1-0.md) | The 1.0 side of the platform error-reporting pipeline (TRUE-78188). | library/app/error/capture.php, library/app/error.php, library/app/exception/business.php, library/app/api/toga2.php, library/app/cloud.php, worker/config.worker.ini, worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php |
16
16
  | [HTTP 500 Error Monitor (App_SystemMonitor_500Error) — and why its \"Error Type\" is not a diagnosis](features/http-500-error-monitor.md) | `App_SystemMonitor_500Error` (`library/app/systemmonitor/500error.php`, title **"HTTP 500 Error Alert"**) is the 1.0 system monitor that watches **`Logs.Api` fo | library/app/systemmonitor/500error.php, worker/crons/infrastructure/system_monitors.php, api2/Controller/Index.php, _underscore/Error.php |
17
17
  | [1.0 MVC Page Pattern & New-App Skeleton](features/mvc-page-pattern-and-app-skeleton.md) | This is the **reusable recipe for standing up a new 1.0 (`App_`) application** and for adding pages to one — the folder-based MVC routing, the page lifecycle, t | library/app/framework.php, library/app/frameworkindex.php, library/app/mvc.php, library/app/database.php, library/app/model.php, library/app/config.php |
@@ -6,10 +6,11 @@ project: Library
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-08-31
9
+ updated: 2026-09-01
10
10
  owners: ["bala"]
11
11
  files:
12
12
  - library/app/email.php
13
+ - library/app/model/logs/email.php
13
14
  - library/app/framework.php
14
15
  related:
15
16
  - ./email-queue-attachments.md
@@ -32,7 +33,7 @@ Verified against `library/app/email.php` on 2026-08-31.
32
33
 
33
34
  | Write | When | How to suppress |
34
35
  |---|---|---|
35
- | `Logs.Email` row (`App_Model_Logs_Email`) | every **inline** send, success or failure | `setIsLogEnabled(false)` |
36
+ | `Logs.Emails` row (`App_Model_Logs_Email`) | every **inline** send, success or failure | `setIsLogEnabled(false)` |
36
37
  | `Common.EmailsQueued` row (`App_Model_Common_EmailsQueued`) | `queueForSending` truthy, **or** outbound email is disabled and `sendEmailEvenIfOutboundEmailIsDisabled` is false | `setQueueForSending(false)` + `sendEmailEvenIfOutboundEmailIsDisabled = true` |
37
38
 
38
39
  The queue-vs-send decision is a single condition (`email.php` ~L313):
@@ -46,6 +47,40 @@ if ($this->queueForSending || (!$this->sendEmailEvenIfOutboundEmailIsDisabled &&
46
47
  a queue insert** with no code change on your side. The log write is inside the `else` branch, so a
47
48
  queued message writes the queue row and **no** log row.
48
49
 
50
+ ## Where the log lives — and how to verify a send actually happened
51
+
52
+ The log row is the **only** durable proof that a 1.0 script really sent a mail, so knowing where
53
+ it lands matters. `App_Model_Logs_Email` (`library/app/model/logs/email.php`) declares
54
+ `TABLENAME = 'Emails'`, `DATABASE = 'db_logs'` and `$_databaseNameOverride = 'Logs'`. Read that
55
+ literally:
56
+
57
+ - **Table is `Emails` (plural), schema `Logs`**, and it lives in the **legacy (v1) environment** —
58
+ the one `db_logs` points at for the 1.0 tier.
59
+ - It is **not** in the prod `Logs` schema, and **not** in a per-client `Logs_<Client>` schema.
60
+ Prod `Logs_Compass` has an `Email` (singular) table used by **2.0**, and it stays empty for
61
+ these 1.0 cron emails. Looking there is the usual wrong turn.
62
+
63
+ Columns worth reading: `dtSent`, `isSuccess`, `error`, `sourceJob`, `instanceId`, `subject`,
64
+ `from`, `to`, `cc`, `bcc`, `body`.
65
+
66
+ ```sql
67
+ # legacy env, schema Logs
68
+ SELECT id, dtSent, isSuccess, error, sourceJob, subject, `to`
69
+ FROM Emails
70
+ WHERE subject LIKE '<subject prefix>%'
71
+ ORDER BY id DESC;
72
+ ```
73
+
74
+ Two reading rules:
75
+
76
+ - **A row exists only if the code reached `->send()` and took the inline branch.** No row means
77
+ either the job never got that far (e.g. it had nothing to report), or the message was queued
78
+ instead (see the table above), or logging was disabled. It does **not** by itself mean a
79
+ failure.
80
+ - **`sourceJob` is `$_SERVER['SCRIPT_NAME']`, which tells you how it ran.** A full
81
+ `/var/www/html/crons/...` path is a scheduled run on the server; a bare filename is a manual
82
+ CLI run someone did by hand.
83
+
49
84
  ## The three switches for a genuinely write-free send
50
85
 
51
86
  All three are required — any one of them left at its default reintroduces a write or swallows the
@@ -54,7 +89,7 @@ send:
54
89
  1. `setQueueForSending(false)` — do not insert into `Common.EmailsQueued`.
55
90
  2. `sendEmailEvenIfOutboundEmailIsDisabled = true` — do not fall into the queue branch just
56
91
  because the `Defaults` flag is off.
57
- 3. `setIsLogEnabled(false)` — do not insert into `Logs.Email`.
92
+ 3. `setIsLogEnabled(false)` — do not insert into `Logs.Emails`.
58
93
 
59
94
  ## `App_Email` only ever *narrows* the recipient list
60
95
 
@@ -114,12 +149,16 @@ php -d include_path=".;<library>;<_underscore>" <path/to/script>.php
114
149
  queue path (see `email-queue-attachments.md`) and the recipient list. Two different behaviors
115
150
  you cannot exercise locally.
116
151
  - **Deleting the log write is not the same as being read-only.** `setIsLogEnabled(false)` only
117
- suppresses `Logs.Email`; the queue insert has its own switch and fires *instead of* sending.
152
+ suppresses `Logs.Emails`; the queue insert has its own switch and fires *instead of* sending.
118
153
  - **Dry-run harness scripts are throwaway.** Keep them out of the repo — they carry hardcoded
119
154
  recipient lists and bypass the real gating. Build one, use it, delete it; document the
120
155
  *technique* here rather than committing the file.
121
156
 
122
157
  ## Change history
158
+ - 2026-09-01 — Corrected the log target: the table is **`Emails`** (plural) in schema **`Logs`**
159
+ on the **legacy** environment, not `Logs.Email` and not a per-client `Logs_<Client>` schema
160
+ (prod `Logs_Compass.Email` is the 2.0 table and stays empty for 1.0 cron mail). Added the
161
+ verification query and the `sourceJob` scheduled-vs-manual tell. (bala)
123
162
  - 2026-08-31 — Documented that `App_Email::send()` writes `Logs.Email` unless
124
163
  `setIsLogEnabled(false)`, that the queue branch also fires when `OUTBOUND_EMAIL_ENABLED` is off,
125
164
  that `cronInitialization()` INSERTs `CronJobExecutions`, and that `send()` only ever narrows
@@ -3,7 +3,7 @@
3
3
  | Doc | Summary | Files |
4
4
  |-----|---------|-------|
5
5
  | [Worker (1.0 Framework) Architecture](architecture.md) | `worker` is the legacy (**1.0** `App_` framework) **background-job tier**. | worker/index.php, worker/_/app/framework.php, worker/crons/, worker/schedules/, worker/ebs/cron.worker.php, worker/.ebextensions/035_cron.worker.config, worker/crons/infrastructure/execute_dbchanges.php, worker/.ebextensions/030_dbchanges.config |
6
- | [Compass MA Sales Order Exception Report](features/compass-ma-sales-order-exception-report.md) | A worker cron that emails operations the "Compass Refresh Exception Report" — Compass `MA%` sales orders whose corresponding Office Depot (ODP) sales order has | worker/crons/toga2/compass/workflow/7_generate_ma_sales_order_exception_report.php |
6
+ | [Compass MA Sales Order Exception Report](features/compass-ma-sales-order-exception-report.md) | A worker cron that emails operations the "Compass Refresh Exception Report" — Compass `MA%` sales orders whose corresponding Office Depot (ODP) sales order has | worker/crons/toga2/compass/workflow/7_generate_ma_sales_order_exception_report.php, worker/schedules/cron.worker.sync.json |
7
7
  | [Compass Manager Approval Reminder Emails (1.0 worker crons)](features/compass-manager-approval-reminder-emails.md) | Two 1.0 worker crons nag approvers about sales orders still waiting on a decision — one per Compass tenant. | worker/crons/toga2/compass/compass_email_reminders.php, worker/crons/toga2/compasscanada/compass_email_reminders.php, worker/schedules/cron.worker.sync.json, worker1.5/crons/toga2/compass/compass_email_reminders.php, worker1.5/schedules/cron.worker.json |
8
8
  | [Compass Partial In-Transit & Delivered Emails (per package)](features/compass-partial-in-transit-delivered-emails.md) | Compass USA and Compass Canada send a **per-package** in-transit email (and a matching delivered email) instead of one email listing the whole order. | worker/crons/toga2/compass/update_salesorder_status_from_odp.php, worker/crons/toga2/compass/send_delivered_email.php, worker/crons/toga2/compasscanada/update_salesorder_status_from_odp.php, worker/crons/toga2/compasscanada/workflow/3_update_salesorder_status_from_grand_and_toy.php, worker/crons/toga2/compasscanada/send_delivered_email.php, worker/crons/toga2/compasscanada/workflow/test_partial_in_transit_email.php, worker/crons/toga2/compasscanada/workflow/test_partial_delivered_email.php, library/app/client/compasscanada.php |
9
9
  | [Elite TOGA 2.0 → TOGaDeskSupport Standalone Attachment Sync](features/elite-togadesk-attachment-sync.md) | `sync_togadesk_elite_attachments.php` is a standalone cron (every 5 minutes) that syncs file attachments from TOGA 2.0 into TOGaDeskSupport for Elite. | worker/crons/toga2/elite/sync_togadesk_elite_attachments.php, worker/crons/toga2/elite/test_sync_togadesk_elite_attachments.php |
@@ -19,4 +19,4 @@
19
19
  | [isFulfillable Multi-Client Backfill (all togasupply clients)](workflows/isfulfillable-multi-client-backfill.md) | One-time backfill that catches up `Items.isFulfillable` on **existing** items across **all 17 togasupply clients** (AIG, Broward Sheriff, Canon, Endeavor Health | worker/crons/toga2/netsuite/backfill_isfulfillable_all_clients.php, library/app/api/toga2.php |
20
20
  | [Deduplicating NetSuite duplicate Items across client DBs (recursive MERGE)](workflows/netsuite-duplicate-items-dedup.md) | A one-time, per-client data cleanup that collapses **duplicate `Items` rows sharing a `partNumber`** in each NetSuite → TOGa Supply client DB down to a single c | library/app/api/toga2.php, worker/crons/toga2/netsuite/common_sync_togasupply.php |
21
21
  | [Onboarding a Client to the NetSuite TOGa Supply Sync](workflows/onboarding-client-to-netsuite-togasupply-sync.md) | How to add a new TOGa 2 client to the per-client NetSuite → TOGa Supply importer (`worker/crons/toga2/netsuite/`). | worker/crons/toga2/netsuite/sync_togasupply.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_elite.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql, dbchanges2/_modules/netsuite/2026-08-05 - CLEAN NETSUITE CLINET.SQL |
22
- | [Tracing a 1.0 worker cron run in production (no stdout, silent skips)](workflows/tracing-a-worker-cron-run-in-production.md) | How to answer *"did this cron actually run, and what did it do?"* on the 1.0 `worker` tier, where **there is no usable stdout** and **a skipped run leaves no tr | worker/ebs/cron.worker.php, worker/schedules/cron.worker.sync.json, worker/schedules/cron.worker.infrastructure.json, worker/.ebextensions/009_setup_phpini.config, library/app/framework.php |
22
+ | [Tracing a 1.0 worker cron run in production (no stdout, silent skips)](workflows/tracing-a-worker-cron-run-in-production.md) | How to answer *"did this cron actually run, and what did it do?"* on the 1.0 `worker` tier, where **there is no usable stdout** and **a skipped run leaves no tr | worker/ebs/cron.worker.php, worker/schedules/cron.worker.sync.json, worker/schedules/cron.worker.infrastructure.json, worker/.ebextensions/009_setup_phpini.config, library/app/framework.php, library/app/email.php, library/app/model/logs/email.php |
@@ -6,11 +6,14 @@ project: Worker
6
6
  client: compass-usa
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-08-21
9
+ updated: 2026-09-01
10
10
  owners: ["bala", "mhammontree"]
11
11
  files:
12
12
  - worker/crons/toga2/compass/workflow/7_generate_ma_sales_order_exception_report.php
13
- related: []
13
+ - worker/schedules/cron.worker.sync.json
14
+ related:
15
+ - ../workflows/tracing-a-worker-cron-run-in-production.md
16
+ - ../../library/features/email-test-mode-and-write-free-sends.md
14
17
  ---
15
18
 
16
19
  ## Summary
@@ -22,6 +25,9 @@ but haven't. Runs against the Compass client DB (`db_prod_compass` / `Client_Com
22
25
  ## Key files / entry points
23
26
  - `worker/crons/toga2/compass/workflow/7_generate_ma_sales_order_exception_report.php` — the
24
27
  whole job: one main SELECT, a per-row NetSuite recheck, and a PhpSpreadsheet Excel email.
28
+ - `worker/schedules/cron.worker.sync.json` — the schedule entry
29
+ **"Exception Report - EDI Office Depot (8am - 4pm)"**, `0 8-14 * * *` (hourly, on the hour;
30
+ the crontab runs **Central** time).
25
31
 
26
32
  ## How it works
27
33
  1. **Main query** walks a two-hop bridge chain to connect the customer order to the vendor order:
@@ -37,7 +43,9 @@ but haven't. Runs against the Compass client DB (`db_prod_compass` / `Client_Com
37
43
  (`CUSTOMER_ID__AGILANT` = 3) sales order tied to that ODP PO number. If found → it back-fills
38
44
  `c_netsuiteInternalSalesOrderId` on the ODP SO and skips the row (drops it off the report
39
45
  permanently). If not found → it writes the row to the Excel.
40
- 4. Emails the workbook to operations only if at least one data row was written.
46
+ 4. Emails the workbook to operations **only if at least one data row was written** — the send is
47
+ guarded by `if ($rowNumber > 2)`, where row 1 is the header. Subject is
48
+ `Compass Refresh Exception Report : <date>`.
41
49
 
42
50
  ## Data model
43
51
  - `Client_Compass.SalesOrders` — the ODP sales order is the one keyed on (`id AS salesOrderId`,
@@ -69,10 +77,19 @@ Compass USA only — this is a Compass-specific integration cron.
69
77
  - **Don't fake the NetSuite id to hide an order.** Writing a bogus `c_netsuiteInternalSalesOrderId`
70
78
  removes it from the report but corrupts the field for anyone who reads it later. Use the stage
71
79
  lever instead.
80
+ - **A silent hour is not a failure.** Because of the `$rowNumber > 2` guard, an hour with zero
81
+ exceptions sends nothing at all. Before treating a missing report as a broken cron, check the
82
+ run itself (`Common.CronJobExecutions`) and the send log — the legacy env's `Logs.Emails`
83
+ (schema `Logs`), `WHERE subject LIKE 'Compass Refresh Exception Report%'`. See
84
+ [Tracing a 1.0 worker cron run](../workflows/tracing-a-worker-cron-run-in-production.md).
72
85
  - `$inProduction = true` switches the DB link to `db_prod_compass`; set false to test against
73
86
  `db_beta_compass`.
74
87
 
75
88
  ## Change history
89
+ - 2026-09-01 — Recorded the schedule (`cron.worker.sync.json`, "Exception Report - EDI Office
90
+ Depot (8am - 4pm)", `0 8-14 * * *`) and the `$rowNumber > 2` send guard, so a silent hour reads
91
+ as "zero exceptions", not a fault; added how to prove a send via the legacy `Logs.Emails` table.
92
+ Verified 2026-09-01: recent runs all logged `isSuccess = 1`. (bala)
76
93
  - 2026-08-21 — Removed the departing employee `mreich@togatech.com` from this cron's hardcoded
77
94
  recipient list (TRUE-80451). Recipients are in-script, not DB-driven. (mhammontree)
78
95
  - 2026-06-18 — Added stage-based exclusion so orders closed/canceled on ODP's end drop off the
@@ -530,6 +530,20 @@ Parameters are stored **per client DB** but accessed **through the TOGa2 API**,
530
530
  a new line is actually created. Full root cause, the prod cleanup migration, and the
531
531
  deploy-fix-before-cleanup sequencing live in the
532
532
  [Compass ODP Duplicate PO-Line Cleanup](../../../clients/compass-usa/workflows/odp-duplicate-po-line-cleanup.md).
533
+ - **Customer-PO created from a sales order was missing `dateOrder` and had `cost` hardcoded 0 (both
534
+ fixed 2026-09-01, shared, `library` only).** In `syncSalesOrderFromNetsuite`
535
+ (`library/app/api/toga2.php`), the "create the customer PO" `POST /purchase-orders` branch
536
+ (~L1961) never set `dateOrder`, so every customer PO built from a sales order had
537
+ `PurchaseOrders.dateOrder` NULL. Fix: add
538
+ `'dateOrder' => App_Date::convertToTimezone($nsOrder->tranDate, 'Y-m-d', 'America/New_York')`
539
+ (the same expression that sets the SO's own `dateOrder` at ~L1394). Separately, all **three**
540
+ PO-item creation sites (`POST /purchase-order-item-sales-order-items`, ~L1863/1937/2051) sent
541
+ `'cost' => 0`; because this PO is the customer PO **for** the sales order, the PO line's cost is
542
+ the SO line's price — changed to `'cost' => $salesOrderItem['price']` (`$itemPrice`, from the
543
+ NetSuite rate or amount/qty fallback built ~L1223, always present). **Both are INSERT-ONLY:** the
544
+ create branches run only when the PO / line does not already exist and nothing PUTs `dateOrder` or
545
+ `cost`, so re-syncs and a second SO sharing the same customer-PO number never overwrite them.
546
+ **Existing NULL/`cost=0` rows are NOT backfilled** — new records only.
533
547
  - **⚠ A single un-deletable renumbered line could freeze the entire SALES_ORDERS section (fixed
534
548
  2026-08-17).** When NetSuite renumbers a line, `syncSalesOrderFromNetsuite` renumbers by
535
549
  **DELETE `/sales-order-items/{uuid}` then re-create in the per-item loop**. Its "pinned row" guard
@@ -881,9 +895,43 @@ Parameters are stored **per client DB** but accessed **through the TOGa2 API**,
881
895
  `'catalog' => ['name' => 'Agilant']` on the inventory-adjustment item-create. Stops all new duplicates.
882
896
  The one-time cleanup of the accumulated ~4,300 duplicate part numbers across all 17 clients is the
883
897
  [NetSuite duplicate-Items dedup workflow](../workflows/netsuite-duplicate-items-dedup.md).
898
+ - **⚠ Inventory-adjustment `/units` reconciliation missed an existing unit on a serial-CASE mismatch →
899
+ 1062 dup-key freeze (fixed 2026-09-01, shared, `library` only).** In
900
+ `syncInventoryAdjustmentFromNetsuite` (`library/app/api/toga2.php` ~L5985) the PHP lookup of existing
901
+ units was keyed by the **returned** unit serial (`$thisLookupUnitsBySerialNumber[$unit->serialNumber]`,
902
+ stored `'n/a'`) but looked up with the NetSuite serial **uppercased** via `strtoupper` (`'N/A'`). **PHP
903
+ array keys are case-sensitive; the `Units.itemId_serialNumber` unique key is case-insensitive** — so
904
+ the existing unit was missed, re-INSERTed, and collided MySQL **1062** ("Duplicate entry '560-N/A'")
905
+ → `EV-10` → the section aborted (fail-loud, no per-record try/catch) and froze at `1-RUNNING`. **Fix:**
906
+ key the lookup by `strtoupper($unit->serialNumber)` so it matches the uppercased NetSuite serial;
907
+ section self-heals on deploy (mode→IDLE, watermark advances). The `Items.uuid=$uuidItem` scoping was
908
+ left intact deliberately — copying the fulfillment "serial-only" lookup would be wrong for the shared
909
+ `'N/A'` placeholder serial (it would reuse another item's placeholder unit). **Durable lesson: when a
910
+ PHP lookup mirrors a case-insensitive DB unique key, normalize the KEY the same way you normalize the
911
+ probe — case-fold both, not one.**
912
+ - **Diagnosis playbook (durable) — localize a frozen inventory-adjustment section without NetSuite
913
+ access.** The 2026-09-01 NYCHH freeze above was diagnosed purely from DB logs, and item-matching was
914
+ *ruled out* (item 560 / partNumber B11B242201 is the only Agilant-catalog item for that part — no
915
+ duplicate second-catalog item, `$uuidItem` resolved correctly; the inventory-adjustment item-create
916
+ already stamps catalog `Agilant`). Chain: **`Client_<Id>.Parameters`** (localize the stuck section:
917
+ `1-RUNNING` + frozen watermark) → **`Logs.Event` filtered by `clientId`** (crash `file:line`, the
918
+ chain `toga2.php → common_sync_togasupply.php → sync*FromNetsuite`, plus the `transactionId`) →
919
+ **`Logs_<Client>.Api` on that `transactionId`** (the untruncated `EV-*`/MySQL rejection + payload).
920
+ A "silent freeze" here is a serial/data collision, not necessarily a catalog/item-matching problem —
921
+ read the untruncated Api-log rejection before assuming item matching.
884
922
 
885
923
  ## Change history
886
924
 
925
+ - 2026-09-01 — **Customer-PO `dateOrder`/`cost` fixes + NYCHH inventory-adjustment serial-case unfreeze
926
+ (shared, `library` only, working-tree — not yet committed/deployed).** `syncSalesOrderFromNetsuite`
927
+ now sets the customer PO's `dateOrder` from the SO `tranDate` (was always NULL) and its PO lines'
928
+ `cost` from the SO line `price` (was hardcoded 0) across all three PO-item create sites — both
929
+ INSERT-only, no backfill of existing rows. `syncInventoryAdjustmentFromNetsuite` now keys the `/units`
930
+ reconciliation lookup by `strtoupper($unit->serialNumber)` so a case-mismatched serial (`'n/a'` vs
931
+ `'N/A'`) matches the case-insensitive `Units.itemId_serialNumber` unique key — was missing the
932
+ existing unit and 1062-colliding, freezing NYCHH `INVENTORY_ADJUSTMENTS` at `1-RUNNING` (watermark
933
+ frozen 2022-12-28). Ruled out item-matching as the cause (single Agilant-catalog item, `$uuidItem`
934
+ correct). All three benefit every togasupply client. (jcardinal)
887
935
  - 2026-09-01 — **TRUE-81576: stopped runaway duplicate `ShippingCarriers` creation (worker + library).**
888
936
  `common_sync_togasupply.php` now seeds the carrier lookup from its own paged `GET /shipping-carriers`
889
937
  loop instead of harvesting carriers out of the `/shipping-methods` loop — a carrier with zero
@@ -6,7 +6,7 @@ project: Worker
6
6
  client: shared
7
7
  type: workflow
8
8
  status: active
9
- updated: 2026-08-28
9
+ updated: 2026-09-01
10
10
  owners: ["bala", "kyalamarthi"]
11
11
  files:
12
12
  - worker/ebs/cron.worker.php
@@ -14,11 +14,14 @@ files:
14
14
  - worker/schedules/cron.worker.infrastructure.json
15
15
  - worker/.ebextensions/009_setup_phpini.config
16
16
  - library/app/framework.php
17
+ - library/app/email.php
18
+ - library/app/model/logs/email.php
17
19
  related:
18
20
  - ../architecture.md
19
21
  - ./diagnosing-frozen-cron-checkins.md
20
22
  - ../features/forecast2-supporting-records-import.md
21
23
  - ../../library/features/cron-execution-monitoring.md
24
+ - ../../library/features/email-test-mode-and-write-free-sends.md
22
25
  - ../../../1.0/standards/backend-php.md
23
26
  ---
24
27
 
@@ -119,6 +122,31 @@ each PO then costing roughly 40 seconds — comfortably past a 5-minute schedule
119
122
  skips while processing are still listed, so accumulated `SENT/` and `OUTBOX/` objects inflate
120
123
  every run.
121
124
 
125
+ ## Step 6 — if the cron's job is to email, check the email log, not the inbox
126
+ "Did the report actually go out?" is answered by the **legacy** environment's `Logs.Emails` table
127
+ (schema `Logs`, table `Emails` — plural). `App_Email::send()` writes one row per **inline** send,
128
+ success or failure. It is **not** in the prod `Logs` schema and **not** in `Logs_<Client>`; a
129
+ prod `Logs_Compass.Email` table exists but is the 2.0 one and stays empty for 1.0 cron mail.
130
+
131
+ ```sql
132
+ # legacy env, schema Logs
133
+ SELECT id, dtSent, isSuccess, error, sourceJob, subject, `to`
134
+ FROM Emails
135
+ WHERE subject LIKE '<subject prefix>%'
136
+ ORDER BY id DESC;
137
+ ```
138
+
139
+ - `isSuccess = 0` with `error` populated → the send was attempted and failed; that is a real fault.
140
+ - **No row is not automatically a fault.** Most reporting crons only call `->send()` when they have
141
+ something to report, so a silent slot usually means "nothing to report". Confirm the run itself
142
+ happened via Step 4's `CronJobExecutions` row before calling it a failure.
143
+ - `sourceJob` is `$_SERVER['SCRIPT_NAME']`: a full `/var/www/html/crons/...` path means a
144
+ **scheduled** server run, a bare filename means someone ran it **by hand**.
145
+
146
+ Detail on what else `send()` writes (the `Common.EmailsQueued` branch, `setIsLogEnabled(false)`,
147
+ `test_mode` recipient rewriting) is in
148
+ [App_Email Side Effects & Test Mode](../../library/features/email-test-mode-and-write-free-sends.md).
149
+
122
150
  ## Deploy-time trap: a PHP 8-only syntax kills the whole cron with no visible error
123
151
  This tier runs **PHP 7.2**, so a **PHP 8 named argument** (`someFunction(items: $x)`) is a
124
152
  **parse error**, not a runtime warning:
@@ -143,6 +171,10 @@ to **2.0 only**. The 1.0 standard wins in `library/`, `worker/` and every other
143
171
  signal, it is a latent bug.
144
172
 
145
173
  ## Change history
174
+ - 2026-09-01 — Added **Step 6: check the email log, not the inbox** — a 1.0 cron's sends land in
175
+ the **legacy** env's `Logs.Emails` (schema `Logs`, table `Emails`), never prod `Logs` or
176
+ `Logs_<Client>`; gave the verification query, and noted that a missing row usually means the job
177
+ had nothing to report and that `sourceJob` distinguishes a scheduled run from a manual one. (bala)
146
178
  - 2026-08-28 — Added **"which of the 7 boxes runs it — and in which timezone"** to Step 1, after a
147
179
  production trace of the Forecast2 supporting-records cron (TRUE-81213). A `cron.worker.<role>.json`
148
180
  job runs on exactly one dynamically-assigned instance: find it via `Vision_Log.Workers` or the EC2
@@ -6,7 +6,7 @@ project: API
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-08-31
9
+ updated: 2026-09-01
10
10
  owners: ["bala", "mhammontree", "apeterson", "jcardinal", "tcox"]
11
11
  files:
12
12
  - api2/.ebextensions/git.php
@@ -22,6 +22,8 @@ related:
22
22
  - ../workflows/environment-configuration-and-provisioning.md
23
23
  - ../workflows/codepipeline-codeconnections-deploy.md
24
24
  - ../../_underscore/features/togaiq-gateway-client.md
25
+ - ./request-logging.md
26
+ - ./v2-deadlock-retry.md
25
27
  - ../../dbchanges2/workflows/framework-branch-ahead-of-schema.md
26
28
  ---
27
29
 
@@ -176,6 +178,46 @@ patched) on `TableViewFields.isGroupable`.
176
178
  the tier trades a stale-code bug for a total outage. Full procedure:
177
179
  [framework branch ahead of schema](../../dbchanges2/workflows/framework-branch-ahead-of-schema.md).
178
180
 
181
+ ## ⚠ Never use a RECENTLY-ADDED `_underscore` symbol from api2 without checking branch containment
182
+
183
+ The cross-repo ordering gotcha below says "land `_underscore` before deploying api2." That is
184
+ necessary but **not sufficient**, because every environment pins its **own** `_underscore` branch
185
+ (see the whole doc above). A framework symbol merged to `_production` may be **absent from `_beta`
186
+ and `_sandbox-dev` for weeks**, so app code that references it is fine on prod and **fatal on the
187
+ sandboxes**.
188
+
189
+ **Concrete (2026-09-01, `TRUE-81056`).** The first version of the duplicate-`transactionId` log fix
190
+ used `\_Database::ERRNO_DUPLICATE_ENTRY` inside a `catch` block. That constant was added by
191
+ `_underscore` commit **`6355d0f0`** and, verified with `git show <branch>:Database.php`:
192
+
193
+ | `_underscore` branch | `ERRNO_DUPLICATE_ENTRY` |
194
+ |---|---|
195
+ | `_production` | present |
196
+ | `_sandbox-client` | present |
197
+ | `_beta` | **absent** |
198
+ | `_sandbox-dev` | **absent** |
199
+
200
+ So on `api.beta.togahub.com` and `api.dev.sandbox.togahub.com` the code fataled with **`Undefined
201
+ constant _Database::ERRNO_DUPLICATE_ENTRY`** — *thrown from inside the catch block*, which means the
202
+ **real** error was replaced by a confusing one about the error handler. Confirmed live in
203
+ `Logs.Event` **7405** (beta) and **7404** (dev-sandbox).
204
+
205
+ **Hardening applied:** api2 now defines its own `MYSQL_ERRNO_DUPLICATE_ENTRY = 1062` and reads
206
+ `\_Database::$lastErrorNumber` through `property_exists()`, so an older framework **degrades
207
+ gracefully instead of fataling**.
208
+
209
+ **The rule, for any app repo (api2, worker2) calling into `_underscore`:**
210
+
211
+ - Before using a framework constant/method/property you (or a teammate) added recently, run
212
+ **`git branch -a --contains <commit>`** in `_underscore` and confirm it is on the branch of
213
+ **every** tier this code will run on — not just `_production`.
214
+ - When the symbol is not everywhere yet, either **wait for it to land on all tiers**, or **guard the
215
+ reference** (`property_exists()` / `defined()` / a local constant) so an older framework falls back
216
+ rather than fatals.
217
+ - **Never reference a not-yet-universal symbol from inside a `catch`/error path.** That is the worst
218
+ place for it: the fatal masks the very error you were trying to handle, and the incident is then
219
+ diagnosed against the wrong symptom.
220
+
179
221
  ## Gotchas / known issues
180
222
 
181
223
  - **⚠ Cross-repo change ordering: land `_underscore` BEFORE deploying api2.** Because api2's EB
@@ -269,7 +311,15 @@ the tier trades a stale-code bug for a total outage. Full procedure:
269
311
  documented in both places so a rotation updates both.
270
312
 
271
313
  ## Change history
272
-
314
+ - 2026-09-01 — Added the rule that **api2 must not reference a recently-added `_underscore` symbol
315
+ without checking branch containment**, since each tier pins its own framework branch. Concrete
316
+ (`TRUE-81056`): `\_Database::ERRNO_DUPLICATE_ENTRY` (added by `_underscore` `6355d0f0`) is present
317
+ only on `_production` and `_sandbox-client`, so referencing it in a `catch` block fataled with
318
+ `Undefined constant` on beta and dev-sandbox (`Logs.Event` 7405 / 7404) — and because the fatal was
319
+ *inside the catch*, it replaced the real error with a confusing one. Hardened api2 to define its own
320
+ `MYSQL_ERRNO_DUPLICATE_ENTRY = 1062` and read `\_Database::$lastErrorNumber` via `property_exists()`.
321
+ Rule: run `git branch -a --contains <commit>` on `_underscore` before using a new framework symbol,
322
+ and never reference a not-yet-universal symbol from an error path. (bala)
273
323
  - 2026-08-31 — Recorded a **worked instance** of the cross-repo ordering rule: **`client-sandbox`
274
324
  went entirely down** — every request 500'd with `Call to undefined method
275
325
  _Model_Core_AclFieldPermission::resolveFieldPermissions()`. `api2@_sandbox-client` (`V2.php:6198`,
@@ -17,6 +17,8 @@ related:
17
17
  - ../architecture.md
18
18
  - ./api-payload-interceptors.md
19
19
  - ./v2-api-error-codes.md
20
+ - ./v2-deadlock-retry.md
21
+ - ./environment-variable-drives-underscore-branch.md
20
22
  - ../../_underscore/features/error-reporting-issue-event.md
21
23
  - ../../../../clients/aig/features/entitlement-intake.md
22
24
  - ../../../../clients/compass-usa/features/mits-po-to-so-item-linking.md
@@ -149,6 +151,116 @@ transaction no matter how many times apiproxy retried. That is why the retry att
149
151
  there even in principle — the two logs are not two views of the same rows. Join the two by
150
152
  `transactionId` when you need the full attempt sequence for one user action.
151
153
 
154
+ ## ⚠ A duplicate `transactionId` must NOT fail the request — save the row, then rename on 1062
155
+
156
+ `Logs_<Client>.Api.transactionId` (and core `Logs.Api.transactionId`) carries a **UNIQUE** key, so
157
+ the log-save at the end of `execute()` is the one place where **two copies of the same request
158
+ collide**. Until 2026-09-01 that collision **turned an already-successful API response into an
159
+ HTTP 500** — the response was fully built, the work was done, and the *audit row* killed it.
160
+
161
+ **The old code was a check-then-insert** (`V2.php` ~L2283-2295 on `_production`):
162
+
163
+ ```php
164
+ // PRE-FIX — structurally cannot work under concurrency
165
+ $existingLog = new $logsApiModelName();
166
+ $existingLog->transactionId = $transactionId;
167
+ if ($existingLog->load()) {
168
+ $transactionId .= ('-' . (new \DateTime())->format('Y-m-d H:i:s.v'));
169
+ $log->transactionId = $transactionId;
170
+ }
171
+ $log->save();
172
+ ```
173
+
174
+ **Why the pre-check can never work.** The rival copy's row is still inside its **uncommitted
175
+ transaction**, so it is invisible to any `SELECT`. Both copies pass the check, both believe the id
176
+ is free, and the second `INSERT` hits the unique key with **MySQL 1062**. The exception escapes the
177
+ logger, unwinds to `Controller/Index.php`, and the caller gets a 500 for work that had already
178
+ succeeded. **No lookup can fix this** — a read cannot see a row that has not been committed yet.
179
+
180
+ **The fix (2026-09-01, `TRUE-81056` / commit `db94a26`)** — `saveApiLogUnderItsOwnTransactionId()`
181
+ and `buildResendTransactionId()` in `V2.php`: **save first, and let the database arbitrate.** On a
182
+ 1062, rename the row to `<transactionId>-1`, then `-2`, and retry. Anything that is *not* a 1062 is
183
+ logged with `error_log()` and swallowed — **a log row must never be the reason a finished request
184
+ fails.** Properties worth knowing:
185
+
186
+ - The suffix is always built from the **originally requested** id, so the third copy is `-2`, never
187
+ `-1-2`. Every copy of one user action still sorts and searches together.
188
+ - The prefix is truncated to `TRANSACTION_ID_MAX_LENGTH` (255) minus the suffix, because a silent
189
+ MySQL column-width chop would re-collide.
190
+ - The loop is **deliberately unbounded and cannot spin**: every UNIQUE column changes per attempt
191
+ (`_Model` mints a fresh `uuid` inside each `save()`, `id` is auto-increment, the resend number only
192
+ climbs), so each try eliminates one taken value and never re-asks for the same one.
193
+ - **It also removes one `SELECT` per API request** against an ~11.5M-row table.
194
+
195
+ ### The duplicate comes from BELOW the application — you cannot stop it, only tolerate it
196
+
197
+ Do not chase this as a double-click. The frontend axios interceptor
198
+ (`toga2-commerce/src/api/axiosInstance.ts`) mints a **fresh uuid per request**, so a genuine
199
+ double-click produces **two different ids**. Two requests sharing one id means the **identical bytes
200
+ were replayed by something under the app**:
201
+
202
+ - **2026-08-18 (production)** — the two copies arrived through **two different Zscaler proxy nodes**
203
+ (`136.226.49.71` and `136.226.49.80`) with **different ALB trace ids**.
204
+ - **2026-08-28** — the copies came over a plain path with **no proxy hop at all**.
205
+
206
+ A `GET` is safe to repeat, so browsers, corporate proxies and load balancers resend it on their own
207
+ when a connection drops — and they resend the original bytes, `transactionId` included. This is the
208
+ same mechanism already documented above for apiproxy's cross-region 5xx retry. **The server must
209
+ tolerate the repeat; the client cannot prevent it.**
210
+
211
+ ### An UNAUTHENTICATED request still writes a log row — so the 500 was reachable with no token
212
+
213
+ A **401 response still reaches the log-save code** and writes a row carrying the caller's
214
+ `transactionId`. That is how the defect was proven from outside with no credentials, and it made the
215
+ 500 a (minor) unauthenticated abuse vector. The fix closes it. When reproducing any log-write
216
+ behavior, remember you do **not** need a valid token to create a row.
217
+
218
+ ### ⚠ Accepted exposure — removing the 500 also removed an ACCIDENTAL write-rollback
219
+
220
+ Recorded deliberately, because it is not obvious. On an uncaught `Throwable`,
221
+ `Controller/Index.php` (~L308) rolls back the request's data changes. The 1062 escaping the logger
222
+ was therefore **silently cancelling the losing duplicate's writes**. With the fix the loser no longer
223
+ throws, so **two concurrent duplicate POSTs would both commit.**
224
+
225
+ The intended guard is the **EV-5 client-`transactionId` replay check** (`V2.php` ~L1971-1977), but it
226
+ is a `load()` on `Logs_Client_Api.transactionId` and therefore has **exactly the same
227
+ uncommitted-read blind spot** as the code that was just removed — under true concurrency it does not
228
+ see the rival either. Every recorded occurrence of this defect has been a **GET**, so nothing has
229
+ been hit.
230
+
231
+ **Status: known, accepted exposure — not a regression to revert.** The correct long-term fix is to
232
+ **claim the `transactionId` at the START of the request instead of the end**, so the UNIQUE key
233
+ enforces idempotency atomically and the loser is rejected before it does any work.
234
+
235
+ ### This is the fourth symptom of one root pattern — check-then-insert on `Logs.Api`
236
+
237
+ Treat any new `Logs.Api` incident as a member of this family before theorizing:
238
+
239
+ | Record | Symptom | What was done |
240
+ |---|---|---|
241
+ | `Logs.Issue 26` | Same defect on `/v2/users/me`, **312 occurrences** | "Fixed" 2026-08-25 (`84fd7a1`) by changing only the **fallback value** to a uuid — the race was left intact |
242
+ | `Logs.Issue 250` | The surviving `/v2/users` case | Same root cause |
243
+ | `Logs.Issue 2L` | Memory exhaustion on `/v2/auth/public` | `36ef31f` — the same pre-check full-table-scanned on a null/empty `transactionId` |
244
+ | `Logs.Issue CT` | 500 on a successful request (this one) | `db94a26` — pre-check removed entirely |
245
+
246
+ The 2026-08-25 work uuid'd the **auto-generated** log-row id, which is why the architecture doc's
247
+ gotcha reads "FIXED". It did not touch the **client-supplied** id, which still went through
248
+ check-then-insert. Both halves had to go.
249
+
250
+ ### Verifying a concurrency fix — confirm the CONSTRAINT exists in the target environment
251
+
252
+ `Logs.Api.transactionId` is **UNIQUE on production, `sandbox-client` and `dev-sandbox`, but only a
253
+ NON-unique index locally.** A local pass therefore proves nothing: the race cannot fail where the
254
+ key does not exist. **Confirm the constraint in the environment you are testing before you trust a
255
+ green result.**
256
+
257
+ The method that worked:
258
+
259
+ 1. **Reproduced on production first try** with two concurrent unauthenticated `curl`s sharing one
260
+ `transactionId`.
261
+ 2. **Proven on `client-sandbox`**: three concurrent copies of the original failing request all
262
+ returned 401, wrote rows `<id>`, `<id>-1`, `<id>-2`, and produced **zero 500s**.
263
+
152
264
  ## Getting the REAL error behind a generic `EO-1` 500 (the three-hop chain)
153
265
 
154
266
  A 500 whose only content is `EO-1` — *"Operation Failed: An internal server error occurred"* —
@@ -288,7 +400,27 @@ re-send.
288
400
  > replay operationally, not by pasting payloads into the KB.
289
401
 
290
402
  ## Change history
291
-
403
+ - 2026-09-01 — **Fixed the duplicate-`transactionId` 500** (`Logs.Issue CT`, `TRUE-81056` /
404
+ `db94a26`): the log-save at the end of `execute()` did a **check-then-insert**, which cannot work
405
+ under concurrency because the rival copy's row is still uncommitted and therefore invisible to the
406
+ `SELECT` — both copies passed, the second `INSERT` hit the UNIQUE key with **1062**, and the
407
+ exception turned an **already-successful response into a 500**. Replaced with
408
+ `saveApiLogUnderItsOwnTransactionId()` / `buildResendTransactionId()`: save first, and on 1062
409
+ rename to `<id>-1`, `-2` and retry, so the database arbitrates; any non-1062 error is `error_log`'d
410
+ and swallowed (a log row must never fail a finished request). Also drops one `SELECT` per request
411
+ against an ~11.5M-row table. Recorded that the duplicate originates **below the app** — the
412
+ toga2-commerce axios interceptor mints a fresh uuid per request, so a double-click yields two
413
+ *different* ids; the 2026-08-18 production pair arrived via two Zscaler nodes
414
+ (`136.226.49.71`/`.80`) with different ALB trace ids, the 2026-08-28 pair with no proxy hop — a GET
415
+ is safe to repeat, so proxies and load balancers resend the original bytes on their own. Recorded
416
+ that a **401 still writes a log row**, which made the 500 reachable unauthenticated. **Accepted a
417
+ known exposure**: the 500 was accidentally triggering the `Controller/Index.php` (~L308)
418
+ `Throwable` rollback of the loser's writes, and the EV-5 replay guard (`V2.php` ~L1971-1977) shares
419
+ the same uncommitted-read blind spot — every recorded case is a GET, and the real fix is to claim
420
+ the `transactionId` at the **start** of the request. Linked the family (`Logs.Issue` 26 / 250 / 2L)
421
+ and the verification rule: `Logs.Api.transactionId` is UNIQUE on prod/`sandbox-client`/`dev-sandbox`
422
+ but **non-unique locally**, so confirm the constraint before trusting a green test. Reported by
423
+ Compass Group; the defect is platform-wide. (bala)
292
424
  - 2026-09-01 — Added **Getting the REAL error behind a generic `EO-1` 500** — the three-hop chain
293
425
  from an opaque 500 to the actual failing field. Key new fact: `internalApiRequest()` logs each
294
426
  **nested** sub-request into `Logs_<Client>.Api` under a **randomly generated `transactionId`**
@@ -6,8 +6,8 @@ project: API
6
6
  client: shared
7
7
  type: feature
8
8
  status: draft
9
- updated: 2026-08-28
10
- owners: [jcardinal]
9
+ updated: 2026-09-01
10
+ owners: [jcardinal, bala]
11
11
  files:
12
12
  - api2/Controller/Index.php
13
13
  - api2/Component/Api/V2/V2.php
@@ -19,6 +19,8 @@ related:
19
19
  - ../../../../clients/compass-usa/features/asn-to-item-fulfillment.md
20
20
  - ../architecture.md
21
21
  - ./v2-api-error-codes.md
22
+ - ./request-logging.md
23
+ - ./environment-variable-drives-underscore-branch.md
22
24
  ---
23
25
 
24
26
  ## Summary
@@ -136,6 +138,22 @@ replay-safe and must stay off the list:
136
138
  idempotent or transaction-scoped, with no irreversible side effect and no consumed/mutated
137
139
  superglobal. Default is to leave a route off the list.
138
140
 
141
+ ## ⚠ `_Database::ERRNO_DUPLICATE_ENTRY` has SHIPPED — but not to every tier
142
+
143
+ Verified 2026-09-01 (`git show <branch>:Database.php` in `_underscore`): the primitives added by
144
+ this work (commit **`6355d0f0`** — `ERRNO_DUPLICATE_ENTRY = 1062`, `$lastErrorNumber`,
145
+ `$retriableUniqueRace`, `wasRetriableUniqueRace()`) are on **`_production` and `_sandbox-client`
146
+ only**. They are **absent from `_beta` and `_sandbox-dev`**.
147
+
148
+ Because api2 clones `_underscore` from a **per-environment branch**, any app-side code that names
149
+ `\_Database::ERRNO_DUPLICATE_ENTRY` **fatals with `Undefined constant` on beta and dev-sandbox** —
150
+ and if the reference sits in a `catch` block, the fatal replaces the real error with a confusing one
151
+ (seen in `Logs.Event` 7405 / 7404 on 2026-09-01). App-side callers should define their own `1062`
152
+ constant and reach `$lastErrorNumber` through `property_exists()` until the framework side is on
153
+ every branch. See
154
+ [ENVIRONMENT decides the `_underscore` branch](./environment-variable-drives-underscore-branch.md)
155
+ and [V2 request logging](./request-logging.md).
156
+
139
157
  ## Gotchas / known issues
140
158
 
141
159
  - Detection is built on the **MySQL errno** (1213/1205), not on matching the response text —
@@ -146,6 +164,13 @@ superglobal. Default is to leave a route off the list.
146
164
  not a substitute for removing the deadlock at its source.
147
165
 
148
166
  ## Change history
167
+ - 2026-09-01 — Recorded the **deployment reach** of this work's `_underscore` primitives: commit
168
+ `6355d0f0` (`ERRNO_DUPLICATE_ENTRY`, `$lastErrorNumber`, `$retriableUniqueRace`,
169
+ `wasRetriableUniqueRace()`) is on **`_production` and `_sandbox-client` only** — **absent from
170
+ `_beta` and `_sandbox-dev`** — so api2 code naming `\_Database::ERRNO_DUPLICATE_ENTRY` fatals with
171
+ `Undefined constant` on those tiers (`Logs.Event` 7405 / 7404). App-side callers must define a local
172
+ 1062 constant and read `$lastErrorNumber` via `property_exists()` until the branches catch up. No
173
+ code change to this feature. (bala)
149
174
  - 2026-08-28 — **Generalized Prong A to a second retriable class: the unique-key (1062)
150
175
  get-or-create race**, fixing the Compass ASN data-loss bug (`Logs.Issue #239`, EV-10) where a
151
176
  concurrent same-`ItemFulfillments.number` INSERT lost the race, `internalApiRequest` threw, and
@@ -6,7 +6,7 @@ project: Library
6
6
  client: nychh
7
7
  type: client-feature
8
8
  status: active
9
- updated: 2026-08-30
9
+ updated: 2026-09-01
10
10
  owners: [jcardinal]
11
11
  files:
12
12
  - library/app/api/toga2.php
@@ -134,11 +134,22 @@ Both are framework-level NetSuite facts — full detail on
134
134
  **`mainline='F'` is the WRONG filter** — NetSuite sometimes marks the first real fulfilled line
135
135
  `mainline='T'`.
136
136
 
137
- ## Two shared-engine fixes in `syncInventoryAdjustmentFromNetsuite` (surfaced here, affect any client)
137
+ ## Three shared-engine fixes in `syncInventoryAdjustmentFromNetsuite` (surfaced here, affect any client)
138
138
 
139
- Both are in the shared library method — see also the
139
+ All in the shared library method — see also the
140
140
  [per-client sync](../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md):
141
141
 
142
+ - **(c) Serial-CASE mismatch missed the existing unit → 1062 freeze (fixed 2026-09-01).** The `/units`
143
+ reconciliation lookup was keyed by the **returned** unit serial (`$unit->serialNumber`, stored `'n/a'`)
144
+ but probed with the NetSuite serial **uppercased** via `strtoupper` (`'N/A'`). PHP array keys are
145
+ case-sensitive but `Units.itemId_serialNumber` is a **case-insensitive** unique key, so the existing
146
+ unit (id 546, itemId 560, serial `'n/a'`) was missed and re-INSERTed → MySQL **1062**
147
+ ("Duplicate entry '560-N/A'") → `EV-10` → section aborted (no per-record try/catch). This froze NYCHH
148
+ `INVENTORY_ADJUSTMENTS` at `1-RUNNING`, watermark stuck 2022-12-28 (integration was recently
149
+ restarted). **Fix:** key the lookup by `strtoupper($unit->serialNumber)`. The `Items.uuid=$uuidItem`
150
+ scoping was **kept deliberately** — copying the fulfillment "serial-only" lookup would be wrong for the
151
+ shared `'N/A'` placeholder serial (would reuse another item's placeholder unit). Section self-heals on
152
+ deploy. `library` only, working-tree, not yet committed/deployed.
142
153
  - **(a) Prune-order FK violation.** The stale-item prune deleted an `InventoryAdjustmentItem` while
143
154
  its child `InventoryAdjustmentItemUnits` still referenced it (FK **RESTRICT**, no cascade) → MySQL
144
155
  **1451** → `EV-11`. **Fix:** delete the child units (`/inventory-adjustment-item-units/{uuid}`)
@@ -197,9 +208,29 @@ this feature on the scheduled cron.
197
208
  - **Empty lookups drop or duplicate items.** The item loop filters to known warehouses and creates
198
209
  items on lookup-miss, so any manual/diagnostic invocation must seed the warehouse/item/manufacturer
199
210
  lookups (the harness does).
211
+ - **A frozen NYCHH `INVENTORY_ADJUSTMENTS` section is NOT necessarily an item-matching / two-catalog
212
+ problem (verified 2026-09-01).** When the section stuck at `1-RUNNING`, item matching was suspected
213
+ (wrong of two intentional catalogs). Verified in prod it was fine: item **560**
214
+ (uuid `4c7b93d2-…`, partNumber **B11B242201**, HYBRID) is the Agilant-catalog item and the **only**
215
+ item with that part number — no duplicate second-catalog item, `$uuidItem` resolved correctly, and the
216
+ inventory-adjustment item-create already stamps catalog `Agilant`. The freeze was purely the serial-CASE
217
+ bug (fix (c) above). **Diagnosis chain (no NetSuite access):** `Client_Nychh.Parameters` (localize the
218
+ stuck section) → `Logs.Event` `clientId=3` (crash `toga2.php:6080` via
219
+ `common_sync_togasupply.php:1180` `syncInventoryAdjustmentFromNetsuite`, + `transactionId`) →
220
+ `Logs_Nychh.Api` on that `transactionId` (untruncated `EV-10`/1062 rejection + payload).
200
221
 
201
222
  ## Change history
202
223
 
224
+ - 2026-09-01 — **Unfroze NYCHH `INVENTORY_ADJUSTMENTS` — serial-CASE mismatch in the `/units` lookup
225
+ (shared `library`, working-tree, not yet committed/deployed).** `syncInventoryAdjustmentFromNetsuite`
226
+ keyed the units lookup by the returned serial (`'n/a'`) but probed with the uppercased NetSuite serial
227
+ (`'N/A'`); PHP array keys are case-sensitive while `Units.itemId_serialNumber` is case-insensitive, so
228
+ the existing unit was missed, re-INSERTed, and 1062-collided → `EV-10` → section aborted at
229
+ `1-RUNNING` (watermark frozen 2022-12-28). Fixed by keying on `strtoupper($unit->serialNumber)`; kept
230
+ the `Items.uuid=$uuidItem` scoping (the shared `'N/A'` placeholder makes serial-only wrong here). Ruled
231
+ out item-matching / two-catalog as the cause (item 560 / B11B242201 is the only Agilant-catalog item,
232
+ `$uuidItem` correct). Diagnosed entirely from `Parameters` → `Logs.Event` (clientId 3) → `Logs_Nychh.Api`
233
+ by `transactionId`, no NetSuite access. (jcardinal)
203
234
  - 2026-08-30 — Verified that NYCHH adjustments with a NULL `itemFulfillmentId`/`itemFulfillmentItemId`
204
235
  are **correct, not a bug**: 0 of 27 NYCHH adjustments populate NetSuite `custbody_stock_adjustment`
205
236
  (~1 in 40 account-wide), so most carry no link. Confirmed the shim's scalar SuiteQL read of the field
@@ -187,4 +187,9 @@ table views. Client-specific DB change-sets live in `dbchanges2/Client_Nychh/`.
187
187
  path on any branch, and neither is in the working tree — so the transfer-order **custom-field
188
188
  write permission** and the **stage status identifier** are, like the ACL layers above, prod/local
189
189
  state with no file behind them. Rewrite them before anyone builds a fresh environment.
190
+ - 🚨 **OPEN, not investigated (found 2026-09-01): recurring api2 `1054` errors for NYCHH.**
191
+ `Logs.Event` **issueId 64**, `clientId 3`: *"Database Error Occurred Error #: 1054 Unknown column
192
+ 'vendor.uuid' in 'where clause'"* on a `PurchaseOrders` GET (`_Query->getFoundRows` via
193
+ `V2.php processRoutePairs`), several occurrences ~2026-09-01 12:29. Flagged for a future look; no
194
+ change made. Likely a TableView/query join referencing `vendor.uuid` that isn't in scope.
190
195
  - This profile is a starting point; expand as more NYCHH-specific behavior is captured.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "toga-ai",
3
- "version": "1.0.731",
3
+ "version": "1.0.733",
4
4
  "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
5
5
  "keywords": [
6
6
  "claude",