toga-ai 1.0.677 → 1.0.679

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -7,7 +7,8 @@
7
7
  | [Compass Manager Approval Reminder Emails (1.0 worker crons)](features/compass-manager-approval-reminder-emails.md) | Two 1.0 worker crons nag approvers about sales orders still waiting on a decision — one per Compass tenant. | worker/crons/toga2/compass/compass_email_reminders.php, worker/crons/toga2/compasscanada/compass_email_reminders.php, worker/schedules/cron.worker.sync.json, worker1.5/crons/toga2/compass/compass_email_reminders.php, worker1.5/schedules/cron.worker.json |
8
8
  | [Compass Partial In-Transit & Delivered Emails (per package)](features/compass-partial-in-transit-delivered-emails.md) | Compass USA and Compass Canada send a **per-package** in-transit email (and a matching delivered email) instead of one email listing the whole order. | worker/crons/toga2/compass/update_salesorder_status_from_odp.php, worker/crons/toga2/compass/send_delivered_email.php, worker/crons/toga2/compasscanada/update_salesorder_status_from_odp.php, worker/crons/toga2/compasscanada/workflow/3_update_salesorder_status_from_grand_and_toy.php, worker/crons/toga2/compasscanada/send_delivered_email.php, worker/crons/toga2/compasscanada/workflow/test_partial_in_transit_email.php, worker/crons/toga2/compasscanada/workflow/test_partial_delivered_email.php, library/app/client/compasscanada.php |
9
9
  | [Elite TOGA 2.0 → TOGaDeskSupport Standalone Attachment Sync](features/elite-togadesk-attachment-sync.md) | `sync_togadesk_elite_attachments.php` is a standalone cron (every 5 minutes) that syncs file attachments from TOGA 2.0 into TOGaDeskSupport for Elite. | worker/crons/toga2/elite/sync_togadesk_elite_attachments.php, worker/crons/toga2/elite/test_sync_togadesk_elite_attachments.php |
10
- | [Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling](features/forecast2-netsuite-reconciliation.md) | CLI tools to **audit** and **repair** drift between the production `Forecast` DB (core2) and NetSuite. | test/@dave/checker.php, worker2/Component/Forecast/SaleImport/SaleImport.php, test/@dave/looper.php, test/@dave/reconcile_netsuite_totals.php, test/@dave/fixer.php, tools/bin/forecast/fixer.php, test/@dave/analyze_netsuite_forecast_diff.php, test/@dave/trueup_sales.php, test/@dave/reconcile_drift_2023plus.php, test/@dave/probe_invoice_gap_2026.php, test/@dave/probe_creditmemo_gap_detail.php, test/@dave/trueup_open_orders.php, test/@dave/loop_trueup_open_orders.php, test/@dave/trueup_opportunities.php, test/@dave/probe_sales_gap_direct.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_profit_invoices.php, test/@dave/probe_profit_gap.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_open_orders.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/schedules/cron.worker.infrastructure.json |
10
+ | [Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling](features/forecast2-netsuite-reconciliation.md) | CLI tools to **audit** and **repair** drift between the production `Forecast` DB (core2) and NetSuite. | test/@dave/checker.php, worker2/Component/Forecast/SaleImport/SaleImport.php, test/@dave/looper.php, test/@dave/reconcile_netsuite_totals.php, test/@dave/fixer.php, tools/bin/forecast/fixer.php, tools/bin/forecast/checker.php, test/@dave/analyze_netsuite_forecast_diff.php, test/@dave/trueup_sales.php, test/@dave/reconcile_drift_2023plus.php, test/@dave/probe_invoice_gap_2026.php, test/@dave/probe_creditmemo_gap_detail.php, test/@dave/trueup_open_orders.php, test/@dave/loop_trueup_open_orders.php, test/@dave/trueup_opportunities.php, test/@dave/probe_sales_gap_direct.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_profit_invoices.php, test/@dave/probe_profit_gap.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_open_orders.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/schedules/cron.worker.infrastructure.json |
11
+ | [Forecast2 Supporting-Records Nightly Import (1.0 cron) — Customers & ConsolidatedCustomers](features/forecast2-supporting-records-import.md) | The nightly **1.0** pull that keeps the Forecast2 lookup/dimension tables (`Forecast.Accounts`, `Classifications`, `Customers`, `ConsolidatedCustomers`, `Employ | worker/crons/toga2/forecast2/import_supporting_records.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker/schedules/cron.worker.infrastructure.json, worker/.ebextensions/009_setup_phpini.config, worker/crons/sync/netsuite/netsuite_customers.php, library/app/api/netsuite/rest.php, library/app/model/forecast2/customer.php, library/app/model/forecast2/consolidatedcustomer.php |
11
12
  | [NetSuite Sales Order Sales Rep Sourcing (Staples & ODP EDI orders)](features/netsuite-sales-order-sales-rep-sourcing.md) | How the **sales rep** on a NetSuite Sales Order is determined for the two 1.0 `worker` EDI order-creation integrations (Staples cXML and Compass/ODP EDI). | worker/crons/toga2/compass/workflow/5_create_netsuite_sales_orders_from_office_depot_purchase_orders.php, worker/crons/sync/staples/sync_staples_cxml.php, test/@Mark/NetSuite/TRUE_80451_customer_salesrep_diag.php |
12
13
  | [NetSuite → TOGa Supply Per-Client Sync (thin wrappers)](features/netsuite-togasupply-per-client-sync.md) | Syncs NetSuite transactions (sales orders, purchase orders, invoices, item receipts, item fulfillments, inventory adjustments) into each TOGa Supply (2.0) clien | worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_canon.php, worker/crons/toga2/netsuite/sync_togasupply_elite.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql, library/app/api/toga2.php, library/app/api/netsuite/rest.php, library/app/framework.php, library/app/systemmonitor/netsuiteintegration.php, test/@srija/Elite Testing/Service Requests/test_sync_togasupply_elite_section.php, test/@srija/Elite Testing/Service Requests/test_diagnose_togasupply_elite.php |
13
14
  | [OneUptime Server monitor + disk/memory hygiene on the 1.0 worker EB host](features/oneuptime-server-monitor-host-hygiene.md) | The 1.0 `agilant-worker` EB environment runs on the **legacy Amazon Linux 1 PHP 7.2 platform** (Apache httpd/prefork, s3fs mounts, cron) and repeatedly went dow | worker/.ebextensions/040_disk_memory_hygiene.config, worker/.ebextensions/045_oneuptime_agent.config, worker/ebs/cron.worker.php, worker/ebs/mount-s3fs-folders.php, worker/ebs/apache_settings.php, worker/ebs/setup_phpini.php |
@@ -17,4 +18,4 @@
17
18
  | [Diagnosing frozen 1.0 worker cron check-ins (Sentry "missed" flood)](workflows/diagnosing-frozen-cron-checkins.md) | When 1.0 worker cron timestamps freeze and Sentry project `worker1` fills with **`missed`** check-ins, the intuitive diagnosis — a wedged `App_Framework::isProc | worker/.ebextensions/cron.config, library/app/worker.php |
18
19
  | [isFulfillable Multi-Client Backfill (all togasupply clients)](workflows/isfulfillable-multi-client-backfill.md) | One-time backfill that catches up `Items.isFulfillable` on **existing** items across **all 17 togasupply clients** (AIG, Broward Sheriff, Canon, Endeavor Health | worker/crons/toga2/netsuite/backfill_isfulfillable_all_clients.php, library/app/api/toga2.php |
19
20
  | [Onboarding a Client to the NetSuite TOGa Supply Sync](workflows/onboarding-client-to-netsuite-togasupply-sync.md) | How to add a new TOGa 2 client to the per-client NetSuite → TOGa Supply importer (`worker/crons/toga2/netsuite/`). | worker/crons/toga2/netsuite/sync_togasupply.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_elite.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql, dbchanges2/_modules/netsuite/2026-08-05 - CLEAN NETSUITE CLINET.SQL |
20
- | [Tracing a 1.0 worker cron run in production (no stdout, silent skips)](workflows/tracing-a-worker-cron-run-in-production.md) | How to answer *"did this cron actually run, and what did it do?"* on the 1.0 `worker` tier, where **there is no usable stdout** and **a skipped run leaves no tr | worker/ebs/cron.worker.php, worker/schedules/cron.worker.sync.json, library/app/framework.php |
21
+ | [Tracing a 1.0 worker cron run in production (no stdout, silent skips)](workflows/tracing-a-worker-cron-run-in-production.md) | How to answer *"did this cron actually run, and what did it do?"* on the 1.0 `worker` tier, where **there is no usable stdout** and **a skipped run leaves no tr | worker/ebs/cron.worker.php, worker/schedules/cron.worker.sync.json, worker/schedules/cron.worker.infrastructure.json, worker/.ebextensions/009_setup_phpini.config, library/app/framework.php |
@@ -6,7 +6,7 @@ project: Worker
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-08-27
9
+ updated: 2026-08-28
10
10
  owners: [dfranks, jcardinal, kyalamarthi]
11
11
  files:
12
12
  - test/@dave/checker.php
@@ -15,6 +15,7 @@ files:
15
15
  - test/@dave/reconcile_netsuite_totals.php
16
16
  - test/@dave/fixer.php
17
17
  - tools/bin/forecast/fixer.php
18
+ - tools/bin/forecast/checker.php
18
19
  - test/@dave/analyze_netsuite_forecast_diff.php
19
20
  - test/@dave/trueup_sales.php
20
21
  - test/@dave/reconcile_drift_2023plus.php
@@ -643,9 +644,23 @@ None — Forecast2 is a single shared dataset.
643
644
  enqueuer execution log on the NetSuite side).
644
645
  - These tools live in `test/@dave/` (developer tooling), but `trueup_open_orders` has been run
645
646
  against production. The `Defaults`/checkpoint mechanics of the scheduled sync are separate.
647
+ - **⚠ SECURITY — the `tools` repo copies of these scripts hardcode PRODUCTION credentials inline.**
648
+ Found 2026-08-28: `tools/bin/forecast/fixer.php:1424` and `tools/bin/forecast/checker.php:678`
649
+ carry live production **Forecast read-replica** database credentials **inline in source**, not via
650
+ `worker/config.worker.ini`. Per the team security rule these are **committed secrets and must be
651
+ treated as compromised and rotated**; deleting the lines is not sufficient because they are in git
652
+ history. Values are deliberately **not** reproduced here, and must never be pasted into a knowledge
653
+ doc, a session file, or a ticket. Not yet actioned — **needs its own ticket**, and a senior should
654
+ sequence the rotation against whatever else reads that replica.
646
655
 
647
656
  ## Change history
648
657
 
658
+ - 2026-08-28 — **⚠ Security finding (not fixed):** the `tools` repo copies of these scripts —
659
+ `tools/bin/forecast/fixer.php:1424` and `tools/bin/forecast/checker.php:678` — hardcode **live
660
+ production Forecast read-replica credentials inline in source**, rather than reading
661
+ `worker/config.worker.ini`. Treat as compromised, rotate, and remember the values are in git history
662
+ so removing the lines is not enough. No values recorded here. Surfaced incidentally while
663
+ investigating TRUE-81213. (kyalamarthi)
649
664
  - 2026-08-27 — **Taught the nightly open-orders discrepancy-fix cron about `locationId` — and turned it
650
665
  into its own backfill (TRUE-79162).** The cron had **zero** occurrences of "location" in 775 lines
651
666
  (line SuiteQL, both SQL builders, and `$compare` all omitted it) and it delete+reinserts rows, so it
@@ -0,0 +1,210 @@
1
+ ---
2
+ title: Forecast2 Supporting-Records Nightly Import (1.0 cron) — Customers & ConsolidatedCustomers
3
+ framework: "1.0"
4
+ repo: worker
5
+ project: Worker
6
+ client: shared
7
+ type: feature
8
+ status: active
9
+ updated: 2026-08-28
10
+ owners: ["kyalamarthi"]
11
+ files:
12
+ - worker/crons/toga2/forecast2/import_supporting_records.php
13
+ - worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php
14
+ - worker/schedules/cron.worker.infrastructure.json
15
+ - worker/.ebextensions/009_setup_phpini.config
16
+ - worker/crons/sync/netsuite/netsuite_customers.php
17
+ - library/app/api/netsuite/rest.php
18
+ - library/app/model/forecast2/customer.php
19
+ - library/app/model/forecast2/consolidatedcustomer.php
20
+ related:
21
+ - ./forecast2-netsuite-reconciliation.md
22
+ - ../workflows/tracing-a-worker-cron-run-in-production.md
23
+ - ../architecture.md
24
+ - ../../../../2.0/apps/worker2/features/netsuite-supporting-record-webhook-importer.md
25
+ - ../../../../2.0/apps/worker2/features/netsuite-supporting-record-backfill-worker.md
26
+ ---
27
+
28
+ ## Summary
29
+
30
+ The nightly **1.0** pull that keeps the Forecast2 lookup/dimension tables (`Forecast.Accounts`,
31
+ `Classifications`, `Customers`, `ConsolidatedCustomers`, `Employees`) in step with NetSuite. It is the
32
+ older sibling — and the standing backstop — of the
33
+ [2.0 webhook importer](../../../../2.0/apps/worker2/features/netsuite-supporting-record-webhook-importer.md),
34
+ and it is what ultimately feeds **Power BI**'s customer dimension, including the
35
+ **Consolidated Business Name** column.
36
+
37
+ This doc goes deep on the **Customers** section, because that is where the interesting behavior
38
+ lives: it is the only section that writes a **parent** table (`ConsolidatedCustomers`) as a
39
+ side-effect of walking **children** (`Customers`), and that asymmetry is what produced TRUE-81213.
40
+
41
+ ## Key files / entry points
42
+
43
+ - **`worker/crons/toga2/forecast2/import_supporting_records.php`** — the scheduled wrapper. Sets the
44
+ `SHOULD_SYNC_*` consts (`ACCOUNTS`, `CLASSIFICATIONS`, `CUSTOMERS`, `EMPLOYEES` = true; jobs/items/
45
+ sales/open-orders/opportunities = false) and then `require`s the shared body. Wraps the run in
46
+ `App_Worker::checkIn('forecast-import-supporting-records')` / `checkOut()`.
47
+ - **`worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php`** — the shared body, one
48
+ `if (SHOULD_SYNC_X …)` block per record type, executed **in file order**:
49
+ accounts (`:85`) → classifications (`:126`) → **customers (`:193`)** → employees (`:290`) → items,
50
+ sales, opportunities, open orders (all off for this wrapper).
51
+ - **`App_Api_Netsuite_Rest::listCustomers()`** (`library/app/api/netsuite/rest.php:294`) — the source
52
+ query. **No date/`lastmodifieddate` filter**: every pass pulls *every* NetSuite customer.
53
+ - **`worker/schedules/cron.worker.infrastructure.json`** — schedule entry *"Forecast 2.0 - Import
54
+ supporting records (2:00 AM)."*, `0 2 * * *`, `active: 1`.
55
+
56
+ ## Where and when it actually runs
57
+
58
+ - The entry lives in the **`infrastructure`** role schedule, so it runs on **exactly one** of the 7
59
+ production `worker` instances — whichever currently holds that role. Role assignment is dynamic;
60
+ the live roster is `Vision_Log.Workers` (`workerName`, `workerInstanceId`, `dtHeartbeat`) and the
61
+ instances are also EC2-tagged `Worker=<ROLE>-WORKER`. Instance ids are ephemeral — a stale heartbeat
62
+ gets the instance terminated and the role re-claimed by its replacement. See
63
+ [Worker architecture](../architecture.md) and
64
+ [tracing a cron run](../workflows/tracing-a-worker-cron-run-in-production.md).
65
+ - **`0 2 * * *` is 2:00 AM CENTRAL, not UTC.** `worker/.ebextensions/009_setup_phpini.config` symlinks
66
+ `/etc/localtime` to `America/Chicago`, so `crond` — and the production MySQL `NOW()` you compare
67
+ against — are both on Central (07:00 UTC while CDT is in effect).
68
+ `periodic_forecast_discrepancy_fix.php` is scheduled at the **same instant on the same instance**.
69
+ - **Manual trigger:** `php /var/www/html/crons/toga2/forecast2/import_supporting_records.php` — the
70
+ path is exactly the `cron` value from the schedule JSON, rooted at `/var/www/html/crons/`.
71
+
72
+ ## How it works — the Customers section (`:193` onward)
73
+
74
+ Two writes happen per NetSuite customer, and **they are guarded differently**. This is the single
75
+ most important thing to understand about this cron.
76
+
77
+ 1. **Block A — the parent (`ConsolidatedCustomers`), lines ~214-253, UNGUARDED.**
78
+ The custom field `custentity10` (`NETSUITE_CUSTOM_FIELD_ID__CONSOLIDATED_CUSTOMER`) is read off the
79
+ customer, and its referenced group is upserted into `ConsolidatedCustomers` keyed on
80
+ `netsuiteInternalId` (create if unseen, rename if the name differs), then cached in
81
+ `$lookupConsolidatedCustomers`. This block has always worked.
82
+ 2. **Block B — the child link (`Customers.consolidatedCustomerId`), lines ~256-259 onward, GUARDED.**
83
+ The row is written only when it is new **or** its change guard trips.
84
+
85
+ **`ConsolidatedCustomers` is not a synced dimension table.** Nothing anywhere enumerates NetSuite's
86
+ consolidated-customer list; parents exist **only as a by-product of walking customers**. Consequences
87
+ worth internalizing before you diagnose anything here:
88
+
89
+ - The parent table is **complete and then some** — prod holds **2,742** rows with **0** NULL/empty
90
+ names against **2,729** distinct `custentity10` values actually in use. It carries a *surplus*
91
+ because **nothing ever deletes from it**.
92
+ - Therefore *"a Consolidated Business Name is blank"* is almost never a **missing group**; it is a
93
+ **missing link**. The two failures have different causes and different fixes.
94
+
95
+ ## The TRUE-81213 fix (`&&` → `||`), commit `b846e97c`
96
+
97
+ The Block B guard read:
98
+
99
+ ```php
100
+ name != entityId && consolidatedCustomerId != $consolidatedCustomerId
101
+ ```
102
+
103
+ so an existing customer was rewritten **only when BOTH changed**. Names almost never change, which
104
+ made `Forecast.Customers.consolidatedCustomerId` effectively **WRITE-ONCE** — set at first insert and
105
+ never repaired afterwards. A Consolidated Customer assigned in NetSuite *after* the customer row
106
+ already existed was never linked, and Power BI reported a blank Consolidated Business Name for it
107
+ forever. The operator is now `||` (either field differing triggers the update), with an inline
108
+ comment recording why.
109
+
110
+ **Blast radius was ~19× the ticket.** The ticket described 33 strategic accounts; the real population
111
+ was ~788. Production, before → after the first corrected run:
112
+
113
+ | Measure | Before | After |
114
+ |---|---|---|
115
+ | Customers with a `consolidatedCustomerId` | 4,062 | **4,855** (+793) |
116
+ | Strategic accounts with a blank Consolidated Business Name | 33 | **1** |
117
+
118
+ The "33" was only the visible tip: those are the strategic accounts that *also* had `accountType`
119
+ populated, and `accountType` exists for just **253 of 9,213** customers. The number that actually
120
+ predicted the outcome was NetSuite's **4,850 customers carrying a `custentity10`** — which the +793
121
+ matched almost exactly.
122
+
123
+ ## Diagnosing a blank Consolidated Business Name
124
+
125
+ 1. **Count orphan groups before anything else.** `ConsolidatedCustomers` rows with **zero** member
126
+ customers were the bug's signature (**192** of them pre-fix) — parents dutifully created by Block A
127
+ while Block B refused to link the child. A high orphan count means *broken links*; genuinely
128
+ *missing groups* would instead show up as customers whose `custentity10` has no row at all. Do not
129
+ conflate the two.
130
+ 2. **Do not use `accountType` as a proxy for coverage.** It is set on 2.7% of customers, so any
131
+ "blank" count filtered by it under-reports the real population by more than an order of magnitude.
132
+ 3. **Compare against NetSuite's `custentity10` population**, not against the ticket's row count.
133
+
134
+ ## Causes that look right and are not (both ruled out in production)
135
+
136
+ - **The SuiteQL alias-casing bug in the 2.0 handler is NOT this.**
137
+ `worker2/Worker/Netsuite/Customer.php` selects `BUILTIN.DF(custentity10) AS consolidatedName` and
138
+ reads `$record->consolidatedName`; SuiteQL lower-cases aliases, so that read is always `NULL`. It has
139
+ caused **zero** data damage: **0 NULL names across all 2,742 rows**. `custentity10` is *also* selected
140
+ **unaliased**, so the FK link still resolves; only a brand-new parent's display name could be lost,
141
+ and this nightly cron creates parents first, with correct names, via its own lowercase alias. Still a
142
+ real latent bug — just not this one.
143
+ - **`accountType` being populated does NOT mean the webhook ran.** Prod `Core.WorkerJobs` has only
144
+ `Netsuite/Customer/put` **30** + `post` **5** = **35** jobs *ever* (all since 2026-08-19), whereas
145
+ `Netsuite/CustomerAccountTypeBackfill/Backfill` ran **twice** on 2026-08-19. The backfill issues a raw
146
+ `UPDATE … SET accountType` and never touches `consolidatedCustomerId` — which is exactly why 253
147
+ customers carry an `accountType` next to a stale consolidated link.
148
+
149
+ ## Why no backfill migration was needed — and why one would have been wrong
150
+
151
+ `listCustomers()` has **no date filter**, so the corrected cron re-links the whole population on its
152
+ next pass. That is not theory: one run wrote 793 rows. **The fix is self-healing; ship the code and
153
+ wait one night.**
154
+
155
+ A `dbchanges` SQL backfill would have been the **wrong tool**, not merely redundant: the
156
+ customer→group mapping exists *only* in NetSuite's `custentity10`. The database cannot derive it, and
157
+ name-matching would have silently assigned customers to the **wrong** group. Contrast
158
+ [`accountType`](../../../../2.0/apps/worker2/features/netsuite-supporting-record-backfill-worker.md),
159
+ which genuinely did need a backfill worker — precisely because the 1.0 model does not declare that
160
+ column, so no nightly writer exists for it at all.
161
+
162
+ **Rule of thumb:** if the nightly full pull owns the column, fix the pull. Write a backfill only for a
163
+ column the pull cannot or will not write.
164
+
165
+ ## Gotchas / known issues
166
+
167
+ - **A manual run silently no-ops while another run is in flight.** `import_supporting_records.php`
168
+ opens with `if (App_Framework::isProcessRunning()) exit;` — no output, no
169
+ `Common.CronJobExecutions` row. Check `ps -ef` for the script path before concluding your manual run
170
+ did nothing. (The guard is a substring match, so an editor or `tail -f` holding that path blocks it
171
+ too — see [tracing a cron run](../workflows/tracing-a-worker-cron-run-in-production.md).)
172
+ - **Customers is the THIRD section.** A PHP warning anywhere in accounts or classifications routes to
173
+ `App_Error::handleError` → `exit` on this tier, so the process dies **before customers is ever
174
+ reached**. "Customers didn't sync" is frequently "an earlier section blew up".
175
+ - **The 1.0 `worker` app cannot be booted on a laptop** — it dies on a missing `vendor/autoload.php`.
176
+ There is no local path to running this cron; probe NetSuite from a 2.0 harness instead (see
177
+ [running worker2 locally](../../../../2.0/apps/worker2/workflows/running-worker2-locally.md)).
178
+ - **⚠ Do not confuse this pipeline with `crons/sync/netsuite/netsuite_customers.php`** — a *different*
179
+ customer sync writing `db_netsuite.NetsuiteCustomers` (which feeds `PowerBISales`), and it is
180
+ **dead**. Line 20 calls `App_NetSuite::listAllCustomers()`, which **does not exist** anywhere in
181
+ `library` (only `listCustomers`, `listAllCustomersForSalesRep`, `listAllCustomersForParentCustomer`).
182
+ It is `active: 1` every 4 hours in `cron.worker.sync.json`, so it fatals on **every** run and has been
183
+ starving its table. The same file is also insert-only (no `else` branch) and its unguarded
184
+ `foreach ($customer->customFieldList->customField …)` would abort the whole run on a customer with no
185
+ custom fields. **Not fixed — needs its own ticket.**
186
+
187
+ ## Change history
188
+
189
+ - 2026-08-28 — **TRUE-81213 fixed (`b846e97c`, verified in production).** The Block B change guard used
190
+ `&&` where it meant `||`, making `Customers.consolidatedCustomerId` write-once and blanking Power BI's
191
+ Consolidated Business Name for every customer whose group was assigned after its row existed. First
192
+ corrected run: consolidated links **4,062 → 4,855 (+793)**, blank strategic accounts **33 → 1**.
193
+ Created this doc to record what the investigation established beyond the fix: `ConsolidatedCustomers`
194
+ is a **by-product** of the unguarded Block A (complete, 2,742 rows, surplus, never deleted from), so
195
+ the bug's signature is **192 orphan groups**, not missing groups; `accountType` (253/9,213) is a
196
+ badly-biased proxy for coverage; the SuiteQL `consolidatedName` casing bug and "the webhook ran" were
197
+ both **ruled out**; and the fix is **self-healing** because `listCustomers()` has no date filter, so a
198
+ SQL backfill was unnecessary *and* would have mis-assigned groups. Also recorded where/when the cron
199
+ runs (infrastructure role, 2:00 AM **Central**) and the dead
200
+ `sync/netsuite/netsuite_customers.php` sibling. (kyalamarthi)
201
+
202
+ ## Related docs
203
+
204
+ - [NetSuite Supporting-Record Webhook Importer](../../../../2.0/apps/worker2/features/netsuite-supporting-record-webhook-importer.md)
205
+ — the 2.0 real-time replacement this cron backstops.
206
+ - [NetSuite Supporting-Record Backfill Worker](../../../../2.0/apps/worker2/features/netsuite-supporting-record-backfill-worker.md)
207
+ — the bulk reconciliation path, and when it is the wrong tool.
208
+ - [Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling](./forecast2-netsuite-reconciliation.md).
209
+ - [Tracing a 1.0 worker cron run in production](../workflows/tracing-a-worker-cron-run-in-production.md).
210
+ - [Worker (1.0 Framework) Architecture](../architecture.md) — the 7-instance role election.
@@ -6,15 +6,18 @@ project: Worker
6
6
  client: shared
7
7
  type: workflow
8
8
  status: active
9
- updated: 2026-08-25
10
- owners: ["bala"]
9
+ updated: 2026-08-28
10
+ owners: ["bala", "kyalamarthi"]
11
11
  files:
12
12
  - worker/ebs/cron.worker.php
13
13
  - worker/schedules/cron.worker.sync.json
14
+ - worker/schedules/cron.worker.infrastructure.json
15
+ - worker/.ebextensions/009_setup_phpini.config
14
16
  - library/app/framework.php
15
17
  related:
16
18
  - ../architecture.md
17
19
  - ./diagnosing-frozen-cron-checkins.md
20
+ - ../features/forecast2-supporting-records-import.md
18
21
  - ../../library/features/cron-execution-monitoring.md
19
22
  - ../../../1.0/standards/backend-php.md
20
23
  ---
@@ -41,6 +44,31 @@ frequency makes you read a normal gap as an outage.
41
44
  Also check whether the job is registered in the env you are testing in at all: this one is **not**
42
45
  in `cron.beta.json`, so beta never runs it on a schedule. Nothing on beta is evidence about it.
43
46
 
47
+ ### Which of the 7 boxes runs it — and in which timezone
48
+
49
+ Two facts you need before you go looking for evidence, both easy to get wrong:
50
+
51
+ - **The role file names the instance.** A job in `cron.worker.<role>.json` runs on **exactly one**
52
+ instance: whichever currently holds that role. There is **no static instance→role mapping** —
53
+ assignment is dynamic and claim-the-first-free-slot (see
54
+ [Worker architecture](../architecture.md)). To find today's box: read
55
+ **`Vision_Log.Workers`** (`workerName`, `workerInstanceId`, `dtHeartbeat`), or look for the EC2 tag
56
+ **`Worker=<ROLE>-WORKER`**. Instance ids are **ephemeral** — a stale heartbeat gets the instance
57
+ terminated and its role re-claimed by the replacement, so an instance id you noted last week is
58
+ probably meaningless today. Worked example: `toga2/forecast2/import_supporting_records.php` is in
59
+ `cron.worker.infrastructure.json`, so only the current **`infrastructure`** worker ever runs it.
60
+ - **⚠ The crontab runs on CENTRAL time, not UTC.** `.ebextensions/009_setup_phpini.config` symlinks
61
+ `/etc/localtime` to `America/Chicago`, so `0 2 * * *` means **2:00 AM Central** (07:00 UTC under
62
+ CDT). The production MySQL also returns Central from `NOW()`, so timestamps you pull back agree
63
+ with the schedule — but anything you compare against a UTC source (CloudWatch, an AWS console, a
64
+ NetSuite timestamp) is off by 5–6 hours. Most "the cron didn't run at the time I expected"
65
+ reports are this.
66
+
67
+ **To run it by hand:** `php /var/www/html/crons/<path exactly as it appears in the schedule JSON>` —
68
+ e.g. `php /var/www/html/crons/toga2/forecast2/import_supporting_records.php`. Do this on the
69
+ role-holding instance, and read Step 4's overlap-guard note first: a manual run **silently exits**
70
+ if a scheduled run is already in flight.
71
+
44
72
  ## Step 2 — echo/print goes nowhere useful, so do not debug with it
45
73
  The crontab line is assembled in `worker/ebs/cron.worker.php`, and the two blocks that build it
46
74
  differ:
@@ -115,6 +143,15 @@ to **2.0 only**. The 1.0 standard wins in `library/`, `worker/` and every other
115
143
  signal, it is a latent bug.
116
144
 
117
145
  ## Change history
146
+ - 2026-08-28 — Added **"which of the 7 boxes runs it — and in which timezone"** to Step 1, after a
147
+ production trace of the Forecast2 supporting-records cron (TRUE-81213). A `cron.worker.<role>.json`
148
+ job runs on exactly one dynamically-assigned instance: find it via `Vision_Log.Workers` or the EC2
149
+ tag `Worker=<ROLE>-WORKER`, never a remembered instance id. And **the crontab runs on Central**
150
+ (`.ebextensions/009_setup_phpini.config` symlinks `/etc/localtime` to `America/Chicago`), as does
151
+ production MySQL `NOW()` — so `0 2 * * *` is 2:00 AM Central, and any comparison against a UTC
152
+ source is off by 5–6 hours. Recorded the manual-trigger form
153
+ `php /var/www/html/crons/<schedule-json path>` and the fact that it silently exits under the overlap
154
+ guard. (kyalamarthi)
118
155
  - 2026-08-25 — Documented from a prod diagnostic session on the Compass ODP 850 importer: the
119
156
  schedule JSON (not the file header) is authoritative, worker cron stdout is unreadable (base
120
157
  block's `WORKER_ERROR` redirect commented out, role block writes a random filename),
@@ -6,7 +6,7 @@ project: Worker
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-08-19
9
+ updated: 2026-08-28
10
10
  owners: ["kyalamarthi"]
11
11
  files:
12
12
  - worker2/Worker/Netsuite/CustomerAccountTypeBackfill.php
@@ -17,6 +17,7 @@ files:
17
17
  related:
18
18
  - ./netsuite-supporting-record-webhook-importer.md
19
19
  - ../workflows/running-worker2-locally.md
20
+ - ../../../../1.0/apps/worker/features/forecast2-supporting-records-import.md
20
21
  - ../../../../1.0/apps/library/features/netsuite-suiteql-api-reference.md
21
22
  ---
22
23
 
@@ -79,6 +80,31 @@ only ongoing writer** — and:
79
80
  - **Therefore: after any bulk reclassification in NetSuite, re-run this backfill.** It is the only
80
81
  thing that will bring `Forecast` back into agreement.
81
82
 
83
+ ## When a backfill is the WRONG tool
84
+
85
+ The test is **"does an existing writer already own this column, over the whole population?"** — not
86
+ "is the data currently wrong?".
87
+
88
+ - **A full, unfiltered nightly pull makes the fix self-healing — ship the code and wait one night.**
89
+ TRUE-81213 (blank Consolidated Business Names) looked exactly like backfill work, and was not:
90
+ `App_Api_Netsuite_Rest::listCustomers()` pulls **every** NetSuite customer with **no date filter**, so
91
+ the corrected cron re-linked **793 rows on its very next pass**. No migration, no worker.
92
+ - **Worse than redundant — a SQL backfill would have been actively wrong there.** The customer→group
93
+ mapping exists **only** in NetSuite's `custentity10`; the database cannot derive it, so the only
94
+ available SQL heuristic (name matching) would have silently assigned customers to the **wrong**
95
+ consolidated group.
96
+ - **`accountType` is the contrasting case that genuinely needs this worker:** the 1.0 model does not
97
+ declare the column, so **no** nightly writer exists — there is nothing to self-heal.
98
+
99
+ ### Corollary: a populated column does NOT prove the webhook ran
100
+
101
+ Do not infer webhook coverage from data being present. Prod `Core.WorkerJobs` holds only
102
+ `Netsuite/Customer/put` **30** + `post` **5** = **35** customer jobs *ever* (all since 2026-08-19),
103
+ while `Netsuite/CustomerAccountTypeBackfill/Backfill` ran **twice** on 2026-08-19 — and **this backfill
104
+ issues a raw `UPDATE … SET accountType`, touching nothing else**. That is precisely why 253 customers
105
+ carried a correct `accountType` alongside a stale `consolidatedCustomerId`, which is the misreading
106
+ that sent TRUE-81213's investigation down the wrong path. **Count the jobs; do not infer them.**
107
+
82
108
  ## Production results (2026-08-19)
83
109
 
84
110
  - **248 of 9,183** NetSuite customers have an Account Type set — **2.7%**. The other 97% are legitimately
@@ -87,13 +113,14 @@ only ongoing writer** — and:
87
113
 
88
114
  ## Gotchas / known issues
89
115
 
90
- - **⚠ The 1.0 nightly cron is effectively INSERT-ONLY for customers — latent bug, NOT fixed.**
91
- `worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php:256-259` guards the update with
92
- `name != … && consolidatedCustomerId != …` where it plainly means `||`, so an existing row is only
93
- rewritten when **both** values changed. Renames therefore never propagate from NetSuite. Do not
94
- assume the nightly cron will heal a stale `Forecast.Customers` row — it will not.
95
- Evidence (prod, 2026-08-19): Forecast holds **9,206** customers vs **9,183** in NetSuite, i.e. 23
96
- stale rows that were never pruned either.
116
+ - **✅ The 1.0 nightly cron was effectively INSERT-ONLY for customers — FIXED 2026-08-28.**
117
+ `worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php:256-259` guarded the update with
118
+ `name != … && consolidatedCustomerId != …` where it meant `||`, so an existing row was rewritten only
119
+ when **both** values changed. TRUE-81213 / commit `b846e97c` changed it to `||`; verified in
120
+ production. The cron now **does** heal a stale `name` or `consolidatedCustomerId`
121
+ (**not** `accountType` — see above, the 1.0 model does not declare it). Historical evidence of the
122
+ bug (prod, 2026-08-19): Forecast held **9,206** customers vs **9,183** in NetSuite; the cron still
123
+ never prunes deleted customers.
97
124
  - **`dryRun` is `true` by default** — a run that "did nothing" almost certainly just needed
98
125
  `dryRun=false`. Read the returned summary before concluding the data was already correct.
99
126
  - **SuiteQL lower-cases every column alias**, so a camelCase alias read back as camelCase is silently
@@ -106,6 +133,15 @@ only ongoing writer** — and:
106
133
 
107
134
  ## Change history
108
135
 
136
+ - 2026-08-28 — Added **"When a backfill is the WRONG tool"** after TRUE-81213 proved the point: a
137
+ blank-Consolidated-Business-Name defect that looked like backfill work was **self-healing**, because
138
+ `listCustomers()` pulls the full population with no date filter (the corrected cron re-linked 793 rows
139
+ in one pass) — and a SQL backfill would have been *actively wrong*, since the customer→group mapping
140
+ lives only in NetSuite's `custentity10` and name-matching would mis-assign groups. Also recorded the
141
+ corollary that **a populated column does not prove the webhook ran** (35 customer webhook jobs ever,
142
+ vs. two runs of this backfill's raw `UPDATE … SET accountType`), and marked the 1.0 cron's
143
+ `&&`-should-be-`||` guard **fixed** (`b846e97c`) — it is now a real backstop for `name` and
144
+ `consolidatedCustomerId`. (kyalamarthi)
109
145
  - 2026-08-19 — Created after building `_Worker_Netsuite_CustomerAccountTypeBackfill` (TRUE-80206) on
110
146
  the `ItemFulfillableBackfill` pattern: recorded the batching recipe (page 1,000 / SuiteQL `IN` 500 /
111
147
  UPDATE 500), the `dryRun`-defaults-to-true and countable-summary-on-every-path contract, and the
@@ -123,4 +159,6 @@ only ongoing writer** — and:
123
159
  - [Running worker2 locally against real NetSuite](../workflows/running-worker2-locally.md) — how this was
124
160
  tested before it touched production.
125
161
  - [NetSuite SuiteQL/REST API Reference](../../../../1.0/apps/library/features/netsuite-suiteql-api-reference.md)
162
+ - [Forecast2 Supporting-Records Nightly Import (1.0 cron)](../../../../1.0/apps/worker/features/forecast2-supporting-records-import.md)
163
+ — the full pull that made TRUE-81213 self-healing, and the orphan-group diagnostic.
126
164
  — SuiteQL mechanics, `BUILTIN.DF`, the lower-cased-alias rule, and the custom-field map.
@@ -6,7 +6,7 @@ project: Worker
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-08-19
9
+ updated: 2026-08-28
10
10
  owners: ["dfranks", "bala", "kyalamarthi"]
11
11
  files:
12
12
  - worker2/Worker/Netsuite/Employee.php
@@ -36,6 +36,7 @@ related:
36
36
  - ../../_underscore/features/forecast-sale-import.md
37
37
  - ../../_underscore/features/netsuite-rest-client.md
38
38
  - ../architecture.md
39
+ - ../../../../1.0/apps/worker/features/forecast2-supporting-records-import.md
39
40
  ---
40
41
 
41
42
  ## Summary
@@ -95,8 +96,9 @@ recipe and its two recurring variants so a new one is a fill-in-the-blanks job,
95
96
  records, the parent/supervisor backfill). It shares `import_supporting_records.php` with the other
96
97
  sections, so **never disable the wrapper.**
97
98
  ⚠ **But do not assume the backstop covers your new column** — see *the backstop is narrower than it
98
- looks* under Gotchas. A column the 1.0 model does not declare has **no** nightly writer at all, and
99
- for Customers the cron is effectively insert-only. When there is no backstop, ship a
99
+ looks* under Gotchas. A column the 1.0 model does not declare has **no** nightly writer at all.
100
+ (Customers' cron *was* effectively insert-only until TRUE-81213 — fixed 2026-08-28.)
101
+ When there is no backstop, ship a
100
102
  [backfill worker](./netsuite-supporting-record-backfill-worker.md) with the feature.
101
103
 
102
104
  ## Recurring variants
@@ -119,7 +121,8 @@ recipe and its two recurring variants so a new one is a fill-in-the-blanks job,
119
121
  `custentity_account_type` (NetSuite Entity Field "Account Type", internal id **10719**) maps
120
122
  **1 = Strategic Emerging · 2 = Not Strategic · 3 = Strategic Growth · 4 = Strategic Maintain ·
121
123
  5 = Strategic Draining**. Power BI reads `Forecast.Customers.accountType` directly, so a raw id
122
- renders as `3` on the report. Only **248 of 9,183** customers (2.7%) have the field set at all —
124
+ renders as `3` on the report. Only **248 of 9,183** customers (2.7%) have the field set at all
125
+ (**253 of 9,213** as of 2026-08-28) —
123
126
  mostly-blank is correct, not a symptom. If you are unsure of a custom field's type, let the
124
127
  [backfill worker's dry run probe it](./netsuite-supporting-record-backfill-worker.md).
125
128
  - **A column with TWO writers must agree byte-for-byte on format** (Customers `name`). SOAP returned a
@@ -141,10 +144,17 @@ recipe and its two recurring variants so a new one is a fill-in-the-blanks job,
141
144
  aliases, or read the lower-cased key.** This has now caused two separate bugs:
142
145
  - it broke the first cut of the account-type backfill — every customer was misreported as "blank in
143
146
  NetSuite" **while the job reported success**; only testing caught it;
144
- - **still open, not fixed:** `worker2/Worker/Netsuite/Customer.php` (~line 147) selects
145
- `BUILTIN.DF(custentity10) AS consolidatedName` and reads `$record->consolidatedName`, so
146
- **consolidated-customer names are never set or refreshed** by the webhook. Fix it the next time that
147
- handler is opened.
147
+ - **still open in `_production`, but measured harmless (2026-08-28):**
148
+ `worker2/Worker/Netsuite/Customer.php` (~line 147) selects
149
+ `BUILTIN.DF(custentity10) AS consolidatedName` and reads `$record->consolidatedName`, so the
150
+ webhook never sets a consolidated-customer **name**. Production damage is **zero — 0 NULL/empty
151
+ names across all 2,742 `Forecast.ConsolidatedCustomers` rows** — because `custentity10` is *also*
152
+ selected **unaliased**, so the FK link still resolves, and the nightly 1.0 cron creates every parent
153
+ first with a correct name via its own lowercase `consol_name` alias. Only a brand-new parent seen by
154
+ the webhook *before* the cron could lose its display name. So: still worth fixing, but it is **not**
155
+ a live data-loss path and it was **not** the cause of TRUE-81213 (see below).
156
+ ⚠ The fix currently exists **only as an uncommitted working-tree edit** on branch `TRUE-80742`,
157
+ which has no commits of its own — it is committed to no branch and will be lost by a checkout.
148
158
  - **SuiteQL over the REST client has NO server-side placeholder binding.** `_Component_Api_Netsuite::send`
149
159
  passes the query string as-is, so the **`(int)` cast on an already-int id is the injection control** —
150
160
  the same codebase-wide idiom used by `lookupId()` and the Item/Opportunity/Employee handlers. A
@@ -163,14 +173,22 @@ recipe and its two recurring variants so a new one is a fill-in-the-blanks job,
163
173
  covers columns the **1.0** model declares. `App_Model_Forecast2_Customer`
164
174
  (`library/app/model/forecast2/customer.php`) does not declare `accountType`, so the 2 AM
165
175
  "Forecast 2.0 — Import supporting records" cron can neither write **nor clobber** it — the webhook is
166
- the **only** ongoing writer. Worse, for Customers the cron is effectively **insert-only**: its change
167
- guard (`common_import_sales_from_netsuite.php:256-259`) uses `name != … && consolidatedCustomerId != …`
168
- where it means `||`, so an existing row is rewritten only when *both* changed and renames never
169
- propagate (prod: 9,206 Forecast customers vs 9,183 in NetSuite). **Latent bug, not fixed.** And because
176
+ the **only** ongoing writer. And because
170
177
  a NetSuite **Mass Update / CSV import without "Run Server SuiteScript"** fires no User Event, a bulk
171
178
  reclassification reaches neither writer. Ship a
172
179
  [backfill worker](./netsuite-supporting-record-backfill-worker.md) for any column in this position and
173
180
  re-run it after every bulk change.
181
+ - **✅ RESOLVED 2026-08-28 — the customer cron is no longer insert-only.** This doc previously recorded
182
+ the `&&`-should-be-`||` change guard at
183
+ `worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php:256-259` as a **latent bug, not
184
+ fixed**. It is **fixed** — TRUE-81213, commit **`b846e97c`**, deployed and **verified in production**.
185
+ The guard rewrote an existing customer only when `name` **AND** `consolidatedCustomerId` had *both*
186
+ changed, so `consolidatedCustomerId` was effectively **write-once** and Power BI showed a blank
187
+ Consolidated Business Name for every customer whose group was assigned after its row existed. First
188
+ corrected run: consolidated links **4,062 → 4,855 (+793)**, blank strategic accounts **33 → 1**.
189
+ The nightly cron therefore now *is* a genuine backstop for `name` and `consolidatedCustomerId`
190
+ (`accountType` remains unbacked — that part of the bullet above still stands). Full write-up:
191
+ [Forecast2 supporting-records nightly import](../../../../1.0/apps/worker/features/forecast2-supporting-records-import.md).
174
192
  - **⚠ Unrelated observation, production, updated 2026-08-19: the item job types are still at 0% success,
175
193
  the cause is now known, and the volume is growing.** `Core.WorkerJobs` in prod shows
176
194
  `Netsuite/InventoryItem/post|put` at **~1,163 failures / 0 successes**, all with
@@ -182,6 +200,17 @@ recipe and its two recurring variants so a new one is a fill-in-the-blanks job,
182
200
 
183
201
  ## Change history
184
202
 
203
+ - 2026-08-28 — **The legacy customer cron's `&&`-should-be-`||` change guard is FIXED** (TRUE-81213,
204
+ commit `b846e97c`, verified in production) — this doc had it recorded as "latent bug, not fixed".
205
+ It had made `Forecast.Customers.consolidatedCustomerId` **write-once**, blanking Power BI's
206
+ Consolidated Business Name; the corrected cron self-healed **+793** links (4,062 → 4,855) on its first
207
+ run and took blank strategic accounts from **33 → 1**. So the nightly cron *is* now a real backstop for
208
+ `name` + `consolidatedCustomerId` (still not for `accountType`). Also **measured** the still-open
209
+ `consolidatedName` alias-casing bug: **zero** production damage (0 NULL names in 2,742 rows) because
210
+ `custentity10` is selected unaliased and the cron creates parents first — and noted that its fix is
211
+ sitting uncommitted on branch `TRUE-80742`. Deep dive lives in the new
212
+ [1.0 supporting-records import doc](../../../../1.0/apps/worker/features/forecast2-supporting-records-import.md).
213
+ (kyalamarthi)
185
214
  - 2026-08-19 — **Customers went live for the first time (TRUE-80206)** and produced four additions.
186
215
  (1) **Root cause of the whole ticket:** the `customscript_ue_amq_enqueue` deployment for Customer
187
216
  (deployment internal id 9, `customdeploy9`) sat at `Status = TESTING` / `isdeployed = F` while all 15
@@ -213,6 +242,8 @@ recipe and its two recurring variants so a new one is a fill-in-the-blanks job,
213
242
 
214
243
  - [NetSuite Supporting-Record Backfill Worker](./netsuite-supporting-record-backfill-worker.md) — the bulk
215
244
  reconciliation counterpart; ship one whenever the legacy cron does not back your column.
245
+ - [Forecast2 Supporting-Records Nightly Import (1.0 cron)](../../../../1.0/apps/worker/features/forecast2-supporting-records-import.md)
246
+ — the legacy backstop this recipe replaces, and the `ConsolidatedCustomers` linkage in detail.
216
247
  - [NetSuite → TOGA Opportunity Sync](./netsuite-opportunity-sync.md) — header+children transaction sibling.
217
248
  - [NetSuite → Forecast Open-Orders Sync](./netsuite-salesorder-open-orders-sync.md).
218
249
  - [Forecast.Sales NetSuite import engine](../../_underscore/features/forecast-sale-import.md) — the
@@ -5,7 +5,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
5
5
  ## 1.0 framework
6
6
 
7
7
  - **library** (Library) _(framework core)_ — 20 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
8
- - **worker** (Worker) — 28 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
8
+ - **worker** (Worker) — 29 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
9
9
  - **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
10
10
  - **worker1.5** (Worker 1.5) — 0 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
11
11
  - **togadesk** (TOGa Desk) — 13 doc(s) → [1.0/apps/togadesk/INDEX.md](1.0/apps/togadesk/INDEX.md)
@@ -40,6 +40,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
40
40
  - **websocket** (WebSocket Server) — 2 doc(s) → [standalone/apps/websocket/INDEX.md](standalone/apps/websocket/INDEX.md)
41
41
  - **forward** (Forwarder) — 3 doc(s) → [standalone/apps/forward/INDEX.md](standalone/apps/forward/INDEX.md)
42
42
  - **claude** (Claude Harness) — 8 doc(s) → [standalone/apps/claude/INDEX.md](standalone/apps/claude/INDEX.md)
43
+ - **powerbi** (Power BI) — 0 doc(s) → [standalone/apps/powerbi/INDEX.md](standalone/apps/powerbi/INDEX.md)
43
44
 
44
45
  ## Clients
45
46
 
@@ -8,11 +8,13 @@ apps:
8
8
  - dbchanges2
9
9
  - api2
10
10
  - togatech
11
+ - worker
12
+ - library
11
13
  project: _Underscore
12
14
  client: true
13
15
  type: profile
14
16
  status: active
15
- updated: 2026-08-13
17
+ updated: 2026-08-28
16
18
  owners: [jcardinal, kyalamarthi, tcox]
17
19
  files: []
18
20
  related:
@@ -236,5 +236,12 @@
236
236
  "framework": "standalone",
237
237
  "role": "app",
238
238
  "dependsOn": []
239
+ },
240
+ {
241
+ "repo": "powerbi",
242
+ "project": "Power BI",
243
+ "framework": "standalone",
244
+ "role": "app",
245
+ "dependsOn": []
239
246
  }
240
247
  ]
@@ -0,0 +1,87 @@
1
+ ---
2
+ type: session
3
+ slug: bdr-blox-talos-drawer
4
+ title: BDR landing drawer rendered by the shared blox Talos component
5
+ author: tcox
6
+ repos: [bdr, toga-blox]
7
+ framework: "2.0"
8
+ client: shared
9
+ created: 2026-08-28
10
+ updated: 2026-08-28
11
+ status: active
12
+ ---
13
+
14
+ # Session: bdr-blox-talos-drawer
15
+ **Date:** 2026-08-28
16
+ **Project/Repo:** bdr (`C:\WWW\BDR`) + toga-blox (`C:\WWW\toga-blox-npm`, branch `TRUE-80692`) (2.0)
17
+ **Task:** Make the BDR `/landing` Talos chat drawer render through the shared `@agilant/toga-blox` Talos component (TRUE-80965/80966) while looking identical to the hand-rolled ChatPanel, by adding an opt-in `drawer` variant + feature toggles to blox rather than a BDR-side bridge.
18
+
19
+ ---
20
+
21
+ ## What WORKED
22
+ <!-- Include specific file paths and evidence -->
23
+ - blox `variant="drawer"` in `toga-blox-npm/src/components/Talos/{types.ts,theme.ts,helpers.tsx,TalosMessage.tsx,TalosPanel.tsx,Talos.module.css}` (uncommitted on `TRUE-80692`). Evidence: headless-Chrome computed-style diff vs the old ChatPanel is identical for drawer/head/title/body/bubble/chip/btn/input/send/foot/scrim; screenshots indistinguishable.
24
+ - `TalosResponse` extended with `message` (light markdown), `suggestions[]`, `cta` so BDR's `createTalosChatAdapter` plugs in directly. Evidence: typed send on `/landing` hit api.togaiq.com (session mint 200, chat 200 in ~6-7s), reply rendered with bold + chips from backend suggestions.
25
+ - `features` toggles + optional `expanded`/`width` + `footer`/`titleIcon`/`closeIcon`/`sendIcon` slots + `branding.avatarNode` + 13 theme keys. Docked mode regression (temporary route, then deleted): hero, 4 default tiles, stub reply w/ bold, user-initials avatar, history/new chat/reopen, expand+scrim (`onDockWidthChange` 0), re-dock (380), keyboard resize (+16), ESC unmount, launcher reopen, zero console errors.
26
+ - BDR `src/components/TalosDrawer.tsx` (new host wrapper) + `SecurityLanding.tsx` ChatPanel→TalosDrawer swap. Evidence: `/landing` header CTA opens the drawer, "Call me now" routes to SecurityCallNow, cursor-following light works on the portaled footer buttons (`--mx/--my` set, `::after` opacity 1). tsc, lint, 385/385 vitest, `next build` green.
27
+ - Booking link fixed: `src/content/security.ts primaryCtaHref` → `https://cal.com/team/toga-technology/free-security-scan` (curl 200).
28
+ - Local link recipe (documented in `2.0/apps/toga-blox/workflows/local-link-into-a-consumer-app.md`): junction `BDR/node_modules/@agilant/toga-blox -> C:\WWW\toga-blox-npm`; inside blox, `node_modules/react`+`react-dom` replaced by junctions to BDR's React 19 (originals kept as `react.r18`/`react-dom.r18`); blox `postcss.config.cjs` renamed `.off`; import the Talos SUBPATH `@agilant/toga-blox/dist/components/Talos/index.js`.
29
+ - Blox's 40 failing tests are pre-existing (same count with Talos changes stashed and under React 18).
30
+ - `/capture` pushed `484e2a6` to `_main` (5 docs + session doc).
31
+
32
+ ## What did NOT work — DO NOT RETRY THESE
33
+ <!-- Exact failure reasons — do not vague-ify -->
34
+ - Importing the package barrel `@agilant/toga-blox` in Next: Turbopack build fails `Cannot find module 'tailwindcss'` while processing `dist/components/AdvancedSelect/AdvancedSelect.module.css` etc. — the barrel drags every blox component's CSS in and Next resolves blox's `postcss.config.cjs`. Use the Talos subpath import.
35
+ - Keeping blox `postcss.config.cjs` in place even with the subpath import: same `Cannot find module 'tailwindcss'` — Next walks up from the linked `Talos.module.css` into the blox checkout. Must stay renamed while linked (published tarball ships only `dist` + README, so not a prod issue).
36
+ - `cmd /c mklink /D` for the BDR symlink: "You do not have sufficient privilege". Use `mklink /J` (junction) or `npm install --no-save <folder>`.
37
+ - Node-side probe of the Talos API with a forged `Origin: http://localhost:3000` header: session mint 403 — the origin gate only passes for a real browser on localhost:3000; the adapter fell back to scripted replies. Test in a browser, not node.
38
+ - Running `next dev` while another instance holds `.next/dev` lock: refused to start ("Run taskkill /PID … to stop it"). Kill the listener by PID from `netstat`, never `taskkill /IM node.exe`.
39
+ - Hitting `/talos-*` routes during `npm run build` in blox: 500 with "Failed to read source code from …dist/components/Talos/index.js" because blox `prebuild` does `rm -rf dist`; recovers on next request.
40
+ - Long multi-command bash heredoc chains (CSS append + mklink + build in one call): "unexpected EOF while looking for matching `'". Split into separate calls.
41
+ - A theme `surface` key alone does not make the docked panel dark: `.field` gradient + `--talos-suggest-bg` / `--talos-talos-bubble-bg` were hardcoded rgba whites (now exposed as theme keys for the drawer; docked still uses `fieldBg`).
42
+
43
+ ## Not tried yet (candidates for next session)
44
+ - Commit blox changes on `TRUE-80692` (or a branch off it) and get apeterson's review; a PR onto `TRUE-80692` vs. a note first is tcox's call.
45
+ - Publish blox with Talos (needs React 19 peer decision: widen `peerDependencies.react` to `^18 || ^19`), then `npm install @agilant/toga-blox@<tag>` in BDR and remove the junction — BDR cannot deploy until then.
46
+ - Delete `BDR/src/components/ChatPanel.tsx` + its `.chatveil/.chatpanel*` CSS block in `security.css` once signed off (kept for one-line rollback).
47
+ - Verify blox `RichText` italic: a live reply rendered "for *your*environment" with the space after `<em>` apparently dropped; old ChatPanel showed literal asterisks.
48
+ - Optional: render `cta` in the drawer (`features.cta`) once backend `cta_url` is aligned with the Cal.com link.
49
+ - Cross-link the Next/Turbopack link hazard from `2.0/standards/frontend.md` §11 (senior owner).
50
+
51
+ ## Current file state
52
+ | File | Status | Notes |
53
+ |------|--------|-------|
54
+ | `toga-blox-npm/src/components/Talos/types.ts` | modified, uncommitted | variant/features/welcome/slots, TalosResponse additions, `paras` optional |
55
+ | `toga-blox-npm/src/components/Talos/theme.ts` | modified, uncommitted | 13 new keys + TOKEN_MAP |
56
+ | `toga-blox-npm/src/components/Talos/helpers.tsx` | modified, uncommitted | `toMarkdownBlocks`, `TalosMarkdown` |
57
+ | `toga-blox-npm/src/components/Talos/TalosMessage.tsx` | modified, uncommitted | variant-aware classes, avatarNode, markdown |
58
+ | `toga-blox-npm/src/components/Talos/TalosPanel.tsx` | modified, uncommitted | drawer render path, feature gating, chips, footer slot |
59
+ | `toga-blox-npm/src/components/Talos/Talos.module.css` | modified, uncommitted | `.chips`, `.footerSlot`, `.veil/.drawer/.d*` blocks |
60
+ | `toga-blox-npm/postcss.config.cjs` | renamed to `.off` (dev-only, shows as deleted) | restore before committing |
61
+ | `toga-blox-npm/node_modules/react{,-dom}` | junctions to BDR React 19 (dev-only) | originals at `react.r18`/`react-dom.r18` |
62
+ | `toga-blox-npm/dist/` | rebuilt with drawer | consumed by BDR via junction |
63
+ | `BDR/node_modules/@agilant/toga-blox` | junction → `C:\WWW\toga-blox-npm` | not in package.json |
64
+ | `BDR/src/components/TalosDrawer.tsx` | new, uncommitted | host wrapper |
65
+ | `BDR/src/flow/screens/SecurityLanding.tsx` | modified, uncommitted | ChatPanel → TalosDrawer |
66
+ | `BDR/src/components/ChatPanel.tsx` | unchanged, now unused | delete after sign-off |
67
+ | `BDR/src/content/security.ts` | modified (Cal.com href) | may already be committed by tcox |
68
+ | `BDR/.env.local` | gitignored | commented `TOGA_SECURITY_CAMPAIGN_UUID` + go-live checklist |
69
+ | `BDR/src/app/talos-blox`, `talos-docked-check` | deleted | scratch routes |
70
+
71
+ ## Decisions made
72
+ - Extend blox with a `drawer` variant + props instead of a BDR-side response bridge — tcox: "we should only have added on or used what was there"; keeps BDR a pure consumer. Rejected: `talosBloxAdapter.ts` mapping layer (built, then deleted).
73
+ - Drawer stays mounted while closed (slide-out animation parity); docked keeps unmount-on-close.
74
+ - Drawer defaults `features.cta=false` so replies look exactly like the old drawer (which ignored `cta`).
75
+ - Theme values restated as literals in `TalosDrawer.tsx` because the portal lands outside `.security-shell`; footer buttons wrapped in a `.security-shell` (display:contents) div so shell `.btn` rules apply.
76
+ - Junction + single React for local dev rather than editing `next.config` (no project-file change for a dev-only concern).
77
+ - Booking goes straight to Cal.com (no schedule screen), per earlier design decision.
78
+
79
+ ## Blockers
80
+ - BDR deploy blocked until blox TRUE-80692 + drawer are merged and published (React 19 peer to settle). Local link only.
81
+ - Blox changes live only in tcox's working tree; apeterson owns the branch.
82
+
83
+ ## Exact next step
84
+ > In `C:\WWW\toga-blox-npm`: `mv postcss.config.cjs.off postcss.config.cjs`, review `git diff src/components/Talos`, then commit on `TRUE-80692` (or a branch off it) with `feat: add drawer variant and feature toggles to Talos` and ping apeterson for review — BDR `TalosDrawer.tsx` is ready and waiting on that publish.
85
+
86
+ ---
87
+ _Saved by /session-save on 2026-08-28_
@@ -0,0 +1,107 @@
1
+ ---
2
+ type: session
3
+ slug: openorderitems-location-fix
4
+ title: Fix Forecast.OpenOrderItems.locationId NULL on 67% of rows
5
+ author: kyalamarthi
6
+ repos: [worker, worker2, dbchanges2, library, test]
7
+ framework: "both"
8
+ client: shared
9
+ status: active
10
+ created: 2026-08-28
11
+ updated: 2026-08-28
12
+ ---
13
+
14
+ # Session: openorderitems-location-fix
15
+ **Date:** 2026-08-28
16
+ **Project/Repo:** worker2 + worker + dbchanges2 (2.0, with 1.0 cron/library findings)
17
+ **Task:** Populate `Forecast.OpenOrderItems.locationId`, which was NULL on 67% of rows, starving Power BI of the warehouse dimension on open orders (TRUE-79162; TRUE-79078 for the reverted `tl.location` select).
18
+
19
+ ---
20
+
21
+ ## What WORKED
22
+
23
+ - **Measured the gap definitively** via the `toga-db` MCP against prod `Forecast`: **2,883 of 4,308 rows NULL (67%)** across **1,366 of 1,827 orders**, and critically **ZERO orders have a mix** of NULL and populated rows — proof it is writer-attribution, not per-line data. The table is live: the NULL count drifted 2,883 -> 2,844 within an hour as webhooks fired.
24
+ - **Disproved the documented root cause.** The KB said `Forecast.Locations` is empty. It is healthy: 184 rows, 172 with `netsuiteParentLocationId`, 12 roots, **0 rows with a missing parent**. The leaf->root rollup has everything it needs, and `SyncAll` did NOT need to run first.
25
+ - **Pinned the real cause with git.** `locationId` is written only as a side effect of a `salesOrder` webhook re-import, which shipped **2026-06-29** (`worker2` ec66962, `_underscore` b9a4472c) with no backfill. The NULL-rate-by-month gradient matches that cutover exactly: 2026-08 1.2%, 2026-07 16%, 2026-06 77%, 2026-05 and earlier ~90-100%.
26
+ - **Found the second cause:** the live 3 AM cron `periodic_forecast_discrepancy_fix_open_orders.php` (`active: 1`) had **zero occurrences of "location" in 775 lines** and delete+reinserts rows, so anything it recreated came back NULL.
27
+ - **The window split that reshaped the whole fix:** 1,711 rows / 750 orders fall INSIDE the cron's `[today-2y, today+10y]` window; **1,133 rows / 608 orders fall before it** (back to 2019-04-24) and are scanned by nothing.
28
+ - **Built and ran a read-only probe** `test/@dave/probe_ooi_location_gap.php` (verified zero write statements) against live NetSuite: 4,387 importable lines, **100% resolvable, 0 legitimately-NULL, 0 dimension gaps**. Expected values are 6 roots (71 New York 2,506 lines; 2: 1,262; 137: 297; 139: 263; 67: 36; 138: 23).
29
+ - **Verified the line join key at scale:** `OpenOrderItems.lineNumber` == `transactionline.id` — 99 of 99 randomly sampled prod rows plus all 7 multi-root orders matched, **zero join misses**. `linesequencenumber` diverges on 2,061 of 4,387 lines (~47%).
30
+ - **Verified order 235272 end to end by hand** (SalesOrd internalId 6087746, status B): 6 lines all NULL locally, all on NetSuite leaf `4` = "New York", which is itself a root -> `Locations.id 71`. Note `tranid 235272` is reused across three NetSuite records (Invoice 5397885, ItemShip 5418115, SalesOrd 6087746).
31
+ - **Phase 1 (cron fix) implemented:** 5 edits — `location AS locationid` in the line SQL; `Locations` lookup + `$hasLocationCol` / `netsuiteParentLocationId` guards via the existing `forecastColumnExists()`; `resolveOoiLocationId()` copied from `tools/bin/forecast/fixer.php:1328-1342`; `locationId` threaded through both SQL builders; a null-safe `locationId` entry in `$compare`. Lint clean, no PHP 7.4+ tokens introduced.
32
+ - **A stub harness that renders the generated SQL caught a real bug** before any run: the first ODKU clause produced `classificationId = VALUES(classificationId),,` plus a missing separator before `itemId`, which would have broken **every insert the cron makes**. `php -l` did not catch it.
33
+ - **Phase 2 (backfill worker) verified end to end** — `_Worker_Netsuite_OpenOrderLocationBackfill`, against live NetSuite with a local `Forecast` fixture, on order 6087746: dry run reported 6 rows leaf 4 -> 71 and **wrote nothing**; live run updated 6; DB showed all 6 = 71; immediate re-run reported *"nothing to do, 0 rows"*.
34
+ - **Phase 3:** added `ORDER BY id` to `Location.php:89` (offset-paged with no stable sort — latent drop/duplicate bug), and wrote the dbchanges2 `Core/2026-08-27a` migration scheduling `Netsuite/Location/SyncAll` daily 2:30 AM. **Tested the migration twice** against a throwaway local `CronJobs` table: inserts once, guard holds on re-run.
35
+ - **Unblocked local worker2 execution** (see failures below for the traps): app-root-first `include_path`, blanked `[database]` password, created local `ClientLogs` + `Forecast` schemas.
36
+ - **Built `test/@dave/nsq2.php`**, a working read-only SuiteQL runner, and used it to confirm the restored `listSalesOrders()` query shape returns `location: "4"` / `location_name: "New York"` from live NetSuite.
37
+ - **PRs opened** on branch `TRUE-80326` for `worker`, `worker2`, `dbchanges2` — all three committed and pushed.
38
+ - **`/capture` published 7 docs**, commit `59f2cd8` to `agilantsolutions/claude` `_main`, `validate: OK (30 repos)`.
39
+
40
+ ## What did NOT work — DO NOT RETRY THESE
41
+
42
+ - **`test/@dave/Junk Drawer/NetSuite/nsq.php` cannot start at all.** Its bootstrap does `realpath(__DIR__ . '/../../../worker')`, which from its current folder resolves to `test/worker` — a path that does not exist (the file was moved deeper without fixing it). Use `test/@dave/nsq2.php` instead.
43
+ - **Booting worker2 locally with only `_underscore` prepended to `include_path` fails.** Exact error: `Exception: Could not determine application root!` thrown from `C:\www\library\_.php:17`. Cause: `class _underscore` lives in `worker2/_.php`, and `C:\www\library\_.php` (the unrelated 1.0 framework file, already on the path) **shadows** it. The **app root must come FIRST**: `set_include_path(<worker2> . PATH_SEPARATOR . <_underscore> . PATH_SEPARATOR . get_include_path())`.
44
+ - **The framework's own error handler crashes while reporting that failure**, masking the real exception: `Error: Class "_underscore" not found in C:\www\_underscore\Error.php:373`. Wrap `require 'index.php'` in try/catch to see the true error.
45
+ - **NetSuite auth from a laptop fails on the database, not on credentials.** First `Access denied for user 'root'@'localhost' (using password: YES)` at `_underscore/Database.php:495`, then `Unknown database 'clientlogs'`. Root cause: **`send()` does NOT call `setLogging(false)`** — there are zero `setLogging` occurrences in `_underscore/Component/Api/Netsuite/Netsuite.php` and all three `_ApiRequest` constructions pass only 4 args, so logging defaults ON and every call writes to `DB_CLIENT_LOGS`. The KB previously claimed the opposite (now corrected). A laptop with no `ClientLogs` schema cannot make even a read-only SuiteQL call.
46
+ - **`GROUP_CONCAT` for bulk row export truncates silently** at 1,024 chars (~100 of 700 rows) — `group_concat_max_len` default. Fix: the optimizer hint `SELECT /*+ SET_VAR(group_concat_max_len=4194304) */ …`, which works through the `toga-db` MCP and returned all 4,308 rows in two blobs.
47
+ - **Bash heredocs above roughly 900 whitespace-separated tokens fail** with `unexpected EOF while looking for matching quote` — a command-length limit, not a quoting error. The write is rejected atomically (no partial file). Split into ~500-token chunks.
48
+ - **`php -l` cannot catch malformed SQL string assembly.** The ODKU double-comma bug lint-passed cleanly. Only rendering the built SQL through a stub harness exposed it.
49
+ - **Wrote the dbchanges2 migration into the WRONG checkout.** `C:\www\2.0\dbchanges2` is a **stale duplicate** (branch TRUE-80742, newest commit 2026-08-19, 144 `Core/` files). The live repo is **`C:\www\dbchanges2`** (`_main`, committed 2026-08-27, 151 files). The developer could not see the file. Same trap already documented for `worker2` and `_underscore`; the `repo-path-dbchanges2` memory has been corrected. `worker` and `library` have only one checkout each.
50
+ - **`git checkout -- <file>` is refused by the GateGuard destructive-command hook** even after presenting the required facts — unlike the write gate, it does not mark-and-allow on retry. Reverse the edit with the Edit tool instead (or set `ECC_GATEGUARD=off`).
51
+ - **I twice asserted `import_open_orders.php` is `active: 0`. It is `active: 1`.** This came from an early exploration report and was repeated without re-verification. It matters a great deal — see Blockers.
52
+ - **My MySQL error-1093 explanation was wrong.** I attributed the clean run to the `FROM (SELECT 1) AS placeholder` form. bala pushed a tested correction (MySQL 8.0.30): **1093 is UPDATE/DELETE-only**, and an unwrapped self-referencing `INSERT … SELECT … WHERE NOT EXISTS` runs clean. The migration is fine; the reasoning was not.
53
+ - **`knowledge.js kickoff-preflight`'s feature matcher has a dead zone.** `--q="forecast locations netsuite backfill import"` matched **0** feature docs; `--q=netsuite` matched **21** (~74k tokens, `estimate.heavy`). Neither is usable — hand-pick the read list.
54
+ - **`knowledge.js publish` was deliberately NOT used for the capture.** A second capture agent was writing the same repo concurrently, and `publish` does `git add knowledge/`, which would have swept its unapproved drafts into the commit. Only the 7 intended files were staged, so the auto-generated `INDEX.md` files are briefly stale on the remote.
55
+
56
+ ## Not tried yet (candidates for next session)
57
+
58
+ - **THE LIVE THREAD — three writers to `Forecast.Locations`, unverified.** `common_import_sales_from_netsuite.php:1636-1934` **auto-INSERTs and RENAMEs** `Locations` rows and writes `locationId` itself. It is gated behind `SHOULD_SYNC_OPEN_ORDERS` (`:1585`), which `import_open_orders.php` sets — and that cron is **active: 1**. It has been dormant since June only because `$nsItem->location` was always null; `tl.location` came back in `library` on **2026-08-28** (jcardinal's NYCHH transfer-order work, `rest.php:448`), so **this code just woke up**. It now coexists with `_Worker_Netsuite_Location::SyncAll` (newly scheduled) and the fixed 3 AM cron. Nobody has checked whether they agree on `Locations.name`.
59
+ - **The backfill has never been run against prod** — only the local fixture. Sequence: `dryRun: true` scoped with `dateOrderTo: "2024-08-26"`, read the "order no longer open" list, then live.
60
+ - **PHP 7.2 lint never performed.** `worker` and `library` run 7.2 in prod; only `C:\xampp\php` (8.2) exists here.
61
+ - **Open/closed status of the 608 pre-window orders never checked in prod.** Some of those 1,133 rows are likely stale and should be removed rather than given a location — the backfill reports them deliberately instead of filling them.
62
+ - **Non-prod parity migration not written.** dev-sandbox has no `Locations` table and no `OpenOrderItems.locationId`, so none of this is testable outside prod. Complicated by MySQL 8 having no `ADD COLUMN IF NOT EXISTS`.
63
+ - **Leaf-level location as a new column** — deferred, if reporting ever wants sub-location granularity.
64
+ - **Post-deploy verification:** confirm the NULL census drops by ~1,711 after the next 3 AM run, and does **not** climb the following night.
65
+ - **Security ticket + rotation:** `tools/bin/forecast/fixer.php:1424` and `checker.php:678` carry inline production database credentials (surfaced by a concurrent capture agent; values deliberately not recorded).
66
+
67
+ ## Current file state
68
+
69
+ | File | Status | Notes |
70
+ |------|--------|-------|
71
+ | `worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_open_orders.php` | Modified, **committed** (PR, branch `TRUE-80326`) | 5 edits: `location AS locationid` in line SQL; `Locations` lookup + `$hasLocationCol` guards; `resolveOoiLocationId()`; `locationId` in both builders (converted to array params); `locationId` in `$compare`. Lint clean on 8.2 only. |
72
+ | `worker2/Worker/Netsuite/OpenOrderLocationBackfill.php` | **New**, committed (PR) | `dryRun` default true, reuses `_Component_Forecast_Db::resolveLocationId`, `AND locationId IS NULL` guard, reports non-open orders instead of filling them. Verified on the local fixture, never run in prod. |
73
+ | `worker2/Worker/Netsuite/Location.php` | Modified, committed (PR) | Added `ORDER BY id` to the offset-paged `SELECT id, name, parent FROM location`. |
74
+ | `dbchanges2/Core/2026-08-27a - Insert - Netsuite Location SyncAll CronJob.sql` | **New**, committed (PR) | In the LIVE repo `C:\www\dbchanges2`. Daily 2:30 AM Central, `isActive=1`, `NOT EXISTS`-guarded, pre-generated v4 UUID literal. |
75
+ | `library/app/api/netsuite/rest.php` | **Reverted — no local change** | Phase 4 was written, verified against live NetSuite, then reverted at the developer's request. Subsequently found **unnecessary**: `tl.location` is back at `:448` upstream as of `27ae6a68` (2026-08-28). Patch + full-file backup remain in the session scratchpad but are now redundant. |
76
+ | `test/@dave/probe_ooi_location_gap.php` | **New**, uncommitted (untracked) | Read-only auditor; zero write statements verified by grep. Takes `sos.txt` + `locations.txt` as argv. |
77
+ | `test/@dave/nsq2.php` | **New**, uncommitted (untracked) | Working read-only SuiteQL runner replacing the broken `nsq.php`. |
78
+ | `test/@dave/ooi_location_gap.csv` | **New**, uncommitted | 4,387 rows of per-line results. Advised NOT to commit (production order identifiers). |
79
+ | `test/@dave/ooi_expected_location_by_order.csv` | **New**, uncommitted | 1,366 order -> expected `locationId` rows. Advised NOT to commit. |
80
+ | `worker2/Config/dev-kyalamarthi-laptop.ini` | Modified, untracked by git | `[database] password` blanked to match local MySQL (root, no password). Backup in scratchpad. **Local only — never commit.** |
81
+ | Local MySQL `ClientLogs` schema | **Created** | 6 tables from `dbchanges2/Logs_Client/2026-08-14 - BLANK CLIENT LOGS DATABASE.sql`. Required because every NetSuite call logs. |
82
+ | Local MySQL `Forecast` schema | **Created** | Minimal fixture: `Locations` (184 rows) + `OpenOrderItems` (6 canary rows for order 235272). |
83
+ | `knowledge/` (team repo) | **Pushed**, commit `59f2cd8` | 7 docs updated by `/capture`. |
84
+
85
+ ## Decisions made
86
+
87
+ - **`locationId` stores the ROOT warehouse, not the line's leaf sub-location.** Rationale: all 1,425 already-populated rows hold one of 6 roots; `resolveLocationId` rolls up deliberately because reporting wants the top level; storing leaves would give backfilled rows a different meaning from existing ones. Data: 25+ distinct leaves collapse to 6 roots, and within an order the leaf almost never varies (15 of 1,366 orders; only 7 resolve to >1 root, each with exactly 1 NULL row). **Rejected:** leaf storage. **Deferred:** leaf as a NEW column.
88
+ - **Fix the cron FIRST, backfill second.** Rationale: the cron's delete+reinsert erases the column, so a backfill-first order regrows the gap the same night. Adding `locationId` to `$compare` also makes the cron repair all 1,711 in-window rows by itself, shrinking the backfill from 2,883 rows to 1,133.
89
+ - **A targeted backfill worker, not a re-import.** **Rejected:** enqueuing `Netsuite/SalesOrder/put` for 1,366 orders — the KB documents a REST-sublist read-lag race that can WIPE an order's rows, and re-import needlessly recomputes revenue/profit. **Rejected:** a one-off `dbchanges2` UPDATE — not repeatable, and that repo has never contained an `UPDATE OpenOrderItems`.
90
+ - **The backfill reuses `_Component_Forecast_Db::resolveLocationId`** rather than adding a third copy of the rollup, so it cannot disagree with the live importer.
91
+ - **The probe deliberately does NOT use `resolveLocationId`** for classification, because that function returns NULL both when NetSuite has no location and when the leaf is unknown to `Locations` — collapsing exactly the two buckets an audit must separate.
92
+ - **Converted the cron's two SQL builders to a single array parameter** (team rule: 4+ params must use an associative array) rather than adding a 13th positional argument.
93
+ - **Guarded the `Core.CronJobs` insert with `NOT EXISTS`** even though every sibling registration in that folder is unguarded — dbchanges2 rule #7 now requires it.
94
+ - **`library/rest.php` split into its own PR** (TRUE-79078) rather than bundled, because it is an independent regression with tier-wide blast radius. Later moot — it landed upstream.
95
+
96
+ ## Blockers
97
+
98
+ - **UNRESOLVED AND TIME-SENSITIVE: three writers now target `Forecast.Locations`.** The 1.0 rollup at `common_import_sales_from_netsuite.php:1636-1934` auto-INSERTs and RENAMEs rows there; it is reached by `import_open_orders.php`, which is **`active: 1`**, and it went live again when `tl.location` returned to `library` on 2026-08-28. It has not been reconciled against `_Worker_Netsuite_Location::SyncAll` (newly scheduled 2:30 AM) or the fixed 3 AM cron. Both run tonight.
99
+ - **No PHP 7.2 binary on this machine** (only `C:\xampp\php`, 8.2), so 7.2 compatibility of the `worker` change is unproven. A single 7.4+ token in the 1.0 tier is a production outage; the PR should not be marked lint-verified.
100
+ - **Nothing in this work is testable outside production** — dev-sandbox lacks the `Locations` table and the `locationId` column entirely.
101
+
102
+ ## Exact next step
103
+
104
+ > Before tonight's crons run, confirm the newly re-animated 1.0 rollup cannot fight `SyncAll` over `Forecast.Locations.name`. Run `git -C C:\www\worker grep -n "INSERT INTO Locations" -A 6 crons/toga2/forecast2/common_import_sales_from_netsuite.php` and compare the `rootName` it writes (from `App_Api_Netsuite_Rest::listLocations()`) against the `name` that `_Worker_Netsuite_Location::SyncAll` stores (NetSuite `location.name`) for the same 12 root ids. If they can differ, gate one writer before 2:30 AM.
105
+
106
+ ---
107
+ _Saved by /session-save on 2026-08-28_
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "toga-ai",
3
- "version": "1.0.677",
3
+ "version": "1.0.679",
4
4
  "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
5
5
  "keywords": [
6
6
  "claude",