toga-ai 1.0.231 → 1.0.232

package/knowledge/1.0/apps/worker/features/forecast2-netsuite-reconciliation.md

@@ -405,11 +405,27 @@ None — Forecast2 is a single shared dataset.
   longer open — so it **removes** billed/closed orders rather than resurrecting them. It is therefore
   **not** a resurrection risk against the webhook `removeAll` path; it is in fact the current backstop
   deleter for orders the webhook never sees billed (the invoice-transform gap).
+- **`fixer.php`'s OpenOrderItems path does NOT carry `locationId` / `quantityBackordered` /
+  `amountDue` — mirror any new open-order column here or the reconciler reverts it.** `fixer.php`
+  already handles anchor-line **`amountDue` on the Sales path**, guarded by a column-existence check
+  `$hasAmountDueCol = forecastColumnExists('Sales','amountDue')` (so it's a no-op until the prod
+  column lands). Its **OOI** path — the lookup SELECT, change-detection, the
+  `buildOoiInsertSql`/`buildOoiUpdateSql` helpers, and the SuiteQL `transactionline` query — does
+  **not** include `locationId`/`quantityBackordered`/`amountDue`. Per the compute-identically
+  invariant, any field added to the open-order **importer** (TRUE-79162) must also be added to this
+  OOI path (and ideally behind the same `forecastColumnExists` guard pattern) or `fixer.php`/`looper`
+  will overwrite the importer's value to NULL on its next run.
 - These tools live in `test/@dave/` (developer tooling), but `trueup_open_orders` has been run
   against production. The `Defaults`/checkpoint mechanics of the scheduled sync are separate.
 
 ## Change history
 
+- 2026-06-29 — **Recorded that `fixer.php`'s OOI path lacks `locationId`/`quantityBackordered`/
+  `amountDue`** (planning for TRUE-79162). The Sales path already handles anchor-line `amountDue`
+  behind a `forecastColumnExists('Sales','amountDue')` guard; the OpenOrderItems path (lookup SELECT,
+  change-detection, `buildOoiInsertSql`/`buildOoiUpdateSql`, SuiteQL `transactionline`) carries none
+  of the three new open-order fields. Per the compute-identically invariant, any field added to the
+  open-order importer must be mirrored here or the reconciler reverts it. (dfranks)
 - 2026-06-29 — **Found/fixed the NULL-rate phantom open-order profit drift; extended PROFIT comparison to all
   three categories.** Cost-only NS open-order lines (zero revenue) can have a **NULL `rate`**; the FIND
   profit SQL did `tl.rate - unitCost` and `NULL - x = NULL` silently dropped the line from `SUM`, inflating

package/knowledge/2.0/apps/worker2/features/netsuite-salesorder-open-orders-sync.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-06-25
+updated: 2026-06-29
 owners: ["dfranks"]
 files:
   - worker2/Worker/Netsuite/SalesOrder.php
@@ -127,12 +127,51 @@ The table lives in the **`Forecast` schema on the core2 cluster** (reader
 sales-order FK column is **`netsuiteSalesOrderInternalId`** (`int unsigned`) — note this is *not* the
 same column name used by `Forecast.Sales` (see the column-name gotcha below).
 
+### Model/DB drift to know before backfilling location / backorder / amountDue (TRUE-79162)
+
+A planned backfill of **location, backorder, and amount-due** data onto open orders runs into three
+schema/model facts (all verified against the prod core2 reader; nothing built yet):
+
+- **`_Model_Forecast_OpenOrderItem` (`_underscore/Model/Forecast/OpenOrderItem.php`) does NOT declare
+  `locationId` or `quantityBackordered` — even though both columns already exist in the prod
+  `Forecast.OpenOrderItems` table** (`locationId` `int unsigned NULL`, `quantityBackordered`
+  `decimal(15,4) NULL`). The model declares only `id, netsuiteSalesOrderInternalId, dateOrder,
+  orderNumber, customerId, salesRepEmployeeId, classificationId, accountId, itemId, lineNumber,
+  revenue, profit`. **Writing those two fields through the 2.0 importer requires adding the
+  properties to the model first** — the column existing in the DB is not sufficient.
+- **`Forecast.Locations` is EMPTY in prod (0 rows).** Schema: `id, name, netsuiteInternalLocationId`
+  (UNIQUE, nullable), `typeId` — **no parent/hierarchy column.** Any open-order `locationId`
+  resolution returns NULL until this table is populated from NetSuite, so **populating `Locations` is
+  a hard prerequisite** for the location backfill. NetSuite returns a **leaf sub-location** but the
+  warehouse dashboard wants the **top-level location**, so a leaf→root rollup is needed (and there is
+  no parent column on `Locations` today to express it).
+- **`Forecast.OpenOrderItems` has NO `amountDue` column** (neither does `Forecast.Sales` in prod —
+  see the [Sales import doc](../../_underscore/features/forecast-sale-import.md)), and `amountDue`
+  appears in no worker2 Netsuite handler or `_Model_Forecast_*`. Importing open-order amountDue
+  needs a **new column + model field**. The amountDue source semantics from the prior Sales work
+  (TRUE-78923) — REST `amountRemaining` / SuiteQL `foreignamountunpaid`, **anchor-line only, store
+  RAW POSITIVE** — carry over, **but `amountRemaining` is an AR/invoice field**, so for an *unbilled*
+  sales order it may return null/0. **Live-probe an open SO before finalizing** the open-order
+  amountDue source.
+
 ## Client variations
 
 None — uniform (platform-wide Forecast2 sync).
 
 ## Gotchas / known issues
 
+- **A new synced `OpenOrderItems` column must be wired into BOTH `buildOpenLineRows()` AND the
+  `syncOpenLines()` change-detection diff — not just the write — or no-diff updates silently drop
+  it.** The forward-sync path is `_Worker_Netsuite_SalesOrder`'s private `buildOpenLineRows()`
+  (assembles each `OpenOrderItems` row array) and `syncOpenLines()` (change-detection + upsert).
+  This is the established forecast2 change-detection gotcha applied to open orders: a column added
+  only to the row build but not to the `syncOpenLines()` diff block is **discarded on a no-change
+  UPDATE** (the diff sees no difference, so the new value never persists). Any
+  locationId/quantityBackordered/amountDue addition (TRUE-79162) must touch both. Forward sync now
+  flows through this webhook importer; the legacy 1.0 cron
+  (`worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php` open-order section) is
+  superseded for it. **Also mirror any new field into the reconciler `test/@dave/fixer.php` OOI path**
+  or it reverts the value on its next run (see the reconciliation doc).
 - **Billing an SO via invoice-transform flips it to "Billed" WITHOUT firing the SalesOrder afterSubmit UE → no edit/PUT webhook → `removeAll()` never runs → the OpenOrderItems rows linger (this is the real cause of the "billed order not removed" problem — NOT a removeAll-didn't-persist bug).** When a NetSuite Sales Order is billed by **creating an Invoice from it**, the SO's `status` transitions to `Billed` as a **side-effect of the invoice transform** — this does **not** trigger the SalesOrder `afterSubmit` User Event, so the AMQ enqueuer (`ue_api_msg_queue_enqueue`, deployed per-record-type) **never emits an edit/PUT webhook for the SO**. Consequently `_Worker_Netsuite_SalesOrder` is never invoked for the billing change, `importOpenOrder`'s status gate (which *would* call `removeAll` on a non-open status) never runs, and the order's `OpenOrderItems` rows survive until the nightly discrepancy-fix cron deletes them. **Confirmed live 2026-06-24/25 on SO 7190415** (tranId 280387): only a `create` webhook delivery exists in `Logs.Webhook` (2026-06-24 23:48, 16 rows inserted) — **NO** edit/delete delivery at billing time (~09:41 UTC 2026-06-25); `Core.WorkerJobs` shows only `Netsuite/Webhook` + `Netsuite/SalesOrder/post` (no put/delete); `Logs.Api` has only the original NS GET + a 200 `OPEN_ORDER_IMPORT` breadcrumb — **no 204 removeAll breadcrumb**. The invoice (7190621) that billed it **also produced no webhook** — its 16 `Forecast.Sales` rows came from the **legacy 5-min pull cron**, not the webhook. **This corrects the earlier suspicion that "removeAll fired but DB didn't persist"** — both `removeAll` paths *do* commit `DB_FORECAST`; `removeAll` was simply **never called**. **Diagnostic ladder for a lingering billed order:** `Logs.Webhook` (was an edit/delete even delivered? — here, no) → `Core.WorkerJobs` (was a put/delete processed? — no) → `Logs.Api` `source='OPEN_ORDER_IMPORT'` (any 204 removeAll breadcrumb? — no). **Fix direction (not yet implemented):** the SalesOrder-side enqueuer must also fire on the invoice-transform status change — either deploy the AMQ enqueuer on **Invoice** so the bill event drives an SO re-sync, or have the SalesOrder handler re-evaluate the SO when its child invoice arrives; until then the **stopgap is the continuous `loop_trueup_open_orders.php --by-lastmodified` runner** (see the reconciliation doc), which windows on `lastmodifieddate` so it catches an order billed today whose `tranDate` predates the window. The `removeAll` path in `SalesOrder.php` was re-confirmed correct in this session.
 - **Wrong FK column name → MySQL 1054 that masquerades as "order missing".** `OpenOrderItems`'s
   sales-order key is **`netsuiteSalesOrderInternalId`**. Do **NOT** query it with
@@ -297,6 +336,16 @@ test fixture (it surfaced the stale SO 7181316 above).
 
 ## Change history
 
+- 2026-06-29 — **Recorded the model/DB drift + prerequisites for the open-order location/backorder/
+  amountDue backfill (TRUE-79162, planning only — no code written).** `_Model_Forecast_OpenOrderItem`
+  declares neither `locationId` nor `quantityBackordered` though both columns already exist in prod
+  `OpenOrderItems` (so the model needs the properties before the importer can write them);
+  `Forecast.Locations` is empty in prod with no parent column (hard prerequisite + leaf→root rollup
+  needed); `OpenOrderItems` has no `amountDue` column (new column + field required, and
+  `amountRemaining` is an AR field that may be null/0 on an unbilled SO — probe first). Added the
+  gotcha that any new synced column must be wired into **both** `buildOpenLineRows()` and the
+  `syncOpenLines()` change-detection diff (and mirrored in `fixer.php`'s OOI path) or it is dropped on
+  no-diff updates. (dfranks)
 - 2026-06-25 — **Root-caused "billed SO not removed from OpenOrderItems in real time" to a missing
   webhook, NOT a removeAll-didn't-persist bug.** Billing an SO via invoice-transform flips it to
   `Billed` as a side-effect that does **not** fire the SalesOrder `afterSubmit` UE → the AMQ enqueuer

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.231",
+  "version": "1.0.232",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",