toga-ai 1.0.230 → 1.0.232

package/knowledge/1.0/apps/worker/features/forecast2-netsuite-reconciliation.md

@@ -405,11 +405,27 @@ None — Forecast2 is a single shared dataset.
   longer open — so it **removes** billed/closed orders rather than resurrecting them. It is therefore
   **not** a resurrection risk against the webhook `removeAll` path; it is in fact the current backstop
   deleter for orders the webhook never sees billed (the invoice-transform gap).
+- **`fixer.php`'s OpenOrderItems path does NOT carry `locationId` / `quantityBackordered` /
+  `amountDue` — mirror any new open-order column here or the reconciler reverts it.** `fixer.php`
+  already handles anchor-line **`amountDue` on the Sales path**, guarded by a column-existence check
+  `$hasAmountDueCol = forecastColumnExists('Sales','amountDue')` (so it's a no-op until the prod
+  column lands). Its **OOI** path — the lookup SELECT, change-detection, the
+  `buildOoiInsertSql`/`buildOoiUpdateSql` helpers, and the SuiteQL `transactionline` query — does
+  **not** include `locationId`/`quantityBackordered`/`amountDue`. Per the compute-identically
+  invariant, any field added to the open-order **importer** (TRUE-79162) must also be added to this
+  OOI path (and ideally behind the same `forecastColumnExists` guard pattern) or `fixer.php`/`looper`
+  will overwrite the importer's value to NULL on its next run.
 - These tools live in `test/@dave/` (developer tooling), but `trueup_open_orders` has been run
   against production. The `Defaults`/checkpoint mechanics of the scheduled sync are separate.
 
 ## Change history
 
+- 2026-06-29 — **Recorded that `fixer.php`'s OOI path lacks `locationId`/`quantityBackordered`/
+  `amountDue`** (planning for TRUE-79162). The Sales path already handles anchor-line `amountDue`
+  behind a `forecastColumnExists('Sales','amountDue')` guard; the OpenOrderItems path (lookup SELECT,
+  change-detection, `buildOoiInsertSql`/`buildOoiUpdateSql`, SuiteQL `transactionline`) carries none
+  of the three new open-order fields. Per the compute-identically invariant, any field added to the
+  open-order importer must be mirrored here or the reconciler reverts it. (dfranks)
 - 2026-06-29 — **Found/fixed the NULL-rate phantom open-order profit drift; extended PROFIT comparison to all
   three categories.** Cost-only NS open-order lines (zero revenue) can have a **NULL `rate`**; the FIND
   profit SQL did `tl.rate - unitCost` and `NULL - x = NULL` silently dropped the line from `SUM`, inflating

package/knowledge/2.0/apps/worker2/features/netsuite-salesorder-open-orders-sync.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-06-25
+updated: 2026-06-29
 owners: ["dfranks"]
 files:
   - worker2/Worker/Netsuite/SalesOrder.php
@@ -127,12 +127,51 @@ The table lives in the **`Forecast` schema on the core2 cluster** (reader
 sales-order FK column is **`netsuiteSalesOrderInternalId`** (`int unsigned`) — note this is *not* the
 same column name used by `Forecast.Sales` (see the column-name gotcha below).
 
+### Model/DB drift to know before backfilling location / backorder / amountDue (TRUE-79162)
+
+A planned backfill of **location, backorder, and amount-due** data onto open orders runs into three
+schema/model facts (all verified against the prod core2 reader; nothing built yet):
+
+- **`_Model_Forecast_OpenOrderItem` (`_underscore/Model/Forecast/OpenOrderItem.php`) does NOT declare
+  `locationId` or `quantityBackordered` — even though both columns already exist in the prod
+  `Forecast.OpenOrderItems` table** (`locationId` `int unsigned NULL`, `quantityBackordered`
+  `decimal(15,4) NULL`). The model declares only `id, netsuiteSalesOrderInternalId, dateOrder,
+  orderNumber, customerId, salesRepEmployeeId, classificationId, accountId, itemId, lineNumber,
+  revenue, profit`. **Writing those two fields through the 2.0 importer requires adding the
+  properties to the model first** — the column existing in the DB is not sufficient.
+- **`Forecast.Locations` is EMPTY in prod (0 rows).** Schema: `id, name, netsuiteInternalLocationId`
+  (UNIQUE, nullable), `typeId` — **no parent/hierarchy column.** Any open-order `locationId`
+  resolution returns NULL until this table is populated from NetSuite, so **populating `Locations` is
+  a hard prerequisite** for the location backfill. NetSuite returns a **leaf sub-location** but the
+  warehouse dashboard wants the **top-level location**, so a leaf→root rollup is needed (and there is
+  no parent column on `Locations` today to express it).
+- **`Forecast.OpenOrderItems` has NO `amountDue` column** (neither does `Forecast.Sales` in prod —
+  see the [Sales import doc](../../_underscore/features/forecast-sale-import.md)), and `amountDue`
+  appears in no worker2 Netsuite handler or `_Model_Forecast_*`. Importing open-order amountDue
+  needs a **new column + model field**. The amountDue source semantics from the prior Sales work
+  (TRUE-78923) — REST `amountRemaining` / SuiteQL `foreignamountunpaid`, **anchor-line only, store
+  RAW POSITIVE** — carry over, **but `amountRemaining` is an AR/invoice field**, so for an *unbilled*
+  sales order it may return null/0. **Live-probe an open SO before finalizing** the open-order
+  amountDue source.
+
 ## Client variations
 
 None — uniform (platform-wide Forecast2 sync).
 
 ## Gotchas / known issues
 
+- **A new synced `OpenOrderItems` column must be wired into BOTH `buildOpenLineRows()` AND the
+  `syncOpenLines()` change-detection diff — not just the write — or no-diff updates silently drop
+  it.** The forward-sync path is `_Worker_Netsuite_SalesOrder`'s private `buildOpenLineRows()`
+  (assembles each `OpenOrderItems` row array) and `syncOpenLines()` (change-detection + upsert).
+  This is the established forecast2 change-detection gotcha applied to open orders: a column added
+  only to the row build but not to the `syncOpenLines()` diff block is **discarded on a no-change
+  UPDATE** (the diff sees no difference, so the new value never persists). Any
+  locationId/quantityBackordered/amountDue addition (TRUE-79162) must touch both. Forward sync now
+  flows through this webhook importer; the legacy 1.0 cron
+  (`worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php` open-order section) is
+  superseded for it. **Also mirror any new field into the reconciler `test/@dave/fixer.php` OOI path**
+  or it reverts the value on its next run (see the reconciliation doc).
 - **Billing an SO via invoice-transform flips it to "Billed" WITHOUT firing the SalesOrder afterSubmit UE → no edit/PUT webhook → `removeAll()` never runs → the OpenOrderItems rows linger (this is the real cause of the "billed order not removed" problem — NOT a removeAll-didn't-persist bug).** When a NetSuite Sales Order is billed by **creating an Invoice from it**, the SO's `status` transitions to `Billed` as a **side-effect of the invoice transform** — this does **not** trigger the SalesOrder `afterSubmit` User Event, so the AMQ enqueuer (`ue_api_msg_queue_enqueue`, deployed per-record-type) **never emits an edit/PUT webhook for the SO**. Consequently `_Worker_Netsuite_SalesOrder` is never invoked for the billing change, `importOpenOrder`'s status gate (which *would* call `removeAll` on a non-open status) never runs, and the order's `OpenOrderItems` rows survive until the nightly discrepancy-fix cron deletes them. **Confirmed live 2026-06-24/25 on SO 7190415** (tranId 280387): only a `create` webhook delivery exists in `Logs.Webhook` (2026-06-24 23:48, 16 rows inserted) — **NO** edit/delete delivery at billing time (~09:41 UTC 2026-06-25); `Core.WorkerJobs` shows only `Netsuite/Webhook` + `Netsuite/SalesOrder/post` (no put/delete); `Logs.Api` has only the original NS GET + a 200 `OPEN_ORDER_IMPORT` breadcrumb — **no 204 removeAll breadcrumb**. The invoice (7190621) that billed it **also produced no webhook** — its 16 `Forecast.Sales` rows came from the **legacy 5-min pull cron**, not the webhook. **This corrects the earlier suspicion that "removeAll fired but DB didn't persist"** — both `removeAll` paths *do* commit `DB_FORECAST`; `removeAll` was simply **never called**. **Diagnostic ladder for a lingering billed order:** `Logs.Webhook` (was an edit/delete even delivered? — here, no) → `Core.WorkerJobs` (was a put/delete processed? — no) → `Logs.Api` `source='OPEN_ORDER_IMPORT'` (any 204 removeAll breadcrumb? — no). **Fix direction (not yet implemented):** the SalesOrder-side enqueuer must also fire on the invoice-transform status change — either deploy the AMQ enqueuer on **Invoice** so the bill event drives an SO re-sync, or have the SalesOrder handler re-evaluate the SO when its child invoice arrives; until then the **stopgap is the continuous `loop_trueup_open_orders.php --by-lastmodified` runner** (see the reconciliation doc), which windows on `lastmodifieddate` so it catches an order billed today whose `tranDate` predates the window. The `removeAll` path in `SalesOrder.php` was re-confirmed correct in this session.
 - **Wrong FK column name → MySQL 1054 that masquerades as "order missing".** `OpenOrderItems`'s
   sales-order key is **`netsuiteSalesOrderInternalId`**. Do **NOT** query it with
@@ -297,6 +336,16 @@ test fixture (it surfaced the stale SO 7181316 above).
 
 ## Change history
 
+- 2026-06-29 — **Recorded the model/DB drift + prerequisites for the open-order location/backorder/
+  amountDue backfill (TRUE-79162, planning only — no code written).** `_Model_Forecast_OpenOrderItem`
+  declares neither `locationId` nor `quantityBackordered` though both columns already exist in prod
+  `OpenOrderItems` (so the model needs the properties before the importer can write them);
+  `Forecast.Locations` is empty in prod with no parent column (hard prerequisite + leaf→root rollup
+  needed); `OpenOrderItems` has no `amountDue` column (new column + field required, and
+  `amountRemaining` is an AR field that may be null/0 on an unbilled SO — probe first). Added the
+  gotcha that any new synced column must be wired into **both** `buildOpenLineRows()` and the
+  `syncOpenLines()` change-detection diff (and mirrored in `fixer.php`'s OOI path) or it is dropped on
+  no-diff updates. (dfranks)
 - 2026-06-25 — **Root-caused "billed SO not removed from OpenOrderItems in real time" to a missing
   webhook, NOT a removeAll-didn't-persist bug.** Billing an SO via invoice-transform flips it to
   `Billed` as a side-effect that does **not** fire the SalesOrder `afterSubmit` UE → the AMQ enqueuer

package/knowledge/sessions/2026-06-29-surface-ui-config-layer-jcardinal.md

@@ -0,0 +1,73 @@
+---
+type: session
+slug: surface-ui-config-layer
+title: Surface UI config layer — deploy fixes + 3 screen migrations
+author: jcardinal
+repos: [dbchanges2, _underscore, api2, toga25-supply]
+framework: "2.0"
+client: shared
+status: active
+created: 2026-06-29
+updated: 2026-06-29
+---
+
+# Session: surface-ui-config-layer
+**Date:** 2026-06-29
+**Project/Repo:** dbchanges2 / _underscore / api2 / toga25-supply (2.0)
+**Task:** Continue the platform-wide "Surface" UI presentation/config layer: fix the deployed layer's 403 + FE crash on dev.sandbox, then refactor toga25-supply screens onto Surface (Items done prior; VendorItems + Inventory this session; SalesOrders remaining).
+
+---
+
+## What WORKED
+- **403 (EZ-1) on `GET /v2/surfaces/meta` fixed.** Root cause found by reading V2.php `getRecordScriptPhpMethod` (~line 5803): RecordScript dispatch is authorized by the **Client-DB `AclRecordScripts`** table (`WHERE recordScriptId=? AND roleId IN(...)`), NOT `AclRecordPermissions`. Confirmed on dev.sandbox: surfaces meta script (id 25) had ZERO `AclRecordScripts` rows, while working TableView/Page `meta` scripts (ids 1,2) grant roles 1,3,4. Fix file `dbchanges2/Client/2026-06-29c - SurfaceRecordScriptAcl.sql` grants meta→{1,3,4}, debug→{3}; id-agnostic (route-join, works on 333 or 2300), NOT EXISTS-guarded. Developer applied it on dev.sandbox → 403 cleared.
+- **Latent fatal fixed** — `dbchanges2/Core/2026-06-29c - SurfaceDebugPhpMethodFix.sql`: idempotent UPDATE renaming surfaces `debug` RecordScript phpMethod `debug`→`metaDebug` on pre-rename envs.
+- **FE "s.elements is not iterable" crash fixed.** Root cause: V2 nests a scripted-API return under a route-keyed slot of `data` (e.g. `data.meta`/`data.surfaces`), one level deeper than the hook unwrapped. `useFetchSurfaceMeta.ts` `extractBundle()` now finds the bundle by structural fingerprint (the `elements` array) across `raw`/`raw.data`/`raw.data.<key>`, with empty-elements fallback. Hardened `SurfaceSection.tsx` + `SurfaceActionBar.tsx` to bail to null unless `elements` is a real array.
+- **VendorItems screen migrated** (screen 2/5). Seed `dbchanges2/Core/2026-06-29d - VendorItemsSurfaceSeed.sql` (vendor-items-list TABLE→TableView 10, vendor-items-list-actions BUTTON_BAR). FE rewired to bundle; legacy `vendorItemsPageFields.json` deleted + removed from fieldsConfig registry. Mirrors the prior Items migration exactly.
+- **Inventory screen migrated presentation-only** (screen 3/5). Seed `dbchanges2/Core/2026-06-29e - InventorySurfaceSeed.sql` (inventory-list TABLE recordId/tableViewId NULL, carries groupByLabels+entityTags in config; inventory-list-actions BUTTON_BAR; Actions anchored to items(21)). FE sources title/buttons/entityTags/groupByLabels from the surface; grouping topology stays in JSON.
+- **/capture PUSHED** — 4 surface feature docs updated on team `_main`.
+
+## What did NOT work — DO NOT RETRY THESE
+- **Granting only `AclRecordPermissions` (record-level READ) does NOT authorize a RecordScript call.** The earlier fix `dbchanges2/Core/2026-06-29b - SurfaceMetaPublicReadAcl.sql` granted Public(1) READ (perm 313, complete logic chain) and the 403 PERSISTED — because dispatch checks `AclRecordScripts`, a different table. Record perms are necessary but NOT sufficient. Do not re-investigate the AclRecordPermissions chain for script 403s.
+- **Unwrapping only `raw.data` in the meta hook** left the FE reading `.elements` on the `{meta:{...}}` wrapper → "not iterable". Hard-coding a single nesting key (`data.meta` vs `data.surfaces`) is brittle — the structural-fingerprint extractor is the right approach.
+- **Swapping `SurfaceSection` into `SalesOrderSummaryGrid` would DESTROY bespoke rendering.** The grid renders via blox `DetailSection`/`OrderSectionCard`/`BaseTotals` (the last with client recurring-lease logic `_totalLease!==0`). Generic label/value `SurfaceSection` cannot represent totals or location cards. Do not attempt a generic-section swap there.
+
+## Not tried yet (candidates for next session)
+- **SalesOrders detail sections via a REGISTERED-RENDERER approach**: map surface element `config.kind` → bespoke component (detailSection→DetailSection, locationCard→OrderSectionCard, totalsCard→BaseTotals), keeping the recurring-lease logic in code/_underscore. Reconcile the early-seeded order-details/ship-to/bill-to/order-summary SECTION surfaces with the grid's actual client-keyed, data-driven section model (they don't currently map cleanly).
+- **add-notes (adminNotes) migration** — `getDetailSections.tsx` + `AdminNotesSection`; currently gated by `tenantFields[key].isVisible`. Move the section visibility gating to Surface while keeping the bespoke component.
+- **Compass approval modals** (`SalesOrderApprovalModalsLayout`) — the plan's explicitly-deferred bespoke surface; needs the blessed-component registry + `_underscore` capability methods, not generic config.
+- **Confirm BLANK_CLIENT_DATABASE provisioning** carries the `AclRecordScripts` grants for surface scripts (else a newly-provisioned client repeats the 403). Determine whether BLANK is a snapshot or replays dated migrations.
+
+## Current file state
+| File | Status | Notes |
+|------|--------|-------|
+| dbchanges2/Client/2026-06-29c - SurfaceRecordScriptAcl.sql | created | The 403 fix; applied on dev.sandbox. Fan out to other client DBs + BLANK. |
+| dbchanges2/Core/2026-06-29c - SurfaceDebugPhpMethodFix.sql | created | debug→metaDebug phpMethod fix for pre-rename envs. |
+| dbchanges2/Core/2026-06-29d - VendorItemsSurfaceSeed.sql | created | VendorItems surface seed. |
+| dbchanges2/Core/2026-06-29e - InventorySurfaceSeed.sql | created | Inventory presentation-only seed. |
+| toga25-supply/src/surface/useFetchSurfaceMeta.ts | modified | extractBundle() structural-fingerprint unwrap. |
+| toga25-supply/src/surface/SurfaceSection.tsx | modified | Array.isArray(elements) guard. |
+| toga25-supply/src/surface/SurfaceActionBar.tsx | modified | Array.isArray(elements) guard. |
+| toga25-supply/src/pages/VendorItems/VendorItemsPage.tsx | modified | Title + button bar from bundle. |
+| toga25-supply/src/pages/VendorItems/viewModel/useVendorItemsPageViewModel.tsx | modified | Dropped useClientFields/vendorItemsPageFields; in-memory columnsButton default. |
+| toga25-supply/src/pages/VendorItems/viewModel/FIELDS/DEFAULT/vendorItemsPageFields.json | deleted | Migrated to Surface. |
+| toga25-supply/src/fieldsConfig/index.ts | modified | Removed vendorItemsPageFields (import/type/Omit/const/2 injections). |
+| toga25-supply/src/pages/Inventory/Inventory.tsx | modified | Title + Refresh/GroupBy icon+label from bundle. |
+| toga25-supply/src/pages/Inventory/viewModel/useInventoryPageViewModel.tsx | modified | entityTags/groupByLabels/title/buttons from surface; groupings still from JSON. |
+| toga25-supply/src/pages/Inventory/viewModel/FIELDS/index.ts | modified | entityTags/labels @deprecated optional on InventoryGroupingsConfig. |
+| toga25-supply/.../Inventory/.../DEFAULT/inventoryGroupings.json | modified | Stripped labels+entityTags; groupings only. |
+| toga25-supply/.../Inventory/.../NYCHH/inventoryGroupings.json | modified | Same strip (file is byte-identical to DEFAULT). |
+
+## Decisions made
+- **Inventory = presentation-only migration** (developer-approved). The `groupings[].levels[]` topology (fetchSlug/role/additionalDataSlug/urlParamKey) is data-fetch wiring → stays JSON; only title/buttons/modal-copy/entityTags moved to Surface. Rejected: migrating the whole groupings blob into surface config (would erode the config/logic boundary — the plan's #1 risk).
+- **SalesOrders deferred to a design-first session** (developer chose to stop here after the bespoke nature was discovered). Rejected: rushing a generic-section swap on this expensive thread (would destroy bespoke totals/location rendering and produce unverifiable output).
+- **Surface RecordScripts must be granted in `AclRecordScripts` (Client DB) per role** — this is the canonical dispatch-authorization mechanism; encode it in every surface ACL seed and in BLANK.
+
+## Blockers
+- **Cannot run `tsc`/build locally** — private `@agilant/toga-blox` registry needs auth (npm E401); FE changes verified by reading, built/deployed by the developer on dev.sandbox.
+- **File-vs-DB id drift** — seed files use renumbered ids (surfaces record 333), but dev.sandbox holds the originally-provisioned ids (2300). All fixes were made id-agnostic (match by route) to span both. Not blocking, but keep using route-based lookups.
+
+## Exact next step
+> In a fresh session: open `toga25-supply/src/pages/SalesOrders/view/SalesOrderRecordModalLayout/view/layoutComponents/SalesOrderSummaryGrid.tsx` and design a **registered-renderer** that maps a surface element's `config.kind` (detailSection / locationCard / totalsCard) to its existing bespoke component (DetailSection / OrderSectionCard / BaseTotals), sourcing section title + visibility + field bindings from the `order-details`/`ship-to`/`bill-to`/`order-summary` surface bundles while keeping the recurring-lease logic in code. First reconcile those 4 seeded surfaces against the grid's actual client-keyed section model (query their SurfaceElements: order-details=4, ship-to=2, bill-to=1, order-summary=4).
+
+---
+_Saved by /session-save on 2026-06-29_

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.230",
+  "version": "1.0.232",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",