toga-ai 1.0.224 → 1.0.225

package/knowledge/1.0/apps/worker/features/forecast2-netsuite-reconciliation.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-06-26
+updated: 2026-06-29
 owners: [dfranks]
 files:
   - test/@dave/checker.php
@@ -174,6 +174,22 @@ by reconciling a chosen tranDate range directly against NetSuite.
   stays bounded under the existing 1G `memory_limit`. (This corrected a prior docblock claim that `$sqls`
   was never accumulated globally.)
 
+- **PROFIT is now compared for all THREE categories, not just Sales.** `checker.php` and `fixer.php` were
+  previously profit-aware only for Sales; they now compute and compare profit for **Open Orders** and
+  **Opportunities** too.
+  - **Open-order profit** mirrors fixer's per-line economics: `qtyOpen * (rate - unitCost)`,
+    `unitCost = costestimate/quantity` (divide-by-zero-safe), **round-then-sum per line** to match the stored
+    `decimal(14,2)` `OpenOrderItems.profit` column.
+  - **Opportunity profit** = the legacy importer's `Σ(amount − costEstimate)` over qualifying lines = SuiteQL
+    `Σ(-foreignamount + costestimate)`; **revenue** stays the header `projectedtotal` (pulled line-level via
+    `LEFT JOIN` + `MAX(projectedtotal)`). It **must EXCLUDE item-group parent lines** — SuiteQL
+    `itemtype = 'Group'` ↔ `App_Model_Forecast2_Item::NETSUITE_TRANSACTION_TYPE__ITEM_GROUP`, which the
+    importer skips. A dedicated `OPP_EXCLUDED_ITEMTYPES` constant (= the ORD exclusion list + `'Group'`)
+    carries this.
+  - **`fixer.fixOpportunities()` previously did NOT write `projectedProfit` at all.** It now computes
+    `projectedProfit` from the `listOpportunities()` itemList (`Σ(amount − costEstimate)`, skipping ITEM_GROUP
+    and items missing from the `Items` lookup, rounded to cents) so the FIX can actually resolve any
+    opportunity-profit drift the FIND flags (see the compute-identically invariant below).
 - **checker → fixer handoff is WINDOW-based, not id-based (decision).** checker hands fixer a tight day
   window and lets fixer **re-find** the discrepant ids inside it, rather than passing exact ids. Rationale:
   maker≠checker independence (fixer re-derives the drift), and checker has already shrunk the window, so a
@@ -269,6 +285,40 @@ None — Forecast2 is a single shared dataset.
   (NS 852.06 vs FC 852.07). **Correction to any prior note:** the open-orders **2025-04 "unlocalized month" was
   this round-then-sum noise, NOT NS-live-vs-Forecast-replica timing jitter** — the rounding fix eliminated it,
   disproving the earlier replica-jitter hypothesis.
+- **NULL `rate` on cost-only open-order lines silently drops them from a NS profit SUM → permanent phantom
+  profit drift FIX can never resolve.** NetSuite open-order **cost-only** lines (zero revenue — e.g.
+  freight/handling sub-items) can carry a **NULL `rate`**. The FIND profit SQL computed `tl.rate - unitCost`;
+  in SQL `NULL - x = NULL`, so `ROUND(...)` returned NULL and the whole line **silently dropped out of
+  `SUM(...)`**, inflating the NS-side profit. But fixer's FIX (PHP) and the legacy importer treat a NULL rate
+  as **0** (`(float)$rate ?? 0`), so the stored Forecast2 profit was already correct. The two conventions
+  disagreed, so FIND kept flagging profit drift (e.g. Δprofit **$0.54** on many identical **$12.20** SOs;
+  large **$1,365–$9,500** deltas on **$0-revenue** SOs) that FIX could never close — every FIX reported
+  `ins=0/upd=0/del=0` because the line was already right. **Revenue never exposed it** (those lines are $0
+  revenue either way: NULL-dropped on NS vs 0-stored on FC both sum the same), so it surfaced **only in the
+  profit dimension**. **Fix:** wrap rate as `NVL(tl.rate, 0)` in the open-order **revenue AND profit**
+  expressions so FIND uses the same "NULL rate = 0" convention as FIX/importer. Applied in **both**
+  `checker.php` (openorders `nsSub`) and `fixer.php` (`findOpenOrderDiscrepancies`). After the fix all three
+  categories report ✓ in sync — there was never real drift, so **no production write was needed**.
+  **Diagnostic that nailed it:** a per-line probe dumping NS `transactionline`
+  (`lineid`/`item`/`itemtype`/`qty`/`billed`/`rate`/`costestimate` + recomputed profit) alongside the stored
+  `OpenOrderItems` rows, which revealed the NULL `rate` on the cost-only lines. (This is a *distinct*
+  convention-mismatch from the round-then-sum gotcha above — there the divergence was rounding; here it is
+  NULL handling. Both are "FIND and FIX must compute identically.")
+- **INVARIANT: checker FIND, fixer FIND, and fixer FIX must compute a given dimension IDENTICALLY, or `--fix`
+  chases phantoms forever.** Any divergence in inclusion filter, rounding, or NULL handling between the three
+  passes manifests as a discrepancy FIND flags and FIX refuses to resolve (`ins=0/upd=0/del=0`), looping
+  forever. Both the round-then-sum and the NULL-rate gotchas above are instances. When adding/changing a
+  compared dimension, change all three sites in lockstep. `listOpportunities()`
+  (`library/app/api/netsuite/rest.php`) returns line items with `amount = -foreignamount` and
+  `costEstimate = -costestimate`, so the REST-itemList sum and the SuiteQL line-sum are **algebraically
+  identical** — that equivalence is what lets fixer's FIX (REST itemList) and FIND (SuiteQL) agree on
+  opportunity profit.
+- **PHASE 3 post-fix re-check ignores `--category` and re-runs ALL three FIND passes (known inefficiency,
+  NOT yet fixed).** `fixer.php` PHASE 3 (post-fix re-check) re-runs all three category FIND passes
+  unconditionally, ignoring `--category` / `$runCats`. The **Sales** per-transaction aggregate SuiteQL
+  (~694K-row Sales table, paginated 1000/page) is the heaviest op in the tool, so even a category-scoped run
+  (e.g. openorders-only) triggers a **minute-plus full Sales re-scan that looks like a hang**. Suggested fix
+  (not applied): gate each PHASE 3 FIND on `in_array($cat, $runCats)`.
 - **Compare money at 2 decimals.** DB columns store 2dp but PHP `revenue - cost` carries float
   dust (`313.6` vs `313.60000001`); raw `!=` produced thousands of phantom UPDATEs that re-wrote
   identical values (1,839 on one open-orders run). Both tools now compare `round((float)$x, 2)`.
@@ -360,6 +410,23 @@ None — Forecast2 is a single shared dataset.
 
 ## Change history
 
+- 2026-06-29 — **Found/fixed the NULL-rate phantom open-order profit drift; extended PROFIT comparison to all
+  three categories.** Cost-only NS open-order lines (zero revenue) can have a **NULL `rate`**; the FIND
+  profit SQL did `tl.rate - unitCost` and `NULL - x = NULL` silently dropped the line from `SUM`, inflating
+  NS profit, while FIX/importer treat NULL rate as 0 — a permanent phantom (Δprofit $0.54 on identical $12.20
+  SOs; $1,365–$9,500 on $0-revenue SOs) FIX reported `ins/upd/del=0` on forever. Revenue never showed it ($0
+  either way). **Fix:** `NVL(tl.rate,0)` in the open-order revenue AND profit expressions in **both**
+  `checker.php` (openorders `nsSub`) and `fixer.php` (`findOpenOrderDiscrepancies`); all three categories then
+  ✓ in sync, no prod write needed. Localized via a per-line NS `transactionline` probe. Also: checker/fixer
+  now compare **profit for Open Orders and Opportunities**, not just Sales — open-order profit
+  `qtyOpen*(rate-unitCost)` round-then-sum per line; opportunity profit `Σ(amount−costEstimate)` excluding
+  `itemtype='Group'` (= `ITEM_GROUP`, new `OPP_EXCLUDED_ITEMTYPES` constant), revenue = header
+  `projectedtotal`; `fixer.fixOpportunities()` now writes `projectedProfit` (previously never did) from the
+  `listOpportunities()` itemList. Recorded the cross-toolchain **invariant**: checker FIND, fixer FIND, and
+  fixer FIX must compute each dimension identically or `--fix` chases phantoms. **Known inefficiency (not
+  fixed):** `fixer.php` PHASE 3 post-fix re-check re-runs all three FIND passes ignoring `--category`, so a
+  scoped run still triggers the heavy ~694K-row Sales re-scan (looks like a hang); suggested gate on
+  `in_array($cat,$runCats)`. (dfranks)
 - 2026-06-26 — **Added `checker.php` (moment-fingerprint drift detector) + `looper.php` (loop runner), and
   found/fixed the open-orders round-then-sum rounding gotcha.** `checker.php` confirms sync in seconds and
   localizes real drift via a 5-number per-bucket fingerprint (`n,rev,profit,idSum,idSq`; power-sums mod the

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.224",
+  "version": "1.0.225",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",