toga-ai 1.0.256 → 1.0.257

package/knowledge/1.0/apps/library/features/netsuite-suiteql-api-reference.md

@@ -6,7 +6,7 @@ project: Library
 client: shared
 type: feature
 status: active
-updated: 2026-06-25
+updated: 2026-06-30
 owners: ["dfranks"]
 files:
   - library/app/api/netsuite/rest.php
@@ -330,9 +330,23 @@ numbers. Budget hours for multi-year runs and launch them under `nohup`/`tmux`.
   (PHP 8.0) is for running probes on the laptop only, **not** for compat checking.
 - Field/relationship availability varies by NetSuite account **and** record type — probe the live
   account before assuming a column exists.
+- **`IS NOT NULL` on a custom `transactionline` column inside an aggregate query is a ~15× planner
+  trap — prefer `> 0` (or `NVL(col,0) <> 0`).** Filtering a custom employee-ref column
+  (`custcol_sales_rep_line`) with `AND tl.custcol_sales_rep_line IS NOT NULL` inside a
+  `GROUP BY` / power-sum aggregate (`MOD(SUM(MOD(id*id,P)),P)`) made the planner scan
+  pathologically: **~122s** vs **~8s** for the semantically identical `AND tl.custcol_sales_rep_line
+  > 0` (employee internalIds are always positive) — a ~15× regression purely from the predicate form.
+  Measured against the Forecast2 reconciliation checker (TRUE-79968). Use `> 0` / `NVL(col,0) <> 0`
+  for a custom ref column you only need to test for presence in an aggregate; reserve `IS NOT NULL`
+  for non-aggregate row filters.
 
 ## Change history
 
+- 2026-06-30 — **Recorded the `IS NOT NULL` vs `> 0` planner trap for custom `transactionline`
+  columns in aggregate queries** (TRUE-79968): a custom employee-ref column filtered with
+  `IS NOT NULL` inside a power-sum `GROUP BY` aggregate scanned ~15× slower (~122s) than the
+  identical `> 0` predicate (~8s). Prefer `> 0` / `NVL(col,0) <> 0` for presence tests in
+  aggregates. (dfranks)
 - 2026-06-25 — **Documented Journal Entry line structure + the JE→Forecast.Sales deferral.** JEs
   invert the `mainline` rule — **all** JE `transactionline` rows are `mainline='T'` (no `'F'` detail
   split), so a `mainline='F'` revenue query over a JE returns empty. Also: `tl.salesrep` does not

package/knowledge/1.0/apps/worker/features/forecast2-netsuite-reconciliation.md

@@ -233,6 +233,22 @@ by reconciling a chosen tranDate range directly against NetSuite.
   are correctly excluded **by design** (e.g. JE 7213407 = a Bank↔Equity reclass, 0 allowlist lines → never
   imported — not a miss). An accrual JE and its NetSuite auto-reversal are **separate transactions**, each
   reconciled independently by its own `internalId`.
+- **The rep-only-NS-side / unfiltered-FC-side asymmetry is DELIBERATE and load-bearing — do NOT add
+  a rep filter to any Forecast-side query (TRUE-79968, dfranks 2026-06-30).** The importer now keeps
+  **only JE lines that carry a raw `custcol_sales_rep_line`** (see the sale-import doc). To make the
+  fixer be the **cleanup mechanism** for any pre-existing no-rep JE rows (there is no separate delete
+  script), the **NS-side** JE SuiteQL in both `checker.php` (`journalEntryNsTotals` arm of the sales
+  `nsSub`) and `fixer.php` (`journalEntryNsTotals` FIND + `fixJournalEntries` FIX) is scoped to
+  **rep-bearing lines** (mirroring the importer), while the **Forecast-side** reads in both tools stay
+  **UNFILTERED** (still SUM all JE rows, incl. any `salesRepEmployeeId IS NULL`). That asymmetry is
+  what surfaces the stale rows: NS (rep-only) **<** FC (still holds no-rep rows) → FIND flags the JE →
+  the existing per-`'journalEntry'` orphan-delete in `fixJournalEntries` removes them. **If the FC side
+  were ALSO rep-filtered, both sides would match and the stale rows would be invisible / never purged.**
+  A blanket `DELETE WHERE salesRepEmployeeId IS NULL` is the wrong fix — locally-null also means
+  "rep present in NetSuite but unmapped to `Forecast.Employees`," which must survive; the fixer sweep
+  correctly re-derives "no raw rep" from NetSuite. One-time cleanup =
+  `fixer.php --commit --prod --category sales` over the JE window, then `checker.php` to confirm
+  (in current prod this was a no-op — 0 no-rep JE rows — so the importer filter is preventative).
 - **JE item dimension is NULL today; here is how to enable it later.** `itemId` is null on JE Sales rows
   because the JE import path reads item **only** via the configurable `JE_LINE_ITEM_FIELD` (currently null)
   — it does **not** read native `transactionline.item`, so even a populated native `tl.item` would NOT flow
@@ -468,11 +484,32 @@ None — Forecast2 is a single shared dataset.
   invariant, any field added to the open-order **importer** (TRUE-79162) must also be added to this
   OOI path (and ideally behind the same `forecastColumnExists` guard pattern) or `fixer.php`/`looper`
   will overwrite the importer's value to NULL on its next run.
+- **`IS NOT NULL` on a CUSTOM `transactionline` column inside a SuiteQL aggregate is a ~15× planner
+  trap — use `> 0` instead.** Filtering `custcol_sales_rep_line` (an employee-ref custom column) with
+  `AND tl.custcol_sales_rep_line IS NOT NULL` inside checker's UNION'd moment-fingerprint (the
+  `MOD(SUM(MOD(id*id,P)),P)` power-sum aggregates) made the planner scan pathologically: a full-window
+  checker run measured **~122s** with `IS NOT NULL` vs **~8s** with the semantically identical
+  `AND tl.custcol_sales_rep_line > 0` (employee internalIds are always positive). Applied to
+  `checker.php` (JE arm) and `fixer.php` (both JE queries) with inline comments warning against
+  reverting. See the SuiteQL API reference for the general rule. (Use `> 0`, or `NVL(col,0) <> 0`,
+  never `IS NOT NULL`, for a custom employee-ref `transactionline` column in a SuiteQL aggregate.)
 - These tools live in `test/@dave/` (developer tooling), but `trueup_open_orders` has been run
   against production. The `Defaults`/checkpoint mechanics of the scheduled sync are separate.
 
 ## Change history
 
+- 2026-06-30 — **Made the fixer the cleanup mechanism for no-rep JE rows + killed a ~15× SuiteQL
+  planner trap (TRUE-79968, dfranks).** The importer now keeps only rep-bearing JE lines (see the
+  sale-import doc); to let `fixer.php` purge any pre-existing no-rep `Forecast.Sales` JE rows with no
+  separate delete script, the **NS-side** JE SuiteQL in both `checker.php` and `fixer.php` was scoped
+  to rep-bearing lines while the **FC-side reads stay UNFILTERED** — a deliberate, load-bearing
+  asymmetry (NS rep-only < FC still-has-no-rep → FIND flags → existing per-`'journalEntry'` orphan
+  delete removes them; filtering the FC side too would hide them forever). A blanket
+  `DELETE WHERE salesRepEmployeeId IS NULL` is wrong (locally-null also = NS-present-but-unmapped rep,
+  which must survive). **Perf:** replaced `AND tl.custcol_sales_rep_line IS NOT NULL` with
+  `AND ... > 0` in checker (JE arm) + fixer (both JE queries) — a clean full-window checker run dropped
+  from **~122s to ~8s** purely from the predicate form, with inline anti-revert comments. Current prod
+  had 0 no-rep JE rows, so the cleanup sweep was a no-op (filter is preventative). (dfranks)
 - 2026-06-30 — **Folded JOURNAL ENTRIES into the SALES reconciliation in `fixer.php` + `checker.php`
   (TRUE-79862).** Architecture decision (dfranks): a JE is **not its own category** — JE rows live in the
   same `Forecast.Sales` table (`netsuiteTransactionType='journalEntry'`) as the four sale types, so the

package/knowledge/2.0/apps/_underscore/features/forecast-sale-import.md

@@ -6,7 +6,7 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-06-29
+updated: 2026-06-30
 owners: [dfranks]
 files:
   - _underscore/Component/Forecast/SaleImport/SaleImport.php
@@ -163,6 +163,19 @@ location, memo). Handler `_Worker_Netsuite_JournalEntry`: `post`/`put` → `sync
 item-line upsert/reconcile machinery (`syncLines`/`guardedInsert`/`deleteRows`).
 
 **Mapping model (durable design):**
+- **Only JE lines that carry a sales rep are imported (TRUE-79968, dfranks 2026-06-30).**
+  `buildJournalEntryRows` applies a **PER-LINE** filter **before** the (salesRep,item)
+  bucketing: a GL line whose **raw** `custcol_sales_rep_line` is empty is **skipped**
+  (`continue`). Consequences — a JE with no rep on any line yields **zero** `Forecast.Sales`
+  rows; a mixed JE keeps only its rep-bearing groups; offset/balancing lines (which omit the
+  custom column) drop out naturally. **The test is on the RAW NetSuite line field, NOT the
+  mapped local `salesRepEmployeeId`** — so a rep present in NetSuite but **not yet mapped** to
+  a local `Forecast.Employees` row **still qualifies** (its group keeps a null
+  `salesRepEmployeeId`). This is deliberate: it keeps no-rep JE noise out of Forecast revenue
+  while not silently dropping a real-but-unmapped rep's revenue. (A blanket
+  `DELETE WHERE salesRepEmployeeId IS NULL` would be **wrong** for the same reason — locally
+  null also means "rep present in NS but unmapped"; any cleanup must re-derive "no raw rep"
+  from NetSuite. See the reconciliation doc for how the fixer sweep does exactly that.)
 - **Sales rep is now LIVE on JE lines** as the custom column **`custcol_sales_rep_line`**
   (`JE_LINE_SALESREP_FIELD = 'custcol_sales_rep_line'`, wired 2026-06-29 after live Apr–May
   2026 probing). The value is a NetSuite **employee REFERENCE object** `{links, id, refName}`;
@@ -429,6 +442,20 @@ record is deleted in NetSuite.)
 - The cron's sign handling is not portable here — see Sign convention.
 
 ## Change history
+- 2026-06-30 — **JE import now keeps ONLY rep-bearing lines** (TRUE-79968, dfranks).
+  `buildJournalEntryRows` skips any GL line whose **raw** `custcol_sales_rep_line` is empty,
+  applied **per-line before** the (salesRep,item) bucketing: a no-rep JE → 0 Sales rows, a
+  mixed JE keeps only rep-bearing groups. The filter tests the **RAW NS field**, not the
+  mapped local `salesRepEmployeeId`, so a rep present in NetSuite but unmapped to
+  `Forecast.Employees` still qualifies (group stays null `salesRepEmployeeId`) — which is also
+  why a blanket `DELETE WHERE salesRepEmployeeId IS NULL` is wrong; cleanup must re-derive
+  "no raw rep" from NetSuite. No-rep cleanup is handled by the existing `fixer.php` orphan-
+  delete sweep (see the reconciliation doc), not a separate delete script — in current prod
+  there were 0 no-rep JE rows so the filter is preventative going forward.
+  **Confirmed the JE webhook handler `_Worker_Netsuite_JournalEntry`
+  (worker2/Worker/Netsuite/JournalEntry.php) holds no business logic** — post/put delegate to
+  `_Component_Forecast_SaleImport::syncJournalEntry`, delete to `removeAllJournalEntry`; editing
+  the importer in `_underscore` is the single correct entry point for JE import changes. (dfranks)
 - 2026-06-29 — **Wired JE sales-rep ingestion live** (TRUE-79862): set
   `JE_LINE_SALESREP_FIELD = 'custcol_sales_rep_line'` after live Apr–May 2026 probing
   confirmed the rep is now present on JE lines as that custom column — a NetSuite employee

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.256",
+  "version": "1.0.257",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",