toga-ai 1.0.239 → 1.0.240

package/knowledge/1.0/apps/worker/INDEX.md

@@ -5,7 +5,7 @@
 | [Worker (1.0 Framework) Architecture](architecture.md) | `worker` is the legacy (**1.0** `App_` framework) **background-job tier**. | worker/index.php, worker/_/app/framework.php, worker/crons/, worker/schedules/, worker/ebs/cron.worker.php, worker/.ebextensions/035_cron.worker.config |
 | [Compass MA Sales Order Exception Report](features/compass-ma-sales-order-exception-report.md) | A worker cron that emails operations the "Compass Refresh Exception Report" — Compass `MA%` sales orders whose corresponding Office Depot (ODP) sales order has | worker/crons/toga2/compass/workflow/7_generate_ma_sales_order_exception_report.php |
 | [Compass Partial In-Transit & Delivered Emails (per package)](features/compass-partial-in-transit-delivered-emails.md) | Compass USA and Compass Canada send a **per-package** in-transit email (and a matching delivered email) instead of one email listing the whole order. | worker/crons/toga2/compass/update_salesorder_status_from_odp.php, worker/crons/toga2/compasscanada/workflow/3_update_salesorder_status_from_grand_and_toy.php |
-| [Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling](features/forecast2-netsuite-reconciliation.md) | CLI tools to **audit** and **repair** drift between the production `Forecast` DB (core2) and NetSuite. | test/@dave/checker.php, test/@dave/looper.php, test/@dave/reconcile_netsuite_totals.php, test/@dave/fixer.php, test/@dave/analyze_netsuite_forecast_diff.php, test/@dave/trueup_sales.php, test/@dave/trueup_open_orders.php, test/@dave/loop_trueup_open_orders.php, test/@dave/trueup_opportunities.php, test/@dave/probe_sales_gap_direct.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_profit_invoices.php, test/@dave/probe_profit_gap.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_open_orders.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/schedules/cron.worker.infrastructure.json |
+| [Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling](features/forecast2-netsuite-reconciliation.md) | CLI tools to **audit** and **repair** drift between the production `Forecast` DB (core2) and NetSuite. | test/@dave/checker.php, _underscore/Component/Forecast/SaleImport/SaleImport.php, test/@dave/looper.php, test/@dave/reconcile_netsuite_totals.php, test/@dave/fixer.php, test/@dave/analyze_netsuite_forecast_diff.php, test/@dave/trueup_sales.php, test/@dave/trueup_open_orders.php, test/@dave/loop_trueup_open_orders.php, test/@dave/trueup_opportunities.php, test/@dave/probe_sales_gap_direct.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_profit_invoices.php, test/@dave/probe_profit_gap.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_open_orders.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/schedules/cron.worker.infrastructure.json |
 | [NetSuite → TOGa Supply Per-Client Sync (thin wrappers)](features/netsuite-togasupply-per-client-sync.md) | Syncs NetSuite transactions (sales orders, purchase orders, invoices, item receipts, item fulfillments, inventory adjustments) into each TOGa Supply (2.0) clien | worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_canon.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql |
 | [Prudential: Send Shipments for the Day report (daily cron)](features/send-shipments-for-the-day.md) | Daily cron (9:00 PM) that emails Prudential and Dell stakeholders an Excel report of all devices shipped that day, including tracking number, serial number, emp | worker/crons/notifications/reports/send_shipments_for_the_day.php |
 | [Onboarding a Client to the NetSuite TOGa Supply Sync](workflows/onboarding-client-to-netsuite-togasupply-sync.md) | How to add a new TOGa 2 client to the per-client NetSuite → TOGa Supply importer (`worker/crons/toga2/netsuite/`). | worker/crons/toga2/netsuite/sync_togasupply.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql |

package/knowledge/1.0/apps/worker/features/forecast2-netsuite-reconciliation.md

@@ -6,10 +6,11 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-06-29
+updated: 2026-06-30
 owners: [dfranks]
 files:
   - test/@dave/checker.php
+  - _underscore/Component/Forecast/SaleImport/SaleImport.php
   - test/@dave/looper.php
   - test/@dave/reconcile_netsuite_totals.php
   - test/@dave/fixer.php
@@ -32,6 +33,7 @@ related:
   - ../architecture.md
   - ../../library/features/netsuite-suiteql-rest-shim.md
   - ../../library/features/netsuite-suiteql-api-reference.md
+  - ../../_underscore/features/forecast-sale-import.md
 ---
 
 ## Summary
@@ -202,6 +204,47 @@ by reconciling a chosen tranDate range directly against NetSuite.
   on `tranDate`, not NS `lastmodifieddate`, so a **symmetric** lastmodified fingerprint isn't possible — it
   would require NS-pull-then-id-lookup or the recency-pruned approach.
 
+- **JOURNAL ENTRIES are reconciled WITHIN the SALES category, not as a separate category (TRUE-79862
+  architecture decision, dfranks).** A JE is **not its own reconciliation category** — JE rows live in
+  the **same `Forecast.Sales` table** as the four sale types (`netsuiteTransactionType='journalEntry'`),
+  written by the same `_Component_Forecast_SaleImport` importer (see the sale-import doc's *JournalEntry
+  import* section). So the **sales pass covers the WHOLE `Sales` table**, sale-types + JEs together:
+  - **`fixer.php`** — `findSalesDiscrepancies` merges, on the **NS side**, the sale-type per-line aggregate
+    with `journalEntryNsTotals()` (a GL-account-grouped JE helper); the **FC side reads ALL `Sales` rows**
+    (no type filter); each discrepancy is tagged **`kind` (`'sale'` | `'je'`)**. The FIX dispatch routes
+    sale ids to `fixSales()` and JE ids to `fixJournalEntries()`. The per-corrector DELETE scoping stays
+    **per-type** (`fixSales` delete-candidates scoped to the 4 sale types; `fixJournalEntries` scoped to
+    `'journalEntry'`) so neither corrector deletes the other's rows.
+  - **`checker.php`** — the sales `nsSub` is a **`UNION ALL`** of the sale-type arm and a JE arm; the sales
+    `fcSub` reads **all `Sales` rows**. checker needs **no per-item/JE-specific logic** because its
+    fingerprint compares per-transaction (per-id) totals + the id-set, which are **item-agnostic**.
+- **JE→`Forecast.Sales` reconciliation math + scope (mirrors `_Component_Forecast_SaleImport::buildJournalEntryRows`).**
+  Only JE lines posting to the revenue/cost **account-NUMBER allowlist** are sales-relevant: revenue
+  `41100/41300/41500`, cost `51100/51200` (exact-number, never prefix — see the sale-import doc for the
+  sub-account traps). `revenue = Σ(−foreignamount)` over revenue-acct lines; `profit = Σ(−foreignamount)`
+  over **all** allowlisted lines (cost lines net profit down via debit-positive `foreignamount`). Group by
+  **(local salesRep, item)** → one Sales row per group; synthetic `lineNumber` from
+  `ksort(json_encode([salesRepId,itemId]))` then `1..N` — **fixer MUST key identically** or its upsert
+  dupes against the webhook importer's rows. JE `foreignamount`s are **cent-precision**, so a per-JE
+  sum-then-round equals the importer's per-salesRep round-then-sum **exactly** — this is what keeps
+  checker's "in sync" == fixer's "nothing to fix" for JEs (the compute-identically invariant applied to JEs).
+  Scope facts: of **~1235 JEs** in 2025-01-01..present, only **~295** touch a revenue/cost account; **~34**
+  of those net to **0 rev AND 0 profit** and import no rows. JEs hitting only bank/equity/tax/AP accounts
+  are correctly excluded **by design** (e.g. JE 7213407 = a Bank↔Equity reclass, 0 allowlist lines → never
+  imported — not a miss). An accrual JE and its NetSuite auto-reversal are **separate transactions**, each
+  reconciled independently by its own `internalId`.
+- **JE item dimension is NULL today; here is how to enable it later.** `itemId` is null on JE Sales rows
+  because the JE import path reads item **only** via the configurable `JE_LINE_ITEM_FIELD` (currently null)
+  — it does **not** read native `transactionline.item`, so even a populated native `tl.item` would NOT flow
+  through. To enable: (1) set `JE_LINE_ITEM_FIELD` to the source field (`'item'` for native, or a
+  `custcol_*` if a custom column is added) — `journalLineLookup` then resolves it via
+  `lookupId('Items','netsuiteItemInternalId',…)` and the group key already carries the item slot; AND (2)
+  update **`fixer.php`** to SELECT the item column in its JE SuiteQL, resolve NS item→local id (reuse
+  `createMissingForecastItemFromNetSuite` self-heal), and extend the group key to `(salesRep, item)`.
+  **`checker.php` needs no change.** Consider whether item should self-heal (salesRep does **not** — see
+  the sale-import doc). The salesRep dimension itself is the JE line custom column `custcol_sales_rep_line`
+  (wired under TRUE-79862).
+
 ## Data model
 
 `Forecast.Sales`, `Forecast.OpenOrderItems` on the **core2** cluster
@@ -214,6 +257,16 @@ None — Forecast2 is a single shared dataset.
 
 ## Gotchas / known issues
 
+- **CROSS-TYPE CONTAMINATION: once JEs share `Forecast.Sales`, any FC Sales read missing a type scope
+  treats JE rows as rogue sales.** Discovered live (TRUE-79862): the SALES fix **deleted 74 JE rows
+  (8 JEs) as "stale sales"** because `findSalesDiscrepancies`'s FC query had **no type filter**, so JE
+  rows looked like FC-only sales that the NS-sales aggregate never returns. **Correct resolution is NOT
+  to filter JEs OUT of the sales pass** — it is to reconcile the **whole table together** (NS side =
+  sale-types `UNION` JE; FC side = all rows) and **route the FIX by `kind`** (`'sale'`→`fixSales`,
+  `'je'`→`fixJournalEntries`), keeping each corrector's DELETE scoping per-type so neither deletes the
+  other's rows. (See the JE-within-Sales architecture under *How it works*.) General lesson: when two
+  transaction families share one table, an audit read with no type scope manufactures phantom FC-only
+  deltas for the other family.
 - **An NS-DELETED invoice leaves STALE cost-only `Forecast.Sales` rows the add/update-only import can
   never remove.** The 5-min `import_sales.php` cron is **add/update-only — it has no delete path** (only
   the nightly discrepancy-fix deletes). So when an invoice is **deleted in NetSuite**, its already-imported
@@ -420,6 +473,24 @@ None — Forecast2 is a single shared dataset.
 
 ## Change history
 
+- 2026-06-30 — **Folded JOURNAL ENTRIES into the SALES reconciliation in `fixer.php` + `checker.php`
+  (TRUE-79862).** Architecture decision (dfranks): a JE is **not its own category** — JE rows live in the
+  same `Forecast.Sales` table (`netsuiteTransactionType='journalEntry'`) as the four sale types, so the
+  sales pass reconciles the **whole table**. `fixer.findSalesDiscrepancies` merges the NS sale-type
+  aggregate with `journalEntryNsTotals()`, reads ALL FC Sales rows, tags each diff `kind` (`sale`|`je`),
+  and routes the FIX (`fixSales`/`fixJournalEntries`) with per-type DELETE scoping. `checker`'s sales
+  `nsSub` becomes a `UNION ALL` (sale-types + JE) over all FC Sales rows — no JE-specific logic since its
+  fingerprint is per-id/item-agnostic. **Gotcha recorded: cross-type contamination** — the initial
+  un-typed FC Sales read deleted 74 JE rows (8 JEs) as "stale sales"; fixed by reconciling the families
+  together + routing by kind, not by filtering JEs out. **JE math/scope** (mirrors
+  `buildJournalEntryRows`): allowlist accts 41100/41300/41500 rev + 51100/51200 cost; group by
+  (salesRep,item) with `ksort(json_encode([salesRep,item]))` synthetic lineNumber — fixer keys identically
+  to the importer or it dupes; cent-precision foreignamounts make sum-then-round == importer round-then-sum
+  (compute-identically holds for JEs). Of ~1235 JEs (2025→present) only ~295 hit a rev/cost acct, ~34 net
+  to $0 and import nothing; bank/equity/tax/AP-only JEs excluded by design (not misses). Accrual + NS
+  auto-reversal reconcile independently per internalId. **Item dimension is null today** (`JE_LINE_ITEM_FIELD`
+  null; native `tl.item` not read); enabling it needs the importer field set AND fixer SELECT/resolve/group-
+  by-(salesRep,item) — checker needs no change. (dfranks)
 - 2026-06-29 — **Recorded that `fixer.php`'s OOI path lacks `locationId`/`quantityBackordered`/
   `amountDue`** (planning for TRUE-79162). The Sales path already handles anchor-line `amountDue`
   behind a `forecastColumnExists('Sales','amountDue')` guard; the OpenOrderItems path (lookup SELECT,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.239",
+  "version": "1.0.240",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",