toga-ai 1.0.57 → 1.0.58

package/knowledge/1.0/apps/library/INDEX.md

@@ -4,3 +4,4 @@
 |-----|---------|-------|
 | [Library (1.0 Framework) Architecture](architecture.md) | `library` is the shared library repository for **all 1.0 (legacy) applications** — the `App_` framework. | library/_.php, library/app/, library/browser/ |
 | [Elite Freshservice Sync (library)](features/elite-freshservice-sync.md) | `App_Api_Toga2` in `library/app/api/toga2.php` orchestrates bidirectional sync between TOGA 2 and TOGaDesk. | library/app/api/toga2.php |
+| [NetSuite SuiteQL/REST Shim — Field Semantics](features/netsuite-suiteql-rest-shim.md) | `App_Api_Netsuite_Rest` is the REST/SuiteQL replacement for the deprecated NetSuite SOAP toolkit. | library/app/api/netsuite/rest.php |

package/knowledge/1.0/apps/library/features/netsuite-suiteql-rest-shim.md

@@ -0,0 +1,117 @@
+---
+title: NetSuite SuiteQL/REST Shim — Field Semantics
+framework: "1.0"
+repo: library
+project: Library
+client: shared
+type: feature
+status: active
+updated: 2026-06-11
+owners: [dfranks]
+files:
+  - library/app/api/netsuite/rest.php
+related:
+  - ../architecture.md
+  - ../../worker/features/forecast2-netsuite-reconciliation.md
+---
+
+## Summary
+
+`App_Api_Netsuite_Rest` is the REST/SuiteQL replacement for the deprecated NetSuite SOAP
+toolkit. Its `list*()` methods return **shim objects whose PHP class/shape matches the old
+SOAP records**, so existing cron code keeps working. This doc captures the **non-obvious
+SuiteQL/REST field semantics** discovered while building and reconciling the Forecast2 sync —
+the things that silently produce wrong totals or duplicate rows if you don't know them.
+
+Production NetSuite code must use REST (SuiteQL/REST), not SOAP — SOAP is being phased out and
+is OK only for ad-hoc troubleshooting.
+
+## Key files / entry points
+
+- `library/app/api/netsuite/rest.php`
+  - `listSales($from, $to, $entityIds, $dateField)` — Invoice/CashSale/CashRfnd/CustCred.
+  - `listSalesOrders($from, $to, $entityIds)` — SalesOrd.
+  - Both do a SuiteQL header query, then (historically) a **per-id REST GET** per record for
+    fields SuiteQL doesn't expose on `transaction`, then a chunked `transactionline` query.
+
+## How it works
+
+Header SuiteQL → per-record detail → line SuiteQL, assembled into SOAP-shaped shims. Sign and
+line-number conventions below are applied so the shim matches what SOAP returned and what the
+Forecast2 tables already store.
+
+## Field semantics / gotchas (verified on the live account)
+
+- **Line number = `transactionline.id`, NOT `linesequencenumber`.** `tl.id` equals the REST
+  `item[].line` value (the stable SOAP-era line key); `linesequencenumber` is physical order and
+  re-maps/collides when SOAP-era records are edited. Verified: sales orders 106 orders / 344
+  lines (2019–2029, incl. edited orders with line-id gaps), invoices + cash sales 62/62.
+  `listSales` historically resolved line via `uniquekey → REST lineUniqueKey → line`; `tl.id`
+  gives the same answer in bulk without the per-id GET.
+
+- **ShipItem lines have `costestimate = NULL`** (100% of 42K+ lines, 2025–26). In any profit
+  expression `SUM(-foreignamount + costestimate)`, a NULL cost makes the whole term NULL and SQL
+  **drops the row from the SUM** — so shipping *revenue* still counts but shipping *profit* does
+  not, silently understating NS profit. Always `NVL(tl.costestimate, 0)`. (This caused a fake
+  $663K "gap" in the audit tools — see the reconciliation doc.)
+
+- **Cash refunds & credit memos carry extra COGS/inventory lines.** SuiteQL `transactionline`
+  with `mainline='F' AND item>0` returns, per item, the customer line **plus** a `CUSTOMERRETURN`
+  and an `ASSET` posting (all `item>0`), which REST's `item[].items` sublist omits. They are
+  distinguished by **`tl.iscogs = 'T'`** (and `accountinglinetype` IN `CUSTOMERRETURN`/`ASSET`).
+  Filter **`tl.iscogs = 'F'`** to get exactly the customer-facing lines. Invoices/cash sales do
+  not have this (0 extra lines). Without the filter, bulk imports duplicate refund/memo lines.
+
+- **Sign conventions are per-type and self-consistent.** SuiteQL returns
+  `foreignamount`/`quantity`/`costestimate` **negative for invoices & cash sales, positive for
+  credit memos & cash refunds**. A uniform `-foreignamount` therefore yields **positive** revenue
+  for invoices and **negative** for reversals — the agreed Forecast2 convention. Do NOT also apply
+  a per-type factor (it double-negates reversals back to positive). The shim flips
+  `amount = -foreignamount`, `costEstimate = -costestimate`.
+
+- **`createdFrom` is not a usable SuiteQL column.** `transaction.createdfrom` is **always NULL**
+  in SuiteQL (26/26). It is recoverable from `previoustransactionlinelink.previousdoc` excluding
+  `previoustype = 'Opprtnty'` (an invoice links to both its SO and its opportunity; REST's
+  `createdFrom` is the SO). **But this is non-deterministic for multi-link transactions** (credit
+  memos return several non-Opprtnty `previousdoc` rows in arbitrary order) — two runs can pick
+  different values. REST `createdFrom` (per-id GET) is the only deterministic source. Treat the
+  link table as best-effort, not authoritative.
+
+- **AR open balance = `transaction.foreignamountunpaid`** (== REST `amountRemaining`, 12/12 on
+  open invoices). It is **NULL on fully-paid invoices and on cash sales/refunds** — wrap in
+  `NVL(...,0)` for a "settled = 0.00" convention. Valid balance columns are **`foreignamountunpaid`,
+  `foreignamountpaid`, `foreigntotal`** only; `amountremaining` and `amountunpaid` do **not exist**
+  and 400 the query.
+
+- **SO originating line** lives in `previoustransactionlinelink` (correlated subquery on
+  `previoustype='SalesOrd'`), because `transactionline` has **no `orderline` column**.
+
+- **TO_DATE rejects impossible dates** (e.g. `2026-06-31`) with an opaque HTTP 400 — validate
+  calendar dates before building SuiteQL.
+
+## Data model
+
+Reads NetSuite `transaction`, `transactionline`, `previoustransactionlinelink` via SuiteQL and
+`/record/v1/{type}/{id}` via REST. Writes nothing.
+
+## Client variations
+
+None — uniform across clients (NetSuite is a single shared account).
+
+## Gotchas / known issues
+
+- The shim must stay PHP 7.2-compatible (library is 7.2 prod): no arrow functions, typed props,
+  `??=`, or `match`. Lint with `C:\xampp7\php\php.exe -l` before deploying.
+- Field availability varies by NetSuite account **and** record type — probe the live account
+  before assuming a column/relationship exists.
+
+## Change history
+
+- 2026-06-11 — Documented bulk SuiteQL field semantics (tl.id==line, ShipItem NULL cost, iscogs
+  COGS filter, createdFrom non-determinism, foreignamountunpaid, sign conventions) surfaced while
+  moving the Forecast2 trueup tools off per-id REST GETs. (dfranks)
+
+## Related docs
+
+- `../architecture.md` — Library core architecture.
+- `../../worker/features/forecast2-netsuite-reconciliation.md` — the tooling that applies these.

package/knowledge/1.0/apps/worker/INDEX.md

@@ -3,3 +3,4 @@
 | Doc | Summary | Files |
 |-----|---------|-------|
 | [Worker (1.0 Framework) Architecture](architecture.md) | `worker` is the legacy (**1.0** `App_` framework) **background-job tier**. | worker/index.php, worker/_/app/framework.php, worker/crons/, worker/schedules/, worker/ebs/cron.worker.php, worker/.ebextensions/035_cron.worker.config |
+| [Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling](features/forecast2-netsuite-reconciliation.md) | CLI tools to **audit** and **repair** drift between the production `Forecast` DB (core2) and NetSuite. | test/@dave/reconcile_netsuite_totals.php, test/@dave/analyze_netsuite_forecast_diff.php, test/@dave/trueup_sales.php, test/@dave/trueup_open_orders.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |

package/knowledge/1.0/apps/worker/features/forecast2-netsuite-reconciliation.md

@@ -0,0 +1,111 @@
+---
+title: Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling
+framework: "1.0"
+repo: worker
+project: Worker
+client: shared
+type: feature
+status: active
+updated: 2026-06-11
+owners: [dfranks]
+files:
+  - test/@dave/reconcile_netsuite_totals.php
+  - test/@dave/analyze_netsuite_forecast_diff.php
+  - test/@dave/trueup_sales.php
+  - test/@dave/trueup_open_orders.php
+  - worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php
+related:
+  - ../architecture.md
+  - ../../library/features/netsuite-suiteql-rest-shim.md
+---
+
+## Summary
+
+CLI tools to **audit** and **repair** drift between the production `Forecast` DB (core2) and
+NetSuite. The 5-min importer (`common_import_sales_from_netsuite.php`) filters on
+`lastmodifieddate` and the nightly discrepancy-fix only looks back 30 days, so older rows whose
+NetSuite values change (e.g. cost re-valuation) are never re-pulled. These tools close that gap
+by reconciling a chosen tranDate range directly against NetSuite.
+
+## Key files / entry points
+
+- `reconcile_netsuite_totals.php [from] [to]` — category grand totals NS vs Forecast2 (Sales,
+  Sales Profit, Open Orders, Opportunities) with deltas. Read-only. Connects explicitly to the
+  **prod core2 reader**; NetSuite via SuiteQL SUMs.
+- `analyze_netsuite_forecast_diff.php [from] [to]` — decomposes the delta **per transaction** into
+  NS_ONLY (missing from FC), FC_ONLY (stale/extra), DRIFT (value differs). Read-only.
+- `trueup_sales.php --from --to [--chunk-days N] [--prod] [--dry-run]` — makes `Forecast.Sales`
+  match NetSuite for a tranDate range (insert/update/delete per line).
+- `trueup_open_orders.php --from= --to= [--prod] [--dry-run] [--verbose]` — same for
+  `Forecast.OpenOrderItems` (currently-open SOs whose tranDate falls in range).
+
+## How it works
+
+- **Bulk SuiteQL (not per-id GET).** Both trueup tools were rewritten to fetch detail via chunked
+  `WHERE transaction IN (...)` SuiteQL using `tl.id` as the line number (see the shim doc for why
+  that's valid), replacing the ~1.5s-per-record REST GET. Result: a day that took ~48 min now
+  runs in seconds (parity verified — `trueup_sales` matched the old `listSales` path 1813/1813
+  records, 3675/3675 line cells on a sample day). `trueup_open_orders` builds its header+lines
+  per chunk; `trueup_sales` does it in a local `fetchSalesBulk()` that returns the same shim shape
+  `App_Api_Netsuite_Rest::listSales()` produced, so the reconciliation loop is unchanged.
+- **`--prod` targeting (self-contained).** trueup writes via `App_Database(...,'db_forecast2')`.
+  On dev laptops that registry config points at **localhost XAMPP**; `--prod` overrides it in
+  memory at runtime by reading `[database_forecast2]` from `worker/config.worker.ini` (the core2
+  **writer**). `App_Database::registerDatabaseConnect` reads the registry config lazily on first
+  query, so the override also covers the keepalive reconnect. **Do not edit any `config.*.ini`** to
+  target prod — that mutates shared dev state and silently leaves the laptop pointed at the writer.
+- **Chunking + keepalive.** trueup_sales chunks by `--chunk-days` (default 7) and commits per
+  chunk so a crash resumes. Both ping/reconnect `db_forecast2` before DB work (Aurora drops idle
+  links during long NS calls).
+
+## Data model
+
+`Forecast.Sales`, `Forecast.OpenOrderItems` on the **core2** cluster
+(reader `reader1.core.database.togahub.com`, writer `writer.core.database.togahub.com`). Source
+of truth is NetSuite; FC is made to match.
+
+## Client variations
+
+None — Forecast2 is a single shared dataset.
+
+## Gotchas / known issues
+
+- **NVL the cost in NS profit SUMs.** `reconcile`/`analyze` compute NS profit as
+  `SUM(-foreignamount + NVL(costestimate,0))`. Without `NVL`, ShipItem lines (NULL cost) drop from
+  the SUM and NS profit is understated, producing a **fake FC-over-NS "gap"** ($663K across 2025).
+  After the fix, the real residual is the opposite sign and small: **~$87K NS-over-FC**,
+  concentrated **Dec 2025–Apr 2026**, driven by NetSuite re-valuing `costestimate` on older
+  invoices that no `lastmodifieddate`-based sync re-pulls. (Revenue reconciles to the penny; only
+  profit drifts.)
+- **Compare money at 2 decimals.** DB columns store 2dp but PHP `revenue - cost` carries float
+  dust (`313.6` vs `313.60000001`); raw `!=` produced thousands of phantom UPDATEs that re-wrote
+  identical values (1,839 on one open-orders run). Both tools now compare `round((float)$x, 2)`.
+- **`trueup_sales` does not manage `createdFrom` or `amountDue`.**
+  - `createdFrom` — non-deterministic from the link table (see shim doc); the REST-based importer
+    owns it. trueup preserves it on UPDATE, leaves NULL on INSERT.
+  - `amountDue` (`Forecast.Sales`) and `dtPendingBilling` (`OpenOrderItems`) are **not yet in
+    prod** — those columns exist only in local dev (TRUE-78923 / TRUE-79081 migrations are
+    local-only). Both trueup tools have those fields **removed** so they run against prod. ⚠ The
+    prod migrations (`dbchanges2/Forecast/2026-06-09 *.sql`) must be applied **before** this
+    session's importer/discrepancy-fix changes (which write those columns) deploy, or production
+    crashes with `Unknown column`.
+- **Always dry-run prod first** and confirm the change counts are real drift, not artifacts —
+  e.g. 2023-01-03 showed 240 "updates" that were entirely createdFrom backfill + profit float-dust
+  (zero real drift); after the fixes it correctly reports `unchanged=240`.
+- These tools live in `test/@dave/` (developer tooling), but `trueup_open_orders` has been run
+  against production. The `Defaults`/checkpoint mechanics of the scheduled sync are separate.
+
+## Change history
+
+- 2026-06-11 — `trueup_sales` rewritten to bulk SuiteQL (`fetchSalesBulk`, ~300× faster);
+  `amountDue` + `createdFrom` dropped; float-dust fixed; `--prod` flag added. `reconcile`/`analyze`
+  profit queries `NVL(costestimate,0)` — established the $663K gap was an audit artifact and the
+  real drift is ~$87K NS-over-FC in Dec 2025–Apr 2026. (dfranks)
+- 2026-06-10 — `trueup_open_orders` bulk SuiteQL refactor + `--prod`; phantom float-dust UPDATEs
+  eliminated; Aurora idle-drop reconnect. (dfranks)
+
+## Related docs
+
+- `../../library/features/netsuite-suiteql-rest-shim.md` — the SuiteQL/REST field semantics these
+  tools rely on.
+- `../architecture.md` — Worker architecture (forecast2 sync lives in `crons/toga2/forecast2/`).

package/knowledge/INDEX.md

@@ -4,8 +4,8 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 
 ## 1.0 framework
 
-- **library** (Library) _(framework core)_ — 2 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
-- **worker** (Worker) — 2 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
+- **library** (Library) _(framework core)_ — 3 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
+- **worker** (Worker) — 3 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
 
 ## 2.0 framework
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.57",
+  "version": "1.0.58",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",