toga-ai 1.0.198 → 1.0.199

package/knowledge/1.0/apps/worker/INDEX.md

@@ -5,7 +5,7 @@
 | [Worker (1.0 Framework) Architecture](architecture.md) | `worker` is the legacy (**1.0** `App_` framework) **background-job tier**. | worker/index.php, worker/_/app/framework.php, worker/crons/, worker/schedules/, worker/ebs/cron.worker.php, worker/.ebextensions/035_cron.worker.config |
 | [Compass MA Sales Order Exception Report](features/compass-ma-sales-order-exception-report.md) | A worker cron that emails operations the "Compass Refresh Exception Report" — Compass `MA%` sales orders whose corresponding Office Depot (ODP) sales order has | worker/crons/toga2/compass/workflow/7_generate_ma_sales_order_exception_report.php |
 | [Compass Partial In-Transit & Delivered Emails (per package)](features/compass-partial-in-transit-delivered-emails.md) | Compass USA and Compass Canada send a **per-package** in-transit email (and a matching delivered email) instead of one email listing the whole order. | worker/crons/toga2/compass/update_salesorder_status_from_odp.php, worker/crons/toga2/compasscanada/workflow/3_update_salesorder_status_from_grand_and_toy.php |
-| [Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling](features/forecast2-netsuite-reconciliation.md) | CLI tools to **audit** and **repair** drift between the production `Forecast` DB (core2) and NetSuite. | test/@dave/reconcile_netsuite_totals.php, test/@dave/analyze_netsuite_forecast_diff.php, test/@dave/trueup_sales.php, test/@dave/trueup_open_orders.php, test/@dave/trueup_opportunities.php, test/@dave/probe_sales_gap_direct.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_profit_invoices.php, test/@dave/probe_profit_gap.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
+| [Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling](features/forecast2-netsuite-reconciliation.md) | CLI tools to **audit** and **repair** drift between the production `Forecast` DB (core2) and NetSuite. | test/@dave/reconcile_netsuite_totals.php, test/@dave/analyze_netsuite_forecast_diff.php, test/@dave/trueup_sales.php, test/@dave/trueup_open_orders.php, test/@dave/loop_trueup_open_orders.php, test/@dave/trueup_opportunities.php, test/@dave/probe_sales_gap_direct.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_profit_invoices.php, test/@dave/probe_profit_gap.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_open_orders.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/schedules/cron.worker.infrastructure.json |
 | [NetSuite → TOGa Supply Per-Client Sync (thin wrappers)](features/netsuite-togasupply-per-client-sync.md) | Syncs NetSuite transactions (sales orders, purchase orders, invoices, item receipts, item fulfillments, inventory adjustments) into each TOGa Supply (2.0) clien | worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_canon.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql |
 | [Prudential: Send Shipments for the Day report (daily cron)](features/send-shipments-for-the-day.md) | Daily cron (9:00 PM) that emails Prudential and Dell stakeholders an Excel report of all devices shipped that day, including tracking number, serial number, emp | worker/crons/notifications/reports/send_shipments_for_the_day.php |
 | [Onboarding a Client to the NetSuite TOGa Supply Sync](workflows/onboarding-client-to-netsuite-togasupply-sync.md) | How to add a new TOGa 2 client to the per-client NetSuite → TOGa Supply importer (`worker/crons/toga2/netsuite/`). | worker/crons/toga2/netsuite/sync_togasupply.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql |

package/knowledge/1.0/apps/worker/features/forecast2-netsuite-reconciliation.md

@@ -6,13 +6,14 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-06-24
+updated: 2026-06-25
 owners: [dfranks]
 files:
   - test/@dave/reconcile_netsuite_totals.php
   - test/@dave/analyze_netsuite_forecast_diff.php
   - test/@dave/trueup_sales.php
   - test/@dave/trueup_open_orders.php
+  - test/@dave/loop_trueup_open_orders.php
   - test/@dave/trueup_opportunities.php
   - test/@dave/probe_sales_gap_direct.php
   - test/@dave/probe_missing_oo_timing.php
@@ -21,6 +22,9 @@ files:
   - test/@dave/probe_profit_invoices.php
   - test/@dave/probe_profit_gap.php
   - worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php
+  - worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_open_orders.php
+  - worker/crons/toga2/forecast2/import_open_orders.php
+  - worker/schedules/cron.worker.infrastructure.json
 related:
   - ../architecture.md
   - ../../library/features/netsuite-suiteql-rest-shim.md
@@ -49,8 +53,29 @@ by reconciling a chosen tranDate range directly against NetSuite.
   NS_ONLY (missing from FC), FC_ONLY (stale/extra), DRIFT (value differs). Read-only.
 - `trueup_sales.php --from --to [--chunk-days N] [--prod] [--dry-run]` — makes `Forecast.Sales`
   match NetSuite for a tranDate range (insert/update/delete per line).
-- `trueup_open_orders.php --from= --to= [--prod] [--dry-run] [--verbose]` — same for
-  `Forecast.OpenOrderItems` (currently-open SOs whose tranDate falls in range).
+- `trueup_open_orders.php --from= --to= [--prod] [--dry-run] [--verbose] [--quiet] [--by-lastmodified]`
+  — same for `Forecast.OpenOrderItems` (currently-open SOs whose tranDate falls in range).
+  - **`--by-lastmodified`** windows **both** passes on `lastmodifieddate` instead of `tranDate`:
+    Step 1 (open-SO reconcile) and Step 3b (stale cleanup). `lastmodifieddate` is a TIMESTAMP →
+    bounded `TO_TIMESTAMP(from 00:00:00)..(to 23:59:59)` in the NetSuite **ET** session tz. Under this
+    mode Step 3b **inverts the lookup**: it asks NS for SOs **modified in the window** that are no longer
+    open (`status NOT IN B/D/E/F`) via id-only paged SuiteQL (**no per-id GET, no per-row request** —
+    cost scales with SOs modified, not with local row count), then deletes the local rows for any still
+    held. This catches an order **billed in the window whose `tranDate` predates it** — exactly the
+    invoice-transform billing case a `tranDate` window misses (see the open-orders sync doc).
+  - **`--quiet`** suppresses all progress (`out()` no-op) and prints the Step 4 results block **only**
+    when `insert+update+delete > 0` (a no-change pass prints nothing); transient NS failures + fatals
+    still go to stderr.
+  - **`--verbose`** was trimmed — removed per-line MATCH/EXCLUDE/NOOP and the always-on per-SO summary;
+    the per-SO header now prints lazily, once, only when the order has an INSERT/UPDATE/DELETE.
+- `loop_trueup_open_orders.php [--prod] [--dry-run] [--verbose] [--quiet] [--sleep=N] [--days-back=N]`
+  — continuous runner that recomputes a rolling `[today−N .. today]` window each pass and re-invokes
+  `trueup_open_orders.php --by-lastmodified` until Ctrl-C. Uses `PHP_BINARY` + `passthru` with
+  `escapeshellarg` over a **fixed flag allowlist** (`--prod`/`--dry-run`/`--verbose`/`--quiet`); survives
+  a failed pass. With `--quiet` it also suppresses its own per-pass banner + the sleeping line, so the
+  only output is a trueup Step 4 block when rows actually change. **This is the stopgap for the real-time
+  open-order removal gap** (an SO billed via invoice-transform fires no webhook → `removeAll` never runs —
+  see the open-orders sync doc).
 - `trueup_opportunities.php --from --to [--prod] [--dry-run]` — same for `Forecast.Opportunities`
   by tranDate range. Pure SuiteQL (no per-id GET), so it's fast: a 2026-YTD prod run reconciled
   ~4,000 opps in ~13s to a $0 delta.
@@ -206,11 +231,39 @@ None — Forecast2 is a single shared dataset.
   revenue. `trueup_sales.php` repairs them: it compares against NS by trandate, detects the sign
   mismatch, and delete+re-inserts with the correct sign. See the shim/reference docs for why a
   uniform `-foreignamount` is correct and an extra per-type `$factor` double-negates.
+- **`OpenOrderItems.dtPendingBilling` does not exist in prod and NO prod code writes it.** A `SHOW
+  COLUMNS` on the core2 reader confirms the column is **present locally, absent in prod** — the
+  TRUE-79081 migration was never deployed. `periodic_forecast_discrepancy_fix_open_orders.php` **intentionally
+  does not write it** (explicit guard comments; writing it would raise MySQL 1054) and worker2's
+  `_Worker_Netsuite_SalesOrder` never references it. So **nothing currently sets
+  `OpenOrderItems.dtPendingBilling` in prod.** (This **corrects** the earlier note that it was "fixed in
+  both writers" on 2026-06-15 — the writers were since changed to **skip** it pending the prod migration.)
+- **Of the legacy open-orders crons, only the discrepancy-fix is active — and it CLEANS billed orders,
+  it does not resurrect them.** `import_open_orders.php` is **`"active": 0` (DISABLED)** in
+  `worker/schedules/cron.worker.infrastructure.json`, but its flag `SHOULD_SYNC_OPEN_ORDERS = true` is
+  still set — a **latent re-enable footgun** (flipping `active` back on would resume the cron with its
+  known cascade/date-window quirks). The **only active legacy writer** of `OpenOrderItems` is
+  `periodic_forecast_discrepancy_fix_open_orders.php` (`active:1`, daily 3:00 AM): it INSERTs only for
+  currently-**open** SOs (status B/D/E/F) and its **Step-3b stale-cleanup DELETEs** rows whose SO is no
+  longer open — so it **removes** billed/closed orders rather than resurrecting them. It is therefore
+  **not** a resurrection risk against the webhook `removeAll` path; it is in fact the current backstop
+  deleter for orders the webhook never sees billed (the invoice-transform gap).
 - These tools live in `test/@dave/` (developer tooling), but `trueup_open_orders` has been run
   against production. The `Defaults`/checkpoint mechanics of the scheduled sync are separate.
 
 ## Change history
 
+- 2026-06-25 — **`trueup_open_orders` gained `--by-lastmodified` / `--quiet` / leaner `--verbose`; added
+  `loop_trueup_open_orders.php` as the stopgap for the real-time billed-order removal gap.**
+  `--by-lastmodified` windows both Step 1 and Step 3b on `lastmodifieddate` (ET TIMESTAMP bounds) and
+  inverts Step 3b to an id-only paged SuiteQL "modified-and-no-longer-open" lookup (no per-id GET), so it
+  catches an order billed in-window whose `tranDate` predates it. `--quiet` prints only a non-zero Step 4
+  block. `loop_trueup_open_orders.php` is a continuous rolling-window runner (PHP_BINARY + passthru over a
+  fixed flag allowlist). Also recorded that **`OpenOrderItems.dtPendingBilling` is local-schema-only and no
+  prod code writes it** (corrects the earlier "fixed in both writers" note — both writers now skip it), and
+  that of the legacy open-orders crons **only `periodic_forecast_discrepancy_fix_open_orders.php` is active**
+  and it cleans up billed/closed orders rather than resurrecting them (while `import_open_orders.php` is
+  `active:0` but still flags `SHOULD_SYNC_OPEN_ORDERS=true` — a latent re-enable footgun). (dfranks)
 - 2026-06-24 — **Sales audit correctness pass.** Fixed a phantom Sales delta: `reconcile`'s NS Sales
   SUM/diff now applies the sync's status exclusions (Unapproved Payment / Voided / Rejected) so it can't
   count rows the sync never stores (closed revenue delta to $0). Added a Sales per-transaction presence

package/knowledge/2.0/apps/_underscore/features/forecast-sale-import.md

@@ -6,7 +6,7 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-06-24
+updated: 2026-06-25
 owners: [dfranks]
 files:
   - _underscore/Component/Forecast/SaleImport/SaleImport.php
@@ -154,4 +154,11 @@ would be needed only for a future real-time item webhook, **not** for this Sales
 - The cron's sign handling is not portable here — see Sign convention.
 
 ## Change history
+- 2026-06-25 — **Verified `sync()` end-to-end across all four sale types** (prod NS reads, local Forecast
+  writes; 40+ records spanning invoice/cashSale/creditMemo/cashRefund): revenue reconciled **to the penny**
+  against an independent SuiteQL `SUM(-foreignamount)` oracle on every record; the uniform sign factor held
+  (invoice/cashSale positive, creditMemo/cashRefund negative); kit handling correct (itemGroup line and the
+  no-item-id summary line dropped, the real component line kept); tax-reversal credit memos correctly net to
+  $0 line revenue; the guarded `INSERT … WHERE NOT EXISTS` proved idempotent (re-sync → 0 duplicates); and
+  item self-heal worked for old records. No code change — confirms the existing behavior. (dfranks)
 - 2026-06-24 — Built the Forecast.Sales real-time webhook importer (shared `_Component_Forecast_SaleImport` engine + four thin worker2 handlers), replacing the legacy cron SALES section; decided the uniform-factor sign convention against the raw REST record; deduped `fetchRecord`/sublist pagination onto `_Component_Forecast_Db`. (dfranks)

package/knowledge/2.0/apps/worker2/features/netsuite-opportunity-sync.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-06-18
+updated: 2026-06-25
 owners: ["dfranks"]
 files:
   - worker2/Worker/Netsuite.php
@@ -321,6 +321,25 @@ None — platform-wide Forecast sync.
   HTTP call so a concurrent drainer pass skips it (`Sending` ∉ sendable), narrowing the double-send
   window to the load→claim gap. `findSendableIds` must list every sendable status
   (`Created`,`Pending`,`Retry`,`Sending`) — omitting `Created` silently matches nothing (`candidates:0`).
+- **The enqueuer is generic over all record types but RELEASED only on Opportunity + Sales Order.**
+  `ue_api_msg_queue_enqueue` (`customscript_ue_amq_enqueue`) is generic via `RECORD_TYPE_MAP`, fires
+  `afterSubmit` for create/edit/xedit/delete, has **no execution-context filter**, and runs with
+  `DEV_OVERRIDE.enabled=false` in prod (posts to `webhook.togahub.com/netsuite`). Probe-confirmed it is
+  **`Released` on only two record types**: Opportunity (`customdeploy1`) and Sales Order
+  (`customdeploy2`). So Invoice/CashSale/CreditMemo/CashRefund (the Forecast.Sales types) are **not yet
+  enqueued via webhook** — which is why a billing invoice fires no webhook (see the open-orders doc's
+  invoice-transform gap). Adding a real-time sale-import webhook requires releasing this same enqueuer on
+  those four types.
+- **`lib_amq_queue.js` is a LIBRARY module (no `@NScriptType`) → it has NO deployment of its own, and its
+  `N/log` output surfaces in the CALLING script's execution log.** Its `transmit()` wraps `https.request`
+  in a try, but the surrounding `record.load`/claim and `markFailure`'s `submitFields` are **outside** that
+  try — so a throw there escapes to `ss_amq_drain`'s per-row catch. A live drainer error
+  `{"queueId":40529,"error":"UNEXPECTED_ERROR: An unexpected SuiteScript error has occurred"}` (a TypeError,
+  observed 2026-06-25 1:10am) originates in this transmit path and shows up under **`ss_amq_drain`'s**
+  execution log (not a library log of its own). **Durable error detail is also persisted on the API Message
+  Queue custom record** fields `custrecordcustrecord_amq_errormsg` / `_resp_status` / `_resp_body`
+  (doubled-prefix ids — see the doubled-id gotcha), which are **SuiteQL-queryable** for after-the-fact
+  diagnosis even though the calling-script log rolls off.
 
 ## CU→NS direction (future — not yet built): reverse mapping
 
@@ -345,6 +364,15 @@ same **skip-if-unchanged** compare on the extracted values, and **actor-identity
 trigger a CU→NS write. The NS→CU change-detection above is the complementary backstop, not a substitute.
 
 ## Change history
+- 2026-06-25 — **Characterized the AMQ enqueuer scope + a live drainer TypeError.** The enqueuer is
+  generic over all record types via `RECORD_TYPE_MAP` (no context filter, prod `DEV_OVERRIDE.enabled=false`)
+  but probe-confirmed `Released` on **only Opportunity (`customdeploy1`) + Sales Order (`customdeploy2`)** —
+  so the four Forecast.Sales record types are not yet webhook-enqueued. Recorded that `lib_amq_queue.js` is a
+  **library module** (no `@NScriptType`/own deployment) whose `N/log` surfaces in the **calling** script's
+  (`ss_amq_drain`) log, and that an `UNEXPECTED_ERROR` (TypeError, queueId 40529, 2026-06-25 1:10am)
+  originates in its `transmit()` path where `record.load`/claim + `markFailure.submitFields` sit **outside**
+  the `https.request` try. Durable error detail also persists on the queue record's
+  `custrecordcustrecord_amq_errormsg`/`_resp_status`/`_resp_body` fields (SuiteQL-queryable). (dfranks)
 - 2026-06-24 — **Default presales-lead assignee fallback.** `buildCustomFields()` now wires the
   previously-unused `DEFAULT_PRESALES_LEAD_USER_ID = '87374309'` (Cory Martin, comartin@togatech.com —
   verified active via ClickUp `GET /team`) into the else-branch when `resolvePresalesUserId()` returns

package/knowledge/2.0/apps/worker2/features/netsuite-salesorder-open-orders-sync.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-06-23
+updated: 2026-06-25
 owners: ["dfranks"]
 files:
   - worker2/Worker/Netsuite/SalesOrder.php
@@ -133,6 +133,7 @@ None — uniform (platform-wide Forecast2 sync).
 
 ## Gotchas / known issues
 
+- **Billing an SO via invoice-transform flips it to "Billed" WITHOUT firing the SalesOrder afterSubmit UE → no edit/PUT webhook → `removeAll()` never runs → the OpenOrderItems rows linger (this is the real cause of the "billed order not removed" problem — NOT a removeAll-didn't-persist bug).** When a NetSuite Sales Order is billed by **creating an Invoice from it**, the SO's `status` transitions to `Billed` as a **side-effect of the invoice transform** — this does **not** trigger the SalesOrder `afterSubmit` User Event, so the AMQ enqueuer (`ue_api_msg_queue_enqueue`, deployed per-record-type) **never emits an edit/PUT webhook for the SO**. Consequently `_Worker_Netsuite_SalesOrder` is never invoked for the billing change, `importOpenOrder`'s status gate (which *would* call `removeAll` on a non-open status) never runs, and the order's `OpenOrderItems` rows survive until the nightly discrepancy-fix cron deletes them. **Confirmed live 2026-06-24/25 on SO 7190415** (tranId 280387): only a `create` webhook delivery exists in `Logs.Webhook` (2026-06-24 23:48, 16 rows inserted) — **NO** edit/delete delivery at billing time (~09:41 UTC 2026-06-25); `Core.WorkerJobs` shows only `Netsuite/Webhook` + `Netsuite/SalesOrder/post` (no put/delete); `Logs.Api` has only the original NS GET + a 200 `OPEN_ORDER_IMPORT` breadcrumb — **no 204 removeAll breadcrumb**. The invoice (7190621) that billed it **also produced no webhook** — its 16 `Forecast.Sales` rows came from the **legacy 5-min pull cron**, not the webhook. **This corrects the earlier suspicion that "removeAll fired but DB didn't persist"** — both `removeAll` paths *do* commit `DB_FORECAST`; `removeAll` was simply **never called**. **Diagnostic ladder for a lingering billed order:** `Logs.Webhook` (was an edit/delete even delivered? — here, no) → `Core.WorkerJobs` (was a put/delete processed? — no) → `Logs.Api` `source='OPEN_ORDER_IMPORT'` (any 204 removeAll breadcrumb? — no). **Fix direction (not yet implemented):** the SalesOrder-side enqueuer must also fire on the invoice-transform status change — either deploy the AMQ enqueuer on **Invoice** so the bill event drives an SO re-sync, or have the SalesOrder handler re-evaluate the SO when its child invoice arrives; until then the **stopgap is the continuous `loop_trueup_open_orders.php --by-lastmodified` runner** (see the reconciliation doc), which windows on `lastmodifieddate` so it catches an order billed today whose `tranDate` predates the window. The `removeAll` path in `SalesOrder.php` was re-confirmed correct in this session.
 - **Wrong FK column name → MySQL 1054 that masquerades as "order missing".** `OpenOrderItems`'s
   sales-order key is **`netsuiteSalesOrderInternalId`**. Do **NOT** query it with
   `netsuiteTransactionInternalId` — that is the **`Forecast.Sales`** column. On `OpenOrderItems` it
@@ -296,6 +297,16 @@ test fixture (it surfaced the stale SO 7181316 above).
 
 ## Change history
 
+- 2026-06-25 — **Root-caused "billed SO not removed from OpenOrderItems in real time" to a missing
+  webhook, NOT a removeAll-didn't-persist bug.** Billing an SO via invoice-transform flips it to
+  `Billed` as a side-effect that does **not** fire the SalesOrder `afterSubmit` UE → the AMQ enqueuer
+  emits no edit/PUT webhook → `_Worker_Netsuite_SalesOrder` never runs → `removeAll()` is never called
+  (both removeAll paths confirmed to commit `DB_FORECAST` correctly — the call simply never happens).
+  Confirmed live on SO 7190415: only a `create` delivery in `Logs.Webhook`, no put/delete in
+  `Core.WorkerJobs`, no 204 removeAll breadcrumb in `Logs.Api`; the billing invoice produced no webhook
+  either (its Sales rows came from the legacy pull cron). Recorded the `Logs.Webhook`→`WorkerJobs`→
+  `OPEN_ORDER_IMPORT`-breadcrumb diagnostic ladder and the stopgap (`loop_trueup_open_orders.php
+  --by-lastmodified`). (dfranks)
 - 2026-06-24 — **Outbound push (`CreateNetSuite`/`UpdateNetSuite`/`Sync`) converted SOAP →
   REST-only** via `_Component_Api_Netsuite`: item + customer lookups now SuiteQL
   (`resolveNetSuiteCustomerId()` throws on 0/>1 name matches), `buildNetSuiteOrder()` emits a REST

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.198",
+  "version": "1.0.199",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",