toga-ai 1.0.153 → 1.0.154

package/knowledge/1.0/apps/worker/INDEX.md

@@ -5,7 +5,7 @@
 | [Worker (1.0 Framework) Architecture](architecture.md) | `worker` is the legacy (**1.0** `App_` framework) **background-job tier**. | worker/index.php, worker/_/app/framework.php, worker/crons/, worker/schedules/, worker/ebs/cron.worker.php, worker/.ebextensions/035_cron.worker.config |
 | [Compass MA Sales Order Exception Report](features/compass-ma-sales-order-exception-report.md) | A worker cron that emails operations the "Compass Refresh Exception Report" — Compass `MA%` sales orders whose corresponding Office Depot (ODP) sales order has | worker/crons/toga2/compass/workflow/7_generate_ma_sales_order_exception_report.php |
 | [Compass Partial In-Transit & Delivered Emails (per package)](features/compass-partial-in-transit-delivered-emails.md) | Compass USA and Compass Canada send a **per-package** in-transit email (and a matching delivered email) instead of one email listing the whole order. | worker/crons/toga2/compass/update_salesorder_status_from_odp.php, worker/crons/toga2/compasscanada/workflow/3_update_salesorder_status_from_grand_and_toy.php |
-| [Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling](features/forecast2-netsuite-reconciliation.md) | CLI tools to **audit** and **repair** drift between the production `Forecast` DB (core2) and NetSuite. | test/@dave/reconcile_netsuite_totals.php, test/@dave/analyze_netsuite_forecast_diff.php, test/@dave/trueup_sales.php, test/@dave/trueup_open_orders.php, test/@dave/trueup_opportunities.php, test/@dave/probe_sales_gap_direct.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
+| [Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling](features/forecast2-netsuite-reconciliation.md) | CLI tools to **audit** and **repair** drift between the production `Forecast` DB (core2) and NetSuite. | test/@dave/reconcile_netsuite_totals.php, test/@dave/analyze_netsuite_forecast_diff.php, test/@dave/trueup_sales.php, test/@dave/trueup_open_orders.php, test/@dave/trueup_opportunities.php, test/@dave/probe_sales_gap_direct.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
 | [NetSuite → TOGa Supply Per-Client Sync (thin wrappers)](features/netsuite-togasupply-per-client-sync.md) | Syncs NetSuite transactions (sales orders, purchase orders, invoices, item receipts, item fulfillments, inventory adjustments) into each TOGa Supply (2.0) clien | worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_canon.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql |
 | [Prudential: Send Shipments for the Day report (daily cron)](features/send-shipments-for-the-day.md) | Daily cron (9:00 PM) that emails Prudential and Dell stakeholders an Excel report of all devices shipped that day, including tracking number, serial number, emp | worker/crons/notifications/reports/send_shipments_for_the_day.php |
 | [Onboarding a Client to the NetSuite TOGa Supply Sync](workflows/onboarding-client-to-netsuite-togasupply-sync.md) | How to add a new TOGa 2 client to the per-client NetSuite → TOGa Supply importer (`worker/crons/toga2/netsuite/`). | worker/crons/toga2/netsuite/sync_togasupply.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql |

package/knowledge/1.0/apps/worker/features/forecast2-netsuite-reconciliation.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-06-11
+updated: 2026-06-22
 owners: [dfranks]
 files:
   - test/@dave/reconcile_netsuite_totals.php
@@ -15,6 +15,8 @@ files:
   - test/@dave/trueup_open_orders.php
   - test/@dave/trueup_opportunities.php
   - test/@dave/probe_sales_gap_direct.php
+  - test/@dave/probe_missing_oo_timing.php
+  - test/@dave/probe_missing_oo_createdby.php
   - worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php
 related:
   - ../architecture.md
@@ -116,6 +118,27 @@ None — Forecast2 is a single shared dataset.
   - **Pre-2024 legacy open SOs** — ordered before the sync window.
   When a user asks about a date range (e.g. 2026 YTD), **filter the open-orders comparison to that
   trandate window manually** — within the normal window (2024→present) the delta is rounding-only.
+- **`reconcile`'s Open Orders headline is product-only; shipping is NS-only and NOT a delta.**
+  Forecast2 never stores the synthetic shipping line (the `OpenOrderItems` writer drops it — see the
+  open-orders sync doc), so its shipping column is structurally `$0`. Summing NS product+shipping in
+  the headline therefore manufactures a phantom delta (e.g. a clean book reported a fake `$7,547`
+  "gap" that was 100% the NS shipping line). Fixed 2026-06-22: the **Open Orders** row compares
+  **product-only on both sides** (reconciles to ~$0; the residual is float rounding across orders),
+  and shipping prints as a separate `+ shipping` info row marked `(NS-only)` via `reportInfoRow()`.
+  Compare the **product** figure for sync health.
+- **The per-order presence diff is the real sync-health signal — and it splits misses into two
+  buckets.** `reconcile` calls `openOrderPerOrderDiff()` after the totals: for the window it lists
+  open SOs *missing from Forecast2* (open in NS, zero rows synced), *missing from NetSuite* (FC rows
+  whose SO is no longer open — benign, `trueup_open_orders` cleanup deletes them), and product-revenue
+  mismatches. A non-zero "missing from Forecast2" count is a **real** gap (distinct from the benign
+  shipping/rounding deltas above). Root-causing those misses (2026-06-22) showed **two independent
+  causes** needing different fixes (detailed in the open-orders sync doc): (A) orders the AMQ enqueuer
+  never fired for — **no `Core.WorkerJobs` row at all** for the SO id; and (B) orders that processed
+  `isSuccess=1` but wrote zero rows via the read-lag→destructive-delete. Diagnose by bucket:
+  `probe_missing_oo_timing.php` (NS status/timestamps confirm the orders are genuinely open) +
+  `probe_missing_oo_createdby.php` (creator/context — integration/Web-Services-origin orders cluster
+  here) + a `Core.WorkerJobs` lookup by `JSON_EXTRACT(parameters,'$.internalId')` (no row → bucket A;
+  rows all `isSuccess=1` yet zero OOI rows → bucket B).
 - **Legacy credit-memo wrong-sign rows.** Some `Forecast.Sales` credit-memo/cash-refund rows were
   stored with **flipped signs** by a prior sync version (positive instead of negative), inflating
   revenue. `trueup_sales.php` repairs them: it compares against NS by trandate, detects the sign
@@ -126,6 +149,12 @@ None — Forecast2 is a single shared dataset.
 
 ## Change history
 
+- 2026-06-22 — **`reconcile` Open Orders made apples-to-apples.** Headline row now compares
+  product-only on both sides (shipping is NS-only, prints as a separate `(NS-only)` info row) — kills
+  a phantom shipping-only "delta". Documented the per-order presence diff as the real sync-health
+  signal and the **two missing-from-Forecast2 buckets** (enqueuer-never-fired vs read-lag
+  destructive-delete) with a bucket-based diagnostic recipe; added `probe_missing_oo_timing.php` +
+  `probe_missing_oo_createdby.php`. (dfranks)
 - 2026-06-11 — Documented the `App_Database` sandbox-dev read pitfall (bypass with direct core2
   mysqli), `reconcile`'s open-orders live-snapshot semantics (future-dated + pre-2024 legacy SOs
   explain apparent gaps), the legacy credit-memo wrong-sign repair, and added `trueup_opportunities`

package/knowledge/2.0/apps/worker2/INDEX.md

@@ -8,5 +8,5 @@
 | [Elite Freshservice Sync (worker2)](features/elite-freshservice-sync.md) | `_Worker_Elite` processes Freshservice webhook events and syncs them into TOGA 2. | worker2/Worker/Elite.php, worker2/Config/dev-kmaramreddy-laptop.ini |
 | [Monitoring Framework (Orchestrator + Child Monitors)](features/monitoring-framework.md) | A unified, DB-driven monitoring framework for business-critical data flows (Compass POs, Prudential asset imports, AIG closed claims, …). | worker2/Worker/Monitor.php, worker2/Worker/Monitors/, worker2/Worker/Notification/Email.php, dbchanges2/Core/2026-05-21 - Monitors.sql |
 | [NetSuite → TOGA Opportunity Sync (API Message Queue + worker2 webhook)](features/netsuite-opportunity-sync.md) | Outbound sync from NetSuite to TOGA for the record types the Forecast2 importer pulls (opportunities first; sales/items/etc. | worker2/Worker/Netsuite.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Controller/Index.php, _underscore/Worker.php, test/@dave/NetSuite/api-message-queue/lib_amq_queue.js, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/NetSuite/api-message-queue/ue_amq_drain.js, test/@dave/NetSuite/api-message-queue/ss_amq_drain.js, test/@dave/NetSuite/api-message-queue/DEPLOY_RUNBOOK.md, test/@dave/clickup/backfill_opportunity_numbers.php, test/@dave/clickup/probe_opportunity_fields.php, test/@dave/probe_clickup_desc_match.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
-| [NetSuite → Forecast Open-Orders Sync (salesOrder webhook → OpenOrderItems)](features/netsuite-salesorder-open-orders-sync.md) | Webhook-driven, single-record port of the legacy open-orders importer (TRUE-79142). | worker2/Worker/Netsuite/SalesOrder.php, worker2/Worker/Netsuite.php, test/@dave/probe_salesorder_rest_shape.php, test/@dave/probe_open_order_lines.php, test/@dave/check_so_status.php, test/@dave/check_so_history.php, test/@dave/probe_so_rest_lines.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
+| [NetSuite → Forecast Open-Orders Sync (salesOrder webhook → OpenOrderItems)](features/netsuite-salesorder-open-orders-sync.md) | Webhook-driven, single-record port of the legacy open-orders importer (TRUE-79142). | worker2/Worker/Netsuite/SalesOrder.php, worker2/Worker/Netsuite.php, test/@dave/probe_salesorder_rest_shape.php, test/@dave/probe_open_order_lines.php, test/@dave/check_so_status.php, test/@dave/check_so_history.php, test/@dave/probe_so_rest_lines.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
 | [Teams Meeting Transcript Export](features/teams-transcript-export.md) | `_Worker_Team_Transcripts` (action `Team/Transcripts/Export`) polls Microsoft Graph for Teams meeting transcripts produced by a set of organizers, classifies ea | worker2/Worker/Team/Transcripts.php, worker2/Config/production.ini, worker2/Database/TeamsTranscriptExports.sql, dbchanges2/Core/2026-06-18a - Teams Transcript Export schedule.sql |

package/knowledge/2.0/apps/worker2/features/netsuite-salesorder-open-orders-sync.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-06-18
+updated: 2026-06-22
 owners: ["dfranks"]
 files:
   - worker2/Worker/Netsuite/SalesOrder.php
@@ -16,6 +16,8 @@ files:
   - test/@dave/check_so_status.php
   - test/@dave/check_so_history.php
   - test/@dave/probe_so_rest_lines.php
+  - test/@dave/probe_missing_oo_timing.php
+  - test/@dave/probe_missing_oo_createdby.php
   - worker/crons/toga2/forecast2/import_open_orders.php
   - worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php
 related:
@@ -120,6 +122,26 @@ None — uniform (platform-wide Forecast2 sync).
   `project_amq_realtime_requirement` in dev memory). Diagnose with `probe_open_order_lines.php` (SuiteQL vs
   Forecast) + `probe_so_rest_lines.php` (live REST sublist count) + the `WorkerJobs` `dtStarted`/`instanceId`
   timing (to confirm serial-vs-concurrent before blaming ordering).
+- **A missing open order has TWO distinct causes — diagnose which before "fixing" it.** When
+  `reconcile_netsuite_totals.php`'s per-order diff reports an open SO with **zero `OpenOrderItems`
+  rows**, it's one of two unrelated failures, told apart by `Core.WorkerJobs` (lookup by
+  `JSON_EXTRACT(parameters,'$.internalId')`):
+  - **Bucket A — the webhook never fired (no `WorkerJobs` row at all).** Nothing enqueued → the
+    handler never ran. The enqueuer (NetSuite-side AMQ User Event) doesn't fire for every context —
+    **integration/Web-Services-origin orders and backend/scheduled mass-edits cluster here** (a UE
+    only fires in the execution contexts its deployment enables). Confirm the order is genuinely open
+    + see its creator/context with `probe_missing_oo_timing.php` + `probe_missing_oo_createdby.php`.
+    This is **NetSuite-side** (enqueuer deployment/context), *not* a worker2 handler bug — the
+    read-lag fixes below do nothing for it. Backstop: the scheduled drainer / `trueup_open_orders`.
+  - **Bucket B — the handler ran and returned `isSuccess=1` but wrote zero rows** (the read-lag →
+    destructive-delete below). All items present in `Forecast.Items` (no throw), order genuinely open.
+  **Diagnostic for whether the self-heal path is even active:** `deferReimport` re-enqueues with an
+  incremented `attempt` parameter, and `logImportBreadcrumb` writes a per-order row to **`Logs.Api`
+  `source = 'OPEN_ORDER_IMPORT'`** whose `responseCode` encodes the outcome (**200** synced / **202**
+  deferred-reenqueued / **204** removeAll / **409** wrote-but-short = silent zero-rows / **500** gave
+  up after `MAX_IMPORT_ATTEMPTS`). If every `WorkerJobs` row for the id has a **NULL `attempt`** and
+  there are **no `OPEN_ORDER_IMPORT` breadcrumbs** for it, the self-heal/defer path isn't executing
+  for that order — so a bucket-B miss has no retry and freezes at zero rows.
 - **REST shape ≠ SOAP shape.** The cron reads the SOAP-shim shape; this handler reads the REST record
   (`status->refName`, line `quantityBilled`, `class->refName`, `entity->id`, `salesRep->id`,
   `shippingCost`). Verified against live orders via `test/@dave/probe_salesorder_rest_shape.php`.
@@ -174,6 +196,15 @@ and are a candidate to extract into a shared `_Component_Forecast_Db` before the
 
 ## Change history
 
+- 2026-06-22 — **Separated "missing open order" into two distinct buckets** with a discriminating
+  diagnostic (`Core.WorkerJobs` lookup by `parameters.internalId`): **(A) webhook never enqueued** (no
+  `WorkerJobs` row — NetSuite-side enqueuer not firing for integration/Web-Services/backend-mass-edit
+  contexts; not a handler bug) vs **(B) processed `isSuccess=1`, zero rows** (the read-lag destructive
+  delete). Recorded the self-heal signal-reading (NULL `attempt` on every job + no `OPEN_ORDER_IMPORT`
+  Logs.Api breadcrumb ⇒ self-heal/defer path not active for that order; breadcrumb `responseCode`
+  legend 200/202/204/409/500). Reconfirmed every sampled miss carries a NetSuite **item group**, and
+  all group/member items resolved in `Forecast.Items` (so isSuccess=1 = no unknown-item throw). Added
+  `probe_missing_oo_timing.php` + `probe_missing_oo_createdby.php`. (dfranks)
 - 2026-06-18 — **Root-caused the open-order delta to REST `item.items` read-after-write lag** (not the
   drainer/enqueue fixes, not ordering). SO 7170669 committed 4 of 15 open lines despite 14 *strictly
   sequential, single-instance, in-order* PUTs — the last authoritative read returned a lagged partial

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.153",
+  "version": "1.0.154",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",