toga-ai 1.0.180 → 1.0.181

package/knowledge/1.0/apps/worker/INDEX.md

@@ -5,7 +5,7 @@
 | [Worker (1.0 Framework) Architecture](architecture.md) | `worker` is the legacy (**1.0** `App_` framework) **background-job tier**. | worker/index.php, worker/_/app/framework.php, worker/crons/, worker/schedules/, worker/ebs/cron.worker.php, worker/.ebextensions/035_cron.worker.config |
 | [Compass MA Sales Order Exception Report](features/compass-ma-sales-order-exception-report.md) | A worker cron that emails operations the "Compass Refresh Exception Report" — Compass `MA%` sales orders whose corresponding Office Depot (ODP) sales order has | worker/crons/toga2/compass/workflow/7_generate_ma_sales_order_exception_report.php |
 | [Compass Partial In-Transit & Delivered Emails (per package)](features/compass-partial-in-transit-delivered-emails.md) | Compass USA and Compass Canada send a **per-package** in-transit email (and a matching delivered email) instead of one email listing the whole order. | worker/crons/toga2/compass/update_salesorder_status_from_odp.php, worker/crons/toga2/compasscanada/workflow/3_update_salesorder_status_from_grand_and_toy.php |
-| [Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling](features/forecast2-netsuite-reconciliation.md) | CLI tools to **audit** and **repair** drift between the production `Forecast` DB (core2) and NetSuite. | test/@dave/reconcile_netsuite_totals.php, test/@dave/analyze_netsuite_forecast_diff.php, test/@dave/trueup_sales.php, test/@dave/trueup_open_orders.php, test/@dave/trueup_opportunities.php, test/@dave/probe_sales_gap_direct.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
+| [Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling](features/forecast2-netsuite-reconciliation.md) | CLI tools to **audit** and **repair** drift between the production `Forecast` DB (core2) and NetSuite. | test/@dave/reconcile_netsuite_totals.php, test/@dave/analyze_netsuite_forecast_diff.php, test/@dave/trueup_sales.php, test/@dave/trueup_open_orders.php, test/@dave/trueup_opportunities.php, test/@dave/probe_sales_gap_direct.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
 | [NetSuite → TOGa Supply Per-Client Sync (thin wrappers)](features/netsuite-togasupply-per-client-sync.md) | Syncs NetSuite transactions (sales orders, purchase orders, invoices, item receipts, item fulfillments, inventory adjustments) into each TOGa Supply (2.0) clien | worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_canon.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql |
 | [Prudential: Send Shipments for the Day report (daily cron)](features/send-shipments-for-the-day.md) | Daily cron (9:00 PM) that emails Prudential and Dell stakeholders an Excel report of all devices shipped that day, including tracking number, serial number, emp | worker/crons/notifications/reports/send_shipments_for_the_day.php |
 | [Onboarding a Client to the NetSuite TOGa Supply Sync](workflows/onboarding-client-to-netsuite-togasupply-sync.md) | How to add a new TOGa 2 client to the per-client NetSuite → TOGa Supply importer (`worker/crons/toga2/netsuite/`). | worker/crons/toga2/netsuite/sync_togasupply.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql |

package/knowledge/1.0/apps/worker/features/forecast2-netsuite-reconciliation.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-06-22
+updated: 2026-06-23
 owners: [dfranks]
 files:
   - test/@dave/reconcile_netsuite_totals.php
@@ -17,6 +17,7 @@ files:
   - test/@dave/probe_sales_gap_direct.php
   - test/@dave/probe_missing_oo_timing.php
   - test/@dave/probe_missing_oo_createdby.php
+  - test/@dave/probe_drift_so_dates.php
   - worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php
 related:
   - ../architecture.md
@@ -67,6 +68,13 @@ by reconciling a chosen tranDate range directly against NetSuite.
 - **Chunking + keepalive.** trueup_sales chunks by `--chunk-days` (default 7) and commits per
   chunk so a crash resumes. Both ping/reconnect `db_forecast2` before DB work (Aurora drops idle
   links during long NS calls).
+- **`probe_drift_so_dates.php` (read-only drift classifier).** For a list of NS SO internalIds it
+  reports, per order: NS `trandate` / `lastmodifieddate` (ET) / status / total; NS current open
+  product-line count + open $; and the matching `Forecast.OpenOrderItems` row count + $ (direct mysqli
+  to the core2 `Forecast` DB, column `netsuiteSalesOrderInternalId`). Bootstraps the 2.0
+  `_Component_Api_Netsuite` adapter via the worker2 chdir+require `index.php` pattern. Use it to
+  **classify a drift set as whole-missing vs. present-but-different**, and to scope a
+  `Logs.Webhook`/`WorkerJobs` search window from the lastmodified timestamps.
 
 ## Data model
 
@@ -139,6 +147,18 @@ None — Forecast2 is a single shared dataset.
   `probe_missing_oo_createdby.php` (creator/context — integration/Web-Services-origin orders cluster
   here) + a `Core.WorkerJobs` lookup by `JSON_EXTRACT(parameters,'$.internalId')` (no row → bucket A;
   rows all `isSuccess=1` yet zero OOI rows → bucket B).
+- **Open-orders drift is THREE distinct categories — diagnose per category, don't assume.**
+  Drift between NS open SOs and `Forecast.OpenOrderItems` maps to `trueup_open_orders.php`'s output
+  buckets:
+  - **(a) WHOLE-ORDER-MISSING** — NS has open lines, FC has **0 rows** → the **"Insert"** bucket.
+    **Dominant by dollars** (a single stale partially-fulfilled order can be **>$1M**).
+  - **(b) LINE-LEVEL value drift** — order present in FC but a line's revenue/qty changed → the
+    **"Update"** bucket. **Largest by count** (~1,612 in a mid-2026 run).
+  - **(c) STALE FC ROWS** — FC holds lines NS no longer returns as open → the **"Delete"/stale-cleanup**
+    bucket. **Smallest** (~13–19).
+  The common "drift = dropped line items still in production" assumption is category **(c) — the
+  SMALLEST**; the dominant/highest-$ problem is **whole-order-missing (a)**. Classify the set first
+  (`probe_drift_so_dates.php`) and diagnose per category.
 - **Legacy credit-memo wrong-sign rows.** Some `Forecast.Sales` credit-memo/cash-refund rows were
   stored with **flipped signs** by a prior sync version (positive instead of negative), inflating
   revenue. `trueup_sales.php` repairs them: it compares against NS by trandate, detects the sign
@@ -149,6 +169,13 @@ None — Forecast2 is a single shared dataset.
 
 ## Change history
 
+- 2026-06-23 — **Framed open-orders drift as three distinct categories** mapping to
+  `trueup_open_orders`'s Insert/Update/Delete buckets — whole-order-missing (dominant by $),
+  line-level value drift (largest by count), and stale FC rows (smallest) — correcting the common
+  "dropped line items" assumption (which is the smallest category). Added the read-only drift
+  classifier `probe_drift_so_dates.php` (NS dates/status/open-$ vs `Forecast.OpenOrderItems` count/$
+  via direct core2 mysqli; bootstraps the 2.0 `_Component_Api_Netsuite` adapter) to classify a drift
+  set and scope a Logs.Webhook/WorkerJobs search window. (dfranks)
 - 2026-06-22 — **`reconcile` Open Orders made apples-to-apples.** Headline row now compares
   product-only on both sides (shipping is NS-only, prints as a separate `(NS-only)` info row) — kills
   a phantom shipping-only "delta". Documented the per-order presence diff as the real sync-health

package/knowledge/2.0/apps/worker2/INDEX.md

@@ -10,7 +10,7 @@
 | [Elite Freshservice Sync (worker2)](features/elite-freshservice-sync.md) | `_Worker_Elite` processes Freshservice webhook events and syncs them into TOGA 2. | worker2/Worker/Elite.php, worker2/Config/dev-kmaramreddy-laptop.ini |
 | [Monitoring Framework (Orchestrator + Child Monitors)](features/monitoring-framework.md) | A unified, DB-driven monitoring framework for business-critical data flows (Compass POs, Prudential asset imports, AIG closed claims, …). | worker2/Worker/Monitor.php, worker2/Worker/Monitors/, worker2/Worker/Notification/Email.php, dbchanges2/Core/2026-05-21 - Monitors.sql |
 | [NetSuite → TOGA Opportunity Sync (API Message Queue + worker2 webhook)](features/netsuite-opportunity-sync.md) | Outbound sync from NetSuite to TOGA for the record types the Forecast2 importer pulls (opportunities first; sales/items/etc. | worker2/Worker/Netsuite.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Controller/Index.php, _underscore/Worker.php, test/@dave/NetSuite/api-message-queue/lib_amq_queue.js, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/NetSuite/api-message-queue/ue_amq_drain.js, test/@dave/NetSuite/api-message-queue/ss_amq_drain.js, test/@dave/NetSuite/api-message-queue/DEPLOY_RUNBOOK.md, test/@dave/clickup/backfill_opportunity_numbers.php, test/@dave/clickup/probe_opportunity_fields.php, test/@dave/probe_clickup_desc_match.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
-| [NetSuite → Forecast Open-Orders Sync (salesOrder webhook → OpenOrderItems)](features/netsuite-salesorder-open-orders-sync.md) | Webhook-driven, single-record port of the legacy open-orders importer (TRUE-79142). | worker2/Worker/Netsuite/SalesOrder.php, worker2/Worker/Netsuite.php, test/@dave/probe_salesorder_rest_shape.php, test/@dave/probe_open_order_lines.php, test/@dave/check_so_status.php, test/@dave/check_so_history.php, test/@dave/probe_so_rest_lines.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
+| [NetSuite → Forecast Open-Orders Sync (salesOrder webhook → OpenOrderItems)](features/netsuite-salesorder-open-orders-sync.md) | Webhook-driven, single-record port of the legacy open-orders importer (TRUE-79142). | worker2/Worker/Netsuite/SalesOrder.php, worker2/Worker/Netsuite.php, test/@dave/probe_salesorder_rest_shape.php, test/@dave/probe_open_order_lines.php, test/@dave/check_so_status.php, test/@dave/check_so_history.php, test/@dave/probe_so_rest_lines.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
 | [Startech Webhook Handler (worker2)](features/startech-webhook-handler.md) | Receives inbound webhook events from Startech (Easeedesk) and creates or updates the corresponding ticket in TOGA 2.0. | worker2/Worker/Startech.php |
 | [Teams Meeting Transcript Export](features/teams-transcript-export.md) | `_Worker_Team_Transcripts` (action `Team/Transcripts/Export`) polls Microsoft Graph for Teams meeting transcripts produced by a set of organizers, classifies ea | worker2/Worker/Team/Transcripts.php, worker2/Config/production.ini, worker2/Database/TeamsTranscriptExports.sql, dbchanges2/Core/2026-06-18a - Teams Transcript Export schedule.sql |
 | [WJE Freshservice Sync (worker2)](features/wje-freshservice-sync.md) | WJE ("WJE IT", helpdesk `wje.freshservice.com`) is a **Freshservice**-based help-desk client whose tickets, contacts, assets, groups, categories, and canned res | worker2/Worker/Wje.php, _underscore/Component/Api/Wje/Wje.php, _underscore/Model/Wje/Ticket.php, _underscore/Model/Wje/TicketNote.php, _underscore/Model/Wje/Contact.php, _underscore/Model/Wje/Unit.php, _underscore/Model/Wje/TicketTeam.php, _underscore/Model/Wje/TicketCategory.php, _underscore/Model/Wje/AssetType.php, _underscore/Model/Wje/PredefinedReply.php, library/app/api/wje.php, worker/crons/toga2/wje/import_supporting_records.php, worker/crons/toga2/wje/sync_togasupply_wje.php, worker/crons/notifications/reports/wje/wje_common.php, library/app/systemmonitor/wje.php, dbchanges2/Client_Wje/2024-10-04 - WjeOnboarding.sql |

package/knowledge/2.0/apps/worker2/features/netsuite-salesorder-open-orders-sync.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-06-22
+updated: 2026-06-23
 owners: ["dfranks"]
 files:
   - worker2/Worker/Netsuite/SalesOrder.php
@@ -18,6 +18,7 @@ files:
   - test/@dave/probe_so_rest_lines.php
   - test/@dave/probe_missing_oo_timing.php
   - test/@dave/probe_missing_oo_createdby.php
+  - test/@dave/probe_drift_so_dates.php
   - worker/crons/toga2/forecast2/import_open_orders.php
   - worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php
 related:
@@ -77,12 +78,35 @@ Unique key `(netsuiteSalesOrderInternalId, lineNumber)`. Columns written: `netsu
 `classificationId`, `itemId` (nullable), `lineNumber`, `revenue`, `profit`. The `accountId` FK column
 exists but is **never populated** (the legacy writer never set it — parity).
 
+The table lives in the **`Forecast` schema on the core2 cluster** (reader
+`reader1.core.database.togahub.com` / writer `writer.core.database.togahub.com`), ~4,347 rows. The
+sales-order FK column is **`netsuiteSalesOrderInternalId`** (`int unsigned`) — note this is *not* the
+same column name used by `Forecast.Sales` (see the column-name gotcha below).
+
 ## Client variations
 
 None — uniform (platform-wide Forecast2 sync).
 
 ## Gotchas / known issues
 
+- **Wrong FK column name → MySQL 1054 that masquerades as "order missing".** `OpenOrderItems`'s
+  sales-order key is **`netsuiteSalesOrderInternalId`**. Do **NOT** query it with
+  `netsuiteTransactionInternalId` — that is the **`Forecast.Sales`** column. On `OpenOrderItems` it
+  raises **MySQL ERROR 1054 "Unknown column"**, which (if the query error is swallowed) degrades into a
+  silent **false-negative "order missing"** reading. Verify the schema before concluding an order is
+  absent from Forecast.
+- **A successful `WorkerJobs` row does NOT prove rows were written — two silent zero-write paths.** An
+  `isSuccess=1` / `failureReason = NULL` job is **not** evidence that any `OpenOrderItems` rows persisted.
+  `importOpenOrder` has two success-with-zero-write paths:
+  - **(a) STATUS GATE** — if NetSuite `status->refName` is not in `OPEN_STATUSES` (`Pending Fulfillment`,
+    `Partially Fulfilled`, `Pending Billing/Partially Fulfilled`, `Pending Billing`) or is missing, it
+    calls `removeAll()` (deletes all FC rows) and returns a success string.
+  - **(b) ALL-LINES-FILTERED** — every line is skipped (item group / excluded EWR item id /
+    `revenue == 0 && profit == 0`), leaving `lineRows` empty, so `syncOpenLines` takes its delete-all
+    branch and returns `"synced (0 open lines)"`.
+  Only an **unknown item** throws (`isSuccess=0`). The **only** thing that detects silent zero-rows is a
+  **post-write verification** (count persisted vs. intended); a handler lacking that verification cannot
+  self-detect this state.
 - **Async-recalc race → a transiently-empty REST read PURGES a genuinely-open order (silent data loss).**
   The open-status gate is not the only way rows get deleted: on an **open** order, if the REST GET
   momentarily returns **zero importable open lines**, `syncOpenLines` runs its reconcile-delete with an
@@ -142,6 +166,17 @@ None — uniform (platform-wide Forecast2 sync).
   up after `MAX_IMPORT_ATTEMPTS`). If every `WorkerJobs` row for the id has a **NULL `attempt`** and
   there are **no `OPEN_ORDER_IMPORT` breadcrumbs** for it, the self-heal/defer path isn't executing
   for that order — so a bucket-B miss has no retry and freezes at zero rows.
+  **Full 3-table diagnostic ladder for a single SO (all read-only):**
+  1. **`Logs.Webhook`** (logs cluster `reader1.logs.database.togahub.com`, schema `Logs`) — confirms
+     NetSuite's AMQ push actually **delivered** the `salesOrder` event to `webhook.togahub.com/netsuite`.
+     Filter `route LIKE '%netsuite%'` and REGEXP the internalId in `requestPayload`.
+  2. **`Core.WorkerJobs`** (core2, schema `Core`) — the **processing outcome**.
+     `action LIKE 'Netsuite/SalesOrder/%'`, `JSON_EXTRACT(parameters,'$.internalId')`.
+  3. **`Logs.Api` with `source = 'OPEN_ORDER_IMPORT'`** — the **import breadcrumb**. `responseCode`:
+     **200** persisted (== intended) / **409** silent zero-rows / **202** deferred / **204** removeAll /
+     **500** gave up.
+  **SIGNAL:** zero `OPEN_ORDER_IMPORT` rows in `Logs.Api` despite a successful `WorkerJobs` row ⇒ the
+  running handler **predates the breadcrumb instrumentation**.
 - **REST shape ≠ SOAP shape.** The cron reads the SOAP-shim shape; this handler reads the REST record
   (`status->refName`, line `quantityBilled`, `class->refName`, `entity->id`, `salesRep->id`,
   `shippingCost`). Verified against live orders via `test/@dave/probe_salesorder_rest_shape.php`.
@@ -196,6 +231,15 @@ and are a candidate to extract into a shared `_Component_Forecast_Db` before the
 
 ## Change history
 
+- 2026-06-23 — **Documented the OpenOrderItems schema/FK column, two silent zero-write paths, and the
+  full 3-table diagnostic ladder.** Recorded that the table lives in `Forecast` on core2 (~4,347 rows)
+  with FK `netsuiteSalesOrderInternalId`, and the **1054 false-negative** trap from querying it with the
+  `Forecast.Sales` column `netsuiteTransactionInternalId`. Added that an `isSuccess=1` job proves nothing
+  about rows written — STATUS GATE and ALL-LINES-FILTERED both succeed with zero writes, and only a
+  post-write count detects it. Extended the WorkerJobs/breadcrumb note into the full read-only ladder
+  (`Logs.Webhook` delivery → `Core.WorkerJobs` outcome → `Logs.Api` `OPEN_ORDER_IMPORT` breadcrumb), with
+  the signal that zero breadcrumbs despite a successful job ⇒ handler predates the instrumentation. Added
+  `probe_drift_so_dates.php`. (dfranks)
 - 2026-06-22 — **Separated "missing open order" into two distinct buckets** with a discriminating
   diagnostic (`Core.WorkerJobs` lookup by `parameters.internalId`): **(A) webhook never enqueued** (no
   `WorkerJobs` row — NetSuite-side enqueuer not firing for integration/Web-Services/backend-mass-edit

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.180",
+  "version": "1.0.181",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",