toga-ai 1.0.112 → 1.0.114

package/knowledge/2.0/apps/worker2/INDEX.md

@@ -7,5 +7,6 @@
 | [Creating Worker Actions](features/creating-worker-actions.md) | How to add a new callable Worker action — a PHP class whose `public static` methods are invoked as background jobs (via webhook, cron, or `_Worker::runTask()`). | worker2/Worker/, worker2/Controller/Index.php, _underscore/Worker.php |
 | [Elite Freshservice Sync (worker2)](features/elite-freshservice-sync.md) | `_Worker_Elite` processes Freshservice webhook events and syncs them into TOGA 2. | worker2/Worker/Elite.php, worker2/Config/dev-kmaramreddy-laptop.ini |
 | [Monitoring Framework (Orchestrator + Child Monitors)](features/monitoring-framework.md) | A unified, DB-driven monitoring framework for business-critical data flows (Compass POs, Prudential asset imports, AIG closed claims, …). | worker2/Worker/Monitor.php, worker2/Worker/Monitors/, worker2/Worker/Notification/Email.php, dbchanges2/Core/2026-05-21 - Monitors.sql |
-| [NetSuite → TOGA Opportunity Sync (API Message Queue + worker2 webhook)](features/netsuite-opportunity-sync.md) | Outbound sync from NetSuite to TOGA for the record types the Forecast2 importer pulls (opportunities first; sales/items/etc. | worker2/Worker/Netsuite.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Controller/Index.php, _underscore/Worker.php, test/@dave/NetSuite/api-message-queue/lib_amq_queue.js, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/NetSuite/api-message-queue/ue_amq_drain.js, test/@dave/NetSuite/api-message-queue/ss_amq_drain.js, test/@dave/NetSuite/api-message-queue/DEPLOY_RUNBOOK.md, test/@dave/clickup/backfill_opportunity_numbers.php, test/@dave/clickup/probe_opportunity_fields.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
+| [NetSuite → TOGA Opportunity Sync (API Message Queue + worker2 webhook)](features/netsuite-opportunity-sync.md) | Outbound sync from NetSuite to TOGA for the record types the Forecast2 importer pulls (opportunities first; sales/items/etc. | worker2/Worker/Netsuite.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Controller/Index.php, _underscore/Worker.php, test/@dave/NetSuite/api-message-queue/lib_amq_queue.js, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/NetSuite/api-message-queue/ue_amq_drain.js, test/@dave/NetSuite/api-message-queue/ss_amq_drain.js, test/@dave/NetSuite/api-message-queue/DEPLOY_RUNBOOK.md, test/@dave/clickup/backfill_opportunity_numbers.php, test/@dave/clickup/probe_opportunity_fields.php, test/@dave/probe_clickup_desc_match.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
+| [NetSuite → Forecast Open-Orders Sync (salesOrder webhook → OpenOrderItems)](features/netsuite-salesorder-open-orders-sync.md) | Webhook-driven, single-record port of the legacy open-orders importer (TRUE-79142). | worker2/Worker/Netsuite/SalesOrder.php, worker2/Worker/Netsuite.php, test/@dave/probe_salesorder_rest_shape.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
 | [Teams Meeting Transcript Export](features/teams-transcript-export.md) | `_Worker_Team_Transcripts` (action `Team/Transcripts/Export`) polls Microsoft Graph for Teams meeting transcripts produced by a set of organizers, classifies ea | worker2/Worker/Team/Transcripts.php, worker2/Config/production.ini |

package/knowledge/2.0/apps/worker2/features/netsuite-opportunity-sync.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-06-16
+updated: 2026-06-17
 owners: ["dfranks"]
 files:
   - worker2/Worker/Netsuite.php
@@ -20,8 +20,10 @@ files:
   - test/@dave/NetSuite/api-message-queue/DEPLOY_RUNBOOK.md
   - test/@dave/clickup/backfill_opportunity_numbers.php
   - test/@dave/clickup/probe_opportunity_fields.php
+  - test/@dave/probe_clickup_desc_match.php
   - worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php
 related:
+  - ./netsuite-salesorder-open-orders-sync.md
   - ../architecture.md
 ---
 
@@ -88,8 +90,16 @@ second, independently-gated concern in the same handler.
 
 1. `findClickupTaskByOpportunityNumber($tranId)` — GET the list filtered by the `Opportunity #`
    custom field (`include_closed=true&include_archived=true&custom_fields=[{field_id,operator:'=',value}]`).
-2. **Match → `updateTask()`**: PUT name/description, then POST each custom field individually to
-   `/task/{id}/field/{fieldId}` (ClickUp has no bulk custom-field set on an existing task).
+2. **Match → `updateTask()`**: **diffs first, writes only what changed** (change-detection backstop,
+   since 2026-06-17). PUT name/description **only if** one differs; POST **only** the custom fields
+   that differ (each to `/task/{id}/field/{fieldId}` — ClickUp has no bulk custom-field set). When
+   nothing differs it writes nothing and returns `task unchanged`. This kills the no-op-write echo: a
+   blind rewrite fires a `taskUpdated` webhook on every sync. Comparison is **block-to-block** —
+   `buildDescription()` is regenerated from current NS data and compared to the stored block (no field
+   extraction needed), and `findClickupTaskByOpportunityNumber()` returns the **full task object** so
+   the diff has the existing name/description/custom-field values. Helpers: `clickupFieldMatches()`
+   (text = string compare; Presales Lead users field = add-only, matches when already assigned) and
+   `normalizeDescription()`.
 3. **No match → `createTask()`**: POST to list `901111987449`, `custom_item_id` = `1009`.
 
 Shared helpers: `buildCustomFields()` (the field array, used by both create and update),
@@ -117,7 +127,10 @@ Shared helpers: `buildCustomFields()` (the field array, used by both create and
   JS `Date.toString()` style in **America/Los_Angeles**, e.g. `GMT-0800 (PST)`, auto PST/PDT by date);
   Stage = `entityStatus.refName` (the percent string like "10%", NOT `probability`); Details = `memo`
   (which carries NetSuite's "Missing Required Details…" fallback verbatim for pre-mandatory records).
-  The blank separator lines are a single space. Rendered verbatim — no trailing-period normalization.
+  The blank separator lines are emitted as a single space, but **ClickUp stores them as truly empty
+  lines** (our `\n \n` comes back `\n\n`). Change-detection's `normalizeDescription()` rtrims each line
+  so that whitespace delta doesn't read as a change (otherwise every sync would re-write the
+  description). Confirmed against task 868jh6x08 / opp 73142 via `test/@dave/probe_clickup_desc_match.php`.
 - **Custom-field value mapping** (the non-obvious part — these were swapped before 2026-06-16):
   `Opportunity #` (`a5529cdc-…`) ← `tranId`; `Customer #` (`170dc118-…`) ← `entity->refName`
   (the customer **name**, deliberately — not the NetSuite customer number). Plus `Sales Rep`,
@@ -226,7 +239,37 @@ None — platform-wide Forecast sync.
   window to the load→claim gap. `findSendableIds` must list every sendable status
   (`Created`,`Pending`,`Retry`,`Sending`) — omitting `Created` silently matches nothing (`candidates:0`).
 
+## CU→NS direction (future — not yet built): reverse mapping
+
+The current sync is **NS→CU only**. When the ClickUp→NetSuite direction is built, it cannot reuse the
+block-to-block compare, because the source of the edit is the ClickUp **composite** but the destination
+is **discrete NetSuite fields**. It must **reverse-map** — extract the real NS values out of the wrapper
+before comparing-to / writing-to NetSuite:
+
+- **Title → NS `title`:** the task name is `{opp#} — {customer} — {title}`. Do **not** naively split on
+  ` — ` (a title can contain a dash). Strip the **known** `{Opportunity#} — {Customer#} — ` prefix using
+  the values already on the `Opportunity #` / `Customer #` custom fields as anchors; the remainder is
+  the NS title.
+- **Description → NS `memo`:** take only the lines between the `Details:` marker and the trailing
+  `NetSuite Internal ID:` line. Company / Amount / Expected Close / Stage are NS-derived display, not
+  ClickUp-authored — parse them out and ignore.
+- **Custom fields** are already discrete — no extraction.
+
+Pair this with: a **field-ownership gate** (only write fields ClickUp is authoritative for — Amount/
+Stage/Expected Close are NS-owned, and HubSpot also writes these records, so don't push them back), the
+same **skip-if-unchanged** compare on the extracted values, and **actor-identity suppression** (drop
+`taskUpdated` events authored solely by the ClickUp bot/integration user) so our own NS→CU writes don't
+trigger a CU→NS write. The NS→CU change-detection above is the complementary backstop, not a substitute.
+
 ## Change history
+- 2026-06-17 — ClickUp **change-detection backstop** added to `updateTask()`: diff before write, skip
+  no-op syncs (`task unchanged`), PUT name/description only when changed, POST only changed custom
+  fields. `findClickupTaskByOpportunityNumber()` now returns the full task object; added
+  `clickupFieldMatches()` (add-only users field) + `normalizeDescription()` (per-line rtrim — ClickUp
+  trims our single-space separator lines to empty, which previously read as a permanent change).
+  Verified opp 73142 (changed in NS → correctly flagged) and same-data (→ `unchanged`). Also recorded
+  the future **CU→NS reverse-mapping** design (extract title/memo, field-ownership gate, actor-identity).
+  (dfranks)
 - 2026-06-16 — ClickUp task description switched to a structured block (`buildDescription()` +
   `formatAmount()`/`formatExpectedClose()`): Company/Opportunity/Amount/Expected Close/Stage/Details/
   Internal ID. Field sources confirmed via `probe_opportunity_fields.php`; Expected Close rendered

package/knowledge/2.0/apps/worker2/features/netsuite-salesorder-open-orders-sync.md

@@ -0,0 +1,120 @@
+---
+title: NetSuite → Forecast Open-Orders Sync (salesOrder webhook → OpenOrderItems)
+framework: "2.0"
+repo: worker2
+project: Worker
+client: shared
+type: feature
+status: active
+updated: 2026-06-17
+owners: ["dfranks"]
+files:
+  - worker2/Worker/Netsuite/SalesOrder.php
+  - worker2/Worker/Netsuite.php
+  - test/@dave/probe_salesorder_rest_shape.php
+  - worker/crons/toga2/forecast2/import_open_orders.php
+  - worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php
+related:
+  - ./netsuite-opportunity-sync.md
+  - ../architecture.md
+---
+
+## Summary
+
+Webhook-driven, single-record port of the legacy open-orders importer (TRUE-79142). A NetSuite
+`salesOrder` create/edit/delete arrives at `webhook.togahub.com/netsuite`, the
+`_Worker_NetSuite::Webhook` router dispatches `Netsuite/SalesOrder/{POST,PUT,DELETE}`, and the
+handler syncs the order's currently-**open** line items into `Forecast.OpenOrderItems`. It is the
+real-time equivalent of the batch cron `worker/crons/toga2/forecast2/import_open_orders.php` (whose
+per-record logic lives in `common_import_sales_from_netsuite.php`, OPEN ORDERS section). Mirrors the
+[opportunity sync](./netsuite-opportunity-sync.md) — same shape, different table and gating.
+
+`OpenOrderItems` holds the **unbilled remainder** of approved-but-not-fully-billed sales orders;
+realized revenue (invoices/cash sales/credit memos/cash refunds) lives in `Forecast.Sales` and is a
+separate build (see *Related work*).
+
+## Key files / entry points
+
+- `Worker/Netsuite/SalesOrder.php` — `_Worker_Netsuite_SalesOrder`. **One class, two directions:**
+  the new INBOUND import (`POST`/`PUT`/`DELETE`, REST → Forecast) was *merged into* the pre-existing
+  OUTBOUND push (`Create`/`Update`/`Sync`, Toga → NetSuite via SOAP). The action router forces the
+  import entry points to live here (`salesOrder` → `_Worker_Netsuite_SalesOrder`), so they coexist.
+- `Worker/Netsuite.php` — the shared router (unchanged; no per-recordType edits needed).
+- NetSuite access is **REST only**, via the `_underscore` client `_Component_Api_Netsuite`
+  (`RECORD_SALES_ORDER` + `?expandSubResources=true`) — never the SOAP `NetSuiteService`.
+
+## How it works
+
+1. `POST`/`PUT` → `importOpenOrder($internalId)`; `DELETE` → `removeAll($internalId)`.
+2. `importOpenOrder` GETs the sales order via REST, then gates on **open status** (`status->refName`
+   ∈ `OPEN_STATUSES`: `Pending Fulfillment`, `Partially Fulfilled`, `Pending Billing/Partially
+   Fulfilled`, `Pending Billing`). A non-open **or missing** status → `removeAll` (the order carries
+   no open rows).
+3. Per open line: skip lines with no NS item id, item-group items (`netsuiteTransactionType ==
+   'itemGroup'`), and excluded EWR items (`DO_NOT_IMPORT_OPEN_ORDER_ITEM_IDS`). Throw if the NS item
+   isn't in `Forecast.Items` (surfaces an item-sync gap rather than dropping revenue).
+4. Open economics: `qtyOpen = quantity − quantityBilled`; `revenue = qtyOpen × rate`;
+   `profit = revenue − qtyOpen × unitCost` where `unitCost = costEstimate / quantity`
+   (divide-by-zero-safe). A line with `revenue == 0 && profit == 0` is dropped.
+5. Lookups → local ids: `Customers`/`Employees` by `netsuiteInternalId` (miss → null);
+   `Classifications` create-on-miss, name = last ` : `-segment of `class->refName` (e.g.
+   "True Solutions : SaaS Reseller : CSP" → "CSP").
+6. `syncOpenLines` upserts each open line keyed on `(netsuiteSalesOrderInternalId, lineNumber)`, then
+   deletes any row NetSuite no longer returns as open (orphans + now-closed lines). Commits
+   `DB_FORECAST` (lazy-transaction discipline).
+
+## Data model
+
+`Forecast.OpenOrderItems` — flat, denormalized **leaf** table (no header table, no FK children).
+Unique key `(netsuiteSalesOrderInternalId, lineNumber)`. Columns written: `netsuiteSalesOrderInternalId`,
+`dateOrder` (tranDate +12h), `orderNumber` (tranId), `customerId`, `salesRepEmployeeId`,
+`classificationId`, `itemId` (nullable), `lineNumber`, `revenue`, `profit`. The `accountId` FK column
+exists but is **never populated** (the legacy writer never set it — parity).
+
+## Client variations
+
+None — uniform (platform-wide Forecast2 sync).
+
+## Gotchas / known issues
+
+- **REST shape ≠ SOAP shape.** The cron reads the SOAP-shim shape; this handler reads the REST record
+  (`status->refName`, line `quantityBilled`, `class->refName`, `entity->id`, `salesRep->id`,
+  `shippingCost`). Verified against live orders via `test/@dave/probe_salesorder_rest_shape.php`.
+- **Three deliberate departures from the legacy cron (all intentional):**
+  1. **No date-window gate.** The cron flips `isOrderOpen=false` for tranDate outside −365d/+90d
+     (`common_import_…:1753`) to bound its windowed scan. Irrelevant to a single-id webhook — dropped.
+     (Consequence: a future-dated open SO the cron excludes *would* sync via webhook.)
+  2. **No shipping line.** The cron appends a synthetic `SHIPPING` line for `OpenOrderItems` with a
+     **null** item id, which its own item-id guard then drops — so shipping has **never** persisted to
+     `OpenOrderItems` (confirmed: 0 line-0 rows in prod). We omit it, preserving that behavior. (NB:
+     the cron's *Sales* section is different — it uses real item id **13500**, so shipping *does*
+     persist in `Forecast.Sales`.)
+  3. **Zero-open cascade bug fixed.** The cron's `$isOrderOpen` is order-scoped and never reset
+     per-line, so the first zero-open line (`common_import_…:1848`) poisons every later line on the
+     order — under-reporting open revenue on partially-billed multi-line orders, order-dependent. The
+     webhook skips only the zero line and keeps the order's other open lines (correct per-line
+     semantic). `trueup_open_orders.php` has been masking this in prod data.
+- **No `initialize()`.** It was removed: it only constructed the SOAP `NetSuiteService` (which *throws*
+  without `NS_HOST`/`NS_ENDPOINT` and builds a `SoapClient`), and the framework runs `initialize()`
+  before *every* action — keeping it would couple the REST import to SOAP config. The push methods
+  (`Create`/`Update`/`Sync`) self-construct `NetSuiteService`, so push is unaffected and the Forecast
+  DB is registered globally in `_.php`.
+
+## Related work
+
+`Forecast.Sales` (realized revenue) is fed by four other NS record types — `invoice`, `cashSale`,
+`creditMemo`, `cashRefund` — each routing to its own worker (`_Worker_Netsuite_Invoice`, etc.) over a
+planned shared `_Component_Forecast_SaleImport` engine (apply `$factor = −1` for credit memos / cash
+refunds). The low-level Forecast SQL/lookup helpers (`sqlLiteral`, `buildInsert`, `buildAssignments`,
+`lookupId`, `toSqlDate`, `fetchRecord`) are currently duplicated in `Opportunity.php` + `SalesOrder.php`
+and are a candidate to extract into a shared `_Component_Forecast_Db` before the Sales build.
+
+## Change history
+
+- 2026-06-17 — Initial open-orders importer merged into `_Worker_Netsuite_SalesOrder` (TRUE-79142):
+  REST-only, no date-window gate, no shipping line, cascade bug fixed. (dfranks)
+
+## Related docs
+
+- [NetSuite → TOGA Opportunity Sync](./netsuite-opportunity-sync.md) — the sibling pattern this mirrors.
+- [Worker (worker2) Architecture](../architecture.md)

package/knowledge/INDEX.md

@@ -15,7 +15,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 ## 2.0 framework
 
 - **_underscore** (_Underscore) _(framework core)_ — 7 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
-- **worker2** (Worker) — 7 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
+- **worker2** (Worker) — 8 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
 - **api2** (API) — 3 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
 - **dbchanges2** (Database Changes) _(framework core)_ — 1 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
 - **toga2-supply** (TOGa Supply) — 2 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.112",
+  "version": "1.0.114",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",