toga-ai 1.0.250 → 1.0.251

package/knowledge/2.0/apps/worker2/INDEX.md

@@ -13,7 +13,7 @@
 | [Etilize Catalog Item Import & Refresh](features/etilize-catalog-item-import.md) | Client-generic catalog onboarding from an S3 CSV plus an Etilize re-pull. | worker2/Worker/Etilize/Items.php |
 | [Etilize Item Translation Import](features/etilize-item-translation-import.md) | The abstract worker class `_Worker_Etilize_ItemTranslations` imports **non-English** item text from Etilize into the client's `ItemTranslations` table. | worker2/Worker/Etilize/ItemTranslations.php |
 | [Monitoring Framework (Orchestrator + Child Monitors)](features/monitoring-framework.md) | A unified, DB-driven monitoring framework for business-critical data flows (Compass POs, Prudential asset imports, AIG closed claims, …). | worker2/Worker/Monitor.php, worker2/Worker/Monitors/, worker2/Worker/Monitors/RateEntitlement.php, worker2/Worker/Notification/Email.php, worker2/Worker/Rate.php, dbchanges2/Core/2026-05-21 - Monitors.sql, dbchanges2/Core/2026-06-29a - Rate Entitlement Contract Monitor.sql |
-| [NetSuite → TOGA Opportunity Sync (API Message Queue + worker2 webhook)](features/netsuite-opportunity-sync.md) | Outbound sync from NetSuite to TOGA for the record types the Forecast2 importer pulls (opportunities first; sales/items/etc. | worker2/Worker/Netsuite.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Controller/Index.php, _underscore/Worker.php, test/@dave/NetSuite/api-message-queue/lib_amq_queue.js, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/NetSuite/api-message-queue/ue_amq_drain.js, test/@dave/NetSuite/api-message-queue/ss_amq_drain.js, test/@dave/NetSuite/api-message-queue/DEPLOY_RUNBOOK.md, test/@dave/clickup/backfill_opportunity_numbers.php, test/@dave/clickup/probe_opportunity_fields.php, test/@dave/probe_clickup_desc_match.php, test/@dave/test_model_load_behavior.php, dbchanges2/Forecast/2026-06-25a - Add unique index on Opportunities netsuiteOpportunityInternalId.sql, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
+| [NetSuite ↔ ClickUp / TOGA Opportunity Sync (API Message Queue + worker2 webhook)](features/netsuite-opportunity-sync.md) | Outbound sync from NetSuite to TOGA for the record types the Forecast2 importer pulls (opportunities first; sales/items/etc. | worker2/Worker/Netsuite.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Worker/Clickup.php, worker2/Worker/Clickup/Opportunity.php, worker2/Controller/Index.php, _underscore/Worker.php, test/@dave/NetSuite/api-message-queue/lib_amq_queue.js, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/NetSuite/api-message-queue/ue_amq_drain.js, test/@dave/NetSuite/api-message-queue/ss_amq_drain.js, test/@dave/NetSuite/api-message-queue/DEPLOY_RUNBOOK.md, test/@dave/clickup/backfill_opportunity_numbers.php, test/@dave/clickup/probe_opportunity_fields.php, test/@dave/probe_clickup_desc_match.php, test/@dave/test_model_load_behavior.php, dbchanges2/Forecast/2026-06-25a - Add unique index on Opportunities netsuiteOpportunityInternalId.sql, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
 | [NetSuite → Forecast Open-Orders Sync (salesOrder webhook → OpenOrderItems)](features/netsuite-salesorder-open-orders-sync.md) | Webhook-driven, single-record port of the legacy open-orders importer (TRUE-79142). | worker2/Worker/Netsuite/SalesOrder.php, worker2/Worker/Netsuite.php, test/@dave/probe_salesorder_rest_shape.php, test/@dave/probe_open_order_lines.php, test/@dave/check_so_status.php, test/@dave/check_so_history.php, test/@dave/probe_so_rest_lines.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_open_order_gating.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
 | [NetSuite Supporting-Record Webhook Importer (the reusable recipe)](features/netsuite-supporting-record-webhook-importer.md) | A single **repeatable recipe** for porting a legacy daily-pull NetSuite *supporting-record* importer (the lookup/dimension tables behind Forecast2 — Employees, | worker2/Worker/Netsuite/Employee.php, worker2/Worker/Netsuite/Account.php, worker2/Worker/Netsuite/Classification.php, worker2/Worker/Netsuite/Customer.php, worker2/Worker/Netsuite/Item.php, worker2/Worker/Netsuite.php, _underscore/Model/Forecast/Employee.php, _underscore/Model/Forecast/Account.php, _underscore/Model/Forecast/Classification.php, _underscore/Component/Forecast/Db/Db.php, test/@dave/test_employee_lifecycle.php, test/@dave/test_account_lifecycle.php, test/@dave/test_classification_lifecycle.php, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, worker/crons/toga2/forecast2/import_supporting_records.php |
 | [Background Email-Template Worker (_Worker_Notification_EmailTemplate)](features/notification-email-template.md) | `_Worker_Notification_EmailTemplate::Send(...)` dispatches a **stored, client-defined `EmailTemplates` row off-thread** as a background WorkerJob. | worker2/Worker/Notification/EmailTemplate.php, _underscore/Model/Client/EmailTemplate.php |

package/knowledge/2.0/apps/worker2/features/netsuite-opportunity-sync.md

@@ -1,16 +1,18 @@
 ---
-title: NetSuite → TOGA Opportunity Sync (API Message Queue + worker2 webhook)
+title: NetSuite ↔ ClickUp / TOGA Opportunity Sync (API Message Queue + worker2 webhook)
 framework: "2.0"
 repo: worker2
 project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-06-25
+updated: 2026-06-30
 owners: ["dfranks"]
 files:
   - worker2/Worker/Netsuite.php
   - worker2/Worker/Netsuite/Opportunity.php
+  - worker2/Worker/Clickup.php
+  - worker2/Worker/Clickup/Opportunity.php
   - worker2/Controller/Index.php
   - _underscore/Worker.php
   - test/@dave/NetSuite/api-message-queue/lib_amq_queue.js
@@ -38,7 +40,10 @@ Outbound sync from NetSuite to TOGA for the record types the Forecast2 importer
 record ("API Message Queue"), a scheduled SuiteScript drains it to
 `webhook.togahub.com/netsuite`, and worker2 routes it to a per-recordType handler that upserts
 into the `Forecast` DB. The legacy NetSuite→ClickUp opportunity-task creation is preserved as a
-second, independently-gated concern in the same handler.
+second, independently-gated concern in the same handler. As of 2026-06-30 (TRUE-79181) the ClickUp leg
+is **bidirectional for field + stage**: ClickUp task edits flow back to the linked NetSuite opportunity
+(`_Worker_Clickup_Opportunity`, update-only), and NS→CU now drives the ClickUp task **status** — see
+the "CU→NS direction" section.
 
 ## Key files / entry points
 
@@ -364,29 +369,106 @@ None — platform-wide Forecast sync.
   (doubled-prefix ids — see the doubled-id gotcha), which are **SuiteQL-queryable** for after-the-fact
   diagnosis even though the calling-script log rolls off.
 
-## CU→NS direction (future — not yet built): reverse mapping
+## CU→NS direction (BUILT 2026-06-30, TRUE-79181): reverse sync
 
-The current sync is **NS→CU only**. When the ClickUp→NetSuite direction is built, it cannot reuse the
-block-to-block compare, because the source of the edit is the ClickUp **composite** but the destination
-is **discrete NetSuite fields**. It must **reverse-map** — extract the real NS values out of the wrapper
-before comparing-to / writing-to NetSuite:
+The integration is now **bidirectional** for **field + stage**. ClickUp task edits flow back to the
+linked NetSuite opportunity via a new handler, and NS→CU now drives the ClickUp task **status** (not
+just description text).
 
-- **Title → NS `title`:** the task name is `{opp#} — {customer} — {title}`. Do **not** naively split on
-  ` — ` (a title can contain a dash). Strip the **known** `{Opportunity#} — {Customer#} — ` prefix using
-  the values already on the `Opportunity #` / `Customer #` custom fields as anchors; the remainder is
-  the NS title.
-- **Description → NS `memo`:** take only the lines between the `Details:` marker and the trailing
-  `NetSuite Internal ID:` line. Company / Amount / Expected Close / Stage are NS-derived display, not
-  ClickUp-authored — parse them out and ignore.
-- **Custom fields** are already discrete — no extraction.
+**Handler:** `_Worker_Clickup_Opportunity` (`worker2/Worker/Clickup/Opportunity.php`) — an abstract
+static-class worker, dispatched by `_Worker_Clickup::Webhook` via
+`_Worker::runTask('Clickup/Opportunity/Process', {taskId,event,data})`. Routing is gated by the ClickUp
+**LIST id** (Presales Qualification list `901111987449`) for `taskUpdated`/`taskStatusUpdated` events;
+non-opportunity lists fall through to the legacy sprint handling in `Clickup.php`.
 
-Pair this with: a **field-ownership gate** (only write fields ClickUp is authoritative for — Amount/
-Stage/Expected Close are NS-owned, and HubSpot also writes these records, so don't push them back), the
-same **skip-if-unchanged** compare on the extracted values, and **actor-identity suppression** (drop
-`taskUpdated` events authored solely by the ClickUp bot/integration user) so our own NS→CU writes don't
-trigger a CU→NS write. The NS→CU change-detection above is the complementary backstop, not a substitute.
+**CU→NS is UPDATE-ONLY** (never create/delete):
+1. Resolve the linked NS opportunity from the task's `Opportunity #` custom field
+   (`a5529cdc-8dbb-4165-8bc5-9f5cc691796a`, = NS `tranId`) via a one-row SuiteQL
+   `SELECT id FROM transaction WHERE type='Opprtnty' AND tranid='<n>'`.
+2. GET the opportunity with `expandSubResources=true` — this also supplies the **current** NS values
+   for the diff.
+3. **No match → skip.**
+4. PATCH **only** the NS-owned fields that actually differ (currently `title`, `stage`). A no-op write
+   fires no NS event, so the write cannot re-trigger NS→CU. **Echo prevention is the difference-check
+   alone** — there is no actor-identity / "last-synced-from" suppression field (that hardening was
+   explicitly deferred; see Decisions).
+
+**Shared `STAGE_MAP` — single source of truth, declared on `_Worker_Netsuite_Opportunity`.** Maps NS
+`entityStatus` internalId → ClickUp status on the Presales Qualification list (VERIFIED live 2026-06-30):
+
+| NS internalId | NS stage | ClickUp status |
+|---|---|---|
+| 124 | Closed - Won | `opportunity won` |
+| 14 | Closed Lost | `closed lost` |
+| 125/126/122/123/128 | 10% / 25% / 50% / 75% / 95% | `in progress` |
+| 130 | Alternate Quote | *(unmapped)* |
+
+**CRITICAL non-1:1 gotcha:** the five probability stages all collapse to `in progress`, so the reverse
+map `CU_STATUS_TO_NS_STAGE` only covers the **unambiguous terminal** stages (won/lost). CU→NS leaves a
+non-terminal status alone rather than guessing a percentage. The percentage-reverse mapping is an
+**unresolved stakeholder (Aaron) decision** — do not invent one.
+
+**NS→CU now drives the ClickUp task STATUS** from the NS stage via `STAGE_MAP` (previously the stage was
+only rendered in the description text). Applied on both create and update; the update PUT carries
+**only** the changed fields (`name`/`description`/`status`) to stay echo-safe.
+
+**Outbound NS calls** use the `Logs.Api` two-save pattern (`_Model_Core_Logs_Api`, `DB_LOGS`,
+`DIRECTION_OUT`: save the request row, then save the response/failure row), with
+`\Sentry\captureException($e); throw $e;` on failure so `WorkerJobs` records `isSuccess=0`. Logged
+payloads are length-capped. NS access is the **2.0 adapter `_Component_Api_Netsuite`** (GET/PATCH +
+SuiteQL POST with `Prefer: transient`); the **1.0 `App_Api_Netsuite_Rest` / SOAP toolkit are
+deprecated** for production opportunity code.
+
+### CU→NS gotchas
+
+- **SuiteQL over `_Component_Api_Netsuite` has NO placeholder binding.** When a SuiteQL filter value
+  comes from an **external** source (here a ClickUp custom-field `tranId`, not a local PK), sanitize
+  with an **allowlist-and-REJECT-on-mismatch**, not strip-and-use:
+  `$safe = preg_replace('/[^A-Za-z0-9_-]/','',$v); if ($safe!==$v) return null;`. Reject-on-mismatch is
+  the firewall; silently stripping characters is not enough.
+- **Validate `task_id` format at the webhook entry.** ClickUp native task ids are alphanumeric. The
+  legacy `_Worker_Clickup::Webhook` interpolates the raw `$payload->task_id` into Team-DB SQL at ~20
+  sites (a **pre-existing** SQL-injection exposure, NOT introduced here). A single input-boundary
+  format guard at the top of `Webhook()` closes that class for the raw payload id without rewriting
+  each query. **FLAG:** the broader legacy `$taskId`/`$customTaskId` raw-SQL interpolation throughout
+  `Clickup.php` is real pre-existing debt that warrants a dedicated security ticket.
+- **ClickUp opportunity field ids:** list `901111987449` ("Presales Qualification");
+  `Opportunity #` = `a5529cdc-8dbb-4165-8bc5-9f5cc691796a`,
+  `Customer #` = `170dc118-b3fa-413b-826b-2753e9405851`.
+
+### Deferred (documented follow-up)
+
+- **Notes & attachments (CU→NS)** were intentionally deferred — they require NetSuite custom fields
+  (`custentity_cu_synced_notes` / `custentity_cu_synced_files`) + a File Cabinet folder that **do not
+  exist yet**, plus unconfirmed NS REST note/file shapes. Field + stage sync is bidirectional;
+  notes/attachments remain a documented follow-up.
+- **Actor-identity echo hardening** deferred — the difference-check is the current (and sufficient for
+  field/stage) echo-prevention mechanism. A "drop events authored by our own integration user" gate is
+  the planned hardening if/when no-op-safe writes can't fully cover a future field.
+- **Percentage reverse-mapping** (which NS percentage stage a CU `in progress` should map back to) is
+  blocked on the Aaron stakeholder decision noted above.
 
 ## Change history
+- 2026-06-30 — **Built the CU→NS reverse sync + NS→CU stage drive (TRUE-79181) — integration now
+  bidirectional for field + stage.** New `_Worker_Clickup_Opportunity` handler
+  (`worker2/Worker/Clickup/Opportunity.php`), dispatched by `_Worker_Clickup::Webhook` via
+  `_Worker::runTask('Clickup/Opportunity/Process', …)`, gated by ClickUp **list id `901111987449`** for
+  `taskUpdated`/`taskStatusUpdated`. CU→NS is **update-only**: resolve the linked opp by the task's
+  `Opportunity #` (= `tranId`) via one-row SuiteQL, GET it (`expandSubResources=true`) for the current
+  values, PATCH only the differing NS-owned fields (`title`, `stage`). **Echo prevention = difference-check
+  only** (actor-identity hardening explicitly deferred). Added a shared `STAGE_MAP` on
+  `_Worker_Netsuite_Opportunity` (NS entityStatus internalId → ClickUp status; VERIFIED live: 124→won,
+  14→lost, 125/126/122/123/128→`in progress`, 130 unmapped) — its **five-probability-stages-collapse-to-
+  one** non-1:1 means the reverse map covers only the terminal stages; the percentage reverse-mapping is
+  an open Aaron decision. NS→CU now also drives the ClickUp task **status** from the stage (was
+  description-text only), PUT carrying only changed fields to stay echo-safe. Outbound NS calls use the
+  `Logs.Api` two-save OUT pattern + `Sentry::captureException;throw` on failure; NS access via the 2.0
+  `_Component_Api_Netsuite` adapter (1.0 `App_Api_Netsuite_Rest`/SOAP deprecated for prod). Gotchas
+  recorded: **SuiteQL has no placeholder binding** → external filter values need allowlist-**reject-on-
+  mismatch** sanitize; **validate `task_id` format at the webhook boundary** (the legacy `Clickup.php`
+  interpolates the raw payload id into Team-DB SQL at ~20 sites — pre-existing injection debt flagged for
+  a dedicated security ticket). Notes/attachments CU→NS deferred (need NetSuite custom fields +
+  File-Cabinet folder that don't exist yet). (dfranks)
 - 2026-06-25 — **Fixed the Opportunity duplicate-row amplification + added a unique-index guard.**
   Root cause: `_Model::load()` returns TRUE only on an **exactly-one** match (FALSE for 0 *and* 2+),
   so once an nsId had 2 rows the `load()`-then-upsert handler could never re-find it and every

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.250",
+  "version": "1.0.251",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",