toga-ai 1.0.340 → 1.0.341

package/knowledge/2.0/apps/worker2/INDEX.md

@@ -16,7 +16,7 @@
 | [Etilize Catalog Item Import & Refresh](features/etilize-catalog-item-import.md) | Client-generic catalog onboarding from an S3 CSV plus an Etilize re-pull. | worker2/Worker/Etilize/Items.php |
 | [Etilize Item Translation Import](features/etilize-item-translation-import.md) | The abstract worker class `_Worker_Etilize_ItemTranslations` imports **non-English** item text from Etilize into the client's `ItemTranslations` table. | worker2/Worker/Etilize/ItemTranslations.php |
 | [Monitoring Framework (Orchestrator + Child Monitors)](features/monitoring-framework.md) | A unified, DB-driven monitoring framework for business-critical data flows (Compass POs, Prudential asset imports, AIG closed claims, …). | worker2/Worker/Monitor.php, worker2/Worker/Monitors/, worker2/Worker/Monitors/RateEntitlement.php, worker2/Worker/Notification/Email.php, worker2/Worker/Rate.php, dbchanges2/Core/2026-05-21 - Monitors.sql, dbchanges2/Core/2026-06-29a - Rate Entitlement Contract Monitor.sql |
-| [NetSuite ↔ ClickUp / TOGA Opportunity Sync (API Message Queue + worker2 webhook)](features/netsuite-opportunity-sync.md) | Outbound sync from NetSuite to TOGA for the record types the Forecast2 importer pulls (opportunities first; sales/items/etc. | worker2/Worker/Netsuite.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Worker/Clickup.php, worker2/Worker/Clickup/Opportunity.php, worker2/Controller/Index.php, _underscore/Worker.php, test/@dave/NetSuite/api-message-queue/lib_amq_queue.js, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/NetSuite/api-message-queue/ue_amq_drain.js, test/@dave/NetSuite/api-message-queue/ss_amq_drain.js, test/@dave/NetSuite/api-message-queue/DEPLOY_RUNBOOK.md, test/@dave/clickup/backfill_opportunity_numbers.php, test/@dave/clickup/probe_opportunity_fields.php, test/@dave/probe_clickup_desc_match.php, test/@dave/test_model_load_behavior.php, dbchanges2/Forecast/2026-06-25a - Add unique index on Opportunities netsuiteOpportunityInternalId.sql, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
+| [NetSuite ↔ ClickUp / TOGA Opportunity Sync (API Message Queue + worker2 webhook)](features/netsuite-opportunity-sync.md) | Outbound sync from NetSuite to TOGA for the record types the Forecast2 importer pulls (opportunities first; sales/items/etc. | worker2/Worker/Netsuite.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Worker/Clickup.php, worker2/Worker/Clickup/Opportunity.php, worker2/Controller/Index.php, _underscore/Worker.php, test/@dave/NetSuite/api-message-queue/lib_amq_queue.js, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/NetSuite/api-message-queue/ue_amq_drain.js, test/@dave/NetSuite/api-message-queue/ss_amq_drain.js, test/@dave/NetSuite/api-message-queue/DEPLOY_RUNBOOK.md, test/@dave/clickup/backfill_opportunity_numbers.php, test/@dave/clickup/probe_opportunity_fields.php, test/@dave/probe_clickup_desc_match.php, test/@dave/test_model_load_behavior.php, dbchanges2/Forecast/2026-06-25a - Add unique index on Opportunities netsuiteOpportunityInternalId.sql, _underscore/Model/Forecast/Opportunity.php, test/@dave/approach/TRUE-80044.md, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
 | [NetSuite → Forecast Open-Orders Sync (salesOrder webhook → OpenOrderItems)](features/netsuite-salesorder-open-orders-sync.md) | Webhook-driven, single-record port of the legacy open-orders importer (TRUE-79142). | worker2/Worker/Netsuite/SalesOrder.php, worker2/Worker/Netsuite.php, test/@dave/probe_salesorder_rest_shape.php, test/@dave/probe_open_order_lines.php, test/@dave/check_so_status.php, test/@dave/check_so_history.php, test/@dave/probe_so_rest_lines.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_open_order_gating.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
 | [NetSuite Supporting-Record Webhook Importer (the reusable recipe)](features/netsuite-supporting-record-webhook-importer.md) | A single **repeatable recipe** for porting a legacy daily-pull NetSuite *supporting-record* importer (the lookup/dimension tables behind Forecast2 — Employees, | worker2/Worker/Netsuite/Employee.php, worker2/Worker/Netsuite/Account.php, worker2/Worker/Netsuite/Classification.php, worker2/Worker/Netsuite/Customer.php, worker2/Worker/Netsuite/Item.php, worker2/Worker/Netsuite.php, _underscore/Model/Forecast/Employee.php, _underscore/Model/Forecast/Account.php, _underscore/Model/Forecast/Classification.php, _underscore/Component/Forecast/Db/Db.php, test/@dave/test_employee_lifecycle.php, test/@dave/test_account_lifecycle.php, test/@dave/test_classification_lifecycle.php, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, worker/crons/toga2/forecast2/import_supporting_records.php |
 | [Background Email-Template Worker (_Worker_Notification_EmailTemplate)](features/notification-email-template.md) | `_Worker_Notification_EmailTemplate::Send(...)` dispatches a **stored, client-defined `EmailTemplates` row off-thread** as a background WorkerJob. | worker2/Worker/Notification/EmailTemplate.php, worker2/Worker/Client/True.php, _underscore/Model/Client/EmailTemplate.php |

package/knowledge/2.0/apps/worker2/features/netsuite-opportunity-sync.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-06-30
+updated: 2026-07-15
 owners: ["dfranks"]
 files:
   - worker2/Worker/Netsuite.php
@@ -25,6 +25,8 @@ files:
   - test/@dave/probe_clickup_desc_match.php
   - test/@dave/test_model_load_behavior.php
   - "dbchanges2/Forecast/2026-06-25a - Add unique index on Opportunities netsuiteOpportunityInternalId.sql"
+  - _underscore/Model/Forecast/Opportunity.php
+  - test/@dave/approach/TRUE-80044.md
   - worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php
 related:
   - ./netsuite-salesorder-open-orders-sync.md
@@ -166,6 +168,39 @@ Shared helpers: `buildCustomFields()` (the field array, used by both create and
   removing that `setLogging(false)` is under review (handle the 20–50 KB payload volume the
   prod-appropriate way — retention/sampling/summary — not by defaulting logging off).
 
+## Atomic ClickUp-task claim (TRUE-80044 — decided approach, not yet built)
+
+Concurrent NetSuite opportunity webhooks (a real edit racing NetSuite's hourly `OPP` fallback) can both
+pass the non-atomic find-then-create in `maybeCreateClickupTask()` and create **duplicate** ClickUp
+tasks. The fix is a **race-safe DB claim** on the **existing `Forecast.Opportunities` table** — an added
+`clickupTaskId` column — **not** a new dedicated table. (A prior plan proposed a new
+`NetsuiteOpportunityClickupTask` table in `Client_True`; reviewer Rohan Girish rejected that and mandated
+reusing `Forecast.Opportunities`.)
+
+How the claim is race-safe:
+- **Conditional UPDATE = the atomic claim.** A raw `_Query`:
+  `UPDATE Opportunities SET clickupTaskId='<uuid sentinel>' WHERE netsuiteOpportunityInternalId=X AND
+  clickupTaskId IS NULL`, then commit. `affectedRows === 1` → this webhook won the claim and proceeds to
+  create the task; `=== 0` → another concurrent handler already claimed it → **skip**. The winner then
+  persists the real task id over the sentinel; a `createTask()` failure runs a **compensation-delete**
+  (reset `clickupTaskId` back to `NULL`, guarded by the sentinel uuid so it only clears its own claim).
+- It works because (a) `Forecast.Opportunities` already has a **UNIQUE index on
+  `netsuiteOpportunityInternalId`** (`dbchanges2/Forecast/2026-06-25a`) → the guarded UPDATE locks a
+  single row, and (b) in both `post()` and `put()`, `importOpportunity()` runs and **COMMITS** the
+  Forecast row (`Opportunity.php:289`) **before** `maybeCreateClickupTask()` is called, so the row exists
+  and is committed at claim time.
+- **Raw `_Query` is required** for the guarded UPDATE — the `_underscore` ORM `save()` is load-then-write
+  and cannot express an atomic guarded UPDATE. All non-claim writes stay on the ORM.
+- **Commit the claim BEFORE the ClickUp POST** — releases the single-row lock and dodges the
+  `DB_FORECAST` lazy-transaction drop.
+- **`createTask()` must return the new task id (`:string`)** — it currently returns `void` and discards
+  the ClickUp response; the winner needs the id to persist over the sentinel.
+- **Edge case:** if `importOpportunity()` was skipped (no NS status → no Forecast row), there is nothing
+  to claim; that rare path falls back to today's unguarded create.
+
+Requires a new `dbchanges2/Forecast/` ALTER adding the `clickupTaskId` column, and a
+`_underscore/Model/Forecast/Opportunity.php` field addition. Plan: `test/@dave/approach/TRUE-80044.md`.
+
 ## Data model
 
 Writes `Forecast.Opportunities` (header) + `Forecast.OpportunityItems` (children), faithful to
@@ -287,16 +322,18 @@ None — platform-wide Forecast sync.
   the create branch) you must change the **leading** content or clear the field entirely. Verified both ways
   on opp 74266/internalId 7161054: `74266--` → matched (update branch, skipped); field removed → missed
   (create branch, new task `868k2rfj8`).
-- **ClickUp dedup keys on the `Opportunity #` field, not a DB column.** We deliberately did NOT add
-  a `clickupTaskId` column — the dedup lookup queries ClickUp itself by `Opportunity #` (= tranId).
-  Consequences to know:
+- **ClickUp dedup keys on the `Opportunity #` field, not a DB column.** Historically we did NOT add
+  a `clickupTaskId` column — the dedup lookup queried ClickUp itself by `Opportunity #` (= tranId).
+  **This is being replaced (TRUE-80044) by a race-safe DB-claim approach — see "Atomic ClickUp-task
+  claim" below.** Consequences of the field-lookup approach to know:
   - **Legacy tasks had `Opportunity #` empty** (the old `webhook/` handler never set it), so they
     can't be matched until backfilled — see the backfill tool below. Without backfill, the first
     edit of a pre-existing opp creates a *second* task.
   - **The find-then-create is not atomic.** Two near-simultaneous events for one opp (e.g. the
     NetSuite hourly `OPP` fallback firing alongside a real edit) can both miss and both create. A
     committed local task id would close most of that window; the field-lookup approach can't.
-    ClickUp custom fields aren't unique-constrained, so some race exists regardless.
+    ClickUp custom fields aren't unique-constrained, so some race exists regardless. **TRUE-80044
+    closes this window with the atomic `clickupTaskId` DB claim — see "Atomic ClickUp-task claim" above.**
   - **Update is per-field** (`/task/{id}/field/{id}` — no bulk set), so a mid-loop failure leaves a
     partially-updated task; the thrown error names how far it got. The Presales Lead users field is
     **add-only** on update (we don't read the prior assignee to remove it).
@@ -449,6 +486,19 @@ deprecated** for production opportunity code.
   blocked on the Aaron stakeholder decision noted above.
 
 ## Change history
+- 2026-07-15 — **Decided the atomic ClickUp-task dedup approach (TRUE-80044) — planning/rework, no code
+  shipped.** The non-atomic find-then-create lets concurrent NS opportunity webhooks create duplicate
+  ClickUp tasks. Decided fix: a race-safe **conditional-UPDATE claim** on a new `clickupTaskId` column of
+  the **existing `Forecast.Opportunities`** table (`UPDATE … SET clickupTaskId='<uuid>' WHERE
+  netsuiteOpportunityInternalId=X AND clickupTaskId IS NULL`; commit; `affectedRows===1` wins, `===0`
+  skips), with a sentinel-guarded compensation-delete on `createTask()` failure. Reviewer (Rohan Girish)
+  **rejected** the prior plan's new dedicated `NetsuiteOpportunityClickupTask` table and mandated the
+  column-on-Forecast.Opportunities approach — safe because the table already has a UNIQUE index on
+  `netsuiteOpportunityInternalId` (single-row lock) and `importOpportunity()` commits the row before
+  `maybeCreateClickupTask()` runs. Requires raw `_Query` (ORM `save()` can't express a guarded UPDATE),
+  claim committed before the ClickUp POST (lock release + lazy-transaction), and `createTask()` changed to
+  return the new task id. See "Atomic ClickUp-task claim" section. Plan: `test/@dave/approach/TRUE-80044.md`.
+  (dfranks)
 - 2026-06-30 — **Built the CU→NS reverse sync + NS→CU stage drive (TRUE-79181) — integration now
   bidirectional for field + stage.** New `_Worker_Clickup_Opportunity` handler
   (`worker2/Worker/Clickup/Opportunity.php`), dispatched by `_Worker_Clickup::Webhook` via

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.340",
+  "version": "1.0.341",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",