toga-ai 1.0.896 → 1.0.897

package/knowledge/2.0/apps/_underscore/features/netsuite-rest-client.md

@@ -6,7 +6,7 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
+updated: 2026-09-25
 owners: ["dfranks", "jcardinal", "bala", "kyalamarthi", "ajean"]
 files:
   - _underscore/Component/Api/Netsuite/Netsuite.php
@@ -73,6 +73,10 @@ Shared 2.0 `_underscore` NetSuite REST client (`_Component_Api_Netsuite`) — re
 
 Account 1095849 is **shared by every client**, so an external id must be globally unique and prefixed — never a per-client record number. Worked example: [Toga → NetSuite sales-order push](../../worker2/features/netsuite-salesorder-outbound-push.md) (`toga-so-<salesOrder uuid>`).
 
+### RESTlets — `sendRestlet(scriptId, deployId, payload)` (added 2026-09-24)
+
+Additive. Uses a SEPARATE OAuth2 token with scope `restlets` (widening the shared scope could break every NS call if the integration disallows it); `generateClientAssertion()` takes optional `$scope` (default unchanged). Host = `.suitetalk.api.` swapped to `.restlets.api.`, or `[netsuite] restletHost`. Logging off, TLS verify on. First user: [opportunity documents sync](../../worker2/features/netsuite-opportunity-documents-sync.md).
+
 ### SuiteQL escaping and filtering
 
 **String-literal escaping — double the quote, never backslash.** SuiteQL is **ANSI SQL sent to NetSuite as the JSON `q` string** — it is **not** executed against MySQL. It escapes a single quote by **doubling it** (`''`), not with a backslash. So `_Database::escape()` (mysqli `real_escape_string`, backslash-style) is the **wrong tool** and would corrupt values like `O'Brien`. Use `str_replace("'", "''", $value)`. (worker2's `SalesOrder.php` carries this as a private `escapeSuiteQl()` helper; any new SuiteQL caller must do the same.)
@@ -112,8 +116,13 @@ Account 1095849 is **shared by every client**, so an external id must be globall
 - **Don't escape SuiteQL with `_Database::escape()`** — backslash escaping corrupts `O'Brien`-style values. SuiteQL doubles the quote (`''`).
 - **204-with-empty-body is success, not failure.** Both create (204 + `Location`) and update (204, no body) return no payload; treat a 2xx with empty body as success and key off the status, not the body.
 
+- **`_ApiRequest` core default does NOT verify TLS peers** (found 2026-09-24). `sendRestlet` turns it on; the core default needs its own ticket.
+
 ## Related
 
 - [NetSuite → Forecast Open-Orders Sync](../../worker2/features/netsuite-salesorder-open-orders-sync.md) — first caller of `createRecord()` (the SalesOrder outbound push).
 - [NetSuite → TOGA Opportunity Sync](../../worker2/features/netsuite-opportunity-sync.md) — the two-copies gotcha and the `Logs.Api` logging note.
 - [_underscore Framework Architecture](../architecture.md)
+
+## Change history
+- 2026-09-25 — Added sendRestlet() with separate restlets-scope token (jcardinal)

package/knowledge/2.0/apps/worker2/INDEX.md

@@ -31,11 +31,12 @@
 | [Etilize Catalog Item Import & Refresh](features/etilize-catalog-item-import.md) | Client-generic catalog onboarding: ingest an item catalog from an S3 CSV and enrich each item from the Etilize datasheet, writing directly to `DB_CLIENT`; open |
 | [Etilize Item Translation Import](features/etilize-item-translation-import.md) | Imports non-English item + feature text from Etilize into the client's `ItemTranslations` (and feature sidecar tables), writing directly to `DB_CLIENT`; open wh |
 | [Monitoring Framework (Orchestrator + Child Monitors)](features/monitoring-framework.md) | DB-driven monitoring framework: one orchestrator applies anti-flap + notification rules to any registered "is X healthy?" child check; open when adding a monito |
-| [Bidirectional NetSuite ↔ Forecast ↔ ClickUp opportunity sync — AGREED DESIGN, not built](features/netsuite-clickup-opportunity-bidirectional-design.md) | Agreed-but-unbuilt design for true bidirectional NetSuite↔Forecast↔ClickUp opportunity sync; open before building anything to replace today's create-only sync s |
+| [Bidirectional NetSuite ↔ Forecast ↔ ClickUp opportunity stage sync — BUILT (stage map, echo guard, claim split)](features/netsuite-clickup-opportunity-bidirectional-design.md) | Two-way NetSuite↔ClickUp opportunity STAGE sync (shared `_Component_Opportunity_StageMap`, actor echo guard, claim split, stale-claim takeover); open before tou |
 | [NetSuite Integrations Monitor (Monitor/Operations/NetsuiteIntegrations)](features/netsuite-integrations-monitor.md) | Cross-client health check that flags stuck NetSuite↔2.0 integrations; open when a NetSuite integration hangs or when building an all-client OneUptime sweep. |
 | [NetSuite item → client routing (there is NO client on the NetSuite item record)](features/netsuite-item-client-routing.md) | Evidence that no NetSuite field can route an item to a client; open before designing NetSuite item webhook phase 2 (TRUE-80553) so the "route by a NetSuite fiel |
 | [ClickUp Opportunity Client vs End-Customer Labels (Stakeholders / End Customer)](features/netsuite-opportunity-client-labels.md) | The NetSuite→ClickUp opportunity task's two customer labels (Stakeholders=client, End Customer=`custbody_end_customer`), their backfills/digest and the fuzzy-ma |
-| [NetSuite ↔ ClickUp / TOGA Opportunity Sync (API Message Queue + worker2 webhook)](features/netsuite-opportunity-sync.md) | Outbound NetSuite→Forecast→ClickUp opportunity sync (API Message Queue + worker2 webhook), its create-only ClickUp task claim, triage, and the CU→NS reverse leg |
+| [NetSuite ↔ ClickUp Opportunity Notes & Files Sync (_Worker_Netsuite_OpportunityDocuments + RESTlet)](features/netsuite-opportunity-documents-sync.md) | Polling job syncing NS opportunity User Notes→ClickUp comments and files both ways via a custom RESTlet and the Forecast.OpportunityDocuments ledger; open when |
+| [NetSuite ↔ ClickUp / TOGA Opportunity Sync (API Message Queue + worker2 webhook)](features/netsuite-opportunity-sync.md) | Outbound NetSuite→Forecast→ClickUp opportunity sync (API Message Queue + worker2 webhook), ClickUp task creation claim, triage, and the CU→NS reverse leg; open |
 | [NetSuite → Forecast Open-Orders Sync (salesOrder webhook → OpenOrderItems)](features/netsuite-salesorder-open-orders-sync.md) | Webhook-driven NetSuite `salesOrder` → `Forecast.OpenOrderItems` sync (inbound half of `SalesOrder.php`); open when touching open-orders import, its locationId/ |
 | [Toga → NetSuite Sales-Order Push (multi-client, mapping-driven worker)](features/netsuite-salesorder-outbound-push.md) | Outbound half of `SalesOrder.php` (below the `OUTBOUND PUSH (REST)` banner): pushes a Toga sales order into NetSuite over REST; open when creating/updating a NS |
 | [NetSuite Supporting-Record Backfill Worker (bulk reconciliation recipe)](features/netsuite-supporting-record-backfill-worker.md) | Bulk reconciliation recipe for re-deriving one Forecast column for every row from NetSuite in batches; open when a webhook column needs history fixed or a mass |

package/knowledge/2.0/apps/worker2/features/clickup-project-routing.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
+updated: 2026-09-25
 owners: [jcardinal, ajean]
 files:
   - worker2/Worker/Clickup/Project.php
@@ -130,6 +130,8 @@ Read-only / setup actions, invoked via `curl -X POST https://worker.togahub.com/
 - **⚠ `CLICKUP_SERVICE_ACCOUNT_USER_ID` is stale, and that is a CORRECTNESS bug — not housekeeping.** `Project.php:245` still holds `'12633140'`, **a real person's ClickUp user id**. The dedicated service account now exists: `devteam@togatech.com` = user id **87469444** (username "Development Team"), verified via `GET /user`, with read access confirmed to space `90113928591`, folder `90117060687` and the Presales Qualification list. Because `getServiceAccountUserId()` resolves the id dynamically and only *falls back* to the constant, rotation will not break outright — **but leaving the old id makes that person's genuine human edits match the service-account branch and be silently ignored by `enforceStatusOwnership`.** Update the constant as part of the rotation.
 - **PR #137 also carries a ClickUp rate limiter that is NOT part of this feature — and it was rejected in review.** The open branch `fix/clickup-project-hub-routing` bundles a 533-line `Worker/Clickup/Api.php` limiter (plus dbchanges2 #449) whose premise, coverage, and self-test are all wrong: api2 and tools do **not** call ClickUp (verified by grep), ~24 of ~116 call sites stay unmetered, and its `TestNoDirectApiCalls()` guard cannot see the 18 hand-built `api.clickup.com` sites. Treat rate limiting as a separate subject with its own design — [Outbound third-party API rate budget](../../_underscore/features/outbound-api-rate-budget.md).
 
+- **Opportunity CU→NS dispatch in `Worker/Clickup.php` is gated by folder `90117060687` and does NOT return early** (since 2026-09-24) — these routing handlers still run for opportunity tasks. Our own writes are dropped upstream by the actor echo guard ([stage sync](./netsuite-clickup-opportunity-bidirectional-design.md)).
+
 ## Related
 
 - [Worker (worker2) Architecture](../architecture.md) — always-HTTP-200, commit-before-SQS.
@@ -139,3 +141,6 @@ Read-only / setup actions, invoked via `curl -X POST https://worker.togahub.com/
 - [ClickUp credential rotation](../workflows/clickup-credential-rotation.md)
 - [ClickUp Subtask Activity](./clickup-subtask-activity.md)
 - [Outbound third-party API rate budget](../../_underscore/features/outbound-api-rate-budget.md)
+
+## Change history
+- 2026-09-25 — Noted folder-gated opportunity dispatch, no early return (jcardinal)

package/knowledge/2.0/apps/worker2/features/netsuite-clickup-opportunity-bidirectional-design.md

@@ -1,101 +1,94 @@
 ---
-title: Bidirectional NetSuite ↔ Forecast ↔ ClickUp opportunity sync — AGREED DESIGN, not built
+title: Bidirectional NetSuite ↔ Forecast ↔ ClickUp opportunity stage sync — BUILT (stage map, echo guard, claim split)
 framework: "2.0"
 repo: worker2
 project: Worker
 client: shared
 type: feature
-status: draft
-updated: 2026-09-16
-owners: ["ajean"]
+status: active
+updated: 2026-09-25
+owners: ["ajean", "jcardinal"]
 files:
+  - worker2/Component/Opportunity/StageMap/StageMap.php
+  - worker2/Controller/OpportunityStageMapTest.php
   - worker2/Worker/Netsuite/Opportunity.php
   - worker2/Worker/Clickup/Opportunity.php
+  - worker2/Worker/Clickup.php
   - worker2/Worker/Clickup/Project.php
+  - dbchanges2/Forecast/2026-09-24a - Add dtClickupClaimed to Opportunities.sql
   - worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_opportunities.php
 related:
   - ./netsuite-opportunity-sync.md
+  - ./netsuite-opportunity-documents-sync.md
   - ./clickup-project-routing.md
   - ../workflows/clickup-credential-rotation.md
   - ../../../../1.0/apps/worker/features/forecast2-netsuite-reconciliation.md
 ---
 
-Agreed-but-unbuilt design for true bidirectional NetSuite↔Forecast↔ClickUp opportunity sync; open before building anything to replace today's create-only sync so the agreed decisions are not re-litigated.
+Two-way NetSuite↔ClickUp opportunity STAGE sync (shared `_Component_Opportunity_StageMap`, actor echo guard, claim split, stale-claim takeover); open before touching any opportunity stage/close/reopen logic or the stage map rules.
 
 ## Summary
 
-**Status: designed and agreed, NOT built. No code from this design has shipped.** Recorded because the decisions cost the most conversation of any part of the work and are the least recoverable from reading code — the code does not exist yet. Provenance: decided in a 2026-09-04 planning session with `ajean`.
+Built 2026-09-24 (`jcardinal`, design by `ajean`). Code on worker2 `_production`; flags `SYNC_STAGE_TO_CLICKUP` (NS→CU) and `SYNC_CLOSE_TO_NETSUITE` (CU→NS close/reopen) now `true`. `UPDATE_CLICKUP` (full-task NS→CU rewrite) stays `false`. Live verification still pending — see Handoff. Notes/files are a separate job: [opportunity documents sync](./netsuite-opportunity-documents-sync.md).
 
-Goal: a genuinely bidirectional opportunity sync between NetSuite, `Forecast.Opportunities`, and the ClickUp Opportunities folder. Today's deployed behaviour is **create-only**: the `clickupTaskId` atomic claim doubles as "should we do anything at all", so NetSuite stage changes never reach ClickUp after task creation, and `UPDATE_CLICKUP` is dead behind the claim. See [NetSuite ↔ ClickUp Opportunity Sync](./netsuite-opportunity-sync.md) for what actually runs.
+## How it works
 
-## How it works (the agreed design)
+### Per-field ownership
+NetSuite owns: Opportunity Stage dropdown, close state. ClickUp owns: description (Fluffer output — never diffed from NS), assignees, presales fields. NS→CU writes ONLY the Opportunity Stage dropdown (`d8fd5819-d8f7-494b-8671-aff7fbdeb985`); the ClickUp Automation moves the task list from it ([routing](./clickup-project-routing.md)).
 
-### 1. Per-field ownership, not a global on/off switch
+### Stage map — `_Component_Opportunity_StageMap` (PHP, one file, both directions)
+Replaced `SALES_STAGE_MAP` / `STAGE_OPTION_ORDERINDEX` constants (deleted). Rules from Aaron Jean + Eric, 2026-09-24:
 
-The current `UPDATE_CLICKUP` boolean is the wrong shape — it can only choose "NetSuite clobbers ClickUp" or "NetSuite never updates ClickUp". Replace with **per-field ownership**:
+1. **Likely to Close (`entityStatus`) always wins.** `124` Closed-Won and `130` Alternate Quote → Opportunity Won; `14` → Closed Lost.
+2. NS Sales Stage (`custbody_sales_stage` id) ↔ ClickUp:
+   - `1` Stage0, `2` Stage1 → leave alone.
+   - `3` Stage2 ↔ Pre-Sales Qualification.
+   - `4` Stage3 → Quote (unless task is already BU Discovery or Quote).
+   - `5` Stage4 ↔ Pending Client Response.
+   - `6` Stage5 and `9` Stage6 → Contract Negotiations; CU→NS writes `6` unless NS is already `6`/`9`.
+   - `11` Stage7 Waiting for PO ↔ SOW/Contract Signed.
+   - `7`/`8` retired.
+3. **BU Discovery** counts as matching NS `3` and `4`, so PQ→BU→Quote writes nothing. Moving BACK into BU from a later stage writes Stage3 (id `4`) — **interpretation, confirm with Aaron/Eric.**
+4. **CU close** writes `entityStatus` `124`/`14` (no-op if already `124`/`130`/`14`); stage untouched.
+5. **CU reopen** writes `entityStatus` `125` (10%) + mapped stage, and posts a ClickUp comment + NetSuite User Note.
+6. Unknown/future CU stages: never overwritten by open NS stages, never write NS.
 
-| Owner | Fields |
-|---|---|
-| **NetSuite** | Opportunity Stage dropdown, closing task status, amount, close date |
-| **ClickUp** | description, assignees, presales fields |
+Each entry = target value + "already matching" set (the consistency short-circuit: if the other side already matches, write nothing). `Controller/OpportunityStageMapTest.php` walks 1413 round trips and asserts no loops — rerun it after any map edit.
 
-**The description is the Fluffer's output, so NetSuite must not diff it.** That is the concrete reason a global switch fails: one field (stage) needs NS→CU writes while another (description) must never receive them.
+Option orderindex is resolved LIVE from the field's `type_config.options` (fixes the old orderindex-`0` `array_search ?: null` bug; Pre-Sales Qualification now maps back).
 
-### 2. Actor identity is the primary echo guard; value diff is the backstop
+**Decision (CTO-reviewed): the map stays PHP**, not Forecast lookup tables — rows can't hold the "unless" rules, and PHP avoids a cross-repo deploy. Supersedes the earlier "move to lookup tables" plan.
 
-Today echo prevention is a value difference-check alone. Not sufficient:
-- A value guard **cannot distinguish "we wrote it" from "a human wrote the same value."**
-- It **cannot see the loop through `_Worker_Clickup_Project::handleOpportunityUpdate`**, which reacts to our own writes.
+### CU→NS (`Worker/Clickup/Opportunity.php`)
+- Routed from `Worker/Clickup.php` by Opportunities **folder** `90117060687` (all 8 stage lists), no early return — hub routing handlers still run.
+- **Actor echo guard:** drop an event when every history item's actor is our config-token or component-token user (resolved via `GET /user`). Never drop actor `-1` (ClickUp Automation — a human list-move arrives as `taskUpdated` on the dropdown with actor `-1`). Value diff kept as backstop.
+- **Link check (security):** before any PATCH, require `Forecast.Opportunities.clickupTaskId === taskId`. `Opportunity #` is free text; anyone could point a task at another opp.
+- `logged()` commits `DB_LOGS`.
 
-So the guard becomes **actor identity** — drop events authored by our own integration user — with the value diff kept as a backstop rather than the mechanism. Requires the service-account id to be correct; the stale `CLICKUP_SERVICE_ACCOUNT_USER_ID` is a prerequisite fix, not optional cleanup (see [clickup-project-routing](./clickup-project-routing.md) and [credential rotation](../workflows/clickup-credential-rotation.md)).
+### NS→CU (`Worker/Netsuite/Opportunity.php`) — claim split
+- The `clickupTaskId` claim now guards **creation only**. Linked opportunity → `syncStageToClickup()`: fresh task GET, write dropdown only if the map says so; ClickUp 404 = skip.
+- **Stale PENDING takeover:** `PENDING` older than 30 min (or NULL `dtClickupClaimed`) can be re-claimed. `dtClickupClaimed` is raw-SQL only — deliberately NOT on either ORM model (1.0 or 2.0); code checks `SHOW COLUMNS` first. `endClaim()` / `loadClaimTimestamp()` are fenced on the claim timestamp so a slow job can't overwrite a newer owner.
 
-### 3. Consistency short-circuit — the answer to the many-to-one map
-
-**If the current value on a side already maps to the incoming value, write nothing.** This resolves the stage map's many-to-one ambiguity **in both directions**, replacing the previous position that a reverse map can only cover the unambiguous terminal stages. No percentage has to be guessed: if the existing value is already consistent with the incoming one, there is nothing to decide.
-
-### 4. The stage map moves out of PHP constants into `Forecast` lookup tables
-
-Editable DB lookup tables keyed on **`SalesStages.netsuiteInternalId`**, replacing the `SALES_STAGE_MAP` constant. **Preserve "absent = leave alone"** for unmapped NetSuite stages — an unmapped stage must remain a no-op, not a default or an error.
-
-### 5. `entityStatus` is evaluated BEFORE Sales Stage
-
-Precedence matters because the two fields can disagree:
-
-| NS `entityStatus` | Meaning | ClickUp stage |
-|---|---|---|
-| `124` | Closed - Won | Opportunity Won |
-| `130` | Alternate Quote | Opportunity Won |
-| `14` | Closed Lost | Closed Lost |
-
-`entityStatus` ("Likely to Close") wins; Sales Stage is consulted only when `entityStatus` does not decide. Note `130` **is** mapped here — older docs recorded it as unmapped.
-
-### 6. The reconciler stops INSERTing and reports instead
-
-`periodic_forecast_discrepancy_fix_opportunities.php` **cannot compute `projectedProfit`** — it fetches no line items at all, and profit is Σ(amount − costEstimate). A component that cannot compute a NOT-NULL column should not author rows. It becomes a **reporter**. Evidence, including the `sql_mode`-driven silent `0.00` and the ClickUp-link deletion path, is in [Forecast2 ↔ NetSuite reconciliation](../../../../1.0/apps/worker/features/forecast2-netsuite-reconciliation.md).
-
-### 7. Document sync — content-hash ledger plus a mandatory polling backstop
-
-- **Filename dedup is insufficient**, because **NetSuite dedup-suffixes uploads** — the same document arrives under a different name. Use a **content-hash ledger** in `Forecast`.
-- Detection is **event-first with a mandatory polling backstop**, because **ClickUp has no attachment webhook event**. The polling is not an optimization; without it attachments are never detected.
-
-### Data model (none built)
-
-New/changed storage this design implies:
-- `Forecast` stage lookup tables keyed on `SalesStages.netsuiteInternalId` (replaces the PHP map).
-- A `Forecast` content-hash ledger for document sync.
-
-Additive columns on `Forecast.Opportunities` are safe — no `SELECT *` over that table exists anywhere. But mind the four concurrent writers and the deliberate `clickupTaskId` omission in the 1.0 model, both mapped in [the sync doc's Data model](./netsuite-opportunity-sync.md).
+### Reconciler
+The 1.0 `periodic_forecast_discrepancy_fix_opportunities.php` still INSERTs (Aaron's 2026-09-09 session kept and completed it) — the old "reconciler becomes a reporter" plan is dropped.
 
 ## Gotchas
+- `taskMoved` is not subscribed for space `90113928591`; stage detection relies on the dropdown `taskUpdated`.
+- `Opportunities` FK children (`OpportunityDocuments`) use ON DELETE CASCADE because the 1.0 reconciler/fixer delete `Opportunities` rows.
+- worker2 has no `?controller=` URL routing — run the map test / probes via CLI runner.
 
-- **The ClickUp Automation is a hard dependency of this design.** A ClickUp opportunity stage is **list membership, not a field**, and only the bidirectional Automation on the "Opportunity Stage" dropdown can move a task (v2 API cannot — `PUT` with `list_id` returns 200 and does nothing). Writing the dropdown *is* the stage write. Details: [clickup-project-routing](./clickup-project-routing.md).
-- **`taskMoved` is not subscribed** for space `90113928591`, so the event that most directly signals a human stage change is not currently delivered. Any CU→NS stage detection needs that subscription registered first.
-- **The orderindex-`0` reverse-map bug must be fixed before CU→NS stage reads can be trusted** — Pre-Sales Qualification currently never maps back. See the CU→NS gotchas in [the sync doc](./netsuite-opportunity-sync.md).
-- **Do not implement this on top of the existing claim-as-gate.** Separating "have we created a task?" from "should we sync this field?" is a precondition for every decision above.
+## Handoff (open, 2026-09-25)
+1. Live stage-change test after flag enable (both directions, incl. close + reopen).
+2. Unverified: NS workflows allow the `entityStatus` PATCH; `addNote` fields.
+3. 1.0 PRs library#880, worker#1718, tools#15 unmerged — nightly reconciler still unguarded.
+4. ClickUp webhook migration to the devteam account ([credential rotation](../workflows/clickup-credential-rotation.md)).
+5. Confirm BU Discovery "move back" rule with Aaron/Eric.
 
 ## Related
+- [NetSuite ↔ ClickUp Opportunity Sync](./netsuite-opportunity-sync.md) — transport, creation, triage.
+- [Opportunity documents sync](./netsuite-opportunity-documents-sync.md) — notes/files.
+- [ClickUp routing](./clickup-project-routing.md) — stage-as-list Automation.
 
-- [NetSuite ↔ ClickUp Opportunity Sync](./netsuite-opportunity-sync.md) — what actually runs today (create-only).
-- [ClickUp Project & Opportunity Multi-List Routing](./clickup-project-routing.md) — the stage-as-list Automation dependency.
-- [ClickUp credential rotation](../workflows/clickup-credential-rotation.md) — the stale service-account id.
-- [Forecast2 ↔ NetSuite reconciliation](../../../../1.0/apps/worker/features/forecast2-netsuite-reconciliation.md) — the reconciler data-loss path.
+## Change history
+- 2026-09-25 — Built: StageMap component, actor guard, link check, claim split, stale takeover; flags on. Map stays PHP; reconciler keeps INSERT (jcardinal)
+- 2026-09-16 — Agreed design recorded, not built (ajean)

package/knowledge/2.0/apps/worker2/features/netsuite-opportunity-documents-sync.md

@@ -0,0 +1,60 @@
+---
+title: NetSuite ↔ ClickUp Opportunity Notes & Files Sync (_Worker_Netsuite_OpportunityDocuments + RESTlet)
+framework: "2.0"
+repo: worker2
+project: Worker
+client: shared
+type: feature
+status: active
+updated: 2026-09-25
+owners: ["jcardinal", "ajean"]
+files:
+  - worker2/Worker/Netsuite/OpportunityDocuments.php
+  - _underscore/Component/Api/Netsuite/Netsuite.php
+  - test/@jeff/netsuite/opportunity-documents/rl_opportunity_documents.js
+  - dbchanges2/Forecast/2026-09-24b - Add OpportunityDocuments ledger.sql
+  - dbchanges2/Core/2026-09-24a - Insert Opportunity Documents Sync CronJob.sql
+related:
+  - ./netsuite-clickup-opportunity-bidirectional-design.md
+  - ./netsuite-opportunity-sync.md
+  - ../../_underscore/features/netsuite-rest-client.md
+---
+
+Polling job syncing NS opportunity User Notes→ClickUp comments and files both ways via a custom RESTlet and the Forecast.OpportunityDocuments ledger; open when notes/attachments don't appear, duplicate, or when touching the RESTlet.
+
+## Summary
+
+Built 2026-09-24. Cron action `Netsuite/OpportunityDocuments/Sync` every 5 min (`dryRun` param available). `SYNC_ENABLED = true`. Directions:
+- NS User Notes → CU comments.
+- NS files → CU attachments.
+- CU attachments → NS files (folder `2008026` "ClickUp Opportunity Uploads", private).
+- CU comments → NS: **not synced** (by design).
+
+Go-live forward only: `SYNC_START = '2026-09-24 00:00:00'` UTC (NS side is day-granular). Polling, because NS note/file adds don't fire the opportunity UE and ClickUp has no attachment webhook.
+
+## How it works
+
+- Per run: 24h lookback, max 25 items / 50 tasks, 6MB file cap. Blocked script/web extensions → `SKIPPED`.
+- NS access goes through the RESTlet via `_Component_Api_Netsuite::sendRestlet()` ([REST client](../../_underscore/features/netsuite-rest-client.md)).
+- **RESTlet** `rl_opportunity_documents.js` — actions `changes` / `getFile` / `addFile` / `addNote`. `getFile` only serves files attached to the given opportunity. Prod: script `customscript_rl_opportunity_documents` (internal `2831`), deploy `customdeploy_rl_opportunity_documents`, audience Administrator, param `custscript_opp_docs_upload_folder` = `2008026`. Runs as integration "TOGa OAuth 2.0" (Jeffrey Cardinal / Administrator; RESTlets scope on).
+- **Ledger `Forecast.OpportunityDocuments`:** dedup by `(direction, sourceId)`; `targetId` marks our own copies (echo guard); sha256 per opportunity (NS dedup-suffixes filenames, so names can't dedup). `PENDING` re-claimed after 30 min; `FAILED` after 5 attempts. FK to `Opportunities` ON DELETE CASCADE (1.0 reconciler/fixer delete opp rows).
+- Job fails loud at end if any item failed (WorkerJobs `isSuccess=0`).
+- Query cache is disabled inside `Sync` — `_Query` caches SELECTs by SQL text and writes don't clear it, so ledger re-reads went stale.
+
+## Gotchas (NetSuite SuiteScript / API)
+- Opportunity search type has NO `mainline` filter (`SSS_INVALID_SRCH_FILTER`) — use `search.Type.TRANSACTION` + `type anyof Opprtnty`.
+- A datetime string with `after` on `file.created` / `usernotes.notedate` throws `UNEXPECTED_ERROR` — use `format.Type.DATE` with `onorafter`.
+- NS REST record API has no `note` type (404); SuiteQL note title/note/date are `NOT_EXPOSED`. SuiteQL `file` table IS exposed. Hence the RESTlet.
+- When creating script/deploy/param IDs type only the suffix (`_rl_...`) — NetSuite adds the prefix and will double a typed one.
+- Replacing a RESTlet file = File Cabinet edit/re-upload. Verify it took: identical stack line numbers mean the old file is still live.
+
+## Handoff (open, 2026-09-25)
+1. Live check: task `868k1ytc5` should get the "Sync test" comment; `868jh6x08` should get `TOGA_Technology.065763.v1.18.pdf`.
+2. Unverified: ClickUp `date_updated` bumps on attachment add (the CU→NS poll depends on it); `addNote` field set.
+
+## Related
+- [Stage sync](./netsuite-clickup-opportunity-bidirectional-design.md)
+- [Opportunity sync transport](./netsuite-opportunity-sync.md)
+
+## Change history
+- 2026-09-25 — Created: polling notes/files sync, RESTlet deployed in prod, ledger + cron (jcardinal)

package/knowledge/2.0/apps/worker2/features/netsuite-opportunity-sync.md

@@ -6,13 +6,14 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
+updated: 2026-09-25
 owners: ["dfranks", "kyalamarthi", "ajean", "jcardinal"]
 files:
   - worker2/Worker/Netsuite.php
   - worker2/Worker/Netsuite/Opportunity.php
   - worker2/Worker/Clickup.php
   - worker2/Worker/Clickup/Opportunity.php
+  - worker2/Component/Opportunity/StageMap/StageMap.php
   - worker2/Controller/Index.php
   - _underscore/Worker.php
   - test/@dave/NetSuite/api-message-queue/lib_amq_queue.js
@@ -32,6 +33,7 @@ files:
   - worker2/Worker/Clickup/Project.md
 related:
   - ./netsuite-clickup-opportunity-bidirectional-design.md
+  - ./netsuite-opportunity-documents-sync.md
   - ../workflows/clickup-credential-rotation.md
   - ./netsuite-opportunity-client-labels.md
   - ./netsuite-salesorder-open-orders-sync.md
@@ -41,7 +43,7 @@ related:
   - ../../../../standalone/apps/claude/workflows/mcp-tool-usage.md
 ---
 
-Outbound NetSuite→Forecast→ClickUp opportunity sync (API Message Queue + worker2 webhook), its create-only ClickUp task claim, triage, and the CU→NS reverse leg; open when opportunity tasks are missing/duplicated or when touching any part of this integration.
+Outbound NetSuite→Forecast→ClickUp opportunity sync (API Message Queue + worker2 webhook), ClickUp task creation claim, triage, and the CU→NS reverse leg; open when opportunity tasks are missing/duplicated or when touching any part of this integration. Stage rules live in the stage-sync doc.
 
 ## Summary
 
@@ -119,15 +121,11 @@ How the claim is race-safe:
 
 Required a new `dbchanges2/Forecast/` ALTER adding the `clickupTaskId` column, and a `_underscore/Model/Forecast/Opportunity.php` field addition. Plan: `test/@dave/approach/TRUE-80044.md`.
 
-#### Consequence: `updateTask()` is unreachable — `UPDATE_CLICKUP` alone does NOT revive it
+#### Claim guards creation only (since 2026-09-24)
 
-`maybeCreateClickupTask()` claims the row **before** it ever looks at ClickUp (`Opportunity.php:40`): `UPDATE Opportunities SET clickupTaskId='PENDING' WHERE … AND clickupTaskId IS NULL`. Once `clickupTaskId` holds a real id, **every later webhook for that opportunity returns `skipped (already claimed)` and exits** — it never reaches the find-then-update branch.
+The claim used to double as "do anything at all", making the integration create-only. Now split: a linked opportunity goes to `syncStageToClickup()` (stage dropdown only, behind `SYNC_STAGE_TO_CLICKUP = true`). `updateTask()` is still dead behind `UPDATE_CLICKUP = false` — leave it. A stale `PENDING` (>30 min via `dtClickupClaimed`, or NULL timestamp) is now taken over automatically. Details: [stage sync](./netsuite-clickup-opportunity-bidirectional-design.md).
 
-So `updateTask()` is dead **twice over**: behind `UPDATE_CLICKUP = false` *and* behind the claim. Flipping the constant activates ~180 lines of code that have **never run in production** and still reaches **none** of the 761 existing tasks — only a separate backfill action reaches those (see [client vs end-customer labels](./netsuite-opportunity-client-labels.md#seed-if-empty-backfill-opportunitystakeholderbackfillbackfill)). `UPDATE_CLICKUP` stays `false`. **This is a trap for the next developer:** the constant's own comment says nothing about the claim interaction, so reading the constant alone gives you the wrong model of what enabling it would do — **anyone "just flipping `UPDATE_CLICKUP`" will find it changes nothing.**
-
-**The claim doubles as "should we do anything at all" (confirmed 2026-09-04).** The claim is not only a dedup guard — because it returns `'skipped (already claimed)'` the moment a real task id exists, it is also the switch that decides whether the handler does any ClickUp work. The durable consequence to state plainly: **NetSuite stage changes never reach ClickUp after task creation.** In deployed behaviour this integration is **create-only**, even though the code reads as bidirectional. Any design that wants live NS→CU updates must separate "have we created a task?" from "should we sync this field?" — see [the bidirectional design](./netsuite-clickup-opportunity-bidirectional-design.md).
-
-**`clickupTaskId` semantics, and why orphaned sentinels are permanent.** `NULL` = unclaimed; `PENDING` = a claim in flight; anything else = the real ClickUp task id. A `PENDING` left behind by a crashed or timed-out request is **never cleaned up**: there is no `register_shutdown_function` in `worker2/Controller/Index.php`, so nothing releases the claim on an abnormal exit, and the compensating release depends on `_Model::load()`, which succeeds only on an **exactly-one** match (see the `load()` footgun below). A stranded `PENDING` therefore blocks that opportunity from ever getting a task until someone clears the column by hand. One such row was found live on 2026-09-04.
+`clickupTaskId` semantics: `NULL` = unclaimed; `PENDING` = claim in flight; else the real task id.
 
 ### Triage: is the opportunity sync working? (`WorkerJobs.output` verdicts)
 
@@ -239,29 +237,19 @@ Three facts worth not rediscovering:
 - **No `SELECT *` over `Forecast.Opportunities` exists anywhere**, so **additive columns are safe** — a new column cannot surprise an existing reader.
 - **`OpportunityItems.line` holds NetSuite line *ids*, not a 1..n sequence.** Max observed in prod is **4196**, with **13,220 rows above 255**, and the column is half of a UNIQUE key. The SOAP-vs-REST semantics of that field **must be verified before the 1.0 cron is ever retired** — if REST numbers lines differently, a cutover creates a **second disjoint item set per opportunity** rather than updating the existing one.
 
-### CU→NS direction (BUILT 2026-06-30, TRUE-79181): reverse sync
-
-The integration is now **bidirectional** for **field + stage**. ClickUp task edits flow back to the linked NetSuite opportunity via a new handler, and NS→CU now drives the ClickUp task **status** (not just description text).
-
-**Handler:** `_Worker_Clickup_Opportunity` (`worker2/Worker/Clickup/Opportunity.php`) — an abstract static-class worker, dispatched by `_Worker_Clickup::Webhook` via `_Worker::runTask('Clickup/Opportunity/Process', {taskId,event,data})`. Routing is gated by the ClickUp **LIST id** (Presales Qualification list `901111987449`) for `taskUpdated`/`taskStatusUpdated` events; non-opportunity lists fall through to the legacy sprint handling in `Clickup.php`.
+### CU→NS direction (TRUE-79181; reworked 2026-09-24)
 
-**CU→NS is UPDATE-ONLY** (never create/delete):
-1. Resolve the linked NS opportunity from the task's `Opportunity #` custom field (`a5529cdc-8dbb-4165-8bc5-9f5cc691796a`, = NS `tranId`) via a one-row SuiteQL `SELECT id FROM transaction WHERE type='Opprtnty' AND tranid='<n>'`.
-2. GET the opportunity with `expandSubResources=true` — this also supplies the **current** NS values for the diff.
-3. **No match → skip.**
-4. PATCH **only** the NS-owned fields that actually differ (currently `title`, `stage`). A no-op write fires no NS event, so the write cannot re-trigger NS→CU. **Echo prevention is the difference-check alone** — there is no actor-identity / "last-synced-from" suppression field (that hardening was explicitly deferred; see Deferred).
+**Handler:** `_Worker_Clickup_Opportunity` (`worker2/Worker/Clickup/Opportunity.php`), dispatched by `_Worker_Clickup::Webhook` via `_Worker::runTask('Clickup/Opportunity/Process', {taskId,event,data})`. Routed by Opportunities **folder** `90117060687` (all 8 stage lists) for `taskUpdated`/`taskStatusUpdated`, with no early return (hub routing still runs).
 
-**CORRECTED 2026-09-04 — `STAGE_MAP` does not exist; the shipped constant is `SALES_STAGE_MAP`.** This section previously described a `STAGE_MAP` keyed on NS **`entityStatus`** internalId mapping to a ClickUp task **status** on the Presales Qualification list, with a table including `130` Alternate Quote as *(unmapped)*. **That constant and that table were replaced and no longer describe the code.**
+**UPDATE-ONLY** (never create/delete):
+1. Resolve NS opp from `Opportunity #` (`a5529cdc-8dbb-4165-8bc5-9f5cc691796a`, = `tranId`) via one-row SuiteQL.
+2. **Require `Forecast.Opportunities.clickupTaskId === taskId`** — else skip (free-text field; security).
+3. Drop echo events by actor (our token users; never actor `-1`); GET the opp with `expandSubResources=true`.
+4. PATCH only differing NS-owned fields (`title`, stage/`entityStatus` per `_Component_Opportunity_StageMap`).
 
-What actually ships: **`SALES_STAGE_MAP`**, declared on `_Worker_Netsuite_Opportunity`, keyed on **`custbody_sales_stage`** and mapping to the ClickUp **"Opportunity Stage" dropdown** (`d8fd5819-d8f7-494b-8671-aff7fbdeb985`) — **not** to a task status. That matters because a ClickUp opportunity **stage is list membership, not a field**: writing the dropdown is how NetSuite moves the task, via the bidirectional ClickUp Automation (see *⚠ A healthy sync reads as dead* above and [clickup-project-routing](./clickup-project-routing.md)).
+Stage/close/reopen rules: [stage sync](./netsuite-clickup-opportunity-bidirectional-design.md). Notes/files: [documents sync](./netsuite-opportunity-documents-sync.md).
 
-**The non-1:1 problem is still real, in the other direction.** Several NetSuite stages map to one ClickUp stage, so a reverse map cannot invert cleanly and CU→NS must not guess. The agreed resolution is a **consistency short-circuit** (if the current value already maps to the incoming value, write nothing) plus moving the map out of PHP constants into editable `Forecast` lookup tables — see [the bidirectional design](./netsuite-clickup-opportunity-bidirectional-design.md). Do not invent a percentage reverse-mapping here.
-
-**`entityStatus` is evaluated BEFORE Sales Stage** in the agreed design: `124` Closed-Won and `130` Alternate Quote → **Opportunity Won**, `14` Closed Lost → **Closed Lost**. `130` is no longer unmapped.
-
-**NS→CU now drives the ClickUp task STATUS** from the NS stage via `STAGE_MAP` (previously the stage was only rendered in the description text). Applied on both create and update; the update PUT carries **only** the changed fields (`name`/`description`/`status`) to stay echo-safe.
-
-**Outbound NS calls** use the `Logs.Api` two-save pattern (`_Model_Core_Logs_Api`, `DB_LOGS`, `DIRECTION_OUT`: save the request row, then save the response/failure row), with `\Sentry\captureException($e); throw $e;` on failure so `WorkerJobs` records `isSuccess=0`. Logged payloads are length-capped. NS access is the **2.0 adapter `_Component_Api_Netsuite`** (GET/PATCH + SuiteQL POST with `Prefer: transient`); the **1.0 `App_Api_Netsuite_Rest` / SOAP toolkit are deprecated** for production opportunity code.
+**Outbound NS calls** use the `Logs.Api` two-save pattern (`_Model_Core_Logs_Api`, `DB_LOGS`, `DIRECTION_OUT`), with `\Sentry\captureException($e); throw $e;` on failure. NS access is the 2.0 adapter `_Component_Api_Netsuite`; 1.0 `App_Api_Netsuite_Rest` / SOAP are deprecated for this code.
 
 ## Gotchas
 
@@ -313,16 +301,13 @@ What actually ships: **`SALES_STAGE_MAP`**, declared on `_Worker_Netsuite_Opport
 
 ### CU→NS gotchas
 
-- **⚠ LIVE BUG (found 2026-09-04, NOT fixed): orderindex `0` never maps back, so the most common stage silently never reaches NetSuite.** `worker2/Worker/Clickup/Opportunity.php:207` does `array_search($orderIndex, STAGE_OPTION_ORDERINDEX, true) ?: null`. `array_search` returns the **key**, and Pre-Sales Qualification's key is `0` — which `?:` converts to `null`. So a task sitting in Pre-Sales Qualification (the most common stage) never writes NetSuite stage 3; the write is dropped with no error. Fix is the strict form: `=== false ? null :`. This is the classic `array_search`-with-`?:` trap — any falsy key (`0`, `''`) is indistinguishable from "not found". A prototype fix was written and **deliberately reverted** on 2026-09-04, so the bug is still live.
 - **SuiteQL over `_Component_Api_Netsuite` has NO placeholder binding.** When a SuiteQL filter value comes from an **external** source (here a ClickUp custom-field `tranId`, not a local PK), sanitize with an **allowlist-and-REJECT-on-mismatch**, not strip-and-use: `$safe = preg_replace('/[^A-Za-z0-9_-]/','',$v); if ($safe!==$v) return null;`. Reject-on-mismatch is the firewall; silently stripping characters is not enough.
 - **Validate `task_id` format at the webhook entry.** ClickUp native task ids are alphanumeric. The legacy `_Worker_Clickup::Webhook` interpolates the raw `$payload->task_id` into Team-DB SQL at ~20 sites (a **pre-existing** SQL-injection exposure, NOT introduced here). A single input-boundary format guard at the top of `Webhook()` closes that class for the raw payload id without rewriting each query. **FLAG:** the broader legacy `$taskId`/`$customTaskId` raw-SQL interpolation throughout `Clickup.php` is real pre-existing debt that warrants a dedicated security ticket.
-- **ClickUp opportunity field ids:** list `901111987449` ("Presales Qualification"); `Opportunity #` = `a5529cdc-8dbb-4165-8bc5-9f5cc691796a`, `Customer #` = `170dc118-b3fa-413b-826b-2753e9405851`.
+- **ClickUp opportunity ids:** folder `90117060687` (routing gate); creation list `901111987449` ("Presales Qualification"); `Opportunity #` = `a5529cdc-8dbb-4165-8bc5-9f5cc691796a`, `Customer #` = `170dc118-b3fa-413b-826b-2753e9405851`.
 
-### Deferred (documented follow-up)
+### Deferred
 
-- **Notes & attachments (CU→NS)** were intentionally deferred — they require NetSuite custom fields (`custentity_cu_synced_notes` / `custentity_cu_synced_files`) + a File Cabinet folder that **do not exist yet**, plus unconfirmed NS REST note/file shapes. Field + stage sync is bidirectional; notes/attachments remain a documented follow-up.
-- **Actor-identity echo hardening** deferred — the difference-check is the current (and sufficient for field/stage) echo-prevention mechanism. A "drop events authored by our own integration user" gate is the planned hardening if/when no-op-safe writes can't fully cover a future field.
-- **Percentage reverse-mapping** (which NS percentage stage a CU `in progress` should map back to) is blocked on the Aaron stakeholder decision noted above.
+- Percentage reverse-mapping is no longer needed — the StageMap "already matching" sets replace it.
 
 ## Related
 
@@ -335,3 +320,6 @@ What actually ships: **`SALES_STAGE_MAP`**, declared on `_Worker_Netsuite_Opport
 - [ClickUp credential rotation](../workflows/clickup-credential-rotation.md).
 - [1.0 webhook: NetSuite → ClickUp opportunity sync](../../../../1.0/apps/webhook/features/netsuite-clickup-opportunity-sync.md) — NetSuite scheduled script 2412, which auto-assigns the Presales Lead that gates task creation.
 - [MCP Tool Usage](../../../../standalone/apps/claude/workflows/mcp-tool-usage.md) — the `toga-db` query traps hit while running the verdict census.
+
+## Change history
+- 2026-09-25 — Claim guards creation only; CU→NS routed by folder, link check, actor guard; orderindex-0 bug fixed (jcardinal)

package/knowledge/INDEX.md

@@ -20,7 +20,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 ## 2.0 framework
 
 - **_underscore** (_Underscore) _(framework core)_ — 90 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
-- **worker2** (Worker) — 77 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
+- **worker2** (Worker) — 78 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
 - **api2** (API) — 26 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
 - **dbchanges2** (Database Changes) _(framework core)_ — 19 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
 - **toga2-supply** (TOGa Supply) — 9 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.896",
+  "version": "1.0.897",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",