toga-ai 1.0.786 → 1.0.787

package/knowledge/sessions/2026-09-09-netsuite-clickup-opportunity-sync-ajean.md

@@ -0,0 +1,167 @@
+---
+type: session
+slug: netsuite-clickup-opportunity-sync
+title: Forecast opportunity data-loss hotfix + NetSuite/ClickUp bidirectional sync handoff
+author: ajean
+repos: [library, worker, tools, worker2, _underscore, dbchanges2]
+framework: "both"
+client: shared
+status: active
+created: 2026-09-09
+updated: 2026-09-09
+---
+
+# Session: netsuite-clickup-opportunity-sync
+**Date:** 2026-09-09
+**Project/Repo:** `library` + `worker` + `tools` (1.0), handing off to `worker2` + `_underscore` + `dbchanges2` (2.0)
+**Task:** Fixed a live data-loss path in the nightly Forecast opportunity reconciler (it deleted rows carrying a ClickUp link, and could delete 3 years of rows on a truncated NetSuite read), then opened three PRs. Handing the remaining NetSuite ↔ ClickUp bidirectional sync work to Jeff Cardinal.
+
+> **HANDOFF TO JEFF.** Aaron is passing this to Jeff. Read *Open PRs* and *Aaron's business rules* first — those two sections are the point of this file. The plan it all comes from is at `~/.claude/plans/i-want-to-update-jolly-pebble.md` on Aaron's machine (813 lines, not in git).
+
+---
+
+## Open PRs — all three OPEN, none merged
+
+Merge order matters: **library first**, then worker, then tools.
+
+| PR | Branch | Size | Mergeable | What it does |
+|---|---|---|---|---|
+| [library#880](https://github.com/agilantsolutions/library/pull/880) | `fix/suiteql-fail-loud-on-partial-page` | 2 commits, +51/−6 | unknown (recheck) | Makes the SuiteQL pagers throw on a bad/partial page instead of returning it as complete |
+| [worker#1718](https://github.com/agilantsolutions/worker/pull/1718) | `fix/forecast-opportunity-delete-guard` | 3 commits, +325/−14 | clean | Reconciler: ClickUp-link delete guard, mass-delete abort, real dry-run, INSERT completion |
+| [tools#15](https://github.com/agilantsolutions/tools/pull/15) | `fix/forecast-fixer-clickup-link-guard` | 1 commit, +23/−1 | clean | Same delete guard in `fixer.php` |
+
+**library#880 must merge and deploy first** — `worker`'s deploy clones `library` onto the box, and worker#1718's location-catch commit exists specifically because of 880's new throw.
+
+Each PR body carries the reasoning, the review results, the test steps and the migration notes. Both PRs that got review comments have a reply explaining what was applied and what was refused, with evidence.
+
+**Nothing is deployed.** The reconciler still runs `0 4 * * *` with `"active": 1`, so the mass-delete risk is live until worker#1718 ships or the cron is disabled.
+
+---
+
+## Aaron's business rules for the stage sync (stated this session — authoritative)
+
+These confirm and slightly extend the plan's mapping. Aaron's words, condensed:
+
+1. **NetSuite Sales Stages stay the right fit for Sales Team and Sales Management.** Aaron is open to changing that on team feedback, but it is the current decision.
+2. **Closed Won and Closed Lost were REMOVED from the NetSuite Sales Stage list.** Open vs Closed is carried by the NetSuite **"Likely to Close"** field (`entityStatus`) instead.
+3. **The logic must read BOTH fields** to decide whether a deal moves to Closed Won or Closed Lost.
+4. **"Likely to Close" WINS over Sales Stage, always.** If Likely to Close is `Closed – Won` or `Closed Lost`, the ClickUp opportunity moves to that stage **regardless** of the Sales Stage. Aaron's own example: Sales Stage = `Stage 1 = Client Discovery Conversation` but Likely to Close = `Closed Lost` ⇒ ClickUp must read **Closed Lost**.
+5. **`Alternate Quote` is a third closing status.** It closes an opportunity in NetSuite without affecting Win/Loss percentages. Since Win/Loss will not be run out of ClickUp, **Alternate Quote ⇒ ClickUp Closed Won** is fine.
+6. **More ClickUp stages will be added later between Presales Qualification and Quote/Proposal/SOW/MSA, and they must NOT write back to the NetSuite Sales Stage.** They matter to Services / Presales / Operations, not to Sales. Today the only one is **BU Discovery**.
+7. **NetSuite stays at `Stage 3 = Quote/ROM/Proposal/SOW/MSA` when a ClickUp opportunity moves from BU Discovery to Quote/Proposal/SOW/MSA.** No NetSuite write on that move.
+
+Rules 4–7 match the plan's already-verified id table, so no re-derivation is needed:
+
+| NetSuite `entityStatus` (Likely to Close) | id | ClickUp dropdown | ClickUp list | Task status |
+|---|---|---|---|---|
+| Closed - Won | 124 | Opportunity Won | Closed Won `901111987478` | `opportunity won` |
+| Alternate Quote | 130 | Opportunity Won | Closed Won `901111987478` | `opportunity won` |
+| Closed Lost | 14 | Closed Lost | Closed Lost `901111987480` | `closed lost` |
+
+Open values that must **not** close: 125 (10%), 126 (25%), 122 (50%), 123 (75%), 128 (95%).
+
+Rule 7 is the plan's "ambiguity rule": NetSuite Sales Stage internalId **4** maps to *both* `BU Discovery` and `Quote/Proposal/SOW/MSA Created`, so moving between those two ClickUp lists resolves to the same NetSuite stage and must write nothing.
+
+**Evaluation order for implementation:** apply `entityStatus` (Likely to Close) **first**; fall through to Sales Stage only when it is not one of the three closing values.
+
+---
+
+## Attachments — Aaron's requirement
+
+- **Documents must transfer both ways** between NetSuite and ClickUp.
+- **The opportunity database is the source of truth.** The metadata ledger lives in **`Forecast`** with a real FK to `Opportunities.id` — not in `Client_True`. (This overrides an earlier `/cto` suggestion; the plan records the same decision.)
+- Ledger holds **metadata only, no blobs**. Dedup on a **content hash, never a filename** — NetSuite dedup-suffixes uploads (`file (2).pdf`), so filename matching loops forever.
+- Clone the proven `ClickUpSubtaskActivity` pattern (`dbchanges2/Team/2026-07-14a`): append-only with a re-claimable `status` so a crashed transfer retries.
+- Sequence **ClickUp → NetSuite first** — it needs no framework change. The NetSuite → ClickUp direction is app-level in worker2, **not** a change to `_ApiRequest` (see *Decisions* #8).
+
+---
+
+## What WORKED
+
+- **Confirmed the plan's Phase A defects are still live**, against today's code rather than trusting the plan. `periodic_forecast_discrepancy_fix_opportunities.php` was byte-identical to `origin/_production` and had **zero** `clickupTaskId` references; `tools/bin/forecast/fixer.php` the same. Both delete paths were unguarded (reconciler `:188`, fixer `:523`). Those two are the **only** deleters of `Forecast.Opportunities` anywhere in worker/tools/worker2.
+- **Added the ClickUp-link delete guard in both files.** Guard placed at the delete loop, not at `$deleteCandidates` build time — at build time the count would be all 746 linked rows every night (noise); at delete time it only reports rows NetSuite actually dropped, and it sits closest to the `DELETE`. `php-reviewer` confirmed `$ids` is populated only inside the guarded loop, so there is no bypass.
+- **Found and fixed a worse bug the plan never mentioned.** `App_Api_Netsuite_Rest::suiteqlListAll()` (`library` `rest.php:2536`) `break`s on a bad page and returns partial data; `listOpportunities()` (`:711`) turns an empty header fetch into `return []`. The reconciler then never unsets its delete candidates, so **up to ~26,766 in-window rows plus their `OpportunityItems` children** go into the DELETE. Both pagers now throw.
+- **Proved the `hasMore` guard works** by reflection against the real private method — 9 cases, all passing (items+hasMore true/false and an empty items array accepted; items-without-hasMore, hasMore=null, no items array, non-array items, null and a raw string all rejected). Output is pasted in the library#880 comment.
+- **Completed the reconciler's INSERT** — it wrote 12 of ~20 columns and left `projectedProfit` at `0.00`. Now writes `projectedProfit`, `leadSource`, `forecastCategoryId`, `salesStageId`, `percentToCloseStatusId`.
+- **Added a real dry-run.** Every write path is now behind it, verified individually: the forked UPDATE/INSERT (`executeForkedProcessSql`), both DELETEs, and the three lookup-table `$model->save()` calls in `resolveForecastLookupId()`.
+- **Reviews came back clean.** `sql-reviewer`: SAFE TO MERGE, 0 critical / 0 warnings. `php-reviewer`: 0 critical. `cto`: DISAGREE-WITH-ALTERNATIVE on the original Phase A3, which changed the plan (see *Decisions* #1).
+- **All four touched files pass `php -l`** and a scan for PHP 8-only syntax (1.0 runs 7.2, so local 8.5 linting is not proof on its own).
+
+---
+
+## What did NOT work — DO NOT RETRY THESE
+
+- **The plan's stated reason for Phase A3 is factually wrong. Do not delete the reconciler's INSERT branch on that basis.** The plan says the cron "fetches no line items at all (verified: zero references to `itemList` or `costEstimate`)" and therefore "structurally cannot compute `projectedProfit`". The *file* has no such reference — but the *data is already on the record it holds*. `listOpportunities()` runs a second SuiteQL query for `tl.costestimate` / `tl.foreignamount` (`library` `rest.php:717-728`) and attaches them as `itemList->item[*]` (`:794-809`). The cron just never read them. Correct profit was ~15 lines away with no extra API call.
+- **The plan's claim that `fixer.php`'s upsert also omits `projectedProfit` is wrong.** `fixer.php:541-562` computes it (self-healing missing Items from NetSuite) and writes it at `:574` in both the UPDATE and INSERT paths. So the "that's why it never converges" reasoning in the plan does not hold.
+- **Deleting the INSERT branch would have removed the only automatic repair path.** `cto` verdict: the 10-minute cron's `lastImportEpoch` checkpoint has already swept past a lost row, and the webhook only fires on a NetSuite push — so after a mass delete nothing would re-create the headers. A reconciler that can only delete is a data-loss machine.
+- **The review comment on PR 1718 asking to skip item lines with a missing `costEstimate` is wrong — do not apply it.** The authority is `checker.php:257`: `ROUND(NVL(SUM(-tl.foreignamount + NVL(tl.costestimate,0)), 0), 2)`. `costestimate` **is** wrapped in `NVL(...,0)`, so a null cost counts as zero and the line still contributes. Skipping it would put the computed profit permanently out of step with the checker. Only the **`amount`** half of that comment was valid (`foreignamount` has no `NVL`, so a null there makes the row expression NULL and `SUM` drops it) and that half is applied.
+- **`MAINTENANCE_MODE = true` never meant "dry run" in this file.** It gated `debuggingOutput()` only; the UPDATE, INSERT and DELETE all ran. The sibling `periodic_forecast_discrepancy_fix_open_orders.php:681` uses the same constant name as a genuine write-suppressing dry-run. Same name, same folder, opposite meaning — that was the trap. Now unified.
+- **A `debuggingOutput()`-only report is invisible in production.** `worker/ebs/cron.worker.php:194` runs the cron with the output redirect **commented out**, so stdout goes to the mail spool on an ephemeral EB instance. Anything that must be seen has to go to `error_log()` + `App_Error_Capture`.
+- **The CTO's claim that this cron has no `memory_limit` is wrong.** `worker/ebs/setup_phpini.php:22` sets `2G`. The five in-memory lookups are not a memory risk.
+- **`context-primer` cannot be used during `/kickoff`.** The `kickoff-gate` hook blocks it; priming must happen in the main session and be released with `knowledge.js kickoff-primed`.
+
+---
+
+## Not tried yet (candidates for next session)
+
+- **Phase 0 human-only prerequisites, all still open.** Register `taskMoved` for space `90113928591` (without it the ClickUp→NetSuite path is inert); consolidate the ClickUp token onto `_Config::clickup('token')` across 7+ files; update `CLICKUP_SERVICE_ACCOUNT_USER_ID` at `worker2/Worker/Clickup/Project.php:245` from `12633140` (Jeff's personal id) to the devteam service account `87469444`, **and** the literal comparison at `:481`; determine whether the ClickUp Automation runs as Jeff or a system user.
+- **The ClickUp webhook migration cannot be done by the devteam account.** `GET /team/8577082/webhook` under devteam returns 0 webhooks — ClickUp lists only webhooks created by the calling token's own account. So devteam **cannot enumerate or delete** the existing subscriptions. This must run with Jeff's token or by a workspace admin in the UI: list what exists, recreate each under devteam, confirm events still arrive, and only then delete the originals. Doing it in the other order silently kills the live `taskUpdated` subscription the whole reverse path depends on. The devteam token is held outside every repo at `~/.clickup-devteam-token` (mode 600) — never paste its value anywhere.
+- **The retroactive fix — not started.** Find rows already damaged by past deletes, and clear the one row stranded at `clickupTaskId = 'PENDING'`. Blocked on a production Forecast read (see *Blockers* #1).
+- **Phase 1 schema (dbchanges2).** `Forecast.OpportunityStageMap`, `Forecast.OpportunityCloseStatusMap`, `Forecast.OpportunityDocument`, an index on `clickupTaskId`, echo-guard columns, `syncEnabled`, and widening `opportunityNumber` past `varchar(16)`. Next filename is `2026-09-09a` or later — `2026-09-01a` (endCustomerName) already exists, so **do not re-add that column**. dbchanges2 has **no `_production` branch**; base is `_main`.
+- **Phase 2/3** — split the `clickupTaskId` claim so it guards creation only, add the `PENDING` reaper, move the stage map out of PHP constants into the new Forecast tables, add `taskMoved` to the opportunity route, and add the actor-identity echo guard.
+- **The live orderindex-0 reverse-map bug is still unfixed.** `worker2/Worker/Clickup/Opportunity.php:207` does `array_search($orderIndex, STAGE_OPTION_ORDERINDEX, true) ?: null`. `array_search` returns the **key**, Pre-Sales Qualification's key is `0`, and `?:` turns that into `null` — so a task in Pre-Sales Qualification never writes NetSuite stage 3. That is the most common stage in the funnel. Fix with `=== false ? null :`, and add a regression test for exactly `orderindex 0`.
+- **A regression-test harness.** Neither `worker`, `tools` nor `library` has PHPUnit (`library` has no `require-dev` at all). The plan's Verification section asks for unit tests that currently have nowhere to live. Either stand up a harness or accept dry-run proof.
+- **Whether the location-tree catch should re-throw** (see *Blockers* #2).
+- **`App_Worker::checkIn()`/`checkOut()` monitoring for this cron.** It has no monitor today, so a crash is invisible. Deliberately not added — the Sentry monitor slug `forecast-discrepancy-fix-opportunities` must be registered first or the check-in is a silent no-op (`library/app/worker.php:250-252` returns early with no SDK).
+
+---
+
+## Current file state
+
+All changes are **committed and pushed on feature branches**. Nothing is merged. Working trees are clean.
+
+| File | Status | Notes |
+|------|--------|-------|
+| `library/app/api/netsuite/rest.php` | modified, pushed | Both SuiteQL pagers now call a new `assertSuiteqlPageIsValid()` that throws on a bad page or a missing `hasMore`. Branch `fix/suiteql-fail-loud-on-partial-page` (PR #880). |
+| `worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_opportunities.php` | modified, pushed | ClickUp-link delete guard, mass-delete abort guard, real dry-run, 5 new INSERT columns, statusless skip, 3 new helper functions, 4 new lookups. Branch `fix/forecast-opportunity-delete-guard` (PR #1718). |
+| `worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php` | modified, pushed | The `listLocations()` catch now reports via `error_log()` + `App_Error_Capture` instead of `debuggingOutput()` only. Deliberately still NOT re-thrown. Same branch/PR. |
+| `tools/bin/forecast/fixer.php` | modified, pushed | ClickUp-link delete guard inside `fixOpportunities()`, behind the existing `forecastColumnExists()` probe. Branch `fix/forecast-fixer-clickup-link-guard` (PR #15). |
+| `~/.claude/plans/i-want-to-update-jolly-pebble.md` | unchanged | The 813-line source plan. **Two of its claims are now known wrong** — see *What did NOT work*. Local to Aaron's machine, not in git. |
+
+Local branch state to be aware of: `worker` and `tools` were behind `origin/_production` at the start (6 and 3 commits). All three feature branches were cut fresh from `origin/_production`, so they are current. `tools`'s old local branch `feature/compass-user-field-edits` was already merged upstream as PR #14.
+
+---
+
+## Decisions made
+
+1. **Keep the reconciler's INSERT and complete it, instead of deleting it (`cto` option D).** Rationale: the premise for deleting it was wrong (the line data is already on the record), and the INSERT is the only automatic path that re-creates a row lost outside a NetSuite edit. Rejected: delete-the-INSERT-and-report (the plan's Phase A3, decided 2026-08-27); populate only the 4 header columns and leave profit at 0; add an "incomplete" flag column.
+2. **Fix `suiteqlListAll()` in shared `library` core, not just guard the one cron.** Aaron's call, against the `cto`'s advice to contain it locally. Rationale: a swallowed failure in shared code is a trap for all 13 calling files, not just the one that tripped it. Blast radius was surveyed first — both pagers are `private static` with zero callers outside `rest.php`.
+3. **Require `hasMore`, not just `items`, before accepting a page.** Rationale: the loop ends on `while (!empty($page->hasMore))`, so a response with rows but no flag was silently treated as the last page. `isset()` accepts `hasMore = false`, so normal and empty responses are unaffected. **Residual risk:** could not be checked against a live NetSuite response (connector unauthorized in session) — if any endpoint omits `hasMore`, all 13 callers now throw. Watch this on first deploy.
+4. **`MAINTENANCE_MODE` now means verbose AND dry-run, matching the sibling cron.** Rationale: kills the same-name-opposite-meaning trap between two files in one folder, and makes the fix testable. Rejected: renaming the constant (reaches `debuggingOutput()` for no benefit); adding a separate flag (leaves the trap in place). **Side effect to know:** non-prod (`test_mode = 1`) now dry-runs, so alpha/beta/demo stop writing Forecast from this cron. Production behaviour is unchanged — same writes, just quieter.
+5. **Delete guard placed at the delete loop, not at candidate-build time.** Rationale: reporting stays actionable (only rows NetSuite actually dropped, not all 746 linked rows every night) and the guard sits closest to the `DELETE`. This deviates from the plan's stated instruction.
+6. **A null `amount` skips the profit line; a null `costEstimate` does not.** Rationale: mirrors the `NVL` asymmetry in `checker.php:257` exactly, so the computed value matches what it is compared against. Written into the docblock with the file:line so nobody reverses it.
+7. **The location-tree catch is made visible but NOT re-thrown.** Rationale: the file is required by five import crons, so aborting them on a location fetch failure is a production behaviour change that needs its own decision. Open question for Jeff.
+8. **NetSuite → ClickUp attachment upload needs NO `_ApiRequest` change** (`/cto` verdict 2026-09-04, recorded in the plan). `_ApiRequest` already does multipart: the request switch has no `default:` case, so `payloadEncoding: null` passes the payload through untouched and cURL builds the boundary itself. Do it app-level in worker2 with `new _ApiRequest('POST', $url, ['attachment' => new CURLFile(...)], null)`. Three reasons the original plan was wrong: the blast radius was understated 7× (97 `new _ApiRequest` sites across 35 files, not ~13); `$payloadEncoding` drives both request encoding **and** response decoding, which a multipart-request/JSON-response call cannot express in one field; and `CURLFile` streams from disk so memory never holds the file. **The one core change worth making** is a one-line guard at `ApiRequest.php:268`, where `$log->requestPayload = $this->requestPayload` hands an array-containing-`CURLFile` to `_Database::escape()` — a **fatal**, latent today for all 97 callers.
+9. **Reviews run as subagents, maker ≠ checker.** `php-reviewer` + `sql-reviewer` on every pass; `cto` before the hard-to-reverse call. All findings were verified against source before applying — two review comments turned out to be wrong and were refused with evidence.
+
+---
+
+## Blockers
+
+1. **Production Forecast read not authorized.** The `746 of 26,766` exposure figure comes from 2026-08-27 and is stale. Needed before the retroactive fix can be scoped, and before that number goes in a PR description. Aaron was asked twice and has not answered.
+2. **The location-tree catch decision is open.** Should `common_import_sales_from_netsuite.php`'s `listLocations()` catch re-throw (aborting all five import crons on a location fetch failure) or stay visible-but-non-fatal as it is now?
+3. **NetSuite and ClickUp connectors are unauthorized in-session.** So `custbody_sales_stage`'s field id, the `entityStatus` internalIds, the removal of Closed Won/Lost from the Sales Stage list, and the live webhook subscriptions could not be verified from source — they come from code comments and the Forecast mirror. Confirm against one live opportunity record before seeding the stage map.
+4. **The ClickUp webhook migration needs Jeff's token or a workspace admin.** The devteam account cannot see or delete webhooks it did not create. Hard prerequisite for Phase 3, and only Jeff can do it.
+5. **`webhook` and `test` repos are not checked out** on Aaron's machine. Phase 5's backfill precedent lives at `test/@dave/clickup/backfill_opportunity_numbers.php`.
+6. **No test harness in any of the three 1.0 repos**, so the plan's unit-test verification has nowhere to live.
+
+---
+
+## Exact next step
+
+> Get **library#880** reviewed and merged, then run the rollout in the order written in worker#1718's body: baseline `php tools/bin/forecast/checker.php --category opportunities` (read-only), set `"active": 0` at `worker/schedules/cron.worker.infrastructure.json:63` and deploy, merge and deploy library#880 then worker#1718, then run
+> `php /var/www/html/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_opportunities.php --dry-run`
+> on the infrastructure worker and check four things: the NetSuite record count is **not zero**, the delete set is small, `held back, ClickUp-linked` looks right against the ~746, and `INSERT ... profit=` is non-zero on a sample.
+
+---
+_Saved by /session-save on 2026-09-09_

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.786",
+  "version": "1.0.787",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",