toga-ai 1.0.708 → 1.0.709

package/knowledge/2.0/apps/worker2/INDEX.md

@@ -17,7 +17,7 @@
 | [ClickUp Task Description Fluffer (Talos KB Investigation)](features/clickup-task-fluffer-talos.md) | `_Worker_Clickup_Fluffer` "fluffs" a ClickUp task by sending its name + existing description to the **Talos `dev-core` KB-investigation agent** and writing the | worker2/Worker/Clickup/Fluffer.php, worker2/Worker/Clickup.php, worker2/Config/production.ini, worker2/Config/beta.ini, worker2/Config/dev-rohan-mac.ini |
 | [ClickUp Work Type Automation (Committed / Conditional / Stretch)](features/clickup-work-type-automation.md) | The ClickUp webhook handler (`_Worker_Clickup`) automatically maintains each task's **Work Type** custom field — `Committed`, `Conditional`, or `Stretch` — base | worker2/Worker/Clickup.php, worker2/Tests/Worker/ClickupWorkTypeTest.php |
 | [Compass VIP Support Importer (worker2)](features/compass-vip-support-importer.md) | A worker2 action that ingests Compass's quarterly VIP spreadsheet and assigns each VIP user's support technician by setting `Users.c_supportedByUserId` in `Clie | worker2/Worker/Client/Compass/VipSupport.php |
-| [Creating Worker Actions](features/creating-worker-actions.md) | How to add a new callable Worker action — a PHP class whose `public static` methods are invoked as background jobs (via webhook, cron, or `_Worker::runTask()`). | worker2/Worker/, worker2/Controller/Index.php, _underscore/Worker.php |
+| [Creating Worker Actions](features/creating-worker-actions.md) | How to add a new callable Worker action — a PHP class whose `public static` methods are invoked as background jobs (via webhook, cron, or `_Worker::runTask()`). | worker2/Worker/, worker2/Controller/Index.php, worker2/LambdaFunctions/JobScheduler/lambda_function.py, _underscore/Worker.php |
 | [Cross-account AWS access for worker2 crons (_Component_Aws_Workloads)](features/cross-account-aws-access.md) | `_Component_Aws_Workloads` is the **standard, and only sanctioned, way any new worker2 cron obtains AWS access** — for any account, any region, any AWS SDK clie | worker2/Component/Aws/Workloads/Workloads.php, worker2/Config/production.ini, worker2/Worker/Infrastructure/CloudWatch.php, worker2/Worker/Monitor/Fleet.php, library/app/worker.php, library/app/cloud.php |
 | [Elastic Beanstalk health monitor → OneUptime (ElasticBeanstalkHealth)](features/elastic-beanstalk-health-monitor.md) | `_Worker_Infrastructure_CloudWatch::ElasticBeanstalkHealth()` is a worker2 cron that reads each Elastic Beanstalk (EB) environment's health and reports it to a | worker2/Worker/Infrastructure/CloudWatch.php |
 | [Elite Freshservice Sync (worker2)](features/elite-freshservice-sync.md) | `_Worker_Elite` processes Freshservice webhook events and syncs them into TOGA 2. | worker2/Worker/Elite.php, worker2/Config/dev-kmaramreddy-laptop.ini |

package/knowledge/2.0/apps/worker2/architecture.md

@@ -6,8 +6,8 @@ project: Worker
 client: shared
 type: architecture
 status: active
-updated: 2026-07-29
-owners: [jcardinal]
+updated: 2026-08-31
+owners: [jcardinal, ajean]
 files:
   - worker2/Controller/Index.php
   - worker2/Worker/
@@ -62,7 +62,9 @@ HTTP-triggered job endpoints must enforce auth.
 `Core.WorkerJobs`, so never treat a queue message as the record of a job. Every worker/webhook
 endpoint must return **HTTP 200 on every path, including failure** (failures are recorded in
 `WorkerJobs`): there is no DLQ and the visibility timeout is 3600s, so a single 500 becomes a
-poison-message storm. Workers are `abstract class _Worker_<Name>` with `public static` entry
+poison-message storm — but 3600s is **delivery**, not an execution budget: the watchdog fails any
+job past `maxExecutionTime`, and that override only applies to CronScheduler-fired jobs, so every
+webhook / `runTask` / curl action is hard-capped at **300s** and anything longer must self-chunk. Workers are `abstract class _Worker_<Name>` with `public static` entry
 methods and are invoked **only** through `_Worker::runTask(action, parameters)` — never call a
 worker method directly. This repo is PHP and Python only: never add a `.sql` file here, as all
 schema changes and cron seeds belong to `dbchanges2`. Both environments run **PHP 8.5**, and the
@@ -127,8 +129,18 @@ fields on other tables: the `Email` table (`Worker/Infrastructure/Email/Send.php
 2. **PayloadRecovery** — every minute, recovers S3 fallback payloads into `WorkerJobs`.
 3. **JobScheduler** — every minute. **Part A:** picks up `dtQueued IS NULL AND isSuccess
    IS NULL` (≤100), sets `dtQueued=NOW()`, sends to queue. **Part B (watchdog):** marks
-   jobs `isSuccess=0` when `NOW() > dtStarted + timeout` (CronJobs.maxExecutionTime, else
-   env `MAX_EXECUTION_TIME_WORKER_ACTION` default 300).
+   jobs `isSuccess=0` when `NOW() > dtStarted + timeout` with `dtCompleted IS NULL`.
+   ⚠ **The `CronJobs.maxExecutionTime` override is reachable ONLY for scheduler-fired jobs.**
+   Part B resolves the timeout through `LEFT JOIN CronJobs cj ON wj.cronJobId = cj.id`, and
+   `cronJobId` is NULL on every job the CronScheduler did not fire — webhooks, `_Worker::runTask`
+   enqueues, direct `curl`. Those are **all** capped at env `MAX_EXECUTION_TIME_WORKER_ACTION`
+   (default 300s) with **no per-job override**; inserting a `CronJobs` row with a larger
+   `maxExecutionTime` changes nothing unless the scheduler is what fires the job. And the watchdog
+   only writes to the row — **it does not stop the PHP process**, so an over-budget action keeps
+   running past a false failure row and typically loses its result `UPDATE` to MySQL 2006 "server
+   has gone away" (prod job 1075223, 2026-08-28: failed at 326s, worked to 818s, report lost).
+   Long work must self-chunk — see
+   [creating-worker-actions](./features/creating-worker-actions.md#self-chunking-the-pattern-for-work-that-cannot-finish-in-one-job).
 4. **CronScheduler** — every minute, evaluates active `CronJobs.schedule` against the
    current Central-tz minute (`croniter`); on match INSERTs a `CRON` job and queues it.
 

package/knowledge/2.0/apps/worker2/features/clickup-project-routing.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-06-23
+updated: 2026-08-31
 owners: [jcardinal, ajean]
 files:
   - worker2/Worker/Clickup/Project.php
@@ -14,6 +14,7 @@ files:
 related:
   - ../architecture.md
   - ./creating-worker-actions.md
+  - ./netsuite-opportunity-sync.md
   - ./clickup-connectivity-watchdog.md
   - ./clickup-subtask-activity.md
 ---
@@ -148,6 +149,17 @@ None — uniform across all clients (shared internal automation for Agilant's Cl
 - **POST failures are fatal, DELETE failures are tolerated** — a failed ADD leaves the task
   out of its correct list (surfaced as job failure); a failed DELETE (most likely TASK_035,
   which shouldn't occur on secondary lists) is logged with `[ClickupProject]` and skipped.
+- **The unowned side of the hybrid split bites triage: home-list moves happen that no code explains.**
+  Because the worker provably cannot move a home list (TASK_035), any observed home-list change is
+  either a ClickUp native **Automation** or a **human** — and neither is visible in our logs, nor
+  reliably readable through the API. Live case 2026-08-31: opportunity tasks created by the NetSuite
+  sync into `901111987449` ("Presales Qualification") were found homed in `901111987451`
+  ("Quote/Proposal/SOW/MSA") in the same folder `90117060687` ("Opportunities"), space `90113928591`
+  — and the movement was **not cleanly stage-driven** (two tasks created the same day at the same
+  Stage 25%, only one moved), so the mover is **unidentified**. Practical rule: **never assess a
+  pipeline's health by the contents of one list**; assess by task id or by the whole folder. Detail
+  and evidence in
+  [NetSuite ↔ ClickUp Opportunity Sync](./netsuite-opportunity-sync.md#triage-is-the-opportunity-sync-working-workerjobsoutput-verdicts).
 - **Webhook subscriptions** must cover each space, all → `webhook.togahub.com/clickup`. Routing:
   legacy sprint space `90020178491`, Project Hub `90113939341`, Lifecycle Services
   `90114156087`; Opportunity Hub via `RegisterOpportunityWebhook()`. The hub guard additionally
@@ -164,6 +176,11 @@ Read-only / setup actions, invoked via `curl -X POST https://worker.togahub.com/
 `RESULT: PASS`), `RegisterOpportunityWebhook` and `RegisterHubGuardWebhooks` (one-time setup).
 
 ## Change history
+- 2026-08-31 — Recorded the triage consequence of the hybrid split (read-only investigation, no code
+  changed): worker-created opportunity tasks were observed **homed in `901111987451`
+  "Quote/Proposal/SOW/MSA"** rather than the `901111987449` list they were POSTed to, with the mover
+  **unidentified** (not cleanly stage-driven — ClickUp Automation or a human). Rule added: judge a
+  pipeline's health by task id or folder, never by one list. (ajean)
 - 2026-07-14 — Correction: `TASK_TYPE_EPIC_ID` / `TASK_TYPE_OPPORTUNITY_ID` are still `null` and the Task-Types ClickApp is not reliably enabled on the hub spaces, so `enforceHubPlacement`'s custom-Task-Type keying is inert; structural (space + top-level) identification is what actually runs. Cross-linked the new subtask-activity feature (same hub spaces/pipeline). (ajean)
 - 2026-06-23 — Added the hub-placement guard (`enforceHubPlacement` on `taskCreated`/`taskMoved`): re-anchors misplaced Epics/Opportunities to their hub by custom Task Type + home space, backfills Business Unit/Project Category, and notifies. Added `DiscoverTaskTypes`, `RegisterHubGuardWebhooks`, `TestHubGuardMappings`. Corrected the earlier "Task Types not enabled" note. (ajean)
 - 2026-06-09 — Documented ClickUp secondary multi-list routing (hybrid native-automation + worker design, three space handlers, self-trigger guard). (jcardinal)

package/knowledge/2.0/apps/worker2/features/creating-worker-actions.md

@@ -6,14 +6,16 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-26
+updated: 2026-08-31
 owners: [jcardinal, dfranks, mhammontree, tcox, bala, ajean]
 files:
   - worker2/Worker/
   - worker2/Controller/Index.php
+  - worker2/LambdaFunctions/JobScheduler/lambda_function.py
   - _underscore/Worker.php
 related:
   - ../architecture.md
+  - ./netsuite-opportunity-client-labels.md
   - ../../api2/features/api-payload-interceptors.md
 ---
 
@@ -203,6 +205,103 @@ So **do not design "throw so it retries" — it won't.** To actually retry / sel
 Rule of thumb: `throw` to surface a genuine failure; `runTask`/in-process loop when the work should
 be reattempted.
 
+## ⚠ The real time budget is 300s — and for a hand-triggered action it CANNOT be raised
+
+The 3600s SQS visibility timeout is **not** your budget, and the worker2 docs previously implied it
+was. The **JobScheduler watchdog** (`LambdaFunctions/JobScheduler/lambda_function.py`, Part B) fails
+any job whose `dtStarted` is older than its timeout while `dtCompleted IS NULL` — and it resolves
+that timeout from `CronJobs.maxExecutionTime` **only through
+`LEFT JOIN CronJobs cj ON wj.cronJobId = cj.id`**.
+
+`cronJobId` is **NULL for anything the CronScheduler did not fire** — every webhook job, every
+`_Worker::runTask()` enqueue, every `curl` to the worker. All of those fall back to env
+`MAX_EXECUTION_TIME_WORKER_ACTION`, **default 300s**, and there is **no per-job override**: adding a
+`Core.CronJobs` row with a large `maxExecutionTime` buys you nothing unless the **scheduler** is the
+thing that fires the job (only then is `cronJobId` populated). "I gave the job an hour" is not
+something you can do for an ad-hoc action.
+
+Worse, the watchdog only writes to the row — **the PHP process keeps running**. Measured in
+production (2026-08-28, job `1075223`, a full `OpportunityStakeholderBackfill` dry run):
+
+| t | what happened |
+|---|---|
+| 326s | watchdog stamps `isSuccess=0`, `output = 'watchdog: exceeded maxExecutionTime without completing'` |
+| 326–818s | the action carries on working, unaware it has been declared dead |
+| 818s | its own result `UPDATE` dies on MySQL **2006 "server has gone away"** (`Logs.Issue` ref 6E) — the entire report is lost; HTTP returned 500 carrying only `Error reference: 6E-190` |
+
+So an over-budget action loses its output **twice**: once to a false failure row, once to a dropped
+connection. Design for 300s.
+
+### Self-chunking: the pattern for work that cannot finish in one job
+
+An action that may exceed the budget works for a **sub-budget slice**, then **re-enqueues itself**
+via `_Worker::runTask()` carrying its cursor and running totals in a `resume` parameter. One
+`WorkerJobs` row per chunk, each labelled so a mid-run row is never misread as the final answer
+(`'chunk 4, run CONTINUES'` vs `'run COMPLETE'`). Proven by
+`Worker/Netsuite/OpportunityStakeholderBackfill.php` (`CHUNK_TIME_BUDGET_SECONDS = 180`; live passes
+of 5–11 chunks, 15–32 minutes, 0 failures) — see
+[netsuite-opportunity-client-labels](./netsuite-opportunity-client-labels.md).
+
+```php
+const CHUNK_TIME_BUDGET_SECONDS = 180;
+
+public static function Backfill(array|object $resume = []): string {
+    $startedAt = microtime(true);              // TOP OF THE JOB, not top of the loop
+    $resume    = (array) json_decode(json_encode($resume), true);
+    $cursor    = (int) ($resume['cursor'] ?? 0);
+    $totals    = $resume['totals'] ?? [];
+
+    $rows = self::fetchBatch($cursor);         // keyset: WHERE o.id > $cursor ORDER BY o.id ASC
+    _Database::transactionCommit(self::SOURCE_DB);   // commit the read BEFORE any remote call
+
+    if (!$rows) return self::report($totals, complete: true);
+
+    $processed = 0;
+    foreach ($rows as $row) {
+        // forward-progress invariant: always do at least one before honouring the deadline
+        if ($processed > 0 && (microtime(true) - $startedAt) >= self::CHUNK_TIME_BUDGET_SECONDS) break;
+        self::processOne($row, $totals);
+        $cursor = (int) $row->id;
+        $processed++;
+    }
+
+    _Worker::runTask(self::ACTION, ['resume' => ['cursor' => $cursor, 'totals' => $totals]]);
+    return self::report($totals, complete: false);
+}
+```
+
+Five design points, each of which was a real bug or near-miss:
+
+- **Measure the deadline from the top of the JOB, not the top of the loop.** Startup work (option-list
+  fetch, alias load) then sits *inside* the budget instead of being added on top of it.
+- **Use a KEYSET cursor (`o.id > $cursor` + an explicit `ORDER BY o.id ASC`), never `OFFSET`.** Rows
+  enter the working set continuously, so an `OFFSET` silently skips or repeats rows when a new one
+  lands mid-run. A query with **no `ORDER BY`** is harmless when read once and *meaningless* across
+  jobs — chunking a paginated query is exactly when a missing `ORDER BY` starts corrupting results.
+- **Enforce a forward-progress invariant: every chunk must advance the cursor by ≥1 unit of work.**
+  Without it, a chunk whose budget is already blown at loop start processes nothing, leaves the
+  cursor unchanged, and re-enqueues an identical job — an **unbounded chain of production jobs**.
+- **Bound the carried detail by entry count AND per-entry length.** A count cap alone does not bound
+  bytes: failure entries embed API exception messages. Truncate by walking the tail back until
+  `json_encode()` succeeds — a severed multi-byte character makes `json_encode()` return `false` for
+  the *whole* message, and `runTask` would then hand SQS an empty body.
+- **Commit the source-DB read as soon as the rows are in hand** — before any remote call and before
+  the early returns. Every run ends on an empty-batch hop, which otherwise leaves a lazy transaction
+  open (see [_underscore architecture](../../_underscore/architecture.md)).
+
+### Do NOT retry the chunk hand-off — a failed hand-off must break the chain
+
+`_Worker::runTask()` **mints the idempotency uuid itself, per call** (`_underscore/Worker.php`), and
+that uuid is the only key the dispatcher's dedupe guard matches on. So a send that reaches SQS and
+then throws on the way back would be retried under a **different** uuid, the guard would not match,
+and **both** messages run: the chain **forks** into two chains from one cursor, each spawning its own
+successors — double downstream load and two disagreeing `COMPLETE` reports.
+
+A broken chain is strictly safer: it is loud, it reports the restart cursor, and it costs one manual
+restart. **Single attempt only, deliberately.** Making a retry safe requires threading a stable
+per-cursor uuid through `runTask` *and* the `Controller/Index.php` dedupe — shared framework core,
+cross-repo, and out of scope for a caller. Do not "fix" this back to a retry loop.
+
 ## `debug_mode` changes how `runTask` dispatches — and the debug path can't do read-after-write
 
 `_Worker::runTask` has two completely different behaviours:
@@ -280,6 +379,18 @@ Rules: rename **both sides in one PR and one deploy**; grep every repo for the o
   to PHP arrays fine — only JSON **objects** become `stdClass`.) This bit
   `_Worker_Infrastructure_CloudWatch::ElasticBeanstalkHealth`'s `environments` region-map in
   production — see [cross-account-aws-access.md](./cross-account-aws-access.md).
+  **This applies to BOTH dispatch paths, so no invocation style escapes it:** the stored-row path
+  (`Controller/Index.php` ~L318) and the direct-payload/`runTask` path (~L455) each do a shallow
+  `(array) json_decode($job->parameters)`. So a structured parameter breaks on the **first**
+  dispatch whether it came from a cron row, a `curl`, or a `runTask` re-enqueue.
+  It is also **asymmetric on the return trip**: an associative array you send comes back
+  `stdClass`, while a PHP **list** comes back a plain array — so a shallow `(array)` cast is not
+  enough for anything nested. Verified on PHP 8.5: `null`, `array` and `object` all `TypeError`
+  against a userland `string`/`array` parameter, while **ints coerce silently** (so a wrong-typed
+  numeric fails *later*, not at dispatch). **Fix pattern:** declare `array|object` and normalize
+  with a full round trip — `(array) json_decode(json_encode($param), true)` — which flattens every
+  level at once. Any action taking a structured parameter (a `resume` cursor, a region map, an
+  options bag) needs this.
 - Webhook headers are **lowercased** by API Gateway (`X-GitHub-Event` → `x-github-event`).
 - If the action needs a client DB, register it in `initialize()` — it runs before the method.
 - **Worker actions do NOT auto-register `DB_CLIENT`.** The dispatcher registers only
@@ -324,6 +435,21 @@ Rules: rename **both sides in one PR and one deploy**; grep every repo for the o
   commit-before-SQS transaction pattern that the worker relies on.
 
 ## Change history
+- 2026-08-31 — Documented that **the real time budget for any hand-triggered action is 300s and
+  cannot be raised**: the JobScheduler watchdog only honours `CronJobs.maxExecutionTime` through
+  `LEFT JOIN … ON wj.cronJobId = cj.id`, and `cronJobId` is NULL for every webhook / `runTask` /
+  `curl` job, so they all fall back to `MAX_EXECUTION_TIME_WORKER_ACTION` (300s) — the 3600s SQS
+  visibility timeout is not the limit. Recorded that the watchdog does **not** stop the process
+  (prod job 1075223: failed at 326s, kept working to 818s, then lost its whole report to MySQL 2006
+  on its own result UPDATE). Added the **self-chunking pattern** for over-budget work (re-enqueue
+  self with a `resume` cursor; deadline from the top of the job, keyset cursor never OFFSET,
+  forward-progress invariant, detail bounded by count *and* bytes, commit the source read before any
+  remote call) and the decision **not to retry the chunk hand-off** — `runTask` mints a fresh
+  idempotency uuid per call, so a retry forks the chain instead of deduping. Extended the
+  nested-parameter gotcha: both dispatch paths cast only the top level, the return trip is
+  asymmetric (assoc → `stdClass`, list → array), ints coerce silently, and the fix is a
+  `json_encode`/`json_decode(true)` round trip. Surfaced making
+  `Netsuite/OpportunityStakeholderBackfill` reportable (PR #157). (ajean)
 - 2026-08-26 — Added two operational gotchas: a new action **does not run on a schedule until its
   `Core.CronJobs` row exists** (schedule is data, not code), and the practice of building a
   **dry-run entry point for a rare-gated path** rather than waiting for it to fire — one found a

package/knowledge/2.0/apps/worker2/features/netsuite-opportunity-client-labels.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-26
+updated: 2026-08-31
 owners: ["ajean"]
 files:
   - worker2/Worker/Netsuite/Opportunity.php
@@ -97,9 +97,67 @@ the only way to populate them.
   opportunities affected, and nothing would have surfaced it. Fuzzy matching is opt-in only.
 - **Pacing is 700 ms per REQUEST, not per task** (~85/min against ClickUp's 100/min/token). A task
   costs a variable 1–3 requests, so per-task pacing cannot bound the request rate.
+- **It self-chunks, because a full pass cannot finish inside the 300s watchdog.** The action works
+  for `CHUNK_TIME_BUDGET_SECONDS` (180s) and then re-enqueues **itself** via `_Worker::runTask`,
+  carrying a keyset cursor and running totals in a `resume` parameter — one `WorkerJobs` row per
+  chunk, each labelled `chunk N, run CONTINUES` or `run COMPLETE`. Before this, a full pass was
+  stamped failed by the watchdog at 326s, kept working to 818s and then **lost its entire report**
+  to a MySQL 2006 on its own result UPDATE. The reusable pattern (and why the chunk hand-off is
+  deliberately **not** retried) is in
+  [creating-worker-actions](./creating-worker-actions.md#self-chunking-the-pattern-for-work-that-cannot-finish-in-one-job).
+
+### Operational baseline (what a real pass looks like)
+
+~769–771 task-bearing opportunities → **1,542 field slots** (2 per opportunity). Three production
+passes:
+
+| Pass | Date | Chunks / wall time | Filled | Notes |
+|---|---|---|---|---|
+| 1 | 2026-08-28 | 11 chunks, 32m | 1,112 | first live pass, 0 failures |
+| 2 | 2026-08-31 | 5 chunks, 15m | 39 | after 21 new ClickUp options; 1,143 already set |
+| 3 | 2026-08-31 | 6 chunks, ~17m | 191 | after 100 more options; unmatched 194 → 2, coverage **89.0%** |
+
+Every pass: **0 failures**, and totals always reconcile to `examined × 2`.
+
+- **Chunk throughput depends on mode, sharply:** ~**170** opportunities per chunk read-only vs ~**68**
+  when write-heavy, because each filled field adds a paced request. Size expectations by mode, not by
+  the last run.
+- **Dry run and live agreed EXACTLY on all five counters** (1112 / 29 / 231 / 166 / 0). That equality
+  is a cheap, high-value integrity check — repeat it before every live pass.
 
 ## Gotchas / known issues
 
+- **⚠ THE FUZZY BLIND SPOT — fuzzy-above-floor names are invisible to the digest AND refused by the
+  backfill, so those gaps have no work queue and never fill.** The digest's `collectUnmatched()`
+  skips anything with a non-null option id
+  (`if ($outcome['id'] !== null || !in_array($outcome['reason'], ACTIONABLE_REASONS)) continue;`) —
+  and a **fuzzy hit HAS an id**, so a fuzzy name never reaches the PM email and therefore never gets
+  written to `Team.ClickUpLabelAliases`. Meanwhile the backfill at `exactOnly:true` **refuses** fuzzy.
+  Net effect measured in production: the digest reported **3 outstanding issues while 169 field slots
+  were actually empty** — 167 of them fuzzy skips. Coverage plateaus around 89% and nothing surfaces
+  the gap. Consequence for whoever is clearing the queue: **assigning every alias in the digest does
+  not clear the fuzzy set**, because those names were never queued — verified that
+  `Lee County Sheriff's Office`, `Polk County Sheriff's Office` and `Plant City Police` are all
+  absent from `ClickUpLabelAliases`. **Suggested fix (not implemented):** the digest should report
+  fuzzy-above-floor as its own actionable reason.
+- **Adding ClickUp options creates NEW fuzzy near-misses — the fuzzy count can RISE as gaps are
+  filled.** Adding ~121 options moved names out of *unmatched*, but some landed in *fuzzy, skipped*
+  rather than *filled*: a customer that previously had **no** candidate now has one scoring 85–95%.
+  Observed unmatched **231 → 194 → 2** while fuzzy went **166 → 166 → 167**. So "add more options" is
+  not a complete remedy; past a point the remaining work is **aliases, not options**.
+- **An alias is Pass 0 — ahead of exact AND fuzzy — so a WRONG alias lands on tasks, and
+  seed-if-empty makes it STICKY.** `resolveLabelOption()` (`Opportunity.php` ~L1298) returns
+  confidence `alias` and deliberately bypasses `exactOnly`, which is correct (a human judgement
+  outranks machine guessing) but means a mis-entered alias **will** be written. And because the
+  backfill only fills **empty** fields, no later run corrects it — someone must clear the field in
+  ClickUp by hand. **Operational check worth repeating before every live pass:** audit
+  `ClickUpLabelAliases` for mappings whose **first word differs** from the source name. Doing that on
+  338 hand-entered aliases surfaced `Hewlett Packard Enterprise → HP` sitting alongside
+  `Hewlett-Packard Enterprise-Health and Life Sciences Team → HPE` — HP Inc. and HPE have been
+  separate companies since 2015.
+- **The run report lists only the first 50 fuzzy entries.** With 167 fuzzy skips, the other 117 are
+  unreviewable *anywhere* — a real reporting gap, and the reason the production fuzzy set had to be
+  counted by query rather than read off the report.
 - **`splitCustomerRefName()` must handle composite sub-customer numbers.** The original pattern
   `/^(\d+)\s+(.*)$/` (digits then whitespace) could not parse `"6096:6308 Compass Group"` — the colon
   broke the match, the number stayed in the name, and normalization produced
@@ -124,6 +182,18 @@ the only way to populate them.
   injection before they go into the digest.
 
 ## Change history
+- 2026-08-31 — **Made the backfill self-chunking and ran three live passes (PR #157, merged).** The
+  action now works 180s and re-enqueues itself with a keyset `resume` cursor, one `WorkerJobs` row per
+  chunk labelled CONTINUES/COMPLETE — previously a full pass was declared failed by the 300s watchdog
+  at 326s, kept working to 818s and lost its whole report to MySQL 2006 (pattern now documented in
+  [creating-worker-actions](./creating-worker-actions.md)). Recorded the operational baseline (1,542
+  slots; passes filled 1,112 / 39 / 191 to 89.0% coverage, 0 failures, dry-run and live agreeing
+  exactly on all five counters) and four gaps found while using it: the **fuzzy blind spot** (fuzzy
+  hits have an option id so the digest skips them *and* `exactOnly` refuses them — 3 reported issues
+  vs 169 genuinely empty slots, 167 fuzzy), that **adding options can raise the fuzzy count**
+  (unmatched 231→2 while fuzzy 166→167), that an **alias is Pass 0 and seed-if-empty makes a wrong
+  alias sticky** (audit first-word mismatches — caught `Hewlett Packard Enterprise → HP` vs HPE), and
+  that the run report **truncates the fuzzy list at 50 entries**. (ajean)
 - 2026-08-26 — **Built the client vs end-customer split (TRUE-81010, PR #142, merged).** Stakeholders
   now receives the consolidated parent (the client, NS `custentity10`) and a new **End Customer**
   field receives the opportunity `entity`; resolution is local via

package/knowledge/2.0/apps/worker2/features/netsuite-opportunity-sync.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-27
+updated: 2026-08-31
 owners: ["dfranks", "kyalamarthi", "ajean"]
 files:
   - worker2/Worker/Netsuite.php
@@ -31,7 +31,10 @@ files:
 related:
   - ./netsuite-opportunity-client-labels.md
   - ./netsuite-salesorder-open-orders-sync.md
+  - ./clickup-project-routing.md
   - ../architecture.md
+  - ../../../../1.0/apps/webhook/features/netsuite-clickup-opportunity-sync.md
+  - ../../../../standalone/apps/claude/workflows/mcp-tool-usage.md
 ---
 
 ## Summary
@@ -119,6 +122,23 @@ the "CU→NS direction" section.
    (text = string compare; Presales Lead users field = add-only, matches when already assigned) and
    `normalizeDescription()`.
 3. **No match → `createTask()`**: POST to list `901111987449`, `custom_item_id` = `1009`.
+   **That is only the task's list at BIRTH** — something outside the worker moves new opportunity
+   tasks to a different **home** list, which makes a healthy sync look dead. See
+   *⚠ A healthy sync reads as dead* under Triage below before concluding tasks aren't being created.
+
+**The Presales-Lead gate DELAYS a task; it does not permanently lose one.** The natural (and wrong)
+reading is *"no lead at creation time ⇒ that opportunity never gets a task."* Because the gate runs
+on **both** `post()` and `put()`, and the claim stays open while
+`Forecast.Opportunities.clickupTaskId` is still NULL, whichever later edit webhook first sees a
+non-empty Presales Lead wins the claim and creates the task **then** — however old the opportunity
+is. Verified live 2026-08-31: NS internalId **6682331** (opportunity **71113**, "6519 Hydro Resources
+— AI Training", `Forecast.Opportunities.id` 70133 — an *old* opportunity) received task
+`868kyqxq2` from a `Netsuite/Opportunity/put`, verdict
+`task created (stakeholder "Hydro Resources"; end customer "Hydro Resources")`. That is a **first**
+claim on a never-claimed opportunity, so it does **not** contradict the rule that once
+`clickupTaskId` holds a real id every later webhook returns `skipped (already claimed)` (see
+*Consequence: `updateTask()` is unreachable* below). So when triaging a backlog of
+`skipped (no presales lead)`, read it as **"queued, awaiting a lead"**, not "lost".
 
 Shared helpers: `buildCustomFields()` (the field array, used by both create and update),
 `resolvePresalesUserId()` (cached `Client_True.Users.c_clickupUserIdentifier` lookup → else a
@@ -226,6 +246,84 @@ reaches **none** of the 761 existing tasks — only a separate backfill action r
 says nothing about the claim interaction, so reading the constant alone gives you the wrong model of
 what enabling it would do.
 
+## Triage: is the opportunity sync working? (`WorkerJobs.output` verdicts)
+
+**Start here** whenever anyone reports *"no new opportunity tasks in ClickUp."* One query answers it
+in seconds — and, importantly, **with no NetSuite or ClickUp access at all.** That matters: the
+NetSuite MCP connector needs interactive OAuth and is routinely unavailable in a session (it was on
+2026-08-31), which blocks the two NetSuite-side checks this doc otherwise sends you to — the
+`scriptdeployment` status/audience query and the API Message Queue
+`custrecordcustrecord_amq_errormsg` inspection. The verdict census below is a genuine **end-to-end
+substitute** for "is it working".
+
+Every `Netsuite/Opportunity/post|put` job writes a JSON result into `Core.WorkerJobs.output`:
+
+```json
+{"db":"created|updated opportunity <internalId> (id=<forecastId>, items=<n>)","clickup":"<verdict>"}
+```
+
+**In practice there are only three `clickup` verdicts.** Census over 2026-08-20 → 2026-08-31
+(1,218 jobs, prod):
+
+| Verdict | Count | Share |
+|---|---|---|
+| `skipped (no presales lead)` | 1,047 | 86% |
+| `skipped (already claimed)` | 150 | 12% |
+| `task created` (optionally + the stakeholder/end-customer label suffix) | 21 | 1.7% |
+
+- **The healthy baseline is ~2% of opportunity events producing a ClickUp task.** A **high
+  `no presales lead` share is NOT a fault** — it is the normal shape of the traffic, and those are
+  queued rather than lost (see the gate-delay note under *ClickUp opportunity task*). **Zero tasks
+  created on a business day** is the signal worth chasing.
+- Zero error verdicts and zero `(label lookup failed)` in that window — independent confirmation
+  that the 2026-08-26 `logged()` NOT-NULL-column fix (PRs #150 + #151) is holding in production.
+- **Query gotcha:** the `toga-db` MCP validator rejects the literal keyword `UPDATE` **even inside a
+  string literal or a column alias**, so match the `skipped update (update disabled)` verdict with
+  `LIKE '%disabled%'` and alias the column `skippedDisabled`, never `updateDisabled`. See
+  [MCP Tool Usage](../../../../standalone/apps/claude/workflows/mcp-tool-usage.md).
+
+### ⚠ A healthy sync reads as dead: new tasks do NOT stay in "Presales Qualification"
+
+**This is the first thing to check when someone says "no opportunity tasks have been created."**
+`maybeCreateClickupTask()` only ever POSTs to list `901111987449` ("Presales Qualification"), and
+ClickUp's v2 API **cannot** alter a task's home list (`400 TASK_035`) — **so the worker cannot be
+the mover.** Yet observed live 2026-08-31, worker-created tasks report their **home** list as
+`901111987451` ("Quote/Proposal/SOW/MSA"), in folder `90117060687` ("Opportunities"), space
+`90113928591` ("Opportunity & RFP Hub") — same folder and space, different list.
+
+| Task | Opportunity (NS internalId) | Created (CDT) | Current home list |
+|---|---|---|---|
+| `868kyq7e1` | 75850 (7435817) | 2026-08-31 08:26 | Quote/Proposal/SOW/MSA |
+| `868kyqxq2` | 71113 (6682331) | 2026-08-31 09:09 | Quote/Proposal/SOW/MSA |
+| `868kxtf1n` | 75809 (7429824) | 2026-08-27 18:02 | Quote/Proposal/SOW/MSA |
+| `868kxgexk` | 75791 (7427497) | 2026-08-27 10:25 | **still Presales Qualification** |
+
+**Consequence: a developer or PM watching the "Presales Qualification" list will believe the sync is
+dead while it is working.** Verify by **task id**, or across the whole **Opportunities folder** —
+never by that one list.
+
+**Open question — the mover is NOT identified.** It is not cleanly stage-driven: `868kxtf1n` and
+`868kxgexk` were created the same day, both at Stage 25%, and only one moved. So it is either a
+ClickUp native **Automation** with a condition not visible through the API, or a **human** moving
+them. Do not assert a cause. Either way it is consistent with the documented hybrid split —
+home-list movement is owned by ClickUp native Automations, secondary multi-list membership by the
+worker: see
+[ClickUp Project & Opportunity Multi-List Routing](./clickup-project-routing.md).
+
+### Watch-item: 2026-08-28 — 74 opportunity events, 0 tasks created (unconfirmed)
+
+Tasks created per day in that window: Aug 24 = 1, Aug 25 = 7, Aug 26 = 2, Aug 27 = 4,
+**Aug 28 = 0**, Aug 29–30 (weekend) = 0, Aug 31 = 2. Aug 28 was a **business day with 74 opportunity
+jobs, 72 of them `skipped (no presales lead)`, and zero tasks created** — the only anomalous weekday.
+
+**Cause NOT confirmed.** On a single data point it is just as likely that nobody assigned presales
+leads that Friday. *If it recurs*, the first candidate to check is NetSuite scheduled script **2412**
+("Update Opp Status / % to Close / Presales Lead"), which auto-assigns the Presales Lead each morning
+and already has its own regression history — see
+[1.0 webhook: NetSuite → ClickUp opportunity sync](../../../../1.0/apps/webhook/features/netsuite-clickup-opportunity-sync.md).
+Diagnostic: re-run the per-day verdict census, and if the leads really are unset in NetSuite, check
+2412's deployment status/audience before reading any PHP.
+
 ## Data model
 
 Writes `Forecast.Opportunities` (header) + `Forecast.OpportunityItems` (children), faithful to
@@ -554,7 +652,20 @@ deprecated** for production opportunity code.
   blocked on the Aaron stakeholder decision noted above.
 
 ## Change history
-
+- 2026-08-31 — **Triage knowledge from a read-only "is the opportunity sync dead?" investigation — no
+  code changed.** Added a *Triage* section: the `Core.WorkerJobs.output` `clickup` verdict census is
+  the fast, NetSuite-free answer, and it has exactly **three** verdicts in practice, with a healthy
+  baseline of only **~2% of opportunity events producing a task** (86% are `skipped (no presales
+  lead)`, which is normal traffic shape, not a fault). Recorded the visibility trap that made a
+  working sync look dead: the worker only ever POSTs to list `901111987449` "Presales Qualification"
+  and **cannot** move a home list (TASK_035), yet new tasks are found homed in `901111987451`
+  "Quote/Proposal/SOW/MSA" — mover **not identified** (not cleanly stage-driven; ClickUp Automation
+  or a human), so verify by task id/folder, never by that one list. Corrected the reading of the
+  Presales-Lead gate: it **delays** a task, it does not lose one — a later edit webhook wins the
+  still-open claim whenever the lead is finally set (proven live on opportunity 71113 / internalId
+  6682331). Logged 2026-08-28 (74 events, 0 tasks) as an **unconfirmed** watch-item with NS script
+  2412 as the candidate to check on recurrence. Also confirmed the 2026-08-26 `logged()` fix is
+  holding (zero error / `(label lookup failed)` verdicts). (ajean)
 - 2026-08-27 — **Corrected the "NetSuite calls aren't logged to `Logs.Api`" gotcha — they are.** No
   `setLogging` call exists in `_underscore/Component/Api/Netsuite/Netsuite.php`, so `_ApiRequest`'s
   `$isLoggingEnabled = true` default holds and every NetSuite call (read-only SuiteQL included) writes a
@@ -704,3 +815,9 @@ deprecated** for production opportunity code.
 ## Related docs
 - `2.0/apps/worker2/architecture.md` — the WorkerJobs/SQS/Lambda model and always-HTTP-200 worker.
 - `2.0/apps/_underscore/architecture.md` — the `_Database::register()` lazy-transaction gotcha.
+- [ClickUp Project & Opportunity Multi-List Routing](./clickup-project-routing.md) — why home-list
+  movement is owned by ClickUp native Automations and not by any worker.
+- [1.0 webhook: NetSuite → ClickUp opportunity sync](../../../../1.0/apps/webhook/features/netsuite-clickup-opportunity-sync.md)
+  — NetSuite scheduled script 2412, which auto-assigns the Presales Lead that gates task creation.
+- [MCP Tool Usage](../../../../standalone/apps/claude/workflows/mcp-tool-usage.md) — the `toga-db`
+  query traps hit while running the verdict census.

package/knowledge/standalone/apps/claude/workflows/mcp-tool-usage.md

@@ -6,7 +6,7 @@ project: Claude Harness
 client: shared
 type: workflow
 status: active
-updated: 2026-08-26
+updated: 2026-08-31
 owners: ["jcardinal", "tcox", "ajean"]
 files:
   - claude/.claude/skills/kickoff/SKILL.md
@@ -141,8 +141,10 @@ trust `Core.DatabaseHosts`, not old migrations.)
 **⚠ The write-blocker is a naive string scan — it rejects `UPDATE`/`REPLACE` even inside a string
 literal.** `toga_query` refuses the whole statement if those words appear anywhere in it, so a
 perfectly read-only query like
-`… WHERE output LIKE '%skipped update%'` fails validation. Work around it by matching on a fragment
-that avoids the keyword (`LIKE '%skipped%'`), or by pulling the rows and filtering client-side —
+`… WHERE output LIKE '%skipped update%'` fails validation. **A column alias trips it too** —
+`… AS updateDisabled` is rejected on the same scan, so the alias has to be renamed
+(`skippedDisabled`), not just the predicate. Work around it by matching on a fragment that avoids the
+keyword (`LIKE '%disabled%'`, `LIKE '%skipped%'`), or by pulling the rows and filtering client-side —
 don't conclude the data isn't there.
 
 **Production sessions render in US/Central (UTC−5); most other timestamps you're comparing are
@@ -195,6 +197,9 @@ Schema names are **case-sensitive** — copy the exact value from `toga_list_sch
   and why. Database → what is actually true right now.
 
 ## Change history
+- 2026-08-31 — Extended the `toga_query` write-blocker trap: the naive keyword scan also rejects a
+  **column alias** containing `UPDATE` (`… AS updateDisabled`), not just a string literal — rename the
+  alias as well as the predicate. Hit while censusing `WorkerJobs.output` sync verdicts. (ajean)
 - 2026-08-26 — Added two `toga-db` query traps: the write-blocker is a **naive string scan** that
   rejects any statement containing `UPDATE`/`REPLACE` **even inside a string literal** (so
   `LIKE '%skipped update%'` fails validation on a read-only SELECT), and **production DB sessions

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.708",
+  "version": "1.0.709",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",