toga-ai 1.0.813 → 1.0.814

package/knowledge/2.0/apps/worker2/architecture.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: architecture
 status: active
-updated: 2026-08-31
+updated: 2026-09-15
 owners: [jcardinal, ajean]
 files:
   - worker2/Controller/Index.php
@@ -213,6 +213,16 @@ file `Worker/Team/Github.php`.
 `_Worker_Infrastructure_Worker_Cleanup`: `WorkerJobs()` deletes `isSuccess=1` rows >90 days
 (≤1000/run); `WebhookLogs()` deletes `Logs.Webhook` >90 days. Two daily CronJobs at 2 AM.
 
+**⚠ The OBSERVED `Core.WorkerJobs` retention is ~30 days, not 90 — unexplained.** Measured
+2026-09-15 in prod: `MIN(dtCreated)` across the **whole** table is **2026-08-16 02:00:46**
+(790,597 rows). Either the cron does not behave as described above, or something else prunes the
+table. **Not investigated** — no cause is asserted here.
+
+**Triage consequence: a missing `WorkerJobs` row is NOT proof the job never ran.** Any "there is no
+job for this record, so the webhook never fired" conclusion is invalid past the ~30-day floor. That
+exact wrong call was made and retracted on 2026-09-15. Same failure family as *an empty log table is
+not proof a code path never ran*.
+
 ## Cron actions must return a result on BOTH paths
 
 **A new cron action is expected to return a result for a success as well as a failure.** Both

package/knowledge/2.0/apps/worker2/features/netsuite-opportunity-client-labels.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-09-02
+updated: 2026-09-15
 owners: ["ajean"]
 files:
   - worker2/Worker/Netsuite/Opportunity.php
@@ -143,6 +143,25 @@ legitimately be running before the column lands. An **ungated write does not deg
 breaks the ENTIRE opportunity sync**, because `importOpportunity()`'s save names a column that does
 not exist and *every* NetSuite webhook fails. A label enhancement must not be able to do that.
 
+**⚠ CORRECTED 2026-09-15 — this gate did not prevent that outcome; it happened anyway, for 9 days.**
+The predicted failure ("it breaks the ENTIRE opportunity sync") occurred 2026-09-01 → 2026-09-10
+with the gate in place, because the gate checks the **DB column** and the failing write went through
+the **ORM**: the column existed, but `_underscore`'s `_Model_Forecast_Opportunity` had not yet
+declared the `endCustomerName` property, so `_Model` rejected the field. **The deploy has two
+independent halves and they ship from different repos** — the dbchanges2 migration *and* the
+`_underscore` model declaration. General rule: **gate on the layer the write actually goes through,
+not the layer that is easiest to query.**
+
+So there are now **two** gates (worker2 PR #168, open against `_production` as of 2026-09-15):
+
+| Gate | Checks | Use for |
+|---|---|---|
+| `hasEndCustomerColumn()` (unchanged) | `information_schema` column | **raw `_Query` SQL only** — the digest + both backfills |
+| `canWriteEndCustomerNameToModel()` (new) | the above **and** `property_exists(_Model_Forecast_Opportunity::class,'endCustomerName')` | **`importOpportunity()`'s ORM write only** |
+
+Each half `error_log`s on its own, naming the repo that is behind. Full incident detail:
+[the outage gotcha on the sync doc](./netsuite-opportunity-sync.md#gotchas--known-issues).
+
 - `information_schema.COLUMNS` check, **memoised per process** in a `static` (the answer cannot
   change mid-job, and the import runs per webhook) — `static`, not a constant, so a long-lived
   worker picks the column up on its next job.
@@ -348,6 +367,13 @@ A one-shot rem+add corrector was scoped and deliberately **not** built.
   injection before they go into the digest.
 
 ## Change history
+- 2026-09-15 — **Corrected the deploy gate: `hasEndCustomerColumn()` did NOT protect the ORM write,
+  and the sync was down 9 days (2026-09-01 → 2026-09-10).** The column existed; `_underscore`'s
+  `_Model_Forecast_Opportunity` had not declared the `endCustomerName` property, so `_Model`
+  rejected the field on every `Netsuite/Opportunity/post|put` (611 failed jobs / 310 opportunities).
+  Added `canWriteEndCustomerNameToModel()` for the ORM write and scoped `hasEndCustomerColumn()` to
+  raw `_Query` callers (worker2 PR #168, open). Lesson: gate on the layer the write goes through —
+  the model declaration is a second deploy half and it lives in a different repo. (ajean)
 - 2026-09-02 — **End Customer now comes from NetSuite `custbody_end_customer`, else mirrors the
   client (worker2 PR #158 + dbchanges2 PR #477; re-import PR #159 — all merged).** Root cause: End
   Customer was derived from the opportunity `entity`, which **is the partner** on a partner-routed

package/knowledge/2.0/apps/worker2/features/netsuite-opportunity-sync.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-09-10
+updated: 2026-09-15
 owners: ["dfranks", "kyalamarthi", "ajean", "jcardinal"]
 files:
   - worker2/Worker/Netsuite.php
@@ -144,6 +144,12 @@ claim on a never-claimed opportunity, so it does **not** contradict the rule tha
 *Consequence: `updateTask()` is unreachable* below). So when triaging a backlog of
 `skipped (no presales lead)`, read it as **"queued, awaiting a lead"**, not "lost".
 
+**Confirmed end-to-end a second time, 2026-09-15.** Opportunity **75504** (NS internalId 7377655,
+`Forecast.Opportunities.id` 94076) had no task; replaying its webhook returned
+`{"db":"updated opportunity 7377655 (id=94076, items=1)","clickup":"skipped (no presales lead)"}`.
+Assigning the Presales Lead in NetSuite produced the task. **This gate is the single most common
+reason a specific opportunity has no ClickUp task** — check it before suspecting the sync.
+
 Shared helpers: `buildCustomFields()` (the field array, used by both create and update),
 `resolvePresalesUserId()` (cached `Client_True.Users.c_clickupUserIdentifier` lookup → else a
 `GET /team` roster scan that back-fills the cache via `cacheClickupUser()`).
@@ -306,6 +312,57 @@ Every `Netsuite/Opportunity/post|put` job writes a JSON result into `Core.Worker
   string literal or a column alias**, so match the `skipped update (update disabled)` verdict with
   `LIKE '%disabled%'` and alias the column `skippedDisabled`, never `updateDisabled`. See
   [MCP Tool Usage](../../../../standalone/apps/claude/workflows/mcp-tool-usage.md).
+- **⚠ `Core.WorkerJobs` holds only ~30 days — a missing job row is NOT proof the webhook never
+  fired.** Measured 2026-09-15: `MIN(dtCreated)` across the **whole** table is **2026-08-16
+  02:00:46** (790,597 rows). So any "this opportunity has no job, therefore NetSuite never sent it"
+  conclusion is invalid past that horizon — a wrong call made and retracted mid-session on
+  2026-09-15. Same failure family as *an empty log table is not proof a code path never ran*. Note
+  the **discrepancy**: `worker2/architecture.md` describes the cleanup cron as deleting `isSuccess=1`
+  rows older than **90 days**, but the observed floor is ~30 — either the cron's behaviour or that
+  description is wrong. Trust the observed value and check before relying on either.
+- **`skipped (no presales lead)` has TWO causes that look identical in the DB.** The gate tests the
+  **resolved email**, not the employee reference — so it fires both when NetSuite's
+  `custbody_ctc_solution_architect` is **unset** *and* when it is set to an employee whose record has
+  **no email**. Do not read the verdict as "the field is empty in NetSuite."
+- **Scale reference for triage** — prod `Forecast.Opportunities` census, 2026-09-15. Most
+  opportunities have no ClickUp task, and that is normal:
+
+  | `netsuiteStatus` | Total | No task | With task |
+  |---|---|---|---|
+  | In Progress | 2,506 | 2,025 | 481 (~19%) |
+  | Closed - Won | 52,931 | 52,808 | 123 |
+  | Closed - Lost | 19,235 | 19,042 | 193 |
+
+### Backfill = replay the webhook, and bulk replay is safe
+
+Re-firing the ingestion endpoint is the repair mechanism for any opportunity the sync missed:
+
+```
+curl -X POST https://webhook.togahub.com/netsuite -H 'Content-Type: application/json' \
+  -d '{"recordType":"opportunity","eventType":"edit","internalId":<id>}'
+```
+
+**Why bulk replay is safe** (exercised at scale 2026-09-15 — 264 replays at 0.5 s spacing, all 264
+accepted by the ingestion Lambda; 265 jobs succeeded, 1 failed, 14 ClickUp tasks created, 1
+`already claimed`, 250 `no presales lead`):
+
+- The atomic `clickupTaskId` claim makes an already-claimed opportunity return
+  `skipped (already claimed)` and write **nothing** to ClickUp — **no duplicate tasks**.
+- An opportunity with no presales lead returns `skipped (no presales lead)` and stays queued.
+- `importOpportunity()` is self-healing, so the replay also repairs the Forecast row.
+
+**A `404` on replay is the correct signal for a NetSuite-deleted opportunity, not a defect.** The
+single failure in that run (NS internalId **7442501**) was NetSuite REST HTTP 404
+*"The opportunity does not exist"* — the record had been deleted in NetSuite, which is also why it
+was the one row missing from Forecast.
+
+### Open item — item-sync watchdog timeouts (NOT investigated)
+
+`Netsuite/InventoryItem/put|post` and `Netsuite/NonInventoryItem/*` logged **427 failures** over
+2026-09-01 → 2026-09-10, all with output `watchdog: exceeded maxExecutionTime without completing`.
+**Different root cause from the `endCustomerName` outage** (execution time, not a model error) and
+deliberately out of scope on 2026-09-15. Recorded as a known open item; needs its own ticket. No
+cause is asserted here.
 
 ### ⚠ A healthy sync reads as dead: new tasks do NOT stay in "Presales Qualification"
 
@@ -341,6 +398,17 @@ Full list ids and the API constraints are in
 re-derive them here. **Practical rule is unchanged:** verify by task id or across the whole
 Opportunities folder, never by one list.
 
+**More evidence, 2026-09-15 — and a counter-example that pins down *when* the Automation moves a
+task.** After the outage fix, new tasks again landed spread across folder `90117060687`
+("Opportunities"), space `90113928591`, **none** in Presales Qualification (`901111987449`):
+`868m40zf5` (opp 76045 Adyen, 9/10) → Pending Client Response · `868m484vn` (opp 76021 Seminole
+Gaming, 9/11) → BU Discovery (`901112518588`) · `868m5e958` (opp 75740, 9/15) and `868m5fmc3`
+(opp 76115, 9/15) → Contract Negotiations (`901111987463`). **But the 14 tasks created by the
+backfill replay the same day DID stay in Presales Qualification** (e.g. `868m5gexk` opp 72119,
+`868m5ghgt` opp 76051). So the Automation moves a task **only when the stage dropdown changes**, not
+on creation — a task sitting in Presales Qualification means its stage has not moved since birth,
+not that the mover is broken.
+
 **The Automation is load-bearing.** NetSuite writes the dropdown and ClickUp→NetSuite reads the
 human's list move; both directions run through it. Treat any change to that Automation as a
 **breaking change** to this integration.
@@ -383,26 +451,25 @@ the legacy cron's OPPORTUNITIES section:
   letter, so the handler runs a one-row SuiteQL query for it.
 - **`endCustomerName`** (added 2026-09-01, `dbchanges2/Forecast/2026-09-01a …`) holds NetSuite
   `custbody_end_customer`'s `refName` — a **name, not a FK**, because nothing upserts an
-  end-customer `Customers` row. The write and both reads are gated on the column existing
-  (`hasEndCustomerColumn()`), since the migration is applied by hand and an unknown column would fail
-  the whole query and break every opportunity webhook. See
+  end-customer `Customers` row. **CORRECTED 2026-09-15 — this bullet used to say the write and both
+  reads are "gated on the column existing (`hasEndCustomerColumn()`)", which reads as "safe". It was
+  not: that gate checks the DB column and the model write took the sync down for 9 days anyway.**
+  There are now **two** gates, and they are not interchangeable:
+  - `hasEndCustomerColumn()` — `information_schema` column check. Correct **only** for raw `_Query`
+    SQL (the digest and the two backfills).
+  - `canWriteEndCustomerNameToModel()` — `hasEndCustomerColumn() && property_exists(
+    _Model_Forecast_Opportunity::class, 'endCustomerName')`. The **only** gate valid for
+    `importOpportunity()`'s ORM write.
+
+  See the *`endCustomerName` 9-day outage* gotcha below and
   [the labels doc](./netsuite-opportunity-client-labels.md).
-  - **⚠ `hasEndCustomerColumn()` guards the DB COLUMN, NOT the ORM MODEL field — and that gap took the
-    whole opportunity sync down for 8 days (2026-09-02 → 2026-09-10).** The guard does a `SHOW COLUMNS`
-    on `Forecast.Opportunities`, so once the migration was applied it passed and the handler ran
-    `$opportunity->endCustomerName = …`. But the 2.0 model `_Model_Forecast_Opportunity`
-    (`_underscore/Model/Forecast/Opportunity.php`) never **declared** the field, so `_Model.__set`
-    (`_underscore/Model.php`) threw *"There is no field called 'endCustomerName' in the
-    '_Model_Forecast_Opportunity' model."* on **every** `Netsuite/Opportunity/post|put` — 100% failure,
-    every opportunity import dead. `Core.WorkerJobs` census: Sep 1 = 115 ok / 60 fail (last good day),
-    Sep 2 → Sep 10 = 0 ok / 100% fail (61 fails on Sep 10 alone). This was a **half-deploy**: the
-    migration + the handler write shipped, the model field declaration did not.
-    **Durable rule: a new column that the handler writes must ship THREE things together — the
-    `dbchanges2` migration, the handler write, AND the `_Model` field declaration.** A column-existence
-    guard (`SHOW COLUMNS`) proves nothing about the ORM; both ORMs enumerate *declared* properties, so an
-    undeclared field is a hard `__set` throw, not a silent skip. Fixed 2026-09-10 by declaring
-    `public $endCustomerName = self::FIELD_CHAR;` (matching sibling char fields clickupTaskId/title/memo
-    over the varchar column); deployed to worker2 `_production`, sync recovered.
+  - **Durable rule from the outage: a new column the handler writes must ship THREE things together —
+    the `dbchanges2` migration, the handler write, AND the `_Model` field declaration.** A
+    column-existence guard proves nothing about the ORM; both ORMs enumerate *declared* properties, so
+    an undeclared field is a hard `__set` throw, not a silent skip. Fixed 2026-09-10 by declaring
+    `public $endCustomerName = self::FIELD_CHAR;` (matching sibling char fields
+    `clickupTaskId`/`title`/`memo` over the varchar column). Full incident detail in the
+    *`endCustomerName` 9-day outage* gotcha below — do not restate it here.
 - Writes use raw `_Query` against `_underscore::DB_FORECAST` and **must**
   `_Database::transactionCommit(DB_FORECAST)` (lazy-transaction gotcha).
 
@@ -439,6 +506,40 @@ None — platform-wide Forecast sync.
 
 ## Gotchas / known issues
 
+- **⚠ The `endCustomerName` 9-day total outage — a column-existence gate does NOT protect an ORM
+  write (2026-09-01 → 2026-09-10).** Every `Netsuite/Opportunity/post|put` job threw
+  `There is no field called 'endCustomerName' in the '_Model_Forecast_Opportunity' model.`
+  (`_underscore/Model.php:498`) from 2026-09-01 12:01 to 2026-09-10 15:48 CDT. **The whole
+  NetSuite→Forecast→ClickUp opportunity sync was dead for 9 days and every job was red.**
+  - **Cause — a deploy split across TWO repos.** worker2 shipped the `$values['endCustomerName']`
+    write (PR #158) while `_underscore/Model/Forecast/Opportunity.php` had not yet declared the
+    property. `_Model` builds its field map from **declared properties** in the constructor, so
+    `__set` rejected the unknown field. The DB column existed (the migration had been applied) — the
+    missing half was the model declaration, in a different repo.
+  - **Blast radius:** the throw lands in `importOpportunity()` **before** the Forecast save and
+    **before** `maybeCreateClickupTask()`, so affected opportunities got **neither** a Forecast row
+    **nor** a ClickUp task. Measured in prod `Core.WorkerJobs`: **611 failed jobs, 310 distinct
+    opportunities, 264 still with no ClickUp task**, 1 (NS internalId 7442501) absent from Forecast.
+  - **Ended by `_underscore` commit `34f76ee5` "Add end customer field to Opportunity"** (Jeff
+    Cardinal, 2026-09-10 15:51:31 CDT — `public $endCustomerName = self::FIELD_CHAR;`) — three
+    minutes after the last failure. Per-day `WorkerJobs` census: Sep 1 = 115 ok / 60 fail (the
+    failures start midday, which is why this is written up as both "8 days" and "9 days"),
+    Sep 2 → Sep 10 = **0 ok / 100% fail**.
+  - **The lesson, and it generalizes: gate on the layer the write actually goes through, not the
+    layer that is easiest to query.** `hasEndCustomerColumn()`'s own docblock predicted this exact
+    failure ("it breaks the ENTIRE opportunity sync") and still did not stop it, because it ran
+    `SHOW COLUMNS` against `Forecast.Opportunities` while the write went through the ORM. A column
+    guard protects raw SQL only. **A model-declaration gap is a second, independent half of the
+    deploy, and it ships from a different repo.**
+  - **Fix (worker2 PR #168, open against `_production` as of 2026-09-15, not yet merged).**
+    `hasEndCustomerColumn()` is byte-identical and now documented as being for **raw `_Query`
+    reads/writes only** — its three callers (`OpportunityStakeholderDigest.php:180`,
+    `OpportunityStakeholderBackfill.php:579`, `OpportunityEndCustomerBackfill.php:91`) all build
+    plain SQL. New `canWriteEndCustomerNameToModel()` = `hasEndCustomerColumn() && property_exists(
+    _Model_Forecast_Opportunity::class, 'endCustomerName')` is used **only** by
+    `importOpportunity()`. Each half `error_log`s separately, naming the repo that is behind. No
+    migration. **No regression test** — worker2 has no test coverage under `Worker/` to hang one off
+    (only `Controller/CompassPeopleTest.php` and `Controller/ClickupDesignTest.php` exist).
 - **`_Model::load()` returns TRUE only on an EXACTLY-ONE match — a duplicate-row self-amplifier in
   any load-then-upsert handler.** `_Model::load()` (`_underscore/Model.php:788–808`) returns FALSE
   for **both** 0 rows **and** 2+ rows — it succeeds only when the lookup matches exactly one row.
@@ -775,6 +876,27 @@ deprecated** for production opportunity code.
   blocked on the Aaron stakeholder decision noted above.
 
 ## Change history
+- 2026-09-15 — **Root-caused a 9-day TOTAL outage of the opportunity sync and split the deploy gate
+  (worker2 PR #168, open).** Same incident jcardinal fixed on the `_underscore` side below; this
+  entry adds the worker2-side gate and the measured blast radius. From 2026-09-01 12:01 to
+  2026-09-10 15:48 CDT every `Netsuite/Opportunity/post|put` threw `There is no field called
+  'endCustomerName' in the '_Model_Forecast_Opportunity' model` — worker2 shipped the write
+  (PR #158) before `_underscore` declared the property, and `_Model` builds its field map from
+  declared properties. **611 failed jobs / 310 opportunities / 264 left with no ClickUp task** (1,
+  NS internalId 7442501, absent from Forecast); ended by `_underscore` `34f76ee5`. The existing
+  `hasEndCustomerColumn()` gate did not help because it checks the **DB column** while the write
+  goes through the **ORM** — corrected that claim in *Data model* and added
+  `canWriteEndCustomerNameToModel()` (`hasEndCustomerColumn() && property_exists(...)`) used only by
+  `importOpportunity()`; `hasEndCustomerColumn()` is now scoped to raw `_Query` callers.
+  **Also recorded:** `Core.WorkerJobs` retains only **~30 days** (MIN `dtCreated` 2026-08-16), so a
+  missing job row proves nothing — and that contradicts architecture.md's 90-day cleanup
+  description; **bulk webhook replay is safe** (264 replays, 0 duplicate tasks — the claim, the
+  presales gate and self-healing import make it so) and a NetSuite **404 on replay = the record was
+  deleted**; `skipped (no presales lead)` has **two** causes (field unset, or the employee has no
+  email); a fresh `Forecast.Opportunities` census; more evidence for the list-visibility trap plus
+  the counter-example that the Automation moves a task **only on a stage-dropdown change**, not on
+  creation. Flagged an unrelated open item: **427** `Netsuite/(Non)InventoryItem/*` watchdog
+  execution-time failures in the same window, not investigated. (ajean)
 - 2026-09-10 — **Fixed an 8-day total opportunity-sync outage caused by a half-deploy: a missing ORM
   model field.** `Forecast.Opportunities.endCustomerName` (column added ~Sep 1) and the worker2 handler
   write shipped, but `_Model_Forecast_Opportunity` (`_underscore/Model/Forecast/Opportunity.php`) never

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.813",
+  "version": "1.0.814",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",