toga-ai 1.0.813 → 1.0.815

package/knowledge/2.0/apps/_underscore/INDEX.md

@@ -11,6 +11,7 @@
 | [Assortment Name Translation (AssortmentTranslations sidecar)](features/assortment-name-translation.md) | Serves Assortment (product-grouping) **names** in multiple languages by adding a per-language **sidecar** table `AssortmentTranslations`, reusing the platform's |
 | [Asynchronous Query Execution (writes-only, via Worker)](features/async-query-execution.md) | `_Query` can run a **write** query asynchronously so a long/slow write does not hold a request-scoped DB connection open long enough to hit **"MySQL server has |
 | [FIELD_AUTOINCREMENT record numbering (SA / TA prefixes, NULL-only trigger, no padding)](features/autoincrement-record-numbering.md) | A human-facing record number (`SA100001`, `TA100000`) is generated by the ORM, not by MySQL. |
+| [Calculated-column filter options (`<field>FilterOptions` model static → table meta)](features/calculated-column-filter-options.md) | A seam that lets a **calculated (`FIELD_SQL`) table column supply its own filter options**, so the front end renders a multi-select instead of a free-text searc |
 | [FIELD_SQL calculated fields — the underscore-prefix + same-name-method contract](features/calculated-sql-fields.md) | A `FIELD_SQL` (calculated) field on a `_Model` is bound by a **two-part contract that `_Model` enforces by throwing at model-construction time**, not by convent |
 | [Carrier Shipping Labels (UPS/FedEx) & NetSuite Item Fulfillment](features/carrier-shipping-labels.md) | Backend mechanics behind TOGa Supply's Fulfill & Ship: buying a carrier label (UPS/FedEx), persisting it, and creating the NetSuite Item Fulfillment with tracki |
 | [Running 2.0 code from a bare CLI script (bootstrap + transactions)](features/cli-script-bootstrap.md) | A throwaway CLI script (a data check, a backfill dry-run, a render harness) that wants the real 2.0 framework — `_Model`, `_Query`, `_Database` — is **not** the |

package/knowledge/2.0/apps/_underscore/features/calculated-column-filter-options.md

@@ -0,0 +1,128 @@
+---
+title: Calculated-column filter options (`<field>FilterOptions` model static → table meta)
+framework: "2.0"
+repo: _underscore
+project: _Underscore
+client: shared
+type: feature
+status: active
+updated: 2026-09-15
+owners: [apeterson]
+files:
+  - _underscore/Model/Client/TableView.php
+  - _underscore/Model/Compass/SalesOrder.php
+  - _underscore/Model.php
+related:
+  - sales-order-status-filter-surface.md
+  - tableview-joins.md
+  - ../../toga-blox/features/table.md
+  - ../../api2/features/tableview-field-metadata.md
+---
+
+## What it is
+
+A seam that lets a **calculated (`FIELD_SQL`) table column supply its own filter options**, so the
+front end renders a multi-select instead of a free-text search.
+
+Before this, `filterOptions` in table meta was filled in **only** for `TYPE__STATUS` columns, via
+`getStatusFilterOptions()` (a Surface `FILTER_SET` bound to a status vocabulary — see
+[sales-order-status-filter-surface](sales-order-status-filter-surface.md)). That path cannot serve a
+column whose options are **live data** (e.g. the list of active Admin users): seeding a surface
+element per user would go stale constantly.
+
+First and only consumer today: the Compass **Assigned Admin** column on the sales-orders table.
+
+## How it works
+
+`_Model_Client_TableView::meta()`, in the per-field loop right after
+`$modelFieldConfig = (new $modelName)->getFieldConfig();`, calls
+`$modelName::{$field}FilterOptions()` when **all three** hold:
+
+1. the column has no `filterOptions` yet,
+2. `isFilterable` is set,
+3. the model field config type is `_Model::FIELD_SQL`,
+
+and the static exists (`method_exists` guard). Suffix constant:
+`_Model_Client_TableView::FILTER_OPTIONS_METHOD_SUFFIX = 'FilterOptions'`.
+
+Returning **null** means "no opinion" — the column keeps its free-text search rather than rendering
+an empty dropdown. Same contract as `getStatusFilterOptions()`.
+
+**This is not a new convention.** `_underscore/Model.php` (the `FIELD_SQL` case, ~line 135) already
+throws when a `FIELD_SQL` field has no static of the same name — that is why `_assignedAdmin()`
+exists. The seam just adds a `FilterOptions` suffix to the same name-derived rule.
+
+It is a **no-op for every existing column**: nothing else declares a `<field>FilterOptions` static.
+
+### The Compass example
+
+`_Model_Compass_SalesOrder::_assignedAdminFilterOptions(): ?array` runs a raw `_Query` on
+`_underscore::DB_CLIENT` joining `Users` / `Users_Roles` / `Roles`, filtered
+`Users.isActive = 1 AND Roles.name = ROLE_NAME__ADMIN` (`'Admin'`), and returns
+`[{slug, name}]` where **both** are `CONCAT(firstName, ' ', lastName)`.
+
+- **The value must be the NAME, not a uuid.** `_assignedAdmin()` is a correlated subquery that
+  resolves to the assignee's `CONCAT(firstName, ' ', lastName)`, so the filter compares against that
+  string; a uuid option would match zero rows.
+- **Both Compass tenants are covered by inheritance** — `_Model_Compass_Usa_SalesOrder` and
+  `_Model_Compass_Canada_SalesOrder` both extend `_Model_Compass_SalesOrder`. No per-client copy.
+- Verified locally: 13 options for `Client_Compass`, 5 for `Client_CompassCanada`. Compass Canada
+  also has a role `Agilant - Administrators`, correctly excluded by the exact `name = 'Admin'` match.
+
+## Why a model static and not a config column
+
+A CTO review compared this against adding a `TableViewFields` column holding a query descriptor
+(route + where + label field) and **agreed** on the model static: the descriptor could not express
+the required two-table `Users_Roles` / `Roles` join without growing into a query language inside a
+config column, and it would spend a multi-tenant schema change to serve one column. Revisit the
+generic version at roughly **4–5** such columns, driven by real examples.
+
+## Gotchas
+
+- **`filterOptions` alone is the front-end gate.** `PrimaryTableHeaderCell` renders
+  `HeaderFilterMultiselect` whenever `isFilterable && filterOptions`, and the free-text
+  `HeaderFilterSearch` only when `isFilterable && !filterOptions`. There is **no type gate** on the
+  multi-select branch — a `STRING` column with options gets a multi-select. `recordRoute` and
+  `surfaceSlug` appear on status columns only because a status column's options come from a
+  `FILTER_SET` surface; they are **not** required. `recordRoute` is only a *fallback* fetch
+  (`needsFetchedOptions = !!field.recordRoute && !field.filterOptions`), so
+  `filterOptions` with `recordRoute: null` is fine. Verified against
+  `@agilant/toga-blox 1.1.2-production.144` (the build toga25-supply pins), in `node_modules/dist`.
+- **The `$recordField` may be a `_Model_Client_CustomRecordField`, not a `_Model_Core_RecordField`.**
+  The Compass Assigned Admin/Assigned Manager columns are backed by `CustomRecordFields` in the
+  **Client** DB: `Client_Compass.TableViewFields` id 192 slug `assigned-admin` →
+  `customRecordFieldId` 96 (`recordFieldId` NULL); `Client_CompassCanada.TableViewFields` id 211 →
+  `customRecordFieldId` 57. Both custom fields are `field = '_assignedAdmin'`, `type = 'STRING'`,
+  `recordId = 14`. Both classes expose `->field` and `->type` identically, so field-name-keyed logic
+  works either way — just do not assume a Core RecordField.
+- **Key on the FIELD name, not the column slug.** The slug is `assigned-admin` (hyphen); the model
+  field is `_assignedAdmin`. The seam uses the field name.
+- **Type `STRING` (not STATUS/SELECT) is what lets the branch fire** — the existing `switch` in
+  `meta()` leaves `filterOptions` null for it.
+- **⚠ Name-as-identifier limit.** The option value is a display name, not a user id. Two admins with
+  the same full name collapse into one option, and picking it matches both of their orders.
+  `SELECT DISTINCT` dedupes strings, not people. Forced by `_assignedAdmin()` resolving to a name —
+  the same pattern `_assignedManager()` already uses.
+- **Matching a role by its human-readable `name` is fragile** (flagged by `sql-reviewer`): a client
+  renaming the `Admin` role silently empties the list. A stable role id would be sturdier.
+- **⚠ Filtering a calculated column lands in HAVING, not WHERE.**
+  `api2/Component/Api/V2/V2.php` `splitWhereExpressionsIntoWhereHaving` routes a where on a
+  calculated field into `HAVING`, so the `_assignedAdmin` correlated subquery is evaluated for every
+  row surviving the WHERE, **before LIMIT** — and again for the pagination count query. On a large
+  Compass `SalesOrders` table this can be slow, and it will not fail loudly (api2's Apache gateway
+  timeout is 1800s). This is pre-existing behaviour of filtering **any** calculated column, not new
+  here, and it was **not** measured with `EXPLAIN` at production size. The cheaper target would be
+  the indexed `ApprovalDecisions.assignedToUserId` with uuid option slugs — but the current
+  `filterOptions` contract has no way to say "filter on a different column than the one displayed",
+  so that stays a fallback design.
+- The multi-select's comma-joined `contains` encoding over-includes and breaks on commas — see
+  [toga-blox table](../../toga-blox/features/table.md). It affects this column and Order Status alike.
+
+## Change history
+- 2026-09-15 — Built the seam (`FILTER_OPTIONS_METHOD_SUFFIX` + the guarded branch in
+  `_Model_Client_TableView::meta()`) and its first consumer,
+  `_Model_Compass_SalesOrder::_assignedAdminFilterOptions()` (active Admin users, covering both
+  Compass tenants by inheritance). TRUE-81388, `_underscore` commit `7d92b0df`. Recorded that
+  `filterOptions` alone gates the front-end multi-select, that these columns are `CustomRecordFields`
+  rather than Core `RecordFields`, the name-as-identifier limit, and the HAVING performance risk.
+  (apeterson)

package/knowledge/2.0/apps/_underscore/features/sales-order-status-filter-surface.md

@@ -6,7 +6,7 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-08-25
+updated: 2026-09-15
 owners: [apeterson]
 files:
   - _underscore/Model/Client/TableView.php
@@ -22,6 +22,7 @@ files:
   - dbchanges2/Client_Quad/2026-08-24b - SalesOrderStatusFilterHides.sql
   - dbchanges2/Client_Nychh/2026-08-24b - SalesOrderStatusFilterHides.sql
 related:
+  - calculated-column-filter-options.md
   - surface-resolver.md
   - item-fulfillment-stage-lifecycle-and-order-status.md
   - ../../dbchanges2/features/surface-layer-schema.md
@@ -137,6 +138,11 @@ states from the NetSuite billing sync.
   metadata DB change (see [toga-blox table](../../toga-blox/features/table.md)).
 
 ## Change history
+- 2026-09-15 — Cross-linked the new calculated-column seam: a `FIELD_SQL` column can now emit its own
+  `filterOptions` from a `<field>FilterOptions` model static, so `getStatusFilterOptions()` is no
+  longer the only producer. `recordRoute`/`surfaceSlug` are status-specific, not requirements for a
+  multi-select. See
+  [calculated-column-filter-options](calculated-column-filter-options.md). (apeterson)
 - 2026-08-25 — Built the feature end to end: Core `FILTER_SET` surface 44 + BADGE elements 147–160
   (labels/colours from the `sales-order-status` vocabulary, per-client hiding via `SurfaceOverrides`
   `IS_VISIBLE=0`); `_Model_Client_TableView` emits `surfaceSlug` + `filterOptions` on `TYPE__STATUS`

package/knowledge/2.0/apps/toga-blox/features/table.md

@@ -163,8 +163,25 @@ layer and emits it — see
 [sales-order-status-filter-surface](../../_underscore/features/sales-order-status-filter-surface.md).
 **The blox side never derives a surface slug by naming convention; it renders what meta gives it.**
 
+**`filterOptions` alone is the gate.** `PrimaryTableHeaderCell` renders `HeaderFilterMultiselect`
+when `isFilterable && filterOptions`, and `HeaderFilterSearch` only when
+`isFilterable && !filterOptions` (and the type is not CURRENCY/NUMBER/DATETIME/DATE). There is **no
+type gate** on the multi-select branch — a `STRING` column with options gets a multi-select, and
+`recordRoute` / `surfaceSlug` are **not** required (`recordRoute` is only the fallback fetch:
+`needsFetchedOptions = !!field.recordRoute && !field.filterOptions`). The backend can therefore feed
+a calculated column's options from a model static — see
+[calculated-column-filter-options](../../_underscore/features/calculated-column-filter-options.md).
+
 ## Gotchas
 
+- **⚠ OPEN — the multi-select filter matches by SUBSTRING, not exactly.** `HeaderFilterMultiselect`
+  comma-joins the picked slugs and `src/api/tableData/getDataTableData.ts` turns that into
+  `where: { or: [ { field: { contains: value } }, … ] }`. Picking "Jon Smith" also matches
+  "Jon Smithson" — it over-includes rows.
+- **⚠ OPEN — an option value containing a comma silently returns zero rows.** It splits into two
+  bogus filter terms, with no error. Both bugs are pre-existing and affect **every** multi-select
+  column including Order Status; both fix in one place — send a list / exact-equals operator instead
+  of a comma-joined `contains`. Not fixed as of 2026-09-15.
 - **⚠ `useTableData` APPENDS `.uuid` to `additionalData.slug` — unless the slug is on a hardcoded
   allow-list.** The list is exactly `catalogId`, `customerId`, `Items.catalogId`,
   `SalesOrderItems.uuid`, `SalesOrders.uuid` (verified in `@agilant/toga-blox`
@@ -238,6 +255,10 @@ layer and emits it — see
   requires `npm run build` before it takes effect at runtime.
 
 ## Change history
+- 2026-09-15 — Recorded that `filterOptions` alone gates the multi-select (no type gate;
+  `recordRoute`/`surfaceSlug` not required) and logged two pre-existing multi-select filter bugs:
+  substring `contains` matching over-includes rows, and a comma inside an option value returns zero
+  rows. Found while adding the Compass Assigned Admin filter (TRUE-81388); neither fixed. (apeterson)
 - 2026-09-04 — Recorded the `useTableData` **`.uuid` append + hardcoded allow-list** contract
   (`catalogId`, `customerId`, `Items.catalogId`, `SalesOrderItems.uuid`, `SalesOrders.uuid`,
   verified in `1.1.2-production.144`). A dotted slug that is not on the list becomes

package/knowledge/2.0/apps/worker2/architecture.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: architecture
 status: active
-updated: 2026-08-31
+updated: 2026-09-15
 owners: [jcardinal, ajean]
 files:
   - worker2/Controller/Index.php
@@ -213,6 +213,16 @@ file `Worker/Team/Github.php`.
 `_Worker_Infrastructure_Worker_Cleanup`: `WorkerJobs()` deletes `isSuccess=1` rows >90 days
 (≤1000/run); `WebhookLogs()` deletes `Logs.Webhook` >90 days. Two daily CronJobs at 2 AM.
 
+**⚠ The OBSERVED `Core.WorkerJobs` retention is ~30 days, not 90 — unexplained.** Measured
+2026-09-15 in prod: `MIN(dtCreated)` across the **whole** table is **2026-08-16 02:00:46**
+(790,597 rows). Either the cron does not behave as described above, or something else prunes the
+table. **Not investigated** — no cause is asserted here.
+
+**Triage consequence: a missing `WorkerJobs` row is NOT proof the job never ran.** Any "there is no
+job for this record, so the webhook never fired" conclusion is invalid past the ~30-day floor. That
+exact wrong call was made and retracted on 2026-09-15. Same failure family as *an empty log table is
+not proof a code path never ran*.
+
 ## Cron actions must return a result on BOTH paths
 
 **A new cron action is expected to return a result for a success as well as a failure.** Both

package/knowledge/2.0/apps/worker2/features/netsuite-opportunity-client-labels.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-09-02
+updated: 2026-09-15
 owners: ["ajean"]
 files:
   - worker2/Worker/Netsuite/Opportunity.php
@@ -143,6 +143,25 @@ legitimately be running before the column lands. An **ungated write does not deg
 breaks the ENTIRE opportunity sync**, because `importOpportunity()`'s save names a column that does
 not exist and *every* NetSuite webhook fails. A label enhancement must not be able to do that.
 
+**⚠ CORRECTED 2026-09-15 — this gate did not prevent that outcome; it happened anyway, for 9 days.**
+The predicted failure ("it breaks the ENTIRE opportunity sync") occurred 2026-09-01 → 2026-09-10
+with the gate in place, because the gate checks the **DB column** and the failing write went through
+the **ORM**: the column existed, but `_underscore`'s `_Model_Forecast_Opportunity` had not yet
+declared the `endCustomerName` property, so `_Model` rejected the field. **The deploy has two
+independent halves and they ship from different repos** — the dbchanges2 migration *and* the
+`_underscore` model declaration. General rule: **gate on the layer the write actually goes through,
+not the layer that is easiest to query.**
+
+So there are now **two** gates (worker2 PR #168, open against `_production` as of 2026-09-15):
+
+| Gate | Checks | Use for |
+|---|---|---|
+| `hasEndCustomerColumn()` (unchanged) | `information_schema` column | **raw `_Query` SQL only** — the digest + both backfills |
+| `canWriteEndCustomerNameToModel()` (new) | the above **and** `property_exists(_Model_Forecast_Opportunity::class,'endCustomerName')` | **`importOpportunity()`'s ORM write only** |
+
+Each half `error_log`s on its own, naming the repo that is behind. Full incident detail:
+[the outage gotcha on the sync doc](./netsuite-opportunity-sync.md#gotchas--known-issues).
+
 - `information_schema.COLUMNS` check, **memoised per process** in a `static` (the answer cannot
   change mid-job, and the import runs per webhook) — `static`, not a constant, so a long-lived
   worker picks the column up on its next job.
@@ -348,6 +367,13 @@ A one-shot rem+add corrector was scoped and deliberately **not** built.
   injection before they go into the digest.
 
 ## Change history
+- 2026-09-15 — **Corrected the deploy gate: `hasEndCustomerColumn()` did NOT protect the ORM write,
+  and the sync was down 9 days (2026-09-01 → 2026-09-10).** The column existed; `_underscore`'s
+  `_Model_Forecast_Opportunity` had not declared the `endCustomerName` property, so `_Model`
+  rejected the field on every `Netsuite/Opportunity/post|put` (611 failed jobs / 310 opportunities).
+  Added `canWriteEndCustomerNameToModel()` for the ORM write and scoped `hasEndCustomerColumn()` to
+  raw `_Query` callers (worker2 PR #168, open). Lesson: gate on the layer the write goes through —
+  the model declaration is a second deploy half and it lives in a different repo. (ajean)
 - 2026-09-02 — **End Customer now comes from NetSuite `custbody_end_customer`, else mirrors the
   client (worker2 PR #158 + dbchanges2 PR #477; re-import PR #159 — all merged).** Root cause: End
   Customer was derived from the opportunity `entity`, which **is the partner** on a partner-routed

package/knowledge/2.0/apps/worker2/features/netsuite-opportunity-sync.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-09-10
+updated: 2026-09-15
 owners: ["dfranks", "kyalamarthi", "ajean", "jcardinal"]
 files:
   - worker2/Worker/Netsuite.php
@@ -144,6 +144,12 @@ claim on a never-claimed opportunity, so it does **not** contradict the rule tha
 *Consequence: `updateTask()` is unreachable* below). So when triaging a backlog of
 `skipped (no presales lead)`, read it as **"queued, awaiting a lead"**, not "lost".
 
+**Confirmed end-to-end a second time, 2026-09-15.** Opportunity **75504** (NS internalId 7377655,
+`Forecast.Opportunities.id` 94076) had no task; replaying its webhook returned
+`{"db":"updated opportunity 7377655 (id=94076, items=1)","clickup":"skipped (no presales lead)"}`.
+Assigning the Presales Lead in NetSuite produced the task. **This gate is the single most common
+reason a specific opportunity has no ClickUp task** — check it before suspecting the sync.
+
 Shared helpers: `buildCustomFields()` (the field array, used by both create and update),
 `resolvePresalesUserId()` (cached `Client_True.Users.c_clickupUserIdentifier` lookup → else a
 `GET /team` roster scan that back-fills the cache via `cacheClickupUser()`).
@@ -306,6 +312,57 @@ Every `Netsuite/Opportunity/post|put` job writes a JSON result into `Core.Worker
   string literal or a column alias**, so match the `skipped update (update disabled)` verdict with
   `LIKE '%disabled%'` and alias the column `skippedDisabled`, never `updateDisabled`. See
   [MCP Tool Usage](../../../../standalone/apps/claude/workflows/mcp-tool-usage.md).
+- **⚠ `Core.WorkerJobs` holds only ~30 days — a missing job row is NOT proof the webhook never
+  fired.** Measured 2026-09-15: `MIN(dtCreated)` across the **whole** table is **2026-08-16
+  02:00:46** (790,597 rows). So any "this opportunity has no job, therefore NetSuite never sent it"
+  conclusion is invalid past that horizon — a wrong call made and retracted mid-session on
+  2026-09-15. Same failure family as *an empty log table is not proof a code path never ran*. Note
+  the **discrepancy**: `worker2/architecture.md` describes the cleanup cron as deleting `isSuccess=1`
+  rows older than **90 days**, but the observed floor is ~30 — either the cron's behaviour or that
+  description is wrong. Trust the observed value and check before relying on either.
+- **`skipped (no presales lead)` has TWO causes that look identical in the DB.** The gate tests the
+  **resolved email**, not the employee reference — so it fires both when NetSuite's
+  `custbody_ctc_solution_architect` is **unset** *and* when it is set to an employee whose record has
+  **no email**. Do not read the verdict as "the field is empty in NetSuite."
+- **Scale reference for triage** — prod `Forecast.Opportunities` census, 2026-09-15. Most
+  opportunities have no ClickUp task, and that is normal:
+
+  | `netsuiteStatus` | Total | No task | With task |
+  |---|---|---|---|
+  | In Progress | 2,506 | 2,025 | 481 (~19%) |
+  | Closed - Won | 52,931 | 52,808 | 123 |
+  | Closed - Lost | 19,235 | 19,042 | 193 |
+
+### Backfill = replay the webhook, and bulk replay is safe
+
+Re-firing the ingestion endpoint is the repair mechanism for any opportunity the sync missed:
+
+```
+curl -X POST https://webhook.togahub.com/netsuite -H 'Content-Type: application/json' \
+  -d '{"recordType":"opportunity","eventType":"edit","internalId":<id>}'
+```
+
+**Why bulk replay is safe** (exercised at scale 2026-09-15 — 264 replays at 0.5 s spacing, all 264
+accepted by the ingestion Lambda; 265 jobs succeeded, 1 failed, 14 ClickUp tasks created, 1
+`already claimed`, 250 `no presales lead`):
+
+- The atomic `clickupTaskId` claim makes an already-claimed opportunity return
+  `skipped (already claimed)` and write **nothing** to ClickUp — **no duplicate tasks**.
+- An opportunity with no presales lead returns `skipped (no presales lead)` and stays queued.
+- `importOpportunity()` is self-healing, so the replay also repairs the Forecast row.
+
+**A `404` on replay is the correct signal for a NetSuite-deleted opportunity, not a defect.** The
+single failure in that run (NS internalId **7442501**) was NetSuite REST HTTP 404
+*"The opportunity does not exist"* — the record had been deleted in NetSuite, which is also why it
+was the one row missing from Forecast.
+
+### Open item — item-sync watchdog timeouts (NOT investigated)
+
+`Netsuite/InventoryItem/put|post` and `Netsuite/NonInventoryItem/*` logged **427 failures** over
+2026-09-01 → 2026-09-10, all with output `watchdog: exceeded maxExecutionTime without completing`.
+**Different root cause from the `endCustomerName` outage** (execution time, not a model error) and
+deliberately out of scope on 2026-09-15. Recorded as a known open item; needs its own ticket. No
+cause is asserted here.
 
 ### ⚠ A healthy sync reads as dead: new tasks do NOT stay in "Presales Qualification"
 
@@ -341,6 +398,17 @@ Full list ids and the API constraints are in
 re-derive them here. **Practical rule is unchanged:** verify by task id or across the whole
 Opportunities folder, never by one list.
 
+**More evidence, 2026-09-15 — and a counter-example that pins down *when* the Automation moves a
+task.** After the outage fix, new tasks again landed spread across folder `90117060687`
+("Opportunities"), space `90113928591`, **none** in Presales Qualification (`901111987449`):
+`868m40zf5` (opp 76045 Adyen, 9/10) → Pending Client Response · `868m484vn` (opp 76021 Seminole
+Gaming, 9/11) → BU Discovery (`901112518588`) · `868m5e958` (opp 75740, 9/15) and `868m5fmc3`
+(opp 76115, 9/15) → Contract Negotiations (`901111987463`). **But the 14 tasks created by the
+backfill replay the same day DID stay in Presales Qualification** (e.g. `868m5gexk` opp 72119,
+`868m5ghgt` opp 76051). So the Automation moves a task **only when the stage dropdown changes**, not
+on creation — a task sitting in Presales Qualification means its stage has not moved since birth,
+not that the mover is broken.
+
 **The Automation is load-bearing.** NetSuite writes the dropdown and ClickUp→NetSuite reads the
 human's list move; both directions run through it. Treat any change to that Automation as a
 **breaking change** to this integration.
@@ -383,26 +451,25 @@ the legacy cron's OPPORTUNITIES section:
   letter, so the handler runs a one-row SuiteQL query for it.
 - **`endCustomerName`** (added 2026-09-01, `dbchanges2/Forecast/2026-09-01a …`) holds NetSuite
   `custbody_end_customer`'s `refName` — a **name, not a FK**, because nothing upserts an
-  end-customer `Customers` row. The write and both reads are gated on the column existing
-  (`hasEndCustomerColumn()`), since the migration is applied by hand and an unknown column would fail
-  the whole query and break every opportunity webhook. See
+  end-customer `Customers` row. **CORRECTED 2026-09-15 — this bullet used to say the write and both
+  reads are "gated on the column existing (`hasEndCustomerColumn()`)", which reads as "safe". It was
+  not: that gate checks the DB column and the model write took the sync down for 9 days anyway.**
+  There are now **two** gates, and they are not interchangeable:
+  - `hasEndCustomerColumn()` — `information_schema` column check. Correct **only** for raw `_Query`
+    SQL (the digest and the two backfills).
+  - `canWriteEndCustomerNameToModel()` — `hasEndCustomerColumn() && property_exists(
+    _Model_Forecast_Opportunity::class, 'endCustomerName')`. The **only** gate valid for
+    `importOpportunity()`'s ORM write.
+
+  See the *`endCustomerName` 9-day outage* gotcha below and
   [the labels doc](./netsuite-opportunity-client-labels.md).
-  - **⚠ `hasEndCustomerColumn()` guards the DB COLUMN, NOT the ORM MODEL field — and that gap took the
-    whole opportunity sync down for 8 days (2026-09-02 → 2026-09-10).** The guard does a `SHOW COLUMNS`
-    on `Forecast.Opportunities`, so once the migration was applied it passed and the handler ran
-    `$opportunity->endCustomerName = …`. But the 2.0 model `_Model_Forecast_Opportunity`
-    (`_underscore/Model/Forecast/Opportunity.php`) never **declared** the field, so `_Model.__set`
-    (`_underscore/Model.php`) threw *"There is no field called 'endCustomerName' in the
-    '_Model_Forecast_Opportunity' model."* on **every** `Netsuite/Opportunity/post|put` — 100% failure,
-    every opportunity import dead. `Core.WorkerJobs` census: Sep 1 = 115 ok / 60 fail (last good day),
-    Sep 2 → Sep 10 = 0 ok / 100% fail (61 fails on Sep 10 alone). This was a **half-deploy**: the
-    migration + the handler write shipped, the model field declaration did not.
-    **Durable rule: a new column that the handler writes must ship THREE things together — the
-    `dbchanges2` migration, the handler write, AND the `_Model` field declaration.** A column-existence
-    guard (`SHOW COLUMNS`) proves nothing about the ORM; both ORMs enumerate *declared* properties, so an
-    undeclared field is a hard `__set` throw, not a silent skip. Fixed 2026-09-10 by declaring
-    `public $endCustomerName = self::FIELD_CHAR;` (matching sibling char fields clickupTaskId/title/memo
-    over the varchar column); deployed to worker2 `_production`, sync recovered.
+  - **Durable rule from the outage: a new column the handler writes must ship THREE things together —
+    the `dbchanges2` migration, the handler write, AND the `_Model` field declaration.** A
+    column-existence guard proves nothing about the ORM; both ORMs enumerate *declared* properties, so
+    an undeclared field is a hard `__set` throw, not a silent skip. Fixed 2026-09-10 by declaring
+    `public $endCustomerName = self::FIELD_CHAR;` (matching sibling char fields
+    `clickupTaskId`/`title`/`memo` over the varchar column). Full incident detail in the
+    *`endCustomerName` 9-day outage* gotcha below — do not restate it here.
 - Writes use raw `_Query` against `_underscore::DB_FORECAST` and **must**
   `_Database::transactionCommit(DB_FORECAST)` (lazy-transaction gotcha).
 
@@ -439,6 +506,40 @@ None — platform-wide Forecast sync.
 
 ## Gotchas / known issues
 
+- **⚠ The `endCustomerName` 9-day total outage — a column-existence gate does NOT protect an ORM
+  write (2026-09-01 → 2026-09-10).** Every `Netsuite/Opportunity/post|put` job threw
+  `There is no field called 'endCustomerName' in the '_Model_Forecast_Opportunity' model.`
+  (`_underscore/Model.php:498`) from 2026-09-01 12:01 to 2026-09-10 15:48 CDT. **The whole
+  NetSuite→Forecast→ClickUp opportunity sync was dead for 9 days and every job was red.**
+  - **Cause — a deploy split across TWO repos.** worker2 shipped the `$values['endCustomerName']`
+    write (PR #158) while `_underscore/Model/Forecast/Opportunity.php` had not yet declared the
+    property. `_Model` builds its field map from **declared properties** in the constructor, so
+    `__set` rejected the unknown field. The DB column existed (the migration had been applied) — the
+    missing half was the model declaration, in a different repo.
+  - **Blast radius:** the throw lands in `importOpportunity()` **before** the Forecast save and
+    **before** `maybeCreateClickupTask()`, so affected opportunities got **neither** a Forecast row
+    **nor** a ClickUp task. Measured in prod `Core.WorkerJobs`: **611 failed jobs, 310 distinct
+    opportunities, 264 still with no ClickUp task**, 1 (NS internalId 7442501) absent from Forecast.
+  - **Ended by `_underscore` commit `34f76ee5` "Add end customer field to Opportunity"** (Jeff
+    Cardinal, 2026-09-10 15:51:31 CDT — `public $endCustomerName = self::FIELD_CHAR;`) — three
+    minutes after the last failure. Per-day `WorkerJobs` census: Sep 1 = 115 ok / 60 fail (the
+    failures start midday, which is why this is written up as both "8 days" and "9 days"),
+    Sep 2 → Sep 10 = **0 ok / 100% fail**.
+  - **The lesson, and it generalizes: gate on the layer the write actually goes through, not the
+    layer that is easiest to query.** `hasEndCustomerColumn()`'s own docblock predicted this exact
+    failure ("it breaks the ENTIRE opportunity sync") and still did not stop it, because it ran
+    `SHOW COLUMNS` against `Forecast.Opportunities` while the write went through the ORM. A column
+    guard protects raw SQL only. **A model-declaration gap is a second, independent half of the
+    deploy, and it ships from a different repo.**
+  - **Fix (worker2 PR #168, open against `_production` as of 2026-09-15, not yet merged).**
+    `hasEndCustomerColumn()` is byte-identical and now documented as being for **raw `_Query`
+    reads/writes only** — its three callers (`OpportunityStakeholderDigest.php:180`,
+    `OpportunityStakeholderBackfill.php:579`, `OpportunityEndCustomerBackfill.php:91`) all build
+    plain SQL. New `canWriteEndCustomerNameToModel()` = `hasEndCustomerColumn() && property_exists(
+    _Model_Forecast_Opportunity::class, 'endCustomerName')` is used **only** by
+    `importOpportunity()`. Each half `error_log`s separately, naming the repo that is behind. No
+    migration. **No regression test** — worker2 has no test coverage under `Worker/` to hang one off
+    (only `Controller/CompassPeopleTest.php` and `Controller/ClickupDesignTest.php` exist).
 - **`_Model::load()` returns TRUE only on an EXACTLY-ONE match — a duplicate-row self-amplifier in
   any load-then-upsert handler.** `_Model::load()` (`_underscore/Model.php:788–808`) returns FALSE
   for **both** 0 rows **and** 2+ rows — it succeeds only when the lookup matches exactly one row.
@@ -775,6 +876,27 @@ deprecated** for production opportunity code.
   blocked on the Aaron stakeholder decision noted above.
 
 ## Change history
+- 2026-09-15 — **Root-caused a 9-day TOTAL outage of the opportunity sync and split the deploy gate
+  (worker2 PR #168, open).** Same incident jcardinal fixed on the `_underscore` side below; this
+  entry adds the worker2-side gate and the measured blast radius. From 2026-09-01 12:01 to
+  2026-09-10 15:48 CDT every `Netsuite/Opportunity/post|put` threw `There is no field called
+  'endCustomerName' in the '_Model_Forecast_Opportunity' model` — worker2 shipped the write
+  (PR #158) before `_underscore` declared the property, and `_Model` builds its field map from
+  declared properties. **611 failed jobs / 310 opportunities / 264 left with no ClickUp task** (1,
+  NS internalId 7442501, absent from Forecast); ended by `_underscore` `34f76ee5`. The existing
+  `hasEndCustomerColumn()` gate did not help because it checks the **DB column** while the write
+  goes through the **ORM** — corrected that claim in *Data model* and added
+  `canWriteEndCustomerNameToModel()` (`hasEndCustomerColumn() && property_exists(...)`) used only by
+  `importOpportunity()`; `hasEndCustomerColumn()` is now scoped to raw `_Query` callers.
+  **Also recorded:** `Core.WorkerJobs` retains only **~30 days** (MIN `dtCreated` 2026-08-16), so a
+  missing job row proves nothing — and that contradicts architecture.md's 90-day cleanup
+  description; **bulk webhook replay is safe** (264 replays, 0 duplicate tasks — the claim, the
+  presales gate and self-healing import make it so) and a NetSuite **404 on replay = the record was
+  deleted**; `skipped (no presales lead)` has **two** causes (field unset, or the employee has no
+  email); a fresh `Forecast.Opportunities` census; more evidence for the list-visibility trap plus
+  the counter-example that the Automation moves a task **only on a stage-dropdown change**, not on
+  creation. Flagged an unrelated open item: **427** `Netsuite/(Non)InventoryItem/*` watchdog
+  execution-time failures in the same window, not investigated. (ajean)
 - 2026-09-10 — **Fixed an 8-day total opportunity-sync outage caused by a half-deploy: a missing ORM
   model field.** `Forecast.Opportunities.endCustomerName` (column added ~Sep 1) and the worker2 handler
   write shipped, but `_Model_Forecast_Opportunity` (`_underscore/Model/Forecast/Opportunity.php`) never

package/knowledge/INDEX.md

@@ -18,7 +18,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 
 ## 2.0 framework
 
-- **_underscore** (_Underscore) _(framework core)_ — 84 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
+- **_underscore** (_Underscore) _(framework core)_ — 85 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
 - **worker2** (Worker) — 69 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
 - **api2** (API) — 26 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
 - **dbchanges2** (Database Changes) _(framework core)_ — 19 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.813",
+  "version": "1.0.815",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",