toga-ai 1.0.618 → 1.0.620

package/knowledge/1.0/apps/library/features/netsuite-suiteql-api-reference.md

@@ -6,8 +6,8 @@ project: Library
 client: shared
 type: feature
 status: active
-updated: 2026-07-21
-owners: ["dfranks", "jcardinal"]
+updated: 2026-08-19
+owners: ["dfranks", "jcardinal", "kyalamarthi"]
 files:
   - library/app/api/netsuite/rest.php
   - library/ssl/netsuite_ec_key.pem
@@ -99,6 +99,17 @@ In production (worker/EB) `send()` is fine — the Logs DB is reachable there.
   `'YYYY-MM'` for bucketing, `NVL(...)`, and `BUILTIN.DF(field)` for a reference field's display
   value (e.g. `BUILTIN.DF(entity)` → customer name, `BUILTIN.DF(status)` → "Pending Fulfillment") —
   a cheap alternative to per-id GETs for display strings.
+- **⚠ Column ALIASES come back LOWER-CASED.** `SELECT … AS accountTypeLabel` returns the response key
+  **`accounttypelabel`**. Reading it back in camelCase (`$row->accountTypeLabel`) yields **`NULL` with no
+  error, no warning and no exception** — the query succeeded, the job reports success, and every value is
+  quietly empty. Confirmed empirically on this account (`BUILTIN.DF(sd.recordtype) AS recordTypeName` →
+  `recordtypename`). **Write all-lowercase aliases, or read the lower-cased key.** This has caused two
+  separate defects in the 2.0 customer sync, one of which is still live.
+- **⚠ A List/Record custom field selected RAW returns the list member's INTERNAL ID, not its label.**
+  Nothing errors — you simply store `3` where the report expects "Strategic Growth". Wrap it:
+  `BUILTIN.DF(custentity_<x>)`. This applies to **every** List/Record custom field, and it is not
+  discoverable from the column name, so **probe the field before trusting a raw select** (issue the
+  `BUILTIN.DF` form once and compare). See the `customer` table section for a worked example.
 - **`ORDER BY` inside a `GROUP BY` query is rejected** with `Invalid or unsupported search`. Sort in
   PHP after fetching all pages.
 - **Correlated subqueries are supported** (used in `listSales()` for `orderLine` — see the
@@ -261,6 +272,15 @@ work; the pk is `nkey`). Columns: `addr1`, `addr2`, `addressee`, `attention`, `c
   4968, script id `custentity10`); SOAP `customFieldList` CF internalId 4968 maps to this.
 - `entityid` and `companyname` are split in SuiteQL; SOAP concatenated them as
   `"<entityid> <companyname>"`. When they're equal, return the value once (don't duplicate).
+  `App_Api_Netsuite_Rest::listCustomers()` is the authoritative implementation of that rule — any other
+  writer of a customer *name* column must mirror it exactly, or the two writers will fight
+  (`"6096"` vs `"6096 ODP Veyer (B2B)"`). Verified 2026-08-19 on four production-sourced names.
+- **`BUILTIN.DF(custentity_account_type)`** = display label of the **Account Type** entity field
+  (NetSuite internal id **10719**). It is a **List/Record** field, so a raw `SELECT
+  custentity_account_type` gives you the member's internal id. Live map (verified 2026-08-19):
+  **1 = Strategic Emerging · 2 = Not Strategic · 3 = Strategic Growth · 4 = Strategic Maintain ·
+  5 = Strategic Draining.** Sparsely populated — **248 of 9,183** customers (2.7%) have it set, so a
+  mostly-empty result is correct, not a broken query.
 
 ## `systemnote` table — field-change forensics
 
@@ -283,7 +303,8 @@ field changed, **including sublist LINE edits that fire no User Event and theref
 |---------------|-----------|-------------|
 | 3149 | `CUSTBODY_END_CUSTOMER` | End Customer (on transaction) |
 | 7097 | `CUSTBODY_STOCKING_ORDER` | Stocking Order flag |
-| 4968 | `custentity10` | Consolidated Customer (on customer record) |
+| 4968 | `custentity10` | Consolidated Customer (on customer record) — List/Record; needs `BUILTIN.DF` |
+| 10719 | `custentity_account_type` | Account Type (on customer record) — List/Record; needs `BUILTIN.DF` |
 | 6840 | `custbody_forecast_category` | Forecast Category (on opportunity) |
 | 6841 | `custbody_sales_stage` | Sales Stage (on opportunity) |
 | 3942 | `custbody_ctc_account` (`CF_ACCOUNT`) | Account |
@@ -371,6 +392,15 @@ numbers. Budget hours for multi-year runs and launch them under `nohup`/`tmux`.
 
 ## Change history
 
+- 2026-08-19 — Added two SuiteQL mechanics that fail **silently**, both found porting the customer
+  account-type sync (TRUE-80206): **column aliases come back lower-cased** (`AS accountTypeLabel` →
+  `accounttypelabel`, so a camelCase read is `NULL` with no error — it made a backfill report success while
+  writing nothing), and **a List/Record custom field selected raw returns the member's internal id, not its
+  label** — always `BUILTIN.DF()`. Documented `custentity_account_type` (internal id 10719, 1–5 label map,
+  populated on only 2.7% of customers) in the `customer` section and the custom-field table, and noted that
+  `listCustomers()`'s entityid/companyname concatenation is the rule any other writer of a customer name
+  must mirror byte-for-byte. Also recorded (elsewhere) that `scriptdeployment` is queryable via SuiteQL but
+  absent from the REST record catalog. (kyalamarthi)
 - 2026-07-21 — **Recorded the "one PHP 7.4+ token = total outage" blast-radius lesson** on the PHP
   7.2 compat gotcha (folded in from a retired project-local CLAUDE.md). On 2026-06-10 a single
   `static fn(...)` in `rest.php` parse-errored `App_Api_Netsuite_Rest` on class load and crashed ALL

package/knowledge/2.0/apps/dbchanges2/workflows/nonprod-metadata-drift-repair.md

@@ -6,8 +6,8 @@ project: Database Changes
 client: shared
 type: workflow
 status: active
-updated: 2026-08-17
-owners: ["mhammontree", "bala"]
+updated: 2026-08-19
+owners: ["mhammontree", "bala", "kyalamarthi"]
 files:
   - dbchanges2/Core/2026-06-30a - ItemFulfillmentStageDefaultInterceptor.sql
   - dbchanges2/Client_Compass/2026-08-06 - RemoveBrokenSalesOrderItemPostPostInterceptor.sql
@@ -233,8 +233,37 @@ blocked by the id-divergence hazard in item 2 above, which is why both go to jca
 - `item-fulfillments.returnAddressId` — the **column exists** but there is **no Core RecordField**.
   Only needed for the return-label flow.
 
+## ⚠ It is not only metadata — the `Forecast/` DDL fan-out is stale too (dev-sandbox, 2026-08-19)
+
+The rest of this workflow is about **metadata** rows drifting. The same "someone has to remember to run
+it there" gap applies to plain **DDL**, and on dev-sandbox the `Forecast/` folder looks like it has not
+been applied since roughly **February 2026**:
+
+| Column | Migration date | dev-sandbox |
+|---|---|---|
+| `Employees.departmentId` | 2026-02 | present |
+| `Customers.accountType` | 2026-07-14 | **missing** |
+| `Sales.amountDue` | 2026-06-09 | **missing** |
+| `Opportunities.clickupTaskId` | 2026-07-20 | **missing** |
+
+All three missing columns are merged in `dbchanges2` `_main`. Note the shape differs from the `Client/`
+fan-out gap documented above: this is not a few skipped dates, it is a **continuous ~6-month tail**.
+
+**Consequence: dev-sandbox cannot be used to test recent Forecast work.** A dry run there dies on the
+missing column, which reads like an application bug and burns an afternoon. Until the `Forecast/` fan-out
+is caught up, test against a **local Forecast fixture** instead — see
+[running worker2 locally](../../worker2/workflows/running-worker2-locally.md). Before assuming a
+`Forecast` column exists in any non-prod environment, check the environment, not the repo.
+
 ## Change history
 
+- 2026-08-19 — Recorded that the drift is **not only metadata**: dev-sandbox's `Forecast/` **DDL** fan-out
+  appears unapplied since ~Feb 2026 — it has `Employees.departmentId` (2026-02) but is missing
+  `Sales.amountDue` (2026-06-09), `Customers.accountType` (2026-07-14) and `Opportunities.clickupTaskId`
+  (2026-07-20), all merged in `_main`. Unlike the `Client/` gap this is a continuous ~6-month tail, not a
+  few skipped dates. Practical effect: **dev-sandbox cannot test recent Forecast work** — a dry run fails
+  on the missing column and looks like an application bug; use a local Forecast fixture instead. Surfaced
+  while testing the customer account-type sync (TRUE-80206). (kyalamarthi)
 - 2026-08-17 (later pass) — Extended the fan-out gap: it **also hits `_modules/<module>/`**, not just
   `Client/`. Beta `Client_Aig` had never run `_modules/netsuite/2026-07-10a - UnitInventoryFields.sql`
   despite `_modules.txt` listing `netsuite`, so `Units` lacked six `c_` columns prod has → 1054/EV-12

package/knowledge/2.0/apps/worker2/INDEX.md

@@ -29,7 +29,8 @@
 | [NetSuite ↔ ClickUp / TOGA Opportunity Sync (API Message Queue + worker2 webhook)](features/netsuite-opportunity-sync.md) | Outbound sync from NetSuite to TOGA for the record types the Forecast2 importer pulls (opportunities first; sales/items/etc. | worker2/Worker/Netsuite.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Worker/Clickup.php, worker2/Worker/Clickup/Opportunity.php, worker2/Controller/Index.php, _underscore/Worker.php, test/@dave/NetSuite/api-message-queue/lib_amq_queue.js, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/NetSuite/api-message-queue/ue_amq_drain.js, test/@dave/NetSuite/api-message-queue/ss_amq_drain.js, test/@dave/NetSuite/api-message-queue/DEPLOY_RUNBOOK.md, test/@dave/clickup/backfill_opportunity_numbers.php, test/@dave/clickup/probe_opportunity_fields.php, test/@dave/probe_clickup_desc_match.php, test/@dave/test_model_load_behavior.php, dbchanges2/Forecast/2026-06-25a - Add unique index on Opportunities netsuiteOpportunityInternalId.sql, _underscore/Model/Forecast/Opportunity.php, test/@dave/approach/TRUE-80044.md, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
 | [NetSuite → Forecast Open-Orders Sync (salesOrder webhook → OpenOrderItems)](features/netsuite-salesorder-open-orders-sync.md) | Webhook-driven, single-record port of the legacy open-orders importer (TRUE-79142). | worker2/Worker/Netsuite/SalesOrder.php, worker2/Worker/Netsuite.php, worker2/Component/Forecast/Db/Db.php, worker2/Worker/Netsuite/Location.php, test/@dave/Junk Drawer/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/Junk Drawer/NetSuite/api-message-queue/ue_amq_invoice_resync_salesorder.js, test/@dave/probe_salesorder_rest_shape.php, test/@dave/probe_open_order_lines.php, test/@dave/check_so_status.php, test/@dave/check_so_history.php, test/@dave/probe_so_rest_lines.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_open_order_gating.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
 | [Toga → NetSuite Sales-Order Push (multi-client, mapping-driven worker)](features/netsuite-salesorder-outbound-push.md) | The **outbound** half of `worker2/Worker/Netsuite/SalesOrder.php` (everything from the `OUTBOUND PUSH (REST)` banner down) pushes a Toga sales order **into** Ne | worker2/Worker/Netsuite/SalesOrder.php, test/@Bala/tests/netsuite_salesorder_payload_tests.php, worker/crons/toga2/prudential/transmissions_to_netsuite.php |
-| [NetSuite Supporting-Record Webhook Importer (the reusable recipe)](features/netsuite-supporting-record-webhook-importer.md) | A single **repeatable recipe** for porting a legacy daily-pull NetSuite *supporting-record* importer (the lookup/dimension tables behind Forecast2 — Employees, | worker2/Worker/Netsuite/Employee.php, worker2/Worker/Netsuite/Account.php, worker2/Worker/Netsuite/Classification.php, worker2/Worker/Netsuite/Customer.php, worker2/Worker/Netsuite/Item.php, worker2/Worker/Netsuite.php, _underscore/Model/Forecast/Employee.php, _underscore/Model/Forecast/Account.php, _underscore/Model/Forecast/Classification.php, _underscore/Component/Forecast/Db/Db.php, test/@dave/test_employee_lifecycle.php, test/@dave/test_account_lifecycle.php, test/@dave/test_classification_lifecycle.php, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, worker/crons/toga2/forecast2/import_supporting_records.php |
+| [NetSuite Supporting-Record Backfill Worker (bulk reconciliation recipe)](features/netsuite-supporting-record-backfill-worker.md) | The **bulk counterpart** to the [supporting-record webhook importer](./netsuite-supporting-record-webhook-importer.md). | worker2/Worker/Netsuite/CustomerAccountTypeBackfill.php, worker2/Worker/Netsuite/ItemFulfillableBackfill.php, worker2/Worker/Netsuite/Customer.php, library/app/model/forecast2/customer.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
+| [NetSuite Supporting-Record Webhook Importer (the reusable recipe)](features/netsuite-supporting-record-webhook-importer.md) | A single **repeatable recipe** for porting a legacy daily-pull NetSuite *supporting-record* importer (the lookup/dimension tables behind Forecast2 — Employees, | worker2/Worker/Netsuite/Employee.php, worker2/Worker/Netsuite/Account.php, worker2/Worker/Netsuite/Classification.php, worker2/Worker/Netsuite/Customer.php, worker2/Worker/Netsuite/Item.php, worker2/Worker/Netsuite.php, _underscore/Model/Forecast/Employee.php, _underscore/Model/Forecast/Account.php, _underscore/Model/Forecast/Classification.php, _underscore/Component/Forecast/Db/Db.php, test/@dave/test_employee_lifecycle.php, test/@dave/test_account_lifecycle.php, test/@dave/test_classification_lifecycle.php, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/NetSuite/api-message-queue/DEPLOY_RUNBOOK.md, worker/crons/toga2/forecast2/import_supporting_records.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker2/Worker/Netsuite/CustomerAccountTypeBackfill.php, library/app/api/netsuite/rest.php, library/app/model/forecast2/customer.php |
 | [Background Email-Template Worker (_Worker_Notification_EmailTemplate)](features/notification-email-template.md) | `_Worker_Notification_EmailTemplate::Send(...)` dispatches a **stored, client-defined `EmailTemplates` row off-thread** as a background WorkerJob. | worker2/Worker/Notification/EmailTemplate.php, worker2/Worker/Client/True.php, _underscore/Model/Client/EmailTemplate.php |
 | [DB-Driven Notification (Internal) Email](features/notification-email.md) | Internal/notification emails (merge-conflict alerts, ops notices — anything system-generated, not client-facing transactional mail) are sent through one worker | worker2/Worker/Notification/Email.php, _underscore/Model/Client/EmailTemplate.php, dbchanges2/Client/2026-06-23a - EmailTemplateWrapper.sql, dbchanges2/Client_True/2026-06-23a - EmailTemplateWrapper.sql |
 | [NYCHH Asset-Tag Backfill (worker2)](features/nychh-asset-tag-backfill.md) | Keeps NYC Health & Hospitals (`Client_Nychh`) unit asset tags and MAC addresses synced from NetSuite. | worker2/Worker/Client/Nychh.php, worker2/Worker/Client/Nychh/AssetTagBackfill.php, worker/crons/toga2/netsuite/verify_fulfillment_asset_tag_sync_nychh.php, worker/crons/toga2/netsuite/backfill_all_asset_tags_from_netsuite_nychh.php |
@@ -46,4 +47,5 @@
 | [VAPI Webhook Handler (worker2 — AI-BDR end-of-call processing)](features/vapi-webhook-handler.md) | `_Worker_Vapi` ([worker2/Worker/Vapi.php](worker2/Worker/Vapi.php)) is the **PHP side of the AI-BDR call loop** — the webhook that receives VAPI's end-of-call r | worker2/Worker/Vapi.php, worker2/Worker/Ai/Bdr/Vapi.php, worker2/Controller/Index.php |
 | [WJE Freshservice Sync (worker2)](features/wje-freshservice-sync.md) | WJE ("WJE IT", helpdesk `wje.freshservice.com`) is a **Freshservice**-based help-desk client whose tickets, contacts, assets, groups, categories, and canned res | worker2/Worker/Wje.php, _underscore/Component/Api/Wje/Wje.php, _underscore/Model/Wje/Ticket.php, _underscore/Model/Wje/TicketNote.php, _underscore/Model/Wje/Contact.php, _underscore/Model/Wje/Unit.php, _underscore/Model/Wje/TicketTeam.php, _underscore/Model/Wje/TicketCategory.php, _underscore/Model/Wje/AssetType.php, _underscore/Model/Wje/PredefinedReply.php, library/app/api/wje.php, worker/crons/toga2/wje/import_supporting_records.php, worker/crons/toga2/wje/sync_togasupply_wje.php, worker/crons/notifications/reports/wje/wje_common.php, library/app/systemmonitor/wje.php, dbchanges2/Client_Wje/2024-10-04 - WjeOnboarding.sql |
 | [PHP Runtime Upgrade on Elastic Beanstalk (worker2 8.3 → 8.5 + PhpSpreadsheet 1.x → 3.x)](workflows/php-runtime-upgrade-dependency-audit.md) | The procedure used to move worker2 from **PHP 8.3 to PHP 8.5** on Elastic Beanstalk, and the dependency work that had to land first. | worker2/composer.json, worker2/composer.lock, worker2/Worker/Team/Sprint.php, worker2/Worker/Client/TowFoundation/ProcessReceipts.php, worker2/Worker/Forecast/Import.php |
+| [Running worker2 locally against real NetSuite (and what EB does instead)](workflows/running-worker2-locally.md) | How to boot **worker2** on a developer machine and run a real worker action against the **live NetSuite** account and a **local** `Forecast` database. | worker2/index.php, worker2/composer.json, worker2/.ebextensions/php_include_underscore.config, worker2/.ebextensions/git.json, worker2/.ebextensions/git.php, worker2/Config/production.ini, _underscore/Component/Api/Netsuite/Netsuite.php, dbchanges2/Logs_Client/2026-04-08_BLANK_CLIENT_LOGS_DATABASE.SQL |
 | [Ticket → ClickUp Pseudocode Planning (Talos-grounded)](workflows/ticket-to-pseudocode-planning.md) | A repeatable procedure for turning a ClickUp ticket into a reviewed, formatted implementation plan posted back to the ticket's `📝 Pseudocode` custom field. | test/@dave/clickup_md2delta.js |

package/knowledge/2.0/apps/worker2/features/elastic-beanstalk-health-monitor.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-17
+updated: 2026-08-19
 owners: [jcardinal]
 files:
   - worker2/Worker/Infrastructure/CloudWatch.php
@@ -28,7 +28,10 @@ OneUptime string-matches — OneUptime cannot compare numbers on a pushed body.
 
 The load-bearing design decision this doc records is the **paging (alarm) criteria**: page
 on EB `Degraded`/`Severe`, but **suppress the common "an occasional HTTP 500 tripped
-Degraded" false alarm** unless 5xx errors actually dominate the request mix.
+Degraded/Severe" false alarm** unless 5xx errors actually dominate the request mix (by
+share) or reach a real absolute volume. The 5xx gate applies to **both** `Degraded` and
+`Severe`, and only when EB attributes the problem **solely** to 5xx — a mixed cause always
+pages.
 
 **Critical behavior:** the method is **non-fatal and never throws** — it always returns a
 string. worker2 has **no DLQ and a 3600s SQS visibility timeout**, so any uncaught 500
@@ -53,15 +56,33 @@ itself runs non-fatal.
 Per environment, `alarm=HIGH` (page) when the EB `HealthStatus` is `Degraded` **or**
 `Severe`. `Suspended` never pages.
 
-- **`Severe` always pages.**
-- **`Degraded` for a non-5xx reason** (latency, instances down, …) **always pages.**
-- **`Degraded` attributed to HTTP 5xx errors pages only when 5xx errors dominate** — i.e.
-  the 5xx share of requests is at/above `HTTP_5XX_ALARM_RATIO` (default 0.80). An occasional
-  500 that trips Degraded is noise and must **not** page; a flood is a real problem and
-  **must** page.
-- **Tiny-sample guard:** below `HTTP_5XX_MIN_REQUEST_COUNT` (default 20) requests in the
-  window, do **not** trust the ratio — page rather than let a 2-of-3 blip read as "80%
-  failing."
+The **5xx gate applies to both `Degraded` and `Severe`.** A non-5xx or mixed-cause
+`Degraded`/`Severe` — and any env with no stated cause — **always pages**; the gate only
+ever suppresses an env EB attributes **solely** to 5xx.
+
+- **Non-5xx or mixed-cause `Degraded`/`Severe`** (latency, instances down, a failed deploy,
+  or a 5xx trickle *alongside* a real non-5xx failure) **always pages.**
+- **Solely-5xx `Degraded`/`Severe` pages only on a genuine flood** — either:
+  - the **absolute** 5xx count in the window is at/above `HTTP_5XX_ALARM_ABSOLUTE_COUNT`
+    (default 20), **or**
+  - the **5xx share** of requests is at/above `HTTP_5XX_ALARM_RATIO` (default 0.80).
+- **Zero requests in the window → no page** (nothing observed; also avoids divide-by-zero).
+
+**Why the gate now covers `Severe` too:** EB escalates an environment to `Severe` precisely
+when the 5xx share is high, so real 5xx floods reach `Severe` *first* and never touch the
+gated `Degraded` branch. When the gate applied only to `Degraded`, the 80% threshold was
+effectively dead — a real 50%-5xx `Severe` event paged despite the setting. Gating both
+tiers makes the threshold actually govern.
+
+**Why "solely" 5xx, not "any" 5xx cause:** a mixed-cause env (a 5xx trickle plus a real
+non-5xx failure such as a failed deploy) must not be silenced by a sub-threshold ratio. The
+ratio gate only applies when **every** stated cause references 5xx (and there is at least
+one); anything else pages.
+
+**Why an absolute floor in addition to the ratio:** the ratio alone is magnitude-blind —
+60k failing out of 100k reads as 60% and would be suppressed, though it is plainly a flood.
+`HTTP_5XX_ALARM_ABSOLUTE_COUNT` pages such an env regardless of share; tune it to the busiest
+monitored environment's request volume.
 
 ### Fail-safe: a blind read pages, never reads healthy
 
@@ -76,18 +97,23 @@ signal — same alarm-vs-probe split as the multi-client monitors in
 | Constant | Default | Meaning |
 |---|---|---|
 | `HEALTH_ALARM_STATUSES` | `['Degraded','Severe']` | HealthStatus values that page |
-| `HTTP_5XX_ALARM_RATIO` | `0.80` | 5xx share (0.0–1.0) at/above which a 5xx-driven `Degraded` pages |
-| `HTTP_5XX_MIN_REQUEST_COUNT` | `20` | Below this many requests in the window, page rather than trust the ratio |
+| `HTTP_5XX_ALARM_RATIO` | `0.80` | 5xx share (0.0–1.0) at/above which a solely-5xx `Degraded`/`Severe` pages |
+| `HTTP_5XX_ALARM_ABSOLUTE_COUNT` | `20` | Absolute 5xx count in the window at/above which a solely-5xx `Degraded`/`Severe` pages regardless of share |
 | `HTTP_5XX_CAUSE_MARKERS` | `['5xx','http 5']` | Case-insensitive substrings identifying a 5xx-attributed EB `Cause` |
 
 ### Implementation shape
 
 - `shouldPageForHealth(healthStatus, causes, applicationMetrics): [bool, ?float]` — returns
   the page decision and the observed 5xx ratio.
-- `causesAttributeTo5xx(causes): bool` — substring-matches `HTTP_5XX_CAUSE_MARKERS` against
-  the EB `Causes` strings (case-insensitive).
+- `causesAttributeSolelyTo5xx(causes): bool` — true only when **every** EB `Cause` references
+  5xx **and there is at least one** cause. Replaces the former `causesAttributeTo5xx()`, which
+  returned true if *any* cause mentioned 5xx and so let a mixed-cause env be gated.
+- `causeIsFivexx(cause): bool` — helper that substring-matches `HTTP_5XX_CAUSE_MARKERS`
+  against a single EB `Cause` string (case-insensitive); `causesAttributeSolelyTo5xx` requires
+  it to hold for all causes.
 - The ratio is computed as `StatusCodes.Status5xx / RequestCount` from `ApplicationMetrics`
-  (raw counts — see below), **not** by parsing the percentage out of the `Causes` text.
+  (raw counts — see below), **not** by parsing the percentage out of the `Causes` text. The
+  absolute-count floor reads `StatusCodes.Status5xx` directly.
 - `describeEnvironmentHealth` is called with
   `AttributeNames = ['HealthStatus','Status','Color','Causes','ApplicationMetrics']`.
 - Per-environment report + OneUptime payload now include the per-env `alarm` token and the
@@ -147,9 +173,10 @@ iteration:** a deploy that fails fast *without ever degrading health* is not cau
   **whole run** via the outer backstop instead of paging just that region's envs as blind.
   Candidate follow-up now that blind reads page.
 - **No first-party test harness exists in worker2** (all tests are vendor/).
-  `shouldPageForHealth()` and `causesAttributeTo5xx()` are pure and ideal to unit-test
-  (ratio at exactly 0.80; count 19 vs 20; `Severe` over a 5xx cause; empty causes; null
-  metrics) — deferred pending a harness.
+  `shouldPageForHealth()` and `causesAttributeSolelyTo5xx()` are pure and ideal to unit-test
+  (ratio at exactly 0.80; absolute count 19 vs 20; `Severe` over a solely-5xx cause; a
+  mixed 5xx + non-5xx cause; empty causes; null metrics; zero requests) — deferred pending a
+  harness.
 
 ## Gotchas / known issues
 
@@ -164,8 +191,28 @@ iteration:** a deploy that fails fast *without ever degrading health* is not cau
   — never parse the percentage out of `Causes`.**
 - **`oneuptimeUrl` is a push credential** — it arrives as a cron parameter; never log it or
   record its value in a doc.
+- **Near-idle ratio edge (residual, accepted).** With the tiny-sample guard removed, a
+  near-idle env whose window holds a tiny all-error sample (e.g. its only request being a
+  500 = 100%) still trips the ratio and pages. Optional future guard: require a minimum 5xx
+  count before the ratio applies (the absolute floor gates high volume, not this low-volume
+  edge).
 
 ## Change history
+- 2026-08-19 — Overhauled the 5xx alarm gating in `shouldPageForHealth()`. The 80% share
+  gate now applies to **both** `Degraded` and `Severe` (was `Degraded`-only, which left the
+  threshold dead because EB escalates real 5xx floods straight to `Severe` — a 50%-5xx
+  `Severe` had been paging despite the 0.80 setting). This Severe-gating trade-off was made
+  deliberately with an independent architecture second opinion on record (cto returned
+  DISAGREE-WITH-ALTERNATIVE; developer accepted). Narrowed 5xx classification: replaced
+  `causesAttributeTo5xx()` (any cause mentions 5xx) with `causesAttributeSolelyTo5xx()` (every
+  cause references 5xx, ≥1) + a `causeIsFivexx()` helper, so a mixed cause (5xx trickle beside
+  a real non-5xx failure) always pages — a security-review gap the Severe change would have
+  widened. Added `HTTP_5XX_ALARM_ABSOLUTE_COUNT` (20): a solely-5xx env pages regardless of
+  share once the absolute 5xx count reaches the floor (closes the ratio's magnitude-blindness,
+  e.g. 60k of 100k). Removed `HTTP_5XX_MIN_REQUEST_COUNT` (was 20) and its tiny-sample
+  page-anyway guard — a small number of 5xx is now judged purely on the ratio; below the
+  absolute floor, page only if share ≥ 0.80, and zero requests → no page. Accepted residual:
+  a near-idle env with a tiny all-error sample still trips the ratio. (jcardinal)
 - 2026-08-17 — Created. Documented the `ElasticBeanstalkHealth` alarm/paging criteria (page
   on `Degraded`/`Severe`; suppress a 5xx-driven `Degraded` unless the 5xx share ≥
   `HTTP_5XX_ALARM_RATIO` 0.80 with an `HTTP_5XX_MIN_REQUEST_COUNT` 20 tiny-sample guard;
@@ -178,5 +225,3 @@ iteration:** a deploy that fails fast *without ever degrading health* is not cau
   covers the reads). Recorded the reverted `['Severe']`-only + deployment-failure-detector
   direction and the accepted residual gap (a deploy that fails without degrading health).
   (jcardinal)
-</content>
-</invoke>

package/knowledge/2.0/apps/worker2/features/netsuite-opportunity-sync.md

@@ -6,8 +6,8 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-07-15
-owners: ["dfranks"]
+updated: 2026-08-19
+owners: ["dfranks", "kyalamarthi"]
 files:
   - worker2/Worker/Netsuite.php
   - worker2/Worker/Netsuite/Opportunity.php
@@ -276,6 +276,22 @@ None — platform-wide Forecast sync.
   enqueuer's `scriptnote` AUDIT log going quiet and the missing opps having zero log entries. When opps go
   missing, **check the enqueuer deployment's Status AND Audience (All Roles/All Employees = T)**, not just
   "is it Released." Backfill the gap window with `trueup_opportunities.php`.
+- **⚠ The enqueuer is deployed PER RECORD TYPE, and a record type left at `Testing` / `isdeployed = F`
+  enqueues NOTHING — forever, silently.** Same failure family as the empty-audience incident above, and it
+  is the default state of a new deployment (runbook §3: "Status Testing — fires only for you → safe"),
+  so it is missed by *forgetting* §6's promotion step rather than by doing anything wrong.
+  Live case, found 2026-08-19 (TRUE-80206): `customscript_ue_amq_enqueue` for **Customer** (deployment
+  internal id **9**, script id `customdeploy9`) sat at `status = TESTING` / `isdeployed = F` while the
+  other **15** record types were `RELEASED` / `T`. Consequence: **zero** `Netsuite/Customer/*` jobs had
+  ever run in production — the handler, the model and the migration had all been merged since 2026-07-14
+  and every one of the 9,206 rows was still NULL. Nothing errored, because nothing ran.
+  - **`Testing` fires only for the deploying user**, and `isdeployed = F` stops it entirely — so a
+    developer testing in the UI sees the sync "work" while production gets nothing.
+  - **Diagnose it with SuiteQL:** the `scriptdeployment` table exposes `status` / `isdeployed` per
+    deployment, and querying all record types side by side is what made the one-of-sixteen outlier
+    obvious. **`scriptdeployment` is NOT in the NetSuite REST record catalog**, so you can *read* it via
+    SuiteQL but you **cannot change it via the API** — promotion is a UI (or SDF) action only.
+  - Whenever a record type "isn't syncing", check this **before** reading any PHP.
 - **NetSuite SuiteQL renders timestamps in TWO different zones — do not compare them naïvely.** Stored
   datetime fields (a custom record's `created`, transaction dates, `systemnote.date`) and the NetSuite UI
   render in the **account/user TZ (Eastern here)**, but **`SYSDATE` and `scriptnote.date` render in
@@ -486,6 +502,14 @@ deprecated** for production opportunity code.
   blocked on the Aaron stakeholder decision noted above.
 
 ## Change history
+- 2026-08-19 — Recorded a second, worse instance of the "`Released` ≠ active" enqueuer-deployment failure
+  family: the AMQ enqueuer is deployed **per record type**, and **Customer** (`customdeploy9`, deployment
+  internal id 9) was still at `status = TESTING` / `isdeployed = F` while the other 15 types were
+  `RELEASED`/`T` — so **no `Netsuite/Customer/*` job had ever run in production** since the code merged on
+  2026-07-14 (root cause of TRUE-80206; resolved by setting Released + Deployed in the NetSuite UI, after
+  which the first-ever job succeeded in 0.9 s). Added the diagnostic: the `scriptdeployment` table is
+  readable via **SuiteQL** (comparing all record types side by side is what exposed the outlier) but is
+  **absent from the REST record catalog**, so the promotion can only be done in the UI/SDF. (kyalamarthi)
 - 2026-07-15 — **Decided the atomic ClickUp-task dedup approach (TRUE-80044) — planning/rework, no code
   shipped.** The non-atomic find-then-create lets concurrent NS opportunity webhooks create duplicate
   ClickUp tasks. Decided fix: a race-safe **conditional-UPDATE claim** on a new `clickupTaskId` column of

package/knowledge/2.0/apps/worker2/features/netsuite-supporting-record-backfill-worker.md

@@ -0,0 +1,126 @@
+---
+title: NetSuite Supporting-Record Backfill Worker (bulk reconciliation recipe)
+framework: "2.0"
+repo: worker2
+project: Worker
+client: shared
+type: feature
+status: active
+updated: 2026-08-19
+owners: ["kyalamarthi"]
+files:
+  - worker2/Worker/Netsuite/CustomerAccountTypeBackfill.php
+  - worker2/Worker/Netsuite/ItemFulfillableBackfill.php
+  - worker2/Worker/Netsuite/Customer.php
+  - library/app/model/forecast2/customer.php
+  - worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php
+related:
+  - ./netsuite-supporting-record-webhook-importer.md
+  - ../workflows/running-worker2-locally.md
+  - ../../../../1.0/apps/library/features/netsuite-suiteql-api-reference.md
+---
+
+## Summary
+
+The **bulk counterpart** to the [supporting-record webhook importer](./netsuite-supporting-record-webhook-importer.md).
+The webhook keeps a `Forecast` column current *going forward*, one record at a time. It cannot fix
+history, and it never fires for a NetSuite **bulk** change. A backfill worker re-derives one column
+for **every** row from NetSuite in batches, and is the reconciliation path after any mass edit.
+
+Two exist and they share one recipe: `Netsuite/ItemFulfillableBackfill/…` (the original) and
+`Netsuite/CustomerAccountTypeBackfill/Backfill` (TRUE-80206). Treat the recipe as fill-in-the-blanks —
+the interesting decisions are already made.
+
+## Key files / entry points
+
+- **`worker2/Worker/Netsuite/CustomerAccountTypeBackfill.php`** —
+  `abstract class _Worker_Netsuite_CustomerAccountTypeBackfill`.
+  Action **`Netsuite/CustomerAccountTypeBackfill/Backfill`**, parameters:
+  - `dryRun` — **defaults to `true`.** A caller must opt *in* to writing (`dryRun=false`).
+  - `netsuiteInternalId` — optional; restricts the run to a single customer (the fast way to
+    verify one record end-to-end without a full pass).
+- **`worker2/Worker/Netsuite/ItemFulfillableBackfill.php`** — the file to clone for the next one.
+
+## How it works (the recipe)
+
+1. **Page the local side.** Read the `Forecast` rows to reconcile in pages of **1,000**.
+2. **Ask NetSuite in batches.** One SuiteQL query per batch of **≤ 500** ids
+   (`… WHERE id IN (<ids>)`). Ids are local PKs / NetSuite internal ids cast with `(int)` — the
+   same cast-only injection control the webhook handlers use (there is no placeholder binding on
+   the SuiteQL REST endpoint).
+3. **Write back in batches of ≤ 500**, skipping rows already correct.
+4. **`dryRun` defaults to `true`** so an accidental invocation is a report, not a mutation.
+5. **Return a countable summary on EVERY path** (scanned / found in NetSuite / already correct /
+   updated / skipped) — worker2 actions must return a result on both the success and the
+   nothing-to-do path, or the job looks hung.
+6. **Idempotent by construction.** An immediate re-run must report *N already correct / 0 updated*.
+   That re-run is the acceptance test.
+
+### The dry run doubles as the field-type probe
+
+`CustomerAccountTypeBackfill::detectAccountTypeExpression()` issues **one** probe query using
+`BUILTIN.DF(<field>)` and falls back to the raw column if that errors, then prints the expression it
+chose **and the values it would store**. So the first dry run answers "is this custom field a
+List/Record or a scalar?" before a single row is written. Copy this when backfilling any custom
+field whose type you have not personally confirmed — guessing wrong stores internal ids
+(see the [List/Record trap](./netsuite-supporting-record-webhook-importer.md)).
+
+## Why this worker exists — do not treat it as a throwaway script
+
+`Forecast.Customers.accountType` has **no nightly backstop**. `App_Model_Forecast2_Customer`
+(`library/app/model/forecast2/customer.php`) does not declare the column, so the 2 AM
+"Forecast 2.0 — Import supporting records" cron can neither write it nor clobber it. That omission
+is deliberate (the model-omits-externally-managed-columns rule), but it means **the webhook is the
+only ongoing writer** — and:
+
+- **Bulk NetSuite changes fire no User Event.** A Mass Update, or a CSV import without
+  *"Run Server SuiteScript and Trigger Workflows"*, updates the record without enqueuing anything.
+  The webhook never sees it.
+- **Therefore: after any bulk reclassification in NetSuite, re-run this backfill.** It is the only
+  thing that will bring `Forecast` back into agreement.
+
+## Production results (2026-08-19)
+
+- **248 of 9,183** NetSuite customers have an Account Type set — **2.7%**. The other 97% are legitimately
+  blank; do not read "blank" as "broken".
+- Full backfill: **248 updated in 8.5 s**. Immediate re-run: **248 already correct / 0 updated**.
+
+## Gotchas / known issues
+
+- **⚠ The 1.0 nightly cron is effectively INSERT-ONLY for customers — latent bug, NOT fixed.**
+  `worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php:256-259` guards the update with
+  `name != … && consolidatedCustomerId != …` where it plainly means `||`, so an existing row is only
+  rewritten when **both** values changed. Renames therefore never propagate from NetSuite. Do not
+  assume the nightly cron will heal a stale `Forecast.Customers` row — it will not.
+  Evidence (prod, 2026-08-19): Forecast holds **9,206** customers vs **9,183** in NetSuite, i.e. 23
+  stale rows that were never pruned either.
+- **`dryRun` is `true` by default** — a run that "did nothing" almost certainly just needed
+  `dryRun=false`. Read the returned summary before concluding the data was already correct.
+- **SuiteQL lower-cases every column alias**, so a camelCase alias read back as camelCase is silently
+  `NULL` — this broke the first version of this backfill (every customer misreported as "blank in
+  NetSuite" while the job reported success). See the
+  [importer gotchas](./netsuite-supporting-record-webhook-importer.md).
+- Do not drive a local test of these through `_Worker::runTask` — the dev config's worker queue URL
+  points at production SQS. Call the method in-process; see
+  [running worker2 locally](../workflows/running-worker2-locally.md).
+
+## Change history
+
+- 2026-08-19 — Created after building `_Worker_Netsuite_CustomerAccountTypeBackfill` (TRUE-80206) on
+  the `ItemFulfillableBackfill` pattern: recorded the batching recipe (page 1,000 / SuiteQL `IN` 500 /
+  UPDATE 500), the `dryRun`-defaults-to-true and countable-summary-on-every-path contract, and the
+  **dry-run-as-field-type-probe** trick (`detectAccountTypeExpression()` probes `BUILTIN.DF` once and
+  falls back to the raw column). Recorded **why the worker must be kept**: `Forecast.Customers.accountType`
+  has no nightly backstop (the 1.0 model does not declare it) and bulk NetSuite edits fire no User Event,
+  so this is the only reconciliation path. Also recorded the latent 1.0 cron `&&`-should-be-`||` change
+  guard that makes the customer cron insert-only (9,206 Forecast rows vs 9,183 in NetSuite). Prod run:
+  248/9,183 (2.7%) updated in 8.5 s, re-run idempotent. (kyalamarthi)
+
+## Related docs
+
+- [NetSuite Supporting-Record Webhook Importer](./netsuite-supporting-record-webhook-importer.md) — the
+  real-time single-record sibling; the Customers variant and the List/Record `BUILTIN.DF` rule live there.
+- [Running worker2 locally against real NetSuite](../workflows/running-worker2-locally.md) — how this was
+  tested before it touched production.
+- [NetSuite SuiteQL/REST API Reference](../../../../1.0/apps/library/features/netsuite-suiteql-api-reference.md)
+  — SuiteQL mechanics, `BUILTIN.DF`, the lower-cased-alias rule, and the custom-field map.

package/knowledge/2.0/apps/worker2/features/netsuite-supporting-record-webhook-importer.md

@@ -6,8 +6,8 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-10
-owners: ["dfranks", "bala"]
+updated: 2026-08-19
+owners: ["dfranks", "bala", "kyalamarthi"]
 files:
   - worker2/Worker/Netsuite/Employee.php
   - worker2/Worker/Netsuite/Account.php
@@ -23,8 +23,14 @@ files:
   - test/@dave/test_account_lifecycle.php
   - test/@dave/test_classification_lifecycle.php
   - test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js
+  - test/@dave/NetSuite/api-message-queue/DEPLOY_RUNBOOK.md
   - worker/crons/toga2/forecast2/import_supporting_records.php
+  - worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php
+  - worker2/Worker/Netsuite/CustomerAccountTypeBackfill.php
+  - library/app/api/netsuite/rest.php
+  - library/app/model/forecast2/customer.php
 related:
+  - ./netsuite-supporting-record-backfill-worker.md
   - ./netsuite-opportunity-sync.md
   - ./netsuite-salesorder-open-orders-sync.md
   - ../../_underscore/features/forecast-sale-import.md
@@ -76,15 +82,24 @@ recipe and its two recurring variants so a new one is a fill-in-the-blanks job,
    *trivially* on merge — keep every helper method).
 5. **Enqueuer:** add `'<record>':'<record>'` to `RECORD_TYPE_MAP` in
    `test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js` (account/customer/employee/
-   classification entries already exist).
+   classification entries already exist). **Then promote that record type's `customscript_ue_amq_enqueue`
+   deployment to `Status = Released` AND `Deployed = T` in NetSuite** (runbook §6, "Before production").
+   A new deployment defaults to `Testing` / `isdeployed = F`, which fires for **nobody** — the code can be
+   merged and correct for weeks while zero jobs ever run. This step was missed for Customer and was the
+   root cause of TRUE-80206; see [the enqueuer deployment gotchas](./netsuite-opportunity-sync.md).
+   **The port is not done until you have seen a real job for the new record type in `Core.WorkerJobs`.**
 6. **e2e harness** `test/@dave/test_<record>_lifecycle.php`: clone an existing one. It drives NetSuite
    REST (check/create/get/update/delete) + the in-process worker verbs (post/put/verb-delete) + a local
    dump of the resulting `Forecast` rows. Boots via chdir `worker2` + `require index.php`.
 7. **Cron:** leave the legacy `SHOULD_SYNC_<X>` daily pull **ON** as a backstop (and, for self-ref
    records, the parent/supervisor backfill). It shares `import_supporting_records.php` with the other
    sections, so **never disable the wrapper.**
+   ⚠ **But do not assume the backstop covers your new column** — see *the backstop is narrower than it
+   looks* under Gotchas. A column the 1.0 model does not declare has **no** nightly writer at all, and
+   for Customers the cron is effectively insert-only. When there is no backstop, ship a
+   [backfill worker](./netsuite-supporting-record-backfill-worker.md) with the feature.
 
-## Two recurring variants
+## Recurring variants
 
 - **Self-referential parent/supervisor FK** (`Employees.supervisorEmployeeId`,
   `Classifications.parentClassificationId`) = **RESOLVE-OR-NULL, never recursive upsert.** Resolve the
@@ -98,9 +113,38 @@ recipe and its two recurring variants so a new one is a fill-in-the-blanks job,
   an unmapped value passes through raw. **Keep the const in sync with the shim.**
 - **Name from a hierarchical NetSuite `name`/`fullname`** = the **leaf segment**
   (`explode(' : ')` + `array_pop`), matching the SOAP/cron/Item resolver.
+- **List/Record custom field → `BUILTIN.DF()` is MANDATORY** (Customers `accountType`). Selecting a
+  List/Record custom field **raw** returns the list member's **internal id**, not its label, and nothing
+  errors — you just store `3`. Select `BUILTIN.DF(<field>)` instead. Concretely:
+  `custentity_account_type` (NetSuite Entity Field "Account Type", internal id **10719**) maps
+  **1 = Strategic Emerging · 2 = Not Strategic · 3 = Strategic Growth · 4 = Strategic Maintain ·
+  5 = Strategic Draining**. Power BI reads `Forecast.Customers.accountType` directly, so a raw id
+  renders as `3` on the report. Only **248 of 9,183** customers (2.7%) have the field set at all —
+  mostly-blank is correct, not a symptom. If you are unsure of a custom field's type, let the
+  [backfill worker's dry run probe it](./netsuite-supporting-record-backfill-worker.md).
+- **A column with TWO writers must agree byte-for-byte on format** (Customers `name`). SOAP returned a
+  single `entityId` of the form `"<entityid> <companyname>"`; SuiteQL splits those into two columns, so
+  the obvious `SELECT entityid` stores `"6096"` where the legacy 1.0 daily cron stores
+  `"6096 ODP Veyer (B2B)"`. Both write `Forecast.Customers.name`, so shipping the bare form would have
+  silently truncated the ~6 customer names a day the webhook touches. `Customer.php` therefore has a
+  private **`buildCustomerName()`** that mirrors `App_Api_Netsuite_Rest::listCustomers()`
+  (`library/app/api/netsuite/rest.php`) **including its "entityid identical to companyname → emit it
+  once" rule**. Verified byte-identical against four production-sourced names, one with an apostrophe.
+  **Before porting any record whose legacy cron still runs, diff the two writers' output on real rows.**
 
 ## Gotchas / known issues
 
+- **⚠ SuiteQL LOWER-CASES every column alias — reading it back in camelCase silently yields `NULL`.**
+  `SELECT … AS accountTypeLabel` returns the key **`accounttypelabel`**; `$row->accountTypeLabel` is then
+  `NULL` with no error, no warning and no exception. Confirmed empirically on this account
+  (`BUILTIN.DF(sd.recordtype) AS recordTypeName` came back as `recordtypename`). **Use all-lowercase
+  aliases, or read the lower-cased key.** This has now caused two separate bugs:
+  - it broke the first cut of the account-type backfill — every customer was misreported as "blank in
+    NetSuite" **while the job reported success**; only testing caught it;
+  - **still open, not fixed:** `worker2/Worker/Netsuite/Customer.php` (~line 147) selects
+    `BUILTIN.DF(custentity10) AS consolidatedName` and reads `$record->consolidatedName`, so
+    **consolidated-customer names are never set or refreshed** by the webhook. Fix it the next time that
+    handler is opened.
 - **SuiteQL over the REST client has NO server-side placeholder binding.** `_Component_Api_Netsuite::send`
   passes the query string as-is, so the **`(int)` cast on an already-int id is the injection control** —
   the same codebase-wide idiom used by `lookupId()` and the Item/Opportunity/Employee handlers. A
@@ -114,14 +158,44 @@ recipe and its two recurring variants so a new one is a fill-in-the-blanks job,
   no migration needed.
 - **`save()` does not diff** — see recipe step 2: the change-detect guard before `save()` is what makes
   a no-change webhook a true no-op.
-- **⚠ Unrelated observation, production, 2026-08-10: three of these job types are at 0% success and
-  nobody is watching.** `Core.WorkerJobs` in prod shows `Netsuite/InventoryItem/post` (266 jobs),
-  `Netsuite/InventoryItem/put` (1,014) and `Netsuite/ItemGroup/post` (5) with **zero successes**, and
-  they are still firing daily. Recorded here as a flag for whoever owns these importers — it was not
-  investigated (surfaced incidentally while auditing WorkerJobs for the Elite sales-order push).
+- **⚠ The backstop is narrower than it looks — some columns have NO nightly writer at all.**
+  Recipe step 7 says leave the legacy cron on as a backstop, and that is right, but the backstop only
+  covers columns the **1.0** model declares. `App_Model_Forecast2_Customer`
+  (`library/app/model/forecast2/customer.php`) does not declare `accountType`, so the 2 AM
+  "Forecast 2.0 — Import supporting records" cron can neither write **nor clobber** it — the webhook is
+  the **only** ongoing writer. Worse, for Customers the cron is effectively **insert-only**: its change
+  guard (`common_import_sales_from_netsuite.php:256-259`) uses `name != … && consolidatedCustomerId != …`
+  where it means `||`, so an existing row is rewritten only when *both* changed and renames never
+  propagate (prod: 9,206 Forecast customers vs 9,183 in NetSuite). **Latent bug, not fixed.** And because
+  a NetSuite **Mass Update / CSV import without "Run Server SuiteScript"** fires no User Event, a bulk
+  reclassification reaches neither writer. Ship a
+  [backfill worker](./netsuite-supporting-record-backfill-worker.md) for any column in this position and
+  re-run it after every bulk change.
+- **⚠ Unrelated observation, production, updated 2026-08-19: the item job types are still at 0% success,
+  the cause is now known, and the volume is growing.** `Core.WorkerJobs` in prod shows
+  `Netsuite/InventoryItem/post|put` at **~1,163 failures / 0 successes**, all with
+  `watchdog: exceeded maxExecutionTime without completing`, still firing daily; `NonInventoryItem` (55),
+  `ItemGroup` (9) and `ServiceItem` (2) are likewise at 0%. The 2026-08-10 snapshot of the same audit was
+  `InventoryItem/post` 266 / `InventoryItem/put` 1,014 / `ItemGroup/post` 5. **Still not investigated** —
+  recorded as a standing flag for whoever owns the item importers. The failure string points at the
+  watchdog/execution-time budget rather than at NetSuite or the data.
 
 ## Change history
 
+- 2026-08-19 — **Customers went live for the first time (TRUE-80206)** and produced four additions.
+  (1) **Root cause of the whole ticket:** the `customscript_ue_amq_enqueue` deployment for Customer
+  (deployment internal id 9, `customdeploy9`) sat at `Status = TESTING` / `isdeployed = F` while all 15
+  other record types were RELEASED/T — so **zero** `Netsuite/Customer/*` jobs had ever run in production
+  and all 9,206 rows were NULL, despite code, model and migration being merged since 2026-07-14. Recipe
+  step 5 now carries the promote-the-deployment step and the "not done until you see a real job" rule.
+  (2) **List/Record custom fields need `BUILTIN.DF()`** — the raw column stores the internal id;
+  documented `custentity_account_type` (id 10719) and its 1–5 label map. (3) **SuiteQL lower-cases column
+  aliases**, silently returning `NULL` for a camelCase read — it broke the backfill's first cut and is
+  **still live** in `Customer.php`'s `consolidatedName`. (4) **Two writers on one column must agree
+  byte-for-byte** — added `buildCustomerName()` mirroring `App_Api_Netsuite_Rest::listCustomers()` so the
+  webhook stops short of truncating names the 1.0 cron stores concatenated. Also recorded that the legacy
+  cron is **not** a universal backstop (no `accountType` in the 1.0 model; `&&`-should-be-`||` change
+  guard makes it insert-only) and refreshed the 0%-success item-job flag. (kyalamarthi)
 - 2026-08-10 — Flagged a **production health observation** (not investigated): prod `Core.WorkerJobs`
   has `Netsuite/InventoryItem/post` (266), `Netsuite/InventoryItem/put` (1,014) and
   `Netsuite/ItemGroup/post` (5) at **0% success**, still running daily and apparently unmonitored.
@@ -137,6 +211,8 @@ recipe and its two recurring variants so a new one is a fill-in-the-blanks job,
 
 ## Related docs
 
+- [NetSuite Supporting-Record Backfill Worker](./netsuite-supporting-record-backfill-worker.md) — the bulk
+  reconciliation counterpart; ship one whenever the legacy cron does not back your column.
 - [NetSuite → TOGA Opportunity Sync](./netsuite-opportunity-sync.md) — header+children transaction sibling.
 - [NetSuite → Forecast Open-Orders Sync](./netsuite-salesorder-open-orders-sync.md).
 - [Forecast.Sales NetSuite import engine](../../_underscore/features/forecast-sale-import.md) — the

package/knowledge/2.0/apps/worker2/workflows/running-worker2-locally.md

@@ -0,0 +1,123 @@
+---
+title: Running worker2 locally against real NetSuite (and what EB does instead)
+framework: "2.0"
+repo: worker2
+project: Worker
+client: shared
+type: workflow
+status: active
+updated: 2026-08-19
+owners: ["kyalamarthi"]
+files:
+  - worker2/index.php
+  - worker2/composer.json
+  - worker2/.ebextensions/php_include_underscore.config
+  - worker2/.ebextensions/git.json
+  - worker2/.ebextensions/git.php
+  - worker2/Config/production.ini
+  - _underscore/Component/Api/Netsuite/Netsuite.php
+  - dbchanges2/Logs_Client/2026-04-08_BLANK_CLIENT_LOGS_DATABASE.SQL
+related:
+  - ../features/netsuite-supporting-record-backfill-worker.md
+  - ../features/netsuite-supporting-record-webhook-importer.md
+  - ../../_underscore/features/cli-script-bootstrap.md
+  - ../../dbchanges2/workflows/nonprod-metadata-drift-repair.md
+  - ../../api2/features/environment-variable-drives-underscore-branch.md
+---
+
+## Summary
+
+How to boot **worker2** on a developer machine and run a real worker action against the **live
+NetSuite** account and a **local** `Forecast` database. Written from testing the customer
+account-type work (TRUE-80206), where dev-sandbox was unusable (see the schema gotcha below) and
+production was obviously not an option.
+
+Five things have to be true, and four of them are supplied by `.ebextensions` on Elastic Beanstalk
+— which is exactly why they are invisible until you try to run the app locally. Each step below
+pairs the local move with the EB mechanism it replaces.
+
+> For a *bare CLI script* that wants the 2.0 framework (rather than the worker app itself), use
+> [Running 2.0 code from a bare CLI script](../../_underscore/features/cli-script-bootstrap.md) —
+> its transaction and `ENVIRONMENT` traps apply here too.
+
+## Steps
+
+1. **`composer install` — `vendor/` is gitignored.** A fresh checkout has no dependencies.
+   If the machine lacks `ext-zip`, install with `--ignore-platform-req=ext-zip`; the only consumer
+   is phpspreadsheet, which a NetSuite/Forecast action never touches.
+
+2. **Make `_underscore` resolvable on the include path.** `worker2/index.php` is just
+   `require '_underscore.php'`, resolved through PHP's `include_path` — there is no relative path
+   and no composer entry to follow. Locally, prepend your `_underscore` checkout with
+   `set_include_path()` in the harness before the require.
+   *On EB:* `worker2/.ebextensions/php_include_underscore.config` writes the same entry into the
+   PHP config, and `.ebextensions/git.php` clones `_underscore` next to the app at deploy time
+   (branch = `_` + lowercased `ENVIRONMENT`, per
+   [the api2 write-up of the same mechanism](../../api2/features/environment-variable-drives-underscore-branch.md)).
+
+3. **Set `ENVIRONMENT` and add a local `Config/<name>.ini`.** `_Environment::$name` comes from the
+   `ENVIRONMENT` env var and selects `worker2/Config/<name>.ini` (the repo already carries one
+   `dev-<developer>-*.ini` per developer — clone the nearest). It needs at minimum:
+   - `[database]` pointed at `localhost`;
+   - a complete `[netsuite]` block. The key naming the signing certificate must be repointed from
+     the **server** path to your local copy under `worker2/Config/` — the shipped value is an
+     absolute server path and will not resolve on a laptop.
+
+4. **Create a local `ClientLogs` schema — authentication writes to it.**
+   `_Component_Api_Netsuite::authenticate()` logs the OAuth token request to the client-logs
+   database, so NetSuite auth **fails on a machine that has no `ClientLogs`**, before any query
+   runs. Seed one from `dbchanges2/Logs_Client/2026-04-08_BLANK_CLIENT_LOGS_DATABASE.SQL`.
+   This is the step that looks like a NetSuite credential problem and is not.
+
+5. **Create a minimal local `Forecast` fixture.** For customer work only the `Customers` table is
+   needed — build just the tables the action touches rather than cloning the whole schema.
+
+Then invoke the action **in-process** (call the static method directly). Do **not** route it
+through `_Worker::runTask` locally: the dev config's `[cloud] aws_worker_queue_url` points at the
+**production** SQS queue.
+
+## Gotchas / known issues
+
+- **⚠ dev-sandbox's `Forecast` schema is ~6 months stale — do not test recent Forecast work there.**
+  Confirmed 2026-08-19: dev-sandbox has `Employees.departmentId` (a 2026-02 migration) but is
+  **missing** `Sales.amountDue` (2026-06-09), `Opportunities.clickupTaskId` (2026-07-20) and
+  `Customers.accountType` (2026-07-14) — all merged in `dbchanges2` `_main`. The `Forecast/` folder
+  appears not to have been applied to dev-sandbox since roughly February 2026. A dry run there dies
+  on the missing column, which reads like a code bug. Use a local fixture instead, and see
+  [repairing non-prod drift](../../dbchanges2/workflows/nonprod-metadata-drift-repair.md).
+- **⚠ SECURITY — live credentials are committed in this repo; they are locations to fix, not
+  settings to copy.** Do not paste any of these values anywhere, including a knowledge doc:
+  - `worker2/.ebextensions/git.json` carries a **live GitHub personal access token**, used so
+    Elastic Beanstalk can clone `_underscore` into the staging directory at deploy time. It is in
+    **git history**, so replacing the file is *not* sufficient — the token must be **rotated**.
+  - `worker2/Config/production.ini` carries live production database credentials.
+  - `test/@dave/CLAUDE.md` contains a production database password inline in an example.
+  Recommended direction (not yet actioned): AWS Secrets Manager or EB environment properties, with
+  the ini files reduced to non-secret settings. Raise this with a senior before a rotation, since
+  the deploy clone breaks the moment the token changes.
+- **A local config that "works" proves nothing about the EB tier.** Branch and config both derive
+  from `ENVIRONMENT`, not from the EB environment's name — the api2 doc linked above documents a
+  live tier where the two disagree.
+
+## Change history
+
+- 2026-08-19 — Created while testing the customer account-type sync (TRUE-80206) with no usable
+  shared environment: recorded the five-step local boot (composer install with the `ext-zip` opt-out,
+  `set_include_path()` for `_underscore` vs. EB's `php_include_underscore.config`, `ENVIRONMENT` →
+  `Config/<name>.ini` with a local `[database]` and a repointed NetSuite certificate path, the
+  **`ClientLogs` schema that `_Component_Api_Netsuite::authenticate()` requires before any query**,
+  and a `Customers`-only `Forecast` fixture). Recorded that **dev-sandbox's `Forecast` schema is ~6
+  months behind `dbchanges2` `_main`** (missing `Sales.amountDue`, `Opportunities.clickupTaskId`,
+  `Customers.accountType`), which is why a local fixture is the fallback. Flagged committed live
+  credentials in `.ebextensions/git.json` (GitHub token — in git history, **rotation required**),
+  `Config/production.ini` and `test/@dave/CLAUDE.md`. (kyalamarthi)
+
+## Related docs
+
+- [Running 2.0 code from a bare CLI script](../../_underscore/features/cli-script-bootstrap.md) — the
+  framework-level bootstrap, transaction and `sql_mode` traps.
+- [NetSuite Supporting-Record Backfill Worker](../features/netsuite-supporting-record-backfill-worker.md)
+  — the action this harness was built to test.
+- [Repairing non-prod metadata drift](../../dbchanges2/workflows/nonprod-metadata-drift-repair.md).
+- [`ENVIRONMENT` decides the `_underscore` branch and Config file](../../api2/features/environment-variable-drives-underscore-branch.md)
+  — the api2 sibling of steps 2–3.

package/knowledge/INDEX.md

@@ -19,7 +19,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 ## 2.0 framework
 
 - **_underscore** (_Underscore) _(framework core)_ — 60 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
-- **worker2** (Worker) — 51 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
+- **worker2** (Worker) — 53 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
 - **api2** (API) — 24 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
 - **dbchanges2** (Database Changes) _(framework core)_ — 8 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
 - **toga2-supply** (TOGa Supply) — 7 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.618",
+  "version": "1.0.620",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",