toga-ai 1.0.734 → 1.0.735
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/1.0/apps/library/features/netsuite-suiteql-api-reference.md +7 -3
- package/knowledge/1.0/apps/tools/INDEX.md +1 -0
- package/knowledge/1.0/apps/tools/features/clickup-label-aliases-admin.md +106 -0
- package/knowledge/2.0/apps/worker2/INDEX.md +2 -2
- package/knowledge/2.0/apps/worker2/features/creating-worker-actions.md +28 -1
- package/knowledge/2.0/apps/worker2/features/netsuite-opportunity-client-labels.md +203 -16
- package/knowledge/2.0/apps/worker2/features/netsuite-opportunity-sync.md +16 -3
- package/knowledge/2.0/apps/worker2/features/netsuite-supporting-record-backfill-worker.md +39 -5
- package/knowledge/INDEX.md +1 -1
- package/package.json +1 -1
|
@@ -6,8 +6,8 @@ project: Library
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-
|
|
10
|
-
owners: ["dfranks", "jcardinal", "kyalamarthi"]
|
|
9
|
+
updated: 2026-09-02
|
|
10
|
+
owners: ["dfranks", "jcardinal", "kyalamarthi", "ajean"]
|
|
11
11
|
files:
|
|
12
12
|
- library/app/api/netsuite/rest.php
|
|
13
13
|
- library/ssl/netsuite_ec_key.pem
|
|
@@ -325,7 +325,7 @@ field changed, **including sublist LINE edits that fire no User Event and theref
|
|
|
325
325
|
|
|
326
326
|
| NS internalId | Script id | Description |
|
|
327
327
|
|---------------|-----------|-------------|
|
|
328
|
-
| 3149 | `CUSTBODY_END_CUSTOMER` | End Customer (on transaction) |
|
|
328
|
+
| 3149 | `CUSTBODY_END_CUSTOMER` | End Customer (on transaction) — **List/Record** (a customer ref); raw SuiteQL needs `BUILTIN.DF`, while the **REST record** returns `{id, refName}` under the **lower-cased** key `custbody_end_customer` |
|
|
329
329
|
| 7097 | `CUSTBODY_STOCKING_ORDER` | Stocking Order flag |
|
|
330
330
|
| 4968 | `custentity10` | Consolidated Customer (on customer record) — List/Record; needs `BUILTIN.DF` |
|
|
331
331
|
| 10719 | `custentity_account_type` | Account Type (on customer record) — List/Record; needs `BUILTIN.DF` |
|
|
@@ -454,6 +454,10 @@ numbers. Budget hours for multi-year runs and launch them under `nohup`/`tmux`.
|
|
|
454
454
|
|
|
455
455
|
## Change history
|
|
456
456
|
|
|
457
|
+
- 2026-09-02 — Noted the shape of `custbody_end_customer` (internalId **3149**) now that worker2 reads
|
|
458
|
+
it: it is a **List/Record** customer reference, so SuiteQL needs `BUILTIN.DF` while the REST record
|
|
459
|
+
hands back `{id, refName}` — and the payload uses the **lower-cased** script id, not the uppercase
|
|
460
|
+
form this table listed, so a reader should try both casings. (ajean)
|
|
457
461
|
- 2026-08-29 — Corrected `fetchInvoiceFile` (deploy/verify session): it is **RESTlet-based**, not a
|
|
458
462
|
REST record GET — `GET /record/v1/file/{id}` 404s (`file` record not in this account's REST Record
|
|
459
463
|
Service), SuiteQL `file` is metadata-only, `media.nl` isn't OAuth-fetchable, so a SuiteScript 2.1
|
|
@@ -3,6 +3,7 @@
|
|
|
3
3
|
| Doc | Summary | Files |
|
|
4
4
|
|-----|---------|-------|
|
|
5
5
|
| [Tools (1.0 Internal-Tools App) Architecture](architecture.md) | **Tools** is a standalone 1.0 (`App_`) application that houses many small internal tools behind simple interfaces, gated by Client_True staff persona. | tools/index.php, tools/.ebextensions/020_setup_git_libraries.config, tools/_/app/framework.php, tools/_/app/frameworkindex.php, tools/assets/img/favicon/favicon.ico, tools/assets/img/favicon/favicon-32x32.png, tools/assets/img/favicon/favicon-16x16.png, tools/assets/img/favicon/apple-touch-icon.png, tools/_/app/auth.php, tools/_/app/nav.php, tools/common/header.php, tools/common/footer.php, tools/mvc/get.php, tools/mvc/_TEMPLATE/get.php, tools/docs/ADDING_A_TOOL.md, tools/assets/css/theme.css, tools/assets/js/theme.js |
|
|
6
|
+
| [/clickup/aliases — ClickUp Label-Alias Admin (Tools → 2.0 Team schema)](features/clickup-label-aliases-admin.md) | The human side of the NetSuite→ClickUp **Stakeholders / End Customer** labels. | tools/mvc/clickup/aliases/get.php, tools/mvc/clickup/aliases/post.php, tools/_/app/clickup/aliases.php, tools/_/app/nav.php |
|
|
6
7
|
| [Developer Dashboard (ClickUp Sprint, tools /developer)](features/clickup-sprint-dashboard.md) | A native **ClickUp sprint dashboard** in the 1.0 `tools` app at route `/developer` (the **"Developer Dashboard"**, renamed from `/clickup/react` / "Clickup" — s | tools/_/app/clickup/sprint.php, tools/v2/sprints/tile/index.php, tools/v2/sprints/current/index.php, tools/v2/sprints/status-breakdown/index.php, tools/v2/sprints/worktype-breakdown/index.php, tools/v2/sprints/points-by-dev/index.php, tools/v2/sprints/burndown/index.php, tools/assets/clickup/sprint-dashboard.html, tools/mvc/developer/get.php, tools/mvc/login/get.php, tools/_/app/nav.php |
|
|
7
8
|
| [CloudFront Client Setup](features/cloudfront-client-setup.md) | An SSO-gated admin tool at **`/devops/cloudfront-clients`** in the Tools 1.0 app that onboards a client onto **CloudFront + Route 53 across multiple AWS account | tools/_/app/devops/cloudfront.php, tools/mvc/devops/cloudfront-clients/get.php, tools/mvc/devops/cloudfront-clients/post.php, tools/assets/js/cloudfront-clients.js, tools/assets/css/cloudfront-clients.css, tools/_/app/nav.php, tools/_/app/frameworkindex.php, tools/config.production.ini |
|
|
8
9
|
| [Design Demo Admin](features/design-demo-admin.md) | A self-serve admin UI at **`/design`** in the SSO-protected **Tools** app that lets the design team publish self-contained "Claude Design" HTML exports as **ver | tools/_/app/design/github.php, tools/mvc/design/get.php, tools/mvc/design/post.php, tools/assets/css/design.css, tools/assets/js/design.js, tools/_/app/frameworkindex.php, tools/_/app/nav.php, tools/composer.json |
|
|
@@ -0,0 +1,106 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: /clickup/aliases — ClickUp Label-Alias Admin (Tools → 2.0 Team schema)
|
|
3
|
+
framework: "1.0"
|
|
4
|
+
repo: tools
|
|
5
|
+
project: Tools
|
|
6
|
+
client: shared
|
|
7
|
+
type: feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-09-02
|
|
10
|
+
owners: ["ajean"]
|
|
11
|
+
files:
|
|
12
|
+
- tools/mvc/clickup/aliases/get.php
|
|
13
|
+
- tools/mvc/clickup/aliases/post.php
|
|
14
|
+
- tools/_/app/clickup/aliases.php
|
|
15
|
+
- tools/_/app/nav.php
|
|
16
|
+
related:
|
|
17
|
+
- ./mvc-data-access-patterns.md
|
|
18
|
+
- ./errors-curation-console.md
|
|
19
|
+
- ./persona-gated-navigation.md
|
|
20
|
+
- ../../../2.0/apps/worker2/features/netsuite-opportunity-client-labels.md
|
|
21
|
+
---
|
|
22
|
+
|
|
23
|
+
## Summary
|
|
24
|
+
|
|
25
|
+
The human side of the NetSuite→ClickUp **Stakeholders / End Customer** labels. worker2 inserts a row
|
|
26
|
+
in `Team.ClickUpLabelAliases` with `clickupOptionId` NULL for every company name it could not resolve
|
|
27
|
+
to a ClickUp label option; this page is where a person points that name at an existing option (or
|
|
28
|
+
clears one back to unresolved).
|
|
29
|
+
|
|
30
|
+
`Team.ClickUpLabelAliases` is **both the alias store and the work queue** — one table, two jobs. The
|
|
31
|
+
page splits the backlog accordingly: *Needs an alias* (the option probably already exists under
|
|
32
|
+
another spelling) and *Resolved* (editable, and clearable).
|
|
33
|
+
|
|
34
|
+
Derivation, matching, and how an alias is used live in the worker2 doc:
|
|
35
|
+
[ClickUp Opportunity Client vs End-Customer Labels](../../../2.0/apps/worker2/features/netsuite-opportunity-client-labels.md).
|
|
36
|
+
|
|
37
|
+
## Key files / entry points
|
|
38
|
+
|
|
39
|
+
- `tools/mvc/clickup/aliases/get.php` — the page (route `/clickup/aliases`).
|
|
40
|
+
- `tools/mvc/clickup/aliases/post.php` — the save; redirect-after-POST, no JavaScript.
|
|
41
|
+
- `tools/_/app/clickup/aliases.php` — `App_Clickup_Aliases`: `options()`, `optionsUpdatedAt()`,
|
|
42
|
+
`all()`, `setOption()`. Connection alias `db_team` (config section `[database_team]`), tables
|
|
43
|
+
`ClickUpLabelAliases` + `ClickUpLabelOptions`.
|
|
44
|
+
- `tools/_/app/nav.php` — listed under **Developers** next to `/errors`, personas
|
|
45
|
+
`Development Team` / `TOGa Technology`.
|
|
46
|
+
|
|
47
|
+
Field ids are constants on the class: Stakeholders `ec15edfd-5920-4c12-a1dc-94ede4a62438`,
|
|
48
|
+
End Customer `67bb359f-fa59-40e1-b94a-0076450bb7c0`. **The two fields are separate option-id spaces.**
|
|
49
|
+
|
|
50
|
+
## How it works
|
|
51
|
+
|
|
52
|
+
- **The dropdowns read worker2's MIRROR, not ClickUp.** Tools deliberately holds **no ClickUp
|
|
53
|
+
credential**: every ClickUp token on the platform authenticates as an individual person, so a copy
|
|
54
|
+
here would embed someone's identity in a third place. worker2 already needs a token and already
|
|
55
|
+
fetches the option lists for its own matching, so it republishes them into
|
|
56
|
+
`Team.ClickUpLabelOptions` on every digest and backfill run.
|
|
57
|
+
- **Queue order comes from SQL, not PHP** — `all()` has **no `WHERE`** and orders
|
|
58
|
+
`(clickupOptionId IS NOT NULL) ASC, opportunityCount DESC, sourceName ASC`, so unresolved rows sort
|
|
59
|
+
to the top, highest-impact first, and the page renders in queue order without re-sorting.
|
|
60
|
+
- **Hand-written SQL through `App_Database`, matching `/errors`** — a 1.0 app reading a 2.0 shared
|
|
61
|
+
schema, so none of the 2.0 model protections apply and every interpolated value is escaped or cast
|
|
62
|
+
at the call site. See [mvc-data-access-patterns](./mvc-data-access-patterns.md).
|
|
63
|
+
- **The option list is the save allowlist.** `post.php` reads `options()` once and validates the
|
|
64
|
+
submitted uuid against it. Escaping alone would stop injection but would happily store a uuid that
|
|
65
|
+
is not a real option — which the sync then reports as `alias_stale` forever.
|
|
66
|
+
- **A row's FIELD comes from the database, never from the request** — otherwise a caller could attach
|
|
67
|
+
a Stakeholders option id to an End Customer row, storing an id that does not exist on that field.
|
|
68
|
+
- **Clearing is always allowed** (retires the alias, returns the name to the queue); a **submitted id
|
|
69
|
+
that is not in the allowlist is skipped, not written** — which covers both tampering and the stale
|
|
70
|
+
option case (the page keeps flagging it instead of silently wiping the value).
|
|
71
|
+
- **Auth + CSRF are explicit:** both files call
|
|
72
|
+
`App_Auth::requireAuth(['TOGa Technology', 'Development Team'])`, and `post.php` additionally calls
|
|
73
|
+
`App_Page::validateCrossSiteRequestForgery()` — a non-GET request never reaches
|
|
74
|
+
`App_FrameworkIndex_Tools::actionHandler()`, so **the framework does not validate CSRF for POSTs**.
|
|
75
|
+
- Storage holds **plain text**; escaping happens exactly once at display. Escaping on the way in
|
|
76
|
+
would accumulate entities and break equality against the NetSuite name.
|
|
77
|
+
|
|
78
|
+
## Gotchas / known issues
|
|
79
|
+
|
|
80
|
+
- **⚠ An option you just created in ClickUp is INVISIBLE here until worker2 republishes the mirror.**
|
|
81
|
+
The dropdowns are as fresh as the last worker2 run. **Fix: run any digest — it refreshes the mirror
|
|
82
|
+
in ~2 seconds, and `dryRun` works too**, because on the digest `$dryRun` means "do not send the
|
|
83
|
+
email", *not* "do not write". This was exactly the confusion after adding 19 new options.
|
|
84
|
+
- **While the mirror is EMPTY, saving is disabled entirely** (the queue still renders). That is a
|
|
85
|
+
distinct state from a failure, and the notice says so: it fills in on the next worker2 digest or
|
|
86
|
+
backfill run. `optionsUpdatedAt()` shows the freshness date on the page.
|
|
87
|
+
- **A name with no plausible option cannot be fixed here.** ClickUp's public API cannot create a
|
|
88
|
+
`labels` option, so someone has to add it in the ClickUp UI first. Those rows still list, ordered by
|
|
89
|
+
impact.
|
|
90
|
+
- **`?saved=0` is a real outcome, not a no-op** — somebody pressed Save without changing a dropdown,
|
|
91
|
+
or every change was rejected by the allowlist. `get.php` distinguishes "not a save" from "a save
|
|
92
|
+
that changed nothing" via `isset($_GET['saved'])`.
|
|
93
|
+
- **`buildArrayOfRows()` takes its argument by reference** — assign the query result to a variable
|
|
94
|
+
first. Passing `App_Database::query(...)` inline raises *"Only variables should be passed by
|
|
95
|
+
reference"*, which the `E_ALL` handler promotes to a fatal 500.
|
|
96
|
+
- **An alias outranks exact AND fuzzy matching in worker2, and seed-if-empty makes a wrong one
|
|
97
|
+
sticky** — a mis-picked option **will** be written to tasks and no later run corrects it. Audit
|
|
98
|
+
before saving; see the worker2 doc's gotchas.
|
|
99
|
+
|
|
100
|
+
## Change history
|
|
101
|
+
- 2026-09-02 — Documented the page (built 2026-08-27, previously uncaptured), and specifically **why
|
|
102
|
+
the dropdown can look stale**: it reads `Team.ClickUpLabelOptions`, worker2's mirror, because Tools
|
|
103
|
+
holds no ClickUp credential — so newly created options only appear after the next digest/backfill
|
|
104
|
+
run, and saving is disabled while the mirror is empty. Also recorded the SQL-side queue ordering,
|
|
105
|
+
the option-list-as-allowlist save rule, field-from-DB-not-request, and the explicit CSRF call POSTs
|
|
106
|
+
need. (ajean)
|
|
@@ -27,11 +27,11 @@
|
|
|
27
27
|
| [Etilize Item Translation Import](features/etilize-item-translation-import.md) | The abstract worker class `_Worker_Etilize_ItemTranslations` imports **non-English** item text from Etilize into the client's `ItemTranslations` table. | worker2/Worker/Etilize/ItemTranslations.php |
|
|
28
28
|
| [Monitoring Framework (Orchestrator + Child Monitors)](features/monitoring-framework.md) | A unified, DB-driven monitoring framework for business-critical data flows (Compass POs, Prudential asset imports, AIG closed claims, …). | worker2/Worker/Monitor.php, worker2/Worker/Monitors/, worker2/Worker/Monitors/RateEntitlement.php, worker2/Worker/Notification/Email.php, worker2/Worker/Rate.php, dbchanges2/Core/2026-05-21 - Monitors.sql, dbchanges2/Core/2026-06-29a - Rate Entitlement Contract Monitor.sql |
|
|
29
29
|
| [NetSuite Integrations Monitor (Monitor/Operations/NetsuiteIntegrations)](features/netsuite-integrations-monitor.md) | `_Worker_Monitor_Operations::NetsuiteIntegrations()` is a cross-client health check that detects **stuck NetSuite ↔ 2.0 integrations**. | worker2/Worker/Monitor/Operations.php, dbchanges2/Core/2026-08-10a - Netsuite Integrations Monitor.sql, _underscore/Database.php, _underscore/Query.php |
|
|
30
|
-
| [ClickUp Opportunity Client vs End-Customer Labels (Stakeholders / End Customer)](features/netsuite-opportunity-client-labels.md) | The NetSuite→ClickUp opportunity task carries **two** customer labels, not one (TRUE-81010, PR #142): - **Stakeholders** = the **client** — the consolidated par | worker2/Worker/Netsuite/Opportunity.php, worker2/Worker/Netsuite/OpportunityStakeholderDigest.php, worker2/Worker/Netsuite/OpportunityStakeholderBackfill.php |
|
|
30
|
+
| [ClickUp Opportunity Client vs End-Customer Labels (Stakeholders / End Customer)](features/netsuite-opportunity-client-labels.md) | The NetSuite→ClickUp opportunity task carries **two** customer labels, not one (TRUE-81010, PR #142): - **Stakeholders** = the **client** — the consolidated par | worker2/Worker/Netsuite/Opportunity.php, worker2/Worker/Netsuite/OpportunityStakeholderDigest.php, worker2/Worker/Netsuite/OpportunityStakeholderBackfill.php, worker2/Worker/Netsuite/OpportunityEndCustomerBackfill.php, dbchanges2/Forecast/2026-09-01a - Add endCustomerName to Opportunities.sql, tools/mvc/clickup/aliases/get.php, tools/mvc/clickup/aliases/post.php, tools/_/app/clickup/aliases.php |
|
|
31
31
|
| [NetSuite ↔ ClickUp / TOGA Opportunity Sync (API Message Queue + worker2 webhook)](features/netsuite-opportunity-sync.md) | Outbound sync from NetSuite to TOGA for the record types the Forecast2 importer pulls (opportunities first; sales/items/etc. | worker2/Worker/Netsuite.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Worker/Clickup.php, worker2/Worker/Clickup/Opportunity.php, worker2/Controller/Index.php, _underscore/Worker.php, test/@dave/NetSuite/api-message-queue/lib_amq_queue.js, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/NetSuite/api-message-queue/ue_amq_drain.js, test/@dave/NetSuite/api-message-queue/ss_amq_drain.js, test/@dave/NetSuite/api-message-queue/DEPLOY_RUNBOOK.md, test/@dave/clickup/backfill_opportunity_numbers.php, test/@dave/clickup/probe_opportunity_fields.php, test/@dave/probe_clickup_desc_match.php, test/@dave/test_model_load_behavior.php, dbchanges2/Forecast/2026-06-25a - Add unique index on Opportunities netsuiteOpportunityInternalId.sql, _underscore/Model/Forecast/Opportunity.php, test/@dave/approach/TRUE-80044.md, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
|
|
32
32
|
| [NetSuite → Forecast Open-Orders Sync (salesOrder webhook → OpenOrderItems)](features/netsuite-salesorder-open-orders-sync.md) | Webhook-driven, single-record port of the legacy open-orders importer (TRUE-79142). | worker2/Worker/Netsuite/SalesOrder.php, worker2/Worker/Netsuite.php, worker2/Component/Forecast/Db/Db.php, worker2/Worker/Netsuite/Location.php, worker2/Worker/Netsuite/OpenOrderLocationBackfill.php, test/@dave/Junk Drawer/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/Junk Drawer/NetSuite/api-message-queue/ue_amq_invoice_resync_salesorder.js, test/@dave/probe_salesorder_rest_shape.php, test/@dave/probe_open_order_lines.php, test/@dave/check_so_status.php, test/@dave/check_so_history.php, test/@dave/probe_so_rest_lines.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_open_order_gating.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_open_orders.php, test/@dave/probe_ooi_location_gap.php, library/app/api/netsuite/rest.php, dbchanges2/Core/2026-08-27a - Insert - Netsuite Location SyncAll CronJob.sql |
|
|
33
33
|
| [Toga → NetSuite Sales-Order Push (multi-client, mapping-driven worker)](features/netsuite-salesorder-outbound-push.md) | The **outbound** half of `worker2/Worker/Netsuite/SalesOrder.php` (everything from the `OUTBOUND PUSH (REST)` banner down) pushes a Toga sales order **into** Ne | worker2/Worker/Netsuite/SalesOrder.php, test/@Bala/tests/netsuite_salesorder_payload_tests.php, worker/crons/toga2/prudential/transmissions_to_netsuite.php |
|
|
34
|
-
| [NetSuite Supporting-Record Backfill Worker (bulk reconciliation recipe)](features/netsuite-supporting-record-backfill-worker.md) | The **bulk counterpart** to the [supporting-record webhook importer](./netsuite-supporting-record-webhook-importer.md). | worker2/Worker/Netsuite/CustomerAccountTypeBackfill.php, worker2/Worker/Netsuite/ItemFulfillableBackfill.php, worker2/Worker/Netsuite/Customer.php, library/app/model/forecast2/customer.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
|
|
34
|
+
| [NetSuite Supporting-Record Backfill Worker (bulk reconciliation recipe)](features/netsuite-supporting-record-backfill-worker.md) | The **bulk counterpart** to the [supporting-record webhook importer](./netsuite-supporting-record-webhook-importer.md). | worker2/Worker/Netsuite/CustomerAccountTypeBackfill.php, worker2/Worker/Netsuite/ItemFulfillableBackfill.php, worker2/Worker/Netsuite/OpportunityEndCustomerBackfill.php, worker2/Worker/Netsuite/Customer.php, library/app/model/forecast2/customer.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
|
|
35
35
|
| [NetSuite Supporting-Record Webhook Importer (the reusable recipe)](features/netsuite-supporting-record-webhook-importer.md) | A single **repeatable recipe** for porting a legacy daily-pull NetSuite *supporting-record* importer (the lookup/dimension tables behind Forecast2 — Employees, | worker2/Worker/Netsuite/Employee.php, worker2/Worker/Netsuite/Account.php, worker2/Worker/Netsuite/Classification.php, worker2/Worker/Netsuite/Customer.php, worker2/Worker/Netsuite/Item.php, worker2/Worker/Netsuite.php, _underscore/Model/Forecast/Employee.php, _underscore/Model/Forecast/Account.php, _underscore/Model/Forecast/Classification.php, _underscore/Component/Forecast/Db/Db.php, test/@dave/test_employee_lifecycle.php, test/@dave/test_account_lifecycle.php, test/@dave/test_classification_lifecycle.php, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/NetSuite/api-message-queue/DEPLOY_RUNBOOK.md, worker/crons/toga2/forecast2/import_supporting_records.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker2/Worker/Netsuite/CustomerAccountTypeBackfill.php, library/app/api/netsuite/rest.php, library/app/model/forecast2/customer.php |
|
|
36
36
|
| [Background Email-Template Worker (_Worker_Notification_EmailTemplate)](features/notification-email-template.md) | `_Worker_Notification_EmailTemplate::Send(...)` dispatches a **stored, client-defined `EmailTemplates` row off-thread** as a background WorkerJob. | worker2/Worker/Notification/EmailTemplate.php, worker2/Worker/Client/True.php, _underscore/Model/Client/EmailTemplate.php |
|
|
37
37
|
| [DB-Driven Notification (Internal) Email](features/notification-email.md) | Internal/notification emails (merge-conflict alerts, ops notices — anything system-generated, not client-facing transactional mail) are sent through one worker | worker2/Worker/Notification/Email.php, _underscore/Model/Client/EmailTemplate.php, dbchanges2/Client/2026-06-23a - EmailTemplateWrapper.sql, dbchanges2/Client_True/2026-06-23a - EmailTemplateWrapper.sql |
|
|
@@ -6,7 +6,7 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-
|
|
9
|
+
updated: 2026-09-02
|
|
10
10
|
owners: [jcardinal, dfranks, mhammontree, tcox, bala, ajean]
|
|
11
11
|
files:
|
|
12
12
|
- worker2/Worker/
|
|
@@ -423,6 +423,27 @@ Rules: rename **both sides in one PR and one deploy**; grep every repo for the o
|
|
|
423
423
|
- **A new action does not run on a schedule until a `Core.CronJobs` row exists.** The schedule
|
|
424
424
|
registry is **data, not code** — shipping the PHP file changes nothing on its own. Ship the code
|
|
425
425
|
deploy and the `dbchanges2` cron-row migration together (see "Adding a new cron job" above).
|
|
426
|
+
- **⚠ If your code names a column that arrives in a HAND-APPLIED migration, GATE it on the column
|
|
427
|
+
existing.** dbchanges2 migrations are run by a human, so the deploy can legitimately land first.
|
|
428
|
+
The damage is not local: a `save()`/`SELECT` naming a missing column fails the **whole** query, so
|
|
429
|
+
a one-field enhancement can break every webhook the handler serves. (Real case: the
|
|
430
|
+
`Opportunities.endCustomerName` label field would have taken down the entire NetSuite opportunity
|
|
431
|
+
sync — see
|
|
432
|
+
[netsuite-opportunity-client-labels](./netsuite-opportunity-client-labels.md#the-deploy-gate-hasendcustomercolumn--and-why-a-label-feature-needs-one).)
|
|
433
|
+
**Pattern:** check `information_schema.COLUMNS`, memoise the answer in a **`static`** (not a
|
|
434
|
+
constant — a long-lived worker then picks the column up on its next job; it cannot change
|
|
435
|
+
mid-job), `error_log` **once** when it is missing so a silently inert feature does not go unnoticed
|
|
436
|
+
for weeks, and apply the gate at **every** site that names the column — writes *and* reads.
|
|
437
|
+
Proven in `Worker/Netsuite/ItemFulfillableBackfill.php::doesClientColumnExist` and
|
|
438
|
+
`Worker/Sync/ServiceRequest.php`.
|
|
439
|
+
- **Type-CHECK a value that came out of `json_decode`; do not `(string)`-cast it.** For an optional
|
|
440
|
+
field, `trim((string) ($payload->{$key}->name ?? ''))` fails two ways: a nested **array** casts to
|
|
441
|
+
the literal string `"Array"`, which passes a non-empty test and is then used as real data; a nested
|
|
442
|
+
**object** without `__toString` is a **fatal `Error`** that kills the job over an optional field.
|
|
443
|
+
Treat "not a string" as "not set" (`is_string()` first, return `null` otherwise) so a wrong guess
|
|
444
|
+
about the payload shape **degrades** instead of breaking. Verified on PHP 8.5: `null`, arrays and
|
|
445
|
+
objects all `TypeError` against a userland `string` parameter, while **ints coerce silently** — so
|
|
446
|
+
a wrong-typed number fails later, not at the boundary.
|
|
426
447
|
- **For a code path gated behind a RARE condition, build a dry-run entry point and fire it
|
|
427
448
|
deliberately — do not wait for the path to fire naturally.** A dry run enqueued against production
|
|
428
449
|
found a fatal bug in ~2 seconds that two weeks of green telemetry had not: the path was reached
|
|
@@ -435,6 +456,12 @@ Rules: rename **both sides in one PR and one deploy**; grep every repo for the o
|
|
|
435
456
|
commit-before-SQS transaction pattern that the worker relies on.
|
|
436
457
|
|
|
437
458
|
## Change history
|
|
459
|
+
- 2026-09-02 — Added two authoring rules learned on the End Customer label work (worker2 PR #158/#159):
|
|
460
|
+
**gate any code that names a column from a hand-applied dbchanges2 migration** on an
|
|
461
|
+
`information_schema.COLUMNS` check memoised in a `static` (an ungated write fails the whole query
|
|
462
|
+
and can take down every webhook the handler serves, not just the new field), and **type-check
|
|
463
|
+
JSON-decoded values instead of `(string)`-casting them** (a nested array becomes the literal
|
|
464
|
+
`"Array"`; a nested object is a fatal `Error`). (ajean)
|
|
438
465
|
- 2026-08-31 — Documented that **the real time budget for any hand-triggered action is 300s and
|
|
439
466
|
cannot be raised**: the JobScheduler watchdog only honours `CronJobs.maxExecutionTime` through
|
|
440
467
|
`LEFT JOIN … ON wj.cronJobId = cj.id`, and `cronJobId` is NULL for every webhook / `runTask` /
|
|
@@ -6,16 +6,24 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-
|
|
9
|
+
updated: 2026-09-02
|
|
10
10
|
owners: ["ajean"]
|
|
11
11
|
files:
|
|
12
12
|
- worker2/Worker/Netsuite/Opportunity.php
|
|
13
13
|
- worker2/Worker/Netsuite/OpportunityStakeholderDigest.php
|
|
14
14
|
- worker2/Worker/Netsuite/OpportunityStakeholderBackfill.php
|
|
15
|
+
- worker2/Worker/Netsuite/OpportunityEndCustomerBackfill.php
|
|
16
|
+
- "dbchanges2/Forecast/2026-09-01a - Add endCustomerName to Opportunities.sql"
|
|
17
|
+
- tools/mvc/clickup/aliases/get.php
|
|
18
|
+
- tools/mvc/clickup/aliases/post.php
|
|
19
|
+
- tools/_/app/clickup/aliases.php
|
|
15
20
|
related:
|
|
16
21
|
- ./netsuite-opportunity-sync.md
|
|
17
22
|
- ./creating-worker-actions.md
|
|
18
23
|
- ./notification-email.md
|
|
24
|
+
- ./netsuite-supporting-record-backfill-worker.md
|
|
25
|
+
- ../../../../1.0/apps/tools/features/clickup-label-aliases-admin.md
|
|
26
|
+
- ../../../../1.0/apps/library/features/netsuite-suiteql-api-reference.md
|
|
19
27
|
---
|
|
20
28
|
|
|
21
29
|
## Summary
|
|
@@ -23,8 +31,14 @@ related:
|
|
|
23
31
|
The NetSuite→ClickUp opportunity task carries **two** customer labels, not one (TRUE-81010, PR #142):
|
|
24
32
|
|
|
25
33
|
- **Stakeholders** = the **client** — the consolidated parent company we contract with.
|
|
26
|
-
- **End Customer** =
|
|
27
|
-
|
|
34
|
+
- **End Customer** = NetSuite's **`custbody_end_customer`** field (NS internalId 3149) when it is
|
|
35
|
+
populated; otherwise it **mirrors the client** (PR #158 + PR #159, 2026-09-01).
|
|
36
|
+
|
|
37
|
+
> **End Customer was originally derived from the opportunity `entity`. That could never be right for
|
|
38
|
+
> a partner-routed deal** — when we sell *through* a partner, the `entity` **is** the partner.
|
|
39
|
+
> Opportunity 75717 sits on entity `"Staples, Inc."`, so both labels read "Staples" while the real
|
|
40
|
+
> end customer (Legacy Health Services) existed only in the free-text `title`. Only NetSuite knows
|
|
41
|
+
> the end customer, and it has carried it all along in `custbody_end_customer` — nothing read it.
|
|
28
42
|
|
|
29
43
|
Before this change the sync pushed `entity->refName` into **Stakeholders**, flattening the two
|
|
30
44
|
levels of the NetSuite customer hierarchy into one field. Verified against production that `entity`
|
|
@@ -43,6 +57,12 @@ and a **seed-if-empty backfill** for the 761 tasks that already exist.
|
|
|
43
57
|
`Netsuite/OpportunityStakeholderDigest/Daily` (`dryRun=false`, `to=null`).
|
|
44
58
|
- `worker2/Worker/Netsuite/OpportunityStakeholderBackfill.php` — action
|
|
45
59
|
`Netsuite/OpportunityStakeholderBackfill/Backfill` (`dryRun=true`, `taskId=null`, `exactOnly=true`).
|
|
60
|
+
- `worker2/Worker/Netsuite/OpportunityEndCustomerBackfill.php` — action
|
|
61
|
+
`Netsuite/OpportunityEndCustomerBackfill/Backfill` (`dryRun=true`, `onlyMissing=true`); the
|
|
62
|
+
re-import pass that fills the new `endCustomerName` column.
|
|
63
|
+
- `dbchanges2/Forecast/2026-09-01a - Add endCustomerName to Opportunities.sql` — the column.
|
|
64
|
+
- `tools/mvc/clickup/aliases/` + `tools/_/app/clickup/aliases.php` — the human alias queue; see
|
|
65
|
+
[Tools ClickUp label-alias admin](../../../../1.0/apps/tools/features/clickup-label-aliases-admin.md).
|
|
46
66
|
|
|
47
67
|
**ClickUp field ids** on list `901111987449` (Presales Qualification), both `labels` type with ~169
|
|
48
68
|
options each: `Stakeholders` = `ec15edfd-5920-4c12-a1dc-94ede4a62438`,
|
|
@@ -58,20 +78,102 @@ So the client is resolved through Forecast:
|
|
|
58
78
|
Opportunities.customerId → Customers.consolidatedCustomerId → ConsolidatedCustomers.name
|
|
59
79
|
```
|
|
60
80
|
|
|
61
|
-
`deriveLabelNames()` then applies
|
|
81
|
+
`deriveLabelNames()` then applies these rules, each sized against production data:
|
|
82
|
+
|
|
83
|
+
1. **Stakeholders: no consolidated parent → the entity IS the client.** 186 of 759 task-bearing
|
|
84
|
+
opportunities (24.5%) have no parent. Stakeholders then receives exactly what it received before
|
|
85
|
+
the split, so the swap was not a regression for them. (25 distinct entity/parent pairs are
|
|
86
|
+
genuinely different — 83 of 573 parented opportunities — which is the case the split exists for.)
|
|
87
|
+
2. **End Customer: `Opportunities.endCustomerName` when set, else mirror the client.** Both fields
|
|
88
|
+
carry the same value rather than one being left blank — which makes an **empty field an
|
|
89
|
+
unambiguous signal that resolution FAILED**, and nothing else.
|
|
90
|
+
|
|
91
|
+
Net effect measured on production when the split shipped: Stakeholders exact-match rose from
|
|
92
|
+
**31.5% to 36.4%** (47.0% including fuzzy).
|
|
93
|
+
|
|
94
|
+
### Where End Customer comes from (2026-09-01 — this is the current rule)
|
|
95
|
+
|
|
96
|
+
**Decision: NetSuite `custbody_end_customer`, else mirror the client.** The alternative considered
|
|
97
|
+
was falling back to the `entity`. Mirroring won because neither label is ever blank either way, and
|
|
98
|
+
mirroring stops End Customer repeating near-duplicate spellings of the client.
|
|
99
|
+
|
|
100
|
+
**Accepted cost, recorded in the method comment so nobody later "fixes" it as a bug:** where the
|
|
101
|
+
entity is a *genuine* subsidiary, that detail is no longer surfaced. `"Office Depot"` /
|
|
102
|
+
`"ODP Veyer (B2B)"` (19 opportunities) and `"CDW"` / `"CDW_Northwell"` (5) now read as the parent on
|
|
103
|
+
both labels unless NetSuite names an end customer. The old **equal-names collapse** (force End
|
|
104
|
+
Customer = client when the two normalize equal) was deleted with this change — mirroring is now the
|
|
105
|
+
default, so the collapse had nothing left to do.
|
|
106
|
+
|
|
107
|
+
Why the entity could never stand in for the end customer, in numbers: of 771 task-bearing
|
|
108
|
+
opportunities, **599 (78%) had End Customer identical to Stakeholders by construction**, and most of
|
|
109
|
+
the 172 that differed were near-duplicate spellings of one company (`"Masonite"` vs
|
|
110
|
+
`"Masonite Holding, Inc."`) rather than real distinctions. `Forecast.Opportunities` has exactly one
|
|
111
|
+
customer link and had **no** end-customer column — a `%endcustomer%` column search across every
|
|
112
|
+
production schema returned zero.
|
|
113
|
+
|
|
114
|
+
**Reading the field.** `custbody_end_customer` is a **List/Record** field (a customer reference with
|
|
115
|
+
a customer number), so on the **REST record** it arrives as `{id, refName}` like
|
|
116
|
+
`custbody_sales_stage` — *not* a bare string, and no `BUILTIN.DF` is needed on the REST path (a
|
|
117
|
+
**SuiteQL** reader of the same field would need it — see
|
|
118
|
+
[SuiteQL/REST reference](../../../../1.0/apps/library/features/netsuite-suiteql-api-reference.md)).
|
|
119
|
+
`readEndCustomerRefName()` tries **both key casings** (our own reference documents it as
|
|
120
|
+
`CUSTBODY_END_CUSTOMER`, while the REST payload lower-cases its siblings); a miss returns `null`, so
|
|
121
|
+
a wrong guess degrades to the pre-existing mirror behaviour instead of breaking the label.
|
|
122
|
+
|
|
123
|
+
**Stored as the NAME, not a FK to `Customers`.** The `entity` is resolved with
|
|
124
|
+
`lookupId(…netsuiteInternalId)`, which returns NULL on a miss, and — unlike the entity — **nothing
|
|
125
|
+
upserts an end-customer row**. A FK would therefore be NULL exactly when the field *is* populated
|
|
126
|
+
but that customer has not synced yet. The label only needs the name, and the payload always carries
|
|
127
|
+
`refName`.
|
|
128
|
+
|
|
129
|
+
`importOpportunity()` reads it and threads it through `deriveLabelNames()` to all three call sites:
|
|
130
|
+
the create path, the backfill and the digest.
|
|
131
|
+
|
|
132
|
+
**The field is sparse and not uniformly a company name.** Only **30 of 776** task-bearing
|
|
133
|
+
opportunities carry it (3.9%). The rate rises with recency — 1.6% over the oldest 188, ~4–6%
|
|
134
|
+
mid-range, **20% over the newest 10** — consistent with recent adoption, so an early-chunk
|
|
135
|
+
extrapolation *understates* it. Content: 11 clean company names, 17 sub-customer paths, and 2 values
|
|
136
|
+
that are not companies at all (`"Epic Training Support and Imprivata Enrollment Supervision/Support
|
|
137
|
+
Wave 3"`, `"NW SOW -Under $25k"`). Plan for the field to keep collecting that.
|
|
138
|
+
|
|
139
|
+
### The deploy gate (`hasEndCustomerColumn()`) — and why a label feature needs one
|
|
140
|
+
|
|
141
|
+
`endCustomerName` arrives in a **dbchanges2 migration a human applies by hand**, so worker2 can
|
|
142
|
+
legitimately be running before the column lands. An **ungated write does not degrade the label — it
|
|
143
|
+
breaks the ENTIRE opportunity sync**, because `importOpportunity()`'s save names a column that does
|
|
144
|
+
not exist and *every* NetSuite webhook fails. A label enhancement must not be able to do that.
|
|
62
145
|
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
146
|
+
- `information_schema.COLUMNS` check, **memoised per process** in a `static` (the answer cannot
|
|
147
|
+
change mid-job, and the import runs per webhook) — `static`, not a constant, so a long-lived
|
|
148
|
+
worker picks the column up on its next job.
|
|
149
|
+
- `error_log`s **once** when the column is missing, because a silently inert feature goes unnoticed
|
|
150
|
+
for weeks.
|
|
151
|
+
- Applied at **three** sites: the import write, plus the backfill and digest `SELECT`s — which would
|
|
152
|
+
otherwise fail the whole query with *Unknown column* rather than omit one field.
|
|
153
|
+
- Precedent already in the repo: `Worker/Netsuite/ItemFulfillableBackfill.php::doesClientColumnExist`
|
|
154
|
+
and `Worker/Sync/ServiceRequest.php`. See
|
|
155
|
+
[creating-worker-actions](./creating-worker-actions.md#gotchas).
|
|
72
156
|
|
|
73
|
-
|
|
74
|
-
|
|
157
|
+
### End Customer re-import pass (`OpportunityEndCustomerBackfill/Backfill`)
|
|
158
|
+
|
|
159
|
+
`importOpportunity()` only runs on a **NetSuite webhook**, so an opportunity nobody touches again is
|
|
160
|
+
never re-imported and its new column stays NULL — measured **0 of 776 populated** immediately after
|
|
161
|
+
the migration. This action re-reads every opportunity from NetSuite and fills the column.
|
|
162
|
+
|
|
163
|
+
- **It writes ONE column; it is not a re-import.** Calling `importOpportunity()` per row is the
|
|
164
|
+
obvious shortcut and the wrong tool: it rewrites every header column, re-processes line items, and
|
|
165
|
+
shares the claim/label machinery — so a "fill one field" job would carry the whole sync's blast
|
|
166
|
+
radius.
|
|
167
|
+
- **Blank/absent in NetSuite is COUNTED, never written.** Leaving NULL is what makes the label mirror
|
|
168
|
+
the client, and writing `''` would be a value that normalizes to nothing and can never match an
|
|
169
|
+
option.
|
|
170
|
+
- **`onlyMissing` defaults TRUE**, so re-runs are cheap — measured **4** NetSuite calls instead of 11
|
|
171
|
+
on a second pass.
|
|
172
|
+
- **Pacing is 200 ms**, deliberately shorter than the ClickUp pacing, because the NetSuite constraint
|
|
173
|
+
is **concurrency, not requests/minute**, and the job is strictly serial.
|
|
174
|
+
- **Self-chunks using the `OpportunityStakeholderBackfill` pattern verbatim** rather than inventing a
|
|
175
|
+
second shape — see
|
|
176
|
+
[creating-worker-actions](./creating-worker-actions.md#self-chunking-the-pattern-for-work-that-cannot-finish-in-one-job).
|
|
75
177
|
|
|
76
178
|
### Daily digest (`OpportunityStakeholderDigest/Daily`)
|
|
77
179
|
|
|
@@ -116,6 +218,7 @@ passes:
|
|
|
116
218
|
| 1 | 2026-08-28 | 11 chunks, 32m | 1,112 | first live pass, 0 failures |
|
|
117
219
|
| 2 | 2026-08-31 | 5 chunks, 15m | 39 | after 21 new ClickUp options; 1,143 already set |
|
|
118
220
|
| 3 | 2026-08-31 | 6 chunks, ~17m | 191 | after 100 more options; unmatched 194 → 2, coverage **89.0%** |
|
|
221
|
+
| 4 | 2026-09-02 | 5 chunks | 26 | after the End Customer re-import; 0 unmatched, 0 fuzzy, coverage **100%** (1,552 of 1,552 slots at 776 opportunities) |
|
|
119
222
|
|
|
120
223
|
Every pass: **0 failures**, and totals always reconcile to `examined × 2`.
|
|
121
224
|
|
|
@@ -123,7 +226,48 @@ Every pass: **0 failures**, and totals always reconcile to `examined × 2`.
|
|
|
123
226
|
when write-heavy, because each filled field adds a paced request. Size expectations by mode, not by
|
|
124
227
|
the last run.
|
|
125
228
|
- **Dry run and live agreed EXACTLY on all five counters** (1112 / 29 / 231 / 166 / 0). That equality
|
|
126
|
-
is a cheap, high-value integrity check — repeat it before every live pass.
|
|
229
|
+
is a cheap, high-value integrity check — repeat it before every live pass. It has now held on
|
|
230
|
+
**every** pass across this project, including the End Customer re-import (dry run and live both
|
|
231
|
+
reporting 30 populated over 776 NetSuite reads, 5 chunks, 0 failures) — keep using it.
|
|
232
|
+
- **End-to-end verification of the case that opened the investigation:** task `868kw9t2v` now reads
|
|
233
|
+
Stakeholders `"Staples"`, End Customer `"Legacy Health Services"`.
|
|
234
|
+
|
|
235
|
+
### Operating the queue (things you will need and cannot guess)
|
|
236
|
+
|
|
237
|
+
**The Tools alias dropdown reads worker2's MIRROR, not live ClickUp.** Options you just created in
|
|
238
|
+
ClickUp are **invisible** on `/clickup/aliases` until worker2 republishes `Team.ClickUpLabelOptions`
|
|
239
|
+
(`publishLabelOptions()`), and **saving is disabled entirely** while the mirror is empty. **Any
|
|
240
|
+
digest run refreshes it in ~2 seconds, including `dryRun`** — on the digest, `$dryRun` means "do not
|
|
241
|
+
send the email", *not* "do not write". This was the fix when 19 newly-added options could not be
|
|
242
|
+
seen. Details in
|
|
243
|
+
[Tools ClickUp label-alias admin](../../../../1.0/apps/tools/features/clickup-label-aliases-admin.md).
|
|
244
|
+
|
|
245
|
+
**Seed the alias queue by hand with `recordUnresolvedAliases()`.** It is callable directly as a
|
|
246
|
+
worker action, which is how to surface names the digest's fuzzy blind spot hides (see the gotcha
|
|
247
|
+
below — still unfixed):
|
|
248
|
+
|
|
249
|
+
```json
|
|
250
|
+
{"action":"Netsuite/Opportunity/recordUnresolvedAliases",
|
|
251
|
+
"parameters":{"rows":[[fieldId, sourceName, opportunityCount]]}}
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
Its `ON DUPLICATE` refreshes `opportunityCount` and leaves `clickupOptionId` alone, so it is safe to
|
|
255
|
+
re-run and **cannot erase an alias a human assigned**. The field ids are counterintuitive:
|
|
256
|
+
Stakeholders = `ec15edfd-5920-4c12-a1dc-94ede4a62438`, End Customer =
|
|
257
|
+
`67bb359f-fa59-40e1-b94a-0076450bb7c0`.
|
|
258
|
+
|
|
259
|
+
**Correcting an ALREADY-POPULATED label takes a specific order — and clearing alone is unsafe.**
|
|
260
|
+
Seed-if-empty never overwrites, so a corrected alias changes nothing while a stale value sits in the
|
|
261
|
+
field; and clearing *before* the new derivation is deployed just refills the old value on the next
|
|
262
|
+
run. The working sequence:
|
|
263
|
+
|
|
264
|
+
1. Apply the dbchanges2 migration.
|
|
265
|
+
2. Deploy worker2.
|
|
266
|
+
3. Run the End Customer re-import (`OpportunityEndCustomerBackfill/Backfill`).
|
|
267
|
+
4. Clear the affected ClickUp fields (27 tasks by hand this round).
|
|
268
|
+
5. Run `OpportunityStakeholderBackfill/Backfill` to re-seed them.
|
|
269
|
+
|
|
270
|
+
A one-shot rem+add corrector was scoped and deliberately **not** built.
|
|
127
271
|
|
|
128
272
|
## Gotchas / known issues
|
|
129
273
|
|
|
@@ -165,6 +309,28 @@ Every pass: **0 failures**, and totals always reconcile to `examined × 2`.
|
|
|
165
309
|
use this composite form** (1,418 of 9,211) and they are precisely the hierarchical customers this
|
|
166
310
|
feature exists to distinguish. Now `/^(\d+(?::\d+)*)\s+(.*)$/` — still requiring the whitespace so
|
|
167
311
|
`"3M Company"` is not decapitated.
|
|
312
|
+
- **⚠ `splitCustomerRefName()` strips a NUMERIC prefix only — a NAME hierarchy passes through whole,
|
|
313
|
+
so AN ALIAS MUST BE KEYED ON THE FULL PATH STRING.** NetSuite sub-customers also arrive as a name
|
|
314
|
+
path, `" : "`-separated — `"ODP Veyer (B2B) : Compass Group"` — not just the numeric
|
|
315
|
+
`"6096:6308 Compass Group"` the splitter was written for. The name path survives intact, normalizes
|
|
316
|
+
to `"odp veyer b2b compass group"`, and can **never** match an option. Consequence when you queue
|
|
317
|
+
an alias: the full path is the needle `resolveLabelOption()` receives, so an alias entered as just
|
|
318
|
+
`"Compass Group"` **never fires**. **17 of the 30 populated end-customer rows are this shape.**
|
|
319
|
+
- **Type-CHECK a JSON-sourced value; never `(string)`-cast it.** `readEndCustomerRefName()` returns
|
|
320
|
+
`null` for anything that is not a string, because `trim((string) ($opp->{$key}->refName ?? ''))`
|
|
321
|
+
is wrong two ways on decoded JSON: a nested **array** casts to the literal `"Array"` (which passes
|
|
322
|
+
a non-empty test and becomes the label name), and a nested **object** with no `__toString` is a
|
|
323
|
+
**fatal `Error` that takes down the whole import** for an optional field. General rule and the
|
|
324
|
+
PHP 8.5 verification are in
|
|
325
|
+
[creating-worker-actions](./creating-worker-actions.md#gotchas).
|
|
326
|
+
- **`trimDetail()` is already safe to apply repeatedly — do not add an "ends in `...`" guard.**
|
|
327
|
+
A review asked for an early return on entries already carrying the truncation suffix, to stop
|
|
328
|
+
repeated shortening across chunks. Measured: applying it 10 times gives byte-identical output after
|
|
329
|
+
the first pass on both the ASCII and multi-byte paths, because `substr($entry, 0, CARRIED_DETAIL_MAX_CHARS)`
|
|
330
|
+
recovers exactly the same content bytes and the dots sit beyond the cut. The proposed guard would
|
|
331
|
+
return an **over-length entry unmodified** whenever it happened to end in `"..."` — which API error
|
|
332
|
+
text routinely does — defeating the byte cap and breaking the chunk chain, the exact failure the cap
|
|
333
|
+
exists to prevent. Skipped, with the reasoning posted on the PR.
|
|
168
334
|
- **Option ids ride on `$body`, not on extra positional arguments.** `buildCustomFields()` used to
|
|
169
335
|
take the Stakeholders option id as an optional 3rd argument, but `updateTask()` called it as
|
|
170
336
|
`buildCustomFields($body, $presalesUserId)` — two arguments — so that path could never set the
|
|
@@ -182,6 +348,27 @@ Every pass: **0 failures**, and totals always reconcile to `examined × 2`.
|
|
|
182
348
|
injection before they go into the digest.
|
|
183
349
|
|
|
184
350
|
## Change history
|
|
351
|
+
- 2026-09-02 — **End Customer now comes from NetSuite `custbody_end_customer`, else mirrors the
|
|
352
|
+
client (worker2 PR #158 + dbchanges2 PR #477; re-import PR #159 — all merged).** Root cause: End
|
|
353
|
+
Customer was derived from the opportunity `entity`, which **is the partner** on a partner-routed
|
|
354
|
+
deal — opp 75717 read "Staples" on both labels while the real end customer (Legacy Health Services)
|
|
355
|
+
sat only in `title`; 599 of 771 (78%) were identical to Stakeholders by construction. NetSuite has
|
|
356
|
+
carried the field all along (NS internalId 3149) and nothing read it; it is List/Record, so REST
|
|
357
|
+
delivers `{id, refName}` and `readEndCustomerRefName()` tries both key casings. Added
|
|
358
|
+
`Forecast.Opportunities.endCustomerName` (stored as the **name**, not a FK — nothing upserts an
|
|
359
|
+
end-customer row), a `hasEndCustomerColumn()` **deploy gate** at three sites (an ungated write
|
|
360
|
+
breaks the *whole* sync, not just the label), and `Netsuite/OpportunityEndCustomerBackfill/Backfill`
|
|
361
|
+
(writes one column, never calls `importOpportunity()`; blank counted not written; `onlyMissing`
|
|
362
|
+
default true; 200 ms pacing because NetSuite limits concurrency, not rate). Deleted the equal-names
|
|
363
|
+
collapse; **accepted cost:** genuine subsidiaries (ODP Veyer 19, CDW_Northwell 5) now read as the
|
|
364
|
+
parent on both labels. Recorded that the NetSuite field is sparse (30 of 776, rising with recency)
|
|
365
|
+
and not always a company; that `splitCustomerRefName()` passes a **name** path through whole so an
|
|
366
|
+
alias must be keyed on the full `" : "` string (17 of 30 rows); that the Tools alias dropdown reads
|
|
367
|
+
worker2's option **mirror**, so any digest run — `dryRun` included — is what makes new options
|
|
368
|
+
visible; the hand-seeding call for the alias queue; and the migrate → deploy → re-import → clear →
|
|
369
|
+
backfill order needed to correct an already-populated label. Verified end to end: 5 chunks, 0
|
|
370
|
+
failures over 776 reads, coverage 100% (1,552 slots), task 868kw9t2v now Staples / Legacy Health
|
|
371
|
+
Services. (ajean)
|
|
185
372
|
- 2026-08-31 — **Made the backfill self-chunking and ran three live passes (PR #157, merged).** The
|
|
186
373
|
action now works 180s and re-enqueues itself with a keyset `resume` cursor, one `WorkerJobs` row per
|
|
187
374
|
chunk labelled CONTINUES/COMPLETE — previously a full pass was declared failed by the 300s watchdog
|
|
@@ -6,7 +6,7 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-
|
|
9
|
+
updated: 2026-09-02
|
|
10
10
|
owners: ["dfranks", "kyalamarthi", "ajean"]
|
|
11
11
|
files:
|
|
12
12
|
- worker2/Worker/Netsuite.php
|
|
@@ -174,8 +174,11 @@ Shared helpers: `buildCustomFields()` (the field array, used by both create and
|
|
|
174
174
|
(the customer **name**, deliberately — not the NetSuite customer number). Plus `Sales Rep`,
|
|
175
175
|
`Presales Lead Email`, and `Presales Lead` (users field).
|
|
176
176
|
**Since 2026-08-26 the two customer *labels* fields are a separate subject** — `Stakeholders`
|
|
177
|
-
carries the **client** (the consolidated parent) and `End Customer` carries
|
|
178
|
-
`
|
|
177
|
+
carries the **client** (the consolidated parent) and `End Customer` carries NetSuite's
|
|
178
|
+
`custbody_end_customer` (stored as `Opportunities.endCustomerName`), falling back to **mirroring the
|
|
179
|
+
client** when NetSuite names none (changed 2026-09-01; it used to carry the opportunity `entity`,
|
|
180
|
+
which is the *partner* on a partner-routed deal). See
|
|
181
|
+
[ClickUp Opportunity Client vs End-Customer Labels](./netsuite-opportunity-client-labels.md);
|
|
179
182
|
do not re-derive that mapping here.
|
|
180
183
|
- The ClickUp token comes from `[clickup] token` in the worker2 config (legacy webhook received
|
|
181
184
|
it on the payload; the new contract drops it).
|
|
@@ -346,6 +349,12 @@ the legacy cron's OPPORTUNITIES section:
|
|
|
346
349
|
(same as `App_Api_Netsuite_Rest::listOpportunities`) — **not** `entityStatus` (that's the
|
|
347
350
|
sales-stage percent → `percentToCloseStatusId`). The REST record GET doesn't return that
|
|
348
351
|
letter, so the handler runs a one-row SuiteQL query for it.
|
|
352
|
+
- **`endCustomerName`** (added 2026-09-01, `dbchanges2/Forecast/2026-09-01a …`) holds NetSuite
|
|
353
|
+
`custbody_end_customer`'s `refName` — a **name, not a FK**, because nothing upserts an
|
|
354
|
+
end-customer `Customers` row. The write and both reads are gated on the column existing
|
|
355
|
+
(`hasEndCustomerColumn()`), since the migration is applied by hand and an unknown column would fail
|
|
356
|
+
the whole query and break every opportunity webhook. See
|
|
357
|
+
[the labels doc](./netsuite-opportunity-client-labels.md).
|
|
349
358
|
- Writes use raw `_Query` against `_underscore::DB_FORECAST` and **must**
|
|
350
359
|
`_Database::transactionCommit(DB_FORECAST)` (lazy-transaction gotcha).
|
|
351
360
|
|
|
@@ -652,6 +661,10 @@ deprecated** for production opportunity code.
|
|
|
652
661
|
blocked on the Aaron stakeholder decision noted above.
|
|
653
662
|
|
|
654
663
|
## Change history
|
|
664
|
+
- 2026-09-02 — `Opportunities.endCustomerName` added to the header write (dbchanges2 PR #477 /
|
|
665
|
+
worker2 PR #158), gated on the column existing; the **End Customer** label no longer comes from the
|
|
666
|
+
opportunity `entity`. Derivation stays owned by
|
|
667
|
+
[netsuite-opportunity-client-labels](./netsuite-opportunity-client-labels.md). (ajean)
|
|
655
668
|
- 2026-08-31 — **Triage knowledge from a read-only "is the opportunity sync dead?" investigation — no
|
|
656
669
|
code changed.** Added a *Triage* section: the `Core.WorkerJobs.output` `clickup` verdict census is
|
|
657
670
|
the fast, NetSuite-free answer, and it has exactly **three** verdicts in practice, with a healthy
|
|
@@ -6,11 +6,12 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-
|
|
10
|
-
owners: ["kyalamarthi"]
|
|
9
|
+
updated: 2026-09-02
|
|
10
|
+
owners: ["kyalamarthi", "ajean"]
|
|
11
11
|
files:
|
|
12
12
|
- worker2/Worker/Netsuite/CustomerAccountTypeBackfill.php
|
|
13
13
|
- worker2/Worker/Netsuite/ItemFulfillableBackfill.php
|
|
14
|
+
- worker2/Worker/Netsuite/OpportunityEndCustomerBackfill.php
|
|
14
15
|
- worker2/Worker/Netsuite/Customer.php
|
|
15
16
|
- library/app/model/forecast2/customer.php
|
|
16
17
|
- worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php
|
|
@@ -28,9 +29,10 @@ The webhook keeps a `Forecast` column current *going forward*, one record at a t
|
|
|
28
29
|
history, and it never fires for a NetSuite **bulk** change. A backfill worker re-derives one column
|
|
29
30
|
for **every** row from NetSuite in batches, and is the reconciliation path after any mass edit.
|
|
30
31
|
|
|
31
|
-
|
|
32
|
-
`Netsuite/CustomerAccountTypeBackfill/Backfill` (TRUE-80206)
|
|
33
|
-
|
|
32
|
+
Three exist and they share one recipe: `Netsuite/ItemFulfillableBackfill/…` (the original),
|
|
33
|
+
`Netsuite/CustomerAccountTypeBackfill/Backfill` (TRUE-80206) and
|
|
34
|
+
`Netsuite/OpportunityEndCustomerBackfill/Backfill` (2026-09-01, per-record REST reads instead of
|
|
35
|
+
batched SuiteQL). Treat the recipe as fill-in-the-blanks — the interesting decisions are already made.
|
|
34
36
|
|
|
35
37
|
## Key files / entry points
|
|
36
38
|
|
|
@@ -57,6 +59,26 @@ the interesting decisions are already made.
|
|
|
57
59
|
6. **Idempotent by construction.** An immediate re-run must report *N already correct / 0 updated*.
|
|
58
60
|
That re-run is the acceptance test.
|
|
59
61
|
|
|
62
|
+
### Three rules that hold for every one of these (confirmed again on the End Customer pass)
|
|
63
|
+
|
|
64
|
+
- **Write the ONE column. Never call the full importer per row.** Reusing `importOpportunity()` /
|
|
65
|
+
`importCustomer()` is the obvious shortcut and the wrong tool: it rewrites every header column,
|
|
66
|
+
re-processes child rows and shares the claim/label machinery, so a "fill one field" job inherits the
|
|
67
|
+
whole sync's blast radius. A raw `UPDATE … SET <one column>` is the point of the recipe.
|
|
68
|
+
- **Blank/absent in NetSuite is COUNTED, not written.** Leave the column NULL and report the count.
|
|
69
|
+
Writing `''` invents a value that downstream matching can never resolve, and it destroys the
|
|
70
|
+
"NULL = fall back to the default behaviour" signal.
|
|
71
|
+
- **An `onlyMissing`-style filter (default true) is what makes re-runs cheap** — measured 4 NetSuite
|
|
72
|
+
calls instead of 11 on a second pass of the End Customer backfill.
|
|
73
|
+
- **Pacing depends on WHICH limit the API enforces.** ClickUp caps requests/minute, so that backfill
|
|
74
|
+
paces 700 ms per *request*. NetSuite caps **concurrency**, and these jobs are strictly serial, so
|
|
75
|
+
200 ms is enough. Copying a pacing number between APIs is how a backfill ends up 3× slower than it
|
|
76
|
+
needs to be — or throttled.
|
|
77
|
+
- **A per-record (non-batched) variant must self-chunk.** 776 REST reads cannot finish inside the 300s
|
|
78
|
+
watchdog; reuse the `OpportunityStakeholderBackfill` chunking shape verbatim rather than inventing a
|
|
79
|
+
second one — see
|
|
80
|
+
[creating-worker-actions](./creating-worker-actions.md#self-chunking-the-pattern-for-work-that-cannot-finish-in-one-job).
|
|
81
|
+
|
|
60
82
|
### The dry run doubles as the field-type probe
|
|
61
83
|
|
|
62
84
|
`CustomerAccountTypeBackfill::detectAccountTypeExpression()` issues **one** probe query using
|
|
@@ -133,6 +155,16 @@ that sent TRUE-81213's investigation down the wrong path. **Count the jobs; do n
|
|
|
133
155
|
|
|
134
156
|
## Change history
|
|
135
157
|
|
|
158
|
+
- 2026-09-02 — Recorded a **third** instance of the recipe,
|
|
159
|
+
`Netsuite/OpportunityEndCustomerBackfill/Backfill` (worker2 PR #159), which reads **per record over
|
|
160
|
+
REST** instead of batched SuiteQL and therefore **self-chunks**. Generalized four rules it
|
|
161
|
+
re-confirmed: write the one column and never call the full importer per row (that would inherit the
|
|
162
|
+
whole sync's blast radius); **count** a blank NetSuite value instead of writing `''`; an
|
|
163
|
+
`onlyMissing` filter (default true) makes re-runs cheap (4 calls vs 11); and pace against the limit
|
|
164
|
+
the API actually enforces — NetSuite constrains **concurrency** (200 ms is fine for a serial job)
|
|
165
|
+
where ClickUp constrains requests/minute (700 ms per request). Details and the production baseline
|
|
166
|
+
live in
|
|
167
|
+
[netsuite-opportunity-client-labels](./netsuite-opportunity-client-labels.md). (ajean)
|
|
136
168
|
- 2026-08-28 — Added **"When a backfill is the WRONG tool"** after TRUE-81213 proved the point: a
|
|
137
169
|
blank-Consolidated-Business-Name defect that looked like backfill work was **self-healing**, because
|
|
138
170
|
`listCustomers()` pulls the full population with no date filter (the corrected cron re-linked 793 rows
|
|
@@ -158,6 +190,8 @@ that sent TRUE-81213's investigation down the wrong path. **Count the jobs; do n
|
|
|
158
190
|
real-time single-record sibling; the Customers variant and the List/Record `BUILTIN.DF` rule live there.
|
|
159
191
|
- [Running worker2 locally against real NetSuite](../workflows/running-worker2-locally.md) — how this was
|
|
160
192
|
tested before it touched production.
|
|
193
|
+
- [ClickUp Opportunity Client vs End-Customer Labels](./netsuite-opportunity-client-labels.md) — the
|
|
194
|
+
third instance of this recipe, its deploy gate, and what a full production pass looks like.
|
|
161
195
|
- [NetSuite SuiteQL/REST API Reference](../../../../1.0/apps/library/features/netsuite-suiteql-api-reference.md)
|
|
162
196
|
- [Forecast2 Supporting-Records Nightly Import (1.0 cron)](../../../../1.0/apps/worker/features/forecast2-supporting-records-import.md)
|
|
163
197
|
— the full pull that made TRUE-81213 self-healing, and the orphan-group diagnostic.
|
package/knowledge/INDEX.md
CHANGED
|
@@ -14,7 +14,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
14
14
|
- **walmarttechservices** (Walmart Tech Services) — 1 doc(s) → [1.0/apps/walmarttechservices/INDEX.md](1.0/apps/walmarttechservices/INDEX.md)
|
|
15
15
|
- **test** (Test) — 15 doc(s) → [1.0/apps/test/INDEX.md](1.0/apps/test/INDEX.md)
|
|
16
16
|
- **toga** (TOGa) — 3 doc(s) → [1.0/apps/toga/INDEX.md](1.0/apps/toga/INDEX.md)
|
|
17
|
-
- **tools** (Tools) —
|
|
17
|
+
- **tools** (Tools) — 16 doc(s) → [1.0/apps/tools/INDEX.md](1.0/apps/tools/INDEX.md)
|
|
18
18
|
|
|
19
19
|
## 2.0 framework
|
|
20
20
|
|
package/package.json
CHANGED