toga-ai 1.0.107 → 1.0.109
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -7,4 +7,5 @@
|
|
|
7
7
|
| [Creating Worker Actions](features/creating-worker-actions.md) | How to add a new callable Worker action — a PHP class whose `public static` methods are invoked as background jobs (via webhook, cron, or `_Worker::runTask()`). | worker2/Worker/, worker2/Controller/Index.php, _underscore/Worker.php |
|
|
8
8
|
| [Elite Freshservice Sync (worker2)](features/elite-freshservice-sync.md) | `_Worker_Elite` processes Freshservice webhook events and syncs them into TOGA 2. | worker2/Worker/Elite.php, worker2/Config/dev-kmaramreddy-laptop.ini |
|
|
9
9
|
| [Monitoring Framework (Orchestrator + Child Monitors)](features/monitoring-framework.md) | A unified, DB-driven monitoring framework for business-critical data flows (Compass POs, Prudential asset imports, AIG closed claims, …). | worker2/Worker/Monitor.php, worker2/Worker/Monitors/, worker2/Worker/Notification/Email.php, dbchanges2/Core/2026-05-21 - Monitors.sql |
|
|
10
|
+
| [NetSuite → TOGA Opportunity Sync (API Message Queue + worker2 webhook)](features/netsuite-opportunity-sync.md) | Outbound sync from NetSuite to TOGA for the record types the Forecast2 importer pulls (opportunities first; sales/items/etc. | worker2/Worker/Netsuite.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Controller/Index.php, _underscore/Worker.php, test/@dave/NetSuite/api-message-queue/lib_amq_queue.js, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/NetSuite/api-message-queue/ue_amq_drain.js, test/@dave/NetSuite/api-message-queue/ss_amq_drain.js, test/@dave/NetSuite/api-message-queue/DEPLOY_RUNBOOK.md, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
|
|
10
11
|
| [Teams Meeting Transcript Export](features/teams-transcript-export.md) | `_Worker_Team_Transcripts` (action `Team/Transcripts/Export`) polls Microsoft Graph for Teams meeting transcripts produced by a set of organizers, classifies ea | worker2/Worker/Team/Transcripts.php, worker2/Config/production.ini |
|
|
@@ -0,0 +1,156 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: NetSuite → TOGA Opportunity Sync (API Message Queue + worker2 webhook)
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
repo: worker2
|
|
5
|
+
project: Worker
|
|
6
|
+
client: shared
|
|
7
|
+
type: feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-06-16
|
|
10
|
+
owners: ["dfranks"]
|
|
11
|
+
files:
|
|
12
|
+
- worker2/Worker/Netsuite.php
|
|
13
|
+
- worker2/Worker/Netsuite/Opportunity.php
|
|
14
|
+
- worker2/Controller/Index.php
|
|
15
|
+
- _underscore/Worker.php
|
|
16
|
+
- test/@dave/NetSuite/api-message-queue/lib_amq_queue.js
|
|
17
|
+
- test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js
|
|
18
|
+
- test/@dave/NetSuite/api-message-queue/ue_amq_drain.js
|
|
19
|
+
- test/@dave/NetSuite/api-message-queue/ss_amq_drain.js
|
|
20
|
+
- test/@dave/NetSuite/api-message-queue/DEPLOY_RUNBOOK.md
|
|
21
|
+
- worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php
|
|
22
|
+
related:
|
|
23
|
+
- ../architecture.md
|
|
24
|
+
---
|
|
25
|
+
|
|
26
|
+
## Summary
|
|
27
|
+
|
|
28
|
+
Outbound sync from NetSuite to TOGA for the record types the Forecast2 importer pulls
|
|
29
|
+
(opportunities first; sales/items/etc. to follow). Replaces the legacy
|
|
30
|
+
`worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php` *pull* cron with a
|
|
31
|
+
**NetSuite-side push**: a SuiteScript enqueues each create/edit/delete into a NetSuite custom
|
|
32
|
+
record ("API Message Queue"), a scheduled SuiteScript drains it to
|
|
33
|
+
`webhook.togahub.com/netsuite`, and worker2 routes it to a per-recordType handler that upserts
|
|
34
|
+
into the `Forecast` DB. The legacy NetSuite→ClickUp opportunity-task creation is preserved as a
|
|
35
|
+
second, independently-gated concern in the same handler.
|
|
36
|
+
|
|
37
|
+
## Key files / entry points
|
|
38
|
+
|
|
39
|
+
**NetSuite side** (SuiteScript 2.1, kept in the `test` repo at
|
|
40
|
+
`test/@dave/NetSuite/api-message-queue/` — upload all four `.js` to one File Cabinet folder):
|
|
41
|
+
- `lib_amq_queue.js` — shared module: record-type/field ids, `transmit()`, `findSendableIds()`.
|
|
42
|
+
- `ue_api_msg_queue_enqueue.js` — User Event on the synced source records (Opportunity, …):
|
|
43
|
+
writes one queue row per create/edit/delete, then **kicks the scheduled drainer via `N/task`**
|
|
44
|
+
(async, non-blocking). The immediate trigger lives here — see gotchas for why.
|
|
45
|
+
- `ss_amq_drain.js` — Scheduled Script: drains rows (the on-demand kick *and* a cron safety-net).
|
|
46
|
+
Needs **two deployments**: one `Scheduled` (cron) + one `Not Scheduled` (the on-demand slot
|
|
47
|
+
`task.create` submits into).
|
|
48
|
+
- `ue_amq_drain.js` — **RETIRED.** A UE on the queue record can't fire on enqueuer-created rows
|
|
49
|
+
(UE-can't-trigger-UE); undeploy it.
|
|
50
|
+
- The custom record itself: **"API Message Queue"**, internal id **1185**.
|
|
51
|
+
|
|
52
|
+
**worker2 side:**
|
|
53
|
+
- `Worker/Netsuite.php` — `_Worker_Netsuite::Webhook($payload,$headers)` router (TRUE-79184):
|
|
54
|
+
derives `Netsuite/<RecordType>/<verb>` and `_Worker::runTask()`s it.
|
|
55
|
+
- `Worker/Netsuite/Opportunity.php` — `_Worker_Netsuite_Opportunity::{POST,PUT,DELETE}`:
|
|
56
|
+
Forecast DB upsert + ClickUp-task parity.
|
|
57
|
+
|
|
58
|
+
## How it works
|
|
59
|
+
|
|
60
|
+
1. **Enqueue (NetSuite):** on create/edit/delete of a synced record, `ue_api_msg_queue_enqueue`
|
|
61
|
+
writes a queue row — `status='Created'`, `method=POST`, `endpoint`, `payload`
|
|
62
|
+
(`{recordType, internalId, eventType, action}`), `headers`. It maps `record.Type`
|
|
63
|
+
(lowercase) → the camelCase `recordType` worker2 expects (explicit map; e.g.
|
|
64
|
+
`salesorder`→`salesOrder`), and `context.type` → CRUD verb (`create`→POST, `edit`/`xedit`→PUT,
|
|
65
|
+
`delete`→DELETE). It then **kicks the drainer**:
|
|
66
|
+
`task.create({taskType:SCHEDULED_SCRIPT, scriptId:'customscript_ss_amq_drain'})` — scriptId only,
|
|
67
|
+
no deployment id. If no on-demand slot is free the kick is skipped and the cron backs it up.
|
|
68
|
+
2. **Drain (NetSuite):** `ss_amq_drain` runs (from the kick or its cron). `transmit()` claims a row
|
|
69
|
+
(`Created`→`Sending`) before the HTTP call, POSTs to the row's endpoint, then **deletes on 2xx** /
|
|
70
|
+
sets `Retry` (+exponential backoff) / `Failed` (terminal — never auto-deleted; Jeff's data-loss rule).
|
|
71
|
+
3. **Ingest (prod):** `webhook.togahub.com/netsuite` → WebhookIngestion **Python Lambda** →
|
|
72
|
+
INSERT `Core.WorkerJobs` + SQS → returns 200 immediately. Fully async; outcome lives in
|
|
73
|
+
`WorkerJobs.isSuccess`/`failureReason`.
|
|
74
|
+
4. **Route (worker2):** the PHP worker runs `_Worker_Netsuite::Webhook`, which
|
|
75
|
+
`_Worker::runTask('Netsuite/Opportunity/<verb>', {internalId})`.
|
|
76
|
+
5. **Handle (worker2):** `_Worker_Netsuite_Opportunity` fetches the record from NetSuite by id
|
|
77
|
+
and upserts it into `Forecast.Opportunities` (+ `OpportunityItems`), and — only when a
|
|
78
|
+
Presales Lead is set — creates the ClickUp opportunity task.
|
|
79
|
+
|
|
80
|
+
## Data model
|
|
81
|
+
|
|
82
|
+
Writes `Forecast.Opportunities` (header) + `Forecast.OpportunityItems` (children), faithful to
|
|
83
|
+
the legacy cron's OPPORTUNITIES section:
|
|
84
|
+
- Upsert header by `netsuiteOpportunityInternalId`; items insert/update/delete by `line`; no
|
|
85
|
+
header delete; children deleted before the header on DELETE.
|
|
86
|
+
- Lookups: `customerId`/`salesRepEmployeeId` → null on miss; `forecastCategoryId`/`salesStageId`/
|
|
87
|
+
`percentToCloseStatusId` → **create-on-miss** (+name update); unknown line item → **throws**
|
|
88
|
+
(Retries until the item import runs).
|
|
89
|
+
- `netsuiteStatus` (NOT-NULL enum `'In Progress'|'Closed - Won'|'Closed - Lost'`) comes from
|
|
90
|
+
**SuiteQL `transaction.status`** letter mapped `A→In Progress, C→Closed - Won, D→Closed - Lost`
|
|
91
|
+
(same as `App_Api_Netsuite_Rest::listOpportunities`) — **not** `entityStatus` (that's the
|
|
92
|
+
sales-stage percent → `percentToCloseStatusId`). The REST record GET doesn't return that
|
|
93
|
+
letter, so the handler runs a one-row SuiteQL query for it.
|
|
94
|
+
- Writes use raw `_Query` against `_underscore::DB_FORECAST` and **must**
|
|
95
|
+
`_Database::transactionCommit(DB_FORECAST)` (lazy-transaction gotcha).
|
|
96
|
+
|
|
97
|
+
## Client variations
|
|
98
|
+
|
|
99
|
+
None — platform-wide Forecast sync.
|
|
100
|
+
|
|
101
|
+
## Gotchas / known issues
|
|
102
|
+
|
|
103
|
+
- **A User Event can't trigger another User Event → the immediate trigger lives in the enqueuer.**
|
|
104
|
+
The enqueuer (a UE) creating the queue row does NOT fire any UE on the queue record (only the
|
|
105
|
+
*scheduled* drainer's `submitFields` does). So a drain-UE on the queue record never runs for
|
|
106
|
+
enqueued rows — the enqueuer kicks the drainer itself via `task.create`. (`ue_amq_drain.js` is
|
|
107
|
+
retired for exactly this reason. Evidence: its log only ever showed `xedit`/`Sending`, never `create`.)
|
|
108
|
+
- **`task.create(SCHEDULED_SCRIPT)` needs a `Not Scheduled` deployment.** A `Scheduled` deployment is
|
|
109
|
+
reserved for its recurrence → `NO_DEPLOYMENTS_AVAILABLE`; a `Testing` deployment can't be submitted
|
|
110
|
+
at all. Give `ss_amq_drain` a dedicated `Not Scheduled` deployment as the on-demand slot (add more
|
|
111
|
+
for concurrency) and keep a `Scheduled` one for the cron. Submit by **scriptId only** — the
|
|
112
|
+
deployment id is prefix-doubled (`customdeploycustomdeploy_ss_amq_drain`).
|
|
113
|
+
- **worker2 routes EVERY request to `_Controller_Index::worker()`** — `worker2/_.php` sets
|
|
114
|
+
`DEFAULT_CONTROLLER_METHOD = ['_Controller_Index','worker']`, which short-circuits `_Route`'s
|
|
115
|
+
URI matching. There is no per-URL routing, and `.htaccess` denies direct `.php` files, so you
|
|
116
|
+
can't add a real `/netsuite` route or a standalone script — everything enters through `worker()`.
|
|
117
|
+
- **`_Worker::runTask()` debug mode swallows the sub-request result.** In debug it self-POSTs
|
|
118
|
+
`_Config::api('worker')` and does **not** check the response (`_underscore/Worker.php:12`). So a
|
|
119
|
+
failing sub-action (e.g. `Netsuite/Opportunity/PUT` throwing) is invisible: the outer webhook
|
|
120
|
+
still returns 200, the queue row clears, and the DB write never happened. In prod the same
|
|
121
|
+
failure surfaces in `WorkerJobs.isSuccess=0`/`failureReason`. To debug locally, POST the
|
|
122
|
+
`{action,parameters}` envelope straight to the worker and read the response, or check logs —
|
|
123
|
+
don't trust the 200.
|
|
124
|
+
- **Local dev needs `[api] worker` in the worker2 config** — debug-mode `runTask` requires
|
|
125
|
+
`_Config::api('worker')`. Set `worker = "http://worker2/"` (the local Apache vhost) plus the
|
|
126
|
+
standard `[api] endpoint`/`uuid_*`. Without `[api]` you get
|
|
127
|
+
`Configuration group [api] has not been defined`.
|
|
128
|
+
- **No WebhookIngestion Lambda locally.** Dev points the enqueuer at an ngrok tunnel to local
|
|
129
|
+
worker2 via the `DEV_OVERRIDE` block, which also wraps the body in worker2's legacy
|
|
130
|
+
`{action,parameters}` envelope (since there's no Lambda to convert `/netsuite` → an action).
|
|
131
|
+
Set `DEV_OVERRIDE.enabled=false` for prod.
|
|
132
|
+
- **NetSuite doubled-prefix ids.** The queue record and its fields were created with the full
|
|
133
|
+
custom prefix typed into the ID box, so the real ids are doubled:
|
|
134
|
+
type `customrecordcustomrecord_api_msg_queue`, fields `custrecordcustrecord_amq_*`. The
|
|
135
|
+
single-prefix forms do not exist — use the doubled ids everywhere (SuiteScript, SuiteQL, REST).
|
|
136
|
+
- **REST opportunity line `itemType` is an object** (`{id,refName}`), not a string — casting it to
|
|
137
|
+
string throws `Object of class stdClass could not be converted to string`. Normalize before
|
|
138
|
+
matching.
|
|
139
|
+
- **Drain dedup** uses a `Sending` claim status: `transmit()` flips `Created`→`Sending` before the
|
|
140
|
+
HTTP call so a concurrent drainer pass skips it (`Sending` ∉ sendable), narrowing the double-send
|
|
141
|
+
window to the load→claim gap. `findSendableIds` must list every sendable status
|
|
142
|
+
(`Created`,`Pending`,`Retry`,`Sending`) — omitting `Created` silently matches nothing (`candidates:0`).
|
|
143
|
+
|
|
144
|
+
## Change history
|
|
145
|
+
- 2026-06-16 — Trigger correction: the immediate trigger moved INTO the enqueuer (`task.create`
|
|
146
|
+
the drainer) after discovering a UE can't trigger a UE — `ue_amq_drain` retired. `task.create`
|
|
147
|
+
needs a `Not Scheduled` deployment of `ss_amq_drain` (else `NO_DEPLOYMENTS_AVAILABLE`); submit by
|
|
148
|
+
scriptId only (deployment id is prefix-doubled). Fixed an `itemType`-as-object crash in the
|
|
149
|
+
importer. POST/DELETE coded but only PUT (edit) verified live. (dfranks)
|
|
150
|
+
- 2026-06-16 — Built the queue enqueuer/drainer SuiteScripts, the worker2 `/netsuite` router, and
|
|
151
|
+
the `_Worker_Netsuite_Opportunity` Forecast importer (header + items + status), with ClickUp
|
|
152
|
+
parity. Verified opportunity 5483750 imports end-to-end in dev via ngrok. (dfranks)
|
|
153
|
+
|
|
154
|
+
## Related docs
|
|
155
|
+
- `2.0/apps/worker2/architecture.md` — the WorkerJobs/SQS/Lambda model and always-HTTP-200 worker.
|
|
156
|
+
- `2.0/apps/_underscore/architecture.md` — the `_Database::register()` lazy-transaction gotcha.
|
package/knowledge/INDEX.md
CHANGED
|
@@ -15,7 +15,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
15
15
|
## 2.0 framework
|
|
16
16
|
|
|
17
17
|
- **_underscore** (_Underscore) _(framework core)_ — 7 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
|
|
18
|
-
- **worker2** (Worker) —
|
|
18
|
+
- **worker2** (Worker) — 7 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
|
|
19
19
|
- **api2** (API) — 3 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
|
|
20
20
|
- **dbchanges2** (Database Changes) _(framework core)_ — 1 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
|
|
21
21
|
- **toga2-supply** (TOGa Supply) — 2 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)
|
package/package.json
CHANGED