toga-ai 1.0.107 → 1.0.108
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -7,4 +7,5 @@
|
|
|
7
7
|
| [Creating Worker Actions](features/creating-worker-actions.md) | How to add a new callable Worker action — a PHP class whose `public static` methods are invoked as background jobs (via webhook, cron, or `_Worker::runTask()`). | worker2/Worker/, worker2/Controller/Index.php, _underscore/Worker.php |
|
|
8
8
|
| [Elite Freshservice Sync (worker2)](features/elite-freshservice-sync.md) | `_Worker_Elite` processes Freshservice webhook events and syncs them into TOGA 2. | worker2/Worker/Elite.php, worker2/Config/dev-kmaramreddy-laptop.ini |
|
|
9
9
|
| [Monitoring Framework (Orchestrator + Child Monitors)](features/monitoring-framework.md) | A unified, DB-driven monitoring framework for business-critical data flows (Compass POs, Prudential asset imports, AIG closed claims, …). | worker2/Worker/Monitor.php, worker2/Worker/Monitors/, worker2/Worker/Notification/Email.php, dbchanges2/Core/2026-05-21 - Monitors.sql |
|
|
10
|
+
| [NetSuite → TOGA Opportunity Sync (API Message Queue + worker2 webhook)](features/netsuite-opportunity-sync.md) | Outbound sync from NetSuite to TOGA for the record types the Forecast2 importer pulls (opportunities first; sales/items/etc. | worker2/Worker/Netsuite.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Controller/Index.php, _underscore/Worker.php, test/@dave/NetSuite/api-message-queue/lib_amq_queue.js, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/NetSuite/api-message-queue/ue_amq_drain.js, test/@dave/NetSuite/api-message-queue/ss_amq_drain.js, test/@dave/NetSuite/api-message-queue/DEPLOY_RUNBOOK.md, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
|
|
10
11
|
| [Teams Meeting Transcript Export](features/teams-transcript-export.md) | `_Worker_Team_Transcripts` (action `Team/Transcripts/Export`) polls Microsoft Graph for Teams meeting transcripts produced by a set of organizers, classifies ea | worker2/Worker/Team/Transcripts.php, worker2/Config/production.ini |
|
|
@@ -0,0 +1,140 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: NetSuite → TOGA Opportunity Sync (API Message Queue + worker2 webhook)
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
repo: worker2
|
|
5
|
+
project: Worker
|
|
6
|
+
client: shared
|
|
7
|
+
type: feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-06-16
|
|
10
|
+
owners: ["dfranks"]
|
|
11
|
+
files:
|
|
12
|
+
- worker2/Worker/Netsuite.php
|
|
13
|
+
- worker2/Worker/Netsuite/Opportunity.php
|
|
14
|
+
- worker2/Controller/Index.php
|
|
15
|
+
- _underscore/Worker.php
|
|
16
|
+
- test/@dave/NetSuite/api-message-queue/lib_amq_queue.js
|
|
17
|
+
- test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js
|
|
18
|
+
- test/@dave/NetSuite/api-message-queue/ue_amq_drain.js
|
|
19
|
+
- test/@dave/NetSuite/api-message-queue/ss_amq_drain.js
|
|
20
|
+
- test/@dave/NetSuite/api-message-queue/DEPLOY_RUNBOOK.md
|
|
21
|
+
- worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php
|
|
22
|
+
related:
|
|
23
|
+
- ../architecture.md
|
|
24
|
+
---
|
|
25
|
+
|
|
26
|
+
## Summary
|
|
27
|
+
|
|
28
|
+
Outbound sync from NetSuite to TOGA for the record types the Forecast2 importer pulls
|
|
29
|
+
(opportunities first; sales/items/etc. to follow). Replaces the legacy
|
|
30
|
+
`worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php` *pull* cron with a
|
|
31
|
+
**NetSuite-side push**: a SuiteScript enqueues each create/edit/delete into a NetSuite custom
|
|
32
|
+
record ("API Message Queue"), a scheduled SuiteScript drains it to
|
|
33
|
+
`webhook.togahub.com/netsuite`, and worker2 routes it to a per-recordType handler that upserts
|
|
34
|
+
into the `Forecast` DB. The legacy NetSuite→ClickUp opportunity-task creation is preserved as a
|
|
35
|
+
second, independently-gated concern in the same handler.
|
|
36
|
+
|
|
37
|
+
## Key files / entry points
|
|
38
|
+
|
|
39
|
+
**NetSuite side** (SuiteScript 2.1, kept in the `test` repo at
|
|
40
|
+
`test/@dave/NetSuite/api-message-queue/` — upload all four `.js` to one File Cabinet folder):
|
|
41
|
+
- `lib_amq_queue.js` — shared module: record-type/field ids, `transmit()`, `findSendableIds()`.
|
|
42
|
+
- `ue_api_msg_queue_enqueue.js` — User Event on the synced source records (Opportunity, …):
|
|
43
|
+
writes one queue row per create/edit/delete.
|
|
44
|
+
- `ue_amq_drain.js` — User Event on the **API Message Queue** record: async trigger — kicks the
|
|
45
|
+
scheduled drainer via `N/task` (no inline HTTP, so it never blocks the source save).
|
|
46
|
+
- `ss_amq_drain.js` — Scheduled Script: catch-up drainer (retries + anything the UE missed).
|
|
47
|
+
- The custom record itself: **"API Message Queue"**, internal id **1185**.
|
|
48
|
+
|
|
49
|
+
**worker2 side:**
|
|
50
|
+
- `Worker/Netsuite.php` — `_Worker_Netsuite::Webhook($payload,$headers)` router (TRUE-79184):
|
|
51
|
+
derives `Netsuite/<RecordType>/<verb>` and `_Worker::runTask()`s it.
|
|
52
|
+
- `Worker/Netsuite/Opportunity.php` — `_Worker_Netsuite_Opportunity::{POST,PUT,DELETE}`:
|
|
53
|
+
Forecast DB upsert + ClickUp-task parity.
|
|
54
|
+
|
|
55
|
+
## How it works
|
|
56
|
+
|
|
57
|
+
1. **Enqueue (NetSuite):** on create/edit/delete of a synced record, `ue_api_msg_queue_enqueue`
|
|
58
|
+
writes a queue row — `status='Created'`, `method=POST`, `endpoint`, `payload`
|
|
59
|
+
(`{recordType, internalId, eventType, action}`), `headers`. It maps `record.Type`
|
|
60
|
+
(lowercase) → the camelCase `recordType` worker2 expects (explicit map; e.g.
|
|
61
|
+
`salesorder`→`salesOrder`), and `context.type` → CRUD verb (`create`→POST, `edit`/`xedit`→PUT,
|
|
62
|
+
`delete`→DELETE).
|
|
63
|
+
2. **Trigger + drain (NetSuite):** the queue record's `ue_amq_drain` UE fires the scheduled
|
|
64
|
+
`ss_amq_drain` via `N/task` (async); `ss_amq_drain` also runs on its own schedule.
|
|
65
|
+
`transmit()` claims a row (`Created`→`Sending`) before the HTTP call, POSTs to the row's
|
|
66
|
+
endpoint, then **deletes on 2xx** / sets `Retry` (+exponential backoff) / `Failed`
|
|
67
|
+
(terminal — never auto-deleted; Jeff's data-loss rule).
|
|
68
|
+
3. **Ingest (prod):** `webhook.togahub.com/netsuite` → WebhookIngestion **Python Lambda** →
|
|
69
|
+
INSERT `Core.WorkerJobs` + SQS → returns 200 immediately. Fully async; outcome lives in
|
|
70
|
+
`WorkerJobs.isSuccess`/`failureReason`.
|
|
71
|
+
4. **Route (worker2):** the PHP worker runs `_Worker_Netsuite::Webhook`, which
|
|
72
|
+
`_Worker::runTask('Netsuite/Opportunity/<verb>', {internalId})`.
|
|
73
|
+
5. **Handle (worker2):** `_Worker_Netsuite_Opportunity` fetches the record from NetSuite by id
|
|
74
|
+
and upserts it into `Forecast.Opportunities` (+ `OpportunityItems`), and — only when a
|
|
75
|
+
Presales Lead is set — creates the ClickUp opportunity task.
|
|
76
|
+
|
|
77
|
+
## Data model
|
|
78
|
+
|
|
79
|
+
Writes `Forecast.Opportunities` (header) + `Forecast.OpportunityItems` (children), faithful to
|
|
80
|
+
the legacy cron's OPPORTUNITIES section:
|
|
81
|
+
- Upsert header by `netsuiteOpportunityInternalId`; items insert/update/delete by `line`; no
|
|
82
|
+
header delete; children deleted before the header on DELETE.
|
|
83
|
+
- Lookups: `customerId`/`salesRepEmployeeId` → null on miss; `forecastCategoryId`/`salesStageId`/
|
|
84
|
+
`percentToCloseStatusId` → **create-on-miss** (+name update); unknown line item → **throws**
|
|
85
|
+
(Retries until the item import runs).
|
|
86
|
+
- `netsuiteStatus` (NOT-NULL enum `'In Progress'|'Closed - Won'|'Closed - Lost'`) comes from
|
|
87
|
+
**SuiteQL `transaction.status`** letter mapped `A→In Progress, C→Closed - Won, D→Closed - Lost`
|
|
88
|
+
(same as `App_Api_Netsuite_Rest::listOpportunities`) — **not** `entityStatus` (that's the
|
|
89
|
+
sales-stage percent → `percentToCloseStatusId`). The REST record GET doesn't return that
|
|
90
|
+
letter, so the handler runs a one-row SuiteQL query for it.
|
|
91
|
+
- Writes use raw `_Query` against `_underscore::DB_FORECAST` and **must**
|
|
92
|
+
`_Database::transactionCommit(DB_FORECAST)` (lazy-transaction gotcha).
|
|
93
|
+
|
|
94
|
+
## Client variations
|
|
95
|
+
|
|
96
|
+
None — platform-wide Forecast sync.
|
|
97
|
+
|
|
98
|
+
## Gotchas / known issues
|
|
99
|
+
|
|
100
|
+
- **worker2 routes EVERY request to `_Controller_Index::worker()`** — `worker2/_.php` sets
|
|
101
|
+
`DEFAULT_CONTROLLER_METHOD = ['_Controller_Index','worker']`, which short-circuits `_Route`'s
|
|
102
|
+
URI matching. There is no per-URL routing, and `.htaccess` denies direct `.php` files, so you
|
|
103
|
+
can't add a real `/netsuite` route or a standalone script — everything enters through `worker()`.
|
|
104
|
+
- **`_Worker::runTask()` debug mode swallows the sub-request result.** In debug it self-POSTs
|
|
105
|
+
`_Config::api('worker')` and does **not** check the response (`_underscore/Worker.php:12`). So a
|
|
106
|
+
failing sub-action (e.g. `Netsuite/Opportunity/PUT` throwing) is invisible: the outer webhook
|
|
107
|
+
still returns 200, the queue row clears, and the DB write never happened. In prod the same
|
|
108
|
+
failure surfaces in `WorkerJobs.isSuccess=0`/`failureReason`. To debug locally, POST the
|
|
109
|
+
`{action,parameters}` envelope straight to the worker and read the response, or check logs —
|
|
110
|
+
don't trust the 200.
|
|
111
|
+
- **Local dev needs `[api] worker` in the worker2 config** — debug-mode `runTask` requires
|
|
112
|
+
`_Config::api('worker')`. Set `worker = "http://worker2/"` (the local Apache vhost) plus the
|
|
113
|
+
standard `[api] endpoint`/`uuid_*`. Without `[api]` you get
|
|
114
|
+
`Configuration group [api] has not been defined`.
|
|
115
|
+
- **No WebhookIngestion Lambda locally.** Dev points the enqueuer at an ngrok tunnel to local
|
|
116
|
+
worker2 via the `DEV_OVERRIDE` block, which also wraps the body in worker2's legacy
|
|
117
|
+
`{action,parameters}` envelope (since there's no Lambda to convert `/netsuite` → an action).
|
|
118
|
+
Set `DEV_OVERRIDE.enabled=false` for prod.
|
|
119
|
+
- **NetSuite doubled-prefix ids.** The queue record and its fields were created with the full
|
|
120
|
+
custom prefix typed into the ID box, so the real ids are doubled:
|
|
121
|
+
type `customrecordcustomrecord_api_msg_queue`, fields `custrecordcustrecord_amq_*`. The
|
|
122
|
+
single-prefix forms do not exist — use the doubled ids everywhere (SuiteScript, SuiteQL, REST).
|
|
123
|
+
- **REST opportunity line `itemType` is an object** (`{id,refName}`), not a string — casting it to
|
|
124
|
+
string throws `Object of class stdClass could not be converted to string`. Normalize before
|
|
125
|
+
matching.
|
|
126
|
+
- **Drain dedup** uses a `Sending` claim status: `transmit()` flips `Created`→`Sending` before the
|
|
127
|
+
HTTP call so a concurrent pass skips it; the immediate UE trigger only fires for `Created`/
|
|
128
|
+
`Pending`, so the drainer's own `Sending`/`Retry`/`Failed` writes don't re-trigger it (a
|
|
129
|
+
`skip (status not triggerable)` debug line on those is expected). `findSendableIds` must list
|
|
130
|
+
every sendable status (`Created`,`Pending`,`Retry`,`Sending`) — omitting `Created` silently
|
|
131
|
+
matches nothing (`candidates:0`).
|
|
132
|
+
|
|
133
|
+
## Change history
|
|
134
|
+
- 2026-06-16 — Built the queue enqueuer/drainer SuiteScripts, the worker2 `/netsuite` router, and
|
|
135
|
+
the `_Worker_Netsuite_Opportunity` Forecast importer (header + items + status), with ClickUp
|
|
136
|
+
parity. Verified opportunity 5483750 imports end-to-end in dev via ngrok. (dfranks)
|
|
137
|
+
|
|
138
|
+
## Related docs
|
|
139
|
+
- `2.0/apps/worker2/architecture.md` — the WorkerJobs/SQS/Lambda model and always-HTTP-200 worker.
|
|
140
|
+
- `2.0/apps/_underscore/architecture.md` — the `_Database::register()` lazy-transaction gotcha.
|
package/knowledge/INDEX.md
CHANGED
|
@@ -15,7 +15,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
15
15
|
## 2.0 framework
|
|
16
16
|
|
|
17
17
|
- **_underscore** (_Underscore) _(framework core)_ — 7 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
|
|
18
|
-
- **worker2** (Worker) —
|
|
18
|
+
- **worker2** (Worker) — 7 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
|
|
19
19
|
- **api2** (API) — 3 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
|
|
20
20
|
- **dbchanges2** (Database Changes) _(framework core)_ — 1 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
|
|
21
21
|
- **toga2-supply** (TOGa Supply) — 2 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)
|
package/package.json
CHANGED