toga-ai 1.0.233 → 1.0.235
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/2.0/apps/_underscore/INDEX.md +2 -1
- package/knowledge/2.0/apps/_underscore/features/event-publish-sqs.md +60 -0
- package/knowledge/2.0/apps/_underscore/features/forecast-sale-import.md +38 -6
- package/knowledge/INDEX.md +2 -1
- package/knowledge/clients/prudential/profile.md +2 -1
- package/knowledge/clients/tow-foundation/features/receipt-processing.md +2 -1
- package/knowledge/registry.json +10 -0
- package/knowledge/standalone/apps/websocket/INDEX.md +6 -0
- package/knowledge/standalone/apps/websocket/architecture.md +90 -0
- package/knowledge/standalone/apps/websocket/features/sqs-fanout.md +86 -0
- package/package.json +1 -1
|
@@ -7,7 +7,8 @@
|
|
|
7
7
|
| [Assortment Name Translation (AssortmentTranslations sidecar)](features/assortment-name-translation.md) | Serves Assortment (product-grouping) **names** in multiple languages by adding a per-language **sidecar** table `AssortmentTranslations`, reusing the platform's | _underscore/Model/Client/AssortmentTranslation.php, dbchanges2/Client/2026-06-26a - AssortmentTranslations.sql, dbchanges2/Core/2026-06-26a - AssortmentTranslationsRecord.sql, dbchanges2/Client/2026-06-26b - AssortmentTranslationsAcl.sql |
|
|
8
8
|
| [Carrier Shipping Labels (UPS/FedEx) & NetSuite Item Fulfillment](features/carrier-shipping-labels.md) | Backend mechanics behind TOGa Supply's Fulfill & Ship: buying a carrier label (UPS/FedEx), persisting it, and creating the NetSuite Item Fulfillment with tracki | _underscore/Model/Client/ItemFulfillment.php, _underscore/Model/Client/ItemFulfillments/TrackingNumber.php, _underscore/Component/Library/LabelPdf/LabelPdf.php, _underscore/Component/Library/Carriers/Ups/Ups.php, _underscore/Component/Library/Carriers/Fedex/Fedex.php, _underscore/Trait/Netsuite/ItemFulfillment.php, _underscore/Trait/Netsuite/SalesOrder.php, _underscore/Component/Library/NetSuite/NetSuite.php, _underscore/Model/Client/TrackingNumber.php, _underscore/Model/Client/ShippingMethod.php, _underscore/Model.php, _underscore/Cloud.php |
|
|
9
9
|
| [Client Email Template Sending](features/email-template-sending.md) | `_Model_Client_EmailTemplate` sends a stored, client-defined email template by UUID. | _underscore/Model/Client/EmailTemplate.php, _underscore/Model/Client/EmailTemplateOutgoingEmailAddress.php, _underscore/Email.php |
|
|
10
|
-
| [
|
|
10
|
+
| [Record-Changed Event Publishing (_Event::publish to SQS)](features/event-publish-sqs.md) | `_Event::publish()` (in `_underscore/Event.php`) is the PHP side of the real-time event pipeline. | _underscore/Event.php |
|
|
11
|
+
| [Forecast.Sales NetSuite import engine (real-time webhook)](features/forecast-sale-import.md) | Real-time importer that takes a NetSuite **sale** record and writes its lines into `Forecast.Sales` (the Forecast2 revenue table). | _underscore/Component/Forecast/SaleImport/SaleImport.php, _underscore/Component/Forecast/Db/Db.php, _underscore/Component/Api/Netsuite/Netsuite.php, worker2/Worker/Netsuite/Invoice.php, worker2/Worker/Netsuite/CashSale.php, worker2/Worker/Netsuite/CreditMemo.php, worker2/Worker/Netsuite/CashRefund.php, worker2/Worker/Netsuite/JournalEntry.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Worker/Netsuite/SalesOrder.php, dbchanges2/Forecast/2026-06-26a - Add journalEntry to Sales transaction type enum.sql, test/@dave/test_invoice_lifecycle.php, test/@dave/test_je_lifecycle.php, test/@dave/test_creditmemo_lifecycle.php, test/@dave/test_cashsale_lifecycle.php, test/@dave/test_cashrefund_lifecycle.php, test/@dave/test_fetchrecord_routes.php, test/@dave/verify_je_classification.php, test/@dave/probe_je_accounts.php, test/@dave/probe_je_shape.php, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/NetSuite/api-message-queue/dev_ue_api_msg_queue_enqueue.js |
|
|
11
12
|
| [_Model magic-field access (__get without __isset)](features/model-magic-field-access.md) | `_Model` exposes DB columns as "magic" properties via `__get()`, but it defines **no** `__isset()`. | _underscore/Model/Core/Model.php |
|
|
12
13
|
| [NetSuite REST Client (_Component_Api_Netsuite) — record writes & SuiteQL](features/netsuite-rest-client.md) | `_Component_Api_Netsuite` is the **2.0 `_underscore` NetSuite REST client** — the shared primitive every worker2/api2 NetSuite caller uses for record GETs, Suit | _underscore/Component/Api/Netsuite/Netsuite.php |
|
|
13
14
|
| [Per-Client Database Connections & the Local Logs Trap](features/per-client-database-connections.md) | When `_underscore` serves a request for a client it opens **three distinct per-client database connections**, not one. | _underscore/Database.php, _underscore/ApiRequest.php, _underscore/Model/Client/Logs/Api.php |
|
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Record-Changed Event Publishing (_Event::publish to SQS)
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
repo: _underscore
|
|
5
|
+
project: _Underscore
|
|
6
|
+
client: shared
|
|
7
|
+
type: feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-06-29
|
|
10
|
+
owners: [rgirish]
|
|
11
|
+
files:
|
|
12
|
+
- _underscore/Event.php
|
|
13
|
+
related:
|
|
14
|
+
- ../../../../standalone/apps/websocket/features/sqs-fanout.md
|
|
15
|
+
---
|
|
16
|
+
|
|
17
|
+
## What it is
|
|
18
|
+
|
|
19
|
+
`_Event::publish()` (in `_underscore/Event.php`) is the PHP side of the real-time event
|
|
20
|
+
pipeline. After a successful V2 write (POST/PUT/PATCH), it publishes a single JSON message
|
|
21
|
+
to the `toga-events` SQS queue describing what changed. The `websocket` standalone server
|
|
22
|
+
consumes that queue and fans out `record:changed` to browsers. The whole call is wrapped in
|
|
23
|
+
a `try/catch (\Throwable)` so an SQS failure logs but never blocks or errors the API
|
|
24
|
+
response.
|
|
25
|
+
|
|
26
|
+
## How it works
|
|
27
|
+
|
|
28
|
+
`publish(object &$api, string $httpMethod, array $routeVars)`:
|
|
29
|
+
|
|
30
|
+
1. Reads `aws_event_queue_url` / `aws_event_queue_region` from `_Config::cloud()`; returns
|
|
31
|
+
early if no queue URL is configured.
|
|
32
|
+
2. Determines the affected record's `recordRoute`, `recordId`, `uuid`.
|
|
33
|
+
3. Derives `parentRoute` / `parentUuid` from `$routeVars` when the URL is a nested resource
|
|
34
|
+
(`count($routeVars) >= 3` → `parentRoute = $routeVars[0]`, `parentUuid = $routeVars[1]`).
|
|
35
|
+
4. JSON-encodes `{ clientId, clientUuid, recordRoute, recordId, uuid, httpMethod,
|
|
36
|
+
parentRoute, parentUuid, dtEvent }` and enqueues via `_Cloud::addSqsMessage()`.
|
|
37
|
+
|
|
38
|
+
SQS credentials come from `_Config::cloud('aws_event_queue_access_key_id' /
|
|
39
|
+
'..._secret_access_key')` — values live in config/cloud, never in code or this doc.
|
|
40
|
+
|
|
41
|
+
## Gotchas
|
|
42
|
+
|
|
43
|
+
- **`recordRoute` must come from `$routeVars[0]`, not `$api->record->route`.** Post-`POST`
|
|
44
|
+
interceptors can swap `$api->record` to a *different* model before `publish()` runs (e.g.
|
|
45
|
+
`service-requests`' `postPost` switches `$api->record` to the `service-request-bundles`
|
|
46
|
+
model), so `$api->record->route` would point at the wrong route. Read the original URL
|
|
47
|
+
segment `$routeVars[0]`, falling back to `$api->record->route` only when `routeVars` is
|
|
48
|
+
empty.
|
|
49
|
+
- **`$api->response->data` is a PHP array, not a stdClass object.** `processRoutePairs()`
|
|
50
|
+
in V2.php returns e.g. `['serviceRequests' => $outData]`. Using object access
|
|
51
|
+
(`$data->{$topKey}`) on it silently returns null, so `uuid`/`recordId` were always null in
|
|
52
|
+
published events. Cast first: `$data = (array) $api->response->data;` then
|
|
53
|
+
`$topRecord = (object) $data[array_key_first($data)];` before reading `->id` / `->uuid`.
|
|
54
|
+
|
|
55
|
+
## Change history
|
|
56
|
+
- 2026-06-29 — Fixed `recordRoute` to read `$routeVars[0]` (authoritative URL route) because
|
|
57
|
+
post-interceptors can mutate `$api->record` to a different model before publish. (rgirish)
|
|
58
|
+
- 2026-06-29 — Fixed null `uuid`/`recordId` in SQS events: response data is a PHP array, so
|
|
59
|
+
cast `(array)` before `array_key_first()` and `(object)` the top record before property
|
|
60
|
+
access (was using object access on an array → silent null). (rgirish)
|
|
@@ -6,7 +6,7 @@ project: _Underscore
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-06-
|
|
9
|
+
updated: 2026-06-29
|
|
10
10
|
owners: [dfranks]
|
|
11
11
|
files:
|
|
12
12
|
- _underscore/Component/Forecast/SaleImport/SaleImport.php
|
|
@@ -28,6 +28,7 @@ files:
|
|
|
28
28
|
- test/@dave/test_fetchrecord_routes.php
|
|
29
29
|
- test/@dave/verify_je_classification.php
|
|
30
30
|
- test/@dave/probe_je_accounts.php
|
|
31
|
+
- test/@dave/probe_je_shape.php
|
|
31
32
|
- test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js
|
|
32
33
|
- test/@dave/NetSuite/api-message-queue/dev_ue_api_msg_queue_enqueue.js
|
|
33
34
|
related:
|
|
@@ -161,11 +162,27 @@ location, memo). Handler `_Worker_Netsuite_JournalEntry`: `post`/`put` → `sync
|
|
|
161
162
|
item-line upsert/reconcile machinery (`syncLines`/`guardedInsert`/`deleteRows`).
|
|
162
163
|
|
|
163
164
|
**Mapping model (durable design):**
|
|
164
|
-
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
`
|
|
168
|
-
|
|
165
|
+
- **Sales rep is now LIVE on JE lines** as the custom column **`custcol_sales_rep_line`**
|
|
166
|
+
(`JE_LINE_SALESREP_FIELD = 'custcol_sales_rep_line'`, wired 2026-06-29 after live Apr–May
|
|
167
|
+
2026 probing). The value is a NetSuite **employee REFERENCE object** `{links, id, refName}`;
|
|
168
|
+
`.id` is the employee internalId (a **string** in JSON). `journalLineLookup()` resolves it
|
|
169
|
+
via `lookupId('Employees','netsuiteInternalId', .id)` → local `salesRepEmployeeId`. The
|
|
170
|
+
per-run **account-number allowlist isolates exactly the lines that carry the rep** — offset/
|
|
171
|
+
balancing lines omit class/entity/`custcol_sales_rep_line`, so they group to null and are
|
|
172
|
+
dropped. No schema/migration (`salesRepEmployeeId` already on `Forecast.Sales`).
|
|
173
|
+
- **JE lines carry NO item field at all — item ingestion from NetSuite is not possible.**
|
|
174
|
+
Confirmed across multiple Apr–May 2026 JEs: the full union of line keys is
|
|
175
|
+
`{account, class, cleared, debit, credit, custcol_sales_rep_line, entity, line, memo, location}`.
|
|
176
|
+
There is **no native `item` and no custom item column**. `JE_LINE_ITEM_FIELD` stays **null
|
|
177
|
+
by design**; kept lines now group **per-salesRep with a null item**. The dimension that
|
|
178
|
+
exists *instead of* item is **`entity`** (the customer ref). Probe tool:
|
|
179
|
+
`test/@dave/probe_je_shape.php` (dumps line column shapes + custom columns).
|
|
180
|
+
- **No salesRep self-heal (unlike the item path).** `journalLineLookup()` for Employees has
|
|
181
|
+
**no `syncEmployee()` fallback** (the item path self-heals via
|
|
182
|
+
`_Worker_Netsuite_Item::syncItem()`). A rep missing from `Forecast.Employees` silently
|
|
183
|
+
resolves to **null** and re-aggregates under null. It relies on the **supporting-records
|
|
184
|
+
pull** keeping `Forecast.Employees` current. (Sample coverage was complete — rep IDs
|
|
185
|
+
482, 569, 685, 30954, 34877 all present in prod `Forecast.Employees`.)
|
|
169
186
|
- Lines hitting a **revenue** or **cost** GL account are grouped by **(salesRep, item)**:
|
|
170
187
|
`revenue = Σ(credit − debit)` over revenue lines, `cost = Σ(debit − credit)` over cost
|
|
171
188
|
lines, `profit = revenue − cost`.
|
|
@@ -411,6 +428,21 @@ record is deleted in NetSuite.)
|
|
|
411
428
|
- The cron's sign handling is not portable here — see Sign convention.
|
|
412
429
|
|
|
413
430
|
## Change history
|
|
431
|
+
- 2026-06-29 — **Wired JE sales-rep ingestion live** (TRUE-79862): set
|
|
432
|
+
`JE_LINE_SALESREP_FIELD = 'custcol_sales_rep_line'` after live Apr–May 2026 probing
|
|
433
|
+
confirmed the rep is now present on JE lines as that custom column — a NetSuite employee
|
|
434
|
+
**reference** object `{links,id,refName}` whose `.id` (string) resolves via
|
|
435
|
+
`lookupId('Employees','netsuiteInternalId',.id)`. Kept lines now group per-salesRep with a
|
|
436
|
+
null item. One-line change, no schema/migration (`salesRepEmployeeId` already exists).
|
|
437
|
+
**Confirmed JE lines carry NO item field** (native or custom) — line key union is
|
|
438
|
+
`{account,class,cleared,debit,credit,custcol_sales_rep_line,entity,line,memo,location}`; the
|
|
439
|
+
dimension *instead of* item is `entity` (customer ref), so `JE_LINE_ITEM_FIELD` stays null
|
|
440
|
+
by design (item ingestion from NetSuite is not possible). **Gotcha: JE salesRep lookup has
|
|
441
|
+
no self-heal** (the item path does via `syncItem()`) — a rep missing from
|
|
442
|
+
`Forecast.Employees` resolves to null and aggregates under null; relies on the
|
|
443
|
+
supporting-records pull. Still dormant in prod (JE webhook fires only once the AMQ enqueuer
|
|
444
|
+
deploys on the Journal Entry record type). Probe tool: `test/@dave/probe_je_shape.php`.
|
|
445
|
+
(dfranks)
|
|
414
446
|
- 2026-06-26 — **Changed JE revenue/cost classification from `accttype` buckets to an EXACT
|
|
415
447
|
account-NUMBER allowlist** (sales-team-defined): `JE_REVENUE_ACCOUNT_NUMBERS =
|
|
416
448
|
{41100,41300,41500}`, `JE_COST_ACCOUNT_NUMBERS = {51100,51200}`; everything else ignored.
|
package/knowledge/INDEX.md
CHANGED
|
@@ -16,7 +16,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
16
16
|
|
|
17
17
|
## 2.0 framework
|
|
18
18
|
|
|
19
|
-
- **_underscore** (_Underscore) _(framework core)_ —
|
|
19
|
+
- **_underscore** (_Underscore) _(framework core)_ — 19 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
|
|
20
20
|
- **worker2** (Worker) — 22 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
|
|
21
21
|
- **api2** (API) — 7 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
|
|
22
22
|
- **dbchanges2** (Database Changes) _(framework core)_ — 3 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
|
|
@@ -34,6 +34,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
34
34
|
## standalone framework
|
|
35
35
|
|
|
36
36
|
- **togatech** (TOGA Technology Website) — 4 doc(s) → [standalone/apps/togatech/INDEX.md](standalone/apps/togatech/INDEX.md)
|
|
37
|
+
- **websocket** (WebSocket Server) — 2 doc(s) → [standalone/apps/websocket/INDEX.md](standalone/apps/websocket/INDEX.md)
|
|
37
38
|
- **forward** (Forwarder) — 4 doc(s) → [standalone/apps/forward/INDEX.md](standalone/apps/forward/INDEX.md)
|
|
38
39
|
|
|
39
40
|
## Clients
|
|
@@ -60,7 +60,7 @@ Credit Card Receipts/
|
|
|
60
60
|
- **`.docx` files bypass Talos** and route to `extractDocxData()` (Talos returns HTTP 500 on a `.docx` MIME type — it only accepts PDF/image). That method opens the docx as a ZIP, extracts `word/document.xml`, strips tags, and regex-parses `MM/DD/YY: description` lines into `line_items[]`
|
|
61
61
|
- All other types POST to Talos AI `/api/ai/generate` → structured `{vendor_name, invoice_date, total, payment_memo, category, ...}`
|
|
62
62
|
- **Year guard on AI dates** — if Talos returns an `invoice_date` whose year is more than 1 year from the current year (AI hallucination on two-digit year inputs, e.g. `5/13/76` → 1976, `5/28/28` → 2028), the year is clamped to the current year while month/day are preserved
|
|
63
|
-
- Cross-verify amount ± $0.01 against parsed statement; if matched,
|
|
63
|
+
- Cross-verify amount ± $0.01 against parsed statement; if matched, the statement description **may** override `payment_memo`. The statement memo is the **authoritative source for activity-purpose headings** (Grantee meeting, Peer meeting, Grantee event, etc.) — but only when it actually *is* such a heading. The Amex statement's `Description` column for ride-share/taxi charges holds raw bank strings (e.g. `"AplPay LYFT 855-280-0278 CA"`, `"UBER"`), not activity headings. **Priority order (enforced):** (1) statement memo wins only if it is a proper activity heading; (2) otherwise the AI memo is kept — including when the statement only has a raw bank string, because the AI's descriptive memo (e.g. `"Transportation: Lyft ride from East Side at 105th to 925 9th Ave"`) is more useful than a raw Apple Pay string. The override condition is `empty($extracted->payment_memo) || self::isActivityHeadingMemo($matched)`. `isActivityHeadingMemo()` returns true only when the statement memo (left-trimmed, lowercased) starts with a recognised activity-purpose prefix followed by a colon (`grantee meeting:`, `grantee event:`, `peer meeting:`, `colleague meeting:`, `staff theater:`, `professional development:`, `dining:`, `travel:`). This **replaces** the earlier `isBareTranportationMemo()` approach, which fired whenever the AI memo started with `"Transportation:"` and so let the raw bank string overwrite a useful AI memo on ride-share rows (degraded rows 1, 17, 18, 21, 26, 31, 32, 34 on Emily Tow's statement).
|
|
64
64
|
- **Zero-total fallback** — if the extracted/docx total is 0, `lookupStatementTotalByVendor()` derives the amount from the billing statement (see *Statement structure* below)
|
|
65
65
|
- Build base filename (no suffix yet) and store in `$pendingRenames`
|
|
66
66
|
- On extract failure → move to `Archive/exception/` immediately
|
|
@@ -236,6 +236,7 @@ Fatal errors send only to `NOTIFY_EMAIL_DEV` (no CC/BCC).
|
|
|
236
236
|
|
|
237
237
|
## Change history
|
|
238
238
|
|
|
239
|
+
- 2026-06-29 — **Reversed the over-aggressive memo override.** Replaced `isBareTranportationMemo()` (added earlier same day) with `isActivityHeadingMemo()`: the statement memo now overrides the AI memo **only** when it is a proper activity-purpose heading (e.g. `Grantee meeting:`, `Peer meeting:`), not whenever the AI memo started with `"Transportation:"`. The previous approach replaced semi-useful AI memos with raw bank strings (`"AplPay LYFT …"`, `"UBER"`) on ride-share/taxi rows where the statement has no activity heading — degraded 8 rows on Emily Tow's statement (1, 17, 18, 21, 26, 31, 32, 34). New priority: statement heading > AI memo, with AI memo kept for any non-heading statement description. (rgirish)
|
|
239
240
|
- 2026-06-29 — `payment_memo` override now also fires on a **bare** `"Transportation:"` AI memo, not just an empty one: new `isBareTranportationMemo()` lets the cardholder's statement Excel (the authoritative source for activity-purpose headings like "Peer meeting:", "Grantee event:") override the AI's generic transportation heading. Fixes wrong headings on Emily Tow May 2026 transit rows. Added the committed `MoveBack` action (+ `collectFilesRecursiveRaw()`) to restore a person's archived receipts into a billing-cycle folder for reprocessing, replacing the throwaway `/tmp/tow_restore_archives.php` script. (rgirish)
|
|
240
241
|
- 2026-06-25 — Statement parsing hardened for Emily Tow's May 2026 Amex xlsx: `loadStatementExcel()` now iterates `getAllSheets()` (4 weekly-period sheets) instead of `getActiveSheet()` and captures the `Receipt` column; `lookupStatementTotalByVendor()` now **sums** all receipt-label-matched rows (14 × $3.00 "Subway Rides" = $42.00) before falling back to description-keyword match — fixes subway totals reading $0/$9.00. Added `extractDocxData()` so `.docx` receipts bypass Talos (Talos 500s on docx), parsing `MM/DD/YY: description` line items into per-entry Excel rows. Added a year guard clamping AI-hallucinated `invoice_date` years (e.g. 1976, 2028) to the current year. Fixed archive-path bug: SharePoint move now uses the actual folder name (`"Emily Tow CC receipts"`) not the normalized name, fixing 404 on archive moves. 5 more vendors found missing from the QB vendor list (client action). (rgirish)
|
|
241
242
|
- 2026-06-23 — `TowFoundationCategories.php` added: AI category values now mapped to exact QB account strings via `mapCategory()`; `"Transportation"` → `"6610 Travel expense"` etc. (19 mappings). QB vendor substring match minimum lowered from 4 to 3 chars with whole-word boundary guard for short needles — fixes `"CVS"` → `"CVS Pharmacy"`, `"MTA"` → `"MTA Metro card"`, prevents `"UPS"` → `"USPS"` false positive. (rgirish)
|
package/knowledge/registry.json
CHANGED
|
@@ -96,6 +96,16 @@
|
|
|
96
96
|
"role": "app",
|
|
97
97
|
"dependsOn": []
|
|
98
98
|
},
|
|
99
|
+
{
|
|
100
|
+
"repo": "websocket",
|
|
101
|
+
"project": "WebSocket Server",
|
|
102
|
+
"framework": "standalone",
|
|
103
|
+
"role": "app",
|
|
104
|
+
"dependsOn": [
|
|
105
|
+
"api2",
|
|
106
|
+
"_underscore"
|
|
107
|
+
]
|
|
108
|
+
},
|
|
99
109
|
{
|
|
100
110
|
"repo": "webhook",
|
|
101
111
|
"project": "Webhook",
|
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
# websocket (WebSocket Server) — standalone knowledge
|
|
2
|
+
|
|
3
|
+
| Doc | Summary | Files |
|
|
4
|
+
|-----|---------|-------|
|
|
5
|
+
| [WebSocket Server Architecture](architecture.md) | `websocket` (npm package `toga-socket-server`) is the real-time push tier for TOGA 2.0 frontends. | websocket/index.js, websocket/auth.js, websocket/rooms.js, websocket/fanout.js, websocket/metadata.js, websocket/ecosystem.config.js, websocket/aws-setup.sh, websocket/package.json |
|
|
6
|
+
| [SQS Fan-out to Socket.io Rooms](features/sqs-fanout.md) | The fan-out logic that turns one SQS `toga-events` message (published by api2 after a write) into `record:changed` emits to every Socket.io room that should lea | websocket/fanout.js, websocket/metadata.js, websocket/rooms.js |
|
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: WebSocket Server Architecture
|
|
3
|
+
framework: "standalone"
|
|
4
|
+
repo: websocket
|
|
5
|
+
project: WebSocket Server
|
|
6
|
+
client: shared
|
|
7
|
+
type: architecture
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-06-29
|
|
10
|
+
owners: [rgirish]
|
|
11
|
+
files:
|
|
12
|
+
- websocket/index.js
|
|
13
|
+
- websocket/auth.js
|
|
14
|
+
- websocket/rooms.js
|
|
15
|
+
- websocket/fanout.js
|
|
16
|
+
- websocket/metadata.js
|
|
17
|
+
- websocket/ecosystem.config.js
|
|
18
|
+
- websocket/aws-setup.sh
|
|
19
|
+
- websocket/package.json
|
|
20
|
+
related:
|
|
21
|
+
- ../../../2.0/apps/_underscore/features/event-publish-sqs.md
|
|
22
|
+
---
|
|
23
|
+
|
|
24
|
+
## Summary
|
|
25
|
+
|
|
26
|
+
`websocket` (npm package `toga-socket-server`) is the real-time push tier for TOGA 2.0
|
|
27
|
+
frontends. It is a **standalone Node.js / Socket.io server** — it uses neither PHP
|
|
28
|
+
framework (`_underscore` 2.0 nor `App_` 1.0), which is why it lives under the
|
|
29
|
+
`standalone/` knowledge partition. It consumes the `toga-events` SQS queue published by
|
|
30
|
+
`api2` (via `_Event::publish()` in `_underscore`) and fans out `record:changed` events to
|
|
31
|
+
connected browsers over Socket.io rooms. It ships no client-facing CRUD of its own.
|
|
32
|
+
|
|
33
|
+
**Critical rules:**
|
|
34
|
+
- It is a **consumer** of an event contract owned by the PHP side — the event payload
|
|
35
|
+
shape is defined by `_underscore`'s `_Event::publish()`. Changes to the payload must be
|
|
36
|
+
coordinated across both repos (see the related `event-publish-sqs` feature doc).
|
|
37
|
+
- Room keys use **`clientUuid`**, not `clientId` — the per-tenant isolation boundary.
|
|
38
|
+
- All secrets (`TOGA_JWT_SECRET`/`API_SECRET_ACCESS_TOKEN`, `SQS_*`, `DB_*`) come from env
|
|
39
|
+
(`.env` locally, Elastic Beanstalk env vars in prod) or the Core DB. Never commit values.
|
|
40
|
+
- JWT secrets are **sourced at runtime from `Core.Parameters`** (`API_SECRET_ACCESS_TOKEN`
|
|
41
|
+
+ `_PREVIOUS`) so the server stays in sync as api2 rotates them every 3–10 days.
|
|
42
|
+
|
|
43
|
+
## Stack
|
|
44
|
+
|
|
45
|
+
- **Node.js ≥ 18**, **Socket.io 4.7**, **@aws-sdk/client-sqs 3**, **mysql2 3**,
|
|
46
|
+
**jsonwebtoken 9** (HS256), **dotenv**.
|
|
47
|
+
- Process management via **PM2** (`ecosystem.config.js`, 2 workers, auto-restart).
|
|
48
|
+
- Deployed to **AWS Elastic Beanstalk**; `aws-setup.sh` provisions an instance and is
|
|
49
|
+
reusable across multiple environments.
|
|
50
|
+
|
|
51
|
+
## Pipeline
|
|
52
|
+
|
|
53
|
+
```
|
|
54
|
+
api2 PHP write (POST/PUT/PATCH)
|
|
55
|
+
→ _underscore _Event::publish() → SQS toga-events queue
|
|
56
|
+
→ this server polls SQS (long-poll loop)
|
|
57
|
+
→ fanOut() → Socket.io rooms → connected browsers (record:changed)
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
## Files
|
|
61
|
+
|
|
62
|
+
| File | Purpose |
|
|
63
|
+
|---|---|
|
|
64
|
+
| `index.js` | HTTP server, Socket.io init, SQS poll loop startup |
|
|
65
|
+
| `auth.js` | JWT validation middleware (HS256, same secret as api2) |
|
|
66
|
+
| `rooms.js` | `subscribe` / `unsubscribe` handlers; `listRoom()` / `instanceRoom()` helpers |
|
|
67
|
+
| `fanout.js` | SQS message → room emit (fan-out to list + instance + parent + child rooms) |
|
|
68
|
+
| `metadata.js` | Core DB cache (route→id, parent/children maps, JWT secrets); refreshed every 5 min |
|
|
69
|
+
| `ecosystem.config.js` | PM2 config |
|
|
70
|
+
| `aws-setup.sh` | Elastic Beanstalk provisioning |
|
|
71
|
+
|
|
72
|
+
## Room naming
|
|
73
|
+
|
|
74
|
+
```
|
|
75
|
+
List room: toga:{clientUuid}:record:{recordRoute}
|
|
76
|
+
Instance room: toga:{clientUuid}:record:{recordRoute}:{uuid}
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
## Metadata cache (`metadata.js`)
|
|
80
|
+
|
|
81
|
+
A single Core DB connection-per-refresh (every 5 minutes, plus at startup) populates three
|
|
82
|
+
in-memory maps plus the JWT secret list:
|
|
83
|
+
|
|
84
|
+
- `routeMap` — `route` string → `recordId` (from `Core.Records`).
|
|
85
|
+
- `parentMap` — child `recordId` → parent `recordId` (from `Core.InherentRecordChildren`,
|
|
86
|
+
column `inherentChildOfRecordId`).
|
|
87
|
+
- `childrenMap` — parent `recordId` → `Set` of child `recordId`s (inverse of `parentMap`).
|
|
88
|
+
- `jwtSecrets` — `[current, previous]` from `Core.Parameters`, checked in order on verify.
|
|
89
|
+
|
|
90
|
+
Accessors: `getRouteMap()`, `getParentMap()`, `getChildrenMap()`, `getJwtSecrets()`.
|
|
@@ -0,0 +1,86 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: SQS Fan-out to Socket.io Rooms
|
|
3
|
+
framework: "standalone"
|
|
4
|
+
repo: websocket
|
|
5
|
+
project: WebSocket Server
|
|
6
|
+
client: shared
|
|
7
|
+
type: feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-06-29
|
|
10
|
+
owners: [rgirish]
|
|
11
|
+
files:
|
|
12
|
+
- websocket/fanout.js
|
|
13
|
+
- websocket/metadata.js
|
|
14
|
+
- websocket/rooms.js
|
|
15
|
+
related:
|
|
16
|
+
- ../../../../2.0/apps/_underscore/features/event-publish-sqs.md
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
## What it is
|
|
20
|
+
|
|
21
|
+
The fan-out logic that turns one SQS `toga-events` message (published by api2 after a
|
|
22
|
+
write) into `record:changed` emits to every Socket.io room that should learn about the
|
|
23
|
+
change. Implemented in `fanOut(event)` in `websocket/fanout.js`, driven by a continuous
|
|
24
|
+
SQS long-poll loop.
|
|
25
|
+
|
|
26
|
+
## How it works
|
|
27
|
+
|
|
28
|
+
`pollLoop()` long-polls SQS (`MaxNumberOfMessages: 10`, `WaitTimeSeconds: 20`), parses each
|
|
29
|
+
message body to an `event`, calls `fanOut(event)`, then deletes the message. Malformed
|
|
30
|
+
bodies are logged and deleted (not retried); fan-out errors are logged without deleting
|
|
31
|
+
fan-out from blocking the loop. On a receive error it backs off 5s before retrying.
|
|
32
|
+
|
|
33
|
+
`fanOut(event)` reads `{ clientUuid, recordRoute, uuid, httpMethod, dtEvent, parentRoute,
|
|
34
|
+
parentUuid }` and emits to up to four kinds of rooms (all keyed by `clientUuid`):
|
|
35
|
+
|
|
36
|
+
1. **Own list room** — `listRoom(clientUuid, recordRoute)` — always.
|
|
37
|
+
2. **Own instance room** — `instanceRoom(clientUuid, recordRoute, uuid)` — only when `uuid`
|
|
38
|
+
is present.
|
|
39
|
+
3. **Inherent children's list rooms** — for each child of the fired record, emit to that
|
|
40
|
+
child's list room with `{ ...payload, parentRoute: recordRoute }`. Resolved from the
|
|
41
|
+
metadata `childrenMap` (parent recordId → Set of child recordIds), looked up via
|
|
42
|
+
`routeMap` to translate the fired `recordRoute` → `recordId` → child recordIds → child
|
|
43
|
+
routes. This lets a list view of children refresh when the parent changes (e.g. a
|
|
44
|
+
`service-requests` write notifies `service-request-bundles`, `service-request-units`,
|
|
45
|
+
and `service-request-notes` list rooms).
|
|
46
|
+
4. **Parent rooms** — when the event itself is a nested write, notify the parent's list
|
|
47
|
+
room (always) and instance room (only if `parentUuid` present) with
|
|
48
|
+
`{ ...payload, childRoute: recordRoute }`.
|
|
49
|
+
|
|
50
|
+
### Parent-route resolution from metadata
|
|
51
|
+
|
|
52
|
+
When api2 posts a flat URL with no `parentRoute` in the event, `fanOut()` resolves the
|
|
53
|
+
parent itself: it reuses the `recordId` already looked up for step 3, then reads
|
|
54
|
+
`getParentMap().get(recordId)` (child recordId → parent recordId) and reverse-looks-up the
|
|
55
|
+
parent's route from `routeMap`. So parent notifications work even when the publisher did
|
|
56
|
+
not supply `parentRoute`.
|
|
57
|
+
|
|
58
|
+
## Frontend subscription
|
|
59
|
+
|
|
60
|
+
The frontend joins rooms by emitting `subscribe` once per route — there is **no server-side
|
|
61
|
+
array support and none is needed**. Multiple sequential emits each join a distinct
|
|
62
|
+
Socket.io room and are effectively simultaneous from the client's perspective:
|
|
63
|
+
|
|
64
|
+
```js
|
|
65
|
+
const routes = ['service-requests', 'service-request-bundles', 'service-request-units'];
|
|
66
|
+
routes.forEach(route => socket.emit('subscribe', { record: route }));
|
|
67
|
+
socket.on('record:changed', (payload) => { /* refresh the matching list/detail view */ });
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
## Gotchas
|
|
71
|
+
|
|
72
|
+
- Room keys are built from **`clientUuid`**, never `clientId` — using `clientId` cross-wires
|
|
73
|
+
tenants or misses rooms entirely.
|
|
74
|
+
- The children-room lookup is a nested scan of `routeMap` (recordId → route) per child; it
|
|
75
|
+
is correct but O(routes × children) — fine at current route counts, watch if routes grow.
|
|
76
|
+
- Children notification depends on `Core.InherentRecordChildren` being populated and the
|
|
77
|
+
metadata cache being fresh (refreshes every 5 min) — a brand-new parent/child relation
|
|
78
|
+
will not fan out until the next refresh.
|
|
79
|
+
|
|
80
|
+
## Change history
|
|
81
|
+
- 2026-06-29 — Added inherent-children list-room notification: a parent write now emits
|
|
82
|
+
`record:changed` (with `parentRoute`) to each inherent child's list room, via new
|
|
83
|
+
`childrenMap` / `getChildrenMap()` in metadata.js. Parent-route resolution reuses the
|
|
84
|
+
recordId/routeMap lookup from the children step. (rgirish)
|
|
85
|
+
- 2026-06-26 — Initial fan-out loop: SQS long-poll → list + instance + parent rooms, with
|
|
86
|
+
malformed-message drop and fan-out error isolation. (rgirish)
|
package/package.json
CHANGED