toga-ai 1.0.233 → 1.0.235

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -7,7 +7,8 @@
7
7
  | [Assortment Name Translation (AssortmentTranslations sidecar)](features/assortment-name-translation.md) | Serves Assortment (product-grouping) **names** in multiple languages by adding a per-language **sidecar** table `AssortmentTranslations`, reusing the platform's | _underscore/Model/Client/AssortmentTranslation.php, dbchanges2/Client/2026-06-26a - AssortmentTranslations.sql, dbchanges2/Core/2026-06-26a - AssortmentTranslationsRecord.sql, dbchanges2/Client/2026-06-26b - AssortmentTranslationsAcl.sql |
8
8
  | [Carrier Shipping Labels (UPS/FedEx) & NetSuite Item Fulfillment](features/carrier-shipping-labels.md) | Backend mechanics behind TOGa Supply's Fulfill & Ship: buying a carrier label (UPS/FedEx), persisting it, and creating the NetSuite Item Fulfillment with tracki | _underscore/Model/Client/ItemFulfillment.php, _underscore/Model/Client/ItemFulfillments/TrackingNumber.php, _underscore/Component/Library/LabelPdf/LabelPdf.php, _underscore/Component/Library/Carriers/Ups/Ups.php, _underscore/Component/Library/Carriers/Fedex/Fedex.php, _underscore/Trait/Netsuite/ItemFulfillment.php, _underscore/Trait/Netsuite/SalesOrder.php, _underscore/Component/Library/NetSuite/NetSuite.php, _underscore/Model/Client/TrackingNumber.php, _underscore/Model/Client/ShippingMethod.php, _underscore/Model.php, _underscore/Cloud.php |
9
9
  | [Client Email Template Sending](features/email-template-sending.md) | `_Model_Client_EmailTemplate` sends a stored, client-defined email template by UUID. | _underscore/Model/Client/EmailTemplate.php, _underscore/Model/Client/EmailTemplateOutgoingEmailAddress.php, _underscore/Email.php |
10
- | [Forecast.Sales NetSuite import engine (real-time webhook)](features/forecast-sale-import.md) | Real-time importer that takes a NetSuite **sale** record and writes its lines into `Forecast.Sales` (the Forecast2 revenue table). | _underscore/Component/Forecast/SaleImport/SaleImport.php, _underscore/Component/Forecast/Db/Db.php, _underscore/Component/Api/Netsuite/Netsuite.php, worker2/Worker/Netsuite/Invoice.php, worker2/Worker/Netsuite/CashSale.php, worker2/Worker/Netsuite/CreditMemo.php, worker2/Worker/Netsuite/CashRefund.php, worker2/Worker/Netsuite/JournalEntry.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Worker/Netsuite/SalesOrder.php, dbchanges2/Forecast/2026-06-26a - Add journalEntry to Sales transaction type enum.sql, test/@dave/test_invoice_lifecycle.php, test/@dave/test_je_lifecycle.php, test/@dave/test_creditmemo_lifecycle.php, test/@dave/test_cashsale_lifecycle.php, test/@dave/test_cashrefund_lifecycle.php, test/@dave/test_fetchrecord_routes.php, test/@dave/verify_je_classification.php, test/@dave/probe_je_accounts.php, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/NetSuite/api-message-queue/dev_ue_api_msg_queue_enqueue.js |
10
+ | [Record-Changed Event Publishing (_Event::publish to SQS)](features/event-publish-sqs.md) | `_Event::publish()` (in `_underscore/Event.php`) is the PHP side of the real-time event pipeline. | _underscore/Event.php |
11
+ | [Forecast.Sales NetSuite import engine (real-time webhook)](features/forecast-sale-import.md) | Real-time importer that takes a NetSuite **sale** record and writes its lines into `Forecast.Sales` (the Forecast2 revenue table). | _underscore/Component/Forecast/SaleImport/SaleImport.php, _underscore/Component/Forecast/Db/Db.php, _underscore/Component/Api/Netsuite/Netsuite.php, worker2/Worker/Netsuite/Invoice.php, worker2/Worker/Netsuite/CashSale.php, worker2/Worker/Netsuite/CreditMemo.php, worker2/Worker/Netsuite/CashRefund.php, worker2/Worker/Netsuite/JournalEntry.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Worker/Netsuite/SalesOrder.php, dbchanges2/Forecast/2026-06-26a - Add journalEntry to Sales transaction type enum.sql, test/@dave/test_invoice_lifecycle.php, test/@dave/test_je_lifecycle.php, test/@dave/test_creditmemo_lifecycle.php, test/@dave/test_cashsale_lifecycle.php, test/@dave/test_cashrefund_lifecycle.php, test/@dave/test_fetchrecord_routes.php, test/@dave/verify_je_classification.php, test/@dave/probe_je_accounts.php, test/@dave/probe_je_shape.php, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/NetSuite/api-message-queue/dev_ue_api_msg_queue_enqueue.js |
11
12
  | [_Model magic-field access (__get without __isset)](features/model-magic-field-access.md) | `_Model` exposes DB columns as "magic" properties via `__get()`, but it defines **no** `__isset()`. | _underscore/Model/Core/Model.php |
12
13
  | [NetSuite REST Client (_Component_Api_Netsuite) — record writes & SuiteQL](features/netsuite-rest-client.md) | `_Component_Api_Netsuite` is the **2.0 `_underscore` NetSuite REST client** — the shared primitive every worker2/api2 NetSuite caller uses for record GETs, Suit | _underscore/Component/Api/Netsuite/Netsuite.php |
13
14
  | [Per-Client Database Connections & the Local Logs Trap](features/per-client-database-connections.md) | When `_underscore` serves a request for a client it opens **three distinct per-client database connections**, not one. | _underscore/Database.php, _underscore/ApiRequest.php, _underscore/Model/Client/Logs/Api.php |
@@ -0,0 +1,60 @@
1
+ ---
2
+ title: Record-Changed Event Publishing (_Event::publish to SQS)
3
+ framework: "2.0"
4
+ repo: _underscore
5
+ project: _Underscore
6
+ client: shared
7
+ type: feature
8
+ status: active
9
+ updated: 2026-06-29
10
+ owners: [rgirish]
11
+ files:
12
+ - _underscore/Event.php
13
+ related:
14
+ - ../../../../standalone/apps/websocket/features/sqs-fanout.md
15
+ ---
16
+
17
+ ## What it is
18
+
19
+ `_Event::publish()` (in `_underscore/Event.php`) is the PHP side of the real-time event
20
+ pipeline. After a successful V2 write (POST/PUT/PATCH), it publishes a single JSON message
21
+ to the `toga-events` SQS queue describing what changed. The `websocket` standalone server
22
+ consumes that queue and fans out `record:changed` to browsers. The whole call is wrapped in
23
+ a `try/catch (\Throwable)` so an SQS failure logs but never blocks or errors the API
24
+ response.
25
+
26
+ ## How it works
27
+
28
+ `publish(object &$api, string $httpMethod, array $routeVars)`:
29
+
30
+ 1. Reads `aws_event_queue_url` / `aws_event_queue_region` from `_Config::cloud()`; returns
31
+ early if no queue URL is configured.
32
+ 2. Determines the affected record's `recordRoute`, `recordId`, `uuid`.
33
+ 3. Derives `parentRoute` / `parentUuid` from `$routeVars` when the URL is a nested resource
34
+ (`count($routeVars) >= 3` → `parentRoute = $routeVars[0]`, `parentUuid = $routeVars[1]`).
35
+ 4. JSON-encodes `{ clientId, clientUuid, recordRoute, recordId, uuid, httpMethod,
36
+ parentRoute, parentUuid, dtEvent }` and enqueues via `_Cloud::addSqsMessage()`.
37
+
38
+ SQS credentials come from `_Config::cloud('aws_event_queue_access_key_id' /
39
+ '..._secret_access_key')` — values live in config/cloud, never in code or this doc.
40
+
41
+ ## Gotchas
42
+
43
+ - **`recordRoute` must come from `$routeVars[0]`, not `$api->record->route`.** Post-`POST`
44
+ interceptors can swap `$api->record` to a *different* model before `publish()` runs (e.g.
45
+ `service-requests`' `postPost` switches `$api->record` to the `service-request-bundles`
46
+ model), so `$api->record->route` would point at the wrong route. Read the original URL
47
+ segment `$routeVars[0]`, falling back to `$api->record->route` only when `routeVars` is
48
+ empty.
49
+ - **`$api->response->data` is a PHP array, not a stdClass object.** `processRoutePairs()`
50
+ in V2.php returns e.g. `['serviceRequests' => $outData]`. Using object access
51
+ (`$data->{$topKey}`) on it silently returns null, so `uuid`/`recordId` were always null in
52
+ published events. Cast first: `$data = (array) $api->response->data;` then
53
+ `$topRecord = (object) $data[array_key_first($data)];` before reading `->id` / `->uuid`.
54
+
55
+ ## Change history
56
+ - 2026-06-29 — Fixed `recordRoute` to read `$routeVars[0]` (authoritative URL route) because
57
+ post-interceptors can mutate `$api->record` to a different model before publish. (rgirish)
58
+ - 2026-06-29 — Fixed null `uuid`/`recordId` in SQS events: response data is a PHP array, so
59
+ cast `(array)` before `array_key_first()` and `(object)` the top record before property
60
+ access (was using object access on an array → silent null). (rgirish)
@@ -6,7 +6,7 @@ project: _Underscore
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-06-26
9
+ updated: 2026-06-29
10
10
  owners: [dfranks]
11
11
  files:
12
12
  - _underscore/Component/Forecast/SaleImport/SaleImport.php
@@ -28,6 +28,7 @@ files:
28
28
  - test/@dave/test_fetchrecord_routes.php
29
29
  - test/@dave/verify_je_classification.php
30
30
  - test/@dave/probe_je_accounts.php
31
+ - test/@dave/probe_je_shape.php
31
32
  - test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js
32
33
  - test/@dave/NetSuite/api-message-queue/dev_ue_api_msg_queue_enqueue.js
33
34
  related:
@@ -161,11 +162,27 @@ location, memo). Handler `_Worker_Netsuite_JournalEntry`: `post`/`put` → `sync
161
162
  item-line upsert/reconcile machinery (`syncLines`/`guardedInsert`/`deleteRows`).
162
163
 
163
164
  **Mapping model (durable design):**
164
- - A JE line carries **no native item and no salesRep** (confirmed against the live account:
165
- 0 JEs carry an item on a revenue/cost line). The fields the business rule needs **do not
166
- exist in NetSuite yet** — they are stubbed behind constants `JE_LINE_ITEM_FIELD` /
167
- `JE_LINE_SALESREP_FIELD` (currently **null**), so until the fields are added every kept
168
- line groups under `(null, null)` into **one aggregate row**.
165
+ - **Sales rep is now LIVE on JE lines** as the custom column **`custcol_sales_rep_line`**
166
+ (`JE_LINE_SALESREP_FIELD = 'custcol_sales_rep_line'`, wired 2026-06-29 after live Apr–May
167
+ 2026 probing). The value is a NetSuite **employee REFERENCE object** `{links, id, refName}`;
168
+ `.id` is the employee internalId (a **string** in JSON). `journalLineLookup()` resolves it
169
+ via `lookupId('Employees','netsuiteInternalId', .id)` → local `salesRepEmployeeId`. The
170
+ per-run **account-number allowlist isolates exactly the lines that carry the rep** — offset/
171
+ balancing lines omit class/entity/`custcol_sales_rep_line`, so they group to null and are
172
+ dropped. No schema/migration (`salesRepEmployeeId` already on `Forecast.Sales`).
173
+ - **JE lines carry NO item field at all — item ingestion from NetSuite is not possible.**
174
+ Confirmed across multiple Apr–May 2026 JEs: the full union of line keys is
175
+ `{account, class, cleared, debit, credit, custcol_sales_rep_line, entity, line, memo, location}`.
176
+ There is **no native `item` and no custom item column**. `JE_LINE_ITEM_FIELD` stays **null
177
+ by design**; kept lines now group **per-salesRep with a null item**. The dimension that
178
+ exists *instead of* item is **`entity`** (the customer ref). Probe tool:
179
+ `test/@dave/probe_je_shape.php` (dumps line column shapes + custom columns).
180
+ - **No salesRep self-heal (unlike the item path).** `journalLineLookup()` for Employees has
181
+ **no `syncEmployee()` fallback** (the item path self-heals via
182
+ `_Worker_Netsuite_Item::syncItem()`). A rep missing from `Forecast.Employees` silently
183
+ resolves to **null** and re-aggregates under null. It relies on the **supporting-records
184
+ pull** keeping `Forecast.Employees` current. (Sample coverage was complete — rep IDs
185
+ 482, 569, 685, 30954, 34877 all present in prod `Forecast.Employees`.)
169
186
  - Lines hitting a **revenue** or **cost** GL account are grouped by **(salesRep, item)**:
170
187
  `revenue = Σ(credit − debit)` over revenue lines, `cost = Σ(debit − credit)` over cost
171
188
  lines, `profit = revenue − cost`.
@@ -411,6 +428,21 @@ record is deleted in NetSuite.)
411
428
  - The cron's sign handling is not portable here — see Sign convention.
412
429
 
413
430
  ## Change history
431
+ - 2026-06-29 — **Wired JE sales-rep ingestion live** (TRUE-79862): set
432
+ `JE_LINE_SALESREP_FIELD = 'custcol_sales_rep_line'` after live Apr–May 2026 probing
433
+ confirmed the rep is now present on JE lines as that custom column — a NetSuite employee
434
+ **reference** object `{links,id,refName}` whose `.id` (string) resolves via
435
+ `lookupId('Employees','netsuiteInternalId',.id)`. Kept lines now group per-salesRep with a
436
+ null item. One-line change, no schema/migration (`salesRepEmployeeId` already exists).
437
+ **Confirmed JE lines carry NO item field** (native or custom) — line key union is
438
+ `{account,class,cleared,debit,credit,custcol_sales_rep_line,entity,line,memo,location}`; the
439
+ dimension *instead of* item is `entity` (customer ref), so `JE_LINE_ITEM_FIELD` stays null
440
+ by design (item ingestion from NetSuite is not possible). **Gotcha: JE salesRep lookup has
441
+ no self-heal** (the item path does via `syncItem()`) — a rep missing from
442
+ `Forecast.Employees` resolves to null and aggregates under null; relies on the
443
+ supporting-records pull. Still dormant in prod (JE webhook fires only once the AMQ enqueuer
444
+ deploys on the Journal Entry record type). Probe tool: `test/@dave/probe_je_shape.php`.
445
+ (dfranks)
414
446
  - 2026-06-26 — **Changed JE revenue/cost classification from `accttype` buckets to an EXACT
415
447
  account-NUMBER allowlist** (sales-team-defined): `JE_REVENUE_ACCOUNT_NUMBERS =
416
448
  {41100,41300,41500}`, `JE_COST_ACCOUNT_NUMBERS = {51100,51200}`; everything else ignored.
@@ -16,7 +16,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
16
16
 
17
17
  ## 2.0 framework
18
18
 
19
- - **_underscore** (_Underscore) _(framework core)_ — 18 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
19
+ - **_underscore** (_Underscore) _(framework core)_ — 19 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
20
20
  - **worker2** (Worker) — 22 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
21
21
  - **api2** (API) — 7 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
22
22
  - **dbchanges2** (Database Changes) _(framework core)_ — 3 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
@@ -34,6 +34,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
34
34
  ## standalone framework
35
35
 
36
36
  - **togatech** (TOGA Technology Website) — 4 doc(s) → [standalone/apps/togatech/INDEX.md](standalone/apps/togatech/INDEX.md)
37
+ - **websocket** (WebSocket Server) — 2 doc(s) → [standalone/apps/websocket/INDEX.md](standalone/apps/websocket/INDEX.md)
37
38
  - **forward** (Forwarder) — 4 doc(s) → [standalone/apps/forward/INDEX.md](standalone/apps/forward/INDEX.md)
38
39
 
39
40
  ## Clients
@@ -5,11 +5,12 @@ apps:
5
5
  - _underscore
6
6
  - api2
7
7
  - dbchanges2
8
+ - websocket
8
9
  project: _Underscore
9
10
  client: prudential
10
11
  type: profile
11
12
  status: active
12
- updated: 2026-06-18
13
+ updated: 2026-06-29
13
14
  owners: ["jcardinal", "rgirish"]
14
15
  files: []
15
16
  related:
@@ -60,7 +60,7 @@ Credit Card Receipts/
60
60
  - **`.docx` files bypass Talos** and route to `extractDocxData()` (Talos returns HTTP 500 on a `.docx` MIME type — it only accepts PDF/image). That method opens the docx as a ZIP, extracts `word/document.xml`, strips tags, and regex-parses `MM/DD/YY: description` lines into `line_items[]`
61
61
  - All other types POST to Talos AI `/api/ai/generate` → structured `{vendor_name, invoice_date, total, payment_memo, category, ...}`
62
62
  - **Year guard on AI dates** — if Talos returns an `invoice_date` whose year is more than 1 year from the current year (AI hallucination on two-digit year inputs, e.g. `5/13/76` → 1976, `5/28/28` → 2028), the year is clamped to the current year while month/day are preserved
63
- - Cross-verify amount ± $0.01 against parsed statement; if matched, override `payment_memo` with the statement description. The statement memo is the **authoritative source for activity-purpose headings** (Grantee meeting, Peer meeting, Grantee event, etc.); the AI memo is good for location/detail but defaults to a bare `"Transportation:"` heading on ride-share/taxi receipts even when the receipt has handwritten activity notes. The override now fires when **either** the AI `payment_memo` is empty **or** it is a bare transportation memo — `isBareTranportationMemo()` returns true when the memo (left-trimmed) starts with `"Transportation:"`. Previously the override only fired on an empty AI memo, so a wrong `"Transportation:"` heading was never corrected.
63
+ - Cross-verify amount ± $0.01 against parsed statement; if matched, the statement description **may** override `payment_memo`. The statement memo is the **authoritative source for activity-purpose headings** (Grantee meeting, Peer meeting, Grantee event, etc.) — but only when it actually *is* such a heading. The Amex statement's `Description` column for ride-share/taxi charges holds raw bank strings (e.g. `"AplPay LYFT 855-280-0278 CA"`, `"UBER"`), not activity headings. **Priority order (enforced):** (1) statement memo wins only if it is a proper activity heading; (2) otherwise the AI memo is kept — including when the statement only has a raw bank string, because the AI's descriptive memo (e.g. `"Transportation: Lyft ride from East Side at 105th to 925 9th Ave"`) is more useful than a raw Apple Pay string. The override condition is `empty($extracted->payment_memo) || self::isActivityHeadingMemo($matched)`. `isActivityHeadingMemo()` returns true only when the statement memo (left-trimmed, lowercased) starts with a recognised activity-purpose prefix followed by a colon (`grantee meeting:`, `grantee event:`, `peer meeting:`, `colleague meeting:`, `staff theater:`, `professional development:`, `dining:`, `travel:`). This **replaces** the earlier `isBareTranportationMemo()` approach, which fired whenever the AI memo started with `"Transportation:"` and so let the raw bank string overwrite a useful AI memo on ride-share rows (degraded rows 1, 17, 18, 21, 26, 31, 32, 34 on Emily Tow's statement).
64
64
  - **Zero-total fallback** — if the extracted/docx total is 0, `lookupStatementTotalByVendor()` derives the amount from the billing statement (see *Statement structure* below)
65
65
  - Build base filename (no suffix yet) and store in `$pendingRenames`
66
66
  - On extract failure → move to `Archive/exception/` immediately
@@ -236,6 +236,7 @@ Fatal errors send only to `NOTIFY_EMAIL_DEV` (no CC/BCC).
236
236
 
237
237
  ## Change history
238
238
 
239
+ - 2026-06-29 — **Reversed the over-aggressive memo override.** Replaced `isBareTranportationMemo()` (added earlier same day) with `isActivityHeadingMemo()`: the statement memo now overrides the AI memo **only** when it is a proper activity-purpose heading (e.g. `Grantee meeting:`, `Peer meeting:`), not whenever the AI memo started with `"Transportation:"`. The previous approach replaced semi-useful AI memos with raw bank strings (`"AplPay LYFT …"`, `"UBER"`) on ride-share/taxi rows where the statement has no activity heading — degraded 8 rows on Emily Tow's statement (1, 17, 18, 21, 26, 31, 32, 34). New priority: statement heading > AI memo, with AI memo kept for any non-heading statement description. (rgirish)
239
240
  - 2026-06-29 — `payment_memo` override now also fires on a **bare** `"Transportation:"` AI memo, not just an empty one: new `isBareTranportationMemo()` lets the cardholder's statement Excel (the authoritative source for activity-purpose headings like "Peer meeting:", "Grantee event:") override the AI's generic transportation heading. Fixes wrong headings on Emily Tow May 2026 transit rows. Added the committed `MoveBack` action (+ `collectFilesRecursiveRaw()`) to restore a person's archived receipts into a billing-cycle folder for reprocessing, replacing the throwaway `/tmp/tow_restore_archives.php` script. (rgirish)
240
241
  - 2026-06-25 — Statement parsing hardened for Emily Tow's May 2026 Amex xlsx: `loadStatementExcel()` now iterates `getAllSheets()` (4 weekly-period sheets) instead of `getActiveSheet()` and captures the `Receipt` column; `lookupStatementTotalByVendor()` now **sums** all receipt-label-matched rows (14 × $3.00 "Subway Rides" = $42.00) before falling back to description-keyword match — fixes subway totals reading $0/$9.00. Added `extractDocxData()` so `.docx` receipts bypass Talos (Talos 500s on docx), parsing `MM/DD/YY: description` line items into per-entry Excel rows. Added a year guard clamping AI-hallucinated `invoice_date` years (e.g. 1976, 2028) to the current year. Fixed archive-path bug: SharePoint move now uses the actual folder name (`"Emily Tow CC receipts"`) not the normalized name, fixing 404 on archive moves. 5 more vendors found missing from the QB vendor list (client action). (rgirish)
241
242
  - 2026-06-23 — `TowFoundationCategories.php` added: AI category values now mapped to exact QB account strings via `mapCategory()`; `"Transportation"` → `"6610 Travel expense"` etc. (19 mappings). QB vendor substring match minimum lowered from 4 to 3 chars with whole-word boundary guard for short needles — fixes `"CVS"` → `"CVS Pharmacy"`, `"MTA"` → `"MTA Metro card"`, prevents `"UPS"` → `"USPS"` false positive. (rgirish)
@@ -96,6 +96,16 @@
96
96
  "role": "app",
97
97
  "dependsOn": []
98
98
  },
99
+ {
100
+ "repo": "websocket",
101
+ "project": "WebSocket Server",
102
+ "framework": "standalone",
103
+ "role": "app",
104
+ "dependsOn": [
105
+ "api2",
106
+ "_underscore"
107
+ ]
108
+ },
99
109
  {
100
110
  "repo": "webhook",
101
111
  "project": "Webhook",
@@ -0,0 +1,6 @@
1
+ # websocket (WebSocket Server) — standalone knowledge
2
+
3
+ | Doc | Summary | Files |
4
+ |-----|---------|-------|
5
+ | [WebSocket Server Architecture](architecture.md) | `websocket` (npm package `toga-socket-server`) is the real-time push tier for TOGA 2.0 frontends. | websocket/index.js, websocket/auth.js, websocket/rooms.js, websocket/fanout.js, websocket/metadata.js, websocket/ecosystem.config.js, websocket/aws-setup.sh, websocket/package.json |
6
+ | [SQS Fan-out to Socket.io Rooms](features/sqs-fanout.md) | The fan-out logic that turns one SQS `toga-events` message (published by api2 after a write) into `record:changed` emits to every Socket.io room that should lea | websocket/fanout.js, websocket/metadata.js, websocket/rooms.js |
@@ -0,0 +1,90 @@
1
+ ---
2
+ title: WebSocket Server Architecture
3
+ framework: "standalone"
4
+ repo: websocket
5
+ project: WebSocket Server
6
+ client: shared
7
+ type: architecture
8
+ status: active
9
+ updated: 2026-06-29
10
+ owners: [rgirish]
11
+ files:
12
+ - websocket/index.js
13
+ - websocket/auth.js
14
+ - websocket/rooms.js
15
+ - websocket/fanout.js
16
+ - websocket/metadata.js
17
+ - websocket/ecosystem.config.js
18
+ - websocket/aws-setup.sh
19
+ - websocket/package.json
20
+ related:
21
+ - ../../../2.0/apps/_underscore/features/event-publish-sqs.md
22
+ ---
23
+
24
+ ## Summary
25
+
26
+ `websocket` (npm package `toga-socket-server`) is the real-time push tier for TOGA 2.0
27
+ frontends. It is a **standalone Node.js / Socket.io server** — it uses neither PHP
28
+ framework (`_underscore` 2.0 nor `App_` 1.0), which is why it lives under the
29
+ `standalone/` knowledge partition. It consumes the `toga-events` SQS queue published by
30
+ `api2` (via `_Event::publish()` in `_underscore`) and fans out `record:changed` events to
31
+ connected browsers over Socket.io rooms. It ships no client-facing CRUD of its own.
32
+
33
+ **Critical rules:**
34
+ - It is a **consumer** of an event contract owned by the PHP side — the event payload
35
+ shape is defined by `_underscore`'s `_Event::publish()`. Changes to the payload must be
36
+ coordinated across both repos (see the related `event-publish-sqs` feature doc).
37
+ - Room keys use **`clientUuid`**, not `clientId` — the per-tenant isolation boundary.
38
+ - All secrets (`TOGA_JWT_SECRET`/`API_SECRET_ACCESS_TOKEN`, `SQS_*`, `DB_*`) come from env
39
+ (`.env` locally, Elastic Beanstalk env vars in prod) or the Core DB. Never commit values.
40
+ - JWT secrets are **sourced at runtime from `Core.Parameters`** (`API_SECRET_ACCESS_TOKEN`
41
+ + `_PREVIOUS`) so the server stays in sync as api2 rotates them every 3–10 days.
42
+
43
+ ## Stack
44
+
45
+ - **Node.js ≥ 18**, **Socket.io 4.7**, **@aws-sdk/client-sqs 3**, **mysql2 3**,
46
+ **jsonwebtoken 9** (HS256), **dotenv**.
47
+ - Process management via **PM2** (`ecosystem.config.js`, 2 workers, auto-restart).
48
+ - Deployed to **AWS Elastic Beanstalk**; `aws-setup.sh` provisions an instance and is
49
+ reusable across multiple environments.
50
+
51
+ ## Pipeline
52
+
53
+ ```
54
+ api2 PHP write (POST/PUT/PATCH)
55
+ → _underscore _Event::publish() → SQS toga-events queue
56
+ → this server polls SQS (long-poll loop)
57
+ → fanOut() → Socket.io rooms → connected browsers (record:changed)
58
+ ```
59
+
60
+ ## Files
61
+
62
+ | File | Purpose |
63
+ |---|---|
64
+ | `index.js` | HTTP server, Socket.io init, SQS poll loop startup |
65
+ | `auth.js` | JWT validation middleware (HS256, same secret as api2) |
66
+ | `rooms.js` | `subscribe` / `unsubscribe` handlers; `listRoom()` / `instanceRoom()` helpers |
67
+ | `fanout.js` | SQS message → room emit (fan-out to list + instance + parent + child rooms) |
68
+ | `metadata.js` | Core DB cache (route→id, parent/children maps, JWT secrets); refreshed every 5 min |
69
+ | `ecosystem.config.js` | PM2 config |
70
+ | `aws-setup.sh` | Elastic Beanstalk provisioning |
71
+
72
+ ## Room naming
73
+
74
+ ```
75
+ List room: toga:{clientUuid}:record:{recordRoute}
76
+ Instance room: toga:{clientUuid}:record:{recordRoute}:{uuid}
77
+ ```
78
+
79
+ ## Metadata cache (`metadata.js`)
80
+
81
+ A single Core DB connection-per-refresh (every 5 minutes, plus at startup) populates three
82
+ in-memory maps plus the JWT secret list:
83
+
84
+ - `routeMap` — `route` string → `recordId` (from `Core.Records`).
85
+ - `parentMap` — child `recordId` → parent `recordId` (from `Core.InherentRecordChildren`,
86
+ column `inherentChildOfRecordId`).
87
+ - `childrenMap` — parent `recordId` → `Set` of child `recordId`s (inverse of `parentMap`).
88
+ - `jwtSecrets` — `[current, previous]` from `Core.Parameters`, checked in order on verify.
89
+
90
+ Accessors: `getRouteMap()`, `getParentMap()`, `getChildrenMap()`, `getJwtSecrets()`.
@@ -0,0 +1,86 @@
1
+ ---
2
+ title: SQS Fan-out to Socket.io Rooms
3
+ framework: "standalone"
4
+ repo: websocket
5
+ project: WebSocket Server
6
+ client: shared
7
+ type: feature
8
+ status: active
9
+ updated: 2026-06-29
10
+ owners: [rgirish]
11
+ files:
12
+ - websocket/fanout.js
13
+ - websocket/metadata.js
14
+ - websocket/rooms.js
15
+ related:
16
+ - ../../../../2.0/apps/_underscore/features/event-publish-sqs.md
17
+ ---
18
+
19
+ ## What it is
20
+
21
+ The fan-out logic that turns one SQS `toga-events` message (published by api2 after a
22
+ write) into `record:changed` emits to every Socket.io room that should learn about the
23
+ change. Implemented in `fanOut(event)` in `websocket/fanout.js`, driven by a continuous
24
+ SQS long-poll loop.
25
+
26
+ ## How it works
27
+
28
+ `pollLoop()` long-polls SQS (`MaxNumberOfMessages: 10`, `WaitTimeSeconds: 20`), parses each
29
+ message body to an `event`, calls `fanOut(event)`, then deletes the message. Malformed
30
+ bodies are logged and deleted (not retried); fan-out errors are logged without deleting
31
+ fan-out from blocking the loop. On a receive error it backs off 5s before retrying.
32
+
33
+ `fanOut(event)` reads `{ clientUuid, recordRoute, uuid, httpMethod, dtEvent, parentRoute,
34
+ parentUuid }` and emits to up to four kinds of rooms (all keyed by `clientUuid`):
35
+
36
+ 1. **Own list room** — `listRoom(clientUuid, recordRoute)` — always.
37
+ 2. **Own instance room** — `instanceRoom(clientUuid, recordRoute, uuid)` — only when `uuid`
38
+ is present.
39
+ 3. **Inherent children's list rooms** — for each child of the fired record, emit to that
40
+ child's list room with `{ ...payload, parentRoute: recordRoute }`. Resolved from the
41
+ metadata `childrenMap` (parent recordId → Set of child recordIds), looked up via
42
+ `routeMap` to translate the fired `recordRoute` → `recordId` → child recordIds → child
43
+ routes. This lets a list view of children refresh when the parent changes (e.g. a
44
+ `service-requests` write notifies `service-request-bundles`, `service-request-units`,
45
+ and `service-request-notes` list rooms).
46
+ 4. **Parent rooms** — when the event itself is a nested write, notify the parent's list
47
+ room (always) and instance room (only if `parentUuid` present) with
48
+ `{ ...payload, childRoute: recordRoute }`.
49
+
50
+ ### Parent-route resolution from metadata
51
+
52
+ When api2 posts a flat URL with no `parentRoute` in the event, `fanOut()` resolves the
53
+ parent itself: it reuses the `recordId` already looked up for step 3, then reads
54
+ `getParentMap().get(recordId)` (child recordId → parent recordId) and reverse-looks-up the
55
+ parent's route from `routeMap`. So parent notifications work even when the publisher did
56
+ not supply `parentRoute`.
57
+
58
+ ## Frontend subscription
59
+
60
+ The frontend joins rooms by emitting `subscribe` once per route — there is **no server-side
61
+ array support and none is needed**. Multiple sequential emits each join a distinct
62
+ Socket.io room and are effectively simultaneous from the client's perspective:
63
+
64
+ ```js
65
+ const routes = ['service-requests', 'service-request-bundles', 'service-request-units'];
66
+ routes.forEach(route => socket.emit('subscribe', { record: route }));
67
+ socket.on('record:changed', (payload) => { /* refresh the matching list/detail view */ });
68
+ ```
69
+
70
+ ## Gotchas
71
+
72
+ - Room keys are built from **`clientUuid`**, never `clientId` — using `clientId` cross-wires
73
+ tenants or misses rooms entirely.
74
+ - The children-room lookup is a nested scan of `routeMap` (recordId → route) per child; it
75
+ is correct but O(routes × children) — fine at current route counts, watch if routes grow.
76
+ - Children notification depends on `Core.InherentRecordChildren` being populated and the
77
+ metadata cache being fresh (refreshes every 5 min) — a brand-new parent/child relation
78
+ will not fan out until the next refresh.
79
+
80
+ ## Change history
81
+ - 2026-06-29 — Added inherent-children list-room notification: a parent write now emits
82
+ `record:changed` (with `parentRoute`) to each inherent child's list room, via new
83
+ `childrenMap` / `getChildrenMap()` in metadata.js. Parent-route resolution reuses the
84
+ recordId/routeMap lookup from the children step. (rgirish)
85
+ - 2026-06-26 — Initial fan-out loop: SQS long-poll → list + instance + parent rooms, with
86
+ malformed-message drop and fan-out error isolation. (rgirish)
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "toga-ai",
3
- "version": "1.0.233",
3
+ "version": "1.0.235",
4
4
  "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
5
5
  "keywords": [
6
6
  "claude",