toga-ai 1.0.544 → 1.0.545

package/knowledge/2.0/apps/worker2/INDEX.md

@@ -23,6 +23,7 @@
 | [Etilize Catalog Item Import & Refresh](features/etilize-catalog-item-import.md) | Client-generic catalog onboarding from an S3 CSV plus an Etilize re-pull. | worker2/Worker/Etilize/Items.php |
 | [Etilize Item Translation Import](features/etilize-item-translation-import.md) | The abstract worker class `_Worker_Etilize_ItemTranslations` imports **non-English** item text from Etilize into the client's `ItemTranslations` table. | worker2/Worker/Etilize/ItemTranslations.php |
 | [Monitoring Framework (Orchestrator + Child Monitors)](features/monitoring-framework.md) | A unified, DB-driven monitoring framework for business-critical data flows (Compass POs, Prudential asset imports, AIG closed claims, …). | worker2/Worker/Monitor.php, worker2/Worker/Monitors/, worker2/Worker/Monitors/RateEntitlement.php, worker2/Worker/Notification/Email.php, worker2/Worker/Rate.php, dbchanges2/Core/2026-05-21 - Monitors.sql, dbchanges2/Core/2026-06-29a - Rate Entitlement Contract Monitor.sql |
+| [NetSuite Integrations Monitor (Monitor/Operations/NetsuiteIntegrations)](features/netsuite-integrations-monitor.md) | `_Worker_Monitor_Operations::NetsuiteIntegrations()` is a cross-client health check that detects **stuck NetSuite ↔ 2.0 integrations**. | worker2/Worker/Monitor/Operations.php, dbchanges2/Core/2026-08-10a - Netsuite Integrations Monitor.sql, _underscore/Database.php, _underscore/Query.php |
 | [NetSuite ↔ ClickUp / TOGA Opportunity Sync (API Message Queue + worker2 webhook)](features/netsuite-opportunity-sync.md) | Outbound sync from NetSuite to TOGA for the record types the Forecast2 importer pulls (opportunities first; sales/items/etc. | worker2/Worker/Netsuite.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Worker/Clickup.php, worker2/Worker/Clickup/Opportunity.php, worker2/Controller/Index.php, _underscore/Worker.php, test/@dave/NetSuite/api-message-queue/lib_amq_queue.js, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/NetSuite/api-message-queue/ue_amq_drain.js, test/@dave/NetSuite/api-message-queue/ss_amq_drain.js, test/@dave/NetSuite/api-message-queue/DEPLOY_RUNBOOK.md, test/@dave/clickup/backfill_opportunity_numbers.php, test/@dave/clickup/probe_opportunity_fields.php, test/@dave/probe_clickup_desc_match.php, test/@dave/test_model_load_behavior.php, dbchanges2/Forecast/2026-06-25a - Add unique index on Opportunities netsuiteOpportunityInternalId.sql, _underscore/Model/Forecast/Opportunity.php, test/@dave/approach/TRUE-80044.md, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
 | [NetSuite → Forecast Open-Orders Sync (salesOrder webhook → OpenOrderItems)](features/netsuite-salesorder-open-orders-sync.md) | Webhook-driven, single-record port of the legacy open-orders importer (TRUE-79142). | worker2/Worker/Netsuite/SalesOrder.php, worker2/Worker/Netsuite.php, worker2/Component/Forecast/Db/Db.php, worker2/Worker/Netsuite/Location.php, test/@dave/Junk Drawer/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/Junk Drawer/NetSuite/api-message-queue/ue_amq_invoice_resync_salesorder.js, test/@dave/probe_salesorder_rest_shape.php, test/@dave/probe_open_order_lines.php, test/@dave/check_so_status.php, test/@dave/check_so_history.php, test/@dave/probe_so_rest_lines.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_open_order_gating.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
 | [NetSuite Supporting-Record Webhook Importer (the reusable recipe)](features/netsuite-supporting-record-webhook-importer.md) | A single **repeatable recipe** for porting a legacy daily-pull NetSuite *supporting-record* importer (the lookup/dimension tables behind Forecast2 — Employees, | worker2/Worker/Netsuite/Employee.php, worker2/Worker/Netsuite/Account.php, worker2/Worker/Netsuite/Classification.php, worker2/Worker/Netsuite/Customer.php, worker2/Worker/Netsuite/Item.php, worker2/Worker/Netsuite.php, _underscore/Model/Forecast/Employee.php, _underscore/Model/Forecast/Account.php, _underscore/Model/Forecast/Classification.php, _underscore/Component/Forecast/Db/Db.php, test/@dave/test_employee_lifecycle.php, test/@dave/test_account_lifecycle.php, test/@dave/test_classification_lifecycle.php, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, worker/crons/toga2/forecast2/import_supporting_records.php |

package/knowledge/2.0/apps/worker2/features/all-client-email-queue-monitor.md

@@ -15,6 +15,7 @@ files:
   - dbchanges2/Logs_Client/2026-05-21 - Email.sql
 related:
   - ./oneuptime-worker2-monitoring.md
+  - ./netsuite-integrations-monitor.md
   - ./monitoring-framework.md
   - ../../_underscore/features/email-send-pipeline.md
   - ../../_underscore/features/per-client-database-connections.md

package/knowledge/2.0/apps/worker2/features/netsuite-integrations-monitor.md

@@ -0,0 +1,139 @@
+---
+title: NetSuite Integrations Monitor (Monitor/Operations/NetsuiteIntegrations)
+framework: "2.0"
+repo: worker2
+project: Worker
+client: shared
+type: feature
+status: active
+updated: 2026-08-10
+owners: ["jcardinal"]
+files:
+  - worker2/Worker/Monitor/Operations.php
+  - dbchanges2/Core/2026-08-10a - Netsuite Integrations Monitor.sql
+  - _underscore/Database.php
+  - _underscore/Query.php
+related:
+  - ./all-client-email-queue-monitor.md
+  - ./oneuptime-worker2-monitoring.md
+  - ./monitoring-framework.md
+  - ../../_underscore/features/per-client-database-connections.md
+---
+
+## Summary
+
+`_Worker_Monitor_Operations::NetsuiteIntegrations()` is a cross-client health check that
+detects **stuck NetSuite ↔ 2.0 integrations**. It sweeps **every** client's business
+database (`Client_<X>`), looks in the `Parameters` table for rows where
+`` `key` `` LIKE `NETSUITE_EXECUTION_MODE_%` and `` `value` `` LIKE `1-RUNNING`, and pushes
+one aggregate payload to a single OneUptime Incoming Request monitor with the **offending
+integrations named** in the body. A row parked at the `1-RUNNING` sentinel means an
+integration hung mid-run.
+
+This is the second all-client monitor in `_Worker_Monitor_Operations`, modeled exactly on
+its sibling [EmailQueue](./all-client-email-queue-monitor.md) — a "dumb reporter, smart
+monitor": the worker only reports measured values and OneUptime holds the alarm
+thresholds. It is **shared internal operational infrastructure**, not a client feature.
+
+## Key files / entry points
+
+- `worker2/Worker/Monitor/Operations.php` — `abstract class _Worker_Monitor_Operations`.
+  - `public static function NetsuiteIntegrations(): string` — the action, route
+    `Monitor/Operations/NetsuiteIntegrations`.
+  - `private static function resolveClientDatabases(): array` — one `DB_CORE` query
+    returning every client + its **business** database name (`Client_<X>`) + this
+    environment's hosts. Mirrors `resolveClientLogDatabases()` but joins `Core.Clients` on
+    `clientDatabaseId` (not `logDatabaseId`).
+- `dbchanges2/Core/2026-08-10a - Netsuite Integrations Monitor.sql` — the `Core.CronJobs`
+  seed row (see below).
+- Dispatched by action string, not PHP import — the autoloader + `_Worker::runTask()`
+  reach the class via the cron action path.
+
+## How it works
+
+1. **Resolve every client's business database.** `resolveClientDatabases()` joins
+   `Clients` → `Databases` (via `Clients.clientDatabaseId`) → `DatabaseHosts` →
+   `Environments`/`Regions`, all **LEFT OUTER JOINs** so a misconfigured client is
+   reported, not silently dropped (`unresolvedClients`). Closest region first; later rows
+   for the same client are skipped.
+2. **Register each client's business DB under its REAL database name, no alias**, guarded
+   by `if (!isset(_Database::$_registers[$dbName]))`, then query via `_Query`. A per-client
+   loop cannot reuse a shared alias — every iteration would clobber it.
+3. **Sweep `Parameters` per client:**
+   `` SELECT `key`, `value` FROM Parameters WHERE `key` LIKE 'NETSUITE_EXECUTION_MODE_%' AND `value` LIKE '1-RUNNING' ``.
+   The integration **type** is the key with the `NETSUITE_EXECUTION_MODE_` prefix stripped
+   via `substr()` (e.g. `NETSUITE_EXECUTION_MODE_SALES_ORDERS` → `SALES_ORDERS`).
+4. **Bucket each client** into `checked`, `skipped` (no `Parameters` table —
+   `isMissingTableError()`), or `unreachable` (any other query/connection failure). A
+   per-client `try/catch (Throwable)` keeps one bad client from failing the sweep.
+5. **Decide the tokens and push once** to OneUptime.
+
+### Payload contract — `alarm` and `probe` are SEPARATE tokens
+
+Follows the [OneUptime push-metric pattern](./oneuptime-worker2-monitoring.md): OneUptime
+can only string-match a body, so the worker decides and emits tokens.
+
+| Field | Meaning |
+|---|---|
+| `status` | `reporting`, or `error` (reserved for the Core lookup itself failing → `error:client_lookup_failed`) |
+| `alarm` | `DEGRADED` when any stuck integration is found, else `OK` |
+| `probe` | `DEGRADED` when any client DB is unreachable, else `OK` |
+| `stuckCount` | number of stuck integrations found |
+| `offendingIntegrations` | comma-joined `"ClientName: TYPE"` |
+| `clientsChecked`, `clientsSkipped` | coverage counters |
+| `unreachableClients`, `unresolvedClients` | named coverage gaps |
+| `checkedAtUtc` | `gmdate('c')` |
+
+## Cron seed (dbchanges2)
+
+`dbchanges2/Core/2026-08-10a - Netsuite Integrations Monitor.sql` — INSERT into
+`Core.CronJobs`: action `Monitor/Operations/NetsuiteIntegrations`, schedule
+`*/10 * * * *` (every 10 min), `maxExecutionTime` 120, `isActive` 1, `parameters` NULL,
+fixed random uuid. Targets **only** `Core` (no cross-cluster refs), INSERT-only and
+unguarded per dbchanges2 conventions. Modeled on
+`Core/2026-07-29a - AIG Entitlement API Failure Monitor.sql`.
+
+**Shipped `isActive=1` while the OneUptime URL is still a placeholder** (developer's
+choice): the sweep runs but the push fails soft (`error_log`) until the real URL is pasted
+into the worker method.
+
+## Gotchas / known issues
+
+- **`` `key` `` and `` `value` `` are MySQL reserved words** — backtick-quote them in the
+  `Parameters` query.
+- **Single source of truth for the key prefix.** `$KEY_PREFIX` builds both the `LIKE`
+  pattern (via `_Database::escape`) and the `substr()` that derives the integration type;
+  never hardcode the prefix twice.
+- **Never-throw / 200-on-all-paths.** The method always returns a summary string. The
+  OneUptime push is `setLogging(false)` + `setThrowExceptionsOnFailure(false)` **and**
+  wrapped in its own `try/catch`, so a failed ping never fails the worker job.
+- **OneUptime push URL is a credential** — it lives only as an in-file value in the worker
+  method, never committed to dbchanges2 and never recorded in a doc. It is currently a
+  placeholder.
+- **Business DB, not logs DB.** Join `Clients.clientDatabaseId` for `Client_<X>`; the
+  sibling EmailQueue monitor joins `logDatabaseId` for `Logs_<X>`. See
+  [per-client database connections](../../_underscore/features/per-client-database-connections.md).
+- The all-client resolution gotchas from
+  [EmailQueue](./all-client-email-queue-monitor.md) apply verbatim: LEFT OUTER JOIN every
+  level, null host columns unless the environment join matched, detect missing table by
+  MySQL error **number** 1146 (not message text), and remember a passing dev run (single
+  all-in-one endpoint) does not prove cross-cluster correctness.
+
+## Change history
+
+- 2026-08-10 — Built `NetsuiteIntegrations()` + `resolveClientDatabases()` on
+  `_Worker_Monitor_Operations` (all-client `Parameters` sweep for
+  `NETSUITE_EXECUTION_MODE_%` = `1-RUNNING`), plus the `Core.CronJobs` seed
+  (`*/10 * * * *`, `isActive=1`) and an OneUptime Incoming Request monitor import. OneUptime
+  URL still a placeholder; sweep runs and pushes fail-soft until pasted. (jcardinal)
+
+## Related docs
+
+- [All-Client Email Queue Monitor](./all-client-email-queue-monitor.md) — the sibling
+  all-client monitor this one is modeled on.
+- [OneUptime push-metric monitors for 2.0 workers](./oneuptime-worker2-monitoring.md) — the
+  push/token pattern and OneUptime criteria (incl. `{{requestBody.*}}` incident templating).
+- [Per-Client Database Connections](../../_underscore/features/per-client-database-connections.md)
+  — business/log database name resolution.
+</content>
+</invoke>

package/knowledge/2.0/apps/worker2/features/oneuptime-worker2-monitoring.md

@@ -111,6 +111,25 @@ log dive. `status: "error"` stays reserved for the checker being **fully** blind
 resolution query failed). Worked example:
 [All-Client Email Queue Monitor](./all-client-email-queue-monitor.md).
 
+### Incident / alert title & description templating (`{{requestBody.*}}`)
+
+OneUptime incident and alert **titles and descriptions support `{{variable}}` templating**
+(verified against OneUptime docs 2026-08-10). For an **Incoming Request** monitor, the
+POSTed JSON body is addressed via the `requestBody` prefix — e.g.
+`{{requestBody.offendingIntegrations}}`, `{{requestBody.stuckCount}}`,
+`{{requestBody.checkedAtUtc}}`, `{{requestBody.message}}`. Other available vars:
+`{{requestHeaders}}`, `{{requestMethod}}`, `{{incomingRequestReceivedAt}}`.
+
+- **Missing paths resolve to empty string**, so one template covers both a normal degraded
+  payload and an edge case like `error:client_lookup_failed` without a second template.
+- Body-content criteria use `checkOn: "Request Body"` / `filterType: "Contains"`; the
+  heartbeat criterion uses `"Not Recieved In Minutes"` (note OneUptime's misspelling).
+- A degraded criterion can OR several tokens, e.g. Contains `"alarm":"DEGRADED"` **OR**
+  `"error":"client_lookup_failed"`.
+- Reuse the project's existing status/severity ObjectIDs when authoring the import JSON.
+- Docs: https://oneuptime.com/docs/monitor/incident-alert-templating and
+  https://oneuptime.com/docs/en/monitor/incoming-request-monitor
+
 ### Heartbeat / cron cadence timing
 
 The 10-min-Degraded / 15-min-Offline heartbeat thresholds pair with a **5-minute** cron
@@ -192,6 +211,10 @@ check for another client.
   monitors must pass the region (see [Cloud S3 helpers](../../_underscore/features/cloud-s3-helpers.md)).
 
 ## Change history
+- 2026-08-10 — Documented OneUptime incident/alert `{{requestBody.*}}` templating (missing
+  paths → empty string; `Not Recieved In Minutes` heartbeat spelling), discovered while
+  wiring the second all-client `_Worker_Monitor_Operations` monitor —
+  [NetSuite Integrations Monitor](./netsuite-integrations-monitor.md). (jcardinal)
 - 2026-07-30 — Added the multi-client refinement of the payload contract: a **separate `probe`
   token** (`OK`/`DEGRADED`) alongside `alarm`, so a partial logs-cluster outage cannot read as a
   healthy metric, plus naming the offending clients in the body. Recorded that the previously

package/knowledge/INDEX.md

@@ -19,7 +19,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 ## 2.0 framework
 
 - **_underscore** (_Underscore) _(framework core)_ — 48 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
-- **worker2** (Worker) — 42 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
+- **worker2** (Worker) — 43 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
 - **api2** (API) — 22 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
 - **dbchanges2** (Database Changes) _(framework core)_ — 8 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
 - **toga2-supply** (TOGa Supply) — 5 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.544",
+  "version": "1.0.545",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",