toga-ai 1.0.933 → 1.0.935

package/knowledge/1.0/apps/library/features/toga2-api-client-and-bridge.md

@@ -23,7 +23,7 @@ related:
   - ../../worker/features/netsuite-togasupply-per-client-sync.md
   - ../architecture.md
   - ./error-capture-1-0.md
-  - ../../../clients/wje/features/oneuptime-ticket-writes-monitor.md
+  - ../../../../clients/wje/features/oneuptime-monitors.md
   - ../../../clients/elite/features/togadesk-service-request-intake.md
 ---
 
@@ -147,7 +147,7 @@ A large bi-directional sync gated by **two checkpoint parameters** stored in 2.0
 Signature takes `$integrationType` (`TICKETS` or `REPAIR_ORDERS`) plus **seven independent enable flags** and an optional `$monitorTogadeskDepartmentIds[]`:
 - **2.0 → TOGaDesk** (`$isEnabledToga2ToTogadeskIntegration`): pages 2.0 `/tickets` and `/ticket-notes` changed since the checkpoint; routes each to `syncToga2TicketIntoTogadesk1{RepairOrder|Ticket}` by `$integrationType`. Per-ticket exceptions are caught and sent to Sentry so one bad ticket doesn't abort the batch.
 - **TOGaDesk → 2.0** (`$isEnabledTogadeskToToga2Integration`): queries the 1.0 `db_togadesk` DB directly (SQL over `repair_orders`/`repair_order_history` or `tickets`/`tickets_replies`/`comments`, using `GREATEST(...)` of reply/comment timestamps) for the client's department(s) — only the ones the cron passes (`$monitorTogadeskDepartmentIds`) — then calls `syncTogaDesk1{RepairOrder|Ticket}IntoToga2Ticket($id)`.
-  - **Link = `referenceId`.** `tickets.referenceId` = 2.0 ticket uuid; `tickets_replies.referenceId` / `comments.referenceId` = 2.0 ticket-note uuid. Written back **only after a successful POST**. Rows created by the 2.0 → 1.0 direction already carry it. So NULL/`''` `referenceId` = not (yet) in 2.0 — the basis of the [WJE ticket-writes monitor](../../../clients/wje/features/oneuptime-ticket-writes-monitor.md).
+  - **Link = `referenceId`.** `tickets.referenceId` = 2.0 ticket uuid; `tickets_replies.referenceId` / `comments.referenceId` = 2.0 ticket-note uuid. Written back **only after a successful POST**. Rows created by the 2.0 → 1.0 direction already carry it. So NULL/`''` `referenceId` = not (yet) in 2.0 — the basis of the [WJE ticket-writes monitor](../../../../clients/wje/features/oneuptime-monitors.md).
   - A reply with a blank message falls back to `newstatus`; if still PHP-empty (NULL, `''`, `'0'`) it is skipped and never gets a `referenceId`. Comments are always sent.
 - Five further one-way 2.0 → TOGaDesk sub-syncs, each its own flag: Contacts→People, Items/Units→Assets, Predefined-Replies→TicketsPr, Ticket-Teams→Groups, Ticket-Categories→Category/Subcategory/Items.
 - Finishes by PUT-ing both checkpoint parameters back to 2.0.

package/knowledge/2.0/apps/worker2/features/oneuptime-worker2-monitoring.md

@@ -33,7 +33,7 @@ related:
   - ../../../clients/staples/features/oneuptime-sftp-folder-monitors.md
   - ../../../clients/compass-usa/features/oneuptime-ma-refresh-order-monitor.md
   - ../../../clients/compass-usa/features/oneuptime-netsuite-duplicate-sales-orders-monitor.md
-  - ../../../clients/wje/features/oneuptime-ticket-writes-monitor.md
+  - ../../../../clients/wje/features/oneuptime-monitors.md
 ---
 
 OneUptime push-metric monitor pattern for worker2 (2.0): a self-contained cron action decides pass/fail and POSTs a string token to a OneUptime Incoming Request monitor; open when building/tuning a worker2 monitor, its thresholds, or the OneUptime import JSON.
@@ -309,7 +309,7 @@ This is about `Common` only — **worker2 does read other legacy DBs.** `worker2
 `[database1]` config group), already used in prod by `Worker/Sync/SalesOrderStatus.php` and
 `Worker/Client/Aig/ClosedClaims.php`. A 2.0 monitor over TOGaDesk data is possible and preferred (it
 runs outside the 1.0 worker it watches) — e.g. the
-[WJE ticket-writes monitor](../../../clients/wje/features/oneuptime-ticket-writes-monitor.md).
+[WJE monitor suite](../../../../clients/wje/features/oneuptime-monitors.md).
 
 ### Match the lateness criterion to the flow's real time-scale
 
@@ -345,6 +345,11 @@ pattern (first instance: `EmailedAsnImport()`, which intermittently paged on a t
 - **Catch `\Throwable`, not `Exception`,** in the loop so a non-`Exception` `Error` (e.g. from the
   php-imap2 C-extension calls) can't escape the method, skip the socket-timeout restore, or break this
   worker action's "always return a string" invariant.
+- **A delayed re-read must call `_Database::resetQueryCache()` first** — an identical second `_Query`
+  in the same run is served from the query cache, so the "retry" sees the first result. Use a
+  sleep-then-re-read when a row can legitimately arrive already "old" (e.g. WJE
+  `TicketSyncToTogadesk`: a note's `dtStamp` is its Freshservice time) — see
+  [WJE monitor suite](../../../../clients/wje/features/oneuptime-monitors.md).
 
 ### Two kinds of monitor — liveness probes and log-scrapers are complements, not duplicates
 
@@ -568,5 +573,6 @@ production deploy.**
 - [OneUptime 1.0 worker uptime monitoring](../../../1.0/apps/worker/features/oneuptime-worker-uptime-monitoring.md) — the 1.0 push-heartbeat predecessor
 
 ## Change history
+- 2026-10-01 — Added the `resetQueryCache()` re-read trap to the in-run retry pattern; WJE link now points at the 3-monitor suite. (mhammontree)
 - 2026-10-01 — Clarified worker2 reads `TOGaDeskSupport` (`DB_TOGADESK1`) — only `Common` is unreachable; added `Monitor/Wje`. (mhammontree)
 - 2026-09-25 — Added `NetsuiteDuplicateSalesOrders`, the first monitor that queries NetSuite (SuiteQL) instead of a Toga DB. (rgirish)

package/knowledge/2.0/apps/worker2/features/wje-freshservice-sync.md

@@ -28,7 +28,7 @@ files:
   - dbchanges2/Client_Wje/2024-10-04 - WjeOnboarding.sql
 related:
   - clients/wje/profile.md
-  - clients/wje/features/oneuptime-ticket-writes-monitor.md
+  - clients/wje/features/oneuptime-monitors.md
   - 2.0/apps/worker2/features/elite-freshservice-sync.md
   - 1.0/apps/library/features/toga2-api-client-and-bridge.md
   - 2.0/apps/worker2/features/startech-webhook-handler.md
@@ -64,7 +64,7 @@ a **daily batch import** of supporting records. A separate 2-minute cron bridges
   `crons/toga2/wje/sync_togasupply_wje.php` (2-min Togadesk bridge),
   `crons/notifications/reports/wje/{wje_monthly_report,wje_quarterly_report,wje_common}.php`.
 - **Models:** `_underscore/Model/Wje/*` (8 classes, see Data model).
-- **Monitoring:** `library/app/systemmonitor/wje.php` (`App_SystemMonitor_WJE`, Freshservice/webhook errors); `worker2/Worker/Monitor/Wje.php` ([TOGaDesk → 2.0 write monitor](../../../clients/wje/features/oneuptime-ticket-writes-monitor.md)).
+- **Monitoring:** `library/app/systemmonitor/wje.php` (`App_SystemMonitor_WJE`, Freshservice/webhook errors); `worker2/Worker/Monitor/Wje.php` ([WJE OneUptime monitor suite](../../../../clients/wje/features/oneuptime-monitors.md): webhook → 2.0, TOGaDesk → 2.0, 2.0 → TOGaDesk, TOGaDesk/2.0 → Freshservice).
 
 ### Real-time webhook (`_Worker_Wje::Webhook`)
 
@@ -124,7 +124,7 @@ Hardcoded IDs: `CLIENT_UUID_WJE=c2f5180f-…`, `API_UUID_WJE=4750650f-…` (and
 
 - **W1** Payload is wrapped under `freshdesk_webhook`, not top-level. Decode with `json_decode($p, true)` then cast.
 - **W2** Webhook body is thin — `ticket_category`/`group_id` often null; the handler re-fetches `/tickets/{id_numeric}`. Don't trust the webhook body for category/group.
-- **W3** Two id forms: `ticket_id` (string "INC-46687", stored in `c_wjeTicketId`) vs `id_numeric` (46687, used in API calls). Mixing them → 404s / missed matches.
+- **W3** Two id forms: `ticket_id` (string `"INC-66452"`) → stored in `Tickets.number`; `id_numeric` (`66452`) → stored in `c_wjeTicketId` and used in API calls (`Worker/Wje.php` ~L563-564). Mixing them → 404s / missed matches.
 - **W4** Webhook resolves category by **name** against TOGA `TicketCategories`; the **daily import** is what populates `c_wjeCategoryId`. A WJE rename before the import runs breaks name matching — check the import cron first.
 - **W5** Units keyed by asset tag `ASSET-{display_id}`, linked via `c_wjeAssetId`; asset webhook falls back to email matching when `c_wjeUserId` is absent.
 - **W6** Asset import catches per-asset errors and continues; if WJE revokes `/assets` API access it silently skips assets — check logs.
@@ -134,9 +134,10 @@ Hardcoded IDs: `CLIENT_UUID_WJE=c2f5180f-…`, `API_UUID_WJE=4750650f-…` (and
 - **W10** Lazy-transaction rule: write back `c_wje*` identifiers via direct model `->save()`, not an internal API round trip inside an open transaction (cf. Startech G4).
 - **W11** `json_decode($s, true)` + top-level `(object)` cast leaves nested fields as arrays — cast each level you traverse.
 - **W12** TicketStages use Freshservice status codes (Open=2 … Closed=5); status sync depends on those matching live Freshservice statuses.
-- Secrets (`API_SECRET_WJE`, the Freshservice password) are hardcoded in source — read them from the file when needed; never copy the values into docs or commits.
+- **Open item — hardcoded credentials (parked 2026-10-01, separate ticket needed):** `API_SECRET_WJE` is a class const literal in `worker2/Worker/Wje.php`; the Freshservice API key is a literal in `_underscore/Component/Api/Wje/Wje.php` `initialize()` (plus the 1.0 copy, W9). Fix = move to config + rotate. Never copy the values into docs or commits.
 
 ## Change history
+- 2026-10-01 — Corrected W3 (`c_wjeTicketId` = `id_numeric`, `INC-…` → `number`); recorded hardcoded-credential locations as an open item; linked the monitor suite. (mhammontree)
 - 2026-10-01 — Corrected dept 290 (not synced 1.0 → 2.0); linked the new ticket-writes monitor. (mhammontree)
 
 ## Related

package/knowledge/clients/wje/INDEX.md

@@ -2,5 +2,5 @@
 
 | Doc | Framework | Summary |
 |-----|-----------|---------|
-| [WJE: TOGaDesk → TOGa 2 ticket-writes monitor (Monitor/Wje/TicketWritesToToga2)](features/oneuptime-ticket-writes-monitor.md) | 2.0 | worker2 OneUptime push monitor counting WJE TOGaDesk tickets/replies/comments never written to TOGa 2 (NULL referenceId); open when working on WJE bridge monito |
+| [WJE: OneUptime monitor suite (Monitor/Wje — TicketWritesToToga2, TicketSyncToTogadesk, WebhookTicketSync, TicketTransmissionToFreshservice)](features/oneuptime-monitors.md) | 2.0 | Four worker2 OneUptime push monitors on the WJE ticket flow (Freshservice webhook → TOGa 2, TOGaDesk → TOGa 2, TOGa 2 → TOGaDesk, TOGaDesk/TOGa 2 → Freshservice |
 | [Wiss, Janney, Elstner Associates, Inc.](profile.md) | 2.0 | WJE (Wiss, Janney, Elstner / "WJE IT") Freshservice help-desk client, single hardcoded tenant; open for the three sync paths, the TOGaDesk bridge ids/department |

package/knowledge/clients/wje/features/oneuptime-monitors.md

@@ -0,0 +1,131 @@
+---
+title: "WJE: OneUptime monitor suite (Monitor/Wje — TicketWritesToToga2, TicketSyncToTogadesk, WebhookTicketSync, TicketTransmissionToFreshservice)"
+framework: "2.0"
+repo: worker2
+project: Worker
+client: wje
+type: client-feature
+status: active
+updated: 2026-10-01
+owners: ["mhammontree"]
+files:
+  - worker2/Worker/Monitor/Wje.php
+  - worker2/Worker/Client/Wje.php
+  - dbchanges2/Core/2026-09-30b - Insert WJE Ticket Writes Monitor CronJob.sql
+  - dbchanges2/Core/2026-10-01a - Insert WJE Ticket Sync To TOGaDesk Monitor CronJob.sql
+  - dbchanges2/Core/2026-10-01b - Insert WJE Freshservice Webhook Sync Monitor CronJob.sql
+  - dbchanges2/Core/2026-10-01c - Insert WJE Ticket Transmission To Freshservice Monitor CronJob.sql
+  - worker/crons/toga2/wje/sync_togasupply_wje.php
+  - library/app/api/toga2.php
+  - library/app/systemmonitor/wje.php
+related:
+  - ../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md
+  - ../../../1.0/apps/library/features/toga2-api-client-and-bridge.md
+  - ../../../2.0/apps/worker2/features/wje-freshservice-sync.md
+  - ../profile.md
+---
+Four worker2 OneUptime push monitors on the WJE ticket flow (Freshservice webhook → TOGa 2, TOGaDesk → TOGa 2, TOGa 2 → TOGaDesk, TOGaDesk/TOGa 2 → Freshservice); open when working on WJE monitoring, its parameters, activation, or clearing a stuck ticket.
+
+## Summary
+`_Worker_Monitor_Wje` (`worker2/Worker/Monitor/Wje.php`) holds four actions, one per hop. All follow the shared [OneUptime push-metric pattern](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md) — payload contract, criteria, provisioning live there.
+
+| Action | Ticket | Hop | OneUptime monitor name | Cron |
+|---|---|---|---|---|
+| `Monitor/Wje/WebhookTicketSync` | TRUE-80520 | Freshservice webhook → `Client_Wje.Tickets` | "WJE Freshservice Webhook Sync" | 5 min, max 120s |
+| `Monitor/Wje/TicketWritesToToga2` | TRUE-80523 | TOGaDesk dept 269 → TOGa 2 | "WJE Ticket Writes to TOGa 2" | 5 min |
+| `Monitor/Wje/TicketSyncToTogadesk` | TRUE-80521 | TOGa 2 → TOGaDesk | "WJE Ticket Sync to TOGaDesk" | 5 min, max 300s |
+| `Monitor/Wje/TicketTransmissionToFreshservice` | TRUE-80522 | TOGaDesk / TOGa 2 → WJE Freshservice | "WJE Ticket Transmission to Freshservice" | 5 min, max 120s |
+
+- **Status 2026-10-01:** all four committed on the one shared branch `TRUE-80523` (worker2, dbchanges2, worker comment), **not deployed**. Seeds ship `isActive = 0`. Push URLs are credentials — never record them.
+- DBs: `TOGaDeskSupport` via `_underscore::DB_TOGADESK1` (legacy); `Client_Wje` via `_Worker_Client_Wje::DB_CLIENT_WJE` (`worker2/Worker/Client/Wje.php`, mirrors `Client/Prudential.php`, registered in `initialize()`); `Logs` via `_underscore::DB_LOGS` (registered per job in worker2 `Controller/Index.php`). Each query stays in one DB family; compare in PHP.
+- DB failures go through `reportQueryFailure(array $params)` → `status: error` with **only** `errorClass` + `errorNumber` (the message can hold SQL, host, user).
+
+## How it works
+
+### Why backlogs, not checkpoint freshness
+The TOGaDesk bridge catches a per-ticket exception → Sentry only, and **still advances its checkpoint** (`TOGA_LAST_TICKET_INTEGRATION_DATETIME` / `TOGADESK_LAST_TICKET_INTEGRATION_DATETIME`). A failed ticket is never retried unless touched again, so a checkpoint-age check reads healthy. All four monitors look for rows that should exist on the far side and do not (link model: [bridge doc](../../../1.0/apps/library/features/toga2-api-client-and-bridge.md)).
+
+### TicketWritesToToga2 (TOGaDesk → TOGa 2)
+- Counts WJE rows (`tickets.clientid = 153`) in `tickets`, `tickets_replies`, `comments` with `referenceId` NULL or `''`, created on/after `BACKLOG_CUTOFF_DATE = '2025-05-01'`, in `departmentIds`, minus `excludedTicketNumbers`.
+- Skip rules mirror `syncTogaDesk1TicketIntoToga2Ticket` (`library/app/api/toga2.php` ~L12280-12290): a reply with blank message falls back to `newstatus` and is skipped if still PHP-empty (NULL, `''`, `'0'`); it never gets a `referenceId`, so the monitor skips it too. Comments are always sent.
+- `HIGH` when any counted row is `>= alertThresholdMinutes` old. Empty `departmentIds` → `status: error`, `error: no_departments_configured`.
+- Body: `latePendingCount`, `pendingCount`, per-type counts, `oldestPendingMinutes`, `alertThresholdMinutes`, `lateTicketNumbers` (top 10), `departmentIds`, `excludedTicketCount`.
+- Params: `{"alertThresholdMinutes":15,"departmentIds":[269],"excludedTicketNumbers":[]}`. PHP default for `departmentIds` is `[]` **on purpose** — the DB row is the only home of the list. The cron comment in `sync_togasupply_wje.php` points here; keep them in step.
+
+### TicketSyncToTogadesk (TOGa 2 → TOGaDesk)
+- Mapping (from `syncToga2TicketIntoTogadesk1Ticket`): only tickets with `c_wjeTicketId` are synced; note `isExternal = 1` → `tickets_replies.referenceid`, else → `comments.referenceid`; 1.0 dept = `ticketTeam.c_togadeskDepartmentId`, fallback 269 → so **no dept filter**.
+- Query A (`Client_Wje`): `Tickets` with `c_wjeTicketId` (age by `dtCreated`) + their `TicketNotes` (age by `dtStamp` — **`TicketNotes` has no `dtCreated`**), inside window `GREATEST(CAST('2026-09-24' AS DATETIME), NOW() - lookbackDays)` and older than the threshold.
+- Query B (`TOGaDeskSupport`): `referenceId`s of clientid 153 tickets / replies / comments with `timestamp >=` window minus 1 day (the sync copies 2.0 `dtCreated`/`dtStamp` into 1.0 `timestamp`). Compare per type in PHP (trim + lowercase).
+- **Missing → confirm before alarming:** sleep 150s, `_Database::resetQueryCache()`, re-read; `HIGH` only if still missing. A note's `dtStamp` is its Freshservice time, so it can arrive already "old".
+- Params: `{"alertThresholdMinutes":15,"lookbackDays":30,"excludedTicketNumbers":[]}`. Rows stuck longer than `lookbackDays` drop off; use `excludedTicketNumbers` for abandoned or hand-deleted ones.
+- Cutoff 2026-09-24: see Gotchas (21-ticket gap).
+
+### WebhookTicketSync (Freshservice webhook → TOGa 2)
+- Reads `Logs.Webhook` rows with route `WJE_WEBHOOK_ROUTE = '/wje'` in the last `lookbackHours`, groups by payload `freshdesk_webhook.id_numeric` (`JSON_VALID`-guarded), age from the **first** webhook.
+- For ids `>= alertThresholdMinutes` old, checks `Client_Wje.Tickets.c_wjeTicketId` (= `id_numeric`, set by `_Worker_Wje::Webhook`, which has no skip rule for ticket webhooks). Missing → `HIGH`.
+- `id_numeric` comes from a public endpoint → validated as a positive int; others counted in `skippedPayloadCount`.
+- **Scan bound:** `Logs.Webhook` has no index on `dtStamp`/`route` (PK `id` only; ~6k rows/day all clients, ~175k rows). The query bounds on PK: `id > (SELECT GREATEST(MAX(id), N) - N)` with `N = maxScannedWebhookRows` (`GREATEST` avoids a negative bound on small tables). Payload sets `scanWindowShort` when the scanned span < `lookbackHours`.
+- Silence (`minutesSinceLastWebhook`) is reported, **never alarmed** — long quiet spells are normal (see Gotchas).
+- Params: `{"alertThresholdMinutes":30,"lookbackHours":24,"maxScannedWebhookRows":20000,"excludedTicketNumbers":[]}`; PHP defaults are the `DEFAULT_WEBHOOK_*` / `DEFAULT_MAX_SCANNED_WEBHOOK_ROWS` constants.
+
+### TicketTransmissionToFreshservice (TOGaDesk / TOGa 2 → Freshservice)
+- **Sends happen only in 1.0** `App_Api_Toga2::syncTogaDesk1TicketIntoToga2Ticket` (`library/app/api/toga2.php`). `App_Api_Wje::send` throws on non-200/201.
+  - New TOGaDesk ticket → `POST /tickets` only when `$isWjeUuid && !empty($togadeskPerson->referenceId)` (~L12133-12174). Freshservice id saved on 2.0 `Tickets.c_wjeTicketId`.
+  - TOGaDesk reply on a ticket with `c_wjeTicketId` → `POST /tickets/{id}/reply` (~L12298). Id saved on 2.0 `TicketNotes.c_wjeConversationId` (~L12469).
+  - Status/field change → `PUT /tickets/{id}` (~L12775-12877). **Nothing is saved** → blind spot (see Gotchas).
+  - Internal comments are never sent (that code is commented out).
+- Check 1 (new tickets): TOGaDesk tickets (`clientid = 153`, in `departmentIds`) since `FRESHSERVICE_TRANSMISSION_CUTOFF_DATE = '2026-10-01'`, `>= alertThresholdMinutes` old, `people.referenceId` set, `tickets.referenceId` set → read 2.0 `Tickets` by uuid. Present with no `c_wjeTicketId` → `HIGH`. Uuid absent from 2.0 is `TicketWritesToToga2`'s job — not counted here.
+- Check 2 (replies): 2.0 external `TicketNotes` on tickets with `c_wjeTicketId`, since the cutoff, `>= alertThresholdMinutes` old, no `c_wjeConversationId` → `HIGH`.
+- Requester rule = the TOGaDesk link (`people.referenceId`), **not** the 2.0 contact's `c_wjeUserId` — the monitor follows what the sync does (see Gotchas).
+- `lateTicketNumbers` labels: `TD-<togadesk number>` (new ticket) / `WJE-<freshservice number>` (reply). One `excludedTicketNumbers` list serves both — the ranges cannot overlap (TOGaDesk WJE numbers start `153000001`; Freshservice ~66,000 in 2026).
+- Params: `{"alertThresholdMinutes":15,"departmentIds":[269],"excludedTicketNumbers":[]}`. Same empty-`departmentIds` → `status: error` rule as `TicketWritesToToga2` (DB row is the only home of the list).
+- **Clear a late one:** fix the cause, then create the ticket in Freshservice by hand and save its id to 2.0 `c_wjeTicketId` — or add the number to `excludedTicketNumbers`. Touching the ticket does **not** retry a lost POST (see Gotchas).
+
+### Clear a stuck ticket (runbook)
+1. Fix the cause (Sentry has the exception).
+2. Touch the ticket (add a reply/note) so the sync picks it up again — or move the checkpoint back.
+3. If abandoning it, add its ticket number to that monitor's `excludedTicketNumbers`.
+
+### Activation (in order, all four)
+1. Push branch `TRUE-80523` (worker2, dbchanges2, worker).
+2. Deploy worker2.
+3. Run the Core seeds in order: `2026-09-30b`, `2026-10-01a`, `2026-10-01b`, `2026-10-01c`.
+4. Create the four OneUptime Incoming Request monitors with **body rules only**: Contains `"alarm":"HIGH"` → Offline; Contains `"status":"error"` → Offline; Contains `"alarm":"OK"` → Online. Put each push URL in its job config (never in a doc).
+5. Test `TicketWritesToToga2`: set `departmentIds` to `[290]` (has old unsent rows), hand-run → expect HIGH. Restore `[269]`, hand-run → expect recovery.
+6. Hand-run the other three → expect OK.
+7. Set `isActive = 1` on all four.
+8. Only then add the Not Received 10/15 min rules (see the activation-ordering trap in the shared doc).
+9. Tell the team `TicketTransmissionToFreshservice` may go HIGH on the next TOGaDesk-created ticket with a linked requester — a real failure, not noise (see Gotchas).
+
+## Gotchas
+- **Dept 269 is very low volume** (Sept 2026: 3 tickets, 7 replies, 2 comments). "No writes" is normal.
+- **Do not add 290 casually** to `TicketWritesToToga2` — 290 is not synced 1.0 → 2.0 (see [profile](../profile.md)); it would alarm on the known backlog.
+- **Pre-2025-05-01 history never drains:** 21 sendable unsent dept-269 rows 2024-12 → 2025-04-22. Hence `BACKLOG_CUTOFF_DATE`.
+- **21 WJE tickets never reached TOGaDesk** (created in TOGa 2 2026-09-18 → 09-23: 66095–66100, 66137–66144, 66146, 66172–66176, 66207). Every ticket and note since 2026-09-24 did. Likely the 2.0 → 1.0 side failed ~5 days while its checkpoint moved — **cause not confirmed; not being investigated (Mark's call).** Hence the 2026-09-24 floor in `TicketSyncToTogadesk`.
+- **WJE turns OFF new-ticket webhooks 8am–6pm on their workday** (TOGA covers after-hours calls). Updates on open tickets still arrive by day (peak 7–10am handoff). Volume ~20–30 webhooks/weekday, ~5/weekend day; 10h+ quiet nights are normal.
+- **WJE sends only a subset of Freshservice tickets** (e.g. 66209–66257 never came); routing rule unknown. So a Freshservice-API "never sent" check was deliberately not built.
+- **Webhook payload shape:** ticket webhooks carry `freshdesk_webhook.ticket_id` (`"INC-66452"`) + `id_numeric` (`66452`); asset webhooks carry `asset_display_id` instead.
+- **Clock is US/Central everywhere** — `Logs`, `Client_Wje`, `TOGaDeskSupport` (verified: 66452 first webhook 06:59:41, `Tickets.dtCreated` 06:59:50). Spring DST can false-alarm once; autumn can delay an alarm up to 1h.
+- **Re-read needs `_Database::resetQueryCache()`** — an identical second `_Query` in the same run is served from cache.
+- **Query cost (prod, verified, all fine at 5 min):** `TOGaDeskSupport` has **no index on `referenceId`** (`tickets`: clientid etc.; `tickets_replies`: `ticketid`; `comments`: PRIMARY only, ~13k-row full scan). `Client_Wje.TicketNotes` ~7.3k-row full scan; `Tickets.c_wjeTicketId` not indexed. Legacy `wait_timeout` 900s (covers the 150s sleep).
+- **Blind spots:**
+  - `TicketWritesToToga2`: field-only edits (priority/assignee) on a linked ticket create no row. Status changes are covered (they arrive as replies).
+  - `TicketTransmissionToFreshservice`: status/field `PUT`s save nothing, so a failed one is invisible here. Partial cover: the 1.0 client logs the Freshservice error and `App_SystemMonitor_WJE` (`library/app/systemmonitor/wje.php`) emails it — **not verified** that it fires for these `PUT`s.
+  - `WebhookTicketSync`: webhooks WJE never sends; note sync inside the webhook; forged numeric posts to the public `/wje` route can raise a false HIGH (whether the ingestion Lambda verifies the sender is unchecked).
+- **0 of 19 TOGaDesk-created WJE tickets ever reached Freshservice** (`153000001`–`153000022`, since 2024-11): 11 `@8x8.com` phone-system requesters not linked (skipped by design), 7 internal/test (`@goagilant.com` / `@togatech.com`), **3 linked `@wje.com` requesters unsent** (`153000016` 2025-10-04, `153000021` 2026-08-08, `153000022` 2026-08-21). Hence the 2026-10-01 cutoff. **Not being investigated (Mark's call — new/future only).**
+- **Likely permanent-loss path (CTO code reading, unproven):** Freshservice `POST` throws after the 2.0 ticket exists → next run finds the 2.0 ticket by number, takes the "already exists" branch (which never POSTs), saves the TOGaDesk `referenceId` → never retried, never alerted. Expect the monitor to go HIGH on the next linked TOGaDesk-created ticket.
+- **2.0 contact `c_wjeUserId` does not mirror the TOGaDesk link** — `153000021`/`22` requesters are linked in 1.0 but their 2.0 contact has no `c_wjeUserId`. Do not switch the monitor to the 2.0 field.
+- **Replies are healthy:** 0 of ~1,400 external notes on WJE-linked tickets since 2026-05 lack `c_wjeConversationId`.
+- **Known risk, no monitor (out of scope):** if the reply `POST` succeeds but the 2.0 note write fails, the next run sends the reply to WJE again (duplicate).
+- **The ClickUp TRUE-80523 description is fabricated** (Talos auto-generated) — see [MCP tool usage gotcha](../../../standalone/apps/claude/workflows/mcp-tool-usage.md). Trust this doc and the code.
+
+## Related
+- [OneUptime push-metric monitors for 2.0 workers](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md) — shared pattern
+- [App_Api_Toga2 bridge](../../../1.0/apps/library/features/toga2-api-client-and-bridge.md) — `syncWithTogadesk`, `referenceId` write-back
+- [WJE Freshservice sync](../../../2.0/apps/worker2/features/wje-freshservice-sync.md) — webhook handler, `c_wje*` fields
+- [WJE profile](../profile.md)
+
+## Change history
+- 2026-10-01 — Added `TicketTransmissionToFreshservice` (TRUE-80522) on the shared branch; recorded the 19-ticket never-sent finding and the likely permanent-loss path. Not deployed. (mhammontree)
+- 2026-10-01 — Renamed from `oneuptime-ticket-writes-monitor.md` to the suite doc; added `TicketSyncToTogadesk` (TRUE-80521) and `WebhookTicketSync` (TRUE-80520). Not deployed. (mhammontree)
+- 2026-10-01 — Created: `TicketWritesToToga2` built on branch TRUE-80523, not deployed, cron `isActive = 0`. (mhammontree)

package/knowledge/clients/wje/profile.md

@@ -17,10 +17,11 @@ files:
   - worker2/Worker/Wje.php
   - worker/crons/toga2/wje/import_supporting_records.php
   - worker2/Worker/Monitor/Wje.php
+  - worker2/Worker/Client/Wje.php
 related:
   - ../../1.0/apps/library/features/toga2-api-client-and-bridge.md
   - ../../2.0/apps/worker2/features/wje-freshservice-sync.md
-  - ./features/oneuptime-ticket-writes-monitor.md
+  - ./features/oneuptime-monitors.md
 ---
 
 WJE (Wiss, Janney, Elstner / "WJE IT") Freshservice help-desk client, single hardcoded tenant; open for the three sync paths, the TOGaDesk bridge ids/departments, and its monitor.
@@ -37,7 +38,9 @@ Data moves between WJE and TOGA on three paths:
 2. **Daily batch import** (`worker/crons/toga2/wje/import_supporting_records.php`) — imports canned responses, requesters, assets, groups, and the 3-level category hierarchy.
 3. **2-minute TOGaDesk bridge** (`App_Api_Toga2::syncWithTogadesk`) — the cron `worker/crons/toga2/wje/sync_togasupply_wje.php` enables the **full** set of bridge sub-syncs (tickets both directions, ticket-notes, contacts→people, items/units→assets, predefined-replies, ticket-teams→groups, ticket-categories) for TOGaDesk client id 153. **TOGaDesk → 2.0 syncs department 269 only.**
 
-Monitoring: [ticket-writes monitor](./features/oneuptime-ticket-writes-monitor.md) alarms when a dept-269 TOGaDesk row is not written to 2.0.
+Monitoring: [OneUptime monitor suite](./features/oneuptime-monitors.md) — four backlog monitors, one per hop (webhook → 2.0, TOGaDesk dept 269 → 2.0, 2.0 → TOGaDesk, TOGaDesk/2.0 → Freshservice).
+
+Webhook behavior: WJE turns OFF new-ticket webhooks 8am–6pm on their workday (TOGA covers after-hours); updates still arrive by day. WJE sends only a subset of its Freshservice tickets (rule unknown).
 
 ## TOGaDesk departments
 
@@ -50,4 +53,4 @@ Monitoring: [ticket-writes monitor](./features/oneuptime-ticket-writes-monitor.m
 
 - **[wje-freshservice-sync](../../2.0/apps/worker2/features/wje-freshservice-sync.md)** — full integration: webhook handler, import cron, `c_wje*` models, API clients, schema, reports, gotchas.
 - **[toga2-api-client-and-bridge](../../1.0/apps/library/features/toga2-api-client-and-bridge.md)** — the `syncWithTogadesk` bridge mechanism.
-- **[oneuptime-ticket-writes-monitor](./features/oneuptime-ticket-writes-monitor.md)** — the 1.0 → 2.0 write monitor.
+- **[oneuptime-monitors](./features/oneuptime-monitors.md)** — the four WJE OneUptime monitors.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.933",
+  "version": "1.0.935",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",

package/knowledge/clients/wje/features/oneuptime-ticket-writes-monitor.md

@@ -1,80 +0,0 @@
----
-title: "WJE: TOGaDesk → TOGa 2 ticket-writes monitor (Monitor/Wje/TicketWritesToToga2)"
-framework: "2.0"
-repo: worker2
-project: Worker
-client: wje
-type: client-feature
-status: active
-updated: 2026-10-01
-owners: ["mhammontree"]
-files:
-  - worker2/Worker/Monitor/Wje.php
-  - dbchanges2/Core/2026-09-30b - Insert WJE Ticket Writes Monitor CronJob.sql
-  - worker/crons/toga2/wje/sync_togasupply_wje.php
-  - library/app/api/toga2.php
-related:
-  - ../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md
-  - ../../../1.0/apps/library/features/toga2-api-client-and-bridge.md
-  - ../../../2.0/apps/worker2/features/wje-freshservice-sync.md
-  - ../profile.md
----
-worker2 OneUptime push monitor counting WJE TOGaDesk tickets/replies/comments never written to TOGa 2 (NULL referenceId); open when working on WJE bridge monitoring, its departments, or clearing a stuck ticket.
-
-## Summary
-`Monitor/Wje/TicketWritesToToga2` (`_Worker_Monitor_Wje`, TRUE-80523) watches the 1.0 → 2.0 half of the WJE TOGaDesk bridge. It follows the shared [OneUptime push-metric pattern](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md) — read that for payload contract, criteria, and provisioning.
-
-- **Status 2026-10-01:** committed on branch `TRUE-80523` (worker2 + dbchanges2 + worker comment), **not deployed**. Cron seed ships `isActive = 0`; `$ONEUPTIME_URL` is still empty (push credential — never record it).
-- Runs in worker2, outside the 1.0 worker it watches. Reads `TOGaDeskSupport` via `_underscore::DB_TOGADESK1` (see [worker2 legacy DB access](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md)).
-- OneUptime monitor name: **"WJE Ticket Writes to TOGa 2"**.
-
-## How it works
-
-### Why a backlog, not a checkpoint-freshness check
-The bridge catches a per-ticket exception → Sentry only, and **still advances `TOGA_LAST_TICKET_INTEGRATION_DATETIME` past it**. The failed ticket is never retried unless touched again. A checkpoint-age monitor would read healthy. A row with NULL `referenceId` catches both a dead cron and silent per-ticket failures (link model: [bridge doc](../../../1.0/apps/library/features/toga2-api-client-and-bridge.md)).
-
-### What it counts
-WJE rows (`tickets.clientid = 153`) in `tickets`, `tickets_replies`, `comments` where `referenceId` is NULL or `''`, created on/after `BACKLOG_CUTOFF_DATE = '2025-05-01'`, in the configured departments, minus `excludedTicketNumbers`.
-
-Skip rules mirror `syncTogaDesk1TicketIntoToga2Ticket` in `library/app/api/toga2.php` (~L12280-12290): a reply with a blank message falls back to `newstatus` and is skipped if still PHP-empty (NULL, `''`, `'0'`) — it never gets a `referenceId`, so the monitor skips it too. Comments are always sent.
-
-### Decision + payload
-- `alarm: HIGH` when any counted row is `>= alertThresholdMinutes` old; else `alarm: OK`.
-- `status: error` on DB failure (payload carries **only** `errorClass` + `errorNumber` — the exception message can hold SQL, DB host, DB user) and on empty `departmentIds` (`error: no_departments_configured`).
-- Body: `latePendingCount`, `pendingCount`, per-type counts, `oldestPendingMinutes`, `alertThresholdMinutes`, `lateTicketNumbers` (top 10), `departmentIds`, `excludedTicketCount`.
-
-### Parameters (`Core.CronJobs.parameters`, no deploy to change)
-`{"alertThresholdMinutes":15,"departmentIds":[269],"excludedTicketNumbers":[]}`
-- `departmentIds` PHP default is `[]` **on purpose** — the DB row is the only home of the list. Missing list → `status: error`, never a silent hardcoded dept. The cron comment in `sync_togasupply_wje.php` points here; keep the two in step if the sync's departments change.
-- `excludedTicketNumbers` — TOGaDesk ticket numbers the team gives up on, so one permanently failing ticket does not hold the alarm red.
-
-### Clear a stuck ticket (runbook)
-1. Fix the cause (Sentry has the exception).
-2. Add a reply to the ticket in TOGaDesk so the sync picks it up again — or move the checkpoint back.
-3. If abandoning it, add its ticket number to `excludedTicketNumbers`.
-
-### Activation (in order)
-1. Create the OneUptime Incoming Request monitor with **body rules only**: Contains `"alarm":"HIGH"` → Offline + incident; Contains `"status":"error"` → Offline + incident; Contains `"alarm":"OK"` → Online.
-2. Paste its URL into `$ONEUPTIME_URL`.
-3. Push and deploy worker2.
-4. Run the Core seed `2026-09-30b - Insert WJE Ticket Writes Monitor CronJob.sql`.
-5. Test HIGH: set `departmentIds` to `[290]` (has old unsent rows), hand-run. Restore `[269]`, hand-run to see recovery.
-6. Set `isActive = 1`.
-7. Only then add the not-received 10/15 min rules (see the activation-ordering trap in the shared doc).
-
-## Gotchas
-- **Dept 269 is very low volume** (Sept 2026: 3 tickets, 7 replies, 2 comments). "No writes" is normal — this is why it is a backlog monitor, not a "writes stopped" monitor.
-- **Blind spot: field-only edits.** Priority/assignee change on an already-linked ticket creates no row, so a failed push is invisible. Status changes are covered (they arrive as replies).
-- **Clock is US/Central.** `TOGaDeskSupport` `NOW()` and row timestamps are both server US/Central (the `_underscore` driver sets no session `time_zone`). Spring DST can false-alarm once; autumn can delay an alarm up to 1h.
-- **Pre-cutoff history never drains:** 21 sendable unsent dept-269 rows 2024-12 → 2025-04-22 (early integration). That is why the cutoff is 2025-05-01.
-- **Query cost (prod, verified):** `tickets` via index_merge(departmentid, clientid); `tickets_replies` via `ticketid` index; `comments` has **no** `ticketid` index → full scan of ~13k rows (acceptable). Timestamp columns have no `ON UPDATE`.
-- **Do not add 290 casually.** Dept 290 is not synced 1.0 → 2.0 today (see [profile](../profile.md)); pointing the monitor at it would alarm on the known backlog, not a new failure.
-- **The ClickUp TRUE-80523 description is fabricated** (Talos auto-generated) — see [MCP tool usage gotcha](../../../standalone/apps/claude/workflows/mcp-tool-usage.md). Trust this doc and the code.
-
-## Related
-- [OneUptime push-metric monitors for 2.0 workers](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md) — shared pattern
-- [App_Api_Toga2 bridge](../../../1.0/apps/library/features/toga2-api-client-and-bridge.md) — `syncWithTogadesk`, `referenceId` write-back
-- [WJE profile](../profile.md)
-
-## Change history
-- 2026-10-01 — Created: monitor built on branch TRUE-80523, not deployed, cron `isActive = 0`. (mhammontree)