toga-ai 1.0.925 → 1.0.927

package/knowledge/1.0/apps/library/features/cron-execution-monitoring.md

@@ -6,8 +6,8 @@ project: Library
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
-owners: [dfranks, bala]
+updated: 2026-10-01
+owners: [dfranks, bala, mhammontree]
 files:
   - library/app/framework.php
 related:
@@ -48,6 +48,7 @@ Two consequences:
 ## Gotchas
 
 - **⚠ A run skipped by the overlap guard leaves NO ROW — absence is ambiguous.** `cronInitialization()` calls `exitIfProcessRunning()` **before** the `CronJobExecutions` INSERT, and the `cronFinished(true)` that guard then calls does nothing because `cronLogId` was never set. So a blocked run is indistinguishable from a run that never launched: **a missing row for an expected slot means "skipped", not "hung"**, and a hung run looks different (it has `dtCheckIn` with no `dtCheckOut`). `isProcessRunning()` matches **any** `ps -ef` line containing the script path (it only excludes `/bin/sh` and its own pid), so a stray `tail -f` or editor on that path blocks the cron indefinitely.
+- **`CronJobExecutions` is huge (~120M rows) with no index on `job`.** Never query it per run (e.g. as a cron's watermark) — it scans the table. Use a small dedicated table instead.
 - **`db_log` has no `CronJobExecutions` table.** A `cronFinished()` UPDATE that runs against the `db_log` connection throws on **every** 1.0 cron job. Both the INSERT and the UPDATE must target `db_common`.
 - **A `git merge` can silently resurrect pre-refactor logic.** These functions were cleanly consolidated to a single `db_common`-only INSERT/UPDATE pair (commit `0a3a6f75`), but a later `git merge origin/_production` (`509614036605…`) reintroduced the old dual-path structure from `_production`'s still-unmigrated copy — leaving duplicate INSERT/UPDATE blocks and reverting one path's connection argument back to `db_log`. Automatic merge resolution updated the SQL text (renamed the table) but restored the wrong connection name, producing a platform-wide breaking bug disguised as a simple PR-feedback rework. **When a merge touches a function you already refactored on your branch and the base branch never got that refactor, diff against the merge-base — do not just eyeball the current file** — to catch resurrected logic. (Fixed 2026-07-06: consolidated back to one `db_common` INSERT/UPDATE pair.)
 
@@ -55,3 +56,6 @@ Two consequences:
 
 - [Worker architecture](../../worker/architecture.md), [App_Email queued sending & attachments](./email-queue-attachments.md).
 - [OneUptime worker2 monitoring](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md), [Tracing a worker cron run in production](../../worker/workflows/tracing-a-worker-cron-run-in-production.md).
+
+## Change history
+- 2026-10-01 — Added the no-index / ~120M-row gotcha (rejected as a watermark source in TRUE-82180). (mhammontree)

package/knowledge/1.0/apps/library/features/mvc-page-pattern-and-app-skeleton.md

@@ -6,8 +6,8 @@ project: Library
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
-owners: [jcardinal]
+updated: 2026-10-01
+owners: [jcardinal, mhammontree]
 files:
   - library/app/framework.php
   - library/app/frameworkindex.php
@@ -84,9 +84,12 @@ The app depends on the `library` core repo (bootstrapped via `_.php`).
 ## Gotchas
 
 - **Route folders must stay distinct.** Matching is longest-to-shortest, so overlapping route folders can produce surprise matches.
-- **An unset `ENVIRONMENT` loads the wrong config file.** `getenv('ENVIRONMENT')` defaults to `'worker'`, so a missing env var silently loads `config.worker.ini`.
+- **An unset `ENVIRONMENT` loads PRODUCTION config.** `getenv('ENVIRONMENT')` defaults to `'worker'`, so running a 1.0 cron locally without it loads `worker/config.worker.ini` — prod DB hosts, `test_mode=0`, `email_errors=1`. Set `ENVIRONMENT` to your dev config name (`config.dev-<name>.ini`) first. Symptom: errors mention `/var/www/cache`.
 - **Secrets are currently stored in plaintext ini files** — a known practice, not a recommended one. Document where a credential lives, never its value.
 
 ## Related
 
 - [Library architecture](../architecture.md), [togaview architecture](../../togaview/architecture.md).
+
+## Change history
+- 2026-10-01 — Unset-`ENVIRONMENT` gotcha: it loads production config locally; added symptom and fix (TRUE-82180). (mhammontree)

package/knowledge/1.0/apps/library/features/toga2-api-client-and-bridge.md

@@ -6,7 +6,7 @@ project: Library
 client: shared
 type: feature
 status: active
-updated: 2026-09-21
+updated: 2026-10-01
 owners: [jcardinal, mhammontree, bala]
 files:
   - library/app/api/toga2.php
@@ -23,6 +23,7 @@ related:
   - ../../worker/features/netsuite-togasupply-per-client-sync.md
   - ../architecture.md
   - ./error-capture-1-0.md
+  - ../../../clients/wje/features/oneuptime-ticket-writes-monitor.md
   - ../../../clients/elite/features/togadesk-service-request-intake.md
 ---
 
@@ -145,7 +146,9 @@ A large bi-directional sync gated by **two checkpoint parameters** stored in 2.0
 
 Signature takes `$integrationType` (`TICKETS` or `REPAIR_ORDERS`) plus **seven independent enable flags** and an optional `$monitorTogadeskDepartmentIds[]`:
 - **2.0 → TOGaDesk** (`$isEnabledToga2ToTogadeskIntegration`): pages 2.0 `/tickets` and `/ticket-notes` changed since the checkpoint; routes each to `syncToga2TicketIntoTogadesk1{RepairOrder|Ticket}` by `$integrationType`. Per-ticket exceptions are caught and sent to Sentry so one bad ticket doesn't abort the batch.
-- **TOGaDesk → 2.0** (`$isEnabledTogadeskToToga2Integration`): queries the 1.0 `db_togadesk` DB directly (SQL over `repair_orders`/`repair_order_history` or `tickets`/`tickets_replies`/`comments`, using `GREATEST(...)` of reply/comment timestamps) for the client's department(s), then calls `syncTogaDesk1{RepairOrder|Ticket}IntoToga2Ticket($id)`.
+- **TOGaDesk → 2.0** (`$isEnabledTogadeskToToga2Integration`): queries the 1.0 `db_togadesk` DB directly (SQL over `repair_orders`/`repair_order_history` or `tickets`/`tickets_replies`/`comments`, using `GREATEST(...)` of reply/comment timestamps) for the client's department(s) — only the ones the cron passes (`$monitorTogadeskDepartmentIds`) — then calls `syncTogaDesk1{RepairOrder|Ticket}IntoToga2Ticket($id)`.
+  - **Link = `referenceId`.** `tickets.referenceId` = 2.0 ticket uuid; `tickets_replies.referenceId` / `comments.referenceId` = 2.0 ticket-note uuid. Written back **only after a successful POST**. Rows created by the 2.0 → 1.0 direction already carry it. So NULL/`''` `referenceId` = not (yet) in 2.0 — the basis of the [WJE ticket-writes monitor](../../../clients/wje/features/oneuptime-ticket-writes-monitor.md).
+  - A reply with a blank message falls back to `newstatus`; if still PHP-empty (NULL, `''`, `'0'`) it is skipped and never gets a `referenceId`. Comments are always sent.
 - Five further one-way 2.0 → TOGaDesk sub-syncs, each its own flag: Contacts→People, Items/Units→Assets, Predefined-Replies→TicketsPr, Ticket-Teams→Groups, Ticket-Categories→Category/Subcategory/Items.
 - Finishes by PUT-ing both checkpoint parameters back to 2.0.
 
@@ -167,6 +170,7 @@ Signature takes `$integrationType` (`TICKETS` or `REPAIR_ORDERS`) plus **seven i
 - **Two unrelated `sync_togasupply_<client>.php` families.** Files under `worker/crons/toga2/netsuite/` are the **NetSuite supply importer** (`require` the common engine). Files under `worker/crons/toga2/<client>/` (e.g. `aig/`, `wje/`) are this **bridge** (`syncWithToga`/`syncWithTogadesk`). Same filename, different integration, different parameter keys — they coexist harmlessly. Confirm which folder before editing.
 - **`syncWithToga` and the bridge build SQL by hand** with `App_Database::sqlEscape()` rather than the `App_Model` layer; follow the surrounding escaping discipline when modifying.
 - **Per-record error isolation** in the TOGaDesk sync is via `try/catch` → `\Sentry\captureException`; a thrown exception elsewhere (transport, checkpoint read) still aborts the whole run.
+- **⚠ A failed ticket is never retried.** After a per-ticket exception the run still advances `TOGA_LAST_TICKET_INTEGRATION_DATETIME` past it, so the row stays unsent until someone touches the ticket again (new reply) or the checkpoint is moved back. A checkpoint-freshness check reads healthy through this — monitor NULL `referenceId` instead.
 - **Naming `fields` can SUPPRESS a nested relation that a plain request returns.** Observed 2026-08-12 on `GET /states`: a bare request comes back with the nested `country`, but adding a `fields` list drops it. This is the mirror image of the reverse-hasMany rule below — when a nested object you expected is missing, suspect the field list before the data.
 - **A reverse hasMany collection is only returned if you add it to the fetch `fields` whitelist.** The `/contacts` fetch requests an **explicit** `fields` list from the 2.0 metadata API; a reverse hasMany collection (e.g. `contactEmailAddresses.emailAddress`) is **not** returned unless added — so consuming a related collection means editing **both** the fetch field list **and** the consuming code. (The general 2.0-side metadata behavior — reverse relations resolved via the child model's `FOREIGNKEY_MODEL` and named as the camelCase plural of the child model — is a framework-level fact captured separately.)
 - **⚠ Depth alone does NOT return every reverse-hasMany collection — `GET /sales-orders` at depth 4 omits `itemFulfillmentItems` and `invoiceItems`.** Verified 2026-08-17 against prod `Logs_Compass.Api` responses (no fields whitelist): a depth-4 sales-order GET returns `salesOrderItems` and the **PO-link bridge** collections (`salesOrderItemPurchaseOrderItems`, `purchaseOrderItemSalesOrderItems`) but **not** `itemFulfillmentItems` or `invoiceItems`. So any importer code that guards on `$salesOrderItem->itemFulfillmentItems` / `->invoiceItems` from that GET is a **silent no-op** — same reverse-hasMany trap as above, with a concrete consequence: the NetSuite SO renumber fix keys off a recorded "could-not-delete" set instead of trusting those collections (see [per-client sync](../../worker/features/netsuite-togasupply-per-client-sync.md)).
@@ -180,6 +184,7 @@ Signature takes `$integrationType` (`TICKETS` or `REPAIR_ORDERS`) plus **seven i
 - **Pre-existing (low severity, out of scope): a raw-vs-escaped comparison causes a redundant UPDATE every sync.** At the update path (~line 8584) `syncContactToToga1Customer` compares `$existingCustomer['emailAddress'] !== $emailAddress` — the DB value from `App_Database::fetchRow()` is **unescaped**, while `$emailAddress` has already been through `App_Database::sqlEscape()`. For any value containing `'`, `"` or `\` the two can **never** compare equal, so that column is rewritten on **every** run. Same pattern applies to `businessName` and `phoneNumber` just above. **Predates TRUE-79401**, is a redundant write (not data loss), and was deliberately left out of that ticket's scope — recorded here so a future reader knows it is known.
 
 ## Change history
+- 2026-10-01 — Documented the `referenceId` link/write-back, blank-reply skip rule, and that a per-ticket failure advances the checkpoint (never retried). (mhammontree)
 - 2026-09-21 — Recorded a pre-existing committed JWT (dev/NYCHH token) at `toga2.php:641` inside `send()` (the sync call path); flagged to rotate + delete, location-only. (jcardinal)
 
 ## Related

package/knowledge/1.0/apps/worker/INDEX.md

@@ -15,6 +15,7 @@
 | [OneUptime Server monitor + disk/memory hygiene on the 1.0 worker EB host](features/oneuptime-server-monitor-host-hygiene.md) | OneUptime Server monitor (per worker role) plus disk/memory hygiene for the legacy AL1 PHP 7.2 1.0 worker EB host; open for the host-metrics agent, monitor crit |
 | [OneUptime external uptime monitoring for 1.0 workers](features/oneuptime-worker-uptime-monitoring.md) | Per-minute external OneUptime "Incoming Request" heartbeat from every 1.0 worker box (Degraded 3 min / Offline 5 min); open for the heartbeat push, monitor prov |
 | [Prudential: Send Shipments for the Day report (daily cron)](features/send-shipments-for-the-day.md) | Daily 9:00 PM Prudential/Dell cron emailing an Excel of every device shipped today (tracking, serial, XID, FedEx status, vendor); open for the ASN join chain an |
+| [Staples 856 ASN Outbound (NetSuite fulfillments → SFTP, tracked in StaplesAdvanceShipNotices)](features/staples-asn-856-outbound.md) | Hourly cron sending Staples 856 ASNs for shipped NetSuite item fulfillments, one row per fulfillment in `Common.StaplesAdvanceShipNotices`; open for the queue/r |
 | [Staples cXML Order Import (SFTP → NetSuite)](features/staples-cxml-order-import.md) | Hourly cron importing Staples cXML POs from SFTP into NetSuite Sales Orders (writes 855 ack back); open for the per-file flow, saved-search-8412 Location resolu |
 | [UMA Student Roster Sync (SFTP → TOGaDeskSupport customers) + Ivanti Ticket Sync](features/uma-student-sync.md) | Daily SFTP full-roster file loads UMA students into TOGaDeskSupport.customers (clientId 10); open for the file format, insert/deactivate rules, how the Ivanti t |
 | [Walmart Daily Sales Report (send_toga_walmart_report.php)](features/walmart-daily-sales-report.md) | The two Walmart sales workbooks (summary + details) built from `TOGA_Walmart` (in-store) and `TOGA_WalmartEComm` (.COM); open for the pseudo-store 1472 channel |

package/knowledge/1.0/apps/worker/features/staples-asn-856-outbound.md

@@ -0,0 +1,67 @@
+---
+title: Staples 856 ASN Outbound (NetSuite fulfillments → SFTP, tracked in StaplesAdvanceShipNotices)
+framework: "1.0"
+repo: worker
+project: Worker
+client: staples
+type: client-feature
+status: active
+updated: 2026-10-01
+owners: [mhammontree]
+files:
+  - worker/crons/sync/staples/staples_asn_netsuite.php
+  - dbchanges/Common/MH/2026-09-30.sql
+  - worker/schedules/cron.worker.sync.json
+related:
+  - ../../../../clients/staples/profile.md
+  - ./staples-cxml-order-import.md
+  - ../../../../clients/staples/features/oneuptime-sftp-folder-monitors.md
+  - ../../library/features/cron-execution-monitoring.md
+  - ../../library/features/mvc-page-pattern-and-app-skeleton.md
+---
+
+Hourly cron sending Staples 856 ASNs for shipped NetSuite item fulfillments, one row per fulfillment in `Common.StaplesAdvanceShipNotices`; open for the queue/retry/dedupe design, the 855 gate, statuses, manual SKIPPED, alerts, and how to test it locally.
+
+## Summary
+
+`staples_asn_netsuite.php` (`50 * * * *`, `schedules/cron.worker.sync.json`) builds a cXML 856 ASN for each shipped item fulfillment of NetSuite customer **30057** and puts it in Staples SFTP `/out`. Since TRUE-82180 (2026-09-30) every fulfillment is tracked in **`Common.StaplesAdvanceShipNotices`** (`db_common`), so a shipment is retried until sent and never sent twice unless its shipping data changed.
+
+**Rule (decided 2026-09-30, Jeff via Mark):** never send Staples the same ASN twice unless the shipping data changed. The fix stays in 1.0 — 2.0 would need an importer rewrite, and NetSuite use will likely move to an internal replacement app later. Old backlog and POs with no 855 are handled by hand, not in code.
+
+## How it works
+
+Table: one row per item fulfillment, `UNIQUE itemFulfillmentInternalId`. Status `PENDING` (not sent; `lastError` says why it waits) / `SENT` / `FAILED` (gave up, needs a developer) / `SKIPPED` (manual only, never retried).
+
+Each run (`cronInitialization(14400, false)` — must **not** use the read cluster; ends with `cronFinished()` / `closeConnections()`):
+
+1. **Queue.** Search 30057 fulfillments modified in the last 7 days (`ASN_LOOKBACK_SECONDS`). New fulfillment → insert `PENDING`. Existing row whose `dtFulfillmentModified` changed → back to `PENDING` (unless `SKIPPED`).
+2. **Process** up to 200 (`ASN_MAX_ROWS_PER_RUN`) `PENDING` rows, oldest `dtUpdated` first.
+3. **855 gate.** A row waits (stays `PENDING`) until `855_<PO>_*` exists in SFTP `/archive`. Likely reason the gate exists: it limits ASNs to orders that came in through the Staples cXML feed (the 856 needs the original cXML in `/archive/archive_in` for Staples line numbers) and keeps 855 before 856. Kept as a wait, not a drop.
+4. **Dedupe by `shippingHash`** = sha1 of shipping data only (carrier, ship-to, boxes with tracking/weight/items/serials, services). It excludes the random Package identifier, timestamps, and `noticeDate`, so a rebuild of unchanged data gives the same hash. Same hash as last sent → no file. A changed hash re-sends with operation still **`new`** (decided 2026-09-30).
+5. **Pre-go-live shipments** (`tranDate` before `MIN(dtCreated)` of the table, no hash yet) are marked `SENT` without sending if an old `856_<PO>_*` file in `/archive` or `/out` contains `shipmentID= "<tranId>"` (the old job wrote a space before the quote).
+6. **Send.** File name `856_<PO>_<fulfillmentInternalId>_<time>`; raw file logged to FileLog job `STAPLES_ASN_OUT_ORDER`. In test mode (`App_Registry::inTestMode()`) the file is written only to the cache folder — never uploaded to the live SFTP.
+7. **Alerts.** One email (same 3 recipients as before) listing rows not sent after 24 h; each row alerts once (`dtAlerted`, cleared on requeue).
+
+**Errors are per row.** A local `set_error_handler` turns PHP warnings into a row failure (otherwise 1.0 `App_Error` exits the whole run). A row fails up to 5 times (`ASN_MAX_SEND_ATTEMPTS`), then `FAILED`. A `SoapFault` / `RuntimeException` (NetSuite or SFTP down) moves the row to the back and stops the run.
+
+**Stop checking a PO handled by hand:** `UPDATE StaplesAdvanceShipNotices SET status = 'SKIPPED' WHERE poNumber = '<PO>';`
+
+**Deploy order:** merge dbchanges first (`_production` applies on the next worker deploy), then deploy worker.
+
+## Testing locally
+
+1. Load prod `Common` locally and run `dbchanges/Common/MH/2026-09-30.sql`.
+2. Set `ENVIRONMENT` to your dev config name first (see the unset-`ENVIRONMENT` gotcha in [1.0 MVC page pattern](../../library/features/mvc-page-pattern-and-app-skeleton.md)) — otherwise the run uses production config.
+3. Run with PHP 7.2 (`xampp_7_2_33`). It reads prod NetSuite and the live SFTP read-only; test mode keeps files in cache.
+
+Expected (2026-10-01 run): first run 101 rows, 90 matched as already sent by the old job, 0 files, ~640 s; second run ~100 s, nothing sent; changed data or a new shipment = exactly one file; alert once per row.
+
+## Gotchas
+
+- **Old design (pre-TRUE-82180) — why ASNs were late and doubled.** It searched only fulfillments modified in the last 3600 s, so each shipment got one chance; a skipped one went out only when someone edited the fulfillment (days/weeks late, ~half of ASNs). Dedupe checked only `/archive` by prefix `856_<PO>_<n>` where `n` restarted every run, and every fulfillment on the SO was re-sent when one changed.
+- **Do not use `Common.CronJobExecutions` or `Logs.FileLog` as a watermark/dedupe source** — both are too big and unindexed for per-run lookups (see [Cron execution monitoring](../../library/features/cron-execution-monitoring.md) and the FileLog note below). That is why the dedicated table exists.
+- **`Logs.FileLog` has no index on `job` / `fileName`** (verified locally) and `Logs` is ~400 GB — never `LIKE`-scan its BLOB per record; it cannot be pulled locally in full.
+- **Hardcoded credentials (parked).** The SFTP password and the cXML SharedSecret are hardcoded in `staples_asn_netsuite.php`; they should move to config.
+
+## Change history
+- 2026-10-01 — First doc. TRUE-82180 rewrite: `StaplesAdvanceShipNotices` queue with 7-day lookback, shipping-hash dedupe, per-row retry/FAILED, 24 h alert, test mode never uploads, old-job match for pre-go-live shipments. Committed worker `cf5198a27`, dbchanges `8b1f7cb` on branch TRUE-82180; not deployed as of 2026-10-01. (mhammontree)

package/knowledge/2.0/apps/worker2/features/oneuptime-worker2-monitoring.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-09-25
+updated: 2026-10-01
 owners: ["jcardinal", "mhammontree", "bala", "rgirish"]
 files:
   - worker2/Worker/Monitor/Compass.php
@@ -18,6 +18,7 @@ files:
   - worker2/Worker/Monitor/Prudential.php
   - worker2/Worker/Monitor/Nycdoe.php
   - worker2/Worker/Monitor/Staples.php
+  - worker2/Worker/Monitor/Wje.php
   - worker2/Config/production.ini
   - worker2/Config/beta.ini
   - worker2/Config/sandbox-dev.ini
@@ -32,6 +33,7 @@ related:
   - ../../../clients/staples/features/oneuptime-sftp-folder-monitors.md
   - ../../../clients/compass-usa/features/oneuptime-ma-refresh-order-monitor.md
   - ../../../clients/compass-usa/features/oneuptime-netsuite-duplicate-sales-orders-monitor.md
+  - ../../../clients/wje/features/oneuptime-ticket-writes-monitor.md
 ---
 
 OneUptime push-metric monitor pattern for worker2 (2.0): a self-contained cron action decides pass/fail and POSTs a string token to a OneUptime Incoming Request monitor; open when building/tuning a worker2 monitor, its thresholds, or the OneUptime import JSON.
@@ -302,6 +304,13 @@ has no proven connection to the legacy `Common` DB today**, so a 2.0 monitor can
 `MaRefreshOrderExceptions` counts rows in `Client_Compass` instead. If worker2 ever gets that
 connection, a generic "1.0 cron has not checked in" monitor becomes possible.
 
+This is about `Common` only — **worker2 does read other legacy DBs.** `worker2/_.php` registers
+`TOGaDeskSupport` as `_underscore::DB_TOGADESK1` and `Vision_Log` as `DB_VISION_LOGS` at boot (from the
+`[database1]` config group), already used in prod by `Worker/Sync/SalesOrderStatus.php` and
+`Worker/Client/Aig/ClosedClaims.php`. A 2.0 monitor over TOGaDesk data is possible and preferred (it
+runs outside the 1.0 worker it watches) — e.g. the
+[WJE ticket-writes monitor](../../../clients/wje/features/oneuptime-ticket-writes-monitor.md).
+
 ### Match the lateness criterion to the flow's real time-scale
 
 The sibling Compass monitors ship 6 criteria including a DEGRADED incident at **2 missed pushes (10
@@ -559,4 +568,5 @@ production deploy.**
 - [OneUptime 1.0 worker uptime monitoring](../../../1.0/apps/worker/features/oneuptime-worker-uptime-monitoring.md) — the 1.0 push-heartbeat predecessor
 
 ## Change history
+- 2026-10-01 — Clarified worker2 reads `TOGaDeskSupport` (`DB_TOGADESK1`) — only `Common` is unreachable; added `Monitor/Wje`. (mhammontree)
 - 2026-09-25 — Added `NetsuiteDuplicateSalesOrders`, the first monitor that queries NetSuite (SuiteQL) instead of a Toga DB. (rgirish)

package/knowledge/2.0/apps/worker2/features/wje-freshservice-sync.md

@@ -6,8 +6,8 @@ project: Worker
 client: wje
 type: client-feature
 status: active
-updated: 2026-09-16
-owners: ["snaredla"]
+updated: 2026-10-01
+owners: ["snaredla", "mhammontree"]
 files:
   - worker2/Worker/Wje.php
   - _underscore/Component/Api/Wje/Wje.php
@@ -24,9 +24,11 @@ files:
   - worker/crons/toga2/wje/sync_togasupply_wje.php
   - worker/crons/notifications/reports/wje/wje_common.php
   - library/app/systemmonitor/wje.php
+  - worker2/Worker/Monitor/Wje.php
   - dbchanges2/Client_Wje/2024-10-04 - WjeOnboarding.sql
 related:
   - clients/wje/profile.md
+  - clients/wje/features/oneuptime-ticket-writes-monitor.md
   - 2.0/apps/worker2/features/elite-freshservice-sync.md
   - 1.0/apps/library/features/toga2-api-client-and-bridge.md
   - 2.0/apps/worker2/features/startech-webhook-handler.md
@@ -62,7 +64,7 @@ a **daily batch import** of supporting records. A separate 2-minute cron bridges
   `crons/toga2/wje/sync_togasupply_wje.php` (2-min Togadesk bridge),
   `crons/notifications/reports/wje/{wje_monthly_report,wje_quarterly_report,wje_common}.php`.
 - **Models:** `_underscore/Model/Wje/*` (8 classes, see Data model).
-- **Monitoring:** `library/app/systemmonitor/wje.php` (`App_SystemMonitor_WJE`).
+- **Monitoring:** `library/app/systemmonitor/wje.php` (`App_SystemMonitor_WJE`, Freshservice/webhook errors); `worker2/Worker/Monitor/Wje.php` ([TOGaDesk → 2.0 write monitor](../../../clients/wje/features/oneuptime-ticket-writes-monitor.md)).
 
 ### Real-time webhook (`_Worker_Wje::Webhook`)
 
@@ -115,7 +117,7 @@ asset type "Laptop", urgencies (Low/Normal/High/Critical), system params
 `TOGADESK_LAST_TICKET_INTEGRATION_DATETIME` / `TOGA_LAST_TICKET_INTEGRATION_DATETIME`.
 
 Hardcoded IDs: `CLIENT_UUID_WJE=c2f5180f-…`, `API_UUID_WJE=4750650f-…` (and `API_SECRET_WJE`) in
-`worker2/Worker/Wje.php`; TOGaDesk client `153`, dept `269` (internal `290`); default group
+`worker2/Worker/Wje.php`; TOGaDesk client `153`, dept `269` (dept `290` is defined but not synced TOGaDesk → 2.0 — see `clients/wje/profile.md`); default group
 `13000155420` / agent `13000771757`; category form field `13000126389`.
 
 ## Gotchas
@@ -134,6 +136,9 @@ Hardcoded IDs: `CLIENT_UUID_WJE=c2f5180f-…`, `API_UUID_WJE=4750650f-…` (and
 - **W12** TicketStages use Freshservice status codes (Open=2 … Closed=5); status sync depends on those matching live Freshservice statuses.
 - Secrets (`API_SECRET_WJE`, the Freshservice password) are hardcoded in source — read them from the file when needed; never copy the values into docs or commits.
 
+## Change history
+- 2026-10-01 — Corrected dept 290 (not synced 1.0 → 2.0); linked the new ticket-writes monitor. (mhammontree)
+
 ## Related
 
 - `clients/wje/profile.md` — WJE client profile + the 2-min Togadesk bridge sync.

package/knowledge/INDEX.md

@@ -5,7 +5,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 ## 1.0 framework
 
 - **library** (Library) _(framework core)_ — 30 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
-- **worker** (Worker) — 40 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
+- **worker** (Worker) — 41 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
 - **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
 - **worker1.5** (Worker 1.5) — 0 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
 - **togadesk** (TOGa Desk) — 13 doc(s) → [1.0/apps/togadesk/INDEX.md](1.0/apps/togadesk/INDEX.md)
@@ -20,7 +20,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 ## 2.0 framework
 
 - **_underscore** (_Underscore) _(framework core)_ — 94 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
-- **worker2** (Worker) — 80 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
+- **worker2** (Worker) — 81 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
 - **api2** (API) — 26 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
 - **dbchanges2** (Database Changes) _(framework core)_ — 22 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
 - **toga2-supply** (TOGa Supply) — 9 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)

package/knowledge/clients/staples/profile.md

@@ -4,18 +4,20 @@ framework: "1.0"
 apps:
   - worker
   - worker2
+  - dbchanges
   - dbchanges2
 project: Worker
 client: staples
 type: profile
 status: active
-updated: 2026-09-18
+updated: 2026-10-01
 owners: [jcardinal, bala, mhammontree]
 files:
   - worker/crons/sync/staples/sync_staples_cxml.php
   - worker/crons/sync/staples/staples_asn_netsuite.php
 related:
   - ../../1.0/apps/worker/features/staples-cxml-order-import.md
+  - ../../1.0/apps/worker/features/staples-asn-856-outbound.md
   - ./features/oneuptime-sftp-folder-monitors.md
   - ../../1.0/apps/worker/features/netsuite-sales-order-sales-rep-sourcing.md
 ---
@@ -27,7 +29,7 @@ Staples — headless Worker-1.0 integration client (two SFTP crons); open for it
 Staples is a **headless** integration client — no UI, Worker 1.0 only, depending on the `library` (1.0 `App_`) core. Its entire footprint is two crons under `worker/crons/sync/staples/`, both scheduled in `schedules/cron.worker.sync.json`:
 
 1. **`sync_staples_cxml.php`** — `45 * * * *`, "Create Sales Orders in Netsuite From Staples". Inbound: pulls cXML purchase orders off SFTP, creates NetSuite Sales Orders, and writes 855 acknowledgements back. The documented feature — see **`1.0/apps/worker/features/staples-cxml-order-import.md`**.
-2. **`staples_asn_netsuite.php`** — `50 * * * *`, "Create ASN from net suite to staples". Outbound ASN (Advance Ship Notice) from NetSuite back to Staples. (Not yet documented in detail.)
+2. **`staples_asn_netsuite.php`** — `50 * * * *`, "Create ASN from net suite to staples". Outbound 856 ASN (Advance Ship Notice) per shipped NetSuite fulfillment, tracked in `Common.StaplesAdvanceShipNotices`. Rule: never send the same ASN twice unless shipping data changed; stays in 1.0 (decided 2026-09-30). See **`1.0/apps/worker/features/staples-asn-856-outbound.md`**.
 
 **NetSuite identity:** Staples is the NetSuite parent entity/customer **internalId 30057**. Individual Staples buying accounts are NetSuite **child customers** matched by their Staples `CustomerID` against **saved search 8412** (match key `custentity_external_customer_id`; location from `custentity_customer_location`).
 

package/knowledge/clients/wje/INDEX.md

@@ -2,4 +2,5 @@
 
 | Doc | Framework | Summary |
 |-----|-----------|---------|
-| [Wiss, Janney, Elstner Associates, Inc.](profile.md) | 2.0 | WJE (Wiss, Janney, Elstner / "WJE IT") Freshservice help-desk client, single hardcoded tenant; open for the three sync paths and the TOGaDesk bridge ids/default |
+| [WJE: TOGaDesk → TOGa 2 ticket-writes monitor (Monitor/Wje/TicketWritesToToga2)](features/oneuptime-ticket-writes-monitor.md) | 2.0 | worker2 OneUptime push monitor counting WJE TOGaDesk tickets/replies/comments never written to TOGa 2 (NULL referenceId); open when working on WJE bridge monito |
+| [Wiss, Janney, Elstner Associates, Inc.](profile.md) | 2.0 | WJE (Wiss, Janney, Elstner / "WJE IT") Freshservice help-desk client, single hardcoded tenant; open for the three sync paths, the TOGaDesk bridge ids/department |

package/knowledge/clients/wje/features/oneuptime-ticket-writes-monitor.md

@@ -0,0 +1,80 @@
+---
+title: "WJE: TOGaDesk → TOGa 2 ticket-writes monitor (Monitor/Wje/TicketWritesToToga2)"
+framework: "2.0"
+repo: worker2
+project: Worker
+client: wje
+type: client-feature
+status: active
+updated: 2026-10-01
+owners: ["mhammontree"]
+files:
+  - worker2/Worker/Monitor/Wje.php
+  - dbchanges2/Core/2026-09-30b - Insert WJE Ticket Writes Monitor CronJob.sql
+  - worker/crons/toga2/wje/sync_togasupply_wje.php
+  - library/app/api/toga2.php
+related:
+  - ../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md
+  - ../../../1.0/apps/library/features/toga2-api-client-and-bridge.md
+  - ../../../2.0/apps/worker2/features/wje-freshservice-sync.md
+  - ../profile.md
+---
+worker2 OneUptime push monitor counting WJE TOGaDesk tickets/replies/comments never written to TOGa 2 (NULL referenceId); open when working on WJE bridge monitoring, its departments, or clearing a stuck ticket.
+
+## Summary
+`Monitor/Wje/TicketWritesToToga2` (`_Worker_Monitor_Wje`, TRUE-80523) watches the 1.0 → 2.0 half of the WJE TOGaDesk bridge. It follows the shared [OneUptime push-metric pattern](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md) — read that for payload contract, criteria, and provisioning.
+
+- **Status 2026-10-01:** committed on branch `TRUE-80523` (worker2 + dbchanges2 + worker comment), **not deployed**. Cron seed ships `isActive = 0`; `$ONEUPTIME_URL` is still empty (push credential — never record it).
+- Runs in worker2, outside the 1.0 worker it watches. Reads `TOGaDeskSupport` via `_underscore::DB_TOGADESK1` (see [worker2 legacy DB access](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md)).
+- OneUptime monitor name: **"WJE Ticket Writes to TOGa 2"**.
+
+## How it works
+
+### Why a backlog, not a checkpoint-freshness check
+The bridge catches a per-ticket exception → Sentry only, and **still advances `TOGA_LAST_TICKET_INTEGRATION_DATETIME` past it**. The failed ticket is never retried unless touched again. A checkpoint-age monitor would read healthy. A row with NULL `referenceId` catches both a dead cron and silent per-ticket failures (link model: [bridge doc](../../../1.0/apps/library/features/toga2-api-client-and-bridge.md)).
+
+### What it counts
+WJE rows (`tickets.clientid = 153`) in `tickets`, `tickets_replies`, `comments` where `referenceId` is NULL or `''`, created on/after `BACKLOG_CUTOFF_DATE = '2025-05-01'`, in the configured departments, minus `excludedTicketNumbers`.
+
+Skip rules mirror `syncTogaDesk1TicketIntoToga2Ticket` in `library/app/api/toga2.php` (~L12280-12290): a reply with a blank message falls back to `newstatus` and is skipped if still PHP-empty (NULL, `''`, `'0'`) — it never gets a `referenceId`, so the monitor skips it too. Comments are always sent.
+
+### Decision + payload
+- `alarm: HIGH` when any counted row is `>= alertThresholdMinutes` old; else `alarm: OK`.
+- `status: error` on DB failure (payload carries **only** `errorClass` + `errorNumber` — the exception message can hold SQL, DB host, DB user) and on empty `departmentIds` (`error: no_departments_configured`).
+- Body: `latePendingCount`, `pendingCount`, per-type counts, `oldestPendingMinutes`, `alertThresholdMinutes`, `lateTicketNumbers` (top 10), `departmentIds`, `excludedTicketCount`.
+
+### Parameters (`Core.CronJobs.parameters`, no deploy to change)
+`{"alertThresholdMinutes":15,"departmentIds":[269],"excludedTicketNumbers":[]}`
+- `departmentIds` PHP default is `[]` **on purpose** — the DB row is the only home of the list. Missing list → `status: error`, never a silent hardcoded dept. The cron comment in `sync_togasupply_wje.php` points here; keep the two in step if the sync's departments change.
+- `excludedTicketNumbers` — TOGaDesk ticket numbers the team gives up on, so one permanently failing ticket does not hold the alarm red.
+
+### Clear a stuck ticket (runbook)
+1. Fix the cause (Sentry has the exception).
+2. Add a reply to the ticket in TOGaDesk so the sync picks it up again — or move the checkpoint back.
+3. If abandoning it, add its ticket number to `excludedTicketNumbers`.
+
+### Activation (in order)
+1. Create the OneUptime Incoming Request monitor with **body rules only**: Contains `"alarm":"HIGH"` → Offline + incident; Contains `"status":"error"` → Offline + incident; Contains `"alarm":"OK"` → Online.
+2. Paste its URL into `$ONEUPTIME_URL`.
+3. Push and deploy worker2.
+4. Run the Core seed `2026-09-30b - Insert WJE Ticket Writes Monitor CronJob.sql`.
+5. Test HIGH: set `departmentIds` to `[290]` (has old unsent rows), hand-run. Restore `[269]`, hand-run to see recovery.
+6. Set `isActive = 1`.
+7. Only then add the not-received 10/15 min rules (see the activation-ordering trap in the shared doc).
+
+## Gotchas
+- **Dept 269 is very low volume** (Sept 2026: 3 tickets, 7 replies, 2 comments). "No writes" is normal — this is why it is a backlog monitor, not a "writes stopped" monitor.
+- **Blind spot: field-only edits.** Priority/assignee change on an already-linked ticket creates no row, so a failed push is invisible. Status changes are covered (they arrive as replies).
+- **Clock is US/Central.** `TOGaDeskSupport` `NOW()` and row timestamps are both server US/Central (the `_underscore` driver sets no session `time_zone`). Spring DST can false-alarm once; autumn can delay an alarm up to 1h.
+- **Pre-cutoff history never drains:** 21 sendable unsent dept-269 rows 2024-12 → 2025-04-22 (early integration). That is why the cutoff is 2025-05-01.
+- **Query cost (prod, verified):** `tickets` via index_merge(departmentid, clientid); `tickets_replies` via `ticketid` index; `comments` has **no** `ticketid` index → full scan of ~13k rows (acceptable). Timestamp columns have no `ON UPDATE`.
+- **Do not add 290 casually.** Dept 290 is not synced 1.0 → 2.0 today (see [profile](../profile.md)); pointing the monitor at it would alarm on the known backlog, not a new failure.
+- **The ClickUp TRUE-80523 description is fabricated** (Talos auto-generated) — see [MCP tool usage gotcha](../../../standalone/apps/claude/workflows/mcp-tool-usage.md). Trust this doc and the code.
+
+## Related
+- [OneUptime push-metric monitors for 2.0 workers](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md) — shared pattern
+- [App_Api_Toga2 bridge](../../../1.0/apps/library/features/toga2-api-client-and-bridge.md) — `syncWithTogadesk`, `referenceId` write-back
+- [WJE profile](../profile.md)
+
+## Change history
+- 2026-10-01 — Created: monitor built on branch TRUE-80523, not deployed, cron `isActive = 0`. (mhammontree)

package/knowledge/clients/wje/profile.md

@@ -4,22 +4,26 @@ framework: "2.0"
 apps:
   - worker
   - library
+  - worker2
+  - dbchanges2
 project: Library
 client: wje
 type: profile
 status: active
-updated: 2026-09-16
-owners: [jcardinal]
+updated: 2026-10-01
+owners: [jcardinal, mhammontree]
 files:
   - worker/crons/toga2/wje/sync_togasupply_wje.php
   - worker2/Worker/Wje.php
   - worker/crons/toga2/wje/import_supporting_records.php
+  - worker2/Worker/Monitor/Wje.php
 related:
   - ../../1.0/apps/library/features/toga2-api-client-and-bridge.md
   - ../../2.0/apps/worker2/features/wje-freshservice-sync.md
+  - ./features/oneuptime-ticket-writes-monitor.md
 ---
 
-WJE (Wiss, Janney, Elstner / "WJE IT") Freshservice help-desk client, single hardcoded tenant; open for the three sync paths and the TOGaDesk bridge ids/defaults.
+WJE (Wiss, Janney, Elstner / "WJE IT") Freshservice help-desk client, single hardcoded tenant; open for the three sync paths, the TOGaDesk bridge ids/departments, and its monitor.
 
 ## Summary
 
@@ -31,9 +35,19 @@ Data moves between WJE and TOGA on three paths:
 
 1. **Real-time Freshservice webhook** → `_Worker_Wje::Webhook` (worker2) — creates/updates tickets, contacts, assets, groups, categories, and conversation notes, keyed by `c_wje*` custom fields.
 2. **Daily batch import** (`worker/crons/toga2/wje/import_supporting_records.php`) — imports canned responses, requesters, assets, groups, and the 3-level category hierarchy.
-3. **2-minute TOGaDesk bridge** (`App_Api_Toga2::syncWithTogadesk`) — the cron `worker/crons/toga2/wje/sync_togasupply_wje.php` enables the **full** set of bridge sub-syncs (tickets both directions, ticket-notes, contacts→people, items/units→assets, predefined-replies, ticket-teams→groups, ticket-categories) for TOGaDesk client id 153, department 269 (plus internal-tickets department 290).
+3. **2-minute TOGaDesk bridge** (`App_Api_Toga2::syncWithTogadesk`) — the cron `worker/crons/toga2/wje/sync_togasupply_wje.php` enables the **full** set of bridge sub-syncs (tickets both directions, ticket-notes, contacts→people, items/units→assets, predefined-replies, ticket-teams→groups, ticket-categories) for TOGaDesk client id 153. **TOGaDesk → 2.0 syncs department 269 only.**
+
+Monitoring: [ticket-writes monitor](./features/oneuptime-ticket-writes-monitor.md) alarms when a dept-269 TOGaDesk row is not written to 2.0.
+
+## TOGaDesk departments
+
+- **269** — tickets created in TOGaDesk (`153000xxx` numbers). Very low volume. Synced TOGaDesk → 2.0.
+- **290** — WJE Freshservice tickets that arrive in TOGaDesk **from** 2.0 (ticket number = Freshservice INC number, e.g. `65239`; ~100-200/month, all carry `referenceId`). The cron defines `TOGADESK_WJE_INTERNAL_TICKETS_DEPARTMENT_ID = 290` but never passes it (`$monitorTogadeskDepartmentIds` not passed), so 290 is **not** synced TOGaDesk → 2.0. Confirmed with Jonathan 2026-09-30: only 269 is synced today.
+
+**⚠ Known gap (not fixed):** a reply or comment typed in TOGaDesk on a dept-290 ticket is never written to 2.0 and never reaches WJE. Verified: ticket `65239`, a reply on 2026-08-18 has NULL `referenceId` and is absent from `Client_Wje.TicketNotes`. Since 2026-01 ~49 replies + 10 comments are stuck. **If 290 is ever added to the sync, its first run pushes that whole old backlog to WJE at once — decide what to do with it first.**
 
 ## Related
 
-- **[wje-freshservice-sync](../../2.0/apps/worker2/features/wje-freshservice-sync.md)** — full integration: webhook handler, import cron, `c_wje*` models, API clients, schema, reports, monitoring, gotchas.
+- **[wje-freshservice-sync](../../2.0/apps/worker2/features/wje-freshservice-sync.md)** — full integration: webhook handler, import cron, `c_wje*` models, API clients, schema, reports, gotchas.
 - **[toga2-api-client-and-bridge](../../1.0/apps/library/features/toga2-api-client-and-bridge.md)** — the `syncWithTogadesk` bridge mechanism.
+- **[oneuptime-ticket-writes-monitor](./features/oneuptime-ticket-writes-monitor.md)** — the 1.0 → 2.0 write monitor.

package/knowledge/standalone/apps/claude/workflows/mcp-tool-usage.md

@@ -6,8 +6,8 @@ project: Claude Harness
 client: shared
 type: workflow
 status: active
-updated: 2026-09-25
-owners: ["jcardinal", "tcox", "ajean"]
+updated: 2026-10-01
+owners: ["jcardinal", "tcox", "ajean", "mhammontree"]
 files:
   - claude/.claude/skills/kickoff/SKILL.md
   - claude/.claude/skills/capture/SKILL.md
@@ -106,6 +106,7 @@ SELECT name, clientIdentifier FROM Clients WHERE name LIKE '%<keyword>%';
 - **⚠ "Beta" is `dev-sandbox`, NOT `client-beta`.** For the beta API tier (`api.beta.togahub.com`), `Core.DatabaseHosts` maps environment `beta` for `Client_*` schemas to the **dev-sandbox** cluster — so "the beta database" and "the dev-sandbox database" are the same host. The MCP's `client-beta` environment exposes **no schemas** and querying it looks like the data is missing. Query **`dev-sandbox`** whenever someone says "beta". Any frontend pointed at `VITE_API=https://api.beta.togahub.com/v2` is therefore reading/writing dev-sandbox data. (2024-era onboarding SQL shows beta on a us-west-2 `client-cluster`; that has since been consolidated — trust `Core.DatabaseHosts`, not old migrations.)
 - **⚠ The write-blocker is a naive string scan — it rejects `UPDATE`/`REPLACE` even inside a string literal.** `toga_query` refuses the whole statement if those words appear anywhere in it, so a read-only query like `… WHERE output LIKE '%skipped update%'` fails validation. **A column alias trips it too** — `… AS updateDisabled` is rejected on the same scan, so the alias has to be renamed (`skippedDisabled`), not just the predicate. Work around it by matching on a fragment that avoids the keyword (`LIKE '%disabled%'`, `LIKE '%skipped%'`), or by pulling the rows and filtering client-side — don't conclude the data isn't there.
 - **Production sessions render in US/Central (UTC−5); most other timestamps you're comparing are UTC.** GitHub/API/webhook timestamps are UTC, so a naive `dtStamp` vs. commit-time comparison is off by five hours — enough to make a healthy worker fleet look dead. Anchor both sides to UTC (or `CONVERT_TZ`) before drawing a conclusion about "when did this last run".
+- **⚠ Talos auto-generated "KB investigation" text on a ClickUp ticket can be fabricated.** TRUE-80523's description called WJE "Write Job Entry" and cited a CloudWatch/Datadog ADR, a DLQ, and a Migration Sprint retro — none exist. Treat such write-ups as unverified; check every name and source against the code, this KB, and the data before acting on it.
 - **`local` is not this MCP.** If someone means their own machine's database, use the `mysql` CLI via shell, not `toga-db`.
 - **The client roster is wider than `registry.json`, and identifiers are not guessable.** `Core.Clients` held **46 clients** on 2026-07-30 against 17 in `registry.json`. A client absent from the registry is normal, not an error — the registry tracks clients whose *code knowledge* has been captured, not the full book of business. Identifiers do not follow from the display name, so **look them up, never construct them**:
 
@@ -123,6 +124,7 @@ SELECT name, clientIdentifier FROM Clients WHERE name LIKE '%<keyword>%';
   Schema names are **case-sensitive** — copy the exact value from `toga_list_schemas` or `Core.Clients`. `Client_Aig` works; `Client_AIG` and `client_aig` do not.
 
 ## Change history
+- 2026-10-01 — Added gotcha: Talos auto-generated ClickUp ticket write-ups can invent sources (TRUE-80523). (mhammontree)
 - 2026-09-25 — corrected the "TOGa IQ" row: it now names the separate `toga25-iq` app, not the `talos` codebase (rebrand) (jcardinal)
 
 ## Related

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.925",
+  "version": "1.0.927",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",