toga-ai 1.0.596 → 1.0.598

package/knowledge/2.0/apps/worker2/INDEX.md

@@ -18,6 +18,7 @@
 | [Compass VIP Support Importer (worker2)](features/compass-vip-support-importer.md) | A worker2 action that ingests Compass's quarterly VIP spreadsheet and assigns each VIP user's support technician by setting `Users.c_supportedByUserId` in `Clie | worker2/Worker/Client/Compass/VipSupport.php |
 | [Creating Worker Actions](features/creating-worker-actions.md) | How to add a new callable Worker action — a PHP class whose `public static` methods are invoked as background jobs (via webhook, cron, or `_Worker::runTask()`). | worker2/Worker/, worker2/Controller/Index.php, _underscore/Worker.php |
 | [Cross-account AWS access for worker2 crons (_Component_Aws_Workloads)](features/cross-account-aws-access.md) | `_Component_Aws_Workloads` is the **standard, and only sanctioned, way any new worker2 cron obtains AWS access** — for any account, any region, any AWS SDK clie | worker2/Component/Aws/Workloads/Workloads.php, worker2/Config/production.ini, worker2/Worker/Infrastructure/CloudWatch.php |
+| [Elastic Beanstalk health monitor → OneUptime (ElasticBeanstalkHealth)](features/elastic-beanstalk-health-monitor.md) | `_Worker_Infrastructure_CloudWatch::ElasticBeanstalkHealth()` is a worker2 cron that reads each Elastic Beanstalk (EB) environment's **enhanced-health** status | worker2/Worker/Infrastructure/CloudWatch.php |
 | [Elite Freshservice Sync (worker2)](features/elite-freshservice-sync.md) | `_Worker_Elite` processes Freshservice webhook events and syncs them into TOGA 2. | worker2/Worker/Elite.php, worker2/Config/dev-kmaramreddy-laptop.ini |
 | [Error Escalation Cron (Errors::Escalate → ClickUp / email)](features/error-escalation-cron.md) | `_Worker_Infrastructure_Errors::Escalate` (renamed from `SyncWithClickup`) is the sole owner of **escalation, de-escalation, ClickUp ticketing, reminders, busin | worker2/Worker/Infrastructure/Errors.php, worker2/Worker/Notification/Email.php, worker2/Worker/Notification/EmailTemplate.php, worker2/Worker/Client/True.php, worker2/Worker/Clickup/ErrorTask.php, worker2/Worker/Clickup.php, worker2/Controller/Index.php, worker2/Config/production.ini, _underscore/Model/Core/Logs/Issue.php, dbchanges2/Core/2026-07-30a - Error escalation cron job.sql, dbchanges2/Logs/2026-08-03a - Issue clickupPriority.sql, dbchanges2/Core/2026-08-04a - Error neglect digest cron job.sql |
 | [Error-Issue Auto-Resolution & Reopen (frequency-decay lifecycle)](features/error-issue-auto-resolution.md) | The error system could escalate and de-escalate an Issue's *urgency* but had no concept of an Issue being **resolved**. | worker2/Worker/Infrastructure/Errors.php, worker2/Worker/Clickup/ErrorTask.php, _underscore/Model/Core/Logs/Issue.php, tools/mvc/errors/get.php, tools/mvc/errors/issue/get.php, dbchanges2/Logs/2026-08-05a - Issue status baseline and auto-resolution.sql |
@@ -33,7 +34,7 @@
 | [DB-Driven Notification (Internal) Email](features/notification-email.md) | Internal/notification emails (merge-conflict alerts, ops notices — anything system-generated, not client-facing transactional mail) are sent through one worker | worker2/Worker/Notification/Email.php, _underscore/Model/Client/EmailTemplate.php, dbchanges2/Client/2026-06-23a - EmailTemplateWrapper.sql, dbchanges2/Client_True/2026-06-23a - EmailTemplateWrapper.sql |
 | [NYCHH Asset-Tag Backfill (worker2)](features/nychh-asset-tag-backfill.md) | Keeps NYC Health & Hospitals (`Client_Nychh`) unit asset tags and MAC addresses synced from NetSuite. | worker2/Worker/Client/Nychh.php, worker2/Worker/Client/Nychh/AssetTagBackfill.php, worker/crons/toga2/netsuite/verify_fulfillment_asset_tag_sync_nychh.php, worker/crons/toga2/netsuite/backfill_all_asset_tags_from_netsuite_nychh.php |
 | [OneUptime Incident → ClickUp Task Sync (Monitor/Oneuptime/SyncIncidents)](features/oneuptime-incident-clickup-sync.md) | An internal/shared TOGA ops feature: worker2 polls the OneUptime API every 15 minutes for **currently-open** incidents and ensures a ClickUp task exists for eac | worker2/Component/Api/Oneuptime/Oneuptime.php, worker2/Worker/Monitor/Oneuptime.php, worker2/Config/production.ini, dbchanges2/Team/2026-08-10a - OneUptime Incident ClickUp Tasks.sql, dbchanges2/Core/2026-08-10b - OneUptime Incident Sync CronJob.sql, dbchanges2/Team/2026-08-12a - OneUptime Incident Tasks Closed Marker.sql |
-| [OneUptime push-metric monitors for 2.0 workers](features/oneuptime-worker2-monitoring.md) | A second, **OneUptime-reporting** monitoring pattern for the 2.0 worker2 tier, ported from the 1.0 `App_SystemMonitor_Compass` monitors. | worker2/Worker/Monitor/Compass.php, worker2/Worker/Client/Compass.php, worker2/composer.json, _underscore/Cloud.php, worker2/Worker/Monitor/Operations.php |
+| [OneUptime push-metric monitors for 2.0 workers](features/oneuptime-worker2-monitoring.md) | A second, **OneUptime-reporting** monitoring pattern for the 2.0 worker2 tier, ported from the 1.0 `App_SystemMonitor_Compass` monitors. | worker2/Worker/Monitor/Compass.php, worker2/Worker/Client/Compass.php, worker2/composer.json, _underscore/Cloud.php, worker2/Worker/Monitor/Operations.php, worker2/Worker/Monitor/Aig.php, worker2/Worker/Monitor/Prudential.php, worker2/Worker/Monitor/Nycdoe.php, worker2/Config/production.ini, worker2/Config/beta.ini, worker2/Config/sandbox-dev.ini |
 | [Platform Cache Cleanup (_Worker_Platform_Cache — Clean + Truncate)](features/platform-cache-cleanup.md) | `_Worker_Platform_Cache` owns maintenance of the shared **Cache** cluster that backs api2's [multi-client data retrieval](../../api2/features/cross-client-data- | worker2/Worker/Platform/Cache.php, worker2/Controller/Index.php, worker2/_.php, dbchanges2/Core/2026-07-27a - PlatformCacheCleanCron.sql |
 | [Service Request → Sales Order → Purchase Order generation (Sync/ServiceRequest)](features/service-request-sales-order-generation.md) | `_Worker_Sync_ServiceRequest` turns a **Service Request into a Sales Order, and then into one Purchase Order per vendor**, for **any** tenant. | worker2/Worker/Sync/ServiceRequest.php, _underscore/Model/Elite/ServiceRequest.php, dbchanges2/Core/2026-08-06a - Service request and sales order payload interceptors.sql |
 | [Startech Webhook Handler (worker2)](features/startech-webhook-handler.md) | Receives inbound webhook events from Startech (Easeedesk) and creates or updates the corresponding ticket in TOGA 2.0. | worker2/Worker/Startech.php |

package/knowledge/2.0/apps/worker2/features/cross-account-aws-access.md

@@ -17,6 +17,7 @@ related:
   - ../../_underscore/features/config-group-access.md
   - ../../_underscore/features/cloud-s3-helpers.md
   - ./oneuptime-worker2-monitoring.md
+  - ./elastic-beanstalk-health-monitor.md
 ---
 
 ## Summary

package/knowledge/2.0/apps/worker2/features/elastic-beanstalk-health-monitor.md

@@ -0,0 +1,182 @@
+---
+title: Elastic Beanstalk health monitor → OneUptime (ElasticBeanstalkHealth)
+framework: "2.0"
+repo: worker2
+project: Worker
+client: shared
+type: feature
+status: active
+updated: 2026-08-17
+owners: [jcardinal]
+files:
+  - worker2/Worker/Infrastructure/CloudWatch.php
+related:
+  - ./cross-account-aws-access.md
+  - ./oneuptime-worker2-monitoring.md
+  - ./monitoring-framework.md
+---
+
+## Summary
+
+`_Worker_Infrastructure_CloudWatch::ElasticBeanstalkHealth()` is a worker2 cron that reads
+each Elastic Beanstalk (EB) environment's **enhanced-health** status and reports it to a
+OneUptime **Incoming Request** monitor, which pages the team. It is a **"dumb reporter,
+smart monitor"** heartbeat (the same Pattern-B contract as
+[OneUptime push-metric monitors](./oneuptime-worker2-monitoring.md)): the worker decides
+pass/fail and POSTs decided **string tokens** (`alarm=HIGH/OK`, `probe=DEGRADED/OK`) that
+OneUptime string-matches — OneUptime cannot compare numbers on a pushed body.
+
+The load-bearing design decision this doc records is the **paging (alarm) criteria**: page
+on EB `Degraded`/`Severe`, but **suppress the common "an occasional HTTP 500 tripped
+Degraded" false alarm** unless 5xx errors actually dominate the request mix.
+
+**Critical behavior:** the method is **non-fatal and never throws** — it always returns a
+string. worker2 has **no DLQ and a 3600s SQS visibility timeout**, so any uncaught 500
+becomes a poison-message storm; an outer `catch(\Throwable)` is the backstop and the push
+itself runs non-fatal.
+
+## Key files / entry points
+
+- `worker2/Worker/Infrastructure/CloudWatch.php` — `_Worker_Infrastructure_CloudWatch`,
+  action `ElasticBeanstalkHealth`, dispatched via
+  `_Worker::runTask('Infrastructure/CloudWatch/ElasticBeanstalkHealth', {awsAccountId,
+  environments, oneuptimeUrl})`.
+- AWS access is obtained through `_Component_Aws_Workloads::assumeCredentials()` (STS-assume
+  a read-only role per account, one assume reused across regions) — see
+  [Cross-account AWS access](./cross-account-aws-access.md), which also documents the
+  `environments` region→env-names map parameter shape.
+- `oneuptimeUrl` arrives as a **cron parameter and is a push credential** — never log it and
+  never record its value in a doc (see gotchas).
+
+## Alarm / paging criteria (the decision logic)
+
+Per environment, `alarm=HIGH` (page) when the EB `HealthStatus` is `Degraded` **or**
+`Severe`. `Suspended` never pages.
+
+- **`Severe` always pages.**
+- **`Degraded` for a non-5xx reason** (latency, instances down, …) **always pages.**
+- **`Degraded` attributed to HTTP 5xx errors pages only when 5xx errors dominate** — i.e.
+  the 5xx share of requests is at/above `HTTP_5XX_ALARM_RATIO` (default 0.80). An occasional
+  500 that trips Degraded is noise and must **not** page; a flood is a real problem and
+  **must** page.
+- **Tiny-sample guard:** below `HTTP_5XX_MIN_REQUEST_COUNT` (default 20) requests in the
+  window, do **not** trust the ratio — page rather than let a 2-of-3 blip read as "80%
+  failing."
+
+### Fail-safe: a blind read pages, never reads healthy
+
+If the health read (`describeEnvironmentHealth`) throws `AwsException` for an environment,
+the monitor sets **`alarm=HIGH`** for that environment (a blind read must never masquerade
+as healthy) **and** sets **`probe=DEGRADED`** (coverage gap, distinct from a measured-bad
+signal — same alarm-vs-probe split as the multi-client monitors in
+[OneUptime push-metric monitors](./oneuptime-worker2-monitoring.md)).
+
+### Configurable constants (top of the class)
+
+| Constant | Default | Meaning |
+|---|---|---|
+| `HEALTH_ALARM_STATUSES` | `['Degraded','Severe']` | HealthStatus values that page |
+| `HTTP_5XX_ALARM_RATIO` | `0.80` | 5xx share (0.0–1.0) at/above which a 5xx-driven `Degraded` pages |
+| `HTTP_5XX_MIN_REQUEST_COUNT` | `20` | Below this many requests in the window, page rather than trust the ratio |
+| `HTTP_5XX_CAUSE_MARKERS` | `['5xx','http 5']` | Case-insensitive substrings identifying a 5xx-attributed EB `Cause` |
+
+### Implementation shape
+
+- `shouldPageForHealth(healthStatus, causes, applicationMetrics): [bool, ?float]` — returns
+  the page decision and the observed 5xx ratio.
+- `causesAttributeTo5xx(causes): bool` — substring-matches `HTTP_5XX_CAUSE_MARKERS` against
+  the EB `Causes` strings (case-insensitive).
+- The ratio is computed as `StatusCodes.Status5xx / RequestCount` from `ApplicationMetrics`
+  (raw counts — see below), **not** by parsing the percentage out of the `Causes` text.
+- `describeEnvironmentHealth` is called with
+  `AttributeNames = ['HealthStatus','Status','Color','Causes','ApplicationMetrics']`.
+- Per-environment report + OneUptime payload now include the per-env `alarm` token and the
+  observed `fivexxRatio`.
+
+## How EB enhanced-health is read (durable AWS reference)
+
+Facts verified against current AWS docs; they inform why the logic above is shaped as it is.
+
+- **HealthStatus severity ladder:** `Ok → Info → Warning → Degraded → Severe` (plus the grey
+  states `Pending`/`Unknown`/`Suspended`/`NoData`). `Degraded` is the "high failure" tier
+  and is **routinely tripped by benign transients** (Auto Scaling scale-up, a mid-deploy
+  dip), which is exactly why `Degraded` alone is noisy. `Severe` = "very high failure /
+  environment effectively not serving."
+- **`ApplicationMetrics` returns RAW COUNTS, not percentages** — despite the API-reference
+  prose saying "percentage"/"per second". Fields: `Duration` (Integer seconds, usually 10),
+  `RequestCount` (Integer, **total** requests over the window),
+  `StatusCodes.{Status2xx,Status3xx,Status4xx,Status5xx}` (Integer counts). AWS's own
+  example: 2xx 3391 + 5xx 843 = RequestCount 4234. With no traffic, `RequestCount=0` and
+  `StatusCodes` may be **absent** → treat as "no data," not "zero failures." The 5xx-ratio
+  denominator is `RequestCount` (the authoritative total).
+- **Why an env is Degraded comes from `Causes`:** EB `Causes` strings literally contain e.g.
+  `"19.9 % of the requests are failing with HTTP 5xx."` A substring check reliably
+  **classifies** 5xx-driven vs. not (fail-safe: a wording change → treated as a normal
+  `Degraded` → pages). The **ratio itself must come from `ApplicationMetrics`**, never by
+  parsing the number out of `Causes`.
+- **A failed application DEPLOYMENT is NOT reliably reflected in HealthStatus** — it can read
+  `Warning`/`Degraded`, or fail before any red request-failure signal. The authoritative
+  structured deploy signal is per-instance `describeInstancesHealth` →
+  `Deployment.Status ∈ {'In Progress','Deployed','Failed'}`. (This detector was evaluated and
+  removed this iteration — see Design history.)
+- **IAM:** `describeEnvironmentHealth` / `describeInstancesHealth` / `describeEvents` are all
+  covered by the managed `AWSElasticBeanstalkReadOnly` policy and require enhanced health
+  enabled. No per-call charge.
+
+## OneUptime monitor config (external, not code)
+
+Paging depends on the OneUptime Incoming-Request monitor's **string-match** criteria on the
+POSTed body — `Contains "alarm":"HIGH"` (and, if desired, `Contains "probe":"DEGRADED"`).
+This is external OneUptime configuration, not worker2 code.
+
+## Design history — rejected direction (this session)
+
+An earlier iteration set `HEALTH_ALARM_STATUSES = ['Severe']` (dropping `Degraded` entirely)
+and **added a separate deployment-failure detector** (`describeInstancesHealth` →
+`Deployment.Status = 'Failed'`) to still catch deploy failures. That was **reverted** after a
+scope change: the team decided they **do** want to page on `Degraded` (to catch a flood of
+500s and non-500 degradations), with only the occasional-500 case suppressed via the 80%
+ratio. The deployment-failure detector was **removed** — deploy failures that surface as
+`Degraded` are now covered by the `Degraded` alarm. **Residual gap accepted for this
+iteration:** a deploy that fails fast *without ever degrading health* is not caught.
+
+## Known gaps / follow-ups (not done)
+
+- **Per-region client construction is not individually guarded.** Each
+  `new ElasticBeanstalkClient` is not in its own try/catch, so a bad region/creds aborts the
+  **whole run** via the outer backstop instead of paging just that region's envs as blind.
+  Candidate follow-up now that blind reads page.
+- **No first-party test harness exists in worker2** (all tests are vendor/).
+  `shouldPageForHealth()` and `causesAttributeTo5xx()` are pure and ideal to unit-test
+  (ratio at exactly 0.80; count 19 vs 20; `Severe` over a 5xx cause; empty causes; null
+  metrics) — deferred pending a harness.
+
+## Gotchas / known issues
+
+- **Keep the method non-fatal — never let it throw.** worker2 has no DLQ and a 3600s SQS
+  visibility timeout, so any uncaught 500 becomes a poison-message storm. The push runs
+  non-fatal and an outer `catch(\Throwable)` is the backstop.
+- **A blind health read must page, not read healthy** — an `AwsException` on
+  `describeEnvironmentHealth` sets `alarm=HIGH` + `probe=DEGRADED` for that env.
+- **`ApplicationMetrics` is raw counts, not percentages** — divide `Status5xx` by
+  `RequestCount`; `RequestCount=0`/absent `StatusCodes` means "no data," not "zero failures."
+- **Classify 5xx-attribution from `Causes` text, but take the ratio from `ApplicationMetrics`
+  — never parse the percentage out of `Causes`.**
+- **`oneuptimeUrl` is a push credential** — it arrives as a cron parameter; never log it or
+  record its value in a doc.
+
+## Change history
+- 2026-08-17 — Created. Documented the `ElasticBeanstalkHealth` alarm/paging criteria (page
+  on `Degraded`/`Severe`; suppress a 5xx-driven `Degraded` unless the 5xx share ≥
+  `HTTP_5XX_ALARM_RATIO` 0.80 with an `HTTP_5XX_MIN_REQUEST_COUNT` 20 tiny-sample guard;
+  `Severe` and non-5xx `Degraded` always page), the blind-read fail-safe (`alarm=HIGH` +
+  `probe=DEGRADED`), the four configurable constants, and the `shouldPageForHealth` /
+  `causesAttributeTo5xx` helper shape. Captured durable EB enhanced-health facts (severity
+  ladder; `ApplicationMetrics` returns raw counts not percentages; `Causes` classifies but
+  `ApplicationMetrics` sets the ratio; failed deploys aren't reliably in HealthStatus —
+  `describeInstancesHealth.Deployment.Status` is authoritative; `AWSElasticBeanstalkReadOnly`
+  covers the reads). Recorded the reverted `['Severe']`-only + deployment-failure-detector
+  direction and the accepted residual gap (a deploy that fails without degrading health).
+  (jcardinal)
+</content>
+</invoke>

package/knowledge/2.0/apps/worker2/features/oneuptime-worker2-monitoring.md

@@ -6,14 +6,20 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-07-30
-owners: ["jcardinal"]
+updated: 2026-08-17
+owners: ["jcardinal", "mhammontree"]
 files:
   - worker2/Worker/Monitor/Compass.php
   - worker2/Worker/Client/Compass.php
   - worker2/composer.json
   - _underscore/Cloud.php
   - worker2/Worker/Monitor/Operations.php
+  - worker2/Worker/Monitor/Aig.php
+  - worker2/Worker/Monitor/Prudential.php
+  - worker2/Worker/Monitor/Nycdoe.php
+  - worker2/Config/production.ini
+  - worker2/Config/beta.ini
+  - worker2/Config/sandbox-dev.ini
 related:
   - ./oneuptime-incident-clickup-sync.md
   - ./monitoring-framework.md
@@ -136,6 +142,69 @@ POSTed JSON body is addressed via the `requestBody` prefix — e.g.
 The 10-min-Degraded / 15-min-Offline heartbeat thresholds pair with a **5-minute** cron
 cadence. An initial 3/5-min setting false-alarmed on a single missed ping.
 
+### Two kinds of monitor — liveness probes and log-scrapers are complements, not duplicates
+
+A monitor that **scrapes log rows** for arrived-and-failed calls (`Monitor/Aig/EntitlementApiFailures`,
+the five 1.0 `App_SystemMonitor_ServiceNow*` classes) is **structurally blind to a fully dead
+endpoint**: a refused connection logs nothing, produces zero failure rows, and reads as healthy.
+Pair every failure-rate monitor on an integration endpoint with a **liveness probe** that actually
+touches it. Rules for the probe:
+
+- Use an **unauthenticated GET** so the probe can never create or mutate a record.
+- **Any structured HTTP response below 500 is HEALTHY** — it proves DNS, TLS, the load balancer,
+  routing, and the auth layer are all alive. A `401` from an unauthenticated call to a protected
+  route is a *passing* result, not a failure. Only ≥500 / connect failure / timeout means dead.
+  (Verified 2026-08-17: `api-writer.togahub.com/v2/health` → 200; unauthenticated `/v2/entitlements`
+  → 401; both `alarm:"OK"`.)
+- api2 short-circuits its health routes before any DB or Sentry work
+  (`api2/Controller/Index.php` `HEALTH_CHECK_ROUTES = ['/health','/v2/health']` → 200), so probing
+  them is cheap and does not depend on a client DB being reachable.
+- **Token validity is out of scope for a worker2 probe** when the OAuth client lives in 1.0
+  (e.g. NYCDOE ServiceNow's `App_Api_NYCDOEV2` in `library`). Re-implementing it in worker2 would
+  duplicate credentials into a second repo and race the shared access token. Accept and record the
+  coverage gap rather than forking the client.
+
+### Tunable thresholds live in `Core.CronJobs.parameters`, everything else stays a local
+
+The alarm threshold is the **one** setting that goes in the DB, as
+`parameters = '{"alertThreshold":N}'`, so it can be retuned without a deploy. worker2 spreads
+`parameters` as **named PHP arguments**, so the JSON key must match the method's parameter name
+**exactly** or the call fails. Every other monitor setting (URLs, cutoffs, window sizes) stays an
+`ALL_CAPS` in-method local.
+
+**Echo `alertThreshold` in the pushed body next to the measured count.** A historical alarm is
+otherwise unreadable once the threshold has been retuned.
+
+### Backlog monitors must exclude known-permanently-stranded rows
+
+A backlog count that includes rows which will *never* drain pins the monitor HIGH from its first
+run, and a monitor that is always red trains people to ignore it. Define a `BACKLOG_CUTOFF_DATE`
+local at the date the monitor was written and count only rows on/after it. Report stranded/unmapped
+row counts as **diagnostic fields that deliberately do not drive the alarm**.
+
+### Monitors are safe to deploy inert
+
+Ship with the `Core.CronJobs` row `isActive = 0`. Each method must guard on an empty push URL, log
+`no OneUptime push URL configured`, and still record its measurement in `WorkerJobs` — so the code
+can deploy and be hand-run before the monitor exists in OneUptime.
+
+### Calibrate thresholds against the tier that runs the business logic
+
+Several monitors live in worker2 but watch a pipeline whose crons run in the **1.0 worker** tier
+against a 2.0 client DB (worker2 only READs it) — Compass and Prudential both work this way.
+Thresholds must be derived from the **1.0 cron cadences and per-run throughput**, not from
+worker2's own `*/5` schedule.
+
+### End-to-end test procedure (no deploy required)
+
+Testing **recovery** is the step people skip and the most valuable one:
+
+1. `UPDATE Core.CronJobs SET parameters = '{"alertThreshold":-1}'` for the action.
+2. Hand-run the action → confirm `alarm:"HIGH"`, the monitor flips **Offline**, an incident opens,
+   and the notification arrives.
+3. Restore the real threshold and hand-run again → confirm the OK-gated criterion returns the
+   monitor to **Online** and resolves the incident.
+
 ## Provisioning a monitor (runbook)
 
 1. Write the monitor method on `_Worker_Monitor_Compass` (or a new `_Worker_Monitor_<X>`),
@@ -144,9 +213,14 @@ cadence. An initial 3/5-min setting false-alarmed on a single missed ping.
    **push credential**; never log it and never record its value in a doc.
 3. Provision the OneUptime monitor by importing the monitor's OneUptime resource-export
    JSON (authored under a local staging dir, e.g. `d:\STAGE` — these are import artifacts,
-   not repo files) and wiring the Contains criteria above.
-4. Add the `Core.CronJobs` row: `action = 'Monitor/Compass/<Method>'`, schedule
-   `*/5 * * * *`, `maxExecutionTime` 120s, `parameters` NULL. Then deploy worker2.
+   not repo files) and wiring **only the body-`Contains` criteria** at this point.
+   The **push URL does not exist until the monitor is saved** — OneUptime generates it, it cannot
+   be pre-derived, so the monitor must exist before the worker code can be finished.
+4. Add the `Core.CronJobs` row: `action = 'Monitor/<Class>/<Method>'`, schedule
+   `*/5 * * * *`, `maxExecutionTime` 120s, `parameters` NULL (or
+   `{"alertThreshold":N}`), `isActive = 0`. Then deploy worker2.
+5. Flip `isActive = 1` and **confirm a push has landed**. Only then add the
+   **absence criteria** (not-received 10/15 min) — see the activation-ordering trap below.
 
 ### CronJob scheduling for these monitors
 
@@ -200,8 +274,15 @@ check for another client.
 - OneUptime Incoming Request bodies are matched **as strings only** (Contains/NotContains) —
   no numeric comparison, no JSON key targeting. Always emit a decided token, never a raw
   number, for push monitors.
-- The OneUptime push URL is a credential — keep it as an in-file constant in the method;
-  never log it or write its value into knowledge docs.
+- **ACTIVATION-ORDERING TRAP.** Creating a monitor with the **absence** criteria already wired
+  while its cron is still `isActive = 0` makes it go **Offline and open an incident within 15
+  minutes** with nothing pushing. Wire body-match criteria at creation; add absence criteria only
+  after `isActive = 1` and a confirmed push — or mute the monitor until then.
+- The OneUptime push URL is a credential — a holder can spoof a healthy heartbeat and thereby
+  silence a real outage. Keep it as an in-file `$ONEUPTIME_URL` local in the method (the
+  established team pattern — 11 pre-existing instances across `Monitor/Compass.php`,
+  `Monitor/Aig.php`, `Monitor/Operations.php`); never log it and never write its value into a doc.
+  Moving these to `Config/production.ini` is a recognized future improvement, not yet done.
 - **`App_Client_Compass` (1.0) is unavailable in worker2.** Referencing it throws a
   class-not-found `Error`, which `catch (Exception)` does NOT catch — the whole monitor
   dies uncaught. Use `_Worker_Client_Compass::` constants only.
@@ -212,6 +293,19 @@ check for another client.
   monitors must pass the region (see [Cloud S3 helpers](../../_underscore/features/cloud-s3-helpers.md)).
 
 ## Change history
+- 2026-08-17 — TRUE-80587: added five monitors and generalized this doc beyond Compass —
+  `Monitor/Prudential/{SalesOrderGenerationBacklog,DellPurchaseOrderTransmissionBacklog,
+  OrderShippedUpdateBacklog}`, `Monitor/Aig/EntitlementApiHealth`,
+  `Monitor/Nycdoe/ServiceNowEndpointHealth` (cron rows in
+  `dbchanges2/Core/2026-08-07a - Prudential AIG ServiceNow OneUptime monitor crons.sql`).
+  New durable sections: **liveness probes vs. log-scrapers are complements** (a scraper cannot see
+  a dead endpoint; <500 including 401 is healthy), the **`alertThreshold` DB parameter** convention
+  (named-argument spread; echo it in the body), **excluding permanently-stranded rows** via a
+  `BACKLOG_CUTOFF_DATE`, **deploy-inert** (`isActive = 0` + empty-URL guard), calibrating against
+  the **1.0** tier's cadences, the no-deploy **end-to-end + recovery test procedure**, and the
+  **activation-ordering trap** (absence criteria before first push = false incident). See
+  [Prudential OneUptime order-pipeline monitors](../../../clients/prudential/features/oneuptime-order-pipeline-monitors.md).
+  (mhammontree)
 - 2026-08-10 — Noted that worker2 gained its **first OneUptime API *reader*** (this doc covers
   the push-only half; OneUptime was previously push-only, "dumb reporter, smart monitor"). The
   reader polls open incidents and files ClickUp tasks — see

package/knowledge/2.0/standards/backend-php.md

@@ -5,7 +5,7 @@ project: _Underscore
 client: shared
 type: standard
 status: active
-updated: 2026-08-10
+updated: 2026-08-17
 owners: [jcardinal, mhammontree, dfranks]
 files: []
 related:
@@ -269,6 +269,22 @@ WHERE
 
 * If necessary, add comments to clarify complex sections of SQL code. Comments should be concise and informative.
 * Prefer to use the `#` character to begin a comment rather than `--`.
+* **Do not write a prose preamble on a migration file.** Comment each statement with a
+  short description of what it does and why. A multi-paragraph narrative over mechanical
+  `INSERT`s is not documentation — it rots, it repeats the ticket, and it buries the SQL.
+  Rollout/runbook steps belong on the ClickUp ticket; a reusable pattern belongs in the
+  knowledge base; the migration file gets one line per statement.
+* **This is not a licence to strip PHP DocBlocks.** DocBlocks on classes and methods remain
+  required (see DocBlocks below) and are a different artifact. A docblock that records *why*
+  a query is written a certain way — e.g. "`Customers` is `LEFT` joined on purpose so the
+  monitor can see rows the transmission crons cannot" — is load-bearing: it prevents a future
+  "cleanup" to `INNER JOIN` that would silently blind the monitor. Keep those.
+* Do not comment a migration with a dependency warning about an earlier file **in the same
+  folder**. The dbchanges2 executor applies each folder's files in **alphabetical order**, so an
+  earlier `YYYY-MM-DD` file in that folder is guaranteed already applied. A **cross-folder**
+  dependency (e.g. a `Client_<Name>/` file relying on a `Core/` change) is a different matter —
+  those are separate executor passes against separate clusters, so noting the ordering there is
+  legitimate and should be kept.
 
 ```sql
 # Bad
@@ -832,5 +848,11 @@ See: 2.0/apps/worker2/features/cross-account-aws-access.md
 
 ## Change history
 
+- 2026-08-17 — SQL Comments: banned prose preambles on migration files (one short description per
+  statement; runbooks go on the ticket, patterns in the knowledge base), while explicitly protecting
+  PHP DocBlocks that record *why* a query is shaped a certain way. Also scoped the "no
+  earlier-file dependency warning" rule to **within one folder** — `Core/` and `Client_<Name>/` are
+  separate executor passes against separate clusters, so a cross-folder ordering note is legitimate
+  and must be kept. (mhammontree)
 - 2026-08-10 — Added "AWS access (worker2)" section: new worker2 AWS code must use `_Component_Aws_Workloads` (instance-role → per-account `WorkloadsRuntime` STS assume-role); legacy static key grandfathered for existing paths only. (jcardinal)
 - 2026-08-01 — Recorded the Logs-cluster singular table-naming exception to the otherwise-plural table convention. (jcardinal)

package/knowledge/INDEX.md

@@ -19,7 +19,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 ## 2.0 framework
 
 - **_underscore** (_Underscore) _(framework core)_ — 58 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
-- **worker2** (Worker) — 49 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
+- **worker2** (Worker) — 50 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
 - **api2** (API) — 24 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
 - **dbchanges2** (Database Changes) _(framework core)_ — 8 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
 - **toga2-supply** (TOGa Supply) — 6 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)

package/knowledge/clients/aig/INDEX.md

@@ -3,5 +3,5 @@
 | Doc | Framework | Summary | Files |
 |-----|-----------|---------|-------|
 | [AIG Contract Reconciliation & Dealer Programs (STS_001 / SA_001)](features/contract-reconciliation.md) | 2.0 | How to reconcile an **AIG contract sales sheet** against the `Client_Aig` tenant, and the durable finding that the **Staples Advantage.com (`SA_001`) program do | _underscore/Model/Client/Entitlement.php, _underscore/Model/Aig/Entitlement.php, _underscore/Model/Client/Contact.php, _underscore/Model/Aig/Contact.php |
-| [AIG Entitlement Intake & SaleItem Code Resolution](features/entitlement-intake.md) | 2.0 | The `_Model_Aig_Entitlement::prePost` interceptor below is the code path for AIG protection-plan entitlements arriving as `api2` V2 JSON POSTs — and that path * | _underscore/Model/Aig/Entitlement.php, _underscore/Model/Aig/Unit.php, api2/Component/Api/V2/V2.php, api2/Config/sandbox-dev.ini, dbchanges2/Client_Aig/2026-06-18a - TRUE-79534 AIG SaleItem codes.sql, dbchanges2/Client_Aig/2026-08-17a - TRUE-80562 AIG API identifier overrides.sql |
+| [AIG Entitlement Intake & SaleItem Code Resolution](features/entitlement-intake.md) | 2.0 | The `_Model_Aig_Entitlement::prePost` interceptor below is the code path for AIG protection-plan entitlements arriving as `api2` V2 JSON POSTs — and that path * | _underscore/Model/Aig/Entitlement.php, _underscore/Model/Aig/Unit.php, api2/Component/Api/V2/V2.php, api2/Config/sandbox-dev.ini, dbchanges2/Client_Aig/2026-06-18a - TRUE-79534 AIG SaleItem codes.sql, dbchanges2/Client_Aig/2026-08-17a - TRUE-80562 AIG API identifier overrides.sql, worker2/Worker/Monitor/Aig.php, api2/Controller/Index.php |
 | [AIG (Staples Protection Plan)](profile.md) | 2.0 | AIG is the warranty underwriter behind the **Staples Protection Plan** retail program. |  |

package/knowledge/clients/aig/features/entitlement-intake.md

@@ -14,7 +14,10 @@ files:
   - api2/Config/sandbox-dev.ini
   - dbchanges2/Client_Aig/2026-06-18a - TRUE-79534 AIG SaleItem codes.sql
   - dbchanges2/Client_Aig/2026-08-17a - TRUE-80562 AIG API identifier overrides.sql
+  - worker2/Worker/Monitor/Aig.php
+  - api2/Controller/Index.php
 related:
+  - 2.0/apps/worker2/features/oneuptime-worker2-monitoring.md
   - clients/aig/features/contract-reconciliation.md
   - 1.0/apps/library/features/toga2-api-client-and-bridge.md
   - 2.0/apps/api2/features/request-logging.md
@@ -379,6 +382,23 @@ When AIG sends a new "Active SaleItemID" spreadsheet (columns `SaleItemID`, `Des
    local pass is not evidence.** See the gotcha and
    [local vs prod MySQL config parity](../../../2.0/apps/dbchanges2/workflows/local-vs-prod-mysql-config-parity.md).
 
+## Monitoring the intake endpoint — the failure-scraper cannot see a dead endpoint
+
+`Monitor/Aig/EntitlementApiFailures` (worker2 → OneUptime) counts **arrived-and-failed** intake
+calls from the log tables. It is structurally blind to a **fully dead** endpoint: a refused
+connection logs nothing, yields zero failure rows, and reads as healthy. TRUE-80587 therefore added
+`Monitor/Aig/EntitlementApiHealth`, a **liveness probe** alongside it — an **unauthenticated GET**
+(so it can never create a record) where **any structured HTTP response below 500 is a PASS**.
+
+Verified 2026-08-17: `api-writer.togahub.com/v2/health` → **200**; unauthenticated
+`/v2/entitlements` → **401**. The 401 is a *healthy* answer — it proves DNS, TLS, the load balancer,
+routing, and the auth layer are all alive. `api2/Controller/Index.php`
+`HEALTH_CHECK_ROUTES = ['/health','/v2/health']` short-circuits to 200 before any DB or Sentry work,
+so the probe does not depend on `Client_Aig` being reachable.
+
+See the shared pattern:
+[OneUptime push-metric monitors for 2.0 workers](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md).
+
 ## Client variations
 
 This is AIG-specific behavior (the `_Model_Aig_Entitlement` override); other clients do not run
@@ -485,6 +505,10 @@ this interceptor or use this dual-purpose Items pattern.
   This is a general dbchanges2 temp-table-staging rule, not AIG-specific.
 
 ## Change history
+- 2026-08-17 — TRUE-80587: added `Monitor/Aig/EntitlementApiHealth`, an unauthenticated-GET liveness
+  probe on the intake endpoint, because the existing `EntitlementApiFailures` log-scraper cannot see
+  a fully dead endpoint. Documented that <500 (including the unauthenticated **401**) is a PASS.
+  (mhammontree)
 
 - 2026-08-17 (later pass) — **Correction + final verification, supersedes two points in the entry
   below.** (1) The nested-`Units` "cross-tenant field list" open question is **WITHDRAWN** — the six

package/knowledge/clients/aig/profile.md

@@ -6,11 +6,12 @@ apps:
   - api2
   - dbchanges2
   - library
+  - worker2
 project: API
 client: aig
 type: profile
 status: active
-updated: 2026-07-28
+updated: 2026-08-17
 owners: ["mhammontree"]
 files: []
 related:
@@ -36,6 +37,9 @@ go out under the Staples Protection Plan brand and link to `staplesprotection.to
   NetSuite item/SO/invoice traits).
 - **`dbchanges2`** — `Client_Aig/` schema + reference-data migrations (e.g. the SaleItem code
   catalog in `Client_Aig.Items`).
+- **`worker2`** — OneUptime monitors: `Monitor/Aig/EntitlementApiFailures` (log-scraper) plus
+  `Monitor/Aig/EntitlementApiHealth` (liveness probe of the intake endpoint). See
+  features/entitlement-intake.md and the shared worker2 OneUptime monitoring doc.
 - **`library` (1.0)** — the `App_Api_Toga2` bridge syncs 2.0 AIG contacts/entitlements down into the
   legacy 1.0 `TOGA_AIG` DB (`Customers`/`ServiceRequests`), run by a worker cron every ~10 min. Note
   the two AIG databases: **`TOGA_AIG`** = V1 legacy sync target; **`Client_Aig`** = V2 prod tenant. See

package/knowledge/clients/nycdoe/INDEX.md

@@ -3,5 +3,5 @@
 | Doc | Framework | Summary | Files |
 |-----|-----------|---------|-------|
 | [NYCDOE Ticket Hold-Status Sync (ServiceNow ⇄ TOGaDesk)](features/hold-status-sync.md) | 1.0 | DOE ticket **hold** status must round-trip between ServiceNow (SNOW) and TOGaDesk and **stay held** — holds are SLA-bearing in both systems. | worker/crons/sync/nycdoe/send_ticket_updates.php, worker/crons/sync/nycdoe/process_tickets.php, worker/crons/sync/nycdoe/send_request_item_updates.php, library/app/model/togadesk/repairorder.php, library/app/api/nycdoev2.php, togadesk/desk/includes/classes/class.repair.php |
-| [NYCDOE ServiceNow / ASN Integration](features/servicenow-integration.md) | 1.0 | The NYCDOE/ServiceNow integration mirrors DOE's ServiceNow tickets (Incidents + RITMs) into local tables, turns vendor shipment notices into NetSuite Sales Orde | worker/crons/sync/nycdoe/import_asn.php, worker/crons/sync/nycdoe/import_inc.php, worker/crons/sync/nycdoe/legacy_import_asn.php, worker/crons/sync/nycdoe/legacy_process_asn_queue.php, worker/crons/sync/nycdoe/process_tickets.php, worker/crons/sync/nycdoe/1_send_asn_to_netsuite.php, worker/crons/sync/nycdoe/2_send_serials_to_netsuite.php, worker/crons/sync/nycdoe/3_create_installation_ticket.php, worker/crons/sync/nycdoe/test_multi_po_receipt_resolution.php, worker/crons/sync/nycdoe/send_ticket_updates.php, worker/crons/sync/nycdoe/send_request_item_updates.php, worker/crons/sync/nycdoe/send_nycdoe_proof_of_delivery.php, worker/crons/sync/nycdoe/sync_nycdoe_locations.php, worker/crons/sync/nycdoe/receive_edi_purchase_orders.php, worker/crons/sync/nycdoe/send_edi_open_invoices.php, worker/crons/notifications/nycdoe/, worker/schedules/cron.worker.sync.json, worker/schedules/cron.worker.notification.json, library/app/api/nycdoe.php, library/app/api/nycdoev2.php, library/app/nycdoe.php, library/app/asnprocessor/manufacturer.php, library/app/asnprocessor/apple.php, library/app/asnprocessor/lenovo.php, library/app/asnprocessor/lexmark.php, library/app/asnprocessor/acer.php, library/app/edi.php, library/app/netsuite.php, dbchanges/Core/SK/, dbchanges/TOGaDeskSupport/SK/ |
+| [NYCDOE ServiceNow / ASN Integration](features/servicenow-integration.md) | 1.0 | The NYCDOE/ServiceNow integration mirrors DOE's ServiceNow tickets (Incidents + RITMs) into local tables, turns vendor shipment notices into NetSuite Sales Orde | worker/crons/sync/nycdoe/import_asn.php, worker/crons/sync/nycdoe/import_inc.php, worker/crons/sync/nycdoe/legacy_import_asn.php, worker/crons/sync/nycdoe/legacy_process_asn_queue.php, worker/crons/sync/nycdoe/process_tickets.php, worker/crons/sync/nycdoe/1_send_asn_to_netsuite.php, worker/crons/sync/nycdoe/2_send_serials_to_netsuite.php, worker/crons/sync/nycdoe/3_create_installation_ticket.php, worker/crons/sync/nycdoe/test_multi_po_receipt_resolution.php, worker/crons/sync/nycdoe/send_ticket_updates.php, worker/crons/sync/nycdoe/send_request_item_updates.php, worker/crons/sync/nycdoe/send_nycdoe_proof_of_delivery.php, worker/crons/sync/nycdoe/sync_nycdoe_locations.php, worker/crons/sync/nycdoe/receive_edi_purchase_orders.php, worker/crons/sync/nycdoe/send_edi_open_invoices.php, worker/crons/notifications/nycdoe/, worker/schedules/cron.worker.sync.json, worker/schedules/cron.worker.notification.json, library/app/api/nycdoe.php, library/app/api/nycdoev2.php, library/app/nycdoe.php, library/app/asnprocessor/manufacturer.php, library/app/asnprocessor/apple.php, library/app/asnprocessor/lenovo.php, library/app/asnprocessor/lexmark.php, library/app/asnprocessor/acer.php, library/app/edi.php, library/app/netsuite.php, dbchanges/Core/SK/, dbchanges/TOGaDeskSupport/SK/, worker2/Worker/Monitor/Nycdoe.php |
 | [New York City Department of Education](profile.md) | 1.0 | NYC DOE (New York City Department of Education) is a TOGA client whose entire integration runs in the **1.0 worker tier** (~30 cron scripts under `worker/crons/ |  |

package/knowledge/clients/nycdoe/features/servicenow-integration.md

@@ -6,7 +6,7 @@ project: Worker
 client: nycdoe
 type: client-feature
 status: active
-updated: 2026-08-14
+updated: 2026-08-17
 owners: [mhammontree, sking]
 files:
   - worker/crons/sync/nycdoe/import_asn.php
@@ -39,8 +39,10 @@ files:
   - library/app/netsuite.php
   - dbchanges/Core/SK/
   - dbchanges/TOGaDeskSupport/SK/
+  - worker2/Worker/Monitor/Nycdoe.php
 related:
   - ../profile.md
+  - ../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md
   - hold-status-sync.md
   - ../../../1.0/apps/worker/architecture.md
   - ../../../1.0/apps/dbchanges/workflows/authoring-and-shipping-sql-files.md
@@ -551,6 +553,28 @@ use the toga DB MCP + `Logs.API` instead of running prod code locally.
   `__get` throws "No field or quick query defined for key …". Read that column via **raw SQL**,
   not the model.
 
+## Monitoring the ServiceNow endpoint (worker2 liveness probe — TRUE-80587)
+
+The five 1.0 `App_SystemMonitor_ServiceNow*` monitors scrape logs for **arrived-and-failed** calls,
+so they are **structurally blind to a fully dead endpoint**: a refused connection logs nothing,
+produces zero failure rows, and reads as healthy. `Monitor/Nycdoe/ServiceNowEndpointHealth`
+(`worker2/Worker/Monitor/Nycdoe.php`, OneUptime push) is the complement — a **liveness probe** that
+actually touches the endpoint.
+
+- **Unauthenticated GET**, so it can never create a record.
+- **Any structured HTTP response below 500 is a PASS** — it proves DNS, TLS, the load balancer,
+  routing, and the auth layer are alive. A `401` is a *healthy* answer, not a failure.
+- **Accepted coverage gap: token validity is NOT monitored.** The OAuth client
+  `App_Api_NYCDOEV2` lives in the 1.0 `library` repo, which worker2 cannot reach. Re-implementing
+  it in worker2 would duplicate DOE's credentials into a second repo and race the shared access
+  token. An expired/revoked token will surface via the 1.0 failure monitors, not this probe — do
+  not "fix" this by forking the client.
+
+This probe is the client's **only 2.0 footprint**. DOE is **1.0 by default** — `worker2` appears in
+the client's `apps:` scope solely so this monitor loads, and its presence is **not** the start of a
+2.0 migration (there is no `Client_Nycdoe` 2.0 tenant). All business logic stays in the 1.0 tier. See
+[OneUptime push-metric monitors for 2.0 workers](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md).
+
 ## Operating rules when changing this integration
 
 1. **Trace the full pipeline downstream before fixing** — the constraint that makes an edit
@@ -563,6 +587,10 @@ use the toga DB MCP + `Logs.API` instead of running prod code locally.
    of the consumer query; `php -l` every touched file.
 
 ## Change history
+- 2026-08-17 — TRUE-80587: added `Monitor/Nycdoe/ServiceNowEndpointHealth` (worker2 → OneUptime), an
+  unauthenticated-GET liveness probe complementing the log-scraping 1.0 `App_SystemMonitor_ServiceNow*`
+  monitors, which cannot see a dead endpoint. Token validity is a deliberately accepted coverage gap
+  (`App_Api_NYCDOEV2` is 1.0-only). First 2.0 code for this client. (mhammontree)
 - 2026-08-14 — Tied the DOE **missing-asset-tag** class to the existing "Lenovo Off-Layout ASN File"
   bug: the +1 shift puts the serial in `assetTag` and means the real asset-tag column (`$cols[24]`)
   is **never read**, so no unit from an off-layout file can get a tag (still **open** — needs the

package/knowledge/clients/nycdoe/profile.md

@@ -6,11 +6,12 @@ apps:
   - library
   - togadesk
   - dbchanges
+  - worker2
 project: Worker
 client: nycdoe
 type: profile
 status: active
-updated: 2026-07-29
+updated: 2026-08-17
 owners: [mhammontree, sking]
 files: []
 related:
@@ -27,8 +28,13 @@ proof-of-delivery back to ServiceNow. A parallel EDI relationship (850 POs in, i
 runs over DOE's SFTP.
 
 ## Platforms & data
-- **1.0 worker tier only** — no 2.0 footprint. Local mirror tables live in `db_common`
+- **1.0 worker tier** — all business logic. Local mirror tables live in `db_common`
   (`NYCDOETickets`, `NYCDOELocations`, `AdvanceShippingNotice*`).
+- **`worker2` (2.0) — monitoring only.** DOE's *only* 2.0 code is the OneUptime liveness probe
+  `Monitor/Nycdoe/ServiceNowEndpointHealth` (`worker2/Worker/Monitor/Nycdoe.php`, TRUE-80587).
+  This client remains **1.0 by default** — `worker2` is in `apps:` solely so that monitor loads.
+  It is **not** the start of a 2.0 migration and there is no `Client_Nycdoe` 2.0 tenant; do not
+  read it as a platform move.
 - **TOGa Desk:** DOE is client id **16** (`db_togadesk`) — repair orders, managed service
   orders, and the "#received / #ordered" Receiving Summary UI.
 - **NetSuite:** Sales Orders, PO reconciliation, item receipts, Item Fulfillments, invoices

package/knowledge/clients/prudential/INDEX.md

@@ -5,6 +5,7 @@
 | [Prudential: Dell ASN units PRE/POST interceptor (legacy key + flat tracking)](features/dell-asn-units-interceptor.md) | 2.0 | After the tracking-number bridge migration, the ASN unit route was renamed (`advance-shipping-notice-units` → `advance-shipping-notice-item-units`), so the inhe | _underscore/Model/Prudential/AdvanceShippingNotice.php, dbchanges2/Client_Prudential/2026-06-10 - AsnUnitsInterceptor.sql |
 | [Prudential: Dell LCH IOP transmissions (LCHRequestV2)](features/dell-lch-iop-transmissions.md) | 1.0 | Outbound order transmissions from the 1.0 worker tier to Dell's Lifecycle Hub (LCH) ITSM integration. | library/app/api/delllch.php, library/app/apitransaction.php, worker/crons/toga2/prudential/transmissions_to_dell_usa.php, worker/crons/toga2/prudential/transmissions_to_dell_ireland.php, worker/crons/toga2/prudential/transmissions_to_dell_india.php, worker/crons/toga2/prudential/generate_sales_and_purchase_orders_from_service_requests.php |
 | [Prudential: Device information import + unit→contact linking (import_device_information.php)](features/device-information-import-and-contact-linking.md) | 1.0 | Prudential's **device-sync** cron (`worker/crons/toga2/prudential/import_device_information.php`) pulls device/asset records (from ServiceNow / the Dell CMDB fe | worker/crons/toga2/prudential/import_device_information.php, worker/crons/toga2/prudential/backfill_unit_contacts.php, dbchanges2/Client_Prudential/2026-07-07 - Contact Dedup Merge.sql |
+| [Prudential: OneUptime order-pipeline backlog monitors (Monitor/Prudential/*)](features/oneuptime-order-pipeline-monitors.md) | 2.0 | Three worker2 OneUptime push monitors (TRUE-80587) that watch the Prudential/Dell order pipeline for **stalls**, which were previously invisible — a stuck pipel | worker2/Worker/Monitor/Prudential.php, worker2/Config/production.ini, dbchanges2/Core/2026-08-07a - Prudential AIG ServiceNow OneUptime monitor crons.sql, dbchanges2/Client_Prudential/2026-04-08 - Insert Customers.sql |
 | [Prudential: Service Request Regional Address Validation](features/service-request-address-validation.md) | 2.0 | The `prePost` interceptor on `_Model_Prudential_ServiceRequest` validates `deliverToAddress` fields differently depending on which Prudential regional customer | _underscore/Model/Prudential/ServiceRequest.php, _underscore/Test/Prudential/ServiceRequestTest.php |
 | [Prudential: Service Request rejection alert email](features/service-request-rejection-alert-email.md) | 2.0 | When a Prudential ServiceNow→TOGa service-request submission (`POST /v2/service-requests`) is **rejected by validation**, TOGa now sends a real-time internal al | worker2/Worker/Client/Prudential/reports/ReqRejectionEmail.php, _underscore/Model/Prudential/ServiceRequest.php |
 | [Prudential Order Shipped Email — transmit_ordershipped_updates_prudential.php](features/transmit-ordershipped-email.md) | 1.0 | Cron script that transmits "Order Shipped" updates to ServiceNow (RITM) and sends a shipped notification email to the end user. | worker/crons/toga2/prudential/transmit_ordershipped_updates_prudential.php, worker/crons/toga2/prudential/transmit_closecomplete_updates_prudential.php, worker/crons/toga2/prudential/transmit_rejected_cancelled_updates_prudential.php, worker/crons/toga2/prudential/generate_sales_and_purchase_orders_from_service_requests.php, worker/crons/toga2/prudential_beta/generate_sales_and_purchase_orders_from_service_requests.php, worker/crons/notifications/reports/prudential_exception_report.php |

package/knowledge/clients/prudential/features/oneuptime-order-pipeline-monitors.md

@@ -0,0 +1,108 @@
+---
+title: "Prudential: OneUptime order-pipeline backlog monitors (Monitor/Prudential/*)"
+framework: "2.0"
+repo: worker2
+project: Worker
+client: prudential
+type: client-feature
+status: active
+updated: 2026-08-17
+owners: ["mhammontree"]
+files:
+  - worker2/Worker/Monitor/Prudential.php
+  - worker2/Config/production.ini
+  - dbchanges2/Core/2026-08-07a - Prudential AIG ServiceNow OneUptime monitor crons.sql
+  - dbchanges2/Client_Prudential/2026-04-08 - Insert Customers.sql
+related:
+  - ../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md
+  - ../profile.md
+  - ./transmit-ordershipped-email.md
+  - ./dell-lch-iop-transmissions.md
+  - ../../../2.0/apps/worker2/features/service-request-sales-order-generation.md
+---
+
+## Summary
+
+Three worker2 OneUptime push monitors (TRUE-80587) that watch the Prudential/Dell order pipeline
+for **stalls**, which were previously invisible — a stuck pipeline produced no error, no email,
+and no alert. They implement the shared pattern in
+[OneUptime push-metric monitors for 2.0 workers](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md);
+read that first for the payload contract, criteria set, and provisioning runbook.
+
+- `Monitor/Prudential/SalesOrderGenerationBacklog` — Service Requests not yet turned into Sales Orders.
+- `Monitor/Prudential/DellPurchaseOrderTransmissionBacklog` — POs not yet transmitted to Dell.
+- `Monitor/Prudential/OrderShippedUpdateBacklog` — order-shipped updates not yet transmitted.
+
+## Key files / entry points
+
+- `worker2/Worker/Monitor/Prudential.php` — `_Worker_Monitor_Prudential`; one `public static`
+  action method per monitor, each self-contained (in-method `ALL_CAPS` locals + `$push` closure).
+- `dbchanges2/Core/2026-08-07a - Prudential AIG ServiceNow OneUptime monitor crons.sql` — the
+  `Core.CronJobs` rows (shipped `isActive = 0`, `parameters = '{"alertThreshold":N}'`).
+
+## How it works
+
+### The monitors read a 2.0 DB but watch 1.0 crons
+
+Prudential's order-pipeline **business logic runs in the 1.0 worker tier** against the 2.0
+`Client_Prudential` database. The monitors live in worker2 and only **READ** that database — the
+same split as the Compass monitors. Thresholds are therefore calibrated against the **1.0 cron
+cadences**, not worker2's own `*/5` schedule:
+
+| 1.0 cron | Cadence | Throughput |
+|---|---|---|
+| `generate_sales_and_purchase_orders` | every 3 min | 1 request per run (≈20/hour) |
+| `transmissions_to_dell_{usa,ireland,india}` | every 5 min | one region each |
+| `transmit_ordershipped_updates_prudential` | every 2 min | — |
+
+### `BACKLOG_CUTOFF_DATE` — excluding permanently-stranded rows
+
+`BACKLOG_CUTOFF_DATE = 2026-08-07`. Rows before it are excluded because they will **never** drain:
+
+- ~38 Service Requests orphaned by the **2026-07-17 Dell regional-split cutover**.
+- Two-phase order-shipped stranding — `transmit_ordershipped_updates_prudential` stamps the
+  transmitted column in Phase 1, so a Phase-2 crash strands that PO's email forever (see the
+  [profile gotchas](../profile.md) and [transmit-ordershipped-email](./transmit-ordershipped-email.md)).
+
+Counting them would pin the monitor HIGH on its very first run, which trains people to ignore it.
+
+### Diagnostic fields that deliberately do NOT drive the alarm
+
+`DellPurchaseOrderTransmissionBacklog` also reports:
+
+- `strandedCount` — `ServiceRequests.customerId IS NULL`.
+- `unmappedCustomerCount` — customer is not one of the three regional customer uuids.
+
+Both are informational only. The three regional transmission crons **INNER JOIN `Customers` and
+filter to a single region uuid each**, so these rows are invisible to *every* transmitter — they
+are a data-integrity problem, not a transmission backlog, and alarming on them would misroute the
+response.
+
+**The monitor `LEFT JOIN`s `Customers` on purpose** so it can see rows the crons cannot. Changing
+it to an `INNER JOIN` would silently blind the monitor — the DocBlock on the method says so, and
+that docblock is load-bearing; do not "tidy" it away.
+
+## Reference data (verified 2026-08-17)
+
+- The USA / Ireland / India regional customer uuids are seeded in
+  `dbchanges2/Client_Prudential/2026-04-08 - Insert Customers.sql` — **not** `2024-01-01.sql`, which
+  an in-code comment incorrectly claims.
+- `Client_Prudential.Vendors.id = 1` is Dell.
+
+## Gotchas / known issues
+
+- Thresholds are DB-tunable (`Core.CronJobs.parameters` → `{"alertThreshold":N}`) — retune there,
+  never by editing PHP. The JSON key must match the method's parameter name exactly.
+- The crons ship `isActive = 0`; the monitors do nothing until activated, and each guards on an
+  empty push URL. Do not add OneUptime absence criteria until a push has landed.
+
+## Change history
+- 2026-08-17 — TRUE-80587: built the three backlog monitors; set `BACKLOG_CUTOFF_DATE` to exclude
+  the ~38 Dell regional-split orphans and the two-phase order-shipped strandings; made
+  `strandedCount` / `unmappedCustomerCount` non-alarming diagnostics behind a deliberate
+  `LEFT JOIN Customers`; calibrated thresholds against the 1.0 cron cadences; corrected the
+  regional-customer-uuid seed-file reference to `2026-04-08 - Insert Customers.sql`. (mhammontree)
+
+## Related docs
+- [OneUptime push-metric monitors for 2.0 workers](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md) — the shared pattern, payload contract, and runbook.
+- [Prudential Financial profile](../profile.md)

package/knowledge/clients/prudential/profile.md

@@ -15,9 +15,10 @@ client: prudential
 type: profile
 status: active
 updated: 2026-08-17
-owners: ["jcardinal", "rgirish", "bala"]
+owners: ["jcardinal", "rgirish", "bala", "mhammontree"]
 files: []
 related:
+  - features/oneuptime-order-pipeline-monitors.md
   - features/dell-asn-units-interceptor.md
   - features/service-request-address-validation.md
   - features/service-request-rejection-alert-email.md
@@ -81,7 +82,14 @@ order-status transmissions.
   (E_ALL→ErrorException), so `!== false` guards do not work — use the `safeFetchPdf()` pattern. See
   the exception report and the transmit-ordershipped-email feature doc.
 
+## Monitoring
+- `Monitor/Prudential/{SalesOrderGenerationBacklog,DellPurchaseOrderTransmissionBacklog,OrderShippedUpdateBacklog}`
+  (worker2 → OneUptime) watch the order pipeline for stalls. Thresholds are DB-tunable and the
+  monitors deliberately ignore pre-2026-08-07 permanently-stranded rows. See
+  features/oneuptime-order-pipeline-monitors.md.
+
 ## Related docs
+- OneUptime order-pipeline backlog monitors (worker2).
 - Dell ASN units interceptor.
 - Dell LCH IOP transmissions (LCHRequestV2, outbound 1.0).
 - 2.0 _underscore: Tracking-Number Bridge Migration.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.596",
+  "version": "1.0.598",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",