toga-ai 1.0.866 → 1.0.867
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -6,7 +6,7 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
9
|
+
updated: 2026-09-22
|
|
10
10
|
owners: [jcardinal]
|
|
11
11
|
files:
|
|
12
12
|
- worker2/Worker/Monitor/Clickup.php
|
|
@@ -25,14 +25,19 @@ worker2 cron that checks the team's ClickUp webhook subscriptions and pages via
|
|
|
25
25
|
|
|
26
26
|
Catches the **silent webhook outage** EB health checks cannot see: if ClickUp suspends or fails our inbound webhook (`https://webhook.togahub.com/clickup`), inbound tasks stop arriving and nothing else notices.
|
|
27
27
|
|
|
28
|
-
Distinct from
|
|
28
|
+
Distinct from the sibling email watchdog [`_Worker_Clickup_Health`](./clickup-connectivity-watchdog.md): the watchdog emails on `status = "failing"` in business hours; this monitor reports to OneUptime around the clock and only escalates on `"suspended"`, a missing subscription, or a blind read (see the alarm rule below). A plain `"failing"` status is **not** alarmed here — that softer signal stays with the email watchdog.
|
|
29
29
|
|
|
30
30
|
## How it works
|
|
31
31
|
|
|
32
32
|
- Reads subscriptions: `_Component_Api_Clickup::send('GET','/team')` → for each team `_Component_Api_Clickup::send('GET','/team/{id}/webhook')`.
|
|
33
|
-
- POSTs a metric body via `_ApiRequest` to a OneUptime incoming-request heartbeat URL: `alarm` (`OK`/`DEGRADED`), `probe` (`OK`/`DEGRADED`), plus `webhookCount`, `
|
|
34
|
-
- **Alarm rule
|
|
35
|
-
-
|
|
33
|
+
- POSTs a metric body via `_ApiRequest` to a OneUptime incoming-request heartbeat URL: `alarm` (`OK`/`DEGRADED`/`HIGH`), `probe` (`OK`/`DEGRADED`), plus `webhookCount`, `okCount`, `suspendedCount`, `blindCount`, `failureCount`, `failures`, `checkedAtUtc`.
|
|
34
|
+
- **Alarm rule (three levels, revised 2026-09-22 — quieter than the original "not active = bad"):**
|
|
35
|
+
- `HIGH` (full failure / Offline) — **blind read**: the ClickUp call threw, OR a monitored `togahub.com` webhook has no readable `health.status`. "We don't know the status."
|
|
36
|
+
- `DEGRADED` — a monitored `togahub.com` webhook has `health.status = "suspended"`, OR no `togahub.com` subscription exists at all. A known outage we can see.
|
|
37
|
+
- `OK` — every monitored webhook is readable and not suspended (so `active`, `failing`, any other readable status → OK).
|
|
38
|
+
- Precedence: `HIGH` > `DEGRADED` > `OK`. `probe` is `OK` on any successful read (even when `alarm=HIGH` from a blind webhook) and `DEGRADED` only when the ClickUp read itself failed.
|
|
39
|
+
- **OneUptime criteria must match the tokens** (config lives outside the repo): Request Body Contains `"alarm":"HIGH"` → Offline + incident; Contains `"alarm":"DEGRADED"` → Degraded; received in 1 min AND Contains `"alarm":"OK"` → Online (OK-gated recovery).
|
|
40
|
+
- **Push URL is an in-file `$ONEUPTIME_URL` local, not `[oneuptime]` config** — the established team convention for every `Worker/Monitor/*` file (Operations, Staples, Compass, Aig, Prudential, Nycdoe, Fleet). A cso review suggested reading it from config; the developer rejected that on purpose — the heartbeat URL is treated as low-sensitivity and stays inline, matching the siblings. See the credential note in [OneUptime push-metric monitors](./oneuptime-worker2-monitoring.md). The `$push` guard still soft-fails on a blank URL so the job can deploy inert.
|
|
36
41
|
- Cron row: `dbchanges2/Core/2026-09-21c - Insert ClickUp Webhook Health Monitor CronJob.sql`. OneUptime import artifact `clickup-webhook-health.json` lives in a local staging dir, not the repo (import artifacts are not version-controlled — see the pattern doc's provisioning runbook).
|
|
37
42
|
|
|
38
43
|
## Gotchas
|
|
@@ -42,6 +47,7 @@ Distinct from and **stricter than** the sibling email watchdog [`_Worker_Clickup
|
|
|
42
47
|
- Keep the metric POST non-fatal — a monitoring outage must never break the worker job.
|
|
43
48
|
|
|
44
49
|
## Change history
|
|
50
|
+
- 2026-09-22 — Reworked the alarm rule from "any status ≠ active → DEGRADED" to three levels: `HIGH` on a blind read (ClickUp call failed, or a webhook has no readable `health.status`), `DEGRADED` on `"suspended"` or no `togahub.com` subscription, `OK` for everything else (`active`/`failing`/other). Original was too sensitive. Also hardened: blind-path OneUptime push wrapped in try/catch (non-fatal), `health` guarded against a non-object, stale "Left BLANK" URL comment removed (URL is live). Payload fields changed: `activeCount` → `okCount`, added `suspendedCount`/`blindCount`. OneUptime monitor criteria must be set to match `"alarm":"HIGH"` and `"alarm":"DEGRADED"`. (jcardinal)
|
|
45
51
|
- 2026-09-21 — Created `_Worker_Monitor_Clickup::CheckWebhook` (OneUptime, `*/10`, stricter than the email watchdog). Recreated the inbound webhook under the service account (`pk_87469444`, new id `bd6168ab-…`, deleting the old OAuth-owned one) so both this monitor and `_Worker_Clickup_Health` can see it — the two were previously blind to it (token-ownership gotcha above). (jcardinal)
|
|
46
52
|
|
|
47
53
|
## Related
|
package/package.json
CHANGED