toga-ai 1.0.894 → 1.0.896
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/1.0/apps/library/INDEX.md +1 -1
- package/knowledge/1.0/apps/library/features/http-500-error-monitor.md +23 -34
- package/knowledge/1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md +3 -1
- package/knowledge/1.0/apps/worker/workflows/tracing-a-worker-cron-run-in-production.md +1 -1
- package/knowledge/2.0/apps/_underscore/features/error-reporting-issue-event.md +2 -3
- package/knowledge/2.0/apps/toga25-supply/features/persisted-query-cache.md +10 -5
- package/knowledge/2.0/apps/worker2/INDEX.md +3 -3
- package/knowledge/2.0/apps/worker2/features/clickup-design-sprint-automation.md +12 -9
- package/knowledge/2.0/apps/worker2/features/clickup-work-type-automation.md +12 -2
- package/knowledge/2.0/apps/worker2/features/oneuptime-incident-clickup-sync.md +21 -21
- package/knowledge/2.0/apps/worker2/features/qa-qc-review-pipeline.md +18 -13
- package/knowledge/2.0/apps/worker2/features/team-sprint-management.md +7 -3
- package/knowledge/2.0/apps/worker2/workflows/clickup-credential-rotation.md +13 -7
- package/package.json +1 -1
|
@@ -14,7 +14,7 @@
|
|
|
14
14
|
| [Branded HTML Email Templates (App_Email_Template)](features/email-templates.md) | Branded HTML email base class `App_Email_Template` (header/body/footer); open when building or fixing a 1.0 branded email or its Outlook rendering. |
|
|
15
15
|
| [App_Email Side Effects & Test Mode (how to send a real email that writes nothing)](features/email-test-mode-and-write-free-sends.md) | 1.0 `App_Email::send()` side effects (log/queue writes, test-mode recipient rewrite) and how to send a genuinely write-free test; open before writing any 1.0 dr |
|
|
16
16
|
| [Error Capture in 1.0 (App_Error_Capture → shared 2.0 Logs DB)](features/error-capture-1-0.md) | 1.0 error-reporting into the shared 2.0 Logs DB (`App_Error_Capture::captureException`), mirroring 2.0's Issue/Event; open when adding a 1.0 capture path or deb |
|
|
17
|
-
| [HTTP 500 Error Monitor (App_SystemMonitor_500Error) —
|
|
17
|
+
| [HTTP 500 Error Monitor (App_SystemMonitor_500Error) — email-only count; diagnose via error.id](features/http-500-error-monitor.md) | 1.0 monitor `App_SystemMonitor_500Error`: counts `Logs.Api` responseCode=500 in the last 5 min and emails devteam; no ClickUp. |
|
|
18
18
|
| [1.0 MVC Page Pattern & New-App Skeleton](features/mvc-page-pattern-and-app-skeleton.md) | Recipe for a new 1.0 (`App_`) app and for adding pages: folder-based MVC routing, page lifecycle, models, multi-DB config; open when standing up a 1.0 app or ad |
|
|
19
19
|
| [NetSuite File Cabinet Content Retrieval via RESTlet (fetchInvoiceFile)](features/netsuite-filecabinet-restlet.md) | How 1.0 pulls NetSuite File Cabinet binary content (invoice PDFs) via a SuiteScript RESTlet (`fetchInvoiceFile`); open when fetching NetSuite file bytes over RE |
|
|
20
20
|
| [assetType from NetSuite itemtype during Item Sync (opt-in per client)](features/netsuite-item-assettype-sync.md) | Stamp `Items.assetTypeId` from NetSuite `item.itemtype` during item sync (opt-in per client, fill-NULL-only); open before enabling asset-type mapping for a clie |
|
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
---
|
|
2
|
-
title: "HTTP 500 Error Monitor (App_SystemMonitor_500Error) —
|
|
2
|
+
title: "HTTP 500 Error Monitor (App_SystemMonitor_500Error) — email-only count; diagnose via error.id"
|
|
3
3
|
framework: "1.0"
|
|
4
4
|
repo: library
|
|
5
5
|
project: Library
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
10
|
-
owners: ["bala"]
|
|
9
|
+
updated: 2026-09-25
|
|
10
|
+
owners: ["bala", "jcardinal"]
|
|
11
11
|
files:
|
|
12
12
|
- library/app/systemmonitor/500error.php
|
|
13
13
|
- worker/crons/infrastructure/system_monitors.php
|
|
@@ -19,67 +19,56 @@ related:
|
|
|
19
19
|
- ./cron-execution-monitoring.md
|
|
20
20
|
- ../../../../2.0/apps/_underscore/features/error-reporting-issue-event.md
|
|
21
21
|
- ../../../../2.0/apps/api2/features/v2-api-error-codes.md
|
|
22
|
+
- ../../../../2.0/apps/worker2/features/error-escalation-cron.md
|
|
22
23
|
---
|
|
23
24
|
|
|
24
|
-
1.0 monitor `App_SystemMonitor_500Error
|
|
25
|
+
1.0 monitor `App_SystemMonitor_500Error`: counts `Logs.Api` responseCode=500 in the last 5 min and emails devteam; no ClickUp. Open when a "HTTP 500 Error Alert" email arrives — diagnose via `error.id`.
|
|
25
26
|
|
|
26
27
|
## Summary
|
|
27
28
|
|
|
28
|
-
`App_SystemMonitor_500Error` (`library/app/systemmonitor/500error.php`, title **"HTTP 500 Error Alert"**)
|
|
29
|
+
`App_SystemMonitor_500Error` (`library/app/systemmonitor/500error.php`, title **"HTTP 500 Error Alert"**) counts **`Logs.Api` rows with `responseCode = 500`** in the last 5 minutes. Any row → returns not-OK → the `App_SystemMonitor` base class emails `devteam@togatech.com`. It carries no per-error detail.
|
|
29
30
|
|
|
30
|
-
|
|
31
|
+
**It creates NO ClickUp tasks (removed 2026-09-25, jcardinal decision).** 500s must not go to ClickUp from here; the 2.0 error pipeline (`Logs.Issue`/`Logs.Event` + [escalation cron](../../../../2.0/apps/worker2/features/error-escalation-cron.md)) owns error tracking. Do not re-add ClickUp task creation.
|
|
31
32
|
|
|
32
33
|
## How it works
|
|
33
34
|
|
|
34
|
-
- **Schedule.** `minuteFrequency = 5`, `minHourToRun = 8`, `maxHourToRun = 17`, `alertRemindFrequency = 240`. Registered in the shared dispatcher `worker/crons/infrastructure/system_monitors.php`
|
|
35
|
-
- **Query.** `getStatus()`
|
|
36
|
-
- **Non-production rows are filtered by a referer heuristic** — a `referer:` header containing `beta`/`gamma`, or lacking `https://` or `.com`, marks the row non-prod and skips it.
|
|
37
|
-
- **Grouping.** `groupErrors()` keys each group on `referer . ' HTTP 500 Error: ' . $errorMessage` and creates/updates one ClickUp task per key in list `901110669877` (or the current dev-sprint list when it resolves one). Two ClickUp custom fields are set from substring matches on the request headers: a **stakeholder** map (`compass`, `prudential`, `quad`, `spglobal`, `nychh`, `growrk`) and a **platform** map (`togasupply`, `togacommerce`, `togadesk`), each falling back to a default uuid.
|
|
35
|
+
- **Schedule.** `minuteFrequency = 5`, `minHourToRun = 8`, `maxHourToRun = 17`, `alertRemindFrequency = 240`. Registered in the shared dispatcher `worker/crons/infrastructure/system_monitors.php` (same 300s shared budget — see [NetSuite Sync Alert Monitor](./netsuite-sync-alert-monitor.md)).
|
|
36
|
+
- **Query.** `getStatus()` runs `SELECT id FROM Logs.Api WHERE responseCode = 500 AND dtStamp >= NOW() - INTERVAL 5 MINUTE` on `db_toga2_logs`. Zero rows = OK; else `[false, "<N> 500 Errors were found…"]`. No cursor, no grouping, no non-prod filter.
|
|
38
37
|
|
|
39
|
-
###
|
|
38
|
+
### Diagnosing an alert: `error.id` (verified lookup recipe)
|
|
40
39
|
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
**api2 leaves those identifiers empty on every production 500 by design.** `api2/Controller/Index.php` (~L271-285) populates `message` and `trace` **only** when `_Environment::isDebugMode()` is true, because `getTrace()` returns frame **arguments** verbatim and any stack with `_Database::register()` on it would hand the database username and password to anyone who could force a 500. See the api2 [error-codes doc](../../../../2.0/apps/api2/features/v2-api-error-codes.md). So the fallback is not a failure — it is the **normal** state in production. Its only meaning is "this 500 came from production api2."
|
|
44
|
-
|
|
45
|
-
### The real diagnostic: `error.id` (verified lookup recipe)
|
|
46
|
-
|
|
47
|
-
The api2 envelope carries `error: { code, id }`. **`id` is the quotable Core Logs reference**, in the form `<Logs.Issue.reference>-<Logs.Event.eventNumber>` (e.g. `78-859`), produced by `_Error::captureException()` (`_underscore/Error.php:334+`), which resolves/creates a `Logs.Issue` by fingerprint, writes a `Logs.Event`, and returns the reference. Three queries get you from an alert to the exception:
|
|
40
|
+
api2 leaves `message`/`trace` empty on production 500s by design (`api2/Controller/Index.php` ~L271-285 fills them only in debug mode, to keep DB credentials out of stack args — see [error codes](../../../../2.0/apps/api2/features/v2-api-error-codes.md)). The envelope carries `error: { code, id }`; **`id` = `<Logs.Issue.reference>-<Logs.Event.eventNumber>`** (e.g. `78-859`), from `_Error::captureException()` (`_underscore/Error.php:334+`).
|
|
48
41
|
|
|
49
42
|
```sql
|
|
50
|
-
# 1.
|
|
43
|
+
# 1. Recent 500s and their envelopes (read error.id from responsePayload)
|
|
51
44
|
SELECT id, route, method, dtStamp, responsePayload
|
|
52
45
|
FROM Logs.Api
|
|
53
|
-
WHERE
|
|
46
|
+
WHERE responseCode = 500 AND dtStamp >= NOW() - INTERVAL 1 HOUR;
|
|
54
47
|
|
|
55
|
-
# 2. The exception
|
|
48
|
+
# 2. The exception + full framework stack (reference = part before the dash)
|
|
56
49
|
SELECT id, reference, status, errorMessage, trace, totalOccurrences, dtLastOccurred
|
|
57
50
|
FROM Logs.Issue
|
|
58
51
|
WHERE reference = '78';
|
|
59
52
|
|
|
60
|
-
# 3. The
|
|
53
|
+
# 3. The occurrence — and WHICH TENANT
|
|
61
54
|
SELECT clientId, dtCreated, errorMessage
|
|
62
55
|
FROM Logs.Event
|
|
63
56
|
WHERE issueId = <issue.id> AND eventNumber = <n>;
|
|
64
57
|
```
|
|
65
58
|
|
|
66
|
-
- `Logs.
|
|
67
|
-
-
|
|
68
|
-
- **api2's own catch block is the ONLY capture point for the inbound-API 500 surface.** api2 handles `Throwable` in its front controller, so `_Error::exceptionHandler` never fires for API traffic. If api2's catch did not capture it, nothing did.
|
|
59
|
+
- `Logs.Event.clientId` is the authoritative tenant.
|
|
60
|
+
- **api2's own catch block is the ONLY capture point for inbound-API 500s** (`_Error::exceptionHandler` never fires for API traffic). If api2's catch did not capture it, nothing did.
|
|
69
61
|
|
|
70
|
-
Full
|
|
62
|
+
Full Issue/Event mechanics: [error reporting](../../../../2.0/apps/_underscore/features/error-reporting-issue-event.md).
|
|
71
63
|
|
|
72
64
|
## Gotchas
|
|
73
65
|
|
|
74
|
-
-
|
|
75
|
-
- **
|
|
76
|
-
- **
|
|
77
|
-
-
|
|
78
|
-
- **`Logs.Issue.status = RESOLVED` does not mean fixed** — it means *stopped firing* (the worker2 frequency-decay cron auto-resolves on consecutive quiet windows). A live defect on a low-traffic tenant reads as RESOLVED the moment that tenant goes quiet. See [error-issue auto-resolution](../../../../2.0/apps/worker2/features/error-issue-auto-resolution.md).
|
|
79
|
-
- **Hardcoded ClickUp API token in source.** `CLICK_UP_API_ACCESS_TOKEN` is a literal class constant in this file at line 14 (pre-existing, same pattern as the NetSuite Sync Alert monitor). It belongs in config/secrets; treat the in-repo value as compromised. (Location only — the value is never copied into the KB.)
|
|
80
|
-
- **Stakeholder/platform attribution is a substring guess.** `str_contains($headers, 'growrk')` style matching mis-attributes any request whose headers happen to mention another tenant, and silently defaults when nothing matches. Use `Logs.Event.clientId` for the real tenant.
|
|
66
|
+
- **The count is not user impact.** apiproxy retries a 5xx across regions, so one user action makes several `Logs.Api` 500 rows within seconds, on different `instanceId`s, **sharing one `transactionId`**. Check the transaction id first.
|
|
67
|
+
- **The 5-minute window is a sample, not the incident.** Use `Logs.Issue.dtLastOccurred` + `totalOccurrences` for the real timeline.
|
|
68
|
+
- **No non-prod filter any more.** The old referer heuristic went with the ClickUp code; every 500 row in the window counts.
|
|
69
|
+
- **`Logs.Issue.status = RESOLVED` means stopped firing, not fixed** — see [error-issue auto-resolution](../../../../2.0/apps/worker2/features/error-issue-auto-resolution.md).
|
|
81
70
|
|
|
82
71
|
## Related
|
|
83
72
|
|
|
84
73
|
- [Error capture in 1.0](./error-capture-1-0.md), [NetSuite sync alert monitor](./netsuite-sync-alert-monitor.md), [Cron execution monitoring](./cron-execution-monitoring.md).
|
|
85
|
-
- [2.0 error reporting Issue/Event](../../../../2.0/apps/_underscore/features/error-reporting-issue-event.md), [v2 API error codes](../../../../2.0/apps/api2/features/v2-api-error-codes.md).
|
|
74
|
+
- [2.0 error reporting Issue/Event](../../../../2.0/apps/_underscore/features/error-reporting-issue-event.md), [v2 API error codes](../../../../2.0/apps/api2/features/v2-api-error-codes.md), [error escalation cron](../../../../2.0/apps/worker2/features/error-escalation-cron.md).
|
|
@@ -6,7 +6,7 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
9
|
+
updated: 2026-09-25
|
|
10
10
|
owners: ["dfranks", "bala", "jcardinal", "mhammontree", "snaredla", "rgirish"]
|
|
11
11
|
files:
|
|
12
12
|
- worker/crons/toga2/netsuite/common_sync_togasupply.php
|
|
@@ -1208,6 +1208,8 @@ library (or vice versa) crashes GroWrk and Adyen on their next sync run.
|
|
|
1208
1208
|
|
|
1209
1209
|
## Gotchas / known issues
|
|
1210
1210
|
|
|
1211
|
+
- **The 2026-09-17 transient-read retry covers NetSuite reads only** (`App_Api_Netsuite_Rest::send`), **not** api2 calls via `App_Api_Toga2::send`. An api2 blip (e.g. "cURL error with API transaction: Empty response body from API (HTTP 200)") still aborts the section for that run and drops the window to `288000-IDLE`; it self-heals next run. One such failure is not a collapse — check the window grows back before acting. Seen NYCHH ITEM_FULFILLMENTS 2026-09-25, issue 873 (`common_sync_togasupply.php:1287` → `toga2.php:6630`).
|
|
1212
|
+
|
|
1211
1213
|
- **⚠ Sales-order downstream LINE links were deleted on every PO re-sync (fixed 2026-09-24).** In
|
|
1212
1214
|
`syncPurchaseOrderFromNetsuite` `$incomingSalesOrderItemUuid` was initialized null and never
|
|
1213
1215
|
assigned, so the existing-line branch deleted every `SalesOrderItems_PurchaseOrderItems` link and
|
|
@@ -116,7 +116,7 @@ Detail on what else `send()` writes (the `Common.EmailsQueued` branch, `setIsLog
|
|
|
116
116
|
```
|
|
117
117
|
The script then does nothing on every tick — and per Steps 2 and 4 you see no output and no `CronJobExecutions` row, i.e. it looks exactly like "the cron never fired". Typed parameters and return types (`: void`, `: array`, `: bool`, `?array`, `?object`, `object $x`) are PHP 7.0/7.1 and **are** safe here; already used in `worker/crons/toga2/compass/backfill_items_isfulfillable.php` and `update_item_fulfillments_in_netsuite.php`.
|
|
118
118
|
- `1.0/standards/backend-php.md` already forbids named arguments in 1.0 — but a workspace `CLAUDE.md` / `.github/copilot-instructions.md` that says *"use PHP 8 Named Parameters"* applies to **2.0 only**. The 1.0 standard wins in `library/`, `worker/` and every other 1.0 app.
|
|
119
|
-
- **Do not read a PHP 8 function call in existing code as proof the tier is PHP 8.** `library/app/systemmonitor/500error.php`
|
|
119
|
+
- **Do not read a PHP 8 function call in existing code as proof the tier is PHP 8.** A 1.0 file using `str_contains()` (as `library/app/systemmonitor/500error.php` did until 2026-09-25) is not a version signal; it is a latent bug.
|
|
120
120
|
- **Deploy-time trap: a `&` (or any shell metacharacter) in the cron FILENAME silently kills it.** `ebs/cron.worker.php:41` builds each crontab line by plain string concatenation (`'php /var/www/html/crons/' . $cron->cron`) with **no quoting and no escaping**. Cron parses that line as a shell command, so a `&` in the filename is read as *"run everything before this in the background and start a new command."*
|
|
121
121
|
- Worked example (found 2026-09-03): a developer renamed the Compass Canada status cron to `crons/toga2/compasscanada/update_salesorder_status_from_g&t.php` and repointed `schedules/cron.worker.sync.json` at it. The crontab line becomes two commands — `php .../update_salesorder_status_from_g` (a file that does not exist) and a stray `t.php` — so the real script **never executes**, on every tick, with no error anywhere. Looks exactly like "the cron never fired".
|
|
122
122
|
- **Rule: a cron filename must be plain `[a-z0-9_]` plus `.php`.** No `&`, no spaces, no `;`, `|`, `$`, `(`, `)`, `*`, `?`, or quotes. The *display name* in `schedules/*.json` is free text and may contain anything; only the `cron` path is shell-parsed.
|
|
@@ -506,9 +506,8 @@ WHERE issueId = <issue.id> AND eventNumber = <n>;
|
|
|
506
506
|
[api2 request logging](../../api2/features/request-logging.md). Read occurrence counts through
|
|
507
507
|
it or you will overstate user impact.
|
|
508
508
|
- The **1.0 alert surface** that fires on these is
|
|
509
|
-
[App_SystemMonitor_500Error](../../../../1.0/apps/library/features/http-500-error-monitor.md) —
|
|
510
|
-
|
|
511
|
-
exists.
|
|
509
|
+
[App_SystemMonitor_500Error](../../../../1.0/apps/library/features/http-500-error-monitor.md) — an
|
|
510
|
+
email-only count with no per-error detail, which is why this recipe exists.
|
|
512
511
|
|
|
513
512
|
#### Shutdown handler
|
|
514
513
|
|
|
@@ -6,8 +6,8 @@ project: TOGa 2.5 Supply
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
10
|
-
owners: ["apeterson"]
|
|
9
|
+
updated: 2026-09-25
|
|
10
|
+
owners: ["apeterson", "jcardinal"]
|
|
11
11
|
files:
|
|
12
12
|
- toga25-supply/src/App.tsx
|
|
13
13
|
- toga25-supply/src/contexts/AuthContext.tsx
|
|
@@ -52,15 +52,16 @@ const localStoragePersister = createSyncStoragePersister({
|
|
|
52
52
|
|
|
53
53
|
`PersistQueryClientProvider` (`src/App.tsx:33`) wraps the app. The `QueryClient` default is `staleTime: 0` (`src/App.tsx:18`).
|
|
54
54
|
|
|
55
|
-
### What is persisted — everything except
|
|
55
|
+
### What is persisted — everything except four namespaces
|
|
56
56
|
|
|
57
|
-
`shouldDehydrateQuery` (`src/App.tsx:63-70`) keeps TanStack's default (persist successful queries) **minus**
|
|
57
|
+
`shouldDehydrateQuery` (`src/App.tsx:63-70`) keeps TanStack's default (persist successful queries) **minus** four query-key namespaces:
|
|
58
58
|
|
|
59
59
|
| Excluded namespace | Why |
|
|
60
60
|
|---|---|
|
|
61
61
|
| `talos` | chat titles / assistant ids are per-session user data; must not sit on disk |
|
|
62
62
|
| `surface-meta` | the surface bundle carries **permission and feature-gate** state |
|
|
63
63
|
| `surface-meta-group` | same bundle, grouped endpoint |
|
|
64
|
+
| `table-meta` | TableView column/join config is schema-shaped; a stale copy sends dropped joins/columns (see gotcha) |
|
|
64
65
|
|
|
65
66
|
The surface exclusion is the important one. `useFetchSurfaceMeta` runs at `staleTime: Infinity`, so a *persisted* copy would never refetch — a permission **grant**, or worse a **revoke**, would not reach a browser that already had one until the persister's 24h `maxAge` expired. Excluding it costs one request per page load; the in-memory cache still covers every route change. `staleTime: Infinity` is safe there **only because** it is excluded from persisting.
|
|
66
67
|
|
|
@@ -69,7 +70,7 @@ The surface exclusion is the important one. `useFetchSurfaceMeta` runs at `stale
|
|
|
69
70
|
| Query key | Request |
|
|
70
71
|
|---|---|
|
|
71
72
|
| `["table-data", slug, …]` | `getDataTableData` (blox) |
|
|
72
|
-
| `["table-meta", slug]` | `GET /table-views/meta` (blox `
|
|
73
|
+
| `["table-meta", slug]` | `GET /table-views/meta` (blox `useFetchTablePageMeta.ts:15`) — **not persisted** |
|
|
73
74
|
| `["status-colors", route]` | `GET /{route}?fields=slug,colorHex&sort=sortOrder` |
|
|
74
75
|
| `["currentUser"]` | `GET /users` |
|
|
75
76
|
| `["clientFields", slug, role]` | client `FIELDS` config |
|
|
@@ -143,6 +144,9 @@ Type-check **both** `-p tsconfig.app.json` and `-p tsconfig.node.json` — `vite
|
|
|
143
144
|
|
|
144
145
|
## Gotchas
|
|
145
146
|
|
|
147
|
+
- **⚠ Never persist a query that carries schema-shaped config (TableView meta, column/join lists).** Before 2026-09-25 `table-meta` was persisted: after any TableView column/join removal, each browser's first load restored the OLD meta and fired the grid with the dropped join → api2 **1054 "Unknown column … in on clause"** in bursts, then meta refetched (`staleTime: 0`) and the grid recovered. A page reload does NOT avoid it (cache is in localStorage). Proven: NYCHH TableView change `Client_Nychh/2026-09-24a` (dropped `TransferOrders.purchaseOrderId` join) → prod `Logs.Event` issue 872. `table-data` / `table-data-infinite` stay persisted — safe because gated `enabled: !!tableViewMeta` (only a brief stale-row flash).
|
|
148
|
+
- **Diagnosis:** a 1054 on a column a migration dropped, while the prod TableView is already clean = stale persisted front-end config, not a DB fix. Confirm: the error burst ends seconds before the same user's first good query in `Logs_<Tenant>.Api`.
|
|
149
|
+
|
|
146
150
|
- **⚠ A non-zero `staleTime` on a persisted query serves yesterday's data and never fetches.** Data hooks in this app use `staleTime: 0`. Hooks that currently break this rule **and are persisted**: `useClientFields`, `useCurrentUser`, `useStatusTypeValues`, `useStatusColors`, `useSalesOrderUuidForServiceRequest` (all `Infinity`), and the three VendorItem form hooks (`5 * 60_000`). Each can serve a previous session's answer.
|
|
147
151
|
- `useTalosSession` and `useTalosThreads` also set a non-zero `staleTime`, but their keys start with `talos`, so they are **not** persisted — in-memory only, safe.
|
|
148
152
|
- **⚠ An edit to a client `FIELDS` JSON looks like it does nothing — and it is NOT Vite HMR.** `useClientFields` is `staleTime: Infinity` **and** persisted, so the old config is frozen in `supply-chain-query-cache`. Clear that key to see the edit. Proper fix: exclude `clientFields` in `shouldDehydrateQuery` (`src/App.tsx:42`).
|
|
@@ -152,4 +156,5 @@ Type-check **both** `-p tsconfig.app.json` and `-p tsconfig.node.json` — `vite
|
|
|
152
156
|
- The `maxAge` is one timestamp for the entire blob — you cannot expire a single query through it.
|
|
153
157
|
|
|
154
158
|
## Change history
|
|
159
|
+
- 2026-09-25 — Excluded `table-meta` from persistence (stale TableView meta caused api2 1054 bursts after a join removal; NYCHH issue 872). Deployed to prod. (jcardinal)
|
|
155
160
|
- 2026-09-24 — Noted the frozen-`FIELDS`-JSON symptom (persisted `useClientFields` at `staleTime: Infinity`, mistaken for broken Vite HMR) and the `shouldDehydrateQuery` fix. (apeterson)
|
|
@@ -10,7 +10,7 @@
|
|
|
10
10
|
| [Callback Scheduling ("call me back") — worker2 AI-BDR](features/callback-scheduling.md) | The AI-BDR callback path (prospect says "call me back" → dialer redials) and its production defects; open when working on Vapi callback capture/parse/schedule o |
|
|
11
11
|
| [ClickUp Connectivity Watchdog](features/clickup-connectivity-watchdog.md) | Cron watchdog that emails when the ClickUp integration goes silent in business hours; open when touching ClickUp webhook health or `_Worker_Clickup_Health`. |
|
|
12
12
|
| [ClickUp Custom-Field Conventions (constant IDs, runtime option values, Task Type migration)](features/clickup-custom-field-conventions.md) | How worker2 code must reference ClickUp custom fields; open before reading/writing any ClickUp custom field or touching the Task Type migration. |
|
|
13
|
-
| [ClickUp Design Sprint Automation (Final Design Outcome)](features/clickup-design-sprint-automation.md) | `_Worker_Clickup_Design` drives the design-sprint workflow in ClickUp (spawn iteration / complete task on Final Design Outcome, daily
|
|
13
|
+
| [ClickUp Design Sprint Automation (Final Design Outcome)](features/clickup-design-sprint-automation.md) | `_Worker_Clickup_Design` drives the design-sprint workflow in ClickUp (FDO default on create, spawn iteration / complete task on Final Design Outcome, daily rev |
|
|
14
14
|
| [ClickUp General Automation (Sub-Environment reservation, Has UI / Has Code roll-ups)](features/clickup-general-automation.md) | `_Worker_Clickup_GeneralAutomation` reacts to ClickUp custom-field changes on a Task; open when working on sub-environment reservation or Has UI / Has Code roll |
|
|
15
15
|
| [ClickUp GitHub-tab Auto-linking & Ticket-id Branch Naming](features/clickup-github-autolink.md) | How ClickUp surfaces branches/PRs/commits in a ticket's GitHub tab and the branch/PR-title naming that triggers it; open when naming branches for ticket work or |
|
|
16
16
|
| [ClickUp Project & Opportunity Multi-List Routing](features/clickup-project-routing.md) | Routes ClickUp tasks into secondary multi-list memberships by custom-field values; open for opportunity/epic routing, the stage-is-a-list constraint, or webhook |
|
|
@@ -18,7 +18,7 @@
|
|
|
18
18
|
| [ClickUp Subtask Activity → Parent Opportunity/Epic Comments](features/clickup-subtask-activity.md) | Mirrors subtask status/comment activity up to its top-level Opportunity/Epic as one comment; open for the subtask fan-out, its dedup ledger, or structural ances |
|
|
19
19
|
| [ClickUp Task Description Fluffer (Talos KB Investigation)](features/clickup-task-fluffer-talos.md) | Async worker that "fluffs" a new ClickUp task by running its name+description through the Talos `dev-core` KB agent and writing the report back; open for the fl |
|
|
20
20
|
| [ClickUp Webhook Health Monitor → OneUptime (Monitor/Clickup/CheckWebhook)](features/clickup-webhook-health-monitor.md) | worker2 cron that checks the team's ClickUp webhook subscriptions and pages via a OneUptime Incoming-Request monitor; open when tuning the ClickUp webhook alarm |
|
|
21
|
-
| [ClickUp Work Type Automation (Committed / Conditional / Stretch)](features/clickup-work-type-automation.md) | Auto-maintains each ClickUp task's Work Type (Committed/Conditional/Stretch) from its dependencies, status, and due date
|
|
21
|
+
| [ClickUp Work Type Automation (Committed / Conditional / Stretch)](features/clickup-work-type-automation.md) | Auto-maintains each ClickUp task's Work Type (Committed/Conditional/Stretch) from its dependencies, status, and due date, and sets new-task defaults on taskCrea |
|
|
22
22
|
| [worker2 Codeception integration tests (2.0 test-framework pilot)](features/codeception-integration-tests.md) | First 2.0 app-level automated test suite: Codeception 5 / PHPUnit 11 in worker2 (pilot, Jeff's direction), real-DB integration tests with hard local-only safety |
|
|
23
23
|
| [worker2 — Committed Secrets & Config Exposure (debt register)](features/committed-secrets-and-config-debt.md) | Standing list of credentials committed to the worker2 repo (locations only); open before adding a credential to worker2 or assuming a config value is private. |
|
|
24
24
|
| [Compass VIP Support Importer (worker2)](features/compass-vip-support-importer.md) | worker2 action that ingests Compass's quarterly VIP spreadsheet and sets each VIP's support tech (`Users.c_supportedByUserId` in `Client_Compass`); open when wo |
|
|
@@ -44,7 +44,7 @@
|
|
|
44
44
|
| [Background Email-Template Worker (_Worker_Notification_EmailTemplate)](features/notification-email-template.md) | Worker action that dispatches a stored client-defined `EmailTemplates` row by UUID off-thread; open for sending a complete pre-authored transactional template a |
|
|
45
45
|
| [DB-Driven Notification (Internal) Email](features/notification-email.md) | Worker action that wraps an ad-hoc subject/body in the client's DB-stored branded shell and sends it; open for internal/system notification email (merge alerts, |
|
|
46
46
|
| [NYCHH Asset-Tag Backfill (worker2)](features/nychh-asset-tag-backfill.md) | Backfills NYCHH unit asset tags + MAC addresses from NetSuite serial (inventory-number) records; open when the fulfillment sync misses a tag added to a serial a |
|
|
47
|
-
| [OneUptime Incident
|
|
47
|
+
| [OneUptime Incident → ClickUp Task Sync (Monitor/Oneuptime/SyncIncidents)](features/oneuptime-incident-clickup-sync.md) | worker2 polls OneUptime every 15 min for open incidents and creates/auto-closes a ClickUp task per incident; open for the incident→ClickUp sync, its claim-fir |
|
|
48
48
|
| [OneUptime push-metric monitors for 2.0 workers](features/oneuptime-worker2-monitoring.md) | OneUptime push-metric monitor pattern for worker2 (2.0): a self-contained cron action decides pass/fail and POSTs a string token to a OneUptime Incoming Request |
|
|
49
49
|
| [Platform Cache Cleanup (_Worker_Platform_Cache — Clean + Truncate)](features/platform-cache-cleanup.md) | `_Worker_Platform_Cache` maintenance of the shared Cache cluster behind api2 cross-client retrieval; open when touching cache reaping/truncate or the Clean cron |
|
|
50
50
|
| [QA/QC Branch Automation (auto-mirror + rebuild-from-ledger revert)](features/qa-qc-branch-automation.md) | worker2-centralized GitHub branch automation (mirror, ledger-revert, the single guarded door to `_production`); open when touching QA/QC branch merges or revert |
|
|
@@ -6,8 +6,8 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
10
|
-
owners: ["ajean"]
|
|
9
|
+
updated: 2026-09-25
|
|
10
|
+
owners: ["ajean", "jcardinal"]
|
|
11
11
|
files:
|
|
12
12
|
- worker2/Worker/Clickup/Design.php
|
|
13
13
|
- worker2/Worker/Clickup.php
|
|
@@ -18,13 +18,13 @@ related:
|
|
|
18
18
|
- ./team-sprint-management.md
|
|
19
19
|
---
|
|
20
20
|
|
|
21
|
-
`_Worker_Clickup_Design` drives the design-sprint workflow in ClickUp (spawn iteration / complete task on Final Design Outcome, daily review reminders); open when working on design-sprint automation, FDO handling, or the spurious-email / no-spawn bugs.
|
|
21
|
+
`_Worker_Clickup_Design` drives the design-sprint workflow in ClickUp (FDO default on create, spawn iteration / complete task on Final Design Outcome, daily review reminders); open when working on design-sprint automation, FDO handling, or the spurious-email / no-spawn bugs.
|
|
22
22
|
|
|
23
23
|
## Summary
|
|
24
24
|
|
|
25
25
|
`_Worker_Clickup_Design` replaces a set of native ClickUp automations. When a task's **Final Design Outcome** (FDO) drop-down is set to an actionable value, the worker spawns an iteration task or completes the task; a daily cron reminds the Design Approver of pending reviews.
|
|
26
26
|
|
|
27
|
-
**Current state (2026-
|
|
27
|
+
**Current state (2026-09-25):** live on both design folders (`DESIGN_SPRINTS_FOLDER_IDS`: True Design Sprints `90115939842`, Design Sprints `90116585900`) + Backlog Design lists. Approve→complete and iterate→spawn verified working after the 9/21 multi-folder gate fix (commit `6dcebe0`); earlier misses on the new folder were the old single-folder gate. History of the 2026-07 no-op / no-spawn fixes (worker2 PR #98, #113 + _underscore PR #669) is in Gotchas. The native ClickUp automations masking the original no-op must stay disabled.
|
|
28
28
|
|
|
29
29
|
## How it works
|
|
30
30
|
|
|
@@ -32,15 +32,16 @@ related:
|
|
|
32
32
|
|
|
33
33
|
- `Worker/Clickup/Design.php` → `_Worker_Clickup_Design` — the design-sprint engine.
|
|
34
34
|
- `handleFinalDesignOutcome(string $taskId, ?string $webhookFdoValue = null)` — decides spawn vs. complete; branches on **all five** FDO options explicitly (no silent fall-through). Optional `$webhookFdoValue` is the authoritative new value from the webhook, preferred over the (possibly stale) snapshot — see Gotchas (read-after-write).
|
|
35
|
-
- `
|
|
35
|
+
- `handleTaskCreated($taskId, $taskDetails)` — called from `_Worker_Clickup` `taskCreated` (wrapped in `setFieldSafely`). Sets FDO = **In Progress** when empty; design-scope gated. Replaces a native ClickUp automation that existed only on the old True Design Sprints folder (decision: code, not a ClickUp automation).
|
|
36
|
+
- `isDesignSprintTask(...)` — the spawn/handler gate. Admits: any `DESIGN_SPRINTS_FOLDER_IDS` folder, **OR** (the Backlog folder `BACKLOG_FOLDER_ID` AND the list name matches `DESIGN_LIST_NAME_PATTERN`), **OR** the `DESIGN_LIST_IDS` allowlist (fallback). Fixes the "backlog Design list silently skipped" root cause — see Gotchas.
|
|
36
37
|
- `resolveFinalDesignOutcomeOptionId(object $taskDetails, ?string $overrideValue = null)` (Design.php:298-317) — maps the drop-down value to an option ID; **string-compares** value/orderindex/id (see Gotchas). When `$overrideValue` is given (the webhook `after` value) it wins over the snapshot's field value; the option *list* is still read from the snapshot.
|
|
37
38
|
- `handleTaskEnteredIterated()` — **new**; fired when a task ENTERS "iterated" status. Sets FDO = "In Review" then increments the Iteration custom field by +1 (idempotent — see Gotchas). Folder-gated via `isInDesignSprintsFolder()`.
|
|
38
39
|
- `isInDesignSprintsFolder(...)` — **new** private folder-gate helper (reused by the iterated-entry trigger).
|
|
39
|
-
- `spawnIterationTask(...)` / `buildIterationCreatePayload(...)` / `carryDesignApprover(...)` / `carryComments(...)` (
|
|
40
|
+
- `spawnIterationTask(...)` / `buildIterationCreatePayload(...)` / `carryDesignApprover(...)` / `carryComments(...)` / `carryDropdownField(...)` — duplicate the task in status `STATUS_NEW_TASK` (`in progress`) and copy description, Design Approver, Task Knowledge (`CLICK_UP_CUSTOM_FIELD_ID__TASK_KNOWLEDGE`, const in Design.php), Work Effort (`CLICK_UP_CUSTOM_FIELD_ID__WORKEFFORT`), and all user comments; the duplicate no longer bumps Iteration (moved to the iterated-entry trigger) and is set to FDO = "In Progress" post-create.
|
|
40
41
|
- `completeTask(...)` — closes an approved / cancelled task (also used on iterated → complete).
|
|
41
42
|
- `SendReviewReminders(...)` — daily reminder cron action; now also fetches the Backlog folder's lists and includes those matching `DESIGN_LIST_NAME_PATTERN`, mirroring `isDesignSprintTask()` so newly-created backlog Design lists get reminders without a code change.
|
|
42
43
|
- `findExistingSpawnedIteration(...)` — its previously-silent `catch` now logs.
|
|
43
|
-
- `
|
|
44
|
+
- `DESIGN_SPRINTS_FOLDER_IDS`, `BACKLOG_FOLDER_ID` (`'90116192732'`), and `DESIGN_LIST_NAME_PATTERN` (`/(?<!non-)\bdesign\b/i`) constants.
|
|
44
45
|
- `Worker/Clickup.php` `_Worker_Clickup::extractCustomFieldAfterValue(object $historyItem): ?string` (**new**) — normalizes a webhook `history_items[].after` custom-field value (scalar orderindex / object / JSON array with `id`|`orderindex`). Distinguishes a **missing** `after` (returns `null` → caller falls back to the snapshot) from an **explicit null** `after` (returns `''` sentinel → field was cleared → resolve to no outcome, do NOT act on the stale previous option).
|
|
45
46
|
- `Worker/Clickup.php` `Webhook` — the `switch ($historyItem->after->status)` block (see Gotchas re: after vs. before) now has a `case _Worker_Clickup_Design::STATUS_ITERATED` dispatching to `handleTaskEnteredIterated()`.
|
|
46
47
|
- `Worker/Clickup.php` — routes ClickUp webhooks.
|
|
@@ -57,8 +58,8 @@ related:
|
|
|
57
58
|
- **"Cancelled"** → `completeTask` (terminal; newly handled — Cancelled now closes the task).
|
|
58
59
|
- **"In Progress" / "In Review"** → explicit no-op.
|
|
59
60
|
- unrecognized → logged (no more silent implicit-else fall-through).
|
|
60
|
-
3. **Iterated-entry trigger
|
|
61
|
-
4. **Duplicate field carry-over:** the spawned iteration task copies description, Design Approver, and **all user comments** (`carryComments()` — GET `/task/{id}/comment`, re-posted oldest-first with `[Original by <user> on <date>]` attribution, non-fatal). It copies **Iteration unchanged** (omitted if the source has none — the +1 now lives only on the iterated-entry trigger, so a review round is never double-counted) and is set to FDO = **"In Progress"** post-create (POST `/task/{id}/field/{id}` drop_down pattern).
|
|
61
|
+
3. **Iterated-entry trigger:** when a design-sprint task ENTERS the "iterated" status, `handleTaskEnteredIterated()` sets FDO = **"In Review"** and increments the **Iteration** custom field by +1. This now drives the review state + round count on review submission; previously nothing fired on entering "iterated". Folder-gated.
|
|
62
|
+
4. **Duplicate field carry-over:** the spawned iteration task starts in `in progress` and copies description, Design Approver, Task Knowledge, Work Effort, and **all user comments** (`carryComments()` — GET `/task/{id}/comment`, re-posted oldest-first with `[Original by <user> on <date>]` attribution, non-fatal). It copies **Iteration unchanged** (omitted if the source has none — the +1 now lives only on the iterated-entry trigger, so a review round is never double-counted) and is set to FDO = **"In Progress"** post-create (POST `/task/{id}/field/{id}` drop_down pattern).
|
|
62
63
|
5. Daily reminder cron `Core.CronJobs` id 21 ("Design Review Reminders", `0 9 * * 1-5`, action `Clickup/Design/SendReviewReminders`) nudges the approver about pending reviews.
|
|
63
64
|
|
|
64
65
|
### Data model / configuration
|
|
@@ -71,6 +72,8 @@ Configuration lives as ClickUp identifiers (folder IDs, custom-field IDs, drop-d
|
|
|
71
72
|
|
|
72
73
|
## Gotchas
|
|
73
74
|
|
|
75
|
+
- **Webhooks triggered by the automation's own ClickUp user ARE delivered.** Do not blame self-suppression when a chained step (e.g. spawn → FDO write) seems not to fire; check the folder gate first.
|
|
76
|
+
- **Status names must exist in every gated list.** `STATUS_NEW_TASK = 'in progress'` was verified present in the design lists via prod webhooks (the old `'to do'` is not used).
|
|
74
77
|
- **The ClickUp API auth is a single hard-coded personal token — a dead token silently kills ALL ClickUp automation.** The integration authenticates via one committed class constant, `_Component_Api_Clickup::ACCESS_TOKEN` (in `_underscore/Component/Api/Clickup/Clickup.php`), identical on `_production` and feature branches. Known secret-in-committed-code risk. When the token is revoked/expired, `GET /user` and `GET /task` return **HTTP 401 "Oauth token not found" / error `OAUTH_019`**; `getTaskDetails` then throws and **every** Clickup/Webhook worker job fails — presenting as "the automation silently does nothing." This happened mid-session on 2026-07-07 (~09:49). **Fix:** regenerate the token in ClickUp, update the `ACCESS_TOKEN` constant, and deploy `_underscore`. (Do not record the token value anywhere.)
|
|
75
78
|
- **Iterated-entry trigger is idempotent — order matters.** `handleTaskEnteredIterated()` writes **FDO = In Review FIRST** as the marker, THEN increments Iteration. A re-delivered / retried webhook sees FDO already "In Review" and skips, so **Iteration can never compound-double-increment**. Worst case is a single missed +1, visible as a failed WorkerJob. Reordering these two writes reintroduces the double-count risk.
|
|
76
79
|
- **`taskStatusUpdated` dispatch structure (route-safety facts).** The `switch ($historyItem->after->status)` block sits **OUTSIDE** the `if (!is_null($sprintNumber))` guard, so entering-status handlers fire even for design tasks with no "sprint N" tag — **each case must self-guard on `sprintNumber`** (the individual cases do). No webhook loop: custom-field writes emit `taskUpdated` (not `taskStatusUpdated`) and land on the FDO no-op cases; the Iteration write's custom_field is named "Iteration" so it never re-enters the FDO dispatch gate.
|
|
@@ -6,16 +6,17 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
9
|
+
updated: 2026-09-25
|
|
10
10
|
owners: ["jcardinal"]
|
|
11
11
|
files:
|
|
12
12
|
- worker2/Worker/Clickup.php
|
|
13
13
|
- worker2/Tests/Worker/ClickupWorkTypeTest.php
|
|
14
14
|
related:
|
|
15
15
|
- ./clickup-project-routing.md
|
|
16
|
+
- ../workflows/clickup-credential-rotation.md
|
|
16
17
|
---
|
|
17
18
|
|
|
18
|
-
Auto-maintains each ClickUp task's Work Type (Committed/Conditional/Stretch) from its dependencies, status, and due date; open when work-type promotion/demotion
|
|
19
|
+
Auto-maintains each ClickUp task's Work Type (Committed/Conditional/Stretch) from its dependencies, status, and due date, and sets new-task defaults on taskCreated; open when work-type promotion/demotion or taskCreated field writes misbehave.
|
|
19
20
|
|
|
20
21
|
## Summary
|
|
21
22
|
|
|
@@ -41,6 +42,13 @@ Flow:
|
|
|
41
42
|
- `Stretch` with a due date inside the current sprint → set Committed/Conditional as above.
|
|
42
43
|
6. When a task is depended on by another (`$dependency->depends_on == $taskId`), it recurses into the blocking direction so completing a task re-evaluates the tasks it was blocking. The `$dependsOnTaskId` argument breaks two-node cycles.
|
|
43
44
|
|
|
45
|
+
### New-task defaults (`taskCreated`)
|
|
46
|
+
|
|
47
|
+
- Every optional field write in `taskCreated` goes through private `setFieldSafely($taskId, $fieldLabel, callable)`: try/catch → `error_log` + Sentry. One refused field never aborts the job (the later Clickup/Fluffer/Process dispatch still runs).
|
|
48
|
+
- **Work Type = Unplanned** when empty, only if the list name starts with `Sprint` AND `isTaskInQaQcScope()` (Dev Sprints folder). Option id resolved by name at runtime (`WORK_TYPE_UNPLANNED`). Non-`Sprint` lists get no Work Type write (jcardinal decision).
|
|
49
|
+
- **Previous Sprint Work Type** = the Work Type name (or `Unplanned` when empty), resolved by name via the field's own options.
|
|
50
|
+
- Design FDO default is dispatched from here too — see [design sprint automation](./clickup-design-sprint-automation.md).
|
|
51
|
+
|
|
44
52
|
Data model: reads/writes the `Team` database (`Tasks`, `Developers`, `Tasks_Developers`, `Sprints`) and the ClickUp REST API. The Work Type itself lives in ClickUp as a custom field (`CLICK_UP_CUSTOM_FIELD_ID__WORKTYPE`), not in the DB.
|
|
45
53
|
|
|
46
54
|
## Gotchas
|
|
@@ -50,6 +58,8 @@ Data model: reads/writes the `Team` database (`Tasks`, `Developers`, `Tasks_Deve
|
|
|
50
58
|
- **No PHPUnit harness** in worker2. The regression test `Tests/Worker/ClickupWorkTypeTest.php` is a plain-PHP script (reflection on `isStatusComplete`) run with `php Tests/Worker/ClickupWorkTypeTest.php`.
|
|
51
59
|
- **ClickUp dropdown custom-field `value` is the option ORDERINDEX, not its name.** The webhook payload for a dropdown/labels custom field delivers the selected option's integer `orderindex`, *not* the option's display name. Any name-keyed lookup (e.g. `$lookupPreviousSprintWorkTypeByName`) will silently never match if fed the raw `value`. Resolve the name first via the field's own `type_config->options` — build a map keyed by `orderindex` (`$workTypesByIndex[$opt->orderindex] = $opt`) and read `->name` from it. This is the proven pattern in `_Worker_Team_Sprint::SprintLock` (`Worker/Team/Sprint.php`, the Work Type resolve ~L3894 and the POST ~L3971). Guard the lookup with `isset($workTypesByIndex[$value])` so a stale/deleted option orderindex doesn't warn.
|
|
52
60
|
|
|
61
|
+
- **2026-09-21..25 outage: every taskCreated failed (~113 jobs, 0 Fluffer jobs).** The automation user lacked edit rights on Previous Sprint Work Type → HTTP 401 `ACCESS_606`; the throw killed the rest of the job. Fixed by field permissions + `setFieldSafely`. Any new optional write must use `setFieldSafely`. Account-change check: [credential rotation](../workflows/clickup-credential-rotation.md).
|
|
62
|
+
|
|
53
63
|
## Related
|
|
54
64
|
|
|
55
65
|
- [ClickUp Project & Opportunity Multi-List Routing](./clickup-project-routing.md)
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
---
|
|
2
|
-
title: OneUptime Incident
|
|
2
|
+
title: OneUptime Incident → ClickUp Task Sync (Monitor/Oneuptime/SyncIncidents)
|
|
3
3
|
framework: "2.0"
|
|
4
4
|
repo: worker2
|
|
5
5
|
project: Worker
|
|
@@ -23,20 +23,20 @@ related:
|
|
|
23
23
|
- ../../_underscore/features/component-model-namespace-registration.md
|
|
24
24
|
---
|
|
25
25
|
|
|
26
|
-
worker2 polls OneUptime every 15 min for open incidents and creates/auto-closes a ClickUp task per incident; open for the incident
|
|
26
|
+
worker2 polls OneUptime every 15 min for open incidents and creates/auto-closes a ClickUp task per incident; open for the incident→ClickUp sync, its claim-first dedup, or the auto-close pass.
|
|
27
27
|
|
|
28
28
|
## Summary
|
|
29
29
|
|
|
30
|
-
An internal/shared TOGA ops feature: worker2 polls the OneUptime API every 15 minutes for **currently-open** incidents and ensures a ClickUp task exists for each, so operational incidents become actionable tickets automatically without duplicates. This is the **first *reader* of the OneUptime API** in worker2
|
|
30
|
+
An internal/shared TOGA ops feature: worker2 polls the OneUptime API every 15 minutes for **currently-open** incidents and ensures a ClickUp task exists for each, so operational incidents become actionable tickets automatically without duplicates. This is the **first *reader* of the OneUptime API** in worker2 — until now OneUptime was **push-only** (workers POST metrics/heartbeats to Incoming-Request monitors; see [OneUptime push-metric monitors](./oneuptime-worker2-monitoring.md)). The model was "dumb reporter, smart monitor"; this adds the reader half.
|
|
31
31
|
|
|
32
32
|
## How it works
|
|
33
33
|
|
|
34
34
|
### Entry points
|
|
35
35
|
|
|
36
|
-
- `worker2/Component/Api/Oneuptime/Oneuptime.php`
|
|
37
|
-
- `worker2/Worker/Monitor/Oneuptime.php`
|
|
36
|
+
- `worker2/Component/Api/Oneuptime/Oneuptime.php` — `_Component_Api_Oneuptime`, the OneUptime API read client, modeled on `_Component_Api_Togaiq`. Reads a new `[oneuptime]` config group (endpoint, api_key, project_id) via `_Config`; `isConfigured()` guard; `send()` wraps `_ApiRequest` with `APIKey` + `ProjectID` headers; `getActiveIncidents()` returns normalized open incidents (filters out resolved **in PHP**).
|
|
37
|
+
- `worker2/Worker/Monitor/Oneuptime.php` — `_Worker_Monitor_Oneuptime::SyncIncidents(): string`, the cron action. Returns a structured summary string on every path (`incidentsPolled` / `created` / `skipped` / `failed`, plus `closed` / `closeFailed`).
|
|
38
38
|
|
|
39
|
-
Canonical worker2 scheduled-3rd-party-API-job shape (see [Creating Worker Actions](./creating-worker-actions.md)): a `Core.CronJobs` row (cron expr, Central tz)
|
|
39
|
+
Canonical worker2 scheduled-3rd-party-API-job shape (see [Creating Worker Actions](./creating-worker-actions.md)): a `Core.CronJobs` row (cron expr, Central tz) → CronScheduler Lambda enqueues → `Controller/Index::worker()` dispatches action `Monitor/Oneuptime/SyncIncidents` to `_Worker_Monitor_Oneuptime::SyncIncidents`. HTTP goes through `_ApiRequest`; secrets come from `_Config::oneuptime()` reading `Config/<env>.ini`; ClickUp uses `_Component_Api_Clickup::send()`.
|
|
40
40
|
|
|
41
41
|
### Open-only
|
|
42
42
|
|
|
@@ -44,14 +44,14 @@ Only currently-open incidents are synced. The list route returns all incidents a
|
|
|
44
44
|
|
|
45
45
|
### Claim-first (unique-key) dedup
|
|
46
46
|
|
|
47
|
-
For each open incident, INSERT a ledger row into `OneUptimeIncidentTasks` keyed on the OneUptime incident id **before** calling ClickUp. A unique-key violation means a task already exists
|
|
47
|
+
For each open incident, INSERT a ledger row into `OneUptimeIncidentTasks` keyed on the OneUptime incident id **before** calling ClickUp. A unique-key violation means a task already exists → skip. On success, create the ClickUp task via `_Component_Api_Clickup::send('POST','/list/{listId}/task', …)` and attach the returned id to the ledger row; on failure, **delete the claim and re-throw**. Established worker2 way to prevent duplicate ClickUp tasks under overlapping cron runs — pattern copied from `_Worker_Infrastructure_Errors::createClickupTask`. `INSERT … WHERE NOT EXISTS` is **insufficient** under REPEATABLE READ; the UNIQUE key is the concurrency arbiter.
|
|
48
48
|
|
|
49
49
|
### Close pass (auto-close on incident resolution)
|
|
50
50
|
|
|
51
51
|
The sync was originally **create-only**; a ClickUp task opened for an incident was never closed once the incident resolved, so a recurrence (a *new* OneUptime incident with the same monitor title) opened a second task that **looked like a duplicate** but was not. `SyncIncidents()` now runs a **close pass first** each cycle, independent of how many incidents are currently open, via `closeResolvedIncidents()`:
|
|
52
52
|
|
|
53
53
|
- SELECT ledger rows `WHERE clickupIdentifier IS NOT NULL AND dtClosed IS NULL`.
|
|
54
|
-
- For each, call `_Component_Api_Oneuptime::getIncidentResolvedState(string $incidentId): ?bool`
|
|
54
|
+
- For each, call `_Component_Api_Oneuptime::getIncidentResolvedState(string $incidentId): ?bool` — `true` = resolved, `false` = confirmed still open, `null` = unknown. **Only a definite `true` closes a task**; `null` means "leave alone" so an ambiguous read never closes a live incident's ticket. Uses a new per-incident route (`ROUTE__INCIDENT_ITEM`, `/api/incident/{id}/get-item`, still a PLACEHOLDER) because resolved incidents no longer appear in `getActiveIncidents()` and must be re-checked individually. A 2xx response missing `currentIncidentState.isResolvedState` logs a warning (incident id only, no body/PII) so a wrong placeholder shape is detectable rather than silently returning `null` forever.
|
|
55
55
|
- To close, PUT `/task/{id}` with the list's **closed-type status NAME** (ClickUp has no generic close verb), resolved per-list via `GET /list/{id}` `statuses[]` where `type == 'closed'`, cached per run in `$closedStatusByList`. Then stamp `dtClosed = NOW()`.
|
|
56
56
|
- `dtClosed` is stamped **only after a successful close**, so a failure retries next run. Per-incident failures are caught (`Throwable`) + `error_log` + counted, not fatal; whole-run failures (e.g. no sprint list) still re-throw for queue retry.
|
|
57
57
|
- All new ClickUp calls in the close path are throttled with `usleep(_Worker_Clickup_Project::RATE_LIMIT_DELAY_US)` (120ms = 500/min), matching `_Worker_Infrastructure_Errors`.
|
|
@@ -60,36 +60,36 @@ Helpers added: `closeResolvedIncidents()`, `closeClickupTask()`, `resolveClosedS
|
|
|
60
60
|
|
|
61
61
|
### Target list = current sprint list
|
|
62
62
|
|
|
63
|
-
The ClickUp task lands in the **current sprint** list, resolved by `GET /folder/{id}` and matching the list whose `start_date`/`due_date` window contains now (mirrors `_Worker_Team_Sprint::SprintLock`
|
|
63
|
+
The ClickUp task lands in the **current sprint** list, resolved by `GET /folder/{id}` and matching the list whose `start_date`/`due_date` window contains now (mirrors `_Worker_Team_Sprint::SprintLock` — see [Team Sprint Management](./team-sprint-management.md) and the sprint-promotion note in [Error Escalation Cron](./error-escalation-cron.md)). Folder id is read from `[clickup] sprint_folder_id` with a constant fallback (`90116585845 — Dev Sprints; repointed by TRUE-82052 on 2026-09-21`).
|
|
64
64
|
|
|
65
65
|
### Data model
|
|
66
66
|
|
|
67
|
-
`OneUptimeIncidentTasks` (CREATE on the **Team** cluster
|
|
67
|
+
`OneUptimeIncidentTasks` (CREATE on the **Team** cluster — chosen over Logs; `dbchanges2/Team/2026-08-10a`): `id`, `uuid char(36)`, `dtCreated`, `oneuptimeIncidentId varchar(64)`, `clickupIdentifier varchar(64) NULL`, `dtClosed DATETIME NULL` (added `dbchanges2/Team/2026-08-12a`, `AFTER clickupIdentifier`). The **UNIQUE KEY on `oneuptimeIncidentId`** is the dedup contract, same rationale as `IssueClickupTasks`. `dtClosed` marks a ledger row whose ClickUp task has been closed following incident resolution (NULL = still open / not yet closed).
|
|
68
68
|
|
|
69
69
|
### Provisioning / deploy
|
|
70
70
|
|
|
71
71
|
- `dbchanges2/Core/2026-08-10b` inserts the `Core.CronJobs` row: action `Monitor/Oneuptime/SyncIncidents`, schedule `*/15 * * * *`, `maxExecutionTime` 120, seeded **`isActive = 0`**. Flip to 1 only after (a) the ledger table exists, (b) the `[oneuptime]` config group is populated, (c) the incident-list route is confirmed.
|
|
72
|
-
- `[oneuptime]` config group (endpoint, api_key, project_id) added to `worker2/Config/production.ini` with placeholders
|
|
72
|
+
- `[oneuptime]` config group (endpoint, api_key, project_id) added to `worker2/Config/production.ini` with placeholders — **values filled by the developer, not committed**. Credentials live only in the ini files.
|
|
73
73
|
|
|
74
74
|
### Configuration / credentials
|
|
75
75
|
|
|
76
|
-
- `[oneuptime]`
|
|
77
|
-
- `[clickup]`
|
|
76
|
+
- `[oneuptime]` — `endpoint`, `api_key`, `project_id`. The hosted OneUptime SaaS base is `https://oneuptime.com` (confirmed from `worker/.ebextensions/045_oneuptime_agent.config`); the REST API path is appended to that base.
|
|
77
|
+
- `[clickup]` — `sprint_folder_id` (fallback constant `90116585845 — Dev Sprints; repointed by TRUE-82052 on 2026-09-21`) plus the ClickUp token. ClickUp/OneUptime token **values** live in `worker2/Config/<env>.ini` — location only, never the value.
|
|
78
78
|
|
|
79
79
|
## Gotchas
|
|
80
80
|
|
|
81
81
|
- **Same-title ClickUp tickets from OneUptime are expected on incident recurrence.** Dedup is **per-incident-id, not per-title**: the `oneuptimeIncidentId` UNIQUE key + claim-first insert only prevents duplicates for the *same* incident id. Two tasks with the same monitor title minutes apart are two *different* OneUptime incidents, not a dedup failure. The symptom that looked like duplication was actually the missing auto-close: incident #1 resolved but its task never closed, then a fresh incident opened a second task. Auto-close closes #1 on resolution so a recurrence (#2) is the only open ticket.
|
|
82
|
-
- **ClickUp has no generic "close" verb.** A task is closed by PUTting the exact NAME of the status whose `type == 'closed'`, which is **list-specific**
|
|
82
|
+
- **ClickUp has no generic "close" verb.** A task is closed by PUTting the exact NAME of the status whose `type == 'closed'`, which is **list-specific** — resolved per-list from `GET /list/{id}` `statuses[]` and cached per run.
|
|
83
83
|
- Throttle every ClickUp call in the close path with `usleep(RATE_LIMIT_DELAY_US)` (120ms) or rate-limit failures degrade into `closeFailed` counts.
|
|
84
|
-
- **Both OneUptime routes remain PLACEHOLDERS**
|
|
85
|
-
- **OneUptime cannot filter open-only server-side here**
|
|
84
|
+
- **Both OneUptime routes remain PLACEHOLDERS** — the incident-list route (`/api/incident/get-list`) *and* the new per-incident route (`/api/incident/{id}/get-item`, `ROUTE__INCIDENT_ITEM`) are unverified against the live OneUptime API. Auto-close **fails safe** (only a definite resolved `true` closes anything; `null` leaves the task alone) but won't fire until the route/field shape is confirmed. Do not enable the cron (`isActive = 1`) until the routes and returned fields are verified.
|
|
85
|
+
- **OneUptime cannot filter open-only server-side here** — resolution filtering happens in PHP, so the returned field set must actually carry the resolved/open state.
|
|
86
86
|
- Ship the worker2 code deploy and both dbchanges2 migrations **together**; the cron row is inert until the action is deployed and the ledger table exists.
|
|
87
|
-
- **Deploy order for the auto-close change:** run `dbchanges2/Team/2026-08-12a` (the `dtClosed` column) **before** the worker2 code deploys
|
|
87
|
+
- **Deploy order for the auto-close change:** run `dbchanges2/Team/2026-08-12a` (the `dtClosed` column) **before** the worker2 code deploys — the new close-pass queries reference `dtClosed`.
|
|
88
88
|
|
|
89
89
|
## Related
|
|
90
90
|
|
|
91
|
-
- [OneUptime push-metric monitors](./oneuptime-worker2-monitoring.md)
|
|
92
|
-
- [Error Escalation Cron](./error-escalation-cron.md)
|
|
93
|
-
- [Team Sprint Management](./team-sprint-management.md)
|
|
94
|
-
- [Creating Worker Actions](./creating-worker-actions.md)
|
|
91
|
+
- [OneUptime push-metric monitors](./oneuptime-worker2-monitoring.md) — the push-only reporter half.
|
|
92
|
+
- [Error Escalation Cron](./error-escalation-cron.md) — the `_Worker_Infrastructure_Errors` claim-first pattern this copies, and sprint promotion.
|
|
93
|
+
- [Team Sprint Management](./team-sprint-management.md) — the current-sprint-list resolution mirrored here.
|
|
94
|
+
- [Creating Worker Actions](./creating-worker-actions.md) — the scheduled-3rd-party-API-job shape.
|
|
95
95
|
- [Component/model namespace registration](../../_underscore/features/component-model-namespace-registration.md).
|
|
@@ -5,8 +5,8 @@ repo: worker2
|
|
|
5
5
|
project: Worker
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
|
-
status:
|
|
9
|
-
updated: 2026-09-
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-09-25
|
|
10
10
|
owners: [jcardinal]
|
|
11
11
|
files:
|
|
12
12
|
- worker2/Worker/Clickup/QaReview.php
|
|
@@ -26,14 +26,14 @@ related:
|
|
|
26
26
|
- ./clickup-custom-field-conventions.md
|
|
27
27
|
---
|
|
28
28
|
|
|
29
|
-
ClickUp-driven QA/QC review + promotion chain (Dev→QA→QC→Client Review→Stage→Prod) for worker2; open when touching a review gate, the QC/Stage crons, or the Feature/Hotfix flows.
|
|
29
|
+
ClickUp-driven QA/QC review + promotion chain (Dev→QA→QC→Client Review→Stage→Prod) for worker2; open when touching a review gate, the QC/Stage crons, or the Feature/Hotfix flows. Live in prod on the Dev Sprints folder.
|
|
30
30
|
|
|
31
31
|
## Summary
|
|
32
32
|
|
|
33
33
|
The ClickUp-driven review + promotion chain for the platform-wide QA/QC dev process (per the approved
|
|
34
34
|
`QA-QC Plan.txt`), moving a task through **Development → QA → QC → (Client Review) → Stage →
|
|
35
|
-
Production**. Several
|
|
36
|
-
|
|
35
|
+
Production**. Several workers plus entry points in the shared ClickUp webhook handler. **Live in
|
|
36
|
+
production** (deployed on `_production`).
|
|
37
37
|
|
|
38
38
|
- `_Worker_Clickup_QaReview` — the UI → FE → BE review chain (branches by Task Type; Hotfix BE-approve
|
|
39
39
|
jumps straight to QC-Hotfix).
|
|
@@ -47,9 +47,10 @@ deployed yet** — on the `feature-qa-qc` branch working tree.
|
|
|
47
47
|
- `_Worker_Clickup_BeReview` — **auto** BE-approve driven by GitHub PR reviews (removes the manual
|
|
48
48
|
BE-review step).
|
|
49
49
|
|
|
50
|
-
**Scope gate —
|
|
51
|
-
**"(TEST) True Dev Sprints"
|
|
52
|
-
`
|
|
50
|
+
**Scope gate — Dev Sprints folder only.** `_Worker_Clickup::QA_QC_FOLDER_IDS` = `['90116585845']`
|
|
51
|
+
(**Dev Sprints**, the live sprint folder; formerly named "(TEST) True Dev Sprints"). The retired True
|
|
52
|
+
Dev Sprints folder `90020432579` was removed 2026-09-25. `isTaskInQaQcScope` matches the home folder
|
|
53
|
+
or any list the task is in (Tasks-in-Multiple-Lists).
|
|
53
54
|
|
|
54
55
|
When a review is **denied**, the task is routed to **Rework** and its code changes are **reverted** from
|
|
55
56
|
the QA/QC branch — see [QA/QC branch automation](./qa-qc-branch-automation.md) for how the revert
|
|
@@ -138,13 +139,17 @@ when a task *left* a status; the QA/QC entry must be on entering the status.
|
|
|
138
139
|
- **The unit-test gate is a stub (`runUnitTests()` returns true).** It does not block anything yet.
|
|
139
140
|
- **Custom-field option VALUES are resolved at runtime, IDs are constants** — see
|
|
140
141
|
[ClickUp custom-field conventions](./clickup-custom-field-conventions.md).
|
|
141
|
-
- **The automation only fires inside
|
|
142
|
-
the fan-out, `handleDevelopmentFinished`,
|
|
143
|
-
verify the folder before expecting the flow to run.
|
|
142
|
+
- **The automation only fires inside `QA_QC_FOLDER_IDS` (Dev Sprints).** `isTaskInQaQcScope` gates
|
|
143
|
+
the fan-out, `handleDevelopmentFinished`, `QcBatch`, and the taskCreated Work Type default. A task
|
|
144
|
+
outside that folder is ignored — verify the folder before expecting the flow to run.
|
|
145
|
+
- **Dev Finished with neither Has UI Changes nor Has Code Changes checked → `qa approved` (skips
|
|
146
|
+
review).** Intended (jcardinal confirmed); do not "fix".
|
|
144
147
|
- **A Feature advances only when ALL subtasks clear a gate** — a single lagging subtask holds the whole
|
|
145
148
|
feature. Subtasks carry no code; only the feature branch does.
|
|
146
|
-
- **`_underscore CLICK_UP_CUSTOM_FIELD_ID__CLIENT_REVIEW` was repointed** to the
|
|
147
|
-
|
|
149
|
+
- **`_underscore CLICK_UP_CUSTOM_FIELD_ID__CLIENT_REVIEW` was repointed** to the Dev Sprints
|
|
150
|
+
folder's field (`2c4ab51a-…`); all field ids were verified against that folder.
|
|
151
|
+
- **The automation user needs edit rights on every field it writes** — a missing field permission
|
|
152
|
+
returns HTTP 401 `ACCESS_606`. See [credential rotation](../workflows/clickup-credential-rotation.md).
|
|
148
153
|
|
|
149
154
|
### Known security gaps (discovered, not yet fixed — separate tickets)
|
|
150
155
|
|
|
@@ -6,7 +6,7 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
9
|
+
updated: 2026-09-25
|
|
10
10
|
owners: ["jcardinal", "kyalamarthi"]
|
|
11
11
|
files:
|
|
12
12
|
- worker2/Worker/Team/Sprint.php
|
|
@@ -355,8 +355,12 @@ see [TOGa IQ Sprint Dashboard API](../../api2/features/sprint-dashboard-api.md)
|
|
|
355
355
|
- **`SprintLock` is NOT a safe way to repopulate empty tables.** It captures before it reads, so it
|
|
356
356
|
looks like a bootstrap (its `getSprint()` call is gated behind `$generateSprintLockReport`), but
|
|
357
357
|
its capture block (~L4016-4030) **writes back to ClickUp for every task** — sets Previous Sprint
|
|
358
|
-
Work Type
|
|
359
|
-
|
|
358
|
+
Work Type and sets "Added at sprint launch" = true. Run outside the real launch ceremony it
|
|
359
|
+
corrupts those fields across the whole sprint.
|
|
360
|
+
- **The ClickUp "Condition" field (On Track) is retired (2026-09-25).** Not in the QA/QC plan, not in
|
|
361
|
+
the Dev Sprints folder, never read by code. The `SprintLock` write was removed (it would fail on the
|
|
362
|
+
new folder). `CLICK_UP_CUSTOM_FIELD_ID__CONDITION` / `CONDITION__ON_TRACK` still exist in
|
|
363
|
+
`_underscore` but are unused — do not add new writes.
|
|
360
364
|
- **`SprintDaily` returns nothing on success** — a production run reads `Successfully Executed
|
|
361
365
|
(4.255 seconds): <no return message>`, so a run that worked is indistinguishable from one that
|
|
362
366
|
never fired. Breaks the return-on-both-paths rule in the
|
|
@@ -6,13 +6,12 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: workflow
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
10
|
-
owners: ["ajean"]
|
|
9
|
+
updated: 2026-09-25
|
|
10
|
+
owners: ["ajean", "jcardinal"]
|
|
11
11
|
files:
|
|
12
12
|
- _underscore/Component/Api/Clickup/Clickup.php
|
|
13
13
|
- worker2/Worker/Team/Sprint.php
|
|
14
14
|
- worker2/Worker/Clickup/Project.php
|
|
15
|
-
- library/app/systemmonitor/500error.php
|
|
16
15
|
- library/app/systemmonitor/netsuiteintegration.php
|
|
17
16
|
- library/app/systemmonitor/emailspam.php
|
|
18
17
|
related:
|
|
@@ -50,7 +49,6 @@ breaking the legacy 1.0 crons**. Two facts make this a procedure rather than a c
|
|
|
50
49
|
| `worker2` | `Worker/Team/Sprint.php` |
|
|
51
50
|
| `worker2` | `Config/production.ini` → `[clickup] token` |
|
|
52
51
|
| `worker2` | a dev `.ini` (not committed) |
|
|
53
|
-
| `library` | `app/systemmonitor/500error.php` |
|
|
54
52
|
| `library` | `app/systemmonitor/netsuiteintegration.php` |
|
|
55
53
|
|
|
56
54
|
**Token B — a second, OLDER token.** Easy to miss, and the reason a "complete" rotation still breaks
|
|
@@ -109,12 +107,20 @@ is a live correctness bug in the actor guard, independent of any rotation.
|
|
|
109
107
|
`'12633140'` to `'87469444'`. See the correctness note above — this step is not cosmetic.
|
|
110
108
|
7. Replace the token in every Token A location, and Token B in `library/app/systemmonitor/emailspam.php`
|
|
111
109
|
plus the three `worker/crons/infrastructure/*` crons.
|
|
112
|
-
8.
|
|
113
|
-
|
|
114
|
-
|
|
110
|
+
8. In ClickUp, give the new account **edit rights on every custom field the automation writes**
|
|
111
|
+
(Work Type, Previous Sprint Work Type, Environment, Final Design Outcome, the QA/QC review fields,
|
|
112
|
+
and so on). Space/list access is not enough — fields have their own permissions.
|
|
113
|
+
9. Deploy worker2, and deploy/verify the 1.0 `library` + `worker` crons.
|
|
114
|
+
10. Only now, with the **original** token, delete the old webhooks recorded in step 2.
|
|
115
|
+
11. Revoke the old token(s).
|
|
115
116
|
|
|
116
117
|
## Gotchas
|
|
117
118
|
|
|
119
|
+
- **Field-level edit permission is separate from space/list access.** After the automation moved to
|
|
120
|
+
user 87469444 (2026-09-21), it could not edit Previous Sprint Work Type / Environment → HTTP 401
|
|
121
|
+
`[ACCESS_606] You do not have permission to do this action` → every `taskCreated` job failed for 4
|
|
122
|
+
days. Check field permissions for every field the automation writes whenever the account changes.
|
|
123
|
+
Code side: [work type automation](../features/clickup-work-type-automation.md) (`setFieldSafely`).
|
|
118
124
|
- **The hardcoded token in `_underscore/Component/Api/Clickup/Clickup.php` is in committed git
|
|
119
125
|
history.** Rotating the value does not remove it from history — treat the old token as compromised and
|
|
120
126
|
revoke it, do not merely stop using it.
|
package/package.json
CHANGED