@checkstack/healthcheck-backend 1.1.3 → 1.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,104 @@
1
1
  # @checkstack/healthcheck-backend
2
2
 
3
+ ## 1.2.0
4
+
5
+ ### Minor Changes
6
+
7
+ - ba07ae2: Quiet down notification spam on flapping systems, auto-open incidents when a check goes critical, and let operators land directly on the broken checks.
8
+
9
+ Notification policy lives **per healthcheck assignment** (one row per `system × configuration`). Different checks on the same system are fully independent — disabling a setting on one check does not affect the others. Defaults preserve existing behaviour for `suppressDeEscalations`; **auto-incident defaults to on** for new and existing assignments.
10
+
11
+ - **`suppressDeEscalations`** (off by default). When on, transitions from a worse state to a better-but-still-failing state (e.g. `unhealthy → degraded`) no longer fire a notification. Escalations and full recoveries to `healthy` are unaffected. Resolved per assignment (the just-ran check is the one driving any aggregate transition).
12
+ - **`autoOpenIncidentOnUnhealthy`** (on by default). Either of two independent triggers can open the auto-incident:
13
+ - **`sustainedUnhealthyTrigger`** (default 30 min) — opens when the check stays continuously unhealthy for the configured duration. Catches real outages.
14
+ - **`flappingTrigger`** (default 3 transitions in 60 min) — opens when the check flips to unhealthy that many times in the window. Catches persistent flapping where each unhealthy phase is too brief for the sustained trigger.
15
+ Each trigger can be individually disabled. One incident per system: triggering checks attach to an existing active auto-incident.
16
+ - **`useNotificationSuppression`** (on by default, only meaningful when auto-open is on). Controls whether the auto-opened incident is created with `suppressNotifications: true` — leaving this off opens the incident but still pings operators on each transition.
17
+ - **`skipDuringMaintenance`** (on by default). No auto-incident is opened while the system has an active maintenance window with suppression. The system is intentionally down and shouldn't trip the on-call.
18
+ - **`autoCloseAfterMinutes`** (default 30). Auto-close cooldown is now per-assignment and snapshotted per-incident at open time — later policy edits don't alter in-flight incidents. Setting `null` ("Never auto-close") leaves the incident for manual resolution.
19
+ - **Require-recovery rule.** After any auto-incident closes (manual or auto), no new auto-incident can open until the check has logged at least one healthy run. Prevents a "operator dismissed but it's still broken" loop.
20
+ - **Auto-close worker** ticks every 60s and resolves auto-opened incidents whose systems have been healthy for their per-row `cooldownMinutes`. Rows with `null` cooldown are skipped entirely. Per-incident: failed close attempts are logged but never abort the sweep.
21
+ - **`incidentResolved` hook subscriber** syncs the auto-incident mapping when an operator manually resolves the incident, so the require-recovery rule sees the close immediately.
22
+ - **Platform-wide defaults.** New admin RPCs `getPlatformNotificationDefaults` / `setPlatformNotificationDefaults` (under the existing `healthcheck.configuration.{read,manage}` access rules) let operators set notification policy once for the whole instance. Per-assignment rows with `notificationPolicy: null` inherit the platform defaults at read time. UI: a "Notification defaults" button in the Assignment IDE opens a modal editor. The per-assignment Notifications panel shows an inheritance banner — "Using platform defaults" (read-only) with an "Override" button, or "Custom override" with a "Use platform defaults" button to revert. The all-or-nothing model keeps the mental model simple: each assignment is either fully inherited or fully overridden.
23
+ - **New service-level RPCs** on the incident plugin (`createAutoIncident`, `resolveAutoIncident`) let other plugins open/close incidents without a user context. Reused by the healthcheck auto-incident flow.
24
+ - **Health-state notification CTA** now deep-links to `?filter=failing` on the system detail page for non-recovery transitions (label changes to "View failing checks"). The system overview gains an `All / Failing / Healthy` segmented filter wired to the same `?filter=…` param.
25
+ - **Notification bell badge** now counts collapse groups instead of raw rows, so the number matches what the user sees in the notifications list. Built on `COUNT(DISTINCT COALESCE(collapse_key, id))` — notifications without a collapse key still each count as one.
26
+ - **`statusFilter` on `getHistory` / `getDetailedHistory`** lets the run-history page and the drawer's Recent Runs panel filter to `All / Healthy / Failing` via shared pills, with the page resetting to the first page on filter change.
27
+ - **Pagination defaults aligned with selector options.** Several pages defaulted to a page size (5 or 20) that wasn't in the dropdown's options (`[10, 25, 50, 100]`), so the page-size `<Select>` rendered empty. The drawer's Recent Runs now defaults to 10; the Run History, History List, and Delivery Logs pages now default to 25.
28
+
29
+ Includes Drizzle migrations adding the `notification_policy` jsonb column to `system_health_checks`, plus two new tables: `health_check_unhealthy_transitions` (for threshold counting) and `health_check_auto_incidents` (for mapping back to incident ids during auto-close).
30
+
31
+ ### Patch Changes
32
+
33
+ - Updated dependencies [ba07ae2]
34
+ - @checkstack/healthcheck-common@1.2.0
35
+ - @checkstack/incident-common@1.3.0
36
+ - @checkstack/incident-backend@1.2.0
37
+ - @checkstack/backend-api@0.17.1
38
+ - @checkstack/satellite-backend@0.3.6
39
+ - @checkstack/cache-api@0.3.5
40
+ - @checkstack/catalog-backend@1.1.6
41
+ - @checkstack/command-backend@0.1.30
42
+ - @checkstack/gitops-backend@0.3.6
43
+ - @checkstack/integration-backend@0.1.30
44
+ - @checkstack/queue-api@0.3.5
45
+ - @checkstack/cache-utils@0.2.10
46
+
47
+ ## 1.1.4
48
+
49
+ ### Patch Changes
50
+
51
+ - f23f3c9: Add `correlationMiddleware` to `@checkstack/backend-api` and apply it
52
+ to every plugin/core router so each request carries a stable
53
+ `x-correlation-id` (read from the inbound header, or freshly minted
54
+ via `crypto.randomUUID()` when absent) and an auto-injected child
55
+ logger bound with `{ correlationId, pluginId, userId? }`. The ID is
56
+ echoed back on the response header so the caller can correlate their
57
+ client-side trace to the server logs.
58
+
59
+ The `Logger` interface in `@checkstack/backend-api` now formally
60
+ documents the structured-metadata convention (`logger.info("msg",
61
+ { ...meta })`) alongside the long-standing varargs shape. Winston's
62
+ splat handling already routes both shapes through the same vararg
63
+ slot, so existing call sites are unaffected. A new optional
64
+ `Logger.child(meta)` method captures the metadata-binding contract the
65
+ new middleware relies on; production loggers always implement it,
66
+ minimal test mocks may omit it (the middleware falls back gracefully).
67
+
68
+ `RpcContext` grew two optional `Headers` bags, `requestHeaders` and
69
+ `responseHeaders`, populated by the outer Hono `/api/*` and `/rest/*`
70
+ handlers in `@checkstack/backend`. They are write-through observation
71
+ points for middleware; an `RpcContext` constructed without them (S2S
72
+ clients, tests) keeps working — the echo is a silent no-op and the ID
73
+ is still bound onto the child logger for server-side correlation.
74
+
75
+ The scaffolding template in `@checkstack/scripts` was updated so any
76
+ new plugin generated via `bun run create` wires the middleware in the
77
+ expected `.use(correlationMiddleware).use(autoAuthMiddleware)` order
78
+ out of the box.
79
+
80
+ - Updated dependencies [f23f3c9]
81
+ - Updated dependencies [f23f3c9]
82
+ - Updated dependencies [f23f3c9]
83
+ - Updated dependencies [f23f3c9]
84
+ - @checkstack/common@0.11.0
85
+ - @checkstack/backend-api@0.17.0
86
+ - @checkstack/catalog-backend@1.1.5
87
+ - @checkstack/command-backend@0.1.29
88
+ - @checkstack/gitops-backend@0.3.5
89
+ - @checkstack/integration-backend@0.1.29
90
+ - @checkstack/satellite-backend@0.3.5
91
+ - @checkstack/notification-common@1.2.0
92
+ - @checkstack/catalog-common@2.2.2
93
+ - @checkstack/gitops-common@0.4.1
94
+ - @checkstack/healthcheck-common@1.1.2
95
+ - @checkstack/incident-common@1.2.2
96
+ - @checkstack/maintenance-common@1.2.2
97
+ - @checkstack/signal-common@0.2.4
98
+ - @checkstack/cache-api@0.3.4
99
+ - @checkstack/queue-api@0.3.4
100
+ - @checkstack/cache-utils@0.2.9
101
+
3
102
  ## 1.1.3
4
103
 
5
104
  ### Patch Changes
@@ -0,0 +1 @@
1
+ ALTER TABLE "system_health_checks" ADD COLUMN "notification_policy" jsonb;
@@ -0,0 +1,20 @@
1
+ CREATE TABLE "health_check_auto_incidents" (
2
+ "id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
3
+ "incident_id" uuid NOT NULL,
4
+ "system_id" text NOT NULL,
5
+ "configuration_id" uuid NOT NULL,
6
+ "opened_at" timestamp DEFAULT now() NOT NULL,
7
+ "closed_at" timestamp
8
+ );
9
+ --> statement-breakpoint
10
+ CREATE TABLE "health_check_unhealthy_transitions" (
11
+ "id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
12
+ "configuration_id" uuid NOT NULL,
13
+ "system_id" text NOT NULL,
14
+ "transitioned_at" timestamp DEFAULT now() NOT NULL
15
+ );
16
+ --> statement-breakpoint
17
+ ALTER TABLE "health_check_auto_incidents" ADD CONSTRAINT "health_check_auto_incidents_configuration_id_health_check_configurations_id_fk" FOREIGN KEY ("configuration_id") REFERENCES "health_check_configurations"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
18
+ ALTER TABLE "health_check_unhealthy_transitions" ADD CONSTRAINT "health_check_unhealthy_transitions_configuration_id_health_check_configurations_id_fk" FOREIGN KEY ("configuration_id") REFERENCES "health_check_configurations"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
19
+ CREATE INDEX "health_check_auto_incidents_active_by_system_idx" ON "health_check_auto_incidents" USING btree ("system_id","closed_at");--> statement-breakpoint
20
+ CREATE INDEX "health_check_unhealthy_transitions_lookup_idx" ON "health_check_unhealthy_transitions" USING btree ("configuration_id","system_id","transitioned_at");
@@ -0,0 +1,2 @@
1
+ ALTER TABLE "health_check_auto_incidents" ADD COLUMN "cooldown_minutes" integer;--> statement-breakpoint
2
+ CREATE INDEX "health_check_auto_incidents_last_close_idx" ON "health_check_auto_incidents" USING btree ("configuration_id","system_id","closed_at");
@@ -0,0 +1,447 @@
1
+ {
2
+ "id": "7fabe9e0-a078-43fa-aa33-548b1f6f7ee9",
3
+ "prevId": "446eaec3-7894-4425-b7db-00c7761ca83f",
4
+ "version": "7",
5
+ "dialect": "postgresql",
6
+ "tables": {
7
+ "public.health_check_aggregates": {
8
+ "name": "health_check_aggregates",
9
+ "schema": "",
10
+ "columns": {
11
+ "id": {
12
+ "name": "id",
13
+ "type": "uuid",
14
+ "primaryKey": true,
15
+ "notNull": true,
16
+ "default": "gen_random_uuid()"
17
+ },
18
+ "configuration_id": {
19
+ "name": "configuration_id",
20
+ "type": "uuid",
21
+ "primaryKey": false,
22
+ "notNull": true
23
+ },
24
+ "system_id": {
25
+ "name": "system_id",
26
+ "type": "text",
27
+ "primaryKey": false,
28
+ "notNull": true
29
+ },
30
+ "bucket_start": {
31
+ "name": "bucket_start",
32
+ "type": "timestamp",
33
+ "primaryKey": false,
34
+ "notNull": true
35
+ },
36
+ "bucket_size": {
37
+ "name": "bucket_size",
38
+ "type": "bucket_size",
39
+ "typeSchema": "public",
40
+ "primaryKey": false,
41
+ "notNull": true
42
+ },
43
+ "run_count": {
44
+ "name": "run_count",
45
+ "type": "integer",
46
+ "primaryKey": false,
47
+ "notNull": true
48
+ },
49
+ "healthy_count": {
50
+ "name": "healthy_count",
51
+ "type": "integer",
52
+ "primaryKey": false,
53
+ "notNull": true
54
+ },
55
+ "degraded_count": {
56
+ "name": "degraded_count",
57
+ "type": "integer",
58
+ "primaryKey": false,
59
+ "notNull": true
60
+ },
61
+ "unhealthy_count": {
62
+ "name": "unhealthy_count",
63
+ "type": "integer",
64
+ "primaryKey": false,
65
+ "notNull": true
66
+ },
67
+ "latency_sum_ms": {
68
+ "name": "latency_sum_ms",
69
+ "type": "integer",
70
+ "primaryKey": false,
71
+ "notNull": false
72
+ },
73
+ "avg_latency_ms": {
74
+ "name": "avg_latency_ms",
75
+ "type": "integer",
76
+ "primaryKey": false,
77
+ "notNull": false
78
+ },
79
+ "min_latency_ms": {
80
+ "name": "min_latency_ms",
81
+ "type": "integer",
82
+ "primaryKey": false,
83
+ "notNull": false
84
+ },
85
+ "max_latency_ms": {
86
+ "name": "max_latency_ms",
87
+ "type": "integer",
88
+ "primaryKey": false,
89
+ "notNull": false
90
+ },
91
+ "p95_latency_ms": {
92
+ "name": "p95_latency_ms",
93
+ "type": "integer",
94
+ "primaryKey": false,
95
+ "notNull": false
96
+ },
97
+ "aggregated_result": {
98
+ "name": "aggregated_result",
99
+ "type": "jsonb",
100
+ "primaryKey": false,
101
+ "notNull": false
102
+ },
103
+ "tdigest_state": {
104
+ "name": "tdigest_state",
105
+ "type": "jsonb",
106
+ "primaryKey": false,
107
+ "notNull": false
108
+ },
109
+ "source_id": {
110
+ "name": "source_id",
111
+ "type": "text",
112
+ "primaryKey": false,
113
+ "notNull": false
114
+ },
115
+ "source_label": {
116
+ "name": "source_label",
117
+ "type": "text",
118
+ "primaryKey": false,
119
+ "notNull": false
120
+ }
121
+ },
122
+ "indexes": {},
123
+ "foreignKeys": {
124
+ "health_check_aggregates_configuration_id_health_check_configurations_id_fk": {
125
+ "name": "health_check_aggregates_configuration_id_health_check_configurations_id_fk",
126
+ "tableFrom": "health_check_aggregates",
127
+ "tableTo": "health_check_configurations",
128
+ "columnsFrom": [
129
+ "configuration_id"
130
+ ],
131
+ "columnsTo": [
132
+ "id"
133
+ ],
134
+ "onDelete": "cascade",
135
+ "onUpdate": "no action"
136
+ }
137
+ },
138
+ "compositePrimaryKeys": {},
139
+ "uniqueConstraints": {
140
+ "health_check_aggregates_bucket_unique": {
141
+ "name": "health_check_aggregates_bucket_unique",
142
+ "nullsNotDistinct": true,
143
+ "columns": [
144
+ "configuration_id",
145
+ "system_id",
146
+ "bucket_start",
147
+ "bucket_size",
148
+ "source_id"
149
+ ]
150
+ }
151
+ },
152
+ "policies": {},
153
+ "checkConstraints": {},
154
+ "isRLSEnabled": false
155
+ },
156
+ "public.health_check_configurations": {
157
+ "name": "health_check_configurations",
158
+ "schema": "",
159
+ "columns": {
160
+ "id": {
161
+ "name": "id",
162
+ "type": "uuid",
163
+ "primaryKey": true,
164
+ "notNull": true,
165
+ "default": "gen_random_uuid()"
166
+ },
167
+ "name": {
168
+ "name": "name",
169
+ "type": "text",
170
+ "primaryKey": false,
171
+ "notNull": true
172
+ },
173
+ "strategy_id": {
174
+ "name": "strategy_id",
175
+ "type": "text",
176
+ "primaryKey": false,
177
+ "notNull": true
178
+ },
179
+ "config": {
180
+ "name": "config",
181
+ "type": "jsonb",
182
+ "primaryKey": false,
183
+ "notNull": true
184
+ },
185
+ "collectors": {
186
+ "name": "collectors",
187
+ "type": "jsonb",
188
+ "primaryKey": false,
189
+ "notNull": false
190
+ },
191
+ "interval_seconds": {
192
+ "name": "interval_seconds",
193
+ "type": "integer",
194
+ "primaryKey": false,
195
+ "notNull": true
196
+ },
197
+ "is_template": {
198
+ "name": "is_template",
199
+ "type": "boolean",
200
+ "primaryKey": false,
201
+ "notNull": false,
202
+ "default": false
203
+ },
204
+ "paused": {
205
+ "name": "paused",
206
+ "type": "boolean",
207
+ "primaryKey": false,
208
+ "notNull": true,
209
+ "default": false
210
+ },
211
+ "created_at": {
212
+ "name": "created_at",
213
+ "type": "timestamp",
214
+ "primaryKey": false,
215
+ "notNull": true,
216
+ "default": "now()"
217
+ },
218
+ "updated_at": {
219
+ "name": "updated_at",
220
+ "type": "timestamp",
221
+ "primaryKey": false,
222
+ "notNull": true,
223
+ "default": "now()"
224
+ }
225
+ },
226
+ "indexes": {},
227
+ "foreignKeys": {},
228
+ "compositePrimaryKeys": {},
229
+ "uniqueConstraints": {},
230
+ "policies": {},
231
+ "checkConstraints": {},
232
+ "isRLSEnabled": false
233
+ },
234
+ "public.health_check_runs": {
235
+ "name": "health_check_runs",
236
+ "schema": "",
237
+ "columns": {
238
+ "id": {
239
+ "name": "id",
240
+ "type": "uuid",
241
+ "primaryKey": true,
242
+ "notNull": true,
243
+ "default": "gen_random_uuid()"
244
+ },
245
+ "configuration_id": {
246
+ "name": "configuration_id",
247
+ "type": "uuid",
248
+ "primaryKey": false,
249
+ "notNull": true
250
+ },
251
+ "system_id": {
252
+ "name": "system_id",
253
+ "type": "text",
254
+ "primaryKey": false,
255
+ "notNull": true
256
+ },
257
+ "status": {
258
+ "name": "status",
259
+ "type": "health_check_status",
260
+ "typeSchema": "public",
261
+ "primaryKey": false,
262
+ "notNull": true
263
+ },
264
+ "latency_ms": {
265
+ "name": "latency_ms",
266
+ "type": "integer",
267
+ "primaryKey": false,
268
+ "notNull": false
269
+ },
270
+ "result": {
271
+ "name": "result",
272
+ "type": "jsonb",
273
+ "primaryKey": false,
274
+ "notNull": false
275
+ },
276
+ "source_id": {
277
+ "name": "source_id",
278
+ "type": "text",
279
+ "primaryKey": false,
280
+ "notNull": false
281
+ },
282
+ "source_label": {
283
+ "name": "source_label",
284
+ "type": "text",
285
+ "primaryKey": false,
286
+ "notNull": false
287
+ },
288
+ "timestamp": {
289
+ "name": "timestamp",
290
+ "type": "timestamp",
291
+ "primaryKey": false,
292
+ "notNull": true,
293
+ "default": "now()"
294
+ }
295
+ },
296
+ "indexes": {},
297
+ "foreignKeys": {
298
+ "health_check_runs_configuration_id_health_check_configurations_id_fk": {
299
+ "name": "health_check_runs_configuration_id_health_check_configurations_id_fk",
300
+ "tableFrom": "health_check_runs",
301
+ "tableTo": "health_check_configurations",
302
+ "columnsFrom": [
303
+ "configuration_id"
304
+ ],
305
+ "columnsTo": [
306
+ "id"
307
+ ],
308
+ "onDelete": "cascade",
309
+ "onUpdate": "no action"
310
+ }
311
+ },
312
+ "compositePrimaryKeys": {},
313
+ "uniqueConstraints": {},
314
+ "policies": {},
315
+ "checkConstraints": {},
316
+ "isRLSEnabled": false
317
+ },
318
+ "public.system_health_checks": {
319
+ "name": "system_health_checks",
320
+ "schema": "",
321
+ "columns": {
322
+ "system_id": {
323
+ "name": "system_id",
324
+ "type": "text",
325
+ "primaryKey": false,
326
+ "notNull": true
327
+ },
328
+ "configuration_id": {
329
+ "name": "configuration_id",
330
+ "type": "uuid",
331
+ "primaryKey": false,
332
+ "notNull": true
333
+ },
334
+ "enabled": {
335
+ "name": "enabled",
336
+ "type": "boolean",
337
+ "primaryKey": false,
338
+ "notNull": true,
339
+ "default": true
340
+ },
341
+ "state_thresholds": {
342
+ "name": "state_thresholds",
343
+ "type": "jsonb",
344
+ "primaryKey": false,
345
+ "notNull": false
346
+ },
347
+ "retention_config": {
348
+ "name": "retention_config",
349
+ "type": "jsonb",
350
+ "primaryKey": false,
351
+ "notNull": false
352
+ },
353
+ "satellite_ids": {
354
+ "name": "satellite_ids",
355
+ "type": "jsonb",
356
+ "primaryKey": false,
357
+ "notNull": false
358
+ },
359
+ "include_local": {
360
+ "name": "include_local",
361
+ "type": "boolean",
362
+ "primaryKey": false,
363
+ "notNull": true,
364
+ "default": true
365
+ },
366
+ "notification_policy": {
367
+ "name": "notification_policy",
368
+ "type": "jsonb",
369
+ "primaryKey": false,
370
+ "notNull": false
371
+ },
372
+ "created_at": {
373
+ "name": "created_at",
374
+ "type": "timestamp",
375
+ "primaryKey": false,
376
+ "notNull": true,
377
+ "default": "now()"
378
+ },
379
+ "updated_at": {
380
+ "name": "updated_at",
381
+ "type": "timestamp",
382
+ "primaryKey": false,
383
+ "notNull": true,
384
+ "default": "now()"
385
+ }
386
+ },
387
+ "indexes": {},
388
+ "foreignKeys": {
389
+ "system_health_checks_configuration_id_health_check_configurations_id_fk": {
390
+ "name": "system_health_checks_configuration_id_health_check_configurations_id_fk",
391
+ "tableFrom": "system_health_checks",
392
+ "tableTo": "health_check_configurations",
393
+ "columnsFrom": [
394
+ "configuration_id"
395
+ ],
396
+ "columnsTo": [
397
+ "id"
398
+ ],
399
+ "onDelete": "cascade",
400
+ "onUpdate": "no action"
401
+ }
402
+ },
403
+ "compositePrimaryKeys": {
404
+ "system_health_checks_system_id_configuration_id_pk": {
405
+ "name": "system_health_checks_system_id_configuration_id_pk",
406
+ "columns": [
407
+ "system_id",
408
+ "configuration_id"
409
+ ]
410
+ }
411
+ },
412
+ "uniqueConstraints": {},
413
+ "policies": {},
414
+ "checkConstraints": {},
415
+ "isRLSEnabled": false
416
+ }
417
+ },
418
+ "enums": {
419
+ "public.bucket_size": {
420
+ "name": "bucket_size",
421
+ "schema": "public",
422
+ "values": [
423
+ "hourly",
424
+ "daily"
425
+ ]
426
+ },
427
+ "public.health_check_status": {
428
+ "name": "health_check_status",
429
+ "schema": "public",
430
+ "values": [
431
+ "healthy",
432
+ "unhealthy",
433
+ "degraded"
434
+ ]
435
+ }
436
+ },
437
+ "schemas": {},
438
+ "sequences": {},
439
+ "roles": {},
440
+ "policies": {},
441
+ "views": {},
442
+ "_meta": {
443
+ "columns": {},
444
+ "schemas": {},
445
+ "tables": {}
446
+ }
447
+ }