@checkstack/healthcheck-backend 1.11.1 → 1.12.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +270 -0
- package/package.json +29 -29
- package/src/automations.test.ts +43 -1
- package/src/automations.ts +22 -3
- package/src/hooks.ts +7 -0
- package/src/index.ts +21 -4
- package/src/queue-executor.test.ts +108 -16
- package/src/queue-executor.ts +116 -9
- package/src/router-pause-recompute.test.ts +142 -0
- package/src/router.ts +47 -0
- package/src/service-env-filter.test.ts +299 -0
- package/src/service-paused-filter.test.ts +391 -0
- package/src/service-rollup-worst-wins.test.ts +205 -0
- package/src/service.ts +274 -28
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,275 @@
|
|
|
1
1
|
# @checkstack/healthcheck-backend
|
|
2
2
|
|
|
3
|
+
## 1.12.0
|
|
4
|
+
|
|
5
|
+
### Minor Changes
|
|
6
|
+
|
|
7
|
+
- 52c55bf: Anomaly baselines are now per-environment, so the env-scoped
|
|
8
|
+
`HealthCheckDrawer` shows the clicked env's baseline (not a cross-env
|
|
9
|
+
one). Closes the follow-up noted in `healthcheck-per-env-rollup`.
|
|
10
|
+
|
|
11
|
+
## What changed
|
|
12
|
+
|
|
13
|
+
- **`anomaly_baselines`** now carries a nullable `environment_id`
|
|
14
|
+
column, and its unique constraint grew to
|
|
15
|
+
`(systemId, configurationId, environmentId, fieldPath)` with
|
|
16
|
+
`NULLS NOT DISTINCT` — so there is exactly one baseline per
|
|
17
|
+
`(system, config, env, path)` tuple, and the env-less slice (`NULL`)
|
|
18
|
+
stays a single row (the pre-feature cross-env baseline, preserved as
|
|
19
|
+
the env-less row until the next analyzer tick rewrites per-env rows).
|
|
20
|
+
Existing rows backfill to `environment_id = NULL` with no data work.
|
|
21
|
+
- **Baseline analyzer** (`jobs/baseline-analyzer.ts`) now fans out per
|
|
22
|
+
environment within each assignment: runs are grouped by
|
|
23
|
+
`environmentId` (null = env-less), stats are computed per env, and
|
|
24
|
+
the upsert targets the 4-tuple. The cache key gained an env segment
|
|
25
|
+
(`baseline:${config}:${system}:${env ?? "<none>"}:${path}`) and the
|
|
26
|
+
`ANOMALY_BASELINE_UPDATED` signal payload now carries `environmentId`.
|
|
27
|
+
Previously the analyzer computed one cross-env batch per assignment.
|
|
28
|
+
- **Inline detector** (`detector.ts`) resolves the per-env baseline:
|
|
29
|
+
the lookup matches `environmentId` when present or `IS NULL` for the
|
|
30
|
+
env-less slice, and the cache key matches the analyzer's env segment.
|
|
31
|
+
`environmentId` is threaded from the `checkCompleted` hook (see
|
|
32
|
+
below); it defaults to `null` (env-less) so a caller that omits it
|
|
33
|
+
resolves the env-less baseline rather than failing.
|
|
34
|
+
- **`getAnomalyBaselines` RPC** now accepts an optional
|
|
35
|
+
`environmentId: string | null` filter and surfaces `environmentId` on
|
|
36
|
+
every `AnomalyBaselineDto`. Tristate semantics, mirroring
|
|
37
|
+
`getHistory`: `undefined` → all envs (no predicate), `null` → env-less
|
|
38
|
+
slice (`IS NULL`), a string → that env. The service predicate is at
|
|
39
|
+
the DB layer.
|
|
40
|
+
- **`HealthCheckDrawer`** threads `item.environmentId` (already on its
|
|
41
|
+
props) into the baselines query, so the drawer's anomaly overlay
|
|
42
|
+
resolves server-side to the clicked env's baseline only — matching the
|
|
43
|
+
env-scoping already applied to its history table and charts. The
|
|
44
|
+
latency chart tolerates the new field (it picks the single
|
|
45
|
+
`"latencyMs"` baseline, which the env filter guarantees is unique).
|
|
46
|
+
- **`getRunsForAnalysis`** (healthcheck) now returns `environmentId`
|
|
47
|
+
on each run so the analyzer can group by env. Additive optional
|
|
48
|
+
field; only the analyzer consumes it.
|
|
49
|
+
- **`checkCompleted` / `checkFailed` hooks** (healthcheck) now carry
|
|
50
|
+
`environmentId: string | null` on their payloads, sourced from the
|
|
51
|
+
per-env execution loop. Only the anomaly detector subscribes to
|
|
52
|
+
`checkCompleted` (it was updated); the failure-path emit (rollup
|
|
53
|
+
error) passes `null`.
|
|
54
|
+
|
|
55
|
+
## Notes
|
|
56
|
+
|
|
57
|
+
- Anomaly _rows_ (`anomalies` table) remain cross-env by design in this
|
|
58
|
+
step — only baselines are env-scoped, matching the scoped task. A
|
|
59
|
+
detector run for env A and env B's normal value still share one
|
|
60
|
+
`(system, config, path)` anomaly row; env-scoping the anomalies table
|
|
61
|
+
is tracked as a separate follow-up so this change stays focused on
|
|
62
|
+
the drawer's baseline overlay.
|
|
63
|
+
- The `checkCompleted` / `checkFailed` payload change is technically
|
|
64
|
+
breaking for hook subscribers that destructure the payload, but the
|
|
65
|
+
only in-tree subscriber (the anomaly plugin) was updated in lockstep.
|
|
66
|
+
External webhook subscribers receive an additional field and are not
|
|
67
|
+
affected unless they reject unknown keys (uncommon).
|
|
68
|
+
- Migration `0006_sad_retro_girl.sql` drops + recreates the unique
|
|
69
|
+
constraint with `NULLS NOT DISTINCT` and adds the column. It applies
|
|
70
|
+
cleanly to fresh and already-populated DBs (existing NULL-env rows
|
|
71
|
+
remain unique under the new key).
|
|
72
|
+
|
|
73
|
+
- d9f4654: Fix team-scoped health-check management being invisible. Health-check
|
|
74
|
+
configuration team grants are keyed on `healthcheck.healthcheck` (the RPC
|
|
75
|
+
middleware derives the grant key from the configuration access rule's
|
|
76
|
+
`resource`, and that rule is `accessPair("healthcheck", ...)`), but the frontend
|
|
77
|
+
capability gate, the route `manageCapability`, and the Teams grant-name resolver
|
|
78
|
+
all declared `healthcheck.configuration`. Because the two never matched, a user
|
|
79
|
+
who could manage a health check via a team grant (without the global manage
|
|
80
|
+
rule) saw none of the health-check management surfaces, and health-check grant
|
|
81
|
+
names did not resolve in the Teams admin UI.
|
|
82
|
+
|
|
83
|
+
`healthCheckResourceTypes.configuration` now resolves to `healthcheck.healthcheck`
|
|
84
|
+
(with a regression test pinning it to the middleware's grant key), the resolver
|
|
85
|
+
registers under the same type, and the create/edit/assignments routes gain the
|
|
86
|
+
`manageCapability` they were missing so team-scoped health-check managers (and,
|
|
87
|
+
for create/assign, system managers) can reach them. This is a non-breaking fix:
|
|
88
|
+
no stored access-rule id or grant key changes.
|
|
89
|
+
|
|
90
|
+
- 21e0d88: Paused health-check configurations no longer contribute to their systems'
|
|
91
|
+
health aggregate, pausing one now closes any open SLO downtime event it was
|
|
92
|
+
keeping open, and the system overview's "Health Checks" list renders a
|
|
93
|
+
"Paused" pill for paused checks instead of their stale run-evaluated status.
|
|
94
|
+
|
|
95
|
+
Previously, pausing a configuration only skipped execution — its stale
|
|
96
|
+
failing runs inside the evaluation window kept the system's rollup status
|
|
97
|
+
`degraded`/`unhealthy`, which in turn kept any open SLO downtime event open
|
|
98
|
+
until those runs aged out, and the system overview list still showed the
|
|
99
|
+
paused check as "Unhealthy". Now:
|
|
100
|
+
|
|
101
|
+
- `getSystemHealthStatus` excludes paused configurations from the worst-
|
|
102
|
+
wins aggregate, so a system whose only failing check is paused reads
|
|
103
|
+
healthy (and paused checks no longer drive the system's red badge).
|
|
104
|
+
- The `pauseConfiguration` RPC recomputes the rollup `health` entity for
|
|
105
|
+
every system the config is enabled-assigned to. If the recomputed
|
|
106
|
+
aggregate transitions degraded → healthy, the existing `HEALTH_ENTITY_KIND`
|
|
107
|
+
"recovered" edge fires and the SLO engine closes the open downtime event
|
|
108
|
+
at the pause time. If the system stays degraded (other failing checks),
|
|
109
|
+
the event correctly stays open.
|
|
110
|
+
- `resumeConfiguration` intentionally does NOT recompute. The next actual
|
|
111
|
+
run drives any degraded transition: if the check still fails, a fresh
|
|
112
|
+
downtime event opens (the previous one was closed on pause, so the
|
|
113
|
+
`handleSystemDown` idempotent guard doesn't suppress it); if it now
|
|
114
|
+
passes, no event opens. This avoids fabricating a downtime from stale
|
|
115
|
+
last-known state when the underlying condition may have been fixed
|
|
116
|
+
during the pause.
|
|
117
|
+
- `getSystemHealthOverview` now returns a `paused` boolean per check. The
|
|
118
|
+
system overview's "Health Checks" list renders a "Paused" pill (unknown
|
|
119
|
+
tone) for paused checks instead of the run-evaluated status, while still
|
|
120
|
+
showing the pre-pause sparkline for context. Paused checks only appear
|
|
121
|
+
under the "All" filter tab, not "Failing" or "Healthy".
|
|
122
|
+
|
|
123
|
+
- 52c55bf: Per-environment health semantics: rollup no longer masks sibling outages,
|
|
124
|
+
and notifications + automation windows are env-scoped.
|
|
125
|
+
|
|
126
|
+
## The bug
|
|
127
|
+
|
|
128
|
+
When a `(system, configuration)` assignment fanned out to multiple
|
|
129
|
+
environments and only some of them failed, the system rollup could
|
|
130
|
+
read **healthy** (masking a permanently-failing env), or **flap**
|
|
131
|
+
healthy↔degraded/unhealthy tick-by-tick whenever env insertion order
|
|
132
|
+
drifted, because the rollup derivation flattened every env's runs into
|
|
133
|
+
one `timestamp DESC` list and handed the interleaved list to the
|
|
134
|
+
threshold evaluator. The default `consecutive` mode walks newest-first
|
|
135
|
+
and breaks the streak on the first interleaving env, so the rollup
|
|
136
|
+
collapsed to whichever single env's status the most recent run landed
|
|
137
|
+
on. Each flap fired an escalation/recovery notification + a
|
|
138
|
+
`system_health_changed` trigger event.
|
|
139
|
+
|
|
140
|
+
## What changed
|
|
141
|
+
|
|
142
|
+
- **`getSystemHealthStatus(systemId)` rollup** now groups the latest
|
|
143
|
+
run window by `environmentId`, evaluates the threshold window PER
|
|
144
|
+
ENVIRONMENT, and takes worst-wins across envs within each association
|
|
145
|
+
(unhealthy > degraded > healthy) before worst-wins across associations.
|
|
146
|
+
This is stable regardless of env insertion order or multi-pod racing.
|
|
147
|
+
For a single-env (or env-less-only) assignment this reduces to the
|
|
148
|
+
pre-existing flat-window behavior. Per-env and env-less slices
|
|
149
|
+
(`environmentId: string` / `null`) are unchanged.
|
|
150
|
+
- **`getSystemHealthOverview`** now groups runs per `(configurationId,
|
|
151
|
+
environmentId)`, evaluates each env's slice on its own monotonic run
|
|
152
|
+
window, and worst-wins across envs — mirroring `getSystemHealthStatus`.
|
|
153
|
+
The response carries `environmentId` on every `recentRuns[]` entry,
|
|
154
|
+
and adds `perEnvironment[]` per check (one entry per env with its own
|
|
155
|
+
`status` and env-scoped `recentRuns`) so a frontend can render one
|
|
156
|
+
row per `(check, environment)` pair, surfacing per-env outages the
|
|
157
|
+
rollup intentionally hides in the aggregate view. The top-level
|
|
158
|
+
`recentRuns[]` and `status` keep their pre-existing shape for
|
|
159
|
+
backwards compatibility (single-env checks are unchanged).
|
|
160
|
+
- **`HealthCheckSystemOverview`** (frontend) now flattens multi-env
|
|
161
|
+
assignments into one row per `(check, environment)` — each row carries
|
|
162
|
+
the check name, an env pill (resolved via the same
|
|
163
|
+
`getSystemEnvironments` query the drawer already uses), the per-env
|
|
164
|
+
status, sparkline, and last-run. With the "Failing"/"Healthy" filter
|
|
165
|
+
now scoped per env, a permanently-failing environment surfaces as its
|
|
166
|
+
own failing row beside its healthy sibling, instead of being masked by
|
|
167
|
+
the rollup's worst-wins / latest-wins. Single-env and env-less
|
|
168
|
+
assignments render the historical single row (no env pill). Clicking
|
|
169
|
+
any env row opens the check-level drawer, scoped to that env via the
|
|
170
|
+
server-side env filter on the queries below — the drawer's run
|
|
171
|
+
history table, charts, and tiles all see only the (check, environment)
|
|
172
|
+
pair the operator clicked, never a mixed-env pool.
|
|
173
|
+
- **`getHistory`, `getDetailedHistory`, `getRunStats`,
|
|
174
|
+
`getAggregatedHistory`, and `getDetailedAggregatedHistory`** now accept
|
|
175
|
+
an optional `environmentId: string | null` input that filters
|
|
176
|
+
server-side at the DB layer (`environment_id = $X` for an env, `IS
|
|
177
|
+
NULL` for the env-less slice, no predicate when omitted). The drawer's
|
|
178
|
+
charts and Recent Runs table pass the clicked row's `environmentId`
|
|
179
|
+
so the pagination, totals, and buckets reflect only that env — a
|
|
180
|
+
client-side filter would double-paginate and miscount totals; the
|
|
181
|
+
filter is at the DB so the data is honest end-to-end. The aggregated
|
|
182
|
+
history applies the env filter to all three tiers the cross-tier
|
|
183
|
+
aggregation engine reads (raw `health_check_runs` + hourly and daily
|
|
184
|
+
`health_check_aggregates`), since both tables are env-keyed. Single-env
|
|
185
|
+
and env-less rows omit the filter, so historical callers are
|
|
186
|
+
unchanged.
|
|
187
|
+
- **Anomaly baselines are NOT yet env-scoped** — `anomaly_baselines` is
|
|
188
|
+
keyed on `(systemId, configurationId, fieldPath)` with no
|
|
189
|
+
`environmentId` column, and the detector computes a single baseline
|
|
190
|
+
across all envs of an assignment. Scoping the drawer's anomaly overlay
|
|
191
|
+
per env needs a schema migration + a per-env detector rewrite, and is
|
|
192
|
+
tracked as a follow-up. The drawer continues to show the cross-env
|
|
193
|
+
baseline next to the (now env-scoped) history + charts.
|
|
194
|
+
- **`system_health_changed` / `system_degraded` / `system_healthy`
|
|
195
|
+
triggers** now partition by `(systemId, environmentId)` instead of
|
|
196
|
+
the bare `systemId` when the trigger fires from a per-env change.
|
|
197
|
+
Two failing environments of one system now fire two distinct events
|
|
198
|
+
with independent flapping/dwell/dedup windows — operators can author
|
|
199
|
+
per-env automations and get per-env notifications. A bare rollup
|
|
200
|
+
transition (`environmentId` absent) partitions on `systemId` alone,
|
|
201
|
+
so existing recipes that read only `payload.systemId` keep working.
|
|
202
|
+
- **`notifyStateChange`** now accepts `environmentId` +
|
|
203
|
+
`environmentName`. Per-env notifications get an env-qualified title
|
|
204
|
+
(`"System health critical (prod): ..."`) and body, and an
|
|
205
|
+
env-qualified collapse key (`systemHealthCollapseKey(systemId, envId)`)
|
|
206
|
+
so two failing envs render as two independent cards instead of
|
|
207
|
+
merging into one. Suppression checks (maintenance/incident) remain
|
|
208
|
+
system-scoped.
|
|
209
|
+
|
|
210
|
+
## Notes
|
|
211
|
+
|
|
212
|
+
- Each failing env now fires its own `system_health_changed` event with
|
|
213
|
+
its own partition — this is the documented migration away from the
|
|
214
|
+
bug-report flapping cadence into a per-env flap cadence. Operators
|
|
215
|
+
with existing `window:` / `dwell:` recipes on `system_health_changed`
|
|
216
|
+
may see different refire cadence per env (one flapping env no longer
|
|
217
|
+
drowns out its steady sibling). To opt back into the pooled
|
|
218
|
+
historical behavior, an automation recipe can override its own
|
|
219
|
+
`partitionBy: (p) => p.systemId`.
|
|
220
|
+
- `SYSTEM_STATUS_CHANGED` remains rollup-only (one broadcast per tick
|
|
221
|
+
on the rollup status transition): it drives low-noise cache
|
|
222
|
+
invalidation for `SystemHealthBadge` and `DependencyBadge`, and the
|
|
223
|
+
per-env trigger events above already cover per-env automation needs.
|
|
224
|
+
|
|
225
|
+
- d2d49cf: Show the environment for fanned-out runs in the dashboard Recent Activity feed.
|
|
226
|
+
The `healthcheck.run.completed` signal now carries optional `environmentId` and
|
|
227
|
+
`environmentName` fields, populated at the two per-environment fan-out broadcast
|
|
228
|
+
sites in the run executor. The Dashboard "Recent activity" terminal feed renders
|
|
229
|
+
the environment name inline (`system (config) @ env -> status`) when a run was
|
|
230
|
+
fanned out to an environment. Runs that are not environment-scoped omit both
|
|
231
|
+
fields and render exactly as before, so their behavior is unchanged.
|
|
232
|
+
|
|
233
|
+
### Patch Changes
|
|
234
|
+
|
|
235
|
+
- Updated dependencies [52c55bf]
|
|
236
|
+
- Updated dependencies [d1b71b6]
|
|
237
|
+
- Updated dependencies [7c18b25]
|
|
238
|
+
- Updated dependencies [d9f4654]
|
|
239
|
+
- Updated dependencies [21e0d88]
|
|
240
|
+
- Updated dependencies [52c55bf]
|
|
241
|
+
- Updated dependencies [e430fbe]
|
|
242
|
+
- Updated dependencies [eab80e3]
|
|
243
|
+
- Updated dependencies [53666a7]
|
|
244
|
+
- Updated dependencies [d2d49cf]
|
|
245
|
+
- Updated dependencies [0d912a3]
|
|
246
|
+
- @checkstack/healthcheck-common@1.10.0
|
|
247
|
+
- @checkstack/notification-common@1.5.0
|
|
248
|
+
- @checkstack/ai-backend@0.10.2
|
|
249
|
+
- @checkstack/common@0.19.0
|
|
250
|
+
- @checkstack/backend-api@0.27.0
|
|
251
|
+
- @checkstack/incident-common@1.7.0
|
|
252
|
+
- @checkstack/incident-backend@1.9.0
|
|
253
|
+
- @checkstack/maintenance-common@1.8.0
|
|
254
|
+
- @checkstack/catalog-common@2.6.0
|
|
255
|
+
- @checkstack/status-page-common@0.5.0
|
|
256
|
+
- @checkstack/catalog-backend@1.6.2
|
|
257
|
+
- @checkstack/sdk@0.118.1
|
|
258
|
+
- @checkstack/satellite-backend@0.7.5
|
|
259
|
+
- @checkstack/automation-backend@0.10.4
|
|
260
|
+
- @checkstack/script-packages-backend@0.3.19
|
|
261
|
+
- @checkstack/ai-common@0.6.3
|
|
262
|
+
- @checkstack/cache-api@0.3.16
|
|
263
|
+
- @checkstack/cache-utils@0.2.21
|
|
264
|
+
- @checkstack/command-backend@0.2.15
|
|
265
|
+
- @checkstack/gitops-backend@0.5.15
|
|
266
|
+
- @checkstack/gitops-common@0.6.8
|
|
267
|
+
- @checkstack/queue-api@0.3.16
|
|
268
|
+
- @checkstack/secrets-backend@0.2.15
|
|
269
|
+
- @checkstack/secrets-common@0.2.8
|
|
270
|
+
- @checkstack/signal-common@0.2.14
|
|
271
|
+
- @checkstack/status-page-backend@0.4.2
|
|
272
|
+
|
|
3
273
|
## 1.11.1
|
|
4
274
|
|
|
5
275
|
### Patch Changes
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@checkstack/healthcheck-backend",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.12.0",
|
|
4
4
|
"license": "Elastic-2.0",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "src/index.ts",
|
|
@@ -14,32 +14,32 @@
|
|
|
14
14
|
"lint:code": "eslint . --max-warnings 0"
|
|
15
15
|
},
|
|
16
16
|
"dependencies": {
|
|
17
|
-
"@checkstack/backend-api": "0.
|
|
18
|
-
"@checkstack/ai-backend": "0.10.
|
|
19
|
-
"@checkstack/ai-common": "0.6.
|
|
20
|
-
"@checkstack/script-packages-backend": "0.3.
|
|
21
|
-
"@checkstack/cache-api": "0.3.
|
|
22
|
-
"@checkstack/cache-utils": "0.2.
|
|
23
|
-
"@checkstack/catalog-backend": "1.6.
|
|
24
|
-
"@checkstack/catalog-common": "2.
|
|
25
|
-
"@checkstack/command-backend": "0.2.
|
|
26
|
-
"@checkstack/common": "0.
|
|
27
|
-
"@checkstack/gitops-backend": "0.5.
|
|
28
|
-
"@checkstack/gitops-common": "0.6.
|
|
29
|
-
"@checkstack/healthcheck-common": "1.
|
|
30
|
-
"@checkstack/secrets-common": "0.2.
|
|
31
|
-
"@checkstack/secrets-backend": "0.2.
|
|
32
|
-
"@checkstack/incident-backend": "1.
|
|
33
|
-
"@checkstack/incident-common": "1.
|
|
34
|
-
"@checkstack/automation-backend": "0.10.
|
|
35
|
-
"@checkstack/maintenance-common": "1.
|
|
36
|
-
"@checkstack/notification-common": "1.
|
|
37
|
-
"@checkstack/queue-api": "0.3.
|
|
38
|
-
"@checkstack/satellite-backend": "0.7.
|
|
39
|
-
"@checkstack/sdk": "0.
|
|
40
|
-
"@checkstack/signal-common": "0.2.
|
|
41
|
-
"@checkstack/status-page-backend": "0.4.
|
|
42
|
-
"@checkstack/status-page-common": "0.
|
|
17
|
+
"@checkstack/backend-api": "0.27.0",
|
|
18
|
+
"@checkstack/ai-backend": "0.10.2",
|
|
19
|
+
"@checkstack/ai-common": "0.6.3",
|
|
20
|
+
"@checkstack/script-packages-backend": "0.3.19",
|
|
21
|
+
"@checkstack/cache-api": "0.3.16",
|
|
22
|
+
"@checkstack/cache-utils": "0.2.21",
|
|
23
|
+
"@checkstack/catalog-backend": "1.6.2",
|
|
24
|
+
"@checkstack/catalog-common": "2.6.0",
|
|
25
|
+
"@checkstack/command-backend": "0.2.15",
|
|
26
|
+
"@checkstack/common": "0.19.0",
|
|
27
|
+
"@checkstack/gitops-backend": "0.5.15",
|
|
28
|
+
"@checkstack/gitops-common": "0.6.8",
|
|
29
|
+
"@checkstack/healthcheck-common": "1.10.0",
|
|
30
|
+
"@checkstack/secrets-common": "0.2.8",
|
|
31
|
+
"@checkstack/secrets-backend": "0.2.15",
|
|
32
|
+
"@checkstack/incident-backend": "1.9.0",
|
|
33
|
+
"@checkstack/incident-common": "1.7.0",
|
|
34
|
+
"@checkstack/automation-backend": "0.10.4",
|
|
35
|
+
"@checkstack/maintenance-common": "1.8.0",
|
|
36
|
+
"@checkstack/notification-common": "1.5.0",
|
|
37
|
+
"@checkstack/queue-api": "0.3.16",
|
|
38
|
+
"@checkstack/satellite-backend": "0.7.5",
|
|
39
|
+
"@checkstack/sdk": "0.118.1",
|
|
40
|
+
"@checkstack/signal-common": "0.2.14",
|
|
41
|
+
"@checkstack/status-page-backend": "0.4.2",
|
|
42
|
+
"@checkstack/status-page-common": "0.5.0",
|
|
43
43
|
"@hono/zod-validator": "^0.7.6",
|
|
44
44
|
"drizzle-orm": "^0.45.0",
|
|
45
45
|
"hono": "^4.12.25",
|
|
@@ -51,8 +51,8 @@
|
|
|
51
51
|
},
|
|
52
52
|
"devDependencies": {
|
|
53
53
|
"@checkstack/drizzle-helper": "0.0.6",
|
|
54
|
-
"@checkstack/scripts": "0.
|
|
55
|
-
"@checkstack/test-utils-backend": "0.1.
|
|
54
|
+
"@checkstack/scripts": "0.7.0",
|
|
55
|
+
"@checkstack/test-utils-backend": "0.1.49",
|
|
56
56
|
"@checkstack/tsconfig": "0.0.7",
|
|
57
57
|
"@types/bun": "^1.0.0",
|
|
58
58
|
"@types/tdigest": "^0.1.5",
|
package/src/automations.test.ts
CHANGED
|
@@ -67,7 +67,7 @@ describe("healthcheck triggers", () => {
|
|
|
67
67
|
});
|
|
68
68
|
|
|
69
69
|
|
|
70
|
-
it("extracts systemId as the contextKey on all three", () => {
|
|
70
|
+
it("extracts systemId as the contextKey on all three (bare rollup, no environmentId)", () => {
|
|
71
71
|
const degradedOrChanged = {
|
|
72
72
|
systemId: "sys-1",
|
|
73
73
|
previousStatus: "healthy",
|
|
@@ -89,6 +89,48 @@ describe("healthcheck triggers", () => {
|
|
|
89
89
|
"sys-1",
|
|
90
90
|
);
|
|
91
91
|
});
|
|
92
|
+
|
|
93
|
+
it("partitions the contextKey per-(system, environment) when environmentId is present", () => {
|
|
94
|
+
// Per-env partition: two failing envs of one system share a SYSTEM
|
|
95
|
+
// but NOT a flapping/dwell/dedup window — automations get N distinct
|
|
96
|
+
// `system_health_changed` events with independent partitions. The bare
|
|
97
|
+
// rollup change (no environmentId) keys on the bare systemId alone so
|
|
98
|
+
// existing recipes keep working.
|
|
99
|
+
const prodSpread = {
|
|
100
|
+
systemId: "sys-1",
|
|
101
|
+
environmentId: "prod",
|
|
102
|
+
previousStatus: "healthy",
|
|
103
|
+
newStatus: "unhealthy",
|
|
104
|
+
healthyChecks: 1,
|
|
105
|
+
totalChecks: 2,
|
|
106
|
+
timestamp: "2026-05-29T11:00:00Z",
|
|
107
|
+
} as const;
|
|
108
|
+
const stagingSpread = {
|
|
109
|
+
...prodSpread,
|
|
110
|
+
environmentId: "staging",
|
|
111
|
+
} as const;
|
|
112
|
+
const rollup = {
|
|
113
|
+
systemId: "sys-1",
|
|
114
|
+
previousStatus: "degraded",
|
|
115
|
+
newStatus: "unhealthy",
|
|
116
|
+
healthyChecks: 1,
|
|
117
|
+
totalChecks: 2,
|
|
118
|
+
timestamp: "2026-05-29T11:00:00Z",
|
|
119
|
+
} as const;
|
|
120
|
+
|
|
121
|
+
expect(systemHealthChangedTrigger.contextKey?.(prodSpread)).toBe(
|
|
122
|
+
"sys-1::prod",
|
|
123
|
+
);
|
|
124
|
+
expect(systemHealthChangedTrigger.contextKey?.(stagingSpread)).toBe(
|
|
125
|
+
"sys-1::staging",
|
|
126
|
+
);
|
|
127
|
+
// Rollup stays system-scoped.
|
|
128
|
+
expect(systemHealthChangedTrigger.contextKey?.(rollup)).toBe("sys-1");
|
|
129
|
+
// The directional triggers use the SAME partition function (they share
|
|
130
|
+
// the schema's { systemId, environmentId? } fields by structural type).
|
|
131
|
+
expect(systemDegradedTrigger.contextKey?.(prodSpread)).toBe("sys-1::prod");
|
|
132
|
+
expect(systemHealthyTrigger.contextKey?.(prodSpread)).toBe("sys-1::prod");
|
|
133
|
+
});
|
|
92
134
|
});
|
|
93
135
|
|
|
94
136
|
describe("assignmentArtifactType", () => {
|
package/src/automations.ts
CHANGED
|
@@ -90,6 +90,24 @@ const checkFailedPayloadSchema = z.object({
|
|
|
90
90
|
|
|
91
91
|
// ─── Triggers ──────────────────────────────────────────────────────────
|
|
92
92
|
|
|
93
|
+
// Per-(system, environment) partition key for all entity-driven system-health
|
|
94
|
+
// triggers. Carrying `environmentId` here — NOT just on the payload — makes
|
|
95
|
+
// the automation engine's dwell / flapping / dedup windows partition per-env:
|
|
96
|
+
// one environment flapping doesn't get drowned out by the steady sibling,
|
|
97
|
+
// and N failing envs fire N distinct `system_health_changed` events with
|
|
98
|
+
// independent partitions (see the changeset "Make healthcheck triggers
|
|
99
|
+
// env-scoped"). `payload.systemId` stays the bare systemId, so existing
|
|
100
|
+
// recipes that read only `systemId` keep working. A bare rollup change
|
|
101
|
+
// (per `environmentId = null`) partitions on the bare systemId alone —
|
|
102
|
+
// the historical system-scoped window. The three trigger payloads all
|
|
103
|
+
// expose `{ systemId, environmentId? }`, so the partition function is shared
|
|
104
|
+
// via an inline closure that's structural-compatible with each.
|
|
105
|
+
function systemHealthPartitionKey<
|
|
106
|
+
P extends { systemId: string; environmentId?: string },
|
|
107
|
+
>(p: P): string {
|
|
108
|
+
return p.environmentId ? `${p.systemId}::${p.environmentId}` : p.systemId;
|
|
109
|
+
}
|
|
110
|
+
|
|
93
111
|
export const systemDegradedTrigger: TriggerDefinition<
|
|
94
112
|
z.infer<typeof systemDegradedPayloadSchema>
|
|
95
113
|
> = {
|
|
@@ -105,7 +123,8 @@ export const systemDegradedTrigger: TriggerDefinition<
|
|
|
105
123
|
setup: makeEntityDrivenTriggerSetup<
|
|
106
124
|
z.infer<typeof systemDegradedPayloadSchema>
|
|
107
125
|
>(),
|
|
108
|
-
|
|
126
|
+
// Per-(system, env); bare systemId for the rollup change (env-less).
|
|
127
|
+
contextKey: (p) => systemHealthPartitionKey(p),
|
|
109
128
|
contextKeyLabel: "system",
|
|
110
129
|
};
|
|
111
130
|
|
|
@@ -122,7 +141,7 @@ export const systemHealthyTrigger: TriggerDefinition<
|
|
|
122
141
|
setup: makeEntityDrivenTriggerSetup<
|
|
123
142
|
z.infer<typeof systemHealthyPayloadSchema>
|
|
124
143
|
>(),
|
|
125
|
-
contextKey: (p) => p
|
|
144
|
+
contextKey: (p) => systemHealthPartitionKey(p),
|
|
126
145
|
contextKeyLabel: "system",
|
|
127
146
|
};
|
|
128
147
|
|
|
@@ -140,7 +159,7 @@ export const systemHealthChangedTrigger: TriggerDefinition<
|
|
|
140
159
|
setup: makeEntityDrivenTriggerSetup<
|
|
141
160
|
z.infer<typeof systemHealthChangedPayloadSchema>
|
|
142
161
|
>(),
|
|
143
|
-
contextKey: (p) => p
|
|
162
|
+
contextKey: (p) => systemHealthPartitionKey(p),
|
|
144
163
|
contextKeyLabel: "system",
|
|
145
164
|
};
|
|
146
165
|
|
package/src/hooks.ts
CHANGED
|
@@ -46,6 +46,12 @@ export const healthCheckHooks = {
|
|
|
46
46
|
latencyMs: number | undefined;
|
|
47
47
|
result: Record<string, unknown> | undefined;
|
|
48
48
|
timestamp: string;
|
|
49
|
+
/**
|
|
50
|
+
* Environment the run was executed for. null = the env-less slice (no
|
|
51
|
+
* environment membership). Forwarded by the anomaly plugin so its inline
|
|
52
|
+
* detector resolves the per-env baseline rather than a cross-env one.
|
|
53
|
+
*/
|
|
54
|
+
environmentId: string | null;
|
|
49
55
|
}>("healthcheck.check.completed"),
|
|
50
56
|
|
|
51
57
|
/**
|
|
@@ -64,5 +70,6 @@ export const healthCheckHooks = {
|
|
|
64
70
|
latencyMs: number | undefined;
|
|
65
71
|
result: Record<string, unknown> | undefined;
|
|
66
72
|
timestamp: string;
|
|
73
|
+
environmentId: string | null;
|
|
67
74
|
}>("healthcheck.check.failed"),
|
|
68
75
|
} as const;
|
package/src/index.ts
CHANGED
|
@@ -1,12 +1,14 @@
|
|
|
1
1
|
import {
|
|
2
2
|
setupHealthCheckWorker,
|
|
3
3
|
bootstrapHealthChecks,
|
|
4
|
+
recomputeSystemRollupHealth,
|
|
4
5
|
} from "./queue-executor";
|
|
5
6
|
import { setupRetentionJob } from "./retention-job";
|
|
6
7
|
import * as schema from "./schema";
|
|
7
8
|
import {
|
|
8
9
|
healthCheckAccessRules,
|
|
9
10
|
healthCheckAccess,
|
|
11
|
+
healthCheckResourceTypes,
|
|
10
12
|
pluginMetadata,
|
|
11
13
|
healthCheckContract,
|
|
12
14
|
healthcheckRoutes,
|
|
@@ -242,10 +244,14 @@ export default createBackendPlugin({
|
|
|
242
244
|
const typedDb = database as SafeDatabase<typeof schema>;
|
|
243
245
|
|
|
244
246
|
// Resolve/search health-check configurations by name for the Teams admin
|
|
245
|
-
// UI (team grants are stored as opaque
|
|
246
|
-
//
|
|
247
|
-
//
|
|
248
|
-
|
|
247
|
+
// UI (team grants are stored as opaque `<type>:<configId>` rows, where
|
|
248
|
+
// <type> is `healthCheckResourceTypes.configuration` — i.e.
|
|
249
|
+
// `healthcheck.healthcheck`, the key the RPC middleware derives from the
|
|
250
|
+
// configuration access rule's resource). This MUST match that grant key,
|
|
251
|
+
// or grant names never resolve. Lets the auth backend render grants by
|
|
252
|
+
// name and power the grant picker without depending on healthcheck
|
|
253
|
+
// internals.
|
|
254
|
+
resourceResolverRegistry.register(healthCheckResourceTypes.configuration, {
|
|
249
255
|
resolveNames: async (ids) => {
|
|
250
256
|
if (ids.length === 0) return new Map();
|
|
251
257
|
const rows = await typedDb
|
|
@@ -480,6 +486,17 @@ export default createBackendPlugin({
|
|
|
480
486
|
maintenanceClient,
|
|
481
487
|
logger,
|
|
482
488
|
signalService,
|
|
489
|
+
recomputeSystemRollupHealth: (systemId) =>
|
|
490
|
+
recomputeSystemRollupHealth({
|
|
491
|
+
systemId,
|
|
492
|
+
// Reuse the COMPUTE-ON-READ service instance bound to the
|
|
493
|
+
// `health` entity read accessor — it's the same db/registry
|
|
494
|
+
// the rollup write inside `executeHealthCheckJob` uses.
|
|
495
|
+
service,
|
|
496
|
+
getHealthEntity: () => healthEntity,
|
|
497
|
+
advisoryLock,
|
|
498
|
+
logger,
|
|
499
|
+
}),
|
|
483
500
|
});
|
|
484
501
|
rpc.registerRouter(healthCheckRouter, healthCheckContract);
|
|
485
502
|
|