@checkstack/healthcheck-backend 1.21.3 → 1.23.0

package/CHANGELOG.md

@@ -1,5 +1,319 @@
 # @checkstack/healthcheck-backend
 
+## 1.23.0
+
+### Minor Changes
+
+- 88f4333: Per-satellite offline threshold, connectivity notifications, and stop satellite-only checks going silent
+
+  **A satellite going offline was invisible, and so were its checks.** Three
+  related changes:
+
+  **Per-satellite offline threshold.** The 45-second global constant is now a
+  per-satellite override (**Offline after**, 2 minutes to 24 hours), because
+  tolerance is a property of the link, not of the platform: a satellite on a flaky
+  uplink needs grace that should not be forced on every other satellite. The
+  threshold is carried on every row read by `computeStatus`, so the entity read,
+  the admin list and the heartbeat monitor cannot disagree about the same
+  satellite. Additive, nullable column - existing satellites keep the default.
+
+  **Connectivity notifications.** Satellites are now a notification target with a
+  **Satellite connectivity** subscription: a warning when a satellite stops
+  heartbeating, informational when it returns. A reconnect only notifies if the
+  satellite was actually offline, so a redeploy is not an event. (The same
+  transitions remain available as `satellite.heartbeat_lost` / `.connected`
+  automation triggers for anyone wanting different routing.)
+
+  **Satellite-only checks no longer go silent.** BUG FIX: a check with
+  `includeLocal: false` whose satellites were all offline recorded NOTHING, so it
+  displayed its last known status indefinitely - a dead probe was indistinguishable
+  from a passing one. The core now records a `degraded` run with a clear message.
+  Degraded rather than unhealthy because the target may be fine; what failed is our
+  ability to observe it. Liveness that cannot be resolved is treated as "executing"
+  so a transient lookup failure cannot mark the whole fleet degraded at once.
+
+  Checks also surface staleness: a last run older than five intervals (minimum ten
+  minutes) is highlighted, so an ageing status is visible even with no run to
+  explain it. Paused checks are never stale, and neither is a RETIRED slice - one
+  whose environment was removed or whose satellite was unassigned - because
+  warning about something you retired on purpose trains operators to ignore the
+  badge.
+
+  The unobservable run does NOT notify subscribers. One offline satellite degrades
+  every check assigned to it in the same tick, and `healthy -> degraded` is an
+  escalation, so notifying per check would turn a single root cause into one alert
+  per check. The satellite's own connectivity subscription reports the cause once;
+  the runs are still recorded, so health and the UI stay honest.
+
+  Satellite liveness is cached on the shared platform cache with a 5s TTL. The
+  executor asks per tick of every satellite-only check and the read is a full
+  scan, so the uncached version scaled with the number of such checks. The TTL is
+  well below the smallest offline threshold the schema allows, so a cached answer
+  can lag a transition by one tick but never span one.
+
+  Corrects the user guide, which claimed offline satellites produced failed runs -
+  they produced nothing at all.
+
+### Patch Changes
+
+- Updated dependencies [88f4333]
+- Updated dependencies [1deaac5]
+- Updated dependencies [88f4333]
+- Updated dependencies [88f4333]
+- Updated dependencies [88f4333]
+- Updated dependencies [88f4333]
+- Updated dependencies [88f4333]
+- Updated dependencies [1deaac5]
+- Updated dependencies [56e5375]
+- Updated dependencies [88f4333]
+  - @checkstack/common@0.24.0
+  - @checkstack/healthcheck-common@1.19.1
+  - @checkstack/status-page-common@0.7.0
+  - @checkstack/incident-common@1.11.0
+  - @checkstack/maintenance-common@1.11.0
+  - @checkstack/command-backend@0.3.0
+  - @checkstack/incident-backend@1.14.0
+  - @checkstack/satellite-backend@0.10.0
+  - @checkstack/status-page-backend@0.7.0
+  - @checkstack/notification-common@1.9.0
+  - @checkstack/ai-backend@0.11.5
+  - @checkstack/backend-api@0.35.0
+  - @checkstack/satellite-common@0.12.0
+  - @checkstack/automation-backend@0.11.9
+  - @checkstack/secrets-backend@0.3.10
+  - @checkstack/ai-common@0.6.8
+  - @checkstack/cache-api@0.3.21
+  - @checkstack/catalog-backend@1.10.2
+  - @checkstack/catalog-common@2.8.2
+  - @checkstack/gitops-backend@0.5.28
+  - @checkstack/gitops-common@0.7.5
+  - @checkstack/healthcheck-execution@0.35.1
+  - @checkstack/queue-api@0.4.1
+  - @checkstack/script-packages-backend@0.4.7
+  - @checkstack/sdk@0.136.1
+  - @checkstack/secrets-common@0.3.4
+  - @checkstack/signal-common@0.3.2
+  - @checkstack/cache-utils@0.3.2
+
+## 1.22.0
+
+### Minor Changes
+
+- be74b01: Evaluate health per probe location, so a failing satellite can no longer read as healthy
+
+  Thanks to @stuajnht for reporting: a system whose local check succeeded and
+  whose satellite check failed was shown as **healthy**, and the report correctly
+  guessed the cause - one combined verdict where there should have been one per
+  location.
+
+  A check's runs were grouped into slices by environment alone, so both locations'
+  runs landed in the same slice and were handed to the threshold evaluator as one
+  interleaved stream. In the default `consecutive` mode the streak breaks on every
+  alternation, no threshold is ever reached, and evaluation falls through to its
+  healthy default. A satellite failing 100% of the time was therefore invisible
+  for as long as a local check succeeded between its runs.
+
+  A slice is now an **(environment, source)** pair - one environment as probed
+  from one location - and each is evaluated on its own window, with the worst
+  result deciding the check. This is the same rule environments already followed;
+  the source dimension was simply never considered. Both the system rollup and the
+  system overview were affected, and both are fixed.
+
+  Related correctness fixes that fall out of keying slices by source:
+
+  - A **de-assigned satellite** (or the core after **Include local** is turned
+    off) stops counting immediately instead of dragging the rollup with its last
+    failures until they age out of the window. Its history moves under **Old
+    checks**.
+  - **Per-satellite environment scoping** is honoured when resolving slices, so a
+    satellite narrowed to production no longer keeps a stale staging slice alive.
+  - A satellite scoped to run env-less while the core fans out keeps its slice
+    live; the "has a live environment slice" question is now answered per
+    location, as the backend already did.
+
+  The system overview shows one row per slice and names the location (for example
+  **EU West**) as soon as a check runs from more than one place. A check that only
+  ever runs on the core shows no location label - there is nothing to
+  disambiguate.
+
+  `checkStatuses[].slices` and the overview's per-slice entries carry the
+  breakdown (`sourceId`, `sourceLabel`, `sourceOrphaned`) on the wire, and
+  `sliceCount` / `failingSliceCount` now count locations as well as environments -
+  so a check probing one environment from the core and one satellite contributes
+  2 to the dashboard's "X of Y checks failing" denominator, not 1.
+
+- be74b01: Satellites run per environment, and can be scoped to specific ones
+
+  Satellites were handed no environment information at all, so every result they
+  reported was stored env-less. On a system with environments that meant satellite
+  checks contributed nothing to per-environment health - and, until the preceding
+  fix, were labelled "Old checks" for it.
+
+  A satellite now fans out exactly as the local executor does:
+
+  - `getAssignmentsForSatellite` resolves each assignment's effective environments
+    and sends them with the assignment.
+  - The agent schedules ONE run per environment and reports each result with its
+    `environmentId`, so per-environment history, charts and rollups include
+    satellite results.
+  - Collectors on a satellite now receive the `environment` run-context block, so
+    `{{ environment.<key> }}` templating resolves there exactly as it does locally.
+
+  **A satellite can also be scoped to specific environments.** Without that, every
+  satellite would probe every environment - a staging-network satellite would start
+  failing prod checks it has no route to, and one per-environment slice would merge
+  results from satellites in different networks. A new `satelliteEnvironmentIds`
+  map on the assignment scopes each satellite: an absent key means "all
+  environments" (so every existing assignment behaves exactly as before), `[]` means
+  one env-less run, and a list narrows to those ids. A satellite can only ever
+  narrow the assignment's own selector, never widen it.
+
+  Both protocol additions are optional, for version skew in either direction: an
+  older satellite sends no `environmentId` and its runs are stored env-less as they
+  always were, while an older core sends no environments and the agent falls back to
+  a single env-less run.
+
+  The assignment's Execution panel gains a per-satellite environment picker,
+  shown for each assigned satellite once the system has environments.
+
+  Thanks to [@stuajnht](https://github.com/stuajnht) for the valuable feedback.
+
+- be74b01: Drive satellite health results through the same reactive/notify path as local runs
+
+  A satellite-detected health change previously did almost nothing on the core:
+  `ingestSatelliteResult` inserted the run row and invalidated the cache, and
+  stopped there. A LOCAL run additionally drives the whole reactive layer - the
+  `health` entity write (which fires the ENTITY_CHANGED that automations and
+  triggers key on), the state-transition record, the subscriber notification, the
+  checkCompleted/checkFailed automation hooks, and the realtime signals. So a
+  satellite that detected an outage fired **no notifications, no automations, no
+  transition record, and no realtime signal** - satellite monitoring was
+  effectively silent.
+
+  Both paths now run through ONE shared function, `persistRunAndReact`, so a
+  satellite result reacts exactly like a local one. The host binds the service
+  dependencies once and hands the router a narrowed reactor, so the local and
+  satellite callers cannot pass different dependencies and drift apart again
+  (`ingestSatelliteResult` was itself a duplicated-and-drifted copy of the local
+  persistence path - this removes the duplication that caused it). Ingest now
+  splits into `processSatelliteResult` (evaluate assertions, strip ephemeral
+  fields, resolve the check name) plus the shared reactive path.
+
+  Also fixed: a satellite collector's transport error is now annotated as
+  `_collectorError` on the stored result, matching a local run - the satellite
+  previously dropped that annotation.
+
+  Coverage: added tests that a satellite result is routed through the shared
+  reactor with its processed payload (guarding against a silent regression back
+  to insert-only), and extracted the satellite's `executeAssignment` into a
+  testable module with tests for custom-field template expansion, probe-measured
+  timings, the `_collectorError` annotation, and the strategy-not-loaded path.
+
+- be74b01: Expand system/environment custom fields in satellite health checks, via one shared execution engine
+
+  Thanks to @stuajnht for reporting: a system or environment custom field
+  referenced with `{{ system.metadata.<key> }}` / `{{ environment.<key> }}` in a
+  health check was NOT expanded when the check ran on a satellite - the raw
+  template reached the probe. The core queue executor grew a per-run templating
+  pass, but the satellite's execution loop was a hand-maintained COPY that never
+  did, so the two drifted.
+
+  The fix removes the copy. A new lean package `@checkstack/healthcheck-execution`
+  owns the shared execution engine - render the strategy + collector
+  `x-templatable` fields against the run's environment/system context, build the
+  transport client, run the collectors, close the client - and BOTH the core
+  queue executor and the satellite now run through it. Templating, the
+  secret-then-template ordering, and the per-collector fan-out therefore cannot
+  drift between core and satellite again. Each side keeps only its genuine edges
+  as injected hooks: the core resolves secrets from its database and does
+  migrate-on-read; the satellite resolves them just-in-time over its socket.
+
+  Also fixed: transport sub-phase timings (DNS / connect / TLS / wait / transfer)
+  are now measured AT THE PROBE and reported by satellites, so a satellite run's
+  `metadata.timings` matches a local run's. The core cannot derive the timing of a
+  probe it did not run - and may have no route to a target a satellite can reach -
+  so the satellite must produce these; the core persists them as-is.
+
+- be74b01: Stop reporting systems as healthy when nothing has measured them
+
+  A system whose health check had never produced a run reported `healthy` - so it
+  showed green in the catalog, kept its group green, and read "operational" on the
+  public status page. A system with no checks at all did the same. For a
+  monitoring product that is the worst possible default: the one state you must
+  never invent is the reassuring one.
+
+  `getSystemHealthStatus` began each check at `healthy` and each system's
+  aggregate at `healthy`, then only ever downgraded. With no runs to examine,
+  nothing downgraded them. `HealthCheckStatus` had no way to say "not measured".
+
+  A new `SystemHealthStatus` adds `unknown` for systems and their checks. It is
+  deliberately NOT a run status - a run that happened is always healthy, degraded
+  or unhealthy, and the database enum stays three-valued. Now:
+
+  - A check with no runs is `unknown`, not `healthy`.
+  - A system reports `unknown` when no check contributed a signal. A system with
+    one healthy check and one never-run check still reads `healthy`: it has
+    positive evidence, and the unmeasured check is visible on its own page.
+  - The catalog reports `unknown` by OMISSION, which its group rollup already
+    treats as "no signal" - so a group with an unmeasured member stops claiming to
+    be healthy. That is the reported bug.
+  - The public status page maps it to its existing `unknown`, which is ignored for
+    the overall banner unless everything is unknown. One unmeasured system no
+    longer claims "operational" for itself, and does not panic the whole page.
+  - A first measurement records a transition with a NULL `fromStatus` - the column
+    was already nullable for exactly this case - instead of pretending the system
+    was healthy beforehand.
+  - Automations matching on `unhealthy` do not fire for a merely unmeasured
+    system, which is correct: an unmeasured system is not a detected outage.
+
+  Dependency warnings deliberately keep their current behaviour: an unmeasured
+  upstream raises no warning, and a never-run check is dropped from the evaluation
+  rather than counted as passing.
+
+  Note that pausing a system's only check now leaves it `unknown` rather than
+  `healthy`. Paused failures still do not keep a system degraded - that behaviour
+  is unchanged - but with nothing running, the system is genuinely unmeasured.
+
+  Thanks to [@stuajnht](https://github.com/stuajnht) for the valuable feedback.
+
+### Patch Changes
+
+- Updated dependencies [be74b01]
+- Updated dependencies [be74b01]
+- Updated dependencies [be74b01]
+- Updated dependencies [be74b01]
+- Updated dependencies [be74b01]
+- Updated dependencies [be74b01]
+- Updated dependencies [be74b01]
+- Updated dependencies [be74b01]
+- Updated dependencies [be74b01]
+- Updated dependencies [be74b01]
+- Updated dependencies [be74b01]
+- Updated dependencies [be74b01]
+- Updated dependencies [be74b01]
+- Updated dependencies [be74b01]
+- Updated dependencies [be74b01]
+- Updated dependencies [be74b01]
+  - @checkstack/ai-backend@0.11.4
+  - @checkstack/notification-common@1.8.0
+  - @checkstack/incident-backend@1.13.6
+  - @checkstack/healthcheck-common@1.19.0
+  - @checkstack/satellite-backend@0.9.4
+  - @checkstack/healthcheck-execution@0.35.0
+  - @checkstack/status-page-backend@0.6.6
+  - @checkstack/status-page-common@0.6.5
+  - @checkstack/automation-backend@0.11.8
+  - @checkstack/secrets-backend@0.3.9
+  - @checkstack/catalog-backend@1.10.1
+  - @checkstack/catalog-common@2.8.1
+  - @checkstack/incident-common@1.10.5
+  - @checkstack/maintenance-common@1.10.5
+  - @checkstack/script-packages-backend@0.4.6
+  - @checkstack/sdk@0.135.1
+  - @checkstack/backend-api@0.34.1
+  - @checkstack/command-backend@0.2.27
+  - @checkstack/gitops-backend@0.5.27
+
 ## 1.21.3
 
 ### Patch Changes

package/drizzle/0021_amazing_wolf_cub.sql

@@ -0,0 +1,8 @@
+-- Per-satellite environment scoping for a health-check assignment.
+--
+-- Deliberately NO backfill: the column's "key absent" semantics already mean
+-- "this satellite runs every environment the assignment resolves to", so a NULL
+-- column reproduces exactly the behaviour every existing row has today. Writing
+-- an explicit map for every row would only freeze today's satellite list into
+-- data that has to be maintained.
+ALTER TABLE "system_health_checks" ADD COLUMN "satellite_environment_ids" jsonb;