@checkstack/healthcheck-backend 1.17.0 → 1.19.0

package/CHANGELOG.md

@@ -1,5 +1,564 @@
 # @checkstack/healthcheck-backend
 
+## 1.19.0
+
+### Minor Changes
+
+- 43e4484: Fix an N+1 in the catalog manager: the per-system "Health Checks" count badge
+  fired one `getSystemAssociations` request per system row, each holding a pooled
+  Postgres connection that contended with the background health-check run
+  executor and could exhaust the pool on large catalogs.
+
+  - Add `getBulkAssignedHealthCheckCounts({ systemIds })` to healthcheck, which
+    returns per-system assignment counts (0 for systems with no assignments) from
+    ONE grouped `COUNT(*) ... GROUP BY system_id` query. Read authorization
+    matches the per-system endpoint it replaces (`configuration.read` +
+    `catalog.system` read via `recordKey`), so a team-scoped user only sees counts
+    for systems they may read.
+  - `CatalogSystemActionsSlot` now passes `visibleSystemIds` (every system id in
+    the row's list) so a per-row filler can bulk-fetch for the whole visible set
+    in a single deduped request instead of one request per row. This mirrors how
+    `CatalogBrowseHealthSlot` / `SystemSignalsSlot` already pass `systemIds`.
+  - The health-check count badge now reads its count from that one deduped bulk
+    query. N visible rows cause 1 request instead of N.
+
+  State & scale: the counts are derived on read from the shared
+  `system_health_checks` table, so every pod returns the same answer; no
+  process-local or duplicated state is introduced.
+
+- 43e4484: Name the failing health check in system-health notifications. The notification
+  body now names the check that drove the transition (in addition to the system
+  and environment), and a `healthcheck.healthcheck` subject is pushed alongside
+  the `catalog.system` subject, deep-linked to the check's run history. Recovery
+  notifications stay system-level. Adds a `createHealthcheckSubject` builder to
+  `healthcheck-common`.
+
+  Thanks to [@stuajnht](https://github.com/stuajnht) for the valuable feedback.
+
+- 43e4484: Status pages can now publish only a subset of catalog environments. The page
+  builder gains a "Published environments" picker (empty = all environments, the
+  backward-compatible default). When a non-empty set is selected, the page omits
+  status, incidents, maintenances and uptime for systems that belong to none of
+  the selected environments.
+
+  - Status pages store an optional `publishedEnvironmentIds` set (new nullable
+    `published_environment_ids` column; NULL = all environments, so existing pages
+    are unchanged) exposed on `StatusPage`, `createStatusPage`, and
+    `updateStatusPage`.
+  - The scope is threaded onto `WidgetResolveContext.publishedEnvironmentIds` as
+    opaque strings and passed identically to `resolvePublic`,
+    `resolveScopedSystems`, and `resolveScopedSystemsDetailed` (and the email
+    subscribe clamp + fan-out), so what a page shows, offers for subscription, and
+    emails about all agree.
+  - Health widgets recompute per environment: they read the per-environment health
+    matrix and roll up only the selected environments. `getBulkRunStats` and
+    `getRunStats` gain an optional `environmentIds` filter so uptime counts only
+    runs recorded in the selected environments.
+  - Incident and maintenance widgets filter their feed and scope by intersecting
+    each item's affected systems with the environment-visible systems. Incidents
+    and maintenance windows carry no environment of their own, so a system in
+    several environments makes its items visible on a page publishing ANY of them
+    (the multi-environment caveat).
+
+### Patch Changes
+
+- 43e4484: fix(healthcheck): disabling an environment for an assignment now clears its stale slice from the rollup and overview immediately
+
+  Disabling an environment for a health-check assignment (removing it from the
+  assignment's `environmentIds`) stopped that environment from fanning out, but a
+  check that was FAILING there kept dragging the system health rollup/badge to
+  unhealthy and kept showing as a live failing row in the system overview. Because
+  the rollup is recomputed by an event-driven consumer subscribed to per-env health
+  CHANGES, and a disabled env produces no further runs (so no change event fires),
+  the stale unhealthy status was never recomputed away - it only cleared
+  incidentally, once the disabled env's runs aged out of the bounded run window
+  (which needs the assignment's OTHER active environments to produce enough newer
+  runs first). With a single active/failing env, it could persist until retention.
+
+  Scope: this reconciles environments DISABLED/removed ON THE ASSIGNMENT (its
+  `systemHealthChecks.environmentIds` selector - switching to Specific and
+  deselecting, or None).
+
+  Fixes:
+
+  - The rollup aggregation (`getSystemHealthStatus`) and the per-check status in
+    `getSystemHealthOverview` now consider only CURRENTLY-EFFECTIVE environment
+    slices, derived from the durable `systemHealthChecks.environmentIds` selector
+    (catalog-free, identical on every pod). A slice whose environment was disabled
+    for the assignment, or the stale env-less slice of a check that now fans out,
+    no longer contributes.
+
+  Known limitation: under an "all-environments" assignment (`environmentIds` is
+  `null`), an environment removed only from the system's CATALOG MEMBERSHIP (rather
+  than disabled on the assignment) can still contribute to the backend rollup/badge
+  until the assignment is re-evaluated, because the rollup read path is
+  intentionally catalog-free for horizontal-scale correctness (it must return the
+  same answer on every pod without a per-read catalog lookup). This is pre-existing;
+  the frontend overview, which can see membership, still orphans such a slice.
+
+  - Each environment is now windowed by its OWN query in the rollup, instead of a
+    single shared `LIMIT` across the mixed-env pool. The old shared window
+    truncated per-env evaluation for checks that fan out to many environments (or
+    with large threshold windows); every environment now gets its full evaluation
+    depth.
+  - Changing an assignment's environment set now triggers an immediate rollup
+    recompute for that system, so the persisted `health` entity (badge + SLO
+    downtime) converges at once rather than waiting for stale runs to age out.
+  - The system-overview frontend tucks a slice whose environment was disabled for
+    the assignment under "Old checks" (system membership alone could not detect it,
+    since the environment is still part of the system). `getSystemHealthOverview`
+    now returns each check's `environmentIds` selector to drive this.
+
+  Shared pure helpers `selectorIncludesEnvironment` / `isEnvSliceEffective` /
+  `selectEffectiveEnvKeys` are added to `@checkstack/healthcheck-common` so the
+  backend and frontend agree on effective-slice detection.
+
+- 43e4484: Batch hot-path scoped-db reads/writes into single transactions to cut per-query round-trips.
+
+  The scoped-db proxy wraps every standalone query in its own `BEGIN → SET LOCAL search_path → query → COMMIT`, so a path issuing N sequential queries paid N round-trips and checked out a connection N times. These reads/writes now run under one `withScopedTransaction`, collapsing the batch to a single `SET LOCAL` on one connection. Behavior is unchanged:
+
+  - healthcheck: `getSystemHealthOverview`'s `1 + N·(2+E)` read fan-out.
+  - incident/maintenance: `getIncident`/`getMaintenance` (4 reads), `getManyEntityStates`, `listOpenIncidentsBySystem` / `getActiveMaintenancesBySystem`, `getMaintenanceWindowsForRange`; the `list*` / `*ForSystem` per-row `N+1` system lookups collapsed to a single set-based `inArray` read; maintenance `transitionStatus` update+insert made atomic; `addUpdate`/`editUpdate`/`addLink` use `.returning()` instead of a follow-up re-select.
+  - ai: `appendMessage`, memory `saveOrUpdate`.
+  - notification: `resolveInheritedGroups`.
+  - status-page: subscriber `verify` (4 reads) and `unsubscribe` (3 reads).
+  - announcement: `getActiveAnnouncements` / `dismissAnnouncement` / `createAnnouncement`.
+  - gitops: `upsertProvenance`.
+
+- 43e4484: Eliminate N+1 RPC fan-outs in the public status-page widget resolvers.
+
+  Each of these widgets renders a PUBLIC page, so every per-item RPC was real
+  external DB load. Three bulk-by-id endpoints replace the per-item fetches:
+
+  - `healthcheck-common`: new `getBulkRunStats({ systemIds, startDate, endDate,
+maxBuckets })` -> `{ stats: Record<systemId, RunStats> }`. The `systemHealth`
+    widget's uptime column now issues ONE request for all systems instead of one
+    `getRunStats` per system. Systems with no runs in the window are omitted, so
+    the resolver's output is unchanged.
+  - `incident-common`: new `getBulkIncidentUpdates({ incidentIds })` ->
+    `{ updates: Record<incidentId, IncidentUpdate[]> }`. The incidents widget now
+    fetches every selected incident's update timeline in ONE request instead of
+    one `getIncident` per incident.
+  - `maintenance-common`: new `getBulkMaintenanceUpdates({ maintenanceIds })` ->
+    `{ updates: Record<maintenanceId, MaintenanceUpdate[]> }` (symmetric with the
+    incident endpoint) for the maintenance widget.
+
+  The new update endpoints apply the same per-item audience filter as
+  `getIncident` / `getMaintenance`, so internal/logged-in updates and author
+  identity never leak to a non-manager caller. Each endpoint is keyed by the
+  resource id and gated with the record post-filter (`recordKey`) matching the
+  single endpoint's read scope, mirroring `getBulkSystemHealthStatus` /
+  `getBulkIncidentsForSystems`. Widget DTO output is unchanged - this is a pure
+  request-count optimization.
+
+- 43e4484: Status page enhancements:
+
+  - Group-status widget can collapse its member rows while every member is
+    operational (auto-expanding on any issue or maintenance).
+  - New "Announcements" status-page widget, contributed fully externally by the
+    announcement plugin: it surfaces active `visibility: "all"` announcements
+    through a public-safe DTO (title/message/severity/timestamps only) and never
+    affects the page status rollup.
+  - Incident and maintenance widgets can scope by catalog GROUPS with per-system
+    exceptions. Scope is resolved at read time (`(systemIds ∪ members(groupIds)) −
+excludedSystemIds`), so members added to a group later are reflected
+    automatically. The builder gets a nested group/system picker.
+  - Incident and maintenance items on a public page link to dedicated public
+    detail pages, gated server-side to items the page's published widgets actually
+    surface (no enumeration, no internal-field leak). The custom-domain public
+    bundle gains a minimal in-memory router for the two detail pages.
+  - Fix the custom-domain "Cannot connect to Checkstack backend" screen: a
+    configured-but-not-servable custom domain now serves the lean public
+    "not available" page instead of the admin shell; the public bundle skips the
+    cross-origin `/api/config` probe; CORS admits resolved custom domains; the
+    request origin is normalized for proxy scheme/port variance; and re-saving an
+    unchanged custom domain no longer clears its verification.
+  - Anonymous email subscriptions (double opt-in) for incident updates, opt-in per
+    status page (`emailSubscriptionsEnabled`, default off): a new
+    `status_page_subscribers` table, public subscribe/verify/unsubscribe
+    procedures with constant-time responses that fail closed when the page has not
+    enabled subscriptions, and team-scoped admin list/remove + an enable toggle in
+    the builder. Emails are delivered through a new `sendRawEmail` primitive in
+    notification-backend that sends to an arbitrary external address (no auth
+    account) via every enabled email strategy (SMTP), with a mandatory unsubscribe
+    link.
+  - Incident/maintenance update fan-out to subscribers via a new
+    `notificationAudienceExtensionPoint` in notification-backend. Every
+    notification funnelled through `notifyForSubscription` (incident, maintenance,
+    health - all unchanged) now also invokes each registered audience sink exactly
+    once, enriched with the affected systems and their catalog groups (resolved
+    from notification-backend's own resource-parent graph, never a domain import).
+    status-page-backend contributes a sink that, AT SEND TIME, matches each
+    notification's affected systems against the systems each published + public +
+    email-enabled page currently surfaces in its incident/maintenance widgets
+    (honoring group membership and per-system exclusions) and emails that page's
+    verified subscribers. Send-time scoping against the live layout is the privacy
+    boundary: a page only ever emails about systems its widgets surface right now.
+    Because `notifyForSubscription` is a single-pod point RPC, each notification
+    fans out exactly once cluster-wide.
+  - Subscriber reconcile on page deletion: the subscriber FK is `ON DELETE
+CASCADE` and page deletion also explicitly purges subscribers (invalidating
+    pending verify/unsubscribe tokens) - no orphan rows, no post-deletion send.
+    Removing all systems from a page or disabling email is intentionally NOT a
+    prune: send-time scoping plus the email-enabled gate make those subscribers
+    dormant with no data loss, and re-enabling restores the audience without a
+    re-subscribe.
+  - Send-time scoping is single-source: the fan-out asks each event-feed widget for
+    its CURRENT effective system scope (the same live catalog group expansion the
+    widget renders from) instead of a parallel copy of group membership, so it can
+    never over- or under-deliver relative to what the page shows.
+  - `sendRawEmail` in notification-backend is now `userType: "service"` (was an
+    authenticated procedure gated on `notification.send`). Sending to an arbitrary
+    address is an open-relay / email-bomb primitive, so it is callable only by a
+    trusted backend-to-backend caller (the status-page subscriber mailer), never by
+    an end user.
+  - Incident/maintenance widgets gain an optional per-system PUBLIC label override
+    (`systemLabels`), the same override path the system-health widget uses, so the
+    public incident/maintenance detail pages present clean labels instead of raw
+    catalog names.
+  - The anonymous subscribe endpoint adds a coarse per-page quota (max new
+    subscribers per rolling hour, counted over durable rows so it holds across
+    pods) on top of the per-(page,email) cooldown, capping verification-email
+    amplification. The quota is CONFIGURABLE per status page (new nullable
+    `email_subscribers_hourly_quota` column; null uses the default of 50, so
+    existing pages are unchanged), validated as a positive integer up to 5000,
+    editable in the builder next to the email opt-in toggle and gated by the same
+    page-manage capability.
+  - Email verification is now per-page configurable and backed by a platform-global
+    once-per-address registry:
+    - New `email_verification_required` column (boolean, default true) on
+      `status_pages`, exposed on the admin StatusPage DTO + `updateStatusPage`
+      input (same page-manage gate) with a builder toggle. When OFF, a new
+      subscriber is created active immediately - no verification email, and the
+      address is NOT written to the global registry (the operator's trust choice
+      for e.g. an internal page).
+    - New `status_page_verified_emails` table: one row per normalized address that
+      has completed verification on ANY page. When a verification-required page is
+      subscribed by an already-globally-verified address, the row is created active
+      immediately and a COURTESY email (with one-click unsubscribe) is sent instead
+      of a verification email, so a malicious add is always caught. `verify` upserts
+      the address into this registry and activates every other pending row for the
+      same address in one update (confirm once, all pages).
+    - Fan-out is unchanged: it still gates on the per-row `verified` flag; the
+      registry only governs whether a NEW subscribe short-circuits to active.
+
+  BREAKING CHANGE: `sendRawEmail` is now service-only. Any (non-existent in-tree)
+  authenticated caller must invoke it through a trusted service client instead.
+
+  Thanks to [@stuajnht](https://github.com/stuajnht) for the valuable feedback.
+
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+  - @checkstack/ai-backend@0.10.10
+  - @checkstack/automation-backend@0.11.1
+  - @checkstack/catalog-common@2.7.0
+  - @checkstack/catalog-backend@1.7.0
+  - @checkstack/healthcheck-common@1.16.0
+  - @checkstack/backend-api@0.31.1
+  - @checkstack/incident-common@1.10.0
+  - @checkstack/incident-backend@1.12.0
+  - @checkstack/maintenance-common@1.10.0
+  - @checkstack/notification-common@1.6.0
+  - @checkstack/status-page-backend@0.5.0
+  - @checkstack/gitops-backend@0.5.22
+  - @checkstack/secrets-backend@0.3.4
+  - @checkstack/status-page-common@0.6.0
+  - @checkstack/satellite-backend@0.8.4
+  - @checkstack/sdk@0.127.1
+  - @checkstack/command-backend@0.2.22
+  - @checkstack/script-packages-backend@0.4.1
+
+## 1.18.0
+
+### Minor Changes
+
+- 8aae4e2: Count fanned-out environment slices in the dashboard's "X of Y checks failing".
+
+  The dashboard problem card counted CHECKS, so a system with a single check that
+  fans out to three environments showed "Unhealthy 1 of 1 checks failing" even
+  when only one of the three environments was failing. It now counts (check ×
+  environment) slices: that system reads "1 of 3 checks failing", and a system
+  with a three-environment check plus a single-environment check with one
+  environment failing reads "1 of 4 checks failing". An env-less check counts as a
+  single slice, so a system with no environments reads exactly as before.
+
+  The per-check status DTO (`SystemCheckStatus`, returned by
+  `getSystemHealthStatus` / `getBulkSystemHealthStatus` /
+  `getBulkSystemHealthMatrix`) gains two fields: `sliceCount` (environment slices
+  this check currently fans out to, always >= 1) and `failingSliceCount` (how many
+  of those slices are non-healthy). `deriveHealthcheckSignals` sums them across
+  checks for the honest numerator/denominator.
+
+- d0eddc9: Rework health-check scheduling to one recurring job per
+  `(configuration, system, environment)` slice and add a slow-check bulkhead so a
+  slow or unreachable check can no longer starve the healthy ones.
+
+  Previously a single recurring job per `(configuration, system)` fanned out over
+  every environment sequentially inside one tick, so the job held a concurrency
+  slot for the sum of all its environments, and a slow environment stalled its
+  siblings. Now each environment slice is its own recurring job that holds a slot
+  only for its own probe. A convergence reconciler (k8s-controller style) derives
+  the desired per-env job set from Postgres + catalog membership and converges the
+  queue toward it (schedule missing, cancel orphans, reschedule interval changes),
+  so it is self-healing across pods and stays correct as catalog membership
+  changes. It runs at boot, and system-scoped after an assignment or GitOps
+  change. `run_now` enqueues one one-off job per effective environment.
+
+  The system rollup (the bare `<systemId>` health entity every badge, SLO rule and
+  dependency map reads) is recomputed by an event-driven, debounced consumer that
+  subscribes to per-environment health changes and recomputes once per system per
+  window, instead of inline on every tick. Notifications stay owned by the
+  per-environment runs, so the rollup notification is structurally deduplicated.
+
+  The bulkhead classifies each slice's recent runs: a slice whose last K runs were
+  slow transport failures (held its slot ~the full timeout) is admitted to a
+  capped, pod-local lane (single-flight per slice) and probed with a timeout shrunk
+  toward its own healthy-latency baseline, or DEFERRED (recording nothing, freeing
+  the slot) when the lane is full or a prior run is still in flight. The adaptive
+  timeout has four deadlock guardrails: no baseline means no shrink, the baseline
+  uses only healthy runs, every Nth suspect run re-probes at the full timeout, and
+  an absolute floor. A healthy slice is never gated and always runs at the full
+  timeout. A new `checkstack.healthcheck.deferred{reason}` counter records
+  bulkhead deferrals.
+
+  Measured with the scale harness (240 checks, 20% unreachable, concurrency 10, 5s
+  timeout, 35s): with the bulkhead off the queue backlog climbs unbounded to 774
+  while 60 slow checks pin slots; with it on the backlog stays bounded (drains to
+  0), completions roughly triple (288 → 862), and slot-pinning timeouts drop
+  (60 → 12) as 207 suspect runs are deferred.
+
+  `@checkstack/test-utils-backend` gains a `withTransactionMock` helper that adds a
+  `.transaction(cb)` passthrough to a mock database, so tests can exercise code
+  that batches reads/writes through `withScopedTransaction`.
+
+  BREAKING CHANGE: the internal `HealthCheckJobPayload` now requires an
+  `environmentId` field and recurring health-check job IDs are per-environment
+  (`healthcheck:<config>:<system>[:<env>]`). This is an internal queue contract
+  with no external package API surface; on upgrade the reconciler cancels the
+  old-format jobs and schedules the per-environment set at boot.
+
+- 8aae4e2: Show the last successful run per check (or per check+environment when fanned
+  out) in the system overview.
+
+  Each overview row that is currently degraded or unhealthy now shows when it was
+  last healthy (for example "Healthy until 2h ago", or "Never healthy" when it has
+  never succeeded), so operators can see at a glance since when a system has been
+  degraded or unhealthy without opening the drawer.
+
+  `getSystemHealthOverview` gains a `lastSuccessfulRunAt` field at both the check
+  level (most recent healthy run across all of the check's environments) and per
+  environment (`perEnvironment[].lastSuccessfulRunAt`). It is computed with a
+  dedicated max-per-environment aggregate query OUTSIDE the bounded sparkline
+  window, so it stays accurate even when a check has been failing for far longer
+  than the last runs shown in the sparkline.
+
+### Patch Changes
+
+- 8aae4e2: Stop sending a duplicate notification when a fanned-out system goes unhealthy.
+
+  A health check that fans out across environments notified once per environment
+  ("... is unhealthy in environment X") AND once more for the system rollup
+  ("... is unhealthy") in the same tick, so operators received two notifications
+  describing the same outage. The rollup transition is always driven by the very
+  environment(s) that already notified, so the rollup notification is now
+  suppressed whenever any environment notified this tick. It is still sent as a
+  fallback when no environment notified (e.g. every per-env delivery was
+  suppressed by policy/maintenance or threw), so a real status change is never
+  left entirely unannounced, and a system with no environments is unaffected.
+
+  Only the redundant user-facing notification is dropped: the rollup state
+  transition is still recorded and the `SYSTEM_STATUS_CHANGED` signal is still
+  broadcast, so SLO downtime, the dependency graph, the frontend, and automations
+  (which subscribe to the per-env and rollup entity changes) are unchanged.
+
+- d0eddc9: Cut health-check connection churn and de-cluster the scheduling "thundering
+  herd" so per-run durations stop varying wildly for the same check against the
+  same target. Grounded in live OpenTelemetry phase histograms: per-run wall time
+  was dominated by TCP/TLS connection setup under a self-inflicted burst, not by
+  slow targets, CPU, or the database.
+
+  - **In-memory queue now honors `startDelay` in `scheduleRecurring`.** It was
+    silently dropped, so every recurring job (health checks included) fired
+    immediately on boot and then on a boot-anchored interval grid - keeping all
+    equal-interval checks phase-aligned forever. `scheduleRecurring` now defers the
+    first execution by `startDelay` and anchors the recurrence to that first fire,
+    matching the queue contract and the BullMQ backend's intent. Jobs scheduled
+    without `startDelay` are unchanged (first run is immediate).
+  - **The BullMQ queue now honors `startDelay` in `scheduleRecurring` too.** It also
+    dropped `startDelay`, and its `every` scheduler captures the grid phase from
+    whenever `upsertJobScheduler` first runs - so a bootstrap loop scheduling many
+    equal-interval jobs at ~the same instant handed them all the same phase.
+    `scheduleRecurring` now pins the first fire to `now + startDelay` via the
+    scheduler's `startDate`, which shifts the whole recurrence, so the same jittered
+    `startDelay` de-clusters checks on the Redis backend identically to the
+    in-memory one. Cron schedules (absolute times) are unaffected.
+  - **The health-check scheduler jitters each check's first fire** by a small,
+    deterministic fraction of its interval (stable across restarts, keyed on the
+    check). A synchronized set of checks now spreads across the interval instead of
+    hammering their targets at the same instant. Because the queue anchors the
+    recurrence to the first fire, this offset persists for every subsequent run.
+  - **The HTTP collector refreshes its TCP/TLS connect-timing probe in the
+    background, per origin, and never awaits it.** Bun's `fetch` already pools and
+    reuses connections across runs (verified: warm reuse survives 20s+ idle gaps),
+    but the timing probe opened a fresh handshake on EVERY run - mis-reporting the
+    reused request's real latency and doubling the connection count under a burst.
+    The probe now refreshes a per-origin sample at most once per TTL (60s) and runs
+    fully in the background: it is NEVER on a request's critical path. Pinned to one
+    resolved IP, the probe can be far slower than the reused fetch (e.g. an
+    intermittent IPv6 SYN retry the real request never pays), and per the collector
+    contract best-effort timing must never delay the check - the previous code
+    `await`ed it, so a slow probe's refresh run showed up as a latency outlier. The
+    `connect`/`tls` phases are now explicitly a cached, per-host estimate.
+  - **The run detail UI now labels the estimate.** The timing-breakdown caption
+    clarifies that DNS, wait, and transfer are measured on the request, while
+    connection and TLS setup are an estimate sampled from a periodic per-host probe
+    and cached briefly (about a minute), so an operator does not read the cached
+    connect/TLS value as a per-run measurement.
+
+  Behaviour is otherwise unchanged: health status and assertions are the same;
+  there are simply far fewer connections, the herd is spread out, and the timing
+  breakdown can no longer be inflated by a slow best-effort probe. No configuration
+  or API changes.
+
+- d0eddc9: Cut the per-tick database work of the health-check executor by batching
+  scoped-database queries, and fix a dashboard "Recent activity" rendering bug.
+
+  The scoped-database proxy has to wrap every standalone query in its own
+  transaction so `SET LOCAL search_path` applies to it, which means a hot path
+  issuing many sequential queries pays the `BEGIN` / `SET LOCAL` / `COMMIT`
+  round-trips once per query and checks a connection out that many times. Two
+  changes remove most of that overhead on the health-check path:
+
+  - **New `withScopedTransaction` helper (`@checkstack/backend-api`).** A reusable
+    primitive for running several scoped queries under a SINGLE `SET LOCAL
+search_path` transaction, plus `ScopedTransaction` / `ScopedQueryRunner`
+    types so a helper can accept either the scoped db or a transaction handle.
+    Use it on any scoped-db hot path that issues 2+ queries in sequence.
+  - **`getSystemHealthStatus` is now batched.** It was a `1 + N` read fan-out (one
+    associations query, then one run-window query per enabled check) run as `1 +
+N` separate proxy transactions. It now runs as ONE transaction. This is the
+    hottest read on the platform - each check tick reads it several times, and the
+    dashboard, RPC router, and AI system-signals all call it - so the reduction in
+    transaction volume and connection churn is broad. The reads are also now a
+    single consistent snapshot.
+  - **The executor's run + aggregate writes are batched.** Each persisted run
+    previously issued the run `INSERT`, the aggregate `SELECT`, and the aggregate
+    `UPSERT` as three separate proxy transactions; they now run in one
+    transaction and commit atomically (the run and the aggregate it feeds can no
+    longer be persisted apart).
+
+  Behaviour is unchanged: the derived health status, transition detection, and
+  signals are identical; only the number of database transactions per tick drops.
+
+  Also fixes a dashboard bug where the "Recent activity" feed generated React keys
+  from `configurationName` plus a millisecond timestamp, so results from different
+  systems sharing a check name that completed in the same millisecond collided on
+  one key and React mis-reconciled the list (visually duplicated/omitted entries).
+  Keys are now derived from the system, configuration, and environment ids.
+
+- d0eddc9: Add opt-in OpenTelemetry metrics with a Prometheus exporter so a performance
+  investigation can be grounded in real numbers from a running instance instead of
+  guesses.
+
+  The layer is **off by default and free when off**: the instruments are OTel
+  no-ops until a `MeterProvider` is registered, so the hot paths pay nothing until
+  you opt in.
+
+  - **`@checkstack/backend-api` gains an `instrumentation` module** exporting lazy,
+    memoized instrument accessors any plugin can record through:
+    `dbTransactionsCounter`, `dbQueriesCounter`, `healthcheckExecutionHistogram`,
+    `healthcheckPhaseHistogram`, `queueEnqueuedCounter`, `queueProcessedCounter`.
+    Each looks up its instrument once and is a no-op until the host registers a
+    provider, so callers can record unconditionally.
+  - **`@checkstack/backend` owns the SDK bootstrap.** `startMetrics()` registers a
+    global `MeterProvider` + Prometheus exporter when `CHECKSTACK_METRICS_ENABLED`
+    is set (host `127.0.0.1`, port `9464` by default, both overridable via
+    `CHECKSTACK_METRICS_HOST` / `CHECKSTACK_METRICS_PORT`). The exporter runs its
+    OWN HTTP server, NOT a route on the app, so it carries no app-auth surface. It
+    also registers host-owned observable instruments:
+    `checkstack.db.pool.connections` (admin/lock pool active/idle/waiting) and
+    `checkstack.runtime.event_loop_delay` (setInterval-drift histogram = JS-thread
+    block time).
+  - **The scoped-DB proxy records DB transactions/queries per plugin schema**, so
+    `db_transactions_total` minus `db_queries_total` per schema is exactly the
+    number of batched transactions - a live check that `withScopedTransaction`
+    batching is taking effect.
+  - **The health-check executor records execution + per-phase histograms**
+    (`connect`, `wait`, ...) so a high `connect` p95 with a low `wait` points at
+    connection establishment rather than a slow target or a CPU-bound platform.
+  - **The in-memory queue records enqueued/processed counters** per queue and
+    status.
+
+  No behaviour changes when disabled. Enable with `CHECKSTACK_METRICS_ENABLED=1`
+  and scrape `http://127.0.0.1:9464/metrics`. See the backend observability guide
+  for the full metric list and interpretation.
+
+- Updated dependencies [8aae4e2]
+- Updated dependencies [f93ee7a]
+- Updated dependencies [f93ee7a]
+- Updated dependencies [8aae4e2]
+- Updated dependencies [d0eddc9]
+- Updated dependencies [d0eddc9]
+- Updated dependencies [f93ee7a]
+- Updated dependencies [f93ee7a]
+- Updated dependencies [d0eddc9]
+- Updated dependencies [d0eddc9]
+- Updated dependencies [8aae4e2]
+- Updated dependencies [d0eddc9]
+- Updated dependencies [f93ee7a]
+  - @checkstack/healthcheck-common@1.15.0
+  - @checkstack/common@0.22.0
+  - @checkstack/catalog-common@2.6.3
+  - @checkstack/ai-backend@0.10.9
+  - @checkstack/backend-api@0.31.0
+  - @checkstack/automation-backend@0.11.0
+  - @checkstack/incident-common@1.9.0
+  - @checkstack/incident-backend@1.11.0
+  - @checkstack/maintenance-common@1.9.0
+  - @checkstack/script-packages-backend@0.4.0
+  - @checkstack/satellite-backend@0.8.3
+  - @checkstack/sdk@0.126.1
+  - @checkstack/ai-common@0.6.6
+  - @checkstack/cache-api@0.3.19
+  - @checkstack/catalog-backend@1.6.9
+  - @checkstack/command-backend@0.2.21
+  - @checkstack/gitops-backend@0.5.21
+  - @checkstack/gitops-common@0.7.3
+  - @checkstack/notification-common@1.5.3
+  - @checkstack/queue-api@0.3.19
+  - @checkstack/secrets-backend@0.3.3
+  - @checkstack/secrets-common@0.3.2
+  - @checkstack/signal-common@0.2.17
+  - @checkstack/status-page-backend@0.4.8
+  - @checkstack/status-page-common@0.5.3
+  - @checkstack/cache-utils@0.2.24
+
 ## 1.17.0
 
 ### Minor Changes

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@checkstack/healthcheck-backend",
-  "version": "1.17.0",
+  "version": "1.19.0",
   "license": "Elastic-2.0",
   "type": "module",
   "main": "src/index.ts",
@@ -14,32 +14,32 @@
     "lint:code": "eslint . --max-warnings 0"
   },
   "dependencies": {
-    "@checkstack/ai-backend": "0.10.8",
-    "@checkstack/ai-common": "0.6.5",
-    "@checkstack/automation-backend": "0.10.10",
-    "@checkstack/backend-api": "0.30.0",
-    "@checkstack/cache-api": "0.3.18",
-    "@checkstack/cache-utils": "0.2.23",
-    "@checkstack/catalog-backend": "1.6.8",
-    "@checkstack/catalog-common": "2.6.2",
-    "@checkstack/command-backend": "0.2.20",
-    "@checkstack/common": "0.21.0",
-    "@checkstack/gitops-backend": "0.5.20",
-    "@checkstack/gitops-common": "0.7.2",
-    "@checkstack/healthcheck-common": "1.14.0",
-    "@checkstack/incident-backend": "1.10.0",
-    "@checkstack/incident-common": "1.8.0",
-    "@checkstack/maintenance-common": "1.8.2",
-    "@checkstack/notification-common": "1.5.2",
-    "@checkstack/queue-api": "0.3.18",
-    "@checkstack/satellite-backend": "0.8.2",
-    "@checkstack/script-packages-backend": "0.3.24",
-    "@checkstack/sdk": "0.125.1",
-    "@checkstack/secrets-backend": "0.3.2",
-    "@checkstack/secrets-common": "0.3.1",
-    "@checkstack/signal-common": "0.2.16",
-    "@checkstack/status-page-backend": "0.4.7",
-    "@checkstack/status-page-common": "0.5.2",
+    "@checkstack/ai-backend": "0.10.10",
+    "@checkstack/ai-common": "0.6.6",
+    "@checkstack/automation-backend": "0.11.1",
+    "@checkstack/backend-api": "0.31.1",
+    "@checkstack/cache-api": "0.3.19",
+    "@checkstack/cache-utils": "0.2.24",
+    "@checkstack/catalog-backend": "1.7.0",
+    "@checkstack/catalog-common": "2.7.0",
+    "@checkstack/command-backend": "0.2.22",
+    "@checkstack/common": "0.22.0",
+    "@checkstack/gitops-backend": "0.5.22",
+    "@checkstack/gitops-common": "0.7.3",
+    "@checkstack/healthcheck-common": "1.16.0",
+    "@checkstack/incident-backend": "1.12.0",
+    "@checkstack/incident-common": "1.10.0",
+    "@checkstack/maintenance-common": "1.10.0",
+    "@checkstack/notification-common": "1.6.0",
+    "@checkstack/queue-api": "0.3.19",
+    "@checkstack/satellite-backend": "0.8.4",
+    "@checkstack/script-packages-backend": "0.4.1",
+    "@checkstack/sdk": "0.127.1",
+    "@checkstack/secrets-backend": "0.3.4",
+    "@checkstack/secrets-common": "0.3.2",
+    "@checkstack/signal-common": "0.2.17",
+    "@checkstack/status-page-backend": "0.5.0",
+    "@checkstack/status-page-common": "0.6.0",
     "@hono/zod-validator": "^0.7.6",
     "@orpc/contract": "^1.14.4",
     "@orpc/server": "^1.14.4",
@@ -52,13 +52,16 @@
   },
   "devDependencies": {
     "@checkstack/drizzle-helper": "0.0.6",
-    "@checkstack/scripts": "0.7.2",
-    "@checkstack/test-utils-backend": "0.1.54",
+    "@checkstack/scripts": "0.7.3",
+    "@checkstack/test-utils-backend": "0.1.56",
     "@checkstack/tsconfig": "0.0.7",
     "@types/bun": "^1.0.0",
+    "@types/pg": "^8.20.0",
     "@types/tdigest": "^0.1.5",
+    "bullmq": "^5.66.4",
     "date-fns": "^4.4.0",
     "drizzle-kit": "^0.31.10",
+    "pg": "^8.21.0",
     "typescript": "^5.0.0"
   }
 }