@checkstack/healthcheck-backend 1.18.0 → 1.20.0

package/CHANGELOG.md

@@ -1,5 +1,489 @@
 # @checkstack/healthcheck-backend
 
+## 1.20.0
+
+### Minor Changes
+
+- bd41130: perf(healthcheck): stop recomputing the full system rollup on every check run
+
+  The queue run executor captured the system-wide rollup health
+  (`getSystemHealthStatus(systemId)`) at the start of EVERY check tick - a
+  worst-wins aggregate that fans out an N+1 of windowed `health_check_runs` reads
+  across every check × environment of the system. That value was only ever
+  consumed on the rare catastrophic-failure path (a job that throws before running
+  any probe); the normal success/failure paths record their transition from the
+  per-environment pre-read and never touched it. Under load this was one of the
+  heaviest repeated reads on the hot path.
+
+  The rollup pre-status is now computed lazily, only inside the catastrophic-
+  failure branch that actually uses it. Behavior is unchanged - the catastrophic
+  path reads the same pre-tick rollup (it is reached only when the run threw before
+  inserting anything, so nothing changed in between) - but every normal check tick
+  no longer pays for a full rollup recompute it discards.
+
+- bd41130: perf(healthcheck): add system-leading aggregate and config-reverse indexes
+
+  Add two Postgres indexes (migration 0020) to serve reads that the existing
+  keys cannot cover:
+
+  - `health_check_aggregates_system_bucket_idx` on
+    `(system_id, bucket_size, bucket_start)`. The health-state read omits
+    `configuration_id`, so the leading-`configuration_id` unique index could
+    not be used and the query scanned the aggregates table. This index leads
+    with `system_id` so those reads use an index instead.
+  - `system_health_checks_config_enabled_idx` on `(configuration_id, enabled)`.
+    The reverse lookup in `getSystemIdsForConfiguration` (config-change
+    recompute) filters by `configuration_id`, but the primary key leads with
+    `system_id` and could not serve it. This index makes the config-scoped
+    lookup an index scan.
+
+- bd41130: perf(healthcheck): add the missing composite indexes on health_check_runs
+
+  The status read path reads the last N runs for a (system, check[, environment])
+  slice ordered by `timestamp DESC` on every status read AND on every check
+  execution, but `health_check_runs` had NO secondary indexes - only its primary
+  key. Every such read was a full sequential scan of the (multi-million-row) table
+  plus an in-memory sort, so point reads averaged 50-320 ms and dominated total DB
+  time. Two composite indexes now back these access patterns:
+
+  - `health_check_runs_check_recent_idx` (system_id, configuration_id, timestamp) -
+    the cross-environment newest-run reads and the retention `DELETE`.
+  - `health_check_runs_slice_recent_idx` (system_id, configuration_id,
+    environment_id, timestamp) - the env-scoped slice reads, the per-check
+    DISTINCT-environment discovery, and the per-env last-healthy `max(timestamp)`
+    group-by.
+
+  Both turn full-table seq-scans into index range scans (Postgres scans the btree
+  backward for the `DESC` order).
+
+  > [!IMPORTANT]
+  > Deploy note: the migration builds the indexes with a plain (non-CONCURRENT)
+  > `CREATE INDEX`, which briefly locks writes to `health_check_runs` while each
+  > index builds (the migrator runs every migration in one transaction, so
+  > `CREATE INDEX CONCURRENTLY` is not possible through it). On a very large table
+  > you can build them `CONCURRENTLY` by hand (same names) before deploying; the
+  > migration uses `IF NOT EXISTS`, so it then no-ops.
+
+- bd41130: perf(healthcheck): cache system health status on the shared distributed cache with per-check-vector invalidation
+
+  The per-system derived health status (`getSystemHealthStatus`) is an N+1 over
+  `health_check_runs` across every check × environment, and it backs the highest
+  call-count read paths: the dashboard badges, the bulk status endpoint, the
+  per-(system, check, environment) matrix the dependency map and status-page
+  widgets consume, and the AI system-signals scan. It was only cached for the
+  single/bulk rollup, was invalidated UNCONDITIONALLY on every check run (so a
+  steady-state healthy system evicted its own cache every tick), the matrix
+  endpoint was not cached at all, and the AI signals scan bypassed the cache with
+  its own uncached N+1.
+
+  All four reads now go through a single `HealthCheckCache` facade - built on the
+  **platform `CacheManager`** - that is the ONE sanctioned reader AND invalidator
+  of a system's status:
+
+  - **Reads** (`read` / `readBulk` / `readMatrix`) are served read-through, keyed
+    per `(system, environment)`, holding the RAW (pre-incident-override) status;
+    the router folds incident overrides downstream, so an incident change never
+    touches this cache. The matrix reuses the same per-environment entries the
+    badge path warms. The AI signals contributor now scans candidate systems from
+    the durable table and resolves their statuses through `readBulk`, reusing the
+    warm cache instead of a fresh N+1.
+  - **Invalidation is change-gated on the per-check status VECTOR**, not the run:
+    `reconcile(previous, next)` evicts only when a check actually flipped status
+    (or its slice-failure composition changed) - a `statusFingerprint` invariant
+    to the volatile `evaluatedAt` / `lastRunAt` / `runsConsidered`. A run that
+    leaves the vector unchanged keeps the cache warm. This also catches a per-check
+    flip that leaves the rollup enum unchanged (which the reactive `health` entity
+    view would miss). A per-environment run that changes its slice evicts BOTH its
+    env key AND the system rollup key (the slice feeds the worst-wins rollup), so a
+    simultaneous slice swap - one env recovering as another fails, which the
+    rollup's own fingerprint is blind to - still refreshes the rollup. Sibling
+    environment keys stay warm.
+
+  Cross-pod coherence comes from the SHARED cache backend, not from an application
+  broadcast: with a distributed provider (Redis) an eviction is a `delete` every
+  pod sees immediately. On the default in-memory backend the cache is per-pod and
+  therefore single-instance-only (the Infrastructure Cache UI now warns about
+  this). The cached value is a derivation of the shared `health_check_runs` tables,
+  so a miss recomputes the same answer on every pod; the 15s TTL is only a
+  natural-refresh safety net.
+
+  Enforced by design, not convention:
+
+  - Every status-mutating writer invalidates through the facade: the run executor,
+    the router config/assignment/satellite handlers, the system/satellite lifecycle
+    hooks, AND the GitOps apply path (create/update/delete/associate/disassociate),
+    which writes configs directly on the service rather than through the router and
+    would otherwise have stranded a stale status until the TTL.
+  - A `checkstack/no-direct-system-status-read` lint rule (error) forbids raw
+    `service.getSystemHealthStatus(...)` reads anywhere except the cache facade and
+    the executor / entity-compute paths that must read live to detect a transition.
+  - A `checkstack/no-direct-health-run-insert` lint rule (error) forbids raw
+    `insert(healthCheckRuns)` outside the executor / service run writers.
+
+  The executor's per-run change-gate reads its pre-run baseline INSIDE the
+  per-(system, environment) advisory-lock critical section (not before the probe),
+  so a concurrent same-slice run cannot commit between the baseline read and the
+  insert and cause the gate to miss a real transition.
+
+  Behavior is unchanged for readers (same values, strictly fresher than the prior
+  15s-stale-on-quiet-systems behavior). The `getSystemHealthStatus` /
+  `getBulkSystemHealthStatus` / `getBulkSystemHealthMatrix` RPC contracts are
+  untouched, so cross-plugin callers (dependency, SLO, status-page) need no change.
+
+- bd41130: fix(status-page): scope email subscriptions to published environments and author-selected systems
+
+  Two correctness fixes to status-page email subscriptions:
+
+  - **Health notifications now respect the page's published environments.** A
+    per-environment health transition carries the environment it happened in
+    (`originEnvironmentId`, threaded through `notifyForSubscription` ->
+    `NotificationAudienceEvent` -> the status-page fan-out). A page that publishes
+    a specific environment set is now skipped for a change in an environment it
+    does not publish - so a `development` failure never emails a prod-only page's
+    subscribers, even for a system that is also shown in prod. Pages publishing all
+    environments, and env-less sources (incident, maintenance, whole-system health
+    rollup), are unaffected.
+  - **Notifications are scoped per category to the widgets the author placed.** The
+    send-time fan-out now surfaces a notification only through widgets of its own
+    category: a health status change reaches a page only through a HEALTH widget
+    (`banner` / `systemHealth` / `groupStatus` / `uptime`, which now implement
+    `resolveScopedSystems` and declare `subscriptionCategory: "health"`), an
+    incident only through an incident widget, and so on. A page that lists a
+    system's incidents but never its health no longer emails health subscribers
+    about it, and a health-only page now correctly surfaces its systems for
+    subscription. Health widgets also participate in the public subscribe picker.
+
+  BREAKING CHANGE: on a page publishing a specific environment set, health
+  subscribers now only receive changes that occurred in a published environment
+  (previously any environment of a surfaced system triggered a notification), and a
+  notification is surfaced only by a widget of its own category (previously any
+  scoping widget on the page could surface any category). Legacy subscribers (NULL
+  categories) and all-environment pages are unchanged; no data migration is needed.
+
+### Patch Changes
+
+- Updated dependencies [bd41130]
+- Updated dependencies [bd41130]
+- Updated dependencies [bd41130]
+- Updated dependencies [bd41130]
+- Updated dependencies [bd41130]
+- Updated dependencies [bd41130]
+- Updated dependencies [bd41130]
+- Updated dependencies [bd41130]
+  - @checkstack/backend-api@0.32.0
+  - @checkstack/cache-utils@0.3.0
+  - @checkstack/catalog-backend@1.8.0
+  - @checkstack/ai-backend@0.10.11
+  - @checkstack/incident-backend@1.13.0
+  - @checkstack/notification-common@1.7.0
+  - @checkstack/status-page-backend@0.6.0
+  - @checkstack/automation-backend@0.11.2
+  - @checkstack/command-backend@0.2.23
+  - @checkstack/gitops-backend@0.5.23
+  - @checkstack/satellite-backend@0.8.5
+  - @checkstack/script-packages-backend@0.4.2
+  - @checkstack/secrets-backend@0.3.5
+  - @checkstack/catalog-common@2.7.1
+  - @checkstack/sdk@0.128.1
+  - @checkstack/healthcheck-common@1.16.1
+  - @checkstack/incident-common@1.10.1
+  - @checkstack/maintenance-common@1.10.1
+  - @checkstack/status-page-common@0.6.1
+
+## 1.19.0
+
+### Minor Changes
+
+- 43e4484: Fix an N+1 in the catalog manager: the per-system "Health Checks" count badge
+  fired one `getSystemAssociations` request per system row, each holding a pooled
+  Postgres connection that contended with the background health-check run
+  executor and could exhaust the pool on large catalogs.
+
+  - Add `getBulkAssignedHealthCheckCounts({ systemIds })` to healthcheck, which
+    returns per-system assignment counts (0 for systems with no assignments) from
+    ONE grouped `COUNT(*) ... GROUP BY system_id` query. Read authorization
+    matches the per-system endpoint it replaces (`configuration.read` +
+    `catalog.system` read via `recordKey`), so a team-scoped user only sees counts
+    for systems they may read.
+  - `CatalogSystemActionsSlot` now passes `visibleSystemIds` (every system id in
+    the row's list) so a per-row filler can bulk-fetch for the whole visible set
+    in a single deduped request instead of one request per row. This mirrors how
+    `CatalogBrowseHealthSlot` / `SystemSignalsSlot` already pass `systemIds`.
+  - The health-check count badge now reads its count from that one deduped bulk
+    query. N visible rows cause 1 request instead of N.
+
+  State & scale: the counts are derived on read from the shared
+  `system_health_checks` table, so every pod returns the same answer; no
+  process-local or duplicated state is introduced.
+
+- 43e4484: Name the failing health check in system-health notifications. The notification
+  body now names the check that drove the transition (in addition to the system
+  and environment), and a `healthcheck.healthcheck` subject is pushed alongside
+  the `catalog.system` subject, deep-linked to the check's run history. Recovery
+  notifications stay system-level. Adds a `createHealthcheckSubject` builder to
+  `healthcheck-common`.
+
+  Thanks to [@stuajnht](https://github.com/stuajnht) for the valuable feedback.
+
+- 43e4484: Status pages can now publish only a subset of catalog environments. The page
+  builder gains a "Published environments" picker (empty = all environments, the
+  backward-compatible default). When a non-empty set is selected, the page omits
+  status, incidents, maintenances and uptime for systems that belong to none of
+  the selected environments.
+
+  - Status pages store an optional `publishedEnvironmentIds` set (new nullable
+    `published_environment_ids` column; NULL = all environments, so existing pages
+    are unchanged) exposed on `StatusPage`, `createStatusPage`, and
+    `updateStatusPage`.
+  - The scope is threaded onto `WidgetResolveContext.publishedEnvironmentIds` as
+    opaque strings and passed identically to `resolvePublic`,
+    `resolveScopedSystems`, and `resolveScopedSystemsDetailed` (and the email
+    subscribe clamp + fan-out), so what a page shows, offers for subscription, and
+    emails about all agree.
+  - Health widgets recompute per environment: they read the per-environment health
+    matrix and roll up only the selected environments. `getBulkRunStats` and
+    `getRunStats` gain an optional `environmentIds` filter so uptime counts only
+    runs recorded in the selected environments.
+  - Incident and maintenance widgets filter their feed and scope by intersecting
+    each item's affected systems with the environment-visible systems. Incidents
+    and maintenance windows carry no environment of their own, so a system in
+    several environments makes its items visible on a page publishing ANY of them
+    (the multi-environment caveat).
+
+### Patch Changes
+
+- 43e4484: fix(healthcheck): disabling an environment for an assignment now clears its stale slice from the rollup and overview immediately
+
+  Disabling an environment for a health-check assignment (removing it from the
+  assignment's `environmentIds`) stopped that environment from fanning out, but a
+  check that was FAILING there kept dragging the system health rollup/badge to
+  unhealthy and kept showing as a live failing row in the system overview. Because
+  the rollup is recomputed by an event-driven consumer subscribed to per-env health
+  CHANGES, and a disabled env produces no further runs (so no change event fires),
+  the stale unhealthy status was never recomputed away - it only cleared
+  incidentally, once the disabled env's runs aged out of the bounded run window
+  (which needs the assignment's OTHER active environments to produce enough newer
+  runs first). With a single active/failing env, it could persist until retention.
+
+  Scope: this reconciles environments DISABLED/removed ON THE ASSIGNMENT (its
+  `systemHealthChecks.environmentIds` selector - switching to Specific and
+  deselecting, or None).
+
+  Fixes:
+
+  - The rollup aggregation (`getSystemHealthStatus`) and the per-check status in
+    `getSystemHealthOverview` now consider only CURRENTLY-EFFECTIVE environment
+    slices, derived from the durable `systemHealthChecks.environmentIds` selector
+    (catalog-free, identical on every pod). A slice whose environment was disabled
+    for the assignment, or the stale env-less slice of a check that now fans out,
+    no longer contributes.
+
+  Known limitation: under an "all-environments" assignment (`environmentIds` is
+  `null`), an environment removed only from the system's CATALOG MEMBERSHIP (rather
+  than disabled on the assignment) can still contribute to the backend rollup/badge
+  until the assignment is re-evaluated, because the rollup read path is
+  intentionally catalog-free for horizontal-scale correctness (it must return the
+  same answer on every pod without a per-read catalog lookup). This is pre-existing;
+  the frontend overview, which can see membership, still orphans such a slice.
+
+  - Each environment is now windowed by its OWN query in the rollup, instead of a
+    single shared `LIMIT` across the mixed-env pool. The old shared window
+    truncated per-env evaluation for checks that fan out to many environments (or
+    with large threshold windows); every environment now gets its full evaluation
+    depth.
+  - Changing an assignment's environment set now triggers an immediate rollup
+    recompute for that system, so the persisted `health` entity (badge + SLO
+    downtime) converges at once rather than waiting for stale runs to age out.
+  - The system-overview frontend tucks a slice whose environment was disabled for
+    the assignment under "Old checks" (system membership alone could not detect it,
+    since the environment is still part of the system). `getSystemHealthOverview`
+    now returns each check's `environmentIds` selector to drive this.
+
+  Shared pure helpers `selectorIncludesEnvironment` / `isEnvSliceEffective` /
+  `selectEffectiveEnvKeys` are added to `@checkstack/healthcheck-common` so the
+  backend and frontend agree on effective-slice detection.
+
+- 43e4484: Batch hot-path scoped-db reads/writes into single transactions to cut per-query round-trips.
+
+  The scoped-db proxy wraps every standalone query in its own `BEGIN → SET LOCAL search_path → query → COMMIT`, so a path issuing N sequential queries paid N round-trips and checked out a connection N times. These reads/writes now run under one `withScopedTransaction`, collapsing the batch to a single `SET LOCAL` on one connection. Behavior is unchanged:
+
+  - healthcheck: `getSystemHealthOverview`'s `1 + N·(2+E)` read fan-out.
+  - incident/maintenance: `getIncident`/`getMaintenance` (4 reads), `getManyEntityStates`, `listOpenIncidentsBySystem` / `getActiveMaintenancesBySystem`, `getMaintenanceWindowsForRange`; the `list*` / `*ForSystem` per-row `N+1` system lookups collapsed to a single set-based `inArray` read; maintenance `transitionStatus` update+insert made atomic; `addUpdate`/`editUpdate`/`addLink` use `.returning()` instead of a follow-up re-select.
+  - ai: `appendMessage`, memory `saveOrUpdate`.
+  - notification: `resolveInheritedGroups`.
+  - status-page: subscriber `verify` (4 reads) and `unsubscribe` (3 reads).
+  - announcement: `getActiveAnnouncements` / `dismissAnnouncement` / `createAnnouncement`.
+  - gitops: `upsertProvenance`.
+
+- 43e4484: Eliminate N+1 RPC fan-outs in the public status-page widget resolvers.
+
+  Each of these widgets renders a PUBLIC page, so every per-item RPC was real
+  external DB load. Three bulk-by-id endpoints replace the per-item fetches:
+
+  - `healthcheck-common`: new `getBulkRunStats({ systemIds, startDate, endDate,
+maxBuckets })` -> `{ stats: Record<systemId, RunStats> }`. The `systemHealth`
+    widget's uptime column now issues ONE request for all systems instead of one
+    `getRunStats` per system. Systems with no runs in the window are omitted, so
+    the resolver's output is unchanged.
+  - `incident-common`: new `getBulkIncidentUpdates({ incidentIds })` ->
+    `{ updates: Record<incidentId, IncidentUpdate[]> }`. The incidents widget now
+    fetches every selected incident's update timeline in ONE request instead of
+    one `getIncident` per incident.
+  - `maintenance-common`: new `getBulkMaintenanceUpdates({ maintenanceIds })` ->
+    `{ updates: Record<maintenanceId, MaintenanceUpdate[]> }` (symmetric with the
+    incident endpoint) for the maintenance widget.
+
+  The new update endpoints apply the same per-item audience filter as
+  `getIncident` / `getMaintenance`, so internal/logged-in updates and author
+  identity never leak to a non-manager caller. Each endpoint is keyed by the
+  resource id and gated with the record post-filter (`recordKey`) matching the
+  single endpoint's read scope, mirroring `getBulkSystemHealthStatus` /
+  `getBulkIncidentsForSystems`. Widget DTO output is unchanged - this is a pure
+  request-count optimization.
+
+- 43e4484: Status page enhancements:
+
+  - Group-status widget can collapse its member rows while every member is
+    operational (auto-expanding on any issue or maintenance).
+  - New "Announcements" status-page widget, contributed fully externally by the
+    announcement plugin: it surfaces active `visibility: "all"` announcements
+    through a public-safe DTO (title/message/severity/timestamps only) and never
+    affects the page status rollup.
+  - Incident and maintenance widgets can scope by catalog GROUPS with per-system
+    exceptions. Scope is resolved at read time (`(systemIds ∪ members(groupIds)) −
+excludedSystemIds`), so members added to a group later are reflected
+    automatically. The builder gets a nested group/system picker.
+  - Incident and maintenance items on a public page link to dedicated public
+    detail pages, gated server-side to items the page's published widgets actually
+    surface (no enumeration, no internal-field leak). The custom-domain public
+    bundle gains a minimal in-memory router for the two detail pages.
+  - Fix the custom-domain "Cannot connect to Checkstack backend" screen: a
+    configured-but-not-servable custom domain now serves the lean public
+    "not available" page instead of the admin shell; the public bundle skips the
+    cross-origin `/api/config` probe; CORS admits resolved custom domains; the
+    request origin is normalized for proxy scheme/port variance; and re-saving an
+    unchanged custom domain no longer clears its verification.
+  - Anonymous email subscriptions (double opt-in) for incident updates, opt-in per
+    status page (`emailSubscriptionsEnabled`, default off): a new
+    `status_page_subscribers` table, public subscribe/verify/unsubscribe
+    procedures with constant-time responses that fail closed when the page has not
+    enabled subscriptions, and team-scoped admin list/remove + an enable toggle in
+    the builder. Emails are delivered through a new `sendRawEmail` primitive in
+    notification-backend that sends to an arbitrary external address (no auth
+    account) via every enabled email strategy (SMTP), with a mandatory unsubscribe
+    link.
+  - Incident/maintenance update fan-out to subscribers via a new
+    `notificationAudienceExtensionPoint` in notification-backend. Every
+    notification funnelled through `notifyForSubscription` (incident, maintenance,
+    health - all unchanged) now also invokes each registered audience sink exactly
+    once, enriched with the affected systems and their catalog groups (resolved
+    from notification-backend's own resource-parent graph, never a domain import).
+    status-page-backend contributes a sink that, AT SEND TIME, matches each
+    notification's affected systems against the systems each published + public +
+    email-enabled page currently surfaces in its incident/maintenance widgets
+    (honoring group membership and per-system exclusions) and emails that page's
+    verified subscribers. Send-time scoping against the live layout is the privacy
+    boundary: a page only ever emails about systems its widgets surface right now.
+    Because `notifyForSubscription` is a single-pod point RPC, each notification
+    fans out exactly once cluster-wide.
+  - Subscriber reconcile on page deletion: the subscriber FK is `ON DELETE
+CASCADE` and page deletion also explicitly purges subscribers (invalidating
+    pending verify/unsubscribe tokens) - no orphan rows, no post-deletion send.
+    Removing all systems from a page or disabling email is intentionally NOT a
+    prune: send-time scoping plus the email-enabled gate make those subscribers
+    dormant with no data loss, and re-enabling restores the audience without a
+    re-subscribe.
+  - Send-time scoping is single-source: the fan-out asks each event-feed widget for
+    its CURRENT effective system scope (the same live catalog group expansion the
+    widget renders from) instead of a parallel copy of group membership, so it can
+    never over- or under-deliver relative to what the page shows.
+  - `sendRawEmail` in notification-backend is now `userType: "service"` (was an
+    authenticated procedure gated on `notification.send`). Sending to an arbitrary
+    address is an open-relay / email-bomb primitive, so it is callable only by a
+    trusted backend-to-backend caller (the status-page subscriber mailer), never by
+    an end user.
+  - Incident/maintenance widgets gain an optional per-system PUBLIC label override
+    (`systemLabels`), the same override path the system-health widget uses, so the
+    public incident/maintenance detail pages present clean labels instead of raw
+    catalog names.
+  - The anonymous subscribe endpoint adds a coarse per-page quota (max new
+    subscribers per rolling hour, counted over durable rows so it holds across
+    pods) on top of the per-(page,email) cooldown, capping verification-email
+    amplification. The quota is CONFIGURABLE per status page (new nullable
+    `email_subscribers_hourly_quota` column; null uses the default of 50, so
+    existing pages are unchanged), validated as a positive integer up to 5000,
+    editable in the builder next to the email opt-in toggle and gated by the same
+    page-manage capability.
+  - Email verification is now per-page configurable and backed by a platform-global
+    once-per-address registry:
+    - New `email_verification_required` column (boolean, default true) on
+      `status_pages`, exposed on the admin StatusPage DTO + `updateStatusPage`
+      input (same page-manage gate) with a builder toggle. When OFF, a new
+      subscriber is created active immediately - no verification email, and the
+      address is NOT written to the global registry (the operator's trust choice
+      for e.g. an internal page).
+    - New `status_page_verified_emails` table: one row per normalized address that
+      has completed verification on ANY page. When a verification-required page is
+      subscribed by an already-globally-verified address, the row is created active
+      immediately and a COURTESY email (with one-click unsubscribe) is sent instead
+      of a verification email, so a malicious add is always caught. `verify` upserts
+      the address into this registry and activates every other pending row for the
+      same address in one update (confirm once, all pages).
+    - Fan-out is unchanged: it still gates on the per-row `verified` flag; the
+      registry only governs whether a NEW subscribe short-circuits to active.
+
+  BREAKING CHANGE: `sendRawEmail` is now service-only. Any (non-existent in-tree)
+  authenticated caller must invoke it through a trusted service client instead.
+
+  Thanks to [@stuajnht](https://github.com/stuajnht) for the valuable feedback.
+
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+- Updated dependencies [43e4484]
+  - @checkstack/ai-backend@0.10.10
+  - @checkstack/automation-backend@0.11.1
+  - @checkstack/catalog-common@2.7.0
+  - @checkstack/catalog-backend@1.7.0
+  - @checkstack/healthcheck-common@1.16.0
+  - @checkstack/backend-api@0.31.1
+  - @checkstack/incident-common@1.10.0
+  - @checkstack/incident-backend@1.12.0
+  - @checkstack/maintenance-common@1.10.0
+  - @checkstack/notification-common@1.6.0
+  - @checkstack/status-page-backend@0.5.0
+  - @checkstack/gitops-backend@0.5.22
+  - @checkstack/secrets-backend@0.3.4
+  - @checkstack/status-page-common@0.6.0
+  - @checkstack/satellite-backend@0.8.4
+  - @checkstack/sdk@0.127.1
+  - @checkstack/command-backend@0.2.22
+  - @checkstack/script-packages-backend@0.4.1
+
 ## 1.18.0
 
 ### Minor Changes

package/drizzle/0019_chemical_frightful_four.sql

@@ -0,0 +1,8 @@
+-- NOTE: plain (non-CONCURRENT) CREATE INDEX takes a SHARE lock that blocks
+-- writes to health_check_runs while each index builds. The migrator wraps every
+-- migration in one transaction, so CREATE INDEX CONCURRENTLY (which cannot run
+-- inside a transaction) is not possible here. On a very large table you can
+-- pre-build these CONCURRENTLY by hand (outside the migrator, same names) before
+-- deploying; the IF NOT EXISTS below then makes this migration a no-op.
+CREATE INDEX IF NOT EXISTS "health_check_runs_check_recent_idx" ON "health_check_runs" USING btree ("system_id","configuration_id","timestamp");--> statement-breakpoint
+CREATE INDEX IF NOT EXISTS "health_check_runs_slice_recent_idx" ON "health_check_runs" USING btree ("system_id","configuration_id","environment_id","timestamp");

package/drizzle/0020_certain_mordo.sql

@@ -0,0 +1,2 @@
+CREATE INDEX "health_check_aggregates_system_bucket_idx" ON "health_check_aggregates" USING btree ("system_id","bucket_size","bucket_start");--> statement-breakpoint
+CREATE INDEX "system_health_checks_config_enabled_idx" ON "system_health_checks" USING btree ("configuration_id","enabled");