@checkstack/healthcheck-backend 0.18.1 → 1.0.1

package/CHANGELOG.md

@@ -1,5 +1,176 @@
 # @checkstack/healthcheck-backend
 
+## 1.0.1
+
+### Patch Changes
+
+- 2a749d3: fix: run afterPluginsReady in topological order; merge daily rollups on conflict
+
+  Two resilience fixes for the dependency chain:
+
+  1. **Plugin loader**: Phase 3 (`afterPluginsReady`) now iterates plugins
+     in the same topologically-sorted order as Phase 2 (`init`). Previously
+     it iterated `pendingInits` in registration order, which raced
+     subscription-spec dependencies — catalog's afterPluginsReady registers
+     `catalog.system` and `catalog.group` notification targets, and emitting
+     plugins (incident, maintenance, …) call `registerSubscriptionSpec`
+     against those targets in their own afterPluginsReady. With registration
+     order, an emitter could run before catalog and hit
+     `Target type catalog.group is not registered`. Sorted order encodes
+     the dependency via `spec.target.ownerPlugin`, so the emitter now
+     always runs after the target owner.
+
+  2. **Healthcheck retention job**: the daily rollup now upserts
+     `health_check_aggregates` with `ON CONFLICT DO UPDATE` instead of a
+     plain insert. Previously, late-arriving hourly aggregates (e.g. from
+     a satellite that was offline when the prior rollup ran) would crash
+     the rollup with a unique-constraint violation on
+     `(configuration_id, system_id, bucket_start, bucket_size, source_id)`.
+     The merge sums counts and folds min/max/p95 into the existing daily
+     row.
+
+  - @checkstack/satellite-backend@0.2.19
+
+## 1.0.0
+
+### Major Changes
+
+- 32d52c6: feat: notification target pattern + per-spec subscriptions
+
+  Replaces the all-or-nothing catalog system/group notification model with a
+  platform-level target pattern. Each notification-emitting plugin declares
+  _subscription specs_ against typed _target_ objects exported from the
+  target's owning plugin (catalog ships `catalogSystemTarget` and
+  `catalogGroupTarget`). Notification-backend handles every per-resource
+  group lifecycle, parent-edge inheritance, and legacy-subscription seeding
+  — plugins never author groupId helpers, lifecycle hooks, or migration
+  code again.
+
+  **Plugin-author surface area is now ~12 lines per emitter:**
+
+  ```ts
+  // <plugin>-common
+  const { defineSubscription } = createSubscriptionFactory(pluginMetadata);
+  export const fooSystemSubscription = defineSubscription({
+    localId: "system",
+    target: catalogSystemTarget,
+    display: { title: "Foo Alerts", description: "...", iconName: "Bell" },
+  });
+
+  // <plugin>-backend register()
+  env.registerSubscriptionSpecs([fooSystemSubscription]);
+  //   ^ feeds the plugin loader's dependency sorter — each spec's
+  //     target.ownerPlugin becomes an implicit init-order dep, so this
+  //     plugin automatically waits for catalog (the target owner) to
+  //     finish init + afterPluginsReady before its own runs.
+
+  // <plugin>-backend afterPluginsReady
+  await notificationClient.registerSubscriptionSpec(
+    specToRegistration(fooSystemSubscription)
+  );
+  // dispatch
+  await notificationClient.notifyForSubscription({
+    specId: fooSystemSubscription.specId,
+    resourceKeys: [systemId],
+    title,
+    body,
+    importance,
+    action,
+    collapseKey,
+    subjects,
+  });
+
+  // <plugin>-frontend
+  createNotificationSubscriptionExtension({ spec: fooSystemSubscription });
+  ```
+
+  **Migrated plugins**: anomaly, incident, maintenance, healthcheck,
+  dependency. Each lost its bespoke `notification-groups.ts`,
+  `bootstrap*NotificationGroups`, `ensure*Group`, and inheritance walk —
+  all of that is now centralized in notification-backend's
+  `subscription-engine`.
+
+  **Plugin loader change** (`@checkstack/backend-api`,
+  `@checkstack/backend`): the register-time API gains
+  `env.registerSubscriptionSpecs([...specs])`. The dependency sorter
+  walks `spec.target.ownerPlugin` for every declared spec and adds the
+  target owner as an init-order dependency of the emitting plugin. This
+  guarantees that catalog (the owner of the platform's `system` and
+  `group` targets) completes init + afterPluginsReady before any
+  emitting plugin tries to register its specs against the notification
+  service — no string-prefix heuristics, no manual `dependsOnPlugins`
+  list, no stub rows. Plugins that fail to declare their specs at
+  register time get a clear `Target type X is not registered. Did the
+emitting plugin declare this spec via env.registerSubscriptionSpecs?`
+  error from the dispatcher.
+
+  **Removed** (no backwards compat):
+
+  - `catalogClient.notifySystemSubscribers` and
+    `catalogClient.notifyManySystemSubscribers`
+  - `notificationClient.notifyUsers` and `notificationClient.notifyGroups`
+    as direct dispatch primitives — replaced by spec-bound
+    `notifyForSubscription`
+  - catalog's `bootstrapNotificationGroups` (replaced by
+    `bootstrapNotificationTargets`)
+
+  **Enforcement**: the dispatcher rejects calls referencing unregistered
+  specIds, specs owned by other plugins, or resourceKeys that haven't been
+  pushed via `upsertNotificationResource`. Display metadata for any
+  groupId is recoverable via the spec registry, so audit lists render
+  correct labels even when an emitter's frontend isn't loaded.
+
+  **Per-field anomaly mute** keeps working — it now lives inside the
+  generic SubscriptionRow's optional `SubControls` panel
+  (`AnomalyFieldMuteList`), exposed through the catalog system detail
+  page's notifications card.
+
+  The catalog system detail page renders a "Notifications" card hosting
+  `SystemNotificationSubscriptionsSlot`. The matching group surface is
+  not yet rendered — group-level subscriptions are wired end-to-end on
+  the backend; a follow-up will add the host UI.
+
+  **Migration of existing subscribers**: target types declare a
+  `legacyGroupIdTemplate`; on first registration of each spec,
+  notification-backend reads subscribers from the legacy
+  `catalog.system.<id>` / `catalog.group.<id>` groups and seeds the new
+  spec groups exactly once per (spec × resource) pair, tracked in
+  `subscription_migrations`. Anomaly stays opt-in (its target also
+  declares the template, but the user-explicit nature of the original
+  opt-in flow means the seeding produces the same set of subscribers
+  they already had).
+
+### Patch Changes
+
+- 32d52c6: Bulk notifications affecting multiple systems and collapse lifecycle events into a single card.
+
+  Notifications now carry an optional `subjects` array (the entities they affect) and an optional `collapseKey` (so related notifications collapse into one row per recipient). Incidents, maintenances, anomalies, healthchecks, and dependency-impact events route through these new fields, so an incident affecting three systems produces one in-app notification + one external send per subscriber instead of three. Lifecycle updates for the same entity (created → updated → resolved) also collapse, with an expandable "+N updates" timeline.
+
+  Subject kinds are namespaced as `<pluginId>.<localKind>` and built via type-safe helpers exported from each domain's common package (`createSystemSubject`, `incidentCollapseKey`, etc.). The frontend kind registry (`registerSubjectKind`) lets plugins bind icon + label for their kinds; unknown kinds fall back to a generic chip.
+
+  All notification strategies (SMTP, Slack, Discord, Teams, Telegram, Pushover, Gotify, Webex, Backstage) render the affected subjects natively in their format (HTML cards, Slack blocks, Discord embed fields, adaptive cards, markdown lists, etc.).
+
+- Updated dependencies [32d52c6]
+- Updated dependencies [32d52c6]
+- Updated dependencies [32d52c6]
+- Updated dependencies [32d52c6]
+- Updated dependencies [32d52c6]
+- Updated dependencies [32d52c6]
+  - @checkstack/gitops-backend@0.2.6
+  - @checkstack/integration-backend@0.1.22
+  - @checkstack/satellite-backend@0.2.18
+  - @checkstack/notification-common@1.0.0
+  - @checkstack/catalog-backend@1.0.0
+  - @checkstack/catalog-common@2.0.0
+  - @checkstack/incident-common@1.0.0
+  - @checkstack/maintenance-common@1.0.0
+  - @checkstack/healthcheck-common@1.0.0
+  - @checkstack/backend-api@0.14.0
+  - @checkstack/cache-api@0.2.2
+  - @checkstack/command-backend@0.1.22
+  - @checkstack/queue-api@0.2.16
+  - @checkstack/cache-utils@0.2.2
+
 ## 0.18.1
 
 ### Patch Changes

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@checkstack/healthcheck-backend",
-  "version": "0.18.1",
+  "version": "1.0.1",
   "type": "module",
   "main": "src/index.ts",
   "checkstack": {
@@ -13,22 +13,23 @@
     "lint:code": "eslint . --max-warnings 0"
   },
   "dependencies": {
-    "@checkstack/backend-api": "0.13.0",
-    "@checkstack/cache-api": "0.2.0",
-    "@checkstack/cache-utils": "0.2.0",
-    "@checkstack/catalog-backend": "0.7.0",
-    "@checkstack/catalog-common": "1.5.2",
-    "@checkstack/command-backend": "0.1.20",
+    "@checkstack/backend-api": "0.13.1",
+    "@checkstack/cache-api": "0.2.1",
+    "@checkstack/cache-utils": "0.2.1",
+    "@checkstack/catalog-backend": "0.7.1",
+    "@checkstack/catalog-common": "1.5.3",
+    "@checkstack/command-backend": "0.1.21",
     "@checkstack/common": "0.7.0",
-    "@checkstack/gitops-backend": "0.2.4",
+    "@checkstack/gitops-backend": "0.2.5",
     "@checkstack/gitops-common": "0.2.1",
-    "@checkstack/healthcheck-common": "0.12.0",
-    "@checkstack/incident-common": "0.4.9",
-    "@checkstack/integration-backend": "0.1.20",
-    "@checkstack/maintenance-common": "0.4.11",
-    "@checkstack/queue-api": "0.2.14",
-    "@checkstack/satellite-backend": "0.2.16",
-    "@checkstack/signal-common": "0.1.10",
+    "@checkstack/healthcheck-common": "0.13.0",
+    "@checkstack/incident-common": "0.5.0",
+    "@checkstack/integration-backend": "0.1.21",
+    "@checkstack/maintenance-common": "0.5.0",
+    "@checkstack/notification-common": "0.3.0",
+    "@checkstack/queue-api": "0.2.15",
+    "@checkstack/satellite-backend": "0.2.17",
+    "@checkstack/signal-common": "0.2.0",
     "@hono/zod-validator": "^0.7.6",
     "drizzle-orm": "^0.45.0",
     "hono": "^4.12.14",
@@ -39,11 +40,12 @@
   "devDependencies": {
     "@checkstack/drizzle-helper": "0.0.4",
     "@checkstack/scripts": "0.1.2",
-    "@checkstack/test-utils-backend": "0.1.20",
+    "@checkstack/test-utils-backend": "0.1.21",
     "@checkstack/tsconfig": "0.0.5",
     "@types/bun": "^1.0.0",
     "@types/tdigest": "^0.1.5",
     "date-fns": "^4.1.0",
+    "drizzle-kit": "^0.31.10",
     "typescript": "^5.0.0"
   }
 }

package/src/index.ts

@@ -10,7 +10,13 @@ import {
   pluginMetadata,
   healthCheckContract,
   healthcheckRoutes,
+  healthcheckSystemSubscription,
+  healthcheckGroupSubscription,
 } from "@checkstack/healthcheck-common";
+import {
+  NotificationApi,
+  specToRegistration,
+} from "@checkstack/notification-common";
 import {
   createBackendPlugin,
   coreServices,
@@ -67,6 +73,10 @@ export default createBackendPlugin({
   metadata: pluginMetadata,
   register(env) {
     env.registerAccessRules(healthCheckAccessRules);
+    env.registerSubscriptionSpecs([
+      healthcheckSystemSubscription,
+      healthcheckGroupSubscription,
+    ]);
 
     // Register hooks as integration events
     const integrationEvents = env.getExtensionPoint(
@@ -179,6 +189,9 @@ export default createBackendPlugin({
         // Create incident client for notification suppression checks
         const incidentClient = rpcClient.forPlugin(IncidentApi);
 
+        // Notification client for spec-bound dispatch
+        const notificationClient = rpcClient.forPlugin(NotificationApi);
+
         // Create gitops client for provenance lock checks
         const gitOpsClient = rpcClient.forPlugin(GitOpsApi);
 
@@ -191,6 +204,7 @@ export default createBackendPlugin({
 
         // Setup queue-based health check worker
         await setupHealthCheckWorker({
+          notificationClient,
           db: database,
           registry: healthCheckRegistry,
           collectorRegistry,
@@ -255,6 +269,7 @@ export default createBackendPlugin({
         logger,
         onHook,
         emitHook,
+        rpcClient,
         healthCheckRegistry,
         collectorRegistry,
       }) => {
@@ -267,6 +282,19 @@ export default createBackendPlugin({
           logger,
         });
 
+        // Notification subscription specs. Per-resource group lifecycle
+        // is owned by notification-backend now — healthcheck just
+        // declares the specs it dispatches under.
+        const afterNotificationClient = rpcClient.forPlugin(NotificationApi);
+        await Promise.all([
+          afterNotificationClient.registerSubscriptionSpec(
+            specToRegistration(healthcheckSystemSubscription),
+          ),
+          afterNotificationClient.registerSubscriptionSpec(
+            specToRegistration(healthcheckGroupSubscription),
+          ),
+        ]);
+
         // Register GitOps documentation now that registries are populated
         registerHealthcheckGitOpsDocumentation({
           kindRegistry,

package/src/queue-executor.test.ts

@@ -189,6 +189,9 @@ describe("Queue-Based Health Check Executor", () => {
         catalogClient: mockCatalogClient as unknown as Parameters<
           typeof setupHealthCheckWorker
         >[0]["catalogClient"],
+        notificationClient: { notifyForSubscription: () => Promise.resolve({ notifiedCount: 0 }) } as unknown as Parameters<
+          typeof setupHealthCheckWorker
+        >[0]["notificationClient"],
         maintenanceClient: mockMaintenanceClient as unknown as Parameters<
           typeof setupHealthCheckWorker
         >[0]["maintenanceClient"],
@@ -383,6 +386,9 @@ describe("Queue-Based Health Check Executor", () => {
         catalogClient: mockCatalogClient as unknown as Parameters<
           typeof setupHealthCheckWorker
         >[0]["catalogClient"],
+        notificationClient: { notifyForSubscription: () => Promise.resolve({ notifiedCount: 0 }) } as unknown as Parameters<
+          typeof setupHealthCheckWorker
+        >[0]["notificationClient"],
         maintenanceClient: mockMaintenanceClient as unknown as Parameters<
           typeof setupHealthCheckWorker
         >[0]["maintenanceClient"],

package/src/queue-executor.ts

@@ -24,9 +24,16 @@ import {
   type HealthCheckStatus,
   stripEphemeralFields,
 } from "@checkstack/healthcheck-common";
-import { CatalogApi, catalogRoutes } from "@checkstack/catalog-common";
+import {
+  CatalogApi,
+  catalogRoutes,
+  createSystemSubject,
+} from "@checkstack/catalog-common";
+import { systemHealthCollapseKey } from "@checkstack/healthcheck-common";
 import { MaintenanceApi } from "@checkstack/maintenance-common";
 import { IncidentApi } from "@checkstack/incident-common";
+import { NotificationApi } from "@checkstack/notification-common";
+import { healthcheckSystemSubscription } from "@checkstack/healthcheck-common";
 import { resolveRoute, type InferClient, extractErrorMessage} from "@checkstack/common";
 import { HealthCheckService } from "./service";
 import { healthCheckHooks } from "./hooks";
@@ -37,6 +44,7 @@ type Db = SafeDatabase<typeof schema>;
 type CatalogClient = InferClient<typeof CatalogApi>;
 type MaintenanceClient = InferClient<typeof MaintenanceApi>;
 type IncidentClient = InferClient<typeof IncidentApi>;
+type NotificationClient = InferClient<typeof NotificationApi>;
 
 /**
  * Emit the checkCompleted hook if available.
@@ -138,6 +146,7 @@ async function notifyStateChange(props: {
   previousStatus: HealthCheckStatus;
   newStatus: HealthCheckStatus;
   catalogClient: CatalogClient;
+  notificationClient: NotificationClient;
   maintenanceClient: MaintenanceClient;
   incidentClient: IncidentClient;
   logger: Logger;
@@ -148,6 +157,7 @@ async function notifyStateChange(props: {
     previousStatus,
     newStatus,
     catalogClient,
+    notificationClient,
     maintenanceClient,
     incidentClient,
     logger,
@@ -225,14 +235,25 @@ async function notifyStateChange(props: {
     systemId,
   });
 
+  void catalogClient; // parents are resolved server-side via stored target edges
+
   try {
-    await catalogClient.notifySystemSubscribers({
-      systemId,
+    await notificationClient.notifyForSubscription({
+      specId: healthcheckSystemSubscription.specId,
+      resourceKeys: [systemId],
       title,
       body,
       importance,
       action: { label: "View System", url: systemDetailPath },
-      includeGroupSubscribers: true,
+      collapseKey: systemHealthCollapseKey(systemId),
+      subjects: [
+        createSystemSubject({
+          id: systemId,
+          name: systemName,
+          url: systemDetailPath,
+          status: newStatus,
+        }),
+      ],
     });
     logger.debug(
       `Notified subscribers: ${previousStatus} → ${newStatus} for system ${systemId}`,
@@ -257,6 +278,7 @@ async function executeHealthCheckJob(props: {
   logger: Logger;
   signalService: SignalService;
   catalogClient: CatalogClient;
+  notificationClient: NotificationClient;
   maintenanceClient: MaintenanceClient;
   incidentClient: IncidentClient;
   getEmitHook: () => EmitHookFn | undefined;
@@ -270,6 +292,7 @@ async function executeHealthCheckJob(props: {
     logger,
     signalService,
     catalogClient,
+    notificationClient,
     maintenanceClient,
     incidentClient,
     getEmitHook,
@@ -575,6 +598,7 @@ async function executeHealthCheckJob(props: {
       const newState = await service.getSystemHealthStatus(systemId);
       if (newState.status !== previousStatus) {
         await notifyStateChange({
+        notificationClient,
           systemId,
           systemName,
           previousStatus,
@@ -669,6 +693,7 @@ async function executeHealthCheckJob(props: {
     const newState = await service.getSystemHealthStatus(systemId);
     if (newState.status !== previousStatus) {
       await notifyStateChange({
+        notificationClient,
         systemId,
         systemName,
         previousStatus,
@@ -800,6 +825,7 @@ async function executeHealthCheckJob(props: {
     const newState = await service.getSystemHealthStatus(systemId);
     if (newState.status !== previousStatus) {
       await notifyStateChange({
+        notificationClient,
         systemId,
         systemName,
         previousStatus,
@@ -868,6 +894,7 @@ export async function setupHealthCheckWorker(props: {
   queueManager: QueueManager;
   signalService: SignalService;
   catalogClient: CatalogClient;
+  notificationClient: NotificationClient;
   maintenanceClient: MaintenanceClient;
   incidentClient: IncidentClient;
   getEmitHook: () => EmitHookFn | undefined;
@@ -881,6 +908,7 @@ export async function setupHealthCheckWorker(props: {
     queueManager,
     signalService,
     catalogClient,
+    notificationClient,
     maintenanceClient,
     incidentClient,
     getEmitHook,
@@ -901,6 +929,7 @@ export async function setupHealthCheckWorker(props: {
         logger,
         signalService,
         catalogClient,
+        notificationClient,
         maintenanceClient,
         incidentClient,
         getEmitHook,

package/src/retention-job.ts

@@ -6,7 +6,7 @@ import {
   healthCheckAggregates,
   DEFAULT_RETENTION_CONFIG,
 } from "./schema";
-import { eq, and, lt } from "drizzle-orm";
+import { eq, and, lt, sql } from "drizzle-orm";
 import type { QueueManager } from "@checkstack/queue-api";
 
 type Db = SafeDatabase<typeof schema>;
@@ -228,23 +228,59 @@ async function rollupHourlyAggregates(params: RollupParams) {
     const p95LatencyMs =
       p95Values.length > 0 ? Math.max(...p95Values) : undefined;
 
-    // Insert daily aggregate
-    await db.insert(healthCheckAggregates).values({
-      configurationId,
-      systemId,
-      bucketStart: bucket.bucketStart,
-      bucketSize: "daily",
-      runCount,
-      healthyCount,
-      degradedCount,
-      unhealthyCount,
-      latencySumMs: latencySumMs > 0 ? latencySumMs : undefined,
-      avgLatencyMs,
-      minLatencyMs,
-      maxLatencyMs,
-      p95LatencyMs,
-      aggregatedResult: undefined, // Cannot combine result across hours
-    });
+    // Upsert the daily aggregate. A row may already exist for this
+    // (configurationId, systemId, day, daily, sourceId=null) tuple if a
+    // prior rollup ran and then late-arriving hourly buckets (e.g. from
+    // a satellite that was offline) were rolled up afterwards. Merge in
+    // that case rather than crashing — sums add, min/max/p95 fold.
+    const newLatencySum = latencySumMs > 0 ? latencySumMs : undefined;
+    await db
+      .insert(healthCheckAggregates)
+      .values({
+        configurationId,
+        systemId,
+        bucketStart: bucket.bucketStart,
+        bucketSize: "daily",
+        runCount,
+        healthyCount,
+        degradedCount,
+        unhealthyCount,
+        latencySumMs: newLatencySum,
+        avgLatencyMs,
+        minLatencyMs,
+        maxLatencyMs,
+        p95LatencyMs,
+        aggregatedResult: undefined, // Cannot combine result across hours
+      })
+      .onConflictDoUpdate({
+        target: [
+          healthCheckAggregates.configurationId,
+          healthCheckAggregates.systemId,
+          healthCheckAggregates.bucketStart,
+          healthCheckAggregates.bucketSize,
+          healthCheckAggregates.sourceId,
+        ],
+        set: {
+          runCount: sql`${healthCheckAggregates.runCount} + ${runCount}`,
+          healthyCount: sql`${healthCheckAggregates.healthyCount} + ${healthyCount}`,
+          degradedCount: sql`${healthCheckAggregates.degradedCount} + ${degradedCount}`,
+          unhealthyCount: sql`${healthCheckAggregates.unhealthyCount} + ${unhealthyCount}`,
+          latencySumMs: sql`COALESCE(${healthCheckAggregates.latencySumMs}, 0) + ${newLatencySum ?? 0}`,
+          avgLatencyMs: sql`CASE WHEN (${healthCheckAggregates.runCount} + ${runCount}) > 0 THEN (COALESCE(${healthCheckAggregates.latencySumMs}, 0) + ${newLatencySum ?? 0}) / (${healthCheckAggregates.runCount} + ${runCount}) ELSE ${healthCheckAggregates.avgLatencyMs} END`,
+          minLatencyMs:
+            minLatencyMs === undefined
+              ? sql`${healthCheckAggregates.minLatencyMs}`
+              : sql`LEAST(COALESCE(${healthCheckAggregates.minLatencyMs}, ${minLatencyMs}), ${minLatencyMs})`,
+          maxLatencyMs:
+            maxLatencyMs === undefined
+              ? sql`${healthCheckAggregates.maxLatencyMs}`
+              : sql`GREATEST(COALESCE(${healthCheckAggregates.maxLatencyMs}, ${maxLatencyMs}), ${maxLatencyMs})`,
+          p95LatencyMs:
+            p95LatencyMs === undefined
+              ? sql`${healthCheckAggregates.p95LatencyMs}`
+              : sql`GREATEST(COALESCE(${healthCheckAggregates.p95LatencyMs}, ${p95LatencyMs}), ${p95LatencyMs})`,
+        },
+      });
 
     // Delete processed hourly aggregates
     for (const hourly of bucket.aggregates) {