@checkstack/incident-backend 1.1.4 → 1.2.0

package/CHANGELOG.md

@@ -1,5 +1,137 @@
 # @checkstack/incident-backend
 
+## 1.2.0
+
+### Minor Changes
+
+- ba07ae2: Quiet down notification spam on flapping systems, auto-open incidents when a check goes critical, and let operators land directly on the broken checks.
+
+  Notification policy lives **per healthcheck assignment** (one row per `system × configuration`). Different checks on the same system are fully independent — disabling a setting on one check does not affect the others. Defaults preserve existing behaviour for `suppressDeEscalations`; **auto-incident defaults to on** for new and existing assignments.
+
+  - **`suppressDeEscalations`** (off by default). When on, transitions from a worse state to a better-but-still-failing state (e.g. `unhealthy → degraded`) no longer fire a notification. Escalations and full recoveries to `healthy` are unaffected. Resolved per assignment (the just-ran check is the one driving any aggregate transition).
+  - **`autoOpenIncidentOnUnhealthy`** (on by default). Either of two independent triggers can open the auto-incident:
+    - **`sustainedUnhealthyTrigger`** (default 30 min) — opens when the check stays continuously unhealthy for the configured duration. Catches real outages.
+    - **`flappingTrigger`** (default 3 transitions in 60 min) — opens when the check flips to unhealthy that many times in the window. Catches persistent flapping where each unhealthy phase is too brief for the sustained trigger.
+      Each trigger can be individually disabled. One incident per system: triggering checks attach to an existing active auto-incident.
+  - **`useNotificationSuppression`** (on by default, only meaningful when auto-open is on). Controls whether the auto-opened incident is created with `suppressNotifications: true` — leaving this off opens the incident but still pings operators on each transition.
+  - **`skipDuringMaintenance`** (on by default). No auto-incident is opened while the system has an active maintenance window with suppression. The system is intentionally down and shouldn't trip the on-call.
+  - **`autoCloseAfterMinutes`** (default 30). Auto-close cooldown is now per-assignment and snapshotted per-incident at open time — later policy edits don't alter in-flight incidents. Setting `null` ("Never auto-close") leaves the incident for manual resolution.
+  - **Require-recovery rule.** After any auto-incident closes (manual or auto), no new auto-incident can open until the check has logged at least one healthy run. Prevents a "operator dismissed but it's still broken" loop.
+  - **Auto-close worker** ticks every 60s and resolves auto-opened incidents whose systems have been healthy for their per-row `cooldownMinutes`. Rows with `null` cooldown are skipped entirely. Per-incident: failed close attempts are logged but never abort the sweep.
+  - **`incidentResolved` hook subscriber** syncs the auto-incident mapping when an operator manually resolves the incident, so the require-recovery rule sees the close immediately.
+  - **Platform-wide defaults.** New admin RPCs `getPlatformNotificationDefaults` / `setPlatformNotificationDefaults` (under the existing `healthcheck.configuration.{read,manage}` access rules) let operators set notification policy once for the whole instance. Per-assignment rows with `notificationPolicy: null` inherit the platform defaults at read time. UI: a "Notification defaults" button in the Assignment IDE opens a modal editor. The per-assignment Notifications panel shows an inheritance banner — "Using platform defaults" (read-only) with an "Override" button, or "Custom override" with a "Use platform defaults" button to revert. The all-or-nothing model keeps the mental model simple: each assignment is either fully inherited or fully overridden.
+  - **New service-level RPCs** on the incident plugin (`createAutoIncident`, `resolveAutoIncident`) let other plugins open/close incidents without a user context. Reused by the healthcheck auto-incident flow.
+  - **Health-state notification CTA** now deep-links to `?filter=failing` on the system detail page for non-recovery transitions (label changes to "View failing checks"). The system overview gains an `All / Failing / Healthy` segmented filter wired to the same `?filter=…` param.
+  - **Notification bell badge** now counts collapse groups instead of raw rows, so the number matches what the user sees in the notifications list. Built on `COUNT(DISTINCT COALESCE(collapse_key, id))` — notifications without a collapse key still each count as one.
+  - **`statusFilter` on `getHistory` / `getDetailedHistory`** lets the run-history page and the drawer's Recent Runs panel filter to `All / Healthy / Failing` via shared pills, with the page resetting to the first page on filter change.
+  - **Pagination defaults aligned with selector options.** Several pages defaulted to a page size (5 or 20) that wasn't in the dropdown's options (`[10, 25, 50, 100]`), so the page-size `<Select>` rendered empty. The drawer's Recent Runs now defaults to 10; the Run History, History List, and Delivery Logs pages now default to 25.
+
+  Includes Drizzle migrations adding the `notification_policy` jsonb column to `system_health_checks`, plus two new tables: `health_check_unhealthy_transitions` (for threshold counting) and `health_check_auto_incidents` (for mapping back to incident ids during auto-close).
+
+### Patch Changes
+
+- Updated dependencies [ba07ae2]
+  - @checkstack/incident-common@1.3.0
+  - @checkstack/backend-api@0.17.1
+  - @checkstack/cache-api@0.3.5
+  - @checkstack/catalog-backend@1.1.6
+  - @checkstack/command-backend@0.1.30
+  - @checkstack/integration-backend@0.1.30
+  - @checkstack/cache-utils@0.2.10
+
+## 1.1.5
+
+### Patch Changes
+
+- f23f3c9: Phase 12 of the v1 polishing plan: three coordinated cleanup items that
+  close out half-finished features ahead of v1.0.
+
+  `@checkstack/incident-backend` adds focused unit-test coverage for
+  `IncidentService.hasActiveIncidentWithSuppression` in
+  `core/incident-backend/src/service.test.ts`. The new tests exercise the
+  real query-builder logic against a programmable mock data source and
+  pin down the active-only silencing contract: returns `true` only when
+  an unresolved incident with `suppressNotifications=true` is associated
+  with the queried `systemId`; returns `false` for resolved incidents,
+  incidents with `suppressNotifications=false`, systems with no incident
+  associations, and other systems' silenced incidents. No runtime
+  changes; the service code was already correct end-to-end (write path
+  through `IncidentEditor`, read path through the healthcheck queue
+  executor and dependency notifications). A companion docs page,
+  `docs/src/content/docs/architecture/alert-silencing.md`, documents the
+  contract, the two read sites, and the dispatch paths silencing does
+  NOT cover so users aren't surprised when an unaware channel keeps
+  firing.
+
+  `@checkstack/auth-frontend` surfaces inline role assignment inside the
+  user-creation dialog so admins can pick role(s) atomically with the
+  create call. `CreateUserDialog` now renders a checkbox list of
+  assignable roles (those with `isAssignable !== false`); on submit,
+  `UsersTab` awaits `createCredentialUser`, then immediately calls
+  `updateUserRoles` with the selected role IDs. On partial failure
+  (user created, role assignment failed) the UI surfaces a warning toast
+  naming the recovery path rather than silently misreporting success. No
+  new endpoints — reuses the existing `createCredentialUser` +
+  `updateUserRoles` contract pair. A companion docs page,
+  `docs/src/content/docs/architecture/users-and-teams.md`, documents the
+  identity / role / team model, the two S2S endpoints
+  (`checkResourceTeamAccess`, `getAccessibleResourceIds`) other plugins
+  should call to honour team grants, and explicitly defers audit
+  logging, CSV export, team-scoped resource-management UI, and deletion
+  side-effect handling to v1.1.
+
+  The third item — deleting the empty `core/status-frontend/` and
+  `core/status-page-backend/` shells — is tooling-only and intentionally
+  ships without a changeset; neither shell had a `package.json`, source
+  file, or downstream importer.
+
+- f23f3c9: Add `correlationMiddleware` to `@checkstack/backend-api` and apply it
+  to every plugin/core router so each request carries a stable
+  `x-correlation-id` (read from the inbound header, or freshly minted
+  via `crypto.randomUUID()` when absent) and an auto-injected child
+  logger bound with `{ correlationId, pluginId, userId? }`. The ID is
+  echoed back on the response header so the caller can correlate their
+  client-side trace to the server logs.
+
+  The `Logger` interface in `@checkstack/backend-api` now formally
+  documents the structured-metadata convention (`logger.info("msg",
+{ ...meta })`) alongside the long-standing varargs shape. Winston's
+  splat handling already routes both shapes through the same vararg
+  slot, so existing call sites are unaffected. A new optional
+  `Logger.child(meta)` method captures the metadata-binding contract the
+  new middleware relies on; production loggers always implement it,
+  minimal test mocks may omit it (the middleware falls back gracefully).
+
+  `RpcContext` grew two optional `Headers` bags, `requestHeaders` and
+  `responseHeaders`, populated by the outer Hono `/api/*` and `/rest/*`
+  handlers in `@checkstack/backend`. They are write-through observation
+  points for middleware; an `RpcContext` constructed without them (S2S
+  clients, tests) keeps working — the echo is a silent no-op and the ID
+  is still bound onto the child logger for server-side correlation.
+
+  The scaffolding template in `@checkstack/scripts` was updated so any
+  new plugin generated via `bun run create` wires the middleware in the
+  expected `.use(correlationMiddleware).use(autoAuthMiddleware)` order
+  out of the box.
+
+- Updated dependencies [f23f3c9]
+- Updated dependencies [f23f3c9]
+- Updated dependencies [f23f3c9]
+- Updated dependencies [f23f3c9]
+  - @checkstack/common@0.11.0
+  - @checkstack/backend-api@0.17.0
+  - @checkstack/catalog-backend@1.1.5
+  - @checkstack/command-backend@0.1.29
+  - @checkstack/integration-backend@0.1.29
+  - @checkstack/notification-common@1.2.0
+  - @checkstack/integration-common@0.5.0
+  - @checkstack/auth-common@0.7.1
+  - @checkstack/catalog-common@2.2.2
+  - @checkstack/incident-common@1.2.2
+  - @checkstack/signal-common@0.2.4
+  - @checkstack/cache-api@0.3.4
+  - @checkstack/cache-utils@0.2.9
+
 ## 1.1.4
 
 ### Patch Changes

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@checkstack/incident-backend",
-  "version": "1.1.4",
+  "version": "1.2.0",
   "license": "Elastic-2.0",
   "type": "module",
   "main": "src/index.ts",
@@ -14,27 +14,27 @@
     "lint:code": "eslint . --max-warnings 0"
   },
   "dependencies": {
-    "@checkstack/backend-api": "0.15.3",
-    "@checkstack/cache-api": "0.3.2",
-    "@checkstack/cache-utils": "0.2.7",
-    "@checkstack/incident-common": "1.2.0",
-    "@checkstack/catalog-common": "2.2.0",
-    "@checkstack/catalog-backend": "1.1.3",
-    "@checkstack/notification-common": "1.1.0",
-    "@checkstack/auth-common": "0.7.0",
-    "@checkstack/command-backend": "0.1.27",
-    "@checkstack/signal-common": "0.2.3",
-    "@checkstack/integration-backend": "0.1.27",
-    "@checkstack/integration-common": "0.4.0",
-    "@checkstack/common": "0.10.0",
+    "@checkstack/backend-api": "0.17.0",
+    "@checkstack/cache-api": "0.3.4",
+    "@checkstack/cache-utils": "0.2.9",
+    "@checkstack/incident-common": "1.2.2",
+    "@checkstack/catalog-common": "2.2.2",
+    "@checkstack/catalog-backend": "1.1.5",
+    "@checkstack/notification-common": "1.2.0",
+    "@checkstack/auth-common": "0.7.1",
+    "@checkstack/command-backend": "0.1.29",
+    "@checkstack/signal-common": "0.2.4",
+    "@checkstack/integration-backend": "0.1.29",
+    "@checkstack/integration-common": "0.5.0",
+    "@checkstack/common": "0.11.0",
     "drizzle-orm": "^0.45.0",
     "zod": "^4.2.1",
     "@orpc/server": "^1.13.2"
   },
   "devDependencies": {
     "@checkstack/drizzle-helper": "0.0.5",
-    "@checkstack/scripts": "0.3.2",
-    "@checkstack/test-utils-backend": "0.1.27",
+    "@checkstack/scripts": "0.3.3",
+    "@checkstack/test-utils-backend": "0.1.29",
     "@checkstack/tsconfig": "0.0.7",
     "@types/bun": "^1.0.0",
     "drizzle-kit": "^0.31.10",

package/src/router.ts

@@ -5,6 +5,7 @@ import {
 } from "@checkstack/incident-common";
 import {
   autoAuthMiddleware,
+  correlationMiddleware,
   Logger,
   type RpcContext,
 } from "@checkstack/backend-api";
@@ -89,6 +90,7 @@ export function createRouter(
 
   const os = implement(incidentContract)
     .$context<RpcContext>()
+    .use(correlationMiddleware)
     .use(autoAuthMiddleware);
 
   return os.router({
@@ -393,6 +395,96 @@ export function createRouter(
         return { suppressed };
       }),
 
+    createAutoIncident: os.createAutoIncident.handler(
+      async ({ input, context }) => {
+        // No user context for service-initiated incidents; createdBy
+        // stays null and the timeline shows the originating plugin via
+        // the hook payload.
+        const result = await service.createIncident(input);
+
+        await cache.invalidateForMutation({
+          incidentId: result.id,
+          systemIds: result.systemIds,
+        });
+
+        await signalService.broadcast(INCIDENT_UPDATED, {
+          incidentId: result.id,
+          systemIds: result.systemIds,
+          action: "created",
+        });
+
+        await context.emitHook(incidentHooks.incidentCreated, {
+          incidentId: result.id,
+          systemIds: result.systemIds,
+          title: result.title,
+          description: result.description,
+          severity: result.severity,
+          status: result.status,
+          createdAt: result.createdAt.toISOString(),
+        });
+
+        const systemNames = await resolveSystemNames(result.systemIds);
+        await notifyAffectedSystems({
+          catalogClient,
+          notificationClient,
+          logger,
+          incidentId: result.id,
+          incidentTitle: result.title,
+          systemIds: result.systemIds,
+          systemNames,
+          action: "created",
+          severity: result.severity,
+        });
+
+        return { id: result.id };
+      },
+    ),
+
+    resolveAutoIncident: os.resolveAutoIncident.handler(
+      async ({ input, context }) => {
+        const result = await service.resolveIncident(input.id, input.message);
+        // Idempotent: a missing or already-resolved incident is treated
+        // as success so the auto-close worker can be re-run safely.
+        if (!result) {
+          return { success: true };
+        }
+
+        await cache.invalidateForMutation({
+          incidentId: result.id,
+          systemIds: result.systemIds,
+        });
+
+        await signalService.broadcast(INCIDENT_UPDATED, {
+          incidentId: result.id,
+          systemIds: result.systemIds,
+          action: "resolved",
+        });
+
+        await context.emitHook(incidentHooks.incidentResolved, {
+          incidentId: result.id,
+          systemIds: result.systemIds,
+          title: result.title,
+          severity: result.severity,
+          resolvedAt: new Date().toISOString(),
+        });
+
+        const systemNames = await resolveSystemNames(result.systemIds);
+        await notifyAffectedSystems({
+          catalogClient,
+          notificationClient,
+          logger,
+          incidentId: result.id,
+          incidentTitle: result.title,
+          systemIds: result.systemIds,
+          systemNames,
+          action: "resolved",
+          severity: result.severity,
+        });
+
+        return { success: true };
+      },
+    ),
+
     addLink: os.addLink.handler(async ({ input }) => {
       // Verify incident exists so the FK violation surfaces as NOT_FOUND.
       const incident = await service.getIncident(input.incidentId);

package/src/service.test.ts

@@ -0,0 +1,126 @@
+import { describe, it, expect, mock, beforeEach } from "bun:test";
+import { IncidentService } from "./service";
+
+/**
+ * Programmable mock DB that records each `select(...).from(...).where(...)`
+ * (and optional `.limit(...)`) chain and returns a configurable row array
+ * per invocation. Tests exercise the real query-builder calls inside
+ * `IncidentService`, only swapping out the terminal data source.
+ */
+function createProgrammableSelectDb(resultsByCall: unknown[][]) {
+  let callIndex = 0;
+
+  const nextResult = (): unknown[] => {
+    const result = resultsByCall[callIndex] ?? [];
+    callIndex += 1;
+    return result;
+  };
+
+  const select = mock((projection?: Record<string, unknown>) => {
+    void projection;
+    const rows = nextResult();
+
+    const limit = mock(() => Promise.resolve(rows));
+    const whereResult = Object.assign(Promise.resolve(rows), { limit });
+    const where = mock(() => whereResult);
+    const fromResult = Object.assign(Promise.resolve(rows), { where });
+    const from = mock(() => fromResult);
+
+    return { from };
+  });
+
+  return {
+    db: { select } as unknown,
+    select,
+    getCallCount: () => callIndex,
+  };
+}
+
+describe("IncidentService.hasActiveIncidentWithSuppression", () => {
+  let dbHelper: ReturnType<typeof createProgrammableSelectDb>;
+  let service: IncidentService;
+
+  const setup = (resultsByCall: unknown[][]) => {
+    dbHelper = createProgrammableSelectDb(resultsByCall);
+    service = new IncidentService(dbHelper.db as never);
+  };
+
+  beforeEach(() => {
+    dbHelper = createProgrammableSelectDb([]);
+  });
+
+  it("returns true when an active incident with suppressNotifications=true exists for the system", async () => {
+    setup([
+      // 1st query: incidentSystems lookup for systemId="sys-1"
+      [{ incidentId: "inc-1" }],
+      // 2nd query: incidents lookup with .where(active AND suppression).limit(1)
+      [{ id: "inc-1" }],
+    ]);
+
+    const result = await service.hasActiveIncidentWithSuppression("sys-1");
+
+    expect(result).toBe(true);
+    expect(dbHelper.getCallCount()).toBe(2);
+  });
+
+  it("returns false when no incidents are associated with the system", async () => {
+    setup([
+      // 1st query: empty -> short-circuits before the 2nd query
+      [],
+    ]);
+
+    const result = await service.hasActiveIncidentWithSuppression("sys-1");
+
+    expect(result).toBe(false);
+    // Only one query should have run; the incidents lookup is skipped.
+    expect(dbHelper.getCallCount()).toBe(1);
+  });
+
+  it("returns false when the matching incident is resolved (silencing is scoped to active incidents)", async () => {
+    setup([
+      // 1st query: the system has an incident association.
+      [{ incidentId: "inc-resolved" }],
+      // 2nd query: the WHERE clause filters out resolved incidents, so the
+      // limit(1) projection finds nothing. The real query builder enforces
+      // this via `ne(incidents.status, "resolved")`.
+      [],
+    ]);
+
+    const result = await service.hasActiveIncidentWithSuppression("sys-1");
+
+    expect(result).toBe(false);
+    expect(dbHelper.getCallCount()).toBe(2);
+  });
+
+  it("returns false when the matching incident has suppressNotifications=false", async () => {
+    setup([
+      // 1st query: the system has an incident association.
+      [{ incidentId: "inc-no-suppress" }],
+      // 2nd query: the WHERE clause filters by suppressNotifications=true,
+      // so a row with suppressNotifications=false is excluded — the result
+      // set is empty.
+      [],
+    ]);
+
+    const result = await service.hasActiveIncidentWithSuppression("sys-1");
+
+    expect(result).toBe(false);
+    expect(dbHelper.getCallCount()).toBe(2);
+  });
+
+  it("filters by systemId — does not return true for another system's silenced incident", async () => {
+    // The systemId filter is enforced by the WHERE clause on the
+    // incidentSystems lookup. Querying "sys-other" returns an empty
+    // association set even though "sys-1" has a silenced incident, so the
+    // method short-circuits to false.
+    setup([
+      // 1st query for systemId="sys-other": no associations.
+      [],
+    ]);
+
+    const result = await service.hasActiveIncidentWithSuppression("sys-other");
+
+    expect(result).toBe(false);
+    expect(dbHelper.getCallCount()).toBe(1);
+  });
+});