@checkstack/healthcheck-backend 1.16.0 → 1.17.0

package/CHANGELOG.md

@@ -1,5 +1,139 @@
 # @checkstack/healthcheck-backend
 
+## 1.17.0
+
+### Minor Changes
+
+- 390d9cf: Add a **Container** health-check strategy for monitoring Docker and Podman
+  containers that expose no external service of their own. It reports container
+  existence, running state, healthcheck status, exit code, restart count, and
+  OOM-killed via the **Container Status** collector, and CPU/memory usage via the
+  **Container Stats** collector. Both collectors issue only read (GET) requests
+  against the runtime REST API.
+
+  The check runs wherever the executor runs: locally on the core instance (the
+  default) to watch containers that share a host with Checkstack, or on a
+  satellite pinned to another host.
+
+  Critically, Checkstack never touches the raw container socket. The strategy
+  talks the Docker Engine / Podman libpod API over either a unix socket path or an
+  `http(s)` endpoint, so operators point it at a **read-only socket-proxy**
+  (`lscr.io/linuxserver/socket-proxy` with `POST=0`) running next to whichever
+  Checkstack instance runs the check - core or a satellite - or at a rootless
+  Podman socket. The raw socket is mounted only into the proxy; even a compromised
+  instance can only read container state, never control the host. A stopped or missing container is a successful collection whose metrics
+  feed assertions (following the transport-failure-vs-metric rule) - only an
+  unreachable runtime endpoint fails the check. Container `exec` probes are
+  intentionally not offered because they would require write access to the socket.
+
+  To support in-product setup guidance, the health-check strategy contract gains
+  an optional `setupInstructions` (Markdown) field, surfaced in the DTO and
+  rendered as a collapsible "Setup guide" callout above the strategy config fields
+  in the editor. The Container strategy populates it with the secure proxy setup.
+
+  The hardened socket-proxy compose is maintained as a single canonical file
+  (`deploy/socket-proxy/docker-compose.yml`) that operators `include:` from their
+  core or satellite compose, so the read-only / `POST=0` / internal-network
+  hardening is defined in exactly one place; the docs and the in-product setup
+  guide reference it rather than duplicating the YAML.
+
+  Also removes a stale hand-written `HealthCheckStrategyDto` interface in
+  `@checkstack/healthcheck-common` that shadowed (and lagged behind) the
+  Zod-inferred DTO; the inferred type from `schemas.ts` is now the single source
+  of truth and correctly carries `resultSchema`, `aggregatedResultSchema`, and the
+  new `setupInstructions`.
+
+  Thanks to [@stuajnht](https://github.com/stuajnht) for the valuable feedback
+  that shaped this release.
+
+- fc64fad: Dependencies can now be scoped to a specific environment and/or health check of
+  the upstream system, each with its own severity - a "matrix" of scope cells.
+
+  Previously a dependency watched the upstream's overall health (any check, any
+  environment) at the edge's impact type, with optional per-check rules. That
+  default is unchanged: with no scope cells configured, the dependency behaves
+  exactly as before. Now each cell pins a check (a specific configuration, or
+  "any"), an environment (a specific environment, or "any"), and a severity
+  (informational / degraded / critical). When a dependency has any cells, only
+  those slices are watched (they replace the whole-system watch) and the worst
+  result across cells wins. This lets you express, e.g., "System A depends on
+  System B only in `prod`", or "only when B's TLS check in `prod` fails", and lets
+  different cells carry different severities.
+
+  Because each environment is evaluated on its own slice, a scoped dependency
+  catches an environment-specific outage that the upstream's overall status
+  (worst-wins across environments) would otherwise hide. The dependency evaluator
+  now reads per-(check, environment) health via a new
+  `@checkstack/healthcheck-common` bulk contract `getBulkSystemHealthMatrix` (and
+  its `@checkstack/healthcheck-backend` implementation), which returns each
+  system's cross-environment rollup plus a per-environment slice. Incident
+  overrides still fold into the overall rollup, so incident-forced statuses keep
+  propagating through dependencies.
+
+  The scope-cell store gains a nullable `environment_id` column and makes
+  `health_check_id` nullable (forward-only migration; existing rows keep working
+  as "any check, any environment"). The dependency editor's per-check panel
+  becomes a scope-matrix editor with check + environment + severity rows.
+
+  Transitive (multi-hop) dependencies still cascade using the upstream's overall
+  status; per-environment cascades across multiple hops are not yet propagated.
+
+- 9d30324: Incidents can now optionally override the health status of their affected
+  systems. When creating or editing an incident you can pick "Override system
+  health" (Degraded or Unhealthy); while the incident is active (not resolved)
+  that status is folded into every affected system's derived health via
+  worst-wins, so it shows on every health surface (status pages, dashboards,
+  dependency map, catalog badges). A health check reporting a worse status still
+  wins, and the override lifts automatically when the incident resolves. This
+  covers components that no automated check can monitor (e.g. a running app whose
+  licenses were revoked so it won't open).
+
+  The override is a deliberate operator choice, independent of the incident's
+  severity. A new service-typed incident RPC `getActiveHealthOverrides` exposes
+  active overrides per system, which `@checkstack/healthcheck-backend` reads and
+  folds into `getSystemHealthStatus`. The system-health response gains an optional
+  `override` field naming the contributing incident so UIs can explain why a
+  system reads unhealthy when its checks look fine. The system health badge uses
+  it to show, on hover, when a status was forced by an incident.
+
+  The dashboard "problem system" signal attributes an override-forced status to
+  the incident ("Forced by incident: <title>") instead of misreporting
+  "0 of N checks failing", while a genuinely worse health check still drives the
+  signal and its detail. Public status pages reflect the forced status but never
+  carry the incident title (the widget DTOs project only the status), so an
+  override cannot leak the name of a hidden incident.
+
+  Behavior change: a system's derived health now reflects active incident
+  overrides in addition to its health checks. Adds a forward-only migration for
+  the new nullable `incidents.health_override` column.
+
+  Thanks to [@stuajnht](https://github.com/stuajnht) for the valuable feedback
+  that shaped this release.
+
+### Patch Changes
+
+- Updated dependencies [390d9cf]
+- Updated dependencies [390d9cf]
+- Updated dependencies [fc64fad]
+- Updated dependencies [fc64fad]
+- Updated dependencies [9d30324]
+- Updated dependencies [9d30324]
+- Updated dependencies [b218e3e]
+  - @checkstack/ai-backend@0.10.8
+  - @checkstack/backend-api@0.30.0
+  - @checkstack/healthcheck-common@1.14.0
+  - @checkstack/incident-common@1.8.0
+  - @checkstack/incident-backend@1.10.0
+  - @checkstack/automation-backend@0.10.10
+  - @checkstack/catalog-backend@1.6.8
+  - @checkstack/command-backend@0.2.20
+  - @checkstack/gitops-backend@0.5.20
+  - @checkstack/satellite-backend@0.8.2
+  - @checkstack/script-packages-backend@0.3.24
+  - @checkstack/secrets-backend@0.3.2
+  - @checkstack/status-page-backend@0.4.7
+  - @checkstack/sdk@0.125.1
+
 ## 1.16.0
 
 ### Minor Changes

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@checkstack/healthcheck-backend",
-  "version": "1.16.0",
+  "version": "1.17.0",
   "license": "Elastic-2.0",
   "type": "module",
   "main": "src/index.ts",
@@ -14,31 +14,31 @@
     "lint:code": "eslint . --max-warnings 0"
   },
   "dependencies": {
-    "@checkstack/ai-backend": "0.10.7",
+    "@checkstack/ai-backend": "0.10.8",
     "@checkstack/ai-common": "0.6.5",
-    "@checkstack/automation-backend": "0.10.9",
-    "@checkstack/backend-api": "0.29.1",
+    "@checkstack/automation-backend": "0.10.10",
+    "@checkstack/backend-api": "0.30.0",
     "@checkstack/cache-api": "0.3.18",
     "@checkstack/cache-utils": "0.2.23",
-    "@checkstack/catalog-backend": "1.6.7",
+    "@checkstack/catalog-backend": "1.6.8",
     "@checkstack/catalog-common": "2.6.2",
-    "@checkstack/command-backend": "0.2.19",
+    "@checkstack/command-backend": "0.2.20",
     "@checkstack/common": "0.21.0",
-    "@checkstack/gitops-backend": "0.5.19",
+    "@checkstack/gitops-backend": "0.5.20",
     "@checkstack/gitops-common": "0.7.2",
-    "@checkstack/healthcheck-common": "1.13.0",
-    "@checkstack/incident-backend": "1.9.5",
-    "@checkstack/incident-common": "1.7.2",
+    "@checkstack/healthcheck-common": "1.14.0",
+    "@checkstack/incident-backend": "1.10.0",
+    "@checkstack/incident-common": "1.8.0",
     "@checkstack/maintenance-common": "1.8.2",
     "@checkstack/notification-common": "1.5.2",
     "@checkstack/queue-api": "0.3.18",
-    "@checkstack/satellite-backend": "0.8.1",
-    "@checkstack/script-packages-backend": "0.3.23",
-    "@checkstack/sdk": "0.123.1",
-    "@checkstack/secrets-backend": "0.3.1",
+    "@checkstack/satellite-backend": "0.8.2",
+    "@checkstack/script-packages-backend": "0.3.24",
+    "@checkstack/sdk": "0.125.1",
+    "@checkstack/secrets-backend": "0.3.2",
     "@checkstack/secrets-common": "0.3.1",
     "@checkstack/signal-common": "0.2.16",
-    "@checkstack/status-page-backend": "0.4.6",
+    "@checkstack/status-page-backend": "0.4.7",
     "@checkstack/status-page-common": "0.5.2",
     "@hono/zod-validator": "^0.7.6",
     "@orpc/contract": "^1.14.4",
@@ -53,7 +53,7 @@
   "devDependencies": {
     "@checkstack/drizzle-helper": "0.0.6",
     "@checkstack/scripts": "0.7.2",
-    "@checkstack/test-utils-backend": "0.1.53",
+    "@checkstack/test-utils-backend": "0.1.54",
     "@checkstack/tsconfig": "0.0.7",
     "@types/bun": "^1.0.0",
     "@types/tdigest": "^0.1.5",

package/src/index.ts

@@ -524,6 +524,35 @@ export default createBackendPlugin({
           logger,
           signalService,
           configSecrets,
+          // Fold active incident health overrides into the user-facing system
+          // health reads (worst-wins). Reuses the incident client already built
+          // above; maps the incident rows to the source-agnostic override shape
+          // the fold expects. Kept out of the shared deriver so SLO/AI/entity
+          // paths stay checks-only (see router.ts).
+          incidentHealthOverrideReader: {
+            getActiveOverrides: async (systemIds) => {
+              const { overrides } =
+                await incidentClient.getActiveHealthOverrides({ systemIds });
+              const mapped: Record<
+                string,
+                {
+                  status: "degraded" | "unhealthy";
+                  source: string;
+                  reason: string;
+                  sourceId?: string;
+                }[]
+              > = {};
+              for (const [systemId, list] of Object.entries(overrides)) {
+                mapped[systemId] = list.map((o) => ({
+                  status: o.status,
+                  source: "incident",
+                  reason: o.incidentTitle,
+                  sourceId: o.incidentId,
+                }));
+              }
+              return mapped;
+            },
+          },
           recomputeSystemRollupHealth: (systemId) =>
             recomputeSystemRollupHealth({
               systemId,

package/src/router.ts

@@ -39,6 +39,11 @@ import { CatalogApi } from "@checkstack/catalog-common";
 import { MaintenanceApi } from "@checkstack/maintenance-common";
 import type { Logger } from "@checkstack/backend-api";
 import type { HealthCheckCache } from "./cache";
+import {
+  applySystemHealthOverrides,
+  type SystemHealthOverrideReader,
+} from "./system-health-override";
+import type { SystemHealthStatusResponse } from "@checkstack/healthcheck-common";
 
 /**
  * Creates the healthcheck router using contract-based implementation.
@@ -81,6 +86,17 @@ export const createHealthCheckRouter = (opts: {
    * router MUST receive it or writes would store inline secrets verbatim.
    */
   configSecrets?: HealthCheckSecretsDeps;
+  /**
+   * Reads active incident health overrides and folds them into the two
+   * user-facing system-health reads (single + bulk) via worst-wins, so a system
+   * shows the status an active incident forces even when its checks look fine.
+   * Applied OUTSIDE the status cache (always live, so an override lifts the
+   * instant its incident resolves) and ONLY in these RPC handlers - never in the
+   * shared `getSystemHealthStatus` deriver, whose other callers (SLO downtime,
+   * the AI signals scan, the persisted `health` entity) must stay checks-only.
+   * Optional so tests / no-incident deployments simply skip the fold.
+   */
+  incidentHealthOverrideReader?: SystemHealthOverrideReader;
 }) => {
   const {
     database,
@@ -94,6 +110,7 @@ export const createHealthCheckRouter = (opts: {
     logger,
     signalService,
     recomputeSystemRollupHealth,
+    incidentHealthOverrideReader,
   } = opts;
   // Create service instance once - shared across all handlers
   const service = new HealthCheckService(
@@ -105,6 +122,43 @@ export const createHealthCheckRouter = (opts: {
     opts.configSecrets,
   );
 
+  /**
+   * Fold active incident health overrides into a batch of checks-only system
+   * statuses via worst-wins. Reads overrides for all systems in ONE incident RPC
+   * (or none, when no reader is wired). Resilient by design: incidents are a
+   * best-effort enrichment of health, so if the read fails the checks-only
+   * statuses are returned unchanged rather than failing the whole health read.
+   */
+  const foldIncidentOverrides = async (
+    statuses: Record<string, SystemHealthStatusResponse>,
+  ): Promise<Record<string, SystemHealthStatusResponse>> => {
+    const systemIds = Object.keys(statuses);
+    if (!incidentHealthOverrideReader || systemIds.length === 0) {
+      return statuses;
+    }
+    let overridesBySystem: Awaited<
+      ReturnType<SystemHealthOverrideReader["getActiveOverrides"]>
+    >;
+    try {
+      overridesBySystem =
+        await incidentHealthOverrideReader.getActiveOverrides(systemIds);
+    } catch (error) {
+      logger.warn(
+        "Failed to read incident health overrides; returning checks-only status",
+        { error: extractErrorMessage(error) },
+      );
+      return statuses;
+    }
+    const folded: Record<string, SystemHealthStatusResponse> = {};
+    for (const [systemId, base] of Object.entries(statuses)) {
+      folded[systemId] = applySystemHealthOverrides({
+        base,
+        overrides: overridesBySystem[systemId] ?? [],
+      });
+    }
+    return folded;
+  };
+
   // Create contract implementer with context type AND auto auth middleware
   const os = implement(healthCheckContract)
     .$context<RpcContext>()
@@ -201,6 +255,7 @@ export const createHealthCheckRouter = (opts: {
         displayName: r.strategy.displayName,
         description: r.strategy.description,
         category: (r.strategy.category ?? "other") as StrategyCategory,
+        setupInstructions: r.strategy.setupInstructions,
         configSchema: toJsonSchema(r.strategy.config.schema),
         resultSchema: r.strategy.result
           ? toJsonSchemaWithChartMeta(r.strategy.result.schema)
@@ -622,9 +677,13 @@ export const createHealthCheckRouter = (opts: {
     ),
     getSystemHealthStatus: os.getSystemHealthStatus.handler(
       async ({ input }) => {
-        return cache.wrapSystemHealthStatus(input.systemId, () =>
+        const base = await cache.wrapSystemHealthStatus(input.systemId, () =>
           service.getSystemHealthStatus(input.systemId),
         );
+        const folded = await foldIncidentOverrides({
+          [input.systemId]: base,
+        });
+        return folded[input.systemId]!;
       },
     ),
 
@@ -634,10 +693,7 @@ export const createHealthCheckRouter = (opts: {
         // and invalidated by id on mutations, so dashboards with overlapping
         // (but non-identical) system sets share cache entries. See
         // ./cache.ts for the key/TTL/invalidation contract.
-        const statuses: Record<
-          string,
-          Awaited<ReturnType<typeof service.getSystemHealthStatus>>
-        > = {};
+        const statuses: Record<string, SystemHealthStatusResponse> = {};
         await Promise.all(
           input.systemIds.map(async (systemId) => {
             statuses[systemId] = await cache.wrapSystemHealthStatus(
@@ -646,6 +702,37 @@ export const createHealthCheckRouter = (opts: {
             );
           }),
         );
+        return { statuses: await foldIncidentOverrides(statuses) };
+      },
+    ),
+
+    getBulkSystemHealthMatrix: os.getBulkSystemHealthMatrix.handler(
+      async ({ input }) => {
+        const matrix = await service.getBulkSystemHealthMatrix(input.systemIds);
+
+        // Fold active incident overrides into each system's OVERALL rollup, so
+        // an incident-forced status still propagates through dependencies (as
+        // it does via getBulkSystemHealthStatus). Per-environment slices track
+        // health-check status only - incidents force whole-system health, which
+        // any-environment (env=null) dependency cells read from this rollup.
+        const overallOnly: Record<string, SystemHealthStatusResponse> = {};
+        for (const [systemId, m] of Object.entries(matrix)) {
+          overallOnly[systemId] = {
+            status: m.status,
+            evaluatedAt: new Date(),
+            checkStatuses: m.checkStatuses,
+          };
+        }
+        const folded = await foldIncidentOverrides(overallOnly);
+
+        const statuses: Record<string, (typeof matrix)[string]> = {};
+        for (const [systemId, m] of Object.entries(matrix)) {
+          statuses[systemId] = {
+            status: folded[systemId]?.status ?? m.status,
+            checkStatuses: m.checkStatuses,
+            environments: m.environments,
+          };
+        }
         return { statuses };
       },
     ),

package/src/service.ts

@@ -41,6 +41,7 @@ import {
   gte,
   lte,
   isNull,
+  isNotNull,
   inArray,
 } from "drizzle-orm";
 import { ORPCError } from "@orpc/server";
@@ -1341,6 +1342,98 @@ export class HealthCheckService {
     return Object.fromEntries(entries);
   }
 
+  /**
+   * Bulk per-(system, check, environment) health for the given systems.
+   *
+   * For each system returns the cross-environment rollup (status +
+   * checkStatuses, same as {@link getSystemHealthStatus}) PLUS a slice per
+   * environment the system has runs for. Consumers that scope by environment
+   * (the dependency map) must read the per-environment slice, because the
+   * rollup deliberately hides a single failing environment.
+   *
+   * Cost scales with the number of environments each system actually fans out
+   * to (`1 + #envs` status evaluations per system); systems with only env-less
+   * runs cost the same as a plain rollup read. Not on any per-run hot path.
+   */
+  async getBulkSystemHealthMatrix(systemIds: string[]): Promise<
+    Record<
+      string,
+      {
+        status: HealthCheckStatus;
+        checkStatuses: SystemHealthStatusResponse["checkStatuses"];
+        environments: Record<
+          string,
+          {
+            status: HealthCheckStatus;
+            checkStatuses: SystemHealthStatusResponse["checkStatuses"];
+          }
+        >;
+      }
+    >
+  > {
+    const result: Record<
+      string,
+      {
+        status: HealthCheckStatus;
+        checkStatuses: SystemHealthStatusResponse["checkStatuses"];
+        environments: Record<
+          string,
+          {
+            status: HealthCheckStatus;
+            checkStatuses: SystemHealthStatusResponse["checkStatuses"];
+          }
+        >;
+      }
+    > = {};
+
+    await Promise.all(
+      systemIds.map(async (systemId) => {
+        const overall = await this.getSystemHealthStatus(systemId);
+
+        // Environments this system actually has runs for (env-less excluded -
+        // it is folded into the rollup and never a real environment id).
+        const envRows = await this.db
+          .selectDistinct({ environmentId: healthCheckRuns.environmentId })
+          .from(healthCheckRuns)
+          .where(
+            and(
+              eq(healthCheckRuns.systemId, systemId),
+              isNotNull(healthCheckRuns.environmentId),
+            ),
+          );
+
+        const environments: Record<
+          string,
+          {
+            status: HealthCheckStatus;
+            checkStatuses: SystemHealthStatusResponse["checkStatuses"];
+          }
+        > = {};
+        await Promise.all(
+          envRows.map(async ({ environmentId }) => {
+            if (!environmentId) return;
+            const envStatus = await this.getSystemHealthStatus(
+              systemId,
+              environmentId,
+            );
+            environments[environmentId] = {
+              status: envStatus.status,
+              checkStatuses: envStatus.checkStatuses,
+            };
+          }),
+        );
+
+        result[systemId] = {
+          status: overall.status,
+          checkStatuses: overall.checkStatuses,
+          environments,
+        };
+      }),
+    );
+
+    return result;
+  }
+
   /**
    * Get comprehensive health overview for a system.
    * Returns all health checks with their last 25 runs for sparkline visualization.

package/src/status-page/widgets.ts

@@ -91,7 +91,17 @@ async function healthStatuses(
   const { statuses } = await ctx.rpcClient
     .forPlugin(HealthCheckApi)
     .getBulkSystemHealthStatus({ systemIds: ids });
-  return statuses;
+  // Project to ONLY the derived status. `getBulkSystemHealthStatus` folds active
+  // incident overrides into `status` (so the public page shows the forced
+  // status), but the response also carries `override.reason` = the incident
+  // TITLE. Status pages are public and incidents may be hidden, so we drop
+  // everything but the status here - the incident name must never reach a public
+  // widget DTO.
+  const projected: Record<string, { status: string } | undefined> = {};
+  for (const [systemId, value] of Object.entries(statuses)) {
+    projected[systemId] = value ? { status: value.status } : undefined;
+  }
+  return projected;
 }
 
 function publicStatus({

package/src/system-health-override.test.ts

@@ -0,0 +1,94 @@
+import { describe, expect, it } from "bun:test";
+import {
+  applySystemHealthOverrides,
+  worstHealthStatus,
+  type SystemHealthOverrideInput,
+} from "./system-health-override";
+import type { SystemHealthStatusResponse } from "@checkstack/healthcheck-common";
+
+const at = new Date("2026-07-05T00:00:00.000Z");
+
+function base(
+  status: SystemHealthStatusResponse["status"],
+): SystemHealthStatusResponse {
+  return { status, evaluatedAt: at, checkStatuses: [] };
+}
+
+function override(
+  status: "degraded" | "unhealthy",
+  sourceId: string,
+): SystemHealthOverrideInput {
+  return {
+    status,
+    source: "incident",
+    reason: `Incident ${sourceId}`,
+    sourceId,
+  };
+}
+
+describe("worstHealthStatus", () => {
+  it("orders unhealthy > degraded > healthy", () => {
+    expect(worstHealthStatus("healthy", "degraded")).toBe("degraded");
+    expect(worstHealthStatus("degraded", "unhealthy")).toBe("unhealthy");
+    expect(worstHealthStatus("unhealthy", "degraded")).toBe("unhealthy");
+    expect(worstHealthStatus("healthy", "healthy")).toBe("healthy");
+  });
+});
+
+describe("applySystemHealthOverrides", () => {
+  it("returns the base unchanged (no override field) when there are no overrides", () => {
+    const result = applySystemHealthOverrides({
+      base: base("healthy"),
+      overrides: [],
+    });
+    expect(result.status).toBe("healthy");
+    expect(result.override).toBeUndefined();
+  });
+
+  it("raises a healthy system to the override status and records the source", () => {
+    const result = applySystemHealthOverrides({
+      base: base("healthy"),
+      overrides: [override("unhealthy", "inc-1")],
+    });
+    expect(result.status).toBe("unhealthy");
+    expect(result.override).toEqual({
+      status: "unhealthy",
+      source: "incident",
+      reason: "Incident inc-1",
+      sourceId: "inc-1",
+    });
+  });
+
+  it("applies an override even when the system has no health checks", () => {
+    const result = applySystemHealthOverrides({
+      base: base("healthy"), // no-checks default is healthy
+      overrides: [override("degraded", "inc-2")],
+    });
+    expect(result.status).toBe("degraded");
+    expect(result.override?.status).toBe("degraded");
+  });
+
+  it("keeps the worse HEALTH CHECK status when a check is worse than the override", () => {
+    // Override says degraded, but a check reports unhealthy -> unhealthy wins.
+    const result = applySystemHealthOverrides({
+      base: base("unhealthy"),
+      overrides: [override("degraded", "inc-3")],
+    });
+    expect(result.status).toBe("unhealthy");
+    // The override is still surfaced (it contributed), at its own status.
+    expect(result.override?.status).toBe("degraded");
+  });
+
+  it("picks the worst override when several incidents override the same system", () => {
+    const result = applySystemHealthOverrides({
+      base: base("healthy"),
+      overrides: [
+        override("degraded", "inc-a"),
+        override("unhealthy", "inc-b"),
+        override("degraded", "inc-c"),
+      ],
+    });
+    expect(result.status).toBe("unhealthy");
+    expect(result.override?.sourceId).toBe("inc-b");
+  });
+});

package/src/system-health-override.ts

@@ -0,0 +1,93 @@
+import type {
+  HealthCheckStatus,
+  SystemHealthStatusResponse,
+  SystemHealthOverride,
+} from "@checkstack/healthcheck-common";
+
+/**
+ * Worst-wins ordering for the derived health vocabulary: a higher rank is a
+ * worse status. Mirrors the inline `unhealthy > degraded > healthy` ordering the
+ * per-check rollup uses in `service.ts`, extracted so the incident-override fold
+ * shares the exact same comparison.
+ */
+const HEALTH_RANK: Record<HealthCheckStatus, number> = {
+  healthy: 0,
+  degraded: 1,
+  unhealthy: 2,
+};
+
+/** Returns whichever status is worse (ties return `a`). */
+export function worstHealthStatus(
+  a: HealthCheckStatus,
+  b: HealthCheckStatus,
+): HealthCheckStatus {
+  return HEALTH_RANK[b] > HEALTH_RANK[a] ? b : a;
+}
+
+/**
+ * A non-health-check contribution to a system's health, as consumed by the
+ * fold. Kept source-agnostic (`source`/`sourceId`) so the health plugin does not
+ * hard-code incident semantics; the incident-backed reader maps its rows into
+ * this shape.
+ */
+export interface SystemHealthOverrideInput {
+  /** The status this contributor forces. Never `healthy` in practice. */
+  status: HealthCheckStatus;
+  /** Contributor kind, e.g. "incident". */
+  source: string;
+  /** Human-readable reason, e.g. the incident title. */
+  reason: string;
+  /** Opaque id of the contributing record, e.g. the incident id. */
+  sourceId?: string;
+}
+
+/**
+ * Reads active health overrides for a set of systems. Implemented in the plugin
+ * wiring over the incident RPC; injected into the health service so the service
+ * stays free of a direct incident dependency and tests can stub it.
+ */
+export interface SystemHealthOverrideReader {
+  getActiveOverrides(
+    systemIds: string[],
+  ): Promise<Record<string, SystemHealthOverrideInput[]>>;
+}
+
+/**
+ * Fold a system's active overrides into its health-check-derived status via
+ * worst-wins. The overall status becomes the worst of the checks-derived status
+ * and every override, so an override that raises a system to `degraded` never
+ * masks a health check reporting `unhealthy` (the worse status always wins), and
+ * an override applies even when the system has no health checks at all.
+ *
+ * The worst contributing override (if any) is surfaced on `override` so a UI can
+ * explain why a system reads worse than its checks alone. When the overrides are
+ * empty the base response is returned unchanged (no `override`).
+ */
+export function applySystemHealthOverrides({
+  base,
+  overrides,
+}: {
+  base: SystemHealthStatusResponse;
+  overrides: SystemHealthOverrideInput[];
+}): SystemHealthStatusResponse {
+  if (overrides.length === 0) return base;
+
+  // The override that reads worst wins the `override` slot; a tie keeps the
+  // first (query order), which for incidents is a stable, arbitrary pick.
+  let worst = overrides[0]!;
+  for (const candidate of overrides) {
+    if (HEALTH_RANK[candidate.status] > HEALTH_RANK[worst.status]) {
+      worst = candidate;
+    }
+  }
+
+  const status = worstHealthStatus(base.status, worst.status);
+  const override: SystemHealthOverride = {
+    status: worst.status,
+    source: worst.source,
+    reason: worst.reason,
+    sourceId: worst.sourceId,
+  };
+
+  return { ...base, status, override };
+}