mandrel-platform 0.21.0 → 0.25.0

package/templates/runbooks/runner-fleet-health.md

@@ -0,0 +1,150 @@
+# Runner Fleet Health Monitor
+
+> **Self-contained runbook** (not a thin stub). Unlike most templates in this
+> directory, there is no canonical `docs/runbooks/` counterpart to link — this
+> file IS the process, mirroring `runner-provisioning.md`. It documents the
+> scheduled `.github/workflows/runner-fleet-health.yml` monitor (Story #258):
+> what it checks, the token scope it needs, the alert semantics, and the
+> operator response when it fires.
+
+---
+
+## Why this exists
+
+All self-hosted runners across the fleet (`domio`, `athportal`, `swarm-os`)
+are **co-resident on one operator Mac**. If that host sleeps, reboots for an
+OS update, fills its disk, or a launchd runner service dies, **every
+consumer's CI and deploy-trigger jobs silently queue** ("waiting for a
+runner") with no alert. Nothing else watches this:
+
+- The Better Stack uptime unit (`uptime-apply.yml`) monitors the **deployed
+  apps**, not the runners.
+- Deploy pipelines are `workflow_run`-gated, so a wedged runner can silently
+  stall staging indefinitely — no failed job, no notification, just a queue
+  that never drains.
+
+> **Roster note:** `Beestera/swarm-os` is monitored only *indirectly*. No
+> `dsj1984`-owned token can read another org's runner API (fine-grained PATs
+> are bound to one resource owner; the Beestera org rejects classic PATs), so
+> its roster entry would permanently false-positive as `0/3` degraded.
+> Because its runners share the Mac with the rostered repos, a wedged
+> **host** still trips the `domio`/`athportal` rows; what goes unwatched is a
+> swarm-os-only launchd service death and its stale queue. See
+> `$comment_swarm_os` in `scripts/runner-fleet-consumers.json`.
+
+`runner-fleet-health.yml` is the standing check that catches this fast.
+
+## What it checks
+
+Runs on a schedule (~every 15 minutes) plus `workflow_dispatch`, on
+`ubuntu-latest` (deliberately GitHub-hosted so it keeps running when the Mac
+is down). For each repo in `scripts/runner-fleet-consumers.json` it calls
+`GET /repos/{owner}/{repo}/actions/runners` and:
+
+1. **Offline runners** — flags any runner whose `status != online`.
+2. **Count shortfall** — flags fewer online runners matching the repo's
+   expected `labels` set than its configured `expectedCount`.
+3. **Stale queued runs** (optional signal) — a `queued`/`waiting` workflow run
+   older than `staleQueuedMinutes` (default 20) with no online runner matching
+   its labels. This catches the case where the runner *looks* present in the
+   roster count but is actually wedged and not claiming jobs.
+
+It renders a per-repo dashboard on `GITHUB_STEP_SUMMARY`.
+
+## Config-driven roster
+
+Adding, removing, or resizing a runner needs **only a config edit** —
+`scripts/runner-fleet-consumers.json`:
+
+```jsonc
+{
+  "defaultStaleQueuedMinutes": 20,
+  "repos": [
+    { "name": "domio", "repo": "dsj1984/domio", "expectedCount": 3, "labels": ["self-hosted", "macOS", "ARM64", "domio-runner"] },
+    // ... one object per repo
+  ],
+}
+```
+
+## Token scope: `PIN_DRIFT_TOKEN`
+
+The monitor reuses the same fine-grained PAT `pin-drift.yml` already
+provisions (`secrets.PIN_DRIFT_TOKEN`), falling back to the built-in
+`github.token` when the secret is absent (the built-in token only grants read
+access to the workflow's own repo — cross-repo rows then surface as `⚠️
+error` rather than hard-failing this repo's own row).
+
+For the runner reads, `PIN_DRIFT_TOKEN` must carry, on every rostered repo:
+
+- **Administration: read** — required by `GET .../actions/runners` (the
+  self-hosted runner list is an admin-surface endpoint; `actions:read` is
+  NOT sufficient for it).
+- **Actions: read** — required by `GET .../actions/runs` (the stale-queue
+  check).
+
+Resource-owner caveat: a fine-grained PAT is bound to a **single** resource
+owner, and the Beestera org rejects classic PATs — which is exactly why
+`Beestera/swarm-os` is off the roster (see the roster note above). Every
+rostered repo must be readable by the ONE token this workflow gets; a repo
+the token cannot see 404s and false-positives as degraded, so extend the
+roster only together with a credential that covers the new repo.
+
+When the token lacks visibility, GitHub returns **404** (not 403) and the
+script treats the empty runner list as a real shortfall — the repo's row
+reads `❌ degraded` with `0/N` online even when the runners are healthy. A
+fleet-wide `0/N` across every repo is the token-misconfiguration signature;
+check the secret before touching the runner host.
+
+## Alert semantics
+
+Alert-only by design (no host-side remediation) — the monitor never touches
+the runner host itself. One channel fires on an unhealthy repo, deliberately
+without adding a new external dependency:
+
+- **Native GitHub failed-workflow notification.** The job script exits
+  non-zero when any repo is unhealthy, so GitHub's own email/notification
+  settings fire the standard "workflow run failed" alert to whoever
+  watches this repo. No tracking issues are filed — the dashboard detail
+  lives on the failed run's job summary.
+
+A future Slack/PagerDuty push could layer on top of this later — deliberately
+deferred (see the Story's Out of Scope) to avoid a new external dependency for
+the initial alert-only default.
+
+## Operator response
+
+When the scheduled workflow run fails:
+
+1. **Read the dashboard** on the workflow run's job summary — it names which
+   signal fired (offline runner, count shortfall, or stale queued run) and
+   for which repo.
+2. **Wake or reboot the Mac** if it's asleep, powered off, or unresponsive
+   over SSH.
+3. **Check disk space** (`df -h`) — a full disk is a common launchd-runner
+   death cause; free space and restart the affected runner service(s).
+4. **Restart the launchd runner service(s)** for the affected repo:
+   ```bash
+   cd <RUNNER_DIR>          # see templates/runbooks/runner-provisioning.md
+   ./svc.sh stop && ./svc.sh start
+   ./svc.sh status          # expect: Started · running
+   ```
+5. **Re-run the monitor** (`workflow_dispatch` from the Actions tab, or wait
+   for the next 15-minute tick) to confirm recovery — a green run means the
+   fleet reports healthy again.
+
+## Out of scope (Story #258)
+
+- Host-side remediation / auto-recovery (waking the Mac, restarting services)
+  — this monitor is alert-only; the operator performs the response above by
+  hand.
+- Host disk-usage monitoring — not exposable via the runners API anyway, and
+  tracked separately.
+- External paging integrations beyond the native failed-workflow
+  notification.
+- Cross-repo runner isolation / ephemeral-runner questions — explicitly
+  deferred.
+
+## Project-Specific Notes
+
+<!-- Record host quirks, roster changes, or false-positive tuning
+     (staleQueuedMinutes overrides) specific to this fleet. -->

package/templates/workflows/deploy-staging-run.yml

@@ -0,0 +1,117 @@
+name: deploy-staging-run
+
+# Canonical staging-deploy caller template — DEPLOY half (Story #272).
+#
+# > **Why this file runs on `workflow_dispatch`.** Its sibling
+# > `deploy-staging.yml` fires on CI-green (`workflow_run`) and DISPATCHES this
+# > workflow. The deploy lives here, on `workflow_dispatch`, because the shared
+# > `deploy-cloudflare.yml`'s `environment:`-gated jobs (`check-env` /
+# > `migration` / `deploy` / `boot-smoke`) are silently SKIPPED on a
+# > `workflow_run` event but run normally on `workflow_dispatch` (Story #272).
+# > Adopt this file together with `deploy-staging.yml`; `platform-sync`
+# > materializes both.
+#
+# > **Thin local caller.** The defence-in-depth deploy core lives in the shared
+# > `dsj1984/mandrel-platform` `deploy-cloudflare.yml` reusable workflow — see
+# > https://github.com/dsj1984/mandrel-platform/blob/main/docs/reusable-workflows.md#deploy-cloudflareyml.
+# > This file only holds <PROJECT_NAME>-specific values (worker names, build
+# > step, secret mapping). When the deploy PROCESS changes, that change lands
+# > upstream in mandrel-platform — not here.
+#
+# Replace every <PLACEHOLDER> with your project's real values:
+#   <MANDREL_PLATFORM_SHA>   the pinned mandrel-platform commit SHA (resolve via
+#                            `node scripts/platform-sync.mjs --ref <release-tag>`
+#                            from the consumer repo root, or `git ls-remote`).
+#   <MANDREL_PLATFORM_TAG>   the human-readable release tag matching the SHA
+#                            above (trailing `# <tag>` comment).
+#   <WORKERS_CSV>            comma-separated Worker names for this env, e.g.
+#                            "api,web".
+#   <BUILD_COMMAND>          optional build command (omit build-command /
+#                            build-artifact entirely if the deploy job's default
+#                            checkout is build-ready).
+#
+# See the full input/secret contract:
+# https://github.com/dsj1984/mandrel-platform/blob/main/docs/reusable-workflows.md#deploy-cloudflareyml
+
+on:
+  # Dispatched by deploy-staging.yml on CI-green, and available for manual
+  # on-demand deploys (UI "Run workflow" + `gh workflow run`). The optional sha
+  # input records the CI-verified commit; the deploy itself runs against the
+  # main tip (`--ref main`).
+  workflow_dispatch:
+    inputs:
+      sha:
+        description: >
+          Commit SHA that passed CI (informational — the deploy runs against
+          the current main tip). Populated automatically when dispatched by
+          deploy-staging.yml.
+        required: false
+        type: string
+
+permissions:
+  contents: read
+
+# Serialize staging deploys: QUEUE, don't cancel (Story #284 / audit H2). An
+# in-flight `wrangler d1 migrations apply` must never be cancelled mid-apply by
+# the next dispatch — a half-applied forward-only migration has no automatic
+# restore. `cancel-in-progress: false` lets a second dispatch wait for the
+# first to finish. (The dispatcher half keeps `cancel-in-progress: true`:
+# cancelling a superseded PENDING dispatch is safe.) The shared
+# deploy-cloudflare.yml additionally serializes per-environment.
+concurrency:
+  group: deploy-staging-run
+  cancel-in-progress: false
+
+jobs:
+  # M4: sha-drift preflight (Story #284). The CI-green gate is TOCTOU-racy — a
+  # commit A goes green and dispatches, but commit B (CI pending, possibly
+  # later red) may already be on main, so the deploy would ship B under A's
+  # green credential. When `sha` is supplied (dispatcher path), assert the main
+  # tip still equals it and FAIL loudly on drift, naming both SHAs. A job with
+  # `uses:` cannot carry `steps:`, so this preflight is a SEPARATE preceding
+  # job that the `uses:` deploy job `needs:`. Manual dispatches without `sha`
+  # skip the assertion (the operator is deploying the current tip on purpose).
+  sha-drift-preflight:
+    name: Sha-drift preflight (deploy what CI verified)
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - name: Checkout main tip
+        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+        with:
+          ref: main
+          persist-credentials: false
+      - name: Assert HEAD matches the CI-verified sha
+        env:
+          INPUT_SHA: ${{ inputs.sha }}
+        shell: bash
+        run: |
+          set -euo pipefail
+          if [ -z "${INPUT_SHA}" ]; then
+            echo "No sha input — manual dispatch deploying the current main tip; skipping sha-drift preflight."
+            exit 0
+          fi
+          head="$(git rev-parse HEAD)"
+          if [ "${head}" != "${INPUT_SHA}" ]; then
+            echo "::error::sha drift: main tip is ${head} but the dispatched (CI-verified) sha is ${INPUT_SHA}. main advanced after CI went green; refusing to deploy an unverified tip." >&2
+            exit 1
+          fi
+          echo "sha-drift preflight OK: main tip ${head} matches the CI-verified sha ${INPUT_SHA}."
+
+  deploy:
+    name: Staging deploy (shared deploy-cloudflare.yml)
+    needs: [sha-drift-preflight]
+    uses: dsj1984/mandrel-platform/.github/workflows/deploy-cloudflare.yml@<MANDREL_PLATFORM_SHA> # <MANDREL_PLATFORM_TAG>
+    with:
+      environment: staging
+      gh-environment: staging
+      workers: <WORKERS_CSV>
+      migrate: true
+      # db-engine defaults to 'd1'. Set db-engine + migrate-command +
+      # snapshot-command for a non-D1 engine (e.g. Turso) — see the contract
+      # doc's "command seams" section.
+    # Frozen secret allowlist: only {CLOUDFLARE_*, TURSO_*} cross into the
+    # shared workflow. Map your project's secret NAMES onto these slots.
+    secrets:
+      CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
+      CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}

package/templates/workflows/deploy-staging.yml

@@ -1,89 +1,150 @@
 name: deploy-staging
 
-# Canonical staging-deploy caller template (Story #175).
+# Canonical staging-deploy caller template — DISPATCHER half (Story #175,
+# reworked for Story #272).
+#
+# > **Why two files (Story #272).** A reusable workflow's `environment:`-gated
+# > jobs (`deploy-cloudflare.yml`'s `check-env` / `migration` / `deploy` /
+# > `boot-smoke`) are **silently skipped on a `workflow_run` event** — a
+# > documented GitHub limitation. The previous single-file template called
+# > `deploy-cloudflare.yml` DIRECTLY from `on: workflow_run`, so those jobs
+# > skipped and **0 workers deployed while the run reported green** (all-skipped,
+# > none failed). One consumer hid 40+ consecutive non-deploys this way.
+# >
+# > The fix: this file is now a thin **dispatcher** that fires on CI-green and
+# > re-launches the deploy via **`workflow_dispatch`** — the event on which the
+# > `environment:` jobs DO run — against its sibling `deploy-staging-run.yml`.
+# > `workflow_dispatch` and `repository_dispatch` are the two events that
+# > "always create workflow runs" even when triggered with the built-in
+# > `GITHUB_TOKEN`, so **no PAT is required** for this same-repo dispatch — just
+# > `permissions: actions: write` below. (Cross-repo dispatch, like
+# > `smoke-dispatch.yml`, still needs a PAT; same repo does not.)
 #
 # > **Thin local caller.** The defence-in-depth deploy core (secret-isolation
 # > audit -> CF env gate -> migration (snapshot + apply) -> deploy ->
-# > boot-smoke + auto-rollback) AND the CI-green guard both live in the shared
+# > boot-smoke + auto-rollback) lives in the shared
 # > `dsj1984/mandrel-platform` `deploy-cloudflare.yml` reusable workflow — see
 # > https://github.com/dsj1984/mandrel-platform/blob/main/docs/reusable-workflows.md#deploy-cloudflareyml.
-# > This file only holds <PROJECT_NAME>-specific values (worker names, build
-# > step, secret mapping). When the deploy PROCESS changes, that change lands
-# > upstream in mandrel-platform — not here.
-#
-# One paved road (operator decision 2026-07-01, D4): every consumer triggers
-# staging deploy via `workflow_run` on its own CI workflow, gated on
-# `conclusion == 'success'`. `workflow_run` fires on BOTH a successful AND a
-# failed upstream run, so a caller-side guard against a red run used to be
-# REQUIRED here — every consumer hand-copied the same `preflight` job (see
-# mandrel-platform Story #175 context). That guard now lives INSIDE
-# `deploy-cloudflare.yml` itself as a job-level `if:` on its entry jobs
-# (`github.event` inside a reusable workflow is the CALLER's event, so the
-# shared workflow can see and gate on the `workflow_run` conclusion even
-# though it cannot own this file's `on:` block); a red upstream run skips
-# the whole chain with zero runner spin-ups (mandrel-platform Story #237).
-# This template needs NO caller-side preflight guard as a result —
-# copy it as-is and fill in the placeholders below.
+# > The <PROJECT_NAME>-specific values (worker names, build step, secret
+# > mapping) live in the sibling `deploy-staging-run.yml`. When the deploy
+# > PROCESS changes, that change lands upstream in mandrel-platform — not here.
 #
-# Replace every <PLACEHOLDER> with your project's real values:
-#   <CI_WORKFLOW_NAME>        the `name:` of the workflow this deploy should
-#                              gate on (e.g. "quality", "CI", "PR Quality").
-#                              Must match EXACTLY — GitHub matches
-#                              `workflow_run.workflows` by workflow name, not
-#                              file path.
-#   <MANDREL_PLATFORM_SHA>     the pinned mandrel-platform commit SHA (resolve
-#                              via `node scripts/platform-sync.mjs --ref
-#                              <release-tag>` from the consumer repo root, or
-#                              hand-resolve via `git ls-remote`).
-#   <MANDREL_PLATFORM_TAG>     the human-readable release tag matching the SHA
-#                              above (trailing `# <tag>` comment).
-#   <WORKERS_CSV>              comma-separated Worker names for this env, e.g.
-#                              "api,web".
-#   <BUILD_COMMAND>            optional build command (omit build-command /
-#                              build-artifact entirely if the deploy job's
-#                              default checkout is build-ready).
+# Adopt BOTH files together: this `deploy-staging.yml` (dispatcher) and
+# `deploy-staging-run.yml` (the actual deploy). `platform-sync` materializes
+# both. Replace every <PLACEHOLDER>:
+#   <CI_WORKFLOW_NAME>   the `name:` of the workflow this deploy gates on (e.g.
+#                        "quality", "CI"). Must match EXACTLY — GitHub matches
+#                        `workflow_run.workflows` by workflow name, not path.
 #
 # See the full input/secret contract:
 # https://github.com/dsj1984/mandrel-platform/blob/main/docs/reusable-workflows.md#deploy-cloudflareyml
 
 on:
-  # CI-green gate: fires when <CI_WORKFLOW_NAME> finishes on main. The shared
-  # deploy-cloudflare.yml's job-level CI-green gate skips the entire job
-  # chain (zero runners) unless the upstream conclusion was 'success' — no
-  # caller-side guard needed.
+  # CI-green gate: fires when <CI_WORKFLOW_NAME> completes on main. Unlike the
+  # old shape, this does NOT call the deploy directly — a `workflow_run` deploy
+  # would skip every `environment:` job. It only DISPATCHES the deploy (below)
+  # on success, so the actual deploy runs on `workflow_dispatch` where those
+  # jobs execute. A red upstream run simply does not dispatch — there is no
+  # green-but-didn't-deploy run at all.
   workflow_run:
     workflows: [<CI_WORKFLOW_NAME>]
     branches: [main]
     types: [completed]
-  # Manual on-demand trigger (UI "Run workflow" + `gh workflow run`).
-  # workflow_dispatch always passes the shared workflow's CI-green guard
-  # (operator-intentional, no upstream conclusion to gate on).
-  workflow_dispatch:
 
+# actions:write lets the built-in GITHUB_TOKEN dispatch deploy-staging-run.yml
+# via the workflow_dispatch API. No PAT needed for a same-repo dispatch —
+# workflow_dispatch always creates a run even from GITHUB_TOKEN.
 permissions:
   contents: read
+  actions: write
 
-# Cancel an in-flight staging deploy when a newer commit lands on main — only
-# the freshest tip of main should reach the staging surfaces. The shared
-# deploy-cloudflare.yml additionally serializes per-environment.
+# Only the freshest tip of main should reach staging: cancel an in-flight
+# DISPATCH when a newer CI run completes. (The deploy itself is serialized
+# separately in deploy-staging-run.yml and per-environment inside the shared
+# deploy-cloudflare.yml.)
 concurrency:
-  group: deploy-staging
+  group: deploy-staging-dispatch
   cancel-in-progress: true
 
 jobs:
-  deploy:
-    name: Staging deploy (shared deploy-cloudflare.yml)
-    uses: dsj1984/mandrel-platform/.github/workflows/deploy-cloudflare.yml@<MANDREL_PLATFORM_SHA> # <MANDREL_PLATFORM_TAG>
-    with:
-      environment: staging
-      gh-environment: staging
-      workers: <WORKERS_CSV>
-      migrate: true
-      # db-engine defaults to 'd1'. Set db-engine + migrate-command +
-      # snapshot-command for a non-D1 engine (e.g. Turso) — see the contract
-      # doc's "command seams" section.
-    # Frozen secret allowlist: only {CLOUDFLARE_*, TURSO_*} cross into the
-    # shared workflow. Map your project's secret NAMES onto these slots.
-    secrets:
-      CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
-      CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
+  dispatch:
+    name: Dispatch staging deploy on CI-green
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    # CI-green gate — three load-bearing conditions, all required (Story #284):
+    #   1. `conclusion == 'success'` — `workflow_run` fires on both success and
+    #      failure; without this a red main would still deploy.
+    #   2. `event == 'push'` — the upstream CI run must itself have been a push
+    #      to the repo, not a `pull_request` run. `workflow_run.branches:
+    #      [main]` filters on the *head branch NAME*, so a fork PR whose head
+    #      branch is literally named `main` with green CI otherwise satisfies
+    #      the branch filter and the success guard (audit M3).
+    #   3. `head_repository.full_name == github.repository` — the CI run must
+    #      have originated from THIS repo, not a fork. Belt-and-suspenders with
+    #      condition 2: even a same-name fork branch cannot spoof same-repo
+    #      provenance, so an external contributor can no longer attacker-time a
+    #      staging deploy off their fork's green CI.
+    if: >-
+      ${{ github.event.workflow_run.conclusion == 'success'
+        && github.event.workflow_run.event == 'push'
+        && github.event.workflow_run.head_repository.full_name == github.repository }}
+    steps:
+      - name: Dispatch deploy-staging-run.yml (workflow_dispatch)
+        env:
+          GH_TOKEN: ${{ github.token }}
+          REPO: ${{ github.repository }}
+          SHA: ${{ github.event.workflow_run.head_sha }}
+        shell: bash
+        run: |
+          set -euo pipefail
+
+          # L3: the dispatch is otherwise fire-and-forget — a transient GitHub
+          # API 5xx fails `gh workflow run`, and the only signal is a red run in
+          # a low-visibility dispatcher while that green commit never reaches
+          # staging. Retry with backoff, then VERIFY a run was actually created
+          # (a 2xx from the dispatch API is not proof a run materialized).
+          dispatched=false
+          for attempt in 1 2 3; do
+            if gh workflow run deploy-staging-run.yml \
+              --repo "${REPO}" \
+              --ref main \
+              -f sha="${SHA}"; then
+              dispatched=true
+              break
+            fi
+            echo "::warning::gh workflow run attempt ${attempt} failed; retrying after backoff."
+            sleep $((attempt * 5))
+          done
+
+          if [ "${dispatched}" != "true" ]; then
+            echo "::error::Failed to dispatch deploy-staging-run.yml after 3 attempts for ${SHA}." >&2
+            exit 1
+          fi
+
+          # Verify a runner workflow run was actually created. The dispatch API
+          # returns before the run row is queryable, so poll `gh run list` for a
+          # recent workflow_dispatch run of the runner workflow. Fail loudly if
+          # none appears — a silent non-creation is exactly the failure mode
+          # this check exists to surface.
+          created=false
+          for attempt in 1 2 3 4 5; do
+            count="$(gh run list \
+              --repo "${REPO}" \
+              --workflow deploy-staging-run.yml \
+              --event workflow_dispatch \
+              --limit 5 \
+              --json databaseId --jq 'length' 2>/dev/null || echo 0)"
+            if [ "${count:-0}" -gt 0 ]; then
+              created=true
+              break
+            fi
+            echo "Run not visible yet (attempt ${attempt}); waiting for the run row to appear."
+            sleep $((attempt * 3))
+          done
+
+          if [ "${created}" != "true" ]; then
+            echo "::error::Dispatched deploy-staging-run.yml for ${SHA} but no runner workflow run appeared via 'gh run list'." >&2
+            exit 1
+          fi
+
+          echo "Dispatched and verified staging deploy for ${SHA} (deploy runs on workflow_dispatch so environment: jobs execute)."