backupdrill 2.5.0 → 2.7.0

package/README.md

@@ -59,7 +59,7 @@ Configure via **env vars**, a **config file**, or **flags** (precedence: flags >
 ### Env vars
 
 ```bash
-export BACKUPDRILL_DATABASE_URL="postgresql://postgres.<ref>:<pw>@aws-0-<region>.pooler.supabase.com:5432/postgres"
+export BACKUPDRILL_DATABASE_URL="postgresql://postgres.<ref>:<pw>@<pooler-host>:5432/postgres"
 export BACKUPDRILL_PROJECT_NAME="my-app"
 export BACKUPDRILL_S3_ENDPOINT="https://<account>.r2.cloudflarestorage.com"  # omit for AWS S3
 export BACKUPDRILL_S3_REGION="auto"
@@ -98,7 +98,7 @@ backupdrill backup --config backupdrill.config.json
 
 ## Get the connection string
 
-Use the **Session Pooler** string from your Supabase dashboard (Project Settings → Database → Connection string → Session pooler). The default `postgres` user works as-is — it owns your tables, so `pg_dump` passes row-level security.
+Use the **Session pooler** string: in the Supabase dashboard, open your project, click **Connect** at the top of the page and pick **Session pooler**. Copy the whole string rather than typing the host yourself, because the pooler host can't be worked out from the region. Replace `[YOUR-PASSWORD]` with the database password (percent-encode characters like `@` or `#`). The default `postgres` user works as-is — it owns your tables, so `pg_dump` passes row-level security.
 
 **Least-privilege role (advanced, usually not viable on Supabase):** `pg_dump` needs `select` on *sequences* too, and it **errors on any RLS-enabled table the role can't bypass** — and most Supabase tables have RLS on, while `bypassrls` can't be granted without superuser. Only use a dedicated role if your schema has no RLS tables:
 
@@ -158,7 +158,9 @@ backupdrill estimate --plan pro   # measures your DB + Storage, prints the proje
 <prefix>/<project>/<timestamp>/
   ├── dump.pgcustom              # pg_dump --format=custom (public schema by default)
   ├── storage/<bucket>/<key>     # your Storage files (if configured)
+  ├── RECOVERY.md                # how to restore this snapshot
   └── manifest.json              # schemas, tables, row counts, file list, sizes + sha256
+<prefix>/<project>.drills/<finished-at>-<passed|failed>-<tag>.json  # only with drill --record: one file per drill
 ```
 
 ## Restoring & drilling
@@ -174,6 +176,8 @@ Backups record your installed Postgres extensions in the manifest, and the drill
 
 The sandbox restore runs up to 4 parallel `pg_restore` jobs, sized from the CPUs and memory Docker reports, so a 2 vCPU / 8 GB machine gets 2. Table data and index builds run in parallel. Foreign keys, triggers and policies run on a single connection to keep their error messages separate, including errors about Supabase-managed objects such as `auth.users`. `backupdrill restore` into a real Supabase project stays on a single connection.
 
+Ctrl+C (or a SIGTERM from whatever runs the drill) stops it and cleans up: the CLI stops a running `--check-cmd` and `pg_restore`, removes the sandbox container and its volume, deletes the downloaded dump, and exits with 130 (143 for SIGTERM). `--keep` doesn't apply to an interrupted drill. Press Ctrl+C a second time to exit without waiting for the cleanup.
+
 ### Run your own checks against the restored copy
 
 The drill's structural checks (tables, rows, checksums) can't know what "working" means for *your* app — business invariants only have answers in your code. `--check-cmd` hands you the restored sandbox before it's destroyed:
@@ -184,7 +188,7 @@ backupdrill drill --check-cmd "npm run smoke"   # BACKUPDRILL_SANDBOX_URL points
 
 Your command runs after all structural checks pass, with `BACKUPDRILL_SANDBOX_URL` set to the sandbox's connection string. Exit 0 means pass; the result shows up in the report as an `app checks` line, and a non-zero exit fails the drill (the report keeps structural and app-level failures clearly separate). Entirely optional — without the flag, nothing changes and the report has no `app checks` line. There's a 10-minute timeout (the whole process tree is killed), and `--keep` leaves the sandbox container running when a drill fails so you can inspect it — the report JSON then includes a `keptSandbox` object with the container id and connection string (`docker rm -f <id>` when done).
 
-**What this can and cannot verify, honestly:** `BACKUPDRILL_SANDBOX_URL` connects as the sandbox **superuser**, and superusers bypass row-level security. The sandbox does carry a stub of the Supabase runtime — an `auth` schema whose `uid()`/`jwt()`/`role()`/`email()` return null, and the `anon`/`authenticated`/`service_role` roles — so functions, column defaults and policies that reference them restore instead of being skipped. The sandbox deliberately does not create Supabase-managed *tables* such as `auth.users`: an index, constraint or policy that points at one is reported as a skipped managed object (these are restored in a second, post-data pass where that exemption applies), while a *view* over one is created in the strict first pass and fails the drill — a real gap in what a public-only backup can rebuild, shown rather than hidden. A function *body* that selects from `auth.users` is different: PostgreSQL does not validate string-literal bodies (`AS $$ … $$`) during a restore (`check_function_bodies` is off in every dump), so such a function is recreated and the drill passes — it would only fail when called. The exception is an SQL-standard `BEGIN ATOMIC … END` body, which is parsed and dependency-checked at creation regardless of that setting, so one of those referencing a managed table fails the drill like a view does. That is exactly what `--check-cmd` is for: if your application depends on such a function, call it there. So use `--check-cmd` for data and business invariants ("orders reference existing users", "the row count I care about is sane"). It is **not** an RLS behavior test — `auth.uid()` is a stub and you are a superuser; verifying RLS takes an authenticated client stack with real JWTs, which no restore sandbox can fake generically.
+**What this can and cannot verify, honestly:** `BACKUPDRILL_SANDBOX_URL` connects as the sandbox **superuser**, and superusers bypass row-level security. The sandbox does carry a stub of the Supabase runtime — an `auth` schema whose `uid()`/`jwt()`/`role()`/`email()` return null, and the `anon`/`authenticated`/`service_role` roles — so functions, column defaults and policies that reference them restore instead of being skipped. The sandbox deliberately does not create Supabase-managed *tables* such as `auth.users`: an index, constraint or policy that points at one is reported as a skipped managed object (these are restored in a second, post-data pass where that exemption applies). A skipped foreign key into `auth.users` still matters for a real restore, see [Tables that reference auth.users](#tables-that-reference-authusers). A *view* over one is created in the strict first pass instead, and fails the drill — a real gap in what a public-only backup can rebuild, shown rather than hidden. A function *body* that selects from `auth.users` is different: PostgreSQL does not validate string-literal bodies (`AS $$ … $$`) during a restore (`check_function_bodies` is off in every dump), so such a function is recreated and the drill passes — it would only fail when called. The exception is an SQL-standard `BEGIN ATOMIC … END` body, which is parsed and dependency-checked at creation regardless of that setting, so one of those referencing a managed table fails the drill like a view does. That is exactly what `--check-cmd` is for: if your application depends on such a function, call it there. So use `--check-cmd` for data and business invariants ("orders reference existing users", "the row count I care about is sane"). It is **not** an RLS behavior test — `auth.uid()` is a stub and you are a superuser; verifying RLS takes an authenticated client stack with real JWTs, which no restore sandbox can fake generically.
 
 When you actually need to recover, `restore` puts the database back into a fresh
 target project (through the same engine the drills use, with verification) and
@@ -200,6 +204,8 @@ backupdrill restore --database --target-supabase-url https://<ref>.supabase.co -
 
 Prefer files on disk instead? Skip the Storage target and pass `--storage-dir ./recovered`.
 
+`--dry-run` writes nothing to the target. Besides checking the target (not the source project, schemas empty, Postgres and `pg_restore` versions, extensions available), it downloads the dump to a temporary directory, checks its sha256 and looks for foreign keys that point outside the backup (see [Tables that reference auth.users](#tables-that-reference-authusers)). The real run repeats those checks before it writes anything; a foreign key into a table that one of the recorded extensions creates is checked again once the extensions are installed. The database restore then runs in a single transaction: if any statement fails, nothing from the snapshot stays on the target, and you can run it again on the same project once you've fixed the cause. (If the connection drops while it commits, the error says so and asks you to check the target first.) Extensions the run installed stay installed, since the next run needs them too, and the empty-target check doesn't count objects that belong to an extension (except an extension's configuration tables, whose rows a dump can carry). One transaction holds a lock on every table, index and sequence until it commits, so a very large schema can run out of lock slots: on Supabase's smallest compute size, about a thousand tables (partitions count as tables). The restore then stops with nothing written and tells you how to raise `max_locks_per_transaction`.
+
 Or restore the dump by hand — into an **empty** database, without `--clean`:
 
 ```bash
@@ -209,8 +215,35 @@ pg_restore --no-owner --no-privileges --dbname "<target-connection-string>" dump
 Don't use `--clean` against a Supabase project: dropping and recreating the `public`
 schema silently wipes Supabase's default grants for `anon`/`authenticated` (the dump
 carries no ACLs), leaving the restored API returning 401s. One error is expected and
-harmless: `schema "public" already exists` — the target always has it. The
-`backupdrill restore` command above handles all of this (plus verification) for you.
+harmless: `schema "public" already exists` — the target always has it. An error
+on a foreign key into `auth.users` means the users have to be copied in first (see
+the next section). The `backupdrill restore` command above handles all of this (plus
+verification) for you.
+
+### Tables that reference auth.users
+
+By default a backup covers the `public` schema only, so your Auth users are not in it. Plenty of Supabase apps have a table like `profiles` whose `id` is a foreign key to `auth.users(id)`. When the restore recreates that key in a new project, Postgres checks every `profiles` row against `auth.users`, and in a new project that table is empty, so the restore fails.
+
+A drill can't catch this. Its sandbox has no `auth.users` at all, so the drill reports that foreign key as a skipped Supabase-managed object and passes. `restore --dry-run` checks it against the real target and tells you how many rows have nothing to point at:
+
+```
+✗ references outside the backup: public.profiles → auth.users (profiles_id_fkey): 200 of 200 row(s) point at auth.users rows the target doesn't have
+```
+
+While that check fails, no snapshot data is written. Copy the users into the new project first, then run the restore. While the old project is still reachable, the Postgres tools can do it. Use the Session pooler string of each project, and type the passwords at hidden prompts so they stay out of your shell history:
+
+```bash
+read -rs OLD_PW; read -rs NEW_PW
+PGPASSWORD="$OLD_PW" pg_dump --data-only --table=auth.users --table=auth.identities \
+  "postgresql://postgres.<old-ref>@<old-pooler-host>:5432/postgres" |
+  PGPASSWORD="$NEW_PW" psql -v ON_ERROR_STOP=1 \
+  "postgresql://postgres.<new-ref>@<new-pooler-host>:5432/postgres"
+unset OLD_PW NEW_PW
+```
+
+`auth.identities` holds each user's sign-in methods; users copied with it can sign in to the new project with their old password. If the old project is gone, the users can only come from a backup that includes the `auth` schema, such as Supabase's own daily backups on a paid plan. A backupdrill snapshot of `public` doesn't have them.
+
+Triggers you created on `auth.users`, such as the common one that inserts a `profiles` row on sign-up, belong to that table, so they aren't in the backup either. Create them again after the restore.
 
 ## 2.0 migration — `restore` flags changed
 
@@ -261,6 +294,80 @@ Both pings of a run carry the same `rid`, so Healthchecks pairs them even if two
 
 A ping never changes the result of a run. Each request times out after 10 seconds and is retried twice on network errors and 5xx responses; if it still fails, the CLI prints a warning and carries on, with the same exit code it would have had without a ping URL.
 
+## Prometheus metrics
+
+`backupdrill exporter` serves [Prometheus](https://prometheus.io) metrics about the backups in your bucket, and about the restore drills you run with `--record`. It reads the bucket each time Prometheus scrapes it and never writes, so a key with read-only access to the bucket is enough.
+
+```bash
+# reads the same BACKUPDRILL_S3_* and BACKUPDRILL_PROJECT_NAME settings as backup and drill
+backupdrill exporter                  # http://127.0.0.1:10059/metrics
+backupdrill exporter --host 0.0.0.0   # when Prometheus runs on another machine
+```
+
+It listens on `127.0.0.1` unless you pass `--host`, because every scrape costs a few requests against your bucket and the port shouldn't end up open to the internet by accident. `--port` changes the port.
+
+Drill results only show up if the drill leaves them in the bucket. Add `--record` to your drill job: when the drill finishes, it writes one small JSON file to `<prefix>/<project>.drills/`, named after the time it finished, whether it passed and a random tag (`2026-10-20T04-12-09-311Z-passed-3f9a1c07.json`). Each drill writes a file under its own name instead of updating a shared one, so the order in which two overlapping drills finish writing doesn't matter. Only the outcome, the time, the snapshot name and the restore time are stored, never the check details. A drill that stops early (Docker isn't running, the bucket is empty) is recorded as not passed. `--record` needs write access to the bucket; if the write fails, the drill prints a warning and its own result and exit code stay the same. Without `--record`, `drill` only reads the bucket, as before.
+
+```bash
+backupdrill drill --record
+```
+
+Point Prometheus at it. One exporter watches one project. The exporter doesn't add a project label, so set one in the scrape config:
+
+```yaml
+scrape_configs:
+  - job_name: backupdrill
+    scrape_interval: 1m
+    static_configs:
+      - targets: ["backup-host:10059"]
+        labels:
+          project: my-app
+```
+
+For a second project, run a second exporter with `--project-name` and a different `--port`.
+
+| Metric | Meaning |
+|---|---|
+| `backupdrill_up` | 1 if the bucket was read, 0 if not (the reason goes to the exporter's log) |
+| `backupdrill_scrape_duration_seconds` | How long reading the bucket took |
+| `backupdrill_build_info{version}` | CLI version, always 1 |
+| `backupdrill_snapshots` | Snapshot folders for the project, complete or not |
+| `backupdrill_backup_last_success_timestamp_seconds` | When the newest complete snapshot finished (`createdAt` in its `manifest.json`); 0 if there is none |
+| `backupdrill_latest_snapshot_dump_bytes` | Size of the database dump in that snapshot |
+| `backupdrill_latest_snapshot_storage_files`, `_storage_bytes` | Storage files in that snapshot and their total size; 0 for a database-only backup |
+| `backupdrill_latest_snapshot_tables`, `_estimated_rows` | Tables captured, and the row count Postgres estimated at backup time |
+| `backupdrill_drill_last_run_timestamp_seconds` | When the last recorded drill finished, passed or not; 0 if none |
+| `backupdrill_drill_last_run_passed` | 1 if that drill passed, 0 if not |
+| `backupdrill_drill_last_success_timestamp_seconds` | When the last recorded drill that passed finished; 0 if none |
+| `backupdrill_drill_last_success_restore_duration_seconds` | How long the database restore took in that drill |
+
+A snapshot counts as complete once its `manifest.json` exists, because the backup writes it last. While there is no complete snapshot, the `latest_snapshot` metrics are left out. `backupdrill_drill_last_run_passed` appears once a drill has been recorded, and the restore duration once a recorded drill has passed. A manifest dated more than 10 minutes in the future counts as a failed read (`backupdrill_up 0`), and a drill file named with such a time is skipped with a warning in the log: either would hold back the alerts that work out an age from that timestamp. Check the clock on the machine that wrote it.
+
+Alert rules to start from. Save them as `backupdrill-alerts.yml`, list that file under `rule_files` in `prometheus.yml` and reload Prometheus; `promtool check rules` passes on them:
+
+```yaml
+groups:
+  - name: backupdrill
+    rules:
+      - alert: BackupdrillBucketUnreadable
+        expr: backupdrill_up == 0
+        for: 15m
+      - alert: SupabaseBackupMissing
+        # daily backups; use 8 * 86400 for weekly
+        expr: time() - backupdrill_backup_last_success_timestamp_seconds > 26 * 3600
+        for: 15m
+      - alert: SupabaseBackupShrank
+        expr: backupdrill_latest_snapshot_dump_bytes < 0.7 * max_over_time(backupdrill_latest_snapshot_dump_bytes[7d])
+      - alert: SupabaseRestoreDrillFailed
+        expr: backupdrill_drill_last_run_passed == 0
+      - alert: SupabaseRestoreDrillMissing
+        # weekly drills
+        expr: time() - backupdrill_drill_last_success_timestamp_seconds > 8 * 86400
+        for: 1h
+```
+
+What a scrape costs: one list request per 1,000 snapshot folders and one per 1,000 drill files, plus reads of the newest `manifest.json` and the newest passing drill file. The manifest is fetched with `If-None-Match`, so when it hasn't changed the bucket answers `304 Not Modified` and nothing is downloaded. That matters once you back up Storage files, since the manifest lists every file. At a one-minute interval this comes to about 86,000 list requests and 86,000 small reads a month; check your storage provider's request pricing. Each read of the bucket stops half a second before Prometheus's scrape timeout (10 seconds if there's no Prometheus header), so a slow bucket shows up as `backupdrill_up 0` instead of a failed scrape.
+
 ## Roadmap
 
 - [x] Database backup → your bucket, with checksummed manifest
@@ -270,6 +377,7 @@ A ping never changes the result of a run. Each request times out after 10 second
 - [x] Storage-file integrity check inside the drill (checksums, sampled by default)
 - [x] `restore` — recover a snapshot into a fresh Supabase project: database through the drill-grade engine with verification, Storage uploaded back with per-file reconciliation (local download remains an option)
 - [x] GitHub Action wrapper
+- [x] `exporter` — Prometheus metrics for backups and recorded drills
 
 For scheduling, automated restore drills, and alerting, use the hosted service at [backupdrill.com](https://backupdrill.com).
 
@@ -278,9 +386,10 @@ For scheduling, automated restore drills, and alerting, use the hosted service a
 `backupdrill` follows [semantic versioning](https://semver.org/). From **1.0.0** onwards the
 following is the stable public surface — breaking it requires a major version bump:
 
-- the commands (`backup`, `drill`, `restore`, `estimate`) and their flags
+- the commands (`backup`, `drill`, `restore`, `estimate`, `exporter`) and their flags
 - the `BACKUPDRILL_*` environment variables and the config-file keys
-- the layout written to your bucket, and the shape of `manifest.json`
+- the layout written to your bucket, and the shape of `manifest.json` and the drill record files
+- the names and meaning of the metrics `exporter` serves
 
 `manifest.json` carries a `schemaVersion` so the format can evolve safely. A snapshot written by
 a newer release is **refused with a clear message** rather than parsed on a best-effort basis —

package/dist/deadline.d.ts

@@ -0,0 +1,9 @@
+/**
+ * 到点就放弃,不管 run 停没停。只把 AbortSignal 交给 SDK 不够:它在两次重试之间按
+ * Retry-After 睡觉时不看 signal,503 带 Retry-After: 2 就能把 1 秒的时限拖到 2 秒多。
+ * signal 仍然传下去;结束时(成功、失败、超时)一律 abort,让并行中还没回来的请求收手,
+ * 不然一个先失败、一个挂着的 Promise.all 会把挂着的那个连接一直留着。
+ */
+export declare function withDeadline<T>(run: (signal: AbortSignal) => Promise<T>, ms: number): Promise<T>;
+/** 超时类错误(withDeadline 的、AbortSignal 的)统一认出来,好给一句人话 */
+export declare function isTimeout(error: unknown): boolean;

package/dist/deadline.js

@@ -0,0 +1,29 @@
+/**
+ * 到点就放弃,不管 run 停没停。只把 AbortSignal 交给 SDK 不够:它在两次重试之间按
+ * Retry-After 睡觉时不看 signal,503 带 Retry-After: 2 就能把 1 秒的时限拖到 2 秒多。
+ * signal 仍然传下去;结束时(成功、失败、超时)一律 abort,让并行中还没回来的请求收手,
+ * 不然一个先失败、一个挂着的 Promise.all 会把挂着的那个连接一直留着。
+ */
+export async function withDeadline(run, ms) {
+    const controller = new AbortController();
+    let timer;
+    const deadline = new Promise((_, reject) => {
+        timer = setTimeout(() => {
+            const error = new Error(`timed out after ${ms}ms`);
+            error.name = "TimeoutError";
+            reject(error);
+        }, ms);
+    });
+    try {
+        return await Promise.race([run(controller.signal), deadline]);
+    }
+    finally {
+        clearTimeout(timer);
+        controller.abort();
+    }
+}
+/** 超时类错误(withDeadline 的、AbortSignal 的)统一认出来,好给一句人话 */
+export function isTimeout(error) {
+    const name = error?.name;
+    return name === "TimeoutError" || name === "AbortError";
+}

package/dist/drill-record.d.ts

@@ -0,0 +1,53 @@
+import { type S3Client } from "@aws-sdk/client-s3";
+import { type BackupConfig } from "./config.js";
+import type { DrillReport } from "./drill.js";
+/**
+ * `drill --record` 写进桶里的演练结论。`exporter` 只读桶,演练结果不落桶它就无从得知。
+ *
+ * 每次演练写一个新对象,从不覆盖:
+ *   <prefix>/<project>.drills/<结束时间>-<passed|failed>-<8 位随机>.json
+ * 为什么不是一份可覆盖的"最近一次":两次演练重叠时,先结束的那次可能后写进桶,把更新的
+ * "没通过"盖成更早的"通过",失败告警就没了;先读后写也挡不住(不是原子的),而各家
+ * S3 兼容存储对条件写入的支持参差不齐。每次一个对象,"哪次最新"留给读的一方按名字判断,
+ * 写入顺序就无所谓了。
+ *
+ * 为什么放在 <project>.drills/,和快照目录并排而不是在它里面:resolveSnapshot(包括已经发出去的
+ * 旧版本)按字典序取 <project>/ 下最大的目录当最新快照,里面多一个 drills/ 会被当成快照。
+ *
+ * 文件名本身带着结束时间和结论,exporter 列一下目录就知道最近一次和最近一次通过是哪个;
+ * 只为了恢复用时才读一次通过的那份。只记结论,不记检查明细:失败明细里可能有 pg_restore
+ * 回显的表数据,而桶的读权限可能比演练日志给得更宽(exporter 只要只读 key)。
+ */
+/** 版本规则同 manifest:缺省不接受,更高的版本明确报错让用户升级,不按旧结构硬读。 */
+export declare const DRILL_RECORD_SCHEMA_VERSION = 1;
+export interface DrillRecord {
+    schemaVersion: number;
+    tool: "backupdrill-cli";
+    toolVersion: string;
+    /** 演练结束的时间(ISO8601),与文件名里的时间相同 */
+    finishedAt: string;
+    pass: boolean;
+    /** 演练的快照名;演练在选定快照之前就失败了(没装 Docker、桶里没有快照)时没有 */
+    snapshot?: string;
+    /** 恢复用时,只有通过的演练才记:没通过时这个数可能只是半程,拿来估恢复时间会误导 */
+    restoreSeconds?: number;
+}
+/** 演练记录所在的"目录",带尾斜杠 */
+export declare function drillRecordsPrefix(config: BackupConfig): string;
+export declare function drillRecordName(record: Pick<DrillRecord, "finishedAt" | "pass">, unique?: string): string;
+/** 从文件名读出结束时间(毫秒)和结论;不是记录文件的名字返回 null */
+export declare function parseDrillRecordName(name: string): {
+    finishedAt: number;
+    pass: boolean;
+} | null;
+/** report 为 null = 演练中途抛错、没有报告;这也算一次没通过的演练。 */
+export declare function buildDrillRecord(report: DrillReport | null, finishedAt: Date): DrillRecord;
+/** 写一个新对象,返回它的 key。不读桶:只需要写权限 */
+export declare function writeDrillRecord(s3: S3Client, config: BackupConfig, record: DrillRecord, abortSignal?: AbortSignal): Promise<string>;
+/**
+ * CLI 的 `drill --record`:写不进去只警告,演练本身的结论和退出码不变。
+ * 常见原因是给演练配的是只读 key;这种情况下 exporter 的演练时间会一直不动,告警规则会发现。
+ */
+export declare function recordDrillOutcome(config: BackupConfig, report: DrillReport | null, s3?: S3Client, timeoutMs?: number): Promise<void>;
+/** 读桶里的演练记录。结构不对就报错:exporter 宁可报 up=0,也不报一个读错的结论。 */
+export declare function parseDrillRecord(json: string, file: string): DrillRecord;

package/dist/drill-record.js

@@ -0,0 +1,126 @@
+import { randomBytes } from "node:crypto";
+import { PutObjectCommand } from "@aws-sdk/client-s3";
+import { requireBucket } from "./config.js";
+import { isTimeout, withDeadline } from "./deadline.js";
+import { log } from "./log.js";
+import { targetClient } from "./snapshots.js";
+import { TOOL_VERSION } from "./version.js";
+/**
+ * `drill --record` 写进桶里的演练结论。`exporter` 只读桶,演练结果不落桶它就无从得知。
+ *
+ * 每次演练写一个新对象,从不覆盖:
+ *   <prefix>/<project>.drills/<结束时间>-<passed|failed>-<8 位随机>.json
+ * 为什么不是一份可覆盖的"最近一次":两次演练重叠时,先结束的那次可能后写进桶,把更新的
+ * "没通过"盖成更早的"通过",失败告警就没了;先读后写也挡不住(不是原子的),而各家
+ * S3 兼容存储对条件写入的支持参差不齐。每次一个对象,"哪次最新"留给读的一方按名字判断,
+ * 写入顺序就无所谓了。
+ *
+ * 为什么放在 <project>.drills/,和快照目录并排而不是在它里面:resolveSnapshot(包括已经发出去的
+ * 旧版本)按字典序取 <project>/ 下最大的目录当最新快照,里面多一个 drills/ 会被当成快照。
+ *
+ * 文件名本身带着结束时间和结论,exporter 列一下目录就知道最近一次和最近一次通过是哪个;
+ * 只为了恢复用时才读一次通过的那份。只记结论,不记检查明细:失败明细里可能有 pg_restore
+ * 回显的表数据,而桶的读权限可能比演练日志给得更宽(exporter 只要只读 key)。
+ */
+/** 版本规则同 manifest:缺省不接受,更高的版本明确报错让用户升级,不按旧结构硬读。 */
+export const DRILL_RECORD_SCHEMA_VERSION = 1;
+/** 演练记录所在的"目录",带尾斜杠 */
+export function drillRecordsPrefix(config) {
+    const storage = requireBucket(config);
+    return `${storage.prefix}/${config.projectName}.drills/`;
+}
+/**
+ * 时间戳写法同快照目录名(把 ISO 里的 : 和 . 换成 -),定长,按名字排序就是按时间排序。
+ * 末尾的随机段:两次演练在同一毫秒结束、结论又相同时,只靠时间和结论会撞名、后写的覆盖先写的
+ */
+const RECORD_NAME = /^(\d{4}-\d{2}-\d{2})T(\d{2})-(\d{2})-(\d{2})-(\d{3})Z-(passed|failed)-[0-9a-f]{8}\.json$/;
+export function drillRecordName(record, unique = randomBytes(4).toString("hex")) {
+    return `${record.finishedAt.replace(/[:.]/g, "-")}-${record.pass ? "passed" : "failed"}-${unique}.json`;
+}
+/** 从文件名读出结束时间(毫秒)和结论;不是记录文件的名字返回 null */
+export function parseDrillRecordName(name) {
+    const m = RECORD_NAME.exec(name);
+    if (!m)
+        return null;
+    const finishedAt = Date.parse(`${m[1]}T${m[2]}:${m[3]}:${m[4]}.${m[5]}Z`);
+    return Number.isFinite(finishedAt) ? { finishedAt, pass: m[6] === "passed" } : null;
+}
+/** report 为 null = 演练中途抛错、没有报告;这也算一次没通过的演练。 */
+export function buildDrillRecord(report, finishedAt) {
+    return {
+        schemaVersion: DRILL_RECORD_SCHEMA_VERSION,
+        tool: "backupdrill-cli",
+        toolVersion: TOOL_VERSION,
+        finishedAt: finishedAt.toISOString(),
+        pass: report?.pass === true,
+        ...(report ? { snapshot: report.snapshot } : {}),
+        ...(report?.pass ? { restoreSeconds: report.restoreSeconds } : {}),
+    };
+}
+/** 写一个新对象,返回它的 key。不读桶:只需要写权限 */
+export async function writeDrillRecord(s3, config, record, abortSignal) {
+    const key = drillRecordsPrefix(config) + drillRecordName(record);
+    await s3.send(new PutObjectCommand({
+        Bucket: requireBucket(config).bucket,
+        Key: key,
+        Body: JSON.stringify(record, null, 2),
+        ContentType: "application/json",
+    }), { abortSignal });
+    return key;
+}
+/** 记一笔最多等这么久:桶接了请求却不回应时,演练不能卡在这里,后面还要退出、发监控 ping */
+const RECORD_TIMEOUT_MS = 30_000;
+/**
+ * CLI 的 `drill --record`:写不进去只警告,演练本身的结论和退出码不变。
+ * 常见原因是给演练配的是只读 key;这种情况下 exporter 的演练时间会一直不动,告警规则会发现。
+ */
+export async function recordDrillOutcome(config, report, s3 = targetClient(config), timeoutMs = RECORD_TIMEOUT_MS) {
+    const record = buildDrillRecord(report, new Date());
+    try {
+        const key = await withDeadline((signal) => writeDrillRecord(s3, config, record, signal), timeoutMs);
+        log.ok(`Drill result recorded → s3://${requireBucket(config).bucket}/${key}`);
+    }
+    catch (error) {
+        const reason = isTimeout(error)
+            ? `no answer from the bucket within ${timeoutMs / 1000}s`
+            : error.message;
+        log.warn(`Could not record the drill result (${reason}). ` +
+            `--record needs write access to the bucket. The drill's own result is unchanged.`);
+    }
+}
+function malformed(file, what) {
+    throw new Error(`${file} is malformed: ${what}`);
+}
+/** 读桶里的演练记录。结构不对就报错:exporter 宁可报 up=0,也不报一个读错的结论。 */
+export function parseDrillRecord(json, file) {
+    let parsed;
+    try {
+        parsed = JSON.parse(json);
+    }
+    catch (error) {
+        throw new Error(`${file} is not valid JSON: ${error.message}`);
+    }
+    if (typeof parsed !== "object" || parsed === null || Array.isArray(parsed)) {
+        malformed(file, "root is not an object");
+    }
+    const r = parsed;
+    if (typeof r.schemaVersion !== "number" || !Number.isInteger(r.schemaVersion) || r.schemaVersion < 1) {
+        malformed(file, `schemaVersion ${JSON.stringify(r.schemaVersion)}`);
+    }
+    if (r.schemaVersion > DRILL_RECORD_SCHEMA_VERSION) {
+        throw new Error(`${file} uses schema version ${r.schemaVersion}, but this BackupDrill understands up to ` +
+            `${DRILL_RECORD_SCHEMA_VERSION}. Upgrade the CLI (npm i -g backupdrill@latest) to read it.`);
+    }
+    if (typeof r.finishedAt !== "string" || !Number.isFinite(Date.parse(r.finishedAt))) {
+        malformed(file, "finishedAt");
+    }
+    if (typeof r.pass !== "boolean")
+        malformed(file, "pass");
+    if (r.snapshot !== undefined && typeof r.snapshot !== "string")
+        malformed(file, "snapshot");
+    if (r.restoreSeconds !== undefined &&
+        (typeof r.restoreSeconds !== "number" || !Number.isFinite(r.restoreSeconds) || r.restoreSeconds < 0)) {
+        malformed(file, "restoreSeconds");
+    }
+    return r;
+}

package/dist/drill.js

@@ -1,4 +1,5 @@
 import { execFile, spawn } from "node:child_process";
+import { randomBytes } from "node:crypto";
 import { mkdtemp, rm } from "node:fs/promises";
 import { tmpdir } from "node:os";
 import { join } from "node:path";
@@ -10,6 +11,7 @@ import { SANDBOX_MANAGED_ERROR, classifyBlocks, finalizePass, installExtensions,
 import { connectPg } from "./supabase-ca.js";
 import { localPgRestoreMajor, restoreToolBreaksOnTarget, sandboxMajorFor } from "./pgbin.js";
 import { log } from "./log.js";
+import { onInterrupt, stopOnInterrupt, throwIfInterrupted } from "./interrupt.js";
 const execFileAsync = promisify(execFile);
 // Storage 校验默认抽样上限(避免每次演练重下全部文件产生大量 egress);超限如实记录
 const STORAGE_SAMPLE_CAP = 100;
@@ -133,6 +135,26 @@ export function sandboxDataVolumeTarget(major) {
     // PG18 将镜像的 VOLUME 上移一层;挂错路径会额外生成无法按名字回收的匿名卷。
     return major >= 18 ? "/var/lib/postgresql" : "/var/lib/postgresql/data";
 }
+/**
+ * 沙箱镜像先单独拉好。让 docker run 自己拉的话,拉取和建容器在同一个命令里:中断落在拉取中途,
+ * 停掉命令也说不准守护进程之后还会不会把容器建出来(交叉审查)。拆开之后,慢的那一步(第一次拉镜像
+ * 可能好几分钟)什么都不建,中断时直接停掉;docker run 只剩建容器和启动。
+ */
+async function ensureImage(image) {
+    const present = await execFileAsync("docker", ["image", "inspect", "--format", "{{.Id}}", image]).then(() => true, () => false);
+    if (present)
+        return;
+    throwIfInterrupted();
+    log.step(`Pulling ${image}…`);
+    const pull = execFileAsync("docker", ["pull", "--quiet", image]);
+    const forget = pull.child ? stopOnInterrupt(pull.child) : () => { };
+    try {
+        await pull;
+    }
+    finally {
+        forget();
+    }
+}
 async function startEphemeralPostgres(image, major, resourceName) {
     const dataVolumeTarget = sandboxDataVolumeTarget(major);
     if (resourceName) {
@@ -144,12 +166,17 @@ async function startEphemeralPostgres(image, major, resourceName) {
     log.step(`Starting ephemeral ${image}…`);
     const resources = await dockerResources();
     const tuning = sandboxTuning(resources.memoryBytes, resources.cpus);
-    const { stdout } = await execFileAsync("docker", [
+    await ensureImage(image);
+    // 名字先定好:Ctrl+C 可能落在 docker run 还没返回 ID 的时候,按名字才删得掉
+    const name = resourceName ?? `backupdrill-drill-${randomBytes(6).toString("hex")}`;
+    throwIfInterrupted();
+    const started = execFileAsync("docker", [
         "run",
         "-d",
         "--rm",
-        ...(resourceName ? ["--name", resourceName, "--mount",
-            `type=volume,source=${resourceName},target=${dataVolumeTarget}`] : []),
+        "--name",
+        name,
+        ...(resourceName ? ["--mount", `type=volume,source=${resourceName},target=${dataVolumeTarget}`] : []),
         "-e",
         "POSTGRES_PASSWORD=drill",
         "-e",
@@ -161,10 +188,59 @@ async function startEphemeralPostgres(image, major, resourceName) {
         "postgres",
         ...tuning.settings.flatMap((kv) => ["-c", kv]),
     ]);
+    // 删容器只有一份:正常收尾(drillDump 的 finally)和中断收尾(interrupt.ts)等的是同一个 promise。
+    // 两边各发一次 docker rm 会撞上 "removal already in progress",后到的那个报删除失败,
+    // 进程又在真正删完之前就退出了(信号测试抓到的竞争)。
+    let volumeName = resourceName ?? null;
+    let runSettled = false;
+    started.then(() => (runSettled = true), () => (runSettled = true));
+    const removeByName = async () => {
+        try {
+            // -v 连匿名卷一起删:中断可能落在下面查卷名之前
+            await execFileAsync("docker", ["rm", "-fv", name], { timeout: 60_000 });
+        }
+        catch (error) {
+            if (!/No such container/i.test(String(error.stderr ?? error))) {
+                log.warn(`could not remove the drill sandbox; remove it with: docker rm -fv ${name}`);
+            }
+        }
+    };
+    let teardownRun = null;
+    const teardown = () => (teardownRun ??= (async () => {
+        if (!runSettled) {
+            // 中断落在 docker run 还没返回的时候。镜像已经在本地(ensureImage),它只剩建容器和启动,
+            // 等它落定再按名字删:中途停掉命令说不准守护进程会不会稍后把容器建出来(交叉审查复现)。
+            // Docker 迟迟不回话就不等了,把容器名告诉用户
+            let timer;
+            const settled = await Promise.race([
+                started.then(() => true, () => true),
+                new Promise((resolve) => (timer = setTimeout(() => resolve(false), 20_000))),
+            ]);
+            clearTimeout(timer);
+            if (!settled) {
+                started.child?.kill("SIGKILL");
+                log.warn(`docker run did not finish within 20s; if a container named ${name} shows up later, ` +
+                    `remove it with: docker rm -fv ${name}`);
+            }
+        }
+        await removeByName();
+        // 显式删卷(--rm 不可靠,见下);幂等,已经没了也不报错
+        if (volumeName)
+            await execFileAsync("docker", ["volume", "rm", "-f", volumeName]).catch(() => { });
+    })());
+    const forgetInterruptCleanup = onInterrupt(teardown);
+    let stdout;
+    try {
+        ({ stdout } = await started);
+    }
+    catch (error) {
+        forgetInterruptCleanup();
+        throw error;
+    }
     const containerId = stdout.trim();
     // postgres 镜像把数据目录声明成 VOLUME → 每次 run 都建一个匿名卷。生产实测 `--rm` **没有**把它
     // 带走(一次演练遗留 37.6 GB,三次失败把 59 GB 根盘灌满),所以记下卷名,销毁容器后显式删。
-    const volumeName = resourceName ?? await execFileAsync("docker", [
+    volumeName = resourceName ?? await execFileAsync("docker", [
         "inspect",
         "-f",
         '{{range .Mounts}}{{if eq .Type "volume"}}{{.Name}}{{end}}{{end}}',
@@ -198,7 +274,7 @@ async function startEphemeralPostgres(image, major, resourceName) {
                     "postgres",
                     "-q",
                 ]);
-                return { containerId, port, connString, volumeName, restoreJobs: tuning.restoreJobs };
+                return { containerId, port, connString, restoreJobs: tuning.restoreJobs, teardown, forgetInterruptCleanup };
             }
             catch {
                 await new Promise((r) => setTimeout(r, 500));
@@ -207,9 +283,8 @@ async function startEphemeralPostgres(image, major, resourceName) {
         throw new Error("ephemeral Postgres did not become ready within 30s");
     }
     catch (error) {
-        await execFileAsync("docker", ["rm", "-f", containerId]).catch(() => { });
-        if (volumeName)
-            await execFileAsync("docker", ["volume", "rm", "-f", volumeName]).catch(() => { });
+        await teardown();
+        forgetInterruptCleanup();
         throw error;
     }
 }
@@ -381,6 +456,8 @@ export function runAppCheck(command, connString, timeoutMs = APP_CHECK_TIMEOUT_M
             // 孙进程会在沙箱销毁后继续跑(xreview 抓出的资源泄漏)。
             detached: true,
         });
+        // 也因为自成一组,终端的 Ctrl+C 到不了它:中断时由我们按组停掉
+        stopOnInterrupt(proc, { group: true });
         let timedOut = false;
         const timer = setTimeout(() => {
             timedOut = true;
@@ -484,6 +561,7 @@ export async function drillDump(dumpPath, manifest, snapshot, preChecks = [], op
             });
         }
         await opts.supervision?.observe({ phase: "restore" });
+        throwIfInterrupted();
         const started = Date.now();
         log.step("Restoring into ephemeral Postgres…");
         let postData;
@@ -514,7 +592,11 @@ export async function drillDump(dumpPath, manifest, snapshot, preChecks = [], op
             detail: postData.failures.length > 0
                 ? `${postData.failures.length} failed: ${postData.failures[0]}`
                 : postData.supabaseSkipped > 0
-                    ? `user objects restored; ${postData.supabaseSkipped} Supabase-managed object(s) skipped (Supabase-managed tables such as auth.users do not exist in the drill sandbox)`
+                    ? // 说清跳过的是什么,以及它对真实恢复意味着什么:2026-10-22 实验里演练 PASS,
+                        // 恢复却死在被跳过的那条 FK → auth.users 上,因为新项目的 auth.users 是空的
+                        `user objects restored; ${postData.supabaseSkipped} Supabase-managed object(s) skipped ` +
+                            `(they reference tables such as auth.users, which the drill sandbox doesn't have; ` +
+                            `restoring into a new project needs the rows they point at to be there first)`
                     : "all post-data objects restored",
         });
         await opts.supervision?.observe({ phase: "verify" });
@@ -523,6 +605,7 @@ export async function drillDump(dumpPath, manifest, snapshot, preChecks = [], op
         // 语义层钩子:只在结构层全绿时才跑——结构已失败时再跑应用检查,只会把
         // 一次失败报成两层失败,误导排查方向。没配置则不跑、报告里不出现该行。
         if (appCheckCommand && checks.every((c) => c.pass)) {
+            throwIfInterrupted();
             log.step("Running app checks…");
             checks.push(await runAppCheck(appCheckCommand, pg.connString));
         }
@@ -559,11 +642,10 @@ export async function drillDump(dumpPath, manifest, snapshot, preChecks = [], op
         else {
             await observeCleanup(opts.supervision);
             log.step("Tearing down ephemeral Postgres…");
-            await execFileAsync("docker", ["rm", "-f", pg.containerId]).catch(() => { });
-            // 显式删匿名卷(--rm 不可靠,见 startEphemeralPostgres);幂等,已被 --rm 带走也不报错
-            if (pg.volumeName)
-                await execFileAsync("docker", ["volume", "rm", "-f", pg.volumeName]).catch(() => { });
+            await pg.teardown();
         }
+        // 删完了,或者按 --keep 有意留着:之后再收到信号都不该再动它
+        pg.forgetInterruptCleanup();
     }
 }
 /** 演练一份快照:从桶里下载→drillDump(含 Storage 校验)→报告。 */
@@ -576,6 +658,10 @@ export async function runDrill(config, opts = {}) {
     const manifest = parseManifest(await reader.readManifest());
     const workdir = opts.supervision?.workdir ?? await mkdtemp(join(tmpdir(), "backupdrill-"));
     const dumpPath = join(workdir, "dump.pgcustom");
+    // 下载的 dump 在这里;托管模式的目录归 supervisor 管
+    const forgetWorkdir = opts.supervision
+        ? () => { }
+        : onInterrupt(() => rm(workdir, { recursive: true, force: true }).catch(() => { }));
     try {
         await opts.supervision?.observe({ phase: "download", dumpBytes: manifest.dump.bytes });
         log.step("Downloading dump…");
@@ -623,5 +709,6 @@ export async function runDrill(config, opts = {}) {
     finally {
         if (!opts.supervision)
             await rm(workdir, { recursive: true, force: true }).catch(() => { });
+        forgetWorkdir();
     }
 }