backupdrill 0.1.1 → 0.1.2

package/README.md

@@ -45,9 +45,26 @@ export BACKUPDRILL_S3_SECRET_ACCESS_KEY="…"
 backupdrill backup
 ```
 
+All recognized variables:
+
+| Variable | Purpose | Default |
+|---|---|---|
+| `BACKUPDRILL_DATABASE_URL` | Source connection string (falls back to `DATABASE_URL`) | — (required) |
+| `BACKUPDRILL_PROJECT_NAME` | Readable name used in object keys and the manifest | `supabase` |
+| `BACKUPDRILL_SCHEMAS` | Comma-separated schemas to dump | `public` |
+| `BACKUPDRILL_S3_ENDPOINT` | S3-compatible endpoint (R2/B2/self-hosted; omit for AWS S3) | — |
+| `BACKUPDRILL_S3_REGION` | Destination bucket region | `auto` |
+| `BACKUPDRILL_S3_BUCKET` | Destination bucket | — (required) |
+| `BACKUPDRILL_S3_ACCESS_KEY_ID`, `BACKUPDRILL_S3_SECRET_ACCESS_KEY` | Destination credentials | — (required) |
+| `BACKUPDRILL_S3_PREFIX` | Key prefix inside the bucket | `backupdrill` |
+| `BACKUPDRILL_S3_FORCE_PATH_STYLE` | `true`/`false` — path-style addressing (R2/MinIO) | `true` when an endpoint is set |
+| `BACKUPDRILL_SUPABASE_STORAGE_*` | Storage-file source (endpoint/region/keys/buckets) — see [Backing up Storage files](#backing-up-storage-files-too) | — |
+| `BACKUPDRILL_PG_DUMP` | Path to the `pg_dump` binary to use | `pg_dump` on `PATH` |
+| `BACKUPDRILL_PG_RESTORE` | Path to `pg_restore` (used by `drill` and `restore`) | next to `BACKUPDRILL_PG_DUMP`, else `pg_restore` on `PATH` |
+
 ### Config file
 
-Copy `backupdrill.config.example.json` → `backupdrill.config.json` (it's gitignored) and fill it in. Keep secrets in env vars even when using a file.
+Copy `backupdrill.config.example.json` → `backupdrill.config.json` (it's gitignored) and fill it in. Keep secrets in env vars even when using a file. To also back up Storage files, start from `backupdrill.config.with-storage.example.json` instead — only copy the `supabaseStorage` block once you actually have S3 access keys for your project's Storage.
 
 ```bash
 backupdrill backup --config backupdrill.config.json
@@ -75,7 +92,7 @@ Supabase's own database restore only brings back `storage.objects` **metadata**
 
 1. Dashboard → Project → **Storage → Settings → S3 Access Keys** → enable the S3 connection and generate an **Access Key ID + Secret Access Key** (these bypass RLS; keep them server-side).
 2. Copy the **endpoint** and **region** from the same page.
-3. Provide them (env shown; config-file keys are `supabaseStorage.*`):
+3. Provide them (env shown; config-file keys are `supabaseStorage.*` — see `backupdrill.config.with-storage.example.json`):
 
 ```bash
 export BACKUPDRILL_SUPABASE_STORAGE_ENDPOINT="https://<ref>.storage.supabase.co/storage/v1/s3"
@@ -120,13 +137,15 @@ backupdrill estimate --plan pro   # measures your DB + Storage, prints the proje
 
 ## Restoring & drilling
 
-Prove a backup is restorable, without touching anything real — `drill` restores the latest snapshot into a throwaway Postgres (needs Docker) and checks it:
+Prove a backup is restorable, without touching anything real — `drill` restores the latest snapshot into a throwaway Postgres and checks it. It needs **Docker** (for the throwaway Postgres) and **`pg_restore`** (same major-version rule as `pg_dump`; set `BACKUPDRILL_PG_RESTORE` if it's not on your `PATH`):
 
 ```bash
 backupdrill drill                    # latest snapshot; add --snapshot <timestamp> to pick one
 backupdrill drill --verify-all-files # also checksum every Storage file, not just a sample
 ```
 
+Backups record your installed Postgres extensions in the manifest, and the drill pre-installs them in the sandbox before restoring — including switching to a pgvector-enabled sandbox image when your database uses `vector` columns.
+
 When you actually need to recover, `restore` puts the database back into a target you name and pulls the Storage files down to a local folder:
 
 ```bash

package/dist/backup.js

@@ -50,11 +50,26 @@ async function inspectDatabase(databaseUrl, schemas) {
     try {
         const version = await client.query("show server_version");
         const versionNum = await client.query("show server_version_num");
-        // 只统计将被 dump 的 schema,manifest 才与转储内容一致
-        const tables = await client.query(`select schemaname as schema, relname as name, n_live_tup as estimated_rows
-         from pg_stat_user_tables
-        where schemaname = any($1::text[])
-        order by schemaname, relname`, [schemas]);
+        // 只统计将被 dump 的 schema,manifest 才与转储内容一致。
+        // 口径 = pg_class relkind in ('r','p','m'):普通表 + 分区父表 + 物化视图,
+        // 与 pg_dump 实际转储的关系集合一致,且必须与 drill 校验端(verifyRestored)
+        // 用同一查询——两端口径不同曾导致含 matview 的库演练必然误报 FAIL。
+        // 行数估算仍取 n_live_tup(planner 统计);分区父表本身不存行,天然为 0。
+        const tables = await client.query(`select n.nspname as schema, c.relname as name,
+              coalesce(s.n_live_tup, 0) as estimated_rows
+         from pg_class c
+         join pg_namespace n on n.oid = c.relnamespace
+         left join pg_stat_all_tables s on s.relid = c.oid
+        where c.relkind in ('r', 'p', 'm')
+          and n.nspname = any($1::text[])
+        order by 1, 2`, [schemas]);
+        // 非内置扩展清单(排除必装的 plpgsql):--schema 转储不含 CREATE EXTENSION,
+        // 演练沙箱要靠它在恢复前把扩展类型(如 vector)装回来
+        const extensions = await client.query(`select e.extname as name, e.extversion as version, n.nspname as schema
+         from pg_extension e
+         join pg_namespace n on n.oid = e.extnamespace
+        where e.extname <> 'plpgsql'
+        order by e.extname`);
         return {
             serverVersion: version.rows[0].server_version,
             serverVersionNum: Number(versionNum.rows[0].server_version_num),
@@ -63,6 +78,7 @@ async function inspectDatabase(databaseUrl, schemas) {
                 name: r.name,
                 estimatedRows: Number(r.estimated_rows),
             })),
+            extensions: extensions.rows,
         };
     }
     finally {
@@ -184,6 +200,7 @@ export async function runBackup(config) {
             tableCount: db.tables.length,
             estimatedRowTotal: db.tables.reduce((n, t) => n + t.estimatedRows, 0),
             tables: db.tables,
+            extensions: db.extensions,
         },
         dump: { key: dumpKey, format: "custom", bytes, sha256 },
         storage: storageResult,

package/dist/drill.js

@@ -5,12 +5,22 @@ import { join } from "node:path";
 import { promisify } from "node:util";
 import { Client } from "pg";
 import { targetClient, resolveSnapshot, getObjectText, downloadToFile, hashObject, } from "./snapshots.js";
+import { resolvePgRestoreBin } from "./pgbin.js";
 import { log } from "./log.js";
 const execFileAsync = promisify(execFile);
 // Storage 校验默认抽样上限(避免每次演练重下全部文件产生大量 egress);超限如实记录
 const STORAGE_SAMPLE_CAP = 100;
-async function startEphemeralPostgres(major) {
-    const image = `postgres:${major}-alpine`;
+/**
+ * 沙箱镜像选择:alpine 镜像只带 contrib 扩展,装不了 pgvector——含 vector 列的
+ * 备份在裸镜像里恢复必然硬崩("type vector does not exist")。manifest 记录了
+ * vector 时换用 pgvector 官方镜像(同 major),其余维持轻量 alpine。
+ */
+function sandboxImage(manifest) {
+    const major = parseInt(manifest.database.serverVersion, 10);
+    const hasVector = (manifest.database.extensions ?? []).some((e) => e.name === "vector");
+    return hasVector ? `pgvector/pgvector:pg${major}` : `postgres:${major}-alpine`;
+}
+async function startEphemeralPostgres(image) {
     log.step(`Starting ephemeral ${image}…`);
     const { stdout } = await execFileAsync("docker", [
         "run",
@@ -64,6 +74,39 @@ async function startEphemeralPostgres(major) {
         throw error;
     }
 }
+function quoteIdent(name) {
+    return `"${name.replace(/"/g, '""')}"`;
+}
+/**
+ * 恢复前把 manifest 记录的扩展装进沙箱(--schema 转储不含 CREATE EXTENSION)。
+ * 必须装进与源库相同的 schema:dump 里的类型是限定名(如 extensions.vector)。
+ * 装不上的扩展如实收集返回,**不在这里失败**——Supabase 项目普遍装着沙箱镜像
+ * 没有的托管扩展(pg_graphql、supabase_vault…),public 转储通常并不引用它们;
+ * 只有恢复真的因此失败时才升级为分类错误(见 drillDump)。
+ */
+async function installExtensions(connString, extensions) {
+    if (!extensions.length)
+        return [];
+    const client = new Client({ connectionString: connString });
+    await client.connect();
+    const unavailable = [];
+    try {
+        for (const ext of extensions) {
+            try {
+                await client.query(`create schema if not exists ${quoteIdent(ext.schema)}`);
+                // CASCADE 自动带上依赖扩展,免受 manifest 里的安装顺序摆布
+                await client.query(`create extension if not exists ${quoteIdent(ext.name)} schema ${quoteIdent(ext.schema)} cascade`);
+            }
+            catch {
+                unavailable.push(ext.name);
+            }
+        }
+    }
+    finally {
+        await client.end();
+    }
+    return unavailable;
+}
 function spawnPgRestore(bin, args) {
     return new Promise((resolve, reject) => {
         const proc = spawn(bin, args, { stdio: ["ignore", "ignore", "pipe"] });
@@ -73,6 +116,10 @@ function spawnPgRestore(bin, args) {
         proc.on("close", (code) => resolve({ code, stderr }));
     });
 }
+// 只有这些特征说明恢复失败源于沙箱缺扩展(限定类型/schema 解析不到、扩展装不上)。
+// 真实 Supabase 项目的 manifest 几乎必然含沙箱装不上的托管扩展(pg_graphql 等),
+// 若不按特征过滤,任何硬失败(归档损坏、OOM…)都会被误归因到沙箱。
+const MISSING_EXTENSION_ERROR = /type "[^"]+" does not exist|schema "[^"]+" does not exist|extension "[^"]+" is not available/i;
 // 演练沙箱是裸 Postgres,Supabase 托管的 schema/角色必然缺席。post-data 里
 // 引用它们的失败是"环境预期",不是备份坏了;其余失败才是演练要抓的。
 const SUPABASE_MANAGED_ERROR = /schema "(auth|storage|realtime|vault|extensions|graphql[a-z_]*)" does not exist|role "(authenticated|anon|service_role|supabase_[a-z_]+)" does not exist|\bauth\.uid\b|\bauth\.jwt\b/i;
@@ -116,10 +163,7 @@ export function postDataResult(code, stderr) {
  *    (policy TO authenticated、FK → auth.users)才归类为沙箱预期,记入报告而非失败。
  */
 async function pgRestore(dumpPath, connString) {
-    const pgRestoreBin = process.env.BACKUPDRILL_PG_RESTORE ||
-        (process.env.BACKUPDRILL_PG_DUMP
-            ? process.env.BACKUPDRILL_PG_DUMP.replace(/pg_dump$/, "pg_restore")
-            : "pg_restore");
+    const pgRestoreBin = resolvePgRestoreBin();
     const common = ["--no-owner", "--no-privileges", "--dbname", connString, dumpPath];
     const first = await spawnPgRestore(pgRestoreBin, [
         "--clean",
@@ -141,17 +185,39 @@ async function verifyRestored(connString, manifest) {
     await client.connect();
     try {
         const schemas = manifest.database.schemas;
-        const tablesRes = await client.query(`select schemaname as schema, tablename as name
-         from pg_tables where schemaname = any($1::text[])`, [schemas]);
+        // 口径必须与 manifest 统计端(backup.ts inspectDatabase)完全一致:
+        // pg_class relkind in ('r','p','m')。此前这里用 pg_tables(不含 matview),
+        // manifest 用 pg_stat_user_tables(含 matview)——任何有 matview 的库
+        // 演练必然误报"表数不符 + 缺表"。
+        const tablesRes = await client.query(`select n.nspname as schema, c.relname as name,
+              c.relkind as kind, c.relispopulated as populated
+         from pg_class c
+         join pg_namespace n on n.oid = c.relnamespace
+        where c.relkind in ('r', 'p', 'm')
+          and n.nspname = any($1::text[])
+        order by 1, 2`, [schemas]);
         const restoredTables = tablesRes.rows;
         // 逐表精确计数(演练要证明数据真回来了,估算值不够)
         const counts = new Map();
+        const unpopulatedMatviews = new Set();
         let rowTotal = 0;
         for (const t of restoredTables) {
+            const key = `${t.schema}.${t.name}`;
+            // 未填充的 matview 无法 count(*)(Postgres 直接报错)。它要么备份时就没数据
+            // (dump 里本来就没有 REFRESH),要么 REFRESH 在 post-data 阶段失败——后者已由
+            // post-data 检查如实记录,这里按 0 行处理即可。
+            if (t.kind === "m" && !t.populated) {
+                counts.set(key, 0);
+                unpopulatedMatviews.add(key);
+                continue;
+            }
             const c = await client.query(`select count(*)::bigint as n from "${t.schema}"."${t.name}"`);
             const n = Number(c.rows[0].n);
-            counts.set(`${t.schema}.${t.name}`, n);
-            rowTotal += n;
+            counts.set(key, n);
+            // 分区父表的 count(*) 是子分区行的总和,子分区又各自计一次;
+            // rowTotal 只累计叶子关系,避免翻倍。
+            if (t.kind !== "p")
+                rowTotal += n;
         }
         const checks = [];
         // 1. 表数量一致
@@ -170,8 +236,13 @@ async function verifyRestored(connString, manifest) {
                 ? "all manifest tables present"
                 : `missing: ${missing.map((t) => `${t.schema}.${t.name}`).join(", ")}`,
         });
-        // 3. 备份时有数据的表,还原后不能是空的(最阴险的"备了半个库"故障)
-        const emptied = manifest.database.tables.filter((t) => t.estimatedRows > 0 && (counts.get(`${t.schema}.${t.name}`) ?? 0) === 0);
+        // 3. 备份时有数据的表,还原后不能是空的(最阴险的"备了半个库"故障)。
+        // 未填充的 matview 豁免:REFRESH 失败已由 post-data 检查裁决(用户对象失败
+        // = 演练失败,Supabase 托管依赖 = 预期跳过),这里再报一次会把环境预期
+        // 误标成备份损坏。
+        const emptied = manifest.database.tables.filter((t) => t.estimatedRows > 0 &&
+            !unpopulatedMatviews.has(`${t.schema}.${t.name}`) &&
+            (counts.get(`${t.schema}.${t.name}`) ?? 0) === 0);
         checks.push({
             name: "populated tables came back",
             pass: emptied.length === 0,
@@ -226,12 +297,39 @@ async function verifyStorage(s3, config, manifest, verifyAll) {
  */
 export async function drillDump(dumpPath, manifest, snapshot, preChecks = []) {
     const checks = [...preChecks];
-    const major = parseInt(manifest.database.serverVersion, 10);
-    const pg = await startEphemeralPostgres(major);
+    const pg = await startEphemeralPostgres(sandboxImage(manifest));
     try {
+        // 旧 manifest(≤0.1.1)没有 extensions 字段 → 不装任何扩展,行为与从前一致
+        const extensions = manifest.database.extensions ?? [];
+        const unavailable = await installExtensions(pg.connString, extensions);
+        if (extensions.length) {
+            // 装不上 ≠ 演练失败:扩展本体不在 public 转储里,没丢任何已备份的数据;
+            // 若恢复真的需要它,下面的 pgRestore 会失败并给出分类错误
+            checks.push({
+                name: "sandbox extensions",
+                pass: true,
+                detail: unavailable.length
+                    ? `installed ${extensions.length - unavailable.length}/${extensions.length}; ` +
+                        `sandbox image cannot install: ${unavailable.join(", ")}`
+                    : `installed ${extensions.map((e) => e.name).join(", ")}`,
+            });
+        }
         const started = Date.now();
         log.step("Restoring into ephemeral Postgres…");
-        const postData = await pgRestore(dumpPath, pg.connString);
+        let postData;
+        try {
+            postData = await pgRestore(dumpPath, pg.connString);
+        }
+        catch (error) {
+            // 沙箱装不上扩展 + 错误特征命中"缺类型/schema/扩展" → 假设式归因到沙箱环境
+            // (不对备份健康下断言);其余失败与扩展无关,必须原样抛出
+            const message = error.message;
+            if (unavailable.length && MISSING_EXTENSION_ERROR.test(message)) {
+                throw new Error(`restore failed; likely because the drill sandbox cannot install ` +
+                    `extension(s) ${unavailable.join(", ")}. Original error: ${message}`);
+            }
+            throw error;
+        }
         const restoreSeconds = Math.round((Date.now() - started) / 100) / 10;
         checks.push({ name: "pg_restore", pass: true, detail: `completed in ${restoreSeconds}s` });
         // 用户自己的 post-data 对象(索引/约束/触发器)必须全部恢复;Supabase 托管对象
@@ -273,15 +371,27 @@ export async function runDrill(config, opts = {}) {
     try {
         log.step("Downloading dump…");
         const { sha256: sha } = await downloadToFile(s3, config.storage.bucket, manifest.dump.key, dumpPath);
+        const integrityPass = sha === manifest.dump.sha256;
         const preChecks = [
             {
                 name: "archive integrity",
-                pass: sha === manifest.dump.sha256,
-                detail: sha === manifest.dump.sha256
+                pass: integrityPass,
+                detail: integrityPass
                     ? `sha256 matches (${sha.slice(0, 12)}…)`
                     : `sha256 MISMATCH — got ${sha.slice(0, 12)}…, manifest ${manifest.dump.sha256.slice(0, 12)}…`,
             },
         ];
+        // 归档校验和不符 = 演练已经 FAIL,继续恢复只会产生误导性失败;短路如实报告
+        if (!integrityPass) {
+            return {
+                snapshot,
+                pass: false,
+                checks: preChecks,
+                restoreSeconds: 0,
+                restoredTableCount: 0,
+                restoredRowTotal: 0,
+            };
+        }
         // Storage 文件完整性(备份含 Storage 时才校验)
         if (manifest.storage && manifest.storage.files.length) {
             log.step("Verifying Storage files…");

package/dist/lib.d.ts

@@ -5,7 +5,7 @@ export { runEstimate, project } from "./estimate.js";
 export { loadConfig } from "./config.js";
 export { targetClient, resolveSnapshot, getObjectText, downloadToFile, hashObject, } from "./snapshots.js";
 export type { BackupConfig, StorageConfig, SupabaseStorageConfig, } from "./config.js";
-export type { Manifest, StorageFile, TableStat } from "./manifest.js";
+export type { ExtensionInfo, Manifest, StorageFile, TableStat } from "./manifest.js";
 export type { DrillReport, DrillCheck } from "./drill.js";
 export type { RestoreResult } from "./restore.js";
 export type { EgressPricing, EgressProjection } from "./estimate.js";

package/dist/manifest.d.ts

@@ -8,6 +8,17 @@ export interface TableStat {
     name: string;
     estimatedRows: number;
 }
+/**
+ * 源库安装的非内置扩展(pg_extension,排除必装的 plpgsql)。
+ * --schema=public 的转储不含 CREATE EXTENSION,但表定义会引用扩展类型
+ * (如 extensions.vector)——沙箱恢复前必须按这份清单把扩展装进**同名 schema**,
+ * dump 里的限定名才解析得到(Supabase 惯例装在 "extensions" schema)。
+ */
+export interface ExtensionInfo {
+    name: string;
+    version: string;
+    schema: string;
+}
 export interface Manifest {
     tool: "backupdrill-cli";
     toolVersion: string;
@@ -20,6 +31,7 @@ export interface Manifest {
         tableCount: number;
         estimatedRowTotal: number;
         tables: TableStat[];
+        extensions?: ExtensionInfo[];
     };
     dump: {
         key: string;

package/dist/pgbin.d.ts

@@ -0,0 +1,7 @@
+/**
+ * pg_restore 二进制解析:显式 BACKUPDRILL_PG_RESTORE 最高优先;否则从
+ * BACKUPDRILL_PG_DUMP 同目录推导。推导未命中(结果仍含 pg_dump 或与原值相同,
+ * 如指向包装脚本/带版本后缀的路径)时回退到 PATH 上的 pg_restore——
+ * 绝不能把 pg_dump 当恢复器执行。
+ */
+export declare function resolvePgRestoreBin(): string;

package/dist/pgbin.js

@@ -0,0 +1,17 @@
+/**
+ * pg_restore 二进制解析:显式 BACKUPDRILL_PG_RESTORE 最高优先;否则从
+ * BACKUPDRILL_PG_DUMP 同目录推导。推导未命中(结果仍含 pg_dump 或与原值相同,
+ * 如指向包装脚本/带版本后缀的路径)时回退到 PATH 上的 pg_restore——
+ * 绝不能把 pg_dump 当恢复器执行。
+ */
+export function resolvePgRestoreBin() {
+    if (process.env.BACKUPDRILL_PG_RESTORE)
+        return process.env.BACKUPDRILL_PG_RESTORE;
+    const pgDump = process.env.BACKUPDRILL_PG_DUMP;
+    if (pgDump) {
+        const derived = pgDump.replace(/pg_dump$/, "pg_restore");
+        if (derived !== pgDump && !derived.includes("pg_dump"))
+            return derived;
+    }
+    return "pg_restore";
+}

package/dist/restore.js

@@ -6,12 +6,10 @@ import { join, dirname } from "node:path";
 import { pipeline } from "node:stream/promises";
 import { GetObjectCommand } from "@aws-sdk/client-s3";
 import { targetClient, resolveSnapshot, getObjectText, downloadToFile, } from "./snapshots.js";
+import { resolvePgRestoreBin } from "./pgbin.js";
 import { log } from "./log.js";
 function pgRestore(dumpPath, targetUrl) {
-    const bin = process.env.BACKUPDRILL_PG_RESTORE ||
-        (process.env.BACKUPDRILL_PG_DUMP
-            ? process.env.BACKUPDRILL_PG_DUMP.replace(/pg_dump$/, "pg_restore")
-            : "pg_restore");
+    const bin = resolvePgRestoreBin();
     return new Promise((resolve, reject) => {
         const proc = spawn(bin, ["--clean", "--if-exists", "--no-owner", "--no-privileges", "--dbname", targetUrl, dumpPath], { stdio: ["ignore", "ignore", "pipe"] });
         let stderr = "";

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "backupdrill",
-  "version": "0.1.1",
+  "version": "0.1.2",
   "description": "Open-source CLI to back up a Supabase project — Postgres database and Storage files — to your own S3/R2/B2 bucket, with a checksummed manifest.",
   "type": "module",
   "bin": {
@@ -33,18 +33,28 @@
   },
   "keywords": [
     "supabase",
+    "supabase-backup",
     "backup",
+    "database-backup",
     "postgres",
+    "postgresql",
     "pg_dump",
+    "restore",
     "s3",
+    "cloudflare-r2",
+    "backblaze-b2",
     "disaster-recovery"
   ],
   "license": "MIT",
+  "author": "BackupDrill <support@backupdrill.com> (https://backupdrill.com)",
   "homepage": "https://backupdrill.com",
   "repository": {
     "type": "git",
     "url": "git+https://github.com/backupdrill/cli.git"
   },
+  "bugs": {
+    "url": "https://github.com/backupdrill/cli/issues"
+  },
   "dependencies": {
     "@aws-sdk/client-s3": "^3.658.0",
     "@aws-sdk/lib-storage": "^3.658.0",