backupdrill 0.1.1 → 0.1.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +27 -8
- package/dist/backup.js +22 -5
- package/dist/drill.d.ts +8 -0
- package/dist/drill.js +151 -23
- package/dist/estimate.d.ts +1 -1
- package/dist/estimate.js +11 -9
- package/dist/index.js +3 -2
- package/dist/lib.d.ts +1 -1
- package/dist/manifest.d.ts +12 -0
- package/dist/pgbin.d.ts +7 -0
- package/dist/pgbin.js +17 -0
- package/dist/restore.js +2 -4
- package/package.json +11 -1
package/README.md
CHANGED
|
@@ -9,7 +9,7 @@ Backups stream through the CLI on your machine (or your CI) straight to your buc
|
|
|
9
9
|
## Why
|
|
10
10
|
|
|
11
11
|
- Supabase's own database restore only brings back `storage.objects` **metadata**, not your actual Storage files.
|
|
12
|
-
- Pro-plan backups keep 7 days and can't be pulled into your own infrastructure; PITR is $100/
|
|
12
|
+
- Pro-plan backups keep 7 days and can't be pulled into your own infrastructure; PITR is $100/month per project.
|
|
13
13
|
- A `pg_dump` you've never restored is a guess.
|
|
14
14
|
|
|
15
15
|
This CLI gets your database into a bucket **you** control, with a manifest you can check.
|
|
@@ -45,9 +45,26 @@ export BACKUPDRILL_S3_SECRET_ACCESS_KEY="…"
|
|
|
45
45
|
backupdrill backup
|
|
46
46
|
```
|
|
47
47
|
|
|
48
|
+
All recognized variables:
|
|
49
|
+
|
|
50
|
+
| Variable | Purpose | Default |
|
|
51
|
+
|---|---|---|
|
|
52
|
+
| `BACKUPDRILL_DATABASE_URL` | Source connection string (falls back to `DATABASE_URL`) | — (required) |
|
|
53
|
+
| `BACKUPDRILL_PROJECT_NAME` | Readable name used in object keys and the manifest | `supabase` |
|
|
54
|
+
| `BACKUPDRILL_SCHEMAS` | Comma-separated schemas to dump | `public` |
|
|
55
|
+
| `BACKUPDRILL_S3_ENDPOINT` | S3-compatible endpoint (R2/B2/self-hosted; omit for AWS S3) | — |
|
|
56
|
+
| `BACKUPDRILL_S3_REGION` | Destination bucket region | `auto` |
|
|
57
|
+
| `BACKUPDRILL_S3_BUCKET` | Destination bucket | — (required) |
|
|
58
|
+
| `BACKUPDRILL_S3_ACCESS_KEY_ID`, `BACKUPDRILL_S3_SECRET_ACCESS_KEY` | Destination credentials | — (required) |
|
|
59
|
+
| `BACKUPDRILL_S3_PREFIX` | Key prefix inside the bucket | `backupdrill` |
|
|
60
|
+
| `BACKUPDRILL_S3_FORCE_PATH_STYLE` | `true`/`false` — path-style addressing (R2/MinIO) | `true` when an endpoint is set |
|
|
61
|
+
| `BACKUPDRILL_SUPABASE_STORAGE_*` | Storage-file source (endpoint/region/keys/buckets) — see [Backing up Storage files](#backing-up-storage-files-too) | — |
|
|
62
|
+
| `BACKUPDRILL_PG_DUMP` | Path to the `pg_dump` binary to use | `pg_dump` on `PATH` |
|
|
63
|
+
| `BACKUPDRILL_PG_RESTORE` | Path to `pg_restore` (used by `drill` and `restore`) | next to `BACKUPDRILL_PG_DUMP`, else `pg_restore` on `PATH` |
|
|
64
|
+
|
|
48
65
|
### Config file
|
|
49
66
|
|
|
50
|
-
Copy `backupdrill.config.example.json` → `backupdrill.config.json` (it's gitignored) and fill it in. Keep secrets in env vars even when using a file.
|
|
67
|
+
Copy `backupdrill.config.example.json` → `backupdrill.config.json` (it's gitignored) and fill it in. Keep secrets in env vars even when using a file. To also back up Storage files, start from `backupdrill.config.with-storage.example.json` instead — only copy the `supabaseStorage` block once you actually have S3 access keys for your project's Storage.
|
|
51
68
|
|
|
52
69
|
```bash
|
|
53
70
|
backupdrill backup --config backupdrill.config.json
|
|
@@ -75,7 +92,7 @@ Supabase's own database restore only brings back `storage.objects` **metadata**
|
|
|
75
92
|
|
|
76
93
|
1. Dashboard → Project → **Storage → Settings → S3 Access Keys** → enable the S3 connection and generate an **Access Key ID + Secret Access Key** (these bypass RLS; keep them server-side).
|
|
77
94
|
2. Copy the **endpoint** and **region** from the same page.
|
|
78
|
-
3. Provide them (env shown; config-file keys are `supabaseStorage.*`):
|
|
95
|
+
3. Provide them (env shown; config-file keys are `supabaseStorage.*` — see `backupdrill.config.with-storage.example.json`):
|
|
79
96
|
|
|
80
97
|
```bash
|
|
81
98
|
export BACKUPDRILL_SUPABASE_STORAGE_ENDPOINT="https://<ref>.storage.supabase.co/storage/v1/s3"
|
|
@@ -89,19 +106,19 @@ With those set, `backupdrill backup` also copies every Storage file into your bu
|
|
|
89
106
|
|
|
90
107
|
## Egress & cost — read this before scheduling daily backups
|
|
91
108
|
|
|
92
|
-
A backup pulls data **out** of Supabase, which counts as **egress on your Supabase bill**. Verified rates (2026-07-
|
|
109
|
+
A backup pulls data **out** of Supabase, which counts as **egress on your Supabase bill**. Verified rates (2026-07-10, [egress docs](https://supabase.com/docs/guides/platform/manage-your-usage/egress) + [pricing](https://supabase.com/pricing)):
|
|
93
110
|
|
|
94
111
|
- Backup traffic (pg_dump + first-time file downloads) is **uncached egress: $0.09/GB** above your plan's included allowance.
|
|
95
112
|
- Included uncached egress: **Free 5 GB · Pro 250 GB · Team 250 GB** per month.
|
|
96
113
|
- **Spend Cap is ON by default (Pro/Team).** If a backup pushes you past the quota, Supabase **throttles your whole project's egress** for the rest of the month and the backup fails — turn Spend Cap OFF to be billed the overage instead.
|
|
97
|
-
- **Free
|
|
114
|
+
- **Free never bills overage.** Exceeding the included 5 GB/month starts Supabase's Fair Use process: a notification to your billing email, then a grace period, then egress restrictions if usage stays over (the restriction clears at the next billing cycle or when you upgrade). Sustained large backups are not viable on Free.
|
|
98
115
|
|
|
99
116
|
Rough cost (Pro, Spend Cap OFF, backups = your only egress):
|
|
100
117
|
|
|
101
118
|
| DB + Storage | Weekly | Daily |
|
|
102
119
|
|---|---|---|
|
|
103
|
-
| 10 GB | $0/
|
|
104
|
-
| 50 GB | ~$0/
|
|
120
|
+
| 10 GB | $0/month (within 250 GB) | ~$4.50/month |
|
|
121
|
+
| 50 GB | ~$0/month | **~$112/month** ⚠ |
|
|
105
122
|
|
|
106
123
|
Cost scales with **size × frequency**. Run `backupdrill estimate` to project it for *your* project, and prefer weekly (or a smaller scope) once your data is large. The compressed dump is smaller than raw DB size, so real DB egress is below the `estimate` upper bound; Storage egress is exact.
|
|
107
124
|
|
|
@@ -120,13 +137,15 @@ backupdrill estimate --plan pro # measures your DB + Storage, prints the proje
|
|
|
120
137
|
|
|
121
138
|
## Restoring & drilling
|
|
122
139
|
|
|
123
|
-
Prove a backup is restorable, without touching anything real — `drill` restores the latest snapshot into a throwaway Postgres
|
|
140
|
+
Prove a backup is restorable, without touching anything real — `drill` restores the latest snapshot into a throwaway Postgres and checks it. It needs **Docker** (for the throwaway Postgres) and **`pg_restore`** (same major-version rule as `pg_dump`; set `BACKUPDRILL_PG_RESTORE` if it's not on your `PATH`):
|
|
124
141
|
|
|
125
142
|
```bash
|
|
126
143
|
backupdrill drill # latest snapshot; add --snapshot <timestamp> to pick one
|
|
127
144
|
backupdrill drill --verify-all-files # also checksum every Storage file, not just a sample
|
|
128
145
|
```
|
|
129
146
|
|
|
147
|
+
Backups record your installed Postgres extensions in the manifest, and the drill pre-installs them in the sandbox before restoring — including switching to a pgvector-enabled sandbox image when your database uses `vector` columns.
|
|
148
|
+
|
|
130
149
|
When you actually need to recover, `restore` puts the database back into a target you name and pulls the Storage files down to a local folder:
|
|
131
150
|
|
|
132
151
|
```bash
|
package/dist/backup.js
CHANGED
|
@@ -50,11 +50,26 @@ async function inspectDatabase(databaseUrl, schemas) {
|
|
|
50
50
|
try {
|
|
51
51
|
const version = await client.query("show server_version");
|
|
52
52
|
const versionNum = await client.query("show server_version_num");
|
|
53
|
-
// 只统计将被 dump 的 schema,manifest
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
53
|
+
// 只统计将被 dump 的 schema,manifest 才与转储内容一致。
|
|
54
|
+
// 口径 = pg_class relkind in ('r','p','m'):普通表 + 分区父表 + 物化视图,
|
|
55
|
+
// 与 pg_dump 实际转储的关系集合一致,且必须与 drill 校验端(verifyRestored)
|
|
56
|
+
// 用同一查询——两端口径不同曾导致含 matview 的库演练必然误报 FAIL。
|
|
57
|
+
// 行数估算仍取 n_live_tup(planner 统计);分区父表本身不存行,天然为 0。
|
|
58
|
+
const tables = await client.query(`select n.nspname as schema, c.relname as name,
|
|
59
|
+
coalesce(s.n_live_tup, 0) as estimated_rows
|
|
60
|
+
from pg_class c
|
|
61
|
+
join pg_namespace n on n.oid = c.relnamespace
|
|
62
|
+
left join pg_stat_all_tables s on s.relid = c.oid
|
|
63
|
+
where c.relkind in ('r', 'p', 'm')
|
|
64
|
+
and n.nspname = any($1::text[])
|
|
65
|
+
order by 1, 2`, [schemas]);
|
|
66
|
+
// 非内置扩展清单(排除必装的 plpgsql):--schema 转储不含 CREATE EXTENSION,
|
|
67
|
+
// 演练沙箱要靠它在恢复前把扩展类型(如 vector)装回来
|
|
68
|
+
const extensions = await client.query(`select e.extname as name, e.extversion as version, n.nspname as schema
|
|
69
|
+
from pg_extension e
|
|
70
|
+
join pg_namespace n on n.oid = e.extnamespace
|
|
71
|
+
where e.extname <> 'plpgsql'
|
|
72
|
+
order by e.extname`);
|
|
58
73
|
return {
|
|
59
74
|
serverVersion: version.rows[0].server_version,
|
|
60
75
|
serverVersionNum: Number(versionNum.rows[0].server_version_num),
|
|
@@ -63,6 +78,7 @@ async function inspectDatabase(databaseUrl, schemas) {
|
|
|
63
78
|
name: r.name,
|
|
64
79
|
estimatedRows: Number(r.estimated_rows),
|
|
65
80
|
})),
|
|
81
|
+
extensions: extensions.rows,
|
|
66
82
|
};
|
|
67
83
|
}
|
|
68
84
|
finally {
|
|
@@ -184,6 +200,7 @@ export async function runBackup(config) {
|
|
|
184
200
|
tableCount: db.tables.length,
|
|
185
201
|
estimatedRowTotal: db.tables.reduce((n, t) => n + t.estimatedRows, 0),
|
|
186
202
|
tables: db.tables,
|
|
203
|
+
extensions: db.extensions,
|
|
187
204
|
},
|
|
188
205
|
dump: { key: dumpKey, format: "custom", bytes, sha256 },
|
|
189
206
|
storage: storageResult,
|
package/dist/drill.d.ts
CHANGED
|
@@ -1,5 +1,13 @@
|
|
|
1
1
|
import type { BackupConfig } from "./config.js";
|
|
2
2
|
import type { Manifest } from "./manifest.js";
|
|
3
|
+
/**
|
|
4
|
+
* 无放回随机抽样:部分 Fisher-Yates,只洗出前 n 位(O(n) 次交换,不必全 shuffle)。
|
|
5
|
+
* 此前恒取 manifest 顺序的前 CAP 个 → 超过 CAP 的项目里排序靠后的文件永远不被校验;
|
|
6
|
+
* 随机抽样让每次演练覆盖不同子集,损坏的文件长期必然被抽中。
|
|
7
|
+
* 随机源用 Math.random 足够:演练是一次性进程,无可复现性要求——校验失败的文件名
|
|
8
|
+
* 会原样列进报告,定位问题不依赖重放同一批样本。
|
|
9
|
+
*/
|
|
10
|
+
export declare function sampleStorageFiles<T>(files: T[], n: number): T[];
|
|
3
11
|
export interface DrillCheck {
|
|
4
12
|
name: string;
|
|
5
13
|
pass: boolean;
|
package/dist/drill.js
CHANGED
|
@@ -5,12 +5,39 @@ import { join } from "node:path";
|
|
|
5
5
|
import { promisify } from "node:util";
|
|
6
6
|
import { Client } from "pg";
|
|
7
7
|
import { targetClient, resolveSnapshot, getObjectText, downloadToFile, hashObject, } from "./snapshots.js";
|
|
8
|
+
import { resolvePgRestoreBin } from "./pgbin.js";
|
|
8
9
|
import { log } from "./log.js";
|
|
9
10
|
const execFileAsync = promisify(execFile);
|
|
10
11
|
// Storage 校验默认抽样上限(避免每次演练重下全部文件产生大量 egress);超限如实记录
|
|
11
12
|
const STORAGE_SAMPLE_CAP = 100;
|
|
12
|
-
|
|
13
|
-
|
|
13
|
+
/**
|
|
14
|
+
* 无放回随机抽样:部分 Fisher-Yates,只洗出前 n 位(O(n) 次交换,不必全 shuffle)。
|
|
15
|
+
* 此前恒取 manifest 顺序的前 CAP 个 → 超过 CAP 的项目里排序靠后的文件永远不被校验;
|
|
16
|
+
* 随机抽样让每次演练覆盖不同子集,损坏的文件长期必然被抽中。
|
|
17
|
+
* 随机源用 Math.random 足够:演练是一次性进程,无可复现性要求——校验失败的文件名
|
|
18
|
+
* 会原样列进报告,定位问题不依赖重放同一批样本。
|
|
19
|
+
*/
|
|
20
|
+
export function sampleStorageFiles(files, n) {
|
|
21
|
+
if (files.length <= n)
|
|
22
|
+
return files;
|
|
23
|
+
const pool = [...files];
|
|
24
|
+
for (let i = 0; i < n; i++) {
|
|
25
|
+
const j = i + Math.floor(Math.random() * (pool.length - i));
|
|
26
|
+
[pool[i], pool[j]] = [pool[j], pool[i]];
|
|
27
|
+
}
|
|
28
|
+
return pool.slice(0, n);
|
|
29
|
+
}
|
|
30
|
+
/**
|
|
31
|
+
* 沙箱镜像选择:alpine 镜像只带 contrib 扩展,装不了 pgvector——含 vector 列的
|
|
32
|
+
* 备份在裸镜像里恢复必然硬崩("type vector does not exist")。manifest 记录了
|
|
33
|
+
* vector 时换用 pgvector 官方镜像(同 major),其余维持轻量 alpine。
|
|
34
|
+
*/
|
|
35
|
+
function sandboxImage(manifest) {
|
|
36
|
+
const major = parseInt(manifest.database.serverVersion, 10);
|
|
37
|
+
const hasVector = (manifest.database.extensions ?? []).some((e) => e.name === "vector");
|
|
38
|
+
return hasVector ? `pgvector/pgvector:pg${major}` : `postgres:${major}-alpine`;
|
|
39
|
+
}
|
|
40
|
+
async function startEphemeralPostgres(image) {
|
|
14
41
|
log.step(`Starting ephemeral ${image}…`);
|
|
15
42
|
const { stdout } = await execFileAsync("docker", [
|
|
16
43
|
"run",
|
|
@@ -64,6 +91,39 @@ async function startEphemeralPostgres(major) {
|
|
|
64
91
|
throw error;
|
|
65
92
|
}
|
|
66
93
|
}
|
|
94
|
+
function quoteIdent(name) {
|
|
95
|
+
return `"${name.replace(/"/g, '""')}"`;
|
|
96
|
+
}
|
|
97
|
+
/**
|
|
98
|
+
* 恢复前把 manifest 记录的扩展装进沙箱(--schema 转储不含 CREATE EXTENSION)。
|
|
99
|
+
* 必须装进与源库相同的 schema:dump 里的类型是限定名(如 extensions.vector)。
|
|
100
|
+
* 装不上的扩展如实收集返回,**不在这里失败**——Supabase 项目普遍装着沙箱镜像
|
|
101
|
+
* 没有的托管扩展(pg_graphql、supabase_vault…),public 转储通常并不引用它们;
|
|
102
|
+
* 只有恢复真的因此失败时才升级为分类错误(见 drillDump)。
|
|
103
|
+
*/
|
|
104
|
+
async function installExtensions(connString, extensions) {
|
|
105
|
+
if (!extensions.length)
|
|
106
|
+
return [];
|
|
107
|
+
const client = new Client({ connectionString: connString });
|
|
108
|
+
await client.connect();
|
|
109
|
+
const unavailable = [];
|
|
110
|
+
try {
|
|
111
|
+
for (const ext of extensions) {
|
|
112
|
+
try {
|
|
113
|
+
await client.query(`create schema if not exists ${quoteIdent(ext.schema)}`);
|
|
114
|
+
// CASCADE 自动带上依赖扩展,免受 manifest 里的安装顺序摆布
|
|
115
|
+
await client.query(`create extension if not exists ${quoteIdent(ext.name)} schema ${quoteIdent(ext.schema)} cascade`);
|
|
116
|
+
}
|
|
117
|
+
catch {
|
|
118
|
+
unavailable.push(ext.name);
|
|
119
|
+
}
|
|
120
|
+
}
|
|
121
|
+
}
|
|
122
|
+
finally {
|
|
123
|
+
await client.end();
|
|
124
|
+
}
|
|
125
|
+
return unavailable;
|
|
126
|
+
}
|
|
67
127
|
function spawnPgRestore(bin, args) {
|
|
68
128
|
return new Promise((resolve, reject) => {
|
|
69
129
|
const proc = spawn(bin, args, { stdio: ["ignore", "ignore", "pipe"] });
|
|
@@ -73,6 +133,10 @@ function spawnPgRestore(bin, args) {
|
|
|
73
133
|
proc.on("close", (code) => resolve({ code, stderr }));
|
|
74
134
|
});
|
|
75
135
|
}
|
|
136
|
+
// 只有这些特征说明恢复失败源于沙箱缺扩展(限定类型/schema 解析不到、扩展装不上)。
|
|
137
|
+
// 真实 Supabase 项目的 manifest 几乎必然含沙箱装不上的托管扩展(pg_graphql 等),
|
|
138
|
+
// 若不按特征过滤,任何硬失败(归档损坏、OOM…)都会被误归因到沙箱。
|
|
139
|
+
const MISSING_EXTENSION_ERROR = /type "[^"]+" does not exist|schema "[^"]+" does not exist|extension "[^"]+" is not available/i;
|
|
76
140
|
// 演练沙箱是裸 Postgres,Supabase 托管的 schema/角色必然缺席。post-data 里
|
|
77
141
|
// 引用它们的失败是"环境预期",不是备份坏了;其余失败才是演练要抓的。
|
|
78
142
|
const SUPABASE_MANAGED_ERROR = /schema "(auth|storage|realtime|vault|extensions|graphql[a-z_]*)" does not exist|role "(authenticated|anon|service_role|supabase_[a-z_]+)" does not exist|\bauth\.uid\b|\bauth\.jwt\b/i;
|
|
@@ -116,10 +180,7 @@ export function postDataResult(code, stderr) {
|
|
|
116
180
|
* (policy TO authenticated、FK → auth.users)才归类为沙箱预期,记入报告而非失败。
|
|
117
181
|
*/
|
|
118
182
|
async function pgRestore(dumpPath, connString) {
|
|
119
|
-
const pgRestoreBin =
|
|
120
|
-
(process.env.BACKUPDRILL_PG_DUMP
|
|
121
|
-
? process.env.BACKUPDRILL_PG_DUMP.replace(/pg_dump$/, "pg_restore")
|
|
122
|
-
: "pg_restore");
|
|
183
|
+
const pgRestoreBin = resolvePgRestoreBin();
|
|
123
184
|
const common = ["--no-owner", "--no-privileges", "--dbname", connString, dumpPath];
|
|
124
185
|
const first = await spawnPgRestore(pgRestoreBin, [
|
|
125
186
|
"--clean",
|
|
@@ -141,17 +202,39 @@ async function verifyRestored(connString, manifest) {
|
|
|
141
202
|
await client.connect();
|
|
142
203
|
try {
|
|
143
204
|
const schemas = manifest.database.schemas;
|
|
144
|
-
|
|
145
|
-
|
|
205
|
+
// 口径必须与 manifest 统计端(backup.ts inspectDatabase)完全一致:
|
|
206
|
+
// pg_class relkind in ('r','p','m')。此前这里用 pg_tables(不含 matview),
|
|
207
|
+
// manifest 用 pg_stat_user_tables(含 matview)——任何有 matview 的库
|
|
208
|
+
// 演练必然误报"表数不符 + 缺表"。
|
|
209
|
+
const tablesRes = await client.query(`select n.nspname as schema, c.relname as name,
|
|
210
|
+
c.relkind as kind, c.relispopulated as populated
|
|
211
|
+
from pg_class c
|
|
212
|
+
join pg_namespace n on n.oid = c.relnamespace
|
|
213
|
+
where c.relkind in ('r', 'p', 'm')
|
|
214
|
+
and n.nspname = any($1::text[])
|
|
215
|
+
order by 1, 2`, [schemas]);
|
|
146
216
|
const restoredTables = tablesRes.rows;
|
|
147
217
|
// 逐表精确计数(演练要证明数据真回来了,估算值不够)
|
|
148
218
|
const counts = new Map();
|
|
219
|
+
const unpopulatedMatviews = new Set();
|
|
149
220
|
let rowTotal = 0;
|
|
150
221
|
for (const t of restoredTables) {
|
|
222
|
+
const key = `${t.schema}.${t.name}`;
|
|
223
|
+
// 未填充的 matview 无法 count(*)(Postgres 直接报错)。它要么备份时就没数据
|
|
224
|
+
// (dump 里本来就没有 REFRESH),要么 REFRESH 在 post-data 阶段失败——后者已由
|
|
225
|
+
// post-data 检查如实记录,这里按 0 行处理即可。
|
|
226
|
+
if (t.kind === "m" && !t.populated) {
|
|
227
|
+
counts.set(key, 0);
|
|
228
|
+
unpopulatedMatviews.add(key);
|
|
229
|
+
continue;
|
|
230
|
+
}
|
|
151
231
|
const c = await client.query(`select count(*)::bigint as n from "${t.schema}"."${t.name}"`);
|
|
152
232
|
const n = Number(c.rows[0].n);
|
|
153
|
-
counts.set(
|
|
154
|
-
|
|
233
|
+
counts.set(key, n);
|
|
234
|
+
// 分区父表的 count(*) 是子分区行的总和,子分区又各自计一次;
|
|
235
|
+
// rowTotal 只累计叶子关系,避免翻倍。
|
|
236
|
+
if (t.kind !== "p")
|
|
237
|
+
rowTotal += n;
|
|
155
238
|
}
|
|
156
239
|
const checks = [];
|
|
157
240
|
// 1. 表数量一致
|
|
@@ -170,8 +253,13 @@ async function verifyRestored(connString, manifest) {
|
|
|
170
253
|
? "all manifest tables present"
|
|
171
254
|
: `missing: ${missing.map((t) => `${t.schema}.${t.name}`).join(", ")}`,
|
|
172
255
|
});
|
|
173
|
-
// 3. 备份时有数据的表,还原后不能是空的(最阴险的"备了半个库"故障)
|
|
174
|
-
|
|
256
|
+
// 3. 备份时有数据的表,还原后不能是空的(最阴险的"备了半个库"故障)。
|
|
257
|
+
// 未填充的 matview 豁免:REFRESH 失败已由 post-data 检查裁决(用户对象失败
|
|
258
|
+
// = 演练失败,Supabase 托管依赖 = 预期跳过),这里再报一次会把环境预期
|
|
259
|
+
// 误标成备份损坏。
|
|
260
|
+
const emptied = manifest.database.tables.filter((t) => t.estimatedRows > 0 &&
|
|
261
|
+
!unpopulatedMatviews.has(`${t.schema}.${t.name}`) &&
|
|
262
|
+
(counts.get(`${t.schema}.${t.name}`) ?? 0) === 0);
|
|
175
263
|
checks.push({
|
|
176
264
|
name: "populated tables came back",
|
|
177
265
|
pass: emptied.length === 0,
|
|
@@ -193,7 +281,7 @@ async function verifyRestored(connString, manifest) {
|
|
|
193
281
|
*/
|
|
194
282
|
async function verifyStorage(s3, config, manifest, verifyAll) {
|
|
195
283
|
const files = manifest.storage.files;
|
|
196
|
-
const sample = verifyAll ? files : files
|
|
284
|
+
const sample = verifyAll ? files : sampleStorageFiles(files, STORAGE_SAMPLE_CAP);
|
|
197
285
|
// 备份写入时的键:<snapshot base>/storage/<bucket>/<key>,base 从 dump.key 反推
|
|
198
286
|
const base = manifest.dump.key.replace(/\/dump\.pgcustom$/, "");
|
|
199
287
|
const mismatches = [];
|
|
@@ -208,15 +296,16 @@ async function verifyStorage(s3, config, manifest, verifyAll) {
|
|
|
208
296
|
mismatches.push(`${f.bucket}/${f.key} (unreadable)`);
|
|
209
297
|
}
|
|
210
298
|
}
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
299
|
+
// 抽样时如实说明这是随机样本(N of M);全量(--verify-all-files 或文件数 ≤ 上限)说 all
|
|
300
|
+
const scope = sample.length === files.length
|
|
301
|
+
? `all ${files.length} files`
|
|
302
|
+
: `random sample of ${sample.length} of ${files.length} files`;
|
|
214
303
|
return {
|
|
215
304
|
name: "storage file integrity",
|
|
216
305
|
pass: mismatches.length === 0,
|
|
217
306
|
detail: mismatches.length === 0
|
|
218
|
-
? `${scope}
|
|
219
|
-
: `${mismatches.length}
|
|
307
|
+
? `${scope} match their manifest checksums`
|
|
308
|
+
: `${mismatches.length} failed in ${scope}: ${mismatches.slice(0, 5).join(", ")}`,
|
|
220
309
|
};
|
|
221
310
|
}
|
|
222
311
|
// ── 主流程 ──────────────────────────────────────────────────────────
|
|
@@ -226,12 +315,39 @@ async function verifyStorage(s3, config, manifest, verifyAll) {
|
|
|
226
315
|
*/
|
|
227
316
|
export async function drillDump(dumpPath, manifest, snapshot, preChecks = []) {
|
|
228
317
|
const checks = [...preChecks];
|
|
229
|
-
const
|
|
230
|
-
const pg = await startEphemeralPostgres(major);
|
|
318
|
+
const pg = await startEphemeralPostgres(sandboxImage(manifest));
|
|
231
319
|
try {
|
|
320
|
+
// 旧 manifest(≤0.1.1)没有 extensions 字段 → 不装任何扩展,行为与从前一致
|
|
321
|
+
const extensions = manifest.database.extensions ?? [];
|
|
322
|
+
const unavailable = await installExtensions(pg.connString, extensions);
|
|
323
|
+
if (extensions.length) {
|
|
324
|
+
// 装不上 ≠ 演练失败:扩展本体不在 public 转储里,没丢任何已备份的数据;
|
|
325
|
+
// 若恢复真的需要它,下面的 pgRestore 会失败并给出分类错误
|
|
326
|
+
checks.push({
|
|
327
|
+
name: "sandbox extensions",
|
|
328
|
+
pass: true,
|
|
329
|
+
detail: unavailable.length
|
|
330
|
+
? `installed ${extensions.length - unavailable.length}/${extensions.length}; ` +
|
|
331
|
+
`sandbox image cannot install: ${unavailable.join(", ")}`
|
|
332
|
+
: `installed ${extensions.map((e) => e.name).join(", ")}`,
|
|
333
|
+
});
|
|
334
|
+
}
|
|
232
335
|
const started = Date.now();
|
|
233
336
|
log.step("Restoring into ephemeral Postgres…");
|
|
234
|
-
|
|
337
|
+
let postData;
|
|
338
|
+
try {
|
|
339
|
+
postData = await pgRestore(dumpPath, pg.connString);
|
|
340
|
+
}
|
|
341
|
+
catch (error) {
|
|
342
|
+
// 沙箱装不上扩展 + 错误特征命中"缺类型/schema/扩展" → 假设式归因到沙箱环境
|
|
343
|
+
// (不对备份健康下断言);其余失败与扩展无关,必须原样抛出
|
|
344
|
+
const message = error.message;
|
|
345
|
+
if (unavailable.length && MISSING_EXTENSION_ERROR.test(message)) {
|
|
346
|
+
throw new Error(`restore failed; likely because the drill sandbox cannot install ` +
|
|
347
|
+
`extension(s) ${unavailable.join(", ")}. Original error: ${message}`);
|
|
348
|
+
}
|
|
349
|
+
throw error;
|
|
350
|
+
}
|
|
235
351
|
const restoreSeconds = Math.round((Date.now() - started) / 100) / 10;
|
|
236
352
|
checks.push({ name: "pg_restore", pass: true, detail: `completed in ${restoreSeconds}s` });
|
|
237
353
|
// 用户自己的 post-data 对象(索引/约束/触发器)必须全部恢复;Supabase 托管对象
|
|
@@ -273,15 +389,27 @@ export async function runDrill(config, opts = {}) {
|
|
|
273
389
|
try {
|
|
274
390
|
log.step("Downloading dump…");
|
|
275
391
|
const { sha256: sha } = await downloadToFile(s3, config.storage.bucket, manifest.dump.key, dumpPath);
|
|
392
|
+
const integrityPass = sha === manifest.dump.sha256;
|
|
276
393
|
const preChecks = [
|
|
277
394
|
{
|
|
278
395
|
name: "archive integrity",
|
|
279
|
-
pass:
|
|
280
|
-
detail:
|
|
396
|
+
pass: integrityPass,
|
|
397
|
+
detail: integrityPass
|
|
281
398
|
? `sha256 matches (${sha.slice(0, 12)}…)`
|
|
282
399
|
: `sha256 MISMATCH — got ${sha.slice(0, 12)}…, manifest ${manifest.dump.sha256.slice(0, 12)}…`,
|
|
283
400
|
},
|
|
284
401
|
];
|
|
402
|
+
// 归档校验和不符 = 演练已经 FAIL,继续恢复只会产生误导性失败;短路如实报告
|
|
403
|
+
if (!integrityPass) {
|
|
404
|
+
return {
|
|
405
|
+
snapshot,
|
|
406
|
+
pass: false,
|
|
407
|
+
checks: preChecks,
|
|
408
|
+
restoreSeconds: 0,
|
|
409
|
+
restoredTableCount: 0,
|
|
410
|
+
restoredRowTotal: 0,
|
|
411
|
+
};
|
|
412
|
+
}
|
|
285
413
|
// Storage 文件完整性(备份含 Storage 时才校验)
|
|
286
414
|
if (manifest.storage && manifest.storage.files.length) {
|
|
287
415
|
log.step("Verifying Storage files…");
|
package/dist/estimate.d.ts
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
import type { BackupConfig } from "./config.js";
|
|
2
2
|
/**
|
|
3
|
-
* Supabase egress 定价(2026-07-
|
|
3
|
+
* Supabase egress 定价(2026-07-10 经核实,来源见 README「Egress & cost」)。
|
|
4
4
|
* 单价 = 超出计划包含额度后的每 GB 价;included = 各计划每月包含的 egress。
|
|
5
5
|
* 备份 = 把数据拉出 Supabase = egress,计入用户账单。这是 PRD §4.3 的上线否决项:
|
|
6
6
|
* 让用户"我们帮你算过账"而不是收到意外账单。
|
package/dist/estimate.js
CHANGED
|
@@ -59,10 +59,10 @@ export async function runEstimate(config, pricing) {
|
|
|
59
59
|
console.error(` Plan: ${pricing.planLabel} — ${pricing.includedGb} GB egress included, ` +
|
|
60
60
|
`then $${pricing.pricePerGb}/GB`);
|
|
61
61
|
console.error("");
|
|
62
|
-
console.error(" Frequency Runs/
|
|
63
|
-
console.error(" ─────────
|
|
62
|
+
console.error(" Frequency Runs/month Monthly egress Overage Est. cost/month");
|
|
63
|
+
console.error(" ───────── ────────── ────────────── ─────── ───────────────");
|
|
64
64
|
for (const r of projection.rows) {
|
|
65
|
-
console.error(` ${r.frequency.padEnd(9)} ${String(r.runsPerMonth).padStart(
|
|
65
|
+
console.error(` ${r.frequency.padEnd(9)} ${String(r.runsPerMonth).padStart(10)} ` +
|
|
66
66
|
`${(r.monthlyEgressGb + " GB").padStart(14)} ` +
|
|
67
67
|
`${(r.overageGb + " GB").padStart(7)} ` +
|
|
68
68
|
`$${r.monthlyCost.toFixed(2)}`);
|
|
@@ -76,11 +76,13 @@ export async function runEstimate(config, pricing) {
|
|
|
76
76
|
const weekly = projection.rows.find((r) => r.frequency === "weekly");
|
|
77
77
|
console.error("");
|
|
78
78
|
if (pricing.hardCap) {
|
|
79
|
-
// Free
|
|
79
|
+
// Free:超额不计费,走 Fair Use 通知 → 宽限期 → 限制;持续超额的大备份在 Free 跑不通
|
|
80
80
|
if (daily && daily.overageGb > 0) {
|
|
81
|
-
log.warn(`${pricing.planLabel} egress is
|
|
82
|
-
|
|
83
|
-
`
|
|
81
|
+
log.warn(`${pricing.planLabel} plan egress is never billed, but exceeding the included ` +
|
|
82
|
+
`${pricing.includedGb} GB/month starts Supabase's Fair Use process: a ` +
|
|
83
|
+
`notification, then a grace period, then egress restrictions. Daily backups ` +
|
|
84
|
+
`(~${daily.monthlyEgressGb} GB/month) would keep you permanently over the ` +
|
|
85
|
+
`line — sustained large backups are not viable on Free. ` +
|
|
84
86
|
`Use weekly at most, or upgrade to Pro.`);
|
|
85
87
|
}
|
|
86
88
|
}
|
|
@@ -93,9 +95,9 @@ export async function runEstimate(config, pricing) {
|
|
|
93
95
|
"numbers above are what you'd pay.");
|
|
94
96
|
const worst = daily?.monthlyCost ?? 0;
|
|
95
97
|
if (worst >= SUBSCRIPTION_FLOOR_USD) {
|
|
96
|
-
log.warn(`At daily frequency the egress overage alone (~$${worst.toFixed(0)}/
|
|
98
|
+
log.warn(`At daily frequency the egress overage alone (~$${worst.toFixed(0)}/month) is ` +
|
|
97
99
|
`in the range of a paid backup subscription. Consider weekly ` +
|
|
98
|
-
`(~$${(weekly?.monthlyCost ?? 0).toFixed(0)}/
|
|
100
|
+
`(~$${(weekly?.monthlyCost ?? 0).toFixed(0)}/month overage) or a smaller scope.`);
|
|
99
101
|
}
|
|
100
102
|
}
|
|
101
103
|
return projection;
|
package/dist/index.js
CHANGED
|
@@ -7,9 +7,10 @@ import { runDrill } from "./drill.js";
|
|
|
7
7
|
import { runRestore } from "./restore.js";
|
|
8
8
|
import { log } from "./log.js";
|
|
9
9
|
import { TOOL_VERSION } from "./version.js";
|
|
10
|
-
// Supabase egress 定价表(2026-07-
|
|
10
|
+
// Supabase egress 定价表(2026-07-10 核实,来源见 README「Egress & cost」)。
|
|
11
11
|
// 备份流量 100% 未缓存,故用 uncached 单价 $0.09/GB;included 用 uncached 额度。
|
|
12
|
-
// Free
|
|
12
|
+
// Free 永不计费:超额走 Fair Use 流程(通知 → 宽限期 → 限制),持续大备份不可行;
|
|
13
|
+
// Pro/Team 由 Spend Cap 决定超额限流还是计费。
|
|
13
14
|
const EGRESS_PRICING = {
|
|
14
15
|
free: { planLabel: "Free", includedGb: 5, pricePerGb: 0.09, hardCap: true },
|
|
15
16
|
pro: { planLabel: "Pro", includedGb: 250, pricePerGb: 0.09, hardCap: false },
|
package/dist/lib.d.ts
CHANGED
|
@@ -5,7 +5,7 @@ export { runEstimate, project } from "./estimate.js";
|
|
|
5
5
|
export { loadConfig } from "./config.js";
|
|
6
6
|
export { targetClient, resolveSnapshot, getObjectText, downloadToFile, hashObject, } from "./snapshots.js";
|
|
7
7
|
export type { BackupConfig, StorageConfig, SupabaseStorageConfig, } from "./config.js";
|
|
8
|
-
export type { Manifest, StorageFile, TableStat } from "./manifest.js";
|
|
8
|
+
export type { ExtensionInfo, Manifest, StorageFile, TableStat } from "./manifest.js";
|
|
9
9
|
export type { DrillReport, DrillCheck } from "./drill.js";
|
|
10
10
|
export type { RestoreResult } from "./restore.js";
|
|
11
11
|
export type { EgressPricing, EgressProjection } from "./estimate.js";
|
package/dist/manifest.d.ts
CHANGED
|
@@ -8,6 +8,17 @@ export interface TableStat {
|
|
|
8
8
|
name: string;
|
|
9
9
|
estimatedRows: number;
|
|
10
10
|
}
|
|
11
|
+
/**
|
|
12
|
+
* 源库安装的非内置扩展(pg_extension,排除必装的 plpgsql)。
|
|
13
|
+
* --schema=public 的转储不含 CREATE EXTENSION,但表定义会引用扩展类型
|
|
14
|
+
* (如 extensions.vector)——沙箱恢复前必须按这份清单把扩展装进**同名 schema**,
|
|
15
|
+
* dump 里的限定名才解析得到(Supabase 惯例装在 "extensions" schema)。
|
|
16
|
+
*/
|
|
17
|
+
export interface ExtensionInfo {
|
|
18
|
+
name: string;
|
|
19
|
+
version: string;
|
|
20
|
+
schema: string;
|
|
21
|
+
}
|
|
11
22
|
export interface Manifest {
|
|
12
23
|
tool: "backupdrill-cli";
|
|
13
24
|
toolVersion: string;
|
|
@@ -20,6 +31,7 @@ export interface Manifest {
|
|
|
20
31
|
tableCount: number;
|
|
21
32
|
estimatedRowTotal: number;
|
|
22
33
|
tables: TableStat[];
|
|
34
|
+
extensions?: ExtensionInfo[];
|
|
23
35
|
};
|
|
24
36
|
dump: {
|
|
25
37
|
key: string;
|
package/dist/pgbin.d.ts
ADDED
package/dist/pgbin.js
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* pg_restore 二进制解析:显式 BACKUPDRILL_PG_RESTORE 最高优先;否则从
|
|
3
|
+
* BACKUPDRILL_PG_DUMP 同目录推导。推导未命中(结果仍含 pg_dump 或与原值相同,
|
|
4
|
+
* 如指向包装脚本/带版本后缀的路径)时回退到 PATH 上的 pg_restore——
|
|
5
|
+
* 绝不能把 pg_dump 当恢复器执行。
|
|
6
|
+
*/
|
|
7
|
+
export function resolvePgRestoreBin() {
|
|
8
|
+
if (process.env.BACKUPDRILL_PG_RESTORE)
|
|
9
|
+
return process.env.BACKUPDRILL_PG_RESTORE;
|
|
10
|
+
const pgDump = process.env.BACKUPDRILL_PG_DUMP;
|
|
11
|
+
if (pgDump) {
|
|
12
|
+
const derived = pgDump.replace(/pg_dump$/, "pg_restore");
|
|
13
|
+
if (derived !== pgDump && !derived.includes("pg_dump"))
|
|
14
|
+
return derived;
|
|
15
|
+
}
|
|
16
|
+
return "pg_restore";
|
|
17
|
+
}
|
package/dist/restore.js
CHANGED
|
@@ -6,12 +6,10 @@ import { join, dirname } from "node:path";
|
|
|
6
6
|
import { pipeline } from "node:stream/promises";
|
|
7
7
|
import { GetObjectCommand } from "@aws-sdk/client-s3";
|
|
8
8
|
import { targetClient, resolveSnapshot, getObjectText, downloadToFile, } from "./snapshots.js";
|
|
9
|
+
import { resolvePgRestoreBin } from "./pgbin.js";
|
|
9
10
|
import { log } from "./log.js";
|
|
10
11
|
function pgRestore(dumpPath, targetUrl) {
|
|
11
|
-
const bin =
|
|
12
|
-
(process.env.BACKUPDRILL_PG_DUMP
|
|
13
|
-
? process.env.BACKUPDRILL_PG_DUMP.replace(/pg_dump$/, "pg_restore")
|
|
14
|
-
: "pg_restore");
|
|
12
|
+
const bin = resolvePgRestoreBin();
|
|
15
13
|
return new Promise((resolve, reject) => {
|
|
16
14
|
const proc = spawn(bin, ["--clean", "--if-exists", "--no-owner", "--no-privileges", "--dbname", targetUrl, dumpPath], { stdio: ["ignore", "ignore", "pipe"] });
|
|
17
15
|
let stderr = "";
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "backupdrill",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.3",
|
|
4
4
|
"description": "Open-source CLI to back up a Supabase project — Postgres database and Storage files — to your own S3/R2/B2 bucket, with a checksummed manifest.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -33,18 +33,28 @@
|
|
|
33
33
|
},
|
|
34
34
|
"keywords": [
|
|
35
35
|
"supabase",
|
|
36
|
+
"supabase-backup",
|
|
36
37
|
"backup",
|
|
38
|
+
"database-backup",
|
|
37
39
|
"postgres",
|
|
40
|
+
"postgresql",
|
|
38
41
|
"pg_dump",
|
|
42
|
+
"restore",
|
|
39
43
|
"s3",
|
|
44
|
+
"cloudflare-r2",
|
|
45
|
+
"backblaze-b2",
|
|
40
46
|
"disaster-recovery"
|
|
41
47
|
],
|
|
42
48
|
"license": "MIT",
|
|
49
|
+
"author": "BackupDrill <support@backupdrill.com> (https://backupdrill.com)",
|
|
43
50
|
"homepage": "https://backupdrill.com",
|
|
44
51
|
"repository": {
|
|
45
52
|
"type": "git",
|
|
46
53
|
"url": "git+https://github.com/backupdrill/cli.git"
|
|
47
54
|
},
|
|
55
|
+
"bugs": {
|
|
56
|
+
"url": "https://github.com/backupdrill/cli/issues"
|
|
57
|
+
},
|
|
48
58
|
"dependencies": {
|
|
49
59
|
"@aws-sdk/client-s3": "^3.658.0",
|
|
50
60
|
"@aws-sdk/lib-storage": "^3.658.0",
|