backupdrill 2.3.0 → 2.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +33 -0
- package/dist/backup.d.ts +5 -1
- package/dist/backup.js +28 -58
- package/dist/config.d.ts +18 -2
- package/dist/config.js +29 -1
- package/dist/drill.d.ts +3 -0
- package/dist/drill.js +19 -12
- package/dist/index.js +30 -6
- package/dist/lib.d.ts +4 -2
- package/dist/lib.js +3 -1
- package/dist/ping.d.ts +31 -0
- package/dist/ping.js +217 -0
- package/dist/restore.d.ts +3 -0
- package/dist/restore.js +15 -11
- package/dist/snapshot-store.d.ts +54 -0
- package/dist/snapshot-store.js +76 -0
- package/dist/snapshots.d.ts +1 -1
- package/dist/snapshots.js +11 -8
- package/package.json +10 -2
- package/plugin/LICENSE +21 -0
- package/plugin/README.md +79 -0
- package/plugin/skills/supabase-backup/SKILL.md +137 -0
- package/plugin/skills/supabase-backup/assets/scheduled-backup.yml +119 -0
package/README.md
CHANGED
|
@@ -42,6 +42,16 @@ The [`plugin/`](plugin/) folder is a plugin for Claude Code and Cursor. It teach
|
|
|
42
42
|
|
|
43
43
|
Details are in [plugin/README.md](plugin/README.md).
|
|
44
44
|
|
|
45
|
+
## Use it from Pi
|
|
46
|
+
|
|
47
|
+
This npm package is also a Pi package. Installing it gives Pi the same skill the Claude Code plugin uses:
|
|
48
|
+
|
|
49
|
+
```sh
|
|
50
|
+
pi install npm:backupdrill
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
Start a new Pi session and ask it to back up your Supabase project or run a restore drill. The skill runs the CLI through `npx`, so Node.js 20 or newer has to be on your `PATH`. Everything else it needs, and where your keys go, is in [plugin/README.md](plugin/README.md).
|
|
54
|
+
|
|
45
55
|
## Usage
|
|
46
56
|
|
|
47
57
|
Configure via **env vars**, a **config file**, or **flags** (precedence: flags > env > file).
|
|
@@ -76,6 +86,7 @@ All recognized variables:
|
|
|
76
86
|
| `BACKUPDRILL_SUPABASE_STORAGE_*` | Storage-file source (endpoint/region/keys/buckets) — see [Backing up Storage files](#backing-up-storage-files-too) | — |
|
|
77
87
|
| `BACKUPDRILL_PG_DUMP` | Path to the `pg_dump` binary to use | `pg_dump` on `PATH` |
|
|
78
88
|
| `BACKUPDRILL_PG_RESTORE` | Path to `pg_restore` (used by `drill` and `restore`) | next to `BACKUPDRILL_PG_DUMP`, else `pg_restore` on `PATH` |
|
|
89
|
+
| `BACKUPDRILL_PING_URL` | Healthchecks.io ping URL for `backup` and `drill` — see [Alerts with Healthchecks.io](#alerts-with-healthchecksio) | — |
|
|
79
90
|
|
|
80
91
|
### Config file
|
|
81
92
|
|
|
@@ -224,10 +235,32 @@ Want this running on a schedule without paying for a server? Run it in GitHub Ac
|
|
|
224
235
|
2. Add these repo secrets (Settings → Secrets and variables → Actions):
|
|
225
236
|
- **Required:** `BACKUPDRILL_DATABASE_URL`, `BACKUPDRILL_S3_BUCKET`, `BACKUPDRILL_S3_ACCESS_KEY_ID`, `BACKUPDRILL_S3_SECRET_ACCESS_KEY`
|
|
226
237
|
- **Optional (destination):** `BACKUPDRILL_S3_ENDPOINT` (for R2/B2), `BACKUPDRILL_S3_REGION`, `BACKUPDRILL_PROJECT_NAME`
|
|
238
|
+
- **Optional (alerts):** `BACKUPDRILL_PING_URL` — see [Alerts with Healthchecks.io](#alerts-with-healthchecksio)
|
|
227
239
|
- **Optional (Storage files):** set all four of `BACKUPDRILL_SUPABASE_STORAGE_ENDPOINT`, `BACKUPDRILL_SUPABASE_STORAGE_REGION`, `BACKUPDRILL_SUPABASE_STORAGE_ACCESS_KEY_ID`, `BACKUPDRILL_SUPABASE_STORAGE_SECRET_ACCESS_KEY` to also back up Storage files; leave them unset to back up the database only.
|
|
228
240
|
|
|
229
241
|
Before switching to a **daily** schedule, read the egress-cost section above — every run is egress on your Supabase bill.
|
|
230
242
|
|
|
243
|
+
## Alerts with Healthchecks.io
|
|
244
|
+
|
|
245
|
+
A scheduled backup can fail quietly, or not run at all: a cron entry that never fires, a runner that dies, a workflow GitHub disabled. `backup` and `drill` can report every run to a [Healthchecks.io](https://healthchecks.io) check (a self-hosted Healthchecks works the same), and Healthchecks alerts you when a run fails or a success ping doesn't arrive on time.
|
|
246
|
+
|
|
247
|
+
Create a check whose schedule and grace time match your job, copy its ping URL, then:
|
|
248
|
+
|
|
249
|
+
```bash
|
|
250
|
+
export BACKUPDRILL_PING_URL="https://hc-ping.com/<uuid>"
|
|
251
|
+
backupdrill backup
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
`--ping-url <url>` and a `"pingUrl"` key in the config file work too (same precedence as everything else). Each run sends a start ping, then either a success or a failure ping:
|
|
255
|
+
|
|
256
|
+
- `<url>/start` before it begins, so Healthchecks shows how long the run took and alerts if it runs past the grace time;
|
|
257
|
+
- `<url>` when it succeeds, with a one-line summary in the body (tables, Storage files and the snapshot name; for `drill`, rows restored and restore time);
|
|
258
|
+
- `<url>/fail` when it fails, with the error in the body. For a drill that ran but didn't pass, that's the failed checks. Before it's sent, the CLI filters out the connection strings, passwords, hostnames, IP addresses, project refs and bucket keys it can recognise, plus the parts of Postgres errors that usually quote row values or SQL (`DETAIL`, `CONTEXT`, `Command was`). It's a filter, not a guarantee: a sensitive value it doesn't recognise can still reach Healthchecks. The full error stays in the job's own output.
|
|
259
|
+
|
|
260
|
+
Both pings of a run carry the same `rid`, so Healthchecks pairs them even if two runs overlap. If you schedule both `backup` and `drill`, give each its own check: on a shared check, the daily backup's success pings would keep it up while a weekly drill that stopped running went unnoticed.
|
|
261
|
+
|
|
262
|
+
A ping never changes the result of a run. Each request times out after 10 seconds and is retried twice on network errors and 5xx responses; if it still fails, the CLI prints a warning and carries on, with the same exit code it would have had without a ping URL.
|
|
263
|
+
|
|
231
264
|
## Roadmap
|
|
232
265
|
|
|
233
266
|
- [x] Database backup → your bucket, with checksummed manifest
|
package/dist/backup.d.ts
CHANGED
|
@@ -1,5 +1,9 @@
|
|
|
1
1
|
import type { BackupConfig } from "./config.js";
|
|
2
|
+
import { type SnapshotStore } from "./snapshot-store.js";
|
|
2
3
|
import type { Manifest } from "./manifest.js";
|
|
3
4
|
import { parsePgDumpMajor } from "./pgbin.js";
|
|
4
5
|
export { parsePgDumpMajor };
|
|
5
|
-
export declare function runBackup(config: BackupConfig
|
|
6
|
+
export declare function runBackup(config: BackupConfig, opts?: {
|
|
7
|
+
/** 快照写到哪;省略 = config.storage 指的 S3 兼容桶 */
|
|
8
|
+
store?: SnapshotStore;
|
|
9
|
+
}): Promise<Manifest>;
|
package/dist/backup.js
CHANGED
|
@@ -2,18 +2,15 @@ import { spawn, execFile } from "node:child_process";
|
|
|
2
2
|
import { createHash } from "node:crypto";
|
|
3
3
|
import { Transform } from "node:stream";
|
|
4
4
|
import { promisify } from "node:util";
|
|
5
|
-
import {
|
|
6
|
-
import { Upload } from "@aws-sdk/lib-storage";
|
|
5
|
+
import { s3SnapshotStore } from "./snapshot-store.js";
|
|
7
6
|
import { MANIFEST_SCHEMA_VERSION } from "./manifest.js";
|
|
8
7
|
import { syncStorage } from "./storage.js";
|
|
9
8
|
import { inspectBucketAttrs } from "./storage-catalog.js";
|
|
10
|
-
import { renderRecoveryDoc } from "./recovery-doc.js";
|
|
11
9
|
import { projectRefOf, refFromStorageEndpoint, assertNoHostOverride } from "./restore-engine.js";
|
|
12
10
|
import { log } from "./log.js";
|
|
13
11
|
import { TOOL_VERSION } from "./version.js";
|
|
14
12
|
import { parsePgDumpMajor } from "./pgbin.js";
|
|
15
13
|
import { relationBytes } from "./estimate.js";
|
|
16
|
-
import { dumpPartSizeFor } from "./part-size.js";
|
|
17
14
|
// 解析函数搬到了 pgbin(演练与恢复都要读 pg_restore 版本,放这里会成环);保留旧出口
|
|
18
15
|
export { parsePgDumpMajor };
|
|
19
16
|
import { dumpUrlFor, connectPg, isSupabaseHost } from "./supabase-ca.js";
|
|
@@ -93,8 +90,8 @@ async function inspectDatabase(databaseUrl, schemas) {
|
|
|
93
90
|
await client.end();
|
|
94
91
|
}
|
|
95
92
|
}
|
|
96
|
-
/** 流式跑 pg_dump
|
|
97
|
-
async function
|
|
93
|
+
/** 流式跑 pg_dump,边写快照边算 sha256 与字节数。返回校验和与大小。 */
|
|
94
|
+
async function dumpToSnapshot(config, writer, pgDumpBin, dumpRelationBytes) {
|
|
98
95
|
const hash = createHash("sha256");
|
|
99
96
|
let bytes = 0;
|
|
100
97
|
const meter = new Transform({
|
|
@@ -136,33 +133,26 @@ async function dumpToS3(config, s3, key, pgDumpBin, partSize) {
|
|
|
136
133
|
? resolve()
|
|
137
134
|
: reject(new Error(`pg_dump exited with code ${code}: ${stderr.trim()}`)));
|
|
138
135
|
});
|
|
139
|
-
const upload = new Upload({
|
|
140
|
-
client: s3,
|
|
141
|
-
partSize,
|
|
142
|
-
params: {
|
|
143
|
-
Bucket: config.storage.bucket,
|
|
144
|
-
Key: key,
|
|
145
|
-
Body: meter,
|
|
146
|
-
ContentType: "application/octet-stream",
|
|
147
|
-
},
|
|
148
|
-
});
|
|
149
|
-
const uploadDone = upload.done();
|
|
150
136
|
// 失败双向联动:dump 失败要中止上传(绝不把半截转储当成功);上传失败也必须
|
|
151
137
|
// 杀掉 pg_dump —— 否则上传侧停止消费后,背压会让 pg_dump 永远阻塞在 stdout
|
|
152
138
|
// 写入上,调用方(worker 的串行队列)随之永久挂起。Promise.all 会订阅两侧,
|
|
153
139
|
// 先失败的一方抛出后,另一方稍后的 rejection 也已被视为 handled。
|
|
140
|
+
// 开始上传也放进 try:实现可能在建上传时就同步抛错,那时 pg_dump 已经在跑,同样要杀掉。
|
|
141
|
+
let upload = null;
|
|
154
142
|
try {
|
|
155
|
-
|
|
143
|
+
upload = writer.uploadDump(meter, dumpRelationBytes);
|
|
144
|
+
await Promise.all([dumpDone, upload.done]);
|
|
156
145
|
}
|
|
157
146
|
catch (error) {
|
|
147
|
+
dumpDone.catch(() => { }); // 没走到 Promise.all 时,被杀的 pg_dump 的退出也得有人接住
|
|
158
148
|
dump.kill("SIGKILL");
|
|
159
149
|
meter.destroy();
|
|
160
|
-
await upload
|
|
150
|
+
await upload?.abort().catch(() => { });
|
|
161
151
|
throw error;
|
|
162
152
|
}
|
|
163
153
|
return { bytes, sha256: hash.digest("hex") };
|
|
164
154
|
}
|
|
165
|
-
export async function runBackup(config) {
|
|
155
|
+
export async function runBackup(config, opts = {}) {
|
|
166
156
|
// 源身份先于一切传输(评审第 12 轮):①连接串不许带 ?host=/user= 覆盖——否则
|
|
167
157
|
// 快照记的 ref 与实际读取对象脱节,恢复端的同源保护随之失真;②库与 Storage
|
|
168
158
|
// 跨项目 = 配置错误,全量 dump+同步之后才发现会白烧 egress 还留下半截快照前缀
|
|
@@ -190,29 +180,26 @@ export async function runBackup(config) {
|
|
|
190
180
|
`pg_dump must be >= the server major version. Install postgresql-client-${serverMajor} ` +
|
|
191
181
|
`(or newer) and/or point BACKUPDRILL_PG_DUMP at it.`);
|
|
192
182
|
}
|
|
193
|
-
const
|
|
194
|
-
region: config.storage.region,
|
|
195
|
-
endpoint: config.storage.endpoint,
|
|
196
|
-
forcePathStyle: config.storage.forcePathStyle,
|
|
197
|
-
credentials: {
|
|
198
|
-
accessKeyId: config.storage.accessKeyId,
|
|
199
|
-
secretAccessKey: config.storage.secretAccessKey,
|
|
200
|
-
},
|
|
201
|
-
});
|
|
183
|
+
const store = opts.store ?? s3SnapshotStore(config);
|
|
202
184
|
const timestamp = new Date().toISOString().replace(/[:.]/g, "-");
|
|
203
|
-
const
|
|
204
|
-
const
|
|
205
|
-
|
|
206
|
-
|
|
185
|
+
const writer = await store.createSnapshot(timestamp);
|
|
186
|
+
const storageFiles = writer.storageFiles;
|
|
187
|
+
// 在转储之前拦下:目标放不了 Storage 文件却配了 Storage,转储完才发现等于白跑一遍
|
|
188
|
+
if (config.supabaseStorage && !storageFiles) {
|
|
189
|
+
throw new Error("Storage file backup needs an S3-compatible bucket; this backup destination only holds the database.");
|
|
190
|
+
}
|
|
191
|
+
const dumpKey = writer.dumpKey;
|
|
192
|
+
log.step(`Dumping → ${writer.describe("dump.pgcustom")}`);
|
|
193
|
+
const { bytes, sha256 } = await dumpToSnapshot(config, writer, pgDumpBin, db.relationBytes);
|
|
207
194
|
log.ok(`Database dumped (${(bytes / 1024 / 1024).toFixed(1)} MB, sha256 ${sha256.slice(0, 12)}…)`);
|
|
208
195
|
// Storage 文件同步(源配置存在才做;否则只备数据库)
|
|
209
196
|
let storageResult = null;
|
|
210
|
-
if (config.supabaseStorage) {
|
|
197
|
+
if (config.supabaseStorage && storageFiles) {
|
|
211
198
|
log.step("Syncing Storage files…");
|
|
212
199
|
const synced = await syncStorage(config.supabaseStorage, {
|
|
213
|
-
client: s3,
|
|
214
|
-
bucket:
|
|
215
|
-
base, // syncStorage 内部会加 /storage/<bucket>/<key>
|
|
200
|
+
client: storageFiles.s3,
|
|
201
|
+
bucket: storageFiles.bucket,
|
|
202
|
+
base: storageFiles.base, // syncStorage 内部会加 /storage/<bucket>/<key>
|
|
216
203
|
});
|
|
217
204
|
// v2 bucket 属性采集,通道 = 已有的只读库连接(per-file 元数据已随 GetObject
|
|
218
205
|
// 响应在 syncStorage 内原子捕获)。失败只降级不报错:storage schema 读不到
|
|
@@ -257,26 +244,9 @@ export async function runBackup(config) {
|
|
|
257
244
|
// RECOVERY.md(PRD §5.9.2.1)先于 manifest 落桶:manifest 是快照的"完成标记"
|
|
258
245
|
// (演练/恢复按它找快照)——先写手册再写标记,"快照完成"就蕴含"手册在场";
|
|
259
246
|
// 手册写失败 = 备份失败(同桶同权限,与 manifest 自身的写入风险一致)。
|
|
260
|
-
await
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
snapshot: timestamp,
|
|
265
|
-
bucket: config.storage.bucket,
|
|
266
|
-
endpoint: config.storage.endpoint,
|
|
267
|
-
region: config.storage.region,
|
|
268
|
-
prefix: config.storage.prefix,
|
|
269
|
-
projectName: config.projectName,
|
|
270
|
-
}),
|
|
271
|
-
ContentType: "text/markdown",
|
|
272
|
-
}));
|
|
273
|
-
log.ok(`Recovery runbook written → s3://${config.storage.bucket}/${base}/RECOVERY.md`);
|
|
274
|
-
await s3.send(new PutObjectCommand({
|
|
275
|
-
Bucket: config.storage.bucket,
|
|
276
|
-
Key: `${base}/manifest.json`,
|
|
277
|
-
Body: JSON.stringify(manifest, null, 2),
|
|
278
|
-
ContentType: "application/json",
|
|
279
|
-
}));
|
|
280
|
-
log.ok(`Manifest written → s3://${config.storage.bucket}/${base}/manifest.json`);
|
|
247
|
+
await writer.writeRecoveryDoc(manifest);
|
|
248
|
+
log.ok(`Recovery runbook written → ${writer.describe("RECOVERY.md")}`);
|
|
249
|
+
await writer.writeManifest(manifest);
|
|
250
|
+
log.ok(`Manifest written → ${writer.describe("manifest.json")}`);
|
|
281
251
|
return manifest;
|
|
282
252
|
}
|
package/dist/config.d.ts
CHANGED
|
@@ -26,19 +26,35 @@ export interface BackupConfig {
|
|
|
26
26
|
databaseUrl: string;
|
|
27
27
|
projectName: string;
|
|
28
28
|
schemas: string[];
|
|
29
|
-
storage
|
|
29
|
+
storage?: StorageConfig;
|
|
30
30
|
supabaseStorage?: SupabaseStorageConfig;
|
|
31
31
|
}
|
|
32
|
+
/** CLI 读出来的配置一定带桶:loadConfig 缺了桶直接报错,所以返回类型里 storage 是必填 */
|
|
33
|
+
export type LoadedBackupConfig = BackupConfig & {
|
|
34
|
+
storage: StorageConfig;
|
|
35
|
+
};
|
|
36
|
+
/** 需要桶的地方先过这一道:没配桶时给一句能看懂的报错,而不是读 undefined 的属性。 */
|
|
37
|
+
export declare function requireBucket(config: BackupConfig): StorageConfig;
|
|
32
38
|
interface RawConfig {
|
|
33
39
|
databaseUrl?: string;
|
|
34
40
|
projectName?: string;
|
|
35
41
|
schemas?: string[];
|
|
36
42
|
storage?: Partial<StorageConfig>;
|
|
37
43
|
supabaseStorage?: Partial<SupabaseStorageConfig>;
|
|
44
|
+
pingUrl?: string;
|
|
38
45
|
}
|
|
46
|
+
/**
|
|
47
|
+
* 监控 ping 网址(Healthchecks.io 那种):flag > 环境变量 > 配置文件,空串算没配
|
|
48
|
+
* (GitHub Actions 里没设的 secret 展开就是空串)。不放进 loadConfig:配置缺项时
|
|
49
|
+
* loadConfig 直接抛错,而这恰恰是要报 /fail 的情况,所以网址得先单独读出来。
|
|
50
|
+
*/
|
|
51
|
+
export declare function resolvePingUrl(opts: {
|
|
52
|
+
flag?: string;
|
|
53
|
+
configPath: string;
|
|
54
|
+
}): Promise<string | undefined>;
|
|
39
55
|
/** 合并优先级:显式 flag > 环境变量 > 配置文件。 */
|
|
40
56
|
export declare function loadConfig(opts: {
|
|
41
57
|
configPath: string;
|
|
42
58
|
overrides: RawConfig;
|
|
43
|
-
}): Promise<
|
|
59
|
+
}): Promise<LoadedBackupConfig>;
|
|
44
60
|
export {};
|
package/dist/config.js
CHANGED
|
@@ -1,4 +1,11 @@
|
|
|
1
1
|
import { readFile } from "node:fs/promises";
|
|
2
|
+
/** 需要桶的地方先过这一道:没配桶时给一句能看懂的报错,而不是读 undefined 的属性。 */
|
|
3
|
+
export function requireBucket(config) {
|
|
4
|
+
if (!config.storage) {
|
|
5
|
+
throw new Error("No S3-compatible bucket configured (storage) — set storage.bucket and its keys.");
|
|
6
|
+
}
|
|
7
|
+
return config.storage;
|
|
8
|
+
}
|
|
2
9
|
function fromEnv() {
|
|
3
10
|
const env = process.env;
|
|
4
11
|
const storage = {};
|
|
@@ -47,8 +54,29 @@ async function fromFile(path) {
|
|
|
47
54
|
catch (error) {
|
|
48
55
|
if (error.code === "ENOENT")
|
|
49
56
|
return {};
|
|
50
|
-
|
|
57
|
+
// JSON.parse 的报错会引用出错位置附近的原文,里面可能正好是密钥。本机日志里留着它方便排错;
|
|
58
|
+
// 发给监控(Healthchecks 会把它放进告警邮件)的只用 pingMessage 这句不带原文的
|
|
59
|
+
throw Object.assign(new Error(`Could not read config file ${path}: ${error.message}`), { pingMessage: `Could not read config file ${path}. The job's log has the details.` });
|
|
60
|
+
}
|
|
61
|
+
}
|
|
62
|
+
/**
|
|
63
|
+
* 监控 ping 网址(Healthchecks.io 那种):flag > 环境变量 > 配置文件,空串算没配
|
|
64
|
+
* (GitHub Actions 里没设的 secret 展开就是空串)。不放进 loadConfig:配置缺项时
|
|
65
|
+
* loadConfig 直接抛错,而这恰恰是要报 /fail 的情况,所以网址得先单独读出来。
|
|
66
|
+
*/
|
|
67
|
+
export async function resolvePingUrl(opts) {
|
|
68
|
+
let file = {};
|
|
69
|
+
try {
|
|
70
|
+
file = await fromFile(opts.configPath);
|
|
71
|
+
}
|
|
72
|
+
catch {
|
|
73
|
+
// 配置文件读不了时由 loadConfig 报错,这里只是拿不到文件里的网址
|
|
51
74
|
}
|
|
75
|
+
// 文件内容可能是 null、数组这类非对象的合法 JSON:那是 loadConfig 要报的错,这里只是拿不到网址
|
|
76
|
+
const filePingUrl = file && typeof file === "object" ? file.pingUrl : undefined;
|
|
77
|
+
return [opts.flag, process.env.BACKUPDRILL_PING_URL, filePingUrl]
|
|
78
|
+
.map((value) => (typeof value === "string" ? value.trim() : undefined))
|
|
79
|
+
.find(Boolean);
|
|
52
80
|
}
|
|
53
81
|
// spread 里显式的 undefined 会覆盖前一层的值(CLI 未传的 flag 正是 undefined),
|
|
54
82
|
// 合并前必须剔除,否则 env/配置文件里的 storage 配置会被整体清空。
|
package/dist/drill.d.ts
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
import type { BackupConfig } from "./config.js";
|
|
2
2
|
import type { Manifest } from "./manifest.js";
|
|
3
|
+
import { type SnapshotStore } from "./snapshot-store.js";
|
|
3
4
|
/**
|
|
4
5
|
* 无放回随机抽样:部分 Fisher-Yates,只洗出前 n 位(O(n) 次交换,不必全 shuffle)。
|
|
5
6
|
* 此前恒取 manifest 顺序的前 CAP 个 → 超过 CAP 的项目里排序靠后的文件永远不被校验;
|
|
@@ -92,5 +93,7 @@ export declare function runDrill(config: BackupConfig, opts?: {
|
|
|
92
93
|
appCheckCommand?: string;
|
|
93
94
|
keepSandboxOnFailure?: boolean;
|
|
94
95
|
supervision?: DrillSupervision;
|
|
96
|
+
/** 快照从哪读;省略 = config.storage 指的 S3 兼容桶 */
|
|
97
|
+
store?: SnapshotStore;
|
|
95
98
|
}): Promise<DrillReport>;
|
|
96
99
|
export {};
|
package/dist/drill.js
CHANGED
|
@@ -4,7 +4,8 @@ import { tmpdir } from "node:os";
|
|
|
4
4
|
import { join } from "node:path";
|
|
5
5
|
import { promisify } from "node:util";
|
|
6
6
|
import { parseManifest } from "./manifest.js";
|
|
7
|
-
import {
|
|
7
|
+
import { hashObject } from "./snapshots.js";
|
|
8
|
+
import { s3SnapshotStore } from "./snapshot-store.js";
|
|
8
9
|
import { SANDBOX_MANAGED_ERROR, classifyBlocks, finalizePass, installExtensions, quoteIdent, restoreDatabaseArtifact, installSandboxShim, } from "./restore-engine.js";
|
|
9
10
|
import { connectPg } from "./supabase-ca.js";
|
|
10
11
|
import { localPgRestoreMajor, restoreToolBreaksOnTarget, sandboxMajorFor } from "./pgbin.js";
|
|
@@ -329,16 +330,15 @@ export async function verifyRestored(connString, manifest) {
|
|
|
329
330
|
* Storage 完整性校验:从备份桶重新读回文件、比对 manifest 里记录的 sha256,
|
|
330
331
|
* 证明备份的文件副本真实存在且未损坏。文件多时抽样(默认上限),避免大量 egress。
|
|
331
332
|
*/
|
|
332
|
-
async function verifyStorage(
|
|
333
|
+
async function verifyStorage(location, manifest, verifyAll) {
|
|
333
334
|
const files = manifest.storage.files;
|
|
334
335
|
const sample = verifyAll ? files : sampleStorageFiles(files, STORAGE_SAMPLE_CAP);
|
|
335
|
-
// 备份写入时的键:<snapshot base>/storage/<bucket>/<key
|
|
336
|
-
const base = manifest.dump.key.replace(/\/dump\.pgcustom$/, "");
|
|
336
|
+
// 备份写入时的键:<snapshot base>/storage/<bucket>/<key>
|
|
337
337
|
const mismatches = [];
|
|
338
338
|
for (const f of sample) {
|
|
339
|
-
const objectKey = `${base}/storage/${f.bucket}/${f.key}`;
|
|
339
|
+
const objectKey = `${location.base}/storage/${f.bucket}/${f.key}`;
|
|
340
340
|
try {
|
|
341
|
-
const { sha256 } = await hashObject(s3,
|
|
341
|
+
const { sha256 } = await hashObject(location.s3, location.bucket, objectKey);
|
|
342
342
|
if (sha256 !== f.sha256)
|
|
343
343
|
mismatches.push(`${f.bucket}/${f.key}`);
|
|
344
344
|
}
|
|
@@ -569,17 +569,17 @@ export async function drillDump(dumpPath, manifest, snapshot, preChecks = [], op
|
|
|
569
569
|
/** 演练一份快照:从桶里下载→drillDump(含 Storage 校验)→报告。 */
|
|
570
570
|
export async function runDrill(config, opts = {}) {
|
|
571
571
|
validateSupervision(opts.supervision, opts.appCheckCommand, opts.keepSandboxOnFailure);
|
|
572
|
-
const
|
|
573
|
-
const
|
|
574
|
-
const snapshot =
|
|
572
|
+
const store = opts.store ?? s3SnapshotStore(config);
|
|
573
|
+
const reader = await store.openSnapshot(opts.snapshot);
|
|
574
|
+
const snapshot = reader.name;
|
|
575
575
|
log.step(`Drilling snapshot ${snapshot}`);
|
|
576
|
-
const manifest = parseManifest(await
|
|
576
|
+
const manifest = parseManifest(await reader.readManifest());
|
|
577
577
|
const workdir = opts.supervision?.workdir ?? await mkdtemp(join(tmpdir(), "backupdrill-"));
|
|
578
578
|
const dumpPath = join(workdir, "dump.pgcustom");
|
|
579
579
|
try {
|
|
580
580
|
await opts.supervision?.observe({ phase: "download", dumpBytes: manifest.dump.bytes });
|
|
581
581
|
log.step("Downloading dump…");
|
|
582
|
-
const { sha256: sha } = await
|
|
582
|
+
const { sha256: sha } = await reader.downloadDump(manifest, dumpPath);
|
|
583
583
|
const integrityPass = sha === manifest.dump.sha256;
|
|
584
584
|
const preChecks = [
|
|
585
585
|
{
|
|
@@ -605,7 +605,14 @@ export async function runDrill(config, opts = {}) {
|
|
|
605
605
|
if (manifest.storage && manifest.storage.files.length) {
|
|
606
606
|
await opts.supervision?.observe({ phase: "storage" });
|
|
607
607
|
log.step("Verifying Storage files…");
|
|
608
|
-
|
|
608
|
+
const location = reader.storageFiles(manifest);
|
|
609
|
+
preChecks.push(location
|
|
610
|
+
? await verifyStorage(location, manifest, opts.verifyAllFiles ?? false)
|
|
611
|
+
: {
|
|
612
|
+
name: "storage file integrity",
|
|
613
|
+
pass: false,
|
|
614
|
+
detail: "the manifest lists Storage files, but this snapshot location cannot hold them",
|
|
615
|
+
});
|
|
609
616
|
}
|
|
610
617
|
return await drillDump(dumpPath, manifest, snapshot, preChecks, {
|
|
611
618
|
appCheckCommand: opts.appCheckCommand,
|
package/dist/index.js
CHANGED
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
2
|
import { Command } from "commander";
|
|
3
|
-
import { loadConfig } from "./config.js";
|
|
3
|
+
import { loadConfig, resolvePingUrl } from "./config.js";
|
|
4
|
+
import { backupSummary, createPinger, drillOutcome, pingMessageOf, sensitiveValuesOf } from "./ping.js";
|
|
4
5
|
import { runBackup } from "./backup.js";
|
|
5
6
|
import { runEstimate } from "./estimate.js";
|
|
6
7
|
import { runDrill } from "./drill.js";
|
|
@@ -37,6 +38,10 @@ const EGRESS_PRICING = {
|
|
|
37
38
|
pro: { planLabel: "Pro", includedGb: 250, pricePerGb: 0.09, hardCap: false },
|
|
38
39
|
team: { planLabel: "Team", includedGb: 250, pricePerGb: 0.09, hardCap: false },
|
|
39
40
|
};
|
|
41
|
+
// backup 和 drill 共用:按 Healthchecks.io 的约定报告开始、成功、失败
|
|
42
|
+
const PING_FLAG = "--ping-url <url>";
|
|
43
|
+
const PING_HELP = "Healthchecks.io ping URL (or BACKUPDRILL_PING_URL): pings <url>/start, then <url> on " +
|
|
44
|
+
"success or <url>/fail on failure. A ping that fails only prints a warning";
|
|
40
45
|
const program = new Command();
|
|
41
46
|
program
|
|
42
47
|
.name("backupdrill")
|
|
@@ -55,9 +60,13 @@ program
|
|
|
55
60
|
.option("--endpoint <url>", "S3-compatible endpoint (required for R2/B2; omit for AWS S3)")
|
|
56
61
|
.option("--region <region>", "bucket region (default: auto)")
|
|
57
62
|
.option("--prefix <prefix>", "key prefix inside the bucket (default: backupdrill)")
|
|
63
|
+
.option(PING_FLAG, PING_HELP)
|
|
58
64
|
.action(async (opts) => {
|
|
65
|
+
const ping = createPinger(await resolvePingUrl({ flag: opts.pingUrl, configPath: opts.config }));
|
|
66
|
+
await ping.start();
|
|
67
|
+
let config;
|
|
59
68
|
try {
|
|
60
|
-
|
|
69
|
+
config = await loadConfig({
|
|
61
70
|
configPath: opts.config,
|
|
62
71
|
overrides: {
|
|
63
72
|
databaseUrl: opts.databaseUrl,
|
|
@@ -79,10 +88,13 @@ program
|
|
|
79
88
|
`${storageNote}.`);
|
|
80
89
|
// stdout 输出机器可读结果,便于在 CI / GitHub Action 里消费
|
|
81
90
|
process.stdout.write(JSON.stringify({ ok: true, manifest }) + "\n");
|
|
91
|
+
await ping.success(backupSummary(manifest));
|
|
82
92
|
}
|
|
83
93
|
catch (error) {
|
|
84
|
-
|
|
94
|
+
const message = error.message;
|
|
95
|
+
log.error(message);
|
|
85
96
|
process.exitCode = 1;
|
|
97
|
+
await ping.fail(`backupdrill ${TOOL_VERSION}: backup failed. ${pingMessageOf(error)}`, sensitiveValuesOf(config, opts.databaseUrl));
|
|
86
98
|
}
|
|
87
99
|
});
|
|
88
100
|
program
|
|
@@ -101,13 +113,17 @@ program
|
|
|
101
113
|
"Note: connects as the sandbox superuser, so RLS policies are NOT enforced — " +
|
|
102
114
|
"verify data and business invariants here, not RLS behavior")
|
|
103
115
|
.option("--keep", "if the drill fails, keep the sandbox container running for inspection")
|
|
116
|
+
.option(PING_FLAG, PING_HELP)
|
|
104
117
|
.action(async (opts) => {
|
|
118
|
+
const ping = createPinger(await resolvePingUrl({ flag: opts.pingUrl, configPath: opts.config }));
|
|
119
|
+
await ping.start();
|
|
120
|
+
let config;
|
|
105
121
|
try {
|
|
106
122
|
if (opts.checkCmd !== undefined && !String(opts.checkCmd).trim()) {
|
|
107
123
|
throw new Error("--check-cmd is empty (did an env var fail to expand?) — refusing to run a drill " +
|
|
108
124
|
"that would report a check that never executes");
|
|
109
125
|
}
|
|
110
|
-
|
|
126
|
+
config = await loadConfig({
|
|
111
127
|
configPath: opts.config,
|
|
112
128
|
overrides: {
|
|
113
129
|
projectName: opts.projectName,
|
|
@@ -136,12 +152,20 @@ program
|
|
|
136
152
|
`${report.restoredTableCount} tables / ${report.restoredRowTotal.toLocaleString()} rows ` +
|
|
137
153
|
`restored in ${report.restoreSeconds}s`);
|
|
138
154
|
process.stdout.write(JSON.stringify({ ok: report.pass, report }) + "\n");
|
|
139
|
-
|
|
155
|
+
const outcome = drillOutcome(report);
|
|
156
|
+
if (outcome.pass) {
|
|
157
|
+
await ping.success(outcome.text);
|
|
158
|
+
}
|
|
159
|
+
else {
|
|
140
160
|
process.exitCode = 1;
|
|
161
|
+
await ping.fail(outcome.text, sensitiveValuesOf(config));
|
|
162
|
+
}
|
|
141
163
|
}
|
|
142
164
|
catch (error) {
|
|
143
|
-
|
|
165
|
+
const message = error.message;
|
|
166
|
+
log.error(message);
|
|
144
167
|
process.exitCode = 1;
|
|
168
|
+
await ping.fail(`backupdrill ${TOOL_VERSION}: drill failed. ${pingMessageOf(error)}`, sensitiveValuesOf(config));
|
|
145
169
|
}
|
|
146
170
|
});
|
|
147
171
|
program
|
package/dist/lib.d.ts
CHANGED
|
@@ -5,9 +5,11 @@ export { runRestore } from "./restore.js";
|
|
|
5
5
|
export { projectRefOf, assertNoHostOverride, libpqChildEnv } from "./restore-engine.js";
|
|
6
6
|
export { sameDatabaseTarget } from "./restore.js";
|
|
7
7
|
export { runEstimate, project } from "./estimate.js";
|
|
8
|
-
export { loadConfig } from "./config.js";
|
|
8
|
+
export { loadConfig, requireBucket } from "./config.js";
|
|
9
9
|
export { targetClient, resolveSnapshot, getObjectText, downloadToFile, hashObject, } from "./snapshots.js";
|
|
10
|
-
export
|
|
10
|
+
export { s3SnapshotStore } from "./snapshot-store.js";
|
|
11
|
+
export type { SnapshotStore, SnapshotWriter, SnapshotReader, DumpUpload, StorageFilesLocation, } from "./snapshot-store.js";
|
|
12
|
+
export type { BackupConfig, LoadedBackupConfig, StorageConfig, SupabaseStorageConfig, } from "./config.js";
|
|
11
13
|
export type { BucketAttrs, ExtensionInfo, Manifest, StorageFile, TableStat } from "./manifest.js";
|
|
12
14
|
export type { DrillReport, DrillCheck, DrillSupervision, DrillPhase } from "./drill.js";
|
|
13
15
|
export type { RestoreResult } from "./restore.js";
|
package/dist/lib.js
CHANGED
|
@@ -10,5 +10,7 @@ export { runRestore } from "./restore.js";
|
|
|
10
10
|
export { projectRefOf, assertNoHostOverride, libpqChildEnv } from "./restore-engine.js";
|
|
11
11
|
export { sameDatabaseTarget } from "./restore.js";
|
|
12
12
|
export { runEstimate, project } from "./estimate.js";
|
|
13
|
-
export { loadConfig } from "./config.js";
|
|
13
|
+
export { loadConfig, requireBucket } from "./config.js";
|
|
14
14
|
export { targetClient, resolveSnapshot, getObjectText, downloadToFile, hashObject, } from "./snapshots.js";
|
|
15
|
+
// 快照位置接口:备份、演练、恢复都可以传入自己的实现(托管 worker 的 Google Drive 目标)
|
|
16
|
+
export { s3SnapshotStore } from "./snapshot-store.js";
|
package/dist/ping.d.ts
ADDED
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
import type { BackupConfig } from "./config.js";
|
|
2
|
+
import type { DrillReport } from "./drill.js";
|
|
3
|
+
import type { Manifest } from "./manifest.js";
|
|
4
|
+
export interface Pinger {
|
|
5
|
+
start(): Promise<void>;
|
|
6
|
+
success(summary: string): Promise<void>;
|
|
7
|
+
/** sensitive:这次运行用到的连接串、密钥等,发出去之前从 reason 里抹掉 */
|
|
8
|
+
fail(reason: string, sensitive?: string[]): Promise<void>;
|
|
9
|
+
}
|
|
10
|
+
export declare function createPinger(rawUrl: string | undefined, opts?: {
|
|
11
|
+
timeoutMs?: number;
|
|
12
|
+
attempts?: number;
|
|
13
|
+
retryDelayMs?: number;
|
|
14
|
+
}): Pinger;
|
|
15
|
+
/** backup 成功时发给监控的一行。快照名取对象键里的目录名(和 drill --snapshot 认的是同一个) */
|
|
16
|
+
export declare function backupSummary(manifest: Manifest): string;
|
|
17
|
+
/** drill 跑完(没抛错)后发哪种 ping、带什么:通过 = 成功;没通过 = 失败,列出没过的检查 */
|
|
18
|
+
export declare function drillOutcome(report: DrillReport): {
|
|
19
|
+
pass: boolean;
|
|
20
|
+
text: string;
|
|
21
|
+
};
|
|
22
|
+
/** 报错可以自带一句发给监控用的说法(pingMessage),比如不能复述配置文件原文的那种 */
|
|
23
|
+
export declare function pingMessageOf(error: unknown): string;
|
|
24
|
+
/**
|
|
25
|
+
* 失败原因发出去之前抹掉连接信息。先按值抹(这次运行确实用到的连接串、密码、主机、密钥),
|
|
26
|
+
* 再按形状兜底:pg / S3 的报错会带主机名、IP、整条连接串,配置文件解析失败时 JSON.parse
|
|
27
|
+
* 的报错还会带一截文件原文,这些都不是事先知道的值。
|
|
28
|
+
*/
|
|
29
|
+
export declare function scrubForPing(text: string, sensitive: string[]): string;
|
|
30
|
+
/** 这次运行里不该出现在 ping 里的值:连接串、密码、主机、项目 ref、桶和 Storage 的密钥。 */
|
|
31
|
+
export declare function sensitiveValuesOf(config: BackupConfig | undefined, extraDatabaseUrl?: string): string[];
|