backupdrill 2.6.0 → 2.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +36 -5
- package/dist/drill.js +100 -13
- package/dist/dump-sql.d.ts +52 -0
- package/dist/dump-sql.js +112 -0
- package/dist/external-refs.d.ts +40 -0
- package/dist/external-refs.js +437 -0
- package/dist/index.js +13 -1
- package/dist/interrupt.d.ts +27 -0
- package/dist/interrupt.js +135 -0
- package/dist/recovery-doc.js +18 -5
- package/dist/restore-engine.d.ts +20 -6
- package/dist/restore-engine.js +130 -14
- package/dist/restore.d.ts +7 -0
- package/dist/restore.js +221 -16
- package/package.json +1 -1
- package/plugin/skills/supabase-backup/assets/scheduled-backup.yml +1 -1
package/README.md
CHANGED
|
@@ -59,7 +59,7 @@ Configure via **env vars**, a **config file**, or **flags** (precedence: flags >
|
|
|
59
59
|
### Env vars
|
|
60
60
|
|
|
61
61
|
```bash
|
|
62
|
-
export BACKUPDRILL_DATABASE_URL="postgresql://postgres.<ref>:<pw
|
|
62
|
+
export BACKUPDRILL_DATABASE_URL="postgresql://postgres.<ref>:<pw>@<pooler-host>:5432/postgres"
|
|
63
63
|
export BACKUPDRILL_PROJECT_NAME="my-app"
|
|
64
64
|
export BACKUPDRILL_S3_ENDPOINT="https://<account>.r2.cloudflarestorage.com" # omit for AWS S3
|
|
65
65
|
export BACKUPDRILL_S3_REGION="auto"
|
|
@@ -98,7 +98,7 @@ backupdrill backup --config backupdrill.config.json
|
|
|
98
98
|
|
|
99
99
|
## Get the connection string
|
|
100
100
|
|
|
101
|
-
Use the **Session
|
|
101
|
+
Use the **Session pooler** string: in the Supabase dashboard, open your project, click **Connect** at the top of the page and pick **Session pooler**. Copy the whole string rather than typing the host yourself, because the pooler host can't be worked out from the region. Replace `[YOUR-PASSWORD]` with the database password (percent-encode characters like `@` or `#`). The default `postgres` user works as-is — it owns your tables, so `pg_dump` passes row-level security.
|
|
102
102
|
|
|
103
103
|
**Least-privilege role (advanced, usually not viable on Supabase):** `pg_dump` needs `select` on *sequences* too, and it **errors on any RLS-enabled table the role can't bypass** — and most Supabase tables have RLS on, while `bypassrls` can't be granted without superuser. Only use a dedicated role if your schema has no RLS tables:
|
|
104
104
|
|
|
@@ -176,6 +176,8 @@ Backups record your installed Postgres extensions in the manifest, and the drill
|
|
|
176
176
|
|
|
177
177
|
The sandbox restore runs up to 4 parallel `pg_restore` jobs, sized from the CPUs and memory Docker reports, so a 2 vCPU / 8 GB machine gets 2. Table data and index builds run in parallel. Foreign keys, triggers and policies run on a single connection to keep their error messages separate, including errors about Supabase-managed objects such as `auth.users`. `backupdrill restore` into a real Supabase project stays on a single connection.
|
|
178
178
|
|
|
179
|
+
Ctrl+C (or a SIGTERM from whatever runs the drill) stops it and cleans up: the CLI stops a running `--check-cmd` and `pg_restore`, removes the sandbox container and its volume, deletes the downloaded dump, and exits with 130 (143 for SIGTERM). `--keep` doesn't apply to an interrupted drill. Press Ctrl+C a second time to exit without waiting for the cleanup.
|
|
180
|
+
|
|
179
181
|
### Run your own checks against the restored copy
|
|
180
182
|
|
|
181
183
|
The drill's structural checks (tables, rows, checksums) can't know what "working" means for *your* app — business invariants only have answers in your code. `--check-cmd` hands you the restored sandbox before it's destroyed:
|
|
@@ -186,7 +188,7 @@ backupdrill drill --check-cmd "npm run smoke" # BACKUPDRILL_SANDBOX_URL points
|
|
|
186
188
|
|
|
187
189
|
Your command runs after all structural checks pass, with `BACKUPDRILL_SANDBOX_URL` set to the sandbox's connection string. Exit 0 means pass; the result shows up in the report as an `app checks` line, and a non-zero exit fails the drill (the report keeps structural and app-level failures clearly separate). Entirely optional — without the flag, nothing changes and the report has no `app checks` line. There's a 10-minute timeout (the whole process tree is killed), and `--keep` leaves the sandbox container running when a drill fails so you can inspect it — the report JSON then includes a `keptSandbox` object with the container id and connection string (`docker rm -f <id>` when done).
|
|
188
190
|
|
|
189
|
-
**What this can and cannot verify, honestly:** `BACKUPDRILL_SANDBOX_URL` connects as the sandbox **superuser**, and superusers bypass row-level security. The sandbox does carry a stub of the Supabase runtime — an `auth` schema whose `uid()`/`jwt()`/`role()`/`email()` return null, and the `anon`/`authenticated`/`service_role` roles — so functions, column defaults and policies that reference them restore instead of being skipped. The sandbox deliberately does not create Supabase-managed *tables* such as `auth.users`: an index, constraint or policy that points at one is reported as a skipped managed object (these are restored in a second, post-data pass where that exemption applies)
|
|
191
|
+
**What this can and cannot verify, honestly:** `BACKUPDRILL_SANDBOX_URL` connects as the sandbox **superuser**, and superusers bypass row-level security. The sandbox does carry a stub of the Supabase runtime — an `auth` schema whose `uid()`/`jwt()`/`role()`/`email()` return null, and the `anon`/`authenticated`/`service_role` roles — so functions, column defaults and policies that reference them restore instead of being skipped. The sandbox deliberately does not create Supabase-managed *tables* such as `auth.users`: an index, constraint or policy that points at one is reported as a skipped managed object (these are restored in a second, post-data pass where that exemption applies). A skipped foreign key into `auth.users` still matters for a real restore, see [Tables that reference auth.users](#tables-that-reference-authusers). A *view* over one is created in the strict first pass instead, and fails the drill — a real gap in what a public-only backup can rebuild, shown rather than hidden. A function *body* that selects from `auth.users` is different: PostgreSQL does not validate string-literal bodies (`AS $$ … $$`) during a restore (`check_function_bodies` is off in every dump), so such a function is recreated and the drill passes — it would only fail when called. The exception is an SQL-standard `BEGIN ATOMIC … END` body, which is parsed and dependency-checked at creation regardless of that setting, so one of those referencing a managed table fails the drill like a view does. That is exactly what `--check-cmd` is for: if your application depends on such a function, call it there. So use `--check-cmd` for data and business invariants ("orders reference existing users", "the row count I care about is sane"). It is **not** an RLS behavior test — `auth.uid()` is a stub and you are a superuser; verifying RLS takes an authenticated client stack with real JWTs, which no restore sandbox can fake generically.
|
|
190
192
|
|
|
191
193
|
When you actually need to recover, `restore` puts the database back into a fresh
|
|
192
194
|
target project (through the same engine the drills use, with verification) and
|
|
@@ -202,6 +204,8 @@ backupdrill restore --database --target-supabase-url https://<ref>.supabase.co -
|
|
|
202
204
|
|
|
203
205
|
Prefer files on disk instead? Skip the Storage target and pass `--storage-dir ./recovered`.
|
|
204
206
|
|
|
207
|
+
`--dry-run` writes nothing to the target. Besides checking the target (not the source project, schemas empty, Postgres and `pg_restore` versions, extensions available), it downloads the dump to a temporary directory, checks its sha256 and looks for foreign keys that point outside the backup (see [Tables that reference auth.users](#tables-that-reference-authusers)). The real run repeats those checks before it writes anything; a foreign key into a table that one of the recorded extensions creates is checked again once the extensions are installed. The database restore then runs in a single transaction: if any statement fails, nothing from the snapshot stays on the target, and you can run it again on the same project once you've fixed the cause. (If the connection drops while it commits, the error says so and asks you to check the target first.) Extensions the run installed stay installed, since the next run needs them too, and the empty-target check doesn't count objects that belong to an extension (except an extension's configuration tables, whose rows a dump can carry). One transaction holds a lock on every table, index and sequence until it commits, so a very large schema can run out of lock slots: on Supabase's smallest compute size, about a thousand tables (partitions count as tables). The restore then stops with nothing written and tells you how to raise `max_locks_per_transaction`.
|
|
208
|
+
|
|
205
209
|
Or restore the dump by hand — into an **empty** database, without `--clean`:
|
|
206
210
|
|
|
207
211
|
```bash
|
|
@@ -211,8 +215,35 @@ pg_restore --no-owner --no-privileges --dbname "<target-connection-string>" dump
|
|
|
211
215
|
Don't use `--clean` against a Supabase project: dropping and recreating the `public`
|
|
212
216
|
schema silently wipes Supabase's default grants for `anon`/`authenticated` (the dump
|
|
213
217
|
carries no ACLs), leaving the restored API returning 401s. One error is expected and
|
|
214
|
-
harmless: `schema "public" already exists` — the target always has it.
|
|
215
|
-
|
|
218
|
+
harmless: `schema "public" already exists` — the target always has it. An error
|
|
219
|
+
on a foreign key into `auth.users` means the users have to be copied in first (see
|
|
220
|
+
the next section). The `backupdrill restore` command above handles all of this (plus
|
|
221
|
+
verification) for you.
|
|
222
|
+
|
|
223
|
+
### Tables that reference auth.users
|
|
224
|
+
|
|
225
|
+
By default a backup covers the `public` schema only, so your Auth users are not in it. Plenty of Supabase apps have a table like `profiles` whose `id` is a foreign key to `auth.users(id)`. When the restore recreates that key in a new project, Postgres checks every `profiles` row against `auth.users`, and in a new project that table is empty, so the restore fails.
|
|
226
|
+
|
|
227
|
+
A drill can't catch this. Its sandbox has no `auth.users` at all, so the drill reports that foreign key as a skipped Supabase-managed object and passes. `restore --dry-run` checks it against the real target and tells you how many rows have nothing to point at:
|
|
228
|
+
|
|
229
|
+
```
|
|
230
|
+
✗ references outside the backup: public.profiles → auth.users (profiles_id_fkey): 200 of 200 row(s) point at auth.users rows the target doesn't have
|
|
231
|
+
```
|
|
232
|
+
|
|
233
|
+
While that check fails, no snapshot data is written. Copy the users into the new project first, then run the restore. While the old project is still reachable, the Postgres tools can do it. Use the Session pooler string of each project, and type the passwords at hidden prompts so they stay out of your shell history:
|
|
234
|
+
|
|
235
|
+
```bash
|
|
236
|
+
read -rs OLD_PW; read -rs NEW_PW
|
|
237
|
+
PGPASSWORD="$OLD_PW" pg_dump --data-only --table=auth.users --table=auth.identities \
|
|
238
|
+
"postgresql://postgres.<old-ref>@<old-pooler-host>:5432/postgres" |
|
|
239
|
+
PGPASSWORD="$NEW_PW" psql -v ON_ERROR_STOP=1 \
|
|
240
|
+
"postgresql://postgres.<new-ref>@<new-pooler-host>:5432/postgres"
|
|
241
|
+
unset OLD_PW NEW_PW
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
`auth.identities` holds each user's sign-in methods; users copied with it can sign in to the new project with their old password. If the old project is gone, the users can only come from a backup that includes the `auth` schema, such as Supabase's own daily backups on a paid plan. A backupdrill snapshot of `public` doesn't have them.
|
|
245
|
+
|
|
246
|
+
Triggers you created on `auth.users`, such as the common one that inserts a `profiles` row on sign-up, belong to that table, so they aren't in the backup either. Create them again after the restore.
|
|
216
247
|
|
|
217
248
|
## 2.0 migration — `restore` flags changed
|
|
218
249
|
|
package/dist/drill.js
CHANGED
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import { execFile, spawn } from "node:child_process";
|
|
2
|
+
import { randomBytes } from "node:crypto";
|
|
2
3
|
import { mkdtemp, rm } from "node:fs/promises";
|
|
3
4
|
import { tmpdir } from "node:os";
|
|
4
5
|
import { join } from "node:path";
|
|
@@ -10,6 +11,7 @@ import { SANDBOX_MANAGED_ERROR, classifyBlocks, finalizePass, installExtensions,
|
|
|
10
11
|
import { connectPg } from "./supabase-ca.js";
|
|
11
12
|
import { localPgRestoreMajor, restoreToolBreaksOnTarget, sandboxMajorFor } from "./pgbin.js";
|
|
12
13
|
import { log } from "./log.js";
|
|
14
|
+
import { onInterrupt, stopOnInterrupt, throwIfInterrupted } from "./interrupt.js";
|
|
13
15
|
const execFileAsync = promisify(execFile);
|
|
14
16
|
// Storage 校验默认抽样上限(避免每次演练重下全部文件产生大量 egress);超限如实记录
|
|
15
17
|
const STORAGE_SAMPLE_CAP = 100;
|
|
@@ -133,6 +135,26 @@ export function sandboxDataVolumeTarget(major) {
|
|
|
133
135
|
// PG18 将镜像的 VOLUME 上移一层;挂错路径会额外生成无法按名字回收的匿名卷。
|
|
134
136
|
return major >= 18 ? "/var/lib/postgresql" : "/var/lib/postgresql/data";
|
|
135
137
|
}
|
|
138
|
+
/**
|
|
139
|
+
* 沙箱镜像先单独拉好。让 docker run 自己拉的话,拉取和建容器在同一个命令里:中断落在拉取中途,
|
|
140
|
+
* 停掉命令也说不准守护进程之后还会不会把容器建出来(交叉审查)。拆开之后,慢的那一步(第一次拉镜像
|
|
141
|
+
* 可能好几分钟)什么都不建,中断时直接停掉;docker run 只剩建容器和启动。
|
|
142
|
+
*/
|
|
143
|
+
async function ensureImage(image) {
|
|
144
|
+
const present = await execFileAsync("docker", ["image", "inspect", "--format", "{{.Id}}", image]).then(() => true, () => false);
|
|
145
|
+
if (present)
|
|
146
|
+
return;
|
|
147
|
+
throwIfInterrupted();
|
|
148
|
+
log.step(`Pulling ${image}…`);
|
|
149
|
+
const pull = execFileAsync("docker", ["pull", "--quiet", image]);
|
|
150
|
+
const forget = pull.child ? stopOnInterrupt(pull.child) : () => { };
|
|
151
|
+
try {
|
|
152
|
+
await pull;
|
|
153
|
+
}
|
|
154
|
+
finally {
|
|
155
|
+
forget();
|
|
156
|
+
}
|
|
157
|
+
}
|
|
136
158
|
async function startEphemeralPostgres(image, major, resourceName) {
|
|
137
159
|
const dataVolumeTarget = sandboxDataVolumeTarget(major);
|
|
138
160
|
if (resourceName) {
|
|
@@ -144,12 +166,17 @@ async function startEphemeralPostgres(image, major, resourceName) {
|
|
|
144
166
|
log.step(`Starting ephemeral ${image}…`);
|
|
145
167
|
const resources = await dockerResources();
|
|
146
168
|
const tuning = sandboxTuning(resources.memoryBytes, resources.cpus);
|
|
147
|
-
|
|
169
|
+
await ensureImage(image);
|
|
170
|
+
// 名字先定好:Ctrl+C 可能落在 docker run 还没返回 ID 的时候,按名字才删得掉
|
|
171
|
+
const name = resourceName ?? `backupdrill-drill-${randomBytes(6).toString("hex")}`;
|
|
172
|
+
throwIfInterrupted();
|
|
173
|
+
const started = execFileAsync("docker", [
|
|
148
174
|
"run",
|
|
149
175
|
"-d",
|
|
150
176
|
"--rm",
|
|
151
|
-
|
|
152
|
-
|
|
177
|
+
"--name",
|
|
178
|
+
name,
|
|
179
|
+
...(resourceName ? ["--mount", `type=volume,source=${resourceName},target=${dataVolumeTarget}`] : []),
|
|
153
180
|
"-e",
|
|
154
181
|
"POSTGRES_PASSWORD=drill",
|
|
155
182
|
"-e",
|
|
@@ -161,10 +188,59 @@ async function startEphemeralPostgres(image, major, resourceName) {
|
|
|
161
188
|
"postgres",
|
|
162
189
|
...tuning.settings.flatMap((kv) => ["-c", kv]),
|
|
163
190
|
]);
|
|
191
|
+
// 删容器只有一份:正常收尾(drillDump 的 finally)和中断收尾(interrupt.ts)等的是同一个 promise。
|
|
192
|
+
// 两边各发一次 docker rm 会撞上 "removal already in progress",后到的那个报删除失败,
|
|
193
|
+
// 进程又在真正删完之前就退出了(信号测试抓到的竞争)。
|
|
194
|
+
let volumeName = resourceName ?? null;
|
|
195
|
+
let runSettled = false;
|
|
196
|
+
started.then(() => (runSettled = true), () => (runSettled = true));
|
|
197
|
+
const removeByName = async () => {
|
|
198
|
+
try {
|
|
199
|
+
// -v 连匿名卷一起删:中断可能落在下面查卷名之前
|
|
200
|
+
await execFileAsync("docker", ["rm", "-fv", name], { timeout: 60_000 });
|
|
201
|
+
}
|
|
202
|
+
catch (error) {
|
|
203
|
+
if (!/No such container/i.test(String(error.stderr ?? error))) {
|
|
204
|
+
log.warn(`could not remove the drill sandbox; remove it with: docker rm -fv ${name}`);
|
|
205
|
+
}
|
|
206
|
+
}
|
|
207
|
+
};
|
|
208
|
+
let teardownRun = null;
|
|
209
|
+
const teardown = () => (teardownRun ??= (async () => {
|
|
210
|
+
if (!runSettled) {
|
|
211
|
+
// 中断落在 docker run 还没返回的时候。镜像已经在本地(ensureImage),它只剩建容器和启动,
|
|
212
|
+
// 等它落定再按名字删:中途停掉命令说不准守护进程会不会稍后把容器建出来(交叉审查复现)。
|
|
213
|
+
// Docker 迟迟不回话就不等了,把容器名告诉用户
|
|
214
|
+
let timer;
|
|
215
|
+
const settled = await Promise.race([
|
|
216
|
+
started.then(() => true, () => true),
|
|
217
|
+
new Promise((resolve) => (timer = setTimeout(() => resolve(false), 20_000))),
|
|
218
|
+
]);
|
|
219
|
+
clearTimeout(timer);
|
|
220
|
+
if (!settled) {
|
|
221
|
+
started.child?.kill("SIGKILL");
|
|
222
|
+
log.warn(`docker run did not finish within 20s; if a container named ${name} shows up later, ` +
|
|
223
|
+
`remove it with: docker rm -fv ${name}`);
|
|
224
|
+
}
|
|
225
|
+
}
|
|
226
|
+
await removeByName();
|
|
227
|
+
// 显式删卷(--rm 不可靠,见下);幂等,已经没了也不报错
|
|
228
|
+
if (volumeName)
|
|
229
|
+
await execFileAsync("docker", ["volume", "rm", "-f", volumeName]).catch(() => { });
|
|
230
|
+
})());
|
|
231
|
+
const forgetInterruptCleanup = onInterrupt(teardown);
|
|
232
|
+
let stdout;
|
|
233
|
+
try {
|
|
234
|
+
({ stdout } = await started);
|
|
235
|
+
}
|
|
236
|
+
catch (error) {
|
|
237
|
+
forgetInterruptCleanup();
|
|
238
|
+
throw error;
|
|
239
|
+
}
|
|
164
240
|
const containerId = stdout.trim();
|
|
165
241
|
// postgres 镜像把数据目录声明成 VOLUME → 每次 run 都建一个匿名卷。生产实测 `--rm` **没有**把它
|
|
166
242
|
// 带走(一次演练遗留 37.6 GB,三次失败把 59 GB 根盘灌满),所以记下卷名,销毁容器后显式删。
|
|
167
|
-
|
|
243
|
+
volumeName = resourceName ?? await execFileAsync("docker", [
|
|
168
244
|
"inspect",
|
|
169
245
|
"-f",
|
|
170
246
|
'{{range .Mounts}}{{if eq .Type "volume"}}{{.Name}}{{end}}{{end}}',
|
|
@@ -198,7 +274,7 @@ async function startEphemeralPostgres(image, major, resourceName) {
|
|
|
198
274
|
"postgres",
|
|
199
275
|
"-q",
|
|
200
276
|
]);
|
|
201
|
-
return { containerId, port, connString,
|
|
277
|
+
return { containerId, port, connString, restoreJobs: tuning.restoreJobs, teardown, forgetInterruptCleanup };
|
|
202
278
|
}
|
|
203
279
|
catch {
|
|
204
280
|
await new Promise((r) => setTimeout(r, 500));
|
|
@@ -207,9 +283,8 @@ async function startEphemeralPostgres(image, major, resourceName) {
|
|
|
207
283
|
throw new Error("ephemeral Postgres did not become ready within 30s");
|
|
208
284
|
}
|
|
209
285
|
catch (error) {
|
|
210
|
-
await
|
|
211
|
-
|
|
212
|
-
await execFileAsync("docker", ["volume", "rm", "-f", volumeName]).catch(() => { });
|
|
286
|
+
await teardown();
|
|
287
|
+
forgetInterruptCleanup();
|
|
213
288
|
throw error;
|
|
214
289
|
}
|
|
215
290
|
}
|
|
@@ -381,6 +456,8 @@ export function runAppCheck(command, connString, timeoutMs = APP_CHECK_TIMEOUT_M
|
|
|
381
456
|
// 孙进程会在沙箱销毁后继续跑(xreview 抓出的资源泄漏)。
|
|
382
457
|
detached: true,
|
|
383
458
|
});
|
|
459
|
+
// 也因为自成一组,终端的 Ctrl+C 到不了它:中断时由我们按组停掉
|
|
460
|
+
stopOnInterrupt(proc, { group: true });
|
|
384
461
|
let timedOut = false;
|
|
385
462
|
const timer = setTimeout(() => {
|
|
386
463
|
timedOut = true;
|
|
@@ -484,6 +561,7 @@ export async function drillDump(dumpPath, manifest, snapshot, preChecks = [], op
|
|
|
484
561
|
});
|
|
485
562
|
}
|
|
486
563
|
await opts.supervision?.observe({ phase: "restore" });
|
|
564
|
+
throwIfInterrupted();
|
|
487
565
|
const started = Date.now();
|
|
488
566
|
log.step("Restoring into ephemeral Postgres…");
|
|
489
567
|
let postData;
|
|
@@ -514,7 +592,11 @@ export async function drillDump(dumpPath, manifest, snapshot, preChecks = [], op
|
|
|
514
592
|
detail: postData.failures.length > 0
|
|
515
593
|
? `${postData.failures.length} failed: ${postData.failures[0]}`
|
|
516
594
|
: postData.supabaseSkipped > 0
|
|
517
|
-
?
|
|
595
|
+
? // 说清跳过的是什么,以及它对真实恢复意味着什么:2026-10-22 实验里演练 PASS,
|
|
596
|
+
// 恢复却死在被跳过的那条 FK → auth.users 上,因为新项目的 auth.users 是空的
|
|
597
|
+
`user objects restored; ${postData.supabaseSkipped} Supabase-managed object(s) skipped ` +
|
|
598
|
+
`(they reference tables such as auth.users, which the drill sandbox doesn't have; ` +
|
|
599
|
+
`restoring into a new project needs the rows they point at to be there first)`
|
|
518
600
|
: "all post-data objects restored",
|
|
519
601
|
});
|
|
520
602
|
await opts.supervision?.observe({ phase: "verify" });
|
|
@@ -523,6 +605,7 @@ export async function drillDump(dumpPath, manifest, snapshot, preChecks = [], op
|
|
|
523
605
|
// 语义层钩子:只在结构层全绿时才跑——结构已失败时再跑应用检查,只会把
|
|
524
606
|
// 一次失败报成两层失败,误导排查方向。没配置则不跑、报告里不出现该行。
|
|
525
607
|
if (appCheckCommand && checks.every((c) => c.pass)) {
|
|
608
|
+
throwIfInterrupted();
|
|
526
609
|
log.step("Running app checks…");
|
|
527
610
|
checks.push(await runAppCheck(appCheckCommand, pg.connString));
|
|
528
611
|
}
|
|
@@ -559,11 +642,10 @@ export async function drillDump(dumpPath, manifest, snapshot, preChecks = [], op
|
|
|
559
642
|
else {
|
|
560
643
|
await observeCleanup(opts.supervision);
|
|
561
644
|
log.step("Tearing down ephemeral Postgres…");
|
|
562
|
-
await
|
|
563
|
-
// 显式删匿名卷(--rm 不可靠,见 startEphemeralPostgres);幂等,已被 --rm 带走也不报错
|
|
564
|
-
if (pg.volumeName)
|
|
565
|
-
await execFileAsync("docker", ["volume", "rm", "-f", pg.volumeName]).catch(() => { });
|
|
645
|
+
await pg.teardown();
|
|
566
646
|
}
|
|
647
|
+
// 删完了,或者按 --keep 有意留着:之后再收到信号都不该再动它
|
|
648
|
+
pg.forgetInterruptCleanup();
|
|
567
649
|
}
|
|
568
650
|
}
|
|
569
651
|
/** 演练一份快照:从桶里下载→drillDump(含 Storage 校验)→报告。 */
|
|
@@ -576,6 +658,10 @@ export async function runDrill(config, opts = {}) {
|
|
|
576
658
|
const manifest = parseManifest(await reader.readManifest());
|
|
577
659
|
const workdir = opts.supervision?.workdir ?? await mkdtemp(join(tmpdir(), "backupdrill-"));
|
|
578
660
|
const dumpPath = join(workdir, "dump.pgcustom");
|
|
661
|
+
// 下载的 dump 在这里;托管模式的目录归 supervisor 管
|
|
662
|
+
const forgetWorkdir = opts.supervision
|
|
663
|
+
? () => { }
|
|
664
|
+
: onInterrupt(() => rm(workdir, { recursive: true, force: true }).catch(() => { }));
|
|
579
665
|
try {
|
|
580
666
|
await opts.supervision?.observe({ phase: "download", dumpBytes: manifest.dump.bytes });
|
|
581
667
|
log.step("Downloading dump…");
|
|
@@ -623,5 +709,6 @@ export async function runDrill(config, opts = {}) {
|
|
|
623
709
|
finally {
|
|
624
710
|
if (!opts.supervision)
|
|
625
711
|
await rm(workdir, { recursive: true, force: true }).catch(() => { });
|
|
712
|
+
forgetWorkdir();
|
|
626
713
|
}
|
|
627
714
|
}
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
export interface QualifiedName {
|
|
2
|
+
schema: string;
|
|
3
|
+
name: string;
|
|
4
|
+
}
|
|
5
|
+
export declare function unquoteIdent(ident: string): string;
|
|
6
|
+
export declare function formatQualified(name: QualifiedName): string;
|
|
7
|
+
/** `pg_restore -f - -L <只含 SCHEMA 条目的清单>` 的输出 → 按出现顺序的 schema 名。 */
|
|
8
|
+
export declare function parseCreateSchemas(sql: string): string[];
|
|
9
|
+
export interface ForeignKeyDef {
|
|
10
|
+
constraint: string;
|
|
11
|
+
table: QualifiedName;
|
|
12
|
+
columns: string[];
|
|
13
|
+
references: QualifiedName;
|
|
14
|
+
referencedColumns: string[];
|
|
15
|
+
/**
|
|
16
|
+
* ADD CONSTRAINT 时会不会校验已有的行。NOT VALID(源库就没校验过)和 PG18 的 NOT ENFORCED
|
|
17
|
+
* 都不校验,恢复不会因为缺行而失败,所以不用查。
|
|
18
|
+
*/
|
|
19
|
+
validatesRows: boolean;
|
|
20
|
+
}
|
|
21
|
+
/**
|
|
22
|
+
* pg_dump 的外键语句(dumpConstraint):
|
|
23
|
+
* ALTER [FOREIGN ]TABLE [ONLY ]<表>\n ADD CONSTRAINT <名> FOREIGN KEY (<列>) REFERENCES <表>(<列>)<选项>;
|
|
24
|
+
* 分区父表上的外键没有 ONLY。认不出的形态(如 PG18 的 PERIOD 时态外键)不返回,
|
|
25
|
+
* 调用方拿条目数对账,差多少就如实报多少条没检查。
|
|
26
|
+
*/
|
|
27
|
+
export declare function parseForeignKeys(sql: string): ForeignKeyDef[];
|
|
28
|
+
/** pg_dump 的挂分区语句(dumpTableAttach):ALTER TABLE ONLY <父> ATTACH PARTITION <子> <边界>; */
|
|
29
|
+
export declare function parsePartitionAttachments(sql: string): Array<{
|
|
30
|
+
parent: QualifiedName;
|
|
31
|
+
child: QualifiedName;
|
|
32
|
+
}>;
|
|
33
|
+
/**
|
|
34
|
+
* pg_dump 的建表语句(dumpTableSchema)里每列的类型,按表(schema\0name)→ 列 → 类型。
|
|
35
|
+
* CREATE [UNLOGGED ]TABLE <表> (\n <列> <类型>[ COLLATE …][ DEFAULT …][ NOT NULL],\n CONSTRAINT …\n)…;
|
|
36
|
+
* 类型是 format_type 的输出,本身不含 COLLATE/DEFAULT/NOT NULL/GENERATED 这些词,读到它们为止。
|
|
37
|
+
* 带引号的列名可以跨行,所以按位置往下读,不按行切。
|
|
38
|
+
*/
|
|
39
|
+
export declare function parseTableColumns(sql: string): Map<string, Map<string, string>>;
|
|
40
|
+
/**
|
|
41
|
+
* 数据段的 COPY 头(dumpTableData_copy):"COPY <表> <列清单> FROM stdin;"。没有列的表列清单是空串,
|
|
42
|
+
* 所以那一行中间是两个空格,返回空列表。
|
|
43
|
+
*/
|
|
44
|
+
export declare function parseCopyHeader(line: string): {
|
|
45
|
+
table: QualifiedName;
|
|
46
|
+
columns: string[];
|
|
47
|
+
} | null;
|
|
48
|
+
/**
|
|
49
|
+
* COPY 文本格式的一个字段 → 值(\N = NULL)。转义规则照 PostgreSQL 文档 "COPY / Text Format":
|
|
50
|
+
* \b \f \n \r \t \v \\,\ 加 1–3 位八进制,\x 加 1–2 位十六进制,其余 \c 就是 c 本身。
|
|
51
|
+
*/
|
|
52
|
+
export declare function decodeCopyField(field: string): string | null;
|
package/dist/dump-sql.js
ADDED
|
@@ -0,0 +1,112 @@
|
|
|
1
|
+
// 读 pg_restore 吐出来的 SQL(纯函数,可单测)。只认 pg_dump 自己的输出格式:
|
|
2
|
+
// 名字一律由 fmtId / quote_identifier 生成——[a-z_][a-z0-9_]* 原样,其余加双引号、内嵌引号翻倍,
|
|
3
|
+
// dump 头部把 search_path 设成空串,所以引用的表总是 schema.name 全限定。
|
|
4
|
+
// 不是通用 SQL 解析器,也不需要是:认不出的语句由调用方如实报"没法检查",不猜。
|
|
5
|
+
const QUOTED_IDENT = String.raw `"(?:[^"]|"")+"`;
|
|
6
|
+
const IDENT = String.raw `(?:${QUOTED_IDENT}|[a-z_][a-z0-9_]*)`;
|
|
7
|
+
const QUALIFIED = String.raw `(${IDENT})\.(${IDENT})`;
|
|
8
|
+
const COLUMNS = String.raw `(${IDENT}(?:\s*,\s*${IDENT})*)`;
|
|
9
|
+
export function unquoteIdent(ident) {
|
|
10
|
+
return ident.startsWith('"') ? ident.slice(1, -1).replace(/""/g, '"') : ident;
|
|
11
|
+
}
|
|
12
|
+
function splitColumns(list) {
|
|
13
|
+
return [...list.matchAll(new RegExp(IDENT, "g"))].map((match) => unquoteIdent(match[0]));
|
|
14
|
+
}
|
|
15
|
+
export function formatQualified(name) {
|
|
16
|
+
const show = (part) => (/^[a-z_][a-z0-9_]*$/.test(part) ? part : `"${part.replace(/"/g, '""')}"`);
|
|
17
|
+
return `${show(name.schema)}.${show(name.name)}`;
|
|
18
|
+
}
|
|
19
|
+
/** `pg_restore -f - -L <只含 SCHEMA 条目的清单>` 的输出 → 按出现顺序的 schema 名。 */
|
|
20
|
+
export function parseCreateSchemas(sql) {
|
|
21
|
+
return [...sql.matchAll(new RegExp(String.raw `^CREATE SCHEMA (${IDENT});$`, "gm"))].map((match) => unquoteIdent(match[1]));
|
|
22
|
+
}
|
|
23
|
+
/**
|
|
24
|
+
* pg_dump 的外键语句(dumpConstraint):
|
|
25
|
+
* ALTER [FOREIGN ]TABLE [ONLY ]<表>\n ADD CONSTRAINT <名> FOREIGN KEY (<列>) REFERENCES <表>(<列>)<选项>;
|
|
26
|
+
* 分区父表上的外键没有 ONLY。认不出的形态(如 PG18 的 PERIOD 时态外键)不返回,
|
|
27
|
+
* 调用方拿条目数对账,差多少就如实报多少条没检查。
|
|
28
|
+
*/
|
|
29
|
+
export function parseForeignKeys(sql) {
|
|
30
|
+
const pattern = new RegExp(String.raw `^ALTER (?:FOREIGN )?TABLE (?:ONLY )?${QUALIFIED}\n {4}ADD CONSTRAINT (${IDENT}) ` +
|
|
31
|
+
String.raw `FOREIGN KEY \(${COLUMNS}\) REFERENCES ${QUALIFIED}\s*\(${COLUMNS}\)(.*);$`, "gm");
|
|
32
|
+
return [...sql.matchAll(pattern)].map((match) => {
|
|
33
|
+
// 选项里可能有带引号的列名(ON DELETE SET NULL ("x")),先去掉再找关键字
|
|
34
|
+
const options = match[8].replace(new RegExp(QUOTED_IDENT, "g"), "");
|
|
35
|
+
return {
|
|
36
|
+
constraint: unquoteIdent(match[3]),
|
|
37
|
+
table: { schema: unquoteIdent(match[1]), name: unquoteIdent(match[2]) },
|
|
38
|
+
columns: splitColumns(match[4]),
|
|
39
|
+
references: { schema: unquoteIdent(match[5]), name: unquoteIdent(match[6]) },
|
|
40
|
+
referencedColumns: splitColumns(match[7]),
|
|
41
|
+
validatesRows: !/\bNOT (?:VALID|ENFORCED)\b/.test(options),
|
|
42
|
+
};
|
|
43
|
+
});
|
|
44
|
+
}
|
|
45
|
+
/** pg_dump 的挂分区语句(dumpTableAttach):ALTER TABLE ONLY <父> ATTACH PARTITION <子> <边界>; */
|
|
46
|
+
export function parsePartitionAttachments(sql) {
|
|
47
|
+
const pattern = new RegExp(String.raw `^ALTER TABLE ONLY ${QUALIFIED} ATTACH PARTITION ${QUALIFIED} `, "gm");
|
|
48
|
+
return [...sql.matchAll(pattern)].map((match) => ({
|
|
49
|
+
parent: { schema: unquoteIdent(match[1]), name: unquoteIdent(match[2]) },
|
|
50
|
+
child: { schema: unquoteIdent(match[3]), name: unquoteIdent(match[4]) },
|
|
51
|
+
}));
|
|
52
|
+
}
|
|
53
|
+
/**
|
|
54
|
+
* pg_dump 的建表语句(dumpTableSchema)里每列的类型,按表(schema\0name)→ 列 → 类型。
|
|
55
|
+
* CREATE [UNLOGGED ]TABLE <表> (\n <列> <类型>[ COLLATE …][ DEFAULT …][ NOT NULL],\n CONSTRAINT …\n)…;
|
|
56
|
+
* 类型是 format_type 的输出,本身不含 COLLATE/DEFAULT/NOT NULL/GENERATED 这些词,读到它们为止。
|
|
57
|
+
* 带引号的列名可以跨行,所以按位置往下读,不按行切。
|
|
58
|
+
*/
|
|
59
|
+
export function parseTableColumns(sql) {
|
|
60
|
+
const tables = new Map();
|
|
61
|
+
const header = new RegExp(String.raw `^CREATE (?:UNLOGGED )?TABLE ${QUALIFIED} \(\n`, "gm");
|
|
62
|
+
for (const match of sql.matchAll(header)) {
|
|
63
|
+
const columns = new Map();
|
|
64
|
+
tables.set(`${unquoteIdent(match[1])}\u0000${unquoteIdent(match[2])}`, columns);
|
|
65
|
+
let at = match.index + match[0].length;
|
|
66
|
+
const column = new RegExp(String.raw ` {4}(${IDENT}) (.+?)(?= COLLATE | DEFAULT | NOT NULL| GENERATED |,?\n)`, "y");
|
|
67
|
+
while (at < sql.length && !sql.startsWith(")", at)) {
|
|
68
|
+
if (sql.startsWith(" CONSTRAINT ", at)) {
|
|
69
|
+
at = sql.indexOf("\n", at) + 1 || sql.length;
|
|
70
|
+
continue;
|
|
71
|
+
}
|
|
72
|
+
column.lastIndex = at;
|
|
73
|
+
const found = column.exec(sql);
|
|
74
|
+
if (!found)
|
|
75
|
+
break;
|
|
76
|
+
columns.set(unquoteIdent(found[1]), found[2]);
|
|
77
|
+
at = sql.indexOf("\n", column.lastIndex) + 1 || sql.length;
|
|
78
|
+
}
|
|
79
|
+
}
|
|
80
|
+
return tables;
|
|
81
|
+
}
|
|
82
|
+
/**
|
|
83
|
+
* 数据段的 COPY 头(dumpTableData_copy):"COPY <表> <列清单> FROM stdin;"。没有列的表列清单是空串,
|
|
84
|
+
* 所以那一行中间是两个空格,返回空列表。
|
|
85
|
+
*/
|
|
86
|
+
export function parseCopyHeader(line) {
|
|
87
|
+
const match = new RegExp(String.raw `^COPY ${QUALIFIED} (?:\(${COLUMNS}\))? FROM stdin;$`).exec(line);
|
|
88
|
+
if (!match)
|
|
89
|
+
return null;
|
|
90
|
+
return {
|
|
91
|
+
table: { schema: unquoteIdent(match[1]), name: unquoteIdent(match[2]) },
|
|
92
|
+
columns: match[3] ? splitColumns(match[3]) : [],
|
|
93
|
+
};
|
|
94
|
+
}
|
|
95
|
+
/**
|
|
96
|
+
* COPY 文本格式的一个字段 → 值(\N = NULL)。转义规则照 PostgreSQL 文档 "COPY / Text Format":
|
|
97
|
+
* \b \f \n \r \t \v \\,\ 加 1–3 位八进制,\x 加 1–2 位十六进制,其余 \c 就是 c 本身。
|
|
98
|
+
*/
|
|
99
|
+
export function decodeCopyField(field) {
|
|
100
|
+
if (field === "\\N")
|
|
101
|
+
return null;
|
|
102
|
+
if (!field.includes("\\"))
|
|
103
|
+
return field;
|
|
104
|
+
return field.replace(/\\(?:([0-7]{1,3})|x([0-9a-fA-F]{1,2})|(.))/g, (_, octal, hex, char) => {
|
|
105
|
+
if (octal)
|
|
106
|
+
return String.fromCharCode(parseInt(octal, 8));
|
|
107
|
+
if (hex)
|
|
108
|
+
return String.fromCharCode(parseInt(hex, 16));
|
|
109
|
+
const named = { b: "\b", f: "\f", n: "\n", r: "\r", t: "\t", v: "\v" };
|
|
110
|
+
return named[char] ?? char;
|
|
111
|
+
});
|
|
112
|
+
}
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
import type pg from "pg";
|
|
2
|
+
import { type ForeignKeyDef } from "./dump-sql.js";
|
|
3
|
+
export interface CheckedForeignKey {
|
|
4
|
+
fk: ForeignKeyDef;
|
|
5
|
+
/** 引用列全非空的行数(MATCH SIMPLE:有一列为空就不校验) */
|
|
6
|
+
rows: number;
|
|
7
|
+
/** 其中在目标库找不到被引用行的 */
|
|
8
|
+
missingRows: number;
|
|
9
|
+
}
|
|
10
|
+
export interface ExternalReferenceCheck {
|
|
11
|
+
checked: CheckedForeignKey[];
|
|
12
|
+
/** 恢复一定会失败的原因,一条一句 */
|
|
13
|
+
problems: string[];
|
|
14
|
+
/** 没能检查的,一条一句。恢复在一个事务里跑,这些真出问题也会整体回滚,所以只提示不拦 */
|
|
15
|
+
unchecked: string[];
|
|
16
|
+
/** 有外键要等扩展装上之后才查得了(见 pendingExtensions) */
|
|
17
|
+
waitingForExtensions: boolean;
|
|
18
|
+
}
|
|
19
|
+
/**
|
|
20
|
+
* 读 dump,找出指向 dumpedSchemas 之外的外键,逐条拿目标库对行。只读:目标库上只跑 select。
|
|
21
|
+
* pg_restore 起不来、归档读不了时抛错,由调用方当成 blocker。单条外键查不成(目标端报错、
|
|
22
|
+
* 数据块读不了)只影响它自己,记进 unchecked,不拖累别的外键的结论。
|
|
23
|
+
*/
|
|
24
|
+
export declare function checkExternalReferences(opts: {
|
|
25
|
+
dumpPath: string;
|
|
26
|
+
dumpedSchemas: string[];
|
|
27
|
+
targetUrl: string;
|
|
28
|
+
/**
|
|
29
|
+
* manifest 里记着、目标上还没装的扩展。有它们在,被引用的表不在、引用方列的类型不在,都可能是
|
|
30
|
+
* 扩展装上之后才有的(扩展可以往别的 schema 里建表,只看它登记的 schema 不够,交叉审查复现),
|
|
31
|
+
* 这时报"装完扩展再查",不当成 blocker;正式恢复装完扩展会再查一遍。
|
|
32
|
+
*/
|
|
33
|
+
pendingExtensions?: string[];
|
|
34
|
+
/**
|
|
35
|
+
* 调用方开着的事务(正式恢复里"先装扩展再复查"的那个):在它里面查,看得到还没提交的扩展。
|
|
36
|
+
* 每条查询包在一个保存点里、查完就回滚到保存点,检查本身留不下任何东西。
|
|
37
|
+
*/
|
|
38
|
+
inTransaction?: pg.Client;
|
|
39
|
+
}): Promise<ExternalReferenceCheck>;
|
|
40
|
+
export declare function describe(fk: ForeignKeyDef): string;
|