backupdrill 2.1.4 → 2.1.6

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,4 +1,4 @@
1
- # backupdrill
1
+ # backupdrill — free, open-source Supabase backup CLI
2
2
 
3
3
  Open-source CLI to back up a **Supabase** project to **your own** S3/R2/B2 bucket — the Postgres database and, once you configure its S3 keys, your Storage files — with a checksummed `manifest.json` so you can prove what was captured.
4
4
 
@@ -6,7 +6,7 @@ Backup is half the job. `backupdrill drill` restores your latest snapshot into a
6
6
 
7
7
  Backups stream through the CLI on your machine (or your CI) straight to your bucket. **Your data never touches our servers, because there are none — this is just a CLI.**
8
8
 
9
- > Want the same engine on a schedule, with drills that run themselves, alerts, and emailed reports? That's the hosted service → **[backupdrill.com](https://backupdrill.com)**. This CLI is the DIY layer; the cloud adds scheduling, alerting, and reports on top.
9
+ > This is the exact engine the hosted service runs in production. Want it on a schedule, with drills that run themselves, alerts, and emailed reports? That's **[backupdrill.com](https://backupdrill.com)**. This CLI is the DIY layer; the cloud adds scheduling, alerting, and reports on top.
10
10
 
11
11
  🎥 Prefer watching? [Where do your backups actually go — explained in about a minute](https://backupdrill.com/guides/test-supabase-backup-restore#explainer-video).
12
12
 
@@ -160,7 +160,7 @@ backupdrill drill --check-cmd "npm run smoke" # BACKUPDRILL_SANDBOX_URL points
160
160
 
161
161
  Your command runs after all structural checks pass, with `BACKUPDRILL_SANDBOX_URL` set to the sandbox's connection string. Exit 0 means pass; the result shows up in the report as an `app checks` line, and a non-zero exit fails the drill (the report keeps structural and app-level failures clearly separate). Entirely optional — without the flag, nothing changes and the report has no `app checks` line. There's a 10-minute timeout (the whole process tree is killed), and `--keep` leaves the sandbox container running when a drill fails so you can inspect it — the report JSON then includes a `keptSandbox` object with the container id and connection string (`docker rm -f <id>` when done).
162
162
 
163
- **What this can and cannot verify, honestly:** `BACKUPDRILL_SANDBOX_URL` connects as the sandbox **superuser**, and superusers bypass row-level security — plus policies referencing Supabase-managed roles (`authenticated`, `anon`) are skipped during restore, since those roles don't exist outside the platform. So use `--check-cmd` for data and business invariants ("orders reference existing users", "the row count I care about is sane"). It is **not** an RLS behavior test — verifying RLS takes an authenticated client stack with real JWTs, which no restore sandbox can fake generically.
163
+ **What this can and cannot verify, honestly:** `BACKUPDRILL_SANDBOX_URL` connects as the sandbox **superuser**, and superusers bypass row-level security. The sandbox does carry a stub of the Supabase runtime — an `auth` schema whose `uid()`/`jwt()`/`role()`/`email()` return null, and the `anon`/`authenticated`/`service_role` roles — so functions, column defaults and policies that reference them restore instead of being skipped. The sandbox deliberately does not create Supabase-managed *tables* such as `auth.users`: an index, constraint or policy that points at one is reported as a skipped managed object (these are restored in a second, post-data pass where that exemption applies), while a *view* over one is created in the strict first pass and fails the drill — a real gap in what a public-only backup can rebuild, shown rather than hidden. A function *body* that selects from `auth.users` is different: PostgreSQL does not validate string-literal bodies (`AS $$ … $$`) during a restore (`check_function_bodies` is off in every dump), so such a function is recreated and the drill passes — it would only fail when called. The exception is an SQL-standard `BEGIN ATOMIC … END` body, which is parsed and dependency-checked at creation regardless of that setting, so one of those referencing a managed table fails the drill like a view does. That is exactly what `--check-cmd` is for: if your application depends on such a function, call it there. So use `--check-cmd` for data and business invariants ("orders reference existing users", "the row count I care about is sane"). It is **not** an RLS behavior test — `auth.uid()` is a stub and you are a superuser; verifying RLS takes an authenticated client stack with real JWTs, which no restore sandbox can fake generically.
164
164
 
165
165
  When you actually need to recover, `restore` puts the database back into a fresh
166
166
  target project (through the same engine the drills use, with verification) and
package/dist/drill.js CHANGED
@@ -5,7 +5,7 @@ import { join } from "node:path";
5
5
  import { promisify } from "node:util";
6
6
  import { parseManifest } from "./manifest.js";
7
7
  import { targetClient, resolveSnapshot, getObjectText, downloadToFile, hashObject, } from "./snapshots.js";
8
- import { SANDBOX_MANAGED_ERROR, classifyBlocks, finalizePass, installExtensions, quoteIdent, restoreDatabaseArtifact, } from "./restore-engine.js";
8
+ import { SANDBOX_MANAGED_ERROR, classifyBlocks, finalizePass, installExtensions, quoteIdent, restoreDatabaseArtifact, installSandboxShim, } from "./restore-engine.js";
9
9
  import { connectPg } from "./supabase-ca.js";
10
10
  import { log } from "./log.js";
11
11
  const execFileAsync = promisify(execFile);
@@ -112,8 +112,8 @@ export function postDataResult(code, stderr) {
112
112
  * 的托管对象跳过如实返回。pre-data 的 schema 冲突跳过(dump 自带 CREATE SCHEMA,
113
113
  * 容器恒有 public)是非事件,不计入报告。
114
114
  */
115
- async function pgRestore(dumpPath, connString) {
116
- const result = await restoreDatabaseArtifact({ dumpPath, connString, target: "sandbox" });
115
+ async function pgRestore(dumpPath, connString, dumpedSchemas) {
116
+ const result = await restoreDatabaseArtifact({ dumpPath, connString, target: "sandbox", dumpedSchemas });
117
117
  if (result.preData.failures.length) {
118
118
  throw new Error(`pg_restore failed: ${result.preData.failures.join(" | ")}`);
119
119
  }
@@ -327,6 +327,12 @@ export async function drillDump(dumpPath, manifest, snapshot, preChecks = [], op
327
327
  // 旧 manifest(≤0.1.1)没有 extensions 字段 → 不装任何扩展,行为与从前一致
328
328
  const extensions = manifest.database.extensions ?? [];
329
329
  const unavailable = await installExtensions(pg.connString, extensions);
330
+ // Supabase 运行面的桩(auth.uid() 等 + 标准角色):没有它,参数/列默认值里引用
331
+ // auth.uid() 的函数与表在 pre-data 就建不出来 —— 见 installSandboxShim 头注。
332
+ // 转储自己带了 auth schema 就不建函数,否则 pg_restore 会撞 "already exists"。
333
+ await installSandboxShim(pg.connString, {
334
+ authFunctions: !manifest.database.schemas.includes("auth"),
335
+ });
330
336
  if (extensions.length) {
331
337
  // 装不上 ≠ 演练失败:扩展本体不在 public 转储里,没丢任何已备份的数据;
332
338
  // 若恢复真的需要它,下面的 pgRestore 会失败并给出分类错误
@@ -343,7 +349,7 @@ export async function drillDump(dumpPath, manifest, snapshot, preChecks = [], op
343
349
  log.step("Restoring into ephemeral Postgres…");
344
350
  let postData;
345
351
  try {
346
- postData = await pgRestore(dumpPath, pg.connString);
352
+ postData = await pgRestore(dumpPath, pg.connString, manifest.database.schemas);
347
353
  }
348
354
  catch (error) {
349
355
  // 沙箱装不上扩展 + 错误特征命中"缺类型/schema/扩展" → 假设式归因到沙箱环境
@@ -365,7 +371,7 @@ export async function drillDump(dumpPath, manifest, snapshot, preChecks = [], op
365
371
  detail: postData.failures.length > 0
366
372
  ? `${postData.failures.length} failed: ${postData.failures[0]}`
367
373
  : postData.supabaseSkipped > 0
368
- ? `user objects restored; ${postData.supabaseSkipped} Supabase-managed object(s) skipped (auth schema/roles do not exist in the drill sandbox)`
374
+ ? `user objects restored; ${postData.supabaseSkipped} Supabase-managed object(s) skipped (Supabase-managed tables such as auth.users do not exist in the drill sandbox)`
369
375
  : "all post-data objects restored",
370
376
  });
371
377
  const verify = await verifyRestored(pg.connString, manifest);
@@ -54,12 +54,34 @@ export declare function spawnPgRestore(bin: string, args: string[], extraEnv?: N
54
54
  code: number | null;
55
55
  stderr: string;
56
56
  }>;
57
+ /**
58
+ * 沙箱 allowlist,**按 manifest 实际转储的 schema 生成**(交叉审查 2026-09-11 第五轮):
59
+ * CLI 用户用 BACKUPDRILL_SCHEMAS=public,auth 把 auth 也转了,那 auth 里的对象就是**他自己的**
60
+ * 备份内容 —— 缺了是真失败,不能再当托管缺席豁免;否则一条依赖 auth.normalize_key() 的
61
+ * 唯一索引建不出来,演练照样 PASS。这是 installSandboxShim "转储带 auth 就不建桩"的对称
62
+ * 另一半:桩不建、豁免也不给,两边口径一致。角色豁免不受影响 —— 转储里从来没有 CREATE ROLE。
63
+ *
64
+ * **两端锚定**(第四轮,终结子串匹配这一整类):匹配的是 classifyBlocks 抽出来的**整条主消息**,
65
+ * 必须从头到尾就是一条"缺对象"诊断。不锚定的话用户行值能借道混进来 —— 表达式索引 `(col::uuid)`
66
+ * 撞上一行内容恰好是 `relation "auth.users" does not exist` 的数据,主消息就是
67
+ * `invalid input syntax for type uuid: "relation "auth.users" does not exist"`,子串一匹配,
68
+ * 一条建失败的唯一索引就被报成了预期跳过。旧的 \bauth\.uid\b / \bauth\.jwt\b 裸子串已删:
69
+ * 它们唯一对应的真实诊断 `function auth.uid() does not exist` 由 function 分支覆盖。
70
+ */
71
+ export declare function sandboxManagedError(dumpedSchemas?: string[]): RegExp;
72
+ /** 默认形态(什么托管 schema 都没转):兼容层与"只转 public"的托管 worker 路径。 */
57
73
  export declare const SANDBOX_MANAGED_ERROR: RegExp;
58
74
  export declare const SCHEMA_EXISTS_ERROR: RegExp;
59
75
  /**
60
76
  * 把 pg_restore 的 stderr 拆成单个错误块并按 allowlist 分类。
61
- * 只对 ERROR 原因行分类,不看 "Command was:" 之后的语句文本——否则一个恰好
62
- * 引用 auth.uid 的用户对象因"损坏/语法错误"失败时,会被误判成预期跳过。
77
+ * **只对 ERROR 那一行分类**:不看 "Command was:" 之后的语句文本,也不看 DETAIL /
78
+ * CONTEXT / LINE 行。前者的理由:一个恰好引用 auth.uid 的用户对象因"损坏/语法错误"
79
+ * 失败时,会被误判成预期跳过。后者(交叉审查 2026-09-11 复现):DETAIL 里带的是**行值**
80
+ * —— `Key (message)=(relation "auth.users" does not exist) already exists` 是用户的数据,
81
+ * 不是错误原因,拿它匹配等于让用户数据决定演练结论;一条失败的唯一索引就能这样混过去。
82
+ * Postgres 的主消息恒为单行,且在块的第一行(`pg_restore: error: ... ERROR: …`)。
83
+ * 抽出 `ERROR:` 之后的那段作为主消息,allowlist 对它**整条**匹配(两端锚定,见上);
84
+ * 没有 `ERROR:` 前缀的块(归档器致命错等)整行当主消息,锚定的 allowlist 不会认它 → 失败。
63
85
  */
64
86
  export declare function classifyBlocks(stderr: string, allow: RegExp): ClassifiedPass;
65
87
  /**
@@ -80,6 +102,37 @@ export declare function quoteIdent(name: string): string;
80
102
  * 本就装不上,public 转储通常不引用),真实目标应当阻断(用户能在 dashboard 开启)。
81
103
  */
82
104
  export declare function installExtensions(connString: string, extensions: ExtensionInfo[]): Promise<string[]>;
105
+ /**
106
+ * 沙箱专用:预置 Supabase 的最小运行面,让**引用**它的用户对象能建出来被验证,而不是
107
+ * 被当作"托管对象跳过"。只装桩,不装数据 —— 演练验证的是用户自己的 schema。
108
+ *
109
+ * 触因(2026-09-11,真实用户 105 表 12 GB 库的首次演练):`create function
110
+ * public.is_admin(p uuid default auth.uid())` —— 函数在 pre-data,参数默认值建函数时
111
+ * 就要解析,沙箱没有 auth schema 就整遍硬失败。同一根因还会吃掉更常见的
112
+ * `create table ... (created_by uuid default auth.uid())`:表建不出来、数据也没了,
113
+ * 演练只能报"缺表"。把 pre-data 的容错放宽不解决后者;补上运行面两者都解决。
114
+ *
115
+ * 桩的形状:
116
+ * auth.uid() → uuid、auth.jwt() → jsonb、auth.role() → text、auth.email() → text,
117
+ * 全部返回 null(恢复过程中不会真调它们:COPY 不求值列默认,策略/函数体不执行);
118
+ * 角色 anon / authenticated / service_role(nologin):RLS 策略的 `to authenticated`
119
+ * 要它们在场才能建。转储带 --no-privileges,不会有 GRANT 引用这些角色。
120
+ * 刻意不建 auth.users 之类的表:空的 auth.users 会让 FK 校验在 post-data 真失败,
121
+ * 比"缺表跳过"更糟。FK → auth.users 仍按 SANDBOX_MANAGED_ERROR 的 relation 形态跳过。
122
+ *
123
+ * 转储自带 auth schema 时(CLI 的 BACKUPDRILL_SCHEMAS=public,auth,自托管场景)**不建函数**:
124
+ * pg_restore 的 CREATE FUNCTION auth.uid() 会撞我们的桩报 "already exists",而那是 pre-data、
125
+ * 全严格 → 一次本来能过的演练被 shim 弄挂(交叉审查 2026-09-11)。角色照建:转储里
126
+ * 从来没有 CREATE ROLE。
127
+ *
128
+ * **绝不对真实 Supabase 目标调用**:那里 auth 是真的,create or replace 会覆盖平台函数。
129
+ */
130
+ export declare function sandboxShimSql(opts: {
131
+ authFunctions: boolean;
132
+ }): string;
133
+ export declare function installSandboxShim(connString: string, opts: {
134
+ authFunctions: boolean;
135
+ }): Promise<void>;
83
136
  /**
84
137
  * 两遍恢复(真实首演炸出的设计,现为 drill 与真实恢复共用):
85
138
  * 1. pre-data+data —— 表结构 + 数据。除 schema 冲突(预期)外任何错误都是硬失败。
@@ -110,4 +163,6 @@ export declare function restoreDatabaseArtifact(opts: {
110
163
  dumpPath: string;
111
164
  connString: string;
112
165
  target: RestoreTargetKind;
166
+ /** manifest.database.schemas:沙箱 allowlist 据此剔掉转储自带的托管 schema(见 sandboxManagedError)。 */
167
+ dumpedSchemas?: string[];
113
168
  }): Promise<EngineResult>;
@@ -171,18 +171,70 @@ export function spawnPgRestore(bin, args, extraEnv = {}, opts = { supabaseHost:
171
171
  proc.on("close", (code) => resolve({ code, stderr }));
172
172
  });
173
173
  }
174
- // 演练沙箱是裸 Postgres,Supabase 托管的 schema/角色必然缺席。post-data 里
175
- // 引用它们的失败是"环境预期",不是备份坏了;其余失败才是演练要抓的。
176
- // 真实 Supabase 目标不适用:角色/托管 schema 恒在,post-data 全严格。
177
- export const SANDBOX_MANAGED_ERROR = /schema "(auth|storage|realtime|vault|extensions|graphql[a-z_]*)" does not exist|role "(authenticated|anon|service_role|supabase_[a-z_]+)" does not exist|\bauth\.uid\b|\bauth\.jwt\b/i;
174
+ // 演练沙箱是裸 Postgres,Supabase 托管的 schema/表必然缺席。post-data 里引用它们的
175
+ // 失败是"环境预期",不是备份坏了;其余失败才是演练要抓的。
176
+ // 真实 Supabase 目标不适用:托管 schema/角色恒在,post-data 全严格。
177
+ //
178
+ // 沙箱经 installSandboxShim 预置了 auth schema 的桩函数与三个标准角色(2026-09-11),
179
+ // 所以缺席的形态从 schema/role 级变成了 relation/function 级(FK → auth.users、
180
+ // 调到没桩的 auth.xxx())。旧形态保留:manifest 更老的路径、或 shim 本身没装上时仍会出现。
181
+ // 托管 schema 两张清单。relation/function 级豁免**刻意不含 extensions**(交叉审查
182
+ // 2026-09-11 复现):那个 schema 装的是扩展对象,installExtensions 能装的都装了;里面还缺
183
+ // 函数/表 = 扩展没装上,或用户把自己的函数放进了 extensions —— 两种都该让演练失败并走
184
+ // "沙箱装不上扩展"的归因,吞成跳过就是给一份缺索引的备份盖"通过"章。schema 级形态保留
185
+ // (那是 schema 本身没建出来的老路径)。条目是正则片段(graphql[a-z_]* 覆盖 graphql_public)。
186
+ const MANAGED_SCHEMA_PATTERNS = ["auth", "storage", "realtime", "vault", "extensions", "graphql[a-z_]*"];
187
+ const MANAGED_OBJECT_SCHEMA_PATTERNS = ["auth", "storage", "realtime", "vault", "graphql[a-z_]*"];
188
+ /** 从托管清单里剔掉**这份转储自己带了**的 schema:带了就不再是"沙箱预期缺席"。 */
189
+ function notDumped(patterns, dumpedSchemas) {
190
+ return patterns.filter((pattern) => !dumpedSchemas.some((schema) => new RegExp(`^(?:${pattern})$`, "i").test(schema)));
191
+ }
192
+ /**
193
+ * 沙箱 allowlist,**按 manifest 实际转储的 schema 生成**(交叉审查 2026-09-11 第五轮):
194
+ * CLI 用户用 BACKUPDRILL_SCHEMAS=public,auth 把 auth 也转了,那 auth 里的对象就是**他自己的**
195
+ * 备份内容 —— 缺了是真失败,不能再当托管缺席豁免;否则一条依赖 auth.normalize_key() 的
196
+ * 唯一索引建不出来,演练照样 PASS。这是 installSandboxShim "转储带 auth 就不建桩"的对称
197
+ * 另一半:桩不建、豁免也不给,两边口径一致。角色豁免不受影响 —— 转储里从来没有 CREATE ROLE。
198
+ *
199
+ * **两端锚定**(第四轮,终结子串匹配这一整类):匹配的是 classifyBlocks 抽出来的**整条主消息**,
200
+ * 必须从头到尾就是一条"缺对象"诊断。不锚定的话用户行值能借道混进来 —— 表达式索引 `(col::uuid)`
201
+ * 撞上一行内容恰好是 `relation "auth.users" does not exist` 的数据,主消息就是
202
+ * `invalid input syntax for type uuid: "relation "auth.users" does not exist"`,子串一匹配,
203
+ * 一条建失败的唯一索引就被报成了预期跳过。旧的 \bauth\.uid\b / \bauth\.jwt\b 裸子串已删:
204
+ * 它们唯一对应的真实诊断 `function auth.uid() does not exist` 由 function 分支覆盖。
205
+ */
206
+ export function sandboxManagedError(dumpedSchemas = []) {
207
+ const schemas = notDumped(MANAGED_SCHEMA_PATTERNS, dumpedSchemas).join("|");
208
+ const objects = notDumped(MANAGED_OBJECT_SCHEMA_PATTERNS, dumpedSchemas).join("|");
209
+ const alternatives = [
210
+ ...(schemas ? [`schema "(${schemas})" does not exist`] : []),
211
+ ...(objects
212
+ ? [
213
+ `relation "(${objects})\\.[^"]+" does not exist`,
214
+ // 完整签名:auth.can_read(uuid, uuid) / auth.x(character varying) 都带空格
215
+ `function (${objects})\\.[^(]+\\([^)]*\\) does not exist`,
216
+ ]
217
+ : []),
218
+ `role "(authenticated|anon|service_role|supabase_[a-z_]+)" does not exist`,
219
+ ];
220
+ return new RegExp(`^(?:${alternatives.join("|")})$`, "i");
221
+ }
222
+ /** 默认形态(什么托管 schema 都没转):兼容层与"只转 public"的托管 worker 路径。 */
223
+ export const SANDBOX_MANAGED_ERROR = sandboxManagedError();
178
224
  // pre-data 唯一的预期冲突(见文件头注 3)。目标空门/新容器保证没有其他冲突源,
179
- // 任何别的 "already exists" 都是真冲突,必须失败。
180
- export const SCHEMA_EXISTS_ERROR = /schema "[^"]+" already exists/i;
225
+ // 任何别的 "already exists" 都是真冲突,必须失败。同样两端锚定。
226
+ export const SCHEMA_EXISTS_ERROR = /^schema "[^"]+" already exists$/i;
181
227
  const NEVER_MATCH = /(?!)/;
182
228
  /**
183
229
  * 把 pg_restore 的 stderr 拆成单个错误块并按 allowlist 分类。
184
- * 只对 ERROR 原因行分类,不看 "Command was:" 之后的语句文本——否则一个恰好
185
- * 引用 auth.uid 的用户对象因"损坏/语法错误"失败时,会被误判成预期跳过。
230
+ * **只对 ERROR 那一行分类**:不看 "Command was:" 之后的语句文本,也不看 DETAIL /
231
+ * CONTEXT / LINE 行。前者的理由:一个恰好引用 auth.uid 的用户对象因"损坏/语法错误"
232
+ * 失败时,会被误判成预期跳过。后者(交叉审查 2026-09-11 复现):DETAIL 里带的是**行值**
233
+ * —— `Key (message)=(relation "auth.users" does not exist) already exists` 是用户的数据,
234
+ * 不是错误原因,拿它匹配等于让用户数据决定演练结论;一条失败的唯一索引就能这样混过去。
235
+ * Postgres 的主消息恒为单行,且在块的第一行(`pg_restore: error: ... ERROR: …`)。
236
+ * 抽出 `ERROR:` 之后的那段作为主消息,allowlist 对它**整条**匹配(两端锚定,见上);
237
+ * 没有 `ERROR:` 前缀的块(归档器致命错等)整行当主消息,锚定的 allowlist 不会认它 → 失败。
186
238
  */
187
239
  export function classifyBlocks(stderr, allow) {
188
240
  const blocks = stderr
@@ -191,8 +243,9 @@ export function classifyBlocks(stderr, allow) {
191
243
  let expectedSkips = 0;
192
244
  const failures = [];
193
245
  for (const block of blocks) {
194
- const cause = block.split(/Command was:/i)[0];
195
- if (allow.test(cause)) {
246
+ const firstLine = block.split("\n")[0];
247
+ const message = firstLine.replace(/^.*?\bERROR:\s+/, "").trim();
248
+ if (allow.test(message)) {
196
249
  expectedSkips += 1;
197
250
  }
198
251
  else {
@@ -256,6 +309,61 @@ export async function installExtensions(connString, extensions) {
256
309
  }
257
310
  return unavailable;
258
311
  }
312
+ /**
313
+ * 沙箱专用:预置 Supabase 的最小运行面,让**引用**它的用户对象能建出来被验证,而不是
314
+ * 被当作"托管对象跳过"。只装桩,不装数据 —— 演练验证的是用户自己的 schema。
315
+ *
316
+ * 触因(2026-09-11,真实用户 105 表 12 GB 库的首次演练):`create function
317
+ * public.is_admin(p uuid default auth.uid())` —— 函数在 pre-data,参数默认值建函数时
318
+ * 就要解析,沙箱没有 auth schema 就整遍硬失败。同一根因还会吃掉更常见的
319
+ * `create table ... (created_by uuid default auth.uid())`:表建不出来、数据也没了,
320
+ * 演练只能报"缺表"。把 pre-data 的容错放宽不解决后者;补上运行面两者都解决。
321
+ *
322
+ * 桩的形状:
323
+ * auth.uid() → uuid、auth.jwt() → jsonb、auth.role() → text、auth.email() → text,
324
+ * 全部返回 null(恢复过程中不会真调它们:COPY 不求值列默认,策略/函数体不执行);
325
+ * 角色 anon / authenticated / service_role(nologin):RLS 策略的 `to authenticated`
326
+ * 要它们在场才能建。转储带 --no-privileges,不会有 GRANT 引用这些角色。
327
+ * 刻意不建 auth.users 之类的表:空的 auth.users 会让 FK 校验在 post-data 真失败,
328
+ * 比"缺表跳过"更糟。FK → auth.users 仍按 SANDBOX_MANAGED_ERROR 的 relation 形态跳过。
329
+ *
330
+ * 转储自带 auth schema 时(CLI 的 BACKUPDRILL_SCHEMAS=public,auth,自托管场景)**不建函数**:
331
+ * pg_restore 的 CREATE FUNCTION auth.uid() 会撞我们的桩报 "already exists",而那是 pre-data、
332
+ * 全严格 → 一次本来能过的演练被 shim 弄挂(交叉审查 2026-09-11)。角色照建:转储里
333
+ * 从来没有 CREATE ROLE。
334
+ *
335
+ * **绝不对真实 Supabase 目标调用**:那里 auth 是真的,create or replace 会覆盖平台函数。
336
+ */
337
+ export function sandboxShimSql(opts) {
338
+ const roles = `
339
+ do $$
340
+ declare r text;
341
+ begin
342
+ foreach r in array array['anon', 'authenticated', 'service_role'] loop
343
+ if not exists (select 1 from pg_roles where rolname = r) then
344
+ execute format('create role %I nologin', r);
345
+ end if;
346
+ end loop;
347
+ end $$;`;
348
+ if (!opts.authFunctions)
349
+ return roles;
350
+ return `
351
+ create schema if not exists auth;
352
+ create or replace function auth.uid() returns uuid language sql stable as $$ select null::uuid $$;
353
+ create or replace function auth.jwt() returns jsonb language sql stable as $$ select null::jsonb $$;
354
+ create or replace function auth.role() returns text language sql stable as $$ select null::text $$;
355
+ create or replace function auth.email() returns text language sql stable as $$ select null::text $$;
356
+ ${roles}`;
357
+ }
358
+ export async function installSandboxShim(connString, opts) {
359
+ const client = await connectPg(connString);
360
+ try {
361
+ await client.query(sandboxShimSql(opts));
362
+ }
363
+ finally {
364
+ await client.end();
365
+ }
366
+ }
259
367
  /**
260
368
  * 两遍恢复(真实首演炸出的设计,现为 drill 与真实恢复共用):
261
369
  * 1. pre-data+data —— 表结构 + 数据。除 schema 冲突(预期)外任何错误都是硬失败。
@@ -415,6 +523,6 @@ export async function restoreDatabaseArtifact(opts) {
415
523
  return { preData, postData: { expectedSkips: 0, failures: [] }, ok: false };
416
524
  }
417
525
  const second = await spawnPgRestore(bin, ["--section=post-data", ...common], env, childOpts);
418
- const postData = finalizePass(second.code, second.stderr, classifyBlocks(second.stderr, opts.target === "sandbox" ? SANDBOX_MANAGED_ERROR : NEVER_MATCH));
526
+ const postData = finalizePass(second.code, second.stderr, classifyBlocks(second.stderr, opts.target === "sandbox" ? sandboxManagedError(opts.dumpedSchemas ?? []) : NEVER_MATCH));
419
527
  return { preData, postData, ok: postData.failures.length === 0 };
420
528
  }
@@ -6,7 +6,37 @@ export declare function targetClient(config: BackupConfig): S3Client;
6
6
  export declare function resolveSnapshot(s3: S3Client, config: BackupConfig, explicit?: string): Promise<string>;
7
7
  export declare function getObjectText(s3: S3Client, bucket: string, key: string): Promise<string>;
8
8
  /** 下载对象到本地文件,顺带算 sha256(验证读回的归档没被损坏)。 */
9
- export declare function downloadToFile(s3: S3Client, bucket: string, key: string, dest: string): Promise<{
9
+ export declare const DOWNLOAD_MAX_ATTEMPTS = 8;
10
+ /** 最小的文件句柄接口:downloadToFile 只用这两样。测试可以注入假句柄(短写、慢写)。 */
11
+ export interface WritableHandle {
12
+ write(buffer: Buffer, offset: number, length: number, position: number): Promise<{
13
+ bytesWritten: number;
14
+ }>;
15
+ close(): Promise<void>;
16
+ }
17
+ /**
18
+ * 把一整块写进 position 起的位置,**写满为止**。Node 的 `FileHandle.write` 合约允许一次只写
19
+ * 一部分(返回 bytesWritten),按整块计数会让校验和给一份少了字节的文件盖章(交叉审查
20
+ * 2026-09-12 故障注入:写进 3 字节、报了 8 字节、校验和照样"通过")。
21
+ * 返回真正落盘的字节数;中途抛错时把已确认的字节数挂在错误上,调用方据此对齐哈希与起点。
22
+ */
23
+ export declare function writeFully(fh: WritableHandle, chunk: Buffer, position: number): Promise<number>;
24
+ /**
25
+ * 断点续传下载(2026-09-12):一个 12 GB 的 dump 走一条 HTTPS 流要 50 分钟,对端(用户的桶)
26
+ * 在第 10 分钟关掉连接,整次演练就没了 —— 而且此前没有任何续传:一次事故、两封 FAILED
27
+ * 邮件。这里按 HTTP Range 续传,三条铁律:
28
+ * 1. 续传起点 = **已确认写进 fd 的字节数**,不是流过管道的字节数。pipeline 出错时 Writable
29
+ * 缓冲区里的块会丢,按"流过多少"续传会在文件里留下一个洞;所以自写 Writable,
30
+ * `fh.write` 返回之后才计数、才哈希,哈希与磁盘内容永远一致;
31
+ * 2. 续传请求必须拿到 206:对端不认 Range 就会把整个对象再发一遍,追加上去就是脏文件,
32
+ * 这种情况直接失败,绝不"看起来成功";
33
+ * 3. ETag 变了 = 对象在下载中途被替换,两半拼不成一份,直接失败。
34
+ * 重试 8 次,退避 1s→30s;每次续传写一行日志(worker 的 journal 能看到断在哪、续在哪)。
35
+ */
36
+ export declare function downloadToFile(s3: S3Client, bucket: string, key: string, dest: string, opts?: {
37
+ maxAttempts?: number;
38
+ openFile?: (path: string) => Promise<WritableHandle>;
39
+ }): Promise<{
10
40
  bytes: number;
11
41
  sha256: string;
12
42
  }>;
package/dist/snapshots.js CHANGED
@@ -1,7 +1,8 @@
1
1
  import { createHash } from "node:crypto";
2
- import { createWriteStream } from "node:fs";
2
+ import { open } from "node:fs/promises";
3
3
  import { pipeline } from "node:stream/promises";
4
- import { Transform } from "node:stream";
4
+ import { Writable } from "node:stream";
5
+ import { log } from "./log.js";
5
6
  import { S3Client, ListObjectsV2Command, GetObjectCommand, } from "@aws-sdk/client-s3";
6
7
  /** 读备份桶(用户自己的桶)的 S3 客户端。drill/restore 共用。 */
7
8
  export function targetClient(config) {
@@ -46,19 +47,150 @@ export async function getObjectText(s3, bucket, key) {
46
47
  return res.Body.transformToString();
47
48
  }
48
49
  /** 下载对象到本地文件,顺带算 sha256(验证读回的归档没被损坏)。 */
49
- export async function downloadToFile(s3, bucket, key, dest) {
50
- const res = await s3.send(new GetObjectCommand({ Bucket: bucket, Key: key }));
50
+ export const DOWNLOAD_MAX_ATTEMPTS = 8;
51
+ const DOWNLOAD_BACKOFF_MS = 1000;
52
+ const DOWNLOAD_BACKOFF_CAP_MS = 30_000;
53
+ /**
54
+ * 下载流半路断掉时值不值得从断点续传。SDK 自己的重试只管"请求建立"那一下;
55
+ * `send()` 一旦返回、body 流开始,后面任何断连都直接落到 pipeline 的 reject 上。
56
+ * 对端主动关连接(Node 报 "aborted")、连接重置、超时、5xx/429 都是重试对象;
57
+ * 4xx(404 NoSuchKey、416 越界、403)是事实,重试只会得到同样的答案。
58
+ */
59
+ function isRetryableDownloadError(error) {
60
+ const e = error;
61
+ const status = e.$metadata?.httpStatusCode;
62
+ if (status !== undefined)
63
+ return status >= 500 || status === 429;
64
+ if (["ECONNRESET", "ETIMEDOUT", "EPIPE", "ECONNREFUSED", "EAI_AGAIN"].includes(e.code ?? "")) {
65
+ return true;
66
+ }
67
+ return /aborted|socket hang up|premature close|short body/i.test(e.message ?? "");
68
+ }
69
+ class NonRetryableDownloadError extends Error {
70
+ }
71
+ /**
72
+ * 把一整块写进 position 起的位置,**写满为止**。Node 的 `FileHandle.write` 合约允许一次只写
73
+ * 一部分(返回 bytesWritten),按整块计数会让校验和给一份少了字节的文件盖章(交叉审查
74
+ * 2026-09-12 故障注入:写进 3 字节、报了 8 字节、校验和照样"通过")。
75
+ * 返回真正落盘的字节数;中途抛错时把已确认的字节数挂在错误上,调用方据此对齐哈希与起点。
76
+ */
77
+ export async function writeFully(fh, chunk, position) {
78
+ let done = 0;
79
+ while (done < chunk.length) {
80
+ let bytesWritten;
81
+ try {
82
+ ({ bytesWritten } = await fh.write(chunk, done, chunk.length - done, position + done));
83
+ }
84
+ catch (error) {
85
+ throw Object.assign(error, { confirmedBytes: done });
86
+ }
87
+ if (bytesWritten <= 0) {
88
+ throw Object.assign(new Error(`write returned ${bytesWritten} bytes at position ${position + done}`), {
89
+ confirmedBytes: done,
90
+ });
91
+ }
92
+ done += bytesWritten;
93
+ }
94
+ return done;
95
+ }
96
+ /**
97
+ * 断点续传下载(2026-09-12):一个 12 GB 的 dump 走一条 HTTPS 流要 50 分钟,对端(用户的桶)
98
+ * 在第 10 分钟关掉连接,整次演练就没了 —— 而且此前没有任何续传:一次事故、两封 FAILED
99
+ * 邮件。这里按 HTTP Range 续传,三条铁律:
100
+ * 1. 续传起点 = **已确认写进 fd 的字节数**,不是流过管道的字节数。pipeline 出错时 Writable
101
+ * 缓冲区里的块会丢,按"流过多少"续传会在文件里留下一个洞;所以自写 Writable,
102
+ * `fh.write` 返回之后才计数、才哈希,哈希与磁盘内容永远一致;
103
+ * 2. 续传请求必须拿到 206:对端不认 Range 就会把整个对象再发一遍,追加上去就是脏文件,
104
+ * 这种情况直接失败,绝不"看起来成功";
105
+ * 3. ETag 变了 = 对象在下载中途被替换,两半拼不成一份,直接失败。
106
+ * 重试 8 次,退避 1s→30s;每次续传写一行日志(worker 的 journal 能看到断在哪、续在哪)。
107
+ */
108
+ export async function downloadToFile(s3, bucket, key, dest, opts = {}) {
109
+ const maxAttempts = opts.maxAttempts ?? DOWNLOAD_MAX_ATTEMPTS;
51
110
  const hash = createHash("sha256");
52
- let bytes = 0;
53
- const hasher = new Transform({
54
- transform(chunk, _enc, cb) {
55
- hash.update(chunk);
56
- bytes += chunk.length;
57
- cb(null, chunk);
58
- },
59
- });
60
- await pipeline(res.Body, hasher, createWriteStream(dest));
61
- return { bytes, sha256: hash.digest("hex") };
111
+ const openFile = opts.openFile ?? ((path) => open(path, "w"));
112
+ // 目标文件推迟到首个响应到手后再打开(交叉审查 2026-09-12):"w" 会先截断,404/403 这种
113
+ // 请求都没成的失败不该顺手毁掉调用方路径上已有的文件。
114
+ let fh = null;
115
+ let written = 0; // 已落盘 = 已哈希 = 续传起点
116
+ let total = null;
117
+ let etag;
118
+ // 上一块的写还在途时 pipeline 就可能 reject(源先断)。重试前必须等它落定,否则 Range 用的
119
+ // 是过期的 written,写完成后 written 又往前走,续传起点与磁盘错位(交叉审查 2026-09-12)。
120
+ let inFlight = Promise.resolve();
121
+ try {
122
+ for (let attempt = 1;; attempt++) {
123
+ await inFlight.catch(() => { });
124
+ const from = written; // 本次请求的起点在发请求前冻结,后面的判断都用它
125
+ try {
126
+ const res = await s3.send(new GetObjectCommand({
127
+ Bucket: bucket,
128
+ Key: key,
129
+ ...(from > 0 ? { Range: `bytes=${from}-` } : {}),
130
+ }));
131
+ const body = res.Body;
132
+ const status = res.$metadata.httpStatusCode;
133
+ if (from > 0 && status !== 206) {
134
+ body.destroy();
135
+ throw new NonRetryableDownloadError(`resume rejected: expected 206 Partial Content from byte ${from}, got ${status} — refusing to append`);
136
+ }
137
+ if (etag === undefined) {
138
+ etag = res.ETag;
139
+ }
140
+ else if (res.ETag !== undefined && res.ETag !== etag) {
141
+ body.destroy();
142
+ throw new NonRetryableDownloadError(`object changed during download (ETag ${etag} → ${res.ETag}) — halves would not match`);
143
+ }
144
+ if (total === null && typeof res.ContentLength === "number")
145
+ total = res.ContentLength;
146
+ if (fh === null)
147
+ fh = await openFile(dest);
148
+ const handle = fh;
149
+ const sink = new Writable({
150
+ write(chunk, _enc, cb) {
151
+ inFlight = writeFully(handle, chunk, written).then((n) => {
152
+ hash.update(chunk.subarray(0, n));
153
+ written += n;
154
+ cb();
155
+ }, (err) => {
156
+ // 半块落盘也要如实计入:哈希与起点永远等于磁盘上真有的字节
157
+ const n = err.confirmedBytes ?? 0;
158
+ if (n > 0) {
159
+ hash.update(chunk.subarray(0, n));
160
+ written += n;
161
+ }
162
+ cb(err);
163
+ });
164
+ },
165
+ });
166
+ await pipeline(body, sink);
167
+ await inFlight.catch(() => { });
168
+ if (total !== null && written !== total) {
169
+ // 流"正常"结束但字节不够:对端掐断时 Node 有时不报错只是提前 end
170
+ throw new Error(`short body: got ${written} of ${total} bytes`);
171
+ }
172
+ return { bytes: written, sha256: hash.digest("hex") };
173
+ }
174
+ catch (error) {
175
+ if (error instanceof NonRetryableDownloadError ||
176
+ !isRetryableDownloadError(error) ||
177
+ attempt >= maxAttempts) {
178
+ throw error;
179
+ }
180
+ const wait = Math.min(DOWNLOAD_BACKOFF_MS * 2 ** (attempt - 1), DOWNLOAD_BACKOFF_CAP_MS);
181
+ const reason = (error.message ?? String(error)).split("\n")[0];
182
+ log.step(`Download interrupted at ${(written / 1048576).toFixed(0)} MB (${reason}); ` +
183
+ `resuming from byte ${written} in ${wait / 1000}s (attempt ${attempt + 1}/${maxAttempts})…`);
184
+ await new Promise((r) => setTimeout(r, wait));
185
+ }
186
+ }
187
+ }
188
+ finally {
189
+ // 终态失败时上一块的写可能还在途:先让它落定再关句柄,别让慢写撞上已关闭的 fd
190
+ await inFlight.catch(() => { });
191
+ if (fh !== null)
192
+ await fh.close();
193
+ }
62
194
  }
63
195
  /** 只算对象校验和、不落盘(Storage 文件完整性校验用,省内存/磁盘)。 */
64
196
  export async function hashObject(s3, bucket, key) {
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "backupdrill",
3
- "version": "2.1.4",
3
+ "version": "2.1.6",
4
4
  "description": "Open-source CLI to back up a Supabase project — Postgres database and Storage files — to your own S3/R2/B2 bucket, with a checksummed manifest.",
5
5
  "type": "module",
6
6
  "bin": {